2024-09-21T15:38:54.645 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-21T15:38:54.667 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-21T15:38:54.752 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi/7909998 branch: reef description: rbd/thrash/{base/install clusters/{fixed-2 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/rbd_fsx_rate_limit} email: ceph-qa@ceph.com first_in_suite: false job_id: '7909998' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 - machine: cpus: 1 disk: 40 ram: 8000 volumes: count: 4 size: 30 os_type: centos os_version: 9.stream overrides: admin_socket: branch: reef ceph: conf: client: rbd qos iops burst: 100 rbd qos iops limit: 50 rbd qos schedule tick min: 100 global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from sha1: 19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: reef sha1: a477b1894d5c5a613f23260ca0e6b96d79652ba7 owner: scheduled_teuthology@teuthology priority: 930 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 4623 sha1: 19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 sleep_before_teardown: 0 subset: 80/128 suite: rbd suite_branch: reef suite_path: /home/teuthworker/src/git.ceph.com_ceph_a477b1894d5c5a613f23260ca0e6b96d79652ba7/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: a477b1894d5c5a613f23260ca0e6b96d79652ba7 targets: smithi089.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL48+d9q3iRAlAALR56tg45jkIWRKg3kxeLC7ODFrnc0wrGWBhJ3sdf2RloSTvD2MYJZdpXqS52grJZ+bX7OOY0= smithi100.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNsIHVy2wcThj5nHSqnIlV7S/E775ijm/9edjgh4SScgmVCAEGeOpWl4D118mk+UYWyL14cxEgdrN64tRvQqx8c= tasks: - install: null - ceph: null - thrashosds: timeout: 1200 - rbd_fsx: clients: - client.0 ops: 6000 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-17_22:16:03 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2902 2024-09-21T15:38:54.752 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_a477b1894d5c5a613f23260ca0e6b96d79652ba7/qa; will attempt to use it 2024-09-21T15:38:54.753 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_a477b1894d5c5a613f23260ca0e6b96d79652ba7/qa/tasks 2024-09-21T15:38:54.753 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-21T15:38:54.754 INFO:teuthology.task.internal:Checking packages... 2024-09-21T15:38:54.776 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '19e2a2beb6f41a3ab62650e38614eb29b59dcbb9' 2024-09-21T15:38:54.776 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-21T15:38:54.776 INFO:teuthology.packaging:ref: None 2024-09-21T15:38:54.776 INFO:teuthology.packaging:tag: None 2024-09-21T15:38:54.776 INFO:teuthology.packaging:branch: reef 2024-09-21T15:38:54.776 INFO:teuthology.packaging:sha1: 19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-21T15:38:54.776 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=reef 2024-09-21T15:38:55.043 INFO:teuthology.task.internal:Found packages for ceph version 18.2.4-871.g6a1bdf85 2024-09-21T15:38:55.044 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-21T15:38:55.051 INFO:teuthology.task.internal:no buildpackages task found 2024-09-21T15:38:55.051 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-21T15:38:55.059 INFO:teuthology.task.internal:Saving configuration 2024-09-21T15:38:55.071 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-21T15:38:55.078 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-21T15:38:55.104 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi089.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi/7909998', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 15:33:58.382147', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL48+d9q3iRAlAALR56tg45jkIWRKg3kxeLC7ODFrnc0wrGWBhJ3sdf2RloSTvD2MYJZdpXqS52grJZ+bX7OOY0='} 2024-09-21T15:38:55.129 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi100.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi/7909998', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 15:33:58.380447', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNsIHVy2wcThj5nHSqnIlV7S/E775ijm/9edjgh4SScgmVCAEGeOpWl4D118mk+UYWyL14cxEgdrN64tRvQqx8c='} 2024-09-21T15:38:55.129 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-21T15:38:55.136 INFO:teuthology.task.internal:roles: ubuntu@smithi089.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-09-21T15:38:55.136 INFO:teuthology.task.internal:roles: ubuntu@smithi100.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-09-21T15:38:55.136 INFO:teuthology.run_tasks:Running task console_log... 2024-09-21T15:38:55.191 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f0ba60d3a30>, signals=[15]) 2024-09-21T15:38:55.191 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-21T15:38:55.197 INFO:teuthology.task.internal:Opening connections... 2024-09-21T15:38:55.197 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi089.front.sepia.ceph.com 2024-09-21T15:38:55.199 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T15:38:55.277 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi100.front.sepia.ceph.com 2024-09-21T15:38:55.278 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T15:38:55.350 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-21T15:38:55.358 DEBUG:teuthology.orchestra.run.smithi089:> uname -m 2024-09-21T15:38:55.376 INFO:teuthology.orchestra.run.smithi089.stdout:x86_64 2024-09-21T15:38:55.376 DEBUG:teuthology.orchestra.run.smithi089:> cat /etc/os-release 2024-09-21T15:38:55.434 INFO:teuthology.orchestra.run.smithi089.stdout:NAME="CentOS Stream" 2024-09-21T15:38:55.434 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION="9" 2024-09-21T15:38:55.435 INFO:teuthology.orchestra.run.smithi089.stdout:ID="centos" 2024-09-21T15:38:55.435 INFO:teuthology.orchestra.run.smithi089.stdout:ID_LIKE="rhel fedora" 2024-09-21T15:38:55.435 INFO:teuthology.orchestra.run.smithi089.stdout:VERSION_ID="9" 2024-09-21T15:38:55.435 INFO:teuthology.orchestra.run.smithi089.stdout:PLATFORM_ID="platform:el9" 2024-09-21T15:38:55.435 INFO:teuthology.orchestra.run.smithi089.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-21T15:38:55.435 INFO:teuthology.orchestra.run.smithi089.stdout:ANSI_COLOR="0;31" 2024-09-21T15:38:55.435 INFO:teuthology.orchestra.run.smithi089.stdout:LOGO="fedora-logo-icon" 2024-09-21T15:38:55.435 INFO:teuthology.orchestra.run.smithi089.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-21T15:38:55.435 INFO:teuthology.orchestra.run.smithi089.stdout:HOME_URL="https://centos.org/" 2024-09-21T15:38:55.435 INFO:teuthology.orchestra.run.smithi089.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-21T15:38:55.435 INFO:teuthology.orchestra.run.smithi089.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-21T15:38:55.435 INFO:teuthology.orchestra.run.smithi089.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-21T15:38:55.436 INFO:teuthology.lock.ops:Updating smithi089.front.sepia.ceph.com on lock server 2024-09-21T15:38:55.455 DEBUG:teuthology.orchestra.run.smithi100:> uname -m 2024-09-21T15:38:55.474 INFO:teuthology.orchestra.run.smithi100.stdout:x86_64 2024-09-21T15:38:55.474 DEBUG:teuthology.orchestra.run.smithi100:> cat /etc/os-release 2024-09-21T15:38:55.532 INFO:teuthology.orchestra.run.smithi100.stdout:NAME="CentOS Stream" 2024-09-21T15:38:55.532 INFO:teuthology.orchestra.run.smithi100.stdout:VERSION="9" 2024-09-21T15:38:55.532 INFO:teuthology.orchestra.run.smithi100.stdout:ID="centos" 2024-09-21T15:38:55.532 INFO:teuthology.orchestra.run.smithi100.stdout:ID_LIKE="rhel fedora" 2024-09-21T15:38:55.532 INFO:teuthology.orchestra.run.smithi100.stdout:VERSION_ID="9" 2024-09-21T15:38:55.532 INFO:teuthology.orchestra.run.smithi100.stdout:PLATFORM_ID="platform:el9" 2024-09-21T15:38:55.532 INFO:teuthology.orchestra.run.smithi100.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-21T15:38:55.532 INFO:teuthology.orchestra.run.smithi100.stdout:ANSI_COLOR="0;31" 2024-09-21T15:38:55.532 INFO:teuthology.orchestra.run.smithi100.stdout:LOGO="fedora-logo-icon" 2024-09-21T15:38:55.532 INFO:teuthology.orchestra.run.smithi100.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-21T15:38:55.532 INFO:teuthology.orchestra.run.smithi100.stdout:HOME_URL="https://centos.org/" 2024-09-21T15:38:55.532 INFO:teuthology.orchestra.run.smithi100.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-21T15:38:55.533 INFO:teuthology.orchestra.run.smithi100.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-21T15:38:55.533 INFO:teuthology.orchestra.run.smithi100.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-21T15:38:55.533 INFO:teuthology.lock.ops:Updating smithi100.front.sepia.ceph.com on lock server 2024-09-21T15:38:55.555 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-21T15:38:55.564 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-21T15:38:55.569 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-21T15:38:55.569 DEBUG:teuthology.orchestra.run.smithi089:> test '!' -e /home/ubuntu/cephtest 2024-09-21T15:38:55.571 DEBUG:teuthology.orchestra.run.smithi100:> test '!' -e /home/ubuntu/cephtest 2024-09-21T15:38:55.588 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-21T15:38:55.595 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-21T15:38:55.595 DEBUG:teuthology.orchestra.run.smithi089:> test -z $(ls -A /var/lib/ceph) 2024-09-21T15:38:55.628 DEBUG:teuthology.orchestra.run.smithi100:> test -z $(ls -A /var/lib/ceph) 2024-09-21T15:38:55.665 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-21T15:38:55.718 INFO:teuthology.run_tasks:Running task kernel... 2024-09-21T15:38:55.739 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-21T15:38:55.739 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-21T15:38:55.739 DEBUG:teuthology.orchestra.run.smithi089:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T15:38:55.740 DEBUG:teuthology.orchestra.run.smithi100:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T15:38:55.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T15:38:55.756 DEBUG:teuthology.orchestra.run.smithi100:> uname -r 2024-09-21T15:38:55.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T15:38:55.757 DEBUG:teuthology.orchestra.run.smithi089:> uname -r 2024-09-21T15:38:55.814 INFO:teuthology.orchestra.run.smithi100.stdout:5.14.0-503.el9.x86_64 2024-09-21T15:38:55.814 INFO:teuthology.task.kernel:Running kernel on smithi100: 5.14.0-503.el9.x86_64 2024-09-21T15:38:55.814 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum install -y kernel 2024-09-21T15:38:55.815 INFO:teuthology.orchestra.run.smithi089.stdout:5.14.0-503.el9.x86_64 2024-09-21T15:38:55.815 INFO:teuthology.task.kernel:Running kernel on smithi089: 5.14.0-503.el9.x86_64 2024-09-21T15:38:55.815 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum install -y kernel 2024-09-21T15:38:58.060 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.3 MB 00:00 2024-09-21T15:38:59.488 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - BaseOS 4.3 MB/s | 8.3 MB 00:01 2024-09-21T15:38:59.886 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - AppStream 27 MB/s | 20 MB 00:00 2024-09-21T15:39:03.456 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - AppStream 7.0 MB/s | 20 MB 00:02 2024-09-21T15:39:04.803 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - CRB 5.6 MB/s | 6.5 MB 00:01 2024-09-21T15:39:06.320 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - Extras packages 64 kB/s | 19 kB 00:00 2024-09-21T15:39:07.224 INFO:teuthology.orchestra.run.smithi100.stdout:Extra Packages for Enterprise Linux 29 MB/s | 23 MB 00:00 2024-09-21T15:39:07.656 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.5 MB 00:00 2024-09-21T15:39:09.224 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - Extras packages 54 kB/s | 19 kB 00:00 2024-09-21T15:39:10.120 INFO:teuthology.orchestra.run.smithi089.stdout:Extra Packages for Enterprise Linux 29 MB/s | 23 MB 00:00 2024-09-21T15:39:12.605 INFO:teuthology.orchestra.run.smithi100.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-09-21T15:39:14.545 INFO:teuthology.orchestra.run.smithi100.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-21T15:39:14.545 INFO:teuthology.orchestra.run.smithi100.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-21T15:39:14.651 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-21T15:39:14.657 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-21T15:39:14.657 INFO:teuthology.orchestra.run.smithi100.stdout: Package Architecture Version Repository Size 2024-09-21T15:39:14.657 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-21T15:39:14.658 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-21T15:39:14.658 INFO:teuthology.orchestra.run.smithi100.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-21T15:39:14.658 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-09-21T15:39:14.658 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-21T15:39:14.658 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-21T15:39:14.658 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-21T15:39:14.658 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:39:14.658 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-21T15:39:14.658 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-21T15:39:14.658 INFO:teuthology.orchestra.run.smithi100.stdout:Install 4 Packages 2024-09-21T15:39:14.658 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:39:14.659 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 79 M 2024-09-21T15:39:14.659 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 126 M 2024-09-21T15:39:14.659 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-21T15:39:14.907 INFO:teuthology.orchestra.run.smithi100.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 272 kB/s | 36 kB 00:00 2024-09-21T15:39:15.625 INFO:teuthology.orchestra.run.smithi089.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-21T15:39:17.585 INFO:teuthology.orchestra.run.smithi089.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-21T15:39:17.585 INFO:teuthology.orchestra.run.smithi089.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-21T15:39:17.684 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-21T15:39:17.693 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-21T15:39:17.693 INFO:teuthology.orchestra.run.smithi089.stdout: Package Architecture Version Repository Size 2024-09-21T15:39:17.694 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-21T15:39:17.694 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-09-21T15:39:17.694 INFO:teuthology.orchestra.run.smithi089.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-21T15:39:17.694 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-09-21T15:39:17.694 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-21T15:39:17.694 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-21T15:39:17.694 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-21T15:39:17.694 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:39:17.694 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-09-21T15:39:17.694 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-21T15:39:17.694 INFO:teuthology.orchestra.run.smithi089.stdout:Install 4 Packages 2024-09-21T15:39:17.694 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:39:17.695 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 79 M 2024-09-21T15:39:17.695 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 126 M 2024-09-21T15:39:17.695 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-09-21T15:39:17.860 INFO:teuthology.orchestra.run.smithi089.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 298 kB/s | 36 kB 00:00 2024-09-21T15:39:19.011 INFO:teuthology.orchestra.run.smithi089.stdout:(2/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 27 MB/s | 35 MB 00:01 2024-09-21T15:39:19.090 INFO:teuthology.orchestra.run.smithi100.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 3.6 MB/s | 16 MB 00:04 2024-09-21T15:39:19.161 INFO:teuthology.orchestra.run.smithi089.stdout:(3/4): kernel-core-5.14.0-508.el9.x86_64.rpm 11 MB/s | 16 MB 00:01 2024-09-21T15:39:19.469 INFO:teuthology.orchestra.run.smithi089.stdout:(4/4): kernel-modules-core-5.14.0-508.el9.x86_6 18 MB/s | 29 MB 00:01 2024-09-21T15:39:19.470 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-09-21T15:39:19.470 INFO:teuthology.orchestra.run.smithi089.stdout:Total 44 MB/s | 79 MB 00:01 2024-09-21T15:39:20.150 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-09-21T15:39:20.499 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-09-21T15:39:20.499 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-09-21T15:39:21.465 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-09-21T15:39:21.466 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-09-21T15:39:21.923 INFO:teuthology.orchestra.run.smithi100.stdout:(3/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 4.8 MB/s | 35 MB 00:07 2024-09-21T15:39:22.715 INFO:teuthology.orchestra.run.smithi100.stdout:(4/4): kernel-modules-core-5.14.0-508.el9.x86_6 3.7 MB/s | 29 MB 00:07 2024-09-21T15:39:22.715 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-21T15:39:22.715 INFO:teuthology.orchestra.run.smithi100.stdout:Total 9.8 MB/s | 79 MB 00:08 2024-09-21T15:39:23.402 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-21T15:39:23.768 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-21T15:39:23.768 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-21T15:39:23.902 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-09-21T15:39:24.545 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-21T15:39:24.636 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T15:39:24.719 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-21T15:39:24.720 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-21T15:39:27.171 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-21T15:39:27.394 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T15:39:27.541 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T15:39:27.807 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-21T15:39:27.924 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T15:39:30.691 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T15:39:30.845 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T15:39:32.355 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T15:39:32.391 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T15:39:35.720 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T15:39:35.812 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T15:39:37.226 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T15:39:40.658 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T15:40:15.552 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T15:40:15.597 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-21T15:40:15.946 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T15:40:15.946 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-21T15:40:15.946 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T15:40:15.946 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T15:40:16.738 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T15:40:16.738 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:40:16.738 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-09-21T15:40:16.738 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-21T15:40:16.738 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-21T15:40:16.738 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-21T15:40:16.738 INFO:teuthology.orchestra.run.smithi089.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-21T15:40:16.738 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:40:16.738 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-21T15:40:17.110 DEBUG:teuthology.orchestra.run.smithi089:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T15:40:17.188 INFO:teuthology.orchestra.run.smithi089.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T15:40:17.189 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T15:40:17.190 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-21T15:40:17.190 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-21T15:40:17.190 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-21T15:40:17.190 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-21T15:40:17.190 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi089.front.sepia.ceph.com, path=None, version=distro) 2024-09-21T15:40:17.190 DEBUG:teuthology.orchestra.run.smithi089:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T15:40:17.249 INFO:teuthology.orchestra.run.smithi089.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T15:40:17.249 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T15:40:17.250 DEBUG:teuthology.orchestra.run.smithi089:> sudo rpm -qi grub2-tools 2024-09-21T15:40:17.284 INFO:teuthology.orchestra.run.smithi089.stdout:Name : grub2-tools 2024-09-21T15:40:17.284 INFO:teuthology.orchestra.run.smithi089.stdout:Epoch : 1 2024-09-21T15:40:17.284 INFO:teuthology.orchestra.run.smithi089.stdout:Version : 2.06 2024-09-21T15:40:17.284 INFO:teuthology.orchestra.run.smithi089.stdout:Release : 82.el9 2024-09-21T15:40:17.284 INFO:teuthology.orchestra.run.smithi089.stdout:Architecture: x86_64 2024-09-21T15:40:17.284 INFO:teuthology.orchestra.run.smithi089.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-21T15:40:17.285 INFO:teuthology.orchestra.run.smithi089.stdout:Group : Unspecified 2024-09-21T15:40:17.285 INFO:teuthology.orchestra.run.smithi089.stdout:Size : 8274670 2024-09-21T15:40:17.285 INFO:teuthology.orchestra.run.smithi089.stdout:License : GPLv3+ 2024-09-21T15:40:17.285 INFO:teuthology.orchestra.run.smithi089.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-21T15:40:17.285 INFO:teuthology.orchestra.run.smithi089.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-21T15:40:17.285 INFO:teuthology.orchestra.run.smithi089.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-21T15:40:17.285 INFO:teuthology.orchestra.run.smithi089.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-21T15:40:17.285 INFO:teuthology.orchestra.run.smithi089.stdout:Packager : builder@centos.org 2024-09-21T15:40:17.285 INFO:teuthology.orchestra.run.smithi089.stdout:Vendor : CentOS 2024-09-21T15:40:17.285 INFO:teuthology.orchestra.run.smithi089.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-21T15:40:17.285 INFO:teuthology.orchestra.run.smithi089.stdout:Summary : Support tools for GRUB. 2024-09-21T15:40:17.285 INFO:teuthology.orchestra.run.smithi089.stdout:Description : 2024-09-21T15:40:17.285 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:40:17.285 INFO:teuthology.orchestra.run.smithi089.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-21T15:40:17.285 INFO:teuthology.orchestra.run.smithi089.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-21T15:40:17.286 INFO:teuthology.orchestra.run.smithi089.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-21T15:40:17.286 INFO:teuthology.orchestra.run.smithi089.stdout:hardware devices. 2024-09-21T15:40:17.286 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:40:17.286 INFO:teuthology.orchestra.run.smithi089.stdout:This subpackage provides tools for support of all platforms. 2024-09-21T15:40:17.287 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-21T15:40:17.287 INFO:teuthology.task.kernel:Updating grub on smithi089 to boot 5.14.0-508.el9.x86_64 2024-09-21T15:40:17.287 DEBUG:teuthology.orchestra.run.smithi089:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-21T15:40:17.958 INFO:teuthology.orchestra.run.smithi089.stderr:Generating grub configuration file ... 2024-09-21T15:40:19.154 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T15:40:19.194 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-21T15:40:19.317 INFO:teuthology.orchestra.run.smithi089.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-21T15:40:19.359 INFO:teuthology.orchestra.run.smithi089.stderr:done 2024-09-21T15:40:19.361 DEBUG:teuthology.orchestra.run.smithi089:> mktemp 2024-09-21T15:40:19.388 INFO:teuthology.orchestra.run.smithi089.stdout:/tmp/tmp.txpjg7nzdH 2024-09-21T15:40:19.389 DEBUG:teuthology.orchestra.run.smithi089:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.txpjg7nzdH 2024-09-21T15:40:19.468 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 0666 /tmp/tmp.txpjg7nzdH 2024-09-21T15:40:19.587 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T15:40:19.587 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-21T15:40:19.587 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T15:40:19.587 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T15:40:19.627 DEBUG:teuthology.orchestra.remote:smithi089:/tmp/tmp.txpjg7nzdH is 6KB 2024-09-21T15:40:19.656 DEBUG:teuthology.orchestra.run.smithi089:> rm -fr /tmp/tmp.txpjg7nzdH 2024-09-21T15:40:19.673 DEBUG:teuthology.orchestra.run.smithi089:> sudo /bin/ls /boot/loader/entries || true 2024-09-21T15:40:19.738 INFO:teuthology.orchestra.run.smithi089.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-21T15:40:19.738 INFO:teuthology.orchestra.run.smithi089.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-21T15:40:19.738 INFO:teuthology.orchestra.run.smithi089.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-21T15:40:19.739 INFO:teuthology.orchestra.run.smithi089.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-21T15:40:19.740 DEBUG:teuthology.orchestra.run.smithi089:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-21T15:40:19.892 DEBUG:teuthology.orchestra.run.smithi089:> sudo shutdown -r now 2024-09-21T15:40:20.418 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T15:40:20.418 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:40:20.419 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-21T15:40:20.419 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-21T15:40:20.419 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-21T15:40:20.419 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-21T15:40:20.419 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-21T15:40:20.419 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:40:20.419 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-21T15:40:20.775 DEBUG:teuthology.orchestra.run.smithi100:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T15:40:20.911 INFO:teuthology.orchestra.run.smithi100.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T15:40:20.911 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T15:40:20.911 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-21T15:40:20.911 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-21T15:40:20.911 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-21T15:40:20.912 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-21T15:40:20.912 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi100.front.sepia.ceph.com, path=None, version=distro) 2024-09-21T15:40:20.912 DEBUG:teuthology.orchestra.run.smithi100:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T15:40:20.970 INFO:teuthology.orchestra.run.smithi100.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T15:40:20.971 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T15:40:20.971 DEBUG:teuthology.orchestra.run.smithi100:> sudo rpm -qi grub2-tools 2024-09-21T15:40:21.003 INFO:teuthology.orchestra.run.smithi100.stdout:Name : grub2-tools 2024-09-21T15:40:21.003 INFO:teuthology.orchestra.run.smithi100.stdout:Epoch : 1 2024-09-21T15:40:21.003 INFO:teuthology.orchestra.run.smithi100.stdout:Version : 2.06 2024-09-21T15:40:21.003 INFO:teuthology.orchestra.run.smithi100.stdout:Release : 82.el9 2024-09-21T15:40:21.003 INFO:teuthology.orchestra.run.smithi100.stdout:Architecture: x86_64 2024-09-21T15:40:21.003 INFO:teuthology.orchestra.run.smithi100.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-21T15:40:21.003 INFO:teuthology.orchestra.run.smithi100.stdout:Group : Unspecified 2024-09-21T15:40:21.003 INFO:teuthology.orchestra.run.smithi100.stdout:Size : 8274670 2024-09-21T15:40:21.003 INFO:teuthology.orchestra.run.smithi100.stdout:License : GPLv3+ 2024-09-21T15:40:21.003 INFO:teuthology.orchestra.run.smithi100.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-21T15:40:21.003 INFO:teuthology.orchestra.run.smithi100.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-21T15:40:21.003 INFO:teuthology.orchestra.run.smithi100.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-21T15:40:21.004 INFO:teuthology.orchestra.run.smithi100.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-21T15:40:21.004 INFO:teuthology.orchestra.run.smithi100.stdout:Packager : builder@centos.org 2024-09-21T15:40:21.004 INFO:teuthology.orchestra.run.smithi100.stdout:Vendor : CentOS 2024-09-21T15:40:21.004 INFO:teuthology.orchestra.run.smithi100.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-21T15:40:21.004 INFO:teuthology.orchestra.run.smithi100.stdout:Summary : Support tools for GRUB. 2024-09-21T15:40:21.004 INFO:teuthology.orchestra.run.smithi100.stdout:Description : 2024-09-21T15:40:21.004 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:40:21.004 INFO:teuthology.orchestra.run.smithi100.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-21T15:40:21.004 INFO:teuthology.orchestra.run.smithi100.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-21T15:40:21.004 INFO:teuthology.orchestra.run.smithi100.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-21T15:40:21.004 INFO:teuthology.orchestra.run.smithi100.stdout:hardware devices. 2024-09-21T15:40:21.004 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:40:21.004 INFO:teuthology.orchestra.run.smithi100.stdout:This subpackage provides tools for support of all platforms. 2024-09-21T15:40:21.005 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-21T15:40:21.005 INFO:teuthology.task.kernel:Updating grub on smithi100 to boot 5.14.0-508.el9.x86_64 2024-09-21T15:40:21.005 DEBUG:teuthology.orchestra.run.smithi100:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-21T15:40:21.669 INFO:teuthology.orchestra.run.smithi100.stderr:Generating grub configuration file ... 2024-09-21T15:40:23.067 INFO:teuthology.orchestra.run.smithi100.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-21T15:40:23.099 INFO:teuthology.orchestra.run.smithi100.stderr:done 2024-09-21T15:40:23.101 DEBUG:teuthology.orchestra.run.smithi100:> mktemp 2024-09-21T15:40:23.117 INFO:teuthology.orchestra.run.smithi100.stdout:/tmp/tmp.ORFlAIf0qm 2024-09-21T15:40:23.117 DEBUG:teuthology.orchestra.run.smithi100:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ORFlAIf0qm 2024-09-21T15:40:23.185 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 0666 /tmp/tmp.ORFlAIf0qm 2024-09-21T15:40:23.325 DEBUG:teuthology.orchestra.remote:smithi100:/tmp/tmp.ORFlAIf0qm is 6KB 2024-09-21T15:40:23.345 DEBUG:teuthology.orchestra.run.smithi100:> rm -fr /tmp/tmp.ORFlAIf0qm 2024-09-21T15:40:23.360 DEBUG:teuthology.orchestra.run.smithi100:> sudo /bin/ls /boot/loader/entries || true 2024-09-21T15:40:23.422 INFO:teuthology.orchestra.run.smithi100.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-21T15:40:23.422 INFO:teuthology.orchestra.run.smithi100.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-21T15:40:23.423 INFO:teuthology.orchestra.run.smithi100.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-21T15:40:23.423 INFO:teuthology.orchestra.run.smithi100.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-21T15:40:23.424 DEBUG:teuthology.orchestra.run.smithi100:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-21T15:40:23.578 DEBUG:teuthology.orchestra.run.smithi100:> sudo shutdown -r now 2024-09-21T15:40:49.919 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-21T15:40:49.919 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-09-21T15:40:49.920 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T15:40:53.585 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-21T15:40:53.585 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com' 2024-09-21T15:40:53.586 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T15:41:24.591 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2024-09-21T15:41:33.592 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-09-21T15:41:33.593 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T15:41:51.983 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.89 2024-09-21T15:41:53.590 DEBUG:teuthology.orchestra.remote:timed out 2024-09-21T15:42:02.591 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com' 2024-09-21T15:42:02.591 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T15:42:02.951 DEBUG:teuthology.orchestra.run.smithi100:> true 2024-09-21T15:42:03.337 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi100.front.sepia.ceph.com' 2024-09-21T15:42:03.337 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-508.el9.x86_64"... 2024-09-21T15:42:03.337 DEBUG:teuthology.orchestra.run.smithi100:> uname -r 2024-09-21T15:42:03.352 INFO:teuthology.orchestra.run.smithi100.stdout:5.14.0-508.el9.x86_64 2024-09-21T15:42:03.352 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-21T15:42:03.352 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-21T15:42:03.353 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-21T15:42:03.984 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-09-21T15:42:03.984 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T15:42:04.353 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-21T15:42:04.354 DEBUG:teuthology.orchestra.run.smithi100:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-21T15:42:04.439 INFO:teuthology.orchestra.run.smithi100.stdout:ttyS1 2024-09-21T15:42:04.463 DEBUG:teuthology.parallel:result is None 2024-09-21T15:42:04.638 DEBUG:teuthology.orchestra.run.smithi089:> true 2024-09-21T15:42:05.063 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-09-21T15:42:05.063 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-21T15:42:05.063 DEBUG:teuthology.orchestra.run.smithi089:> uname -r 2024-09-21T15:42:05.078 INFO:teuthology.orchestra.run.smithi089.stdout:5.14.0-508.el9.x86_64 2024-09-21T15:42:05.078 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-21T15:42:05.078 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-21T15:42:05.078 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-21T15:42:06.079 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-21T15:42:06.079 DEBUG:teuthology.orchestra.run.smithi089:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-21T15:42:06.152 INFO:teuthology.orchestra.run.smithi089.stdout:ttyS1 2024-09-21T15:42:06.186 DEBUG:teuthology.parallel:result is None 2024-09-21T15:42:06.186 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-21T15:42:06.195 INFO:teuthology.task.internal:Creating test directory... 2024-09-21T15:42:06.195 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T15:42:06.197 DEBUG:teuthology.orchestra.run.smithi100:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T15:42:06.212 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-21T15:42:06.218 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-21T15:42:06.226 INFO:teuthology.task.internal:Creating archive directory... 2024-09-21T15:42:06.226 DEBUG:teuthology.orchestra.run.smithi089:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T15:42:06.256 DEBUG:teuthology.orchestra.run.smithi100:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T15:42:06.278 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-21T15:42:06.285 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-21T15:42:06.285 DEBUG:teuthology.orchestra.run.smithi089:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T15:42:06.316 DEBUG:teuthology.orchestra.run.smithi100:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T15:42:06.347 INFO:teuthology.orchestra.run.smithi089.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T15:42:06.348 INFO:teuthology.orchestra.run.smithi100.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T15:42:06.358 INFO:teuthology.orchestra.run.smithi089.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T15:42:06.358 INFO:teuthology.orchestra.run.smithi100.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T15:42:06.360 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-21T15:42:06.367 INFO:teuthology.task.internal:Configuring sudo... 2024-09-21T15:42:06.367 DEBUG:teuthology.orchestra.run.smithi089:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T15:42:06.406 DEBUG:teuthology.orchestra.run.smithi100:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T15:42:06.430 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-21T15:42:06.439 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-21T15:42:06.439 DEBUG:teuthology.orchestra.run.smithi089:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T15:42:06.476 DEBUG:teuthology.orchestra.run.smithi100:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T15:42:06.491 DEBUG:teuthology.orchestra.run.smithi089:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T15:42:06.575 DEBUG:teuthology.orchestra.run.smithi089:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T15:42:06.624 DEBUG:teuthology.orchestra.run.smithi089:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T15:42:06.711 DEBUG:teuthology.orchestra.run.smithi089:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T15:42:06.753 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-21T15:42:06.753 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T15:42:06.823 DEBUG:teuthology.orchestra.run.smithi100:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T15:42:06.866 DEBUG:teuthology.orchestra.run.smithi100:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T15:42:06.923 DEBUG:teuthology.orchestra.run.smithi100:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T15:42:07.006 DEBUG:teuthology.orchestra.run.smithi100:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T15:42:07.082 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-21T15:42:07.082 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T15:42:07.140 DEBUG:teuthology.orchestra.run.smithi089:> sudo service rsyslog restart 2024-09-21T15:42:07.142 DEBUG:teuthology.orchestra.run.smithi100:> sudo service rsyslog restart 2024-09-21T15:42:07.196 INFO:teuthology.orchestra.run.smithi089.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T15:42:07.383 INFO:teuthology.orchestra.run.smithi100.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T15:42:07.764 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-21T15:42:07.772 INFO:teuthology.task.internal:Starting timer... 2024-09-21T15:42:07.772 INFO:teuthology.run_tasks:Running task pcp... 2024-09-21T15:42:07.784 INFO:teuthology.run_tasks:Running task selinux... 2024-09-21T15:42:07.794 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-09-21T15:42:07.794 DEBUG:teuthology.orchestra.run.smithi089:> sudo service auditd rotate 2024-09-21T15:42:07.837 INFO:teuthology.orchestra.run.smithi089.stdout:Rotating logs: 2024-09-21T15:42:07.839 DEBUG:teuthology.orchestra.run.smithi100:> sudo service auditd rotate 2024-09-21T15:42:07.961 INFO:teuthology.orchestra.run.smithi100.stdout:Rotating logs: 2024-09-21T15:42:07.964 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-21T15:42:07.965 DEBUG:teuthology.orchestra.run.smithi089:> /usr/sbin/getenforce 2024-09-21T15:42:07.984 INFO:teuthology.orchestra.run.smithi089.stdout:Permissive 2024-09-21T15:42:07.984 DEBUG:teuthology.orchestra.run.smithi100:> /usr/sbin/getenforce 2024-09-21T15:42:08.032 INFO:teuthology.orchestra.run.smithi100.stdout:Permissive 2024-09-21T15:42:08.033 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi089.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi100.front.sepia.ceph.com': 'permissive'} 2024-09-21T15:42:08.033 DEBUG:teuthology.orchestra.run.smithi089:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-21T15:42:08.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T15:42:08.058 DEBUG:teuthology.orchestra.run.smithi100:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-21T15:42:08.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T15:42:08.084 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-21T15:42:08.084 DEBUG:teuthology.orchestra.run.smithi089:> sudo /usr/sbin/setenforce permissive 2024-09-21T15:42:08.148 DEBUG:teuthology.orchestra.run.smithi100:> sudo /usr/sbin/setenforce permissive 2024-09-21T15:42:08.190 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-21T15:42:08.199 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-21T15:42:08.209 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-21T15:42:08.210 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi089.front.sepia.ceph.com,smithi100.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-21T15:45:54.915 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi089.front.sepia.ceph.com'), Remote(name='ubuntu@smithi100.front.sepia.ceph.com')] 2024-09-21T15:45:54.916 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-09-21T15:45:54.917 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi089.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T15:45:54.991 DEBUG:teuthology.orchestra.run.smithi089:> true 2024-09-21T15:45:55.065 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi089.front.sepia.ceph.com' 2024-09-21T15:45:55.066 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com' 2024-09-21T15:45:55.066 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T15:45:55.138 DEBUG:teuthology.orchestra.run.smithi100:> true 2024-09-21T15:45:55.225 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi100.front.sepia.ceph.com' 2024-09-21T15:45:55.225 INFO:teuthology.run_tasks:Running task clock... 2024-09-21T15:45:55.236 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-21T15:45:55.236 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T15:45:55.237 DEBUG:teuthology.orchestra.run.smithi089:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T15:45:55.239 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T15:45:55.240 DEBUG:teuthology.orchestra.run.smithi100:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T15:45:55.265 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-21T15:45:55.279 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-21T15:45:55.295 INFO:teuthology.orchestra.run.smithi100.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-21T15:45:55.303 INFO:teuthology.orchestra.run.smithi089.stderr:sudo: ntpd: command not found 2024-09-21T15:45:55.309 INFO:teuthology.orchestra.run.smithi100.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-21T15:45:55.314 INFO:teuthology.orchestra.run.smithi089.stdout:506 Cannot talk to daemon 2024-09-21T15:45:55.328 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-21T15:45:55.332 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: ntpd: command not found 2024-09-21T15:45:55.343 INFO:teuthology.orchestra.run.smithi089.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-21T15:45:55.343 INFO:teuthology.orchestra.run.smithi100.stdout:506 Cannot talk to daemon 2024-09-21T15:45:55.358 INFO:teuthology.orchestra.run.smithi100.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-21T15:45:55.372 INFO:teuthology.orchestra.run.smithi100.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-21T15:45:55.396 INFO:teuthology.orchestra.run.smithi089.stderr:bash: line 1: ntpq: command not found 2024-09-21T15:45:55.400 INFO:teuthology.orchestra.run.smithi089.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T15:45:55.400 INFO:teuthology.orchestra.run.smithi089.stdout:=============================================================================== 2024-09-21T15:45:55.400 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T15:45:55.400 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T15:45:55.400 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T15:45:55.400 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T15:45:55.428 INFO:teuthology.orchestra.run.smithi100.stderr:bash: line 1: ntpq: command not found 2024-09-21T15:45:55.431 INFO:teuthology.orchestra.run.smithi100.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T15:45:55.431 INFO:teuthology.orchestra.run.smithi100.stdout:=============================================================================== 2024-09-21T15:45:55.431 INFO:teuthology.orchestra.run.smithi100.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T15:45:55.431 INFO:teuthology.orchestra.run.smithi100.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T15:45:55.431 INFO:teuthology.orchestra.run.smithi100.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T15:45:55.431 INFO:teuthology.orchestra.run.smithi100.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T15:45:55.432 INFO:teuthology.run_tasks:Running task install... 2024-09-21T15:45:55.440 DEBUG:teuthology.task.install:project ceph 2024-09-21T15:45:55.440 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '19e2a2beb6f41a3ab62650e38614eb29b59dcbb9'}} 2024-09-21T15:45:55.440 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '19e2a2beb6f41a3ab62650e38614eb29b59dcbb9'} 2024-09-21T15:45:55.440 INFO:teuthology.task.install:Using flavor: default 2024-09-21T15:45:55.445 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-21T15:45:55.445 INFO:teuthology.task.install:extra packages: [] 2024-09-21T15:45:55.445 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '19e2a2beb6f41a3ab62650e38614eb29b59dcbb9', 'tag': None, 'wait_for_package': False} 2024-09-21T15:45:55.446 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-21T15:45:55.447 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '19e2a2beb6f41a3ab62650e38614eb29b59dcbb9', 'tag': None, 'wait_for_package': False} 2024-09-21T15:45:55.447 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-21T15:45:55.591 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/centos/9/flavors/default/ 2024-09-21T15:45:55.592 INFO:teuthology.task.install.rpm:Package version is 18.2.4-855.g19e2a2be 2024-09-21T15:45:55.595 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/centos/9/flavors/default/ 2024-09-21T15:45:55.595 INFO:teuthology.task.install.rpm:Package version is 18.2.4-855.g19e2a2be 2024-09-21T15:45:55.700 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-21T15:45:55.700 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-21T15:45:55.700 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-21T15:45:55.708 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/reef/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-21T15:45:55.708 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-21T15:45:55.708 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-21T15:45:55.726 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-21T15:45:55.727 DEBUG:teuthology.orchestra.run.smithi089:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-21T15:45:55.734 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-21T15:45:55.734 DEBUG:teuthology.orchestra.run.smithi100:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/19e2a2beb6f41a3ab62650e38614eb29b59dcbb9/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-21T15:45:55.792 DEBUG:teuthology.orchestra.run.smithi089:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-21T15:45:55.799 DEBUG:teuthology.orchestra.run.smithi100:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-21T15:45:55.867 DEBUG:teuthology.orchestra.run.smithi089:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-21T15:45:55.875 DEBUG:teuthology.orchestra.run.smithi100:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-21T15:45:55.931 INFO:teuthology.orchestra.run.smithi089.stdout:check_obsoletes = 1 2024-09-21T15:45:55.933 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum clean all 2024-09-21T15:45:55.939 INFO:teuthology.orchestra.run.smithi100.stdout:check_obsoletes = 1 2024-09-21T15:45:55.940 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum clean all 2024-09-21T15:45:56.159 INFO:teuthology.orchestra.run.smithi089.stdout:45 files removed 2024-09-21T15:45:56.177 INFO:teuthology.orchestra.run.smithi100.stdout:45 files removed 2024-09-21T15:45:56.186 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-radosgw 2024-09-21T15:45:56.204 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-radosgw 2024-09-21T15:45:56.742 INFO:teuthology.orchestra.run.smithi089.stdout:ceph packages for x86_64 285 kB/s | 77 kB 00:00 2024-09-21T15:45:56.771 INFO:teuthology.orchestra.run.smithi100.stdout:ceph packages for x86_64 281 kB/s | 77 kB 00:00 2024-09-21T15:45:57.009 INFO:teuthology.orchestra.run.smithi089.stdout:ceph noarch packages 51 kB/s | 11 kB 00:00 2024-09-21T15:45:57.039 INFO:teuthology.orchestra.run.smithi100.stdout:ceph noarch packages 51 kB/s | 11 kB 00:00 2024-09-21T15:45:57.259 INFO:teuthology.orchestra.run.smithi089.stdout:ceph source packages 8.7 kB/s | 1.9 kB 00:00 2024-09-21T15:45:57.306 INFO:teuthology.orchestra.run.smithi100.stdout:ceph source packages 8.1 kB/s | 1.9 kB 00:00 2024-09-21T15:45:57.727 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - BaseOS 19 MB/s | 8.3 MB 00:00 2024-09-21T15:45:59.047 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - BaseOS 4.8 MB/s | 8.3 MB 00:01 2024-09-21T15:46:01.020 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - AppStream 9.2 MB/s | 20 MB 00:02 2024-09-21T15:46:01.673 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - AppStream 13 MB/s | 20 MB 00:01 2024-09-21T15:46:06.016 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.5 MB 00:00 2024-09-21T15:46:06.670 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - CRB 3.5 MB/s | 6.5 MB 00:01 2024-09-21T15:46:07.642 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - Extras packages 79 kB/s | 19 kB 00:00 2024-09-21T15:46:08.213 INFO:teuthology.orchestra.run.smithi089.stdout:CentOS Stream 9 - Extras packages 66 kB/s | 19 kB 00:00 2024-09-21T15:46:08.663 INFO:teuthology.orchestra.run.smithi100.stdout:Extra Packages for Enterprise Linux 25 MB/s | 23 MB 00:00 2024-09-21T15:46:09.384 INFO:teuthology.orchestra.run.smithi089.stdout:Extra Packages for Enterprise Linux 22 MB/s | 23 MB 00:01 2024-09-21T15:46:14.085 INFO:teuthology.orchestra.run.smithi100.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2024-09-21T15:46:14.782 INFO:teuthology.orchestra.run.smithi089.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-09-21T15:46:16.286 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-21T15:46:16.296 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-21T15:46:16.297 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-09-21T15:46:16.297 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-21T15:46:16.297 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-21T15:46:16.297 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-radosgw x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 7.7 M 2024-09-21T15:46:16.297 INFO:teuthology.orchestra.run.smithi100.stdout:Upgrading: 2024-09-21T15:46:16.297 INFO:teuthology.orchestra.run.smithi100.stdout: librados2 x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 3.3 M 2024-09-21T15:46:16.297 INFO:teuthology.orchestra.run.smithi100.stdout: librbd1 x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 3.0 M 2024-09-21T15:46:16.297 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-09-21T15:46:16.297 INFO:teuthology.orchestra.run.smithi100.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-21T15:46:16.297 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-base x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 5.1 M 2024-09-21T15:46:16.297 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-common x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 18 M 2024-09-21T15:46:16.297 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-selinux x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 25 k 2024-09-21T15:46:16.297 INFO:teuthology.orchestra.run.smithi100.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-09-21T15:46:16.298 INFO:teuthology.orchestra.run.smithi100.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-21T15:46:16.298 INFO:teuthology.orchestra.run.smithi100.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-21T15:46:16.298 INFO:teuthology.orchestra.run.smithi100.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-21T15:46:16.298 INFO:teuthology.orchestra.run.smithi100.stdout: libcephfs2 x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 712 k 2024-09-21T15:46:16.298 INFO:teuthology.orchestra.run.smithi100.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-21T15:46:16.298 INFO:teuthology.orchestra.run.smithi100.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-21T15:46:16.298 INFO:teuthology.orchestra.run.smithi100.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-21T15:46:16.298 INFO:teuthology.orchestra.run.smithi100.stdout: libradosstriper1 x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 477 k 2024-09-21T15:46:16.298 INFO:teuthology.orchestra.run.smithi100.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-21T15:46:16.298 INFO:teuthology.orchestra.run.smithi100.stdout: librgw2 x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 4.5 M 2024-09-21T15:46:16.298 INFO:teuthology.orchestra.run.smithi100.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-21T15:46:16.298 INFO:teuthology.orchestra.run.smithi100.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-21T15:46:16.298 INFO:teuthology.orchestra.run.smithi100.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-21T15:46:16.298 INFO:teuthology.orchestra.run.smithi100.stdout: python3-ceph-argparse x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 45 k 2024-09-21T15:46:16.299 INFO:teuthology.orchestra.run.smithi100.stdout: python3-ceph-common x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 129 k 2024-09-21T15:46:16.299 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cephfs x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 162 k 2024-09-21T15:46:16.299 INFO:teuthology.orchestra.run.smithi100.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-21T15:46:16.299 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rados x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 321 k 2024-09-21T15:46:16.299 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rbd x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 300 k 2024-09-21T15:46:16.299 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rgw x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 99 k 2024-09-21T15:46:16.299 INFO:teuthology.orchestra.run.smithi100.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-21T15:46:16.299 INFO:teuthology.orchestra.run.smithi100.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-21T15:46:16.299 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:46:16.299 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-21T15:46:16.299 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-21T15:46:16.299 INFO:teuthology.orchestra.run.smithi100.stdout:Install 28 Packages 2024-09-21T15:46:16.299 INFO:teuthology.orchestra.run.smithi100.stdout:Upgrade 2 Packages 2024-09-21T15:46:16.300 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:46:16.300 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 53 M 2024-09-21T15:46:16.300 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-21T15:46:17.051 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-21T15:46:17.071 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-21T15:46:17.071 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-09-21T15:46:17.071 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-21T15:46:17.071 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-09-21T15:46:17.071 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-radosgw x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 7.7 M 2024-09-21T15:46:17.071 INFO:teuthology.orchestra.run.smithi089.stdout:Upgrading: 2024-09-21T15:46:17.071 INFO:teuthology.orchestra.run.smithi089.stdout: librados2 x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 3.3 M 2024-09-21T15:46:17.071 INFO:teuthology.orchestra.run.smithi089.stdout: librbd1 x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 3.0 M 2024-09-21T15:46:17.072 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-09-21T15:46:17.072 INFO:teuthology.orchestra.run.smithi089.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-21T15:46:17.072 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-base x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 5.1 M 2024-09-21T15:46:17.072 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-common x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 18 M 2024-09-21T15:46:17.072 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-selinux x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 25 k 2024-09-21T15:46:17.072 INFO:teuthology.orchestra.run.smithi089.stdout: fmt x86_64 8.1.1-5.el9 epel 111 k 2024-09-21T15:46:17.072 INFO:teuthology.orchestra.run.smithi089.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-21T15:46:17.072 INFO:teuthology.orchestra.run.smithi089.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-21T15:46:17.072 INFO:teuthology.orchestra.run.smithi089.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-21T15:46:17.072 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs2 x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 712 k 2024-09-21T15:46:17.072 INFO:teuthology.orchestra.run.smithi089.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-21T15:46:17.072 INFO:teuthology.orchestra.run.smithi089.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-21T15:46:17.072 INFO:teuthology.orchestra.run.smithi089.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-21T15:46:17.072 INFO:teuthology.orchestra.run.smithi089.stdout: libradosstriper1 x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 477 k 2024-09-21T15:46:17.072 INFO:teuthology.orchestra.run.smithi089.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-21T15:46:17.072 INFO:teuthology.orchestra.run.smithi089.stdout: librgw2 x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 4.5 M 2024-09-21T15:46:17.073 INFO:teuthology.orchestra.run.smithi089.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-21T15:46:17.073 INFO:teuthology.orchestra.run.smithi089.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-21T15:46:17.073 INFO:teuthology.orchestra.run.smithi089.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-21T15:46:17.073 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-argparse x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 45 k 2024-09-21T15:46:17.073 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-common x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 129 k 2024-09-21T15:46:17.073 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cephfs x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 162 k 2024-09-21T15:46:17.073 INFO:teuthology.orchestra.run.smithi089.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-21T15:46:17.073 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rados x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 321 k 2024-09-21T15:46:17.073 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rbd x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 300 k 2024-09-21T15:46:17.073 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rgw x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 99 k 2024-09-21T15:46:17.073 INFO:teuthology.orchestra.run.smithi089.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-21T15:46:17.073 INFO:teuthology.orchestra.run.smithi089.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-21T15:46:17.073 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:46:17.073 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-09-21T15:46:17.073 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-21T15:46:17.073 INFO:teuthology.orchestra.run.smithi089.stdout:Install 28 Packages 2024-09-21T15:46:17.074 INFO:teuthology.orchestra.run.smithi089.stdout:Upgrade 2 Packages 2024-09-21T15:46:17.074 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:46:17.074 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 53 M 2024-09-21T15:46:17.074 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-09-21T15:46:17.261 INFO:teuthology.orchestra.run.smithi100.stdout:(1/30): ceph-base-18.2.4-855.g19e2a2be.el9.x86_ 9.9 MB/s | 5.1 MB 00:00 2024-09-21T15:46:17.395 INFO:teuthology.orchestra.run.smithi100.stdout:(2/30): ceph-radosgw-18.2.4-855.g19e2a2be.el9.x 12 MB/s | 7.7 MB 00:00 2024-09-21T15:46:17.420 INFO:teuthology.orchestra.run.smithi100.stdout:(3/30): ceph-selinux-18.2.4-855.g19e2a2be.el9.x 159 kB/s | 25 kB 00:00 2024-09-21T15:46:17.462 INFO:teuthology.orchestra.run.smithi100.stdout:(4/30): libcephfs2-18.2.4-855.g19e2a2be.el9.x86 10 MB/s | 712 kB 00:00 2024-09-21T15:46:17.487 INFO:teuthology.orchestra.run.smithi100.stdout:(5/30): libradosstriper1-18.2.4-855.g19e2a2be.e 6.9 MB/s | 477 kB 00:00 2024-09-21T15:46:17.554 INFO:teuthology.orchestra.run.smithi100.stdout:(6/30): python3-ceph-argparse-18.2.4-855.g19e2a 676 kB/s | 45 kB 00:00 2024-09-21T15:46:17.629 INFO:teuthology.orchestra.run.smithi100.stdout:(7/30): python3-ceph-common-18.2.4-855.g19e2a2b 1.7 MB/s | 129 kB 00:00 2024-09-21T15:46:17.813 INFO:teuthology.orchestra.run.smithi100.stdout:(8/30): ceph-common-18.2.4-855.g19e2a2be.el9.x8 17 MB/s | 18 MB 00:01 2024-09-21T15:46:17.838 INFO:teuthology.orchestra.run.smithi100.stdout:(9/30): python3-cephfs-18.2.4-855.g19e2a2be.el9 775 kB/s | 162 kB 00:00 2024-09-21T15:46:17.905 INFO:teuthology.orchestra.run.smithi100.stdout:(10/30): librgw2-18.2.4-855.g19e2a2be.el9.x86_6 10 MB/s | 4.5 MB 00:00 2024-09-21T15:46:17.930 INFO:teuthology.orchestra.run.smithi100.stdout:(11/30): python3-rbd-18.2.4-855.g19e2a2be.el9.x 3.2 MB/s | 300 kB 00:00 2024-09-21T15:46:17.956 INFO:teuthology.orchestra.run.smithi100.stdout:(12/30): python3-rados-18.2.4-855.g19e2a2be.el9 2.2 MB/s | 321 kB 00:00 2024-09-21T15:46:17.981 INFO:teuthology.orchestra.run.smithi100.stdout:(13/30): python3-rgw-18.2.4-855.g19e2a2be.el9.x 1.3 MB/s | 99 kB 00:00 2024-09-21T15:46:18.090 INFO:teuthology.orchestra.run.smithi100.stdout:(14/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 417 kB/s | 45 kB 00:00 2024-09-21T15:46:18.100 INFO:teuthology.orchestra.run.smithi089.stdout:(1/30): ceph-base-18.2.4-855.g19e2a2be.el9.x86_ 9.2 MB/s | 5.1 MB 00:00 2024-09-21T15:46:18.115 INFO:teuthology.orchestra.run.smithi100.stdout:(15/30): boost-program-options-1.75.0-8.el9.x86 580 kB/s | 107 kB 00:00 2024-09-21T15:46:18.140 INFO:teuthology.orchestra.run.smithi100.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 871 kB/s | 160 kB 00:00 2024-09-21T15:46:18.191 INFO:teuthology.orchestra.run.smithi100.stdout:(17/30): python3-prettytable-0.7.2-27.el9.noarc 844 kB/s | 42 kB 00:00 2024-09-21T15:46:18.216 INFO:teuthology.orchestra.run.smithi100.stdout:(18/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.8 MB/s | 292 kB 00:00 2024-09-21T15:46:18.234 INFO:teuthology.orchestra.run.smithi089.stdout:(2/30): ceph-radosgw-18.2.4-855.g19e2a2be.el9.x 11 MB/s | 7.7 MB 00:00 2024-09-21T15:46:18.258 INFO:teuthology.orchestra.run.smithi089.stdout:(3/30): ceph-selinux-18.2.4-855.g19e2a2be.el9.x 159 kB/s | 25 kB 00:00 2024-09-21T15:46:18.266 INFO:teuthology.orchestra.run.smithi100.stdout:(19/30): librdkafka-1.6.1-102.el9.x86_64.rpm 3.7 MB/s | 662 kB 00:00 2024-09-21T15:46:18.308 INFO:teuthology.orchestra.run.smithi100.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 942 kB/s | 111 kB 00:00 2024-09-21T15:46:18.309 INFO:teuthology.orchestra.run.smithi089.stdout:(4/30): libcephfs2-18.2.4-855.g19e2a2be.el9.x86 9.3 MB/s | 712 kB 00:00 2024-09-21T15:46:18.334 INFO:teuthology.orchestra.run.smithi100.stdout:(21/30): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.1 MB/s | 26 kB 00:00 2024-09-21T15:46:18.343 INFO:teuthology.orchestra.run.smithi089.stdout:(5/30): libradosstriper1-18.2.4-855.g19e2a2be.e 5.6 MB/s | 477 kB 00:00 2024-09-21T15:46:18.359 INFO:teuthology.orchestra.run.smithi100.stdout:(22/30): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-09-21T15:46:18.393 INFO:teuthology.orchestra.run.smithi100.stdout:(23/30): gperftools-libs-2.9.1-3.el9.x86_64.rpm 1.8 MB/s | 308 kB 00:00 2024-09-21T15:46:18.410 INFO:teuthology.orchestra.run.smithi089.stdout:(6/30): python3-ceph-argparse-18.2.4-855.g19e2a 674 kB/s | 45 kB 00:00 2024-09-21T15:46:18.418 INFO:teuthology.orchestra.run.smithi100.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-09-21T15:46:18.460 INFO:teuthology.orchestra.run.smithi100.stdout:(25/30): re2-20211101-20.el9.x86_64.rpm 4.5 MB/s | 191 kB 00:00 2024-09-21T15:46:18.485 INFO:teuthology.orchestra.run.smithi100.stdout:(26/30): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.9 MB/s | 839 kB 00:00 2024-09-21T15:46:18.493 INFO:teuthology.orchestra.run.smithi089.stdout:(7/30): python3-ceph-common-18.2.4-855.g19e2a2b 1.5 MB/s | 129 kB 00:00 2024-09-21T15:46:18.560 INFO:teuthology.orchestra.run.smithi089.stdout:(8/30): librgw2-18.2.4-855.g19e2a2be.el9.x86_64 18 MB/s | 4.5 MB 00:00 2024-09-21T15:46:18.569 INFO:teuthology.orchestra.run.smithi100.stdout:(27/30): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-09-21T15:46:18.594 INFO:teuthology.orchestra.run.smithi089.stdout:(9/30): python3-cephfs-18.2.4-855.g19e2a2be.el9 1.6 MB/s | 162 kB 00:00 2024-09-21T15:46:18.619 INFO:teuthology.orchestra.run.smithi089.stdout:(10/30): python3-rados-18.2.4-855.g19e2a2be.el9 5.3 MB/s | 321 kB 00:00 2024-09-21T15:46:18.645 INFO:teuthology.orchestra.run.smithi089.stdout:(11/30): python3-rbd-18.2.4-855.g19e2a2be.el9.x 5.8 MB/s | 300 kB 00:00 2024-09-21T15:46:18.652 INFO:teuthology.orchestra.run.smithi100.stdout:(28/30): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-09-21T15:46:18.670 INFO:teuthology.orchestra.run.smithi089.stdout:(12/30): python3-rgw-18.2.4-855.g19e2a2be.el9.x 1.9 MB/s | 99 kB 00:00 2024-09-21T15:46:18.744 INFO:teuthology.orchestra.run.smithi100.stdout:(29/30): librados2-18.2.4-855.g19e2a2be.el9.x86 13 MB/s | 3.3 MB 00:00 2024-09-21T15:46:18.811 INFO:teuthology.orchestra.run.smithi100.stdout:(30/30): librbd1-18.2.4-855.g19e2a2be.el9.x86_6 12 MB/s | 3.0 MB 00:00 2024-09-21T15:46:18.817 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-21T15:46:18.818 INFO:teuthology.orchestra.run.smithi100.stdout:Total 21 MB/s | 53 MB 00:02 2024-09-21T15:46:18.854 INFO:teuthology.orchestra.run.smithi089.stdout:(13/30): ceph-common-18.2.4-855.g19e2a2be.el9.x 14 MB/s | 18 MB 00:01 2024-09-21T15:46:18.887 INFO:teuthology.orchestra.run.smithi089.stdout:(14/30): boost-program-options-1.75.0-8.el9.x86 441 kB/s | 107 kB 00:00 2024-09-21T15:46:18.892 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-21T15:46:18.927 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-21T15:46:18.928 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-21T15:46:18.929 INFO:teuthology.orchestra.run.smithi089.stdout:(15/30): librabbitmq-0.11.0-7.el9.x86_64.rpm 603 kB/s | 45 kB 00:00 2024-09-21T15:46:18.979 INFO:teuthology.orchestra.run.smithi089.stdout:(16/30): libpmemobj-1.12.1-1.el9.x86_64.rpm 518 kB/s | 160 kB 00:00 2024-09-21T15:46:19.022 INFO:teuthology.orchestra.run.smithi089.stdout:(17/30): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.1 MB/s | 292 kB 00:00 2024-09-21T15:46:19.047 INFO:teuthology.orchestra.run.smithi089.stdout:(18/30): librdkafka-1.6.1-102.el9.x86_64.rpm 4.1 MB/s | 662 kB 00:00 2024-09-21T15:46:19.080 INFO:teuthology.orchestra.run.smithi089.stdout:(19/30): python3-prettytable-0.7.2-27.el9.noarc 419 kB/s | 42 kB 00:00 2024-09-21T15:46:19.123 INFO:teuthology.orchestra.run.smithi089.stdout:(20/30): fmt-8.1.1-5.el9.x86_64.rpm 1.1 MB/s | 111 kB 00:00 2024-09-21T15:46:19.148 INFO:teuthology.orchestra.run.smithi089.stdout:(21/30): libarrow-doc-9.0.0-11.el9.noarch.rpm 1.1 MB/s | 26 kB 00:00 2024-09-21T15:46:19.174 INFO:teuthology.orchestra.run.smithi089.stdout:(22/30): liboath-2.6.7-2.el9.x86_64.rpm 1.9 MB/s | 49 kB 00:00 2024-09-21T15:46:19.199 INFO:teuthology.orchestra.run.smithi089.stdout:(23/30): gperftools-libs-2.9.1-3.el9.x86_64.rpm 2.1 MB/s | 308 kB 00:00 2024-09-21T15:46:19.224 INFO:teuthology.orchestra.run.smithi089.stdout:(24/30): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-21T15:46:19.258 INFO:teuthology.orchestra.run.smithi089.stdout:(25/30): re2-20211101-20.el9.x86_64.rpm 5.6 MB/s | 191 kB 00:00 2024-09-21T15:46:19.300 INFO:teuthology.orchestra.run.smithi089.stdout:(26/30): parquet-libs-9.0.0-11.el9.x86_64.rpm 8.1 MB/s | 839 kB 00:00 2024-09-21T15:46:19.367 INFO:teuthology.orchestra.run.smithi089.stdout:(27/30): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-09-21T15:46:19.450 INFO:teuthology.orchestra.run.smithi089.stdout:(28/30): libarrow-9.0.0-11.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-09-21T15:46:19.534 INFO:teuthology.orchestra.run.smithi089.stdout:(29/30): librados2-18.2.4-855.g19e2a2be.el9.x86 14 MB/s | 3.3 MB 00:00 2024-09-21T15:46:19.545 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-21T15:46:19.545 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-21T15:46:19.618 INFO:teuthology.orchestra.run.smithi089.stdout:(30/30): librbd1-18.2.4-855.g19e2a2be.el9.x86_6 12 MB/s | 3.0 MB 00:00 2024-09-21T15:46:19.622 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-09-21T15:46:19.622 INFO:teuthology.orchestra.run.smithi089.stdout:Total 21 MB/s | 53 MB 00:02 2024-09-21T15:46:19.693 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-09-21T15:46:19.727 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-09-21T15:46:19.727 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-09-21T15:46:20.345 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-09-21T15:46:20.345 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-09-21T15:46:20.921 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-21T15:46:21.020 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-09-21T15:46:21.200 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-09-21T15:46:21.249 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-09-21T15:46:21.622 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-09-21T15:46:21.673 INFO:teuthology.orchestra.run.smithi100.stdout: Upgrading : librados2-2:18.2.4-855.g19e2a2be.el9.x86_64 5/32 2024-09-21T15:46:21.765 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librados2-2:18.2.4-855.g19e2a2be.el9.x86_64 5/32 2024-09-21T15:46:21.777 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-09-21T15:46:21.850 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-rados-2:18.2.4-855.g19e2a2be.el9.x86_64 6/32 2024-09-21T15:46:21.868 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/32 2024-09-21T15:46:21.875 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libcephfs2-2:18.2.4-855.g19e2a2be.el9.x86_64 7/32 2024-09-21T15:46:21.898 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/32 2024-09-21T15:46:21.938 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: libcephfs2-2:18.2.4-855.g19e2a2be.el9.x86_64 7/32 2024-09-21T15:46:21.948 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/32 2024-09-21T15:46:21.975 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-09-21T15:46:22.044 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-ceph-argparse-2:18.2.4-855.g19e2a2be.el9.x 9/32 2024-09-21T15:46:22.126 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cephfs-2:18.2.4-855.g19e2a2be.el9.x86_64 10/32 2024-09-21T15:46:22.152 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libradosstriper1-2:18.2.4-855.g19e2a2be.el9.x86_64 11/32 2024-09-21T15:46:22.219 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: libradosstriper1-2:18.2.4-855.g19e2a2be.el9.x86_64 11/32 2024-09-21T15:46:22.316 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/32 2024-09-21T15:46:22.323 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : re2-1:20211101-20.el9.x86_64 12/32 2024-09-21T15:46:22.360 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 13/32 2024-09-21T15:46:22.372 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : librados2-2:18.2.4-855.g19e2a2be.el9.x86_64 5/32 2024-09-21T15:46:22.426 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-09-21T15:46:22.458 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/32 2024-09-21T15:46:22.463 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librados2-2:18.2.4-855.g19e2a2be.el9.x86_64 5/32 2024-09-21T15:46:22.485 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 16/32 2024-09-21T15:46:22.521 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-09-21T15:46:22.549 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rados-2:18.2.4-855.g19e2a2be.el9.x86_64 6/32 2024-09-21T15:46:22.554 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-09-21T15:46:22.582 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libcephfs2-2:18.2.4-855.g19e2a2be.el9.x86_64 7/32 2024-09-21T15:46:22.645 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libcephfs2-2:18.2.4-855.g19e2a2be.el9.x86_64 7/32 2024-09-21T15:46:22.682 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : liboath-2.6.7-2.el9.x86_64 8/32 2024-09-21T15:46:22.746 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-ceph-argparse-2:18.2.4-855.g19e2a2be.el9.x 9/32 2024-09-21T15:46:22.823 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cephfs-2:18.2.4-855.g19e2a2be.el9.x86_64 10/32 2024-09-21T15:46:22.851 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libradosstriper1-2:18.2.4-855.g19e2a2be.el9.x86_64 11/32 2024-09-21T15:46:22.853 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-09-21T15:46:22.901 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libradosstriper1-2:18.2.4-855.g19e2a2be.el9.x86_64 11/32 2024-09-21T15:46:22.908 INFO:teuthology.orchestra.run.smithi100.stdout: Upgrading : librbd1-2:18.2.4-855.g19e2a2be.el9.x86_64 20/32 2024-09-21T15:46:23.000 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librbd1-2:18.2.4-855.g19e2a2be.el9.x86_64 20/32 2024-09-21T15:46:23.005 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : re2-1:20211101-20.el9.x86_64 12/32 2024-09-21T15:46:23.031 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-rbd-2:18.2.4-855.g19e2a2be.el9.x86_64 21/32 2024-09-21T15:46:23.042 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 13/32 2024-09-21T15:46:23.066 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-09-21T15:46:23.112 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/32 2024-09-21T15:46:23.140 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 15/32 2024-09-21T15:46:23.168 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 16/32 2024-09-21T15:46:23.204 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : fmt-8.1.1-5.el9.x86_64 17/32 2024-09-21T15:46:23.237 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 18/32 2024-09-21T15:46:23.487 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 23/32 2024-09-21T15:46:23.520 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : librgw2-2:18.2.4-855.g19e2a2be.el9.x86_64 24/32 2024-09-21T15:46:23.539 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 19/32 2024-09-21T15:46:23.583 INFO:teuthology.orchestra.run.smithi089.stdout: Upgrading : librbd1-2:18.2.4-855.g19e2a2be.el9.x86_64 20/32 2024-09-21T15:46:23.591 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librgw2-2:18.2.4-855.g19e2a2be.el9.x86_64 24/32 2024-09-21T15:46:23.667 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-rgw-2:18.2.4-855.g19e2a2be.el9.x86_64 25/32 2024-09-21T15:46:23.674 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librbd1-2:18.2.4-855.g19e2a2be.el9.x86_64 20/32 2024-09-21T15:46:23.697 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-ceph-common-2:18.2.4-855.g19e2a2be.el9.x86 26/32 2024-09-21T15:46:23.705 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rbd-2:18.2.4-855.g19e2a2be.el9.x86_64 21/32 2024-09-21T15:46:23.740 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 22/32 2024-09-21T15:46:24.144 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 23/32 2024-09-21T15:46:24.177 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librgw2-2:18.2.4-855.g19e2a2be.el9.x86_64 24/32 2024-09-21T15:46:24.249 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librgw2-2:18.2.4-855.g19e2a2be.el9.x86_64 24/32 2024-09-21T15:46:24.333 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rgw-2:18.2.4-855.g19e2a2be.el9.x86_64 25/32 2024-09-21T15:46:24.363 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-ceph-common-2:18.2.4-855.g19e2a2be.el9.x86 26/32 2024-09-21T15:46:25.793 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-common-2:18.2.4-855.g19e2a2be.el9.x86_64 27/32 2024-09-21T15:46:25.836 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-common-2:18.2.4-855.g19e2a2be.el9.x86_64 27/32 2024-09-21T15:46:26.378 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-common-2:18.2.4-855.g19e2a2be.el9.x86_64 27/32 2024-09-21T15:46:26.414 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-base-2:18.2.4-855.g19e2a2be.el9.x86_64 28/32 2024-09-21T15:46:26.498 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-common-2:18.2.4-855.g19e2a2be.el9.x86_64 27/32 2024-09-21T15:46:26.515 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-base-2:18.2.4-855.g19e2a2be.el9.x86_64 28/32 2024-09-21T15:46:26.515 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-21T15:46:26.516 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-21T15:46:26.516 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:46:26.535 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-common-2:18.2.4-855.g19e2a2be.el9.x86_64 27/32 2024-09-21T15:46:26.549 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-selinux-2:18.2.4-855.g19e2a2be.el9.x86_64 29/32 2024-09-21T15:46:27.079 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-common-2:18.2.4-855.g19e2a2be.el9.x86_64 27/32 2024-09-21T15:46:27.122 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-base-2:18.2.4-855.g19e2a2be.el9.x86_64 28/32 2024-09-21T15:46:27.228 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-base-2:18.2.4-855.g19e2a2be.el9.x86_64 28/32 2024-09-21T15:46:27.228 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-21T15:46:27.228 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-21T15:46:27.228 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:46:27.257 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-selinux-2:18.2.4-855.g19e2a2be.el9.x86_64 29/32 2024-09-21T15:46:36.459 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-selinux-2:18.2.4-855.g19e2a2be.el9.x86_64 29/32 2024-09-21T15:46:36.459 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /sys 2024-09-21T15:46:36.459 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /proc 2024-09-21T15:46:36.459 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /mnt 2024-09-21T15:46:36.459 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /var/tmp 2024-09-21T15:46:36.459 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /home 2024-09-21T15:46:36.459 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /root 2024-09-21T15:46:36.459 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /tmp 2024-09-21T15:46:36.459 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:46:37.226 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-radosgw-2:18.2.4-855.g19e2a2be.el9.x86_64 30/32 2024-09-21T15:46:37.251 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-radosgw-2:18.2.4-855.g19e2a2be.el9.x86_64 30/32 2024-09-21T15:46:37.251 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T15:46:37.252 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-21T15:46:37.252 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T15:46:37.252 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T15:46:37.252 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:46:37.253 INFO:teuthology.orchestra.run.smithi100.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-09-21T15:46:37.345 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-09-21T15:46:37.345 INFO:teuthology.orchestra.run.smithi100.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-09-21T15:46:37.531 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-selinux-2:18.2.4-855.g19e2a2be.el9.x86_64 29/32 2024-09-21T15:46:37.531 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /sys 2024-09-21T15:46:37.531 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /proc 2024-09-21T15:46:37.531 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /mnt 2024-09-21T15:46:37.531 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /var/tmp 2024-09-21T15:46:37.531 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /home 2024-09-21T15:46:37.531 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /root 2024-09-21T15:46:37.531 INFO:teuthology.orchestra.run.smithi089.stdout:skipping the directory /tmp 2024-09-21T15:46:37.532 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:46:38.333 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-radosgw-2:18.2.4-855.g19e2a2be.el9.x86_64 30/32 2024-09-21T15:46:38.358 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-radosgw-2:18.2.4-855.g19e2a2be.el9.x86_64 30/32 2024-09-21T15:46:38.358 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T15:46:38.358 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-21T15:46:38.359 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T15:46:38.359 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T15:46:38.359 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:46:38.360 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-09-21T15:46:38.469 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 31/32 2024-09-21T15:46:38.469 INFO:teuthology.orchestra.run.smithi089.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 32/32 2024-09-21T15:46:38.492 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-09-21T15:46:38.493 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-base-2:18.2.4-855.g19e2a2be.el9.x86_64 1/32 2024-09-21T15:46:38.493 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-common-2:18.2.4-855.g19e2a2be.el9.x86_64 2/32 2024-09-21T15:46:38.493 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-radosgw-2:18.2.4-855.g19e2a2be.el9.x86_64 3/32 2024-09-21T15:46:38.493 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-selinux-2:18.2.4-855.g19e2a2be.el9.x86_64 4/32 2024-09-21T15:46:38.493 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libcephfs2-2:18.2.4-855.g19e2a2be.el9.x86_64 5/32 2024-09-21T15:46:38.493 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libradosstriper1-2:18.2.4-855.g19e2a2be.el9.x86_64 6/32 2024-09-21T15:46:38.493 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librgw2-2:18.2.4-855.g19e2a2be.el9.x86_64 7/32 2024-09-21T15:46:38.493 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-ceph-argparse-2:18.2.4-855.g19e2a2be.el9.x 8/32 2024-09-21T15:46:38.493 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-ceph-common-2:18.2.4-855.g19e2a2be.el9.x86 9/32 2024-09-21T15:46:38.493 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cephfs-2:18.2.4-855.g19e2a2be.el9.x86_64 10/32 2024-09-21T15:46:38.493 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-rados-2:18.2.4-855.g19e2a2be.el9.x86_64 11/32 2024-09-21T15:46:38.493 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-rbd-2:18.2.4-855.g19e2a2be.el9.x86_64 12/32 2024-09-21T15:46:38.494 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-rgw-2:18.2.4-855.g19e2a2be.el9.x86_64 13/32 2024-09-21T15:46:38.494 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-09-21T15:46:38.494 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-09-21T15:46:38.494 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-09-21T15:46:38.494 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-09-21T15:46:38.494 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-09-21T15:46:38.494 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-09-21T15:46:38.494 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-09-21T15:46:38.494 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 21/32 2024-09-21T15:46:38.494 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 22/32 2024-09-21T15:46:38.494 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 23/32 2024-09-21T15:46:38.494 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-09-21T15:46:38.494 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-09-21T15:46:38.494 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 26/32 2024-09-21T15:46:38.495 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : re2-1:20211101-20.el9.x86_64 27/32 2024-09-21T15:46:38.495 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-09-21T15:46:38.495 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librados2-2:18.2.4-855.g19e2a2be.el9.x86_64 29/32 2024-09-21T15:46:38.495 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-09-21T15:46:38.495 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librbd1-2:18.2.4-855.g19e2a2be.el9.x86_64 31/32 2024-09-21T15:46:39.618 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 32/32 2024-09-21T15:46:39.618 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-base-2:18.2.4-855.g19e2a2be.el9.x86_64 1/32 2024-09-21T15:46:39.619 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-common-2:18.2.4-855.g19e2a2be.el9.x86_64 2/32 2024-09-21T15:46:39.619 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-radosgw-2:18.2.4-855.g19e2a2be.el9.x86_64 3/32 2024-09-21T15:46:39.619 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-selinux-2:18.2.4-855.g19e2a2be.el9.x86_64 4/32 2024-09-21T15:46:39.619 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libcephfs2-2:18.2.4-855.g19e2a2be.el9.x86_64 5/32 2024-09-21T15:46:39.619 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libradosstriper1-2:18.2.4-855.g19e2a2be.el9.x86_64 6/32 2024-09-21T15:46:39.619 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librgw2-2:18.2.4-855.g19e2a2be.el9.x86_64 7/32 2024-09-21T15:46:39.619 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-ceph-argparse-2:18.2.4-855.g19e2a2be.el9.x 8/32 2024-09-21T15:46:39.619 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-ceph-common-2:18.2.4-855.g19e2a2be.el9.x86 9/32 2024-09-21T15:46:39.619 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cephfs-2:18.2.4-855.g19e2a2be.el9.x86_64 10/32 2024-09-21T15:46:39.619 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rados-2:18.2.4-855.g19e2a2be.el9.x86_64 11/32 2024-09-21T15:46:39.619 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rbd-2:18.2.4-855.g19e2a2be.el9.x86_64 12/32 2024-09-21T15:46:39.620 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rgw-2:18.2.4-855.g19e2a2be.el9.x86_64 13/32 2024-09-21T15:46:39.620 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/32 2024-09-21T15:46:39.620 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/32 2024-09-21T15:46:39.620 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/32 2024-09-21T15:46:39.620 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/32 2024-09-21T15:46:39.620 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/32 2024-09-21T15:46:39.620 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/32 2024-09-21T15:46:39.620 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : fmt-8.1.1-5.el9.x86_64 20/32 2024-09-21T15:46:39.620 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 21/32 2024-09-21T15:46:39.620 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 22/32 2024-09-21T15:46:39.620 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 23/32 2024-09-21T15:46:39.620 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 24/32 2024-09-21T15:46:39.621 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 25/32 2024-09-21T15:46:39.621 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 26/32 2024-09-21T15:46:39.621 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : re2-1:20211101-20.el9.x86_64 27/32 2024-09-21T15:46:39.621 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 28/32 2024-09-21T15:46:39.621 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librados2-2:18.2.4-855.g19e2a2be.el9.x86_64 29/32 2024-09-21T15:46:39.621 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 30/32 2024-09-21T15:46:39.621 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librbd1-2:18.2.4-855.g19e2a2be.el9.x86_64 31/32 2024-09-21T15:46:39.658 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-09-21T15:46:39.658 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:46:39.658 INFO:teuthology.orchestra.run.smithi100.stdout:Upgraded: 2024-09-21T15:46:39.658 INFO:teuthology.orchestra.run.smithi100.stdout: librados2-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:46:39.658 INFO:teuthology.orchestra.run.smithi100.stdout: librbd1-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:46:39.658 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-21T15:46:39.658 INFO:teuthology.orchestra.run.smithi100.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-21T15:46:39.658 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-base-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:46:39.658 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-common-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:46:39.659 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-radosgw-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:46:39.659 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-selinux-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:46:39.659 INFO:teuthology.orchestra.run.smithi100.stdout: fmt-8.1.1-5.el9.x86_64 2024-09-21T15:46:39.659 INFO:teuthology.orchestra.run.smithi100.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-21T15:46:39.659 INFO:teuthology.orchestra.run.smithi100.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-21T15:46:39.659 INFO:teuthology.orchestra.run.smithi100.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-21T15:46:39.659 INFO:teuthology.orchestra.run.smithi100.stdout: libcephfs2-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:46:39.659 INFO:teuthology.orchestra.run.smithi100.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-21T15:46:39.659 INFO:teuthology.orchestra.run.smithi100.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-21T15:46:39.659 INFO:teuthology.orchestra.run.smithi100.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-21T15:46:39.659 INFO:teuthology.orchestra.run.smithi100.stdout: libradosstriper1-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:46:39.659 INFO:teuthology.orchestra.run.smithi100.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-21T15:46:39.660 INFO:teuthology.orchestra.run.smithi100.stdout: librgw2-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:46:39.660 INFO:teuthology.orchestra.run.smithi100.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-21T15:46:39.660 INFO:teuthology.orchestra.run.smithi100.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-21T15:46:39.660 INFO:teuthology.orchestra.run.smithi100.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-21T15:46:39.660 INFO:teuthology.orchestra.run.smithi100.stdout: python3-ceph-argparse-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:46:39.660 INFO:teuthology.orchestra.run.smithi100.stdout: python3-ceph-common-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:46:39.660 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cephfs-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:46:39.660 INFO:teuthology.orchestra.run.smithi100.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-21T15:46:39.660 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rados-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:46:39.660 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rbd-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:46:39.660 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rgw-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:46:39.660 INFO:teuthology.orchestra.run.smithi100.stdout: re2-1:20211101-20.el9.x86_64 2024-09-21T15:46:39.661 INFO:teuthology.orchestra.run.smithi100.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-21T15:46:39.661 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:46:39.661 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-21T15:46:40.019 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-test 2024-09-21T15:46:40.560 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 32/32 2024-09-21T15:46:40.560 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:46:40.560 INFO:teuthology.orchestra.run.smithi089.stdout:Upgraded: 2024-09-21T15:46:40.560 INFO:teuthology.orchestra.run.smithi089.stdout: librados2-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:46:40.560 INFO:teuthology.orchestra.run.smithi089.stdout: librbd1-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:46:40.560 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-09-21T15:46:40.561 INFO:teuthology.orchestra.run.smithi089.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-21T15:46:40.561 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-base-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:46:40.561 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-common-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:46:40.561 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-radosgw-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:46:40.561 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-selinux-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:46:40.561 INFO:teuthology.orchestra.run.smithi089.stdout: fmt-8.1.1-5.el9.x86_64 2024-09-21T15:46:40.561 INFO:teuthology.orchestra.run.smithi089.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-21T15:46:40.561 INFO:teuthology.orchestra.run.smithi089.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-21T15:46:40.561 INFO:teuthology.orchestra.run.smithi089.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-21T15:46:40.561 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs2-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:46:40.561 INFO:teuthology.orchestra.run.smithi089.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-21T15:46:40.561 INFO:teuthology.orchestra.run.smithi089.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-21T15:46:40.561 INFO:teuthology.orchestra.run.smithi089.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-21T15:46:40.561 INFO:teuthology.orchestra.run.smithi089.stdout: libradosstriper1-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:46:40.562 INFO:teuthology.orchestra.run.smithi089.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-21T15:46:40.562 INFO:teuthology.orchestra.run.smithi089.stdout: librgw2-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:46:40.562 INFO:teuthology.orchestra.run.smithi089.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-21T15:46:40.562 INFO:teuthology.orchestra.run.smithi089.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-21T15:46:40.562 INFO:teuthology.orchestra.run.smithi089.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-21T15:46:40.562 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-argparse-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:46:40.562 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ceph-common-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:46:40.562 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cephfs-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:46:40.562 INFO:teuthology.orchestra.run.smithi089.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-21T15:46:40.562 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rados-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:46:40.562 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rbd-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:46:40.562 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rgw-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:46:40.562 INFO:teuthology.orchestra.run.smithi089.stdout: re2-1:20211101-20.el9.x86_64 2024-09-21T15:46:40.563 INFO:teuthology.orchestra.run.smithi089.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-21T15:46:40.563 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:46:40.563 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-21T15:46:40.600 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:00:26 ago on Sat 21 Sep 2024 03:46:14 PM UTC. 2024-09-21T15:46:40.731 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-21T15:46:40.731 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-21T15:46:40.731 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-09-21T15:46:40.731 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-21T15:46:40.732 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-21T15:46:40.732 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-test x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 40 M 2024-09-21T15:46:40.732 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-09-21T15:46:40.732 INFO:teuthology.orchestra.run.smithi100.stdout: libcephsqlite x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 166 k 2024-09-21T15:46:40.732 INFO:teuthology.orchestra.run.smithi100.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-21T15:46:40.732 INFO:teuthology.orchestra.run.smithi100.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-21T15:46:40.732 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:46:40.732 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-21T15:46:40.732 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-21T15:46:40.732 INFO:teuthology.orchestra.run.smithi100.stdout:Install 4 Packages 2024-09-21T15:46:40.732 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:46:40.733 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 40 M 2024-09-21T15:46:40.733 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 164 M 2024-09-21T15:46:40.733 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-21T15:46:40.886 INFO:teuthology.orchestra.run.smithi100.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.3 MB/s | 304 kB 00:00 2024-09-21T15:46:40.913 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-test 2024-09-21T15:46:40.927 INFO:teuthology.orchestra.run.smithi100.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-21T15:46:40.977 INFO:teuthology.orchestra.run.smithi100.stdout:(3/4): libcephsqlite-18.2.4-855.g19e2a2be.el9.x 749 kB/s | 166 kB 00:00 2024-09-21T15:46:41.500 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:00:27 ago on Sat 21 Sep 2024 03:46:14 PM UTC. 2024-09-21T15:46:41.632 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-21T15:46:41.633 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-21T15:46:41.633 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-09-21T15:46:41.633 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-21T15:46:41.633 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-09-21T15:46:41.633 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-test x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 40 M 2024-09-21T15:46:41.633 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-09-21T15:46:41.633 INFO:teuthology.orchestra.run.smithi089.stdout: libcephsqlite x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 166 k 2024-09-21T15:46:41.634 INFO:teuthology.orchestra.run.smithi089.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-21T15:46:41.634 INFO:teuthology.orchestra.run.smithi089.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-21T15:46:41.634 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:46:41.634 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-09-21T15:46:41.634 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-21T15:46:41.634 INFO:teuthology.orchestra.run.smithi089.stdout:Install 4 Packages 2024-09-21T15:46:41.634 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:46:41.635 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 40 M 2024-09-21T15:46:41.635 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 164 M 2024-09-21T15:46:41.635 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-09-21T15:46:42.010 INFO:teuthology.orchestra.run.smithi089.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 2.5 MB/s | 304 kB 00:00 2024-09-21T15:46:42.035 INFO:teuthology.orchestra.run.smithi100.stdout:(4/4): ceph-test-18.2.4-855.g19e2a2be.el9.x86_6 31 MB/s | 40 MB 00:01 2024-09-21T15:46:42.038 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-21T15:46:42.038 INFO:teuthology.orchestra.run.smithi100.stdout:Total 31 MB/s | 40 MB 00:01 2024-09-21T15:46:42.052 INFO:teuthology.orchestra.run.smithi089.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-09-21T15:46:42.066 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-21T15:46:42.085 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-21T15:46:42.086 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-21T15:46:42.119 INFO:teuthology.orchestra.run.smithi089.stdout:(3/4): libcephsqlite-18.2.4-855.g19e2a2be.el9.x 725 kB/s | 166 kB 00:00 2024-09-21T15:46:42.407 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-21T15:46:42.408 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-21T15:46:42.932 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-21T15:46:43.003 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-21T15:46:43.052 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-21T15:46:43.088 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libcephsqlite-2:18.2.4-855.g19e2a2be.el9.x86_64 3/4 2024-09-21T15:46:43.145 INFO:teuthology.orchestra.run.smithi089.stdout:(4/4): ceph-test-18.2.4-855.g19e2a2be.el9.x86_6 32 MB/s | 40 MB 00:01 2024-09-21T15:46:43.147 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-09-21T15:46:43.147 INFO:teuthology.orchestra.run.smithi089.stdout:Total 27 MB/s | 40 MB 00:01 2024-09-21T15:46:43.175 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-09-21T15:46:43.195 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-09-21T15:46:43.195 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-09-21T15:46:43.524 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-09-21T15:46:43.525 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-09-21T15:46:44.037 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-09-21T15:46:44.124 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-21T15:46:44.169 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-21T15:46:44.197 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libcephsqlite-2:18.2.4-855.g19e2a2be.el9.x86_64 3/4 2024-09-21T15:46:46.771 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: libcephsqlite-2:18.2.4-855.g19e2a2be.el9.x86_64 3/4 2024-09-21T15:46:46.861 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-test-2:18.2.4-855.g19e2a2be.el9.x86_64 4/4 2024-09-21T15:46:47.908 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libcephsqlite-2:18.2.4-855.g19e2a2be.el9.x86_64 3/4 2024-09-21T15:46:47.987 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-test-2:18.2.4-855.g19e2a2be.el9.x86_64 4/4 2024-09-21T15:46:48.857 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-test-2:18.2.4-855.g19e2a2be.el9.x86_64 4/4 2024-09-21T15:46:48.858 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-test-2:18.2.4-855.g19e2a2be.el9.x86_64 1/4 2024-09-21T15:46:48.858 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libcephsqlite-2:18.2.4-855.g19e2a2be.el9.x86_64 2/4 2024-09-21T15:46:48.858 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-21T15:46:49.168 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-21T15:46:49.168 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:46:49.168 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-21T15:46:49.169 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-test-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:46:49.169 INFO:teuthology.orchestra.run.smithi100.stdout: libcephsqlite-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:46:49.169 INFO:teuthology.orchestra.run.smithi100.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-21T15:46:49.169 INFO:teuthology.orchestra.run.smithi100.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-21T15:46:49.169 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:46:49.169 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-21T15:46:49.549 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph 2024-09-21T15:46:49.939 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-test-2:18.2.4-855.g19e2a2be.el9.x86_64 4/4 2024-09-21T15:46:49.939 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-test-2:18.2.4-855.g19e2a2be.el9.x86_64 1/4 2024-09-21T15:46:49.939 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libcephsqlite-2:18.2.4-855.g19e2a2be.el9.x86_64 2/4 2024-09-21T15:46:49.939 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-21T15:46:50.135 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:00:36 ago on Sat 21 Sep 2024 03:46:14 PM UTC. 2024-09-21T15:46:50.265 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-21T15:46:50.265 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:46:50.265 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-09-21T15:46:50.265 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-test-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:46:50.265 INFO:teuthology.orchestra.run.smithi089.stdout: libcephsqlite-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:46:50.265 INFO:teuthology.orchestra.run.smithi089.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-21T15:46:50.265 INFO:teuthology.orchestra.run.smithi089.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-21T15:46:50.265 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:46:50.265 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-21T15:46:50.267 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-21T15:46:50.270 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-21T15:46:50.270 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-09-21T15:46:50.270 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-21T15:46:50.270 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-21T15:46:50.270 INFO:teuthology.orchestra.run.smithi100.stdout: ceph x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 6.5 k 2024-09-21T15:46:50.270 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-09-21T15:46:50.270 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mds x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 2.1 M 2024-09-21T15:46:50.270 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 1.5 M 2024-09-21T15:46:50.271 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-modules-core noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 246 k 2024-09-21T15:46:50.271 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mon x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 4.7 M 2024-09-21T15:46:50.271 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-osd x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 17 M 2024-09-21T15:46:50.271 INFO:teuthology.orchestra.run.smithi100.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-21T15:46:50.271 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-21T15:46:50.271 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-21T15:46:50.271 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-21T15:46:50.271 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-21T15:46:50.271 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-21T15:46:50.271 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-21T15:46:50.271 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-21T15:46:50.271 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-21T15:46:50.271 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-21T15:46:50.271 INFO:teuthology.orchestra.run.smithi100.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-21T15:46:50.272 INFO:teuthology.orchestra.run.smithi100.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-21T15:46:50.272 INFO:teuthology.orchestra.run.smithi100.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-21T15:46:50.272 INFO:teuthology.orchestra.run.smithi100.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-21T15:46:50.272 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-21T15:46:50.272 INFO:teuthology.orchestra.run.smithi100.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-21T15:46:50.272 INFO:teuthology.orchestra.run.smithi100.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-21T15:46:50.272 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-21T15:46:50.272 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-21T15:46:50.272 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-21T15:46:50.272 INFO:teuthology.orchestra.run.smithi100.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-21T15:46:50.272 INFO:teuthology.orchestra.run.smithi100.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-21T15:46:50.272 INFO:teuthology.orchestra.run.smithi100.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-21T15:46:50.272 INFO:teuthology.orchestra.run.smithi100.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-21T15:46:50.272 INFO:teuthology.orchestra.run.smithi100.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-21T15:46:50.272 INFO:teuthology.orchestra.run.smithi100.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-21T15:46:50.273 INFO:teuthology.orchestra.run.smithi100.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-21T15:46:50.273 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:46:50.273 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-21T15:46:50.273 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-21T15:46:50.273 INFO:teuthology.orchestra.run.smithi100.stdout:Install 33 Packages 2024-09-21T15:46:50.273 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:46:50.273 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 29 M 2024-09-21T15:46:50.273 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 104 M 2024-09-21T15:46:50.274 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-21T15:46:50.571 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph 2024-09-21T15:46:50.896 INFO:teuthology.orchestra.run.smithi100.stdout:(1/33): ceph-18.2.4-855.g19e2a2be.el9.x86_64.rp 47 kB/s | 6.5 kB 00:00 2024-09-21T15:46:51.054 INFO:teuthology.orchestra.run.smithi100.stdout:(2/33): ceph-mgr-18.2.4-855.g19e2a2be.el9.x86_6 4.9 MB/s | 1.5 MB 00:00 2024-09-21T15:46:51.113 INFO:teuthology.orchestra.run.smithi100.stdout:(3/33): ceph-mds-18.2.4-855.g19e2a2be.el9.x86_6 5.9 MB/s | 2.1 MB 00:00 2024-09-21T15:46:51.158 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:00:37 ago on Sat 21 Sep 2024 03:46:14 PM UTC. 2024-09-21T15:46:51.180 INFO:teuthology.orchestra.run.smithi100.stdout:(4/33): ceph-mgr-modules-core-18.2.4-855.g19e2a 3.6 MB/s | 246 kB 00:00 2024-09-21T15:46:51.295 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-21T15:46:51.297 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-21T15:46:51.297 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-09-21T15:46:51.297 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-21T15:46:51.297 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-09-21T15:46:51.297 INFO:teuthology.orchestra.run.smithi089.stdout: ceph x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 6.5 k 2024-09-21T15:46:51.297 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-09-21T15:46:51.297 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mds x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 2.1 M 2024-09-21T15:46:51.298 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 1.5 M 2024-09-21T15:46:51.298 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-modules-core noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 246 k 2024-09-21T15:46:51.298 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mon x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 4.7 M 2024-09-21T15:46:51.298 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-osd x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 17 M 2024-09-21T15:46:51.298 INFO:teuthology.orchestra.run.smithi089.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-21T15:46:51.298 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-21T15:46:51.298 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-21T15:46:51.298 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-21T15:46:51.299 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-21T15:46:51.299 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-21T15:46:51.299 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-21T15:46:51.299 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-21T15:46:51.299 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-21T15:46:51.299 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-21T15:46:51.299 INFO:teuthology.orchestra.run.smithi089.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-21T15:46:51.299 INFO:teuthology.orchestra.run.smithi089.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-21T15:46:51.299 INFO:teuthology.orchestra.run.smithi089.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-21T15:46:51.299 INFO:teuthology.orchestra.run.smithi089.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-21T15:46:51.299 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-21T15:46:51.299 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-21T15:46:51.299 INFO:teuthology.orchestra.run.smithi089.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-21T15:46:51.300 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-21T15:46:51.300 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-21T15:46:51.300 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-21T15:46:51.300 INFO:teuthology.orchestra.run.smithi089.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-21T15:46:51.300 INFO:teuthology.orchestra.run.smithi089.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-21T15:46:51.300 INFO:teuthology.orchestra.run.smithi089.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-21T15:46:51.300 INFO:teuthology.orchestra.run.smithi089.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-21T15:46:51.300 INFO:teuthology.orchestra.run.smithi089.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-21T15:46:51.300 INFO:teuthology.orchestra.run.smithi089.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-21T15:46:51.300 INFO:teuthology.orchestra.run.smithi089.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-21T15:46:51.300 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:46:51.300 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-09-21T15:46:51.300 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-21T15:46:51.300 INFO:teuthology.orchestra.run.smithi089.stdout:Install 33 Packages 2024-09-21T15:46:51.301 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:46:51.301 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 29 M 2024-09-21T15:46:51.301 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 104 M 2024-09-21T15:46:51.301 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-09-21T15:46:51.305 INFO:teuthology.orchestra.run.smithi100.stdout:(5/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.0 MB/s | 253 kB 00:00 2024-09-21T15:46:51.389 INFO:teuthology.orchestra.run.smithi100.stdout:(6/33): python3-cryptography-36.0.1-4.el9.x86_6 15 MB/s | 1.2 MB 00:00 2024-09-21T15:46:51.464 INFO:teuthology.orchestra.run.smithi100.stdout:(7/33): ceph-mon-18.2.4-855.g19e2a2be.el9.x86_6 8.2 MB/s | 4.7 MB 00:00 2024-09-21T15:46:51.489 INFO:teuthology.orchestra.run.smithi100.stdout:(8/33): python3-ply-3.11-14.el9.noarch.rpm 1.0 MB/s | 106 kB 00:00 2024-09-21T15:46:51.531 INFO:teuthology.orchestra.run.smithi100.stdout:(9/33): python3-requests-2.25.1-8.el9.noarch.rp 2.9 MB/s | 125 kB 00:00 2024-09-21T15:46:51.581 INFO:teuthology.orchestra.run.smithi089.stdout:(1/33): ceph-18.2.4-855.g19e2a2be.el9.x86_64.rp 52 kB/s | 6.5 kB 00:00 2024-09-21T15:46:51.715 INFO:teuthology.orchestra.run.smithi100.stdout:(10/33): ceph-osd-18.2.4-855.g19e2a2be.el9.x86_ 25 MB/s | 17 MB 00:00 2024-09-21T15:46:51.740 INFO:teuthology.orchestra.run.smithi100.stdout:(11/33): python3-pycparser-2.20-6.el9.noarch.rp 490 kB/s | 135 kB 00:00 2024-09-21T15:46:51.774 INFO:teuthology.orchestra.run.smithi100.stdout:(12/33): python3-urllib3-1.26.5-6.el9.noarch.rp 886 kB/s | 215 kB 00:00 2024-09-21T15:46:51.798 INFO:teuthology.orchestra.run.smithi089.stdout:(2/33): ceph-mds-18.2.4-855.g19e2a2be.el9.x86_6 6.1 MB/s | 2.1 MB 00:00 2024-09-21T15:46:51.840 INFO:teuthology.orchestra.run.smithi089.stdout:(3/33): ceph-mgr-18.2.4-855.g19e2a2be.el9.x86_6 3.8 MB/s | 1.5 MB 00:00 2024-09-21T15:46:51.857 INFO:teuthology.orchestra.run.smithi100.stdout:(13/33): python3-markupsafe-1.1.1-12.el9.x86_64 296 kB/s | 35 kB 00:00 2024-09-21T15:46:51.908 INFO:teuthology.orchestra.run.smithi100.stdout:(14/33): python3-mako-1.1.4-6.el9.noarch.rpm 894 kB/s | 172 kB 00:00 2024-09-21T15:46:51.915 INFO:teuthology.orchestra.run.smithi089.stdout:(4/33): ceph-mgr-modules-core-18.2.4-855.g19e2a 3.2 MB/s | 246 kB 00:00 2024-09-21T15:46:51.933 INFO:teuthology.orchestra.run.smithi100.stdout:(15/33): python3-pytz-2021.1-5.el9.noarch.rpm 321 kB/s | 51 kB 00:00 2024-09-21T15:46:51.961 INFO:teuthology.orchestra.run.smithi100.stdout:(16/33): python3-toml-0.10.2-6.el9.noarch.rpm 401 kB/s | 42 kB 00:00 2024-09-21T15:46:52.012 INFO:teuthology.orchestra.run.smithi100.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 419 kB/s | 43 kB 00:00 2024-09-21T15:46:52.037 INFO:teuthology.orchestra.run.smithi100.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 459 kB/s | 11 kB 00:00 2024-09-21T15:46:52.049 INFO:teuthology.orchestra.run.smithi089.stdout:(5/33): ceph-mon-18.2.4-855.g19e2a2be.el9.x86_6 10 MB/s | 4.7 MB 00:00 2024-09-21T15:46:52.063 INFO:teuthology.orchestra.run.smithi100.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 708 kB/s | 18 kB 00:00 2024-09-21T15:46:52.079 INFO:teuthology.orchestra.run.smithi100.stdout:(20/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-09-21T15:46:52.105 INFO:teuthology.orchestra.run.smithi100.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 554 kB/s | 23 kB 00:00 2024-09-21T15:46:52.124 INFO:teuthology.orchestra.run.smithi089.stdout:(6/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.2 MB/s | 253 kB 00:00 2024-09-21T15:46:52.130 INFO:teuthology.orchestra.run.smithi100.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 386 kB/s | 19 kB 00:00 2024-09-21T15:46:52.155 INFO:teuthology.orchestra.run.smithi100.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 390 kB/s | 20 kB 00:00 2024-09-21T15:46:52.181 INFO:teuthology.orchestra.run.smithi100.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 919 kB/s | 46 kB 00:00 2024-09-21T15:46:52.206 INFO:teuthology.orchestra.run.smithi100.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-09-21T15:46:52.231 INFO:teuthology.orchestra.run.smithi100.stdout:(26/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.4 MB/s | 358 kB 00:00 2024-09-21T15:46:52.256 INFO:teuthology.orchestra.run.smithi100.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-09-21T15:46:52.282 INFO:teuthology.orchestra.run.smithi100.stdout:(28/33): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-09-21T15:46:52.316 INFO:teuthology.orchestra.run.smithi100.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.0 MB/s | 90 kB 00:00 2024-09-21T15:46:52.333 INFO:teuthology.orchestra.run.smithi089.stdout:(7/33): ceph-osd-18.2.4-855.g19e2a2be.el9.x86_6 31 MB/s | 17 MB 00:00 2024-09-21T15:46:52.341 INFO:teuthology.orchestra.run.smithi100.stdout:(30/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-09-21T15:46:52.358 INFO:teuthology.orchestra.run.smithi089.stdout:(8/33): python3-ply-3.11-14.el9.noarch.rpm 455 kB/s | 106 kB 00:00 2024-09-21T15:46:52.366 INFO:teuthology.orchestra.run.smithi100.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 802 kB/s | 20 kB 00:00 2024-09-21T15:46:52.392 INFO:teuthology.orchestra.run.smithi100.stdout:(32/33): python3-webob-1.8.8-2.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-09-21T15:46:52.400 INFO:teuthology.orchestra.run.smithi089.stdout:(9/33): python3-cryptography-36.0.1-4.el9.x86_6 3.5 MB/s | 1.2 MB 00:00 2024-09-21T15:46:52.416 INFO:teuthology.orchestra.run.smithi100.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-09-21T15:46:52.421 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-21T15:46:52.421 INFO:teuthology.orchestra.run.smithi100.stdout:Total 14 MB/s | 29 MB 00:02 2024-09-21T15:46:52.451 INFO:teuthology.orchestra.run.smithi089.stdout:(10/33): python3-requests-2.25.1-8.el9.noarch.r 1.3 MB/s | 125 kB 00:00 2024-09-21T15:46:52.476 INFO:teuthology.orchestra.run.smithi089.stdout:(11/33): python3-pycparser-2.20-6.el9.noarch.rp 949 kB/s | 135 kB 00:00 2024-09-21T15:46:52.501 INFO:teuthology.orchestra.run.smithi089.stdout:(12/33): python3-urllib3-1.26.5-6.el9.noarch.rp 2.1 MB/s | 215 kB 00:00 2024-09-21T15:46:52.539 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-21T15:46:52.577 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-21T15:46:52.578 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-21T15:46:52.635 INFO:teuthology.orchestra.run.smithi089.stdout:(13/33): python3-markupsafe-1.1.1-12.el9.x86_64 219 kB/s | 35 kB 00:00 2024-09-21T15:46:52.677 INFO:teuthology.orchestra.run.smithi089.stdout:(14/33): python3-pytz-2021.1-5.el9.noarch.rpm 291 kB/s | 51 kB 00:00 2024-09-21T15:46:52.702 INFO:teuthology.orchestra.run.smithi089.stdout:(15/33): python3-mako-1.1.4-6.el9.noarch.rpm 685 kB/s | 172 kB 00:00 2024-09-21T15:46:52.744 INFO:teuthology.orchestra.run.smithi089.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 648 kB/s | 43 kB 00:00 2024-09-21T15:46:52.778 INFO:teuthology.orchestra.run.smithi089.stdout:(17/33): python3-cherrypy-18.6.1-2.el9.noarch.r 11 MB/s | 358 kB 00:00 2024-09-21T15:46:52.803 INFO:teuthology.orchestra.run.smithi089.stdout:(18/33): python3-toml-0.10.2-6.el9.noarch.rpm 248 kB/s | 42 kB 00:00 2024-09-21T15:46:52.837 INFO:teuthology.orchestra.run.smithi089.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-09-21T15:46:52.862 INFO:teuthology.orchestra.run.smithi089.stdout:(20/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 128 kB/s | 11 kB 00:00 2024-09-21T15:46:52.896 INFO:teuthology.orchestra.run.smithi089.stdout:(21/33): python3-jaraco-functools-3.5.0-2.el9.n 744 kB/s | 19 kB 00:00 2024-09-21T15:46:52.921 INFO:teuthology.orchestra.run.smithi089.stdout:(22/33): python3-jaraco-collections-3.0.0-8.el9 451 kB/s | 23 kB 00:00 2024-09-21T15:46:52.946 INFO:teuthology.orchestra.run.smithi089.stdout:(23/33): python3-jaraco-classes-3.2.1-5.el9.noa 124 kB/s | 18 kB 00:00 2024-09-21T15:46:52.971 INFO:teuthology.orchestra.run.smithi089.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 267 kB/s | 20 kB 00:00 2024-09-21T15:46:52.997 INFO:teuthology.orchestra.run.smithi089.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 614 kB/s | 46 kB 00:00 2024-09-21T15:46:52.999 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-21T15:46:53.000 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-21T15:46:53.022 INFO:teuthology.orchestra.run.smithi089.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 658 kB/s | 16 kB 00:00 2024-09-21T15:46:53.047 INFO:teuthology.orchestra.run.smithi089.stdout:(27/33): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-09-21T15:46:53.077 INFO:teuthology.orchestra.run.smithi089.stdout:(28/33): python3-tempora-5.0.0-2.el9.noarch.rpm 1.2 MB/s | 36 kB 00:00 2024-09-21T15:46:53.102 INFO:teuthology.orchestra.run.smithi089.stdout:(29/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.1 MB/s | 90 kB 00:00 2024-09-21T15:46:53.136 INFO:teuthology.orchestra.run.smithi089.stdout:(30/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.6 MB/s | 272 kB 00:00 2024-09-21T15:46:53.169 INFO:teuthology.orchestra.run.smithi089.stdout:(31/33): python3-webob-1.8.8-2.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-09-21T15:46:53.194 INFO:teuthology.orchestra.run.smithi089.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 347 kB/s | 20 kB 00:00 2024-09-21T15:46:53.219 INFO:teuthology.orchestra.run.smithi089.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.6 MB/s | 427 kB 00:00 2024-09-21T15:46:53.224 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-09-21T15:46:53.224 INFO:teuthology.orchestra.run.smithi089.stdout:Total 15 MB/s | 29 MB 00:01 2024-09-21T15:46:53.347 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-09-21T15:46:53.385 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-09-21T15:46:53.385 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-09-21T15:46:53.787 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-09-21T15:46:53.787 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-09-21T15:46:54.251 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-21T15:46:54.320 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-09-21T15:46:54.352 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-09-21T15:46:54.384 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-09-21T15:46:54.475 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-09-21T15:46:54.554 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-09-21T15:46:54.596 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/33 2024-09-21T15:46:54.631 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-09-21T15:46:54.661 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-09-21T15:46:54.695 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-09-21T15:46:54.728 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-09-21T15:46:54.769 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-09-21T15:46:54.812 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-09-21T15:46:54.847 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-09-21T15:46:54.885 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-09-21T15:46:54.966 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-09-21T15:46:55.072 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-09-21T15:46:55.130 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-09-21T15:46:55.131 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-09-21T15:46:55.164 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-09-21T15:46:55.205 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-09-21T15:46:55.210 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-09-21T15:46:55.265 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 18/33 2024-09-21T15:46:55.299 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-09-21T15:46:55.313 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-09-21T15:46:55.361 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-09-21T15:46:55.394 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-09-21T15:46:55.425 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-09-21T15:46:55.442 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 6/33 2024-09-21T15:46:55.477 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-09-21T15:46:55.507 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-09-21T15:46:55.541 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-09-21T15:46:55.575 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-09-21T15:46:55.612 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-09-21T15:46:55.616 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-09-21T15:46:55.659 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-09-21T15:46:55.659 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-09-21T15:46:55.701 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-09-21T15:46:55.737 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-09-21T15:46:55.740 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-09-21T15:46:55.821 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-09-21T15:46:55.826 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-09-21T15:46:55.866 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-09-21T15:46:55.969 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-09-21T15:46:55.983 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-09-21T15:46:56.048 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-09-21T15:46:56.101 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 18/33 2024-09-21T15:46:56.140 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-09-21T15:46:56.167 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-modules-core-2:18.2.4-855.g19e2a2be.el9.n 28/33 2024-09-21T15:46:56.190 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-09-21T15:46:56.193 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-2:18.2.4-855.g19e2a2be.el9.x86_64 29/33 2024-09-21T15:46:56.220 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mgr-2:18.2.4-855.g19e2a2be.el9.x86_64 29/33 2024-09-21T15:46:56.220 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T15:46:56.220 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-21T15:46:56.220 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T15:46:56.221 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T15:46:56.221 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:46:56.244 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-09-21T15:46:56.438 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-09-21T15:46:56.502 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-09-21T15:46:56.579 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-09-21T15:46:56.676 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-09-21T15:46:56.709 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-09-21T15:46:56.825 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-09-21T15:46:56.992 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-modules-core-2:18.2.4-855.g19e2a2be.el9.n 28/33 2024-09-21T15:46:57.020 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-2:18.2.4-855.g19e2a2be.el9.x86_64 29/33 2024-09-21T15:46:57.047 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-2:18.2.4-855.g19e2a2be.el9.x86_64 29/33 2024-09-21T15:46:57.047 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T15:46:57.047 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-21T15:46:57.047 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T15:46:57.047 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T15:46:57.048 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:46:57.670 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-osd-2:18.2.4-855.g19e2a2be.el9.x86_64 30/33 2024-09-21T15:46:57.699 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-osd-2:18.2.4-855.g19e2a2be.el9.x86_64 30/33 2024-09-21T15:46:57.699 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T15:46:57.699 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-21T15:46:57.699 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T15:46:57.700 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T15:46:57.700 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:46:58.146 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mon-2:18.2.4-855.g19e2a2be.el9.x86_64 31/33 2024-09-21T15:46:58.169 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mon-2:18.2.4-855.g19e2a2be.el9.x86_64 31/33 2024-09-21T15:46:58.170 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T15:46:58.170 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-21T15:46:58.170 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T15:46:58.170 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T15:46:58.170 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:46:58.372 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mds-2:18.2.4-855.g19e2a2be.el9.x86_64 32/33 2024-09-21T15:46:58.395 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mds-2:18.2.4-855.g19e2a2be.el9.x86_64 32/33 2024-09-21T15:46:58.395 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T15:46:58.395 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-21T15:46:58.395 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T15:46:58.395 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T15:46:58.396 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:46:58.451 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-2:18.2.4-855.g19e2a2be.el9.x86_64 33/33 2024-09-21T15:46:58.454 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-osd-2:18.2.4-855.g19e2a2be.el9.x86_64 30/33 2024-09-21T15:46:58.482 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-osd-2:18.2.4-855.g19e2a2be.el9.x86_64 30/33 2024-09-21T15:46:58.482 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T15:46:58.483 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-21T15:46:58.483 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T15:46:58.483 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T15:46:58.483 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:46:58.930 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mon-2:18.2.4-855.g19e2a2be.el9.x86_64 31/33 2024-09-21T15:46:58.954 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mon-2:18.2.4-855.g19e2a2be.el9.x86_64 31/33 2024-09-21T15:46:58.954 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T15:46:58.954 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-21T15:46:58.954 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T15:46:58.954 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T15:46:58.955 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:46:59.164 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mds-2:18.2.4-855.g19e2a2be.el9.x86_64 32/33 2024-09-21T15:46:59.188 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mds-2:18.2.4-855.g19e2a2be.el9.x86_64 32/33 2024-09-21T15:46:59.188 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T15:46:59.188 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-21T15:46:59.188 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T15:46:59.188 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T15:46:59.189 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:46:59.253 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-2:18.2.4-855.g19e2a2be.el9.x86_64 33/33 2024-09-21T15:47:00.298 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-2:18.2.4-855.g19e2a2be.el9.x86_64 33/33 2024-09-21T15:47:00.298 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-2:18.2.4-855.g19e2a2be.el9.x86_64 1/33 2024-09-21T15:47:00.298 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mds-2:18.2.4-855.g19e2a2be.el9.x86_64 2/33 2024-09-21T15:47:00.298 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-2:18.2.4-855.g19e2a2be.el9.x86_64 3/33 2024-09-21T15:47:00.298 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mon-2:18.2.4-855.g19e2a2be.el9.x86_64 4/33 2024-09-21T15:47:00.298 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-osd-2:18.2.4-855.g19e2a2be.el9.x86_64 5/33 2024-09-21T15:47:00.298 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-modules-core-2:18.2.4-855.g19e2a2be.el9.n 6/33 2024-09-21T15:47:00.298 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-09-21T15:47:00.299 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-09-21T15:47:00.299 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-09-21T15:47:00.299 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-09-21T15:47:00.299 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-09-21T15:47:00.299 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/33 2024-09-21T15:47:00.299 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-09-21T15:47:00.299 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-09-21T15:47:00.299 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-09-21T15:47:00.299 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-09-21T15:47:00.299 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-09-21T15:47:00.299 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-09-21T15:47:00.299 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-09-21T15:47:00.299 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-09-21T15:47:00.299 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-09-21T15:47:00.300 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-09-21T15:47:00.300 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-09-21T15:47:00.300 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-09-21T15:47:00.300 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-09-21T15:47:00.301 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-09-21T15:47:00.301 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-09-21T15:47:00.301 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-09-21T15:47:00.301 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-09-21T15:47:00.301 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-09-21T15:47:00.301 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 31/33 2024-09-21T15:47:00.301 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-09-21T15:47:01.124 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-2:18.2.4-855.g19e2a2be.el9.x86_64 33/33 2024-09-21T15:47:01.125 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-2:18.2.4-855.g19e2a2be.el9.x86_64 1/33 2024-09-21T15:47:01.125 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mds-2:18.2.4-855.g19e2a2be.el9.x86_64 2/33 2024-09-21T15:47:01.125 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-2:18.2.4-855.g19e2a2be.el9.x86_64 3/33 2024-09-21T15:47:01.125 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mon-2:18.2.4-855.g19e2a2be.el9.x86_64 4/33 2024-09-21T15:47:01.125 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-osd-2:18.2.4-855.g19e2a2be.el9.x86_64 5/33 2024-09-21T15:47:01.125 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-modules-core-2:18.2.4-855.g19e2a2be.el9.n 6/33 2024-09-21T15:47:01.125 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-09-21T15:47:01.125 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-09-21T15:47:01.125 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-09-21T15:47:01.125 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-09-21T15:47:01.125 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-09-21T15:47:01.125 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/33 2024-09-21T15:47:01.125 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-09-21T15:47:01.125 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-09-21T15:47:01.125 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-09-21T15:47:01.126 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-09-21T15:47:01.126 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-09-21T15:47:01.126 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-09-21T15:47:01.126 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-09-21T15:47:01.126 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-09-21T15:47:01.126 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-09-21T15:47:01.126 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-09-21T15:47:01.126 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-09-21T15:47:01.126 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-09-21T15:47:01.126 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-09-21T15:47:01.126 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-09-21T15:47:01.126 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-09-21T15:47:01.126 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-09-21T15:47:01.127 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-09-21T15:47:01.127 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-09-21T15:47:01.127 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 31/33 2024-09-21T15:47:01.127 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-09-21T15:47:01.306 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-09-21T15:47:01.306 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:01.307 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-21T15:47:01.307 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:47:01.307 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mds-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:47:01.307 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:47:01.307 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-modules-core-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-21T15:47:01.307 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mon-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:47:01.307 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-osd-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:47:01.307 INFO:teuthology.orchestra.run.smithi100.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-21T15:47:01.307 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-21T15:47:01.307 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-21T15:47:01.307 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-21T15:47:01.307 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-21T15:47:01.308 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-21T15:47:01.308 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-21T15:47:01.308 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-21T15:47:01.308 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-21T15:47:01.308 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-21T15:47:01.308 INFO:teuthology.orchestra.run.smithi100.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-21T15:47:01.308 INFO:teuthology.orchestra.run.smithi100.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-21T15:47:01.308 INFO:teuthology.orchestra.run.smithi100.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-21T15:47:01.308 INFO:teuthology.orchestra.run.smithi100.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-21T15:47:01.308 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-21T15:47:01.308 INFO:teuthology.orchestra.run.smithi100.stdout: python3-ply-3.11-14.el9.noarch 2024-09-21T15:47:01.308 INFO:teuthology.orchestra.run.smithi100.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-21T15:47:01.308 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-21T15:47:01.308 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-21T15:47:01.308 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-21T15:47:01.309 INFO:teuthology.orchestra.run.smithi100.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-21T15:47:01.309 INFO:teuthology.orchestra.run.smithi100.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-21T15:47:01.309 INFO:teuthology.orchestra.run.smithi100.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-21T15:47:01.309 INFO:teuthology.orchestra.run.smithi100.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-21T15:47:01.309 INFO:teuthology.orchestra.run.smithi100.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-21T15:47:01.309 INFO:teuthology.orchestra.run.smithi100.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-21T15:47:01.309 INFO:teuthology.orchestra.run.smithi100.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-21T15:47:01.309 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:01.309 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-21T15:47:01.685 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-base 2024-09-21T15:47:02.111 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-09-21T15:47:02.111 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:02.111 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-09-21T15:47:02.111 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:47:02.111 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mds-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:47:02.111 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:47:02.111 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-modules-core-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-21T15:47:02.112 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mon-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:47:02.112 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-osd-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:47:02.112 INFO:teuthology.orchestra.run.smithi089.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-21T15:47:02.112 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-21T15:47:02.112 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-21T15:47:02.112 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-21T15:47:02.112 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-21T15:47:02.112 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-21T15:47:02.112 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-21T15:47:02.112 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-21T15:47:02.112 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-21T15:47:02.112 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-21T15:47:02.112 INFO:teuthology.orchestra.run.smithi089.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-21T15:47:02.113 INFO:teuthology.orchestra.run.smithi089.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-21T15:47:02.113 INFO:teuthology.orchestra.run.smithi089.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-21T15:47:02.113 INFO:teuthology.orchestra.run.smithi089.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-21T15:47:02.113 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-21T15:47:02.113 INFO:teuthology.orchestra.run.smithi089.stdout: python3-ply-3.11-14.el9.noarch 2024-09-21T15:47:02.113 INFO:teuthology.orchestra.run.smithi089.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-21T15:47:02.113 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-21T15:47:02.113 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-21T15:47:02.113 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-21T15:47:02.113 INFO:teuthology.orchestra.run.smithi089.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-21T15:47:02.113 INFO:teuthology.orchestra.run.smithi089.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-21T15:47:02.113 INFO:teuthology.orchestra.run.smithi089.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-21T15:47:02.113 INFO:teuthology.orchestra.run.smithi089.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-21T15:47:02.113 INFO:teuthology.orchestra.run.smithi089.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-21T15:47:02.113 INFO:teuthology.orchestra.run.smithi089.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-21T15:47:02.113 INFO:teuthology.orchestra.run.smithi089.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-21T15:47:02.114 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:02.114 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-21T15:47:02.273 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:00:48 ago on Sat 21 Sep 2024 03:46:14 PM UTC. 2024-09-21T15:47:02.361 INFO:teuthology.orchestra.run.smithi100.stdout:Package ceph-base-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-21T15:47:02.402 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-21T15:47:02.403 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-09-21T15:47:02.403 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-21T15:47:02.465 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-base 2024-09-21T15:47:02.480 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install cephadm 2024-09-21T15:47:03.051 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:00:49 ago on Sat 21 Sep 2024 03:46:14 PM UTC. 2024-09-21T15:47:03.053 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:00:49 ago on Sat 21 Sep 2024 03:46:14 PM UTC. 2024-09-21T15:47:03.141 INFO:teuthology.orchestra.run.smithi089.stdout:Package ceph-base-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-21T15:47:03.183 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-21T15:47:03.183 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-21T15:47:03.186 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-09-21T15:47:03.186 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-21T15:47:03.186 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-21T15:47:03.187 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-09-21T15:47:03.187 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-21T15:47:03.187 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-21T15:47:03.187 INFO:teuthology.orchestra.run.smithi100.stdout: cephadm noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 226 k 2024-09-21T15:47:03.187 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:03.187 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-21T15:47:03.187 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-21T15:47:03.187 INFO:teuthology.orchestra.run.smithi100.stdout:Install 1 Package 2024-09-21T15:47:03.187 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:03.187 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 226 k 2024-09-21T15:47:03.187 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 217 k 2024-09-21T15:47:03.187 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-21T15:47:03.264 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install cephadm 2024-09-21T15:47:03.404 INFO:teuthology.orchestra.run.smithi100.stdout:cephadm-18.2.4-855.g19e2a2be.el9.noarch.rpm 1.0 MB/s | 226 kB 00:00 2024-09-21T15:47:03.404 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-21T15:47:03.404 INFO:teuthology.orchestra.run.smithi100.stdout:Total 1.0 MB/s | 226 kB 00:00 2024-09-21T15:47:03.404 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-21T15:47:03.409 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-21T15:47:03.410 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-21T15:47:03.412 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-21T15:47:03.412 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-21T15:47:03.481 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-21T15:47:03.832 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:00:49 ago on Sat 21 Sep 2024 03:46:14 PM UTC. 2024-09-21T15:47:03.950 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 1/1 2024-09-21T15:47:03.963 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-21T15:47:03.964 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-21T15:47:03.964 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-09-21T15:47:03.965 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-21T15:47:03.965 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-09-21T15:47:03.965 INFO:teuthology.orchestra.run.smithi089.stdout: cephadm noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 226 k 2024-09-21T15:47:03.965 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:03.965 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-09-21T15:47:03.965 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-21T15:47:03.965 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2024-09-21T15:47:03.965 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:03.965 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 226 k 2024-09-21T15:47:03.966 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 217 k 2024-09-21T15:47:03.966 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-09-21T15:47:04.034 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 1/1 2024-09-21T15:47:04.200 INFO:teuthology.orchestra.run.smithi089.stdout:cephadm-18.2.4-855.g19e2a2be.el9.noarch.rpm 964 kB/s | 226 kB 00:00 2024-09-21T15:47:04.201 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-09-21T15:47:04.201 INFO:teuthology.orchestra.run.smithi089.stdout:Total 957 kB/s | 226 kB 00:00 2024-09-21T15:47:04.201 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-09-21T15:47:04.206 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-09-21T15:47:04.206 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-09-21T15:47:04.209 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-09-21T15:47:04.209 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-09-21T15:47:04.279 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-09-21T15:47:04.384 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 1/1 2024-09-21T15:47:04.633 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 1/1 2024-09-21T15:47:04.633 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:04.633 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-21T15:47:04.633 INFO:teuthology.orchestra.run.smithi100.stdout: cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-21T15:47:04.633 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:04.634 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-21T15:47:04.739 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 1/1 2024-09-21T15:47:04.821 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 1/1 2024-09-21T15:47:04.860 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-immutable-object-cache 2024-09-21T15:47:05.166 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 1/1 2024-09-21T15:47:05.414 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 1/1 2024-09-21T15:47:05.414 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:05.414 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-09-21T15:47:05.415 INFO:teuthology.orchestra.run.smithi089.stdout: cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-21T15:47:05.415 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:05.415 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-21T15:47:05.454 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:00:51 ago on Sat 21 Sep 2024 03:46:14 PM UTC. 2024-09-21T15:47:05.589 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-21T15:47:05.590 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-21T15:47:05.590 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repo Size 2024-09-21T15:47:05.590 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-21T15:47:05.590 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-21T15:47:05.590 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-immutable-object-cache x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 143 k 2024-09-21T15:47:05.591 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:05.591 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-21T15:47:05.591 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-21T15:47:05.591 INFO:teuthology.orchestra.run.smithi100.stdout:Install 1 Package 2024-09-21T15:47:05.591 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:05.591 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 143 k 2024-09-21T15:47:05.591 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 395 k 2024-09-21T15:47:05.591 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-21T15:47:05.662 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-immutable-object-cache 2024-09-21T15:47:05.804 INFO:teuthology.orchestra.run.smithi100.stdout:ceph-immutable-object-cache-18.2.4-855.g19e2a2b 672 kB/s | 143 kB 00:00 2024-09-21T15:47:05.804 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-21T15:47:05.805 INFO:teuthology.orchestra.run.smithi100.stdout:Total 667 kB/s | 143 kB 00:00 2024-09-21T15:47:05.805 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-21T15:47:05.812 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-21T15:47:05.812 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-21T15:47:05.868 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-21T15:47:05.869 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-21T15:47:06.028 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-21T15:47:06.080 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-immutable-object-cache-2:18.2.4-855.g19e2a2be.e 1/1 2024-09-21T15:47:06.104 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.4-855.g19e2a2be.e 1/1 2024-09-21T15:47:06.104 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T15:47:06.104 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-21T15:47:06.104 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:06.256 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:00:52 ago on Sat 21 Sep 2024 03:46:14 PM UTC. 2024-09-21T15:47:06.391 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-21T15:47:06.391 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-21T15:47:06.391 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repo Size 2024-09-21T15:47:06.391 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-21T15:47:06.391 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-09-21T15:47:06.391 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-immutable-object-cache x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 143 k 2024-09-21T15:47:06.392 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:06.392 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-09-21T15:47:06.392 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-21T15:47:06.392 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2024-09-21T15:47:06.392 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:06.392 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 143 k 2024-09-21T15:47:06.392 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 395 k 2024-09-21T15:47:06.393 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-09-21T15:47:06.608 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-immutable-object-cache-18.2.4-855.g19e2a2b 662 kB/s | 143 kB 00:00 2024-09-21T15:47:06.609 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-09-21T15:47:06.609 INFO:teuthology.orchestra.run.smithi089.stdout:Total 659 kB/s | 143 kB 00:00 2024-09-21T15:47:06.609 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-09-21T15:47:06.616 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-09-21T15:47:06.616 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-09-21T15:47:06.672 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-09-21T15:47:06.672 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-09-21T15:47:06.826 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-09-21T15:47:06.884 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-immutable-object-cache-2:18.2.4-855.g19e2a2be.e 1/1 2024-09-21T15:47:06.908 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-immutable-object-cache-2:18.2.4-855.g19e2a2be.e 1/1 2024-09-21T15:47:06.908 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T15:47:06.908 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-21T15:47:06.908 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:07.025 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-immutable-object-cache-2:18.2.4-855.g19e2a2be.e 1/1 2024-09-21T15:47:07.025 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:07.025 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-21T15:47:07.025 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-immutable-object-cache-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:47:07.025 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:07.025 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-21T15:47:07.231 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-mgr 2024-09-21T15:47:07.824 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:00:53 ago on Sat 21 Sep 2024 03:46:14 PM UTC. 2024-09-21T15:47:07.828 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-immutable-object-cache-2:18.2.4-855.g19e2a2be.e 1/1 2024-09-21T15:47:07.828 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:07.829 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-09-21T15:47:07.829 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-immutable-object-cache-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:47:07.829 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:07.829 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-21T15:47:07.915 INFO:teuthology.orchestra.run.smithi100.stdout:Package ceph-mgr-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-21T15:47:07.958 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-21T15:47:07.959 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-09-21T15:47:07.959 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-21T15:47:08.028 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-mgr 2024-09-21T15:47:08.046 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-mgr-dashboard 2024-09-21T15:47:08.622 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:00:54 ago on Sat 21 Sep 2024 03:46:14 PM UTC. 2024-09-21T15:47:08.627 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:00:54 ago on Sat 21 Sep 2024 03:46:14 PM UTC. 2024-09-21T15:47:08.715 INFO:teuthology.orchestra.run.smithi089.stdout:Package ceph-mgr-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-21T15:47:08.753 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-21T15:47:08.754 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-21T15:47:08.754 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-09-21T15:47:08.754 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-21T15:47:08.755 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-21T15:47:08.755 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-dashboard noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 3.5 M 2024-09-21T15:47:08.755 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-09-21T15:47:08.755 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-grafana-dashboards noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 24 k 2024-09-21T15:47:08.755 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-prometheus-alerts noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 16 k 2024-09-21T15:47:08.755 INFO:teuthology.orchestra.run.smithi100.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-21T15:47:08.755 INFO:teuthology.orchestra.run.smithi100.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-21T15:47:08.755 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:08.755 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-21T15:47:08.755 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-21T15:47:08.755 INFO:teuthology.orchestra.run.smithi100.stdout:Install 5 Packages 2024-09-21T15:47:08.755 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:08.756 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 3.8 M 2024-09-21T15:47:08.756 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 87 M 2024-09-21T15:47:08.756 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-21T15:47:08.757 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-21T15:47:08.758 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-09-21T15:47:08.758 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-21T15:47:08.845 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-mgr-dashboard 2024-09-21T15:47:09.258 INFO:teuthology.orchestra.run.smithi100.stdout:(1/5): ceph-grafana-dashboards-18.2.4-855.g19e2 155 kB/s | 24 kB 00:00 2024-09-21T15:47:09.284 INFO:teuthology.orchestra.run.smithi100.stdout:(2/5): ceph-prometheus-alerts-18.2.4-855.g19e2a 90 kB/s | 16 kB 00:00 2024-09-21T15:47:09.334 INFO:teuthology.orchestra.run.smithi100.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 409 kB/s | 31 kB 00:00 2024-09-21T15:47:09.359 INFO:teuthology.orchestra.run.smithi100.stdout:(4/5): python3-routes-2.5.1-5.el9.noarch.rpm 2.4 MB/s | 188 kB 00:00 2024-09-21T15:47:09.419 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:00:55 ago on Sat 21 Sep 2024 03:46:14 PM UTC. 2024-09-21T15:47:09.526 INFO:teuthology.orchestra.run.smithi100.stdout:(5/5): ceph-mgr-dashboard-18.2.4-855.g19e2a2be. 8.3 MB/s | 3.5 MB 00:00 2024-09-21T15:47:09.529 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-21T15:47:09.530 INFO:teuthology.orchestra.run.smithi100.stdout:Total 4.9 MB/s | 3.8 MB 00:00 2024-09-21T15:47:09.530 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-21T15:47:09.537 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-21T15:47:09.537 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-21T15:47:09.550 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-21T15:47:09.551 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-21T15:47:09.551 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-09-21T15:47:09.551 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-21T15:47:09.551 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-09-21T15:47:09.551 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-dashboard noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 3.5 M 2024-09-21T15:47:09.551 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-09-21T15:47:09.551 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-grafana-dashboards noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 24 k 2024-09-21T15:47:09.551 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-prometheus-alerts noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 16 k 2024-09-21T15:47:09.551 INFO:teuthology.orchestra.run.smithi089.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-21T15:47:09.551 INFO:teuthology.orchestra.run.smithi089.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-21T15:47:09.551 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:09.552 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-09-21T15:47:09.552 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-21T15:47:09.552 INFO:teuthology.orchestra.run.smithi089.stdout:Install 5 Packages 2024-09-21T15:47:09.552 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:09.552 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 3.8 M 2024-09-21T15:47:09.552 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 87 M 2024-09-21T15:47:09.552 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-09-21T15:47:09.622 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-21T15:47:09.622 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-21T15:47:09.763 INFO:teuthology.orchestra.run.smithi089.stdout:(1/5): ceph-prometheus-alerts-18.2.4-855.g19e2a 111 kB/s | 16 kB 00:00 2024-09-21T15:47:09.788 INFO:teuthology.orchestra.run.smithi089.stdout:(2/5): ceph-grafana-dashboards-18.2.4-855.g19e2 140 kB/s | 24 kB 00:00 2024-09-21T15:47:09.831 INFO:teuthology.orchestra.run.smithi089.stdout:(3/5): python3-repoze-lru-0.7-16.el9.noarch.rpm 458 kB/s | 31 kB 00:00 2024-09-21T15:47:09.856 INFO:teuthology.orchestra.run.smithi089.stdout:(4/5): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2024-09-21T15:47:09.885 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-21T15:47:09.983 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-09-21T15:47:10.015 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-09-21T15:47:10.023 INFO:teuthology.orchestra.run.smithi089.stdout:(5/5): ceph-mgr-dashboard-18.2.4-855.g19e2a2be. 8.7 MB/s | 3.5 MB 00:00 2024-09-21T15:47:10.026 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-09-21T15:47:10.027 INFO:teuthology.orchestra.run.smithi089.stdout:Total 8.0 MB/s | 3.8 MB 00:00 2024-09-21T15:47:10.027 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-09-21T15:47:10.034 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-09-21T15:47:10.034 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-09-21T15:47:10.043 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-prometheus-alerts-2:18.2.4-855.g19e2a2be.el9.no 3/5 2024-09-21T15:47:10.115 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-09-21T15:47:10.115 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-09-21T15:47:10.376 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-09-21T15:47:10.470 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 1/5 2024-09-21T15:47:10.511 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 2/5 2024-09-21T15:47:10.540 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-prometheus-alerts-2:18.2.4-855.g19e2a2be.el9.no 3/5 2024-09-21T15:47:11.083 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-grafana-dashboards-2:18.2.4-855.g19e2a2be.el9.n 4/5 2024-09-21T15:47:11.113 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-dashboard-2:18.2.4-855.g19e2a2be.el9.noarch 5/5 2024-09-21T15:47:11.581 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-grafana-dashboards-2:18.2.4-855.g19e2a2be.el9.n 4/5 2024-09-21T15:47:11.619 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-dashboard-2:18.2.4-855.g19e2a2be.el9.noarch 5/5 2024-09-21T15:47:12.280 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.4-855.g19e2a2be.el9.noarch 5/5 2024-09-21T15:47:12.280 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-grafana-dashboards-2:18.2.4-855.g19e2a2be.el9.n 1/5 2024-09-21T15:47:12.281 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-dashboard-2:18.2.4-855.g19e2a2be.el9.noarch 2/5 2024-09-21T15:47:12.281 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-prometheus-alerts-2:18.2.4-855.g19e2a2be.el9.no 3/5 2024-09-21T15:47:12.281 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-09-21T15:47:12.564 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-09-21T15:47:12.564 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:12.565 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-21T15:47:12.565 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-grafana-dashboards-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-21T15:47:12.565 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-dashboard-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-21T15:47:12.565 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-prometheus-alerts-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-21T15:47:12.565 INFO:teuthology.orchestra.run.smithi100.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-21T15:47:12.565 INFO:teuthology.orchestra.run.smithi100.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-21T15:47:12.565 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:12.565 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-21T15:47:12.752 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-dashboard-2:18.2.4-855.g19e2a2be.el9.noarch 5/5 2024-09-21T15:47:12.752 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-grafana-dashboards-2:18.2.4-855.g19e2a2be.el9.n 1/5 2024-09-21T15:47:12.752 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-dashboard-2:18.2.4-855.g19e2a2be.el9.noarch 2/5 2024-09-21T15:47:12.752 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-prometheus-alerts-2:18.2.4-855.g19e2a2be.el9.no 3/5 2024-09-21T15:47:12.753 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 4/5 2024-09-21T15:47:12.805 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-21T15:47:13.008 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 5/5 2024-09-21T15:47:13.008 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:13.008 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-09-21T15:47:13.008 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-grafana-dashboards-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-21T15:47:13.008 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-dashboard-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-21T15:47:13.009 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-prometheus-alerts-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-21T15:47:13.009 INFO:teuthology.orchestra.run.smithi089.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-21T15:47:13.009 INFO:teuthology.orchestra.run.smithi089.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-21T15:47:13.009 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:13.009 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-21T15:47:13.261 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-21T15:47:13.376 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:00:59 ago on Sat 21 Sep 2024 03:46:14 PM UTC. 2024-09-21T15:47:13.508 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-21T15:47:13.509 INFO:teuthology.orchestra.run.smithi100.stdout:====================================================================================== 2024-09-21T15:47:13.509 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-09-21T15:47:13.509 INFO:teuthology.orchestra.run.smithi100.stdout:====================================================================================== 2024-09-21T15:47:13.510 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-21T15:47:13.510 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 7.4 M 2024-09-21T15:47:13.510 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-09-21T15:47:13.510 INFO:teuthology.orchestra.run.smithi100.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-21T15:47:13.510 INFO:teuthology.orchestra.run.smithi100.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-21T15:47:13.510 INFO:teuthology.orchestra.run.smithi100.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-21T15:47:13.510 INFO:teuthology.orchestra.run.smithi100.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-21T15:47:13.510 INFO:teuthology.orchestra.run.smithi100.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-21T15:47:13.510 INFO:teuthology.orchestra.run.smithi100.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-21T15:47:13.510 INFO:teuthology.orchestra.run.smithi100.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-21T15:47:13.510 INFO:teuthology.orchestra.run.smithi100.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-21T15:47:13.510 INFO:teuthology.orchestra.run.smithi100.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-21T15:47:13.510 INFO:teuthology.orchestra.run.smithi100.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-21T15:47:13.511 INFO:teuthology.orchestra.run.smithi100.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-21T15:47:13.511 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:13.511 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-21T15:47:13.511 INFO:teuthology.orchestra.run.smithi100.stdout:====================================================================================== 2024-09-21T15:47:13.511 INFO:teuthology.orchestra.run.smithi100.stdout:Install 12 Packages 2024-09-21T15:47:13.511 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:13.511 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 38 M 2024-09-21T15:47:13.511 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 212 M 2024-09-21T15:47:13.512 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-21T15:47:13.800 INFO:teuthology.orchestra.run.smithi100.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 873 kB/s | 191 kB 00:00 2024-09-21T15:47:13.839 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:00:59 ago on Sat 21 Sep 2024 03:46:14 PM UTC. 2024-09-21T15:47:13.867 INFO:teuthology.orchestra.run.smithi100.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 2.7 MB/s | 801 kB 00:00 2024-09-21T15:47:13.901 INFO:teuthology.orchestra.run.smithi100.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 326 kB/s | 33 kB 00:00 2024-09-21T15:47:13.969 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-21T15:47:13.971 INFO:teuthology.orchestra.run.smithi089.stdout:====================================================================================== 2024-09-21T15:47:13.971 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-09-21T15:47:13.971 INFO:teuthology.orchestra.run.smithi089.stdout:====================================================================================== 2024-09-21T15:47:13.971 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-09-21T15:47:13.971 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-diskprediction-local noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 7.4 M 2024-09-21T15:47:13.971 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-09-21T15:47:13.971 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-21T15:47:13.971 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-21T15:47:13.971 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-21T15:47:13.971 INFO:teuthology.orchestra.run.smithi089.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-21T15:47:13.971 INFO:teuthology.orchestra.run.smithi089.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-21T15:47:13.972 INFO:teuthology.orchestra.run.smithi089.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-21T15:47:13.972 INFO:teuthology.orchestra.run.smithi089.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-21T15:47:13.972 INFO:teuthology.orchestra.run.smithi089.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-21T15:47:13.972 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-21T15:47:13.972 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-21T15:47:13.972 INFO:teuthology.orchestra.run.smithi089.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-21T15:47:13.972 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:13.972 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-09-21T15:47:13.972 INFO:teuthology.orchestra.run.smithi089.stdout:====================================================================================== 2024-09-21T15:47:13.972 INFO:teuthology.orchestra.run.smithi089.stdout:Install 12 Packages 2024-09-21T15:47:13.972 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:13.973 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 38 M 2024-09-21T15:47:13.973 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 212 M 2024-09-21T15:47:13.973 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-09-21T15:47:14.059 INFO:teuthology.orchestra.run.smithi100.stdout:(4/12): ceph-mgr-diskprediction-local-18.2.4-85 15 MB/s | 7.4 MB 00:00 2024-09-21T15:47:14.084 INFO:teuthology.orchestra.run.smithi100.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 99 kB/s | 18 kB 00:00 2024-09-21T15:47:14.168 INFO:teuthology.orchestra.run.smithi100.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 9.9 MB/s | 3.0 MB 00:00 2024-09-21T15:47:14.193 INFO:teuthology.orchestra.run.smithi100.stdout:(7/12): openblas-0.3.26-2.el9.x86_64.rpm 288 kB/s | 38 kB 00:00 2024-09-21T15:47:14.236 INFO:teuthology.orchestra.run.smithi100.stdout:(8/12): python3-devel-3.9.19-8.el9.x86_64.rpm 3.6 MB/s | 245 kB 00:00 2024-09-21T15:47:14.319 INFO:teuthology.orchestra.run.smithi100.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.9 MB/s | 250 kB 00:00 2024-09-21T15:47:14.411 INFO:teuthology.orchestra.run.smithi100.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 15 MB/s | 4.9 MB 00:00 2024-09-21T15:47:14.495 INFO:teuthology.orchestra.run.smithi100.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 5.1 MB 00:00 2024-09-21T15:47:14.681 INFO:teuthology.orchestra.run.smithi089.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.1 MB/s | 191 kB 00:00 2024-09-21T15:47:14.723 INFO:teuthology.orchestra.run.smithi089.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.6 MB/s | 801 kB 00:00 2024-09-21T15:47:14.773 INFO:teuthology.orchestra.run.smithi089.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 355 kB/s | 33 kB 00:00 2024-09-21T15:47:14.815 INFO:teuthology.orchestra.run.smithi089.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 439 kB/s | 18 kB 00:00 2024-09-21T15:47:14.858 INFO:teuthology.orchestra.run.smithi089.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 923 kB/s | 38 kB 00:00 2024-09-21T15:47:14.895 INFO:teuthology.orchestra.run.smithi100.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 28 MB/s | 16 MB 00:00 2024-09-21T15:47:14.896 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-21T15:47:14.896 INFO:teuthology.orchestra.run.smithi100.stdout:Total 28 MB/s | 38 MB 00:01 2024-09-21T15:47:15.000 INFO:teuthology.orchestra.run.smithi089.stdout:(6/12): ceph-mgr-diskprediction-local-18.2.4-85 15 MB/s | 7.4 MB 00:00 2024-09-21T15:47:15.058 INFO:teuthology.orchestra.run.smithi089.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 8.9 MB/s | 3.0 MB 00:00 2024-09-21T15:47:15.192 INFO:teuthology.orchestra.run.smithi089.stdout:(8/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 15 MB/s | 4.9 MB 00:00 2024-09-21T15:47:15.246 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-21T15:47:15.272 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-21T15:47:15.272 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-21T15:47:15.276 INFO:teuthology.orchestra.run.smithi089.stdout:(9/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 23 MB/s | 5.1 MB 00:00 2024-09-21T15:47:15.301 INFO:teuthology.orchestra.run.smithi089.stdout:(10/12): python3-devel-3.9.19-8.el9.x86_64.rpm 813 kB/s | 245 kB 00:00 2024-09-21T15:47:15.335 INFO:teuthology.orchestra.run.smithi089.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 1.7 MB/s | 250 kB 00:00 2024-09-21T15:47:15.643 INFO:teuthology.orchestra.run.smithi089.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 44 MB/s | 16 MB 00:00 2024-09-21T15:47:15.644 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-09-21T15:47:15.644 INFO:teuthology.orchestra.run.smithi089.stdout:Total 23 MB/s | 38 MB 00:01 2024-09-21T15:47:15.799 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-21T15:47:15.800 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-21T15:47:15.995 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-09-21T15:47:16.021 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-09-21T15:47:16.021 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-09-21T15:47:16.549 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-09-21T15:47:16.550 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-09-21T15:47:16.558 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-21T15:47:16.630 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-21T15:47:16.668 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-21T15:47:16.770 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-21T15:47:16.811 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-21T15:47:16.987 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-21T15:47:17.034 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-21T15:47:17.115 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-21T15:47:17.305 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-09-21T15:47:17.384 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-21T15:47:17.425 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-21T15:47:17.524 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-21T15:47:17.559 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-21T15:47:17.744 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-21T15:47:17.783 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-21T15:47:17.814 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-21T15:47:17.864 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-21T15:47:17.933 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-21T15:47:18.494 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-21T15:47:18.630 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-21T15:47:19.201 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-21T15:47:19.895 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-21T15:47:20.248 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-21T15:47:20.276 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.4-855.g19e2a2 12/12 2024-09-21T15:47:20.944 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-21T15:47:20.969 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-diskprediction-local-2:18.2.4-855.g19e2a2 12/12 2024-09-21T15:47:22.812 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.4-855.g19e2a2 12/12 2024-09-21T15:47:22.812 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.4-855.g19e2a2 1/12 2024-09-21T15:47:22.812 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-21T15:47:22.812 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-21T15:47:22.812 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-21T15:47:22.812 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-21T15:47:22.812 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-21T15:47:22.812 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-21T15:47:22.812 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-21T15:47:22.812 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-21T15:47:22.812 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-21T15:47:22.812 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-21T15:47:23.239 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-21T15:47:23.239 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:23.240 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-21T15:47:23.240 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-diskprediction-local-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-21T15:47:23.240 INFO:teuthology.orchestra.run.smithi100.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-21T15:47:23.240 INFO:teuthology.orchestra.run.smithi100.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-21T15:47:23.240 INFO:teuthology.orchestra.run.smithi100.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-21T15:47:23.240 INFO:teuthology.orchestra.run.smithi100.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-21T15:47:23.240 INFO:teuthology.orchestra.run.smithi100.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-21T15:47:23.240 INFO:teuthology.orchestra.run.smithi100.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-21T15:47:23.240 INFO:teuthology.orchestra.run.smithi100.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-21T15:47:23.240 INFO:teuthology.orchestra.run.smithi100.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-21T15:47:23.240 INFO:teuthology.orchestra.run.smithi100.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-21T15:47:23.240 INFO:teuthology.orchestra.run.smithi100.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-21T15:47:23.240 INFO:teuthology.orchestra.run.smithi100.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-21T15:47:23.241 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:23.241 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-21T15:47:23.625 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:18.2.4-855.g19e2a2 12/12 2024-09-21T15:47:23.625 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-diskprediction-local-2:18.2.4-855.g19e2a2 1/12 2024-09-21T15:47:23.625 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-21T15:47:23.625 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-21T15:47:23.626 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-21T15:47:23.626 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-21T15:47:23.626 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-21T15:47:23.626 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-21T15:47:23.626 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-21T15:47:23.626 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-21T15:47:23.626 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-21T15:47:23.626 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-21T15:47:23.632 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-mgr-rook 2024-09-21T15:47:24.029 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-21T15:47:24.029 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:24.029 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-09-21T15:47:24.029 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-diskprediction-local-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-21T15:47:24.029 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-21T15:47:24.029 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-21T15:47:24.030 INFO:teuthology.orchestra.run.smithi089.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-21T15:47:24.030 INFO:teuthology.orchestra.run.smithi089.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-21T15:47:24.030 INFO:teuthology.orchestra.run.smithi089.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-21T15:47:24.030 INFO:teuthology.orchestra.run.smithi089.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-21T15:47:24.030 INFO:teuthology.orchestra.run.smithi089.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-21T15:47:24.030 INFO:teuthology.orchestra.run.smithi089.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-21T15:47:24.030 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-21T15:47:24.030 INFO:teuthology.orchestra.run.smithi089.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-21T15:47:24.030 INFO:teuthology.orchestra.run.smithi089.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-21T15:47:24.030 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:24.030 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-21T15:47:24.221 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:10 ago on Sat 21 Sep 2024 03:46:14 PM UTC. 2024-09-21T15:47:24.353 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-21T15:47:24.354 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-21T15:47:24.354 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-09-21T15:47:24.354 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-21T15:47:24.354 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-21T15:47:24.354 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-rook noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 49 k 2024-09-21T15:47:24.354 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-09-21T15:47:24.355 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-21T15:47:24.355 INFO:teuthology.orchestra.run.smithi100.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-21T15:47:24.355 INFO:teuthology.orchestra.run.smithi100.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-21T15:47:24.355 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-21T15:47:24.355 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-21T15:47:24.355 INFO:teuthology.orchestra.run.smithi100.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-21T15:47:24.355 INFO:teuthology.orchestra.run.smithi100.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-21T15:47:24.355 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-21T15:47:24.355 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-21T15:47:24.355 INFO:teuthology.orchestra.run.smithi100.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-21T15:47:24.355 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-21T15:47:24.355 INFO:teuthology.orchestra.run.smithi100.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-21T15:47:24.355 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:24.356 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-21T15:47:24.356 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-21T15:47:24.356 INFO:teuthology.orchestra.run.smithi100.stdout:Install 13 Packages 2024-09-21T15:47:24.356 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:24.356 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 2.2 M 2024-09-21T15:47:24.356 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 26 M 2024-09-21T15:47:24.356 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-21T15:47:24.422 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-mgr-rook 2024-09-21T15:47:24.568 INFO:teuthology.orchestra.run.smithi100.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 158 kB/s | 19 kB 00:00 2024-09-21T15:47:24.586 INFO:teuthology.orchestra.run.smithi100.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 185 kB/s | 26 kB 00:00 2024-09-21T15:47:24.602 INFO:teuthology.orchestra.run.smithi100.stdout:(3/13): ceph-mgr-rook-18.2.4-855.g19e2a2be.el9. 318 kB/s | 49 kB 00:00 2024-09-21T15:47:24.710 INFO:teuthology.orchestra.run.smithi100.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-09-21T15:47:24.736 INFO:teuthology.orchestra.run.smithi100.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.3 MB/s | 222 kB 00:00 2024-09-21T15:47:24.761 INFO:teuthology.orchestra.run.smithi100.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 54 kB 00:00 2024-09-21T15:47:24.786 INFO:teuthology.orchestra.run.smithi100.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 640 kB/s | 32 kB 00:00 2024-09-21T15:47:24.803 INFO:teuthology.orchestra.run.smithi100.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 349 kB/s | 14 kB 00:00 2024-09-21T15:47:24.828 INFO:teuthology.orchestra.run.smithi100.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.2 MB/s | 279 kB 00:00 2024-09-21T15:47:24.862 INFO:teuthology.orchestra.run.smithi100.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.9 MB/s | 220 kB 00:00 2024-09-21T15:47:24.887 INFO:teuthology.orchestra.run.smithi100.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.8 MB/s | 90 kB 00:00 2024-09-21T15:47:24.904 INFO:teuthology.orchestra.run.smithi100.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 786 kB/s | 59 kB 00:00 2024-09-21T15:47:24.946 INFO:teuthology.orchestra.run.smithi100.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.2 MB/s | 1.0 MB 00:00 2024-09-21T15:47:24.948 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-21T15:47:24.949 INFO:teuthology.orchestra.run.smithi100.stdout:Total 3.7 MB/s | 2.2 MB 00:00 2024-09-21T15:47:25.012 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-21T15:47:25.027 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-21T15:47:25.027 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-21T15:47:25.029 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:11 ago on Sat 21 Sep 2024 03:46:14 PM UTC. 2024-09-21T15:47:25.116 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-21T15:47:25.117 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-21T15:47:25.171 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-21T15:47:25.172 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-21T15:47:25.172 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-09-21T15:47:25.172 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-21T15:47:25.172 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-09-21T15:47:25.172 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-rook noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 49 k 2024-09-21T15:47:25.172 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-09-21T15:47:25.172 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-21T15:47:25.172 INFO:teuthology.orchestra.run.smithi089.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-21T15:47:25.172 INFO:teuthology.orchestra.run.smithi089.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-21T15:47:25.172 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-21T15:47:25.172 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-21T15:47:25.173 INFO:teuthology.orchestra.run.smithi089.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-21T15:47:25.173 INFO:teuthology.orchestra.run.smithi089.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-21T15:47:25.173 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-21T15:47:25.173 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-21T15:47:25.173 INFO:teuthology.orchestra.run.smithi089.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-21T15:47:25.173 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-21T15:47:25.173 INFO:teuthology.orchestra.run.smithi089.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-21T15:47:25.173 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:25.173 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-09-21T15:47:25.173 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-21T15:47:25.173 INFO:teuthology.orchestra.run.smithi089.stdout:Install 13 Packages 2024-09-21T15:47:25.173 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:25.174 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 2.2 M 2024-09-21T15:47:25.174 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 26 M 2024-09-21T15:47:25.174 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-09-21T15:47:25.472 INFO:teuthology.orchestra.run.smithi089.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 173 kB/s | 19 kB 00:00 2024-09-21T15:47:25.488 INFO:teuthology.orchestra.run.smithi089.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 201 kB/s | 26 kB 00:00 2024-09-21T15:47:25.514 INFO:teuthology.orchestra.run.smithi089.stdout:(3/13): ceph-mgr-rook-18.2.4-855.g19e2a2be.el9. 322 kB/s | 49 kB 00:00 2024-09-21T15:47:25.606 INFO:teuthology.orchestra.run.smithi089.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.3 MB/s | 159 kB 00:00 2024-09-21T15:47:25.617 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-21T15:47:25.648 INFO:teuthology.orchestra.run.smithi089.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.2 MB/s | 222 kB 00:00 2024-09-21T15:47:25.665 INFO:teuthology.orchestra.run.smithi089.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 914 kB/s | 54 kB 00:00 2024-09-21T15:47:25.723 INFO:teuthology.orchestra.run.smithi089.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.3 MB/s | 279 kB 00:00 2024-09-21T15:47:25.741 INFO:teuthology.orchestra.run.smithi089.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 349 kB/s | 32 kB 00:00 2024-09-21T15:47:25.766 INFO:teuthology.orchestra.run.smithi089.stdout:(9/13): python3-certifi-2023.05.07-4.el9.noarch 140 kB/s | 14 kB 00:00 2024-09-21T15:47:25.783 INFO:teuthology.orchestra.run.smithi089.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 4.3 MB/s | 59 kB 00:00 2024-09-21T15:47:25.801 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-21T15:47:25.808 INFO:teuthology.orchestra.run.smithi089.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.5 MB/s | 90 kB 00:00 2024-09-21T15:47:25.842 INFO:teuthology.orchestra.run.smithi089.stdout:(12/13): python3-google-auth-2.34.0-1.el9.noarc 1.9 MB/s | 220 kB 00:00 2024-09-21T15:47:25.855 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-21T15:47:25.884 INFO:teuthology.orchestra.run.smithi089.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.8 MB/s | 1.0 MB 00:00 2024-09-21T15:47:25.887 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-09-21T15:47:25.888 INFO:teuthology.orchestra.run.smithi089.stdout:Total 3.1 MB/s | 2.2 MB 00:00 2024-09-21T15:47:25.911 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-21T15:47:25.948 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-21T15:47:25.958 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-09-21T15:47:25.975 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-09-21T15:47:25.975 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-09-21T15:47:25.989 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-21T15:47:26.063 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-09-21T15:47:26.064 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-09-21T15:47:26.092 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-21T15:47:26.203 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-21T15:47:26.250 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-21T15:47:26.576 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-09-21T15:47:26.748 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-21T15:47:26.802 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-21T15:47:26.866 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-21T15:47:26.903 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-21T15:47:26.953 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-21T15:47:26.960 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-21T15:47:27.014 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-21T15:47:27.056 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-21T15:47:27.064 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-21T15:47:27.104 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-21T15:47:27.125 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-rook-2:18.2.4-855.g19e2a2be.el9.noarch 13/13 2024-09-21T15:47:27.174 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-21T15:47:27.222 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-21T15:47:27.937 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-21T15:47:27.986 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-21T15:47:27.989 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.4-855.g19e2a2be.el9.noarch 13/13 2024-09-21T15:47:27.989 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-rook-2:18.2.4-855.g19e2a2be.el9.noarch 1/13 2024-09-21T15:47:27.989 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-21T15:47:27.989 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-21T15:47:27.989 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-21T15:47:27.990 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-21T15:47:27.990 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-21T15:47:27.990 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-21T15:47:27.990 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-21T15:47:27.990 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-21T15:47:27.990 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-21T15:47:27.990 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-21T15:47:27.990 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-21T15:47:28.029 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-21T15:47:28.085 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-21T15:47:28.115 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-rook-2:18.2.4-855.g19e2a2be.el9.noarch 13/13 2024-09-21T15:47:28.407 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-21T15:47:28.407 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:28.407 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-21T15:47:28.408 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-rook-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-21T15:47:28.408 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-21T15:47:28.408 INFO:teuthology.orchestra.run.smithi100.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-21T15:47:28.408 INFO:teuthology.orchestra.run.smithi100.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-21T15:47:28.408 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-21T15:47:28.408 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-21T15:47:28.408 INFO:teuthology.orchestra.run.smithi100.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-21T15:47:28.408 INFO:teuthology.orchestra.run.smithi100.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-21T15:47:28.408 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-21T15:47:28.408 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-21T15:47:28.408 INFO:teuthology.orchestra.run.smithi100.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-21T15:47:28.408 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-21T15:47:28.408 INFO:teuthology.orchestra.run.smithi100.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-21T15:47:28.408 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:28.409 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-21T15:47:28.673 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-mgr-cephadm 2024-09-21T15:47:28.903 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-rook-2:18.2.4-855.g19e2a2be.el9.noarch 13/13 2024-09-21T15:47:28.903 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-rook-2:18.2.4-855.g19e2a2be.el9.noarch 1/13 2024-09-21T15:47:28.903 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-21T15:47:28.903 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-21T15:47:28.903 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-21T15:47:28.903 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-21T15:47:28.903 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-21T15:47:28.903 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-21T15:47:28.904 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-21T15:47:28.904 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-21T15:47:28.904 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-21T15:47:28.904 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-21T15:47:28.904 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-21T15:47:29.252 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-21T15:47:29.252 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:29.252 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-09-21T15:47:29.252 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-rook-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-21T15:47:29.252 INFO:teuthology.orchestra.run.smithi089.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-21T15:47:29.252 INFO:teuthology.orchestra.run.smithi089.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-21T15:47:29.252 INFO:teuthology.orchestra.run.smithi089.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-21T15:47:29.252 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-21T15:47:29.252 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-21T15:47:29.252 INFO:teuthology.orchestra.run.smithi089.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-21T15:47:29.252 INFO:teuthology.orchestra.run.smithi089.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-21T15:47:29.252 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-21T15:47:29.252 INFO:teuthology.orchestra.run.smithi089.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-21T15:47:29.253 INFO:teuthology.orchestra.run.smithi089.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-21T15:47:29.253 INFO:teuthology.orchestra.run.smithi089.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-21T15:47:29.253 INFO:teuthology.orchestra.run.smithi089.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-21T15:47:29.253 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:29.253 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-21T15:47:29.261 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:15 ago on Sat 21 Sep 2024 03:46:14 PM UTC. 2024-09-21T15:47:29.393 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-21T15:47:29.394 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-21T15:47:29.394 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-09-21T15:47:29.394 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-21T15:47:29.394 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-21T15:47:29.394 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-cephadm noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 139 k 2024-09-21T15:47:29.394 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-09-21T15:47:29.394 INFO:teuthology.orchestra.run.smithi100.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-21T15:47:29.394 INFO:teuthology.orchestra.run.smithi100.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-21T15:47:29.395 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-21T15:47:29.395 INFO:teuthology.orchestra.run.smithi100.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-21T15:47:29.395 INFO:teuthology.orchestra.run.smithi100.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-21T15:47:29.395 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:29.395 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-21T15:47:29.395 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-21T15:47:29.395 INFO:teuthology.orchestra.run.smithi100.stdout:Install 6 Packages 2024-09-21T15:47:29.395 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:29.396 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 7.0 M 2024-09-21T15:47:29.396 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 33 M 2024-09-21T15:47:29.396 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-21T15:47:29.544 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-mgr-cephadm 2024-09-21T15:47:29.632 INFO:teuthology.orchestra.run.smithi100.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.6 MB/s | 248 kB 00:00 2024-09-21T15:47:29.691 INFO:teuthology.orchestra.run.smithi100.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 9.2 MB/s | 548 kB 00:00 2024-09-21T15:47:29.716 INFO:teuthology.orchestra.run.smithi100.stdout:(3/6): ceph-mgr-cephadm-18.2.4-855.g19e2a2be.el 595 kB/s | 139 kB 00:00 2024-09-21T15:47:29.741 INFO:teuthology.orchestra.run.smithi100.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 1.1 MB/s | 58 kB 00:00 2024-09-21T15:47:29.783 INFO:teuthology.orchestra.run.smithi100.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.1 MB/s | 74 kB 00:00 2024-09-21T15:47:29.884 INFO:teuthology.orchestra.run.smithi100.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 15 MB/s | 6.0 MB 00:00 2024-09-21T15:47:29.886 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-21T15:47:29.887 INFO:teuthology.orchestra.run.smithi100.stdout:Total 14 MB/s | 7.0 MB 00:00 2024-09-21T15:47:29.957 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-21T15:47:29.968 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-21T15:47:29.968 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-21T15:47:30.067 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-21T15:47:30.068 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-21T15:47:30.121 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:16 ago on Sat 21 Sep 2024 03:46:14 PM UTC. 2024-09-21T15:47:30.255 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-21T15:47:30.255 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-21T15:47:30.255 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-09-21T15:47:30.256 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-21T15:47:30.256 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-09-21T15:47:30.256 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-cephadm noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 139 k 2024-09-21T15:47:30.256 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-09-21T15:47:30.256 INFO:teuthology.orchestra.run.smithi089.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-21T15:47:30.256 INFO:teuthology.orchestra.run.smithi089.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-21T15:47:30.256 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-21T15:47:30.256 INFO:teuthology.orchestra.run.smithi089.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-21T15:47:30.256 INFO:teuthology.orchestra.run.smithi089.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-21T15:47:30.256 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:30.256 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-09-21T15:47:30.256 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-21T15:47:30.257 INFO:teuthology.orchestra.run.smithi089.stdout:Install 6 Packages 2024-09-21T15:47:30.257 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:30.257 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 7.0 M 2024-09-21T15:47:30.257 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 33 M 2024-09-21T15:47:30.257 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-09-21T15:47:30.398 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-21T15:47:30.481 INFO:teuthology.orchestra.run.smithi089.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 2.1 MB/s | 248 kB 00:00 2024-09-21T15:47:30.531 INFO:teuthology.orchestra.run.smithi089.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 11 MB/s | 548 kB 00:00 2024-09-21T15:47:30.556 INFO:teuthology.orchestra.run.smithi089.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-09-21T15:47:30.564 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-21T15:47:30.582 INFO:teuthology.orchestra.run.smithi089.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 2.9 MB/s | 74 kB 00:00 2024-09-21T15:47:30.607 INFO:teuthology.orchestra.run.smithi089.stdout:(5/6): ceph-mgr-cephadm-18.2.4-855.g19e2a2be.el 578 kB/s | 139 kB 00:00 2024-09-21T15:47:30.611 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-21T15:47:30.707 INFO:teuthology.orchestra.run.smithi089.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 6.0 MB 00:00 2024-09-21T15:47:30.709 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-09-21T15:47:30.709 INFO:teuthology.orchestra.run.smithi089.stdout:Total 15 MB/s | 7.0 MB 00:00 2024-09-21T15:47:30.779 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-09-21T15:47:30.790 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-09-21T15:47:30.791 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-09-21T15:47:30.889 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-09-21T15:47:30.891 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-09-21T15:47:31.128 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-21T15:47:31.205 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-09-21T15:47:31.222 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-21T15:47:31.282 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-21T15:47:31.306 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 6/6 2024-09-21T15:47:31.385 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-21T15:47:31.443 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-21T15:47:31.977 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-21T15:47:32.064 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-21T15:47:32.091 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 6/6 2024-09-21T15:47:32.091 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 1/6 2024-09-21T15:47:32.091 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-21T15:47:32.091 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-21T15:47:32.092 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-21T15:47:32.092 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-21T15:47:32.125 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-21T15:47:32.156 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-mgr-cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 6/6 2024-09-21T15:47:32.357 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-21T15:47:32.357 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:32.357 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-21T15:47:32.357 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-21T15:47:32.357 INFO:teuthology.orchestra.run.smithi100.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-21T15:47:32.357 INFO:teuthology.orchestra.run.smithi100.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-21T15:47:32.357 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-21T15:47:32.357 INFO:teuthology.orchestra.run.smithi100.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-21T15:47:32.357 INFO:teuthology.orchestra.run.smithi100.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-21T15:47:32.357 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:32.358 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-21T15:47:32.598 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-fuse 2024-09-21T15:47:32.985 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-mgr-cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 6/6 2024-09-21T15:47:32.985 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-mgr-cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 1/6 2024-09-21T15:47:32.985 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-21T15:47:32.985 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-21T15:47:32.985 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-21T15:47:32.986 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-21T15:47:33.182 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:19 ago on Sat 21 Sep 2024 03:46:14 PM UTC. 2024-09-21T15:47:33.276 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-21T15:47:33.277 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:33.277 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-09-21T15:47:33.277 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-mgr-cephadm-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-21T15:47:33.277 INFO:teuthology.orchestra.run.smithi089.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-21T15:47:33.277 INFO:teuthology.orchestra.run.smithi089.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-21T15:47:33.277 INFO:teuthology.orchestra.run.smithi089.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-21T15:47:33.277 INFO:teuthology.orchestra.run.smithi089.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-21T15:47:33.277 INFO:teuthology.orchestra.run.smithi089.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-21T15:47:33.277 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:33.277 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-21T15:47:33.314 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-21T15:47:33.314 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-21T15:47:33.314 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-09-21T15:47:33.314 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-21T15:47:33.314 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-21T15:47:33.315 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-fuse x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 852 k 2024-09-21T15:47:33.315 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-09-21T15:47:33.315 INFO:teuthology.orchestra.run.smithi100.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-21T15:47:33.315 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:33.315 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-21T15:47:33.315 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-21T15:47:33.315 INFO:teuthology.orchestra.run.smithi100.stdout:Install 2 Packages 2024-09-21T15:47:33.315 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:33.316 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 932 k 2024-09-21T15:47:33.316 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 2.7 M 2024-09-21T15:47:33.316 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-21T15:47:33.527 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-fuse 2024-09-21T15:47:33.594 INFO:teuthology.orchestra.run.smithi100.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 529 kB/s | 80 kB 00:00 2024-09-21T15:47:33.719 INFO:teuthology.orchestra.run.smithi100.stdout:(2/2): ceph-fuse-18.2.4-855.g19e2a2be.el9.x86_6 3.0 MB/s | 852 kB 00:00 2024-09-21T15:47:33.720 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-21T15:47:33.720 INFO:teuthology.orchestra.run.smithi100.stdout:Total 2.2 MB/s | 932 kB 00:00 2024-09-21T15:47:33.732 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-21T15:47:33.754 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-21T15:47:33.754 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-21T15:47:33.833 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-21T15:47:33.833 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-21T15:47:34.021 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-21T15:47:34.109 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:20 ago on Sat 21 Sep 2024 03:46:14 PM UTC. 2024-09-21T15:47:34.151 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-21T15:47:34.226 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-fuse-2:18.2.4-855.g19e2a2be.el9.x86_64 2/2 2024-09-21T15:47:34.241 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-21T15:47:34.242 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-21T15:47:34.242 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-09-21T15:47:34.242 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-21T15:47:34.242 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-09-21T15:47:34.242 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-fuse x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 852 k 2024-09-21T15:47:34.242 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-09-21T15:47:34.243 INFO:teuthology.orchestra.run.smithi089.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-21T15:47:34.243 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:34.243 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-09-21T15:47:34.243 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-21T15:47:34.243 INFO:teuthology.orchestra.run.smithi089.stdout:Install 2 Packages 2024-09-21T15:47:34.243 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:34.243 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 932 k 2024-09-21T15:47:34.243 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 2.7 M 2024-09-21T15:47:34.244 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-09-21T15:47:34.544 INFO:teuthology.orchestra.run.smithi089.stdout:(1/2): ceph-fuse-18.2.4-855.g19e2a2be.el9.x86_6 3.0 MB/s | 852 kB 00:00 2024-09-21T15:47:34.895 INFO:teuthology.orchestra.run.smithi089.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 126 kB/s | 80 kB 00:00 2024-09-21T15:47:34.895 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-09-21T15:47:34.895 INFO:teuthology.orchestra.run.smithi089.stdout:Total 1.4 MB/s | 932 kB 00:00 2024-09-21T15:47:34.907 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-fuse-2:18.2.4-855.g19e2a2be.el9.x86_64 2/2 2024-09-21T15:47:34.907 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-fuse-2:18.2.4-855.g19e2a2be.el9.x86_64 1/2 2024-09-21T15:47:34.908 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-09-21T15:47:34.931 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-09-21T15:47:34.931 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-09-21T15:47:35.011 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-09-21T15:47:35.012 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-09-21T15:47:35.170 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-21T15:47:35.170 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:35.170 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-21T15:47:35.170 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-fuse-2:18.2.4-855.g19e2a2be.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-21T15:47:35.170 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:35.170 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-21T15:47:35.205 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-09-21T15:47:35.343 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-21T15:47:35.411 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-fuse-2:18.2.4-855.g19e2a2be.el9.x86_64 2/2 2024-09-21T15:47:35.463 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-volume 2024-09-21T15:47:36.049 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:22 ago on Sat 21 Sep 2024 03:46:14 PM UTC. 2024-09-21T15:47:36.096 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-fuse-2:18.2.4-855.g19e2a2be.el9.x86_64 2/2 2024-09-21T15:47:36.097 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-fuse-2:18.2.4-855.g19e2a2be.el9.x86_64 1/2 2024-09-21T15:47:36.188 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-21T15:47:36.189 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-21T15:47:36.189 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-09-21T15:47:36.189 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-21T15:47:36.189 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-21T15:47:36.189 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-volume noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 264 k 2024-09-21T15:47:36.189 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-09-21T15:47:36.189 INFO:teuthology.orchestra.run.smithi100.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-21T15:47:36.189 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:36.190 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-21T15:47:36.190 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-21T15:47:36.190 INFO:teuthology.orchestra.run.smithi100.stdout:Install 2 Packages 2024-09-21T15:47:36.190 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:36.190 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 342 k 2024-09-21T15:47:36.190 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 1.5 M 2024-09-21T15:47:36.190 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-21T15:47:36.375 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-21T15:47:36.375 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:36.376 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-09-21T15:47:36.376 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-fuse-2:18.2.4-855.g19e2a2be.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-21T15:47:36.376 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:36.376 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-21T15:47:36.534 INFO:teuthology.orchestra.run.smithi100.stdout:(1/2): ceph-volume-18.2.4-855.g19e2a2be.el9.noa 1.2 MB/s | 264 kB 00:00 2024-09-21T15:47:36.647 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install ceph-volume 2024-09-21T15:47:36.775 INFO:teuthology.orchestra.run.smithi100.stdout:(2/2): python3-packaging-20.9-5.el9.noarch.rpm 167 kB/s | 77 kB 00:00 2024-09-21T15:47:36.776 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-21T15:47:36.776 INFO:teuthology.orchestra.run.smithi100.stdout:Total 582 kB/s | 342 kB 00:00 2024-09-21T15:47:36.787 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-21T15:47:36.796 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-21T15:47:36.797 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-21T15:47:36.828 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-21T15:47:36.828 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-21T15:47:36.998 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-21T15:47:37.192 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-09-21T15:47:37.228 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-volume-2:18.2.4-855.g19e2a2be.el9.noarch 2/2 2024-09-21T15:47:37.238 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:23 ago on Sat 21 Sep 2024 03:46:14 PM UTC. 2024-09-21T15:47:37.243 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-volume-2:18.2.4-855.g19e2a2be.el9.noarch 2/2 2024-09-21T15:47:37.243 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T15:47:37.243 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-21T15:47:37.243 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:37.375 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-21T15:47:37.375 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-21T15:47:37.375 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repository Size 2024-09-21T15:47:37.375 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-21T15:47:37.375 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-09-21T15:47:37.375 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-volume noarch 2:18.2.4-855.g19e2a2be.el9 ceph-noarch 264 k 2024-09-21T15:47:37.375 INFO:teuthology.orchestra.run.smithi089.stdout:Installing dependencies: 2024-09-21T15:47:37.376 INFO:teuthology.orchestra.run.smithi089.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-21T15:47:37.376 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:37.376 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-09-21T15:47:37.376 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-21T15:47:37.376 INFO:teuthology.orchestra.run.smithi089.stdout:Install 2 Packages 2024-09-21T15:47:37.376 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:37.376 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 342 k 2024-09-21T15:47:37.377 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 1.5 M 2024-09-21T15:47:37.377 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-09-21T15:47:37.644 INFO:teuthology.orchestra.run.smithi089.stdout:(1/2): ceph-volume-18.2.4-855.g19e2a2be.el9.noa 1.0 MB/s | 264 kB 00:00 2024-09-21T15:47:37.974 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-volume-2:18.2.4-855.g19e2a2be.el9.noarch 1/2 2024-09-21T15:47:38.061 INFO:teuthology.orchestra.run.smithi089.stdout:(2/2): python3-packaging-20.9-5.el9.noarch.rpm 116 kB/s | 77 kB 00:00 2024-09-21T15:47:38.061 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-09-21T15:47:38.062 INFO:teuthology.orchestra.run.smithi089.stdout:Total 498 kB/s | 342 kB 00:00 2024-09-21T15:47:38.074 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-09-21T15:47:38.084 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-09-21T15:47:38.084 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-09-21T15:47:38.121 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-09-21T15:47:38.122 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-09-21T15:47:38.268 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-09-21T15:47:38.297 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-09-21T15:47:38.297 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:38.297 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-21T15:47:38.297 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-volume-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-21T15:47:38.297 INFO:teuthology.orchestra.run.smithi100.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-21T15:47:38.297 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:38.297 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-21T15:47:38.453 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-09-21T15:47:38.497 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : ceph-volume-2:18.2.4-855.g19e2a2be.el9.noarch 2/2 2024-09-21T15:47:38.513 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: ceph-volume-2:18.2.4-855.g19e2a2be.el9.noarch 2/2 2024-09-21T15:47:38.513 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T15:47:38.513 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-21T15:47:38.513 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:38.607 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install librados-devel 2024-09-21T15:47:39.171 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : ceph-volume-2:18.2.4-855.g19e2a2be.el9.noarch 1/2 2024-09-21T15:47:39.202 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:25 ago on Sat 21 Sep 2024 03:46:14 PM UTC. 2024-09-21T15:47:39.336 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-21T15:47:39.337 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-21T15:47:39.337 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repo Size 2024-09-21T15:47:39.338 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-21T15:47:39.338 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-21T15:47:39.338 INFO:teuthology.orchestra.run.smithi100.stdout: librados-devel x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 126 k 2024-09-21T15:47:39.338 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:39.338 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-21T15:47:39.338 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-21T15:47:39.338 INFO:teuthology.orchestra.run.smithi100.stdout:Install 1 Package 2024-09-21T15:47:39.338 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:39.338 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 126 k 2024-09-21T15:47:39.338 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 456 k 2024-09-21T15:47:39.338 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-21T15:47:39.467 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-09-21T15:47:39.467 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:39.467 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-09-21T15:47:39.468 INFO:teuthology.orchestra.run.smithi089.stdout: ceph-volume-2:18.2.4-855.g19e2a2be.el9.noarch 2024-09-21T15:47:39.468 INFO:teuthology.orchestra.run.smithi089.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-21T15:47:39.468 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:39.468 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-21T15:47:39.532 INFO:teuthology.orchestra.run.smithi100.stdout:librados-devel-18.2.4-855.g19e2a2be.el9.x86_64. 653 kB/s | 126 kB 00:00 2024-09-21T15:47:39.533 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-21T15:47:39.533 INFO:teuthology.orchestra.run.smithi100.stdout:Total 647 kB/s | 126 kB 00:00 2024-09-21T15:47:39.533 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-21T15:47:39.539 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-21T15:47:39.539 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-21T15:47:39.602 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-21T15:47:39.602 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-21T15:47:39.753 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-21T15:47:39.789 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install librados-devel 2024-09-21T15:47:39.862 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : librados-devel-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-21T15:47:40.311 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librados-devel-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-21T15:47:40.379 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:26 ago on Sat 21 Sep 2024 03:46:14 PM UTC. 2024-09-21T15:47:40.514 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-21T15:47:40.515 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-21T15:47:40.515 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repo Size 2024-09-21T15:47:40.515 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-21T15:47:40.515 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-09-21T15:47:40.515 INFO:teuthology.orchestra.run.smithi089.stdout: librados-devel x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 126 k 2024-09-21T15:47:40.515 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:40.515 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-09-21T15:47:40.515 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-21T15:47:40.516 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2024-09-21T15:47:40.516 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:40.516 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 126 k 2024-09-21T15:47:40.516 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 456 k 2024-09-21T15:47:40.516 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-09-21T15:47:40.555 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librados-devel-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-21T15:47:40.555 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:40.555 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-21T15:47:40.555 INFO:teuthology.orchestra.run.smithi100.stdout: librados-devel-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:47:40.556 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:40.556 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-21T15:47:40.702 INFO:teuthology.orchestra.run.smithi089.stdout:librados-devel-18.2.4-855.g19e2a2be.el9.x86_64. 681 kB/s | 126 kB 00:00 2024-09-21T15:47:40.751 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-09-21T15:47:40.751 INFO:teuthology.orchestra.run.smithi089.stdout:Total 676 kB/s | 126 kB 00:00 2024-09-21T15:47:40.751 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-09-21T15:47:40.751 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-09-21T15:47:40.752 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-09-21T15:47:40.770 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-09-21T15:47:40.771 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-09-21T15:47:40.792 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install libcephfs2 2024-09-21T15:47:40.921 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-09-21T15:47:41.032 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : librados-devel-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-21T15:47:41.383 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:27 ago on Sat 21 Sep 2024 03:46:14 PM UTC. 2024-09-21T15:47:41.449 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: librados-devel-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-21T15:47:41.473 INFO:teuthology.orchestra.run.smithi100.stdout:Package libcephfs2-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-21T15:47:41.518 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-21T15:47:41.519 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-09-21T15:47:41.519 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-21T15:47:41.600 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install libcephfs-devel 2024-09-21T15:47:41.699 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : librados-devel-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-21T15:47:41.700 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:41.700 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-09-21T15:47:41.700 INFO:teuthology.orchestra.run.smithi089.stdout: librados-devel-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:47:41.700 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:41.700 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-21T15:47:41.957 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install libcephfs2 2024-09-21T15:47:42.176 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:28 ago on Sat 21 Sep 2024 03:46:14 PM UTC. 2024-09-21T15:47:42.310 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-21T15:47:42.311 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-21T15:47:42.311 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repo Size 2024-09-21T15:47:42.311 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-21T15:47:42.311 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-21T15:47:42.311 INFO:teuthology.orchestra.run.smithi100.stdout: libcephfs-devel x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 31 k 2024-09-21T15:47:42.311 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:42.311 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-21T15:47:42.312 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-21T15:47:42.312 INFO:teuthology.orchestra.run.smithi100.stdout:Install 1 Package 2024-09-21T15:47:42.312 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:42.312 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 31 k 2024-09-21T15:47:42.312 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 139 k 2024-09-21T15:47:42.312 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-21T15:47:42.461 INFO:teuthology.orchestra.run.smithi100.stdout:libcephfs-devel-18.2.4-855.g19e2a2be.el9.x86_64 208 kB/s | 31 kB 00:00 2024-09-21T15:47:42.462 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-21T15:47:42.462 INFO:teuthology.orchestra.run.smithi100.stdout:Total 206 kB/s | 31 kB 00:00 2024-09-21T15:47:42.462 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-21T15:47:42.465 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-21T15:47:42.466 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-21T15:47:42.496 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-21T15:47:42.497 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-21T15:47:42.548 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:28 ago on Sat 21 Sep 2024 03:46:14 PM UTC. 2024-09-21T15:47:42.588 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-21T15:47:42.638 INFO:teuthology.orchestra.run.smithi089.stdout:Package libcephfs2-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-21T15:47:42.684 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-21T15:47:42.685 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-09-21T15:47:42.685 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-21T15:47:42.692 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libcephfs-devel-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-21T15:47:42.763 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install libcephfs-devel 2024-09-21T15:47:43.079 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: libcephfs-devel-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-21T15:47:43.317 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libcephfs-devel-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-21T15:47:43.317 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:43.317 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-21T15:47:43.318 INFO:teuthology.orchestra.run.smithi100.stdout: libcephfs-devel-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:47:43.318 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:43.318 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-21T15:47:43.343 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:29 ago on Sat 21 Sep 2024 03:46:14 PM UTC. 2024-09-21T15:47:43.474 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-21T15:47:43.475 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-21T15:47:43.475 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repo Size 2024-09-21T15:47:43.475 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-21T15:47:43.475 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-09-21T15:47:43.475 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs-devel x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 31 k 2024-09-21T15:47:43.475 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:43.475 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-09-21T15:47:43.475 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-21T15:47:43.475 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2024-09-21T15:47:43.475 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:43.476 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 31 k 2024-09-21T15:47:43.476 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 139 k 2024-09-21T15:47:43.476 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-09-21T15:47:43.500 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install librados2 2024-09-21T15:47:43.623 INFO:teuthology.orchestra.run.smithi089.stdout:libcephfs-devel-18.2.4-855.g19e2a2be.el9.x86_64 212 kB/s | 31 kB 00:00 2024-09-21T15:47:43.623 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-09-21T15:47:43.624 INFO:teuthology.orchestra.run.smithi089.stdout:Total 210 kB/s | 31 kB 00:00 2024-09-21T15:47:43.624 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-09-21T15:47:43.627 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-09-21T15:47:43.627 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-09-21T15:47:43.655 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-09-21T15:47:43.655 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-09-21T15:47:43.745 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-09-21T15:47:43.861 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : libcephfs-devel-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-21T15:47:44.081 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:30 ago on Sat 21 Sep 2024 03:46:14 PM UTC. 2024-09-21T15:47:44.171 INFO:teuthology.orchestra.run.smithi100.stdout:Package librados2-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-21T15:47:44.214 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-21T15:47:44.214 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-09-21T15:47:44.214 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-21T15:47:44.225 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: libcephfs-devel-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-21T15:47:44.293 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install librbd1 2024-09-21T15:47:44.464 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : libcephfs-devel-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-21T15:47:44.464 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:44.464 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-09-21T15:47:44.464 INFO:teuthology.orchestra.run.smithi089.stdout: libcephfs-devel-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:47:44.464 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:44.464 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-21T15:47:44.669 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install librados2 2024-09-21T15:47:44.874 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:30 ago on Sat 21 Sep 2024 03:46:14 PM UTC. 2024-09-21T15:47:44.965 INFO:teuthology.orchestra.run.smithi100.stdout:Package librbd1-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-21T15:47:45.007 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-21T15:47:45.008 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-09-21T15:47:45.008 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-21T15:47:45.091 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install python3-rados 2024-09-21T15:47:45.251 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:31 ago on Sat 21 Sep 2024 03:46:14 PM UTC. 2024-09-21T15:47:45.340 INFO:teuthology.orchestra.run.smithi089.stdout:Package librados2-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-21T15:47:45.383 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-21T15:47:45.384 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-09-21T15:47:45.384 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-21T15:47:45.465 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install librbd1 2024-09-21T15:47:45.670 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:31 ago on Sat 21 Sep 2024 03:46:14 PM UTC. 2024-09-21T15:47:45.760 INFO:teuthology.orchestra.run.smithi100.stdout:Package python3-rados-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-21T15:47:45.803 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-21T15:47:45.804 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-09-21T15:47:45.804 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-21T15:47:45.882 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install python3-rgw 2024-09-21T15:47:46.048 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:32 ago on Sat 21 Sep 2024 03:46:14 PM UTC. 2024-09-21T15:47:46.136 INFO:teuthology.orchestra.run.smithi089.stdout:Package librbd1-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-21T15:47:46.178 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-21T15:47:46.180 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-09-21T15:47:46.180 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-21T15:47:46.264 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install python3-rados 2024-09-21T15:47:46.465 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:32 ago on Sat 21 Sep 2024 03:46:14 PM UTC. 2024-09-21T15:47:46.554 INFO:teuthology.orchestra.run.smithi100.stdout:Package python3-rgw-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-21T15:47:46.597 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-21T15:47:46.598 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-09-21T15:47:46.598 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-21T15:47:46.682 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install python3-cephfs 2024-09-21T15:47:46.845 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:32 ago on Sat 21 Sep 2024 03:46:14 PM UTC. 2024-09-21T15:47:46.934 INFO:teuthology.orchestra.run.smithi089.stdout:Package python3-rados-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-21T15:47:46.977 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-21T15:47:46.978 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-09-21T15:47:46.978 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-21T15:47:47.055 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install python3-rgw 2024-09-21T15:47:47.268 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:33 ago on Sat 21 Sep 2024 03:46:14 PM UTC. 2024-09-21T15:47:47.360 INFO:teuthology.orchestra.run.smithi100.stdout:Package python3-cephfs-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-21T15:47:47.402 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-21T15:47:47.403 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-09-21T15:47:47.403 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-21T15:47:47.481 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install python3-rbd 2024-09-21T15:47:47.641 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:33 ago on Sat 21 Sep 2024 03:46:14 PM UTC. 2024-09-21T15:47:47.730 INFO:teuthology.orchestra.run.smithi089.stdout:Package python3-rgw-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-21T15:47:47.774 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-21T15:47:47.775 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-09-21T15:47:47.775 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-21T15:47:47.853 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install python3-cephfs 2024-09-21T15:47:48.065 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:34 ago on Sat 21 Sep 2024 03:46:14 PM UTC. 2024-09-21T15:47:48.154 INFO:teuthology.orchestra.run.smithi100.stdout:Package python3-rbd-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-21T15:47:48.196 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-21T15:47:48.197 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-09-21T15:47:48.198 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-21T15:47:48.271 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install rbd-fuse 2024-09-21T15:47:48.431 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:34 ago on Sat 21 Sep 2024 03:46:14 PM UTC. 2024-09-21T15:47:48.519 INFO:teuthology.orchestra.run.smithi089.stdout:Package python3-cephfs-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-21T15:47:48.561 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-21T15:47:48.563 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-09-21T15:47:48.563 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-21T15:47:48.643 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install python3-rbd 2024-09-21T15:47:48.851 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:34 ago on Sat 21 Sep 2024 03:46:14 PM UTC. 2024-09-21T15:47:48.982 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-21T15:47:48.983 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-21T15:47:48.983 INFO:teuthology.orchestra.run.smithi100.stdout: Package Architecture Version Repository Size 2024-09-21T15:47:48.983 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-21T15:47:48.983 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-21T15:47:48.983 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-fuse x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 87 k 2024-09-21T15:47:48.983 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:48.983 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-21T15:47:48.983 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-21T15:47:48.983 INFO:teuthology.orchestra.run.smithi100.stdout:Install 1 Package 2024-09-21T15:47:48.983 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:48.983 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 87 k 2024-09-21T15:47:48.984 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 227 k 2024-09-21T15:47:48.984 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-21T15:47:49.181 INFO:teuthology.orchestra.run.smithi100.stdout:rbd-fuse-18.2.4-855.g19e2a2be.el9.x86_64.rpm 439 kB/s | 87 kB 00:00 2024-09-21T15:47:49.181 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-21T15:47:49.182 INFO:teuthology.orchestra.run.smithi100.stdout:Total 436 kB/s | 87 kB 00:00 2024-09-21T15:47:49.182 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-21T15:47:49.188 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-21T15:47:49.189 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-21T15:47:49.226 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:35 ago on Sat 21 Sep 2024 03:46:14 PM UTC. 2024-09-21T15:47:49.250 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-21T15:47:49.251 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-21T15:47:49.316 INFO:teuthology.orchestra.run.smithi089.stdout:Package python3-rbd-2:18.2.4-855.g19e2a2be.el9.x86_64 is already installed. 2024-09-21T15:47:49.358 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-21T15:47:49.359 INFO:teuthology.orchestra.run.smithi089.stdout:Nothing to do. 2024-09-21T15:47:49.359 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-21T15:47:49.438 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install rbd-fuse 2024-09-21T15:47:49.438 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-21T15:47:49.528 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : rbd-fuse-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-21T15:47:49.909 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: rbd-fuse-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-21T15:47:50.018 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:36 ago on Sat 21 Sep 2024 03:46:14 PM UTC. 2024-09-21T15:47:50.149 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-21T15:47:50.149 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-21T15:47:50.150 INFO:teuthology.orchestra.run.smithi089.stdout: Package Architecture Version Repository Size 2024-09-21T15:47:50.150 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-21T15:47:50.150 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-09-21T15:47:50.150 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-fuse x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 87 k 2024-09-21T15:47:50.150 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:50.150 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-09-21T15:47:50.150 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-21T15:47:50.150 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2024-09-21T15:47:50.150 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:50.151 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 87 k 2024-09-21T15:47:50.151 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 227 k 2024-09-21T15:47:50.151 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-09-21T15:47:50.157 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : rbd-fuse-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-21T15:47:50.157 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:50.158 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-21T15:47:50.158 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-fuse-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:47:50.158 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:50.158 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-21T15:47:50.335 INFO:teuthology.orchestra.run.smithi089.stdout:rbd-fuse-18.2.4-855.g19e2a2be.el9.x86_64.rpm 470 kB/s | 87 kB 00:00 2024-09-21T15:47:50.336 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-09-21T15:47:50.336 INFO:teuthology.orchestra.run.smithi089.stdout:Total 466 kB/s | 87 kB 00:00 2024-09-21T15:47:50.336 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-09-21T15:47:50.343 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-09-21T15:47:50.343 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-09-21T15:47:50.406 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-09-21T15:47:50.406 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-09-21T15:47:50.440 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install rbd-mirror 2024-09-21T15:47:50.556 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-09-21T15:47:50.648 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : rbd-fuse-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-21T15:47:50.993 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: rbd-fuse-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-21T15:47:51.037 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:37 ago on Sat 21 Sep 2024 03:46:14 PM UTC. 2024-09-21T15:47:51.172 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-21T15:47:51.173 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-21T15:47:51.173 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repo Size 2024-09-21T15:47:51.173 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-21T15:47:51.173 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-21T15:47:51.173 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-mirror x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 3.0 M 2024-09-21T15:47:51.173 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:51.174 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-21T15:47:51.174 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-21T15:47:51.174 INFO:teuthology.orchestra.run.smithi100.stdout:Install 1 Package 2024-09-21T15:47:51.174 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:51.174 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 3.0 M 2024-09-21T15:47:51.174 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 12 M 2024-09-21T15:47:51.174 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-21T15:47:51.256 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : rbd-fuse-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-21T15:47:51.256 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:51.256 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-09-21T15:47:51.256 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-fuse-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:47:51.256 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:51.256 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-21T15:47:51.527 INFO:teuthology.orchestra.run.smithi100.stdout:rbd-mirror-18.2.4-855.g19e2a2be.el9.x86_64.rpm 8.5 MB/s | 3.0 MB 00:00 2024-09-21T15:47:51.527 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-21T15:47:51.528 INFO:teuthology.orchestra.run.smithi100.stdout:Total 8.4 MB/s | 3.0 MB 00:00 2024-09-21T15:47:51.528 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-21T15:47:51.537 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-21T15:47:51.537 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-21T15:47:51.548 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install rbd-mirror 2024-09-21T15:47:51.616 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-21T15:47:51.616 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-21T15:47:52.035 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-21T15:47:52.142 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:38 ago on Sat 21 Sep 2024 03:46:14 PM UTC. 2024-09-21T15:47:52.211 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : rbd-mirror-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-21T15:47:52.236 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: rbd-mirror-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-21T15:47:52.236 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T15:47:52.236 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-21T15:47:52.236 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T15:47:52.236 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T15:47:52.236 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:52.278 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-21T15:47:52.279 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-21T15:47:52.279 INFO:teuthology.orchestra.run.smithi089.stdout: Package Arch Version Repo Size 2024-09-21T15:47:52.279 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-21T15:47:52.279 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-09-21T15:47:52.279 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-mirror x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 3.0 M 2024-09-21T15:47:52.279 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:52.279 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-09-21T15:47:52.279 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-21T15:47:52.279 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2024-09-21T15:47:52.279 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:52.280 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 3.0 M 2024-09-21T15:47:52.280 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 12 M 2024-09-21T15:47:52.280 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-09-21T15:47:52.610 INFO:teuthology.orchestra.run.smithi089.stdout:rbd-mirror-18.2.4-855.g19e2a2be.el9.x86_64.rpm 9.2 MB/s | 3.0 MB 00:00 2024-09-21T15:47:52.610 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-09-21T15:47:52.610 INFO:teuthology.orchestra.run.smithi089.stdout:Total 9.1 MB/s | 3.0 MB 00:00 2024-09-21T15:47:52.610 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-09-21T15:47:52.617 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-09-21T15:47:52.617 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-09-21T15:47:52.697 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-09-21T15:47:52.697 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-09-21T15:47:53.117 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-09-21T15:47:53.191 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : rbd-mirror-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-21T15:47:53.216 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: rbd-mirror-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-21T15:47:53.216 INFO:teuthology.orchestra.run.smithi089.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T15:47:53.216 INFO:teuthology.orchestra.run.smithi089.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-21T15:47:53.216 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T15:47:53.216 INFO:teuthology.orchestra.run.smithi089.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T15:47:53.217 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:53.292 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : rbd-mirror-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-21T15:47:53.292 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:53.292 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-21T15:47:53.292 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-mirror-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:47:53.292 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:53.292 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-21T15:47:53.547 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install rbd-nbd 2024-09-21T15:47:54.135 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:40 ago on Sat 21 Sep 2024 03:46:14 PM UTC. 2024-09-21T15:47:54.274 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-09-21T15:47:54.275 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-21T15:47:54.275 INFO:teuthology.orchestra.run.smithi100.stdout: Package Architecture Version Repository Size 2024-09-21T15:47:54.275 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-21T15:47:54.275 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-09-21T15:47:54.275 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-nbd x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 172 k 2024-09-21T15:47:54.275 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:54.275 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-09-21T15:47:54.275 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-09-21T15:47:54.275 INFO:teuthology.orchestra.run.smithi100.stdout:Install 1 Package 2024-09-21T15:47:54.276 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:54.276 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 172 k 2024-09-21T15:47:54.276 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 494 k 2024-09-21T15:47:54.276 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-09-21T15:47:54.316 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : rbd-mirror-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-21T15:47:54.316 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:54.317 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-09-21T15:47:54.317 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-mirror-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:47:54.317 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:54.317 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-21T15:47:54.481 INFO:teuthology.orchestra.run.smithi100.stdout:rbd-nbd-18.2.4-855.g19e2a2be.el9.x86_64.rpm 842 kB/s | 172 kB 00:00 2024-09-21T15:47:54.481 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-09-21T15:47:54.482 INFO:teuthology.orchestra.run.smithi100.stdout:Total 835 kB/s | 172 kB 00:00 2024-09-21T15:47:54.482 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-09-21T15:47:54.490 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-09-21T15:47:54.490 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-09-21T15:47:54.553 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-09-21T15:47:54.553 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-09-21T15:47:54.572 DEBUG:teuthology.orchestra.run.smithi089:> sudo yum -y install rbd-nbd 2024-09-21T15:47:54.706 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-09-21T15:47:54.812 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : rbd-nbd-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-21T15:47:55.173 INFO:teuthology.orchestra.run.smithi089.stdout:Last metadata expiration check: 0:01:41 ago on Sat 21 Sep 2024 03:46:14 PM UTC. 2024-09-21T15:47:55.246 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: rbd-nbd-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-21T15:47:55.307 INFO:teuthology.orchestra.run.smithi089.stdout:Dependencies resolved. 2024-09-21T15:47:55.308 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-21T15:47:55.308 INFO:teuthology.orchestra.run.smithi089.stdout: Package Architecture Version Repository Size 2024-09-21T15:47:55.308 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-21T15:47:55.308 INFO:teuthology.orchestra.run.smithi089.stdout:Installing: 2024-09-21T15:47:55.308 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-nbd x86_64 2:18.2.4-855.g19e2a2be.el9 ceph 172 k 2024-09-21T15:47:55.308 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:55.308 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction Summary 2024-09-21T15:47:55.309 INFO:teuthology.orchestra.run.smithi089.stdout:================================================================================ 2024-09-21T15:47:55.309 INFO:teuthology.orchestra.run.smithi089.stdout:Install 1 Package 2024-09-21T15:47:55.309 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:55.309 INFO:teuthology.orchestra.run.smithi089.stdout:Total download size: 172 k 2024-09-21T15:47:55.309 INFO:teuthology.orchestra.run.smithi089.stdout:Installed size: 494 k 2024-09-21T15:47:55.309 INFO:teuthology.orchestra.run.smithi089.stdout:Downloading Packages: 2024-09-21T15:47:55.483 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : rbd-nbd-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-21T15:47:55.484 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:55.484 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-09-21T15:47:55.484 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-nbd-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:47:55.484 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:55.484 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-09-21T15:47:55.513 INFO:teuthology.orchestra.run.smithi089.stdout:rbd-nbd-18.2.4-855.g19e2a2be.el9.x86_64.rpm 845 kB/s | 172 kB 00:00 2024-09-21T15:47:55.514 INFO:teuthology.orchestra.run.smithi089.stdout:-------------------------------------------------------------------------------- 2024-09-21T15:47:55.514 INFO:teuthology.orchestra.run.smithi089.stdout:Total 837 kB/s | 172 kB 00:00 2024-09-21T15:47:55.514 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction check 2024-09-21T15:47:55.522 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction check succeeded. 2024-09-21T15:47:55.522 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction test 2024-09-21T15:47:55.589 INFO:teuthology.orchestra.run.smithi089.stdout:Transaction test succeeded. 2024-09-21T15:47:55.589 INFO:teuthology.orchestra.run.smithi089.stdout:Running transaction 2024-09-21T15:47:55.700 DEBUG:teuthology.parallel:result is None 2024-09-21T15:47:55.746 INFO:teuthology.orchestra.run.smithi089.stdout: Preparing : 1/1 2024-09-21T15:47:55.846 INFO:teuthology.orchestra.run.smithi089.stdout: Installing : rbd-nbd-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-21T15:47:56.298 INFO:teuthology.orchestra.run.smithi089.stdout: Running scriptlet: rbd-nbd-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-21T15:47:56.545 INFO:teuthology.orchestra.run.smithi089.stdout: Verifying : rbd-nbd-2:18.2.4-855.g19e2a2be.el9.x86_64 1/1 2024-09-21T15:47:56.546 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:56.546 INFO:teuthology.orchestra.run.smithi089.stdout:Installed: 2024-09-21T15:47:56.546 INFO:teuthology.orchestra.run.smithi089.stdout: rbd-nbd-2:18.2.4-855.g19e2a2be.el9.x86_64 2024-09-21T15:47:56.546 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:56.546 INFO:teuthology.orchestra.run.smithi089.stdout:Complete! 2024-09-21T15:47:56.783 DEBUG:teuthology.parallel:result is None 2024-09-21T15:47:56.783 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-21T15:47:56.928 DEBUG:teuthology.orchestra.run.smithi089:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-21T15:47:56.949 INFO:teuthology.orchestra.run.smithi089.stdout:18.2.4-855.g19e2a2be.el9 2024-09-21T15:47:56.950 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-855.g19e2a2be.el9 2024-09-21T15:47:56.950 INFO:teuthology.task.install:The correct ceph version 18.2.4-855.g19e2a2be is installed. 2024-09-21T15:47:56.951 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=19e2a2beb6f41a3ab62650e38614eb29b59dcbb9 2024-09-21T15:47:57.088 DEBUG:teuthology.orchestra.run.smithi100:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-21T15:47:57.113 INFO:teuthology.orchestra.run.smithi100.stdout:18.2.4-855.g19e2a2be.el9 2024-09-21T15:47:57.113 INFO:teuthology.packaging:The installed version of ceph is 18.2.4-855.g19e2a2be.el9 2024-09-21T15:47:57.113 INFO:teuthology.task.install:The correct ceph version 18.2.4-855.g19e2a2be is installed. 2024-09-21T15:47:57.115 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-21T15:47:57.115 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-21T15:47:57.115 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-21T15:47:57.141 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-21T15:47:57.141 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-21T15:47:57.183 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-21T15:47:57.183 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-21T15:47:57.183 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/usr/bin/daemon-helper 2024-09-21T15:47:57.209 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-21T15:47:57.273 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-21T15:47:57.273 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/usr/bin/daemon-helper 2024-09-21T15:47:57.302 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-21T15:47:57.371 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-21T15:47:57.371 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-21T15:47:57.372 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-21T15:47:57.396 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-21T15:47:57.462 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-21T15:47:57.462 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-21T15:47:57.491 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-21T15:47:57.558 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-21T15:47:57.558 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-21T15:47:57.558 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/usr/bin/stdin-killer 2024-09-21T15:47:57.582 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-21T15:47:57.647 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-21T15:47:57.647 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/usr/bin/stdin-killer 2024-09-21T15:47:57.672 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-21T15:47:57.736 INFO:teuthology.run_tasks:Running task ceph... 2024-09-21T15:47:57.810 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-21T15:47:57.811 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 777 /var/log/ceph 2024-09-21T15:47:57.813 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 777 /var/log/ceph 2024-09-21T15:47:57.838 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-21T15:47:57.838 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-21T15:47:57.878 DEBUG:teuthology.orchestra.run.smithi100:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-21T15:47:57.904 INFO:tasks.ceph:Creating extra log directories... 2024-09-21T15:47:57.904 DEBUG:teuthology.orchestra.run.smithi089:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-21T15:47:57.942 DEBUG:teuthology.orchestra.run.smithi100:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-21T15:47:57.981 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-21T15:47:57.981 INFO:tasks.ceph:config {'conf': {'client': {'rbd qos iops burst': 100, 'rbd qos iops limit': 50, 'rbd qos schedule tick min': 100}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-21T15:47:57.982 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi/7909998', 'branch': 'reef', 'description': 'rbd/thrash/{base/install clusters/{fixed-2 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/rbd_fsx_rate_limit}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7909998', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'machine': {'cpus': 1, 'disk': 40, 'ram': 8000}, 'volumes': {'count': 4, 'size': 30}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'reef'}, 'ceph': {'conf': {'client': {'rbd qos iops burst': 100, 'rbd qos iops limit': 50, 'rbd qos schedule tick min': 100}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from'], 'sha1': '19e2a2beb6f41a3ab62650e38614eb29b59dcbb9'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '19e2a2beb6f41a3ab62650e38614eb29b59dcbb9'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'reef', 'sha1': 'a477b1894d5c5a613f23260ca0e6b96d79652ba7'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 930, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 4623, 'sha1': '19e2a2beb6f41a3ab62650e38614eb29b59dcbb9', 'sleep_before_teardown': 0, 'subset': '80/128', 'suite': 'rbd', 'suite_branch': 'reef', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_a477b1894d5c5a613f23260ca0e6b96d79652ba7/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'a477b1894d5c5a613f23260ca0e6b96d79652ba7', 'targets': {'smithi089.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBL48+d9q3iRAlAALR56tg45jkIWRKg3kxeLC7ODFrnc0wrGWBhJ3sdf2RloSTvD2MYJZdpXqS52grJZ+bX7OOY0=', 'smithi100.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNsIHVy2wcThj5nHSqnIlV7S/E775ijm/9edjgh4SScgmVCAEGeOpWl4D118mk+UYWyL14cxEgdrN64tRvQqx8c='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'timeout': 1200}}, {'rbd_fsx': {'clients': ['client.0'], 'ops': 6000}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-17_22:16:03', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2902'} 2024-09-21T15:47:57.982 DEBUG:teuthology.orchestra.run.smithi089:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-21T15:47:58.007 DEBUG:teuthology.orchestra.run.smithi100:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-21T15:47:58.028 DEBUG:teuthology.orchestra.run.smithi089:> sudo install -d -m0777 -- /var/run/ceph 2024-09-21T15:47:58.064 DEBUG:teuthology.orchestra.run.smithi100:> sudo install -d -m0777 -- /var/run/ceph 2024-09-21T15:47:58.093 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-21T15:47:58.093 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/scratch_devs of=/dev/stdout 2024-09-21T15:47:58.142 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-21T15:47:58.142 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_1 2024-09-21T15:47:58.197 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-21T15:47:58.197 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T15:47:58.197 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-21T15:47:58.197 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T15:47:58.197 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T15:47:58.197 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2024-09-21 15:47:53.467651408 +0000 2024-09-21T15:47:58.197 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2024-09-21 15:45:33.988519976 +0000 2024-09-21T15:47:58.197 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2024-09-21 15:45:33.988519976 +0000 2024-09-21T15:47:58.197 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2024-09-21 15:45:33.988519976 +0000 2024-09-21T15:47:58.198 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-21T15:47:58.259 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2024-09-21T15:47:58.260 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2024-09-21T15:47:58.260 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.000206213 s, 2.5 MB/s 2024-09-21T15:47:58.261 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-21T15:47:58.317 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_2 2024-09-21T15:47:58.372 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-21T15:47:58.372 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T15:47:58.372 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-09-21T15:47:58.372 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T15:47:58.372 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T15:47:58.372 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2024-09-21 15:47:53.467651408 +0000 2024-09-21T15:47:58.372 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2024-09-21 15:45:34.321512315 +0000 2024-09-21T15:47:58.372 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2024-09-21 15:45:34.321512315 +0000 2024-09-21T15:47:58.372 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2024-09-21 15:45:34.321512315 +0000 2024-09-21T15:47:58.372 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-21T15:47:58.435 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2024-09-21T15:47:58.435 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2024-09-21T15:47:58.435 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.000198732 s, 2.6 MB/s 2024-09-21T15:47:58.436 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-21T15:47:58.493 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_3 2024-09-21T15:47:58.548 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-21T15:47:58.548 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T15:47:58.548 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-09-21T15:47:58.548 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T15:47:58.548 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T15:47:58.548 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2024-09-21 15:47:53.467651408 +0000 2024-09-21T15:47:58.548 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2024-09-21 15:45:34.642504930 +0000 2024-09-21T15:47:58.549 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2024-09-21 15:45:34.642504930 +0000 2024-09-21T15:47:58.549 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2024-09-21 15:45:34.642504930 +0000 2024-09-21T15:47:58.549 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-21T15:47:58.611 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2024-09-21T15:47:58.611 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2024-09-21T15:47:58.611 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.000186189 s, 2.7 MB/s 2024-09-21T15:47:58.612 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-21T15:47:58.668 DEBUG:teuthology.orchestra.run.smithi089:> stat /dev/vg_nvme/lv_4 2024-09-21T15:47:58.723 INFO:teuthology.orchestra.run.smithi089.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-21T15:47:58.723 INFO:teuthology.orchestra.run.smithi089.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T15:47:58.723 INFO:teuthology.orchestra.run.smithi089.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-21T15:47:58.723 INFO:teuthology.orchestra.run.smithi089.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T15:47:58.723 INFO:teuthology.orchestra.run.smithi089.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T15:47:58.723 INFO:teuthology.orchestra.run.smithi089.stdout:Access: 2024-09-21 15:47:53.468651385 +0000 2024-09-21T15:47:58.723 INFO:teuthology.orchestra.run.smithi089.stdout:Modify: 2024-09-21 15:45:34.966497476 +0000 2024-09-21T15:47:58.723 INFO:teuthology.orchestra.run.smithi089.stdout:Change: 2024-09-21 15:45:34.966497476 +0000 2024-09-21T15:47:58.723 INFO:teuthology.orchestra.run.smithi089.stdout: Birth: 2024-09-21 15:45:34.966497476 +0000 2024-09-21T15:47:58.723 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-21T15:47:58.786 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records in 2024-09-21T15:47:58.786 INFO:teuthology.orchestra.run.smithi089.stderr:1+0 records out 2024-09-21T15:47:58.786 INFO:teuthology.orchestra.run.smithi089.stderr:512 bytes copied, 0.00022276 s, 2.3 MB/s 2024-09-21T15:47:58.787 DEBUG:teuthology.orchestra.run.smithi089:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-21T15:47:58.843 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-21T15:47:58.843 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-21T15:47:58.843 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/scratch_devs of=/dev/stdout 2024-09-21T15:47:58.859 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-21T15:47:58.859 DEBUG:teuthology.orchestra.run.smithi100:> stat /dev/vg_nvme/lv_1 2024-09-21T15:47:58.914 INFO:teuthology.orchestra.run.smithi100.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-21T15:47:58.915 INFO:teuthology.orchestra.run.smithi100.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T15:47:58.915 INFO:teuthology.orchestra.run.smithi100.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-21T15:47:58.915 INFO:teuthology.orchestra.run.smithi100.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T15:47:58.915 INFO:teuthology.orchestra.run.smithi100.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T15:47:58.915 INFO:teuthology.orchestra.run.smithi100.stdout:Access: 2024-09-21 15:47:52.518440206 +0000 2024-09-21T15:47:58.915 INFO:teuthology.orchestra.run.smithi100.stdout:Modify: 2024-09-21 15:45:33.974739905 +0000 2024-09-21T15:47:58.915 INFO:teuthology.orchestra.run.smithi100.stdout:Change: 2024-09-21 15:45:33.974739905 +0000 2024-09-21T15:47:58.915 INFO:teuthology.orchestra.run.smithi100.stdout: Birth: 2024-09-21 15:45:33.974739905 +0000 2024-09-21T15:47:58.915 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-21T15:47:58.978 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records in 2024-09-21T15:47:58.979 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records out 2024-09-21T15:47:58.979 INFO:teuthology.orchestra.run.smithi100.stderr:512 bytes copied, 0.000210178 s, 2.4 MB/s 2024-09-21T15:47:58.980 DEBUG:teuthology.orchestra.run.smithi100:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-21T15:47:59.036 DEBUG:teuthology.orchestra.run.smithi100:> stat /dev/vg_nvme/lv_2 2024-09-21T15:47:59.092 INFO:teuthology.orchestra.run.smithi100.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-21T15:47:59.092 INFO:teuthology.orchestra.run.smithi100.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T15:47:59.092 INFO:teuthology.orchestra.run.smithi100.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-09-21T15:47:59.093 INFO:teuthology.orchestra.run.smithi100.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T15:47:59.093 INFO:teuthology.orchestra.run.smithi100.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T15:47:59.093 INFO:teuthology.orchestra.run.smithi100.stdout:Access: 2024-09-21 15:47:52.518440206 +0000 2024-09-21T15:47:59.093 INFO:teuthology.orchestra.run.smithi100.stdout:Modify: 2024-09-21 15:45:34.304731124 +0000 2024-09-21T15:47:59.093 INFO:teuthology.orchestra.run.smithi100.stdout:Change: 2024-09-21 15:45:34.304731124 +0000 2024-09-21T15:47:59.093 INFO:teuthology.orchestra.run.smithi100.stdout: Birth: 2024-09-21 15:45:34.304731124 +0000 2024-09-21T15:47:59.093 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-21T15:47:59.156 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records in 2024-09-21T15:47:59.156 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records out 2024-09-21T15:47:59.156 INFO:teuthology.orchestra.run.smithi100.stderr:512 bytes copied, 0.00020475 s, 2.5 MB/s 2024-09-21T15:47:59.157 DEBUG:teuthology.orchestra.run.smithi100:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-21T15:47:59.213 DEBUG:teuthology.orchestra.run.smithi100:> stat /dev/vg_nvme/lv_3 2024-09-21T15:47:59.268 INFO:teuthology.orchestra.run.smithi100.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-21T15:47:59.268 INFO:teuthology.orchestra.run.smithi100.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T15:47:59.268 INFO:teuthology.orchestra.run.smithi100.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-09-21T15:47:59.268 INFO:teuthology.orchestra.run.smithi100.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T15:47:59.268 INFO:teuthology.orchestra.run.smithi100.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T15:47:59.268 INFO:teuthology.orchestra.run.smithi100.stdout:Access: 2024-09-21 15:47:52.519440180 +0000 2024-09-21T15:47:59.268 INFO:teuthology.orchestra.run.smithi100.stdout:Modify: 2024-09-21 15:45:34.627722530 +0000 2024-09-21T15:47:59.268 INFO:teuthology.orchestra.run.smithi100.stdout:Change: 2024-09-21 15:45:34.627722530 +0000 2024-09-21T15:47:59.268 INFO:teuthology.orchestra.run.smithi100.stdout: Birth: 2024-09-21 15:45:34.627722530 +0000 2024-09-21T15:47:59.269 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-21T15:47:59.331 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records in 2024-09-21T15:47:59.331 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records out 2024-09-21T15:47:59.331 INFO:teuthology.orchestra.run.smithi100.stderr:512 bytes copied, 0.000184323 s, 2.8 MB/s 2024-09-21T15:47:59.332 DEBUG:teuthology.orchestra.run.smithi100:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-21T15:47:59.388 DEBUG:teuthology.orchestra.run.smithi100:> stat /dev/vg_nvme/lv_4 2024-09-21T15:47:59.443 INFO:teuthology.orchestra.run.smithi100.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-21T15:47:59.443 INFO:teuthology.orchestra.run.smithi100.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T15:47:59.443 INFO:teuthology.orchestra.run.smithi100.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-09-21T15:47:59.443 INFO:teuthology.orchestra.run.smithi100.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T15:47:59.443 INFO:teuthology.orchestra.run.smithi100.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T15:47:59.443 INFO:teuthology.orchestra.run.smithi100.stdout:Access: 2024-09-21 15:47:52.519440180 +0000 2024-09-21T15:47:59.443 INFO:teuthology.orchestra.run.smithi100.stdout:Modify: 2024-09-21 15:45:34.951713909 +0000 2024-09-21T15:47:59.443 INFO:teuthology.orchestra.run.smithi100.stdout:Change: 2024-09-21 15:45:34.951713909 +0000 2024-09-21T15:47:59.443 INFO:teuthology.orchestra.run.smithi100.stdout: Birth: 2024-09-21 15:45:34.951713909 +0000 2024-09-21T15:47:59.444 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-21T15:47:59.506 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records in 2024-09-21T15:47:59.506 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records out 2024-09-21T15:47:59.506 INFO:teuthology.orchestra.run.smithi100.stderr:512 bytes copied, 0.000212079 s, 2.4 MB/s 2024-09-21T15:47:59.507 DEBUG:teuthology.orchestra.run.smithi100:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-21T15:47:59.562 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-21T15:47:59.563 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi089.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi100.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-09-21T15:47:59.563 INFO:tasks.ceph:Generating config... 2024-09-21T15:47:59.564 INFO:tasks.ceph:[client] rbd qos iops burst = 100 2024-09-21T15:47:59.564 INFO:tasks.ceph:[client] rbd qos iops limit = 50 2024-09-21T15:47:59.564 INFO:tasks.ceph:[client] rbd qos schedule tick min = 100 2024-09-21T15:47:59.564 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-21T15:47:59.564 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-21T15:47:59.564 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-21T15:47:59.564 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-21T15:47:59.564 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-21T15:47:59.564 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-21T15:47:59.564 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-21T15:47:59.564 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-21T15:47:59.565 INFO:tasks.ceph:[osd] bdev async discard = True 2024-09-21T15:47:59.565 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-09-21T15:47:59.565 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-09-21T15:47:59.565 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-21T15:47:59.565 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-21T15:47:59.565 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-21T15:47:59.565 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-21T15:47:59.565 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-21T15:47:59.565 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-21T15:47:59.565 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-21T15:47:59.565 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-21T15:47:59.565 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-21T15:47:59.565 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-21T15:47:59.565 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-09-21T15:47:59.565 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-21T15:47:59.566 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-09-21T15:47:59.566 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-21T15:47:59.566 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-09-21T15:47:59.566 INFO:tasks.ceph:Setting up mon.a... 2024-09-21T15:47:59.566 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-21T15:47:59.618 INFO:teuthology.orchestra.run.smithi089.stdout:creating /etc/ceph/ceph.keyring 2024-09-21T15:47:59.620 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-21T15:47:59.663 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-21T15:47:59.728 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.89'), ('mon.c', '[v2:172.21.15.89:3301,v1:172.21.15.89:6790]'), ('mon.b', '172.21.15.100')] 2024-09-21T15:47:59.728 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.89,[v2:172.21.15.89:3301,v1:172.21.15.89:6790],172.21.15.100', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'rbd qos iops burst': 100, 'rbd qos iops limit': 50, 'rbd qos schedule tick min': 100}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-09-21T15:47:59.729 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-21T15:47:59.729 DEBUG:teuthology.orchestra.run.smithi089:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-21T15:47:59.783 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.89 --addv c '[v2:172.21.15.89:3301,v1:172.21.15.89:6790]' --add b 172.21.15.100 --print /home/ubuntu/cephtest/ceph.monmap 2024-09-21T15:47:59.869 INFO:teuthology.orchestra.run.smithi089.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-21T15:47:59.869 INFO:teuthology.orchestra.run.smithi089.stdout:monmaptool: generated fsid 4a9b64ea-3aaf-4a40-8ce8-02163a0f6e12 2024-09-21T15:47:59.869 INFO:teuthology.orchestra.run.smithi089.stdout:setting min_mon_release = pacific 2024-09-21T15:47:59.869 INFO:teuthology.orchestra.run.smithi089.stdout:epoch 0 2024-09-21T15:47:59.869 INFO:teuthology.orchestra.run.smithi089.stdout:fsid 4a9b64ea-3aaf-4a40-8ce8-02163a0f6e12 2024-09-21T15:47:59.869 INFO:teuthology.orchestra.run.smithi089.stdout:last_changed 2024-09-21T15:47:59.869280+0000 2024-09-21T15:47:59.869 INFO:teuthology.orchestra.run.smithi089.stdout:created 2024-09-21T15:47:59.869280+0000 2024-09-21T15:47:59.870 INFO:teuthology.orchestra.run.smithi089.stdout:min_mon_release 16 (pacific) 2024-09-21T15:47:59.870 INFO:teuthology.orchestra.run.smithi089.stdout:election_strategy: 1 2024-09-21T15:47:59.870 INFO:teuthology.orchestra.run.smithi089.stdout:0: [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] mon.a 2024-09-21T15:47:59.870 INFO:teuthology.orchestra.run.smithi089.stdout:1: [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon.b 2024-09-21T15:47:59.870 INFO:teuthology.orchestra.run.smithi089.stdout:2: [v2:172.21.15.89:3301/0,v1:172.21.15.89:6790/0] mon.c 2024-09-21T15:47:59.870 INFO:teuthology.orchestra.run.smithi089.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-21T15:47:59.871 DEBUG:teuthology.orchestra.run.smithi089:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-21T15:47:59.885 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 4a9b64ea-3aaf-4a40-8ce8-02163a0f6e12... 2024-09-21T15:47:59.886 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-21T15:47:59.928 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-21T15:47:59.968 INFO:teuthology.orchestra.run.smithi089.stdout:[global] 2024-09-21T15:47:59.969 INFO:teuthology.orchestra.run.smithi089.stdout: chdir = "" 2024-09-21T15:47:59.969 INFO:teuthology.orchestra.run.smithi089.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-21T15:47:59.969 INFO:teuthology.orchestra.run.smithi089.stdout: auth supported = cephx 2024-09-21T15:47:59.969 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:59.969 INFO:teuthology.orchestra.run.smithi089.stdout: filestore xattr use omap = true 2024-09-21T15:47:59.969 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:59.969 INFO:teuthology.orchestra.run.smithi089.stdout: mon clock drift allowed = 1.000 2024-09-21T15:47:59.969 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:59.969 INFO:teuthology.orchestra.run.smithi089.stdout: osd crush chooseleaf type = 0 2024-09-21T15:47:59.969 INFO:teuthology.orchestra.run.smithi089.stdout: auth debug = true 2024-09-21T15:47:59.969 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:59.969 INFO:teuthology.orchestra.run.smithi089.stdout: ms die on old message = true 2024-09-21T15:47:59.969 INFO:teuthology.orchestra.run.smithi089.stdout: ms die on bug = true 2024-09-21T15:47:59.969 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:59.970 INFO:teuthology.orchestra.run.smithi089.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-21T15:47:59.970 INFO:teuthology.orchestra.run.smithi089.stdout: mon pg warn max object skew = 0 2024-09-21T15:47:59.970 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:59.970 INFO:teuthology.orchestra.run.smithi089.stdout: # disable pg_autoscaler by default for new pools 2024-09-21T15:47:59.970 INFO:teuthology.orchestra.run.smithi089.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-21T15:47:59.970 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:59.970 INFO:teuthology.orchestra.run.smithi089.stdout: osd pool default size = 2 2024-09-21T15:47:59.970 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:59.970 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd allow primary affinity = true 2024-09-21T15:47:59.970 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd allow pg remap = true 2024-09-21T15:47:59.970 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on legacy crush tunables = false 2024-09-21T15:47:59.970 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on crush straw calc version zero = false 2024-09-21T15:47:59.970 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on no sortbitwise = false 2024-09-21T15:47:59.970 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on osd down out interval zero = false 2024-09-21T15:47:59.971 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on too few osds = false 2024-09-21T15:47:59.971 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-21T15:47:59.971 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-21T15:47:59.971 INFO:teuthology.orchestra.run.smithi089.stdout: mon_allow_pool_size_one = true 2024-09-21T15:47:59.971 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:59.971 INFO:teuthology.orchestra.run.smithi089.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-21T15:47:59.971 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:59.971 INFO:teuthology.orchestra.run.smithi089.stdout: osd default data pool replay window = 5 2024-09-21T15:47:59.971 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:59.971 INFO:teuthology.orchestra.run.smithi089.stdout: mon allow pool delete = true 2024-09-21T15:47:59.971 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:59.971 INFO:teuthology.orchestra.run.smithi089.stdout: mon cluster log file level = debug 2024-09-21T15:47:59.971 INFO:teuthology.orchestra.run.smithi089.stdout: debug asserts on shutdown = true 2024-09-21T15:47:59.971 INFO:teuthology.orchestra.run.smithi089.stdout: mon health detail to clog = false 2024-09-21T15:47:59.971 INFO:teuthology.orchestra.run.smithi089.stdout: mon host = "172.21.15.89,[v2:172.21.15.89:3301,v1:172.21.15.89:6790],172.21.15.100" 2024-09-21T15:47:59.972 INFO:teuthology.orchestra.run.smithi089.stdout: mon client directed command retry = 5 2024-09-21T15:47:59.972 INFO:teuthology.orchestra.run.smithi089.stdout: mon warn on pool no app = False 2024-09-21T15:47:59.972 INFO:teuthology.orchestra.run.smithi089.stdout: ms inject socket failures = 5000 2024-09-21T15:47:59.972 INFO:teuthology.orchestra.run.smithi089.stdout: fsid = 4a9b64ea-3aaf-4a40-8ce8-02163a0f6e12 2024-09-21T15:47:59.972 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:59.972 INFO:teuthology.orchestra.run.smithi089.stdout:[osd] 2024-09-21T15:47:59.972 INFO:teuthology.orchestra.run.smithi089.stdout: osd journal size = 100 2024-09-21T15:47:59.972 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:59.972 INFO:teuthology.orchestra.run.smithi089.stdout: osd scrub load threshold = 5.0 2024-09-21T15:47:59.972 INFO:teuthology.orchestra.run.smithi089.stdout: osd scrub max interval = 600 2024-09-21T15:47:59.972 INFO:teuthology.orchestra.run.smithi089.stdout: osd mclock profile = high_recovery_ops 2024-09-21T15:47:59.972 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:59.972 INFO:teuthology.orchestra.run.smithi089.stdout: osd recover clone overlap = true 2024-09-21T15:47:59.972 INFO:teuthology.orchestra.run.smithi089.stdout: osd recovery max chunk = 1048576 2024-09-21T15:47:59.972 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:59.973 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug shutdown = true 2024-09-21T15:47:59.973 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug op order = true 2024-09-21T15:47:59.973 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug verify stray on activate = true 2024-09-21T15:47:59.973 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:59.973 INFO:teuthology.orchestra.run.smithi089.stdout: osd open classes on start = true 2024-09-21T15:47:59.973 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug pg log writeout = true 2024-09-21T15:47:59.973 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:59.973 INFO:teuthology.orchestra.run.smithi089.stdout: osd deep scrub update digest min age = 30 2024-09-21T15:47:59.973 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:59.973 INFO:teuthology.orchestra.run.smithi089.stdout: osd map max advance = 10 2024-09-21T15:47:59.973 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:59.973 INFO:teuthology.orchestra.run.smithi089.stdout: journal zero on create = true 2024-09-21T15:47:59.973 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:59.973 INFO:teuthology.orchestra.run.smithi089.stdout: filestore ondisk finisher threads = 3 2024-09-21T15:47:59.973 INFO:teuthology.orchestra.run.smithi089.stdout: filestore apply finisher threads = 3 2024-09-21T15:47:59.974 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:59.974 INFO:teuthology.orchestra.run.smithi089.stdout: bdev debug aio = true 2024-09-21T15:47:59.974 INFO:teuthology.orchestra.run.smithi089.stdout: osd debug misdirected ops = true 2024-09-21T15:47:59.974 INFO:teuthology.orchestra.run.smithi089.stdout: bdev async discard = True 2024-09-21T15:47:59.974 INFO:teuthology.orchestra.run.smithi089.stdout: bdev enable discard = True 2024-09-21T15:47:59.974 INFO:teuthology.orchestra.run.smithi089.stdout: bluestore allocator = stupid 2024-09-21T15:47:59.974 INFO:teuthology.orchestra.run.smithi089.stdout: bluestore block size = 96636764160 2024-09-21T15:47:59.974 INFO:teuthology.orchestra.run.smithi089.stdout: bluestore fsck on mount = True 2024-09-21T15:47:59.974 INFO:teuthology.orchestra.run.smithi089.stdout: debug bluefs = 1/20 2024-09-21T15:47:59.974 INFO:teuthology.orchestra.run.smithi089.stdout: debug bluestore = 1/20 2024-09-21T15:47:59.974 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2024-09-21T15:47:59.974 INFO:teuthology.orchestra.run.smithi089.stdout: debug osd = 20 2024-09-21T15:47:59.974 INFO:teuthology.orchestra.run.smithi089.stdout: debug rocksdb = 4/10 2024-09-21T15:47:59.974 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd backfillfull_ratio = 0.85 2024-09-21T15:47:59.974 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd full ratio = 0.9 2024-09-21T15:47:59.974 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd nearfull ratio = 0.8 2024-09-21T15:47:59.975 INFO:teuthology.orchestra.run.smithi089.stdout: osd blocked scrub grace period = 3600 2024-09-21T15:47:59.975 INFO:teuthology.orchestra.run.smithi089.stdout: osd failsafe full ratio = 0.95 2024-09-21T15:47:59.975 INFO:teuthology.orchestra.run.smithi089.stdout: osd max markdown count = 1000 2024-09-21T15:47:59.975 INFO:teuthology.orchestra.run.smithi089.stdout: osd objectstore = bluestore 2024-09-21T15:47:59.975 INFO:teuthology.orchestra.run.smithi089.stdout: osd shutdown pgref assert = True 2024-09-21T15:47:59.975 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:59.975 INFO:teuthology.orchestra.run.smithi089.stdout:[mgr] 2024-09-21T15:47:59.975 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2024-09-21T15:47:59.975 INFO:teuthology.orchestra.run.smithi089.stdout: debug mgr = 20 2024-09-21T15:47:59.975 INFO:teuthology.orchestra.run.smithi089.stdout: debug mon = 20 2024-09-21T15:47:59.975 INFO:teuthology.orchestra.run.smithi089.stdout: debug auth = 20 2024-09-21T15:47:59.975 INFO:teuthology.orchestra.run.smithi089.stdout: mon reweight min pgs per osd = 4 2024-09-21T15:47:59.975 INFO:teuthology.orchestra.run.smithi089.stdout: mon reweight min bytes per osd = 10 2024-09-21T15:47:59.975 INFO:teuthology.orchestra.run.smithi089.stdout: mgr/telemetry/nag = false 2024-09-21T15:47:59.975 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:59.975 INFO:teuthology.orchestra.run.smithi089.stdout:[mon] 2024-09-21T15:47:59.976 INFO:teuthology.orchestra.run.smithi089.stdout: debug ms = 1 2024-09-21T15:47:59.976 INFO:teuthology.orchestra.run.smithi089.stdout: debug mon = 20 2024-09-21T15:47:59.976 INFO:teuthology.orchestra.run.smithi089.stdout: debug paxos = 20 2024-09-21T15:47:59.976 INFO:teuthology.orchestra.run.smithi089.stdout: debug auth = 20 2024-09-21T15:47:59.976 INFO:teuthology.orchestra.run.smithi089.stdout: mon data avail warn = 5 2024-09-21T15:47:59.976 INFO:teuthology.orchestra.run.smithi089.stdout: mon mgr mkfs grace = 240 2024-09-21T15:47:59.976 INFO:teuthology.orchestra.run.smithi089.stdout: mon reweight min pgs per osd = 4 2024-09-21T15:47:59.976 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd reporter subtree level = osd 2024-09-21T15:47:59.976 INFO:teuthology.orchestra.run.smithi089.stdout: mon osd prime pg temp = true 2024-09-21T15:47:59.976 INFO:teuthology.orchestra.run.smithi089.stdout: mon reweight min bytes per osd = 10 2024-09-21T15:47:59.976 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:59.976 INFO:teuthology.orchestra.run.smithi089.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-21T15:47:59.976 INFO:teuthology.orchestra.run.smithi089.stdout: auth mon ticket ttl = 660 # 11m 2024-09-21T15:47:59.976 INFO:teuthology.orchestra.run.smithi089.stdout: auth service ticket ttl = 240 # 4m 2024-09-21T15:47:59.976 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:59.976 INFO:teuthology.orchestra.run.smithi089.stdout: # don't complain about insecure global_id in the test suite 2024-09-21T15:47:59.977 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-21T15:47:59.977 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-21T15:47:59.977 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:59.977 INFO:teuthology.orchestra.run.smithi089.stdout: # 1m isn't quite enough 2024-09-21T15:47:59.977 INFO:teuthology.orchestra.run.smithi089.stdout: mon_down_mkfs_grace = 2m 2024-09-21T15:47:59.977 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:59.977 INFO:teuthology.orchestra.run.smithi089.stdout: mon_warn_on_filestore_osds = false 2024-09-21T15:47:59.977 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:47:59.977 INFO:teuthology.orchestra.run.smithi089.stdout:[client] 2024-09-21T15:47:59.977 INFO:teuthology.orchestra.run.smithi089.stdout: rgw cache enabled = true 2024-09-21T15:47:59.977 INFO:teuthology.orchestra.run.smithi089.stdout: rgw enable ops log = true 2024-09-21T15:47:59.977 INFO:teuthology.orchestra.run.smithi089.stdout: rgw enable usage log = true 2024-09-21T15:47:59.977 INFO:teuthology.orchestra.run.smithi089.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-21T15:47:59.977 INFO:teuthology.orchestra.run.smithi089.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-21T15:47:59.977 INFO:teuthology.orchestra.run.smithi089.stdout: rbd qos iops burst = 100 2024-09-21T15:47:59.978 INFO:teuthology.orchestra.run.smithi089.stdout: rbd qos iops limit = 50 2024-09-21T15:47:59.978 INFO:teuthology.orchestra.run.smithi089.stdout: rbd qos schedule tick min = 100 2024-09-21T15:47:59.978 INFO:teuthology.orchestra.run.smithi089.stdout:[mon.a] 2024-09-21T15:47:59.978 INFO:teuthology.orchestra.run.smithi089.stdout:[mon.c] 2024-09-21T15:47:59.978 INFO:teuthology.orchestra.run.smithi089.stdout:[mon.b] 2024-09-21T15:47:59.978 INFO:teuthology.orchestra.run.smithi100.stdout:[global] 2024-09-21T15:47:59.978 INFO:teuthology.orchestra.run.smithi100.stdout: chdir = "" 2024-09-21T15:47:59.978 INFO:teuthology.orchestra.run.smithi100.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-21T15:47:59.978 INFO:teuthology.orchestra.run.smithi100.stdout: auth supported = cephx 2024-09-21T15:47:59.978 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:59.978 INFO:teuthology.orchestra.run.smithi100.stdout: filestore xattr use omap = true 2024-09-21T15:47:59.979 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:59.979 INFO:teuthology.orchestra.run.smithi100.stdout: mon clock drift allowed = 1.000 2024-09-21T15:47:59.979 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:59.979 INFO:teuthology.orchestra.run.smithi100.stdout: osd crush chooseleaf type = 0 2024-09-21T15:47:59.979 INFO:teuthology.orchestra.run.smithi100.stdout: auth debug = true 2024-09-21T15:47:59.979 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:59.979 INFO:teuthology.orchestra.run.smithi100.stdout: ms die on old message = true 2024-09-21T15:47:59.979 INFO:teuthology.orchestra.run.smithi100.stdout: ms die on bug = true 2024-09-21T15:47:59.979 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:59.979 INFO:teuthology.orchestra.run.smithi100.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-21T15:47:59.979 INFO:teuthology.orchestra.run.smithi100.stdout: mon pg warn max object skew = 0 2024-09-21T15:47:59.979 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:59.979 INFO:teuthology.orchestra.run.smithi100.stdout: # disable pg_autoscaler by default for new pools 2024-09-21T15:47:59.979 INFO:teuthology.orchestra.run.smithi100.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-21T15:47:59.979 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:59.980 INFO:teuthology.orchestra.run.smithi100.stdout: osd pool default size = 2 2024-09-21T15:47:59.980 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:59.980 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd allow primary affinity = true 2024-09-21T15:47:59.980 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd allow pg remap = true 2024-09-21T15:47:59.980 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on legacy crush tunables = false 2024-09-21T15:47:59.980 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on crush straw calc version zero = false 2024-09-21T15:47:59.980 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on no sortbitwise = false 2024-09-21T15:47:59.980 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on osd down out interval zero = false 2024-09-21T15:47:59.980 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on too few osds = false 2024-09-21T15:47:59.980 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-21T15:47:59.980 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-21T15:47:59.980 INFO:teuthology.orchestra.run.smithi100.stdout: mon_allow_pool_size_one = true 2024-09-21T15:47:59.980 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:59.980 INFO:teuthology.orchestra.run.smithi100.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-21T15:47:59.980 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:59.980 INFO:teuthology.orchestra.run.smithi100.stdout: osd default data pool replay window = 5 2024-09-21T15:47:59.981 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:59.981 INFO:teuthology.orchestra.run.smithi100.stdout: mon allow pool delete = true 2024-09-21T15:47:59.981 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:59.981 INFO:teuthology.orchestra.run.smithi100.stdout: mon cluster log file level = debug 2024-09-21T15:47:59.981 INFO:teuthology.orchestra.run.smithi100.stdout: debug asserts on shutdown = true 2024-09-21T15:47:59.981 INFO:teuthology.orchestra.run.smithi100.stdout: mon health detail to clog = false 2024-09-21T15:47:59.981 INFO:teuthology.orchestra.run.smithi100.stdout: mon host = "172.21.15.89,[v2:172.21.15.89:3301,v1:172.21.15.89:6790],172.21.15.100" 2024-09-21T15:47:59.981 INFO:teuthology.orchestra.run.smithi100.stdout: mon client directed command retry = 5 2024-09-21T15:47:59.981 INFO:teuthology.orchestra.run.smithi100.stdout: mon warn on pool no app = False 2024-09-21T15:47:59.981 INFO:teuthology.orchestra.run.smithi100.stdout: ms inject socket failures = 5000 2024-09-21T15:47:59.981 INFO:teuthology.orchestra.run.smithi100.stdout: fsid = 4a9b64ea-3aaf-4a40-8ce8-02163a0f6e12 2024-09-21T15:47:59.981 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:59.981 INFO:teuthology.orchestra.run.smithi100.stdout:[osd] 2024-09-21T15:47:59.981 INFO:teuthology.orchestra.run.smithi100.stdout: osd journal size = 100 2024-09-21T15:47:59.981 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:59.982 INFO:teuthology.orchestra.run.smithi100.stdout: osd scrub load threshold = 5.0 2024-09-21T15:47:59.982 INFO:teuthology.orchestra.run.smithi100.stdout: osd scrub max interval = 600 2024-09-21T15:47:59.982 INFO:teuthology.orchestra.run.smithi100.stdout: osd mclock profile = high_recovery_ops 2024-09-21T15:47:59.982 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:59.982 INFO:teuthology.orchestra.run.smithi100.stdout: osd recover clone overlap = true 2024-09-21T15:47:59.982 INFO:teuthology.orchestra.run.smithi100.stdout: osd recovery max chunk = 1048576 2024-09-21T15:47:59.982 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:59.982 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug shutdown = true 2024-09-21T15:47:59.982 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug op order = true 2024-09-21T15:47:59.982 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug verify stray on activate = true 2024-09-21T15:47:59.982 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:59.982 INFO:teuthology.orchestra.run.smithi100.stdout: osd open classes on start = true 2024-09-21T15:47:59.982 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug pg log writeout = true 2024-09-21T15:47:59.982 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:59.982 INFO:teuthology.orchestra.run.smithi100.stdout: osd deep scrub update digest min age = 30 2024-09-21T15:47:59.983 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:59.983 INFO:teuthology.orchestra.run.smithi100.stdout: osd map max advance = 10 2024-09-21T15:47:59.983 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:59.983 INFO:teuthology.orchestra.run.smithi100.stdout: journal zero on create = true 2024-09-21T15:47:59.983 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:59.983 INFO:teuthology.orchestra.run.smithi100.stdout: filestore ondisk finisher threads = 3 2024-09-21T15:47:59.983 INFO:teuthology.orchestra.run.smithi100.stdout: filestore apply finisher threads = 3 2024-09-21T15:47:59.983 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:59.983 INFO:teuthology.orchestra.run.smithi100.stdout: bdev debug aio = true 2024-09-21T15:47:59.983 INFO:teuthology.orchestra.run.smithi100.stdout: osd debug misdirected ops = true 2024-09-21T15:47:59.983 INFO:teuthology.orchestra.run.smithi100.stdout: bdev async discard = True 2024-09-21T15:47:59.983 INFO:teuthology.orchestra.run.smithi100.stdout: bdev enable discard = True 2024-09-21T15:47:59.983 INFO:teuthology.orchestra.run.smithi100.stdout: bluestore allocator = stupid 2024-09-21T15:47:59.983 INFO:teuthology.orchestra.run.smithi100.stdout: bluestore block size = 96636764160 2024-09-21T15:47:59.983 INFO:teuthology.orchestra.run.smithi100.stdout: bluestore fsck on mount = True 2024-09-21T15:47:59.984 INFO:teuthology.orchestra.run.smithi100.stdout: debug bluefs = 1/20 2024-09-21T15:47:59.984 INFO:teuthology.orchestra.run.smithi100.stdout: debug bluestore = 1/20 2024-09-21T15:47:59.984 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-09-21T15:47:59.984 INFO:teuthology.orchestra.run.smithi100.stdout: debug osd = 20 2024-09-21T15:47:59.984 INFO:teuthology.orchestra.run.smithi100.stdout: debug rocksdb = 4/10 2024-09-21T15:47:59.984 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd backfillfull_ratio = 0.85 2024-09-21T15:47:59.984 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd full ratio = 0.9 2024-09-21T15:47:59.984 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd nearfull ratio = 0.8 2024-09-21T15:47:59.984 INFO:teuthology.orchestra.run.smithi100.stdout: osd blocked scrub grace period = 3600 2024-09-21T15:47:59.984 INFO:teuthology.orchestra.run.smithi100.stdout: osd failsafe full ratio = 0.95 2024-09-21T15:47:59.984 INFO:teuthology.orchestra.run.smithi100.stdout: osd max markdown count = 1000 2024-09-21T15:47:59.984 INFO:teuthology.orchestra.run.smithi100.stdout: osd objectstore = bluestore 2024-09-21T15:47:59.984 INFO:teuthology.orchestra.run.smithi100.stdout: osd shutdown pgref assert = True 2024-09-21T15:47:59.984 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:59.985 INFO:teuthology.orchestra.run.smithi100.stdout:[mgr] 2024-09-21T15:47:59.985 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-09-21T15:47:59.985 INFO:teuthology.orchestra.run.smithi100.stdout: debug mgr = 20 2024-09-21T15:47:59.985 INFO:teuthology.orchestra.run.smithi100.stdout: debug mon = 20 2024-09-21T15:47:59.985 INFO:teuthology.orchestra.run.smithi100.stdout: debug auth = 20 2024-09-21T15:47:59.985 INFO:teuthology.orchestra.run.smithi100.stdout: mon reweight min pgs per osd = 4 2024-09-21T15:47:59.985 INFO:teuthology.orchestra.run.smithi100.stdout: mon reweight min bytes per osd = 10 2024-09-21T15:47:59.985 INFO:teuthology.orchestra.run.smithi100.stdout: mgr/telemetry/nag = false 2024-09-21T15:47:59.985 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:59.985 INFO:teuthology.orchestra.run.smithi100.stdout:[mon] 2024-09-21T15:47:59.985 INFO:teuthology.orchestra.run.smithi100.stdout: debug ms = 1 2024-09-21T15:47:59.985 INFO:teuthology.orchestra.run.smithi100.stdout: debug mon = 20 2024-09-21T15:47:59.985 INFO:teuthology.orchestra.run.smithi100.stdout: debug paxos = 20 2024-09-21T15:47:59.985 INFO:teuthology.orchestra.run.smithi100.stdout: debug auth = 20 2024-09-21T15:47:59.985 INFO:teuthology.orchestra.run.smithi100.stdout: mon data avail warn = 5 2024-09-21T15:47:59.986 INFO:teuthology.orchestra.run.smithi100.stdout: mon mgr mkfs grace = 240 2024-09-21T15:47:59.986 INFO:teuthology.orchestra.run.smithi100.stdout: mon reweight min pgs per osd = 4 2024-09-21T15:47:59.986 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd reporter subtree level = osd 2024-09-21T15:47:59.986 INFO:teuthology.orchestra.run.smithi100.stdout: mon osd prime pg temp = true 2024-09-21T15:47:59.986 INFO:teuthology.orchestra.run.smithi100.stdout: mon reweight min bytes per osd = 10 2024-09-21T15:47:59.986 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:59.986 INFO:teuthology.orchestra.run.smithi100.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-21T15:47:59.986 INFO:teuthology.orchestra.run.smithi100.stdout: auth mon ticket ttl = 660 # 11m 2024-09-21T15:47:59.986 INFO:teuthology.orchestra.run.smithi100.stdout: auth service ticket ttl = 240 # 4m 2024-09-21T15:47:59.986 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:59.986 INFO:teuthology.orchestra.run.smithi100.stdout: # don't complain about insecure global_id in the test suite 2024-09-21T15:47:59.986 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-21T15:47:59.986 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-21T15:47:59.986 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:59.986 INFO:teuthology.orchestra.run.smithi100.stdout: # 1m isn't quite enough 2024-09-21T15:47:59.987 INFO:teuthology.orchestra.run.smithi100.stdout: mon_down_mkfs_grace = 2m 2024-09-21T15:47:59.987 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:59.987 INFO:teuthology.orchestra.run.smithi100.stdout: mon_warn_on_filestore_osds = false 2024-09-21T15:47:59.987 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-09-21T15:47:59.987 INFO:teuthology.orchestra.run.smithi100.stdout:[client] 2024-09-21T15:47:59.987 INFO:teuthology.orchestra.run.smithi100.stdout: rgw cache enabled = true 2024-09-21T15:47:59.987 INFO:teuthology.orchestra.run.smithi100.stdout: rgw enable ops log = true 2024-09-21T15:47:59.987 INFO:teuthology.orchestra.run.smithi100.stdout: rgw enable usage log = true 2024-09-21T15:47:59.987 INFO:teuthology.orchestra.run.smithi100.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-21T15:47:59.987 INFO:teuthology.orchestra.run.smithi100.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-21T15:47:59.987 INFO:teuthology.orchestra.run.smithi100.stdout: rbd qos iops burst = 100 2024-09-21T15:47:59.987 INFO:teuthology.orchestra.run.smithi100.stdout: rbd qos iops limit = 50 2024-09-21T15:47:59.987 INFO:teuthology.orchestra.run.smithi100.stdout: rbd qos schedule tick min = 100 2024-09-21T15:47:59.987 INFO:teuthology.orchestra.run.smithi100.stdout:[mon.a] 2024-09-21T15:47:59.987 INFO:teuthology.orchestra.run.smithi100.stdout:[mon.c] 2024-09-21T15:47:59.988 INFO:teuthology.orchestra.run.smithi100.stdout:[mon.b] 2024-09-21T15:47:59.989 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-21T15:47:59.989 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-21T15:48:00.082 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-21T15:48:00.083 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-21T15:48:00.083 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-21T15:48:00.097 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-21T15:48:00.097 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-21T15:48:00.152 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi089.front.sepia.ceph.com 2024-09-21T15:48:00.152 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-21T15:48:00.152 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-21T15:48:00.152 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-21T15:48:00.225 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-21T15:48:00.225 DEBUG:teuthology.orchestra.run.smithi089:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-21T15:48:00.280 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi100.front.sepia.ceph.com 2024-09-21T15:48:00.280 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-21T15:48:00.280 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-21T15:48:00.280 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-21T15:48:00.314 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-21T15:48:00.314 DEBUG:teuthology.orchestra.run.smithi100:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-21T15:48:00.370 INFO:tasks.ceph:Setting up mon nodes... 2024-09-21T15:48:00.370 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-21T15:48:00.371 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-09-21T15:48:00.420 INFO:teuthology.orchestra.run.smithi089.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-09-21T15:48:00.423 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-21T15:48:00.477 INFO:teuthology.orchestra.run.smithi100.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-21T15:48:00.479 INFO:tasks.ceph:Setting up mds nodes... 2024-09-21T15:48:00.480 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-21T15:48:00.480 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-21T15:48:00.519 INFO:teuthology.orchestra.run.smithi089.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-21T15:48:00.532 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-21T15:48:00.572 INFO:teuthology.orchestra.run.smithi100.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-09-21T15:48:00.584 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-21T15:48:00.585 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi089.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi100.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-09-21T15:48:00.585 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-21T15:48:00.609 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-21T15:48:00.609 INFO:tasks.ceph:role: osd.0 2024-09-21T15:48:00.609 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi089.front.sepia.ceph.com 2024-09-21T15:48:00.609 DEBUG:teuthology.orchestra.run.smithi089:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-21T15:48:00.681 INFO:teuthology.orchestra.run.smithi089.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T15:48:00.681 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T15:48:00.682 INFO:teuthology.orchestra.run.smithi089.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T15:48:00.682 INFO:teuthology.orchestra.run.smithi089.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-21T15:48:00.682 INFO:teuthology.orchestra.run.smithi089.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T15:48:00.682 INFO:teuthology.orchestra.run.smithi089.stdout: = sunit=0 swidth=0 blks 2024-09-21T15:48:00.682 INFO:teuthology.orchestra.run.smithi089.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T15:48:00.682 INFO:teuthology.orchestra.run.smithi089.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-21T15:48:00.682 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T15:48:00.682 INFO:teuthology.orchestra.run.smithi089.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T15:48:01.192 INFO:teuthology.orchestra.run.smithi089.stdout:Discarding blocks...Done. 2024-09-21T15:48:01.194 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi089.front.sepia.ceph.com -o noatime 2024-09-21T15:48:01.194 DEBUG:teuthology.orchestra.run.smithi089:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-21T15:48:01.341 DEBUG:teuthology.orchestra.run.smithi089:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-21T15:48:01.373 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-21T15:48:01.439 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-21T15:48:01.439 INFO:tasks.ceph:role: osd.1 2024-09-21T15:48:01.439 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi089.front.sepia.ceph.com 2024-09-21T15:48:01.439 DEBUG:teuthology.orchestra.run.smithi089:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-21T15:48:01.512 INFO:teuthology.orchestra.run.smithi089.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T15:48:01.512 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T15:48:01.512 INFO:teuthology.orchestra.run.smithi089.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T15:48:01.512 INFO:teuthology.orchestra.run.smithi089.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-21T15:48:01.512 INFO:teuthology.orchestra.run.smithi089.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T15:48:01.512 INFO:teuthology.orchestra.run.smithi089.stdout: = sunit=0 swidth=0 blks 2024-09-21T15:48:01.512 INFO:teuthology.orchestra.run.smithi089.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T15:48:01.512 INFO:teuthology.orchestra.run.smithi089.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-21T15:48:01.513 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T15:48:01.513 INFO:teuthology.orchestra.run.smithi089.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T15:48:01.966 INFO:teuthology.orchestra.run.smithi089.stdout:Discarding blocks...Done. 2024-09-21T15:48:01.968 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi089.front.sepia.ceph.com -o noatime 2024-09-21T15:48:01.968 DEBUG:teuthology.orchestra.run.smithi089:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-21T15:48:02.063 DEBUG:teuthology.orchestra.run.smithi089:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-21T15:48:02.091 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-21T15:48:02.156 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-21T15:48:02.157 INFO:tasks.ceph:role: osd.2 2024-09-21T15:48:02.157 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi089.front.sepia.ceph.com 2024-09-21T15:48:02.157 DEBUG:teuthology.orchestra.run.smithi089:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-21T15:48:02.229 INFO:teuthology.orchestra.run.smithi089.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T15:48:02.229 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T15:48:02.229 INFO:teuthology.orchestra.run.smithi089.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T15:48:02.229 INFO:teuthology.orchestra.run.smithi089.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-21T15:48:02.229 INFO:teuthology.orchestra.run.smithi089.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T15:48:02.229 INFO:teuthology.orchestra.run.smithi089.stdout: = sunit=0 swidth=0 blks 2024-09-21T15:48:02.229 INFO:teuthology.orchestra.run.smithi089.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T15:48:02.229 INFO:teuthology.orchestra.run.smithi089.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-21T15:48:02.229 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T15:48:02.229 INFO:teuthology.orchestra.run.smithi089.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T15:48:02.723 INFO:teuthology.orchestra.run.smithi089.stdout:Discarding blocks...Done. 2024-09-21T15:48:02.725 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi089.front.sepia.ceph.com -o noatime 2024-09-21T15:48:02.725 DEBUG:teuthology.orchestra.run.smithi089:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-21T15:48:02.876 DEBUG:teuthology.orchestra.run.smithi089:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-21T15:48:02.905 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-21T15:48:02.969 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-09-21T15:48:02.970 INFO:tasks.ceph:role: osd.3 2024-09-21T15:48:02.970 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi089.front.sepia.ceph.com 2024-09-21T15:48:02.970 DEBUG:teuthology.orchestra.run.smithi089:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-21T15:48:03.041 INFO:teuthology.orchestra.run.smithi089.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T15:48:03.041 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T15:48:03.041 INFO:teuthology.orchestra.run.smithi089.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T15:48:03.041 INFO:teuthology.orchestra.run.smithi089.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-21T15:48:03.041 INFO:teuthology.orchestra.run.smithi089.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T15:48:03.042 INFO:teuthology.orchestra.run.smithi089.stdout: = sunit=0 swidth=0 blks 2024-09-21T15:48:03.042 INFO:teuthology.orchestra.run.smithi089.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T15:48:03.042 INFO:teuthology.orchestra.run.smithi089.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-21T15:48:03.042 INFO:teuthology.orchestra.run.smithi089.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T15:48:03.042 INFO:teuthology.orchestra.run.smithi089.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T15:48:03.567 INFO:teuthology.orchestra.run.smithi089.stdout:Discarding blocks...Done. 2024-09-21T15:48:03.570 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi089.front.sepia.ceph.com -o noatime 2024-09-21T15:48:03.570 DEBUG:teuthology.orchestra.run.smithi089:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-09-21T15:48:03.718 DEBUG:teuthology.orchestra.run.smithi089:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-21T15:48:03.751 DEBUG:teuthology.orchestra.run.smithi089:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T15:48:03.856 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-21T15:48:03.855+0000 7f79b5022640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-21T15:48:03.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-21T15:48:03.855+0000 7f79b5022640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-21T15:48:03.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-21T15:48:03.855+0000 7f79b5022640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-21T15:48:03.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-21T15:48:03.855+0000 7f79b5022640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-21T15:48:03.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-21T15:48:03.855+0000 7f79b5022640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-09-21T15:48:03.857 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-21T15:48:03.856+0000 7f79b5022640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-21T15:48:06.709 DEBUG:teuthology.orchestra.run.smithi089:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-21T15:48:06.739 DEBUG:teuthology.orchestra.run.smithi089:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T15:48:06.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-21T15:48:06.827+0000 7fddf720f640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-21T15:48:06.828 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-21T15:48:06.827+0000 7fddf720f640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-21T15:48:06.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-21T15:48:06.827+0000 7fddf720f640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-21T15:48:06.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-21T15:48:06.827+0000 7fddf720f640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-21T15:48:06.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-21T15:48:06.827+0000 7fddf720f640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-09-21T15:48:06.829 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-21T15:48:06.828+0000 7fddf720f640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-21T15:48:09.688 DEBUG:teuthology.orchestra.run.smithi089:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-21T15:48:09.717 DEBUG:teuthology.orchestra.run.smithi089:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T15:48:09.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-21T15:48:09.807+0000 7fca6da1a640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-21T15:48:09.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-21T15:48:09.807+0000 7fca6da1a640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-21T15:48:09.808 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-21T15:48:09.807+0000 7fca6da1a640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-21T15:48:09.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-21T15:48:09.807+0000 7fca6da1a640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-21T15:48:09.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-21T15:48:09.807+0000 7fca6da1a640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-09-21T15:48:09.809 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-21T15:48:09.807+0000 7fca6da1a640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-21T15:48:12.658 DEBUG:teuthology.orchestra.run.smithi089:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-21T15:48:12.683 DEBUG:teuthology.orchestra.run.smithi089:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T15:48:12.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-21T15:48:12.766+0000 7f15c2f45640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-21T15:48:12.767 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-21T15:48:12.766+0000 7f15c2f45640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-21T15:48:12.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-21T15:48:12.766+0000 7f15c2f45640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-21T15:48:12.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-21T15:48:12.766+0000 7f15c2f45640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-21T15:48:12.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-21T15:48:12.766+0000 7f15c2f45640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-09-21T15:48:12.768 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-21T15:48:12.767+0000 7f15c2f45640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-21T15:48:15.629 DEBUG:teuthology.orchestra.run.smithi089:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-21T15:48:15.658 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-21T15:48:15.689 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-21T15:48:15.689 INFO:tasks.ceph:role: osd.4 2024-09-21T15:48:15.690 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi100.front.sepia.ceph.com 2024-09-21T15:48:15.690 DEBUG:teuthology.orchestra.run.smithi100:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-21T15:48:15.766 INFO:teuthology.orchestra.run.smithi100.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T15:48:15.766 INFO:teuthology.orchestra.run.smithi100.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T15:48:15.766 INFO:teuthology.orchestra.run.smithi100.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T15:48:15.766 INFO:teuthology.orchestra.run.smithi100.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-21T15:48:15.766 INFO:teuthology.orchestra.run.smithi100.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T15:48:15.767 INFO:teuthology.orchestra.run.smithi100.stdout: = sunit=0 swidth=0 blks 2024-09-21T15:48:15.767 INFO:teuthology.orchestra.run.smithi100.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T15:48:15.767 INFO:teuthology.orchestra.run.smithi100.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-21T15:48:15.767 INFO:teuthology.orchestra.run.smithi100.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T15:48:15.767 INFO:teuthology.orchestra.run.smithi100.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T15:48:16.270 INFO:teuthology.orchestra.run.smithi100.stdout:Discarding blocks...Done. 2024-09-21T15:48:16.272 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi100.front.sepia.ceph.com -o noatime 2024-09-21T15:48:16.272 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-09-21T15:48:16.395 DEBUG:teuthology.orchestra.run.smithi100:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-21T15:48:16.430 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-21T15:48:16.497 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-21T15:48:16.497 INFO:tasks.ceph:role: osd.5 2024-09-21T15:48:16.498 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi100.front.sepia.ceph.com 2024-09-21T15:48:16.498 DEBUG:teuthology.orchestra.run.smithi100:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-21T15:48:16.572 INFO:teuthology.orchestra.run.smithi100.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T15:48:16.572 INFO:teuthology.orchestra.run.smithi100.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T15:48:16.572 INFO:teuthology.orchestra.run.smithi100.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T15:48:16.572 INFO:teuthology.orchestra.run.smithi100.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-21T15:48:16.572 INFO:teuthology.orchestra.run.smithi100.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T15:48:16.573 INFO:teuthology.orchestra.run.smithi100.stdout: = sunit=0 swidth=0 blks 2024-09-21T15:48:16.573 INFO:teuthology.orchestra.run.smithi100.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T15:48:16.573 INFO:teuthology.orchestra.run.smithi100.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-21T15:48:16.573 INFO:teuthology.orchestra.run.smithi100.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T15:48:16.573 INFO:teuthology.orchestra.run.smithi100.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T15:48:17.050 INFO:teuthology.orchestra.run.smithi100.stdout:Discarding blocks...Done. 2024-09-21T15:48:17.051 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi100.front.sepia.ceph.com -o noatime 2024-09-21T15:48:17.051 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-09-21T15:48:17.151 DEBUG:teuthology.orchestra.run.smithi100:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-21T15:48:17.180 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-09-21T15:48:17.246 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-21T15:48:17.246 INFO:tasks.ceph:role: osd.6 2024-09-21T15:48:17.246 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi100.front.sepia.ceph.com 2024-09-21T15:48:17.246 DEBUG:teuthology.orchestra.run.smithi100:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-21T15:48:17.318 INFO:teuthology.orchestra.run.smithi100.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T15:48:17.318 INFO:teuthology.orchestra.run.smithi100.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T15:48:17.318 INFO:teuthology.orchestra.run.smithi100.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T15:48:17.318 INFO:teuthology.orchestra.run.smithi100.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-21T15:48:17.318 INFO:teuthology.orchestra.run.smithi100.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T15:48:17.318 INFO:teuthology.orchestra.run.smithi100.stdout: = sunit=0 swidth=0 blks 2024-09-21T15:48:17.318 INFO:teuthology.orchestra.run.smithi100.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T15:48:17.319 INFO:teuthology.orchestra.run.smithi100.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-21T15:48:17.319 INFO:teuthology.orchestra.run.smithi100.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T15:48:17.319 INFO:teuthology.orchestra.run.smithi100.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T15:48:17.800 INFO:teuthology.orchestra.run.smithi100.stdout:Discarding blocks...Done. 2024-09-21T15:48:17.803 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi100.front.sepia.ceph.com -o noatime 2024-09-21T15:48:17.803 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-09-21T15:48:17.935 DEBUG:teuthology.orchestra.run.smithi100:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-09-21T15:48:17.964 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-09-21T15:48:18.028 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-09-21T15:48:18.028 INFO:tasks.ceph:role: osd.7 2024-09-21T15:48:18.029 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi100.front.sepia.ceph.com 2024-09-21T15:48:18.029 DEBUG:teuthology.orchestra.run.smithi100:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-09-21T15:48:18.100 INFO:teuthology.orchestra.run.smithi100.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-09-21T15:48:18.101 INFO:teuthology.orchestra.run.smithi100.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-21T15:48:18.101 INFO:teuthology.orchestra.run.smithi100.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-21T15:48:18.101 INFO:teuthology.orchestra.run.smithi100.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-09-21T15:48:18.101 INFO:teuthology.orchestra.run.smithi100.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-21T15:48:18.101 INFO:teuthology.orchestra.run.smithi100.stdout: = sunit=0 swidth=0 blks 2024-09-21T15:48:18.101 INFO:teuthology.orchestra.run.smithi100.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-21T15:48:18.101 INFO:teuthology.orchestra.run.smithi100.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-09-21T15:48:18.101 INFO:teuthology.orchestra.run.smithi100.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-21T15:48:18.101 INFO:teuthology.orchestra.run.smithi100.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-21T15:48:18.617 INFO:teuthology.orchestra.run.smithi100.stdout:Discarding blocks...Done. 2024-09-21T15:48:18.619 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi100.front.sepia.ceph.com -o noatime 2024-09-21T15:48:18.619 DEBUG:teuthology.orchestra.run.smithi100:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-09-21T15:48:18.747 DEBUG:teuthology.orchestra.run.smithi100:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-09-21T15:48:18.780 DEBUG:teuthology.orchestra.run.smithi100:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T15:48:18.875 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-21T15:48:18.874+0000 7f8a48bc1640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-21T15:48:18.875 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-21T15:48:18.874+0000 7f8a48bc1640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-21T15:48:18.875 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-21T15:48:18.874+0000 7f8a48bc1640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-21T15:48:18.875 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-21T15:48:18.874+0000 7f8a48bc1640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-21T15:48:18.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-21T15:48:18.874+0000 7f8a48bc1640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-09-21T15:48:18.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-21T15:48:18.874+0000 7f8a48bc1640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-21T15:48:21.723 DEBUG:teuthology.orchestra.run.smithi100:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-21T15:48:21.753 DEBUG:teuthology.orchestra.run.smithi100:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T15:48:21.843 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-21T15:48:21.842+0000 7f22ad71b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-21T15:48:21.844 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-21T15:48:21.842+0000 7f22ad71b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-21T15:48:21.844 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-21T15:48:21.842+0000 7f22ad71b640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-21T15:48:21.844 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-21T15:48:21.842+0000 7f22ad71b640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-21T15:48:21.844 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-21T15:48:21.842+0000 7f22ad71b640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-09-21T15:48:21.844 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-21T15:48:21.843+0000 7f22ad71b640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-21T15:48:24.688 DEBUG:teuthology.orchestra.run.smithi100:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-21T15:48:24.718 DEBUG:teuthology.orchestra.run.smithi100:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T15:48:24.808 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-21T15:48:24.806+0000 7feb3a4d4640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-09-21T15:48:24.808 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-21T15:48:24.806+0000 7feb3a4d4640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-09-21T15:48:24.808 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-21T15:48:24.806+0000 7feb3a4d4640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-09-21T15:48:24.808 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-21T15:48:24.806+0000 7feb3a4d4640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-09-21T15:48:24.808 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-21T15:48:24.806+0000 7feb3a4d4640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-09-21T15:48:24.808 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-21T15:48:24.807+0000 7feb3a4d4640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-09-21T15:48:27.655 DEBUG:teuthology.orchestra.run.smithi100:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-21T15:48:27.680 DEBUG:teuthology.orchestra.run.smithi100:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-21T15:48:27.765 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-21T15:48:27.764+0000 7f43f5bb2640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-09-21T15:48:27.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-21T15:48:27.764+0000 7f43f5bb2640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-09-21T15:48:27.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-21T15:48:27.764+0000 7f43f5bb2640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-09-21T15:48:27.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-21T15:48:27.764+0000 7f43f5bb2640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-09-21T15:48:27.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-21T15:48:27.764+0000 7f43f5bb2640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-09-21T15:48:27.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-21T15:48:27.764+0000 7f43f5bb2640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-09-21T15:48:30.611 DEBUG:teuthology.orchestra.run.smithi100:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-21T15:48:30.640 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-21T15:48:30.641 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-21T15:48:30.641 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-09-21T15:48:30.671 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-21T15:48:30.671 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-21T15:48:30.740 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-21T15:48:30.740 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-21T15:48:30.808 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-21T15:48:30.808 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-21T15:48:30.875 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-21T15:48:30.876 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-21T15:48:30.943 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-21T15:48:30.943 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-21T15:48:30.973 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-21T15:48:30.973 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-21T15:48:31.042 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-21T15:48:31.042 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-21T15:48:31.110 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-21T15:48:31.110 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-09-21T15:48:31.177 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-21T15:48:31.177 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-09-21T15:48:31.243 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-21T15:48:31.243 DEBUG:teuthology.orchestra.run.smithi089:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-21T15:48:31.261 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-21T15:48:31.262 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-09-21T15:48:31.299 INFO:tasks.ceph:Adding keys to all mons... 2024-09-21T15:48:31.299 DEBUG:teuthology.orchestra.run.smithi089:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-21T15:48:31.304 DEBUG:teuthology.orchestra.run.smithi100:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-21T15:48:31.342 INFO:teuthology.orchestra.run.smithi089.stdout:[mgr.y] 2024-09-21T15:48:31.342 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQCw6u5mN9AWGRAA86cLUuXgVtZ3Z9Jc98YSPw== 2024-09-21T15:48:31.342 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.0] 2024-09-21T15:48:31.343 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQCz6u5mK9UKMxAALvWsJQ29C2h/m/2ahngfkw== 2024-09-21T15:48:31.343 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.1] 2024-09-21T15:48:31.343 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQC26u5mO6lcMRAAw8LWZ02c8zK65Yhj2OLV5w== 2024-09-21T15:48:31.343 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.2] 2024-09-21T15:48:31.343 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQC56u5mFlkqMBAAJhlY/KQnJ7ZDDWJEum+S8Q== 2024-09-21T15:48:31.343 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.3] 2024-09-21T15:48:31.343 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQC86u5mG267LRAAIZx7pEDBCOCFUtQUNp54Bw== 2024-09-21T15:48:31.343 INFO:teuthology.orchestra.run.smithi089.stdout:[mgr.x] 2024-09-21T15:48:31.343 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQCw6u5mw21zHBAAAKmAJR97Jzx2pjHZmH9kXg== 2024-09-21T15:48:31.343 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.4] 2024-09-21T15:48:31.343 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQDC6u5mqOQnNBAAXo75Dtoph4aL1lNJ+/Gxcg== 2024-09-21T15:48:31.343 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.5] 2024-09-21T15:48:31.343 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQDF6u5msRFFMhAARFvubBQE+1CdlRzJwEcZdw== 2024-09-21T15:48:31.343 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.6] 2024-09-21T15:48:31.344 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQDI6u5m2CgfMBAAtlQZJjAtgoqvcVayN7sqrA== 2024-09-21T15:48:31.344 INFO:teuthology.orchestra.run.smithi089.stdout:[osd.7] 2024-09-21T15:48:31.344 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQDL6u5maP2bLRAAOuZHxtThCx+Hw/ZpfoaGmg== 2024-09-21T15:48:31.344 INFO:teuthology.orchestra.run.smithi089.stdout:[client.0] 2024-09-21T15:48:31.344 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQCw6u5m+/b4HhAA6h5dr+yG0FHHav/VaSt+1g== 2024-09-21T15:48:31.344 INFO:teuthology.orchestra.run.smithi089.stdout:[client.1] 2024-09-21T15:48:31.344 INFO:teuthology.orchestra.run.smithi089.stdout: key = AQCw6u5mxxwiIhAAmjVhTEEI8QlwBwzhlu0TCg== 2024-09-21T15:48:31.363 INFO:teuthology.orchestra.run.smithi100.stdout:[mgr.y] 2024-09-21T15:48:31.363 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQCw6u5mN9AWGRAA86cLUuXgVtZ3Z9Jc98YSPw== 2024-09-21T15:48:31.363 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.0] 2024-09-21T15:48:31.363 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQCz6u5mK9UKMxAALvWsJQ29C2h/m/2ahngfkw== 2024-09-21T15:48:31.363 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.1] 2024-09-21T15:48:31.363 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQC26u5mO6lcMRAAw8LWZ02c8zK65Yhj2OLV5w== 2024-09-21T15:48:31.363 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.2] 2024-09-21T15:48:31.363 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQC56u5mFlkqMBAAJhlY/KQnJ7ZDDWJEum+S8Q== 2024-09-21T15:48:31.363 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.3] 2024-09-21T15:48:31.363 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQC86u5mG267LRAAIZx7pEDBCOCFUtQUNp54Bw== 2024-09-21T15:48:31.364 INFO:teuthology.orchestra.run.smithi100.stdout:[mgr.x] 2024-09-21T15:48:31.364 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQCw6u5mw21zHBAAAKmAJR97Jzx2pjHZmH9kXg== 2024-09-21T15:48:31.364 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.4] 2024-09-21T15:48:31.364 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQDC6u5mqOQnNBAAXo75Dtoph4aL1lNJ+/Gxcg== 2024-09-21T15:48:31.364 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.5] 2024-09-21T15:48:31.364 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQDF6u5msRFFMhAARFvubBQE+1CdlRzJwEcZdw== 2024-09-21T15:48:31.364 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.6] 2024-09-21T15:48:31.364 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQDI6u5m2CgfMBAAtlQZJjAtgoqvcVayN7sqrA== 2024-09-21T15:48:31.364 INFO:teuthology.orchestra.run.smithi100.stdout:[osd.7] 2024-09-21T15:48:31.364 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQDL6u5maP2bLRAAOuZHxtThCx+Hw/ZpfoaGmg== 2024-09-21T15:48:31.364 INFO:teuthology.orchestra.run.smithi100.stdout:[client.0] 2024-09-21T15:48:31.364 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQCw6u5m+/b4HhAA6h5dr+yG0FHHav/VaSt+1g== 2024-09-21T15:48:31.364 INFO:teuthology.orchestra.run.smithi100.stdout:[client.1] 2024-09-21T15:48:31.364 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQCw6u5mxxwiIhAAmjVhTEEI8QlwBwzhlu0TCg== 2024-09-21T15:48:31.365 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-21T15:48:31.390 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-21T15:48:31.456 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T15:48:31.458 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T15:48:31.509 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T15:48:31.511 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T15:48:31.560 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T15:48:31.563 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T15:48:31.612 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T15:48:31.614 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T15:48:31.664 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-21T15:48:31.666 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-21T15:48:31.716 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T15:48:31.718 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T15:48:31.768 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T15:48:31.770 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T15:48:31.820 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T15:48:31.822 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T15:48:31.872 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T15:48:31.874 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-21T15:48:31.924 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-21T15:48:31.926 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-21T15:48:31.979 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-21T15:48:31.981 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-21T15:48:32.034 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-21T15:48:32.034 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-21T15:48:32.058 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-21T15:48:32.152 DEBUG:teuthology.orchestra.run.smithi089:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-21T15:48:32.177 DEBUG:teuthology.orchestra.run.smithi089:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-21T15:48:32.241 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-21T15:48:32.337 DEBUG:teuthology.orchestra.run.smithi089:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-21T15:48:32.361 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-21T15:48:32.387 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-21T15:48:32.484 DEBUG:teuthology.orchestra.run.smithi100:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-21T15:48:32.509 DEBUG:teuthology.orchestra.run.smithi089:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-21T15:48:32.511 DEBUG:teuthology.orchestra.run.smithi100:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-21T15:48:32.564 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-21T15:48:32.564 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-21T15:48:32.564 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-21T15:48:32.566 INFO:tasks.ceph.mon.a:Started 2024-09-21T15:48:32.567 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-21T15:48:32.567 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-21T15:48:32.569 INFO:tasks.ceph.mon.c:Started 2024-09-21T15:48:32.569 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-21T15:48:32.569 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-21T15:48:32.606 INFO:tasks.ceph.mon.b:Started 2024-09-21T15:48:32.606 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-21T15:48:32.607 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-09-21T15:48:32.607 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-09-21T15:48:32.609 INFO:tasks.ceph.mgr.y:Started 2024-09-21T15:48:32.609 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-21T15:48:32.609 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-21T15:48:32.611 INFO:tasks.ceph.mgr.x:Started 2024-09-21T15:48:32.611 INFO:tasks.ceph:Setting crush tunables to default 2024-09-21T15:48:32.611 DEBUG:teuthology.orchestra.run.smithi089:> sudo ceph --cluster ceph osd crush tunables default 2024-09-21T15:48:37.984 INFO:teuthology.orchestra.run.smithi089.stderr:adjusted tunables profile to default 2024-09-21T15:48:37.996 INFO:tasks.ceph:check_enable_crimson: False 2024-09-21T15:48:37.997 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-21T15:48:37.997 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-21T15:48:37.997 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-21T15:48:38.007 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-09-21T15:48:38.005+0000 7f2e65a1b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-21T15:48:38.008 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-09-21T15:48:38.007+0000 7f2c3db73040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-21T15:48:38.021 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-21T15:48:38.022 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-21T15:48:38.049 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-09-21T15:48:38.048+0000 7f2e65a1b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-21T15:48:38.049 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-09-21T15:48:38.048+0000 7f2c3db73040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-21T15:48:38.073 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-21T15:48:38.073 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-21T15:48:38.138 DEBUG:teuthology.orchestra.run.smithi089:> set -ex 2024-09-21T15:48:38.138 DEBUG:teuthology.orchestra.run.smithi089:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-21T15:48:38.185 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-09-21T15:48:38.184+0000 7f2c3db73040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-21T15:48:38.198 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-09-21T15:48:38.197+0000 7f2e65a1b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-21T15:48:38.202 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-21T15:48:38.203 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-21T15:48:38.227 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-21T15:48:38.228 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-21T15:48:38.237 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-09-21T15:48:38.236+0000 7f2c3db73040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-21T15:48:38.248 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-09-21T15:48:38.247+0000 7f2e65a1b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-21T15:48:38.261 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-21T15:48:38.261 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-09-21T15:48:38.325 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-09-21T15:48:38.325 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-09-21T15:48:38.390 DEBUG:teuthology.orchestra.run.smithi100:> sudo ceph --cluster ceph osd new 7b3fabfa-3fe6-47c7-9d0c-a6036f738771 0 2024-09-21T15:48:38.400 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-09-21T15:48:38.399+0000 7f2c3db73040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-21T15:48:38.407 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-09-21T15:48:38.406+0000 7f2e65a1b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-21T15:48:38.515 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-09-21T15:48:38.514+0000 7f2e65a1b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-21T15:48:38.516 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-09-21T15:48:38.515+0000 7f2c3db73040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-21T15:48:38.699 INFO:teuthology.orchestra.run.smithi100.stderr:2024-09-21T15:48:38.698+0000 7f3c8e9e1640 0 --2- 172.21.15.100:0/225829855 >> v2:172.21.15.100:3300/0 conn(0x7f3c881419e0 0x7f3c88141e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-21T15:48:38.781 INFO:teuthology.orchestra.run.smithi100.stdout:0 2024-09-21T15:48:38.792 DEBUG:teuthology.orchestra.run.smithi100:> sudo ceph --cluster ceph osd new 767ad1bc-84ef-4102-83f5-aacda0917ddb 1 2024-09-21T15:48:38.832 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-09-21T15:48:38.831+0000 7f2e65a1b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-21T15:48:38.879 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-09-21T15:48:38.877+0000 7f2c3db73040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-21T15:48:38.886 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-09-21T15:48:38.885+0000 7f2e65a1b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-21T15:48:38.934 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-09-21T15:48:38.933+0000 7f2c3db73040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-21T15:48:38.973 INFO:teuthology.orchestra.run.smithi100.stdout:1 2024-09-21T15:48:38.984 DEBUG:teuthology.orchestra.run.smithi100:> sudo ceph --cluster ceph osd new 1dbd0d50-aae9-42a7-90a1-aa92697778ae 2 2024-09-21T15:48:39.035 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-09-21T15:48:39.033+0000 7f2e65a1b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-21T15:48:39.090 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-09-21T15:48:39.089+0000 7f2c3db73040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-21T15:48:39.165 INFO:teuthology.orchestra.run.smithi100.stdout:2 2024-09-21T15:48:39.176 DEBUG:teuthology.orchestra.run.smithi100:> sudo ceph --cluster ceph osd new f37cd295-2bac-4df4-a99b-abedc0978dce 3 2024-09-21T15:48:39.355 INFO:teuthology.orchestra.run.smithi100.stdout:3 2024-09-21T15:48:39.366 DEBUG:teuthology.orchestra.run.smithi100:> sudo ceph --cluster ceph osd new 2eeb351d-d51c-4fc3-884e-debecd995aea 4 2024-09-21T15:48:39.544 INFO:teuthology.orchestra.run.smithi100.stdout:4 2024-09-21T15:48:39.554 DEBUG:teuthology.orchestra.run.smithi100:> sudo ceph --cluster ceph osd new b5396267-6823-42f1-bd04-bd8f144fa76c 5 2024-09-21T15:48:39.735 INFO:teuthology.orchestra.run.smithi100.stdout:5 2024-09-21T15:48:39.749 DEBUG:teuthology.orchestra.run.smithi100:> sudo ceph --cluster ceph osd new 955b67e9-73bc-42c1-88e0-3356c105e1ea 6 2024-09-21T15:48:39.925 INFO:teuthology.orchestra.run.smithi100.stdout:6 2024-09-21T15:48:39.935 DEBUG:teuthology.orchestra.run.smithi100:> sudo ceph --cluster ceph osd new 414ea7f0-138d-404e-bce6-fe8dedcdc938 7 2024-09-21T15:48:39.968 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-09-21T15:48:39.967+0000 7f2e65a1b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-21T15:48:40.018 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-09-21T15:48:40.017+0000 7f2e65a1b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-21T15:48:40.089 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-09-21T15:48:40.087+0000 7f2c3db73040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-21T15:48:40.113 INFO:teuthology.orchestra.run.smithi100.stdout:7 2024-09-21T15:48:40.124 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-21T15:48:40.124 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-21T15:48:40.126 INFO:tasks.ceph.osd.0:Started 2024-09-21T15:48:40.126 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-21T15:48:40.126 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-21T15:48:40.128 INFO:tasks.ceph.osd.1:Started 2024-09-21T15:48:40.129 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-21T15:48:40.129 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-21T15:48:40.130 INFO:tasks.ceph.osd.2:Started 2024-09-21T15:48:40.130 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-21T15:48:40.131 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-21T15:48:40.133 INFO:tasks.ceph.osd.3:Started 2024-09-21T15:48:40.133 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-21T15:48:40.133 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-21T15:48:40.135 INFO:tasks.ceph.osd.4:Started 2024-09-21T15:48:40.135 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-21T15:48:40.135 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-21T15:48:40.137 INFO:tasks.ceph.osd.5:Started 2024-09-21T15:48:40.137 INFO:tasks.ceph.osd.6:Restarting daemon 2024-09-21T15:48:40.137 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-09-21T15:48:40.140 INFO:tasks.ceph.osd.6:Started 2024-09-21T15:48:40.141 INFO:tasks.ceph.osd.7:Restarting daemon 2024-09-21T15:48:40.141 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-09-21T15:48:40.143 INFO:tasks.ceph.osd.7:Started 2024-09-21T15:48:40.143 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T15:48:40.144 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-09-21T15:48:40.142+0000 7f2c3db73040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-21T15:48:40.351 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:48:40.351 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":10,"fsid":"4a9b64ea-3aaf-4a40-8ce8-02163a0f6e12","created":"2024-09-21T15:48:37.867501+0000","modified":"2024-09-21T15:48:40.108789+0000","last_up_change":"0.000000","last_in_change":"2024-09-21T15:48:40.108789+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7b3fabfa-3fe6-47c7-9d0c-a6036f738771","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"767ad1bc-84ef-4102-83f5-aacda0917ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1dbd0d50-aae9-42a7-90a1-aa92697778ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f37cd295-2bac-4df4-a99b-abedc0978dce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2eeb351d-d51c-4fc3-884e-debecd995aea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b5396267-6823-42f1-bd04-bd8f144fa76c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"955b67e9-73bc-42c1-88e0-3356c105e1ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"414ea7f0-138d-404e-bce6-fe8dedcdc938","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T15:48:40.362 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-21T15:48:40.362 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-21T15:48:40.430 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-09-21T15:48:40.429+0000 7f2e65a1b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-21T15:48:40.515 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-09-21T15:48:40.514+0000 7f2e65a1b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-21T15:48:40.531 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-09-21T15:48:40.530+0000 7f2c3db73040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-21T15:48:40.613 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-09-21T15:48:40.612+0000 7f2c3db73040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-21T15:48:40.619 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-09-21T15:48:40.618+0000 7f2e65a1b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-21T15:48:40.664 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-09-21T15:48:40.663+0000 7f2e65a1b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-21T15:48:40.717 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-09-21T15:48:40.716+0000 7f2c3db73040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-21T15:48:40.732 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:48:40.731+0000 7f225cc10640 -1 Falling back to public interface 2024-09-21T15:48:40.738 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:48:40.737+0000 7f3d5320d640 -1 Falling back to public interface 2024-09-21T15:48:40.739 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:48:40.737+0000 7fd8645bf640 -1 Falling back to public interface 2024-09-21T15:48:40.739 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:48:40.737+0000 7fd863568640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6801/0: (98) Address already in use 2024-09-21T15:48:40.741 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:48:40.740+0000 7f11b04da640 -1 Falling back to public interface 2024-09-21T15:48:40.747 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:48:40.746+0000 7ff0d5743640 -1 Falling back to public interface 2024-09-21T15:48:40.749 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:48:40.748+0000 7f6e5ab20640 -1 Falling back to public interface 2024-09-21T15:48:40.762 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:48:40.761+0000 7fd099864640 -1 Falling back to public interface 2024-09-21T15:48:40.763 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:48:40.761+0000 7f2f77819640 -1 Falling back to public interface 2024-09-21T15:48:40.763 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:48:40.762+0000 7fd09880d640 -1 PosixStack listen unable to listen on v2:0.0.0.0:6820/0: (98) Address already in use 2024-09-21T15:48:40.765 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-09-21T15:48:40.763+0000 7f2c3db73040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-21T15:48:40.777 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-09-21T15:48:40.775+0000 7f2e65a1b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-21T15:48:40.829 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-09-21T15:48:40.827+0000 7f2e65a1b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-21T15:48:40.873 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-09-21T15:48:40.872+0000 7f2e65a1b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-21T15:48:40.875 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-09-21T15:48:40.874+0000 7f2c3db73040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-21T15:48:40.929 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-09-21T15:48:40.927+0000 7f2c3db73040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-21T15:48:40.972 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-09-21T15:48:40.971+0000 7f2c3db73040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-21T15:48:40.982 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-09-21T15:48:40.981+0000 7f2e65a1b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-21T15:48:41.033 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-09-21T15:48:41.031+0000 7f2e65a1b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-21T15:48:41.083 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-09-21T15:48:41.082+0000 7f2e65a1b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-21T15:48:41.084 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-09-21T15:48:41.082+0000 7f2c3db73040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-21T15:48:41.132 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-09-21T15:48:41.131+0000 7f2c3db73040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-21T15:48:41.149 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-09-21T15:48:41.148+0000 7f2e65a1b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-21T15:48:41.168 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-21T15:48:41.183 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-09-21T15:48:41.182+0000 7f2c3db73040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-21T15:48:41.196 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-09-21T15:48:41.194+0000 7f2e65a1b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-21T15:48:41.241 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-09-21T15:48:41.240+0000 7f2e65a1b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-21T15:48:41.249 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-09-21T15:48:41.248+0000 7f2c3db73040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-21T15:48:41.294 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-09-21T15:48:41.293+0000 7f2c3db73040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-21T15:48:41.325 INFO:teuthology.misc.health.smithi089.stdout: 2024-09-21T15:48:41.325 INFO:teuthology.misc.health.smithi089.stdout:{"epoch":10,"fsid":"4a9b64ea-3aaf-4a40-8ce8-02163a0f6e12","created":"2024-09-21T15:48:37.867501+0000","modified":"2024-09-21T15:48:40.108789+0000","last_up_change":"0.000000","last_in_change":"2024-09-21T15:48:40.108789+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7b3fabfa-3fe6-47c7-9d0c-a6036f738771","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"767ad1bc-84ef-4102-83f5-aacda0917ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1dbd0d50-aae9-42a7-90a1-aa92697778ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f37cd295-2bac-4df4-a99b-abedc0978dce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2eeb351d-d51c-4fc3-884e-debecd995aea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b5396267-6823-42f1-bd04-bd8f144fa76c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"955b67e9-73bc-42c1-88e0-3356c105e1ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"414ea7f0-138d-404e-bce6-fe8dedcdc938","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T15:48:41.335 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-09-21T15:48:41.339 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-09-21T15:48:41.338+0000 7f2c3db73040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-21T15:48:41.878 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-09-21T15:48:41.876+0000 7f2e65a1b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-21T15:48:41.976 INFO:tasks.ceph.mgr.x.smithi100.stderr:2024-09-21T15:48:41.974+0000 7f2c3db73040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-21T15:48:42.895 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-09-21T15:48:42.894+0000 7f2e05f73640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-21T15:48:42.897 INFO:tasks.ceph.mgr.y.smithi089.stderr:2024-09-21T15:48:42.896+0000 7f2e05f73640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-21T15:48:44.206 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:48:44.204+0000 7f225cc10640 -1 osd.7 0 log_to_monitors true 2024-09-21T15:48:44.228 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:48:44.227+0000 7fd8645bf640 -1 osd.2 0 log_to_monitors true 2024-09-21T15:48:44.250 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:48:44.249+0000 7f3d5320d640 -1 osd.0 0 log_to_monitors true 2024-09-21T15:48:44.260 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:48:44.259+0000 7f11b04da640 -1 osd.6 0 log_to_monitors true 2024-09-21T15:48:44.260 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:48:44.259+0000 7f6e5ab20640 -1 osd.3 0 log_to_monitors true 2024-09-21T15:48:44.262 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:48:44.261+0000 7f2f77819640 -1 osd.4 0 log_to_monitors true 2024-09-21T15:48:44.268 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:48:44.267+0000 7ff0d5743640 -1 osd.1 0 log_to_monitors true 2024-09-21T15:48:44.274 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:48:44.273+0000 7fd099864640 -1 osd.5 0 log_to_monitors true 2024-09-21T15:48:48.141 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-21T15:48:48.144 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:48:48.143+0000 7f11abc44640 -1 osd.6 0 waiting for initial osdmap 2024-09-21T15:48:48.158 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:48:48.157+0000 7fd85fd29640 -1 osd.2 0 waiting for initial osdmap 2024-09-21T15:48:48.164 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:48:48.163+0000 7f2257c72640 -1 osd.7 0 waiting for initial osdmap 2024-09-21T15:48:48.165 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:48:48.164+0000 7f11a4277640 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T15:48:48.174 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:48:48.173+0000 7f3d4e426640 -1 osd.0 0 waiting for initial osdmap 2024-09-21T15:48:48.175 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:48:48.173+0000 7fd85835c640 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T15:48:48.178 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:48:48.176+0000 7f2f728e7640 -1 osd.4 0 waiting for initial osdmap 2024-09-21T15:48:48.180 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:48:48.179+0000 7f22502a5640 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T15:48:48.188 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:48:48.187+0000 7f6e5628a640 -1 osd.3 0 waiting for initial osdmap 2024-09-21T15:48:48.190 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:48:48.189+0000 7ff0d0ead640 -1 osd.1 0 waiting for initial osdmap 2024-09-21T15:48:48.192 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:48:48.191+0000 7f2f6af1a640 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T15:48:48.195 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:48:48.194+0000 7f3d46a59640 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T15:48:48.201 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:48:48.200+0000 7fd094fce640 -1 osd.5 0 waiting for initial osdmap 2024-09-21T15:48:48.206 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:48:48.205+0000 7f6e4e8bd640 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T15:48:48.209 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:48:48.208+0000 7ff0c94e0640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T15:48:48.221 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:48:48.220+0000 7fd08d601640 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T15:48:48.424 INFO:teuthology.misc.health.smithi089.stdout: 2024-09-21T15:48:48.424 INFO:teuthology.misc.health.smithi089.stdout:{"epoch":12,"fsid":"4a9b64ea-3aaf-4a40-8ce8-02163a0f6e12","created":"2024-09-21T15:48:37.867501+0000","modified":"2024-09-21T15:48:45.902822+0000","last_up_change":"0.000000","last_in_change":"2024-09-21T15:48:40.108789+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"7b3fabfa-3fe6-47c7-9d0c-a6036f738771","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"767ad1bc-84ef-4102-83f5-aacda0917ddb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1dbd0d50-aae9-42a7-90a1-aa92697778ae","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"f37cd295-2bac-4df4-a99b-abedc0978dce","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"2eeb351d-d51c-4fc3-884e-debecd995aea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"b5396267-6823-42f1-bd04-bd8f144fa76c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"955b67e9-73bc-42c1-88e0-3356c105e1ea","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"414ea7f0-138d-404e-bce6-fe8dedcdc938","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T15:48:48.435 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-09-21T15:48:55.241 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-21T15:48:55.517 INFO:teuthology.misc.health.smithi089.stdout: 2024-09-21T15:48:55.517 INFO:teuthology.misc.health.smithi089.stdout:{"epoch":16,"fsid":"4a9b64ea-3aaf-4a40-8ce8-02163a0f6e12","created":"2024-09-21T15:48:37.867501+0000","modified":"2024-09-21T15:48:52.184043+0000","last_up_change":"2024-09-21T15:48:49.160053+0000","last_in_change":"2024-09-21T15:48:40.108789+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T15:48:49.921079+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b3fabfa-3fe6-47c7-9d0c-a6036f738771","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":3360073633},{"type":"v1","addr":"172.21.15.89:6801","nonce":3360073633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6803","nonce":3360073633},{"type":"v1","addr":"172.21.15.89:6805","nonce":3360073633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6811","nonce":3360073633},{"type":"v1","addr":"172.21.15.89:6813","nonce":3360073633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6807","nonce":3360073633},{"type":"v1","addr":"172.21.15.89:6809","nonce":3360073633}]},"public_addr":"172.21.15.89:6801/3360073633","cluster_addr":"172.21.15.89:6805/3360073633","heartbeat_back_addr":"172.21.15.89:6813/3360073633","heartbeat_front_addr":"172.21.15.89:6809/3360073633","state":["exists","up"]},{"osd":1,"uuid":"767ad1bc-84ef-4102-83f5-aacda0917ddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3187095367},{"type":"v1","addr":"172.21.15.89:6817","nonce":3187095367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":3187095367},{"type":"v1","addr":"172.21.15.89:6819","nonce":3187095367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":3187095367},{"type":"v1","addr":"172.21.15.89:6824","nonce":3187095367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":3187095367},{"type":"v1","addr":"172.21.15.89:6821","nonce":3187095367}]},"public_addr":"172.21.15.89:6817/3187095367","cluster_addr":"172.21.15.89:6819/3187095367","heartbeat_back_addr":"172.21.15.89:6824/3187095367","heartbeat_front_addr":"172.21.15.89:6821/3187095367","state":["exists","up"]},{"osd":2,"uuid":"1dbd0d50-aae9-42a7-90a1-aa92697778ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":4098625737},{"type":"v1","addr":"172.21.15.89:6804","nonce":4098625737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":4098625737},{"type":"v1","addr":"172.21.15.89:6808","nonce":4098625737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4098625737},{"type":"v1","addr":"172.21.15.89:6815","nonce":4098625737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4098625737},{"type":"v1","addr":"172.21.15.89:6812","nonce":4098625737}]},"public_addr":"172.21.15.89:6804/4098625737","cluster_addr":"172.21.15.89:6808/4098625737","heartbeat_back_addr":"172.21.15.89:6815/4098625737","heartbeat_front_addr":"172.21.15.89:6812/4098625737","state":["exists","up"]},{"osd":3,"uuid":"f37cd295-2bac-4df4-a99b-abedc0978dce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":1886828410},{"type":"v1","addr":"172.21.15.89:6825","nonce":1886828410}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1886828410},{"type":"v1","addr":"172.21.15.89:6827","nonce":1886828410}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1886828410},{"type":"v1","addr":"172.21.15.89:6831","nonce":1886828410}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1886828410},{"type":"v1","addr":"172.21.15.89:6829","nonce":1886828410}]},"public_addr":"172.21.15.89:6825/1886828410","cluster_addr":"172.21.15.89:6827/1886828410","heartbeat_back_addr":"172.21.15.89:6831/1886828410","heartbeat_front_addr":"172.21.15.89:6829/1886828410","state":["exists","up"]},{"osd":4,"uuid":"2eeb351d-d51c-4fc3-884e-debecd995aea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6817","nonce":22722842},{"type":"v1","addr":"172.21.15.100:6819","nonce":22722842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":22722842},{"type":"v1","addr":"172.21.15.100:6822","nonce":22722842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":22722842},{"type":"v1","addr":"172.21.15.100:6830","nonce":22722842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":22722842},{"type":"v1","addr":"172.21.15.100:6826","nonce":22722842}]},"public_addr":"172.21.15.100:6819/22722842","cluster_addr":"172.21.15.100:6822/22722842","heartbeat_back_addr":"172.21.15.100:6830/22722842","heartbeat_front_addr":"172.21.15.100:6826/22722842","state":["exists","up"]},{"osd":5,"uuid":"b5396267-6823-42f1-bd04-bd8f144fa76c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":363733545},{"type":"v1","addr":"172.21.15.100:6818","nonce":363733545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":363733545},{"type":"v1","addr":"172.21.15.100:6823","nonce":363733545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6829","nonce":363733545},{"type":"v1","addr":"172.21.15.100:6831","nonce":363733545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":363733545},{"type":"v1","addr":"172.21.15.100:6827","nonce":363733545}]},"public_addr":"172.21.15.100:6818/363733545","cluster_addr":"172.21.15.100:6823/363733545","heartbeat_back_addr":"172.21.15.100:6831/363733545","heartbeat_front_addr":"172.21.15.100:6827/363733545","state":["exists","up"]},{"osd":6,"uuid":"955b67e9-73bc-42c1-88e0-3356c105e1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":205694879},{"type":"v1","addr":"172.21.15.100:6809","nonce":205694879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":205694879},{"type":"v1","addr":"172.21.15.100:6811","nonce":205694879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":205694879},{"type":"v1","addr":"172.21.15.100:6815","nonce":205694879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":205694879},{"type":"v1","addr":"172.21.15.100:6813","nonce":205694879}]},"public_addr":"172.21.15.100:6809/205694879","cluster_addr":"172.21.15.100:6811/205694879","heartbeat_back_addr":"172.21.15.100:6815/205694879","heartbeat_front_addr":"172.21.15.100:6813/205694879","state":["exists","up"]},{"osd":7,"uuid":"414ea7f0-138d-404e-bce6-fe8dedcdc938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":14,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":3120011840},{"type":"v1","addr":"172.21.15.100:6801","nonce":3120011840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":3120011840},{"type":"v1","addr":"172.21.15.100:6803","nonce":3120011840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":3120011840},{"type":"v1","addr":"172.21.15.100:6807","nonce":3120011840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":3120011840},{"type":"v1","addr":"172.21.15.100:6805","nonce":3120011840}]},"public_addr":"172.21.15.100:6801/3120011840","cluster_addr":"172.21.15.100:6803/3120011840","heartbeat_back_addr":"172.21.15.100:6807/3120011840","heartbeat_front_addr":"172.21.15.100:6805/3120011840","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:48:45.230746+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:48:45.303567+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:48:45.210941+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:48:45.218918+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:48:45.301781+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:48:45.248444+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:48:45.275465+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:48:45.166209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T15:48:55.528 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-09-21T15:48:55.528 INFO:tasks.ceph:Creating RBD pool 2024-09-21T15:48:55.528 DEBUG:teuthology.orchestra.run.smithi089:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-21T15:48:56.580 INFO:teuthology.orchestra.run.smithi089.stderr:pool 'rbd' created 2024-09-21T15:48:56.596 DEBUG:teuthology.orchestra.run.smithi089:> rbd --cluster ceph pool init rbd 2024-09-21T15:48:59.605 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-21T15:48:59.605 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-21T15:48:59.606 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-21T15:48:59.606 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-21T15:48:59.606 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-21T15:48:59.929 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:48:59.940 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":5,"flags":0,"active_gid":4100,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6832","nonce":2109618686},{"type":"v1","addr":"172.21.15.89:6833","nonce":2109618686}]},"active_addr":"172.21.15.89:6833/2109618686","active_change":"2024-09-21T15:48:41.879657+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[{"gid":4105,"name":"x","mgr_features":4540138322906710015,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.89:0","nonce":145310024}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.89:0","nonce":4045143050}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.89:0","nonce":4078458943}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.89:0","nonce":3040380786}]}]} 2024-09-21T15:48:59.943 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-21T15:48:59.944 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-21T15:48:59.944 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T15:49:00.218 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:49:00.218 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":20,"fsid":"4a9b64ea-3aaf-4a40-8ce8-02163a0f6e12","created":"2024-09-21T15:48:37.867501+0000","modified":"2024-09-21T15:48:59.585479+0000","last_up_change":"2024-09-21T15:48:49.160053+0000","last_in_change":"2024-09-21T15:48:40.108789+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T15:48:49.921079+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T15:48:55.812982+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":20,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b3fabfa-3fe6-47c7-9d0c-a6036f738771","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":3360073633},{"type":"v1","addr":"172.21.15.89:6801","nonce":3360073633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6803","nonce":3360073633},{"type":"v1","addr":"172.21.15.89:6805","nonce":3360073633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6811","nonce":3360073633},{"type":"v1","addr":"172.21.15.89:6813","nonce":3360073633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6807","nonce":3360073633},{"type":"v1","addr":"172.21.15.89:6809","nonce":3360073633}]},"public_addr":"172.21.15.89:6801/3360073633","cluster_addr":"172.21.15.89:6805/3360073633","heartbeat_back_addr":"172.21.15.89:6813/3360073633","heartbeat_front_addr":"172.21.15.89:6809/3360073633","state":["exists","up"]},{"osd":1,"uuid":"767ad1bc-84ef-4102-83f5-aacda0917ddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3187095367},{"type":"v1","addr":"172.21.15.89:6817","nonce":3187095367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":3187095367},{"type":"v1","addr":"172.21.15.89:6819","nonce":3187095367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":3187095367},{"type":"v1","addr":"172.21.15.89:6824","nonce":3187095367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":3187095367},{"type":"v1","addr":"172.21.15.89:6821","nonce":3187095367}]},"public_addr":"172.21.15.89:6817/3187095367","cluster_addr":"172.21.15.89:6819/3187095367","heartbeat_back_addr":"172.21.15.89:6824/3187095367","heartbeat_front_addr":"172.21.15.89:6821/3187095367","state":["exists","up"]},{"osd":2,"uuid":"1dbd0d50-aae9-42a7-90a1-aa92697778ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":4098625737},{"type":"v1","addr":"172.21.15.89:6804","nonce":4098625737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":4098625737},{"type":"v1","addr":"172.21.15.89:6808","nonce":4098625737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4098625737},{"type":"v1","addr":"172.21.15.89:6815","nonce":4098625737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4098625737},{"type":"v1","addr":"172.21.15.89:6812","nonce":4098625737}]},"public_addr":"172.21.15.89:6804/4098625737","cluster_addr":"172.21.15.89:6808/4098625737","heartbeat_back_addr":"172.21.15.89:6815/4098625737","heartbeat_front_addr":"172.21.15.89:6812/4098625737","state":["exists","up"]},{"osd":3,"uuid":"f37cd295-2bac-4df4-a99b-abedc0978dce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":1886828410},{"type":"v1","addr":"172.21.15.89:6825","nonce":1886828410}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1886828410},{"type":"v1","addr":"172.21.15.89:6827","nonce":1886828410}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1886828410},{"type":"v1","addr":"172.21.15.89:6831","nonce":1886828410}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1886828410},{"type":"v1","addr":"172.21.15.89:6829","nonce":1886828410}]},"public_addr":"172.21.15.89:6825/1886828410","cluster_addr":"172.21.15.89:6827/1886828410","heartbeat_back_addr":"172.21.15.89:6831/1886828410","heartbeat_front_addr":"172.21.15.89:6829/1886828410","state":["exists","up"]},{"osd":4,"uuid":"2eeb351d-d51c-4fc3-884e-debecd995aea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6817","nonce":22722842},{"type":"v1","addr":"172.21.15.100:6819","nonce":22722842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":22722842},{"type":"v1","addr":"172.21.15.100:6822","nonce":22722842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":22722842},{"type":"v1","addr":"172.21.15.100:6830","nonce":22722842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":22722842},{"type":"v1","addr":"172.21.15.100:6826","nonce":22722842}]},"public_addr":"172.21.15.100:6819/22722842","cluster_addr":"172.21.15.100:6822/22722842","heartbeat_back_addr":"172.21.15.100:6830/22722842","heartbeat_front_addr":"172.21.15.100:6826/22722842","state":["exists","up"]},{"osd":5,"uuid":"b5396267-6823-42f1-bd04-bd8f144fa76c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":363733545},{"type":"v1","addr":"172.21.15.100:6818","nonce":363733545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":363733545},{"type":"v1","addr":"172.21.15.100:6823","nonce":363733545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6829","nonce":363733545},{"type":"v1","addr":"172.21.15.100:6831","nonce":363733545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":363733545},{"type":"v1","addr":"172.21.15.100:6827","nonce":363733545}]},"public_addr":"172.21.15.100:6818/363733545","cluster_addr":"172.21.15.100:6823/363733545","heartbeat_back_addr":"172.21.15.100:6831/363733545","heartbeat_front_addr":"172.21.15.100:6827/363733545","state":["exists","up"]},{"osd":6,"uuid":"955b67e9-73bc-42c1-88e0-3356c105e1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":205694879},{"type":"v1","addr":"172.21.15.100:6809","nonce":205694879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":205694879},{"type":"v1","addr":"172.21.15.100:6811","nonce":205694879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":205694879},{"type":"v1","addr":"172.21.15.100:6815","nonce":205694879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":205694879},{"type":"v1","addr":"172.21.15.100:6813","nonce":205694879}]},"public_addr":"172.21.15.100:6809/205694879","cluster_addr":"172.21.15.100:6811/205694879","heartbeat_back_addr":"172.21.15.100:6815/205694879","heartbeat_front_addr":"172.21.15.100:6813/205694879","state":["exists","up"]},{"osd":7,"uuid":"414ea7f0-138d-404e-bce6-fe8dedcdc938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":3120011840},{"type":"v1","addr":"172.21.15.100:6801","nonce":3120011840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":3120011840},{"type":"v1","addr":"172.21.15.100:6803","nonce":3120011840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":3120011840},{"type":"v1","addr":"172.21.15.100:6807","nonce":3120011840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":3120011840},{"type":"v1","addr":"172.21.15.100:6805","nonce":3120011840}]},"public_addr":"172.21.15.100:6801/3120011840","cluster_addr":"172.21.15.100:6803/3120011840","heartbeat_back_addr":"172.21.15.100:6807/3120011840","heartbeat_front_addr":"172.21.15.100:6805/3120011840","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:48:45.230746+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:48:45.303567+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:48:45.210941+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:48:45.218918+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:48:45.301781+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:48:45.248444+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:48:45.275465+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:48:45.166209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T15:49:00.231 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-21T15:49:00.231 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T15:49:00.502 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:49:00.503 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":20,"fsid":"4a9b64ea-3aaf-4a40-8ce8-02163a0f6e12","created":"2024-09-21T15:48:37.867501+0000","modified":"2024-09-21T15:48:59.585479+0000","last_up_change":"2024-09-21T15:48:49.160053+0000","last_in_change":"2024-09-21T15:48:40.108789+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T15:48:49.921079+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T15:48:55.812982+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":20,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b3fabfa-3fe6-47c7-9d0c-a6036f738771","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":3360073633},{"type":"v1","addr":"172.21.15.89:6801","nonce":3360073633}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6803","nonce":3360073633},{"type":"v1","addr":"172.21.15.89:6805","nonce":3360073633}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6811","nonce":3360073633},{"type":"v1","addr":"172.21.15.89:6813","nonce":3360073633}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6807","nonce":3360073633},{"type":"v1","addr":"172.21.15.89:6809","nonce":3360073633}]},"public_addr":"172.21.15.89:6801/3360073633","cluster_addr":"172.21.15.89:6805/3360073633","heartbeat_back_addr":"172.21.15.89:6813/3360073633","heartbeat_front_addr":"172.21.15.89:6809/3360073633","state":["exists","up"]},{"osd":1,"uuid":"767ad1bc-84ef-4102-83f5-aacda0917ddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3187095367},{"type":"v1","addr":"172.21.15.89:6817","nonce":3187095367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":3187095367},{"type":"v1","addr":"172.21.15.89:6819","nonce":3187095367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":3187095367},{"type":"v1","addr":"172.21.15.89:6824","nonce":3187095367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":3187095367},{"type":"v1","addr":"172.21.15.89:6821","nonce":3187095367}]},"public_addr":"172.21.15.89:6817/3187095367","cluster_addr":"172.21.15.89:6819/3187095367","heartbeat_back_addr":"172.21.15.89:6824/3187095367","heartbeat_front_addr":"172.21.15.89:6821/3187095367","state":["exists","up"]},{"osd":2,"uuid":"1dbd0d50-aae9-42a7-90a1-aa92697778ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":4098625737},{"type":"v1","addr":"172.21.15.89:6804","nonce":4098625737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":4098625737},{"type":"v1","addr":"172.21.15.89:6808","nonce":4098625737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4098625737},{"type":"v1","addr":"172.21.15.89:6815","nonce":4098625737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4098625737},{"type":"v1","addr":"172.21.15.89:6812","nonce":4098625737}]},"public_addr":"172.21.15.89:6804/4098625737","cluster_addr":"172.21.15.89:6808/4098625737","heartbeat_back_addr":"172.21.15.89:6815/4098625737","heartbeat_front_addr":"172.21.15.89:6812/4098625737","state":["exists","up"]},{"osd":3,"uuid":"f37cd295-2bac-4df4-a99b-abedc0978dce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":1886828410},{"type":"v1","addr":"172.21.15.89:6825","nonce":1886828410}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1886828410},{"type":"v1","addr":"172.21.15.89:6827","nonce":1886828410}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1886828410},{"type":"v1","addr":"172.21.15.89:6831","nonce":1886828410}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1886828410},{"type":"v1","addr":"172.21.15.89:6829","nonce":1886828410}]},"public_addr":"172.21.15.89:6825/1886828410","cluster_addr":"172.21.15.89:6827/1886828410","heartbeat_back_addr":"172.21.15.89:6831/1886828410","heartbeat_front_addr":"172.21.15.89:6829/1886828410","state":["exists","up"]},{"osd":4,"uuid":"2eeb351d-d51c-4fc3-884e-debecd995aea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6817","nonce":22722842},{"type":"v1","addr":"172.21.15.100:6819","nonce":22722842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":22722842},{"type":"v1","addr":"172.21.15.100:6822","nonce":22722842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":22722842},{"type":"v1","addr":"172.21.15.100:6830","nonce":22722842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":22722842},{"type":"v1","addr":"172.21.15.100:6826","nonce":22722842}]},"public_addr":"172.21.15.100:6819/22722842","cluster_addr":"172.21.15.100:6822/22722842","heartbeat_back_addr":"172.21.15.100:6830/22722842","heartbeat_front_addr":"172.21.15.100:6826/22722842","state":["exists","up"]},{"osd":5,"uuid":"b5396267-6823-42f1-bd04-bd8f144fa76c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":363733545},{"type":"v1","addr":"172.21.15.100:6818","nonce":363733545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":363733545},{"type":"v1","addr":"172.21.15.100:6823","nonce":363733545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6829","nonce":363733545},{"type":"v1","addr":"172.21.15.100:6831","nonce":363733545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":363733545},{"type":"v1","addr":"172.21.15.100:6827","nonce":363733545}]},"public_addr":"172.21.15.100:6818/363733545","cluster_addr":"172.21.15.100:6823/363733545","heartbeat_back_addr":"172.21.15.100:6831/363733545","heartbeat_front_addr":"172.21.15.100:6827/363733545","state":["exists","up"]},{"osd":6,"uuid":"955b67e9-73bc-42c1-88e0-3356c105e1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":205694879},{"type":"v1","addr":"172.21.15.100:6809","nonce":205694879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":205694879},{"type":"v1","addr":"172.21.15.100:6811","nonce":205694879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":205694879},{"type":"v1","addr":"172.21.15.100:6815","nonce":205694879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":205694879},{"type":"v1","addr":"172.21.15.100:6813","nonce":205694879}]},"public_addr":"172.21.15.100:6809/205694879","cluster_addr":"172.21.15.100:6811/205694879","heartbeat_back_addr":"172.21.15.100:6815/205694879","heartbeat_front_addr":"172.21.15.100:6813/205694879","state":["exists","up"]},{"osd":7,"uuid":"414ea7f0-138d-404e-bce6-fe8dedcdc938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":17,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":3120011840},{"type":"v1","addr":"172.21.15.100:6801","nonce":3120011840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":3120011840},{"type":"v1","addr":"172.21.15.100:6803","nonce":3120011840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":3120011840},{"type":"v1","addr":"172.21.15.100:6807","nonce":3120011840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":3120011840},{"type":"v1","addr":"172.21.15.100:6805","nonce":3120011840}]},"public_addr":"172.21.15.100:6801/3120011840","cluster_addr":"172.21.15.100:6803/3120011840","heartbeat_back_addr":"172.21.15.100:6807/3120011840","heartbeat_front_addr":"172.21.15.100:6805/3120011840","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:48:45.230746+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:48:45.303567+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:48:45.210941+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:48:45.218918+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:48:45.301781+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:48:45.248444+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:48:45.275465+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:48:45.166209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T15:49:00.516 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-21T15:49:00.516 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-21T15:49:00.516 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-21T15:49:00.517 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-21T15:49:00.517 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-21T15:49:00.518 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-21T15:49:00.518 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-21T15:49:00.518 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-21T15:49:00.735 INFO:teuthology.orchestra.run.smithi089.stdout:55834574853 2024-09-21T15:49:00.735 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-21T15:49:00.746 INFO:teuthology.orchestra.run.smithi089.stdout:55834574855 2024-09-21T15:49:00.746 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-21T15:49:00.751 INFO:teuthology.orchestra.run.smithi089.stdout:55834574855 2024-09-21T15:49:00.752 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-21T15:49:00.753 INFO:teuthology.orchestra.run.smithi089.stdout:55834574855 2024-09-21T15:49:00.753 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-21T15:49:00.753 INFO:teuthology.orchestra.run.smithi089.stdout:55834574854 2024-09-21T15:49:00.754 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-21T15:49:00.755 INFO:teuthology.orchestra.run.smithi089.stdout:55834574854 2024-09-21T15:49:00.755 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-21T15:49:00.953 INFO:teuthology.orchestra.run.smithi089.stdout:55834574855 2024-09-21T15:49:00.953 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-21T15:49:00.960 INFO:teuthology.orchestra.run.smithi089.stdout:55834574856 2024-09-21T15:49:00.960 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-21T15:49:01.186 INFO:teuthology.orchestra.run.smithi089.stdout:55834574851 2024-09-21T15:49:01.209 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574851 for osd.7 2024-09-21T15:49:01.269 INFO:teuthology.orchestra.run.smithi089.stdout:55834574853 2024-09-21T15:49:01.270 INFO:teuthology.orchestra.run.smithi089.stdout:55834574852 2024-09-21T15:49:01.285 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574852 for osd.4 2024-09-21T15:49:01.289 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574853 for osd.0 2024-09-21T15:49:01.303 INFO:teuthology.orchestra.run.smithi089.stdout:55834574853 2024-09-21T15:49:01.311 INFO:teuthology.orchestra.run.smithi089.stdout:55834574852 2024-09-21T15:49:01.321 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574853 for osd.5 2024-09-21T15:49:01.324 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574852 for osd.1 2024-09-21T15:49:01.325 INFO:teuthology.orchestra.run.smithi089.stdout:55834574853 2024-09-21T15:49:01.339 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574853 for osd.3 2024-09-21T15:49:01.412 INFO:teuthology.orchestra.run.smithi089.stdout:55834574853 2024-09-21T15:49:01.417 INFO:teuthology.orchestra.run.smithi089.stdout:55834574854 2024-09-21T15:49:01.426 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574853 for osd.2 2024-09-21T15:49:01.429 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574854 for osd.6 2024-09-21T15:49:02.210 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-21T15:49:02.286 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-21T15:49:02.289 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-21T15:49:02.321 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-21T15:49:02.325 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-21T15:49:02.340 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-21T15:49:02.427 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-21T15:49:02.430 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-21T15:49:02.667 INFO:teuthology.orchestra.run.smithi089.stdout:55834574853 2024-09-21T15:49:02.685 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.7 2024-09-21T15:49:02.686 DEBUG:teuthology.parallel:result is None 2024-09-21T15:49:02.773 INFO:teuthology.orchestra.run.smithi089.stdout:55834574855 2024-09-21T15:49:02.795 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.5 2024-09-21T15:49:02.795 DEBUG:teuthology.parallel:result is None 2024-09-21T15:49:02.798 INFO:teuthology.orchestra.run.smithi089.stdout:55834574854 2024-09-21T15:49:02.811 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.4 2024-09-21T15:49:02.811 DEBUG:teuthology.parallel:result is None 2024-09-21T15:49:02.848 INFO:teuthology.orchestra.run.smithi089.stdout:55834574855 2024-09-21T15:49:02.858 INFO:teuthology.orchestra.run.smithi089.stdout:55834574854 2024-09-21T15:49:02.864 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.0 2024-09-21T15:49:02.865 DEBUG:teuthology.parallel:result is None 2024-09-21T15:49:02.874 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.1 2024-09-21T15:49:02.874 DEBUG:teuthology.parallel:result is None 2024-09-21T15:49:02.893 INFO:teuthology.orchestra.run.smithi089.stdout:55834574855 2024-09-21T15:49:02.905 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.3 2024-09-21T15:49:02.905 DEBUG:teuthology.parallel:result is None 2024-09-21T15:49:02.907 INFO:teuthology.orchestra.run.smithi089.stdout:55834574856 2024-09-21T15:49:02.920 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574856 got 55834574856 for osd.6 2024-09-21T15:49:02.920 DEBUG:teuthology.parallel:result is None 2024-09-21T15:49:02.922 INFO:teuthology.orchestra.run.smithi089.stdout:55834574855 2024-09-21T15:49:02.935 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574855 got 55834574855 for osd.2 2024-09-21T15:49:02.935 DEBUG:teuthology.parallel:result is None 2024-09-21T15:49:02.935 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-21T15:49:02.935 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T15:49:03.246 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:49:03.246 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-09-21T15:49:03.258 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-09-21T15:49:02.656770+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":218912,"kb_used_data":2744,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755808,"statfs":{"total":773094113280,"available":772869947392,"internally_reserved":0,"allocated":2809856,"data_stored":1719990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.079507"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:48:59.666431+0000","last_change":"2024-09-21T15:48:59.666431+0000","last_active":"2024-09-21T15:48:59.666431+0000","last_peered":"2024-09-21T15:48:59.666431+0000","last_clean":"2024-09-21T15:48:59.666431+0000","last_became_active":"2024-09-21T15:48:57.583665+0000","last_became_peered":"2024-09-21T15:48:57.583665+0000","last_unstale":"2024-09-21T15:48:59.666431+0000","last_undegraded":"2024-09-21T15:48:59.666431+0000","last_fullsized":"2024-09-21T15:48:59.666431+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:52:09.228964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031098399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:48:59.589790+0000","last_change":"2024-09-21T15:48:59.589790+0000","last_active":"2024-09-21T15:48:59.589790+0000","last_peered":"2024-09-21T15:48:59.589790+0000","last_clean":"2024-09-21T15:48:59.589790+0000","last_became_active":"2024-09-21T15:48:57.582121+0000","last_became_peered":"2024-09-21T15:48:57.582121+0000","last_unstale":"2024-09-21T15:48:59.589790+0000","last_undegraded":"2024-09-21T15:48:59.589790+0000","last_fullsized":"2024-09-21T15:48:59.589790+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:33:47.592624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039654700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:49:00.114513+0000","last_change":"2024-09-21T15:49:00.114779+0000","last_active":"2024-09-21T15:49:00.114513+0000","last_peered":"2024-09-21T15:49:00.114513+0000","last_clean":"2024-09-21T15:49:00.114513+0000","last_became_active":"2024-09-21T15:48:57.937107+0000","last_became_peered":"2024-09-21T15:48:57.937107+0000","last_unstale":"2024-09-21T15:49:00.114513+0000","last_undegraded":"2024-09-21T15:49:00.114513+0000","last_fullsized":"2024-09-21T15:49:00.114513+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:38:43.937586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036074600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:48:59.589789+0000","last_change":"2024-09-21T15:48:59.589789+0000","last_active":"2024-09-21T15:48:59.589789+0000","last_peered":"2024-09-21T15:48:59.589789+0000","last_clean":"2024-09-21T15:48:59.589789+0000","last_became_active":"2024-09-21T15:48:57.936973+0000","last_became_peered":"2024-09-21T15:48:57.936973+0000","last_unstale":"2024-09-21T15:48:59.589789+0000","last_undegraded":"2024-09-21T15:48:59.589789+0000","last_fullsized":"2024-09-21T15:48:59.589789+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:41:58.458308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040525700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"20'2","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:48:59.600124+0000","last_change":"2024-09-21T15:48:59.595513+0000","last_active":"2024-09-21T15:48:59.600124+0000","last_peered":"2024-09-21T15:48:59.600124+0000","last_clean":"2024-09-21T15:48:59.600124+0000","last_became_active":"2024-09-21T15:48:57.585496+0000","last_became_peered":"2024-09-21T15:48:57.585496+0000","last_unstale":"2024-09-21T15:48:59.600124+0000","last_undegraded":"2024-09-21T15:48:59.600124+0000","last_fullsized":"2024-09-21T15:48:59.600124+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:01:35.910466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000859595,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:48:59.590261+0000","last_change":"2024-09-21T15:48:59.590261+0000","last_active":"2024-09-21T15:48:59.590261+0000","last_peered":"2024-09-21T15:48:59.590261+0000","last_clean":"2024-09-21T15:48:59.590261+0000","last_became_active":"2024-09-21T15:48:57.936108+0000","last_became_peered":"2024-09-21T15:48:57.936108+0000","last_unstale":"2024-09-21T15:48:59.590261+0000","last_undegraded":"2024-09-21T15:48:59.590261+0000","last_fullsized":"2024-09-21T15:48:59.590261+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:06:25.149550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043235099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:49:00.114548+0000","last_change":"2024-09-21T15:49:00.114730+0000","last_active":"2024-09-21T15:49:00.114548+0000","last_peered":"2024-09-21T15:49:00.114548+0000","last_clean":"2024-09-21T15:49:00.114548+0000","last_became_active":"2024-09-21T15:48:57.585525+0000","last_became_peered":"2024-09-21T15:48:57.585525+0000","last_unstale":"2024-09-21T15:49:00.114548+0000","last_undegraded":"2024-09-21T15:49:00.114548+0000","last_fullsized":"2024-09-21T15:49:00.114548+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:35:35.786526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031691100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"18'1","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:48:59.595056+0000","last_change":"2024-09-21T15:48:59.595056+0000","last_active":"2024-09-21T15:48:59.595056+0000","last_peered":"2024-09-21T15:48:59.595056+0000","last_clean":"2024-09-21T15:48:59.595056+0000","last_became_active":"2024-09-21T15:48:57.585384+0000","last_became_peered":"2024-09-21T15:48:57.585384+0000","last_unstale":"2024-09-21T15:48:59.595056+0000","last_undegraded":"2024-09-21T15:48:59.595056+0000","last_fullsized":"2024-09-21T15:48:59.595056+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:17:00.878205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039559000000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"16'117","reported_seq":164,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T15:48:58.780157+0000","last_change":"2024-09-21T15:48:51.381021+0000","last_active":"2024-09-21T15:48:58.780157+0000","last_peered":"2024-09-21T15:48:58.780157+0000","last_clean":"2024-09-21T15:48:58.780157+0000","last_became_active":"2024-09-21T15:48:51.380092+0000","last_became_peered":"2024-09-21T15:48:51.380092+0000","last_unstale":"2024-09-21T15:48:58.780157+0000","last_undegraded":"2024-09-21T15:48:58.780157+0000","last_fullsized":"2024-09-21T15:48:58.780157+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:50.161884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:50.161884+0000","last_clean_scrub_stamp":"2024-09-21T15:48:50.161884+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:32:51.001923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27788,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344052,"statfs":{"total":96636764160,"available":96608309248,"internally_reserved":0,"allocated":786432,"data_stored":653396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39000000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]}]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":68860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":68879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73399999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":68860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40300000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":68860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59999999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]}]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":68860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76500000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":68879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71599999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27796,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344044,"statfs":{"total":96636764160,"available":96608301056,"internally_reserved":0,"allocated":786432,"data_stored":653396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28199999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29599999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T15:49:03.259 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T15:49:03.527 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:49:03.527 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-09-21T15:49:03.539 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":20,"stamp":"2024-09-21T15:49:02.656770+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":218912,"kb_used_data":2744,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755808,"statfs":{"total":773094113280,"available":772869947392,"internally_reserved":0,"allocated":2809856,"data_stored":1719990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.079507"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:48:59.666431+0000","last_change":"2024-09-21T15:48:59.666431+0000","last_active":"2024-09-21T15:48:59.666431+0000","last_peered":"2024-09-21T15:48:59.666431+0000","last_clean":"2024-09-21T15:48:59.666431+0000","last_became_active":"2024-09-21T15:48:57.583665+0000","last_became_peered":"2024-09-21T15:48:57.583665+0000","last_unstale":"2024-09-21T15:48:59.666431+0000","last_undegraded":"2024-09-21T15:48:59.666431+0000","last_fullsized":"2024-09-21T15:48:59.666431+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:52:09.228964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031098399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:48:59.589790+0000","last_change":"2024-09-21T15:48:59.589790+0000","last_active":"2024-09-21T15:48:59.589790+0000","last_peered":"2024-09-21T15:48:59.589790+0000","last_clean":"2024-09-21T15:48:59.589790+0000","last_became_active":"2024-09-21T15:48:57.582121+0000","last_became_peered":"2024-09-21T15:48:57.582121+0000","last_unstale":"2024-09-21T15:48:59.589790+0000","last_undegraded":"2024-09-21T15:48:59.589790+0000","last_fullsized":"2024-09-21T15:48:59.589790+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:33:47.592624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039654700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:49:00.114513+0000","last_change":"2024-09-21T15:49:00.114779+0000","last_active":"2024-09-21T15:49:00.114513+0000","last_peered":"2024-09-21T15:49:00.114513+0000","last_clean":"2024-09-21T15:49:00.114513+0000","last_became_active":"2024-09-21T15:48:57.937107+0000","last_became_peered":"2024-09-21T15:48:57.937107+0000","last_unstale":"2024-09-21T15:49:00.114513+0000","last_undegraded":"2024-09-21T15:49:00.114513+0000","last_fullsized":"2024-09-21T15:49:00.114513+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:38:43.937586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036074600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:48:59.589789+0000","last_change":"2024-09-21T15:48:59.589789+0000","last_active":"2024-09-21T15:48:59.589789+0000","last_peered":"2024-09-21T15:48:59.589789+0000","last_clean":"2024-09-21T15:48:59.589789+0000","last_became_active":"2024-09-21T15:48:57.936973+0000","last_became_peered":"2024-09-21T15:48:57.936973+0000","last_unstale":"2024-09-21T15:48:59.589789+0000","last_undegraded":"2024-09-21T15:48:59.589789+0000","last_fullsized":"2024-09-21T15:48:59.589789+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:41:58.458308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040525700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"20'2","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:48:59.600124+0000","last_change":"2024-09-21T15:48:59.595513+0000","last_active":"2024-09-21T15:48:59.600124+0000","last_peered":"2024-09-21T15:48:59.600124+0000","last_clean":"2024-09-21T15:48:59.600124+0000","last_became_active":"2024-09-21T15:48:57.585496+0000","last_became_peered":"2024-09-21T15:48:57.585496+0000","last_unstale":"2024-09-21T15:48:59.600124+0000","last_undegraded":"2024-09-21T15:48:59.600124+0000","last_fullsized":"2024-09-21T15:48:59.600124+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:01:35.910466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000859595,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:48:59.590261+0000","last_change":"2024-09-21T15:48:59.590261+0000","last_active":"2024-09-21T15:48:59.590261+0000","last_peered":"2024-09-21T15:48:59.590261+0000","last_clean":"2024-09-21T15:48:59.590261+0000","last_became_active":"2024-09-21T15:48:57.936108+0000","last_became_peered":"2024-09-21T15:48:57.936108+0000","last_unstale":"2024-09-21T15:48:59.590261+0000","last_undegraded":"2024-09-21T15:48:59.590261+0000","last_fullsized":"2024-09-21T15:48:59.590261+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:06:25.149550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043235099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:49:00.114548+0000","last_change":"2024-09-21T15:49:00.114730+0000","last_active":"2024-09-21T15:49:00.114548+0000","last_peered":"2024-09-21T15:49:00.114548+0000","last_clean":"2024-09-21T15:49:00.114548+0000","last_became_active":"2024-09-21T15:48:57.585525+0000","last_became_peered":"2024-09-21T15:48:57.585525+0000","last_unstale":"2024-09-21T15:49:00.114548+0000","last_undegraded":"2024-09-21T15:49:00.114548+0000","last_fullsized":"2024-09-21T15:49:00.114548+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:35:35.786526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031691100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"18'1","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:48:59.595056+0000","last_change":"2024-09-21T15:48:59.595056+0000","last_active":"2024-09-21T15:48:59.595056+0000","last_peered":"2024-09-21T15:48:59.595056+0000","last_clean":"2024-09-21T15:48:59.595056+0000","last_became_active":"2024-09-21T15:48:57.585384+0000","last_became_peered":"2024-09-21T15:48:57.585384+0000","last_unstale":"2024-09-21T15:48:59.595056+0000","last_undegraded":"2024-09-21T15:48:59.595056+0000","last_fullsized":"2024-09-21T15:48:59.595056+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:17:00.878205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039559000000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"16'117","reported_seq":164,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T15:48:58.780157+0000","last_change":"2024-09-21T15:48:51.381021+0000","last_active":"2024-09-21T15:48:58.780157+0000","last_peered":"2024-09-21T15:48:58.780157+0000","last_clean":"2024-09-21T15:48:58.780157+0000","last_became_active":"2024-09-21T15:48:51.380092+0000","last_became_peered":"2024-09-21T15:48:51.380092+0000","last_unstale":"2024-09-21T15:48:58.780157+0000","last_undegraded":"2024-09-21T15:48:58.780157+0000","last_fullsized":"2024-09-21T15:48:58.780157+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:50.161884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:50.161884+0000","last_clean_scrub_stamp":"2024-09-21T15:48:50.161884+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:32:51.001923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27788,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344052,"statfs":{"total":96636764160,"available":96608309248,"internally_reserved":0,"allocated":786432,"data_stored":653396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39000000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]}]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":68860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":68879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73399999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":68860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40300000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":68860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59999999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]}]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":68860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76500000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":68879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71599999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27796,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344044,"statfs":{"total":96636764160,"available":96608301056,"internally_reserved":0,"allocated":786432,"data_stored":653396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28199999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29599999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T15:49:03.540 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-21T15:49:03.540 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-21T15:49:03.540 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-21T15:49:03.540 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-21T15:49:03.842 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:49:03.842 INFO:teuthology.orchestra.run.smithi089.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-21T15:49:03.855 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-21T15:49:03.855 INFO:teuthology.run_tasks:Running task thrashosds... 2024-09-21T15:49:03.865 INFO:tasks.thrashosds:config is {'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-21T15:49:03.866 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'reef'}, 'ceph': {'conf': {'client': {'rbd qos iops burst': 100, 'rbd qos iops limit': 50, 'rbd qos schedule tick min': 100}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from'], 'sha1': '19e2a2beb6f41a3ab62650e38614eb29b59dcbb9'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '19e2a2beb6f41a3ab62650e38614eb29b59dcbb9'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'reef', 'sha1': 'a477b1894d5c5a613f23260ca0e6b96d79652ba7'}} 2024-09-21T15:49:03.866 INFO:tasks.thrashosds:config is {'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0, 'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5} 2024-09-21T15:49:03.866 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-09-21T15:49:04.157 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-21T15:49:04.157 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-21T15:49:04.157 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T15:49:04.426 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:49:04.426 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-09-21T15:49:04.438 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-21T15:49:03.891922+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":218912,"kb_used_data":2744,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755808,"statfs":{"total":773094113280,"available":772869947392,"internally_reserved":0,"allocated":2809856,"data_stored":1719990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000971"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:48:59.666431+0000","last_change":"2024-09-21T15:48:59.666431+0000","last_active":"2024-09-21T15:48:59.666431+0000","last_peered":"2024-09-21T15:48:59.666431+0000","last_clean":"2024-09-21T15:48:59.666431+0000","last_became_active":"2024-09-21T15:48:57.583665+0000","last_became_peered":"2024-09-21T15:48:57.583665+0000","last_unstale":"2024-09-21T15:48:59.666431+0000","last_undegraded":"2024-09-21T15:48:59.666431+0000","last_fullsized":"2024-09-21T15:48:59.666431+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:52:09.228964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031098399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:48:59.589790+0000","last_change":"2024-09-21T15:48:59.589790+0000","last_active":"2024-09-21T15:48:59.589790+0000","last_peered":"2024-09-21T15:48:59.589790+0000","last_clean":"2024-09-21T15:48:59.589790+0000","last_became_active":"2024-09-21T15:48:57.582121+0000","last_became_peered":"2024-09-21T15:48:57.582121+0000","last_unstale":"2024-09-21T15:48:59.589790+0000","last_undegraded":"2024-09-21T15:48:59.589790+0000","last_fullsized":"2024-09-21T15:48:59.589790+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:33:47.592624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039654700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:49:00.114513+0000","last_change":"2024-09-21T15:49:00.114779+0000","last_active":"2024-09-21T15:49:00.114513+0000","last_peered":"2024-09-21T15:49:00.114513+0000","last_clean":"2024-09-21T15:49:00.114513+0000","last_became_active":"2024-09-21T15:48:57.937107+0000","last_became_peered":"2024-09-21T15:48:57.937107+0000","last_unstale":"2024-09-21T15:49:00.114513+0000","last_undegraded":"2024-09-21T15:49:00.114513+0000","last_fullsized":"2024-09-21T15:49:00.114513+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:38:43.937586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036074600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:48:59.589789+0000","last_change":"2024-09-21T15:48:59.589789+0000","last_active":"2024-09-21T15:48:59.589789+0000","last_peered":"2024-09-21T15:48:59.589789+0000","last_clean":"2024-09-21T15:48:59.589789+0000","last_became_active":"2024-09-21T15:48:57.936973+0000","last_became_peered":"2024-09-21T15:48:57.936973+0000","last_unstale":"2024-09-21T15:48:59.589789+0000","last_undegraded":"2024-09-21T15:48:59.589789+0000","last_fullsized":"2024-09-21T15:48:59.589789+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:41:58.458308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040525700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"20'2","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:48:59.600124+0000","last_change":"2024-09-21T15:48:59.595513+0000","last_active":"2024-09-21T15:48:59.600124+0000","last_peered":"2024-09-21T15:48:59.600124+0000","last_clean":"2024-09-21T15:48:59.600124+0000","last_became_active":"2024-09-21T15:48:57.585496+0000","last_became_peered":"2024-09-21T15:48:57.585496+0000","last_unstale":"2024-09-21T15:48:59.600124+0000","last_undegraded":"2024-09-21T15:48:59.600124+0000","last_fullsized":"2024-09-21T15:48:59.600124+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:01:35.910466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000859595,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:48:59.590261+0000","last_change":"2024-09-21T15:48:59.590261+0000","last_active":"2024-09-21T15:48:59.590261+0000","last_peered":"2024-09-21T15:48:59.590261+0000","last_clean":"2024-09-21T15:48:59.590261+0000","last_became_active":"2024-09-21T15:48:57.936108+0000","last_became_peered":"2024-09-21T15:48:57.936108+0000","last_unstale":"2024-09-21T15:48:59.590261+0000","last_undegraded":"2024-09-21T15:48:59.590261+0000","last_fullsized":"2024-09-21T15:48:59.590261+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:06:25.149550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043235099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:49:00.114548+0000","last_change":"2024-09-21T15:49:00.114730+0000","last_active":"2024-09-21T15:49:00.114548+0000","last_peered":"2024-09-21T15:49:00.114548+0000","last_clean":"2024-09-21T15:49:00.114548+0000","last_became_active":"2024-09-21T15:48:57.585525+0000","last_became_peered":"2024-09-21T15:48:57.585525+0000","last_unstale":"2024-09-21T15:49:00.114548+0000","last_undegraded":"2024-09-21T15:49:00.114548+0000","last_fullsized":"2024-09-21T15:49:00.114548+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:35:35.786526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031691100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"18'1","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:48:59.595056+0000","last_change":"2024-09-21T15:48:59.595056+0000","last_active":"2024-09-21T15:48:59.595056+0000","last_peered":"2024-09-21T15:48:59.595056+0000","last_clean":"2024-09-21T15:48:59.595056+0000","last_became_active":"2024-09-21T15:48:57.585384+0000","last_became_peered":"2024-09-21T15:48:57.585384+0000","last_unstale":"2024-09-21T15:48:59.595056+0000","last_undegraded":"2024-09-21T15:48:59.595056+0000","last_fullsized":"2024-09-21T15:48:59.595056+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:17:00.878205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039559000000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"16'117","reported_seq":164,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T15:48:58.780157+0000","last_change":"2024-09-21T15:48:51.381021+0000","last_active":"2024-09-21T15:48:58.780157+0000","last_peered":"2024-09-21T15:48:58.780157+0000","last_clean":"2024-09-21T15:48:58.780157+0000","last_became_active":"2024-09-21T15:48:51.380092+0000","last_became_peered":"2024-09-21T15:48:51.380092+0000","last_unstale":"2024-09-21T15:48:58.780157+0000","last_undegraded":"2024-09-21T15:48:58.780157+0000","last_fullsized":"2024-09-21T15:48:58.780157+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:50.161884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:50.161884+0000","last_clean_scrub_stamp":"2024-09-21T15:48:50.161884+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:32:51.001923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27788,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344052,"statfs":{"total":96636764160,"available":96608309248,"internally_reserved":0,"allocated":786432,"data_stored":653396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39000000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]}]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":68860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":68879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73399999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":68860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40300000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":68860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59999999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]}]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":68860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76500000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":68879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71599999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27796,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344044,"statfs":{"total":96636764160,"available":96608301056,"internally_reserved":0,"allocated":786432,"data_stored":653396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28199999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29599999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T15:49:04.439 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T15:49:04.708 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:49:04.709 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-09-21T15:49:04.722 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-21T15:49:03.891922+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":161,"num_write_kb":2418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":120,"ondisk_log_size":120,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":13,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":7,"kb":754974720,"kb_used":218912,"kb_used_data":2744,"kb_used_omap":12,"kb_used_meta":216051,"kb_avail":754755808,"statfs":{"total":773094113280,"available":772869947392,"internally_reserved":0,"allocated":2809856,"data_stored":1719990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":221236792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000971"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:48:59.666431+0000","last_change":"2024-09-21T15:48:59.666431+0000","last_active":"2024-09-21T15:48:59.666431+0000","last_peered":"2024-09-21T15:48:59.666431+0000","last_clean":"2024-09-21T15:48:59.666431+0000","last_became_active":"2024-09-21T15:48:57.583665+0000","last_became_peered":"2024-09-21T15:48:57.583665+0000","last_unstale":"2024-09-21T15:48:59.666431+0000","last_undegraded":"2024-09-21T15:48:59.666431+0000","last_fullsized":"2024-09-21T15:48:59.666431+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:52:09.228964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031098399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:48:59.589790+0000","last_change":"2024-09-21T15:48:59.589790+0000","last_active":"2024-09-21T15:48:59.589790+0000","last_peered":"2024-09-21T15:48:59.589790+0000","last_clean":"2024-09-21T15:48:59.589790+0000","last_became_active":"2024-09-21T15:48:57.582121+0000","last_became_peered":"2024-09-21T15:48:57.582121+0000","last_unstale":"2024-09-21T15:48:59.589790+0000","last_undegraded":"2024-09-21T15:48:59.589790+0000","last_fullsized":"2024-09-21T15:48:59.589790+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:33:47.592624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039654700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:49:00.114513+0000","last_change":"2024-09-21T15:49:00.114779+0000","last_active":"2024-09-21T15:49:00.114513+0000","last_peered":"2024-09-21T15:49:00.114513+0000","last_clean":"2024-09-21T15:49:00.114513+0000","last_became_active":"2024-09-21T15:48:57.937107+0000","last_became_peered":"2024-09-21T15:48:57.937107+0000","last_unstale":"2024-09-21T15:49:00.114513+0000","last_undegraded":"2024-09-21T15:49:00.114513+0000","last_fullsized":"2024-09-21T15:49:00.114513+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:38:43.937586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036074600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:48:59.589789+0000","last_change":"2024-09-21T15:48:59.589789+0000","last_active":"2024-09-21T15:48:59.589789+0000","last_peered":"2024-09-21T15:48:59.589789+0000","last_clean":"2024-09-21T15:48:59.589789+0000","last_became_active":"2024-09-21T15:48:57.936973+0000","last_became_peered":"2024-09-21T15:48:57.936973+0000","last_unstale":"2024-09-21T15:48:59.589789+0000","last_undegraded":"2024-09-21T15:48:59.589789+0000","last_fullsized":"2024-09-21T15:48:59.589789+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:41:58.458308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040525700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"20'2","reported_seq":17,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:48:59.600124+0000","last_change":"2024-09-21T15:48:59.595513+0000","last_active":"2024-09-21T15:48:59.600124+0000","last_peered":"2024-09-21T15:48:59.600124+0000","last_clean":"2024-09-21T15:48:59.600124+0000","last_became_active":"2024-09-21T15:48:57.585496+0000","last_became_peered":"2024-09-21T15:48:57.585496+0000","last_unstale":"2024-09-21T15:48:59.600124+0000","last_undegraded":"2024-09-21T15:48:59.600124+0000","last_fullsized":"2024-09-21T15:48:59.600124+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:01:35.910466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000859595,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:48:59.590261+0000","last_change":"2024-09-21T15:48:59.590261+0000","last_active":"2024-09-21T15:48:59.590261+0000","last_peered":"2024-09-21T15:48:59.590261+0000","last_clean":"2024-09-21T15:48:59.590261+0000","last_became_active":"2024-09-21T15:48:57.936108+0000","last_became_peered":"2024-09-21T15:48:57.936108+0000","last_unstale":"2024-09-21T15:48:59.590261+0000","last_undegraded":"2024-09-21T15:48:59.590261+0000","last_fullsized":"2024-09-21T15:48:59.590261+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:06:25.149550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043235099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":15,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:49:00.114548+0000","last_change":"2024-09-21T15:49:00.114730+0000","last_active":"2024-09-21T15:49:00.114548+0000","last_peered":"2024-09-21T15:49:00.114548+0000","last_clean":"2024-09-21T15:49:00.114548+0000","last_became_active":"2024-09-21T15:48:57.585525+0000","last_became_peered":"2024-09-21T15:48:57.585525+0000","last_unstale":"2024-09-21T15:49:00.114548+0000","last_undegraded":"2024-09-21T15:49:00.114548+0000","last_fullsized":"2024-09-21T15:49:00.114548+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:35:35.786526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031691100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"18'1","reported_seq":16,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-09-21T15:48:59.595056+0000","last_change":"2024-09-21T15:48:59.595056+0000","last_active":"2024-09-21T15:48:59.595056+0000","last_peered":"2024-09-21T15:48:59.595056+0000","last_clean":"2024-09-21T15:48:59.595056+0000","last_became_active":"2024-09-21T15:48:57.585384+0000","last_became_peered":"2024-09-21T15:48:57.585384+0000","last_unstale":"2024-09-21T15:48:59.595056+0000","last_undegraded":"2024-09-21T15:48:59.595056+0000","last_fullsized":"2024-09-21T15:48:59.595056+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:17:00.878205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039559000000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"16'117","reported_seq":164,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-21T15:48:58.780157+0000","last_change":"2024-09-21T15:48:51.381021+0000","last_active":"2024-09-21T15:48:58.780157+0000","last_peered":"2024-09-21T15:48:58.780157+0000","last_clean":"2024-09-21T15:48:58.780157+0000","last_became_active":"2024-09-21T15:48:51.380092+0000","last_became_peered":"2024-09-21T15:48:51.380092+0000","last_unstale":"2024-09-21T15:48:58.780157+0000","last_undegraded":"2024-09-21T15:48:58.780157+0000","last_fullsized":"2024-09-21T15:48:58.780157+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:50.161884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:50.161884+0000","last_clean_scrub_stamp":"2024-09-21T15:48:50.161884+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:32:51.001923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574853,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27788,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344052,"statfs":{"total":96636764160,"available":96608309248,"internally_reserved":0,"allocated":786432,"data_stored":653396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":27654603},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51900000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39000000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]}]},{"osd":6,"up_from":13,"seq":55834574856,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":68860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55100000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69899999999999995}]}]},{"osd":5,"up_from":13,"seq":55834574855,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":68879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.75}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61399999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52800000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73399999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":68860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69799999999999995}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51000000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40300000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]}]},{"osd":3,"up_from":13,"seq":55834574855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":68860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28899999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.376}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59999999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68500000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53000000000000003}]}]},{"osd":2,"up_from":13,"seq":55834574855,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27220,"kb_used_data":200,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344620,"statfs":{"total":96636764160,"available":96608890880,"internally_reserved":0,"allocated":204800,"data_stored":68860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58999999999999997}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55900000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71199999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72499999999999998}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73799999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76500000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":68879,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42999999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49199999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66100000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71599999999999997}]}]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27796,"kb_used_data":768,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344044,"statfs":{"total":96636764160,"available":96608301056,"internally_reserved":0,"allocated":786432,"data_stored":653396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34000000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28199999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29599999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59799999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T15:49:04.723 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-21T15:49:04.723 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2024-09-21T15:49:04.995 INFO:teuthology.orchestra.run.smithi089.stdout:epoch 21 2024-09-21T15:49:04.995 INFO:teuthology.orchestra.run.smithi089.stdout:fsid 4a9b64ea-3aaf-4a40-8ce8-02163a0f6e12 2024-09-21T15:49:04.995 INFO:teuthology.orchestra.run.smithi089.stdout:created 2024-09-21T15:48:37.867501+0000 2024-09-21T15:49:04.995 INFO:teuthology.orchestra.run.smithi089.stdout:modified 2024-09-21T15:49:02.651462+0000 2024-09-21T15:49:04.995 INFO:teuthology.orchestra.run.smithi089.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-21T15:49:04.995 INFO:teuthology.orchestra.run.smithi089.stdout:crush_version 4 2024-09-21T15:49:04.995 INFO:teuthology.orchestra.run.smithi089.stdout:full_ratio 0.95 2024-09-21T15:49:04.995 INFO:teuthology.orchestra.run.smithi089.stdout:backfillfull_ratio 0.9 2024-09-21T15:49:04.995 INFO:teuthology.orchestra.run.smithi089.stdout:nearfull_ratio 0.85 2024-09-21T15:49:04.995 INFO:teuthology.orchestra.run.smithi089.stdout:require_min_compat_client luminous 2024-09-21T15:49:04.995 INFO:teuthology.orchestra.run.smithi089.stdout:min_compat_client jewel 2024-09-21T15:49:04.995 INFO:teuthology.orchestra.run.smithi089.stdout:require_osd_release reef 2024-09-21T15:49:04.996 INFO:teuthology.orchestra.run.smithi089.stdout:stretch_mode_enabled false 2024-09-21T15:49:04.996 INFO:teuthology.orchestra.run.smithi089.stdout:pool 1 '.mgr' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 16 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 8.00 2024-09-21T15:49:04.996 INFO:teuthology.orchestra.run.smithi089.stdout:pool 2 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 20 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 1.99 2024-09-21T15:49:04.996 INFO:teuthology.orchestra.run.smithi089.stdout:max_osd 8 2024-09-21T15:49:04.996 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.89:6800/3360073633,v1:172.21.15.89:6801/3360073633] [v2:172.21.15.89:6803/3360073633,v1:172.21.15.89:6805/3360073633] exists,up 7b3fabfa-3fe6-47c7-9d0c-a6036f738771 2024-09-21T15:49:04.996 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1 up in weight 1 up_from 13 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:172.21.15.89:6816/3187095367,v1:172.21.15.89:6817/3187095367] [v2:172.21.15.89:6818/3187095367,v1:172.21.15.89:6819/3187095367] exists,up 767ad1bc-84ef-4102-83f5-aacda0917ddb 2024-09-21T15:49:04.996 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2 up in weight 1 up_from 13 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:172.21.15.89:6802/4098625737,v1:172.21.15.89:6804/4098625737] [v2:172.21.15.89:6806/4098625737,v1:172.21.15.89:6808/4098625737] exists,up 1dbd0d50-aae9-42a7-90a1-aa92697778ae 2024-09-21T15:49:04.996 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.89:6823/1886828410,v1:172.21.15.89:6825/1886828410] [v2:172.21.15.89:6826/1886828410,v1:172.21.15.89:6827/1886828410] exists,up f37cd295-2bac-4df4-a99b-abedc0978dce 2024-09-21T15:49:04.996 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.100:6817/22722842,v1:172.21.15.100:6819/22722842] [v2:172.21.15.100:6820/22722842,v1:172.21.15.100:6822/22722842] exists,up 2eeb351d-d51c-4fc3-884e-debecd995aea 2024-09-21T15:49:04.996 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5 up in weight 1 up_from 13 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:172.21.15.100:6816/363733545,v1:172.21.15.100:6818/363733545] [v2:172.21.15.100:6821/363733545,v1:172.21.15.100:6823/363733545] exists,up b5396267-6823-42f1-bd04-bd8f144fa76c 2024-09-21T15:49:04.996 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6 up in weight 1 up_from 13 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:172.21.15.100:6808/205694879,v1:172.21.15.100:6809/205694879] [v2:172.21.15.100:6810/205694879,v1:172.21.15.100:6811/205694879] exists,up 955b67e9-73bc-42c1-88e0-3356c105e1ea 2024-09-21T15:49:04.996 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7 up in weight 1 up_from 13 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:172.21.15.100:6800/3120011840,v1:172.21.15.100:6801/3120011840] [v2:172.21.15.100:6802/3120011840,v1:172.21.15.100:6803/3120011840] exists,up 414ea7f0-138d-404e-bce6-fe8dedcdc938 2024-09-21T15:49:05.007 INFO:tasks.ceph.ceph_manager.ceph:['osd.0 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.89:6800/3360073633,v1:172.21.15.89:6801/3360073633] [v2:172.21.15.89:6803/3360073633,v1:172.21.15.89:6805/3360073633] exists,up 7b3fabfa-3fe6-47c7-9d0c-a6036f738771', 'osd.1 up in weight 1 up_from 13 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:172.21.15.89:6816/3187095367,v1:172.21.15.89:6817/3187095367] [v2:172.21.15.89:6818/3187095367,v1:172.21.15.89:6819/3187095367] exists,up 767ad1bc-84ef-4102-83f5-aacda0917ddb', 'osd.2 up in weight 1 up_from 13 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:172.21.15.89:6802/4098625737,v1:172.21.15.89:6804/4098625737] [v2:172.21.15.89:6806/4098625737,v1:172.21.15.89:6808/4098625737] exists,up 1dbd0d50-aae9-42a7-90a1-aa92697778ae', 'osd.3 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.89:6823/1886828410,v1:172.21.15.89:6825/1886828410] [v2:172.21.15.89:6826/1886828410,v1:172.21.15.89:6827/1886828410] exists,up f37cd295-2bac-4df4-a99b-abedc0978dce', 'osd.4 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.100:6817/22722842,v1:172.21.15.100:6819/22722842] [v2:172.21.15.100:6820/22722842,v1:172.21.15.100:6822/22722842] exists,up 2eeb351d-d51c-4fc3-884e-debecd995aea', 'osd.5 up in weight 1 up_from 13 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:172.21.15.100:6816/363733545,v1:172.21.15.100:6818/363733545] [v2:172.21.15.100:6821/363733545,v1:172.21.15.100:6823/363733545] exists,up b5396267-6823-42f1-bd04-bd8f144fa76c', 'osd.6 up in weight 1 up_from 13 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:172.21.15.100:6808/205694879,v1:172.21.15.100:6809/205694879] [v2:172.21.15.100:6810/205694879,v1:172.21.15.100:6811/205694879] exists,up 955b67e9-73bc-42c1-88e0-3356c105e1ea', 'osd.7 up in weight 1 up_from 13 up_thru 17 down_at 0 last_clean_interval [0,0) [v2:172.21.15.100:6800/3120011840,v1:172.21.15.100:6801/3120011840] [v2:172.21.15.100:6802/3120011840,v1:172.21.15.100:6803/3120011840] exists,up 414ea7f0-138d-404e-bce6-fe8dedcdc938'] 2024-09-21T15:49:05.008 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok config show 2024-09-21T15:49:05.106 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:05.106 INFO:teuthology.orchestra.run.smithi089.stdout: "name": "mon.a", 2024-09-21T15:49:05.106 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster": "ceph", 2024-09-21T15:49:05.106 INFO:teuthology.orchestra.run.smithi089.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-21T15:49:05.106 INFO:teuthology.orchestra.run.smithi089.stdout: "admin_socket_mode": "", 2024-09-21T15:49:05.106 INFO:teuthology.orchestra.run.smithi089.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-09-21T15:49:05.107 INFO:teuthology.orchestra.run.smithi089.stdout: "auth_client_required": "cephx, none", 2024-09-21T15:49:05.107 INFO:teuthology.orchestra.run.smithi089.stdout: "auth_cluster_required": "cephx", 2024-09-21T15:49:05.107 INFO:teuthology.orchestra.run.smithi089.stdout: "auth_debug": "true", 2024-09-21T15:49:05.107 INFO:teuthology.orchestra.run.smithi089.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-21T15:49:05.107 INFO:teuthology.orchestra.run.smithi089.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-21T15:49:05.107 INFO:teuthology.orchestra.run.smithi089.stdout: "auth_service_required": "cephx", 2024-09-21T15:49:05.107 INFO:teuthology.orchestra.run.smithi089.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-21T15:49:05.107 INFO:teuthology.orchestra.run.smithi089.stdout: "auth_supported": "cephx", 2024-09-21T15:49:05.107 INFO:teuthology.orchestra.run.smithi089.stdout: "bdev_aio": "true", 2024-09-21T15:49:05.107 INFO:teuthology.orchestra.run.smithi089.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-21T15:49:05.107 INFO:teuthology.orchestra.run.smithi089.stdout: "bdev_aio_poll_ms": "250", 2024-09-21T15:49:05.107 INFO:teuthology.orchestra.run.smithi089.stdout: "bdev_aio_reap_max": "16", 2024-09-21T15:49:05.107 INFO:teuthology.orchestra.run.smithi089.stdout: "bdev_async_discard": "false", 2024-09-21T15:49:05.107 INFO:teuthology.orchestra.run.smithi089.stdout: "bdev_block_size": "4096", 2024-09-21T15:49:05.107 INFO:teuthology.orchestra.run.smithi089.stdout: "bdev_debug_aio": "false", 2024-09-21T15:49:05.108 INFO:teuthology.orchestra.run.smithi089.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-21T15:49:05.108 INFO:teuthology.orchestra.run.smithi089.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-21T15:49:05.108 INFO:teuthology.orchestra.run.smithi089.stdout: "bdev_debug_inflight_ios": "false", 2024-09-21T15:49:05.108 INFO:teuthology.orchestra.run.smithi089.stdout: "bdev_enable_discard": "false", 2024-09-21T15:49:05.108 INFO:teuthology.orchestra.run.smithi089.stdout: "bdev_flock_retry": "3", 2024-09-21T15:49:05.108 INFO:teuthology.orchestra.run.smithi089.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-21T15:49:05.108 INFO:teuthology.orchestra.run.smithi089.stdout: "bdev_inject_crash": "0", 2024-09-21T15:49:05.108 INFO:teuthology.orchestra.run.smithi089.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-21T15:49:05.108 INFO:teuthology.orchestra.run.smithi089.stdout: "bdev_ioring": "false", 2024-09-21T15:49:05.108 INFO:teuthology.orchestra.run.smithi089.stdout: "bdev_ioring_hipri": "false", 2024-09-21T15:49:05.108 INFO:teuthology.orchestra.run.smithi089.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-21T15:49:05.108 INFO:teuthology.orchestra.run.smithi089.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-21T15:49:05.108 INFO:teuthology.orchestra.run.smithi089.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-21T15:49:05.108 INFO:teuthology.orchestra.run.smithi089.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-21T15:49:05.108 INFO:teuthology.orchestra.run.smithi089.stdout: "bdev_type": "", 2024-09-21T15:49:05.109 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_alloc_size": "1048576", 2024-09-21T15:49:05.109 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_allocator": "hybrid", 2024-09-21T15:49:05.109 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_buffered_io": "true", 2024-09-21T15:49:05.109 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_check_for_zeros": "false", 2024-09-21T15:49:05.109 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-21T15:49:05.109 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-21T15:49:05.109 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_compact_log_sync": "false", 2024-09-21T15:49:05.109 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-21T15:49:05.109 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-21T15:49:05.109 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-21T15:49:05.109 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-21T15:49:05.109 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_max_log_runway": "4194304", 2024-09-21T15:49:05.109 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_max_prefetch": "1048576", 2024-09-21T15:49:05.109 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_min_flush_size": "524288", 2024-09-21T15:49:05.109 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_min_log_runway": "1048576", 2024-09-21T15:49:05.109 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_replay_recovery": "false", 2024-09-21T15:49:05.110 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-21T15:49:05.110 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-21T15:49:05.110 INFO:teuthology.orchestra.run.smithi089.stdout: "bluefs_sync_write": "false", 2024-09-21T15:49:05.110 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-21T15:49:05.110 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-21T15:49:05.110 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-21T15:49:05.110 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_allocation_from_file": "true", 2024-09-21T15:49:05.110 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_allocator": "hybrid", 2024-09-21T15:49:05.110 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-21T15:49:05.110 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-21T15:49:05.110 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-21T15:49:05.110 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-21T15:49:05.110 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-21T15:49:05.110 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-21T15:49:05.110 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-21T15:49:05.111 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_block_create": "true", 2024-09-21T15:49:05.111 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_block_db_create": "false", 2024-09-21T15:49:05.111 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_block_db_path": "", 2024-09-21T15:49:05.111 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_block_db_size": "0", 2024-09-21T15:49:05.111 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_block_path": "", 2024-09-21T15:49:05.111 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_block_preallocate_file": "false", 2024-09-21T15:49:05.111 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_block_size": "107374182400", 2024-09-21T15:49:05.111 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_block_wal_create": "false", 2024-09-21T15:49:05.111 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_block_wal_path": "", 2024-09-21T15:49:05.111 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_block_wal_size": "100663296", 2024-09-21T15:49:05.111 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_bluefs": "true", 2024-09-21T15:49:05.111 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-21T15:49:05.111 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-21T15:49:05.111 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-21T15:49:05.112 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-21T15:49:05.112 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-21T15:49:05.112 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-21T15:49:05.112 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-21T15:49:05.112 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-21T15:49:05.112 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_autotune": "true", 2024-09-21T15:49:05.112 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-21T15:49:05.112 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-21T15:49:05.113 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-21T15:49:05.113 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-21T15:49:05.113 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_size": "0", 2024-09-21T15:49:05.113 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-21T15:49:05.113 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-21T15:49:05.113 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-21T15:49:05.113 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-21T15:49:05.113 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cache_type": "2q", 2024-09-21T15:49:05.113 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-21T15:49:05.113 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_clone_cow": "true", 2024-09-21T15:49:05.113 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-21T15:49:05.113 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-21T15:49:05.113 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-21T15:49:05.114 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-21T15:49:05.114 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-21T15:49:05.114 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-21T15:49:05.114 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-21T15:49:05.114 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_compression_mode": "none", 2024-09-21T15:49:05.114 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-21T15:49:05.114 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_csum_type": "crc32c", 2024-09-21T15:49:05.114 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-21T15:49:05.114 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_debug_freelist": "false", 2024-09-21T15:49:05.114 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-21T15:49:05.114 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-21T15:49:05.114 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_debug_inject_bug21040": "false", 2024-09-21T15:49:05.114 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-21T15:49:05.114 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-21T15:49:05.115 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-21T15:49:05.115 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-21T15:49:05.115 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-21T15:49:05.115 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-21T15:49:05.115 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-21T15:49:05.115 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_debug_prefill": "0.000000", 2024-09-21T15:49:05.115 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-21T15:49:05.115 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-21T15:49:05.115 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-21T15:49:05.115 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_debug_small_allocations": "0", 2024-09-21T15:49:05.115 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-21T15:49:05.115 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_default_buffered_read": "true", 2024-09-21T15:49:05.115 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_default_buffered_write": "false", 2024-09-21T15:49:05.115 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-21T15:49:05.116 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-21T15:49:05.116 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-21T15:49:05.116 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-21T15:49:05.116 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-21T15:49:05.116 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-21T15:49:05.116 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-21T15:49:05.116 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-21T15:49:05.116 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fail_eio": "false", 2024-09-21T15:49:05.116 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-21T15:49:05.116 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-21T15:49:05.116 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-21T15:49:05.116 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-21T15:49:05.116 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-21T15:49:05.116 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-21T15:49:05.116 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_on_mount": "false", 2024-09-21T15:49:05.117 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-21T15:49:05.117 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_on_umount": "false", 2024-09-21T15:49:05.117 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-21T15:49:05.117 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-21T15:49:05.117 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-21T15:49:05.117 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-21T15:49:05.117 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-21T15:49:05.117 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-21T15:49:05.117 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-21T15:49:05.117 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-21T15:49:05.117 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_ignore_data_csum": "false", 2024-09-21T15:49:05.117 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-21T15:49:05.117 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-21T15:49:05.117 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-21T15:49:05.117 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-21T15:49:05.118 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_log_op_age": "5.000000", 2024-09-21T15:49:05.118 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_max_alloc_size": "0", 2024-09-21T15:49:05.118 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_max_blob_size": "0", 2024-09-21T15:49:05.118 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-21T15:49:05.118 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-21T15:49:05.118 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-21T15:49:05.118 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_max_deferred_txc": "32", 2024-09-21T15:49:05.118 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_min_alloc_size": "0", 2024-09-21T15:49:05.118 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-21T15:49:05.118 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-21T15:49:05.118 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_nid_prealloc": "1024", 2024-09-21T15:49:05.118 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-21T15:49:05.118 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-21T15:49:05.118 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-21T15:49:05.119 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-21T15:49:05.119 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_retry_disk_reads": "3", 2024-09-21T15:49:05.119 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_rocksdb_cf": "true", 2024-09-21T15:49:05.119 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-09-21T15:49:05.119 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-09-21T15:49:05.119 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-21T15:49:05.119 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-21T15:49:05.119 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-21T15:49:05.119 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-21T15:49:05.119 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_spdk_mem": "512", 2024-09-21T15:49:05.119 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-21T15:49:05.119 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-21T15:49:05.119 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-21T15:49:05.119 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-21T15:49:05.120 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-21T15:49:05.120 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-21T15:49:05.120 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-21T15:49:05.120 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_tracing": "false", 2024-09-21T15:49:05.120 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-21T15:49:05.120 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-21T15:49:05.120 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-21T15:49:05.120 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-21T15:49:05.120 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-21T15:49:05.120 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-21T15:49:05.120 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-21T15:49:05.120 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-21T15:49:05.120 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-21T15:49:05.120 INFO:teuthology.orchestra.run.smithi089.stdout: "bluestore_zero_block_detection": "false", 2024-09-21T15:49:05.121 INFO:teuthology.orchestra.run.smithi089.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-21T15:49:05.121 INFO:teuthology.orchestra.run.smithi089.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-21T15:49:05.121 INFO:teuthology.orchestra.run.smithi089.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-21T15:49:05.121 INFO:teuthology.orchestra.run.smithi089.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-21T15:49:05.121 INFO:teuthology.orchestra.run.smithi089.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-21T15:49:05.121 INFO:teuthology.orchestra.run.smithi089.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-21T15:49:05.121 INFO:teuthology.orchestra.run.smithi089.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-21T15:49:05.121 INFO:teuthology.orchestra.run.smithi089.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-09-21T15:49:05.121 INFO:teuthology.orchestra.run.smithi089.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-21T15:49:05.121 INFO:teuthology.orchestra.run.smithi089.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-21T15:49:05.121 INFO:teuthology.orchestra.run.smithi089.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-21T15:49:05.121 INFO:teuthology.orchestra.run.smithi089.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-21T15:49:05.121 INFO:teuthology.orchestra.run.smithi089.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-21T15:49:05.121 INFO:teuthology.orchestra.run.smithi089.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-21T15:49:05.121 INFO:teuthology.orchestra.run.smithi089.stdout: "cephx_cluster_require_signatures": "false", 2024-09-21T15:49:05.122 INFO:teuthology.orchestra.run.smithi089.stdout: "cephx_cluster_require_version": "2", 2024-09-21T15:49:05.122 INFO:teuthology.orchestra.run.smithi089.stdout: "cephx_require_signatures": "false", 2024-09-21T15:49:05.122 INFO:teuthology.orchestra.run.smithi089.stdout: "cephx_require_version": "2", 2024-09-21T15:49:05.122 INFO:teuthology.orchestra.run.smithi089.stdout: "cephx_service_require_signatures": "false", 2024-09-21T15:49:05.122 INFO:teuthology.orchestra.run.smithi089.stdout: "cephx_service_require_version": "2", 2024-09-21T15:49:05.122 INFO:teuthology.orchestra.run.smithi089.stdout: "cephx_sign_messages": "true", 2024-09-21T15:49:05.122 INFO:teuthology.orchestra.run.smithi089.stdout: "chdir": "", 2024-09-21T15:49:05.122 INFO:teuthology.orchestra.run.smithi089.stdout: "client_acl_type": "", 2024-09-21T15:49:05.122 INFO:teuthology.orchestra.run.smithi089.stdout: "client_asio_thread_count": "2", 2024-09-21T15:49:05.122 INFO:teuthology.orchestra.run.smithi089.stdout: "client_cache_mid": "0.750000", 2024-09-21T15:49:05.122 INFO:teuthology.orchestra.run.smithi089.stdout: "client_cache_size": "16384", 2024-09-21T15:49:05.122 INFO:teuthology.orchestra.run.smithi089.stdout: "client_caps_release_delay": "5", 2024-09-21T15:49:05.122 INFO:teuthology.orchestra.run.smithi089.stdout: "client_check_pool_perm": "true", 2024-09-21T15:49:05.123 INFO:teuthology.orchestra.run.smithi089.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-21T15:49:05.123 INFO:teuthology.orchestra.run.smithi089.stdout: "client_debug_force_sync_read": "false", 2024-09-21T15:49:05.123 INFO:teuthology.orchestra.run.smithi089.stdout: "client_debug_getattr_caps": "false", 2024-09-21T15:49:05.123 INFO:teuthology.orchestra.run.smithi089.stdout: "client_debug_inject_features": "", 2024-09-21T15:49:05.123 INFO:teuthology.orchestra.run.smithi089.stdout: "client_debug_inject_tick_delay": "0", 2024-09-21T15:49:05.123 INFO:teuthology.orchestra.run.smithi089.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-21T15:49:05.123 INFO:teuthology.orchestra.run.smithi089.stdout: "client_die_on_failed_remount": "false", 2024-09-21T15:49:05.123 INFO:teuthology.orchestra.run.smithi089.stdout: "client_dirsize_rbytes": "true", 2024-09-21T15:49:05.123 INFO:teuthology.orchestra.run.smithi089.stdout: "client_force_lazyio": "false", 2024-09-21T15:49:05.123 INFO:teuthology.orchestra.run.smithi089.stdout: "client_fs": "", 2024-09-21T15:49:05.123 INFO:teuthology.orchestra.run.smithi089.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-21T15:49:05.123 INFO:teuthology.orchestra.run.smithi089.stdout: "client_inject_release_failure": "false", 2024-09-21T15:49:05.123 INFO:teuthology.orchestra.run.smithi089.stdout: "client_max_inline_size": "4096", 2024-09-21T15:49:05.123 INFO:teuthology.orchestra.run.smithi089.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-21T15:49:05.123 INFO:teuthology.orchestra.run.smithi089.stdout: "client_mds_namespace": "", 2024-09-21T15:49:05.124 INFO:teuthology.orchestra.run.smithi089.stdout: "client_metadata": "", 2024-09-21T15:49:05.124 INFO:teuthology.orchestra.run.smithi089.stdout: "client_mount_gid": "-1", 2024-09-21T15:49:05.124 INFO:teuthology.orchestra.run.smithi089.stdout: "client_mount_timeout": "300", 2024-09-21T15:49:05.124 INFO:teuthology.orchestra.run.smithi089.stdout: "client_mount_uid": "-1", 2024-09-21T15:49:05.124 INFO:teuthology.orchestra.run.smithi089.stdout: "client_mountpoint": "/", 2024-09-21T15:49:05.124 INFO:teuthology.orchestra.run.smithi089.stdout: "client_notify_timeout": "10", 2024-09-21T15:49:05.124 INFO:teuthology.orchestra.run.smithi089.stdout: "client_oc": "true", 2024-09-21T15:49:05.124 INFO:teuthology.orchestra.run.smithi089.stdout: "client_oc_max_dirty": "104857600", 2024-09-21T15:49:05.124 INFO:teuthology.orchestra.run.smithi089.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-21T15:49:05.124 INFO:teuthology.orchestra.run.smithi089.stdout: "client_oc_max_objects": "1000", 2024-09-21T15:49:05.124 INFO:teuthology.orchestra.run.smithi089.stdout: "client_oc_size": "209715200", 2024-09-21T15:49:05.124 INFO:teuthology.orchestra.run.smithi089.stdout: "client_oc_target_dirty": "8388608", 2024-09-21T15:49:05.124 INFO:teuthology.orchestra.run.smithi089.stdout: "client_permissions": "true", 2024-09-21T15:49:05.124 INFO:teuthology.orchestra.run.smithi089.stdout: "client_quota": "true", 2024-09-21T15:49:05.125 INFO:teuthology.orchestra.run.smithi089.stdout: "client_quota_df": "true", 2024-09-21T15:49:05.125 INFO:teuthology.orchestra.run.smithi089.stdout: "client_readahead_max_bytes": "0", 2024-09-21T15:49:05.125 INFO:teuthology.orchestra.run.smithi089.stdout: "client_readahead_max_periods": "4", 2024-09-21T15:49:05.125 INFO:teuthology.orchestra.run.smithi089.stdout: "client_readahead_min": "131072", 2024-09-21T15:49:05.125 INFO:teuthology.orchestra.run.smithi089.stdout: "client_reconnect_stale": "false", 2024-09-21T15:49:05.125 INFO:teuthology.orchestra.run.smithi089.stdout: "client_shutdown_timeout": "30", 2024-09-21T15:49:05.125 INFO:teuthology.orchestra.run.smithi089.stdout: "client_snapdir": ".snap", 2024-09-21T15:49:05.125 INFO:teuthology.orchestra.run.smithi089.stdout: "client_tick_interval": "1", 2024-09-21T15:49:05.125 INFO:teuthology.orchestra.run.smithi089.stdout: "client_trace": "", 2024-09-21T15:49:05.125 INFO:teuthology.orchestra.run.smithi089.stdout: "client_try_dentry_invalidate": "false", 2024-09-21T15:49:05.125 INFO:teuthology.orchestra.run.smithi089.stdout: "client_use_faked_inos": "false", 2024-09-21T15:49:05.125 INFO:teuthology.orchestra.run.smithi089.stdout: "client_use_random_mds": "false", 2024-09-21T15:49:05.125 INFO:teuthology.orchestra.run.smithi089.stdout: "clog_to_graylog": "false", 2024-09-21T15:49:05.125 INFO:teuthology.orchestra.run.smithi089.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-21T15:49:05.125 INFO:teuthology.orchestra.run.smithi089.stdout: "clog_to_graylog_port": "12201", 2024-09-21T15:49:05.126 INFO:teuthology.orchestra.run.smithi089.stdout: "clog_to_monitors": "default=true", 2024-09-21T15:49:05.126 INFO:teuthology.orchestra.run.smithi089.stdout: "clog_to_syslog": "false", 2024-09-21T15:49:05.126 INFO:teuthology.orchestra.run.smithi089.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-21T15:49:05.126 INFO:teuthology.orchestra.run.smithi089.stdout: "clog_to_syslog_level": "info", 2024-09-21T15:49:05.126 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "-", 2024-09-21T15:49:05.126 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_network": "", 2024-09-21T15:49:05.126 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_network_interface": "", 2024-09-21T15:49:05.126 INFO:teuthology.orchestra.run.smithi089.stdout: "compressor_zlib_isal": "false", 2024-09-21T15:49:05.126 INFO:teuthology.orchestra.run.smithi089.stdout: "compressor_zlib_level": "5", 2024-09-21T15:49:05.126 INFO:teuthology.orchestra.run.smithi089.stdout: "compressor_zlib_winsize": "-15", 2024-09-21T15:49:05.126 INFO:teuthology.orchestra.run.smithi089.stdout: "compressor_zstd_level": "1", 2024-09-21T15:49:05.126 INFO:teuthology.orchestra.run.smithi089.stdout: "container_image": "docker.io/ceph/daemon-base:latest-master-devel", 2024-09-21T15:49:05.126 INFO:teuthology.orchestra.run.smithi089.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-21T15:49:05.126 INFO:teuthology.orchestra.run.smithi089.stdout: "crimson_alien_op_num_threads": "6", 2024-09-21T15:49:05.127 INFO:teuthology.orchestra.run.smithi089.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-21T15:49:05.127 INFO:teuthology.orchestra.run.smithi089.stdout: "crimson_osd_obc_lru_size": "10", 2024-09-21T15:49:05.127 INFO:teuthology.orchestra.run.smithi089.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-21T15:49:05.127 INFO:teuthology.orchestra.run.smithi089.stdout: "crimson_seastar_smp": "1", 2024-09-21T15:49:05.127 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_location": "", 2024-09-21T15:49:05.127 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_location_hook": "", 2024-09-21T15:49:05.127 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_location_hook_timeout": "10", 2024-09-21T15:49:05.127 INFO:teuthology.orchestra.run.smithi089.stdout: "daemonize": "false", 2024-09-21T15:49:05.127 INFO:teuthology.orchestra.run.smithi089.stdout: "daos_pool": "tank", 2024-09-21T15:49:05.127 INFO:teuthology.orchestra.run.smithi089.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-09-21T15:49:05.127 INFO:teuthology.orchestra.run.smithi089.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-09-21T15:49:05.127 INFO:teuthology.orchestra.run.smithi089.stdout: "dbstore_db_name_prefix": "dbstore", 2024-09-21T15:49:05.127 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_alienstore": "0/5", 2024-09-21T15:49:05.127 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_allow_any_pool_priority": "false", 2024-09-21T15:49:05.128 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_asok": "1/5", 2024-09-21T15:49:05.128 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_asok_assert_abort": "false", 2024-09-21T15:49:05.128 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_asserts_on_shutdown": "true", 2024-09-21T15:49:05.128 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_auth": "20/20", 2024-09-21T15:49:05.128 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_bdev": "1/3", 2024-09-21T15:49:05.128 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_bluefs": "1/5", 2024-09-21T15:49:05.128 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_bluestore": "1/5", 2024-09-21T15:49:05.128 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_buffer": "0/1", 2024-09-21T15:49:05.128 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_ceph_exporter": "1/5", 2024-09-21T15:49:05.128 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_cephfs_mirror": "0/5", 2024-09-21T15:49:05.128 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_cephsqlite": "0/5", 2024-09-21T15:49:05.128 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_client": "0/5", 2024-09-21T15:49:05.128 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_compressor": "1/5", 2024-09-21T15:49:05.128 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_context": "0/1", 2024-09-21T15:49:05.128 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_crush": "1/1", 2024-09-21T15:49:05.129 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_crypto": "1/5", 2024-09-21T15:49:05.129 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_cyanstore": "0/5", 2024-09-21T15:49:05.129 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_deliberately_leak_memory": "false", 2024-09-21T15:49:05.129 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_disable_randomized_ping": "false", 2024-09-21T15:49:05.129 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_dpdk": "1/5", 2024-09-21T15:49:05.129 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_eventtrace": "1/5", 2024-09-21T15:49:05.129 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_filer": "0/1", 2024-09-21T15:49:05.129 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_filestore": "1/3", 2024-09-21T15:49:05.129 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_finisher": "1/1", 2024-09-21T15:49:05.129 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_fuse": "1/5", 2024-09-21T15:49:05.130 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_heartbeat_testing_span": "0", 2024-09-21T15:49:05.130 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_heartbeatmap": "1/5", 2024-09-21T15:49:05.130 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-21T15:49:05.130 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_javaclient": "1/5", 2024-09-21T15:49:05.130 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_journal": "1/3", 2024-09-21T15:49:05.130 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_journaler": "0/5", 2024-09-21T15:49:05.130 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_kstore": "1/5", 2024-09-21T15:49:05.131 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_leveldb": "4/5", 2024-09-21T15:49:05.131 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_lockdep": "0/1", 2024-09-21T15:49:05.131 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_mclock": "1/5", 2024-09-21T15:49:05.131 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_mds": "1/5", 2024-09-21T15:49:05.131 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_mds_balancer": "1/5", 2024-09-21T15:49:05.131 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_mds_locker": "1/5", 2024-09-21T15:49:05.131 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_mds_log": "1/5", 2024-09-21T15:49:05.132 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_mds_log_expire": "1/5", 2024-09-21T15:49:05.132 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_mds_migrator": "1/5", 2024-09-21T15:49:05.132 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_memstore": "1/5", 2024-09-21T15:49:05.132 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_mgr": "2/5", 2024-09-21T15:49:05.132 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_mgrc": "1/5", 2024-09-21T15:49:05.132 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_mon": "20/20", 2024-09-21T15:49:05.132 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_monc": "0/10", 2024-09-21T15:49:05.133 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_ms": "1/1", 2024-09-21T15:49:05.133 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_none": "0/5", 2024-09-21T15:49:05.133 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_objclass": "0/5", 2024-09-21T15:49:05.133 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_objectcacher": "0/5", 2024-09-21T15:49:05.133 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_objecter": "0/1", 2024-09-21T15:49:05.133 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_optracker": "0/5", 2024-09-21T15:49:05.133 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_osd": "1/5", 2024-09-21T15:49:05.134 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_paxos": "20/20", 2024-09-21T15:49:05.134 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_perfcounter": "1/5", 2024-09-21T15:49:05.134 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_prioritycache": "1/5", 2024-09-21T15:49:05.134 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_rados": "0/5", 2024-09-21T15:49:05.134 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_rbd": "0/5", 2024-09-21T15:49:05.134 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_rbd_mirror": "0/5", 2024-09-21T15:49:05.135 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_rbd_pwl": "0/5", 2024-09-21T15:49:05.135 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_rbd_replay": "0/5", 2024-09-21T15:49:05.135 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_refs": "0/0", 2024-09-21T15:49:05.135 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_reserver": "1/1", 2024-09-21T15:49:05.135 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_rgw": "1/5", 2024-09-21T15:49:05.135 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_rgw_access": "1/5", 2024-09-21T15:49:05.135 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_rgw_datacache": "1/5", 2024-09-21T15:49:05.136 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_rgw_dbstore": "1/5", 2024-09-21T15:49:05.136 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_rgw_flight": "1/5", 2024-09-21T15:49:05.136 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_rgw_sync": "1/5", 2024-09-21T15:49:05.136 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_rocksdb": "4/5", 2024-09-21T15:49:05.136 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_seastore": "0/5", 2024-09-21T15:49:05.136 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_seastore_backref": "0/5", 2024-09-21T15:49:05.136 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_seastore_cache": "0/5", 2024-09-21T15:49:05.137 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_seastore_cleaner": "0/5", 2024-09-21T15:49:05.137 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_seastore_device": "0/5", 2024-09-21T15:49:05.137 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_seastore_epm": "0/5", 2024-09-21T15:49:05.137 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-09-21T15:49:05.137 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_seastore_journal": "0/5", 2024-09-21T15:49:05.137 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_seastore_lba": "0/5", 2024-09-21T15:49:05.137 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_seastore_odata": "0/5", 2024-09-21T15:49:05.137 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_seastore_omap": "0/5", 2024-09-21T15:49:05.137 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_seastore_onode": "0/5", 2024-09-21T15:49:05.137 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_seastore_t": "0/5", 2024-09-21T15:49:05.137 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_seastore_tm": "0/5", 2024-09-21T15:49:05.137 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_striper": "0/1", 2024-09-21T15:49:05.137 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_test": "0/5", 2024-09-21T15:49:05.138 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_throttle": "1/1", 2024-09-21T15:49:05.138 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_timer": "0/1", 2024-09-21T15:49:05.138 INFO:teuthology.orchestra.run.smithi089.stdout: "debug_tp": "0/5", 2024-09-21T15:49:05.138 INFO:teuthology.orchestra.run.smithi089.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-21T15:49:05.138 INFO:teuthology.orchestra.run.smithi089.stdout: "device_failure_prediction_mode": "none", 2024-09-21T15:49:05.138 INFO:teuthology.orchestra.run.smithi089.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-21T15:49:05.138 INFO:teuthology.orchestra.run.smithi089.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-09-21T15:49:05.138 INFO:teuthology.orchestra.run.smithi089.stdout: "err_to_graylog": "false", 2024-09-21T15:49:05.138 INFO:teuthology.orchestra.run.smithi089.stdout: "err_to_journald": "false", 2024-09-21T15:49:05.138 INFO:teuthology.orchestra.run.smithi089.stdout: "err_to_stderr": "true", 2024-09-21T15:49:05.138 INFO:teuthology.orchestra.run.smithi089.stdout: "err_to_syslog": "false", 2024-09-21T15:49:05.138 INFO:teuthology.orchestra.run.smithi089.stdout: "event_tracing": "false", 2024-09-21T15:49:05.138 INFO:teuthology.orchestra.run.smithi089.stdout: "exporter_addr": "0.0.0.0", 2024-09-21T15:49:05.139 INFO:teuthology.orchestra.run.smithi089.stdout: "exporter_http_port": "9926", 2024-09-21T15:49:05.139 INFO:teuthology.orchestra.run.smithi089.stdout: "exporter_prio_limit": "5", 2024-09-21T15:49:05.139 INFO:teuthology.orchestra.run.smithi089.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-21T15:49:05.139 INFO:teuthology.orchestra.run.smithi089.stdout: "exporter_sort_metrics": "true", 2024-09-21T15:49:05.139 INFO:teuthology.orchestra.run.smithi089.stdout: "exporter_stats_period": "5", 2024-09-21T15:49:05.139 INFO:teuthology.orchestra.run.smithi089.stdout: "fake_statfs_for_testing": "0", 2024-09-21T15:49:05.139 INFO:teuthology.orchestra.run.smithi089.stdout: "fatal_signal_handlers": "true", 2024-09-21T15:49:05.139 INFO:teuthology.orchestra.run.smithi089.stdout: "filer_max_purge_ops": "10", 2024-09-21T15:49:05.139 INFO:teuthology.orchestra.run.smithi089.stdout: "filer_max_truncate_ops": "128", 2024-09-21T15:49:05.139 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_apply_finisher_threads": "1", 2024-09-21T15:49:05.139 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_blackhole": "false", 2024-09-21T15:49:05.139 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_btrfs_clone_range": "true", 2024-09-21T15:49:05.139 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_btrfs_snap": "true", 2024-09-21T15:49:05.139 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_caller_concurrency": "10", 2024-09-21T15:49:05.140 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_collect_device_partition_information": "true", 2024-09-21T15:49:05.140 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_commit_timeout": "600.000000", 2024-09-21T15:49:05.140 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_debug_inject_read_err": "false", 2024-09-21T15:49:05.140 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_debug_omap_check": "false", 2024-09-21T15:49:05.140 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_debug_random_read_err": "0.000000", 2024-09-21T15:49:05.140 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_debug_verify_split": "false", 2024-09-21T15:49:05.140 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_dump_file": "", 2024-09-21T15:49:05.140 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-21T15:49:05.140 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-21T15:49:05.140 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_fadvise": "true", 2024-09-21T15:49:05.140 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_fail_eio": "true", 2024-09-21T15:49:05.140 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_fd_cache_shards": "16", 2024-09-21T15:49:05.140 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_fd_cache_size": "128", 2024-09-21T15:49:05.140 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_fiemap": "false", 2024-09-21T15:49:05.140 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_fiemap_threshold": "4096", 2024-09-21T15:49:05.141 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-21T15:49:05.141 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-21T15:49:05.141 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_inject_stall": "0", 2024-09-21T15:49:05.141 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_journal_parallel": "false", 2024-09-21T15:49:05.141 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_journal_trailing": "false", 2024-09-21T15:49:05.141 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_journal_writeahead": "false", 2024-09-21T15:49:05.141 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_kill_at": "0", 2024-09-21T15:49:05.141 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-21T15:49:05.141 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-21T15:49:05.141 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-21T15:49:05.141 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-21T15:49:05.141 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-21T15:49:05.141 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_max_inline_xattrs": "0", 2024-09-21T15:49:05.141 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-21T15:49:05.141 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-21T15:49:05.142 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-21T15:49:05.142 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-21T15:49:05.142 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_max_xattr_value_size": "0", 2024-09-21T15:49:05.142 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-21T15:49:05.142 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-21T15:49:05.142 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-21T15:49:05.142 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_merge_threshold": "-10", 2024-09-21T15:49:05.142 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-21T15:49:05.142 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_odsync_write": "false", 2024-09-21T15:49:05.142 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_omap_backend": "rocksdb", 2024-09-21T15:49:05.142 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_omap_backend_path": "", 2024-09-21T15:49:05.142 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-21T15:49:05.142 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-21T15:49:05.142 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-21T15:49:05.143 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_op_thread_timeout": "60", 2024-09-21T15:49:05.143 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_op_threads": "2", 2024-09-21T15:49:05.143 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_punch_hole": "false", 2024-09-21T15:49:05.143 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-21T15:49:05.143 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-21T15:49:05.143 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-21T15:49:05.143 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-21T15:49:05.143 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-21T15:49:05.143 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-21T15:49:05.143 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-21T15:49:05.143 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-21T15:49:05.143 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-21T15:49:05.143 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_queue_max_ops": "50", 2024-09-21T15:49:05.143 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-21T15:49:05.143 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_seek_data_hole": "false", 2024-09-21T15:49:05.144 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_sloppy_crc": "false", 2024-09-21T15:49:05.144 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-21T15:49:05.144 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_splice": "false", 2024-09-21T15:49:05.144 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_split_multiple": "2", 2024-09-21T15:49:05.144 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_split_rand_factor": "20", 2024-09-21T15:49:05.144 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_update_to": "1000", 2024-09-21T15:49:05.144 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-21T15:49:05.144 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-21T15:49:05.144 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-21T15:49:05.144 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-21T15:49:05.144 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-21T15:49:05.144 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-21T15:49:05.144 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_wbthrottle_enable": "true", 2024-09-21T15:49:05.144 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-21T15:49:05.144 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-21T15:49:05.145 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-21T15:49:05.145 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-21T15:49:05.145 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-21T15:49:05.145 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-21T15:49:05.145 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_xfs_extsize": "false", 2024-09-21T15:49:05.145 INFO:teuthology.orchestra.run.smithi089.stdout: "filestore_zfs_snap": "false", 2024-09-21T15:49:05.145 INFO:teuthology.orchestra.run.smithi089.stdout: "fio_dir": "/tmp/fio", 2024-09-21T15:49:05.145 INFO:teuthology.orchestra.run.smithi089.stdout: "fsid": "4a9b64ea-3aaf-4a40-8ce8-02163a0f6e12", 2024-09-21T15:49:05.145 INFO:teuthology.orchestra.run.smithi089.stdout: "fuse_allow_other": "true", 2024-09-21T15:49:05.145 INFO:teuthology.orchestra.run.smithi089.stdout: "fuse_atomic_o_trunc": "true", 2024-09-21T15:49:05.145 INFO:teuthology.orchestra.run.smithi089.stdout: "fuse_big_writes": "true", 2024-09-21T15:49:05.145 INFO:teuthology.orchestra.run.smithi089.stdout: "fuse_debug": "false", 2024-09-21T15:49:05.145 INFO:teuthology.orchestra.run.smithi089.stdout: "fuse_default_permissions": "false", 2024-09-21T15:49:05.145 INFO:teuthology.orchestra.run.smithi089.stdout: "fuse_disable_pagecache": "false", 2024-09-21T15:49:05.145 INFO:teuthology.orchestra.run.smithi089.stdout: "fuse_max_write": "0", 2024-09-21T15:49:05.146 INFO:teuthology.orchestra.run.smithi089.stdout: "fuse_multithreaded": "true", 2024-09-21T15:49:05.146 INFO:teuthology.orchestra.run.smithi089.stdout: "fuse_require_active_mds": "true", 2024-09-21T15:49:05.146 INFO:teuthology.orchestra.run.smithi089.stdout: "fuse_set_user_groups": "true", 2024-09-21T15:49:05.146 INFO:teuthology.orchestra.run.smithi089.stdout: "fuse_splice_move": "true", 2024-09-21T15:49:05.146 INFO:teuthology.orchestra.run.smithi089.stdout: "fuse_splice_read": "true", 2024-09-21T15:49:05.146 INFO:teuthology.orchestra.run.smithi089.stdout: "fuse_splice_write": "true", 2024-09-21T15:49:05.146 INFO:teuthology.orchestra.run.smithi089.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-21T15:49:05.146 INFO:teuthology.orchestra.run.smithi089.stdout: "fuse_use_invalidate_cb": "true", 2024-09-21T15:49:05.146 INFO:teuthology.orchestra.run.smithi089.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-21T15:49:05.146 INFO:teuthology.orchestra.run.smithi089.stdout: "gss_target_name": "ceph", 2024-09-21T15:49:05.146 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_file": "", 2024-09-21T15:49:05.146 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_inject_failure": "0", 2024-09-21T15:49:05.146 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_interval": "5", 2024-09-21T15:49:05.146 INFO:teuthology.orchestra.run.smithi089.stdout: "host": "smithi089", 2024-09-21T15:49:05.147 INFO:teuthology.orchestra.run.smithi089.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-21T15:49:05.147 INFO:teuthology.orchestra.run.smithi089.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-21T15:49:05.147 INFO:teuthology.orchestra.run.smithi089.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-21T15:49:05.147 INFO:teuthology.orchestra.run.smithi089.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-21T15:49:05.147 INFO:teuthology.orchestra.run.smithi089.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-21T15:49:05.147 INFO:teuthology.orchestra.run.smithi089.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-21T15:49:05.147 INFO:teuthology.orchestra.run.smithi089.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-21T15:49:05.147 INFO:teuthology.orchestra.run.smithi089.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-21T15:49:05.147 INFO:teuthology.orchestra.run.smithi089.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-21T15:49:05.147 INFO:teuthology.orchestra.run.smithi089.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-21T15:49:05.147 INFO:teuthology.orchestra.run.smithi089.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-21T15:49:05.147 INFO:teuthology.orchestra.run.smithi089.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-21T15:49:05.147 INFO:teuthology.orchestra.run.smithi089.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-21T15:49:05.147 INFO:teuthology.orchestra.run.smithi089.stdout: "inject_early_sigterm": "false", 2024-09-21T15:49:05.148 INFO:teuthology.orchestra.run.smithi089.stdout: "jaeger_agent_port": "6799", 2024-09-21T15:49:05.148 INFO:teuthology.orchestra.run.smithi089.stdout: "jaeger_tracing_enable": "false", 2024-09-21T15:49:05.148 INFO:teuthology.orchestra.run.smithi089.stdout: "journal_aio": "true", 2024-09-21T15:49:05.148 INFO:teuthology.orchestra.run.smithi089.stdout: "journal_align_min_size": "65536", 2024-09-21T15:49:05.148 INFO:teuthology.orchestra.run.smithi089.stdout: "journal_block_align": "true", 2024-09-21T15:49:05.148 INFO:teuthology.orchestra.run.smithi089.stdout: "journal_block_size": "4096", 2024-09-21T15:49:05.148 INFO:teuthology.orchestra.run.smithi089.stdout: "journal_dio": "true", 2024-09-21T15:49:05.148 INFO:teuthology.orchestra.run.smithi089.stdout: "journal_discard": "false", 2024-09-21T15:49:05.148 INFO:teuthology.orchestra.run.smithi089.stdout: "journal_force_aio": "false", 2024-09-21T15:49:05.148 INFO:teuthology.orchestra.run.smithi089.stdout: "journal_ignore_corruption": "false", 2024-09-21T15:49:05.148 INFO:teuthology.orchestra.run.smithi089.stdout: "journal_max_write_bytes": "10485760", 2024-09-21T15:49:05.148 INFO:teuthology.orchestra.run.smithi089.stdout: "journal_max_write_entries": "100", 2024-09-21T15:49:05.148 INFO:teuthology.orchestra.run.smithi089.stdout: "journal_replay_from": "0", 2024-09-21T15:49:05.148 INFO:teuthology.orchestra.run.smithi089.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-21T15:49:05.148 INFO:teuthology.orchestra.run.smithi089.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-21T15:49:05.149 INFO:teuthology.orchestra.run.smithi089.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-21T15:49:05.149 INFO:teuthology.orchestra.run.smithi089.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-21T15:49:05.149 INFO:teuthology.orchestra.run.smithi089.stdout: "journal_write_header_frequency": "0", 2024-09-21T15:49:05.149 INFO:teuthology.orchestra.run.smithi089.stdout: "journal_zero_on_create": "false", 2024-09-21T15:49:05.149 INFO:teuthology.orchestra.run.smithi089.stdout: "journaler_prefetch_periods": "10", 2024-09-21T15:49:05.149 INFO:teuthology.orchestra.run.smithi089.stdout: "journaler_prezero_periods": "5", 2024-09-21T15:49:05.149 INFO:teuthology.orchestra.run.smithi089.stdout: "journaler_write_head_interval": "15", 2024-09-21T15:49:05.149 INFO:teuthology.orchestra.run.smithi089.stdout: "key": "", 2024-09-21T15:49:05.149 INFO:teuthology.orchestra.run.smithi089.stdout: "keyfile": "", 2024-09-21T15:49:05.149 INFO:teuthology.orchestra.run.smithi089.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-21T15:49:05.149 INFO:teuthology.orchestra.run.smithi089.stdout: "kstore_backend": "rocksdb", 2024-09-21T15:49:05.149 INFO:teuthology.orchestra.run.smithi089.stdout: "kstore_default_stripe_size": "65536", 2024-09-21T15:49:05.149 INFO:teuthology.orchestra.run.smithi089.stdout: "kstore_fsck_on_mount": "false", 2024-09-21T15:49:05.149 INFO:teuthology.orchestra.run.smithi089.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-21T15:49:05.149 INFO:teuthology.orchestra.run.smithi089.stdout: "kstore_max_bytes": "67108864", 2024-09-21T15:49:05.150 INFO:teuthology.orchestra.run.smithi089.stdout: "kstore_max_ops": "512", 2024-09-21T15:49:05.150 INFO:teuthology.orchestra.run.smithi089.stdout: "kstore_nid_prealloc": "1024", 2024-09-21T15:49:05.150 INFO:teuthology.orchestra.run.smithi089.stdout: "kstore_onode_map_size": "1024", 2024-09-21T15:49:05.150 INFO:teuthology.orchestra.run.smithi089.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-21T15:49:05.150 INFO:teuthology.orchestra.run.smithi089.stdout: "kstore_sync_submit_transaction": "false", 2024-09-21T15:49:05.150 INFO:teuthology.orchestra.run.smithi089.stdout: "kstore_sync_transaction": "false", 2024-09-21T15:49:05.150 INFO:teuthology.orchestra.run.smithi089.stdout: "leveldb_block_size": "65536", 2024-09-21T15:49:05.150 INFO:teuthology.orchestra.run.smithi089.stdout: "leveldb_bloom_size": "0", 2024-09-21T15:49:05.150 INFO:teuthology.orchestra.run.smithi089.stdout: "leveldb_cache_size": "536870912", 2024-09-21T15:49:05.150 INFO:teuthology.orchestra.run.smithi089.stdout: "leveldb_compact_on_mount": "false", 2024-09-21T15:49:05.150 INFO:teuthology.orchestra.run.smithi089.stdout: "leveldb_compression": "false", 2024-09-21T15:49:05.150 INFO:teuthology.orchestra.run.smithi089.stdout: "leveldb_log": "", 2024-09-21T15:49:05.150 INFO:teuthology.orchestra.run.smithi089.stdout: "leveldb_log_to_ceph_log": "true", 2024-09-21T15:49:05.150 INFO:teuthology.orchestra.run.smithi089.stdout: "leveldb_max_open_files": "0", 2024-09-21T15:49:05.151 INFO:teuthology.orchestra.run.smithi089.stdout: "leveldb_paranoid": "false", 2024-09-21T15:49:05.151 INFO:teuthology.orchestra.run.smithi089.stdout: "leveldb_write_buffer_size": "33554432", 2024-09-21T15:49:05.151 INFO:teuthology.orchestra.run.smithi089.stdout: "librados_thread_count": "2", 2024-09-21T15:49:05.151 INFO:teuthology.orchestra.run.smithi089.stdout: "lockdep": "false", 2024-09-21T15:49:05.151 INFO:teuthology.orchestra.run.smithi089.stdout: "lockdep_force_backtrace": "false", 2024-09-21T15:49:05.151 INFO:teuthology.orchestra.run.smithi089.stdout: "log_coarse_timestamps": "true", 2024-09-21T15:49:05.151 INFO:teuthology.orchestra.run.smithi089.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-21T15:49:05.151 INFO:teuthology.orchestra.run.smithi089.stdout: "log_flush_on_exit": "false", 2024-09-21T15:49:05.151 INFO:teuthology.orchestra.run.smithi089.stdout: "log_graylog_host": "127.0.0.1", 2024-09-21T15:49:05.151 INFO:teuthology.orchestra.run.smithi089.stdout: "log_graylog_port": "12201", 2024-09-21T15:49:05.151 INFO:teuthology.orchestra.run.smithi089.stdout: "log_max_new": "1000", 2024-09-21T15:49:05.151 INFO:teuthology.orchestra.run.smithi089.stdout: "log_max_recent": "10000", 2024-09-21T15:49:05.151 INFO:teuthology.orchestra.run.smithi089.stdout: "log_stderr_prefix": "", 2024-09-21T15:49:05.151 INFO:teuthology.orchestra.run.smithi089.stdout: "log_stop_at_utilization": "0.970000", 2024-09-21T15:49:05.151 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_file": "true", 2024-09-21T15:49:05.152 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_graylog": "false", 2024-09-21T15:49:05.152 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_journald": "false", 2024-09-21T15:49:05.152 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_stderr": "false", 2024-09-21T15:49:05.152 INFO:teuthology.orchestra.run.smithi089.stdout: "log_to_syslog": "false", 2024-09-21T15:49:05.152 INFO:teuthology.orchestra.run.smithi089.stdout: "max_rotating_auth_attempts": "10", 2024-09-21T15:49:05.152 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-21T15:49:05.152 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_action_on_write_error": "1", 2024-09-21T15:49:05.152 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_alternate_name_max": "8192", 2024-09-21T15:49:05.152 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_asio_thread_count": "2", 2024-09-21T15:49:05.152 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_bal_export_pin": "true", 2024-09-21T15:49:05.152 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_bal_fragment_dirs": "true", 2024-09-21T15:49:05.152 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-21T15:49:05.152 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_bal_fragment_interval": "5", 2024-09-21T15:49:05.152 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-21T15:49:05.152 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-21T15:49:05.153 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_bal_interval": "10", 2024-09-21T15:49:05.153 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_bal_max": "-1", 2024-09-21T15:49:05.153 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_bal_max_until": "-1", 2024-09-21T15:49:05.153 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_bal_merge_size": "50", 2024-09-21T15:49:05.153 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_bal_midchunk": "0.300000", 2024-09-21T15:49:05.153 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-21T15:49:05.153 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_bal_min_start": "0.200000", 2024-09-21T15:49:05.153 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_bal_minchunk": "0.001000", 2024-09-21T15:49:05.153 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_bal_mode": "0", 2024-09-21T15:49:05.153 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_bal_need_max": "1.200000", 2024-09-21T15:49:05.153 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_bal_need_min": "0.800000", 2024-09-21T15:49:05.153 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-21T15:49:05.153 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-21T15:49:05.153 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_bal_split_bits": "3", 2024-09-21T15:49:05.153 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-21T15:49:05.154 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_bal_split_size": "10000", 2024-09-21T15:49:05.154 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-21T15:49:05.154 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_bal_target_decay": "10.000000", 2024-09-21T15:49:05.154 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-21T15:49:05.154 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_beacon_grace": "15.000000", 2024-09-21T15:49:05.154 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_beacon_interval": "4.000000", 2024-09-21T15:49:05.154 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-21T15:49:05.154 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-21T15:49:05.154 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_cache_mid": "0.700000", 2024-09-21T15:49:05.154 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_cache_release_free_interval": "10", 2024-09-21T15:49:05.154 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_cache_reservation": "0.050000", 2024-09-21T15:49:05.154 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-21T15:49:05.154 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_cache_trim_interval": "1", 2024-09-21T15:49:05.154 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_cache_trim_threshold": "262144", 2024-09-21T15:49:05.155 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-21T15:49:05.155 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-21T15:49:05.155 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-21T15:49:05.155 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_client_prealloc_inos": "1000", 2024-09-21T15:49:05.155 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-21T15:49:05.155 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_connect_bootstrapping": "false", 2024-09-21T15:49:05.155 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_damage_table_max_entries": "10000", 2024-09-21T15:49:05.155 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-21T15:49:05.155 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_debug_auth_pins": "false", 2024-09-21T15:49:05.155 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_debug_frag": "false", 2024-09-21T15:49:05.155 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_debug_scatterstat": "false", 2024-09-21T15:49:05.155 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_debug_subtrees": "false", 2024-09-21T15:49:05.155 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_decay_halflife": "5.000000", 2024-09-21T15:49:05.155 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_default_dir_hash": "2", 2024-09-21T15:49:05.155 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_defer_session_stale": "true", 2024-09-21T15:49:05.156 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_deny_all_reconnect": "false", 2024-09-21T15:49:05.156 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_dir_keys_per_op": "16384", 2024-09-21T15:49:05.156 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_dir_max_commit_size": "10", 2024-09-21T15:49:05.156 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_dir_max_entries": "0", 2024-09-21T15:49:05.156 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_dir_prefetch": "true", 2024-09-21T15:49:05.156 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-21T15:49:05.156 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-21T15:49:05.156 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_dump_cache_on_map": "false", 2024-09-21T15:49:05.156 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-21T15:49:05.156 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-21T15:49:05.156 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_early_reply": "true", 2024-09-21T15:49:05.156 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_enable_op_tracker": "true", 2024-09-21T15:49:05.156 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_enforce_unique_name": "true", 2024-09-21T15:49:05.156 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-21T15:49:05.157 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-21T15:49:05.157 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_export_ephemeral_random": "true", 2024-09-21T15:49:05.157 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-21T15:49:05.157 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-09-21T15:49:05.157 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-21T15:49:05.157 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-21T15:49:05.157 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-09-21T15:49:05.157 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-21T15:49:05.157 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-21T15:49:05.157 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-21T15:49:05.157 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_health_summarize_threshold": "10", 2024-09-21T15:49:05.157 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-21T15:49:05.157 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-21T15:49:05.157 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_inject_health_dummy": "false", 2024-09-21T15:49:05.158 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-21T15:49:05.158 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_inject_migrator_session_race": "false", 2024-09-21T15:49:05.158 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-21T15:49:05.158 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-21T15:49:05.158 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-21T15:49:05.158 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_join_fs": "", 2024-09-21T15:49:05.158 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_journal_format": "1", 2024-09-21T15:49:05.158 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-09-21T15:49:05.158 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_kill_create_at": "0", 2024-09-21T15:49:05.158 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_kill_export_at": "0", 2024-09-21T15:49:05.158 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_kill_import_at": "0", 2024-09-21T15:49:05.158 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_kill_journal_at": "0", 2024-09-21T15:49:05.158 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_kill_journal_expire_at": "0", 2024-09-21T15:49:05.159 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_kill_journal_replay_at": "0", 2024-09-21T15:49:05.159 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_kill_link_at": "0", 2024-09-21T15:49:05.159 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_kill_mdstable_at": "0", 2024-09-21T15:49:05.159 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_kill_openc_at": "0", 2024-09-21T15:49:05.159 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_kill_rename_at": "0", 2024-09-21T15:49:05.159 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_log_events_per_segment": "1024", 2024-09-21T15:49:05.159 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_log_max_events": "-1", 2024-09-21T15:49:05.159 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_log_max_segments": "128", 2024-09-21T15:49:05.159 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_log_pause": "false", 2024-09-21T15:49:05.159 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_log_segment_size": "0", 2024-09-21T15:49:05.159 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-21T15:49:05.159 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_log_warn_factor": "2.000000", 2024-09-21T15:49:05.159 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_max_caps_per_client": "1048576", 2024-09-21T15:49:05.160 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_max_completed_flushes": "100000", 2024-09-21T15:49:05.160 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_max_completed_requests": "100000", 2024-09-21T15:49:05.160 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_max_export_size": "20971520", 2024-09-21T15:49:05.160 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_max_file_recover": "32", 2024-09-21T15:49:05.160 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_max_purge_files": "64", 2024-09-21T15:49:05.160 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_max_purge_ops": "8192", 2024-09-21T15:49:05.160 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-21T15:49:05.160 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-21T15:49:05.160 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_max_snaps_per_dir": "100", 2024-09-21T15:49:05.160 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_metrics_update_interval": "2", 2024-09-21T15:49:05.160 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_min_caps_per_client": "100", 2024-09-21T15:49:05.160 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_min_caps_working_set": "10000", 2024-09-21T15:49:05.160 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-21T15:49:05.160 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_numa_node": "-1", 2024-09-21T15:49:05.161 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-21T15:49:05.161 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_op_complaint_time": "30.000000", 2024-09-21T15:49:05.161 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_op_history_duration": "600", 2024-09-21T15:49:05.161 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_op_history_size": "20", 2024-09-21T15:49:05.161 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_op_history_slow_op_size": "20", 2024-09-21T15:49:05.161 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-21T15:49:05.161 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_op_log_threshold": "5", 2024-09-21T15:49:05.161 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_ping_grace": "15", 2024-09-21T15:49:05.161 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_ping_interval": "5", 2024-09-21T15:49:05.161 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-21T15:49:05.161 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-21T15:49:05.161 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_recall_max_caps": "30000", 2024-09-21T15:49:05.161 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-21T15:49:05.162 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-21T15:49:05.162 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-21T15:49:05.162 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_recall_warning_threshold": "262144", 2024-09-21T15:49:05.162 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-21T15:49:05.162 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_replay_interval": "1.000000", 2024-09-21T15:49:05.162 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-21T15:49:05.162 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-21T15:49:05.162 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_root_ino_gid": "0", 2024-09-21T15:49:05.162 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_root_ino_uid": "0", 2024-09-21T15:49:05.162 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-21T15:49:05.162 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-21T15:49:05.162 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-21T15:49:05.163 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-21T15:49:05.163 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-21T15:49:05.163 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-21T15:49:05.163 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-21T15:49:05.163 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-21T15:49:05.163 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-21T15:49:05.163 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-21T15:49:05.163 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_shutdown_check": "0", 2024-09-21T15:49:05.163 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_skip_ino": "0", 2024-09-21T15:49:05.163 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-21T15:49:05.163 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_snap_max_uid": "4294967294", 2024-09-21T15:49:05.163 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_snap_min_uid": "0", 2024-09-21T15:49:05.163 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_snap_rstat": "false", 2024-09-21T15:49:05.164 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_standby_replay_damaged": "false", 2024-09-21T15:49:05.164 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_symlink_recovery": "true", 2024-09-21T15:49:05.164 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-21T15:49:05.164 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_thrash_exports": "0", 2024-09-21T15:49:05.164 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_thrash_fragments": "0", 2024-09-21T15:49:05.164 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_tick_interval": "5.000000", 2024-09-21T15:49:05.164 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_valgrind_exit": "false", 2024-09-21T15:49:05.164 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_verify_backtrace": "1", 2024-09-21T15:49:05.164 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_verify_scatter": "false", 2024-09-21T15:49:05.164 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-21T15:49:05.164 INFO:teuthology.orchestra.run.smithi089.stdout: "mds_wipe_sessions": "false", 2024-09-21T15:49:05.164 INFO:teuthology.orchestra.run.smithi089.stdout: "mempool_debug": "false", 2024-09-21T15:49:05.164 INFO:teuthology.orchestra.run.smithi089.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-21T15:49:05.164 INFO:teuthology.orchestra.run.smithi089.stdout: "memstore_device_bytes": "1073741824", 2024-09-21T15:49:05.165 INFO:teuthology.orchestra.run.smithi089.stdout: "memstore_page_set": "false", 2024-09-21T15:49:05.165 INFO:teuthology.orchestra.run.smithi089.stdout: "memstore_page_size": "65536", 2024-09-21T15:49:05.165 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr_client_bytes": "134217728", 2024-09-21T15:49:05.165 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr_client_messages": "512", 2024-09-21T15:49:05.165 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-21T15:49:05.165 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-21T15:49:05.165 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-21T15:49:05.165 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-21T15:49:05.165 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr_disabled_modules": "", 2024-09-21T15:49:05.165 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-21T15:49:05.165 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr_max_pg_num_change": "128", 2024-09-21T15:49:05.165 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr_mds_bytes": "134217728", 2024-09-21T15:49:05.165 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr_mds_messages": "128", 2024-09-21T15:49:05.165 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-21T15:49:05.166 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr_mon_bytes": "134217728", 2024-09-21T15:49:05.166 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr_mon_messages": "128", 2024-09-21T15:49:05.166 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr_osd_bytes": "536870912", 2024-09-21T15:49:05.166 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr_osd_messages": "8192", 2024-09-21T15:49:05.166 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr_pool": "true", 2024-09-21T15:49:05.166 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-21T15:49:05.166 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr_standby_modules": "true", 2024-09-21T15:49:05.166 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr_stats_period": "5", 2024-09-21T15:49:05.166 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr_stats_threshold": "5", 2024-09-21T15:49:05.166 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr_tick_period": "2", 2024-09-21T15:49:05.166 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-21T15:49:05.167 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-21T15:49:05.167 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_allow_pool_delete": "true", 2024-09-21T15:49:05.167 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_allow_pool_size_one": "true", 2024-09-21T15:49:05.167 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_auth_validate_all_caps": "true", 2024-09-21T15:49:05.167 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-21T15:49:05.167 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-21T15:49:05.167 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_client_bytes": "104857600", 2024-09-21T15:49:05.167 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_client_directed_command_retry": "5", 2024-09-21T15:49:05.167 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-21T15:49:05.167 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-21T15:49:05.168 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-21T15:49:05.168 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-21T15:49:05.168 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_client_hunt_parallel": "3", 2024-09-21T15:49:05.168 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_client_log_interval": "1.000000", 2024-09-21T15:49:05.168 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-21T15:49:05.168 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_client_ping_interval": "10.000000", 2024-09-21T15:49:05.168 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-21T15:49:05.168 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-21T15:49:05.168 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-21T15:49:05.168 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-21T15:49:05.168 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_cluster_log_file_level": "debug", 2024-09-21T15:49:05.168 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_cluster_log_to_file": "true", 2024-09-21T15:49:05.168 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-21T15:49:05.168 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-21T15:49:05.169 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-21T15:49:05.169 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_cluster_log_to_journald": "false", 2024-09-21T15:49:05.169 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-21T15:49:05.169 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-21T15:49:05.169 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-21T15:49:05.169 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-09-21T15:49:05.169 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_compact_on_bootstrap": "false", 2024-09-21T15:49:05.169 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_compact_on_start": "false", 2024-09-21T15:49:05.169 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_compact_on_trim": "true", 2024-09-21T15:49:05.169 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-21T15:49:05.169 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-21T15:49:05.169 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-21T15:49:05.169 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_cpu_threads": "4", 2024-09-21T15:49:05.169 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_crush_min_required_version": "hammer", 2024-09-21T15:49:05.170 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_daemon_bytes": "419430400", 2024-09-21T15:49:05.170 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-21T15:49:05.170 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_data_avail_crit": "5", 2024-09-21T15:49:05.170 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_data_avail_warn": "5", 2024-09-21T15:49:05.170 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_data_size_warn": "16106127360", 2024-09-21T15:49:05.170 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-21T15:49:05.170 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-21T15:49:05.170 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_debug_dump_json": "false", 2024-09-21T15:49:05.170 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-21T15:49:05.170 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_debug_dump_transactions": "false", 2024-09-21T15:49:05.170 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_debug_extra_checks": "false", 2024-09-21T15:49:05.170 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-21T15:49:05.170 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-21T15:49:05.170 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_debug_no_require_quincy": "false", 2024-09-21T15:49:05.171 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_debug_no_require_reef": "false", 2024-09-21T15:49:05.171 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-21T15:49:05.171 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-21T15:49:05.171 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-21T15:49:05.171 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_down_mkfs_grace": "120", 2024-09-21T15:49:05.171 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_election_default_strategy": "1", 2024-09-21T15:49:05.171 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_election_timeout": "5.000000", 2024-09-21T15:49:05.171 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-21T15:49:05.171 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_elector_ping_divisor": "2", 2024-09-21T15:49:05.171 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-21T15:49:05.171 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_enable_op_tracker": "true", 2024-09-21T15:49:05.171 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_fake_pool_delete": "false", 2024-09-21T15:49:05.171 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_force_quorum_join": "false", 2024-09-21T15:49:05.172 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_globalid_prealloc": "10000", 2024-09-21T15:49:05.172 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_health_detail_to_clog": "false", 2024-09-21T15:49:05.172 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_health_log_update_period": "5", 2024-09-21T15:49:05.172 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_health_max_detail": "50", 2024-09-21T15:49:05.172 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_health_to_clog": "true", 2024-09-21T15:49:05.172 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_health_to_clog_interval": "600", 2024-09-21T15:49:05.172 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-21T15:49:05.172 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_host": "172.21.15.89,[v2:172.21.15.89:3301,v1:172.21.15.89:6790],172.21.15.100", 2024-09-21T15:49:05.172 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_host_override": "", 2024-09-21T15:49:05.172 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_initial_members": "", 2024-09-21T15:49:05.172 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-21T15:49:05.172 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-21T15:49:05.172 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-21T15:49:05.172 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-21T15:49:05.172 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-21T15:49:05.173 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_lease": "5.000000", 2024-09-21T15:49:05.173 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-21T15:49:05.173 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-21T15:49:05.173 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_log_full_interval": "50", 2024-09-21T15:49:05.173 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_log_max": "10000", 2024-09-21T15:49:05.173 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_log_max_summary": "50", 2024-09-21T15:49:05.173 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-21T15:49:05.173 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_max_log_epochs": "500", 2024-09-21T15:49:05.173 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-21T15:49:05.173 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-21T15:49:05.173 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_max_osd": "10000", 2024-09-21T15:49:05.173 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_max_pg_per_osd": "10000", 2024-09-21T15:49:05.173 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_max_pool_pg_num": "65536", 2024-09-21T15:49:05.173 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-21T15:49:05.174 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-21T15:49:05.174 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_mds_force_trim_to": "0", 2024-09-21T15:49:05.174 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_mds_skip_sanity": "false", 2024-09-21T15:49:05.174 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_memory_autotune": "true", 2024-09-21T15:49:05.174 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_memory_target": "2147483648", 2024-09-21T15:49:05.174 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_mgr_beacon_grace": "30", 2024-09-21T15:49:05.174 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-21T15:49:05.174 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_mgr_digest_period": "5", 2024-09-21T15:49:05.174 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_mgr_inactive_grace": "60", 2024-09-21T15:49:05.174 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-21T15:49:05.174 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-21T15:49:05.174 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_min_osdmap_epochs": "500", 2024-09-21T15:49:05.174 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_op_complaint_time": "30", 2024-09-21T15:49:05.174 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_op_history_duration": "600", 2024-09-21T15:49:05.175 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_op_history_size": "20", 2024-09-21T15:49:05.175 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_op_history_slow_op_size": "20", 2024-09-21T15:49:05.175 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-21T15:49:05.175 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_op_log_threshold": "5", 2024-09-21T15:49:05.175 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-21T15:49:05.175 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-21T15:49:05.175 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-21T15:49:05.175 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_auto_mark_in": "false", 2024-09-21T15:49:05.175 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-21T15:49:05.175 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-21T15:49:05.175 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-21T15:49:05.175 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_cache_size": "500", 2024-09-21T15:49:05.175 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-21T15:49:05.175 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-21T15:49:05.175 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-21T15:49:05.176 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_down_out_interval": "600", 2024-09-21T15:49:05.176 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-21T15:49:05.176 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-21T15:49:05.176 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_force_trim_to": "0", 2024-09-21T15:49:05.176 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-21T15:49:05.176 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-21T15:49:05.176 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-21T15:49:05.176 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-21T15:49:05.176 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-21T15:49:05.176 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-21T15:49:05.176 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_max_creating_pgs": "1024", 2024-09-21T15:49:05.176 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-21T15:49:05.176 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_min_down_reporters": "2", 2024-09-21T15:49:05.176 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-21T15:49:05.177 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-21T15:49:05.177 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-21T15:49:05.177 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-21T15:49:05.177 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-21T15:49:05.177 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-21T15:49:05.177 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_report_timeout": "900", 2024-09-21T15:49:05.177 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-21T15:49:05.177 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-21T15:49:05.177 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-21T15:49:05.177 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-21T15:49:05.177 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-21T15:49:05.177 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osdmap_full_prune_interval": "10", 2024-09-21T15:49:05.177 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osdmap_full_prune_min": "10000", 2024-09-21T15:49:05.177 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_osdmap_full_prune_txsize": "100", 2024-09-21T15:49:05.177 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-21T15:49:05.177 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_pg_stuck_threshold": "60", 2024-09-21T15:49:05.178 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-21T15:49:05.178 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-21T15:49:05.178 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-21T15:49:05.178 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-21T15:49:05.178 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-21T15:49:05.178 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-21T15:49:05.178 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_probe_timeout": "2.000000", 2024-09-21T15:49:05.178 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_reweight_max_change": "0.050000", 2024-09-21T15:49:05.178 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_reweight_max_osds": "4", 2024-09-21T15:49:05.178 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-21T15:49:05.178 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-21T15:49:05.178 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-21T15:49:05.178 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-21T15:49:05.178 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-21T15:49:05.179 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_scrub_interval": "86400", 2024-09-21T15:49:05.179 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_scrub_max_keys": "100", 2024-09-21T15:49:05.179 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_scrub_timeout": "300", 2024-09-21T15:49:05.179 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_session_timeout": "300", 2024-09-21T15:49:05.179 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_smart_report_timeout": "5", 2024-09-21T15:49:05.179 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_stat_smooth_intervals": "6", 2024-09-21T15:49:05.179 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-21T15:49:05.179 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_stretch_pool_min_size": "2", 2024-09-21T15:49:05.179 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_stretch_pool_size": "4", 2024-09-21T15:49:05.179 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-21T15:49:05.179 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-21T15:49:05.179 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_sync_debug": "false", 2024-09-21T15:49:05.179 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-21T15:49:05.179 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-21T15:49:05.179 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_sync_provider_kill_at": "0", 2024-09-21T15:49:05.180 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_sync_requester_kill_at": "0", 2024-09-21T15:49:05.180 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_sync_timeout": "60.000000", 2024-09-21T15:49:05.180 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_target_pg_per_osd": "100", 2024-09-21T15:49:05.180 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_tick_interval": "5", 2024-09-21T15:49:05.180 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_timecheck_interval": "300.000000", 2024-09-21T15:49:05.180 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-21T15:49:05.180 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_use_min_delay_socket": "false", 2024-09-21T15:49:05.180 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_warn_older_version_delay": "604800", 2024-09-21T15:49:05.180 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-21T15:49:05.180 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-21T15:49:05.180 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-21T15:49:05.180 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_warn_on_filestore_osds": "false", 2024-09-21T15:49:05.180 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-21T15:49:05.180 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-21T15:49:05.181 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-21T15:49:05.181 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_warn_on_misplaced": "false", 2024-09-21T15:49:05.181 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-21T15:49:05.181 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_warn_on_older_version": "true", 2024-09-21T15:49:05.181 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-21T15:49:05.181 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_warn_on_pool_no_app": "false", 2024-09-21T15:49:05.181 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_warn_on_pool_no_redundancy": "false", 2024-09-21T15:49:05.181 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-21T15:49:05.181 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-21T15:49:05.181 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-21T15:49:05.181 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-21T15:49:05.181 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-21T15:49:05.181 INFO:teuthology.orchestra.run.smithi089.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-21T15:49:05.182 INFO:teuthology.orchestra.run.smithi089.stdout: "monmap": "", 2024-09-21T15:49:05.182 INFO:teuthology.orchestra.run.smithi089.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-21T15:49:05.182 INFO:teuthology.orchestra.run.smithi089.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-09-21T15:49:05.182 INFO:teuthology.orchestra.run.smithi089.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-09-21T15:49:05.182 INFO:teuthology.orchestra.run.smithi089.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-21T15:49:05.182 INFO:teuthology.orchestra.run.smithi089.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-09-21T15:49:05.182 INFO:teuthology.orchestra.run.smithi089.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-09-21T15:49:05.183 INFO:teuthology.orchestra.run.smithi089.stdout: "motr_tracing_enabled": "false", 2024-09-21T15:49:05.183 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_async_op_threads": "3", 2024-09-21T15:49:05.183 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-21T15:49:05.183 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_async_rdma_cm": "false", 2024-09-21T15:49:05.183 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_async_rdma_device_name": "", 2024-09-21T15:49:05.183 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_async_rdma_dscp": "96", 2024-09-21T15:49:05.183 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-21T15:49:05.183 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-21T15:49:05.184 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_async_rdma_local_gid": "", 2024-09-21T15:49:05.184 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-21T15:49:05.184 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_async_rdma_port_num": "1", 2024-09-21T15:49:05.184 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-21T15:49:05.184 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-21T15:49:05.184 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-21T15:49:05.184 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-21T15:49:05.184 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_async_rdma_sl": "3", 2024-09-21T15:49:05.184 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_async_rdma_support_srq": "true", 2024-09-21T15:49:05.185 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_async_rdma_type": "ib", 2024-09-21T15:49:05.185 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_async_reap_threshold": "5", 2024-09-21T15:49:05.185 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_bind_before_connect": "false", 2024-09-21T15:49:05.185 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_bind_ipv4": "true", 2024-09-21T15:49:05.185 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_bind_ipv6": "false", 2024-09-21T15:49:05.185 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_bind_msgr1": "true", 2024-09-21T15:49:05.185 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_bind_msgr2": "true", 2024-09-21T15:49:05.185 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_bind_port_max": "7568", 2024-09-21T15:49:05.185 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_bind_port_min": "6800", 2024-09-21T15:49:05.186 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-21T15:49:05.186 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_bind_retry_count": "3", 2024-09-21T15:49:05.186 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_bind_retry_delay": "5", 2024-09-21T15:49:05.186 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_blackhole_client": "false", 2024-09-21T15:49:05.186 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_blackhole_mds": "false", 2024-09-21T15:49:05.186 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_blackhole_mgr": "false", 2024-09-21T15:49:05.187 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_blackhole_mon": "false", 2024-09-21T15:49:05.187 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_blackhole_osd": "false", 2024-09-21T15:49:05.187 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_client_mode": "crc secure", 2024-09-21T15:49:05.187 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_cluster_mode": "crc secure", 2024-09-21T15:49:05.187 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_cluster_type": "", 2024-09-21T15:49:05.187 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_compress_secure": "false", 2024-09-21T15:49:05.187 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_connection_idle_timeout": "900", 2024-09-21T15:49:05.188 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_connection_ready_timeout": "10", 2024-09-21T15:49:05.188 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_crc_data": "true", 2024-09-21T15:49:05.188 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_crc_header": "true", 2024-09-21T15:49:05.188 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_die_on_bad_msg": "false", 2024-09-21T15:49:05.188 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_die_on_bug": "true", 2024-09-21T15:49:05.188 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_die_on_old_message": "true", 2024-09-21T15:49:05.188 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_die_on_skipped_message": "false", 2024-09-21T15:49:05.189 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-21T15:49:05.189 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-21T15:49:05.189 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_dpdk_coremask": "0xF", 2024-09-21T15:49:05.189 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-21T15:49:05.189 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-21T15:49:05.189 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_dpdk_enable_tso": "true", 2024-09-21T15:49:05.189 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-21T15:49:05.189 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-21T15:49:05.189 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_dpdk_hugepages": "", 2024-09-21T15:49:05.189 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-21T15:49:05.189 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-21T15:49:05.189 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_dpdk_lro": "true", 2024-09-21T15:49:05.189 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_dpdk_memory_channel": "4", 2024-09-21T15:49:05.190 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-21T15:49:05.190 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_dpdk_pmd": "", 2024-09-21T15:49:05.190 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_dpdk_port_id": "0", 2024-09-21T15:49:05.190 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-21T15:49:05.190 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-21T15:49:05.190 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_dump_on_send": "false", 2024-09-21T15:49:05.190 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_initial_backoff": "0.200000", 2024-09-21T15:49:05.190 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_inject_delay_max": "1.000000", 2024-09-21T15:49:05.190 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_inject_delay_probability": "0.000000", 2024-09-21T15:49:05.190 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_inject_delay_type": "", 2024-09-21T15:49:05.190 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_inject_internal_delays": "0.000000", 2024-09-21T15:49:05.190 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_inject_network_congestion": "0", 2024-09-21T15:49:05.190 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_inject_socket_failures": "5000", 2024-09-21T15:49:05.191 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_learn_addr_from_peer": "true", 2024-09-21T15:49:05.191 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_max_accept_failures": "4", 2024-09-21T15:49:05.191 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_max_backoff": "15.000000", 2024-09-21T15:49:05.191 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_mon_client_mode": "secure crc", 2024-09-21T15:49:05.191 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-21T15:49:05.191 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_mon_service_mode": "secure crc", 2024-09-21T15:49:05.191 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_osd_compress_min_size": "1024", 2024-09-21T15:49:05.191 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_osd_compress_mode": "none", 2024-09-21T15:49:05.191 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_osd_compression_algorithm": "snappy", 2024-09-21T15:49:05.191 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-21T15:49:05.191 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_pq_min_cost": "65536", 2024-09-21T15:49:05.191 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_public_type": "", 2024-09-21T15:49:05.191 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_service_mode": "crc secure", 2024-09-21T15:49:05.191 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_tcp_listen_backlog": "512", 2024-09-21T15:49:05.191 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_tcp_nodelay": "true", 2024-09-21T15:49:05.192 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-21T15:49:05.192 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_tcp_rcvbuf": "0", 2024-09-21T15:49:05.192 INFO:teuthology.orchestra.run.smithi089.stdout: "ms_type": "async+posix", 2024-09-21T15:49:05.192 INFO:teuthology.orchestra.run.smithi089.stdout: "no_config_file": "false", 2024-09-21T15:49:05.192 INFO:teuthology.orchestra.run.smithi089.stdout: "objecter_completion_locks_per_session": "32", 2024-09-21T15:49:05.192 INFO:teuthology.orchestra.run.smithi089.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-21T15:49:05.192 INFO:teuthology.orchestra.run.smithi089.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-21T15:49:05.192 INFO:teuthology.orchestra.run.smithi089.stdout: "objecter_inflight_ops": "1024", 2024-09-21T15:49:05.192 INFO:teuthology.orchestra.run.smithi089.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-21T15:49:05.192 INFO:teuthology.orchestra.run.smithi089.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-21T15:49:05.192 INFO:teuthology.orchestra.run.smithi089.stdout: "objecter_tick_interval": "5.000000", 2024-09-21T15:49:05.192 INFO:teuthology.orchestra.run.smithi089.stdout: "objecter_timeout": "10.000000", 2024-09-21T15:49:05.192 INFO:teuthology.orchestra.run.smithi089.stdout: "objectstore_blackhole": "false", 2024-09-21T15:49:05.192 INFO:teuthology.orchestra.run.smithi089.stdout: "openssl_engine_opts": "", 2024-09-21T15:49:05.192 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_agent_delay_time": "5.000000", 2024-09-21T15:49:05.193 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_agent_hist_halflife": "1000", 2024-09-21T15:49:05.193 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_agent_max_low_ops": "2", 2024-09-21T15:49:05.193 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_agent_max_ops": "4", 2024-09-21T15:49:05.193 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-21T15:49:05.193 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-21T15:49:05.193 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_agent_slop": "0.020000", 2024-09-21T15:49:05.193 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-21T15:49:05.193 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-21T15:49:05.193 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_asio_thread_count": "2", 2024-09-21T15:49:05.193 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_async_recovery_min_cost": "100", 2024-09-21T15:49:05.193 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-21T15:49:05.193 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-21T15:49:05.193 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_backfill_scan_max": "512", 2024-09-21T15:49:05.193 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_backfill_scan_min": "64", 2024-09-21T15:49:05.193 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_backoff_on_degraded": "false", 2024-09-21T15:49:05.193 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_backoff_on_peering": "false", 2024-09-21T15:49:05.194 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_backoff_on_unfound": "true", 2024-09-21T15:49:05.194 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_beacon_report_interval": "300", 2024-09-21T15:49:05.194 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_bench_duration": "30", 2024-09-21T15:49:05.194 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-21T15:49:05.194 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_bench_max_block_size": "67108864", 2024-09-21T15:49:05.194 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-21T15:49:05.194 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_blkin_trace_all": "false", 2024-09-21T15:49:05.194 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_blocked_scrub_grace_period": "120", 2024-09-21T15:49:05.194 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-21T15:49:05.194 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-09-21T15:49:05.194 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-21T15:49:05.194 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_check_for_log_corruption": "false", 2024-09-21T15:49:05.194 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-21T15:49:05.194 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-21T15:49:05.195 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-09-21T15:49:05.195 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-21T15:49:05.195 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_class_update_on_start": "true", 2024-09-21T15:49:05.195 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_client_message_cap": "256", 2024-09-21T15:49:05.195 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_client_message_size_cap": "524288000", 2024-09-21T15:49:05.195 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_client_op_priority": "63", 2024-09-21T15:49:05.195 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_client_watch_timeout": "30", 2024-09-21T15:49:05.195 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_command_max_records": "256", 2024-09-21T15:49:05.195 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_compact_on_start": "false", 2024-09-21T15:49:05.195 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-21T15:49:05.195 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_crush_chooseleaf_type": "0", 2024-09-21T15:49:05.195 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-21T15:49:05.195 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_crush_update_on_start": "true", 2024-09-21T15:49:05.195 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_crush_update_weight_set": "true", 2024-09-21T15:49:05.195 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-21T15:49:05.195 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-21T15:49:05.196 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-21T15:49:05.196 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-21T15:49:05.196 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-21T15:49:05.196 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_debug_feed_pullee": "-1", 2024-09-21T15:49:05.196 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-21T15:49:05.196 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-21T15:49:05.196 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-21T15:49:05.196 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_debug_misdirected_ops": "false", 2024-09-21T15:49:05.196 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_debug_no_acting_change": "false", 2024-09-21T15:49:05.196 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_debug_no_purge_strays": "false", 2024-09-21T15:49:05.196 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_debug_op_order": "false", 2024-09-21T15:49:05.196 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-21T15:49:05.196 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-21T15:49:05.196 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-21T15:49:05.196 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-21T15:49:05.196 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_debug_shutdown": "false", 2024-09-21T15:49:05.196 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-21T15:49:05.197 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-21T15:49:05.197 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-21T15:49:05.197 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-21T15:49:05.197 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_debug_verify_snaps": "false", 2024-09-21T15:49:05.197 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-21T15:49:05.197 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-21T15:49:05.197 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_deep_scrub_keys": "1024", 2024-09-21T15:49:05.197 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-21T15:49:05.197 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-21T15:49:05.197 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-21T15:49:05.197 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_deep_scrub_stride": "524288", 2024-09-21T15:49:05.197 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-21T15:49:05.197 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_default_data_pool_replay_window": "5", 2024-09-21T15:49:05.197 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_default_notify_timeout": "30", 2024-09-21T15:49:05.197 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_delete_sleep": "0.000000", 2024-09-21T15:49:05.197 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-21T15:49:05.198 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-21T15:49:05.198 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-21T15:49:05.198 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_discard_disconnected_ops": "true", 2024-09-21T15:49:05.198 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_enable_op_tracker": "true", 2024-09-21T15:49:05.198 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-21T15:49:05.198 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_extblkdev_plugins": "vdo", 2024-09-21T15:49:05.198 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-21T15:49:05.198 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-21T15:49:05.198 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_fast_info": "true", 2024-09-21T15:49:05.198 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_fast_shutdown": "true", 2024-09-21T15:49:05.198 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-21T15:49:05.198 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-21T15:49:05.198 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-21T15:49:05.198 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-21T15:49:05.199 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-21T15:49:05.199 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_function_tracing": "false", 2024-09-21T15:49:05.199 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_heartbeat_grace": "20", 2024-09-21T15:49:05.199 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_heartbeat_interval": "6", 2024-09-21T15:49:05.199 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-21T15:49:05.199 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_heartbeat_min_peers": "10", 2024-09-21T15:49:05.199 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_heartbeat_min_size": "2000", 2024-09-21T15:49:05.199 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_heartbeat_stale": "600", 2024-09-21T15:49:05.199 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-21T15:49:05.199 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_hit_set_max_size": "100000", 2024-09-21T15:49:05.199 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_hit_set_min_size": "1000", 2024-09-21T15:49:05.199 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-21T15:49:05.199 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-21T15:49:05.199 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-21T15:49:05.200 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-21T15:49:05.200 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-21T15:49:05.200 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-21T15:49:05.200 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_journal_size": "5120", 2024-09-21T15:49:05.200 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_kill_backfill_at": "0", 2024-09-21T15:49:05.200 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-21T15:49:05.200 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_map_cache_size": "50", 2024-09-21T15:49:05.200 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_map_dedup": "true", 2024-09-21T15:49:05.200 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_map_message_max": "40", 2024-09-21T15:49:05.200 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-21T15:49:05.200 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_map_share_max_epochs": "40", 2024-09-21T15:49:05.200 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_max_attr_name_len": "100", 2024-09-21T15:49:05.200 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_max_attr_size": "0", 2024-09-21T15:49:05.200 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_max_backfills": "1", 2024-09-21T15:49:05.200 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_max_markdown_count": "5", 2024-09-21T15:49:05.200 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_max_markdown_period": "600", 2024-09-21T15:49:05.201 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_max_object_name_len": "2048", 2024-09-21T15:49:05.201 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_max_object_namespace_len": "256", 2024-09-21T15:49:05.201 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_max_object_size": "134217728", 2024-09-21T15:49:05.201 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-21T15:49:05.201 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-21T15:49:05.201 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_max_pg_blocked_by": "16", 2024-09-21T15:49:05.201 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_max_pg_log_entries": "10000", 2024-09-21T15:49:05.201 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-21T15:49:05.201 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_max_pgls": "1024", 2024-09-21T15:49:05.201 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_max_push_cost": "8388608", 2024-09-21T15:49:05.201 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_max_push_objects": "10", 2024-09-21T15:49:05.201 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_max_scrubs": "3", 2024-09-21T15:49:05.201 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-21T15:49:05.201 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_max_trimming_pgs": "2", 2024-09-21T15:49:05.202 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_max_write_op_reply_len": "64", 2024-09-21T15:49:05.202 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_max_write_size": "90", 2024-09-21T15:49:05.202 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-21T15:49:05.202 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-21T15:49:05.202 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-21T15:49:05.202 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-21T15:49:05.202 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-21T15:49:05.202 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-21T15:49:05.202 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-21T15:49:05.202 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-21T15:49:05.202 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mclock_profile": "balanced", 2024-09-21T15:49:05.202 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-21T15:49:05.202 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-21T15:49:05.202 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-21T15:49:05.203 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-21T15:49:05.203 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-21T15:49:05.203 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-21T15:49:05.203 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-21T15:49:05.203 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-21T15:49:05.203 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-21T15:49:05.203 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-21T15:49:05.203 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-21T15:49:05.203 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_memory_base": "805306368", 2024-09-21T15:49:05.203 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_memory_cache_min": "134217728", 2024-09-21T15:49:05.203 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-21T15:49:05.203 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-21T15:49:05.203 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_memory_target": "4294967296", 2024-09-21T15:49:05.203 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_memory_target_autotune": "false", 2024-09-21T15:49:05.203 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-21T15:49:05.204 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_min_pg_log_entries": "250", 2024-09-21T15:49:05.204 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_min_recovery_priority": "0", 2024-09-21T15:49:05.204 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-21T15:49:05.204 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-21T15:49:05.204 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mon_report_interval": "5", 2024-09-21T15:49:05.204 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-21T15:49:05.204 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-21T15:49:05.204 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_num_cache_shards": "32", 2024-09-21T15:49:05.204 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_num_op_tracker_shard": "32", 2024-09-21T15:49:05.204 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_numa_auto_affinity": "true", 2024-09-21T15:49:05.204 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_numa_node": "-1", 2024-09-21T15:49:05.204 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_numa_prefer_iface": "true", 2024-09-21T15:49:05.205 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-09-21T15:49:05.205 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_objecter_finishers": "1", 2024-09-21T15:49:05.205 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_objectstore": "bluestore", 2024-09-21T15:49:05.205 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_objectstore_fuse": "false", 2024-09-21T15:49:05.205 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_objectstore_tracing": "false", 2024-09-21T15:49:05.205 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_op_complaint_time": "30.000000", 2024-09-21T15:49:05.205 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_op_history_duration": "600", 2024-09-21T15:49:05.205 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_op_history_size": "20", 2024-09-21T15:49:05.205 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_op_history_slow_op_size": "20", 2024-09-21T15:49:05.205 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-21T15:49:05.205 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_op_log_threshold": "5", 2024-09-21T15:49:05.205 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_op_num_shards": "0", 2024-09-21T15:49:05.205 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_op_num_shards_hdd": "5", 2024-09-21T15:49:05.205 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_op_num_shards_ssd": "8", 2024-09-21T15:49:05.205 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-21T15:49:05.206 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-21T15:49:05.206 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-21T15:49:05.206 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-21T15:49:05.206 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_op_pq_min_cost": "65536", 2024-09-21T15:49:05.206 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-21T15:49:05.206 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_op_queue_cut_off": "high", 2024-09-21T15:49:05.206 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-21T15:49:05.206 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_op_thread_timeout": "15", 2024-09-21T15:49:05.206 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_open_classes_on_start": "true", 2024-09-21T15:49:05.206 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_os_flags": "0", 2024-09-21T15:49:05.206 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_peering_op_priority": "255", 2024-09-21T15:49:05.206 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pg_delete_cost": "1048576", 2024-09-21T15:49:05.206 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pg_delete_priority": "5", 2024-09-21T15:49:05.206 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-21T15:49:05.207 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-21T15:49:05.207 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-21T15:49:05.207 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pg_log_trim_max": "10000", 2024-09-21T15:49:05.207 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pg_log_trim_min": "100", 2024-09-21T15:49:05.207 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-21T15:49:05.207 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-21T15:49:05.207 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pg_stat_report_interval_max": "500", 2024-09-21T15:49:05.207 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-21T15:49:05.207 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-21T15:49:05.207 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-21T15:49:05.207 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-21T15:49:05.207 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-21T15:49:05.207 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-21T15:49:05.207 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pool_default_crimson": "false", 2024-09-21T15:49:05.207 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-21T15:49:05.208 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-21T15:49:05.208 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-21T15:49:05.208 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-21T15:49:05.208 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-21T15:49:05.208 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-21T15:49:05.208 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-21T15:49:05.208 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-21T15:49:05.208 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pool_default_flags": "0", 2024-09-21T15:49:05.208 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-21T15:49:05.208 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pool_default_min_size": "0", 2024-09-21T15:49:05.208 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-21T15:49:05.208 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pool_default_pg_num": "32", 2024-09-21T15:49:05.208 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pool_default_pgp_num": "0", 2024-09-21T15:49:05.208 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-21T15:49:05.208 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pool_default_size": "2", 2024-09-21T15:49:05.208 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pool_default_type": "replicated", 2024-09-21T15:49:05.208 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-21T15:49:05.209 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-21T15:49:05.209 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_push_per_object_cost": "1000", 2024-09-21T15:49:05.209 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-21T15:49:05.209 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_recover_clone_overlap": "true", 2024-09-21T15:49:05.209 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-21T15:49:05.209 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_recovery_cost": "20971520", 2024-09-21T15:49:05.209 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-21T15:49:05.209 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_recovery_max_active": "0", 2024-09-21T15:49:05.209 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-21T15:49:05.209 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-21T15:49:05.209 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-21T15:49:05.209 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-21T15:49:05.209 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_recovery_max_single_start": "1", 2024-09-21T15:49:05.209 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_recovery_op_priority": "3", 2024-09-21T15:49:05.209 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-21T15:49:05.210 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_recovery_priority": "5", 2024-09-21T15:49:05.210 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-21T15:49:05.210 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_recovery_sleep": "0.000000", 2024-09-21T15:49:05.210 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-21T15:49:05.210 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-21T15:49:05.210 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-21T15:49:05.210 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_repair_during_recovery": "false", 2024-09-21T15:49:05.210 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_requested_scrub_priority": "120", 2024-09-21T15:49:05.210 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-21T15:49:05.210 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-21T15:49:05.210 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_scrub_auto_repair": "false", 2024-09-21T15:49:05.210 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-21T15:49:05.210 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-21T15:49:05.210 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_scrub_begin_hour": "0", 2024-09-21T15:49:05.211 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_scrub_begin_week_day": "0", 2024-09-21T15:49:05.211 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_scrub_chunk_max": "25", 2024-09-21T15:49:05.211 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_scrub_chunk_min": "5", 2024-09-21T15:49:05.211 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_scrub_cost": "52428800", 2024-09-21T15:49:05.211 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_scrub_during_recovery": "false", 2024-09-21T15:49:05.211 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_scrub_end_hour": "0", 2024-09-21T15:49:05.211 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_scrub_end_week_day": "0", 2024-09-21T15:49:05.211 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_scrub_event_cost": "4096", 2024-09-21T15:49:05.211 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-21T15:49:05.211 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-21T15:49:05.211 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_scrub_invalid_stats": "true", 2024-09-21T15:49:05.211 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-21T15:49:05.211 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-21T15:49:05.211 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_scrub_max_preemptions": "5", 2024-09-21T15:49:05.211 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-21T15:49:05.212 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_scrub_priority": "5", 2024-09-21T15:49:05.212 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_scrub_reservation_timeout": "5000", 2024-09-21T15:49:05.212 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_scrub_sleep": "0.000000", 2024-09-21T15:49:05.212 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_scrub_slow_reservation_response": "2200", 2024-09-21T15:49:05.212 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-09-21T15:49:05.212 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-09-21T15:49:05.212 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-21T15:49:05.212 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_skip_data_digest": "false", 2024-09-21T15:49:05.212 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_smart_report_timeout": "5", 2024-09-21T15:49:05.212 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_snap_trim_cost": "1048576", 2024-09-21T15:49:05.212 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_snap_trim_priority": "5", 2024-09-21T15:49:05.212 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-21T15:49:05.212 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-21T15:49:05.212 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-21T15:49:05.212 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-21T15:49:05.212 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-09-21T15:49:05.213 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_stats_update_period_scrubbing": "15", 2024-09-21T15:49:05.213 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-21T15:49:05.213 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_target_transaction_size": "30", 2024-09-21T15:49:05.213 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-21T15:49:05.213 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-21T15:49:05.213 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-21T15:49:05.213 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-21T15:49:05.213 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-21T15:49:05.213 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-21T15:49:05.213 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-21T15:49:05.213 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-21T15:49:05.213 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-21T15:49:05.213 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-21T15:49:05.213 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_tracing": "false", 2024-09-21T15:49:05.213 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_use_stale_snap": "false", 2024-09-21T15:49:05.213 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-21T15:49:05.214 INFO:teuthology.orchestra.run.smithi089.stdout: "osdc_blkin_trace_all": "false", 2024-09-21T15:49:05.214 INFO:teuthology.orchestra.run.smithi089.stdout: "paxos_kill_at": "0", 2024-09-21T15:49:05.214 INFO:teuthology.orchestra.run.smithi089.stdout: "paxos_max_join_drift": "10", 2024-09-21T15:49:05.214 INFO:teuthology.orchestra.run.smithi089.stdout: "paxos_min": "500", 2024-09-21T15:49:05.214 INFO:teuthology.orchestra.run.smithi089.stdout: "paxos_min_wait": "0.050000", 2024-09-21T15:49:05.214 INFO:teuthology.orchestra.run.smithi089.stdout: "paxos_propose_interval": "1.000000", 2024-09-21T15:49:05.214 INFO:teuthology.orchestra.run.smithi089.stdout: "paxos_service_trim_max": "500", 2024-09-21T15:49:05.214 INFO:teuthology.orchestra.run.smithi089.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-21T15:49:05.214 INFO:teuthology.orchestra.run.smithi089.stdout: "paxos_service_trim_min": "250", 2024-09-21T15:49:05.214 INFO:teuthology.orchestra.run.smithi089.stdout: "paxos_stash_full_interval": "25", 2024-09-21T15:49:05.214 INFO:teuthology.orchestra.run.smithi089.stdout: "paxos_trim_max": "500", 2024-09-21T15:49:05.214 INFO:teuthology.orchestra.run.smithi089.stdout: "paxos_trim_min": "250", 2024-09-21T15:49:05.214 INFO:teuthology.orchestra.run.smithi089.stdout: "perf": "true", 2024-09-21T15:49:05.214 INFO:teuthology.orchestra.run.smithi089.stdout: "pid_file": "/var/run/ceph/ceph-mon.a.pid", 2024-09-21T15:49:05.215 INFO:teuthology.orchestra.run.smithi089.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-21T15:49:05.215 INFO:teuthology.orchestra.run.smithi089.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-09-21T15:49:05.215 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "-", 2024-09-21T15:49:05.215 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrv": "", 2024-09-21T15:49:05.215 INFO:teuthology.orchestra.run.smithi089.stdout: "public_bind_addr": "-", 2024-09-21T15:49:05.215 INFO:teuthology.orchestra.run.smithi089.stdout: "public_network": "", 2024-09-21T15:49:05.215 INFO:teuthology.orchestra.run.smithi089.stdout: "public_network_interface": "", 2024-09-21T15:49:05.215 INFO:teuthology.orchestra.run.smithi089.stdout: "qat_compressor_enabled": "false", 2024-09-21T15:49:05.215 INFO:teuthology.orchestra.run.smithi089.stdout: "qat_compressor_session_max_number": "256", 2024-09-21T15:49:05.215 INFO:teuthology.orchestra.run.smithi089.stdout: "rados_mon_op_timeout": "0", 2024-09-21T15:49:05.215 INFO:teuthology.orchestra.run.smithi089.stdout: "rados_osd_op_timeout": "0", 2024-09-21T15:49:05.215 INFO:teuthology.orchestra.run.smithi089.stdout: "rados_tracing": "false", 2024-09-21T15:49:05.215 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_atime_update_interval": "60", 2024-09-21T15:49:05.215 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-21T15:49:05.215 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_balance_parent_reads": "false", 2024-09-21T15:49:05.215 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_balance_snap_reads": "false", 2024-09-21T15:49:05.216 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_blkin_trace_all": "false", 2024-09-21T15:49:05.216 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-21T15:49:05.216 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-21T15:49:05.216 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_cache": "true", 2024-09-21T15:49:05.216 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-21T15:49:05.216 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-21T15:49:05.216 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-21T15:49:05.216 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-21T15:49:05.216 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_cache_policy": "writearound", 2024-09-21T15:49:05.216 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_cache_size": "33554432", 2024-09-21T15:49:05.216 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-21T15:49:05.216 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-21T15:49:05.216 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_clone_copy_on_read": "false", 2024-09-21T15:49:05.216 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_compression_hint": "none", 2024-09-21T15:49:05.216 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_concurrent_management_ops": "10", 2024-09-21T15:49:05.216 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-21T15:49:05.217 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_default_clone_format": "auto", 2024-09-21T15:49:05.217 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_default_data_pool": "", 2024-09-21T15:49:05.217 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_default_features": "61", 2024-09-21T15:49:05.217 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_default_format": "2", 2024-09-21T15:49:05.217 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_default_map_options": "", 2024-09-21T15:49:05.217 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_default_order": "22", 2024-09-21T15:49:05.217 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_default_pool": "rbd", 2024-09-21T15:49:05.217 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-21T15:49:05.217 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_default_stripe_count": "0", 2024-09-21T15:49:05.217 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_default_stripe_unit": "0", 2024-09-21T15:49:05.217 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-21T15:49:05.217 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-21T15:49:05.217 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-21T15:49:05.217 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_enable_alloc_hint": "true", 2024-09-21T15:49:05.217 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-21T15:49:05.218 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_io_scheduler": "simple", 2024-09-21T15:49:05.218 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-21T15:49:05.218 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-21T15:49:05.218 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-21T15:49:05.218 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-21T15:49:05.218 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-21T15:49:05.218 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-21T15:49:05.218 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-21T15:49:05.218 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-21T15:49:05.218 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-21T15:49:05.218 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_journal_order": "24", 2024-09-21T15:49:05.218 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_journal_pool": "", 2024-09-21T15:49:05.218 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_journal_splay_width": "4", 2024-09-21T15:49:05.218 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_localize_parent_reads": "false", 2024-09-21T15:49:05.218 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_localize_snap_reads": "false", 2024-09-21T15:49:05.218 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-21T15:49:05.219 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-21T15:49:05.219 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-21T15:49:05.219 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-21T15:49:05.219 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-21T15:49:05.219 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-21T15:49:05.219 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-21T15:49:05.219 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-21T15:49:05.219 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-21T15:49:05.219 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-21T15:49:05.219 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-21T15:49:05.219 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-21T15:49:05.219 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-21T15:49:05.219 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-21T15:49:05.219 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-21T15:49:05.219 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-21T15:49:05.220 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-21T15:49:05.220 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-21T15:49:05.220 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-21T15:49:05.220 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-21T15:49:05.220 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-21T15:49:05.220 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-21T15:49:05.220 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-21T15:49:05.220 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-21T15:49:05.220 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-21T15:49:05.220 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-21T15:49:05.220 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-21T15:49:05.220 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-21T15:49:05.220 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-21T15:49:05.220 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-21T15:49:05.220 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-21T15:49:05.221 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_mtime_update_interval": "60", 2024-09-21T15:49:05.221 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_non_blocking_aio": "true", 2024-09-21T15:49:05.221 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_op_thread_timeout": "60", 2024-09-21T15:49:05.221 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_op_threads": "1", 2024-09-21T15:49:05.221 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_parent_cache_enabled": "false", 2024-09-21T15:49:05.221 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-21T15:49:05.221 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-21T15:49:05.221 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-21T15:49:05.221 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_plugins": "", 2024-09-21T15:49:05.221 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_qos_bps_burst": "0", 2024-09-21T15:49:05.221 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-21T15:49:05.221 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_qos_bps_limit": "0", 2024-09-21T15:49:05.221 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_qos_exclude_ops": "0", 2024-09-21T15:49:05.221 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_qos_iops_burst": "0", 2024-09-21T15:49:05.221 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-21T15:49:05.221 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_qos_iops_limit": "0", 2024-09-21T15:49:05.221 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-21T15:49:05.222 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-21T15:49:05.222 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-21T15:49:05.222 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-21T15:49:05.222 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-21T15:49:05.222 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-21T15:49:05.222 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-21T15:49:05.222 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-21T15:49:05.222 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-21T15:49:05.222 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-21T15:49:05.222 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-21T15:49:05.222 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-21T15:49:05.222 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-21T15:49:05.222 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-21T15:49:05.222 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_read_from_replica_policy": "default", 2024-09-21T15:49:05.223 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-21T15:49:05.223 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-21T15:49:05.223 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-21T15:49:05.223 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-21T15:49:05.223 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_skip_partial_discard": "true", 2024-09-21T15:49:05.223 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-21T15:49:05.223 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_tracing": "false", 2024-09-21T15:49:05.223 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_validate_names": "true", 2024-09-21T15:49:05.223 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd_validate_pool": "true", 2024-09-21T15:49:05.223 INFO:teuthology.orchestra.run.smithi089.stdout: "restapi_base_url": "", 2024-09-21T15:49:05.223 INFO:teuthology.orchestra.run.smithi089.stdout: "restapi_log_level": "", 2024-09-21T15:49:05.223 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_acl_grants_max_num": "100", 2024-09-21T15:49:05.223 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_admin_entry": "admin", 2024-09-21T15:49:05.223 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-09-21T15:49:05.223 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_backend_store": "rados", 2024-09-21T15:49:05.223 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_barbican_url": "", 2024-09-21T15:49:05.224 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_beast_enable_async": "true", 2024-09-21T15:49:05.224 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-21T15:49:05.224 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-21T15:49:05.224 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-21T15:49:05.224 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-09-21T15:49:05.224 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-21T15:49:05.224 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-21T15:49:05.224 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-09-21T15:49:05.224 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_cache_enabled": "true", 2024-09-21T15:49:05.224 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_cache_expiry_interval": "900", 2024-09-21T15:49:05.224 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_cache_lru_size": "10000", 2024-09-21T15:49:05.224 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_config_store": "rados", 2024-09-21T15:49:05.224 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_content_length_compat": "false", 2024-09-21T15:49:05.224 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_copy_obj_progress": "true", 2024-09-21T15:49:05.224 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-21T15:49:05.224 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_cors_rules_max_num": "100", 2024-09-21T15:49:05.225 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_cross_domain_policy": "", 2024-09-21T15:49:05.225 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-21T15:49:05.225 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_kmip_addr": "", 2024-09-21T15:49:05.225 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-21T15:49:05.225 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-21T15:49:05.225 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-21T15:49:05.225 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-21T15:49:05.225 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_kmip_password": "", 2024-09-21T15:49:05.225 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-21T15:49:05.225 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_kmip_username": "", 2024-09-21T15:49:05.225 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_require_ssl": "true", 2024-09-21T15:49:05.225 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-21T15:49:05.225 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-21T15:49:05.225 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-21T15:49:05.225 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-21T15:49:05.225 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-21T15:49:05.226 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-21T15:49:05.226 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-21T15:49:05.226 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-21T15:49:05.226 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-21T15:49:05.226 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-21T15:49:05.226 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-21T15:49:05.226 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-21T15:49:05.226 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-21T15:49:05.226 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-21T15:49:05.226 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-21T15:49:05.226 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_vault_addr": "", 2024-09-21T15:49:05.226 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_vault_auth": "token", 2024-09-21T15:49:05.226 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_vault_namespace": "", 2024-09-21T15:49:05.226 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_vault_prefix": "", 2024-09-21T15:49:05.226 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-21T15:49:05.227 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-21T15:49:05.227 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-21T15:49:05.227 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-21T15:49:05.227 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_vault_token_file": "", 2024-09-21T15:49:05.227 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-21T15:49:05.227 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_curl_buffersize": "524288", 2024-09-21T15:49:05.227 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-21T15:49:05.227 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_curl_low_speed_time": "300", 2024-09-21T15:49:05.227 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_curl_tcp_keepalive": "0", 2024-09-21T15:49:05.227 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-21T15:49:05.227 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-21T15:49:05.227 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-21T15:49:05.227 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-21T15:49:05.227 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-21T15:49:05.227 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-21T15:49:05.227 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-21T15:49:05.228 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-21T15:49:05.228 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-21T15:49:05.228 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-21T15:49:05.228 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_data_log_changes_size": "1000", 2024-09-21T15:49:05.228 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_data_log_num_shards": "128", 2024-09-21T15:49:05.228 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-21T15:49:05.228 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_data_log_window": "30", 2024-09-21T15:49:05.228 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_data_notify_interval_msec": "0", 2024-09-21T15:49:05.228 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_data_sync_poll_interval": "20", 2024-09-21T15:49:05.228 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_data_sync_spawn_window": "20", 2024-09-21T15:49:05.228 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-21T15:49:05.228 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-21T15:49:05.228 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-21T15:49:05.228 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-21T15:49:05.228 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-21T15:49:05.229 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-21T15:49:05.229 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-21T15:49:05.229 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-21T15:49:05.229 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-21T15:49:05.229 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-21T15:49:05.229 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-21T15:49:05.229 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-21T15:49:05.229 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-21T15:49:05.229 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-21T15:49:05.229 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-21T15:49:05.229 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-21T15:49:05.229 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-21T15:49:05.229 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-21T15:49:05.229 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-21T15:49:05.229 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-21T15:49:05.229 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-21T15:49:05.230 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_dns_name": "", 2024-09-21T15:49:05.230 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_dns_s3website_name": "", 2024-09-21T15:49:05.230 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_dynamic_resharding": "true", 2024-09-21T15:49:05.230 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-21T15:49:05.230 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_enable_gc_threads": "true", 2024-09-21T15:49:05.230 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_enable_lc_threads": "true", 2024-09-21T15:49:05.230 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_enable_ops_log": "false", 2024-09-21T15:49:05.230 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_enable_quota_threads": "true", 2024-09-21T15:49:05.230 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_enable_static_website": "false", 2024-09-21T15:49:05.230 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_enable_usage_log": "false", 2024-09-21T15:49:05.230 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_enforce_swift_acls": "true", 2024-09-21T15:49:05.230 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_exit_timeout_secs": "120", 2024-09-21T15:49:05.230 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_expose_bucket": "false", 2024-09-21T15:49:05.230 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_extended_http_attrs": "", 2024-09-21T15:49:05.230 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_filter": "none", 2024-09-21T15:49:05.231 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-21T15:49:05.231 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_frontends": "beast port=7480", 2024-09-21T15:49:05.231 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-21T15:49:05.231 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_gc_max_deferred": "50", 2024-09-21T15:49:05.231 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-21T15:49:05.231 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_gc_max_objs": "32", 2024-09-21T15:49:05.231 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-21T15:49:05.231 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-21T15:49:05.231 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-21T15:49:05.231 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-21T15:49:05.231 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_gc_processor_period": "3600", 2024-09-21T15:49:05.231 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-21T15:49:05.231 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-21T15:49:05.231 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-21T15:49:05.231 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-21T15:49:05.231 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_init_timeout": "300", 2024-09-21T15:49:05.232 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-21T15:49:05.232 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-09-21T15:49:05.232 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-21T15:49:05.232 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-21T15:49:05.232 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_keystone_admin_domain": "", 2024-09-21T15:49:05.232 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_keystone_admin_password": "", 2024-09-21T15:49:05.232 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_keystone_admin_password_path": "", 2024-09-21T15:49:05.232 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_keystone_admin_project": "", 2024-09-21T15:49:05.232 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_keystone_admin_tenant": "", 2024-09-21T15:49:05.232 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_keystone_admin_token": "", 2024-09-21T15:49:05.232 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_keystone_admin_token_path": "", 2024-09-21T15:49:05.232 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_keystone_admin_user": "", 2024-09-21T15:49:05.232 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_keystone_api_version": "2", 2024-09-21T15:49:05.232 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_keystone_barbican_domain": "", 2024-09-21T15:49:05.232 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_keystone_barbican_password": "", 2024-09-21T15:49:05.232 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_keystone_barbican_project": "", 2024-09-21T15:49:05.233 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-21T15:49:05.233 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_keystone_barbican_user": "", 2024-09-21T15:49:05.233 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-21T15:49:05.233 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-21T15:49:05.233 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-21T15:49:05.233 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-21T15:49:05.233 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-21T15:49:05.233 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_keystone_url": "", 2024-09-21T15:49:05.233 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-21T15:49:05.233 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_lc_debug_interval": "-1", 2024-09-21T15:49:05.233 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_lc_lock_max_time": "90", 2024-09-21T15:49:05.233 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_lc_max_objs": "32", 2024-09-21T15:49:05.233 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_lc_max_rules": "1000", 2024-09-21T15:49:05.233 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_lc_max_worker": "3", 2024-09-21T15:49:05.233 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-21T15:49:05.234 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_lc_thread_delay": "0", 2024-09-21T15:49:05.234 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-21T15:49:05.234 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_ldap_dnattr": "uid", 2024-09-21T15:49:05.234 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-21T15:49:05.234 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_ldap_searchfilter": "", 2024-09-21T15:49:05.234 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-21T15:49:05.234 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_ldap_uri": "ldaps://", 2024-09-21T15:49:05.234 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-21T15:49:05.234 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-21T15:49:05.234 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-21T15:49:05.234 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_log_http_headers": "", 2024-09-21T15:49:05.234 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-21T15:49:05.234 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-21T15:49:05.234 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_log_object_name_utc": "false", 2024-09-21T15:49:05.234 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_luarocks_location": "/tmp/luarocks", 2024-09-21T15:49:05.234 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_max_attr_name_len": "0", 2024-09-21T15:49:05.235 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_max_attr_size": "0", 2024-09-21T15:49:05.235 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-21T15:49:05.235 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_max_chunk_size": "4194304", 2024-09-21T15:49:05.235 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-21T15:49:05.235 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-09-21T15:49:05.235 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-21T15:49:05.235 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_max_listing_results": "1000", 2024-09-21T15:49:05.235 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_max_notify_retries": "10", 2024-09-21T15:49:05.235 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-21T15:49:05.235 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_max_put_param_size": "1048576", 2024-09-21T15:49:05.235 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_max_put_size": "5368709120", 2024-09-21T15:49:05.235 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_max_slo_entries": "1000", 2024-09-21T15:49:05.235 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_md_log_max_shards": "64", 2024-09-21T15:49:05.235 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-21T15:49:05.235 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_meta_sync_poll_interval": "20", 2024-09-21T15:49:05.235 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_meta_sync_spawn_window": "20", 2024-09-21T15:49:05.236 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-21T15:49:05.236 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_mp_lock_max_time": "600", 2024-09-21T15:49:05.236 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-21T15:49:05.236 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-21T15:49:05.236 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-21T15:49:05.236 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-21T15:49:05.236 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-21T15:49:05.236 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-09-21T15:49:05.236 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-21T15:49:05.236 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-21T15:49:05.236 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_nfs_max_gc": "300", 2024-09-21T15:49:05.236 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-21T15:49:05.236 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-21T15:49:05.236 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-21T15:49:05.236 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-21T15:49:05.237 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-21T15:49:05.237 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-21T15:49:05.237 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-21T15:49:05.237 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_num_async_rados_threads": "32", 2024-09-21T15:49:05.237 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_num_control_oids": "8", 2024-09-21T15:49:05.237 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_numa_node": "-1", 2024-09-21T15:49:05.237 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-21T15:49:05.237 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-21T15:49:05.237 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_objexp_chunk_size": "100", 2024-09-21T15:49:05.237 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_objexp_gc_interval": "600", 2024-09-21T15:49:05.237 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-21T15:49:05.237 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-21T15:49:05.237 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-21T15:49:05.237 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_op_thread_timeout": "600", 2024-09-21T15:49:05.237 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_op_tracing": "false", 2024-09-21T15:49:05.237 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_opa_token": "", 2024-09-21T15:49:05.238 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_opa_url": "", 2024-09-21T15:49:05.238 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_opa_verify_ssl": "true", 2024-09-21T15:49:05.238 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-21T15:49:05.238 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-09-21T15:49:05.238 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_ops_log_rados": "false", 2024-09-21T15:49:05.238 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_ops_log_socket_path": "", 2024-09-21T15:49:05.238 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-21T15:49:05.238 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-09-21T15:49:05.238 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-21T15:49:05.238 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_period_push_interval": "2.000000", 2024-09-21T15:49:05.238 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-21T15:49:05.238 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-21T15:49:05.238 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-09-21T15:49:05.238 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_print_continue": "true", 2024-09-21T15:49:05.238 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-21T15:49:05.239 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-21T15:49:05.239 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-21T15:49:05.239 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-21T15:49:05.239 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-21T15:49:05.239 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_rados_tracing": "false", 2024-09-21T15:49:05.239 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_realm": "", 2024-09-21T15:49:05.239 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_realm_id": "", 2024-09-21T15:49:05.239 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-21T15:49:05.239 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_region": "", 2024-09-21T15:49:05.239 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-21T15:49:05.239 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-21T15:49:05.239 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-21T15:49:05.239 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-21T15:49:05.239 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_request_uri": "", 2024-09-21T15:49:05.239 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_reshard_batch_size": "64", 2024-09-21T15:49:05.240 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-21T15:49:05.240 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_reshard_max_aio": "128", 2024-09-21T15:49:05.240 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_reshard_num_logs": "16", 2024-09-21T15:49:05.240 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_reshard_thread_interval": "600", 2024-09-21T15:49:05.240 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_resolve_cname": "false", 2024-09-21T15:49:05.240 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-21T15:49:05.240 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_run_sync_thread": "true", 2024-09-21T15:49:05.240 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-21T15:49:05.240 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-21T15:49:05.240 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-21T15:49:05.240 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-21T15:49:05.240 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-21T15:49:05.240 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-21T15:49:05.240 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-21T15:49:05.240 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-21T15:49:05.241 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_scheduler_type": "throttler", 2024-09-21T15:49:05.241 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_script_uri": "", 2024-09-21T15:49:05.241 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_service_provider_name": "", 2024-09-21T15:49:05.241 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-21T15:49:05.241 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_sts_client_id": "", 2024-09-21T15:49:05.241 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_sts_client_secret": "", 2024-09-21T15:49:05.241 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_sts_entry": "sts", 2024-09-21T15:49:05.241 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_sts_key": "sts", 2024-09-21T15:49:05.241 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-21T15:49:05.241 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_sts_min_session_duration": "900", 2024-09-21T15:49:05.241 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_sts_token_introspection_url": "", 2024-09-21T15:49:05.241 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_swift_account_in_url": "false", 2024-09-21T15:49:05.241 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_swift_auth_entry": "auth", 2024-09-21T15:49:05.241 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_swift_auth_url": "", 2024-09-21T15:49:05.241 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_swift_custom_header": "", 2024-09-21T15:49:05.242 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-21T15:49:05.242 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_swift_need_stats": "true", 2024-09-21T15:49:05.242 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_swift_tenant_name": "", 2024-09-21T15:49:05.242 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_swift_token_expiration": "86400", 2024-09-21T15:49:05.242 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_swift_url": "", 2024-09-21T15:49:05.242 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_swift_url_prefix": "swift", 2024-09-21T15:49:05.242 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-21T15:49:05.242 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-09-21T15:49:05.242 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-21T15:49:05.242 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_sync_lease_period": "120", 2024-09-21T15:49:05.242 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-21T15:49:05.242 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-21T15:49:05.242 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-21T15:49:05.242 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-21T15:49:05.242 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-21T15:49:05.243 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-21T15:49:05.243 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-21T15:49:05.243 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-21T15:49:05.243 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-21T15:49:05.243 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_thread_pool_size": "512", 2024-09-21T15:49:05.243 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_torrent_comment": "", 2024-09-21T15:49:05.243 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_torrent_createby": "", 2024-09-21T15:49:05.243 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_torrent_encoding": "", 2024-09-21T15:49:05.243 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_torrent_flag": "false", 2024-09-21T15:49:05.243 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_torrent_origin": "", 2024-09-21T15:49:05.243 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-21T15:49:05.243 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_torrent_tracker": "", 2024-09-21T15:49:05.243 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_trust_forwarded_https": "false", 2024-09-21T15:49:05.243 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-21T15:49:05.243 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-21T15:49:05.244 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_usage_max_shards": "32", 2024-09-21T15:49:05.244 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_usage_max_user_shards": "1", 2024-09-21T15:49:05.244 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_use_opa_authz": "false", 2024-09-21T15:49:05.244 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-21T15:49:05.244 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-21T15:49:05.244 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_user_max_buckets": "1000", 2024-09-21T15:49:05.244 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_user_policies_max_num": "100", 2024-09-21T15:49:05.244 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-21T15:49:05.244 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-21T15:49:05.244 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-21T15:49:05.244 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-21T15:49:05.244 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_user_unique_email": "true", 2024-09-21T15:49:05.244 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_verify_ssl": "true", 2024-09-21T15:49:05.244 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-21T15:49:05.245 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_zone": "", 2024-09-21T15:49:05.245 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_zone_id": "", 2024-09-21T15:49:05.245 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-21T15:49:05.245 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_zonegroup": "", 2024-09-21T15:49:05.245 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_zonegroup_id": "", 2024-09-21T15:49:05.245 INFO:teuthology.orchestra.run.smithi089.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-21T15:49:05.245 INFO:teuthology.orchestra.run.smithi089.stdout: "rgwlc_auto_session_clear": "true", 2024-09-21T15:49:05.245 INFO:teuthology.orchestra.run.smithi089.stdout: "rgwlc_skip_bucket_step": "false", 2024-09-21T15:49:05.245 INFO:teuthology.orchestra.run.smithi089.stdout: "rocksdb_block_size": "4096", 2024-09-21T15:49:05.245 INFO:teuthology.orchestra.run.smithi089.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-21T15:49:05.245 INFO:teuthology.orchestra.run.smithi089.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-21T15:49:05.245 INFO:teuthology.orchestra.run.smithi089.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-21T15:49:05.245 INFO:teuthology.orchestra.run.smithi089.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-21T15:49:05.245 INFO:teuthology.orchestra.run.smithi089.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-21T15:49:05.245 INFO:teuthology.orchestra.run.smithi089.stdout: "rocksdb_cache_size": "536870912", 2024-09-21T15:49:05.245 INFO:teuthology.orchestra.run.smithi089.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-21T15:49:05.246 INFO:teuthology.orchestra.run.smithi089.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-21T15:49:05.246 INFO:teuthology.orchestra.run.smithi089.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-21T15:49:05.246 INFO:teuthology.orchestra.run.smithi089.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-21T15:49:05.246 INFO:teuthology.orchestra.run.smithi089.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-21T15:49:05.246 INFO:teuthology.orchestra.run.smithi089.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-21T15:49:05.246 INFO:teuthology.orchestra.run.smithi089.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-21T15:49:05.246 INFO:teuthology.orchestra.run.smithi089.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-21T15:49:05.246 INFO:teuthology.orchestra.run.smithi089.stdout: "rocksdb_index_type": "binary_search", 2024-09-21T15:49:05.246 INFO:teuthology.orchestra.run.smithi089.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-21T15:49:05.246 INFO:teuthology.orchestra.run.smithi089.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-21T15:49:05.246 INFO:teuthology.orchestra.run.smithi089.stdout: "rocksdb_partition_filters": "false", 2024-09-21T15:49:05.246 INFO:teuthology.orchestra.run.smithi089.stdout: "rocksdb_perf": "false", 2024-09-21T15:49:05.246 INFO:teuthology.orchestra.run.smithi089.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-21T15:49:05.246 INFO:teuthology.orchestra.run.smithi089.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-21T15:49:05.246 INFO:teuthology.orchestra.run.smithi089.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-21T15:49:05.247 INFO:teuthology.orchestra.run.smithi089.stdout: "run_dir": "/var/run/ceph", 2024-09-21T15:49:05.247 INFO:teuthology.orchestra.run.smithi089.stdout: "seastore_block_create": "true", 2024-09-21T15:49:05.247 INFO:teuthology.orchestra.run.smithi089.stdout: "seastore_cache_lru_size": "67108864", 2024-09-21T15:49:05.247 INFO:teuthology.orchestra.run.smithi089.stdout: "seastore_cbjournal_size": "5368709120", 2024-09-21T15:49:05.247 INFO:teuthology.orchestra.run.smithi089.stdout: "seastore_default_max_object_size": "16777216", 2024-09-21T15:49:05.247 INFO:teuthology.orchestra.run.smithi089.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-21T15:49:05.247 INFO:teuthology.orchestra.run.smithi089.stdout: "seastore_device_size": "53687091200", 2024-09-21T15:49:05.247 INFO:teuthology.orchestra.run.smithi089.stdout: "seastore_journal_batch_capacity": "16", 2024-09-21T15:49:05.247 INFO:teuthology.orchestra.run.smithi089.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-21T15:49:05.247 INFO:teuthology.orchestra.run.smithi089.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-21T15:49:05.247 INFO:teuthology.orchestra.run.smithi089.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-21T15:49:05.247 INFO:teuthology.orchestra.run.smithi089.stdout: "seastore_main_device_type": "SSD", 2024-09-21T15:49:05.247 INFO:teuthology.orchestra.run.smithi089.stdout: "seastore_max_concurrent_transactions": "8", 2024-09-21T15:49:05.247 INFO:teuthology.orchestra.run.smithi089.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-09-21T15:49:05.248 INFO:teuthology.orchestra.run.smithi089.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-09-21T15:49:05.248 INFO:teuthology.orchestra.run.smithi089.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-09-21T15:49:05.248 INFO:teuthology.orchestra.run.smithi089.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-09-21T15:49:05.248 INFO:teuthology.orchestra.run.smithi089.stdout: "seastore_segment_size": "67108864", 2024-09-21T15:49:05.248 INFO:teuthology.orchestra.run.smithi089.stdout: "set_keepcaps": "false", 2024-09-21T15:49:05.248 INFO:teuthology.orchestra.run.smithi089.stdout: "setgroup": "", 2024-09-21T15:49:05.248 INFO:teuthology.orchestra.run.smithi089.stdout: "setuser": "", 2024-09-21T15:49:05.248 INFO:teuthology.orchestra.run.smithi089.stdout: "setuser_match_path": "", 2024-09-21T15:49:05.248 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-21T15:49:05.248 INFO:teuthology.orchestra.run.smithi089.stdout: "thp": "false", 2024-09-21T15:49:05.248 INFO:teuthology.orchestra.run.smithi089.stdout: "threadpool_default_timeout": "60", 2024-09-21T15:49:05.248 INFO:teuthology.orchestra.run.smithi089.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-21T15:49:05.248 INFO:teuthology.orchestra.run.smithi089.stdout: "throttler_perf_counter": "true" 2024-09-21T15:49:05.248 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:05.250 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-21T15:49:05.359 INFO:teuthology.orchestra.run.smithi089.stdout:mon.a: {} 2024-09-21T15:49:05.360 INFO:teuthology.orchestra.run.smithi089.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T15:49:05.369 INFO:teuthology.orchestra.run.smithi089.stdout:mon.b: {} 2024-09-21T15:49:05.369 INFO:teuthology.orchestra.run.smithi089.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T15:49:05.377 INFO:teuthology.orchestra.run.smithi089.stdout:mon.c: {} 2024-09-21T15:49:05.377 INFO:teuthology.orchestra.run.smithi089.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T15:49:05.388 DEBUG:teuthology.orchestra.run.smithi089:> type ceph-objectstore-tool 2024-09-21T15:49:05.402 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-21T15:49:05.402 DEBUG:teuthology.orchestra.run.smithi100:> type ceph-objectstore-tool 2024-09-21T15:49:05.421 INFO:teuthology.orchestra.run.smithi100.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-21T15:49:05.421 INFO:teuthology.run_tasks:Running task rbd_fsx... 2024-09-21T15:49:05.437 INFO:tasks.rbd_fsx:starting rbd_fsx... 2024-09-21T15:49:05.437 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-21T15:49:05.438 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T15:49:05.438 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T15:49:05.438 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-09-21T15:49:05.438 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T15:49:05.438 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T15:49:05.438 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3 2024-09-21T15:49:05.438 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-21T15:49:05.439 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-21T15:49:05.439 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:49:05.439 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-21T15:49:05.439 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:49:05.439 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-21T15:49:05.440 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T15:49:05.440 DEBUG:teuthology.orchestra.run.smithi089:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_librbd_fsx --cluster ceph --id 0 -d -W -R -p 100 -P /home/ubuntu/cephtest/archive -r 1 -w 1 -t 1 -h 1 -l 250000000 -S 0 -N 6000 pool_client.0 image_client.0 2024-09-21T15:49:05.462 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:05.461+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:05.478 INFO:teuthology.orchestra.run.smithi089.stdout:mapped writes DISABLED 2024-09-21T15:49:05.479 INFO:teuthology.orchestra.run.smithi089.stdout:mapped reads DISABLED 2024-09-21T15:49:05.479 INFO:teuthology.orchestra.run.smithi089.stdout:Seed set to 8312 2024-09-21T15:49:05.542 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:05.539+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:05.555 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:05.555 INFO:teuthology.orchestra.run.smithi089.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T15:49:05.555 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:05.556 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:05.556 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:49:05.556 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:49:05.556 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:05.567 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:49:05.586 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-09-21T15:49:05.586 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:49:05.604 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:49:05.604 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:49:05.620 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:49:05.620 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:49:05.636 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:49:05.636 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:49:05.641 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:05.639+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:05.654 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:49:05.672 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:49:05.672 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:49:05.677 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:05.677 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:05.690 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:49:05.694 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:49:05.694 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:49:05.727 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:49:05.728 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:49:05.741 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:05.740+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:05.797 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:05.797 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:05.806 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:49:05.841 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:05.840+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:05.909 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:05.909 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:05.918 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:49:05.943 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:05.942+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:06.020 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:06.020 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:06.029 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:49:06.072 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:06.071+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:06.130 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:06.130 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:06.140 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:49:06.181 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:06.180+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:06.244 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:06.243+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:06.282 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:06.282 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:06.292 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:49:06.345 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:06.343+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:06.394 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:06.394 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:06.403 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:49:06.445 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:06.444+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:06.546 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:06.546 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:06.546 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:06.545+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:06.554 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:49:06.646 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:06.645+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:06.656 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:06.656 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:06.665 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:49:06.683 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is set 2024-09-21T15:49:06.747 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:06.746+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:06.776 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:06.776 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:06.784 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:49:06.868 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:06.867+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:06.927 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:06.927 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:06.935 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:49:06.976 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:06.975+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:07.049 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:07.048+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:07.050 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:07.050 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:07.059 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:49:07.149 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:07.148+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:07.203 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:07.203 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:07.211 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:49:07.256 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:07.255+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:07.313 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:07.313 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:07.321 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:49:07.362 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:07.361+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:07.453 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:07.452+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:07.463 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:07.463 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:07.472 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:49:07.554 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:07.553+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:07.574 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:07.575 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:07.585 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:49:07.655 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:07.654+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:07.688 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:07.688 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:07.697 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T15:49:07.742 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:49:07.786 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:07.785+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:07.839 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:07.839 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:07.847 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T15:49:07.869 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-09-21T15:49:07.869 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:49:07.885 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:49:07.885 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:49:07.892 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:07.891+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:07.901 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:49:07.901 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:49:07.916 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:49:07.917 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:49:07.934 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:49:07.934 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:49:07.949 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:49:07.949 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:49:07.951 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:07.951 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:07.957 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:07.956+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:07.960 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T15:49:07.965 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:49:07.965 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:49:07.981 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:49:07.982 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:49:08.059 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:08.057+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:08.102 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:08.102 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:49:08.102 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:49:08.102 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [] 2024-09-21T15:49:08.102 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:08.111 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T15:49:08.159 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:08.158+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:08.213 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:08.214 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:08.214 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:49:08.214 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:08.222 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T15:49:08.263 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:08.262+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:08.361 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:08.360+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:08.363 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:08.364 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:08.364 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:49:08.364 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:49:08.364 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:08.372 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T15:49:08.462 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:08.461+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:08.516 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:08.516 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:49:08.516 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:49:08.517 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:49:08.517 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.517 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4209.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:49:08.517 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:49:08.517 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 17.322583371, 2024-09-21T15:49:08.517 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.19452909900000001, 2024-09-21T15:49:08.517 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:08.517 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:08.517 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:08.517 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4209", 2024-09-21T15:49:08.517 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/4045143050", 2024-09-21T15:49:08.517 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1 2024-09-21T15:49:08.517 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.517 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:08.518 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.518 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:08.518 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:49:08.518 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:08.518 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.518 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.518 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:08.518 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:49:08.518 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:08.518 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.518 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.518 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:08.518 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191161+0000", 2024-09-21T15:49:08.518 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:08.518 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.519 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.519 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:08.519 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191169+0000", 2024-09-21T15:49:08.519 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0090000000000007e-06 2024-09-21T15:49:08.519 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.519 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.519 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:08.519 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191180+0000", 2024-09-21T15:49:08.519 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0728e-05 2024-09-21T15:49:08.519 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.519 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.519 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:08.519 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191187+0000", 2024-09-21T15:49:08.519 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9009999999999994e-06 2024-09-21T15:49:08.520 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.520 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.520 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:08.520 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191238+0000", 2024-09-21T15:49:08.520 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0287000000000002e-05 2024-09-21T15:49:08.520 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.520 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.520 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-21T15:49:08.520 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191245+0000", 2024-09-21T15:49:08.520 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0709999999999997e-06 2024-09-21T15:49:08.520 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.520 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.520 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:08.520 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380309+0000", 2024-09-21T15:49:08.520 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.189064277 2024-09-21T15:49:08.521 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.521 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.521 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:08.521 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380454+0000", 2024-09-21T15:49:08.521 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014490799999999999 2024-09-21T15:49:08.521 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.521 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.521 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0", 2024-09-21T15:49:08.521 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380610+0000", 2024-09-21T15:49:08.521 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015650199999999999 2024-09-21T15:49:08.521 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.521 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.521 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:08.521 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.381160+0000", 2024-09-21T15:49:08.522 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00054903799999999996 2024-09-21T15:49:08.522 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.522 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.522 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:08.522 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385591+0000", 2024-09-21T15:49:08.522 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.004431272 2024-09-21T15:49:08.522 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.522 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.522 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:08.522 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385651+0000", 2024-09-21T15:49:08.522 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0455999999999999e-05 2024-09-21T15:49:08.522 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.522 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.522 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:08.523 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385691+0000", 2024-09-21T15:49:08.523 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0234999999999997e-05 2024-09-21T15:49:08.523 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:08.523 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:08.523 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:08.523 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.523 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.523 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4209.0:21 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 65536~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:49:08.523 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:48:51.410326+0000", 2024-09-21T15:49:08.523 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 17.103420057000001, 2024-09-21T15:49:08.523 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0075314029999999999, 2024-09-21T15:49:08.523 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:08.524 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:08.524 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:08.524 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4209", 2024-09-21T15:49:08.524 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/4045143050", 2024-09-21T15:49:08.524 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 21 2024-09-21T15:49:08.524 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.524 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:08.524 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.524 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:08.524 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.410326+0000", 2024-09-21T15:49:08.524 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:08.524 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.524 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.524 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:08.525 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.410326+0000", 2024-09-21T15:49:08.525 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:08.525 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.525 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.525 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:08.525 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.410325+0000", 2024-09-21T15:49:08.525 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-21T15:49:08.525 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.525 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.525 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:08.525 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.410397+0000", 2024-09-21T15:49:08.525 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2465000000000005e-05 2024-09-21T15:49:08.525 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.525 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.525 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:08.525 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.410404+0000", 2024-09-21T15:49:08.526 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3180000000000003e-06 2024-09-21T15:49:08.526 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.526 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.526 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:08.526 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.410408+0000", 2024-09-21T15:49:08.526 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9380000000000002e-06 2024-09-21T15:49:08.526 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.526 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.526 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:08.526 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.410975+0000", 2024-09-21T15:49:08.526 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00056738800000000005 2024-09-21T15:49:08.526 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.526 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.526 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:08.527 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.411046+0000", 2024-09-21T15:49:08.527 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0971999999999997e-05 2024-09-21T15:49:08.527 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.527 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.527 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0", 2024-09-21T15:49:08.527 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.411140+0000", 2024-09-21T15:49:08.527 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4302000000000006e-05 2024-09-21T15:49:08.527 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.527 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.527 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:08.528 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.411989+0000", 2024-09-21T15:49:08.528 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00084893900000000005 2024-09-21T15:49:08.528 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.528 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.528 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:08.528 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.417775+0000", 2024-09-21T15:49:08.528 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0057862950000000003 2024-09-21T15:49:08.528 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.528 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.528 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:08.528 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.417821+0000", 2024-09-21T15:49:08.528 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5522000000000002e-05 2024-09-21T15:49:08.528 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.528 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.529 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:08.529 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.417857+0000", 2024-09-21T15:49:08.529 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6109000000000003e-05 2024-09-21T15:49:08.529 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:08.529 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:08.529 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:08.529 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.529 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.529 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4209.0:22 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:49:08.529 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:48:51.410415+0000", 2024-09-21T15:49:08.529 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 17.103330975999999, 2024-09-21T15:49:08.529 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0075417000000000001, 2024-09-21T15:49:08.529 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:08.529 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:08.530 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:08.530 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4209", 2024-09-21T15:49:08.530 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/4045143050", 2024-09-21T15:49:08.530 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 22 2024-09-21T15:49:08.530 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.530 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:08.530 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.530 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:08.530 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.410415+0000", 2024-09-21T15:49:08.530 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:08.530 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.530 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.530 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:08.530 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.410415+0000", 2024-09-21T15:49:08.530 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:08.531 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.531 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.531 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:08.531 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.410414+0000", 2024-09-21T15:49:08.531 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:08.531 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.531 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.531 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:08.531 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.410422+0000", 2024-09-21T15:49:08.531 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5390000000000002e-06 2024-09-21T15:49:08.531 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.531 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.531 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:08.531 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.410426+0000", 2024-09-21T15:49:08.531 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1609999999999996e-06 2024-09-21T15:49:08.532 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.532 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.532 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:08.532 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.410432+0000", 2024-09-21T15:49:08.532 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8769999999999998e-06 2024-09-21T15:49:08.532 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.532 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.532 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:08.532 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.411561+0000", 2024-09-21T15:49:08.532 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001128729 2024-09-21T15:49:08.532 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.532 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.532 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:08.532 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.411644+0000", 2024-09-21T15:49:08.532 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3411999999999994e-05 2024-09-21T15:49:08.533 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.533 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.533 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0", 2024-09-21T15:49:08.533 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.411771+0000", 2024-09-21T15:49:08.533 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000126857 2024-09-21T15:49:08.533 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.533 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.533 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:08.533 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.412241+0000", 2024-09-21T15:49:08.533 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00047017899999999999 2024-09-21T15:49:08.533 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.533 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.533 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:08.533 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.417893+0000", 2024-09-21T15:49:08.533 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0056522430000000004 2024-09-21T15:49:08.533 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.534 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.534 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:08.534 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.417925+0000", 2024-09-21T15:49:08.534 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2103000000000001e-05 2024-09-21T15:49:08.534 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.534 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.534 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:08.534 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.417956+0000", 2024-09-21T15:49:08.534 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1049e-05 2024-09-21T15:49:08.534 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:08.534 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:08.534 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:08.534 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.534 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.534 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4209.0:55 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:49:08.534 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:48:51.450808+0000", 2024-09-21T15:49:08.535 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 17.062937852000001, 2024-09-21T15:49:08.535 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0069101529999999996, 2024-09-21T15:49:08.535 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:08.535 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:08.535 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:08.535 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4209", 2024-09-21T15:49:08.535 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/4045143050", 2024-09-21T15:49:08.535 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 55 2024-09-21T15:49:08.535 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.535 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:08.535 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.535 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:08.535 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.450808+0000", 2024-09-21T15:49:08.535 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:08.536 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.536 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.536 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:08.536 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.450808+0000", 2024-09-21T15:49:08.536 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:08.536 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.536 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.536 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:08.536 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.450807+0000", 2024-09-21T15:49:08.536 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-21T15:49:08.536 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.536 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.536 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:08.536 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.450891+0000", 2024-09-21T15:49:08.537 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4159999999999994e-05 2024-09-21T15:49:08.537 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.537 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.537 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:08.537 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.450902+0000", 2024-09-21T15:49:08.537 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0930999999999999e-05 2024-09-21T15:49:08.537 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.537 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.537 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:08.537 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.450908+0000", 2024-09-21T15:49:08.537 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7989999999999999e-06 2024-09-21T15:49:08.537 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.537 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.537 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:08.537 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.451635+0000", 2024-09-21T15:49:08.538 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00072709100000000002 2024-09-21T15:49:08.538 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.538 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.538 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:08.538 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.451715+0000", 2024-09-21T15:49:08.538 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9936000000000002e-05 2024-09-21T15:49:08.538 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.538 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.538 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0", 2024-09-21T15:49:08.538 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.451836+0000", 2024-09-21T15:49:08.538 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000120559 2024-09-21T15:49:08.538 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.538 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.538 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:08.539 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.452700+0000", 2024-09-21T15:49:08.539 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00086452999999999996 2024-09-21T15:49:08.539 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.539 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.539 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:08.539 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.457583+0000", 2024-09-21T15:49:08.539 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0048826319999999996 2024-09-21T15:49:08.539 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.539 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.539 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:08.539 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.457665+0000", 2024-09-21T15:49:08.539 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2194000000000006e-05 2024-09-21T15:49:08.539 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.539 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.539 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:08.540 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.457718+0000", 2024-09-21T15:49:08.540 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3038999999999998e-05 2024-09-21T15:49:08.540 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:08.540 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:08.540 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:08.540 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.540 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.540 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4209.0:56 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:49:08.540 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:48:51.450916+0000", 2024-09-21T15:49:08.540 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 17.062829735000001, 2024-09-21T15:49:08.540 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0069673809999999999, 2024-09-21T15:49:08.540 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:08.540 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:08.540 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:08.540 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4209", 2024-09-21T15:49:08.540 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/4045143050", 2024-09-21T15:49:08.541 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 56 2024-09-21T15:49:08.541 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.541 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:08.541 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.541 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:08.541 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.450916+0000", 2024-09-21T15:49:08.541 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:08.541 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.541 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.541 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:08.541 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.450916+0000", 2024-09-21T15:49:08.541 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:08.541 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.541 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.541 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:08.541 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.450915+0000", 2024-09-21T15:49:08.542 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:08.542 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.542 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.542 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:08.542 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.450920+0000", 2024-09-21T15:49:08.542 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1339999999999997e-06 2024-09-21T15:49:08.542 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.542 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.542 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:08.542 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.450926+0000", 2024-09-21T15:49:08.542 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8460000000000002e-06 2024-09-21T15:49:08.542 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.542 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.542 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:08.543 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.450931+0000", 2024-09-21T15:49:08.543 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.899e-06 2024-09-21T15:49:08.543 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.543 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.543 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:08.543 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.452288+0000", 2024-09-21T15:49:08.543 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001356929 2024-09-21T15:49:08.543 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.543 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.543 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:08.543 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.452355+0000", 2024-09-21T15:49:08.543 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6456000000000002e-05 2024-09-21T15:49:08.543 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.543 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.543 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0", 2024-09-21T15:49:08.544 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.452428+0000", 2024-09-21T15:49:08.544 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3202999999999997e-05 2024-09-21T15:49:08.544 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.544 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.544 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:08.544 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.452971+0000", 2024-09-21T15:49:08.544 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00054313500000000004 2024-09-21T15:49:08.544 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.544 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.544 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:08.544 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.457782+0000", 2024-09-21T15:49:08.544 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0048112060000000002 2024-09-21T15:49:08.544 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.544 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.544 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:08.545 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.457864+0000", 2024-09-21T15:49:08.545 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1550000000000004e-05 2024-09-21T15:49:08.545 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.545 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.545 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:08.545 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.457883+0000", 2024-09-21T15:49:08.545 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9602999999999998e-05 2024-09-21T15:49:08.545 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:08.545 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:08.545 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:08.545 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.545 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.545 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4209.0:58 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [write 131072~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:49:08.545 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:48:51.458174+0000", 2024-09-21T15:49:08.546 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 17.055571775000001, 2024-09-21T15:49:08.546 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0073108069999999999, 2024-09-21T15:49:08.546 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:08.546 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:08.546 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:08.546 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4209", 2024-09-21T15:49:08.546 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/4045143050", 2024-09-21T15:49:08.546 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 58 2024-09-21T15:49:08.546 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.546 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:08.546 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.546 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:08.546 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.458174+0000", 2024-09-21T15:49:08.546 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:08.547 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.547 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.547 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:08.547 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.458174+0000", 2024-09-21T15:49:08.547 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:08.547 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.547 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.547 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:08.547 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.458173+0000", 2024-09-21T15:49:08.547 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:08.547 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.547 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.547 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:08.547 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.458258+0000", 2024-09-21T15:49:08.547 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4943000000000003e-05 2024-09-21T15:49:08.548 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.548 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.548 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:08.548 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.458267+0000", 2024-09-21T15:49:08.548 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0839999999999998e-06 2024-09-21T15:49:08.548 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.548 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.548 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:08.548 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.458274+0000", 2024-09-21T15:49:08.548 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3589999999999996e-06 2024-09-21T15:49:08.548 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.548 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.548 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:08.548 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.458597+0000", 2024-09-21T15:49:08.549 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000322839 2024-09-21T15:49:08.549 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.549 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.549 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:08.549 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.458674+0000", 2024-09-21T15:49:08.549 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6990999999999995e-05 2024-09-21T15:49:08.549 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.549 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.549 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0", 2024-09-21T15:49:08.549 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.458813+0000", 2024-09-21T15:49:08.549 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013970700000000001 2024-09-21T15:49:08.549 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.549 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.549 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:08.550 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.459501+0000", 2024-09-21T15:49:08.550 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00068760700000000002 2024-09-21T15:49:08.550 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.550 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.550 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:08.550 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.465405+0000", 2024-09-21T15:49:08.550 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0059040780000000001 2024-09-21T15:49:08.550 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.550 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.550 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:08.550 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.465463+0000", 2024-09-21T15:49:08.550 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8266000000000001e-05 2024-09-21T15:49:08.550 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.550 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.550 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:08.551 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.465485+0000", 2024-09-21T15:49:08.551 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1568000000000001e-05 2024-09-21T15:49:08.551 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:08.551 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:08.551 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:08.551 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.551 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.551 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4209.0:59 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:49:08.551 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:48:51.458284+0000", 2024-09-21T15:49:08.551 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 17.055462219999999, 2024-09-21T15:49:08.551 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0072805420000000001, 2024-09-21T15:49:08.551 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:08.551 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:08.551 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:08.552 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4209", 2024-09-21T15:49:08.552 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/4045143050", 2024-09-21T15:49:08.552 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 59 2024-09-21T15:49:08.552 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.552 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:08.552 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.552 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:08.552 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.458284+0000", 2024-09-21T15:49:08.552 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:08.552 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.552 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.552 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:08.552 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.458284+0000", 2024-09-21T15:49:08.552 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:08.552 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.553 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.553 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:08.553 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.458283+0000", 2024-09-21T15:49:08.553 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:08.553 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.553 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.553 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:08.553 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.458288+0000", 2024-09-21T15:49:08.553 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6190000000000002e-06 2024-09-21T15:49:08.553 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.553 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.553 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:08.554 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.458294+0000", 2024-09-21T15:49:08.554 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5289999999999997e-06 2024-09-21T15:49:08.554 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.554 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.554 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:08.554 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.458299+0000", 2024-09-21T15:49:08.554 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5800000000000002e-06 2024-09-21T15:49:08.554 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.554 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.554 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:08.554 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.459188+0000", 2024-09-21T15:49:08.554 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00088962600000000002 2024-09-21T15:49:08.554 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.554 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.555 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:08.555 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.459250+0000", 2024-09-21T15:49:08.555 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1556999999999994e-05 2024-09-21T15:49:08.555 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.555 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.555 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0", 2024-09-21T15:49:08.555 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.459324+0000", 2024-09-21T15:49:08.555 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4325000000000004e-05 2024-09-21T15:49:08.555 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.555 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.555 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:08.555 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.459763+0000", 2024-09-21T15:49:08.555 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00043917400000000002 2024-09-21T15:49:08.555 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.555 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.556 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:08.556 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.465514+0000", 2024-09-21T15:49:08.556 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0057508489999999997 2024-09-21T15:49:08.556 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.556 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.556 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:08.556 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.465545+0000", 2024-09-21T15:49:08.556 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0388000000000001e-05 2024-09-21T15:49:08.556 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.556 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.556 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:08.556 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.465564+0000", 2024-09-21T15:49:08.556 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9510000000000001e-05 2024-09-21T15:49:08.556 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:08.556 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:08.556 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:08.557 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.557 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.557 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4209.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:49:08.557 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:48:51.633532+0000", 2024-09-21T15:49:08.557 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 16.880213747999999, 2024-09-21T15:49:08.557 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0070635749999999999, 2024-09-21T15:49:08.557 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:08.557 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:08.557 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:08.557 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4209", 2024-09-21T15:49:08.557 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/4045143050", 2024-09-21T15:49:08.557 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 117 2024-09-21T15:49:08.557 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.557 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:08.557 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.557 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:08.558 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.633532+0000", 2024-09-21T15:49:08.558 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:08.558 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.558 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.558 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:08.558 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.633532+0000", 2024-09-21T15:49:08.558 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:08.558 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.558 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.558 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:08.558 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.633532+0000", 2024-09-21T15:49:08.558 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:08.558 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.558 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.558 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:08.559 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.633537+0000", 2024-09-21T15:49:08.559 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1359999999999996e-06 2024-09-21T15:49:08.559 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.559 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.559 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:08.559 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.633545+0000", 2024-09-21T15:49:08.559 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6160000000000002e-06 2024-09-21T15:49:08.559 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.559 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.559 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:08.559 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.633550+0000", 2024-09-21T15:49:08.559 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8459999999999999e-06 2024-09-21T15:49:08.559 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.559 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.559 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:08.559 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.639424+0000", 2024-09-21T15:49:08.560 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0058736220000000002 2024-09-21T15:49:08.560 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.560 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.560 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:08.560 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.639480+0000", 2024-09-21T15:49:08.560 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5841000000000003e-05 2024-09-21T15:49:08.560 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.560 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.560 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0", 2024-09-21T15:49:08.560 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.639556+0000", 2024-09-21T15:49:08.560 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6907000000000001e-05 2024-09-21T15:49:08.560 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.560 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.560 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:08.560 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.640102+0000", 2024-09-21T15:49:08.560 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00054595999999999996 2024-09-21T15:49:08.561 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.561 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.561 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:08.561 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.640541+0000", 2024-09-21T15:49:08.561 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00043851300000000002 2024-09-21T15:49:08.561 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.561 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.561 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:08.561 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.640566+0000", 2024-09-21T15:49:08.561 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5123999999999999e-05 2024-09-21T15:49:08.561 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.561 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.561 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:08.561 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.640596+0000", 2024-09-21T15:49:08.561 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9507000000000001e-05 2024-09-21T15:49:08.561 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:08.562 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:08.562 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:08.562 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.562 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.562 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4209.0:132 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:49:08.562 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:48:51.960729+0000", 2024-09-21T15:49:08.562 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 16.553017249, 2024-09-21T15:49:08.562 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.007466833, 2024-09-21T15:49:08.562 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:08.562 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:08.562 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:08.562 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4209", 2024-09-21T15:49:08.562 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/4045143050", 2024-09-21T15:49:08.562 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 132 2024-09-21T15:49:08.562 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.563 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:08.563 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.563 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:08.563 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.960729+0000", 2024-09-21T15:49:08.563 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:08.563 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.563 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.563 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:08.563 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.960729+0000", 2024-09-21T15:49:08.563 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:08.563 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.563 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.563 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:08.563 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.960728+0000", 2024-09-21T15:49:08.563 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:08.564 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.564 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.564 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:08.564 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.960732+0000", 2024-09-21T15:49:08.564 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7850000000000002e-06 2024-09-21T15:49:08.564 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.564 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.564 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:08.564 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.960738+0000", 2024-09-21T15:49:08.564 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4289999999999998e-06 2024-09-21T15:49:08.564 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.564 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.564 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:08.564 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.960743+0000", 2024-09-21T15:49:08.564 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8300000000000003e-06 2024-09-21T15:49:08.564 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.564 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.565 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:08.565 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.964774+0000", 2024-09-21T15:49:08.565 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0040313370000000003 2024-09-21T15:49:08.565 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.565 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.565 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:08.565 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.964818+0000", 2024-09-21T15:49:08.565 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3405999999999997e-05 2024-09-21T15:49:08.565 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.565 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.565 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0", 2024-09-21T15:49:08.565 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.964889+0000", 2024-09-21T15:49:08.565 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1039999999999997e-05 2024-09-21T15:49:08.565 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.565 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.566 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:08.566 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.966777+0000", 2024-09-21T15:49:08.566 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001888287 2024-09-21T15:49:08.566 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.566 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.566 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:08.566 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.968151+0000", 2024-09-21T15:49:08.566 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0013735659999999999 2024-09-21T15:49:08.566 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.566 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.566 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:08.566 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.968175+0000", 2024-09-21T15:49:08.566 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4769e-05 2024-09-21T15:49:08.566 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.566 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.567 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:08.567 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.968195+0000", 2024-09-21T15:49:08.567 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0015999999999999e-05 2024-09-21T15:49:08.567 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:08.567 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:08.567 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:08.567 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.567 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.567 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4209.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:49:08.567 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:48:51.960761+0000", 2024-09-21T15:49:08.567 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 16.552984476999999, 2024-09-21T15:49:08.567 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0083343619999999997, 2024-09-21T15:49:08.567 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:08.567 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:08.568 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:08.568 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4209", 2024-09-21T15:49:08.568 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/4045143050", 2024-09-21T15:49:08.568 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 133 2024-09-21T15:49:08.568 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.568 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:08.568 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.568 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:08.568 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.960761+0000", 2024-09-21T15:49:08.568 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:08.568 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.568 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.568 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:08.568 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.960761+0000", 2024-09-21T15:49:08.568 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:08.568 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.569 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.569 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:08.569 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.960761+0000", 2024-09-21T15:49:08.569 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:08.569 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.569 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.569 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:08.569 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.960765+0000", 2024-09-21T15:49:08.569 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0149999999999997e-06 2024-09-21T15:49:08.569 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.569 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.569 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:08.569 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.960773+0000", 2024-09-21T15:49:08.569 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0299999999999994e-06 2024-09-21T15:49:08.569 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.569 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.570 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:08.570 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.960779+0000", 2024-09-21T15:49:08.570 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5320000000000003e-06 2024-09-21T15:49:08.570 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.570 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.570 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:08.570 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.965149+0000", 2024-09-21T15:49:08.570 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0043699589999999996 2024-09-21T15:49:08.570 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.570 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.570 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:08.570 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.965222+0000", 2024-09-21T15:49:08.570 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2749000000000004e-05 2024-09-21T15:49:08.570 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.570 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.570 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0", 2024-09-21T15:49:08.571 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.965296+0000", 2024-09-21T15:49:08.571 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4240999999999996e-05 2024-09-21T15:49:08.571 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.571 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.571 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:08.571 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.966808+0000", 2024-09-21T15:49:08.571 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001511957 2024-09-21T15:49:08.571 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.571 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.571 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:08.571 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.969047+0000", 2024-09-21T15:49:08.571 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0022384459999999998 2024-09-21T15:49:08.571 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.571 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.571 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:08.572 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.969077+0000", 2024-09-21T15:49:08.572 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9833999999999999e-05 2024-09-21T15:49:08.572 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.572 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.572 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:08.572 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.969096+0000", 2024-09-21T15:49:08.572 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9128999999999999e-05 2024-09-21T15:49:08.572 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:08.572 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:08.572 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:08.572 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.572 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.572 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4209.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:49:08.572 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:48:51.960803+0000", 2024-09-21T15:49:08.573 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 16.552942351999999, 2024-09-21T15:49:08.573 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.008335482, 2024-09-21T15:49:08.573 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:08.573 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:08.573 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:08.573 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4209", 2024-09-21T15:49:08.573 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/4045143050", 2024-09-21T15:49:08.573 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 134 2024-09-21T15:49:08.573 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.573 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:08.573 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.573 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:08.573 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.960803+0000", 2024-09-21T15:49:08.573 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:08.574 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.574 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.574 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:08.574 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.960803+0000", 2024-09-21T15:49:08.574 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:08.574 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.574 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.574 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:08.574 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.960803+0000", 2024-09-21T15:49:08.574 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:08.574 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.574 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.574 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:08.574 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.960912+0000", 2024-09-21T15:49:08.574 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000109607 2024-09-21T15:49:08.575 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.575 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.575 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:08.575 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.960928+0000", 2024-09-21T15:49:08.575 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5417000000000002e-05 2024-09-21T15:49:08.575 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.575 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.575 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:08.575 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.960934+0000", 2024-09-21T15:49:08.575 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5050000000000004e-06 2024-09-21T15:49:08.575 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.575 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.575 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:08.575 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.965542+0000", 2024-09-21T15:49:08.576 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0046079520000000002 2024-09-21T15:49:08.576 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.576 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.576 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:08.576 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.965604+0000", 2024-09-21T15:49:08.576 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1650999999999996e-05 2024-09-21T15:49:08.576 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.576 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.576 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0", 2024-09-21T15:49:08.576 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.965671+0000", 2024-09-21T15:49:08.576 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6582999999999996e-05 2024-09-21T15:49:08.576 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.576 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.576 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:08.577 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.968145+0000", 2024-09-21T15:49:08.577 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0024744120000000001 2024-09-21T15:49:08.577 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.577 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.577 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:08.577 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.969102+0000", 2024-09-21T15:49:08.577 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00095750299999999996 2024-09-21T15:49:08.577 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.577 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.577 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:08.577 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.969123+0000", 2024-09-21T15:49:08.577 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0761e-05 2024-09-21T15:49:08.577 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.577 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.578 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:08.578 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.969139+0000", 2024-09-21T15:49:08.578 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5688999999999998e-05 2024-09-21T15:49:08.578 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:08.578 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:08.578 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:08.578 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.578 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.578 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4209.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:49:08.578 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:48:51.960943+0000", 2024-09-21T15:49:08.578 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 16.552802561, 2024-09-21T15:49:08.578 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0082935030000000007, 2024-09-21T15:49:08.578 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:08.578 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:08.579 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:08.579 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4209", 2024-09-21T15:49:08.579 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/4045143050", 2024-09-21T15:49:08.579 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 135 2024-09-21T15:49:08.579 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.579 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:08.579 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.579 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:08.579 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.960943+0000", 2024-09-21T15:49:08.579 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:08.579 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.579 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.579 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:08.579 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.960943+0000", 2024-09-21T15:49:08.580 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:08.580 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.580 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.580 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:08.580 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.960943+0000", 2024-09-21T15:49:08.580 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:08.580 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.580 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.580 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:08.580 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.960947+0000", 2024-09-21T15:49:08.580 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5109999999999997e-06 2024-09-21T15:49:08.580 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.580 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.580 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:08.580 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.960955+0000", 2024-09-21T15:49:08.581 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0269999999999995e-06 2024-09-21T15:49:08.581 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.581 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.581 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:08.581 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.960965+0000", 2024-09-21T15:49:08.581 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9930000000000007e-06 2024-09-21T15:49:08.581 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.581 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.581 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:08.581 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.966080+0000", 2024-09-21T15:49:08.581 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0051153179999999998 2024-09-21T15:49:08.581 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.581 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.581 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:08.582 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.966125+0000", 2024-09-21T15:49:08.582 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4543999999999999e-05 2024-09-21T15:49:08.582 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.582 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.582 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0", 2024-09-21T15:49:08.582 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.966194+0000", 2024-09-21T15:49:08.582 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9412999999999994e-05 2024-09-21T15:49:08.582 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.582 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.582 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:08.582 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.969145+0000", 2024-09-21T15:49:08.582 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0029502949999999999 2024-09-21T15:49:08.582 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.582 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.582 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:08.583 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.969183+0000", 2024-09-21T15:49:08.583 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8569000000000003e-05 2024-09-21T15:49:08.583 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.583 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.583 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:08.583 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.969210+0000", 2024-09-21T15:49:08.583 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6840999999999999e-05 2024-09-21T15:49:08.583 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.583 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.583 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:08.583 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.969237+0000", 2024-09-21T15:49:08.583 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6614999999999998e-05 2024-09-21T15:49:08.583 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:08.584 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:08.584 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:08.584 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.584 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.584 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4209.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:49:08.584 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:48:51.960971+0000", 2024-09-21T15:49:08.584 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 16.552774802999998, 2024-09-21T15:49:08.584 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0083661589999999997, 2024-09-21T15:49:08.584 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:08.584 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:08.584 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:08.584 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4209", 2024-09-21T15:49:08.584 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/4045143050", 2024-09-21T15:49:08.584 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 136 2024-09-21T15:49:08.584 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.585 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:08.585 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.585 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:08.585 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.960971+0000", 2024-09-21T15:49:08.585 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:08.585 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.585 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.585 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:08.585 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.960971+0000", 2024-09-21T15:49:08.585 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:08.585 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.585 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.585 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:08.585 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.960971+0000", 2024-09-21T15:49:08.585 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:08.586 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.586 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.586 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:08.586 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.960974+0000", 2024-09-21T15:49:08.586 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9430000000000002e-06 2024-09-21T15:49:08.586 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.586 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.586 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:08.586 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.960983+0000", 2024-09-21T15:49:08.586 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1790000000000008e-06 2024-09-21T15:49:08.586 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.586 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.586 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:08.586 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.960992+0000", 2024-09-21T15:49:08.587 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5480000000000007e-06 2024-09-21T15:49:08.587 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.587 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.587 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:08.587 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.966446+0000", 2024-09-21T15:49:08.587 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0054534379999999997 2024-09-21T15:49:08.587 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.587 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.587 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:08.587 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.966488+0000", 2024-09-21T15:49:08.587 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2225999999999999e-05 2024-09-21T15:49:08.587 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.587 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.587 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0", 2024-09-21T15:49:08.587 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.966557+0000", 2024-09-21T15:49:08.588 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8961000000000006e-05 2024-09-21T15:49:08.588 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.588 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.588 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:08.588 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.969148+0000", 2024-09-21T15:49:08.588 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0025910019999999998 2024-09-21T15:49:08.588 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.588 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.588 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:08.588 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.969276+0000", 2024-09-21T15:49:08.588 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012818199999999999 2024-09-21T15:49:08.588 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.588 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.588 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:08.588 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.969317+0000", 2024-09-21T15:49:08.589 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0695999999999998e-05 2024-09-21T15:49:08.589 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.589 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.589 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:08.589 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.969337+0000", 2024-09-21T15:49:08.589 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0398999999999999e-05 2024-09-21T15:49:08.589 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:08.589 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:08.589 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:08.589 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.589 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.589 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4209.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:49:08.589 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:48:51.961000+0000", 2024-09-21T15:49:08.589 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 16.552746204000002, 2024-09-21T15:49:08.589 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0084066750000000006, 2024-09-21T15:49:08.590 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:08.590 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:08.590 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:08.590 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4209", 2024-09-21T15:49:08.590 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/4045143050", 2024-09-21T15:49:08.590 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 137 2024-09-21T15:49:08.590 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.590 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:08.590 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.590 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:08.590 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.961000+0000", 2024-09-21T15:49:08.590 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:08.590 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.590 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.591 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:08.591 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.961000+0000", 2024-09-21T15:49:08.591 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:08.591 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.591 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.591 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:08.591 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.960999+0000", 2024-09-21T15:49:08.591 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:08.591 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.591 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.591 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:08.591 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.961083+0000", 2024-09-21T15:49:08.591 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3438999999999999e-05 2024-09-21T15:49:08.591 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.591 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.591 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:08.592 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.961092+0000", 2024-09-21T15:49:08.592 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5680000000000005e-06 2024-09-21T15:49:08.592 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.592 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.592 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:08.592 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.961098+0000", 2024-09-21T15:49:08.592 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2949999999999999e-06 2024-09-21T15:49:08.592 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.592 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.592 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:08.592 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.966831+0000", 2024-09-21T15:49:08.592 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0057329290000000003 2024-09-21T15:49:08.592 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.592 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.592 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:08.593 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.966875+0000", 2024-09-21T15:49:08.593 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4063000000000001e-05 2024-09-21T15:49:08.593 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.593 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.593 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0", 2024-09-21T15:49:08.593 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.966943+0000", 2024-09-21T15:49:08.593 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7546999999999996e-05 2024-09-21T15:49:08.593 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.593 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.593 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:08.593 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.969152+0000", 2024-09-21T15:49:08.593 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0022087999999999999 2024-09-21T15:49:08.593 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.593 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.593 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:08.593 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.969368+0000", 2024-09-21T15:49:08.594 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00021579499999999999 2024-09-21T15:49:08.594 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.594 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.594 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:08.594 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.969390+0000", 2024-09-21T15:49:08.594 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2259000000000002e-05 2024-09-21T15:49:08.594 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.594 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.594 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:08.594 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.969406+0000", 2024-09-21T15:49:08.594 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6419e-05 2024-09-21T15:49:08.594 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:08.594 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:08.594 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:08.594 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.595 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.595 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4209.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:49:08.595 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:48:51.961106+0000", 2024-09-21T15:49:08.595 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 16.552639967000001, 2024-09-21T15:49:08.595 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0091200770000000007, 2024-09-21T15:49:08.595 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:08.595 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:08.595 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:08.595 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4209", 2024-09-21T15:49:08.595 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/4045143050", 2024-09-21T15:49:08.595 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 138 2024-09-21T15:49:08.595 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.595 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:08.595 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.595 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:08.595 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.961106+0000", 2024-09-21T15:49:08.596 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:08.596 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.596 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.596 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:08.596 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.961106+0000", 2024-09-21T15:49:08.596 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:08.596 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.596 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.596 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:08.596 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.961105+0000", 2024-09-21T15:49:08.596 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:08.596 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.596 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.596 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:08.596 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.961109+0000", 2024-09-21T15:49:08.597 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1860000000000002e-06 2024-09-21T15:49:08.597 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.597 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.597 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:08.597 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.961116+0000", 2024-09-21T15:49:08.597 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3260000000000001e-06 2024-09-21T15:49:08.597 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.597 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.597 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:08.597 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.961120+0000", 2024-09-21T15:49:08.597 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3479999999999997e-06 2024-09-21T15:49:08.597 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.597 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.597 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:08.597 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.967345+0000", 2024-09-21T15:49:08.597 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0062248670000000002 2024-09-21T15:49:08.598 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.598 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.598 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:08.598 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.967389+0000", 2024-09-21T15:49:08.598 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4104999999999999e-05 2024-09-21T15:49:08.598 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.598 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.598 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0", 2024-09-21T15:49:08.598 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.967464+0000", 2024-09-21T15:49:08.598 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5172000000000005e-05 2024-09-21T15:49:08.598 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.598 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.598 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:08.598 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.969430+0000", 2024-09-21T15:49:08.598 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0019654379999999999 2024-09-21T15:49:08.599 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.599 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.599 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:08.599 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.970185+0000", 2024-09-21T15:49:08.599 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00075544000000000002 2024-09-21T15:49:08.599 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.599 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.599 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:08.599 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.970207+0000", 2024-09-21T15:49:08.599 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1492e-05 2024-09-21T15:49:08.599 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.599 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.599 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:08.599 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.970226+0000", 2024-09-21T15:49:08.599 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9244000000000001e-05 2024-09-21T15:49:08.599 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:08.600 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:08.600 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:08.600 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.600 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.600 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4209.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:49:08.600 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:48:51.961135+0000", 2024-09-21T15:49:08.600 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 16.552611069000001, 2024-09-21T15:49:08.600 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.009410112, 2024-09-21T15:49:08.600 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:08.600 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:08.600 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:08.600 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4209", 2024-09-21T15:49:08.600 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/4045143050", 2024-09-21T15:49:08.600 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 139 2024-09-21T15:49:08.600 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.600 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:08.601 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.601 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:08.601 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.961135+0000", 2024-09-21T15:49:08.601 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:08.601 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.601 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.601 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:08.601 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.961135+0000", 2024-09-21T15:49:08.601 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:08.601 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.601 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.601 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:08.601 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.961134+0000", 2024-09-21T15:49:08.601 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:08.601 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.601 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.601 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:08.602 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.961137+0000", 2024-09-21T15:49:08.602 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.446e-06 2024-09-21T15:49:08.602 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.602 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.602 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:08.602 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.961141+0000", 2024-09-21T15:49:08.602 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7740000000000002e-06 2024-09-21T15:49:08.602 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.602 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.602 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:08.602 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.961143+0000", 2024-09-21T15:49:08.602 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7480000000000002e-06 2024-09-21T15:49:08.602 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.602 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.602 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:08.603 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.967756+0000", 2024-09-21T15:49:08.603 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0066129639999999998 2024-09-21T15:49:08.603 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.603 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.603 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:08.603 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.967808+0000", 2024-09-21T15:49:08.603 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1779e-05 2024-09-21T15:49:08.603 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.603 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.603 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0", 2024-09-21T15:49:08.603 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.967881+0000", 2024-09-21T15:49:08.603 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2673e-05 2024-09-21T15:49:08.603 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.603 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.603 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:08.604 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.970389+0000", 2024-09-21T15:49:08.604 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00250782 2024-09-21T15:49:08.604 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.605 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.605 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:08.605 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.970500+0000", 2024-09-21T15:49:08.605 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000111153 2024-09-21T15:49:08.605 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.605 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.605 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:08.605 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.970526+0000", 2024-09-21T15:49:08.605 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5814000000000001e-05 2024-09-21T15:49:08.605 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.606 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.606 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:08.606 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.970545+0000", 2024-09-21T15:49:08.606 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.925e-05 2024-09-21T15:49:08.606 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:08.606 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:08.606 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:08.606 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.606 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.606 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4209.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:49:08.606 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:48:51.961147+0000", 2024-09-21T15:49:08.606 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 16.552599138000001, 2024-09-21T15:49:08.606 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.009443089, 2024-09-21T15:49:08.606 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:08.606 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:08.607 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:08.607 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4209", 2024-09-21T15:49:08.607 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/4045143050", 2024-09-21T15:49:08.607 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 140 2024-09-21T15:49:08.607 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.607 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:08.607 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.607 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:08.607 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.961147+0000", 2024-09-21T15:49:08.607 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:08.607 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.607 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.607 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:08.607 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.961147+0000", 2024-09-21T15:49:08.607 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:08.607 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.608 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.608 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:08.608 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.961146+0000", 2024-09-21T15:49:08.608 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:08.608 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.608 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.608 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:08.608 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.961199+0000", 2024-09-21T15:49:08.608 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2315999999999999e-05 2024-09-21T15:49:08.608 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.608 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.608 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:08.608 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.961203+0000", 2024-09-21T15:49:08.608 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3930000000000001e-06 2024-09-21T15:49:08.608 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.608 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.608 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:08.609 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.961208+0000", 2024-09-21T15:49:08.609 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7700000000000001e-06 2024-09-21T15:49:08.609 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.609 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.609 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:08.609 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.968221+0000", 2024-09-21T15:49:08.609 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00701332 2024-09-21T15:49:08.609 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.609 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.609 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:08.609 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.968278+0000", 2024-09-21T15:49:08.609 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7283e-05 2024-09-21T15:49:08.609 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.609 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.609 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0", 2024-09-21T15:49:08.609 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.968360+0000", 2024-09-21T15:49:08.610 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1422999999999996e-05 2024-09-21T15:49:08.610 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.610 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.610 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:08.610 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.970422+0000", 2024-09-21T15:49:08.610 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0020619010000000001 2024-09-21T15:49:08.610 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.610 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.610 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:08.610 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.970551+0000", 2024-09-21T15:49:08.610 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012950599999999999 2024-09-21T15:49:08.610 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.610 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.610 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:08.610 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.970573+0000", 2024-09-21T15:49:08.610 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2132000000000001e-05 2024-09-21T15:49:08.611 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.611 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.611 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:08.611 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.970590+0000", 2024-09-21T15:49:08.611 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6254000000000001e-05 2024-09-21T15:49:08.611 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:08.611 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:08.611 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:08.611 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.611 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.611 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4209.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:49:08.611 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:48:51.961211+0000", 2024-09-21T15:49:08.611 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 16.552534549000001, 2024-09-21T15:49:08.611 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0094235829999999993, 2024-09-21T15:49:08.611 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:08.612 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:08.612 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:08.612 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4209", 2024-09-21T15:49:08.612 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/4045143050", 2024-09-21T15:49:08.612 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 141 2024-09-21T15:49:08.612 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.612 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:08.612 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.612 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:08.612 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.961211+0000", 2024-09-21T15:49:08.612 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:08.612 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.612 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.612 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:08.613 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.961211+0000", 2024-09-21T15:49:08.613 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:08.613 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.613 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.613 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:08.613 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.961211+0000", 2024-09-21T15:49:08.613 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:08.613 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.613 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.613 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:08.613 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.961213+0000", 2024-09-21T15:49:08.613 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.154e-06 2024-09-21T15:49:08.613 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.613 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.613 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:08.613 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.961216+0000", 2024-09-21T15:49:08.613 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2049999999999998e-06 2024-09-21T15:49:08.614 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.614 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.614 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:08.614 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.961219+0000", 2024-09-21T15:49:08.614 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4700000000000001e-06 2024-09-21T15:49:08.614 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.614 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.614 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:08.614 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.968713+0000", 2024-09-21T15:49:08.614 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0074942489999999997 2024-09-21T15:49:08.614 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.614 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.614 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:08.614 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.968757+0000", 2024-09-21T15:49:08.614 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4172000000000003e-05 2024-09-21T15:49:08.615 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.615 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.615 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0", 2024-09-21T15:49:08.615 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.968827+0000", 2024-09-21T15:49:08.615 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0154999999999995e-05 2024-09-21T15:49:08.615 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.615 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.615 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:08.615 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.970440+0000", 2024-09-21T15:49:08.615 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0016127330000000001 2024-09-21T15:49:08.615 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.615 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.615 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:08.615 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.970593+0000", 2024-09-21T15:49:08.615 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015323499999999999 2024-09-21T15:49:08.615 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.615 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.616 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:08.616 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.970613+0000", 2024-09-21T15:49:08.616 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0157999999999999e-05 2024-09-21T15:49:08.616 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.616 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.616 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:08.616 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.970635+0000", 2024-09-21T15:49:08.616 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1330000000000001e-05 2024-09-21T15:49:08.616 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:08.616 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:08.616 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:08.616 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.616 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.616 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4209.0:158 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T15:49:08.616 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:48:52.417615+0000", 2024-09-21T15:49:08.616 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 16.096130381999998, 2024-09-21T15:49:08.617 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.012373113999999999, 2024-09-21T15:49:08.617 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:08.617 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:08.617 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:08.617 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4209", 2024-09-21T15:49:08.617 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/4045143050", 2024-09-21T15:49:08.617 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 158 2024-09-21T15:49:08.617 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.617 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:08.617 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.617 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:08.617 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:52.417615+0000", 2024-09-21T15:49:08.617 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:08.617 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.617 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.617 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:08.617 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:52.417615+0000", 2024-09-21T15:49:08.618 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:08.618 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.618 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.618 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:08.618 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:52.417615+0000", 2024-09-21T15:49:08.618 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:08.618 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.618 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.618 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:08.618 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:52.417694+0000", 2024-09-21T15:49:08.618 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8679999999999999e-05 2024-09-21T15:49:08.618 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.618 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.618 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:08.618 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:52.417698+0000", 2024-09-21T15:49:08.619 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3290000000000004e-06 2024-09-21T15:49:08.619 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.619 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.619 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:08.619 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:52.417702+0000", 2024-09-21T15:49:08.619 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8290000000000001e-06 2024-09-21T15:49:08.619 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.619 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.619 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:08.619 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:52.420022+0000", 2024-09-21T15:49:08.619 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0023200550000000001 2024-09-21T15:49:08.619 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.619 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.619 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:08.619 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:52.420098+0000", 2024-09-21T15:49:08.619 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5879999999999999e-05 2024-09-21T15:49:08.620 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.620 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.620 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0", 2024-09-21T15:49:08.620 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:52.420219+0000", 2024-09-21T15:49:08.620 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000121495 2024-09-21T15:49:08.620 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.620 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.620 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:08.620 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:52.425464+0000", 2024-09-21T15:49:08.620 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0052448699999999996 2024-09-21T15:49:08.620 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.620 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.620 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:08.620 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:52.429942+0000", 2024-09-21T15:49:08.620 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0044779770000000002 2024-09-21T15:49:08.620 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.621 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.621 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:08.621 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:52.429969+0000", 2024-09-21T15:49:08.621 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7087999999999999e-05 2024-09-21T15:49:08.621 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.621 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.621 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:08.621 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:52.429988+0000", 2024-09-21T15:49:08.621 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9151e-05 2024-09-21T15:49:08.621 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:08.621 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:08.621 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:08.621 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.621 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.621 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4209.0:159 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e16)", 2024-09-21T15:49:08.621 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:48:52.417706+0000", 2024-09-21T15:49:08.622 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 16.096040239000001, 2024-09-21T15:49:08.622 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.012554381999999999, 2024-09-21T15:49:08.622 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:08.622 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:08.622 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:08.622 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4209", 2024-09-21T15:49:08.622 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/4045143050", 2024-09-21T15:49:08.622 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 159 2024-09-21T15:49:08.622 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.622 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:08.622 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.622 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:08.622 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:52.417706+0000", 2024-09-21T15:49:08.622 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:08.622 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.622 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.623 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:08.623 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:52.417706+0000", 2024-09-21T15:49:08.623 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:08.623 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.623 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.623 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:08.623 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:52.417705+0000", 2024-09-21T15:49:08.623 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:08.623 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.623 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.623 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:08.623 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:52.417708+0000", 2024-09-21T15:49:08.623 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2589999999999999e-06 2024-09-21T15:49:08.623 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.623 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.624 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:08.624 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:52.417711+0000", 2024-09-21T15:49:08.624 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1860000000000001e-06 2024-09-21T15:49:08.624 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.624 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.624 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:08.624 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:52.417714+0000", 2024-09-21T15:49:08.624 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1879999999999998e-06 2024-09-21T15:49:08.624 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.624 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.624 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:08.624 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:52.422195+0000", 2024-09-21T15:49:08.624 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.004480866 2024-09-21T15:49:08.624 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.625 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.625 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:08.625 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:52.422242+0000", 2024-09-21T15:49:08.625 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7073000000000001e-05 2024-09-21T15:49:08.625 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.625 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.625 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0", 2024-09-21T15:49:08.625 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:52.422308+0000", 2024-09-21T15:49:08.625 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5884999999999997e-05 2024-09-21T15:49:08.625 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.625 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.625 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:08.625 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:52.427845+0000", 2024-09-21T15:49:08.625 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0055377309999999997 2024-09-21T15:49:08.625 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.625 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.626 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:08.626 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:52.430219+0000", 2024-09-21T15:49:08.626 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0023734400000000001 2024-09-21T15:49:08.626 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.626 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.626 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:08.626 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:52.430242+0000", 2024-09-21T15:49:08.626 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2847999999999999e-05 2024-09-21T15:49:08.626 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:08.626 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:08.626 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:08.626 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:52.430260+0000", 2024-09-21T15:49:08.626 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8172000000000002e-05 2024-09-21T15:49:08.626 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:08.626 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:08.626 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:08.627 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:08.627 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:08.627 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:08.628 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:08.627+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:08.628 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:49:08.705 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T15:49:08.749 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:08.748+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:08.761 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:08.761 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:49:08.761 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:49:08.761 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:08.770 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:49:08.830 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:08.828+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:08.915 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:08.915 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:49:08.915 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:49:08.915 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:49:08.915 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:08.924 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:49:08.968 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:08.967+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:09.030 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:09.031 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:49:09.031 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:49:09.031 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:49:09.031 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.031 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4209.0:1 1.0 e15/14 1:30306672:devicehealth::main.db.0000000000000000:head v 15'1, mlcod=0'0)", 2024-09-21T15:49:09.031 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:48:51.380733+0000", 2024-09-21T15:49:09.031 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.647053864, 2024-09-21T15:49:09.031 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046510550000000003, 2024-09-21T15:49:09.031 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:09.031 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:09.031 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:09.031 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.031 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:09.031 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.380733+0000", 2024-09-21T15:49:09.032 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:09.032 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.032 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.032 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:09.032 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.380733+0000", 2024-09-21T15:49:09.032 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:09.032 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.032 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.032 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:09.032 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.380733+0000", 2024-09-21T15:49:09.032 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:49:09.032 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.032 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.032 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:09.032 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.380745+0000", 2024-09-21T15:49:09.032 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2043e-05 2024-09-21T15:49:09.032 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.033 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.033 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:09.033 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.380756+0000", 2024-09-21T15:49:09.033 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1588e-05 2024-09-21T15:49:09.033 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.033 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.033 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:09.033 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.380764+0000", 2024-09-21T15:49:09.033 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1810000000000001e-06 2024-09-21T15:49:09.033 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.033 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.033 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:09.033 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.380822+0000", 2024-09-21T15:49:09.033 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7503000000000002e-05 2024-09-21T15:49:09.033 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.033 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.034 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:09.034 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.380848+0000", 2024-09-21T15:49:09.034 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6443000000000001e-05 2024-09-21T15:49:09.034 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.034 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:09.034 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.385350+0000", 2024-09-21T15:49:09.034 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045012969999999996 2024-09-21T15:49:09.034 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.034 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:09.034 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.385384+0000", 2024-09-21T15:49:09.034 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4187999999999999e-05 2024-09-21T15:49:09.034 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:09.035 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:09.035 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:09.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.035 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4209.0:18 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'8, mlcod=15'5)", 2024-09-21T15:49:09.035 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:48:51.402576+0000", 2024-09-21T15:49:09.035 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.625210458000002, 2024-09-21T15:49:09.035 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0043380090000000003, 2024-09-21T15:49:09.035 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:09.035 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:09.035 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:09.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.035 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:09.035 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.402576+0000", 2024-09-21T15:49:09.035 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:09.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.036 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:09.036 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.402576+0000", 2024-09-21T15:49:09.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:09.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.036 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:09.036 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.402576+0000", 2024-09-21T15:49:09.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:09.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.036 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:09.036 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.402585+0000", 2024-09-21T15:49:09.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6670000000000008e-06 2024-09-21T15:49:09.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.037 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:09.037 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.402594+0000", 2024-09-21T15:49:09.037 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0319999999999992e-06 2024-09-21T15:49:09.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.037 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:09.037 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.402601+0000", 2024-09-21T15:49:09.037 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2740000000000004e-06 2024-09-21T15:49:09.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.037 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:09.037 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.402630+0000", 2024-09-21T15:49:09.037 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9113000000000001e-05 2024-09-21T15:49:09.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.038 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:09.038 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.402647+0000", 2024-09-21T15:49:09.038 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6994e-05 2024-09-21T15:49:09.038 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.038 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.038 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:09.038 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.406878+0000", 2024-09-21T15:49:09.038 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0042316569999999998 2024-09-21T15:49:09.038 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.038 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.038 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:09.038 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.406914+0000", 2024-09-21T15:49:09.038 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5732999999999998e-05 2024-09-21T15:49:09.038 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:09.038 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:09.038 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:09.039 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.039 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.039 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4209.0:21 1.0 e15/14 1:30306672:devicehealth::main.db.0000000000000000:head v 15'11, mlcod=15'8)", 2024-09-21T15:49:09.039 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:48:51.411258+0000", 2024-09-21T15:49:09.039 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.616528379000002, 2024-09-21T15:49:09.039 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0063623070000000002, 2024-09-21T15:49:09.039 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:09.039 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:09.039 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:09.039 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.039 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:09.039 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.411258+0000", 2024-09-21T15:49:09.039 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:09.039 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.039 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.039 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:09.040 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.411258+0000", 2024-09-21T15:49:09.040 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:09.040 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.040 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:09.040 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.411258+0000", 2024-09-21T15:49:09.040 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:09.040 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.040 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:09.040 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.411364+0000", 2024-09-21T15:49:09.040 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105951 2024-09-21T15:49:09.040 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.040 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:09.040 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.411374+0000", 2024-09-21T15:49:09.041 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0183999999999999e-05 2024-09-21T15:49:09.041 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.041 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.041 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:09.041 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.411380+0000", 2024-09-21T15:49:09.041 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7909999999999999e-06 2024-09-21T15:49:09.041 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.041 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.041 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:09.041 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.411411+0000", 2024-09-21T15:49:09.041 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0309999999999999e-05 2024-09-21T15:49:09.041 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.041 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.041 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:09.041 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.411426+0000", 2024-09-21T15:49:09.041 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5208000000000001e-05 2024-09-21T15:49:09.041 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.042 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.042 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:09.042 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.417585+0000", 2024-09-21T15:49:09.042 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0061587869999999998 2024-09-21T15:49:09.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.042 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.042 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:09.042 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.417621+0000", 2024-09-21T15:49:09.042 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5750000000000002e-05 2024-09-21T15:49:09.042 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:09.042 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:09.042 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:09.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.042 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.042 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4209.0:22 1.0 e15/14 1:30306672:devicehealth::main.db.0000000000000000:head v 15'12, mlcod=15'8)", 2024-09-21T15:49:09.042 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:48:51.411903+0000", 2024-09-21T15:49:09.043 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.615883926999999, 2024-09-21T15:49:09.043 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0057343919999999996, 2024-09-21T15:49:09.043 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:09.043 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:09.043 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:09.043 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.043 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:09.043 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.411903+0000", 2024-09-21T15:49:09.043 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:09.043 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.043 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.043 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:09.043 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.411903+0000", 2024-09-21T15:49:09.043 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:09.043 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.044 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:09.044 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.411902+0000", 2024-09-21T15:49:09.044 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:09.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.044 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:09.044 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.411913+0000", 2024-09-21T15:49:09.044 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1161999999999999e-05 2024-09-21T15:49:09.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.044 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:09.044 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.411921+0000", 2024-09-21T15:49:09.044 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1440000000000003e-06 2024-09-21T15:49:09.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:09.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.411927+0000", 2024-09-21T15:49:09.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5319999999999997e-06 2024-09-21T15:49:09.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:09.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.411962+0000", 2024-09-21T15:49:09.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5539e-05 2024-09-21T15:49:09.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:09.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.411976+0000", 2024-09-21T15:49:09.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3703000000000001e-05 2024-09-21T15:49:09.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.046 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:09.046 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.417623+0000", 2024-09-21T15:49:09.046 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0056472830000000003 2024-09-21T15:49:09.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.046 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:09.046 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.417637+0000", 2024-09-21T15:49:09.046 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3618000000000001e-05 2024-09-21T15:49:09.046 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:09.046 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:09.046 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:09.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.046 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4209.0:39 1.0 e15/14 1:30306672:devicehealth::main.db.0000000000000000:head v 15'25, mlcod=15'22)", 2024-09-21T15:49:09.046 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:48:51.431591+0000", 2024-09-21T15:49:09.047 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.596195656999999, 2024-09-21T15:49:09.047 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029592860000000002, 2024-09-21T15:49:09.047 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:09.047 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:09.047 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:09.047 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.047 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:09.047 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.431591+0000", 2024-09-21T15:49:09.047 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:09.047 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.047 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.047 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:09.047 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.431591+0000", 2024-09-21T15:49:09.047 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:09.047 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.048 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.048 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:09.048 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.431590+0000", 2024-09-21T15:49:09.048 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:09.048 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.048 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.048 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:09.048 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.431678+0000", 2024-09-21T15:49:09.048 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7841000000000001e-05 2024-09-21T15:49:09.048 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.048 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.048 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:09.048 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.431689+0000", 2024-09-21T15:49:09.048 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.084e-05 2024-09-21T15:49:09.048 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.049 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.049 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:09.049 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.431694+0000", 2024-09-21T15:49:09.049 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4820000000000002e-06 2024-09-21T15:49:09.049 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.049 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.049 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:09.049 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.431729+0000", 2024-09-21T15:49:09.049 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4996999999999997e-05 2024-09-21T15:49:09.049 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.049 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.049 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:09.049 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.431743+0000", 2024-09-21T15:49:09.049 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4211e-05 2024-09-21T15:49:09.049 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.050 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.050 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:09.050 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.434519+0000", 2024-09-21T15:49:09.050 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0027758150000000001 2024-09-21T15:49:09.050 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.050 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.050 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:09.050 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.434550+0000", 2024-09-21T15:49:09.050 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1100999999999999e-05 2024-09-21T15:49:09.050 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:09.050 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:09.050 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:09.050 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.050 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.051 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4209.0:47 1.0 e15/14 1:30306672:devicehealth::main.db.0000000000000000:head v 15'29, mlcod=15'27)", 2024-09-21T15:49:09.051 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:48:51.443491+0000", 2024-09-21T15:49:09.051 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.584295441999998, 2024-09-21T15:49:09.051 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033033210000000001, 2024-09-21T15:49:09.051 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:09.051 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:09.051 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:09.051 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.051 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:09.051 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.443491+0000", 2024-09-21T15:49:09.051 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:09.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.051 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.051 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:09.052 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.443491+0000", 2024-09-21T15:49:09.052 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:09.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.052 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.052 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:09.052 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.443490+0000", 2024-09-21T15:49:09.052 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:09.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.052 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.052 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:09.052 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.443497+0000", 2024-09-21T15:49:09.052 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6649999999999997e-06 2024-09-21T15:49:09.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.052 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.052 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:09.052 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.443510+0000", 2024-09-21T15:49:09.053 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2627e-05 2024-09-21T15:49:09.053 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.053 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.053 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:09.053 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.443515+0000", 2024-09-21T15:49:09.053 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7459999999999997e-06 2024-09-21T15:49:09.053 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.053 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.053 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:09.053 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.443546+0000", 2024-09-21T15:49:09.053 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0488999999999999e-05 2024-09-21T15:49:09.053 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.053 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.054 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:09.054 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.443560+0000", 2024-09-21T15:49:09.054 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3869999999999999e-05 2024-09-21T15:49:09.054 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.054 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.054 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:09.054 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.446702+0000", 2024-09-21T15:49:09.054 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0031427640000000001 2024-09-21T15:49:09.054 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.054 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.054 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:09.054 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.446794+0000", 2024-09-21T15:49:09.054 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2009999999999995e-05 2024-09-21T15:49:09.054 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:09.055 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:09.055 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:09.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.055 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.055 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4209.0:53 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'31, mlcod=15'29)", 2024-09-21T15:49:09.055 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:48:51.451017+0000", 2024-09-21T15:49:09.055 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.576769417000001, 2024-09-21T15:49:09.055 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035066849999999998, 2024-09-21T15:49:09.055 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:09.055 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:09.055 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:09.055 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.055 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:09.055 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.451017+0000", 2024-09-21T15:49:09.055 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:09.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.056 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.056 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:09.056 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.451017+0000", 2024-09-21T15:49:09.056 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:09.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.056 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.056 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:09.056 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.451016+0000", 2024-09-21T15:49:09.056 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:09.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.056 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.056 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:09.056 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.451023+0000", 2024-09-21T15:49:09.056 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2770000000000002e-06 2024-09-21T15:49:09.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.057 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:09.057 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.451035+0000", 2024-09-21T15:49:09.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1960000000000001e-05 2024-09-21T15:49:09.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.057 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:09.057 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.451040+0000", 2024-09-21T15:49:09.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6760000000000004e-06 2024-09-21T15:49:09.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.057 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:09.057 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.451083+0000", 2024-09-21T15:49:09.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2413000000000002e-05 2024-09-21T15:49:09.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.058 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:09.058 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.451099+0000", 2024-09-21T15:49:09.058 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6631999999999999e-05 2024-09-21T15:49:09.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.058 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:09.058 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.454498+0000", 2024-09-21T15:49:09.058 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033985700000000001 2024-09-21T15:49:09.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.058 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:09.058 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.454524+0000", 2024-09-21T15:49:09.058 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6013999999999999e-05 2024-09-21T15:49:09.058 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:09.059 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:09.059 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:09.059 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.059 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.059 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4209.0:54 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'32, mlcod=15'29)", 2024-09-21T15:49:09.059 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:48:51.451400+0000", 2024-09-21T15:49:09.059 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.576386393, 2024-09-21T15:49:09.059 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0031372370000000002, 2024-09-21T15:49:09.059 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:09.059 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:09.059 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:09.059 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.059 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:09.059 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.451400+0000", 2024-09-21T15:49:09.059 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:09.060 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.060 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.060 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:09.060 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.451400+0000", 2024-09-21T15:49:09.060 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:09.060 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.060 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.060 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:09.060 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.451399+0000", 2024-09-21T15:49:09.060 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:09.060 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.060 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.060 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:09.060 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.451408+0000", 2024-09-21T15:49:09.060 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.2050000000000002e-06 2024-09-21T15:49:09.060 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.061 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:09.061 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.451420+0000", 2024-09-21T15:49:09.061 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2748000000000001e-05 2024-09-21T15:49:09.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.061 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:09.061 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.451426+0000", 2024-09-21T15:49:09.061 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1369999999999999e-06 2024-09-21T15:49:09.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.061 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:09.061 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.451454+0000", 2024-09-21T15:49:09.061 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7932e-05 2024-09-21T15:49:09.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.062 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:09.062 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.451465+0000", 2024-09-21T15:49:09.062 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.114e-05 2024-09-21T15:49:09.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.062 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:09.062 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.454526+0000", 2024-09-21T15:49:09.062 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0030604439999999998 2024-09-21T15:49:09.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.062 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:09.062 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.454537+0000", 2024-09-21T15:49:09.062 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1517e-05 2024-09-21T15:49:09.063 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:09.063 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:09.063 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:09.063 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.063 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.063 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4209.0:55 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'33, mlcod=15'29)", 2024-09-21T15:49:09.063 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:48:51.451961+0000", 2024-09-21T15:49:09.063 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.575825349999999, 2024-09-21T15:49:09.063 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053240259999999999, 2024-09-21T15:49:09.063 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:09.063 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:09.063 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:09.063 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.063 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:09.063 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.451961+0000", 2024-09-21T15:49:09.064 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:09.064 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.064 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.064 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:09.064 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.451961+0000", 2024-09-21T15:49:09.064 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:09.064 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.064 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.064 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:09.064 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.451961+0000", 2024-09-21T15:49:09.064 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:09.064 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.064 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.064 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:09.065 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.452035+0000", 2024-09-21T15:49:09.065 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4844000000000004e-05 2024-09-21T15:49:09.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.065 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:09.065 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.452045+0000", 2024-09-21T15:49:09.065 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5750000000000007e-06 2024-09-21T15:49:09.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.065 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:09.065 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.452050+0000", 2024-09-21T15:49:09.065 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5549999999999996e-06 2024-09-21T15:49:09.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.065 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:09.066 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.452083+0000", 2024-09-21T15:49:09.066 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3152999999999999e-05 2024-09-21T15:49:09.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.066 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.066 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:09.066 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.452098+0000", 2024-09-21T15:49:09.066 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5469999999999999e-05 2024-09-21T15:49:09.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.066 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.066 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:09.066 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.457252+0000", 2024-09-21T15:49:09.066 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0051533949999999999 2024-09-21T15:49:09.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.066 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.067 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:09.067 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.457285+0000", 2024-09-21T15:49:09.067 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3685e-05 2024-09-21T15:49:09.067 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:09.067 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:09.067 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:09.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.067 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4209.0:56 1.0 e15/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 15'34, mlcod=15'29)", 2024-09-21T15:49:09.067 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:48:51.452539+0000", 2024-09-21T15:49:09.067 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.575247519000001, 2024-09-21T15:49:09.067 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0047685999999999996, 2024-09-21T15:49:09.067 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:09.067 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:09.068 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:09.068 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.068 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:09.068 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.452539+0000", 2024-09-21T15:49:09.068 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:09.068 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.068 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.068 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:09.068 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.452539+0000", 2024-09-21T15:49:09.068 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:09.068 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.068 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.068 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:09.068 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.452538+0000", 2024-09-21T15:49:09.068 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:09.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.069 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:09.069 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.452545+0000", 2024-09-21T15:49:09.069 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8600000000000004e-06 2024-09-21T15:49:09.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.069 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:09.069 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.452558+0000", 2024-09-21T15:49:09.069 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3162999999999999e-05 2024-09-21T15:49:09.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.069 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:09.069 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.452564+0000", 2024-09-21T15:49:09.069 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4360000000000001e-06 2024-09-21T15:49:09.070 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.070 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:09.070 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.454259+0000", 2024-09-21T15:49:09.070 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0016953140000000001 2024-09-21T15:49:09.070 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.070 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:09.070 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.454273+0000", 2024-09-21T15:49:09.070 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4358e-05 2024-09-21T15:49:09.070 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.070 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:09.070 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.457287+0000", 2024-09-21T15:49:09.071 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0030136669999999998 2024-09-21T15:49:09.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.071 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:09.071 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.457308+0000", 2024-09-21T15:49:09.071 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0744999999999999e-05 2024-09-21T15:49:09.071 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:09.071 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:09.071 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:09.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.071 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4209.0:58 1.0 e15/14 1:30306672:devicehealth::main.db.0000000000000000:head v 15'36, mlcod=15'33)", 2024-09-21T15:49:09.071 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:48:51.458953+0000", 2024-09-21T15:49:09.071 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.568833436999999, 2024-09-21T15:49:09.072 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0061908639999999999, 2024-09-21T15:49:09.072 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:09.072 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:09.072 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:09.072 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.072 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:09.072 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.458953+0000", 2024-09-21T15:49:09.072 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:09.072 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.072 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.072 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:09.072 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.458953+0000", 2024-09-21T15:49:09.072 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:09.072 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.072 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.073 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:09.073 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.458953+0000", 2024-09-21T15:49:09.073 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:09.073 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.073 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.073 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:09.073 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.459036+0000", 2024-09-21T15:49:09.073 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3861000000000002e-05 2024-09-21T15:49:09.073 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.073 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.073 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:09.073 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.459046+0000", 2024-09-21T15:49:09.073 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9510000000000001e-06 2024-09-21T15:49:09.073 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.074 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.074 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:09.074 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.459052+0000", 2024-09-21T15:49:09.074 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0850000000000002e-06 2024-09-21T15:49:09.074 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.074 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.074 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:09.074 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.459082+0000", 2024-09-21T15:49:09.074 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9425e-05 2024-09-21T15:49:09.074 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.074 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.074 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:09.074 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.459093+0000", 2024-09-21T15:49:09.074 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1447999999999999e-05 2024-09-21T15:49:09.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:09.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.465112+0000", 2024-09-21T15:49:09.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0060189049999999997 2024-09-21T15:49:09.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:09.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.465144+0000", 2024-09-21T15:49:09.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1829000000000003e-05 2024-09-21T15:49:09.075 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:09.075 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:09.075 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:09.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.076 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4209.0:59 1.0 e15/14 1:30306672:devicehealth::main.db.0000000000000000:head v 15'37, mlcod=15'33)", 2024-09-21T15:49:09.076 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:48:51.459396+0000", 2024-09-21T15:49:09.076 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.568390750999999, 2024-09-21T15:49:09.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0057630119999999996, 2024-09-21T15:49:09.076 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:09.076 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:09.076 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:09.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.076 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:09.076 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.459396+0000", 2024-09-21T15:49:09.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:09.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.076 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:09.076 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.459396+0000", 2024-09-21T15:49:09.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:09.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.077 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.077 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:09.077 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.459395+0000", 2024-09-21T15:49:09.077 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:09.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.077 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.077 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:09.077 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.459405+0000", 2024-09-21T15:49:09.077 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0086e-05 2024-09-21T15:49:09.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.077 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.077 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:09.077 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.459413+0000", 2024-09-21T15:49:09.077 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8560000000000007e-06 2024-09-21T15:49:09.078 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.078 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:09.078 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.459419+0000", 2024-09-21T15:49:09.078 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2630000000000003e-06 2024-09-21T15:49:09.078 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.078 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:09.078 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.459447+0000", 2024-09-21T15:49:09.078 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8065e-05 2024-09-21T15:49:09.078 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.078 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:09.078 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.459462+0000", 2024-09-21T15:49:09.079 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5091e-05 2024-09-21T15:49:09.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.079 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.079 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:09.079 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.465146+0000", 2024-09-21T15:49:09.079 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0056848150000000002 2024-09-21T15:49:09.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.079 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.079 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:09.079 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:51.465159+0000", 2024-09-21T15:49:09.079 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2381999999999999e-05 2024-09-21T15:49:09.079 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:09.079 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:09.079 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:09.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.080 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4209.0:154 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'108, mlcod=15'105)", 2024-09-21T15:49:09.080 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:48:52.418201+0000", 2024-09-21T15:49:09.080 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 16.609585783, 2024-09-21T15:49:09.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0041530610000000004, 2024-09-21T15:49:09.080 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:09.080 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:09.080 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:09.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.080 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:09.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.418201+0000", 2024-09-21T15:49:09.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:09.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.080 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:09.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.418201+0000", 2024-09-21T15:49:09.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:09.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.081 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:09.081 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.418200+0000", 2024-09-21T15:49:09.081 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:09.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.081 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:09.081 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.418206+0000", 2024-09-21T15:49:09.081 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.288e-06 2024-09-21T15:49:09.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.081 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:09.081 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.418216+0000", 2024-09-21T15:49:09.081 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9539999999999999e-06 2024-09-21T15:49:09.082 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.082 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.082 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:09.082 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.418223+0000", 2024-09-21T15:49:09.082 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4710000000000001e-06 2024-09-21T15:49:09.082 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.082 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.082 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:09.082 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.418250+0000", 2024-09-21T15:49:09.082 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7195999999999998e-05 2024-09-21T15:49:09.082 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.082 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.082 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:09.082 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.418266+0000", 2024-09-21T15:49:09.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5840000000000001e-05 2024-09-21T15:49:09.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.083 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:09.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.422329+0000", 2024-09-21T15:49:09.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.004063316 2024-09-21T15:49:09.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.083 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:09.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.422354+0000", 2024-09-21T15:49:09.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4496000000000001e-05 2024-09-21T15:49:09.083 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:09.083 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:09.083 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:09.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.084 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4209.0:155 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'109, mlcod=15'105)", 2024-09-21T15:49:09.084 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:48:52.418672+0000", 2024-09-21T15:49:09.084 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 16.609114626, 2024-09-21T15:49:09.084 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0037005530000000001, 2024-09-21T15:49:09.084 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:09.084 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:09.084 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:09.085 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.085 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:09.085 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.418672+0000", 2024-09-21T15:49:09.085 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:09.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.085 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.085 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:09.086 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.418672+0000", 2024-09-21T15:49:09.086 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:09.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.086 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:09.086 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.418672+0000", 2024-09-21T15:49:09.086 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:09.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.086 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:09.086 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.418759+0000", 2024-09-21T15:49:09.086 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7096999999999996e-05 2024-09-21T15:49:09.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.087 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.087 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:09.087 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.418766+0000", 2024-09-21T15:49:09.087 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4100000000000002e-06 2024-09-21T15:49:09.087 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.087 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.087 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:09.087 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.418771+0000", 2024-09-21T15:49:09.087 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7999999999999998e-06 2024-09-21T15:49:09.087 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.087 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.087 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:09.087 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.418793+0000", 2024-09-21T15:49:09.087 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2178e-05 2024-09-21T15:49:09.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.088 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:09.088 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.418804+0000", 2024-09-21T15:49:09.088 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.046e-05 2024-09-21T15:49:09.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.088 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:09.088 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.422357+0000", 2024-09-21T15:49:09.088 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035532139999999999 2024-09-21T15:49:09.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.088 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:09.088 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.422373+0000", 2024-09-21T15:49:09.088 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5790999999999998e-05 2024-09-21T15:49:09.089 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:09.089 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:09.089 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:09.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.089 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.089 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4209.0:156 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'110, mlcod=15'105)", 2024-09-21T15:49:09.089 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:48:52.419389+0000", 2024-09-21T15:49:09.089 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 16.608397941, 2024-09-21T15:49:09.089 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0034227469999999999, 2024-09-21T15:49:09.089 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:09.089 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:09.089 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:09.089 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.089 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:09.089 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.419389+0000", 2024-09-21T15:49:09.090 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:09.090 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.090 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.090 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:09.090 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.419389+0000", 2024-09-21T15:49:09.090 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:09.090 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.090 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.090 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:09.090 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.419388+0000", 2024-09-21T15:49:09.090 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:09.090 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.090 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.090 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:09.090 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.419394+0000", 2024-09-21T15:49:09.091 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2049999999999999e-06 2024-09-21T15:49:09.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.091 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:09.091 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.419401+0000", 2024-09-21T15:49:09.091 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2049999999999999e-06 2024-09-21T15:49:09.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.091 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:09.091 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.419405+0000", 2024-09-21T15:49:09.091 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6489999999999999e-06 2024-09-21T15:49:09.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.091 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:09.091 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.421776+0000", 2024-09-21T15:49:09.092 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0023709199999999999 2024-09-21T15:49:09.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.092 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:09.092 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.421795+0000", 2024-09-21T15:49:09.092 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9060999999999999e-05 2024-09-21T15:49:09.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.092 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:09.092 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.422794+0000", 2024-09-21T15:49:09.092 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00099828099999999995 2024-09-21T15:49:09.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.092 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:09.092 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.422811+0000", 2024-09-21T15:49:09.093 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7856999999999999e-05 2024-09-21T15:49:09.093 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:09.093 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:09.093 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:09.093 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.093 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.093 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4209.0:157 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'111, mlcod=16'106)", 2024-09-21T15:49:09.093 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:48:52.419859+0000", 2024-09-21T15:49:09.093 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 16.607927926999999, 2024-09-21T15:49:09.093 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029733989999999998, 2024-09-21T15:49:09.093 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:09.093 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:09.093 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:09.093 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.093 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:09.093 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.419859+0000", 2024-09-21T15:49:09.094 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:09.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.094 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.094 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:09.094 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.419859+0000", 2024-09-21T15:49:09.094 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:09.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.094 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.094 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:09.094 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.419858+0000", 2024-09-21T15:49:09.094 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:09.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.094 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.094 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:09.095 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.419864+0000", 2024-09-21T15:49:09.095 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9479999999999996e-06 2024-09-21T15:49:09.095 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.095 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:09.095 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.419871+0000", 2024-09-21T15:49:09.095 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4150000000000004e-06 2024-09-21T15:49:09.095 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.095 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:09.095 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.419876+0000", 2024-09-21T15:49:09.095 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.994e-06 2024-09-21T15:49:09.095 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.095 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:09.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.422123+0000", 2024-09-21T15:49:09.096 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0022471499999999998 2024-09-21T15:49:09.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:09.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.422134+0000", 2024-09-21T15:49:09.096 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1323e-05 2024-09-21T15:49:09.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:09.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.422815+0000", 2024-09-21T15:49:09.096 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00068099299999999996 2024-09-21T15:49:09.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:09.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.422832+0000", 2024-09-21T15:49:09.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7017999999999999e-05 2024-09-21T15:49:09.097 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:09.097 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:09.097 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:09.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.097 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4209.0:158 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'112, mlcod=16'106)", 2024-09-21T15:49:09.097 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:48:52.420312+0000", 2024-09-21T15:49:09.097 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 16.607474186000001, 2024-09-21T15:49:09.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0094451629999999995, 2024-09-21T15:49:09.097 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:09.097 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:09.097 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:09.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.097 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:09.098 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.420312+0000", 2024-09-21T15:49:09.098 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:09.098 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:09.098 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.420312+0000", 2024-09-21T15:49:09.098 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:09.098 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:09.098 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.420312+0000", 2024-09-21T15:49:09.098 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:09.098 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.099 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:09.099 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.420385+0000", 2024-09-21T15:49:09.099 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3291e-05 2024-09-21T15:49:09.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.099 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:09.099 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.420391+0000", 2024-09-21T15:49:09.099 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.789e-06 2024-09-21T15:49:09.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.099 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:09.099 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.420395+0000", 2024-09-21T15:49:09.099 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2540000000000003e-06 2024-09-21T15:49:09.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.100 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:09.100 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.422395+0000", 2024-09-21T15:49:09.100 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0019992230000000001 2024-09-21T15:49:09.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.100 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:09.100 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.422416+0000", 2024-09-21T15:49:09.100 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0922999999999999e-05 2024-09-21T15:49:09.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.100 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:09.100 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.429675+0000", 2024-09-21T15:49:09.100 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0072597419999999996 2024-09-21T15:49:09.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.101 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:09.101 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.429758+0000", 2024-09-21T15:49:09.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.2294999999999997e-05 2024-09-21T15:49:09.101 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:09.101 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:09.101 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:09.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.101 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4209.0:159 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'113, mlcod=16'106)", 2024-09-21T15:49:09.101 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:48:52.422413+0000", 2024-09-21T15:49:09.101 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 16.605373615000001, 2024-09-21T15:49:09.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0077345039999999997, 2024-09-21T15:49:09.101 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:09.102 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:09.102 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:09.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.102 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:09.102 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.422413+0000", 2024-09-21T15:49:09.102 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:09.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.102 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:09.102 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.422413+0000", 2024-09-21T15:49:09.102 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:09.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.102 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:09.102 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.422413+0000", 2024-09-21T15:49:09.103 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:09.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.103 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:09.103 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.422419+0000", 2024-09-21T15:49:09.103 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3160000000000002e-06 2024-09-21T15:49:09.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.103 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:09.103 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.422430+0000", 2024-09-21T15:49:09.103 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1209999999999999e-05 2024-09-21T15:49:09.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.103 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:09.104 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.422436+0000", 2024-09-21T15:49:09.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0129999999999999e-06 2024-09-21T15:49:09.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.104 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:09.104 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.422852+0000", 2024-09-21T15:49:09.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00041611000000000003 2024-09-21T15:49:09.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.104 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:09.104 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.422872+0000", 2024-09-21T15:49:09.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9412999999999999e-05 2024-09-21T15:49:09.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.104 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:09.105 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.430120+0000", 2024-09-21T15:49:09.105 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0072488270000000002 2024-09-21T15:49:09.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.105 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:09.105 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.430147+0000", 2024-09-21T15:49:09.105 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7106000000000001e-05 2024-09-21T15:49:09.105 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:09.105 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:09.105 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:09.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.105 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4209.0:162 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'116, mlcod=16'114)", 2024-09-21T15:49:09.105 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:48:52.433559+0000", 2024-09-21T15:49:09.106 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 16.594227895, 2024-09-21T15:49:09.106 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052075369999999999, 2024-09-21T15:49:09.106 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:09.106 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:09.106 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:09.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.106 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:09.106 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.433559+0000", 2024-09-21T15:49:09.106 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:09.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.106 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:09.106 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.433559+0000", 2024-09-21T15:49:09.106 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:09.107 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.107 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.107 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:09.107 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.433558+0000", 2024-09-21T15:49:09.107 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:09.107 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.107 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.107 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:09.107 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.433564+0000", 2024-09-21T15:49:09.107 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7420000000000003e-06 2024-09-21T15:49:09.107 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.107 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.107 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:09.108 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.433572+0000", 2024-09-21T15:49:09.108 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1790000000000008e-06 2024-09-21T15:49:09.108 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.108 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.108 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:09.108 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.433577+0000", 2024-09-21T15:49:09.108 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6589999999999998e-06 2024-09-21T15:49:09.108 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.108 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.108 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:09.108 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.433605+0000", 2024-09-21T15:49:09.108 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8589999999999999e-05 2024-09-21T15:49:09.108 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.108 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.108 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:09.109 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.433618+0000", 2024-09-21T15:49:09.109 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2964e-05 2024-09-21T15:49:09.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.109 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:09.109 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.438740+0000", 2024-09-21T15:49:09.109 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0051220750000000002 2024-09-21T15:49:09.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.109 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:09.109 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:52.438766+0000", 2024-09-21T15:49:09.109 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5950000000000001e-05 2024-09-21T15:49:09.109 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:09.109 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:09.109 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:09.110 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.110 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.110 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:11 3.1f 3:fa654f18:::rbd_id.image_client.0:head [create,call rbd.set_id in=9b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-21T15:49:09.110 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:08.700822+0000", 2024-09-21T15:49:09.110 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.326964896, 2024-09-21T15:49:09.110 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0028783049999999998, 2024-09-21T15:49:09.110 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:09.110 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:09.110 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:09.110 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:09.110 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:09.110 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 11 2024-09-21T15:49:09.110 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.111 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:09.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.111 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:09.111 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:08.700822+0000", 2024-09-21T15:49:09.111 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:09.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.111 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:09.111 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:08.700822+0000", 2024-09-21T15:49:09.111 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:09.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.111 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:09.111 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:08.700820+0000", 2024-09-21T15:49:09.112 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:09.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.112 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:09.112 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:08.700831+0000", 2024-09-21T15:49:09.112 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0522e-05 2024-09-21T15:49:09.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.112 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:09.112 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:08.700845+0000", 2024-09-21T15:49:09.112 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4042e-05 2024-09-21T15:49:09.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.112 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:09.112 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:08.700854+0000", 2024-09-21T15:49:09.113 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0920000000000004e-06 2024-09-21T15:49:09.113 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.113 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:09.113 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:08.700895+0000", 2024-09-21T15:49:09.113 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1624e-05 2024-09-21T15:49:09.113 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.113 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op must wait for map", 2024-09-21T15:49:09.113 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:08.700899+0000", 2024-09-21T15:49:09.113 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0430000000000001e-06 2024-09-21T15:49:09.113 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.113 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:09.113 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:08.702449+0000", 2024-09-21T15:49:09.114 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0015495540000000001 2024-09-21T15:49:09.114 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.114 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:09.114 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:08.702578+0000", 2024-09-21T15:49:09.114 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012897399999999999 2024-09-21T15:49:09.114 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.114 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-09-21T15:49:09.114 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:08.702734+0000", 2024-09-21T15:49:09.114 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000156358 2024-09-21T15:49:09.114 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.114 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:09.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:08.703531+0000", 2024-09-21T15:49:09.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00079675099999999995 2024-09-21T15:49:09.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:09.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:08.703629+0000", 2024-09-21T15:49:09.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7663000000000003e-05 2024-09-21T15:49:09.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:09.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:08.703662+0000", 2024-09-21T15:49:09.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3367000000000001e-05 2024-09-21T15:49:09.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.116 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:09.116 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:08.703700+0000", 2024-09-21T15:49:09.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7922e-05 2024-09-21T15:49:09.116 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:09.116 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:09.116 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:09.116 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:09.116 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:09.116 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:09.117 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:49:09.118 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:09.116+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:09.218 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:09.217+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:09.262 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:09.262 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:49:09.262 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:49:09.263 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:09.271 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:49:09.319 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:09.318+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:09.374 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:09.374 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:49:09.374 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:49:09.374 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:49:09.375 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:09.383 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:49:09.427 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:09.426+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:09.486 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:09.487 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:49:09.487 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:49:09.487 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:49:09.487 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.487 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4275.0:5 2.2 e19/17 2:5cea7035:::rbd_info:head v 19'1, mlcod=0'0)", 2024-09-21T15:49:09.487 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:48:58.590679+0000", 2024-09-21T15:49:09.487 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.895439099000001, 2024-09-21T15:49:09.487 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00080136699999999999, 2024-09-21T15:49:09.487 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:09.487 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:09.487 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:09.487 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.487 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:09.488 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:58.590679+0000", 2024-09-21T15:49:09.488 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:09.488 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.488 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.488 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:09.488 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:58.590679+0000", 2024-09-21T15:49:09.488 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:09.488 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.488 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.488 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:09.488 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:58.590679+0000", 2024-09-21T15:49:09.488 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:49:09.488 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.489 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.489 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:09.489 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:58.590691+0000", 2024-09-21T15:49:09.489 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2768999999999999e-05 2024-09-21T15:49:09.489 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.489 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.489 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:09.489 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:58.590704+0000", 2024-09-21T15:49:09.489 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2981e-05 2024-09-21T15:49:09.489 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.489 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.489 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:09.489 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:58.590714+0000", 2024-09-21T15:49:09.489 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6010000000000001e-06 2024-09-21T15:49:09.490 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.490 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.490 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:09.490 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:58.590773+0000", 2024-09-21T15:49:09.490 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9284999999999999e-05 2024-09-21T15:49:09.490 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.490 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.490 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:09.490 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:58.590798+0000", 2024-09-21T15:49:09.490 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5109999999999998e-05 2024-09-21T15:49:09.490 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.490 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.490 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:09.491 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:58.591449+0000", 2024-09-21T15:49:09.491 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00065082599999999997 2024-09-21T15:49:09.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.491 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:09.491 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:58.591480+0000", 2024-09-21T15:49:09.491 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0976000000000003e-05 2024-09-21T15:49:09.491 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:09.491 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:09.491 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:09.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.491 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4275.0:7 2.2 e20/17 2:5cea7035:::rbd_info:head v 20'2, mlcod=0'0)", 2024-09-21T15:49:09.492 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:48:59.595176+0000", 2024-09-21T15:49:09.492 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.89094199, 2024-09-21T15:49:09.492 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00079370899999999997, 2024-09-21T15:49:09.492 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:09.492 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:09.492 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:09.492 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.492 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:09.492 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:59.595176+0000", 2024-09-21T15:49:09.492 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:09.492 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.492 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.493 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:09.493 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:59.595176+0000", 2024-09-21T15:49:09.493 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:09.493 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.493 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.493 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:09.493 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:59.595175+0000", 2024-09-21T15:49:09.493 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:09.493 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.493 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.493 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:09.493 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:59.595185+0000", 2024-09-21T15:49:09.493 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6560000000000004e-06 2024-09-21T15:49:09.494 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.494 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.494 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:09.494 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:59.595202+0000", 2024-09-21T15:49:09.494 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6642000000000001e-05 2024-09-21T15:49:09.494 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.494 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.494 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:09.494 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:59.595213+0000", 2024-09-21T15:49:09.494 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1092e-05 2024-09-21T15:49:09.494 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.494 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.494 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:09.495 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:59.595269+0000", 2024-09-21T15:49:09.495 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5847999999999998e-05 2024-09-21T15:49:09.495 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.495 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.495 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:09.495 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:59.595293+0000", 2024-09-21T15:49:09.495 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4476000000000001e-05 2024-09-21T15:49:09.495 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.495 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.495 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:09.495 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:59.595902+0000", 2024-09-21T15:49:09.495 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00060874099999999999 2024-09-21T15:49:09.495 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.495 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.496 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:09.496 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:59.595970+0000", 2024-09-21T15:49:09.496 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7618000000000002e-05 2024-09-21T15:49:09.496 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:09.496 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:09.496 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:09.496 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.496 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.496 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:9 3.1a e25/22 3:5cea7035:::rbd_info:head v 25'2, mlcod=0'0)", 2024-09-21T15:49:09.496 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:08.695923+0000", 2024-09-21T15:49:09.496 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.79019443899999997, 2024-09-21T15:49:09.496 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001386878, 2024-09-21T15:49:09.496 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:09.496 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:09.496 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:09.497 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.497 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:09.497 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:08.695923+0000", 2024-09-21T15:49:09.497 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:09.497 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.497 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.497 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:09.497 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:08.695923+0000", 2024-09-21T15:49:09.497 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:09.497 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.497 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.497 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:09.497 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:08.695923+0000", 2024-09-21T15:49:09.497 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:49:09.498 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.498 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.498 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:09.498 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:08.695930+0000", 2024-09-21T15:49:09.498 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7939999999999997e-06 2024-09-21T15:49:09.498 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.498 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.498 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:09.498 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:08.695948+0000", 2024-09-21T15:49:09.498 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7957000000000001e-05 2024-09-21T15:49:09.498 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.499 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.499 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:09.499 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:08.695958+0000", 2024-09-21T15:49:09.499 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0162e-05 2024-09-21T15:49:09.499 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.499 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.499 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:09.499 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:08.696010+0000", 2024-09-21T15:49:09.499 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1604000000000003e-05 2024-09-21T15:49:09.499 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.499 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.499 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:09.499 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:08.696027+0000", 2024-09-21T15:49:09.500 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7176e-05 2024-09-21T15:49:09.500 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.500 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.500 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:09.500 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:08.697280+0000", 2024-09-21T15:49:09.500 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001253079 2024-09-21T15:49:09.500 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.500 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.500 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:09.500 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:08.697310+0000", 2024-09-21T15:49:09.500 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0335999999999999e-05 2024-09-21T15:49:09.500 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:09.500 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:09.500 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:09.501 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:09.501 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:09.501 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:09.502 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:49:09.543 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:09.542+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:09.621 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:09.620+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:09.646 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:09.646 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:49:09.646 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:49:09.646 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:09.655 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:49:09.693 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is set 2024-09-21T15:49:09.747 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:09.746+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:09.797 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:09.797 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:49:09.797 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:49:09.797 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:49:09.797 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:09.806 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:49:09.847 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:09.846+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:09.923 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:09.921+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:09.950 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:09.950 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:49:09.951 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:49:09.951 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:49:09.951 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.951 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4275.0:1 2.3 e18/17 2:c4c92e5a:::rbd_trash:head v 18'1, mlcod=0'0)", 2024-09-21T15:49:09.951 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:48:57.586388+0000", 2024-09-21T15:49:09.951 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.363879461, 2024-09-21T15:49:09.951 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00075578400000000001, 2024-09-21T15:49:09.951 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:09.951 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:09.951 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:09.951 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.951 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:09.951 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:57.586388+0000", 2024-09-21T15:49:09.951 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:09.952 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.952 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.952 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:09.952 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:57.586388+0000", 2024-09-21T15:49:09.952 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:09.952 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.952 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.952 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:09.952 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:57.586388+0000", 2024-09-21T15:49:09.952 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:49:09.952 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.952 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.952 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:09.952 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:57.586396+0000", 2024-09-21T15:49:09.952 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4339999999999992e-06 2024-09-21T15:49:09.952 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.953 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.953 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:09.953 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:57.586406+0000", 2024-09-21T15:49:09.953 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9639999999999998e-06 2024-09-21T15:49:09.953 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.953 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.953 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:09.953 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:57.586423+0000", 2024-09-21T15:49:09.953 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7005000000000001e-05 2024-09-21T15:49:09.953 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.953 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.953 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:09.953 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:57.586459+0000", 2024-09-21T15:49:09.953 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5828999999999999e-05 2024-09-21T15:49:09.954 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.954 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.954 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:09.954 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:57.586482+0000", 2024-09-21T15:49:09.954 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2841e-05 2024-09-21T15:49:09.954 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.954 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.954 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:09.954 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:57.587111+0000", 2024-09-21T15:49:09.954 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00062914999999999998 2024-09-21T15:49:09.954 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:09.954 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:09.954 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:09.954 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:57.587144+0000", 2024-09-21T15:49:09.954 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2790999999999998e-05 2024-09-21T15:49:09.955 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:09.955 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:09.955 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:09.955 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:09.955 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:09.955 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:09.959 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:49:09.995 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:49:10.043 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:10.042+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:10.107 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:10.107 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:49:10.107 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:49:10.107 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:10.116 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:49:10.122 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-09-21T15:49:10.122 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:49:10.124 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:10.124+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:10.139 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:49:10.139 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:49:10.155 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:49:10.155 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:49:10.171 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:49:10.171 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:49:10.188 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:49:10.188 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:49:10.204 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:49:10.205 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:49:10.221 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:49:10.221 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:49:10.225 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:10.224+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:10.226 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:10.226 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:10.235 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:49:10.238 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:49:10.238 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:49:10.326 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:10.325+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:10.340 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:10.340 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:10.350 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:49:10.427 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:10.426+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:10.443 INFO:teuthology.orchestra.run.smithi089.stdout:skipping zero length punch hole 2024-09-21T15:49:10.443 INFO:teuthology.orchestra.run.smithi089.stdout:skipping zero size read 2024-09-21T15:49:10.459 INFO:teuthology.orchestra.run.smithi089.stdout:3 write 0x7f78322 thru 0x7f78386 (0x65 bytes) 2024-09-21T15:49:10.468 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:10.468 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:10.477 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:49:10.528 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:10.527+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:10.570 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T15:49:10.570 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T15:49:10.570 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-09-21T15:49:10.570 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T15:49:10.570 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T15:49:10.571 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok config set filestore_inject_stall 3 2024-09-21T15:49:10.620 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:10.620 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:10.628 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:49:10.629 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:10.627+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:10.677 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:10.677 INFO:teuthology.orchestra.run.smithi089.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T15:49:10.677 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:10.681 INFO:teuthology.orchestra.run.smithi089.stdout:5 writesame 0xd671c89 thru 0xd67931b data_size 0xd(0x7693 bytes) 2024-09-21T15:49:10.681 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:10.681 INFO:teuthology.orchestra.run.smithi089.stdout:7 write 0x63000ff thru 0x630e368 (0xe26a bytes) 2024-09-21T15:49:10.684 INFO:teuthology.orchestra.run.smithi089.stdout:8 punch from 0x668265a to 0x6682804, (0x1aa bytes) 2024-09-21T15:49:10.684 INFO:teuthology.orchestra.run.smithi089.stdout:10 punch from 0x9137cdb to 0x9144fad, (0xd2d2 bytes) 2024-09-21T15:49:10.684 INFO:teuthology.orchestra.run.smithi089.stdout:11 punch from 0x5021cf4 to 0x502775f, (0x5a6b bytes) 2024-09-21T15:49:10.691 INFO:teuthology.orchestra.run.smithi089.stdout:14 write 0xede7029 thru 0xede72b7 (0x28f bytes) 2024-09-21T15:49:10.697 INFO:teuthology.orchestra.run.smithi089.stdout:15 writesame 0x1838a95 thru 0x183e60b data_size 0x23(0x5b77 bytes) 2024-09-21T15:49:10.700 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:10.700 INFO:teuthology.orchestra.run.smithi089.stdout:18 writesame 0x6471756 thru 0x6471af8 data_size 0x7(0x3a3 bytes) 2024-09-21T15:49:10.702 INFO:teuthology.orchestra.run.smithi089.stdout:19 punch from 0xb366c62 to 0xb37393a, (0xccd8 bytes) 2024-09-21T15:49:10.703 INFO:teuthology.orchestra.run.smithi089.stdout:20 writesame 0x46a7b3a thru 0x46a881c data_size 0x1(0xce3 bytes) 2024-09-21T15:49:10.706 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:10.706 INFO:teuthology.orchestra.run.smithi089.stdout:truncating to largest ever: 0x37d10f4 2024-09-21T15:49:10.719 INFO:teuthology.orchestra.run.smithi089.stdout:26 trunc from 0xede72b8 to 0x37d10f4 2024-09-21T15:49:10.729 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:10.728+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:10.735 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:10.735 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:10.737 INFO:teuthology.orchestra.run.smithi089.stdout:30 read 0x3498a44 thru 0x34a5334 (0xc8f1 bytes) 2024-09-21T15:49:10.737 INFO:teuthology.orchestra.run.smithi089.stdout:31 read 0x223d0fc thru 0x2241d3c (0x4c41 bytes) 2024-09-21T15:49:10.737 INFO:teuthology.orchestra.run.smithi089.stdout:32 read 0xe6238d thru 0xe634a2 (0x1116 bytes) 2024-09-21T15:49:10.737 INFO:teuthology.orchestra.run.smithi089.stdout:34 punch from 0x199bc66 to 0x199e94a, (0x2ce4 bytes) 2024-09-21T15:49:10.744 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:49:10.749 INFO:teuthology.orchestra.run.smithi089.stdout:35 write 0x814236f thru 0x81497aa (0x743c bytes) 2024-09-21T15:49:10.759 INFO:teuthology.orchestra.run.smithi089.stdout:36 trunc from 0x81497ab to 0x1d9f266 2024-09-21T15:49:10.788 INFO:teuthology.orchestra.run.smithi089.stdout:37 write 0xccb65f0 thru 0xccb93a9 (0x2dba bytes) 2024-09-21T15:49:10.791 INFO:teuthology.orchestra.run.smithi089.stdout:truncating to largest ever: 0x7a44e60 2024-09-21T15:49:10.797 INFO:teuthology.orchestra.run.smithi089.stdout:39 trunc from 0xccb93aa to 0x7a44e60 2024-09-21T15:49:10.808 INFO:teuthology.orchestra.run.smithi089.stdout:truncating to largest ever: 0xb726bda 2024-09-21T15:49:10.813 INFO:teuthology.orchestra.run.smithi089.stdout:40 trunc from 0x7a44e60 to 0xb726bda 2024-09-21T15:49:10.820 INFO:teuthology.orchestra.run.smithi089.stdout:41 read 0x72acb46 thru 0x72b3ae9 (0x6fa4 bytes) 2024-09-21T15:49:10.820 INFO:teuthology.orchestra.run.smithi089.stdout:42 punch from 0x88f689e to 0x8901094, (0xa7f6 bytes) 2024-09-21T15:49:10.820 INFO:teuthology.orchestra.run.smithi089.stdout:44 write 0xb07be66 thru 0xb08a9cf (0xeb6a bytes) 2024-09-21T15:49:10.823 INFO:teuthology.orchestra.run.smithi089.stdout:47 read 0x3d02936 thru 0x3d02fa3 (0x66e bytes) 2024-09-21T15:49:10.830 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:10.829+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:10.834 INFO:teuthology.orchestra.run.smithi089.stdout:49 trunc from 0xb726bda to 0x189aadd 2024-09-21T15:49:10.845 INFO:teuthology.orchestra.run.smithi089.stdout:51 punch from 0x121d694 to 0x121e932, (0x129e bytes) 2024-09-21T15:49:10.846 INFO:teuthology.orchestra.run.smithi089.stdout:52 punch from 0xd9b796 to 0xdaadf1, (0xf65b bytes) 2024-09-21T15:49:10.846 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:10.847 INFO:teuthology.orchestra.run.smithi089.stdout:54 trunc from 0x189aadd to 0x4cdbc3 2024-09-21T15:49:10.858 INFO:teuthology.orchestra.run.smithi089.stdout:55 read 0x17ea2b thru 0x180582 (0x1b58 bytes) 2024-09-21T15:49:10.858 INFO:teuthology.orchestra.run.smithi089.stdout:56 read 0x45247c thru 0x459334 (0x6eb9 bytes) 2024-09-21T15:49:10.858 INFO:teuthology.orchestra.run.smithi089.stdout:57 punch from 0x11e78a to 0x1206bf, (0x1f35 bytes) 2024-09-21T15:49:10.858 INFO:teuthology.orchestra.run.smithi089.stdout:58 punch from 0x34d822 to 0x35107e, (0x385c bytes) 2024-09-21T15:49:10.868 INFO:teuthology.orchestra.run.smithi089.stdout:60 write 0x524d7ba thru 0x5259bc5 (0xc40c bytes) 2024-09-21T15:49:10.871 INFO:teuthology.orchestra.run.smithi089.stdout:61 writesame 0xca45f5 thru 0xcacdc3 data_size 0x3(0x87cf bytes) 2024-09-21T15:49:10.879 INFO:teuthology.orchestra.run.smithi089.stdout:62 read 0x2bfd497 thru 0x2c0942e (0xbf98 bytes) 2024-09-21T15:49:10.879 INFO:teuthology.orchestra.run.smithi089.stdout:63 read 0x1679787 thru 0x1684d43 (0xb5bd bytes) 2024-09-21T15:49:10.879 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:10.891 INFO:teuthology.orchestra.run.smithi089.stdout:66 writesame 0xb27dc4d thru 0xb288d73 data_size 0x3(0xb127 bytes) 2024-09-21T15:49:10.891 INFO:teuthology.orchestra.run.smithi089.stdout:68 read 0x2ce26d6 thru 0x2cf0c7c (0xe5a7 bytes) 2024-09-21T15:49:10.892 INFO:teuthology.orchestra.run.smithi089.stdout:69 punch from 0x63de4d2 to 0x63ed4a0, (0xefce bytes) 2024-09-21T15:49:10.892 INFO:teuthology.orchestra.run.smithi089.stdout:72 read 0x94d97ee thru 0x94da756 (0xf69 bytes) 2024-09-21T15:49:10.892 INFO:teuthology.orchestra.run.smithi089.stdout:74 punch from 0x9b06b32 to 0x9b0bb70, (0x503e bytes) 2024-09-21T15:49:10.892 INFO:teuthology.orchestra.run.smithi089.stdout:75 write 0x9c6c6c8 thru 0x9c6faae (0x33e7 bytes) 2024-09-21T15:49:10.894 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:10.894 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:10.894 INFO:teuthology.orchestra.run.smithi089.stdout:78 punch from 0x618885f to 0x6188c4a, (0x3eb bytes) 2024-09-21T15:49:10.895 INFO:teuthology.orchestra.run.smithi089.stdout:80 read 0x1d27e30 thru 0x1d30590 (0x8761 bytes) 2024-09-21T15:49:10.901 INFO:teuthology.orchestra.run.smithi089.stdout:82 writesame 0xbf54789 thru 0xbf56d8c data_size 0x6(0x2604 bytes) 2024-09-21T15:49:10.902 INFO:teuthology.orchestra.run.smithi089.stdout:85 read 0x79b1205 thru 0x79b8fe7 (0x7de3 bytes) 2024-09-21T15:49:10.902 INFO:teuthology.orchestra.run.smithi089.stdout:88 write 0x9871b17 thru 0x987632d (0x4817 bytes) 2024-09-21T15:49:10.903 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:49:10.908 INFO:teuthology.orchestra.run.smithi089.stdout:89 read 0xa11c4eb thru 0xa12aebf (0xe9d5 bytes) 2024-09-21T15:49:10.908 INFO:teuthology.orchestra.run.smithi089.stdout:90 write 0x5674348 thru 0x567cc09 (0x88c2 bytes) 2024-09-21T15:49:10.918 INFO:teuthology.orchestra.run.smithi089.stdout:91 write 0xe3838fe thru 0xe387a89 (0x418c bytes) 2024-09-21T15:49:10.921 INFO:teuthology.orchestra.run.smithi089.stdout:92 write 0xaf6607b thru 0xaf6c5af (0x6535 bytes) 2024-09-21T15:49:10.924 INFO:teuthology.orchestra.run.smithi089.stdout:93 write 0xcf2823b thru 0xcf36f6f (0xed35 bytes) 2024-09-21T15:49:10.927 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:10.937 INFO:teuthology.orchestra.run.smithi089.stdout:95 trunc from 0xe387a8a to 0x5d7d90c 2024-09-21T15:49:10.944 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:10.943+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:10.954 INFO:teuthology.orchestra.run.smithi089.stdout:96 punch from 0x38751ef to 0x3880bf6, (0xba07 bytes) 2024-09-21T15:49:10.954 INFO:teuthology.orchestra.run.smithi089.stdout:97 writesame 0xc2e870 thru 0xc2ee8b data_size 0x11(0x61c bytes) 2024-09-21T15:49:10.964 INFO:teuthology.orchestra.run.smithi089.stdout:98 writesame 0x92f5c4c thru 0x9305a7c data_size 0x3(0xfe31 bytes) 2024-09-21T15:49:10.964 INFO:teuthology.orchestra.run.smithi089.stdout:100 read 0x3e43149 thru 0x3e457ac (0x2664 bytes) 2024-09-21T15:49:10.964 INFO:teuthology.orchestra.run.smithi089.stdout:102 writesame 0x72af09f thru 0x72be8f6 data_size 0xc(0xf858 bytes) 2024-09-21T15:49:10.969 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:10.977 INFO:teuthology.orchestra.run.smithi089.stdout:105 trunc from 0x9305a7d to 0x22feddd 2024-09-21T15:49:11.005 INFO:teuthology.orchestra.run.smithi089.stdout:106 write 0xd64cb3a thru 0xd64e741 (0x1c08 bytes) 2024-09-21T15:49:11.007 INFO:teuthology.orchestra.run.smithi089.stdout:108 write 0x76ced44 thru 0x76d4ee9 (0x61a6 bytes) 2024-09-21T15:49:11.010 INFO:teuthology.orchestra.run.smithi089.stdout:109 read 0x7dd9b07 thru 0x7de32eb (0x97e5 bytes) 2024-09-21T15:49:11.015 INFO:teuthology.orchestra.run.smithi089.stdout:111 trunc from 0xd64e742 to 0x9b49c5e 2024-09-21T15:49:11.023 INFO:teuthology.orchestra.run.smithi089.stdout:112 punch from 0x3ddf0a8 to 0x3de795d, (0x88b5 bytes) 2024-09-21T15:49:11.024 INFO:teuthology.orchestra.run.smithi089.stdout:114 punch from 0x2c3d57b to 0x2c4378c, (0x6211 bytes) 2024-09-21T15:49:11.024 INFO:teuthology.orchestra.run.smithi089.stdout:115 read 0x96f1b77 thru 0x96f3a66 (0x1ef0 bytes) 2024-09-21T15:49:11.031 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:11.030+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:11.032 INFO:teuthology.orchestra.run.smithi089.stdout:116 write 0xd1cb3ae thru 0xd1cfb27 (0x477a bytes) 2024-09-21T15:49:11.035 INFO:teuthology.orchestra.run.smithi089.stdout:117 punch from 0x3065518 to 0x306ec5e, (0x9746 bytes) 2024-09-21T15:49:11.035 INFO:teuthology.orchestra.run.smithi089.stdout:118 write 0x6954de0 thru 0x6964844 (0xfa65 bytes) 2024-09-21T15:49:11.038 INFO:teuthology.orchestra.run.smithi089.stdout:121 punch from 0x4aa8475 to 0x4aada36, (0x55c1 bytes) 2024-09-21T15:49:11.038 INFO:teuthology.orchestra.run.smithi089.stdout:122 write 0xb6883cc thru 0xb68b936 (0x356b bytes) 2024-09-21T15:49:11.043 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:11.043 INFO:teuthology.orchestra.run.smithi089.stdout:124 write 0x26a622a thru 0x26ac07e (0x5e55 bytes) 2024-09-21T15:49:11.047 INFO:teuthology.orchestra.run.smithi089.stdout:truncating to largest ever: 0xd192a06 2024-09-21T15:49:11.047 INFO:teuthology.orchestra.run.smithi089.stdout:126 trunc from 0xd1cfb28 to 0xd192a06 2024-09-21T15:49:11.052 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:11.052 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:11.054 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:11.054 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:11.055 INFO:teuthology.orchestra.run.smithi089.stdout:129 write 0x320ebf0 thru 0x3210a56 (0x1e67 bytes) 2024-09-21T15:49:11.059 INFO:teuthology.orchestra.run.smithi089.stdout:130 write 0x6237b1c thru 0x6244335 (0xc81a bytes) 2024-09-21T15:49:11.062 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:49:11.064 INFO:teuthology.orchestra.run.smithi089.stdout:132 punch from 0x5711bd8 to 0x571c565, (0xa98d bytes) 2024-09-21T15:49:11.064 INFO:teuthology.orchestra.run.smithi089.stdout:134 punch from 0x857e3fa to 0x858dc73, (0xf879 bytes) 2024-09-21T15:49:11.064 INFO:teuthology.orchestra.run.smithi089.stdout:136 read 0x6182541 thru 0x6187e72 (0x5932 bytes) 2024-09-21T15:49:11.064 INFO:teuthology.orchestra.run.smithi089.stdout:137 punch from 0x3fc170c to 0x3fcfc17, (0xe50b bytes) 2024-09-21T15:49:11.064 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:11.064 INFO:teuthology.orchestra.run.smithi089.stdout:140 read 0x968bc75 thru 0x9693a7b (0x7e07 bytes) 2024-09-21T15:49:11.064 INFO:teuthology.orchestra.run.smithi089.stdout:141 write 0x46eb140 thru 0x46f1736 (0x65f7 bytes) 2024-09-21T15:49:11.066 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:11.066 INFO:teuthology.orchestra.run.smithi089.stdout:143 read 0x6b1e7b9 thru 0x6b1ead9 (0x321 bytes) 2024-09-21T15:49:11.066 INFO:teuthology.orchestra.run.smithi089.stdout:145 writesame 0xc7fea63 thru 0xc80e98f data_size 0x4b(0xff2d bytes) 2024-09-21T15:49:11.071 INFO:teuthology.orchestra.run.smithi089.stdout:146 read 0x2c84e40 thru 0x2c8d158 (0x8319 bytes) 2024-09-21T15:49:11.071 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:11.071 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:11.071 INFO:teuthology.orchestra.run.smithi089.stdout:149 read 0x982b773 thru 0x982f73f (0x3fcd bytes) 2024-09-21T15:49:11.071 INFO:teuthology.orchestra.run.smithi089.stdout:151 read 0xa0d55a9 thru 0xa0d8188 (0x2be0 bytes) 2024-09-21T15:49:11.071 INFO:teuthology.orchestra.run.smithi089.stdout:152 writesame 0x3ada9fc thru 0x3ae1a74 data_size 0x1(0x7079 bytes) 2024-09-21T15:49:11.090 INFO:teuthology.orchestra.run.smithi089.stdout:154 writesame 0xd31ce28 thru 0xd327f5d data_size 0x3(0xb136 bytes) 2024-09-21T15:49:11.092 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:11.092 INFO:teuthology.orchestra.run.smithi089.stdout:156 write 0xc2cbb0b thru 0xc2d01b2 (0x46a8 bytes) 2024-09-21T15:49:11.095 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:11.095 INFO:teuthology.orchestra.run.smithi089.stdout:161 punch from 0x14e8070 to 0x14e88c2, (0x852 bytes) 2024-09-21T15:49:11.095 INFO:teuthology.orchestra.run.smithi089.stdout:162 read 0x3ef9409 thru 0x3ef9428 (0x20 bytes) 2024-09-21T15:49:11.095 INFO:teuthology.orchestra.run.smithi089.stdout:163 read 0xcf18e6c thru 0xcf1d0ec (0x4281 bytes) 2024-09-21T15:49:11.095 INFO:teuthology.orchestra.run.smithi089.stdout:164 read 0x2d4cc50 thru 0x2d5aeb7 (0xe268 bytes) 2024-09-21T15:49:11.095 INFO:teuthology.orchestra.run.smithi089.stdout:165 write 0xcf7294f thru 0xcf7fb46 (0xd1f8 bytes) 2024-09-21T15:49:11.099 INFO:teuthology.orchestra.run.smithi089.stdout:166 read 0x9052dc6 thru 0x905babc (0x8cf7 bytes) 2024-09-21T15:49:11.099 INFO:teuthology.orchestra.run.smithi089.stdout:167 punch from 0x42d144d to 0x42d6932, (0x54e5 bytes) 2024-09-21T15:49:11.099 INFO:teuthology.orchestra.run.smithi089.stdout:168 write 0x20e11f4 thru 0x20e4655 (0x3462 bytes) 2024-09-21T15:49:11.102 INFO:teuthology.orchestra.run.smithi089.stdout:169 write 0xc77bf3c thru 0xc78368a (0x774f bytes) 2024-09-21T15:49:11.103 INFO:teuthology.orchestra.run.smithi089.stdout:170 clone 1 order 24 su 8388608 sc 6 2024-09-21T15:49:11.131 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:11.130+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:11.204 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:11.204 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:11.213 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T15:49:11.232 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:11.231+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:11.314 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:11.314 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:11.323 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T15:49:11.366 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:11.365+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:11.424 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:11.425 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:11.433 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T15:49:11.434 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:11.433+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:11.535 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:11.534+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:11.576 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:11.576 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:11.585 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T15:49:11.636 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:11.635+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:11.687 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:11.687 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:11.696 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T15:49:11.707 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T15:49:11.737 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:11.736+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:11.837 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:11.836+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:11.839 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:11.839 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:11.848 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T15:49:11.895 INFO:teuthology.orchestra.run.smithi089.stdout:171 writesame 0x687aec2 thru 0x6888b68 data_size 0x13(0xdca7 bytes) 2024-09-21T15:49:11.909 INFO:teuthology.orchestra.run.smithi089.stdout:173 write 0xe272dd9 thru 0xe2826b9 (0xf8e1 bytes) 2024-09-21T15:49:11.913 INFO:teuthology.orchestra.run.smithi089.stdout:174 punch from 0x14068f4 to 0x1413fa8, (0xd6b4 bytes) 2024-09-21T15:49:11.922 INFO:teuthology.orchestra.run.smithi089.stdout:177 trunc from 0xe2826ba to 0x749cdc7 2024-09-21T15:49:11.938 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:11.937+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:11.942 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:11.942 INFO:teuthology.orchestra.run.smithi089.stdout:181 read 0x401fe67 thru 0x402967b (0x9815 bytes) 2024-09-21T15:49:11.954 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:11.955 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:11.955 INFO:teuthology.orchestra.run.smithi089.stdout:182 write 0xbc9d2ad thru 0xbcac46e (0xf1c2 bytes) 2024-09-21T15:49:11.960 INFO:teuthology.orchestra.run.smithi089.stdout:184 write 0x12cfb1c thru 0x12d12a7 (0x178c bytes) 2024-09-21T15:49:11.964 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:49:11.966 INFO:teuthology.orchestra.run.smithi089.stdout:186 write 0x1236092 thru 0x1241456 (0xb3c5 bytes) 2024-09-21T15:49:11.968 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:11.968 INFO:teuthology.orchestra.run.smithi089.stdout:188 read 0x6e92de7 thru 0x6e9656c (0x3786 bytes) 2024-09-21T15:49:11.969 INFO:teuthology.orchestra.run.smithi089.stdout:189 punch from 0x2a9b886 to 0x2aa0f1e, (0x5698 bytes) 2024-09-21T15:49:11.969 INFO:teuthology.orchestra.run.smithi089.stdout:190 write 0x60dd6ca thru 0x60dfff1 (0x2928 bytes) 2024-09-21T15:49:11.971 INFO:teuthology.orchestra.run.smithi089.stdout:192 write 0x7d0a9a9 thru 0x7d10da7 (0x63ff bytes) 2024-09-21T15:49:11.977 INFO:teuthology.orchestra.run.smithi089.stdout:194 trunc from 0xbcac46f to 0xc5a1790 2024-09-21T15:49:11.985 INFO:teuthology.orchestra.run.smithi089.stdout:195 writesame 0x256bcaf thru 0x25742d6 data_size 0x36(0x8628 bytes) 2024-09-21T15:49:11.989 INFO:teuthology.orchestra.run.smithi089.stdout:197 read 0x27b4386 thru 0x27b5e54 (0x1acf bytes) 2024-09-21T15:49:11.989 INFO:teuthology.orchestra.run.smithi089.stdout:198 read 0x3307a60 thru 0x3315a19 (0xdfba bytes) 2024-09-21T15:49:11.989 INFO:teuthology.orchestra.run.smithi089.stdout:199 writesame 0x43fad3f thru 0x440aa6d data_size 0x91(0xfd2f bytes) 2024-09-21T15:49:11.991 INFO:teuthology.orchestra.run.smithi089.stdout:201 writesame 0x2dfb352 thru 0x2dfc12e data_size 0x1(0xddd bytes) 2024-09-21T15:49:11.995 INFO:teuthology.orchestra.run.smithi089.stdout:202 punch from 0xa074141 to 0xa080c23, (0xcae2 bytes) 2024-09-21T15:49:11.995 INFO:teuthology.orchestra.run.smithi089.stdout:203 write 0xbe678d2 thru 0xbe746e3 (0xce12 bytes) 2024-09-21T15:49:11.996 INFO:teuthology.orchestra.run.smithi089.stdout:204 trunc from 0xc5a1790 to 0xc5b5189 2024-09-21T15:49:12.003 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:12.015 INFO:teuthology.orchestra.run.smithi089.stdout:206 trunc from 0xc5b5189 to 0x1e4c389 2024-09-21T15:49:12.038 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:12.037+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:12.045 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:12.045 INFO:teuthology.orchestra.run.smithi089.stdout:209 read 0x16c6f63 thru 0x16d5a8a (0xeb28 bytes) 2024-09-21T15:49:12.054 INFO:teuthology.orchestra.run.smithi089.stdout:210 trunc from 0x1e4c389 to 0x89ebb39 2024-09-21T15:49:12.061 INFO:teuthology.orchestra.run.smithi089.stdout:211 punch from 0x6841799 to 0x684ef73, (0xd7da bytes) 2024-09-21T15:49:12.061 INFO:teuthology.orchestra.run.smithi089.stdout:212 punch from 0x4a774bf to 0x4a84e11, (0xd952 bytes) 2024-09-21T15:49:12.075 INFO:teuthology.orchestra.run.smithi089.stdout:213 writesame 0xeb2ee67 thru 0xeb35dcc data_size 0x62(0x6f66 bytes) 2024-09-21T15:49:12.075 INFO:teuthology.orchestra.run.smithi089.stdout:214 punch from 0x5732cf1 to 0x573fee8, (0xd1f7 bytes) 2024-09-21T15:49:12.075 INFO:teuthology.orchestra.run.smithi089.stdout:215 write 0x1831d3a thru 0x183c85c (0xab23 bytes) 2024-09-21T15:49:12.076 INFO:teuthology.orchestra.run.smithi089.stdout:216 write 0xb633e84 thru 0xb642d06 (0xee83 bytes) 2024-09-21T15:49:12.081 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:12.081 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:12.081 INFO:teuthology.orchestra.run.smithi089.stdout:217 punch from 0x750c2c3 to 0x7519202, (0xcf3f bytes) 2024-09-21T15:49:12.082 INFO:teuthology.orchestra.run.smithi089.stdout:218 read 0x154c87b thru 0x1559035 (0xc7bb bytes) 2024-09-21T15:49:12.082 INFO:teuthology.orchestra.run.smithi089.stdout:219 write 0xdc477c5 thru 0xdc4c1bb (0x49f7 bytes) 2024-09-21T15:49:12.085 INFO:teuthology.orchestra.run.smithi089.stdout:220 write 0x166d684 thru 0x1671dc0 (0x473d bytes) 2024-09-21T15:49:12.086 INFO:teuthology.orchestra.run.smithi089.stdout:222 writesame 0x2bbaccc thru 0x2bbbba2 data_size 0x1(0xed7 bytes) 2024-09-21T15:49:12.089 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:12.090 INFO:teuthology.orchestra.run.smithi089.stdout:225 read 0x4b8bb2f thru 0x4b8f248 (0x371a bytes) 2024-09-21T15:49:12.090 INFO:teuthology.orchestra.run.smithi089.stdout:226 writesame 0x7351448 thru 0x735873b data_size 0xe(0x72f4 bytes) 2024-09-21T15:49:12.091 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:49:12.108 INFO:teuthology.orchestra.run.smithi089.stdout:229 trunc from 0xeb35dcd to 0x3fc231a 2024-09-21T15:49:12.122 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:12.138 INFO:teuthology.orchestra.run.smithi089.stdout:232 writesame 0xc8278cb thru 0xc82933a data_size 0xc(0x1a70 bytes) 2024-09-21T15:49:12.138 INFO:teuthology.orchestra.run.smithi089.stdout:233 writesame 0x4d7d78c thru 0x4d83210 data_size 0x1(0x5a85 bytes) 2024-09-21T15:49:12.139 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:12.138+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:12.153 INFO:teuthology.orchestra.run.smithi089.stdout:235 read 0x3789d18 thru 0x378ab4a (0xe33 bytes) 2024-09-21T15:49:12.153 INFO:teuthology.orchestra.run.smithi089.stdout:236 read 0x9187011 thru 0x9192c26 (0xbc16 bytes) 2024-09-21T15:49:12.154 INFO:teuthology.orchestra.run.smithi089.stdout:237 write 0x42a29e9 thru 0x42a85ec (0x5c04 bytes) 2024-09-21T15:49:12.155 INFO:teuthology.orchestra.run.smithi089.stdout:239 write 0x78dba9f thru 0x78dc0cd (0x62f bytes) 2024-09-21T15:49:12.157 INFO:teuthology.orchestra.run.smithi089.stdout:240 writesame 0x68086c7 thru 0x6817cba data_size 0x84(0xf5f4 bytes) 2024-09-21T15:49:12.161 INFO:teuthology.orchestra.run.smithi089.stdout:241 write 0xa4d4f59 thru 0xa4d62ac (0x1354 bytes) 2024-09-21T15:49:12.164 INFO:teuthology.orchestra.run.smithi089.stdout:242 punch from 0x3dab7d8 to 0x3dba67b, (0xeea3 bytes) 2024-09-21T15:49:12.164 INFO:teuthology.orchestra.run.smithi089.stdout:243 read 0x8d29b87 thru 0x8d33333 (0x97ad bytes) 2024-09-21T15:49:12.165 INFO:teuthology.orchestra.run.smithi089.stdout:244 write 0x540dd39 thru 0x540e3ed (0x6b5 bytes) 2024-09-21T15:49:12.167 INFO:teuthology.orchestra.run.smithi089.stdout:246 read 0x1077572 thru 0x1086f18 (0xf9a7 bytes) 2024-09-21T15:49:12.167 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:12.167 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:12.168 INFO:teuthology.orchestra.run.smithi089.stdout:249 write 0x5a1b01e thru 0x5a1c93b (0x191e bytes) 2024-09-21T15:49:12.169 INFO:teuthology.orchestra.run.smithi089.stdout:250 write 0x8ed05cf thru 0x8ed71f1 (0x6c23 bytes) 2024-09-21T15:49:12.171 INFO:teuthology.orchestra.run.smithi089.stdout:251 read 0x4f5a6d9 thru 0x4f5b98b (0x12b3 bytes) 2024-09-21T15:49:12.171 INFO:teuthology.orchestra.run.smithi089.stdout:252 read 0x1fccf50 thru 0x1fd8166 (0xb217 bytes) 2024-09-21T15:49:12.181 INFO:teuthology.orchestra.run.smithi089.stdout:253 trunc from 0xc82933b to 0x448b7dd 2024-09-21T15:49:12.197 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:12.197 INFO:teuthology.orchestra.run.smithi089.stdout:255 punch from 0x3589af5 to 0x3598a00, (0xef0b bytes) 2024-09-21T15:49:12.206 INFO:teuthology.orchestra.run.smithi089.stdout:256 write 0x7ba515c thru 0x7ba8737 (0x35dc bytes) 2024-09-21T15:49:12.220 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:12.221 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:12.221 INFO:teuthology.orchestra.run.smithi089.stdout:257 write 0xd006ece thru 0xd00758e (0x6c1 bytes) 2024-09-21T15:49:12.223 INFO:teuthology.orchestra.run.smithi089.stdout:258 writesame 0x141309 thru 0x15089c data_size 0x7c(0xf594 bytes) 2024-09-21T15:49:12.225 INFO:teuthology.orchestra.run.smithi089.stdout:259 read 0xbca2c7a thru 0xbca8680 (0x5a07 bytes) 2024-09-21T15:49:12.225 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:12.225 INFO:teuthology.orchestra.run.smithi089.stdout:263 trunc from 0xd00758f to 0xd091355 2024-09-21T15:49:12.231 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:49:12.231 INFO:teuthology.orchestra.run.smithi089.stdout:264 punch from 0xa67c441 to 0xa67d3e6, (0xfa5 bytes) 2024-09-21T15:49:12.231 INFO:teuthology.orchestra.run.smithi089.stdout:266 writesame 0x2e6b00 thru 0x2f0311 data_size 0x22(0x9812 bytes) 2024-09-21T15:49:12.232 INFO:teuthology.orchestra.run.smithi089.stdout:267 read 0x83dc28b thru 0x83df5e4 (0x335a bytes) 2024-09-21T15:49:12.232 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:12.232 INFO:teuthology.orchestra.run.smithi089.stdout:269 write 0x176fd1d thru 0x177d64e (0xd932 bytes) 2024-09-21T15:49:12.240 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:12.239+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:12.240 INFO:teuthology.orchestra.run.smithi089.stdout:270 trunc from 0xd091355 to 0x70de6c6 2024-09-21T15:49:12.250 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:49:12.265 INFO:teuthology.orchestra.run.smithi089.stdout:271 writesame 0x74d3cee thru 0x74e174f data_size 0x1(0xda62 bytes) 2024-09-21T15:49:12.266 INFO:teuthology.orchestra.run.smithi089.stdout:272 read 0xde5bf7 thru 0xdf1cf4 (0xc0fe bytes) 2024-09-21T15:49:12.266 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:12.266 INFO:teuthology.orchestra.run.smithi089.stdout:274 read 0x21a3060 thru 0x21a643e (0x33df bytes) 2024-09-21T15:49:12.267 INFO:teuthology.orchestra.run.smithi089.stdout:275 write 0x2bfa108 thru 0x2c07a07 (0xd900 bytes) 2024-09-21T15:49:12.268 INFO:teuthology.orchestra.run.smithi089.stdout:276 read 0x2d1aba1 thru 0x2d242ee (0x974e bytes) 2024-09-21T15:49:12.268 INFO:teuthology.orchestra.run.smithi089.stdout:279 read 0x467fec2 thru 0x4682c05 (0x2d44 bytes) 2024-09-21T15:49:12.269 INFO:teuthology.orchestra.run.smithi089.stdout:281 punch from 0x69a9e1b to 0x69b2eb5, (0x909a bytes) 2024-09-21T15:49:12.276 INFO:teuthology.orchestra.run.smithi089.stdout:282 trunc from 0x74e1750 to 0x18ef7b0 2024-09-21T15:49:12.294 INFO:teuthology.orchestra.run.smithi089.stdout:283 punch from 0x109217b to 0x1094c80, (0x2b05 bytes) 2024-09-21T15:49:12.294 INFO:teuthology.orchestra.run.smithi089.stdout:284 read 0xcb6613 thru 0xcc0ed7 (0xa8c5 bytes) 2024-09-21T15:49:12.294 INFO:teuthology.orchestra.run.smithi089.stdout:285 writesame 0xe27135 thru 0xe3599c data_size 0x6f(0xe868 bytes) 2024-09-21T15:49:12.296 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:12.313 INFO:teuthology.orchestra.run.smithi089.stdout:288 write 0xc0db164 thru 0xc0e20c7 (0x6f64 bytes) 2024-09-21T15:49:12.323 INFO:teuthology.orchestra.run.smithi089.stdout:290 writesame 0xe16259c thru 0xe166c3c data_size 0x31(0x46a1 bytes) 2024-09-21T15:49:12.323 INFO:teuthology.orchestra.run.smithi089.stdout:291 write 0xbbb4d3a thru 0xbbb701b (0x22e2 bytes) 2024-09-21T15:49:12.326 INFO:teuthology.orchestra.run.smithi089.stdout:292 write 0x6e0ad63 thru 0x6e15bae (0xae4c bytes) 2024-09-21T15:49:12.330 INFO:teuthology.orchestra.run.smithi089.stdout:293 punch from 0x8f260e8 to 0x8f26922, (0x83a bytes) 2024-09-21T15:49:12.330 INFO:teuthology.orchestra.run.smithi089.stdout:294 write 0xde1f500 thru 0xde2ecf3 (0xf7f4 bytes) 2024-09-21T15:49:12.334 INFO:teuthology.orchestra.run.smithi089.stdout:296 read 0x8f52f6b thru 0x8f59668 (0x66fe bytes) 2024-09-21T15:49:12.334 INFO:teuthology.orchestra.run.smithi089.stdout:297 write 0x9cc97cf thru 0x9cd6391 (0xcbc3 bytes) 2024-09-21T15:49:12.335 INFO:teuthology.orchestra.run.smithi089.stdout:299 writesame 0x5fb5bb6 thru 0x5fbae41 data_size 0x12(0x528c bytes) 2024-09-21T15:49:12.339 INFO:teuthology.orchestra.run.smithi089.stdout:302 write 0x94febb6 thru 0x9505910 (0x6d5b bytes) 2024-09-21T15:49:12.340 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:12.339+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:12.342 INFO:teuthology.orchestra.run.smithi089.stdout:305 write 0xd1259de thru 0xd125f0f (0x532 bytes) 2024-09-21T15:49:12.345 INFO:teuthology.orchestra.run.smithi089.stdout:306 write 0x392f32c thru 0x393a93e (0xb613 bytes) 2024-09-21T15:49:12.347 INFO:teuthology.orchestra.run.smithi089.stdout:307 read 0xbd7a70f thru 0xbd80e6f (0x6761 bytes) 2024-09-21T15:49:12.347 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:12.347 INFO:teuthology.orchestra.run.smithi089.stdout:310 writesame 0xa3c3909 thru 0xa3c42e6 data_size 0x3(0x9de bytes) 2024-09-21T15:49:12.348 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:12.348 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:12.356 INFO:teuthology.orchestra.run.smithi089.stdout:311 trunc from 0xe166c3d to 0x8d7f0de 2024-09-21T15:49:12.361 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:49:12.372 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:12.372 INFO:teuthology.orchestra.run.smithi089.stdout:314 punch from 0x5696447 to 0x569e486, (0x803f bytes) 2024-09-21T15:49:12.372 INFO:teuthology.orchestra.run.smithi089.stdout:315 write 0x8c1819d thru 0x8c1cb60 (0x49c4 bytes) 2024-09-21T15:49:12.374 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:12.380 INFO:teuthology.orchestra.run.smithi089.stdout:317 write 0x986708d thru 0x98683fc (0x1370 bytes) 2024-09-21T15:49:12.381 INFO:teuthology.orchestra.run.smithi089.stdout:318 read 0x81e1047 thru 0x81f0caa (0xfc64 bytes) 2024-09-21T15:49:12.381 INFO:teuthology.orchestra.run.smithi089.stdout:321 read 0x352bb57 thru 0x35393fa (0xd8a4 bytes) 2024-09-21T15:49:12.382 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:12.388 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-09-21T15:49:12.388 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:49:12.396 INFO:teuthology.orchestra.run.smithi089.stdout:325 writesame 0xe8c34bd thru 0xe8d1699 data_size 0x1(0xe1dd bytes) 2024-09-21T15:49:12.407 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:49:12.407 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:49:12.413 INFO:teuthology.orchestra.run.smithi089.stdout:326 trunc from 0xe8d169a to 0x95b193 2024-09-21T15:49:12.425 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:49:12.425 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:49:12.441 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:12.440+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:12.444 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:49:12.444 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:49:12.456 INFO:teuthology.orchestra.run.smithi089.stdout:327 write 0xa779c1a thru 0xa77c5e2 (0x29c9 bytes) 2024-09-21T15:49:12.459 INFO:teuthology.orchestra.run.smithi089.stdout:328 write 0xa4aebc2 thru 0xa4b7012 (0x8451 bytes) 2024-09-21T15:49:12.460 INFO:teuthology.orchestra.run.smithi089.stdout:329 read 0x777910f thru 0x7785493 (0xc385 bytes) 2024-09-21T15:49:12.463 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:49:12.464 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:49:12.466 INFO:teuthology.orchestra.run.smithi089.stdout:330 write 0xa954b2b thru 0xa958f48 (0x441e bytes) 2024-09-21T15:49:12.472 INFO:teuthology.orchestra.run.smithi089.stdout:332 writesame 0x4915b00 thru 0x491a028 data_size 0x1(0x4529 bytes) 2024-09-21T15:49:12.481 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:49:12.481 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:49:12.483 INFO:teuthology.orchestra.run.smithi089.stdout:333 read 0x265c567 thru 0x265cae9 (0x583 bytes) 2024-09-21T15:49:12.483 INFO:teuthology.orchestra.run.smithi089.stdout:336 read 0xa37f017 thru 0xa381bdb (0x2bc5 bytes) 2024-09-21T15:49:12.483 INFO:teuthology.orchestra.run.smithi089.stdout:337 writesame 0x709352a thru 0x7097581 data_size 0x47(0x4058 bytes) 2024-09-21T15:49:12.489 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:12.489 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:49:12.489 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:49:12.490 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:12.490 INFO:teuthology.orchestra.run.smithi089.stdout:338 trunc from 0xa958f49 to 0x55b2364 2024-09-21T15:49:12.499 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:49:12.500 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:49:12.500 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:49:12.504 INFO:teuthology.orchestra.run.smithi089.stdout:339 write 0x278a9d9 thru 0x2795c9c (0xb2c4 bytes) 2024-09-21T15:49:12.509 INFO:teuthology.orchestra.run.smithi089.stdout:340 read 0x18570f4 thru 0x185908c (0x1f99 bytes) 2024-09-21T15:49:12.509 INFO:teuthology.orchestra.run.smithi089.stdout:341 read 0x3d09f8 thru 0x3ded25 (0xe32e bytes) 2024-09-21T15:49:12.517 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:49:12.517 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:49:12.520 INFO:teuthology.orchestra.run.smithi089.stdout:342 write 0x8ed996a thru 0x8edd3cd (0x3a64 bytes) 2024-09-21T15:49:12.535 INFO:teuthology.orchestra.run.smithi089.stdout:343 write 0xec146d7 thru 0xec14e6f (0x799 bytes) 2024-09-21T15:49:12.540 INFO:teuthology.orchestra.run.smithi089.stdout:344 writesame 0xd19e414 thru 0xd1a9fb5 data_size 0x5e(0xbba2 bytes) 2024-09-21T15:49:12.541 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:12.540+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:12.547 INFO:teuthology.orchestra.run.smithi089.stdout:345 read 0xd551625 thru 0xd55a913 (0x92ef bytes) 2024-09-21T15:49:12.547 INFO:teuthology.orchestra.run.smithi089.stdout:346 write 0x45c6e25 thru 0x45cbe8a (0x5066 bytes) 2024-09-21T15:49:12.550 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:12.550 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:12.550 INFO:teuthology.orchestra.run.smithi089.stdout:349 read 0xa27f59e thru 0xa28c409 (0xce6c bytes) 2024-09-21T15:49:12.567 INFO:teuthology.orchestra.run.smithi089.stdout:350 trunc from 0xec14e70 to 0x30f88a3 2024-09-21T15:49:12.601 INFO:teuthology.orchestra.run.smithi089.stdout:351 write 0xde9f20b thru 0xdea4408 (0x51fe bytes) 2024-09-21T15:49:12.604 INFO:teuthology.orchestra.run.smithi089.stdout:352 writesame 0xaf0d17b thru 0xaf11eaf data_size 0x3b(0x4d35 bytes) 2024-09-21T15:49:12.609 INFO:teuthology.orchestra.run.smithi089.stdout:355 write 0x3226858 thru 0x322c11e (0x58c7 bytes) 2024-09-21T15:49:12.613 INFO:teuthology.orchestra.run.smithi089.stdout:356 punch from 0x89d2932 to 0x89e03b3, (0xda81 bytes) 2024-09-21T15:49:12.613 INFO:teuthology.orchestra.run.smithi089.stdout:357 read 0x65928dd thru 0x6595c06 (0x332a bytes) 2024-09-21T15:49:12.613 INFO:teuthology.orchestra.run.smithi089.stdout:358 read 0xb9c2a99 thru 0xb9c8764 (0x5ccc bytes) 2024-09-21T15:49:12.613 INFO:teuthology.orchestra.run.smithi089.stdout:359 write 0x721bf87 thru 0x72245d7 (0x8651 bytes) 2024-09-21T15:49:12.619 INFO:teuthology.orchestra.run.smithi089.stdout:360 punch from 0xd5bdccd to 0xd5be623, (0x956 bytes) 2024-09-21T15:49:12.619 INFO:teuthology.orchestra.run.smithi089.stdout:361 punch from 0x14685f2 to 0x14725db, (0x9fe9 bytes) 2024-09-21T15:49:12.619 INFO:teuthology.orchestra.run.smithi089.stdout:362 write 0x7639e16 thru 0x764351a (0x9705 bytes) 2024-09-21T15:49:12.623 INFO:teuthology.orchestra.run.smithi089.stdout:364 write 0x4932f2e thru 0x493e677 (0xb74a bytes) 2024-09-21T15:49:12.628 INFO:teuthology.orchestra.run.smithi089.stdout:365 trunc from 0xdea4409 to 0xb8dd4ce 2024-09-21T15:49:12.637 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:12.638 INFO:teuthology.orchestra.run.smithi089.stdout:367 read 0x3d5509d thru 0x3d58b88 (0x3aec bytes) 2024-09-21T15:49:12.638 INFO:teuthology.orchestra.run.smithi089.stdout:368 write 0x76ad7bb thru 0x76b9e4f (0xc695 bytes) 2024-09-21T15:49:12.638 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:12.639 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:49:12.639 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:49:12.639 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:49:12.639 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:12.639 INFO:teuthology.orchestra.run.smithi089.stdout:369 punch from 0x463fff6 to 0x4642300, (0x230a bytes) 2024-09-21T15:49:12.639 INFO:teuthology.orchestra.run.smithi089.stdout:370 write 0x60b96ef thru 0x60bf918 (0x622a bytes) 2024-09-21T15:49:12.642 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:12.643 INFO:teuthology.orchestra.run.smithi089.stdout:373 writesame 0x3a01b4b thru 0x3a09b7f data_size 0x1(0x8035 bytes) 2024-09-21T15:49:12.643 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:12.641+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:12.648 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:49:12.667 INFO:teuthology.orchestra.run.smithi089.stdout:375 trunc from 0xb8dd4ce to 0x3d4a93f 2024-09-21T15:49:12.687 INFO:teuthology.orchestra.run.smithi089.stdout:376 write 0x49b6d34 thru 0x49c37c6 (0xca93 bytes) 2024-09-21T15:49:12.701 INFO:teuthology.orchestra.run.smithi089.stdout:377 writesame 0x9800d82 thru 0x9802a1c data_size 0x1(0x1c9b bytes) 2024-09-21T15:49:12.701 INFO:teuthology.orchestra.run.smithi089.stdout:378 read 0x32b97e4 thru 0x32c3d53 (0xa570 bytes) 2024-09-21T15:49:12.701 INFO:teuthology.orchestra.run.smithi089.stdout:381 writesame 0x84c3087 thru 0x84d1c6f data_size 0x29(0xebe9 bytes) 2024-09-21T15:49:12.707 INFO:teuthology.orchestra.run.smithi089.stdout:382 write 0x72e1fb6 thru 0x72ebc6e (0x9cb9 bytes) 2024-09-21T15:49:12.712 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:12.715 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is unset 2024-09-21T15:49:12.726 INFO:teuthology.orchestra.run.smithi089.stdout:387 writesame 0xe78370c thru 0xe7849fb data_size 0x18(0x12f0 bytes) 2024-09-21T15:49:12.726 INFO:teuthology.orchestra.run.smithi089.stdout:388 write 0x989df00 thru 0x98a674e (0x884f bytes) 2024-09-21T15:49:12.731 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:12.731 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:12.731 INFO:teuthology.orchestra.run.smithi089.stdout:392 writesame 0x407c8c4 thru 0x407e957 data_size 0x14(0x2094 bytes) 2024-09-21T15:49:12.734 INFO:teuthology.orchestra.run.smithi089.stdout:393 write 0x28041b1 thru 0x2809c6d (0x5abd bytes) 2024-09-21T15:49:12.737 INFO:teuthology.orchestra.run.smithi089.stdout:394 read 0xc3911ff thru 0xc397328 (0x612a bytes) 2024-09-21T15:49:12.737 INFO:teuthology.orchestra.run.smithi089.stdout:396 read 0x6e8e271 thru 0x6e9a318 (0xc0a8 bytes) 2024-09-21T15:49:12.737 INFO:teuthology.orchestra.run.smithi089.stdout:397 write 0x77e9ce thru 0x785323 (0x6956 bytes) 2024-09-21T15:49:12.739 INFO:teuthology.orchestra.run.smithi089.stdout:398 write 0x1014869 thru 0x10171d4 (0x296c bytes) 2024-09-21T15:49:12.740 INFO:teuthology.orchestra.run.smithi089.stdout:399 writesame 0xc4153f0 thru 0xc4191db data_size 0x6(0x3dec bytes) 2024-09-21T15:49:12.743 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:12.742+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:12.745 INFO:teuthology.orchestra.run.smithi089.stdout:400 punch from 0x8c168ae to 0x8c24f37, (0xe689 bytes) 2024-09-21T15:49:12.745 INFO:teuthology.orchestra.run.smithi089.stdout:401 clone 2 order 19 su 4096 sc 12 2024-09-21T15:49:12.781 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:12.781 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:49:12.781 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:49:12.781 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:49:12.781 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.781 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4275.0:5 2.2 e19/17 2:5cea7035:::rbd_info:head v 19'1, mlcod=0'0)", 2024-09-21T15:49:12.781 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:48:58.590679+0000", 2024-09-21T15:49:12.781 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.188348356000001, 2024-09-21T15:49:12.782 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00080136699999999999, 2024-09-21T15:49:12.782 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:12.782 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:12.782 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:12.782 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.782 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:12.782 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:58.590679+0000", 2024-09-21T15:49:12.782 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:12.782 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.782 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.782 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:12.782 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:58.590679+0000", 2024-09-21T15:49:12.782 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:12.782 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.783 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.783 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:12.783 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:58.590679+0000", 2024-09-21T15:49:12.783 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:49:12.783 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.783 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.783 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:12.783 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:58.590691+0000", 2024-09-21T15:49:12.783 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2768999999999999e-05 2024-09-21T15:49:12.783 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.783 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.783 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:12.783 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:58.590704+0000", 2024-09-21T15:49:12.783 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2981e-05 2024-09-21T15:49:12.784 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.784 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.784 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:12.784 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:58.590714+0000", 2024-09-21T15:49:12.784 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6010000000000001e-06 2024-09-21T15:49:12.784 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.784 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.784 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:12.784 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:58.590773+0000", 2024-09-21T15:49:12.784 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9284999999999999e-05 2024-09-21T15:49:12.784 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.784 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.785 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:12.785 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:58.590798+0000", 2024-09-21T15:49:12.785 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5109999999999998e-05 2024-09-21T15:49:12.785 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.785 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.785 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:12.785 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:58.591449+0000", 2024-09-21T15:49:12.785 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00065082599999999997 2024-09-21T15:49:12.785 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.785 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.785 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:12.785 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:58.591480+0000", 2024-09-21T15:49:12.785 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0976000000000003e-05 2024-09-21T15:49:12.786 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:12.786 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:12.786 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:12.786 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.786 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.786 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4275.0:7 2.2 e20/17 2:5cea7035:::rbd_info:head v 20'2, mlcod=0'0)", 2024-09-21T15:49:12.786 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:48:59.595176+0000", 2024-09-21T15:49:12.786 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.183851247, 2024-09-21T15:49:12.786 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00079370899999999997, 2024-09-21T15:49:12.786 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:12.786 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:12.786 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:12.786 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.787 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:12.787 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:59.595176+0000", 2024-09-21T15:49:12.787 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:12.787 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.787 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.787 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:12.787 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:59.595176+0000", 2024-09-21T15:49:12.787 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:12.787 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.787 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.787 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:12.787 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:59.595175+0000", 2024-09-21T15:49:12.787 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:12.788 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.788 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.788 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:12.788 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:59.595185+0000", 2024-09-21T15:49:12.788 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6560000000000004e-06 2024-09-21T15:49:12.788 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.788 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.788 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:12.788 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:59.595202+0000", 2024-09-21T15:49:12.788 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6642000000000001e-05 2024-09-21T15:49:12.788 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.788 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.788 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:12.788 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:59.595213+0000", 2024-09-21T15:49:12.789 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1092e-05 2024-09-21T15:49:12.789 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.789 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.789 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:12.789 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:59.595269+0000", 2024-09-21T15:49:12.789 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5847999999999998e-05 2024-09-21T15:49:12.789 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.789 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.789 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:12.789 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:59.595293+0000", 2024-09-21T15:49:12.789 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4476000000000001e-05 2024-09-21T15:49:12.789 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.789 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.790 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:12.790 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:59.595902+0000", 2024-09-21T15:49:12.790 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00060874099999999999 2024-09-21T15:49:12.790 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.790 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.790 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:12.790 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:59.595970+0000", 2024-09-21T15:49:12.790 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7618000000000002e-05 2024-09-21T15:49:12.790 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:12.790 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:12.790 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:12.790 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.790 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.791 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:9 3.1a e25/22 3:5cea7035:::rbd_info:head v 25'2, mlcod=0'0)", 2024-09-21T15:49:12.791 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:08.695923+0000", 2024-09-21T15:49:12.791 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.0831036960000002, 2024-09-21T15:49:12.791 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001386878, 2024-09-21T15:49:12.791 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:12.791 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:12.791 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:12.791 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.791 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:12.791 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:08.695923+0000", 2024-09-21T15:49:12.791 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:12.791 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.791 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.792 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:12.792 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:08.695923+0000", 2024-09-21T15:49:12.792 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:12.792 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.792 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.792 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:12.792 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:08.695923+0000", 2024-09-21T15:49:12.792 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:49:12.792 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.792 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.792 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:12.792 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:08.695930+0000", 2024-09-21T15:49:12.792 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7939999999999997e-06 2024-09-21T15:49:12.792 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.793 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.793 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:12.793 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:08.695948+0000", 2024-09-21T15:49:12.793 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7957000000000001e-05 2024-09-21T15:49:12.793 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.793 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.793 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:12.793 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:08.695958+0000", 2024-09-21T15:49:12.793 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0162e-05 2024-09-21T15:49:12.793 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.793 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.793 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:12.793 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:08.696010+0000", 2024-09-21T15:49:12.793 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1604000000000003e-05 2024-09-21T15:49:12.793 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.794 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.794 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:12.794 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:08.696027+0000", 2024-09-21T15:49:12.794 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7176e-05 2024-09-21T15:49:12.794 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.794 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.794 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:12.794 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:08.697280+0000", 2024-09-21T15:49:12.794 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001253079 2024-09-21T15:49:12.794 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.794 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.794 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:12.794 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:08.697310+0000", 2024-09-21T15:49:12.794 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0335999999999999e-05 2024-09-21T15:49:12.795 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:12.795 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:12.795 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:12.795 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.795 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.795 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:854 3.0 3:025e53e7:::rbd_data.1133f3dfbd6b.0000000000000008:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-21T15:49:12.795 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.415330+0000", 2024-09-21T15:49:12.795 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.36369689599999999, 2024-09-21T15:49:12.795 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0011317600000000001, 2024-09-21T15:49:12.795 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:12.795 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:12.795 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:12.795 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:12.795 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:12.795 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 854 2024-09-21T15:49:12.796 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.796 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:12.796 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.796 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:12.796 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.415330+0000", 2024-09-21T15:49:12.796 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:12.796 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.796 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.796 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:12.796 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.415330+0000", 2024-09-21T15:49:12.796 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:12.796 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.796 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.796 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:12.797 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.415328+0000", 2024-09-21T15:49:12.797 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:49:12.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.797 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.797 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:12.797 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.415340+0000", 2024-09-21T15:49:12.797 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1571e-05 2024-09-21T15:49:12.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.797 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.797 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:12.797 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.415359+0000", 2024-09-21T15:49:12.797 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8841000000000001e-05 2024-09-21T15:49:12.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.797 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.798 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:12.798 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.415368+0000", 2024-09-21T15:49:12.798 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5359999999999998e-06 2024-09-21T15:49:12.798 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.798 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.798 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:12.798 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.415422+0000", 2024-09-21T15:49:12.798 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3352e-05 2024-09-21T15:49:12.798 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.798 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.798 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:12.798 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.415512+0000", 2024-09-21T15:49:12.798 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0644999999999997e-05 2024-09-21T15:49:12.798 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.799 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.799 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-09-21T15:49:12.799 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.415621+0000", 2024-09-21T15:49:12.799 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000109027 2024-09-21T15:49:12.799 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.799 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.799 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:12.799 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.416152+0000", 2024-09-21T15:49:12.799 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00053077099999999998 2024-09-21T15:49:12.799 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.799 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.799 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:12.799 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.416414+0000", 2024-09-21T15:49:12.799 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00026228699999999999 2024-09-21T15:49:12.800 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.800 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.800 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:12.800 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.416442+0000", 2024-09-21T15:49:12.800 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7568000000000001e-05 2024-09-21T15:49:12.800 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.800 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.800 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:12.800 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.416462+0000", 2024-09-21T15:49:12.800 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0126999999999999e-05 2024-09-21T15:49:12.800 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:12.800 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:12.800 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:12.800 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.801 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.801 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:857 3.19 3:9b8b44d5:::rbd_data.1133f3dfbd6b.0000000000000000:head [stat,truncate 8388608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-21T15:49:12.801 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.422005+0000", 2024-09-21T15:49:12.801 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.35702186699999999, 2024-09-21T15:49:12.801 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0018289809999999999, 2024-09-21T15:49:12.801 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:12.801 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:12.801 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:12.801 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:12.801 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:12.801 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 857 2024-09-21T15:49:12.801 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.801 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:12.801 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.802 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:12.802 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.422005+0000", 2024-09-21T15:49:12.802 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:12.802 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.802 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.802 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:12.802 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.422005+0000", 2024-09-21T15:49:12.802 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:12.802 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.802 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.802 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:12.802 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.422004+0000", 2024-09-21T15:49:12.802 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:12.802 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.803 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.803 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:12.803 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.422009+0000", 2024-09-21T15:49:12.803 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2809999999999999e-06 2024-09-21T15:49:12.803 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.803 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.803 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:12.803 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.422022+0000", 2024-09-21T15:49:12.803 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2142e-05 2024-09-21T15:49:12.803 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.803 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.803 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:12.803 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.422029+0000", 2024-09-21T15:49:12.803 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4170000000000003e-06 2024-09-21T15:49:12.804 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.804 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.804 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:12.804 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.422079+0000", 2024-09-21T15:49:12.804 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9799000000000002e-05 2024-09-21T15:49:12.804 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.804 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.804 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:12.804 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.422164+0000", 2024-09-21T15:49:12.804 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4678000000000005e-05 2024-09-21T15:49:12.804 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.804 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.804 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:49:12.804 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.422304+0000", 2024-09-21T15:49:12.805 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014015100000000001 2024-09-21T15:49:12.805 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.805 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.805 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:12.805 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.422839+0000", 2024-09-21T15:49:12.805 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00053522300000000004 2024-09-21T15:49:12.805 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.805 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.805 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:12.805 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.423755+0000", 2024-09-21T15:49:12.805 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000916025 2024-09-21T15:49:12.805 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.805 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.805 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:12.806 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.423801+0000", 2024-09-21T15:49:12.806 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5859000000000003e-05 2024-09-21T15:49:12.806 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.806 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.806 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:12.806 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.423834+0000", 2024-09-21T15:49:12.806 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3448000000000002e-05 2024-09-21T15:49:12.806 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:12.806 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:12.806 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:12.806 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.806 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.806 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:887 3.0 3:025e53e7:::rbd_data.1133f3dfbd6b.0000000000000008:head [set-alloc-hint object_size 16777216 write_size 16777216,write 16227354~10697] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-21T15:49:12.806 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.458148+0000", 2024-09-21T15:49:12.807 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.32087899399999997, 2024-09-21T15:49:12.807 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001346725, 2024-09-21T15:49:12.807 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:12.807 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:12.807 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:12.807 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:12.807 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:12.807 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 887 2024-09-21T15:49:12.807 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.807 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:12.807 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.807 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:12.807 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.458148+0000", 2024-09-21T15:49:12.807 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:12.807 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.808 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:12.808 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.458148+0000", 2024-09-21T15:49:12.808 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:12.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.808 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:12.808 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.458147+0000", 2024-09-21T15:49:12.808 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:12.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.808 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:12.808 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.458167+0000", 2024-09-21T15:49:12.808 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0021e-05 2024-09-21T15:49:12.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.808 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:12.809 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.458184+0000", 2024-09-21T15:49:12.809 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7261e-05 2024-09-21T15:49:12.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.809 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:12.809 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.458191+0000", 2024-09-21T15:49:12.809 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0879999999999997e-06 2024-09-21T15:49:12.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.809 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:12.809 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.458243+0000", 2024-09-21T15:49:12.809 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2377999999999999e-05 2024-09-21T15:49:12.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.810 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:12.810 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.458326+0000", 2024-09-21T15:49:12.810 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.2972000000000005e-05 2024-09-21T15:49:12.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.810 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.810 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-09-21T15:49:12.810 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.458473+0000", 2024-09-21T15:49:12.810 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000146456 2024-09-21T15:49:12.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.810 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.810 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:12.810 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.459149+0000", 2024-09-21T15:49:12.810 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00067666199999999997 2024-09-21T15:49:12.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.810 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.810 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:12.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.459443+0000", 2024-09-21T15:49:12.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00029320600000000001 2024-09-21T15:49:12.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:12.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.459473+0000", 2024-09-21T15:49:12.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0873000000000001e-05 2024-09-21T15:49:12.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:12.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.459495+0000", 2024-09-21T15:49:12.812 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1379999999999999e-05 2024-09-21T15:49:12.812 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:12.812 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:12.812 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:12.812 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.812 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.812 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:888 3.0 3:025e53e7:::rbd_data.1133f3dfbd6b.0000000000000008:head [write 13298626~33873] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-21T15:49:12.812 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.459661+0000", 2024-09-21T15:49:12.812 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.31936615800000001, 2024-09-21T15:49:12.812 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012611549999999999, 2024-09-21T15:49:12.812 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:12.812 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:12.813 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:12.813 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:12.813 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:12.813 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 888 2024-09-21T15:49:12.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.813 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:12.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.813 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:12.813 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.459661+0000", 2024-09-21T15:49:12.813 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:12.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.813 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:12.813 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.459661+0000", 2024-09-21T15:49:12.814 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:12.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.814 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:12.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.459660+0000", 2024-09-21T15:49:12.814 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:12.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.814 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:12.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.459690+0000", 2024-09-21T15:49:12.814 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0059000000000001e-05 2024-09-21T15:49:12.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.864 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.865 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:12.865 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.459698+0000", 2024-09-21T15:49:12.865 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1540000000000002e-06 2024-09-21T15:49:12.865 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.865 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.865 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:12.865 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.459703+0000", 2024-09-21T15:49:12.865 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5040000000000004e-06 2024-09-21T15:49:12.865 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.865 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.865 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:12.865 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.459737+0000", 2024-09-21T15:49:12.865 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4147999999999999e-05 2024-09-21T15:49:12.866 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.866 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.866 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:12.866 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.459783+0000", 2024-09-21T15:49:12.866 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5952000000000003e-05 2024-09-21T15:49:12.866 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.866 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.866 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-09-21T15:49:12.866 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.459862+0000", 2024-09-21T15:49:12.866 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8929000000000003e-05 2024-09-21T15:49:12.866 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.866 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.866 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:12.867 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.460512+0000", 2024-09-21T15:49:12.867 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00065006299999999999 2024-09-21T15:49:12.867 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.867 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.867 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:12.867 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.460869+0000", 2024-09-21T15:49:12.867 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00035713200000000002 2024-09-21T15:49:12.867 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.867 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.867 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:12.867 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.460901+0000", 2024-09-21T15:49:12.867 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2289000000000002e-05 2024-09-21T15:49:12.867 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.867 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.867 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:12.868 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.460922+0000", 2024-09-21T15:49:12.868 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0673e-05 2024-09-21T15:49:12.868 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:12.868 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:12.868 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:12.868 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.868 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.868 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:904 3.0 3:025e53e7:::rbd_data.1133f3dfbd6b.0000000000000008:head [write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-21T15:49:12.868 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.483432+0000", 2024-09-21T15:49:12.868 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.29559534399999998, 2024-09-21T15:49:12.868 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012339219999999999, 2024-09-21T15:49:12.868 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:12.868 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:12.868 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:12.868 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:12.868 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:12.869 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 904 2024-09-21T15:49:12.869 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.869 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:12.869 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.869 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:12.869 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.483432+0000", 2024-09-21T15:49:12.869 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:12.869 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.869 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.869 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:12.869 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.483432+0000", 2024-09-21T15:49:12.869 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:12.869 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.869 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.869 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:12.870 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.483431+0000", 2024-09-21T15:49:12.870 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:12.870 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.870 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.870 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:12.870 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.483435+0000", 2024-09-21T15:49:12.870 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2459999999999997e-06 2024-09-21T15:49:12.870 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.870 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.870 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:12.870 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.483443+0000", 2024-09-21T15:49:12.870 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0709999999999995e-06 2024-09-21T15:49:12.870 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.870 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.871 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:12.871 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.483449+0000", 2024-09-21T15:49:12.871 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0889999999999999e-06 2024-09-21T15:49:12.871 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.871 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.871 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:12.871 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.483477+0000", 2024-09-21T15:49:12.871 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8470999999999999e-05 2024-09-21T15:49:12.871 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.871 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.871 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:12.871 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.483518+0000", 2024-09-21T15:49:12.871 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1239000000000003e-05 2024-09-21T15:49:12.871 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.871 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.871 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-09-21T15:49:12.872 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.483621+0000", 2024-09-21T15:49:12.872 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000103069 2024-09-21T15:49:12.872 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.872 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.872 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:12.872 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.484323+0000", 2024-09-21T15:49:12.872 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00070221300000000003 2024-09-21T15:49:12.872 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.872 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.872 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:12.872 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.484619+0000", 2024-09-21T15:49:12.872 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00029503999999999999 2024-09-21T15:49:12.872 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.872 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.872 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:12.873 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.484644+0000", 2024-09-21T15:49:12.873 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5312000000000001e-05 2024-09-21T15:49:12.873 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.873 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.873 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:12.873 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.484666+0000", 2024-09-21T15:49:12.873 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1829999999999999e-05 2024-09-21T15:49:12.873 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:12.873 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:12.873 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:12.873 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.873 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.873 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:906 3.0 3:025e53e7:::rbd_data.1133f3dfbd6b.0000000000000008:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-21T15:49:12.873 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.492569+0000", 2024-09-21T15:49:12.873 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.28645836000000002, 2024-09-21T15:49:12.874 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0011036539999999999, 2024-09-21T15:49:12.874 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:12.874 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:12.874 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:12.874 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:12.874 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:12.874 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 906 2024-09-21T15:49:12.874 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.874 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:12.874 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.874 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:12.874 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.492569+0000", 2024-09-21T15:49:12.874 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:12.874 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.875 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.875 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:12.875 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.492569+0000", 2024-09-21T15:49:12.875 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:12.875 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.875 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.875 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:12.875 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.492567+0000", 2024-09-21T15:49:12.875 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:12.875 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.875 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.875 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:12.875 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.492576+0000", 2024-09-21T15:49:12.875 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1670000000000005e-06 2024-09-21T15:49:12.876 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.876 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.876 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:12.876 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.492593+0000", 2024-09-21T15:49:12.876 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6626e-05 2024-09-21T15:49:12.876 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.876 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.876 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:12.876 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.492600+0000", 2024-09-21T15:49:12.876 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6229999999999999e-06 2024-09-21T15:49:12.876 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.876 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.876 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:12.876 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.492638+0000", 2024-09-21T15:49:12.877 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7880000000000003e-05 2024-09-21T15:49:12.877 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.877 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.877 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:12.877 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.492692+0000", 2024-09-21T15:49:12.877 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4561999999999997e-05 2024-09-21T15:49:12.877 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.877 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.877 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-09-21T15:49:12.877 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.492756+0000", 2024-09-21T15:49:12.877 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3697999999999999e-05 2024-09-21T15:49:12.877 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.877 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.877 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:12.877 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.493355+0000", 2024-09-21T15:49:12.878 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00059938000000000005 2024-09-21T15:49:12.878 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.878 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.878 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:12.878 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.493620+0000", 2024-09-21T15:49:12.878 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00026518400000000001 2024-09-21T15:49:12.878 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.878 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.878 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:12.878 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.493646+0000", 2024-09-21T15:49:12.878 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5474999999999999e-05 2024-09-21T15:49:12.878 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.878 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.878 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:12.879 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.493672+0000", 2024-09-21T15:49:12.879 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6517999999999999e-05 2024-09-21T15:49:12.879 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:12.879 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:12.879 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:12.879 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.879 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.879 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:945 3.0 3:05942dfc:::rbd_data.1133f3dfbd6b.000000000000000e:head [set-alloc-hint object_size 16777216 write_size 16777216,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-21T15:49:12.879 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.545886+0000", 2024-09-21T15:49:12.879 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.23314079500000001, 2024-09-21T15:49:12.879 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001499963, 2024-09-21T15:49:12.879 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:12.879 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:12.879 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:12.879 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:12.880 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:12.880 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 945 2024-09-21T15:49:12.880 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.880 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:12.880 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.880 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:12.880 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.545886+0000", 2024-09-21T15:49:12.880 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:12.880 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.880 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.880 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:12.880 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.545886+0000", 2024-09-21T15:49:12.880 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:12.880 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.880 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.881 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:12.881 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.545885+0000", 2024-09-21T15:49:12.881 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:12.881 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.881 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.881 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:12.881 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.545893+0000", 2024-09-21T15:49:12.881 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4630000000000001e-06 2024-09-21T15:49:12.881 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.881 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.881 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:12.881 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.545909+0000", 2024-09-21T15:49:12.881 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5529e-05 2024-09-21T15:49:12.881 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.881 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.882 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:12.882 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.545916+0000", 2024-09-21T15:49:12.882 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9079999999999998e-06 2024-09-21T15:49:12.882 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.882 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.882 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:12.882 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.545955+0000", 2024-09-21T15:49:12.882 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9051999999999998e-05 2024-09-21T15:49:12.882 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.882 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.882 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:12.882 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.546020+0000", 2024-09-21T15:49:12.882 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5238000000000001e-05 2024-09-21T15:49:12.882 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.883 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.883 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-09-21T15:49:12.883 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.546137+0000", 2024-09-21T15:49:12.883 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000117064 2024-09-21T15:49:12.883 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.883 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.883 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:12.883 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.547081+0000", 2024-09-21T15:49:12.883 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00094388700000000002 2024-09-21T15:49:12.883 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.883 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.883 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:12.883 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.547332+0000", 2024-09-21T15:49:12.883 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00025075100000000001 2024-09-21T15:49:12.884 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.884 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.884 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:12.884 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.547362+0000", 2024-09-21T15:49:12.884 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9935000000000001e-05 2024-09-21T15:49:12.884 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.884 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.884 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:12.884 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.547386+0000", 2024-09-21T15:49:12.884 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4587e-05 2024-09-21T15:49:12.884 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:12.884 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:12.884 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:12.884 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.884 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.885 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:951 3.0 3:05942dfc:::rbd_data.1133f3dfbd6b.000000000000000e:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-21T15:49:12.885 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.571136+0000", 2024-09-21T15:49:12.885 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.207891258, 2024-09-21T15:49:12.885 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001074195, 2024-09-21T15:49:12.885 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:12.885 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:12.885 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:12.885 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:12.885 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:12.885 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 951 2024-09-21T15:49:12.885 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.885 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:12.885 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.885 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:12.885 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.571136+0000", 2024-09-21T15:49:12.886 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:12.886 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.886 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.886 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:12.886 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.571136+0000", 2024-09-21T15:49:12.886 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:12.886 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.886 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.886 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:12.886 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.571134+0000", 2024-09-21T15:49:12.886 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:49:12.886 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.886 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.886 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:12.887 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.571143+0000", 2024-09-21T15:49:12.887 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2890000000000003e-06 2024-09-21T15:49:12.887 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.887 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.887 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:12.887 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.571164+0000", 2024-09-21T15:49:12.887 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0346000000000001e-05 2024-09-21T15:49:12.887 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.887 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.887 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:12.887 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.571175+0000", 2024-09-21T15:49:12.887 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1608e-05 2024-09-21T15:49:12.887 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.887 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.888 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:12.888 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.571218+0000", 2024-09-21T15:49:12.888 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3161999999999998e-05 2024-09-21T15:49:12.888 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.888 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.888 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:12.888 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.571274+0000", 2024-09-21T15:49:12.888 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5995999999999997e-05 2024-09-21T15:49:12.888 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.888 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.888 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-09-21T15:49:12.888 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.571342+0000", 2024-09-21T15:49:12.888 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7562000000000001e-05 2024-09-21T15:49:12.888 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.888 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.889 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:12.889 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.571898+0000", 2024-09-21T15:49:12.889 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00055590399999999997 2024-09-21T15:49:12.889 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.889 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.889 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:12.889 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.572160+0000", 2024-09-21T15:49:12.889 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00026235599999999999 2024-09-21T15:49:12.889 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.889 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.889 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:12.889 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.572189+0000", 2024-09-21T15:49:12.889 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8870999999999999e-05 2024-09-21T15:49:12.889 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.889 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.890 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:12.890 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.572210+0000", 2024-09-21T15:49:12.890 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1047e-05 2024-09-21T15:49:12.890 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:12.890 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:12.890 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:12.890 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.890 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.890 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:954 3.19 3:9b8b44d5:::rbd_data.1133f3dfbd6b.0000000000000000:head [stat,truncate 9406627] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-21T15:49:12.890 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.574673+0000", 2024-09-21T15:49:12.890 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.20435425600000001, 2024-09-21T15:49:12.890 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002593635, 2024-09-21T15:49:12.890 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:12.890 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:12.891 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:12.891 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:12.891 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:12.891 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 954 2024-09-21T15:49:12.891 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.891 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:12.891 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.891 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:12.891 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.574673+0000", 2024-09-21T15:49:12.891 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:12.891 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.891 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.891 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:12.891 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.574673+0000", 2024-09-21T15:49:12.892 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:12.892 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.892 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.892 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:12.892 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.574672+0000", 2024-09-21T15:49:12.892 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:12.892 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.892 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.892 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:12.892 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.574690+0000", 2024-09-21T15:49:12.892 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8482e-05 2024-09-21T15:49:12.892 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.892 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.892 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:12.892 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.574703+0000", 2024-09-21T15:49:12.893 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2869e-05 2024-09-21T15:49:12.893 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.893 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.893 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:12.893 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.574709+0000", 2024-09-21T15:49:12.893 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5249999999999996e-06 2024-09-21T15:49:12.893 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.893 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.893 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:12.893 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.574757+0000", 2024-09-21T15:49:12.893 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8687000000000003e-05 2024-09-21T15:49:12.893 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.893 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.893 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:12.893 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.574841+0000", 2024-09-21T15:49:12.894 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3590000000000004e-05 2024-09-21T15:49:12.894 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.894 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.894 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:49:12.894 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.574992+0000", 2024-09-21T15:49:12.894 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015128 2024-09-21T15:49:12.894 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.894 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.894 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:12.894 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.575483+0000", 2024-09-21T15:49:12.894 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00049023100000000004 2024-09-21T15:49:12.894 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.894 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.894 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:12.895 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.577216+0000", 2024-09-21T15:49:12.895 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0017337559999999999 2024-09-21T15:49:12.895 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.895 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.895 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:12.895 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.577246+0000", 2024-09-21T15:49:12.895 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9683999999999999e-05 2024-09-21T15:49:12.895 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.895 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.895 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:12.895 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.577266+0000", 2024-09-21T15:49:12.895 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0445000000000002e-05 2024-09-21T15:49:12.895 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:12.895 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:12.895 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:12.896 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.896 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.896 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:981 3.19 3:9b8b44d5:::rbd_data.1133f3dfbd6b.0000000000000000:head [stat,write 10643544~22727] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-21T15:49:12.896 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.609187+0000", 2024-09-21T15:49:12.896 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.169840519, 2024-09-21T15:49:12.896 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0040569439999999998, 2024-09-21T15:49:12.896 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:12.896 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:12.896 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:12.896 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:12.896 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:12.896 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 981 2024-09-21T15:49:12.896 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.896 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:12.896 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.897 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:12.897 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.609187+0000", 2024-09-21T15:49:12.897 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:12.897 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.897 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.897 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:12.897 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.609187+0000", 2024-09-21T15:49:12.897 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:12.897 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.897 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.897 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:12.897 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.609184+0000", 2024-09-21T15:49:12.897 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999971 2024-09-21T15:49:12.897 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.897 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.898 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:12.898 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.609221+0000", 2024-09-21T15:49:12.898 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6739999999999997e-05 2024-09-21T15:49:12.898 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.898 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.898 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:12.898 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.609242+0000", 2024-09-21T15:49:12.898 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1154000000000001e-05 2024-09-21T15:49:12.898 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.898 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.898 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:12.898 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.609252+0000", 2024-09-21T15:49:12.898 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0155e-05 2024-09-21T15:49:12.898 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.898 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.899 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:12.899 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.609303+0000", 2024-09-21T15:49:12.899 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1316000000000001e-05 2024-09-21T15:49:12.899 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.899 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.899 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:12.899 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.609392+0000", 2024-09-21T15:49:12.899 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8661999999999999e-05 2024-09-21T15:49:12.899 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.899 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.899 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:49:12.899 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.609550+0000", 2024-09-21T15:49:12.899 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015781200000000001 2024-09-21T15:49:12.899 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.900 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.900 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:12.900 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.610334+0000", 2024-09-21T15:49:12.900 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00078432700000000003 2024-09-21T15:49:12.900 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.900 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.900 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:12.900 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.613192+0000", 2024-09-21T15:49:12.900 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0028579149999999999 2024-09-21T15:49:12.900 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.900 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.900 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:12.900 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.613222+0000", 2024-09-21T15:49:12.900 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9785e-05 2024-09-21T15:49:12.900 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.900 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.901 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:12.901 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.613244+0000", 2024-09-21T15:49:12.901 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1712000000000002e-05 2024-09-21T15:49:12.901 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:12.901 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:12.901 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:12.901 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.901 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.901 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:983 3.0 3:025e53e7:::rbd_data.1133f3dfbd6b.0000000000000008:head [set-alloc-hint object_size 16777216 write_size 16777216,write 2211719~34385] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-21T15:49:12.901 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.615069+0000", 2024-09-21T15:49:12.901 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.16395773399999999, 2024-09-21T15:49:12.901 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0037302559999999999, 2024-09-21T15:49:12.901 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:12.901 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:12.901 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:12.902 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:12.902 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:12.902 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 983 2024-09-21T15:49:12.902 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.902 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:12.902 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.902 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:12.902 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.615069+0000", 2024-09-21T15:49:12.902 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:12.902 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.902 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.902 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:12.902 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.615069+0000", 2024-09-21T15:49:12.902 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:12.903 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.903 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.903 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:12.903 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.615068+0000", 2024-09-21T15:49:12.903 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:12.903 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.903 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.903 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:12.903 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.615086+0000", 2024-09-21T15:49:12.903 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7966e-05 2024-09-21T15:49:12.903 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.903 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.903 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:12.903 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.615101+0000", 2024-09-21T15:49:12.904 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5e-05 2024-09-21T15:49:12.904 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.904 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.904 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:12.904 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.615109+0000", 2024-09-21T15:49:12.904 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7489999999999998e-06 2024-09-21T15:49:12.904 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.904 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.904 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:12.904 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.615155+0000", 2024-09-21T15:49:12.904 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6343e-05 2024-09-21T15:49:12.904 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.904 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.904 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:12.904 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.615207+0000", 2024-09-21T15:49:12.905 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2145000000000003e-05 2024-09-21T15:49:12.905 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.905 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.905 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-09-21T15:49:12.905 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.615293+0000", 2024-09-21T15:49:12.905 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5385999999999999e-05 2024-09-21T15:49:12.905 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.905 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.905 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:12.905 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.615892+0000", 2024-09-21T15:49:12.905 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00059898299999999998 2024-09-21T15:49:12.905 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.905 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.905 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:12.905 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.618751+0000", 2024-09-21T15:49:12.905 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0028590709999999999 2024-09-21T15:49:12.906 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.906 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.906 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:12.906 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.618780+0000", 2024-09-21T15:49:12.906 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8949999999999999e-05 2024-09-21T15:49:12.906 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.906 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.906 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:12.906 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.618800+0000", 2024-09-21T15:49:12.906 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9992000000000001e-05 2024-09-21T15:49:12.906 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:12.906 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:12.906 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:12.906 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.906 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.907 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:984 3.0 3:025e53e7:::rbd_data.1133f3dfbd6b.0000000000000008:head [write 6528534~38661] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-21T15:49:12.907 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.619034+0000", 2024-09-21T15:49:12.907 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.159993053, 2024-09-21T15:49:12.907 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0039438279999999999, 2024-09-21T15:49:12.907 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:12.907 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:12.907 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:12.907 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:12.907 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:12.907 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 984 2024-09-21T15:49:12.907 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.907 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:12.907 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.907 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:12.907 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619034+0000", 2024-09-21T15:49:12.908 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:12.908 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.908 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.908 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:12.908 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619034+0000", 2024-09-21T15:49:12.908 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:12.908 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.908 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.908 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:12.908 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619033+0000", 2024-09-21T15:49:12.908 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:12.908 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.908 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.908 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:12.908 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619050+0000", 2024-09-21T15:49:12.908 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6654999999999999e-05 2024-09-21T15:49:12.909 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.909 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.909 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:12.909 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619064+0000", 2024-09-21T15:49:12.909 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4429e-05 2024-09-21T15:49:12.909 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.909 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.909 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:12.909 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619072+0000", 2024-09-21T15:49:12.909 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.2439999999999993e-06 2024-09-21T15:49:12.909 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.909 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.909 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:12.909 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619114+0000", 2024-09-21T15:49:12.909 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1789999999999998e-05 2024-09-21T15:49:12.910 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.910 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.910 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:12.910 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619195+0000", 2024-09-21T15:49:12.910 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0560000000000001e-05 2024-09-21T15:49:12.910 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.910 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.910 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-09-21T15:49:12.910 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619328+0000", 2024-09-21T15:49:12.910 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000133676 2024-09-21T15:49:12.910 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.910 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.910 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:12.910 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619990+0000", 2024-09-21T15:49:12.911 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00066200999999999999 2024-09-21T15:49:12.911 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.911 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.911 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:12.911 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.622898+0000", 2024-09-21T15:49:12.911 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002907658 2024-09-21T15:49:12.911 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.911 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.911 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:12.911 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.622941+0000", 2024-09-21T15:49:12.911 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3025999999999998e-05 2024-09-21T15:49:12.911 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.911 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.911 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:12.911 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.622978+0000", 2024-09-21T15:49:12.912 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6798999999999998e-05 2024-09-21T15:49:12.912 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:12.912 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:12.912 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:12.912 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.912 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.912 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:999 3.0 3:025e53e7:::rbd_data.1133f3dfbd6b.0000000000000008:head [write 7002043~50837] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-21T15:49:12.912 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.638307+0000", 2024-09-21T15:49:12.912 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.14072024299999999, 2024-09-21T15:49:12.912 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0014479180000000001, 2024-09-21T15:49:12.912 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:12.912 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:12.912 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:12.912 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:12.913 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:12.913 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 999 2024-09-21T15:49:12.913 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.913 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:12.913 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.913 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:12.913 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.638307+0000", 2024-09-21T15:49:12.913 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:12.913 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.913 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.913 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:12.913 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.638307+0000", 2024-09-21T15:49:12.913 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:12.913 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.913 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.914 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:12.914 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.638305+0000", 2024-09-21T15:49:12.914 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:49:12.914 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.914 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.914 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:12.914 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.638329+0000", 2024-09-21T15:49:12.914 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4006e-05 2024-09-21T15:49:12.914 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.914 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.914 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:12.914 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.638347+0000", 2024-09-21T15:49:12.914 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8029999999999998e-05 2024-09-21T15:49:12.914 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.915 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.915 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:12.915 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.638357+0000", 2024-09-21T15:49:12.915 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0213e-05 2024-09-21T15:49:12.915 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.915 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.915 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:12.915 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.638398+0000", 2024-09-21T15:49:12.915 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0815000000000001e-05 2024-09-21T15:49:12.915 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.915 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.915 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:12.915 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.638454+0000", 2024-09-21T15:49:12.915 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5591999999999999e-05 2024-09-21T15:49:12.915 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.916 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.916 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-09-21T15:49:12.916 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.638530+0000", 2024-09-21T15:49:12.916 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6037000000000004e-05 2024-09-21T15:49:12.916 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.916 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.916 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:12.916 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.639328+0000", 2024-09-21T15:49:12.916 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00079817199999999999 2024-09-21T15:49:12.916 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.916 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.916 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:12.916 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.639669+0000", 2024-09-21T15:49:12.916 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00034078800000000002 2024-09-21T15:49:12.917 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.917 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.917 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:12.917 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.639717+0000", 2024-09-21T15:49:12.917 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7895000000000001e-05 2024-09-21T15:49:12.917 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.917 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.917 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:12.917 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.639755+0000", 2024-09-21T15:49:12.917 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8207000000000002e-05 2024-09-21T15:49:12.917 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:12.917 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:12.917 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:12.917 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.917 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.918 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:1005 3.0 3:025e53e7:::rbd_data.1133f3dfbd6b.0000000000000008:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-21T15:49:12.918 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.669413+0000", 2024-09-21T15:49:12.918 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.109614244, 2024-09-21T15:49:12.918 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0011980630000000001, 2024-09-21T15:49:12.918 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:12.918 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:12.918 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:12.918 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:12.918 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:12.918 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1005 2024-09-21T15:49:12.918 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.918 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:12.918 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.918 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:12.918 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.669413+0000", 2024-09-21T15:49:12.919 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:12.919 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.919 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.919 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:12.919 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.669413+0000", 2024-09-21T15:49:12.919 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:12.919 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.919 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.919 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:12.919 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.669411+0000", 2024-09-21T15:49:12.919 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:49:12.919 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.919 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.919 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:12.919 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.669422+0000", 2024-09-21T15:49:12.919 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1316e-05 2024-09-21T15:49:12.920 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.920 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.920 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:12.920 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.669443+0000", 2024-09-21T15:49:12.920 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1185000000000001e-05 2024-09-21T15:49:12.920 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.920 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.920 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:12.920 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.669454+0000", 2024-09-21T15:49:12.920 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0533000000000001e-05 2024-09-21T15:49:12.920 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.920 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.920 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:12.920 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.669495+0000", 2024-09-21T15:49:12.920 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0720000000000003e-05 2024-09-21T15:49:12.920 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.920 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.921 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:12.921 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.669548+0000", 2024-09-21T15:49:12.921 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3338999999999999e-05 2024-09-21T15:49:12.921 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.921 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.921 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-09-21T15:49:12.921 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.669614+0000", 2024-09-21T15:49:12.921 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5616000000000003e-05 2024-09-21T15:49:12.921 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.921 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.921 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:12.921 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.670265+0000", 2024-09-21T15:49:12.921 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00065141299999999999 2024-09-21T15:49:12.921 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.922 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.922 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:12.922 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.670547+0000", 2024-09-21T15:49:12.922 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00028231300000000002 2024-09-21T15:49:12.922 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.922 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.922 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:12.922 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.670574+0000", 2024-09-21T15:49:12.922 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7104e-05 2024-09-21T15:49:12.922 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.922 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.922 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:12.922 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.670611+0000", 2024-09-21T15:49:12.922 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6559e-05 2024-09-21T15:49:12.923 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:12.923 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:12.923 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:12.923 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.923 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.923 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:1042 3.6 e27/22 3:622141b5:::rbd_data.1133f3dfbd6b.000000000000000a:head v 27'4, mlcod=27'2)", 2024-09-21T15:49:12.923 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.703829+0000", 2024-09-21T15:49:12.923 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.075198046000000004, 2024-09-21T15:49:12.923 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029263179999999998, 2024-09-21T15:49:12.923 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:12.923 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:12.923 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:12.923 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.924 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:12.924 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.703829+0000", 2024-09-21T15:49:12.924 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:12.924 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.924 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.924 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:12.924 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.703829+0000", 2024-09-21T15:49:12.924 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:12.924 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.924 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.924 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:12.924 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.703827+0000", 2024-09-21T15:49:12.924 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:12.924 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.924 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.925 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:12.925 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.703889+0000", 2024-09-21T15:49:12.925 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1820000000000001e-05 2024-09-21T15:49:12.925 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.925 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.925 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:12.925 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.703906+0000", 2024-09-21T15:49:12.925 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6409000000000002e-05 2024-09-21T15:49:12.925 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.925 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.925 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:12.925 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.703913+0000", 2024-09-21T15:49:12.925 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1559999999999998e-06 2024-09-21T15:49:12.925 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.925 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.926 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:12.926 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.703964+0000", 2024-09-21T15:49:12.926 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1035000000000002e-05 2024-09-21T15:49:12.926 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.926 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.926 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:12.926 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.704017+0000", 2024-09-21T15:49:12.926 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3301999999999999e-05 2024-09-21T15:49:12.926 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.926 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.926 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:12.926 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.706719+0000", 2024-09-21T15:49:12.926 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0027016560000000002 2024-09-21T15:49:12.926 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.927 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.927 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:12.927 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.706755+0000", 2024-09-21T15:49:12.927 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.659e-05 2024-09-21T15:49:12.927 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:12.927 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:12.927 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:12.927 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.927 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.927 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:1044 3.0 3:025e53e7:::rbd_data.1133f3dfbd6b.0000000000000008:head [set-alloc-hint object_size 16777216 write_size 16777216,write 3022774~40121] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-21T15:49:12.927 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.709547+0000", 2024-09-21T15:49:12.927 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.069480060999999996, 2024-09-21T15:49:12.927 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0027353299999999998, 2024-09-21T15:49:12.927 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:12.927 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:12.928 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:12.928 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:12.928 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:12.928 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1044 2024-09-21T15:49:12.928 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.928 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:12.928 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.928 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:12.928 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.709547+0000", 2024-09-21T15:49:12.928 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:12.928 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.928 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.928 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:12.928 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.709547+0000", 2024-09-21T15:49:12.929 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:12.929 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.929 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.929 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:12.929 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.709546+0000", 2024-09-21T15:49:12.929 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:12.929 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.929 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.929 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:12.929 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.709566+0000", 2024-09-21T15:49:12.929 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0305e-05 2024-09-21T15:49:12.929 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.929 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.929 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:12.929 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.709577+0000", 2024-09-21T15:49:12.930 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.065e-05 2024-09-21T15:49:12.930 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.930 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.930 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:12.930 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.709593+0000", 2024-09-21T15:49:12.930 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6384000000000001e-05 2024-09-21T15:49:12.930 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.930 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.930 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:12.930 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.709636+0000", 2024-09-21T15:49:12.930 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2453000000000002e-05 2024-09-21T15:49:12.930 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.930 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.930 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:12.930 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.709696+0000", 2024-09-21T15:49:12.931 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0779999999999997e-05 2024-09-21T15:49:12.931 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.931 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.931 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-09-21T15:49:12.931 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.709791+0000", 2024-09-21T15:49:12.931 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4903999999999998e-05 2024-09-21T15:49:12.931 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.931 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.931 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:12.931 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.711746+0000", 2024-09-21T15:49:12.931 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0019550980000000002 2024-09-21T15:49:12.931 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.931 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.931 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:12.931 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.712222+0000", 2024-09-21T15:49:12.932 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000475682 2024-09-21T15:49:12.932 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.932 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.932 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:12.932 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.712258+0000", 2024-09-21T15:49:12.932 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6368999999999997e-05 2024-09-21T15:49:12.932 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.932 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.932 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:12.932 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.712282+0000", 2024-09-21T15:49:12.932 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3886000000000001e-05 2024-09-21T15:49:12.932 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:12.932 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:12.932 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:12.932 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.932 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.933 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:1060 3.19 3:9b8b44d5:::rbd_data.1133f3dfbd6b.0000000000000000:head [stat,write 7858638~26966] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-21T15:49:12.933 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.737760+0000", 2024-09-21T15:49:12.933 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.041267267000000003, 2024-09-21T15:49:12.933 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0018353969999999999, 2024-09-21T15:49:12.933 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:12.933 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:12.933 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:12.933 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:12.933 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:12.933 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1060 2024-09-21T15:49:12.933 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.933 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:12.933 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.933 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:12.933 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.737760+0000", 2024-09-21T15:49:12.934 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:12.934 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.934 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.934 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:12.934 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.737760+0000", 2024-09-21T15:49:12.934 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:12.934 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.934 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.934 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:12.934 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.737758+0000", 2024-09-21T15:49:12.934 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:12.934 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.934 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.934 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:12.934 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.737786+0000", 2024-09-21T15:49:12.935 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8042e-05 2024-09-21T15:49:12.935 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.935 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.935 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:12.935 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.737806+0000", 2024-09-21T15:49:12.935 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9918000000000001e-05 2024-09-21T15:49:12.935 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.935 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.935 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:12.935 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.737814+0000", 2024-09-21T15:49:12.935 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3000000000000004e-06 2024-09-21T15:49:12.935 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.935 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.935 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:12.936 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.737853+0000", 2024-09-21T15:49:12.936 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9212000000000003e-05 2024-09-21T15:49:12.936 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.936 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.936 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:12.936 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.737907+0000", 2024-09-21T15:49:12.936 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3887000000000001e-05 2024-09-21T15:49:12.936 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.936 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.936 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:49:12.936 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.738001+0000", 2024-09-21T15:49:12.936 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4549999999999994e-05 2024-09-21T15:49:12.936 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.936 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.937 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:12.937 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.738660+0000", 2024-09-21T15:49:12.937 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00065848600000000001 2024-09-21T15:49:12.937 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.937 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.937 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:12.937 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.739508+0000", 2024-09-21T15:49:12.937 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00084874199999999999 2024-09-21T15:49:12.937 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.937 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.937 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:12.937 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.739556+0000", 2024-09-21T15:49:12.937 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8084000000000002e-05 2024-09-21T15:49:12.937 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:12.937 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:12.938 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:12.938 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.739595+0000", 2024-09-21T15:49:12.938 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8739999999999998e-05 2024-09-21T15:49:12.938 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:12.938 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:12.938 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:12.938 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:12.938 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:12.938 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:12.940 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:49:12.940 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:12.938+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:13.039 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:13.038+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:13.084 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:13.084 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:49:13.084 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:49:13.084 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:13.092 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:49:13.176 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:13.175+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:13.233 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:13.234 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:49:13.234 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:49:13.234 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:49:13.234 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:13.241 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:13.240+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:13.242 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:49:13.343 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:13.341+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:13.385 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:13.385 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:49:13.385 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:49:13.385 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:49:13.385 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.385 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4275.0:1 2.3 e18/17 2:c4c92e5a:::rbd_trash:head v 18'1, mlcod=0'0)", 2024-09-21T15:49:13.385 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:48:57.586388+0000", 2024-09-21T15:49:13.385 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 15.797633810000001, 2024-09-21T15:49:13.386 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00075578400000000001, 2024-09-21T15:49:13.386 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:13.386 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:13.386 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:13.386 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.386 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:13.386 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:57.586388+0000", 2024-09-21T15:49:13.386 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.386 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.386 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.386 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:13.386 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:57.586388+0000", 2024-09-21T15:49:13.386 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.386 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.387 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.387 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:13.387 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:57.586388+0000", 2024-09-21T15:49:13.387 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:49:13.387 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.387 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.387 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:13.387 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:57.586396+0000", 2024-09-21T15:49:13.387 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4339999999999992e-06 2024-09-21T15:49:13.387 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.387 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.387 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:13.387 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:57.586406+0000", 2024-09-21T15:49:13.387 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9639999999999998e-06 2024-09-21T15:49:13.387 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.388 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.388 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:13.388 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:57.586423+0000", 2024-09-21T15:49:13.388 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7005000000000001e-05 2024-09-21T15:49:13.388 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.388 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.388 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:13.388 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:57.586459+0000", 2024-09-21T15:49:13.388 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5828999999999999e-05 2024-09-21T15:49:13.388 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.388 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.388 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:13.388 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:57.586482+0000", 2024-09-21T15:49:13.388 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2841e-05 2024-09-21T15:49:13.388 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.389 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.389 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:13.389 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:57.587111+0000", 2024-09-21T15:49:13.389 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00062914999999999998 2024-09-21T15:49:13.389 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.389 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.389 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:13.389 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:48:57.587144+0000", 2024-09-21T15:49:13.389 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2790999999999998e-05 2024-09-21T15:49:13.389 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.389 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:13.389 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.389 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.389 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.389 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:887 3.0 e27/22 3:025e53e7:::rbd_data.1133f3dfbd6b.0000000000000008:head v 27'14, mlcod=27'12)", 2024-09-21T15:49:13.390 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.458555+0000", 2024-09-21T15:49:13.390 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.92546710799999998, 2024-09-21T15:49:13.390 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00080860600000000002, 2024-09-21T15:49:13.390 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:13.390 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:13.390 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:13.390 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.390 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:13.390 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.458555+0000", 2024-09-21T15:49:13.390 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.390 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.390 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.390 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:13.390 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.458555+0000", 2024-09-21T15:49:13.391 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.391 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.391 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.391 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:13.391 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.458553+0000", 2024-09-21T15:49:13.391 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:49:13.391 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.391 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.391 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:13.391 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.458583+0000", 2024-09-21T15:49:13.391 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0307999999999998e-05 2024-09-21T15:49:13.391 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.391 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.391 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:13.391 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.458603+0000", 2024-09-21T15:49:13.392 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.997e-05 2024-09-21T15:49:13.392 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.392 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.392 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:13.392 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.458614+0000", 2024-09-21T15:49:13.392 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0618999999999999e-05 2024-09-21T15:49:13.392 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.392 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.392 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:13.392 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.458667+0000", 2024-09-21T15:49:13.392 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3625999999999998e-05 2024-09-21T15:49:13.392 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.392 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.392 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:13.393 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.458689+0000", 2024-09-21T15:49:13.393 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1812000000000001e-05 2024-09-21T15:49:13.393 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.393 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.393 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:13.393 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.459327+0000", 2024-09-21T15:49:13.393 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00063789999999999995 2024-09-21T15:49:13.393 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.393 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.393 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:13.393 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.459363+0000", 2024-09-21T15:49:13.393 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6217000000000002e-05 2024-09-21T15:49:13.393 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.393 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:13.393 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.394 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.394 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.394 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:888 3.0 e27/22 3:025e53e7:::rbd_data.1133f3dfbd6b.0000000000000008:head v 27'15, mlcod=27'13)", 2024-09-21T15:49:13.394 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.459921+0000", 2024-09-21T15:49:13.394 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.92410061499999996, 2024-09-21T15:49:13.394 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00084743299999999998, 2024-09-21T15:49:13.394 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:13.394 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:13.394 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:13.394 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.394 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:13.394 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.459921+0000", 2024-09-21T15:49:13.394 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.395 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.395 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.395 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:13.395 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.459921+0000", 2024-09-21T15:49:13.395 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.395 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.395 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.395 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:13.395 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.459921+0000", 2024-09-21T15:49:13.395 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:13.395 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.395 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.395 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:13.395 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.459956+0000", 2024-09-21T15:49:13.396 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5673999999999998e-05 2024-09-21T15:49:13.396 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.396 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.396 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:13.396 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.459970+0000", 2024-09-21T15:49:13.396 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3443e-05 2024-09-21T15:49:13.396 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.396 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.396 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:13.396 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.459976+0000", 2024-09-21T15:49:13.396 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5100000000000004e-06 2024-09-21T15:49:13.396 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.396 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.396 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:13.396 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.460020+0000", 2024-09-21T15:49:13.397 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4132999999999999e-05 2024-09-21T15:49:13.397 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.397 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.397 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:13.397 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.460040+0000", 2024-09-21T15:49:13.397 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9466e-05 2024-09-21T15:49:13.397 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.397 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.397 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:13.397 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.460742+0000", 2024-09-21T15:49:13.397 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00070200500000000001 2024-09-21T15:49:13.397 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.397 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.397 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:13.397 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.460769+0000", 2024-09-21T15:49:13.398 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6834000000000001e-05 2024-09-21T15:49:13.398 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.398 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:13.398 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.398 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.398 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.398 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:904 3.0 e27/22 3:025e53e7:::rbd_data.1133f3dfbd6b.0000000000000008:head v 27'16, mlcod=27'14)", 2024-09-21T15:49:13.398 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.483729+0000", 2024-09-21T15:49:13.398 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.900293173, 2024-09-21T15:49:13.398 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00079325900000000004, 2024-09-21T15:49:13.398 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:13.398 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:13.398 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:13.398 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.399 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:13.399 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.483729+0000", 2024-09-21T15:49:13.399 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.399 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.399 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.399 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:13.399 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.483729+0000", 2024-09-21T15:49:13.399 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.399 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.399 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.399 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:13.399 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.483727+0000", 2024-09-21T15:49:13.399 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:13.400 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.400 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.400 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:13.400 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.483753+0000", 2024-09-21T15:49:13.400 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5922e-05 2024-09-21T15:49:13.400 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.400 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.400 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:13.400 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.483771+0000", 2024-09-21T15:49:13.400 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7972e-05 2024-09-21T15:49:13.400 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.400 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.400 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:13.400 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.483781+0000", 2024-09-21T15:49:13.400 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0064999999999999e-05 2024-09-21T15:49:13.401 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.401 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.401 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:13.401 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.483824+0000", 2024-09-21T15:49:13.401 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2404000000000003e-05 2024-09-21T15:49:13.401 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.401 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.401 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:13.401 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.483845+0000", 2024-09-21T15:49:13.401 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1573000000000002e-05 2024-09-21T15:49:13.401 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.401 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.401 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:13.401 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.484478+0000", 2024-09-21T15:49:13.401 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00063268799999999996 2024-09-21T15:49:13.401 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.402 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.402 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:13.402 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.484522+0000", 2024-09-21T15:49:13.402 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3760000000000001e-05 2024-09-21T15:49:13.402 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.402 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:13.402 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.402 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.402 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.402 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:906 3.0 e27/22 3:025e53e7:::rbd_data.1133f3dfbd6b.0000000000000008:head v 27'17, mlcod=27'15)", 2024-09-21T15:49:13.402 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.492830+0000", 2024-09-21T15:49:13.402 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.89119209099999996, 2024-09-21T15:49:13.402 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00067820000000000001, 2024-09-21T15:49:13.402 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:13.402 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:13.403 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:13.403 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.403 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:13.403 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.492830+0000", 2024-09-21T15:49:13.403 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.403 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.403 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.403 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:13.403 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.492830+0000", 2024-09-21T15:49:13.403 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.403 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.403 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.403 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:13.403 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.492829+0000", 2024-09-21T15:49:13.403 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:13.404 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.404 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.404 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:13.404 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.492839+0000", 2024-09-21T15:49:13.404 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0097e-05 2024-09-21T15:49:13.404 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.404 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.404 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:13.404 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.492854+0000", 2024-09-21T15:49:13.404 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.502e-05 2024-09-21T15:49:13.404 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.404 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.404 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:13.404 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.492862+0000", 2024-09-21T15:49:13.404 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7430000000000008e-06 2024-09-21T15:49:13.404 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.405 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.405 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:13.405 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.492907+0000", 2024-09-21T15:49:13.405 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5009000000000002e-05 2024-09-21T15:49:13.405 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.405 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.405 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:13.405 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.492924+0000", 2024-09-21T15:49:13.405 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6914000000000001e-05 2024-09-21T15:49:13.405 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.405 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.405 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:13.405 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.493477+0000", 2024-09-21T15:49:13.405 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00055308400000000002 2024-09-21T15:49:13.405 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.405 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.406 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:13.406 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.493508+0000", 2024-09-21T15:49:13.406 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0378e-05 2024-09-21T15:49:13.406 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.406 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:13.406 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.406 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.406 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.406 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:945 3.0 e27/22 3:05942dfc:::rbd_data.1133f3dfbd6b.000000000000000e:head v 27'18, mlcod=27'16)", 2024-09-21T15:49:13.406 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.546272+0000", 2024-09-21T15:49:13.406 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.83774984699999999, 2024-09-21T15:49:13.406 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00096418899999999995, 2024-09-21T15:49:13.406 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:13.406 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:13.407 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:13.407 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.407 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:13.407 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.546272+0000", 2024-09-21T15:49:13.407 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.407 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.407 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.407 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:13.407 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.546272+0000", 2024-09-21T15:49:13.407 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.407 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.407 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.407 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:13.407 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.546270+0000", 2024-09-21T15:49:13.407 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:13.408 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.408 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.408 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:13.408 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.546320+0000", 2024-09-21T15:49:13.408 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0043999999999997e-05 2024-09-21T15:49:13.408 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.408 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.408 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:13.408 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.546338+0000", 2024-09-21T15:49:13.408 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7841000000000001e-05 2024-09-21T15:49:13.408 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.408 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.408 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:13.408 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.546348+0000", 2024-09-21T15:49:13.408 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4890000000000001e-06 2024-09-21T15:49:13.408 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.408 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.409 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:13.409 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.546392+0000", 2024-09-21T15:49:13.409 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4363000000000002e-05 2024-09-21T15:49:13.409 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.409 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.409 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:13.409 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.546414+0000", 2024-09-21T15:49:13.409 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2164999999999999e-05 2024-09-21T15:49:13.409 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.409 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.409 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:13.409 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.547211+0000", 2024-09-21T15:49:13.409 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00079653099999999995 2024-09-21T15:49:13.409 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.409 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.409 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:13.410 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.547236+0000", 2024-09-21T15:49:13.410 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5364e-05 2024-09-21T15:49:13.410 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.410 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:13.410 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.410 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.410 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.410 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:951 3.0 e27/22 3:05942dfc:::rbd_data.1133f3dfbd6b.000000000000000e:head v 27'19, mlcod=27'17)", 2024-09-21T15:49:13.410 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.571431+0000", 2024-09-21T15:49:13.410 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.81259124599999999, 2024-09-21T15:49:13.410 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00063992400000000005, 2024-09-21T15:49:13.410 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:13.410 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:13.410 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:13.410 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.411 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:13.411 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.571431+0000", 2024-09-21T15:49:13.411 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.411 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.411 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.411 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:13.411 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.571431+0000", 2024-09-21T15:49:13.411 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.411 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.411 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.411 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:13.411 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.571429+0000", 2024-09-21T15:49:13.411 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:13.411 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.411 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.412 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:13.412 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.571439+0000", 2024-09-21T15:49:13.412 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6870000000000006e-06 2024-09-21T15:49:13.412 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.412 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.412 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:13.412 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.571460+0000", 2024-09-21T15:49:13.412 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0832e-05 2024-09-21T15:49:13.412 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.412 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.412 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:13.412 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.571469+0000", 2024-09-21T15:49:13.412 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1889999999999996e-06 2024-09-21T15:49:13.412 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.412 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.412 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:13.413 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.571521+0000", 2024-09-21T15:49:13.413 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1755000000000002e-05 2024-09-21T15:49:13.413 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.413 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.413 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:13.413 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.571546+0000", 2024-09-21T15:49:13.413 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4742000000000001e-05 2024-09-21T15:49:13.413 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.413 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.413 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:13.413 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.572043+0000", 2024-09-21T15:49:13.413 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00049764700000000004 2024-09-21T15:49:13.413 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.413 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.414 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:13.414 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.572070+0000", 2024-09-21T15:49:13.414 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7151000000000001e-05 2024-09-21T15:49:13.414 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.414 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:13.414 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.414 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.414 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.414 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:983 3.0 e27/22 3:025e53e7:::rbd_data.1133f3dfbd6b.0000000000000008:head v 27'20, mlcod=27'18)", 2024-09-21T15:49:13.414 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.615392+0000", 2024-09-21T15:49:13.414 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.76863003399999996, 2024-09-21T15:49:13.414 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032613970000000001, 2024-09-21T15:49:13.414 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:13.414 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:13.414 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:13.415 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.415 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:13.415 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.615392+0000", 2024-09-21T15:49:13.415 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.415 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.415 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.415 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:13.415 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.615392+0000", 2024-09-21T15:49:13.415 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.415 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.415 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.415 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:13.415 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.615390+0000", 2024-09-21T15:49:13.415 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:13.415 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.415 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.416 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:13.416 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.615431+0000", 2024-09-21T15:49:13.416 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0893999999999998e-05 2024-09-21T15:49:13.416 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.416 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.416 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:13.416 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.615450+0000", 2024-09-21T15:49:13.416 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8961e-05 2024-09-21T15:49:13.416 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.416 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.416 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:13.416 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.615458+0000", 2024-09-21T15:49:13.416 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1620000000000008e-06 2024-09-21T15:49:13.416 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.416 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.416 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:13.417 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.615501+0000", 2024-09-21T15:49:13.417 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2116999999999997e-05 2024-09-21T15:49:13.417 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.417 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.417 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:13.417 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.615518+0000", 2024-09-21T15:49:13.417 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7689e-05 2024-09-21T15:49:13.417 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.417 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.417 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:13.417 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.618621+0000", 2024-09-21T15:49:13.417 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0031022340000000002 2024-09-21T15:49:13.417 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.417 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.417 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:13.418 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.618653+0000", 2024-09-21T15:49:13.418 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2591e-05 2024-09-21T15:49:13.418 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.418 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:13.418 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.418 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.418 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.418 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:984 3.0 e27/22 3:025e53e7:::rbd_data.1133f3dfbd6b.0000000000000008:head v 27'21, mlcod=27'19)", 2024-09-21T15:49:13.418 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.619393+0000", 2024-09-21T15:49:13.418 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.76462920700000003, 2024-09-21T15:49:13.418 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003401837, 2024-09-21T15:49:13.418 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:13.418 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:13.418 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:13.418 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.418 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:13.419 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619393+0000", 2024-09-21T15:49:13.419 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.419 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.419 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.419 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:13.419 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619393+0000", 2024-09-21T15:49:13.419 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.419 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.419 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.419 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:13.419 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619392+0000", 2024-09-21T15:49:13.419 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:13.419 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.419 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.419 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:13.419 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619439+0000", 2024-09-21T15:49:13.420 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7170999999999999e-05 2024-09-21T15:49:13.420 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.420 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.420 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:13.420 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619452+0000", 2024-09-21T15:49:13.420 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2986999999999999e-05 2024-09-21T15:49:13.420 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.420 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.420 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:13.420 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619459+0000", 2024-09-21T15:49:13.420 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6109999999999999e-06 2024-09-21T15:49:13.420 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.420 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.420 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:13.420 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619518+0000", 2024-09-21T15:49:13.420 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9117999999999999e-05 2024-09-21T15:49:13.421 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.421 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.421 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:13.421 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619532+0000", 2024-09-21T15:49:13.421 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4622e-05 2024-09-21T15:49:13.421 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.421 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.421 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:13.421 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.622760+0000", 2024-09-21T15:49:13.421 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032278839999999999 2024-09-21T15:49:13.421 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.421 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.421 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:13.421 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.622794+0000", 2024-09-21T15:49:13.421 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4144999999999999e-05 2024-09-21T15:49:13.421 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.421 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:13.422 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.422 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.422 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.422 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:999 3.0 e27/22 3:025e53e7:::rbd_data.1133f3dfbd6b.0000000000000008:head v 27'22, mlcod=27'20)", 2024-09-21T15:49:13.422 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.638611+0000", 2024-09-21T15:49:13.422 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.74541084099999999, 2024-09-21T15:49:13.422 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00094990500000000002, 2024-09-21T15:49:13.422 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:13.422 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:13.422 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:13.422 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.422 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:13.422 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.638611+0000", 2024-09-21T15:49:13.422 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.422 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.422 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.423 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:13.423 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.638611+0000", 2024-09-21T15:49:13.423 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.423 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.423 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.423 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:13.423 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.638610+0000", 2024-09-21T15:49:13.423 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:13.423 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.423 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.423 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:13.423 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.638659+0000", 2024-09-21T15:49:13.423 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9434999999999997e-05 2024-09-21T15:49:13.423 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.424 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.424 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:13.424 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.638678+0000", 2024-09-21T15:49:13.424 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8992e-05 2024-09-21T15:49:13.424 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.424 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.424 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:13.424 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.638687+0000", 2024-09-21T15:49:13.424 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4379999999999995e-06 2024-09-21T15:49:13.424 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.424 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.424 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:13.424 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.638733+0000", 2024-09-21T15:49:13.424 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6338000000000003e-05 2024-09-21T15:49:13.424 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.425 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.425 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:13.425 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.638757+0000", 2024-09-21T15:49:13.425 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3825999999999999e-05 2024-09-21T15:49:13.425 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.425 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.425 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:13.425 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.639524+0000", 2024-09-21T15:49:13.425 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00076672000000000003 2024-09-21T15:49:13.425 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.425 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.425 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:13.425 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.639561+0000", 2024-09-21T15:49:13.425 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7317999999999997e-05 2024-09-21T15:49:13.426 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.426 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:13.426 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.426 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.426 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.426 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:1005 3.0 e27/22 3:025e53e7:::rbd_data.1133f3dfbd6b.0000000000000008:head v 27'23, mlcod=27'21)", 2024-09-21T15:49:13.426 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.669698+0000", 2024-09-21T15:49:13.426 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.714323767, 2024-09-21T15:49:13.426 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00074938200000000004, 2024-09-21T15:49:13.426 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:13.426 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:13.426 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:13.426 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.427 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:13.427 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.669698+0000", 2024-09-21T15:49:13.427 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.427 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.427 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.427 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:13.427 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.669698+0000", 2024-09-21T15:49:13.427 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.427 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.427 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.427 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:13.427 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.669697+0000", 2024-09-21T15:49:13.427 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:13.427 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.427 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.428 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:13.428 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.669707+0000", 2024-09-21T15:49:13.428 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0567999999999999e-05 2024-09-21T15:49:13.428 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.428 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.428 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:13.428 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.669725+0000", 2024-09-21T15:49:13.428 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7504e-05 2024-09-21T15:49:13.428 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.428 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.428 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:13.428 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.669734+0000", 2024-09-21T15:49:13.428 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.397e-06 2024-09-21T15:49:13.428 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.428 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.429 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:13.429 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.669790+0000", 2024-09-21T15:49:13.429 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5787e-05 2024-09-21T15:49:13.429 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.429 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.429 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:13.429 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.669817+0000", 2024-09-21T15:49:13.429 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6925000000000001e-05 2024-09-21T15:49:13.429 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.429 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.429 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:13.429 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.670413+0000", 2024-09-21T15:49:13.429 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00059617700000000004 2024-09-21T15:49:13.429 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.429 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.430 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:13.430 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.670447+0000", 2024-09-21T15:49:13.430 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4143999999999997e-05 2024-09-21T15:49:13.430 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.430 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:13.430 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.430 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.430 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.430 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:1044 3.0 e27/22 3:025e53e7:::rbd_data.1133f3dfbd6b.0000000000000008:head v 27'24, mlcod=27'22)", 2024-09-21T15:49:13.430 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.709866+0000", 2024-09-21T15:49:13.430 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.67415546299999995, 2024-09-21T15:49:13.430 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0021649210000000002, 2024-09-21T15:49:13.430 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:13.430 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:13.431 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:13.431 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.431 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:13.431 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.709866+0000", 2024-09-21T15:49:13.431 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.431 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.431 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.431 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:13.431 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.709866+0000", 2024-09-21T15:49:13.431 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.431 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.431 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.431 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:13.431 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.709865+0000", 2024-09-21T15:49:13.431 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:13.432 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.432 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.432 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:13.432 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.709892+0000", 2024-09-21T15:49:13.432 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6990000000000001e-05 2024-09-21T15:49:13.432 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.432 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.432 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:13.432 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.709911+0000", 2024-09-21T15:49:13.432 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8386999999999999e-05 2024-09-21T15:49:13.432 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.432 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.432 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:13.433 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.709920+0000", 2024-09-21T15:49:13.433 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0059999999999998e-06 2024-09-21T15:49:13.433 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.433 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.433 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:13.433 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.709964+0000", 2024-09-21T15:49:13.433 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4132999999999999e-05 2024-09-21T15:49:13.433 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.433 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.433 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:13.433 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.709992+0000", 2024-09-21T15:49:13.433 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8204e-05 2024-09-21T15:49:13.433 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.433 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.433 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:13.433 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.711995+0000", 2024-09-21T15:49:13.434 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002002925 2024-09-21T15:49:13.434 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.434 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.434 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:13.434 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.712031+0000", 2024-09-21T15:49:13.434 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6434999999999999e-05 2024-09-21T15:49:13.434 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.434 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:13.434 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.434 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.434 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:13.434 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:13.435 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:49:13.479 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:13.478+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:13.537 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:13.537 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:49:13.537 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:49:13.537 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:13.544 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:13.543+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:13.545 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:49:13.645 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:13.645+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:13.686 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:13.687 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:49:13.687 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:49:13.687 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:49:13.687 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:13.696 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:49:13.746 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:13.745+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:13.777 INFO:teuthology.orchestra.run.smithi089.stdout:truncating image image_client.0-clone1 from 0xe7849fc (overlap 0x95b193) to 0x0 2024-09-21T15:49:13.809 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:13.809 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:49:13.809 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:49:13.809 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:49:13.810 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.810 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:878 3.18 3:1b75505a:::rbd_header.1133f3dfbd6b:head [call rbd.set_size in=11b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-21T15:49:13.810 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.453159+0000", 2024-09-21T15:49:13.810 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.353901276, 2024-09-21T15:49:13.810 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010022399999999999, 2024-09-21T15:49:13.810 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:13.810 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:13.810 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:13.810 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:13.810 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:13.810 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 878 2024-09-21T15:49:13.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.811 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:13.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:13.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.453159+0000", 2024-09-21T15:49:13.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:13.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.453159+0000", 2024-09-21T15:49:13.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:13.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.453157+0000", 2024-09-21T15:49:13.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:49:13.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.812 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.812 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:13.812 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.453167+0000", 2024-09-21T15:49:13.812 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5810000000000003e-06 2024-09-21T15:49:13.812 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.812 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.812 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:13.812 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.453188+0000", 2024-09-21T15:49:13.812 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0786000000000001e-05 2024-09-21T15:49:13.812 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.812 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.812 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:13.812 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.453197+0000", 2024-09-21T15:49:13.812 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6690000000000001e-06 2024-09-21T15:49:13.812 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.812 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.813 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:13.813 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.453239+0000", 2024-09-21T15:49:13.813 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1284e-05 2024-09-21T15:49:13.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.813 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:13.813 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.453298+0000", 2024-09-21T15:49:13.813 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8928000000000002e-05 2024-09-21T15:49:13.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.813 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 0", 2024-09-21T15:49:13.813 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.453418+0000", 2024-09-21T15:49:13.813 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012075500000000001 2024-09-21T15:49:13.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:13.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.454012+0000", 2024-09-21T15:49:13.814 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00059328099999999997 2024-09-21T15:49:13.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.814 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:13.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.454109+0000", 2024-09-21T15:49:13.814 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7644000000000002e-05 2024-09-21T15:49:13.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.814 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:13.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.454139+0000", 2024-09-21T15:49:13.814 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9771999999999999e-05 2024-09-21T15:49:13.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.815 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.815 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:13.815 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.454162+0000", 2024-09-21T15:49:13.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2497000000000002e-05 2024-09-21T15:49:13.815 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.815 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:13.815 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.815 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.815 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:891 3.18 3:1b75505a:::rbd_header.1133f3dfbd6b:head [call rbd.set_size in=11b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-21T15:49:13.815 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.462710+0000", 2024-09-21T15:49:13.815 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.344351125, 2024-09-21T15:49:13.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001118637, 2024-09-21T15:49:13.816 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:13.816 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:13.816 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:13.816 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:13.816 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:13.816 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 891 2024-09-21T15:49:13.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.816 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:13.816 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.816 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:13.816 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.462710+0000", 2024-09-21T15:49:13.816 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.816 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.816 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:13.816 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.462710+0000", 2024-09-21T15:49:13.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.817 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:13.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.462708+0000", 2024-09-21T15:49:13.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:13.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.817 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:13.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.462716+0000", 2024-09-21T15:49:13.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6769999999999995e-06 2024-09-21T15:49:13.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.817 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:13.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.462730+0000", 2024-09-21T15:49:13.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4572999999999999e-05 2024-09-21T15:49:13.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.818 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.818 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:13.818 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.462737+0000", 2024-09-21T15:49:13.818 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3980000000000004e-06 2024-09-21T15:49:13.818 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.818 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.818 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:13.818 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.462783+0000", 2024-09-21T15:49:13.818 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6128999999999998e-05 2024-09-21T15:49:13.818 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.818 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.818 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:13.818 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.462869+0000", 2024-09-21T15:49:13.818 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6108999999999998e-05 2024-09-21T15:49:13.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.819 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 0", 2024-09-21T15:49:13.819 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.463069+0000", 2024-09-21T15:49:13.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00019950400000000001 2024-09-21T15:49:13.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.819 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:13.819 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.463531+0000", 2024-09-21T15:49:13.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00046281700000000002 2024-09-21T15:49:13.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.819 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:13.819 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.463749+0000", 2024-09-21T15:49:13.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00021736499999999999 2024-09-21T15:49:13.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.820 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:13.820 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.463788+0000", 2024-09-21T15:49:13.820 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9456000000000003e-05 2024-09-21T15:49:13.820 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.820 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.820 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:13.820 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.463828+0000", 2024-09-21T15:49:13.820 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9978000000000002e-05 2024-09-21T15:49:13.820 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.820 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:13.820 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.820 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.820 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.820 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:909 3.18 3:1b75505a:::rbd_header.1133f3dfbd6b:head [call rbd.set_size in=11b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-21T15:49:13.820 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.495062+0000", 2024-09-21T15:49:13.821 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.3119988140000001, 2024-09-21T15:49:13.821 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001358176, 2024-09-21T15:49:13.821 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:13.821 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:13.821 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:13.821 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:13.821 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:13.821 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 909 2024-09-21T15:49:13.821 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.821 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:13.821 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.821 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:13.821 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.495062+0000", 2024-09-21T15:49:13.821 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.821 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.821 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.822 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:13.822 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.495062+0000", 2024-09-21T15:49:13.822 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.822 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.822 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.822 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:13.822 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.495060+0000", 2024-09-21T15:49:13.822 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:49:13.822 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.822 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.822 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:13.822 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.495070+0000", 2024-09-21T15:49:13.822 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.397e-06 2024-09-21T15:49:13.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.823 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.823 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:13.823 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.495086+0000", 2024-09-21T15:49:13.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6308e-05 2024-09-21T15:49:13.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.823 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.823 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:13.823 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.495093+0000", 2024-09-21T15:49:13.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.002e-06 2024-09-21T15:49:13.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.823 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.823 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:13.823 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.495131+0000", 2024-09-21T15:49:13.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8418999999999999e-05 2024-09-21T15:49:13.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.824 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:13.824 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.495191+0000", 2024-09-21T15:49:13.824 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0152000000000003e-05 2024-09-21T15:49:13.824 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.824 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 0", 2024-09-21T15:49:13.824 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.495337+0000", 2024-09-21T15:49:13.824 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000145971 2024-09-21T15:49:13.824 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.824 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:13.824 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.496121+0000", 2024-09-21T15:49:13.824 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00078323200000000003 2024-09-21T15:49:13.824 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.825 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:13.825 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.496334+0000", 2024-09-21T15:49:13.825 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00021296499999999999 2024-09-21T15:49:13.825 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.825 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.825 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:13.825 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.496379+0000", 2024-09-21T15:49:13.825 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4994000000000003e-05 2024-09-21T15:49:13.825 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.825 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.825 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:13.825 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.496420+0000", 2024-09-21T15:49:13.825 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1495000000000002e-05 2024-09-21T15:49:13.825 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.825 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:13.825 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.826 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.826 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.826 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:923 3.18 3:1b75505a:::rbd_header.1133f3dfbd6b:head [call rbd.set_size in=11b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-21T15:49:13.826 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.516317+0000", 2024-09-21T15:49:13.826 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.2907435010000001, 2024-09-21T15:49:13.826 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001128293, 2024-09-21T15:49:13.826 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:13.826 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:13.826 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:13.826 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:13.826 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:13.826 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 923 2024-09-21T15:49:13.826 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.826 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:13.826 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.826 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:13.827 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.516317+0000", 2024-09-21T15:49:13.827 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.827 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.827 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.827 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:13.827 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.516317+0000", 2024-09-21T15:49:13.827 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.827 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.827 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.827 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:13.827 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.516315+0000", 2024-09-21T15:49:13.827 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:49:13.827 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.827 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.827 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:13.827 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.516324+0000", 2024-09-21T15:49:13.828 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8140000000000007e-06 2024-09-21T15:49:13.828 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.828 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.828 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:13.828 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.516340+0000", 2024-09-21T15:49:13.828 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5909999999999998e-05 2024-09-21T15:49:13.828 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.828 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.828 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:13.828 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.516349+0000", 2024-09-21T15:49:13.828 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1800000000000002e-06 2024-09-21T15:49:13.828 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.828 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.828 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:13.828 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.516400+0000", 2024-09-21T15:49:13.828 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0794000000000002e-05 2024-09-21T15:49:13.829 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.829 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.829 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:13.829 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.516489+0000", 2024-09-21T15:49:13.829 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8657000000000002e-05 2024-09-21T15:49:13.829 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.829 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.829 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 0", 2024-09-21T15:49:13.829 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.516680+0000", 2024-09-21T15:49:13.829 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000191676 2024-09-21T15:49:13.829 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.829 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.829 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:13.829 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.517149+0000", 2024-09-21T15:49:13.829 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00046852299999999999 2024-09-21T15:49:13.829 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.830 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.830 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:13.830 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.517390+0000", 2024-09-21T15:49:13.830 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000240743 2024-09-21T15:49:13.830 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.830 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.830 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:13.830 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.517422+0000", 2024-09-21T15:49:13.830 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1773999999999998e-05 2024-09-21T15:49:13.830 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.830 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.830 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:13.830 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.517445+0000", 2024-09-21T15:49:13.830 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3943000000000001e-05 2024-09-21T15:49:13.830 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.831 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:13.831 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.831 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.831 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:934 3.18 3:1b75505a:::rbd_header.1133f3dfbd6b:head [call rbd.set_size in=11b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-21T15:49:13.831 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.531402+0000", 2024-09-21T15:49:13.831 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.275658899, 2024-09-21T15:49:13.831 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002356478, 2024-09-21T15:49:13.831 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:13.831 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:13.831 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:13.831 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:13.831 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:13.831 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 934 2024-09-21T15:49:13.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.832 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:13.832 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.832 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:13.832 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.531402+0000", 2024-09-21T15:49:13.832 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.832 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.832 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:13.832 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.531402+0000", 2024-09-21T15:49:13.832 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.832 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.832 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:13.832 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.531400+0000", 2024-09-21T15:49:13.832 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-09-21T15:49:13.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.833 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.833 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:13.833 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.531410+0000", 2024-09-21T15:49:13.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0121e-05 2024-09-21T15:49:13.833 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.833 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.833 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:13.833 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.531429+0000", 2024-09-21T15:49:13.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9728000000000002e-05 2024-09-21T15:49:13.833 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.833 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.833 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:13.833 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.531439+0000", 2024-09-21T15:49:13.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4520000000000003e-06 2024-09-21T15:49:13.833 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.833 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.834 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:13.834 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.531481+0000", 2024-09-21T15:49:13.834 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1838999999999997e-05 2024-09-21T15:49:13.834 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.834 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.834 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:13.834 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.531537+0000", 2024-09-21T15:49:13.834 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5979000000000001e-05 2024-09-21T15:49:13.834 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.834 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.834 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 0", 2024-09-21T15:49:13.834 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.531669+0000", 2024-09-21T15:49:13.834 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000132285 2024-09-21T15:49:13.834 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.835 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.835 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:13.835 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.533483+0000", 2024-09-21T15:49:13.835 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0018143499999999999 2024-09-21T15:49:13.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.835 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.835 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:13.835 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.533701+0000", 2024-09-21T15:49:13.835 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00021753500000000001 2024-09-21T15:49:13.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.835 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.835 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:13.835 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.533730+0000", 2024-09-21T15:49:13.835 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9238e-05 2024-09-21T15:49:13.836 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.836 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.836 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:13.836 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.533758+0000", 2024-09-21T15:49:13.836 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8167999999999999e-05 2024-09-21T15:49:13.836 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.836 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:13.836 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.836 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.836 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.836 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:955 3.1e 3:7ceb6cf6:::rbd_data.1133f3dfbd6b.0000000000000001:head [stat,truncate 8388608] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-21T15:49:13.836 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.574693+0000", 2024-09-21T15:49:13.836 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.2323679359999999, 2024-09-21T15:49:13.836 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010551499999999999, 2024-09-21T15:49:13.836 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:13.836 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:13.837 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:13.837 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:13.837 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:13.837 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 955 2024-09-21T15:49:13.837 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.837 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:13.837 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.837 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:13.837 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.574693+0000", 2024-09-21T15:49:13.837 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.837 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.837 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.837 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:13.837 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.574693+0000", 2024-09-21T15:49:13.837 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.838 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.838 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.838 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:13.838 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.574691+0000", 2024-09-21T15:49:13.838 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:13.838 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.838 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.838 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:13.838 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.574699+0000", 2024-09-21T15:49:13.838 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5290000000000003e-06 2024-09-21T15:49:13.838 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.838 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.838 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:13.838 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.574715+0000", 2024-09-21T15:49:13.838 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5837000000000001e-05 2024-09-21T15:49:13.839 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.839 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.839 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:13.839 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.574723+0000", 2024-09-21T15:49:13.839 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.619e-06 2024-09-21T15:49:13.839 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.839 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.839 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:13.839 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.574768+0000", 2024-09-21T15:49:13.839 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4511999999999997e-05 2024-09-21T15:49:13.839 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.839 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.839 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:13.839 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.574824+0000", 2024-09-21T15:49:13.839 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6799999999999998e-05 2024-09-21T15:49:13.840 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.840 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.840 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-09-21T15:49:13.840 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.574910+0000", 2024-09-21T15:49:13.840 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5583999999999999e-05 2024-09-21T15:49:13.840 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.840 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.840 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:13.840 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.575410+0000", 2024-09-21T15:49:13.840 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00050031200000000004 2024-09-21T15:49:13.840 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.840 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.840 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:13.840 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.575699+0000", 2024-09-21T15:49:13.840 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000288829 2024-09-21T15:49:13.841 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.841 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.841 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:13.841 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.575726+0000", 2024-09-21T15:49:13.841 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6533000000000002e-05 2024-09-21T15:49:13.841 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.841 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.841 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:13.841 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.575748+0000", 2024-09-21T15:49:13.841 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2186000000000001e-05 2024-09-21T15:49:13.841 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.841 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:13.841 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.841 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.842 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.842 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:954 3.19 e27/22 3:9b8b44d5:::rbd_data.1133f3dfbd6b.0000000000000000:head v 27'10, mlcod=27'8)", 2024-09-21T15:49:13.842 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.575075+0000", 2024-09-21T15:49:13.842 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.2319854779999999, 2024-09-21T15:49:13.842 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0020513430000000002, 2024-09-21T15:49:13.842 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:13.842 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:13.842 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:13.842 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.842 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:13.842 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.575075+0000", 2024-09-21T15:49:13.842 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.842 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.842 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.843 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:13.843 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.575075+0000", 2024-09-21T15:49:13.843 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.843 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.843 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.843 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:13.843 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.575074+0000", 2024-09-21T15:49:13.843 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:13.843 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.843 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.843 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:13.843 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.575082+0000", 2024-09-21T15:49:13.843 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5039999999999996e-06 2024-09-21T15:49:13.843 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.843 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.843 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:13.844 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.575091+0000", 2024-09-21T15:49:13.844 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.037e-06 2024-09-21T15:49:13.844 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.844 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.844 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:13.844 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.575096+0000", 2024-09-21T15:49:13.844 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8189999999999999e-06 2024-09-21T15:49:13.844 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.844 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.844 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:13.844 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.575131+0000", 2024-09-21T15:49:13.844 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4820999999999997e-05 2024-09-21T15:49:13.844 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.844 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.844 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:13.844 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.575154+0000", 2024-09-21T15:49:13.844 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3289999999999999e-05 2024-09-21T15:49:13.845 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.845 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.845 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:13.845 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.577103+0000", 2024-09-21T15:49:13.845 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0019491389999999999 2024-09-21T15:49:13.845 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.845 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.845 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:13.845 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.577127+0000", 2024-09-21T15:49:13.845 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3594000000000001e-05 2024-09-21T15:49:13.845 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.845 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:13.845 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.845 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.845 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.845 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:959 3.18 3:1b75505a:::rbd_header.1133f3dfbd6b:head [call rbd.set_size in=11b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-21T15:49:13.846 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.577379+0000", 2024-09-21T15:49:13.846 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.2296812720000001, 2024-09-21T15:49:13.846 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00092499200000000002, 2024-09-21T15:49:13.846 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:13.846 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:13.846 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:13.846 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:13.846 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:13.846 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 959 2024-09-21T15:49:13.846 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.846 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:13.846 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.846 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:13.846 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.577379+0000", 2024-09-21T15:49:13.846 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.846 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.847 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.847 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:13.847 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.577379+0000", 2024-09-21T15:49:13.847 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.847 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.847 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:13.847 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.577379+0000", 2024-09-21T15:49:13.847 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:13.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.847 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.847 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:13.847 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.577383+0000", 2024-09-21T15:49:13.847 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7729999999999999e-06 2024-09-21T15:49:13.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.848 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.848 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:13.848 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.577393+0000", 2024-09-21T15:49:13.848 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3989999999999993e-06 2024-09-21T15:49:13.848 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.848 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.848 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:13.848 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.577398+0000", 2024-09-21T15:49:13.848 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7879999999999997e-06 2024-09-21T15:49:13.848 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.848 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.848 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:13.848 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.577432+0000", 2024-09-21T15:49:13.848 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4310999999999997e-05 2024-09-21T15:49:13.848 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.849 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.849 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:13.849 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.577486+0000", 2024-09-21T15:49:13.849 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4376999999999998e-05 2024-09-21T15:49:13.849 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.849 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.849 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 0", 2024-09-21T15:49:13.849 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.577614+0000", 2024-09-21T15:49:13.849 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012747499999999999 2024-09-21T15:49:13.849 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.849 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.849 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:13.849 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.578072+0000", 2024-09-21T15:49:13.849 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00045838299999999998 2024-09-21T15:49:13.849 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.850 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.850 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:13.850 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.578258+0000", 2024-09-21T15:49:13.850 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000185496 2024-09-21T15:49:13.850 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.850 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.850 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:13.850 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.578284+0000", 2024-09-21T15:49:13.850 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6488000000000001e-05 2024-09-21T15:49:13.850 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.850 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.850 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:13.850 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.578304+0000", 2024-09-21T15:49:13.850 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0299e-05 2024-09-21T15:49:13.851 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.851 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:13.851 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.851 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.851 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.851 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:969 3.18 3:1b75505a:::rbd_header.1133f3dfbd6b:head [call rbd.set_size in=11b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-21T15:49:13.851 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.598630+0000", 2024-09-21T15:49:13.851 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.2084309740000001, 2024-09-21T15:49:13.851 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00095591399999999998, 2024-09-21T15:49:13.851 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:13.851 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:13.851 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:13.851 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:13.851 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:13.851 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 969 2024-09-21T15:49:13.852 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.852 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:13.852 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.852 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:13.852 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.598630+0000", 2024-09-21T15:49:13.852 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.852 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.852 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.852 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:13.852 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.598630+0000", 2024-09-21T15:49:13.852 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.852 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.852 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.852 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:13.852 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.598628+0000", 2024-09-21T15:49:13.853 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:49:13.853 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.853 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:13.853 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.598637+0000", 2024-09-21T15:49:13.853 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.952e-06 2024-09-21T15:49:13.853 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.853 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:13.853 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.598653+0000", 2024-09-21T15:49:13.853 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6121e-05 2024-09-21T15:49:13.853 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.853 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:13.853 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.598667+0000", 2024-09-21T15:49:13.854 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3691e-05 2024-09-21T15:49:13.854 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.854 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.854 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:13.854 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.598709+0000", 2024-09-21T15:49:13.854 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2431000000000001e-05 2024-09-21T15:49:13.854 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.854 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.854 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:13.854 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.598766+0000", 2024-09-21T15:49:13.854 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6721999999999997e-05 2024-09-21T15:49:13.854 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.854 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.854 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 0", 2024-09-21T15:49:13.855 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.598892+0000", 2024-09-21T15:49:13.855 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000126062 2024-09-21T15:49:13.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.855 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:13.855 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.599375+0000", 2024-09-21T15:49:13.855 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00048278600000000002 2024-09-21T15:49:13.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.855 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:13.855 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.599536+0000", 2024-09-21T15:49:13.855 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00016101599999999999 2024-09-21T15:49:13.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.856 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:13.856 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.599565+0000", 2024-09-21T15:49:13.856 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9149000000000002e-05 2024-09-21T15:49:13.856 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.856 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.856 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:13.856 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.599586+0000", 2024-09-21T15:49:13.856 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0715000000000001e-05 2024-09-21T15:49:13.856 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.856 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:13.856 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.856 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.856 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.856 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:981 3.19 e27/22 3:9b8b44d5:::rbd_data.1133f3dfbd6b.0000000000000000:head v 27'11, mlcod=27'9)", 2024-09-21T15:49:13.856 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.609627+0000", 2024-09-21T15:49:13.857 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.1974336349999999, 2024-09-21T15:49:13.857 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0034715589999999999, 2024-09-21T15:49:13.857 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:13.857 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:13.857 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:13.857 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.857 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:13.857 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.609627+0000", 2024-09-21T15:49:13.857 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.857 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.857 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.857 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:13.857 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.609627+0000", 2024-09-21T15:49:13.857 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.857 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.857 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.857 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:13.858 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.609626+0000", 2024-09-21T15:49:13.858 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:13.858 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.858 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.858 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:13.858 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.609647+0000", 2024-09-21T15:49:13.858 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0905999999999999e-05 2024-09-21T15:49:13.858 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.858 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.858 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:13.858 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.609658+0000", 2024-09-21T15:49:13.858 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0669999999999999e-05 2024-09-21T15:49:13.858 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.858 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.858 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:13.858 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.609663+0000", 2024-09-21T15:49:13.859 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9799999999999998e-06 2024-09-21T15:49:13.859 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.859 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.859 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:13.859 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.609695+0000", 2024-09-21T15:49:13.859 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1838999999999998e-05 2024-09-21T15:49:13.859 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.859 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.859 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:13.859 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.609715+0000", 2024-09-21T15:49:13.859 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0437000000000001e-05 2024-09-21T15:49:13.859 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.859 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.859 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:13.859 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.613073+0000", 2024-09-21T15:49:13.859 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003358069 2024-09-21T15:49:13.860 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.860 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.860 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:13.860 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.613099+0000", 2024-09-21T15:49:13.860 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5545999999999999e-05 2024-09-21T15:49:13.860 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.860 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:13.860 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.860 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.860 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.860 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:989 3.18 3:1b75505a:::rbd_header.1133f3dfbd6b:head [call rbd.set_size in=11b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-21T15:49:13.860 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.631979+0000", 2024-09-21T15:49:13.860 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.1750820959999999, 2024-09-21T15:49:13.860 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00093807899999999997, 2024-09-21T15:49:13.860 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:13.861 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:13.861 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:13.861 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:13.861 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:13.861 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 989 2024-09-21T15:49:13.861 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.861 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:13.861 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.861 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:13.861 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.631979+0000", 2024-09-21T15:49:13.861 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.861 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.861 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.861 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:13.861 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.631979+0000", 2024-09-21T15:49:13.861 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.862 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.862 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.862 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:13.862 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.631977+0000", 2024-09-21T15:49:13.862 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:13.862 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.862 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.862 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:13.862 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.631983+0000", 2024-09-21T15:49:13.862 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7010000000000001e-06 2024-09-21T15:49:13.862 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.862 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.862 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:13.862 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.631998+0000", 2024-09-21T15:49:13.862 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4674999999999999e-05 2024-09-21T15:49:13.862 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.863 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.863 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:13.863 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.632004+0000", 2024-09-21T15:49:13.863 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7510000000000002e-06 2024-09-21T15:49:13.863 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.863 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.863 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:13.863 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.632041+0000", 2024-09-21T15:49:13.863 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6808999999999999e-05 2024-09-21T15:49:13.863 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.863 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.863 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:13.863 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.632100+0000", 2024-09-21T15:49:13.863 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8687000000000002e-05 2024-09-21T15:49:13.863 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.863 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.863 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 0", 2024-09-21T15:49:13.864 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.632236+0000", 2024-09-21T15:49:13.864 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013635200000000001 2024-09-21T15:49:13.864 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.864 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.864 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:13.864 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.632659+0000", 2024-09-21T15:49:13.864 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00042297899999999998 2024-09-21T15:49:13.864 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.864 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.864 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:13.864 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.632868+0000", 2024-09-21T15:49:13.864 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000209163 2024-09-21T15:49:13.864 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.864 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.864 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:13.865 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.632896+0000", 2024-09-21T15:49:13.865 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7512e-05 2024-09-21T15:49:13.865 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.865 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.865 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:13.865 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.632917+0000", 2024-09-21T15:49:13.865 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0696e-05 2024-09-21T15:49:13.865 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.865 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:13.865 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.865 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.865 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.865 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:1002 3.1e 3:7ceb6cf6:::rbd_data.1133f3dfbd6b.0000000000000001:head [stat,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-21T15:49:13.865 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.642814+0000", 2024-09-21T15:49:13.865 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.164247179, 2024-09-21T15:49:13.865 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.015295105999999999, 2024-09-21T15:49:13.865 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:13.866 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:13.866 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:13.866 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:13.866 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:13.866 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1002 2024-09-21T15:49:13.866 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.866 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:13.866 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.866 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:13.866 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.642814+0000", 2024-09-21T15:49:13.866 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.866 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.866 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.866 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:13.866 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.642814+0000", 2024-09-21T15:49:13.867 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.867 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.867 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.867 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:13.867 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.642812+0000", 2024-09-21T15:49:13.867 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:13.867 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.867 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.867 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:13.867 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.642821+0000", 2024-09-21T15:49:13.867 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3100000000000006e-06 2024-09-21T15:49:13.867 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.867 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.867 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:13.867 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.642837+0000", 2024-09-21T15:49:13.867 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5831000000000001e-05 2024-09-21T15:49:13.868 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.868 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.868 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:13.868 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.642843+0000", 2024-09-21T15:49:13.868 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1619999999999996e-06 2024-09-21T15:49:13.868 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.868 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.868 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:13.868 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.642894+0000", 2024-09-21T15:49:13.868 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1054999999999998e-05 2024-09-21T15:49:13.868 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.868 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.868 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:13.868 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.642956+0000", 2024-09-21T15:49:13.868 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1618000000000005e-05 2024-09-21T15:49:13.868 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.868 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.869 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-09-21T15:49:13.869 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.645383+0000", 2024-09-21T15:49:13.869 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0024269589999999998 2024-09-21T15:49:13.869 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.869 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.869 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:13.869 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.658018+0000", 2024-09-21T15:49:13.869 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.012634938 2024-09-21T15:49:13.869 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.869 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.869 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:13.869 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.658029+0000", 2024-09-21T15:49:13.869 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1129e-05 2024-09-21T15:49:13.869 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.869 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.869 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:13.870 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.658064+0000", 2024-09-21T15:49:13.870 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5250000000000003e-05 2024-09-21T15:49:13.870 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.870 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.870 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:13.870 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.658109+0000", 2024-09-21T15:49:13.870 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4471999999999997e-05 2024-09-21T15:49:13.870 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.870 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:13.870 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.870 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.870 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.870 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop_reply(client.4403.0:1002 3.1e e27/22 ondisk, result = 0)", 2024-09-21T15:49:13.870 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.652730+0000", 2024-09-21T15:49:13.870 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.1543304729999999, 2024-09-21T15:49:13.870 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052901320000000003, 2024-09-21T15:49:13.871 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:13.871 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:13.871 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:13.871 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.871 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:13.871 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.652730+0000", 2024-09-21T15:49:13.871 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.871 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.871 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.871 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:13.871 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.652730+0000", 2024-09-21T15:49:13.871 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.871 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.871 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.872 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:13.872 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.652729+0000", 2024-09-21T15:49:13.872 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:13.872 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.872 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.872 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:13.872 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.652734+0000", 2024-09-21T15:49:13.872 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7539999999999997e-06 2024-09-21T15:49:13.872 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.872 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.872 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:13.872 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.652747+0000", 2024-09-21T15:49:13.872 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3256e-05 2024-09-21T15:49:13.872 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.872 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.873 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:13.873 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.652756+0000", 2024-09-21T15:49:13.873 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3299999999999999e-06 2024-09-21T15:49:13.873 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.873 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.873 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:13.873 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.658005+0000", 2024-09-21T15:49:13.873 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052498730000000004 2024-09-21T15:49:13.873 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.873 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.873 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:13.873 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.658013+0000", 2024-09-21T15:49:13.873 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2080000000000003e-06 2024-09-21T15:49:13.873 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.873 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.874 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:13.874 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.658020+0000", 2024-09-21T15:49:13.874 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7419999999999997e-06 2024-09-21T15:49:13.874 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.874 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:13.874 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.874 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.874 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.874 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:1008 3.1e 3:7ceb6cf6:::rbd_data.1133f3dfbd6b.0000000000000001:head [stat,truncate 13936959] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-21T15:49:13.874 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.672118+0000", 2024-09-21T15:49:13.874 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.1349422410000001, 2024-09-21T15:49:13.874 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010130969999999999, 2024-09-21T15:49:13.874 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:13.874 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:13.874 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:13.875 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:13.875 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:13.875 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1008 2024-09-21T15:49:13.875 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.875 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:13.875 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.875 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:13.875 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.672118+0000", 2024-09-21T15:49:13.875 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.875 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.875 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.875 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:13.875 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.672118+0000", 2024-09-21T15:49:13.875 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.875 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.875 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.876 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:13.876 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.672117+0000", 2024-09-21T15:49:13.876 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:49:13.876 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.876 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.876 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:13.876 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.672125+0000", 2024-09-21T15:49:13.876 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5210000000000002e-06 2024-09-21T15:49:13.876 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.876 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.876 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:13.876 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.672142+0000", 2024-09-21T15:49:13.876 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6633000000000002e-05 2024-09-21T15:49:13.876 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.876 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.876 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:13.877 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.672149+0000", 2024-09-21T15:49:13.877 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.584e-06 2024-09-21T15:49:13.877 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.877 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.877 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:13.877 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.672190+0000", 2024-09-21T15:49:13.877 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1304999999999998e-05 2024-09-21T15:49:13.877 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.877 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.877 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:13.877 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.672249+0000", 2024-09-21T15:49:13.877 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9172000000000002e-05 2024-09-21T15:49:13.877 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.877 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.877 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-09-21T15:49:13.877 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.672335+0000", 2024-09-21T15:49:13.878 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6447999999999996e-05 2024-09-21T15:49:13.878 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.878 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.878 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:13.878 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.672848+0000", 2024-09-21T15:49:13.878 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00051300200000000003 2024-09-21T15:49:13.878 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.878 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.878 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:13.878 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.673067+0000", 2024-09-21T15:49:13.878 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000218929 2024-09-21T15:49:13.878 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.878 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.878 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:13.879 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.673098+0000", 2024-09-21T15:49:13.879 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0185e-05 2024-09-21T15:49:13.879 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.879 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.879 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:13.879 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.673132+0000", 2024-09-21T15:49:13.879 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3997e-05 2024-09-21T15:49:13.879 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.879 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:13.879 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.879 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.879 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.879 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:1012 3.18 3:1b75505a:::rbd_header.1133f3dfbd6b:head [call rbd.set_size in=11b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-21T15:49:13.879 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.675119+0000", 2024-09-21T15:49:13.879 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.131941775, 2024-09-21T15:49:13.879 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010005979999999999, 2024-09-21T15:49:13.880 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:13.880 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:13.880 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:13.880 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:13.880 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:13.880 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1012 2024-09-21T15:49:13.880 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.880 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:13.880 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.880 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:13.880 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.675119+0000", 2024-09-21T15:49:13.880 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.880 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.880 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.881 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:13.881 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.675119+0000", 2024-09-21T15:49:13.881 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.881 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.881 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.881 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:13.881 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.675118+0000", 2024-09-21T15:49:13.881 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:13.881 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.881 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.881 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:13.881 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.675124+0000", 2024-09-21T15:49:13.881 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9970000000000004e-06 2024-09-21T15:49:13.881 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.881 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.882 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:13.882 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.675138+0000", 2024-09-21T15:49:13.882 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4097000000000001e-05 2024-09-21T15:49:13.882 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.882 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.882 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:13.882 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.675146+0000", 2024-09-21T15:49:13.882 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6029999999999999e-06 2024-09-21T15:49:13.882 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.882 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.882 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:13.882 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.675190+0000", 2024-09-21T15:49:13.882 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4233999999999997e-05 2024-09-21T15:49:13.882 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.882 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.883 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:13.883 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.675247+0000", 2024-09-21T15:49:13.883 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7102999999999999e-05 2024-09-21T15:49:13.883 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.883 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.883 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 0", 2024-09-21T15:49:13.883 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.675381+0000", 2024-09-21T15:49:13.883 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013374099999999999 2024-09-21T15:49:13.883 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.883 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.883 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:13.883 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.675841+0000", 2024-09-21T15:49:13.883 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00046079500000000001 2024-09-21T15:49:13.883 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.883 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.884 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:13.884 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.676067+0000", 2024-09-21T15:49:13.884 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00022593 2024-09-21T15:49:13.884 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.884 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.884 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:13.884 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.676096+0000", 2024-09-21T15:49:13.884 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8541999999999999e-05 2024-09-21T15:49:13.884 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.884 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.884 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:13.884 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.676120+0000", 2024-09-21T15:49:13.884 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3609999999999999e-05 2024-09-21T15:49:13.884 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.884 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:13.885 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.885 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.885 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.885 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:1022 3.18 3:1b75505a:::rbd_header.1133f3dfbd6b:head [call rbd.set_size in=11b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-21T15:49:13.885 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.683758+0000", 2024-09-21T15:49:13.885 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.1233023609999999, 2024-09-21T15:49:13.885 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00095730499999999998, 2024-09-21T15:49:13.885 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:13.885 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:13.885 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:13.885 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:13.885 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:13.885 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1022 2024-09-21T15:49:13.885 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.886 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:13.886 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.886 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:13.886 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.683758+0000", 2024-09-21T15:49:13.886 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.886 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.886 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.886 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:13.886 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.683758+0000", 2024-09-21T15:49:13.886 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.886 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.886 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.886 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:13.886 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.683757+0000", 2024-09-21T15:49:13.886 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:13.887 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.887 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.887 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:13.887 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.683764+0000", 2024-09-21T15:49:13.887 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6789999999999999e-06 2024-09-21T15:49:13.887 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.887 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.887 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:13.887 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.683779+0000", 2024-09-21T15:49:13.887 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5639000000000001e-05 2024-09-21T15:49:13.887 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.887 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.887 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:13.887 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.683787+0000", 2024-09-21T15:49:13.887 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.328e-06 2024-09-21T15:49:13.887 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.888 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.888 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:13.888 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.683831+0000", 2024-09-21T15:49:13.888 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3912999999999998e-05 2024-09-21T15:49:13.888 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.888 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.888 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:13.888 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.683891+0000", 2024-09-21T15:49:13.888 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0736000000000003e-05 2024-09-21T15:49:13.888 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.888 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.888 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 0", 2024-09-21T15:49:13.888 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.684045+0000", 2024-09-21T15:49:13.888 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015364999999999999 2024-09-21T15:49:13.888 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.888 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.889 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:13.889 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.684527+0000", 2024-09-21T15:49:13.889 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00048185799999999999 2024-09-21T15:49:13.889 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.889 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.889 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:13.889 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.684666+0000", 2024-09-21T15:49:13.889 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000138928 2024-09-21T15:49:13.889 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.889 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.889 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:13.889 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.684695+0000", 2024-09-21T15:49:13.889 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9334000000000001e-05 2024-09-21T15:49:13.889 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.889 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.889 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:13.889 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.684716+0000", 2024-09-21T15:49:13.890 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0465000000000001e-05 2024-09-21T15:49:13.890 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.890 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:13.890 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.890 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.890 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.890 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:1032 3.18 3:1b75505a:::rbd_header.1133f3dfbd6b:head [call rbd.set_size in=11b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-21T15:49:13.890 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.696321+0000", 2024-09-21T15:49:13.890 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.110739616, 2024-09-21T15:49:13.890 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0024352829999999999, 2024-09-21T15:49:13.890 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:13.890 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:13.890 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:13.890 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:13.891 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:13.891 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1032 2024-09-21T15:49:13.891 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.891 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:13.891 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.891 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:13.891 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.696321+0000", 2024-09-21T15:49:13.891 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.891 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.891 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.891 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:13.891 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.696321+0000", 2024-09-21T15:49:13.891 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.891 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.891 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.892 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:13.892 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.696319+0000", 2024-09-21T15:49:13.892 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:49:13.892 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.892 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.892 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:13.892 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.696328+0000", 2024-09-21T15:49:13.892 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.935e-06 2024-09-21T15:49:13.892 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.892 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.892 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:13.892 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.696344+0000", 2024-09-21T15:49:13.892 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5948e-05 2024-09-21T15:49:13.892 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.892 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.892 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:13.893 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.696351+0000", 2024-09-21T15:49:13.893 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1180000000000003e-06 2024-09-21T15:49:13.893 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.893 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.893 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:13.893 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.696400+0000", 2024-09-21T15:49:13.893 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9162000000000001e-05 2024-09-21T15:49:13.893 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.893 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.893 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:13.893 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.696490+0000", 2024-09-21T15:49:13.893 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9726000000000001e-05 2024-09-21T15:49:13.893 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.893 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.894 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 0", 2024-09-21T15:49:13.894 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.696684+0000", 2024-09-21T15:49:13.894 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00019393700000000001 2024-09-21T15:49:13.894 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.894 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.894 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:13.894 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.698470+0000", 2024-09-21T15:49:13.894 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001785643 2024-09-21T15:49:13.894 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.894 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.894 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:13.894 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.698667+0000", 2024-09-21T15:49:13.894 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00019782299999999999 2024-09-21T15:49:13.895 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.895 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.895 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:13.895 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.698718+0000", 2024-09-21T15:49:13.895 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0698999999999997e-05 2024-09-21T15:49:13.895 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.895 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.895 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:13.895 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.698756+0000", 2024-09-21T15:49:13.895 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8157000000000001e-05 2024-09-21T15:49:13.895 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.895 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:13.895 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.896 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.896 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.896 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:1046 3.18 3:1b75505a:::rbd_header.1133f3dfbd6b:head [call rbd.set_size in=11b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e27)", 2024-09-21T15:49:13.896 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.720180+0000", 2024-09-21T15:49:13.896 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.0868805699999999, 2024-09-21T15:49:13.896 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0034699409999999998, 2024-09-21T15:49:13.896 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:13.896 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:13.896 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:13.896 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:13.896 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:13.896 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1046 2024-09-21T15:49:13.896 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.896 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:13.897 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.897 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:13.897 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.720180+0000", 2024-09-21T15:49:13.897 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.897 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.897 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.897 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:13.897 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.720180+0000", 2024-09-21T15:49:13.897 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.897 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.897 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.897 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:13.897 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.720178+0000", 2024-09-21T15:49:13.897 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:49:13.897 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.898 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.898 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:13.898 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.720189+0000", 2024-09-21T15:49:13.898 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0592e-05 2024-09-21T15:49:13.898 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.898 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.898 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:13.898 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.720210+0000", 2024-09-21T15:49:13.898 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1412000000000001e-05 2024-09-21T15:49:13.898 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.898 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.898 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:13.898 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.720220+0000", 2024-09-21T15:49:13.899 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6879999999999994e-06 2024-09-21T15:49:13.899 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.899 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.899 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:13.899 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.720277+0000", 2024-09-21T15:49:13.899 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.736e-05 2024-09-21T15:49:13.899 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.899 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.899 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:13.899 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.720372+0000", 2024-09-21T15:49:13.899 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5340000000000005e-05 2024-09-21T15:49:13.899 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.899 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.900 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 0", 2024-09-21T15:49:13.900 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.720536+0000", 2024-09-21T15:49:13.900 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00016351199999999999 2024-09-21T15:49:13.900 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.900 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.900 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:13.900 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.723052+0000", 2024-09-21T15:49:13.900 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002515672 2024-09-21T15:49:13.900 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.900 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.900 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:13.900 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.723581+0000", 2024-09-21T15:49:13.900 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00052973699999999996 2024-09-21T15:49:13.900 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.901 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.901 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:13.901 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.723622+0000", 2024-09-21T15:49:13.901 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0621999999999998e-05 2024-09-21T15:49:13.901 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.901 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.901 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:13.901 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.723650+0000", 2024-09-21T15:49:13.901 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8096999999999999e-05 2024-09-21T15:49:13.901 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.901 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:13.901 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.901 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.901 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.902 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:1058 3.7 3:e2f50d9b:::rbd_data.1133f3dfbd6b.0000000000000005:head [set-alloc-hint object_size 16777216 write_size 16777216,write 16817~23229] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e28)", 2024-09-21T15:49:13.902 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.735890+0000", 2024-09-21T15:49:13.902 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.071170556, 2024-09-21T15:49:13.902 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012606519999999999, 2024-09-21T15:49:13.902 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:13.902 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:13.902 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:13.902 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:13.902 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:13.902 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 1058 2024-09-21T15:49:13.902 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.902 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:13.902 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.902 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:13.903 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.735890+0000", 2024-09-21T15:49:13.903 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.903 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.903 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.903 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:13.903 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.735890+0000", 2024-09-21T15:49:13.903 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.903 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.903 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.903 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:13.903 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.735889+0000", 2024-09-21T15:49:13.904 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:13.904 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.904 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.904 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:13.904 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.735910+0000", 2024-09-21T15:49:13.904 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.128e-05 2024-09-21T15:49:13.904 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.904 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.904 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:13.904 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.735925+0000", 2024-09-21T15:49:13.904 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4979e-05 2024-09-21T15:49:13.904 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.904 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.905 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:13.905 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.735933+0000", 2024-09-21T15:49:13.905 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.405e-06 2024-09-21T15:49:13.905 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.905 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.905 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:13.905 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.735978+0000", 2024-09-21T15:49:13.905 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4369000000000002e-05 2024-09-21T15:49:13.905 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.905 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.905 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:13.905 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.736043+0000", 2024-09-21T15:49:13.905 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5265000000000006e-05 2024-09-21T15:49:13.905 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.906 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.906 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 7", 2024-09-21T15:49:13.906 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.736134+0000", 2024-09-21T15:49:13.906 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0731999999999998e-05 2024-09-21T15:49:13.906 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.906 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.906 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:13.906 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.736934+0000", 2024-09-21T15:49:13.906 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000800473 2024-09-21T15:49:13.906 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.906 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.906 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:13.906 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.737097+0000", 2024-09-21T15:49:13.907 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00016278500000000001 2024-09-21T15:49:13.907 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.907 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.907 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:13.907 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.737127+0000", 2024-09-21T15:49:13.907 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0537000000000003e-05 2024-09-21T15:49:13.907 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.907 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.907 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:13.907 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.737151+0000", 2024-09-21T15:49:13.907 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3323e-05 2024-09-21T15:49:13.907 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.907 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:13.907 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.907 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.907 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.908 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:1060 3.19 e28/22 3:9b8b44d5:::rbd_data.1133f3dfbd6b.0000000000000000:head v 28'12, mlcod=27'10)", 2024-09-21T15:49:13.908 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.738105+0000", 2024-09-21T15:49:13.908 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.0689561460000001, 2024-09-21T15:49:13.908 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012980769999999999, 2024-09-21T15:49:13.908 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:13.908 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:13.908 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:13.908 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.908 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:13.908 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.738105+0000", 2024-09-21T15:49:13.908 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.908 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.908 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.908 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:13.908 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.738105+0000", 2024-09-21T15:49:13.908 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:13.909 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.909 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.909 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:13.909 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.738104+0000", 2024-09-21T15:49:13.909 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:49:13.909 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.909 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.909 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:13.909 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.738120+0000", 2024-09-21T15:49:13.909 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5933000000000001e-05 2024-09-21T15:49:13.909 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.909 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.909 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:13.909 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.738128+0000", 2024-09-21T15:49:13.909 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5120000000000002e-06 2024-09-21T15:49:13.909 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.909 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.910 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:13.910 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.738134+0000", 2024-09-21T15:49:13.910 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4570000000000002e-06 2024-09-21T15:49:13.910 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.910 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.910 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:13.910 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.738180+0000", 2024-09-21T15:49:13.910 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5803999999999997e-05 2024-09-21T15:49:13.910 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.910 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.910 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:13.910 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.738201+0000", 2024-09-21T15:49:13.910 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1305e-05 2024-09-21T15:49:13.910 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.910 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.911 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:13.911 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.739373+0000", 2024-09-21T15:49:13.911 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0011712269999999999 2024-09-21T15:49:13.911 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:13.911 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:13.911 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:13.911 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.739403+0000", 2024-09-21T15:49:13.911 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0077e-05 2024-09-21T15:49:13.911 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.911 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:13.911 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.911 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:13.911 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:13.911 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:13.913 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:49:13.913 INFO:teuthology.orchestra.run.smithi089.stdout:checking clone #0, image image_client.0 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent1 2024-09-21T15:49:13.913 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:13.911+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:14.013 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:14.012+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:14.017 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:14.017 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:14.017 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:49:14.017 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:14.027 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:49:14.056 INFO:teuthology.orchestra.run.smithi089.stdout:402 read 0xab092ac thru 0xab143b7 (0xb10c bytes) 2024-09-21T15:49:14.057 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:14.057 INFO:teuthology.orchestra.run.smithi089.stdout:406 write 0x5445571 thru 0x5446a84 (0x1514 bytes) 2024-09-21T15:49:14.086 INFO:teuthology.orchestra.run.smithi089.stdout:407 punch from 0xd68d133 to 0xd693265, (0x6132 bytes) 2024-09-21T15:49:14.088 INFO:teuthology.orchestra.run.smithi089.stdout:408 trunc from 0xe7849fc to 0xc6e1bdc 2024-09-21T15:49:14.114 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:14.113+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:14.189 INFO:teuthology.orchestra.run.smithi089.stdout:409 trunc from 0xc6e1bdc to 0x5d1dd7b 2024-09-21T15:49:14.197 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:14.197 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:14.197 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:49:14.197 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:49:14.197 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:14.205 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:49:14.247 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:14.246+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:14.317 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:14.316+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:14.353 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:14.353 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:49:14.353 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:49:14.353 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:49:14.353 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.353 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:9 3.1a 3:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e25)", 2024-09-21T15:49:14.353 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:08.690799+0000", 2024-09-21T15:49:14.353 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.6602059410000001, 2024-09-21T15:49:14.354 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0066901119999999998, 2024-09-21T15:49:14.354 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:14.354 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:14.354 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:14.354 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:14.354 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:14.354 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 9 2024-09-21T15:49:14.354 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.354 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:14.354 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.354 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:14.354 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:08.690799+0000", 2024-09-21T15:49:14.354 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:14.354 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.355 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.355 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:14.355 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:08.690799+0000", 2024-09-21T15:49:14.355 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:14.355 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.355 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.355 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:14.355 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:08.690798+0000", 2024-09-21T15:49:14.355 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:14.355 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.355 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.355 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:14.355 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:08.690806+0000", 2024-09-21T15:49:14.355 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5020000000000003e-06 2024-09-21T15:49:14.355 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.356 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.356 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:14.356 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:08.690816+0000", 2024-09-21T15:49:14.356 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0888e-05 2024-09-21T15:49:14.356 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.356 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.356 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:14.356 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:08.690824+0000", 2024-09-21T15:49:14.356 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7020000000000001e-06 2024-09-21T15:49:14.356 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.356 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.356 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:14.356 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:08.690873+0000", 2024-09-21T15:49:14.356 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8698e-05 2024-09-21T15:49:14.357 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.357 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.357 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-21T15:49:14.357 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:08.690878+0000", 2024-09-21T15:49:14.357 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4399999999999996e-06 2024-09-21T15:49:14.357 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.357 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.357 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:14.357 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:08.695589+0000", 2024-09-21T15:49:14.357 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0047107260000000001 2024-09-21T15:49:14.357 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.357 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.357 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:14.358 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:08.695680+0000", 2024-09-21T15:49:14.358 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1299999999999997e-05 2024-09-21T15:49:14.358 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.358 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.358 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 1", 2024-09-21T15:49:14.358 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:08.695797+0000", 2024-09-21T15:49:14.358 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00011678100000000001 2024-09-21T15:49:14.358 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.358 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.358 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:14.358 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:08.696460+0000", 2024-09-21T15:49:14.358 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00066301100000000003 2024-09-21T15:49:14.358 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.359 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.359 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:14.359 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:08.697438+0000", 2024-09-21T15:49:14.359 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00097792100000000009 2024-09-21T15:49:14.359 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.359 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.359 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:14.359 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:08.697468+0000", 2024-09-21T15:49:14.359 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9641999999999998e-05 2024-09-21T15:49:14.359 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.359 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.359 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:14.359 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:08.697490+0000", 2024-09-21T15:49:14.359 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1857000000000001e-05 2024-09-21T15:49:14.359 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:14.360 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:14.360 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:14.360 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.360 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.360 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4403.0:1055 3.1d e28/22 3:bd483619:::rbd_data.1133f3dfbd6b.0000000000000007:head v 28'28, mlcod=27'26)", 2024-09-21T15:49:14.360 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:12.728887+0000", 2024-09-21T15:49:14.360 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 1.6221179590000001, 2024-09-21T15:49:14.360 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0025816039999999999, 2024-09-21T15:49:14.360 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:14.360 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:14.360 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:14.360 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.360 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:14.361 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:12.728887+0000", 2024-09-21T15:49:14.361 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:14.361 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.361 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.361 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:14.361 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:12.728887+0000", 2024-09-21T15:49:14.361 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:14.361 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.361 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.361 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:14.361 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:12.728887+0000", 2024-09-21T15:49:14.361 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:14.361 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.361 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.361 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:14.362 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:12.728924+0000", 2024-09-21T15:49:14.362 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7042000000000002e-05 2024-09-21T15:49:14.362 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.362 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.362 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:14.362 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:12.728933+0000", 2024-09-21T15:49:14.362 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.9050000000000003e-06 2024-09-21T15:49:14.362 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.362 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.362 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:14.362 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:12.728940+0000", 2024-09-21T15:49:14.362 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0720000000000002e-06 2024-09-21T15:49:14.362 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.362 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.362 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:14.363 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:12.728986+0000", 2024-09-21T15:49:14.363 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5957000000000001e-05 2024-09-21T15:49:14.363 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.363 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.363 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:14.363 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:12.729008+0000", 2024-09-21T15:49:14.363 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1580999999999999e-05 2024-09-21T15:49:14.363 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.363 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.363 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:14.363 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:12.731447+0000", 2024-09-21T15:49:14.363 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0024392889999999999 2024-09-21T15:49:14.363 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.363 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.364 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:14.364 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:12.731469+0000", 2024-09-21T15:49:14.364 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2048e-05 2024-09-21T15:49:14.364 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:14.364 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:14.364 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:14.364 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.364 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.364 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4403.0:1056 3.1d e28/22 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:head v 28'29, mlcod=27'27)", 2024-09-21T15:49:14.364 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:12.732040+0000", 2024-09-21T15:49:14.364 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 1.618965317, 2024-09-21T15:49:14.364 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002226904, 2024-09-21T15:49:14.364 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:14.364 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:14.364 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:14.365 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.365 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:14.365 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:12.732040+0000", 2024-09-21T15:49:14.365 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:14.365 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.365 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.365 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:14.365 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:12.732040+0000", 2024-09-21T15:49:14.365 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:14.365 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.365 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.365 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:14.365 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:12.732039+0000", 2024-09-21T15:49:14.365 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:14.365 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.366 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.366 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:14.366 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:12.732068+0000", 2024-09-21T15:49:14.366 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8775000000000001e-05 2024-09-21T15:49:14.366 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.366 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.366 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:14.366 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:12.732077+0000", 2024-09-21T15:49:14.366 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1500000000000005e-06 2024-09-21T15:49:14.366 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.366 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.366 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:14.366 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:12.732081+0000", 2024-09-21T15:49:14.366 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0910000000000003e-06 2024-09-21T15:49:14.366 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.367 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.367 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:14.367 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:12.732110+0000", 2024-09-21T15:49:14.367 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9057e-05 2024-09-21T15:49:14.367 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.367 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.367 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:14.367 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:12.732126+0000", 2024-09-21T15:49:14.367 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5192e-05 2024-09-21T15:49:14.367 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.367 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.367 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:14.367 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:12.734242+0000", 2024-09-21T15:49:14.367 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0021166959999999999 2024-09-21T15:49:14.367 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.367 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.368 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:14.368 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:12.734267+0000", 2024-09-21T15:49:14.368 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.457e-05 2024-09-21T15:49:14.368 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:14.368 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:14.368 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:14.368 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.368 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.368 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4403.0:1117 3.e e29/22 3:74dd1916:::rbd_header.11335fa2fe69:head v 29'24, mlcod=27'22)", 2024-09-21T15:49:14.368 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:13.749934+0000", 2024-09-21T15:49:14.368 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.60107167399999994, 2024-09-21T15:49:14.368 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0022678910000000002, 2024-09-21T15:49:14.368 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:14.368 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:14.368 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:14.368 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.369 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:14.369 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.749934+0000", 2024-09-21T15:49:14.369 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:14.369 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.369 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.369 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:14.369 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.749934+0000", 2024-09-21T15:49:14.369 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:14.369 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.369 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.369 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:14.369 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.749933+0000", 2024-09-21T15:49:14.369 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-21T15:49:14.369 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.369 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.370 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:14.370 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.749940+0000", 2024-09-21T15:49:14.370 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9739999999999996e-06 2024-09-21T15:49:14.370 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.370 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.370 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:14.370 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.749952+0000", 2024-09-21T15:49:14.370 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1800999999999999e-05 2024-09-21T15:49:14.370 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.370 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.370 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:14.370 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.749962+0000", 2024-09-21T15:49:14.370 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0815000000000001e-05 2024-09-21T15:49:14.370 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.370 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.371 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:14.371 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.750000+0000", 2024-09-21T15:49:14.371 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7394999999999997e-05 2024-09-21T15:49:14.371 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.371 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.371 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:14.371 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.750023+0000", 2024-09-21T15:49:14.371 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3274000000000001e-05 2024-09-21T15:49:14.371 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.371 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.371 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:14.371 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.752170+0000", 2024-09-21T15:49:14.371 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0021468849999999999 2024-09-21T15:49:14.371 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.371 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.371 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:14.371 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.752202+0000", 2024-09-21T15:49:14.372 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1526999999999998e-05 2024-09-21T15:49:14.372 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:14.372 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:14.372 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:14.372 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.372 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.372 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4403.0:1118 3.13 e29/22 3:ce51b6aa:::rbd_object_map.11335fa2fe69:head v 29'92, mlcod=27'90)", 2024-09-21T15:49:14.372 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:13.752816+0000", 2024-09-21T15:49:14.372 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.59818885099999997, 2024-09-21T15:49:14.372 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002274243, 2024-09-21T15:49:14.372 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:14.372 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:14.372 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:14.372 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.372 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:14.372 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.752816+0000", 2024-09-21T15:49:14.373 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:14.373 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.373 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.373 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:14.373 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.752816+0000", 2024-09-21T15:49:14.373 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:14.373 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.373 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.373 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:14.373 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.752816+0000", 2024-09-21T15:49:14.373 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:14.373 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.373 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.374 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:14.374 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.752821+0000", 2024-09-21T15:49:14.374 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5779999999999998e-06 2024-09-21T15:49:14.374 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.374 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.374 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:14.374 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.752832+0000", 2024-09-21T15:49:14.374 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0239e-05 2024-09-21T15:49:14.374 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.374 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.374 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:14.374 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.752836+0000", 2024-09-21T15:49:14.375 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8180000000000003e-06 2024-09-21T15:49:14.375 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.375 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.375 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:14.375 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.752872+0000", 2024-09-21T15:49:14.375 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5793000000000001e-05 2024-09-21T15:49:14.375 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.375 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.375 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:14.375 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.752885+0000", 2024-09-21T15:49:14.375 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3087e-05 2024-09-21T15:49:14.375 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.376 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.376 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:14.376 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.755059+0000", 2024-09-21T15:49:14.376 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0021738270000000001 2024-09-21T15:49:14.376 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.376 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.376 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:14.376 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.755091+0000", 2024-09-21T15:49:14.376 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1556000000000001e-05 2024-09-21T15:49:14.376 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:14.376 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:14.376 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:14.376 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.376 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.376 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4403.0:1132 3.e e29/22 3:74dd1916:::rbd_header.11335fa2fe69:head v 29'26, mlcod=29'24)", 2024-09-21T15:49:14.377 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:13.763492+0000", 2024-09-21T15:49:14.377 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.58751371699999999, 2024-09-21T15:49:14.377 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0020899719999999998, 2024-09-21T15:49:14.377 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:14.377 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:14.377 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:14.377 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.377 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:14.377 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.763492+0000", 2024-09-21T15:49:14.377 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:14.377 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.377 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.377 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:14.377 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.763492+0000", 2024-09-21T15:49:14.377 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:14.377 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.378 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.378 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:14.378 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.763491+0000", 2024-09-21T15:49:14.378 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-21T15:49:14.378 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.378 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.378 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:14.378 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.763497+0000", 2024-09-21T15:49:14.378 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4999999999999996e-06 2024-09-21T15:49:14.378 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.378 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.378 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:14.378 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.763509+0000", 2024-09-21T15:49:14.379 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1497e-05 2024-09-21T15:49:14.379 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.379 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.379 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:14.379 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.763515+0000", 2024-09-21T15:49:14.379 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.438e-06 2024-09-21T15:49:14.379 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.379 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.379 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:14.379 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.763552+0000", 2024-09-21T15:49:14.379 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6322000000000002e-05 2024-09-21T15:49:14.379 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.379 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.379 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:14.379 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.763573+0000", 2024-09-21T15:49:14.380 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1056e-05 2024-09-21T15:49:14.380 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.380 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.380 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:14.380 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.765552+0000", 2024-09-21T15:49:14.380 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001979565 2024-09-21T15:49:14.380 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.380 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.380 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:14.380 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.765582+0000", 2024-09-21T15:49:14.380 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9319000000000002e-05 2024-09-21T15:49:14.380 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:14.380 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:14.380 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:14.380 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.380 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.381 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4403.0:1133 3.a e29/22 3:535385f0:::rbd_children:head v 29'13, mlcod=27'11)", 2024-09-21T15:49:14.381 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:13.766287+0000", 2024-09-21T15:49:14.381 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.58471854700000003, 2024-09-21T15:49:14.381 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0019559500000000001, 2024-09-21T15:49:14.381 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:14.381 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:14.381 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:14.381 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.381 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:14.381 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.766287+0000", 2024-09-21T15:49:14.381 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:14.381 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.381 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.381 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:14.381 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.766287+0000", 2024-09-21T15:49:14.381 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:14.382 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.382 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.382 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:14.382 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.766287+0000", 2024-09-21T15:49:14.382 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:14.382 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.382 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.382 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:14.382 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.766290+0000", 2024-09-21T15:49:14.382 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0850000000000001e-06 2024-09-21T15:49:14.382 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.382 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.382 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:14.382 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.766299+0000", 2024-09-21T15:49:14.382 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2799999999999992e-06 2024-09-21T15:49:14.382 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.383 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.383 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:14.383 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.766305+0000", 2024-09-21T15:49:14.383 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9519999999999999e-06 2024-09-21T15:49:14.383 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.383 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.383 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:14.383 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.766343+0000", 2024-09-21T15:49:14.383 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8090000000000003e-05 2024-09-21T15:49:14.383 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.383 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.383 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:14.383 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.766358+0000", 2024-09-21T15:49:14.383 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4888999999999999e-05 2024-09-21T15:49:14.383 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.383 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.383 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:14.384 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.768209+0000", 2024-09-21T15:49:14.384 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001851512 2024-09-21T15:49:14.384 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.384 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.384 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:14.384 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.768243+0000", 2024-09-21T15:49:14.384 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3328999999999999e-05 2024-09-21T15:49:14.384 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:14.384 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:14.384 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:14.384 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.384 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.384 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4403.0:1152 3.1d e29/22 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:head v 29'32, mlcod=28'29)", 2024-09-21T15:49:14.384 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:13.781248+0000", 2024-09-21T15:49:14.384 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.56975773399999996, 2024-09-21T15:49:14.385 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0020040349999999999, 2024-09-21T15:49:14.385 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:14.385 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:14.385 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:14.385 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.385 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:14.385 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.781248+0000", 2024-09-21T15:49:14.385 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:14.385 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.385 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.385 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:14.385 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.781248+0000", 2024-09-21T15:49:14.385 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:14.385 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.385 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.385 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:14.385 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.781247+0000", 2024-09-21T15:49:14.386 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-21T15:49:14.386 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.386 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.386 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:14.386 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.781254+0000", 2024-09-21T15:49:14.386 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4749999999999996e-06 2024-09-21T15:49:14.386 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.386 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.386 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:14.386 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.781267+0000", 2024-09-21T15:49:14.386 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2704999999999999e-05 2024-09-21T15:49:14.386 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.386 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.386 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:14.386 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.781274+0000", 2024-09-21T15:49:14.387 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.849e-06 2024-09-21T15:49:14.387 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.387 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.387 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:14.387 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.781320+0000", 2024-09-21T15:49:14.387 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5664999999999997e-05 2024-09-21T15:49:14.387 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.387 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.387 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:14.387 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.781340+0000", 2024-09-21T15:49:14.387 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0613999999999999e-05 2024-09-21T15:49:14.387 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.387 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.387 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:14.387 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.783224+0000", 2024-09-21T15:49:14.387 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0018838209999999999 2024-09-21T15:49:14.387 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.388 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.388 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:14.388 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.783252+0000", 2024-09-21T15:49:14.388 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7648999999999999e-05 2024-09-21T15:49:14.388 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:14.388 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:14.388 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:14.388 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.388 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.388 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4403.0:1156 3.1d e29/22 3:bd483619:::rbd_data.1133f3dfbd6b.0000000000000007:head v 29'34, mlcod=28'29)", 2024-09-21T15:49:14.388 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:13.782294+0000", 2024-09-21T15:49:14.388 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.56871150500000001, 2024-09-21T15:49:14.388 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0059156499999999997, 2024-09-21T15:49:14.388 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:14.388 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:14.388 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:14.388 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.389 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:14.389 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.782294+0000", 2024-09-21T15:49:14.389 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:14.389 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.389 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.389 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:14.389 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.782294+0000", 2024-09-21T15:49:14.389 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:14.389 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.389 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.389 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:14.389 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.782293+0000", 2024-09-21T15:49:14.389 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-21T15:49:14.389 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.389 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.389 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:14.390 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.782301+0000", 2024-09-21T15:49:14.390 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5680000000000002e-06 2024-09-21T15:49:14.390 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.390 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.390 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:14.390 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.782313+0000", 2024-09-21T15:49:14.390 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2061e-05 2024-09-21T15:49:14.390 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.390 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.390 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:14.390 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.782320+0000", 2024-09-21T15:49:14.390 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8730000000000001e-06 2024-09-21T15:49:14.390 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.390 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.390 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:14.390 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.782381+0000", 2024-09-21T15:49:14.391 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1049000000000005e-05 2024-09-21T15:49:14.391 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.391 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.391 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:14.391 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.782400+0000", 2024-09-21T15:49:14.391 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9174999999999999e-05 2024-09-21T15:49:14.391 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.391 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.391 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:14.391 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.788182+0000", 2024-09-21T15:49:14.391 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0057820809999999997 2024-09-21T15:49:14.391 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.391 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.391 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:14.391 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.788209+0000", 2024-09-21T15:49:14.392 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7676000000000001e-05 2024-09-21T15:49:14.392 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:14.392 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:14.392 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:14.392 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.392 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.392 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4403.0:1154 3.e e29/22 3:7647128d:::rbd_data.1133f3dfbd6b.0000000000000004:head v 29'31, mlcod=29'26)", 2024-09-21T15:49:14.392 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:13.782446+0000", 2024-09-21T15:49:14.392 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.56855959300000003, 2024-09-21T15:49:14.392 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003141965, 2024-09-21T15:49:14.392 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:14.392 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:14.392 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:14.392 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.392 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:14.392 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.782446+0000", 2024-09-21T15:49:14.393 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:14.393 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.393 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.393 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:14.393 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.782446+0000", 2024-09-21T15:49:14.393 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:14.393 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.393 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.393 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:14.393 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.782445+0000", 2024-09-21T15:49:14.393 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-21T15:49:14.393 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.393 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.393 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:14.393 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.782452+0000", 2024-09-21T15:49:14.394 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1389999999999998e-06 2024-09-21T15:49:14.394 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.394 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.394 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:14.394 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.782465+0000", 2024-09-21T15:49:14.394 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2541000000000001e-05 2024-09-21T15:49:14.394 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.394 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.394 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:14.394 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.782471+0000", 2024-09-21T15:49:14.394 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4169999999999997e-06 2024-09-21T15:49:14.394 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.394 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.395 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:14.395 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.782508+0000", 2024-09-21T15:49:14.395 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6733000000000002e-05 2024-09-21T15:49:14.395 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.395 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.395 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:14.395 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.782529+0000", 2024-09-21T15:49:14.395 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1418000000000001e-05 2024-09-21T15:49:14.395 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.395 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.395 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:14.395 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.785546+0000", 2024-09-21T15:49:14.395 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0030171719999999998 2024-09-21T15:49:14.395 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.395 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.395 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:14.396 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:13.785588+0000", 2024-09-21T15:49:14.396 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1332999999999999e-05 2024-09-21T15:49:14.396 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:14.396 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:14.396 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:14.396 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.396 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.396 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4403.0:2072 3.13 e29/22 3:ce51b6aa:::rbd_object_map.11335fa2fe69:head v 29'97, mlcod=29'94)", 2024-09-21T15:49:14.396 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.133063+0000", 2024-09-21T15:49:14.396 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.21794218200000001, 2024-09-21T15:49:14.396 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0027114069999999999, 2024-09-21T15:49:14.396 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:14.396 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:14.396 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:14.396 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.397 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:14.397 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.133063+0000", 2024-09-21T15:49:14.397 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:14.397 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.397 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.397 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:14.397 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.133063+0000", 2024-09-21T15:49:14.397 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:14.397 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.397 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.397 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:14.397 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.133063+0000", 2024-09-21T15:49:14.397 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:14.397 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.397 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.397 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:14.398 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.133069+0000", 2024-09-21T15:49:14.398 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3199999999999996e-06 2024-09-21T15:49:14.398 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.398 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.398 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:14.398 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.133080+0000", 2024-09-21T15:49:14.398 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1439e-05 2024-09-21T15:49:14.398 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.398 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.398 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:14.398 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.133087+0000", 2024-09-21T15:49:14.398 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4470000000000003e-06 2024-09-21T15:49:14.398 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.398 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.399 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:14.399 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.133133+0000", 2024-09-21T15:49:14.399 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5896999999999998e-05 2024-09-21T15:49:14.399 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.399 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.399 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:14.399 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.133153+0000", 2024-09-21T15:49:14.399 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0111000000000001e-05 2024-09-21T15:49:14.399 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.399 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.399 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:14.399 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.135736+0000", 2024-09-21T15:49:14.399 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002583124 2024-09-21T15:49:14.399 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.399 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.400 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:14.400 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.135775+0000", 2024-09-21T15:49:14.400 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8735000000000001e-05 2024-09-21T15:49:14.400 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:14.400 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:14.400 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:14.400 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.400 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.400 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4403.0:2073 3.13 e29/22 3:ce51b6aa:::rbd_object_map.11335fa2fe69:head v 29'98, mlcod=29'94)", 2024-09-21T15:49:14.400 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.133731+0000", 2024-09-21T15:49:14.400 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.21727436899999999, 2024-09-21T15:49:14.400 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0047716700000000004, 2024-09-21T15:49:14.400 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:14.400 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:14.400 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:14.400 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.401 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:14.401 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.133731+0000", 2024-09-21T15:49:14.401 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:14.401 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.401 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.401 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:14.401 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.133731+0000", 2024-09-21T15:49:14.401 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:14.401 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.401 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.401 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:14.401 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.133730+0000", 2024-09-21T15:49:14.401 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:14.401 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.401 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.402 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:14.402 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.133736+0000", 2024-09-21T15:49:14.402 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9839999999999998e-06 2024-09-21T15:49:14.402 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.402 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.402 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:14.402 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.133747+0000", 2024-09-21T15:49:14.402 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0925e-05 2024-09-21T15:49:14.402 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.402 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.402 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:14.402 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.133754+0000", 2024-09-21T15:49:14.402 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3099999999999997e-06 2024-09-21T15:49:14.403 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.403 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.403 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:14.403 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.133794+0000", 2024-09-21T15:49:14.403 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0416000000000001e-05 2024-09-21T15:49:14.403 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.403 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.403 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:14.403 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.133814+0000", 2024-09-21T15:49:14.403 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9638000000000001e-05 2024-09-21T15:49:14.403 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.403 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.403 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:14.403 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.138464+0000", 2024-09-21T15:49:14.404 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0046502430000000001 2024-09-21T15:49:14.404 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.404 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.404 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:14.404 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.138503+0000", 2024-09-21T15:49:14.404 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8769000000000001e-05 2024-09-21T15:49:14.404 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:14.404 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:14.404 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:14.404 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.404 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.404 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4403.0:2074 3.13 e29/22 3:ce51b6aa:::rbd_object_map.11335fa2fe69:head v 29'99, mlcod=29'94)", 2024-09-21T15:49:14.404 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.134594+0000", 2024-09-21T15:49:14.405 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.21641168799999999, 2024-09-21T15:49:14.405 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0039321290000000004, 2024-09-21T15:49:14.405 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:14.405 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:14.405 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:14.405 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.405 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:14.405 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.134594+0000", 2024-09-21T15:49:14.405 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:14.405 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.405 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.405 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:14.405 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.134594+0000", 2024-09-21T15:49:14.405 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:14.405 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.405 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.406 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:14.406 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.134593+0000", 2024-09-21T15:49:14.406 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-21T15:49:14.406 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.406 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.406 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:14.406 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.134599+0000", 2024-09-21T15:49:14.406 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4860000000000002e-06 2024-09-21T15:49:14.406 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.406 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.406 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:14.406 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.134610+0000", 2024-09-21T15:49:14.406 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0696999999999999e-05 2024-09-21T15:49:14.406 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.407 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.407 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:14.407 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.134617+0000", 2024-09-21T15:49:14.407 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7669999999999998e-06 2024-09-21T15:49:14.407 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.407 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.407 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:14.407 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.134654+0000", 2024-09-21T15:49:14.407 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6739999999999997e-05 2024-09-21T15:49:14.407 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.407 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.407 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:14.407 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.134673+0000", 2024-09-21T15:49:14.407 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9792999999999998e-05 2024-09-21T15:49:14.407 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.408 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.408 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:14.408 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.138505+0000", 2024-09-21T15:49:14.408 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003831793 2024-09-21T15:49:14.408 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.408 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.408 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:14.408 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.138526+0000", 2024-09-21T15:49:14.408 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0616e-05 2024-09-21T15:49:14.408 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:14.408 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:14.408 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:14.408 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.408 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.408 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4403.0:2075 3.13 e29/22 3:ce51b6aa:::rbd_object_map.11335fa2fe69:head v 29'100, mlcod=29'94)", 2024-09-21T15:49:14.408 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.135271+0000", 2024-09-21T15:49:14.409 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.21573463400000001, 2024-09-21T15:49:14.409 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0032754099999999999, 2024-09-21T15:49:14.409 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:14.409 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:14.409 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:14.409 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.409 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:14.409 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.135271+0000", 2024-09-21T15:49:14.409 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:14.409 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.409 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.409 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:14.409 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.135271+0000", 2024-09-21T15:49:14.409 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:14.409 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.409 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.409 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:14.410 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.135270+0000", 2024-09-21T15:49:14.410 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:14.410 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.410 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.410 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:14.410 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.135279+0000", 2024-09-21T15:49:14.410 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4020000000000008e-06 2024-09-21T15:49:14.410 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.410 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.410 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:14.410 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.135291+0000", 2024-09-21T15:49:14.410 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2058e-05 2024-09-21T15:49:14.410 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.410 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.410 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:14.410 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.135298+0000", 2024-09-21T15:49:14.411 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2879999999999998e-06 2024-09-21T15:49:14.411 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.411 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.411 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:14.411 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.135360+0000", 2024-09-21T15:49:14.411 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2281999999999998e-05 2024-09-21T15:49:14.411 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.411 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.411 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:14.411 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.135372+0000", 2024-09-21T15:49:14.411 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1627999999999999e-05 2024-09-21T15:49:14.411 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.411 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.411 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:14.411 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.138527+0000", 2024-09-21T15:49:14.411 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0031551399999999999 2024-09-21T15:49:14.411 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.412 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.412 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:14.412 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.138546+0000", 2024-09-21T15:49:14.412 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9313999999999998e-05 2024-09-21T15:49:14.412 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:14.412 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:14.412 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:14.412 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.412 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.412 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4403.0:2076 3.13 e29/22 3:ce51b6aa:::rbd_object_map.11335fa2fe69:head v 29'101, mlcod=29'94)", 2024-09-21T15:49:14.412 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.136123+0000", 2024-09-21T15:49:14.412 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.21488211500000001, 2024-09-21T15:49:14.412 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0052685789999999998, 2024-09-21T15:49:14.412 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:14.412 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:14.412 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:14.412 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.413 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:14.413 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.136123+0000", 2024-09-21T15:49:14.413 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:14.413 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.413 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.413 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:14.413 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.136123+0000", 2024-09-21T15:49:14.413 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:14.413 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.413 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.413 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:14.413 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.136123+0000", 2024-09-21T15:49:14.413 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:14.413 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.413 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.413 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:14.414 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.136129+0000", 2024-09-21T15:49:14.414 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3339999999999998e-06 2024-09-21T15:49:14.414 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.414 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.414 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:14.414 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.136138+0000", 2024-09-21T15:49:14.414 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0990000000000005e-06 2024-09-21T15:49:14.414 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.414 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.414 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:14.414 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.136142+0000", 2024-09-21T15:49:14.414 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1849999999999997e-06 2024-09-21T15:49:14.414 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.414 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.414 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:14.415 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.136176+0000", 2024-09-21T15:49:14.415 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4215999999999999e-05 2024-09-21T15:49:14.415 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.415 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.415 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:14.415 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.136195+0000", 2024-09-21T15:49:14.415 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8813000000000001e-05 2024-09-21T15:49:14.415 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.415 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.415 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:14.415 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.141357+0000", 2024-09-21T15:49:14.415 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0051618710000000002 2024-09-21T15:49:14.415 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.415 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.416 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:14.416 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.141392+0000", 2024-09-21T15:49:14.416 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4678999999999998e-05 2024-09-21T15:49:14.416 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:14.416 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:14.416 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:14.416 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.416 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.416 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4403.0:2082 3.13 e29/22 3:ce51b6aa:::rbd_object_map.11335fa2fe69:head v 29'102, mlcod=29'99)", 2024-09-21T15:49:14.416 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.140470+0000", 2024-09-21T15:49:14.416 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.21053503500000001, 2024-09-21T15:49:14.416 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003569203, 2024-09-21T15:49:14.416 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:14.416 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:14.416 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:14.416 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.417 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:14.417 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.140470+0000", 2024-09-21T15:49:14.417 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:14.417 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.417 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.417 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:14.417 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.140470+0000", 2024-09-21T15:49:14.417 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:14.417 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.417 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.417 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:14.417 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.140470+0000", 2024-09-21T15:49:14.417 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:14.417 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.417 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.417 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:14.418 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.140474+0000", 2024-09-21T15:49:14.418 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7430000000000002e-06 2024-09-21T15:49:14.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.418 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:14.418 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.140484+0000", 2024-09-21T15:49:14.418 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0580000000000004e-06 2024-09-21T15:49:14.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.418 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:14.418 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.140488+0000", 2024-09-21T15:49:14.418 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4209999999999997e-06 2024-09-21T15:49:14.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.418 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:14.419 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.140524+0000", 2024-09-21T15:49:14.419 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5638e-05 2024-09-21T15:49:14.419 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.419 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.419 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:14.419 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.140537+0000", 2024-09-21T15:49:14.419 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.306e-05 2024-09-21T15:49:14.419 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.419 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.419 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:14.419 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.143996+0000", 2024-09-21T15:49:14.419 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0034590369999999999 2024-09-21T15:49:14.419 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.419 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.420 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:14.420 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.144040+0000", 2024-09-21T15:49:14.420 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3810000000000002e-05 2024-09-21T15:49:14.420 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:14.420 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:14.420 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:14.420 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.420 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.420 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4403.0:2083 3.a e29/22 3:55b68c50:::rbd_data.11335fa2fe69.0000000000000191:head v 29'14, mlcod=27'12)", 2024-09-21T15:49:14.420 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.142213+0000", 2024-09-21T15:49:14.420 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.20879215800000001, 2024-09-21T15:49:14.420 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.004555992, 2024-09-21T15:49:14.420 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:14.420 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:14.420 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:14.421 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.421 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:14.421 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.142213+0000", 2024-09-21T15:49:14.421 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:14.421 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.421 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.421 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:14.421 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.142213+0000", 2024-09-21T15:49:14.421 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:14.421 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.421 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.421 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:14.421 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.142212+0000", 2024-09-21T15:49:14.421 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:14.422 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.422 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.422 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:14.422 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.142219+0000", 2024-09-21T15:49:14.422 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3910000000000003e-06 2024-09-21T15:49:14.422 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.422 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.422 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:14.422 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.142233+0000", 2024-09-21T15:49:14.422 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4324e-05 2024-09-21T15:49:14.422 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.422 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.422 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:14.422 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.142240+0000", 2024-09-21T15:49:14.423 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.024e-06 2024-09-21T15:49:14.423 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.423 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.423 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:14.423 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.142284+0000", 2024-09-21T15:49:14.423 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4124e-05 2024-09-21T15:49:14.423 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.423 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.423 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:14.423 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.142305+0000", 2024-09-21T15:49:14.423 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0180999999999999e-05 2024-09-21T15:49:14.423 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.423 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.423 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:14.423 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.146740+0000", 2024-09-21T15:49:14.423 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.004435385 2024-09-21T15:49:14.424 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.424 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.424 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:14.424 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.146769+0000", 2024-09-21T15:49:14.424 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9277000000000001e-05 2024-09-21T15:49:14.424 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:14.424 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:14.424 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:14.424 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.424 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.424 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4403.0:2085 3.13 e29/22 3:ce51b6aa:::rbd_object_map.11335fa2fe69:head v 29'103, mlcod=29'101)", 2024-09-21T15:49:14.424 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.147470+0000", 2024-09-21T15:49:14.424 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.20353545100000001, 2024-09-21T15:49:14.424 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0023658379999999999, 2024-09-21T15:49:14.424 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:14.425 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:14.425 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:14.425 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.425 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:14.425 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.147470+0000", 2024-09-21T15:49:14.425 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:14.425 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.425 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.425 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:14.425 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.147470+0000", 2024-09-21T15:49:14.425 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:14.425 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.425 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.425 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:14.426 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.147469+0000", 2024-09-21T15:49:14.426 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-21T15:49:14.426 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.426 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.426 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:14.426 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.147475+0000", 2024-09-21T15:49:14.426 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4749999999999998e-06 2024-09-21T15:49:14.426 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.426 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.426 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:14.426 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.147494+0000", 2024-09-21T15:49:14.426 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8991000000000002e-05 2024-09-21T15:49:14.426 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.426 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.427 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:14.427 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.147500+0000", 2024-09-21T15:49:14.427 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0789999999999997e-06 2024-09-21T15:49:14.427 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.427 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.427 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:14.427 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.147540+0000", 2024-09-21T15:49:14.427 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9019e-05 2024-09-21T15:49:14.427 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.427 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.427 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:14.427 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.147560+0000", 2024-09-21T15:49:14.427 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.052e-05 2024-09-21T15:49:14.427 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.427 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.428 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:14.428 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.149805+0000", 2024-09-21T15:49:14.428 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0022450959999999998 2024-09-21T15:49:14.428 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.428 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.428 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:14.428 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.149836+0000", 2024-09-21T15:49:14.428 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0604e-05 2024-09-21T15:49:14.428 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:14.428 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:14.428 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:14.428 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.428 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.428 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4403.0:2116 3.e e29/22 3:72b4795c:::rbd_data.11335fa2fe69.0000000000000197:head v 29'36, mlcod=29'34)", 2024-09-21T15:49:14.428 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.192977+0000", 2024-09-21T15:49:14.429 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.158028113, 2024-09-21T15:49:14.429 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0021186080000000001, 2024-09-21T15:49:14.429 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:14.429 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:14.429 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:14.429 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.429 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:14.429 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.192977+0000", 2024-09-21T15:49:14.429 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:14.429 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.429 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.429 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:14.429 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.192977+0000", 2024-09-21T15:49:14.429 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:14.429 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.429 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.430 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:14.430 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.192977+0000", 2024-09-21T15:49:14.430 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:14.430 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.430 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.430 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:14.430 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.192982+0000", 2024-09-21T15:49:14.430 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9559999999999996e-06 2024-09-21T15:49:14.430 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.430 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.430 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:14.430 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.192990+0000", 2024-09-21T15:49:14.430 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8519999999999993e-06 2024-09-21T15:49:14.430 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.431 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.431 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:14.431 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.192994+0000", 2024-09-21T15:49:14.431 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0239999999999999e-06 2024-09-21T15:49:14.431 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.431 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.431 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:14.431 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.193045+0000", 2024-09-21T15:49:14.431 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0315999999999997e-05 2024-09-21T15:49:14.431 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.431 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.431 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:14.431 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.193064+0000", 2024-09-21T15:49:14.431 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9159000000000001e-05 2024-09-21T15:49:14.432 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.432 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.432 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:14.432 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.195072+0000", 2024-09-21T15:49:14.432 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0020085490000000001 2024-09-21T15:49:14.432 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.432 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.432 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:14.432 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.195096+0000", 2024-09-21T15:49:14.432 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3359000000000001e-05 2024-09-21T15:49:14.432 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:14.432 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:14.432 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:14.432 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.433 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.433 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4403.0:3661 3.11 e29/22 3:895cc7c9:::rbd_data.11335fa2fe69.00000000000000b7:head v 29'8, mlcod=29'6)", 2024-09-21T15:49:14.433 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.260527+0000", 2024-09-21T15:49:14.433 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.090478458999999997, 2024-09-21T15:49:14.433 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002074846, 2024-09-21T15:49:14.433 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:14.433 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:14.433 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:14.433 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.433 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:14.433 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.260527+0000", 2024-09-21T15:49:14.433 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:14.433 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.433 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.433 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:14.433 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.260527+0000", 2024-09-21T15:49:14.434 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:14.434 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.434 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.434 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:14.434 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.260526+0000", 2024-09-21T15:49:14.434 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-21T15:49:14.434 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.434 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.434 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:14.434 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.260533+0000", 2024-09-21T15:49:14.434 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5690000000000001e-06 2024-09-21T15:49:14.434 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.435 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.435 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:14.435 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.260543+0000", 2024-09-21T15:49:14.435 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0743e-05 2024-09-21T15:49:14.435 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.435 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.435 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:14.435 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.260550+0000", 2024-09-21T15:49:14.435 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1340000000000001e-06 2024-09-21T15:49:14.435 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.435 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.435 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:14.436 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.260594+0000", 2024-09-21T15:49:14.436 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4087e-05 2024-09-21T15:49:14.436 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.436 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.436 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:14.436 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.260616+0000", 2024-09-21T15:49:14.436 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2399e-05 2024-09-21T15:49:14.436 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.436 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.436 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:14.436 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.262574+0000", 2024-09-21T15:49:14.436 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0019580019999999999 2024-09-21T15:49:14.436 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:14.436 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:14.437 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:14.437 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.262602+0000", 2024-09-21T15:49:14.437 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7651e-05 2024-09-21T15:49:14.437 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:14.437 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:14.437 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:14.437 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:14.437 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:14.437 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:14.438 INFO:teuthology.orchestra.run.smithi089.stdout:411 write 0xd48e9c thru 0xd530ce (0xa233 bytes) 2024-09-21T15:49:14.439 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:49:14.439 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:14.437+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:14.532 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:49:14.557 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:14.557 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:14.557 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:49:14.557 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:14.566 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:49:14.577 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:14.576+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:14.593 INFO:teuthology.orchestra.run.smithi089.stdout:412 read 0x108b159 thru 0x108ed96 (0x3c3e bytes) 2024-09-21T15:49:14.594 INFO:teuthology.orchestra.run.smithi089.stdout:413 punch from 0x1e7afa2 to 0x1e7b991, (0x9ef bytes) 2024-09-21T15:49:14.594 INFO:teuthology.orchestra.run.smithi089.stdout:415 read 0x4caf177 thru 0x4cb17b7 (0x2641 bytes) 2024-09-21T15:49:14.595 INFO:teuthology.orchestra.run.smithi089.stdout:416 read 0x34e50df thru 0x34e82be (0x31e0 bytes) 2024-09-21T15:49:14.600 INFO:teuthology.orchestra.run.smithi089.stdout:419 trunc from 0x5d1dd7b to 0x1b839a2 2024-09-21T15:49:14.639 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:14.638+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:14.662 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-09-21T15:49:14.662 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:49:14.679 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:49:14.679 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:49:14.686 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:14.686 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:14.686 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:49:14.686 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:49:14.686 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:14.697 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:49:14.697 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:49:14.697 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:49:14.712 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:49:14.712 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:49:14.729 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:49:14.730 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:49:14.731 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T15:49:14.746 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:49:14.747 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:49:14.764 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:49:14.764 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:49:14.781 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:14.780+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:14.782 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:49:14.782 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:49:14.841 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:14.840+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:14.869 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:14.869 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:14.879 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T15:49:14.895 INFO:teuthology.orchestra.run.smithi089.stdout:420 flatten 2024-09-21T15:49:14.964 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:14.963+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:15.024 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:15.024 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:15.034 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T15:49:15.075 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:15.074+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:15.144 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:15.142+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:15.178 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:15.178 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:15.187 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T15:49:15.244 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:15.243+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:15.340 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:15.340 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:15.346 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:15.345+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:15.350 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T15:49:15.447 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:15.446+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:15.448 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:15.448 INFO:teuthology.orchestra.run.smithi089.stdout:422 read 0xdf8359 thru 0xe06a07 (0xe6af bytes) 2024-09-21T15:49:15.448 INFO:teuthology.orchestra.run.smithi089.stdout:423 punch from 0x130ef6c to 0x13132eb, (0x437f bytes) 2024-09-21T15:49:15.448 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:15.448 INFO:teuthology.orchestra.run.smithi089.stdout:426 punch from 0xe816b5 to 0xe90c57, (0xf5a2 bytes) 2024-09-21T15:49:15.455 INFO:teuthology.orchestra.run.smithi089.stdout:427 write 0x28590e5 thru 0x2859f5c (0xe78 bytes) 2024-09-21T15:49:15.459 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:15.459 INFO:teuthology.orchestra.run.smithi089.stdout:429 read 0x1e1b812 thru 0x1e20a07 (0x51f6 bytes) 2024-09-21T15:49:15.459 INFO:teuthology.orchestra.run.smithi089.stdout:430 read 0x1c41e94 thru 0x1c46f2f (0x509c bytes) 2024-09-21T15:49:15.467 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:15.467 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:15.468 INFO:teuthology.orchestra.run.smithi089.stdout:431 write 0x39b4ef8 thru 0x39bbdb9 (0x6ec2 bytes) 2024-09-21T15:49:15.478 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T15:49:15.480 INFO:teuthology.orchestra.run.smithi089.stdout:432 read 0x1a0b914 thru 0x1a1081c (0x4f09 bytes) 2024-09-21T15:49:15.487 INFO:teuthology.orchestra.run.smithi089.stdout:433 trunc from 0x39bbdba to 0x8f181da 2024-09-21T15:49:15.492 INFO:teuthology.orchestra.run.smithi089.stdout:434 write 0x16d2f68 thru 0x16de60a (0xb6a3 bytes) 2024-09-21T15:49:15.517 INFO:teuthology.orchestra.run.smithi089.stdout:436 write 0x9b96baf thru 0x9b98bdd (0x202f bytes) 2024-09-21T15:49:15.524 INFO:teuthology.orchestra.run.smithi089.stdout:438 read 0x77e6fb8 thru 0x77f0722 (0x976b bytes) 2024-09-21T15:49:15.524 INFO:teuthology.orchestra.run.smithi089.stdout:440 write 0x59aef2e thru 0x59b3d2d (0x4e00 bytes) 2024-09-21T15:49:15.535 INFO:teuthology.orchestra.run.smithi089.stdout:441 trunc from 0x9b98bde to 0xab62798 2024-09-21T15:49:15.541 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:15.541 INFO:teuthology.orchestra.run.smithi089.stdout:443 read 0x77a9a3e thru 0x77b38bf (0x9e82 bytes) 2024-09-21T15:49:15.541 INFO:teuthology.orchestra.run.smithi089.stdout:444 read 0x119e133 thru 0x11aa926 (0xc7f4 bytes) 2024-09-21T15:49:15.545 INFO:teuthology.orchestra.run.smithi089.stdout:445 trunc from 0xab62798 to 0x802d940 2024-09-21T15:49:15.558 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:15.558 INFO:teuthology.orchestra.run.smithi089.stdout:449 writesame 0x45c3fbc thru 0x45c6a03 data_size 0x52(0x2a48 bytes) 2024-09-21T15:49:15.562 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:15.561+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:15.577 INFO:teuthology.orchestra.run.smithi089.stdout:450 write 0xeab6ebe thru 0xeac384c (0xc98f bytes) 2024-09-21T15:49:15.595 INFO:teuthology.orchestra.run.smithi089.stdout:452 read 0x8c552b3 thru 0x8c55cdf (0xa2d bytes) 2024-09-21T15:49:15.595 INFO:teuthology.orchestra.run.smithi089.stdout:455 writesame 0x263aef thru 0x273757 data_size 0x7(0xfc69 bytes) 2024-09-21T15:49:15.607 INFO:teuthology.orchestra.run.smithi089.stdout:457 read 0x9686ad3 thru 0x969286f (0xbd9d bytes) 2024-09-21T15:49:15.607 INFO:teuthology.orchestra.run.smithi089.stdout:458 read 0x5b319f2 thru 0x5b38b67 (0x7176 bytes) 2024-09-21T15:49:15.608 INFO:teuthology.orchestra.run.smithi089.stdout:459 punch from 0xce7af22 to 0xce83f34, (0x9012 bytes) 2024-09-21T15:49:15.608 INFO:teuthology.orchestra.run.smithi089.stdout:460 read 0xe18eb05 thru 0xe18fa0d (0xf09 bytes) 2024-09-21T15:49:15.608 INFO:teuthology.orchestra.run.smithi089.stdout:461 write 0xba0d896 thru 0xba0eb16 (0x1281 bytes) 2024-09-21T15:49:15.611 INFO:teuthology.orchestra.run.smithi089.stdout:462 write 0x2e99e79 thru 0x2ea9645 (0xf7cd bytes) 2024-09-21T15:49:15.632 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:15.632 INFO:teuthology.orchestra.run.smithi089.stdout:464 read 0x6507277 thru 0x650d5c5 (0x634f bytes) 2024-09-21T15:49:15.632 INFO:teuthology.orchestra.run.smithi089.stdout:466 writesame 0xb6f5e9 thru 0xb7c53d data_size 0x1(0xcf55 bytes) 2024-09-21T15:49:15.642 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:15.642 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:15.649 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:15.648+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:15.651 INFO:teuthology.orchestra.run.smithi089.stdout:469 writesame 0x152526d thru 0x152dd30 data_size 0x53(0x8ac4 bytes) 2024-09-21T15:49:15.652 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T15:49:15.674 INFO:teuthology.orchestra.run.smithi089.stdout:470 trunc from 0xeac384d to 0x201fd15 2024-09-21T15:49:15.688 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T15:49:15.688 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T15:49:15.688 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-21T15:49:15.689 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-21T15:49:15.721 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is unset 2024-09-21T15:49:15.727 INFO:teuthology.orchestra.run.smithi089.stdout:472 writesame 0xabfbae8 thru 0xac017e6 data_size 0x13(0x5cff bytes) 2024-09-21T15:49:15.728 INFO:teuthology.orchestra.run.smithi089.stdout:473 write 0x674ef91 thru 0x675a0d5 (0xb145 bytes) 2024-09-21T15:49:15.747 INFO:teuthology.orchestra.run.smithi089.stdout:474 trunc from 0xac017e7 to 0x92c4591 2024-09-21T15:49:15.749 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:15.748+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:15.753 INFO:teuthology.orchestra.run.smithi089.stdout:475 trunc from 0x92c4591 to 0x87db1c4 2024-09-21T15:49:15.758 INFO:teuthology.orchestra.run.smithi089.stdout:476 read 0x80ef92d thru 0x80f5bb3 (0x6287 bytes) 2024-09-21T15:49:15.758 INFO:teuthology.orchestra.run.smithi089.stdout:477 writesame 0x2ce8474 thru 0x2cede7b data_size 0x8(0x5a08 bytes) 2024-09-21T15:49:15.775 INFO:teuthology.orchestra.run.smithi089.stdout:478 trunc from 0x87db1c4 to 0x2782465 2024-09-21T15:49:15.798 INFO:teuthology.orchestra.run.smithi089.stdout:truncating to largest ever: 0xd470b73 2024-09-21T15:49:15.807 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:15.807 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:15.811 INFO:teuthology.orchestra.run.smithi089.stdout:480 trunc from 0x2782465 to 0xd470b73 2024-09-21T15:49:15.817 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:49:15.834 INFO:teuthology.orchestra.run.smithi089.stdout:482 trunc from 0xd470b73 to 0x34bf73c 2024-09-21T15:49:15.840 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:15.841 INFO:teuthology.orchestra.run.smithi089.stdout:484 trunc from 0x34bf73c to 0x251423c 2024-09-21T15:49:15.851 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:15.849+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:15.851 INFO:teuthology.orchestra.run.smithi089.stdout:485 read 0x179aae0 thru 0x17a711d (0xc63e bytes) 2024-09-21T15:49:15.851 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:15.851 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:15.851 INFO:teuthology.orchestra.run.smithi089.stdout:490 read 0xbdb2a9 thru 0xbdf7dd (0x4535 bytes) 2024-09-21T15:49:15.867 INFO:teuthology.orchestra.run.smithi089.stdout:491 trunc from 0x251423c to 0xc5ec660 2024-09-21T15:49:15.872 INFO:teuthology.orchestra.run.smithi089.stdout:492 read 0x130b4a7 thru 0x1311acc (0x6626 bytes) 2024-09-21T15:49:15.873 INFO:teuthology.orchestra.run.smithi089.stdout:493 write 0x5eda163 thru 0x5ee4185 (0xa023 bytes) 2024-09-21T15:49:15.897 INFO:teuthology.orchestra.run.smithi089.stdout:494 write 0xe10dbe0 thru 0xe11842e (0xa84f bytes) 2024-09-21T15:49:15.914 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:15.914 INFO:teuthology.orchestra.run.smithi089.stdout:498 write 0x1f9cb3c thru 0x1fac919 (0xfdde bytes) 2024-09-21T15:49:15.932 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:15.932 INFO:teuthology.orchestra.run.smithi089.stdout:501 punch from 0xe01659e to 0xe025155, (0xebb7 bytes) 2024-09-21T15:49:15.932 INFO:teuthology.orchestra.run.smithi089.stdout:502 writesame 0x6474cc3 thru 0x6477fe8 data_size 0x1(0x3326 bytes) 2024-09-21T15:49:15.934 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:15.934 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:15.942 INFO:teuthology.orchestra.run.smithi089.stdout:505 read 0xb07110b thru 0xb073ed1 (0x2dc7 bytes) 2024-09-21T15:49:15.942 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:15.942 INFO:teuthology.orchestra.run.smithi089.stdout:508 write 0x66f46a3 thru 0x6702d29 (0xe687 bytes) 2024-09-21T15:49:15.943 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:49:15.959 INFO:teuthology.orchestra.run.smithi089.stdout:510 write 0x2dfea0d thru 0x2e09a1c (0xb010 bytes) 2024-09-21T15:49:15.959 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:15.958+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:15.974 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:15.974 INFO:teuthology.orchestra.run.smithi089.stdout:513 read 0xa0e2309 thru 0xa0eab86 (0x887e bytes) 2024-09-21T15:49:15.974 INFO:teuthology.orchestra.run.smithi089.stdout:514 read 0xf4b4f thru 0x100cba (0xc16c bytes) 2024-09-21T15:49:15.975 INFO:teuthology.orchestra.run.smithi089.stdout:515 punch from 0x385f0ee to 0x38645ad, (0x54bf bytes) 2024-09-21T15:49:15.984 INFO:teuthology.orchestra.run.smithi089.stdout:516 trunc from 0xe11842f to 0x7115f4e 2024-09-21T15:49:16.004 INFO:teuthology.orchestra.run.smithi089.stdout:518 write 0xa07129a thru 0xa07dfa7 (0xcd0e bytes) 2024-09-21T15:49:16.026 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:49:16.026 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:16.026 INFO:teuthology.orchestra.run.smithi089.stdout: "epoch": 30, 2024-09-21T15:49:16.026 INFO:teuthology.orchestra.run.smithi089.stdout: "fsid": "4a9b64ea-3aaf-4a40-8ce8-02163a0f6e12", 2024-09-21T15:49:16.026 INFO:teuthology.orchestra.run.smithi089.stdout: "created": "2024-09-21T15:48:37.867501+0000", 2024-09-21T15:49:16.026 INFO:teuthology.orchestra.run.smithi089.stdout: "modified": "2024-09-21T15:49:15.715519+0000", 2024-09-21T15:49:16.026 INFO:teuthology.orchestra.run.smithi089.stdout: "last_up_change": "2024-09-21T15:48:49.160053+0000", 2024-09-21T15:49:16.026 INFO:teuthology.orchestra.run.smithi089.stdout: "last_in_change": "2024-09-21T15:48:40.108789+0000", 2024-09-21T15:49:16.026 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T15:49:16.026 INFO:teuthology.orchestra.run.smithi089.stdout: "flags_num": 5799936, 2024-09-21T15:49:16.026 INFO:teuthology.orchestra.run.smithi089.stdout: "flags_set": [ 2024-09-21T15:49:16.027 INFO:teuthology.orchestra.run.smithi089.stdout: "pglog_hardlimit", 2024-09-21T15:49:16.027 INFO:teuthology.orchestra.run.smithi089.stdout: "purged_snapdirs", 2024-09-21T15:49:16.027 INFO:teuthology.orchestra.run.smithi089.stdout: "recovery_deletes", 2024-09-21T15:49:16.027 INFO:teuthology.orchestra.run.smithi089.stdout: "sortbitwise" 2024-09-21T15:49:16.027 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-21T15:49:16.027 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_version": 4, 2024-09-21T15:49:16.027 INFO:teuthology.orchestra.run.smithi089.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T15:49:16.027 INFO:teuthology.orchestra.run.smithi089.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T15:49:16.027 INFO:teuthology.orchestra.run.smithi089.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T15:49:16.027 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_snapshot": "", 2024-09-21T15:49:16.027 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_max": 3, 2024-09-21T15:49:16.027 INFO:teuthology.orchestra.run.smithi089.stdout: "max_osd": 8, 2024-09-21T15:49:16.027 INFO:teuthology.orchestra.run.smithi089.stdout: "require_min_compat_client": "luminous", 2024-09-21T15:49:16.027 INFO:teuthology.orchestra.run.smithi089.stdout: "min_compat_client": "jewel", 2024-09-21T15:49:16.028 INFO:teuthology.orchestra.run.smithi089.stdout: "require_osd_release": "reef", 2024-09-21T15:49:16.028 INFO:teuthology.orchestra.run.smithi089.stdout: "allow_crimson": false, 2024-09-21T15:49:16.028 INFO:teuthology.orchestra.run.smithi089.stdout: "pools": [ 2024-09-21T15:49:16.028 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.028 INFO:teuthology.orchestra.run.smithi089.stdout: "pool": 1, 2024-09-21T15:49:16.028 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_name": ".mgr", 2024-09-21T15:49:16.028 INFO:teuthology.orchestra.run.smithi089.stdout: "create_time": "2024-09-21T15:48:49.921079+0000", 2024-09-21T15:49:16.028 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-21T15:49:16.028 INFO:teuthology.orchestra.run.smithi089.stdout: "flags_names": "hashpspool", 2024-09-21T15:49:16.028 INFO:teuthology.orchestra.run.smithi089.stdout: "type": 1, 2024-09-21T15:49:16.028 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 2, 2024-09-21T15:49:16.028 INFO:teuthology.orchestra.run.smithi089.stdout: "min_size": 1, 2024-09-21T15:49:16.028 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_rule": 0, 2024-09-21T15:49:16.029 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_count": 0, 2024-09-21T15:49:16.029 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_target": 0, 2024-09-21T15:49:16.029 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T15:49:16.029 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T15:49:16.029 INFO:teuthology.orchestra.run.smithi089.stdout: "object_hash": 2, 2024-09-21T15:49:16.029 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_autoscale_mode": "off", 2024-09-21T15:49:16.029 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num": 1, 2024-09-21T15:49:16.029 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num": 1, 2024-09-21T15:49:16.029 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num_target": 1, 2024-09-21T15:49:16.029 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_target": 1, 2024-09-21T15:49:16.029 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_pending": 1, 2024-09-21T15:49:16.029 INFO:teuthology.orchestra.run.smithi089.stdout: "last_pg_merge_meta": { 2024-09-21T15:49:16.029 INFO:teuthology.orchestra.run.smithi089.stdout: "source_pgid": "0.0", 2024-09-21T15:49:16.029 INFO:teuthology.orchestra.run.smithi089.stdout: "ready_epoch": 0, 2024-09-21T15:49:16.030 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_started": 0, 2024-09-21T15:49:16.030 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_clean": 0, 2024-09-21T15:49:16.030 INFO:teuthology.orchestra.run.smithi089.stdout: "source_version": "0'0", 2024-09-21T15:49:16.030 INFO:teuthology.orchestra.run.smithi089.stdout: "target_version": "0'0" 2024-09-21T15:49:16.030 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.030 INFO:teuthology.orchestra.run.smithi089.stdout: "last_change": "16", 2024-09-21T15:49:16.030 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend": "0", 2024-09-21T15:49:16.030 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T15:49:16.030 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T15:49:16.030 INFO:teuthology.orchestra.run.smithi089.stdout: "auid": 0, 2024-09-21T15:49:16.030 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_mode": "selfmanaged", 2024-09-21T15:49:16.030 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_seq": 0, 2024-09-21T15:49:16.031 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_epoch": 0, 2024-09-21T15:49:16.031 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_snaps": [], 2024-09-21T15:49:16.031 INFO:teuthology.orchestra.run.smithi089.stdout: "removed_snaps": "[]", 2024-09-21T15:49:16.031 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_bytes": 0, 2024-09-21T15:49:16.031 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_objects": 0, 2024-09-21T15:49:16.031 INFO:teuthology.orchestra.run.smithi089.stdout: "tiers": [], 2024-09-21T15:49:16.031 INFO:teuthology.orchestra.run.smithi089.stdout: "tier_of": -1, 2024-09-21T15:49:16.031 INFO:teuthology.orchestra.run.smithi089.stdout: "read_tier": -1, 2024-09-21T15:49:16.031 INFO:teuthology.orchestra.run.smithi089.stdout: "write_tier": -1, 2024-09-21T15:49:16.031 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_mode": "none", 2024-09-21T15:49:16.031 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_bytes": 0, 2024-09-21T15:49:16.031 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_objects": 0, 2024-09-21T15:49:16.031 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T15:49:16.032 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T15:49:16.032 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T15:49:16.032 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_flush_age": 0, 2024-09-21T15:49:16.032 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_evict_age": 0, 2024-09-21T15:49:16.032 INFO:teuthology.orchestra.run.smithi089.stdout: "erasure_code_profile": "", 2024-09-21T15:49:16.032 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_params": { 2024-09-21T15:49:16.032 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "none" 2024-09-21T15:49:16.032 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.032 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_period": 0, 2024-09-21T15:49:16.032 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_count": 0, 2024-09-21T15:49:16.032 INFO:teuthology.orchestra.run.smithi089.stdout: "use_gmt_hitset": true, 2024-09-21T15:49:16.032 INFO:teuthology.orchestra.run.smithi089.stdout: "min_read_recency_for_promote": 0, 2024-09-21T15:49:16.033 INFO:teuthology.orchestra.run.smithi089.stdout: "min_write_recency_for_promote": 0, 2024-09-21T15:49:16.033 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T15:49:16.033 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_search_last_n": 0, 2024-09-21T15:49:16.033 INFO:teuthology.orchestra.run.smithi089.stdout: "grade_table": [], 2024-09-21T15:49:16.033 INFO:teuthology.orchestra.run.smithi089.stdout: "stripe_width": 0, 2024-09-21T15:49:16.033 INFO:teuthology.orchestra.run.smithi089.stdout: "expected_num_objects": 0, 2024-09-21T15:49:16.033 INFO:teuthology.orchestra.run.smithi089.stdout: "fast_read": false, 2024-09-21T15:49:16.033 INFO:teuthology.orchestra.run.smithi089.stdout: "options": { 2024-09-21T15:49:16.033 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_max": 32, 2024-09-21T15:49:16.033 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_min": 1 2024-09-21T15:49:16.033 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.033 INFO:teuthology.orchestra.run.smithi089.stdout: "application_metadata": { 2024-09-21T15:49:16.033 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": {} 2024-09-21T15:49:16.033 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.033 INFO:teuthology.orchestra.run.smithi089.stdout: "read_balance": { 2024-09-21T15:49:16.034 INFO:teuthology.orchestra.run.smithi089.stdout: "score_acting": 8, 2024-09-21T15:49:16.034 INFO:teuthology.orchestra.run.smithi089.stdout: "score_stable": 8, 2024-09-21T15:49:16.034 INFO:teuthology.orchestra.run.smithi089.stdout: "optimal_score": 0.25, 2024-09-21T15:49:16.034 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_acting": 2, 2024-09-21T15:49:16.034 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_stable": 2, 2024-09-21T15:49:16.034 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity_weighted": 1, 2024-09-21T15:49:16.034 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity": 1, 2024-09-21T15:49:16.034 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity_weighted": 1 2024-09-21T15:49:16.034 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:16.034 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.034 INFO:teuthology.orchestra.run.smithi089.stdout: "pool": 2, 2024-09-21T15:49:16.034 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_name": "rbd", 2024-09-21T15:49:16.035 INFO:teuthology.orchestra.run.smithi089.stdout: "create_time": "2024-09-21T15:48:55.812982+0000", 2024-09-21T15:49:16.035 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 8193, 2024-09-21T15:49:16.035 INFO:teuthology.orchestra.run.smithi089.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T15:49:16.035 INFO:teuthology.orchestra.run.smithi089.stdout: "type": 1, 2024-09-21T15:49:16.035 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 2, 2024-09-21T15:49:16.035 INFO:teuthology.orchestra.run.smithi089.stdout: "min_size": 1, 2024-09-21T15:49:16.035 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_rule": 0, 2024-09-21T15:49:16.035 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_count": 0, 2024-09-21T15:49:16.035 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_target": 0, 2024-09-21T15:49:16.035 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T15:49:16.035 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T15:49:16.035 INFO:teuthology.orchestra.run.smithi089.stdout: "object_hash": 2, 2024-09-21T15:49:16.035 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_autoscale_mode": "off", 2024-09-21T15:49:16.036 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num": 8, 2024-09-21T15:49:16.036 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num": 8, 2024-09-21T15:49:16.036 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num_target": 8, 2024-09-21T15:49:16.036 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_target": 8, 2024-09-21T15:49:16.036 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_pending": 8, 2024-09-21T15:49:16.036 INFO:teuthology.orchestra.run.smithi089.stdout: "last_pg_merge_meta": { 2024-09-21T15:49:16.036 INFO:teuthology.orchestra.run.smithi089.stdout: "source_pgid": "0.0", 2024-09-21T15:49:16.036 INFO:teuthology.orchestra.run.smithi089.stdout: "ready_epoch": 0, 2024-09-21T15:49:16.036 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_started": 0, 2024-09-21T15:49:16.036 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_clean": 0, 2024-09-21T15:49:16.036 INFO:teuthology.orchestra.run.smithi089.stdout: "source_version": "0'0", 2024-09-21T15:49:16.036 INFO:teuthology.orchestra.run.smithi089.stdout: "target_version": "0'0" 2024-09-21T15:49:16.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.036 INFO:teuthology.orchestra.run.smithi089.stdout: "last_change": "20", 2024-09-21T15:49:16.037 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend": "0", 2024-09-21T15:49:16.037 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T15:49:16.037 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T15:49:16.037 INFO:teuthology.orchestra.run.smithi089.stdout: "auid": 0, 2024-09-21T15:49:16.037 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_mode": "selfmanaged", 2024-09-21T15:49:16.037 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_seq": 3, 2024-09-21T15:49:16.037 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_epoch": 20, 2024-09-21T15:49:16.037 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_snaps": [], 2024-09-21T15:49:16.037 INFO:teuthology.orchestra.run.smithi089.stdout: "removed_snaps": "[]", 2024-09-21T15:49:16.037 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_bytes": 0, 2024-09-21T15:49:16.037 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_objects": 0, 2024-09-21T15:49:16.037 INFO:teuthology.orchestra.run.smithi089.stdout: "tiers": [], 2024-09-21T15:49:16.037 INFO:teuthology.orchestra.run.smithi089.stdout: "tier_of": -1, 2024-09-21T15:49:16.038 INFO:teuthology.orchestra.run.smithi089.stdout: "read_tier": -1, 2024-09-21T15:49:16.038 INFO:teuthology.orchestra.run.smithi089.stdout: "write_tier": -1, 2024-09-21T15:49:16.038 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_mode": "none", 2024-09-21T15:49:16.038 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_bytes": 0, 2024-09-21T15:49:16.038 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_objects": 0, 2024-09-21T15:49:16.038 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T15:49:16.038 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T15:49:16.038 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T15:49:16.038 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_flush_age": 0, 2024-09-21T15:49:16.038 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_evict_age": 0, 2024-09-21T15:49:16.038 INFO:teuthology.orchestra.run.smithi089.stdout: "erasure_code_profile": "", 2024-09-21T15:49:16.038 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_params": { 2024-09-21T15:49:16.038 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "none" 2024-09-21T15:49:16.039 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.039 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_period": 0, 2024-09-21T15:49:16.039 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_count": 0, 2024-09-21T15:49:16.039 INFO:teuthology.orchestra.run.smithi089.stdout: "use_gmt_hitset": true, 2024-09-21T15:49:16.039 INFO:teuthology.orchestra.run.smithi089.stdout: "min_read_recency_for_promote": 0, 2024-09-21T15:49:16.039 INFO:teuthology.orchestra.run.smithi089.stdout: "min_write_recency_for_promote": 0, 2024-09-21T15:49:16.039 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T15:49:16.039 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_search_last_n": 0, 2024-09-21T15:49:16.039 INFO:teuthology.orchestra.run.smithi089.stdout: "grade_table": [], 2024-09-21T15:49:16.039 INFO:teuthology.orchestra.run.smithi089.stdout: "stripe_width": 0, 2024-09-21T15:49:16.039 INFO:teuthology.orchestra.run.smithi089.stdout: "expected_num_objects": 0, 2024-09-21T15:49:16.039 INFO:teuthology.orchestra.run.smithi089.stdout: "fast_read": false, 2024-09-21T15:49:16.039 INFO:teuthology.orchestra.run.smithi089.stdout: "options": {}, 2024-09-21T15:49:16.039 INFO:teuthology.orchestra.run.smithi089.stdout: "application_metadata": { 2024-09-21T15:49:16.040 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd": {} 2024-09-21T15:49:16.040 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.040 INFO:teuthology.orchestra.run.smithi089.stdout: "read_balance": { 2024-09-21T15:49:16.040 INFO:teuthology.orchestra.run.smithi089.stdout: "score_acting": 1.9900000095367432, 2024-09-21T15:49:16.040 INFO:teuthology.orchestra.run.smithi089.stdout: "score_stable": 1.9900000095367432, 2024-09-21T15:49:16.040 INFO:teuthology.orchestra.run.smithi089.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T15:49:16.040 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_acting": 1.75, 2024-09-21T15:49:16.040 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_stable": 1.75, 2024-09-21T15:49:16.040 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity_weighted": 1, 2024-09-21T15:49:16.040 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity": 1, 2024-09-21T15:49:16.040 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity_weighted": 1 2024-09-21T15:49:16.040 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:16.040 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.040 INFO:teuthology.orchestra.run.smithi089.stdout: "pool": 3, 2024-09-21T15:49:16.041 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_name": "pool_client.0", 2024-09-21T15:49:16.041 INFO:teuthology.orchestra.run.smithi089.stdout: "create_time": "2024-09-21T15:49:05.523888+0000", 2024-09-21T15:49:16.041 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 8193, 2024-09-21T15:49:16.041 INFO:teuthology.orchestra.run.smithi089.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T15:49:16.041 INFO:teuthology.orchestra.run.smithi089.stdout: "type": 1, 2024-09-21T15:49:16.041 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 2, 2024-09-21T15:49:16.041 INFO:teuthology.orchestra.run.smithi089.stdout: "min_size": 1, 2024-09-21T15:49:16.041 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_rule": 0, 2024-09-21T15:49:16.041 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_count": 0, 2024-09-21T15:49:16.041 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_target": 0, 2024-09-21T15:49:16.041 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T15:49:16.041 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T15:49:16.041 INFO:teuthology.orchestra.run.smithi089.stdout: "object_hash": 2, 2024-09-21T15:49:16.041 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_autoscale_mode": "off", 2024-09-21T15:49:16.041 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num": 32, 2024-09-21T15:49:16.042 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num": 32, 2024-09-21T15:49:16.042 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num_target": 32, 2024-09-21T15:49:16.042 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_target": 32, 2024-09-21T15:49:16.042 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_pending": 32, 2024-09-21T15:49:16.042 INFO:teuthology.orchestra.run.smithi089.stdout: "last_pg_merge_meta": { 2024-09-21T15:49:16.042 INFO:teuthology.orchestra.run.smithi089.stdout: "source_pgid": "0.0", 2024-09-21T15:49:16.042 INFO:teuthology.orchestra.run.smithi089.stdout: "ready_epoch": 0, 2024-09-21T15:49:16.042 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_started": 0, 2024-09-21T15:49:16.042 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_clean": 0, 2024-09-21T15:49:16.042 INFO:teuthology.orchestra.run.smithi089.stdout: "source_version": "0'0", 2024-09-21T15:49:16.042 INFO:teuthology.orchestra.run.smithi089.stdout: "target_version": "0'0" 2024-09-21T15:49:16.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.042 INFO:teuthology.orchestra.run.smithi089.stdout: "last_change": "29", 2024-09-21T15:49:16.042 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend": "0", 2024-09-21T15:49:16.042 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T15:49:16.043 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T15:49:16.043 INFO:teuthology.orchestra.run.smithi089.stdout: "auid": 0, 2024-09-21T15:49:16.043 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_mode": "selfmanaged", 2024-09-21T15:49:16.043 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_seq": 5, 2024-09-21T15:49:16.043 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_epoch": 29, 2024-09-21T15:49:16.043 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_snaps": [], 2024-09-21T15:49:16.043 INFO:teuthology.orchestra.run.smithi089.stdout: "removed_snaps": "[]", 2024-09-21T15:49:16.043 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_bytes": 0, 2024-09-21T15:49:16.043 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_objects": 0, 2024-09-21T15:49:16.043 INFO:teuthology.orchestra.run.smithi089.stdout: "tiers": [], 2024-09-21T15:49:16.043 INFO:teuthology.orchestra.run.smithi089.stdout: "tier_of": -1, 2024-09-21T15:49:16.043 INFO:teuthology.orchestra.run.smithi089.stdout: "read_tier": -1, 2024-09-21T15:49:16.044 INFO:teuthology.orchestra.run.smithi089.stdout: "write_tier": -1, 2024-09-21T15:49:16.044 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_mode": "none", 2024-09-21T15:49:16.044 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_bytes": 0, 2024-09-21T15:49:16.044 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_objects": 0, 2024-09-21T15:49:16.044 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T15:49:16.044 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T15:49:16.044 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T15:49:16.044 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_flush_age": 0, 2024-09-21T15:49:16.044 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_evict_age": 0, 2024-09-21T15:49:16.044 INFO:teuthology.orchestra.run.smithi089.stdout: "erasure_code_profile": "", 2024-09-21T15:49:16.044 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_params": { 2024-09-21T15:49:16.044 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "none" 2024-09-21T15:49:16.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.044 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_period": 0, 2024-09-21T15:49:16.044 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_count": 0, 2024-09-21T15:49:16.045 INFO:teuthology.orchestra.run.smithi089.stdout: "use_gmt_hitset": true, 2024-09-21T15:49:16.045 INFO:teuthology.orchestra.run.smithi089.stdout: "min_read_recency_for_promote": 0, 2024-09-21T15:49:16.045 INFO:teuthology.orchestra.run.smithi089.stdout: "min_write_recency_for_promote": 0, 2024-09-21T15:49:16.045 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T15:49:16.045 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_search_last_n": 0, 2024-09-21T15:49:16.045 INFO:teuthology.orchestra.run.smithi089.stdout: "grade_table": [], 2024-09-21T15:49:16.045 INFO:teuthology.orchestra.run.smithi089.stdout: "stripe_width": 0, 2024-09-21T15:49:16.045 INFO:teuthology.orchestra.run.smithi089.stdout: "expected_num_objects": 0, 2024-09-21T15:49:16.045 INFO:teuthology.orchestra.run.smithi089.stdout: "fast_read": false, 2024-09-21T15:49:16.045 INFO:teuthology.orchestra.run.smithi089.stdout: "options": {}, 2024-09-21T15:49:16.045 INFO:teuthology.orchestra.run.smithi089.stdout: "application_metadata": { 2024-09-21T15:49:16.045 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd": {} 2024-09-21T15:49:16.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.045 INFO:teuthology.orchestra.run.smithi089.stdout: "read_balance": { 2024-09-21T15:49:16.046 INFO:teuthology.orchestra.run.smithi089.stdout: "score_acting": 1.5, 2024-09-21T15:49:16.046 INFO:teuthology.orchestra.run.smithi089.stdout: "score_stable": 1.5, 2024-09-21T15:49:16.046 INFO:teuthology.orchestra.run.smithi089.stdout: "optimal_score": 1, 2024-09-21T15:49:16.046 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_acting": 1.5, 2024-09-21T15:49:16.046 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_stable": 1.5, 2024-09-21T15:49:16.046 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity_weighted": 1, 2024-09-21T15:49:16.046 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity": 1, 2024-09-21T15:49:16.046 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity_weighted": 1 2024-09-21T15:49:16.046 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:16.046 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:16.046 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-21T15:49:16.046 INFO:teuthology.orchestra.run.smithi089.stdout: "osds": [ 2024-09-21T15:49:16.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.046 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 0, 2024-09-21T15:49:16.046 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "7b3fabfa-3fe6-47c7-9d0c-a6036f738771", 2024-09-21T15:49:16.047 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-09-21T15:49:16.047 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-09-21T15:49:16.047 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-09-21T15:49:16.047 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-09-21T15:49:16.047 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-09-21T15:49:16.047 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-09-21T15:49:16.047 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-09-21T15:49:16.047 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 22, 2024-09-21T15:49:16.047 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-09-21T15:49:16.047 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-09-21T15:49:16.047 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-09-21T15:49:16.047 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:49:16.047 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.048 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:49:16.048 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6800", 2024-09-21T15:49:16.048 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3360073633 2024-09-21T15:49:16.048 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.048 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.048 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:49:16.048 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6801", 2024-09-21T15:49:16.048 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3360073633 2024-09-21T15:49:16.048 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:16.048 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:16.048 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.048 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-09-21T15:49:16.048 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:49:16.048 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.048 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:49:16.049 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6803", 2024-09-21T15:49:16.049 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3360073633 2024-09-21T15:49:16.049 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.049 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.049 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:49:16.049 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6805", 2024-09-21T15:49:16.049 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3360073633 2024-09-21T15:49:16.049 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:16.049 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:16.049 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.049 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-09-21T15:49:16.049 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:49:16.049 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.050 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:49:16.050 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6811", 2024-09-21T15:49:16.050 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3360073633 2024-09-21T15:49:16.050 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.050 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.050 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:49:16.050 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6813", 2024-09-21T15:49:16.050 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3360073633 2024-09-21T15:49:16.050 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:16.050 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:16.050 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.050 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-09-21T15:49:16.051 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:49:16.051 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.051 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:49:16.051 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6807", 2024-09-21T15:49:16.051 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3360073633 2024-09-21T15:49:16.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.051 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.051 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:49:16.051 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6809", 2024-09-21T15:49:16.051 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3360073633 2024-09-21T15:49:16.051 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:16.051 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:16.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.052 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.89:6801/3360073633", 2024-09-21T15:49:16.052 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.89:6805/3360073633", 2024-09-21T15:49:16.052 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.89:6813/3360073633", 2024-09-21T15:49:16.052 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.89:6809/3360073633", 2024-09-21T15:49:16.052 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-09-21T15:49:16.052 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-09-21T15:49:16.052 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-09-21T15:49:16.052 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:16.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.052 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.052 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 1, 2024-09-21T15:49:16.052 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "767ad1bc-84ef-4102-83f5-aacda0917ddb", 2024-09-21T15:49:16.052 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-09-21T15:49:16.053 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-09-21T15:49:16.053 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-09-21T15:49:16.053 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-09-21T15:49:16.053 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-09-21T15:49:16.053 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-09-21T15:49:16.053 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-09-21T15:49:16.053 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 22, 2024-09-21T15:49:16.053 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-09-21T15:49:16.053 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-09-21T15:49:16.053 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-09-21T15:49:16.053 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:49:16.053 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.053 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:49:16.053 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6816", 2024-09-21T15:49:16.053 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3187095367 2024-09-21T15:49:16.054 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.054 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.054 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:49:16.054 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6817", 2024-09-21T15:49:16.054 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3187095367 2024-09-21T15:49:16.054 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:16.054 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:16.054 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.054 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-09-21T15:49:16.054 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:49:16.054 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.054 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:49:16.054 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6818", 2024-09-21T15:49:16.054 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3187095367 2024-09-21T15:49:16.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.055 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.055 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:49:16.055 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6819", 2024-09-21T15:49:16.055 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3187095367 2024-09-21T15:49:16.055 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:16.055 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:16.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.055 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-09-21T15:49:16.055 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:49:16.055 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.055 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:49:16.055 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6822", 2024-09-21T15:49:16.055 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3187095367 2024-09-21T15:49:16.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.056 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.056 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:49:16.056 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6824", 2024-09-21T15:49:16.056 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3187095367 2024-09-21T15:49:16.056 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:16.056 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:16.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.056 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-09-21T15:49:16.056 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:49:16.056 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.056 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:49:16.056 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6820", 2024-09-21T15:49:16.056 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3187095367 2024-09-21T15:49:16.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.056 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.057 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:49:16.057 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6821", 2024-09-21T15:49:16.057 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3187095367 2024-09-21T15:49:16.057 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:16.057 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:16.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.057 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.89:6817/3187095367", 2024-09-21T15:49:16.057 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.89:6819/3187095367", 2024-09-21T15:49:16.057 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.89:6824/3187095367", 2024-09-21T15:49:16.057 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.89:6821/3187095367", 2024-09-21T15:49:16.057 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-09-21T15:49:16.057 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-09-21T15:49:16.057 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-09-21T15:49:16.057 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:16.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.058 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 2, 2024-09-21T15:49:16.058 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "1dbd0d50-aae9-42a7-90a1-aa92697778ae", 2024-09-21T15:49:16.058 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-09-21T15:49:16.058 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-09-21T15:49:16.058 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-09-21T15:49:16.058 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-09-21T15:49:16.058 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-09-21T15:49:16.058 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-09-21T15:49:16.058 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-09-21T15:49:16.058 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 17, 2024-09-21T15:49:16.058 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-09-21T15:49:16.058 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-09-21T15:49:16.059 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-09-21T15:49:16.059 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:49:16.059 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.059 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:49:16.059 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6802", 2024-09-21T15:49:16.059 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 4098625737 2024-09-21T15:49:16.059 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.059 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.059 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:49:16.059 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6804", 2024-09-21T15:49:16.059 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 4098625737 2024-09-21T15:49:16.059 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:16.059 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:16.060 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.060 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-09-21T15:49:16.060 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:49:16.060 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.060 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:49:16.060 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6806", 2024-09-21T15:49:16.060 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 4098625737 2024-09-21T15:49:16.060 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.060 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.060 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:49:16.060 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6808", 2024-09-21T15:49:16.060 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 4098625737 2024-09-21T15:49:16.060 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:16.060 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:16.060 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.061 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-09-21T15:49:16.061 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:49:16.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.061 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:49:16.061 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6814", 2024-09-21T15:49:16.061 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 4098625737 2024-09-21T15:49:16.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.061 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:49:16.061 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6815", 2024-09-21T15:49:16.061 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 4098625737 2024-09-21T15:49:16.061 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:16.061 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:16.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.062 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-09-21T15:49:16.062 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:49:16.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.062 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:49:16.062 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6810", 2024-09-21T15:49:16.062 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 4098625737 2024-09-21T15:49:16.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.062 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:49:16.062 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6812", 2024-09-21T15:49:16.062 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 4098625737 2024-09-21T15:49:16.062 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:16.062 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:16.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.063 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.89:6804/4098625737", 2024-09-21T15:49:16.063 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.89:6808/4098625737", 2024-09-21T15:49:16.063 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.89:6815/4098625737", 2024-09-21T15:49:16.063 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.89:6812/4098625737", 2024-09-21T15:49:16.063 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-09-21T15:49:16.063 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-09-21T15:49:16.063 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-09-21T15:49:16.063 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:16.063 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.063 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.063 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 3, 2024-09-21T15:49:16.063 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "f37cd295-2bac-4df4-a99b-abedc0978dce", 2024-09-21T15:49:16.063 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-09-21T15:49:16.064 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-09-21T15:49:16.064 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-09-21T15:49:16.064 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-09-21T15:49:16.064 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-09-21T15:49:16.064 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-09-21T15:49:16.064 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-09-21T15:49:16.064 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 22, 2024-09-21T15:49:16.064 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-09-21T15:49:16.064 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-09-21T15:49:16.064 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-09-21T15:49:16.064 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:49:16.064 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.064 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:49:16.064 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6823", 2024-09-21T15:49:16.065 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 1886828410 2024-09-21T15:49:16.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.065 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:49:16.065 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6825", 2024-09-21T15:49:16.065 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 1886828410 2024-09-21T15:49:16.065 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:16.065 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:16.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.065 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-09-21T15:49:16.065 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:49:16.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.065 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:49:16.065 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6826", 2024-09-21T15:49:16.065 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 1886828410 2024-09-21T15:49:16.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.066 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.066 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:49:16.066 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6827", 2024-09-21T15:49:16.066 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 1886828410 2024-09-21T15:49:16.066 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:16.066 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:16.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.066 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-09-21T15:49:16.066 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:49:16.066 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.066 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:49:16.066 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6830", 2024-09-21T15:49:16.066 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 1886828410 2024-09-21T15:49:16.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.067 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:49:16.067 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6831", 2024-09-21T15:49:16.067 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 1886828410 2024-09-21T15:49:16.067 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:16.067 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:16.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.067 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-09-21T15:49:16.067 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:49:16.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.067 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:49:16.067 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6828", 2024-09-21T15:49:16.067 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 1886828410 2024-09-21T15:49:16.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.068 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.068 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:49:16.068 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6829", 2024-09-21T15:49:16.068 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 1886828410 2024-09-21T15:49:16.068 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:16.068 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:16.068 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.068 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.89:6825/1886828410", 2024-09-21T15:49:16.068 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.89:6827/1886828410", 2024-09-21T15:49:16.068 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.89:6831/1886828410", 2024-09-21T15:49:16.068 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.89:6829/1886828410", 2024-09-21T15:49:16.068 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-09-21T15:49:16.068 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-09-21T15:49:16.068 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-09-21T15:49:16.069 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:16.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.069 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 4, 2024-09-21T15:49:16.069 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "2eeb351d-d51c-4fc3-884e-debecd995aea", 2024-09-21T15:49:16.069 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-09-21T15:49:16.069 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-09-21T15:49:16.069 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-09-21T15:49:16.069 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-09-21T15:49:16.069 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-09-21T15:49:16.069 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-09-21T15:49:16.069 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-09-21T15:49:16.069 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 22, 2024-09-21T15:49:16.069 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-09-21T15:49:16.069 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-09-21T15:49:16.070 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-09-21T15:49:16.070 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:49:16.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.070 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:49:16.070 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6817", 2024-09-21T15:49:16.070 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 22722842 2024-09-21T15:49:16.070 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.070 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:49:16.070 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6819", 2024-09-21T15:49:16.070 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 22722842 2024-09-21T15:49:16.070 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:16.070 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:16.070 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.071 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-09-21T15:49:16.071 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:49:16.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.071 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:49:16.071 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6820", 2024-09-21T15:49:16.071 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 22722842 2024-09-21T15:49:16.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.071 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:49:16.071 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6822", 2024-09-21T15:49:16.071 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 22722842 2024-09-21T15:49:16.071 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:16.071 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:16.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.071 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-09-21T15:49:16.072 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:49:16.072 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.072 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:49:16.072 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6828", 2024-09-21T15:49:16.072 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 22722842 2024-09-21T15:49:16.072 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.072 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.072 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:49:16.072 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6830", 2024-09-21T15:49:16.072 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 22722842 2024-09-21T15:49:16.072 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:16.072 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:16.072 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.073 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-09-21T15:49:16.073 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:49:16.073 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.073 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:49:16.073 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6824", 2024-09-21T15:49:16.073 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 22722842 2024-09-21T15:49:16.073 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.073 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.073 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:49:16.073 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6826", 2024-09-21T15:49:16.073 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 22722842 2024-09-21T15:49:16.073 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:16.073 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:16.074 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.074 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.100:6819/22722842", 2024-09-21T15:49:16.074 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.100:6822/22722842", 2024-09-21T15:49:16.074 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.100:6830/22722842", 2024-09-21T15:49:16.074 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.100:6826/22722842", 2024-09-21T15:49:16.074 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-09-21T15:49:16.074 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-09-21T15:49:16.074 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-09-21T15:49:16.074 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:16.074 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.074 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.074 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 5, 2024-09-21T15:49:16.074 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "b5396267-6823-42f1-bd04-bd8f144fa76c", 2024-09-21T15:49:16.075 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-09-21T15:49:16.075 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-09-21T15:49:16.075 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-09-21T15:49:16.075 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-09-21T15:49:16.075 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-09-21T15:49:16.075 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-09-21T15:49:16.075 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-09-21T15:49:16.075 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 22, 2024-09-21T15:49:16.075 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-09-21T15:49:16.075 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-09-21T15:49:16.075 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-09-21T15:49:16.075 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:49:16.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.075 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:49:16.076 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6816", 2024-09-21T15:49:16.076 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 363733545 2024-09-21T15:49:16.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.076 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:49:16.076 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6818", 2024-09-21T15:49:16.076 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 363733545 2024-09-21T15:49:16.076 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:16.076 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:16.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.076 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-09-21T15:49:16.076 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:49:16.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.076 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:49:16.077 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6821", 2024-09-21T15:49:16.077 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 363733545 2024-09-21T15:49:16.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.077 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.077 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:49:16.077 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6823", 2024-09-21T15:49:16.077 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 363733545 2024-09-21T15:49:16.077 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:16.077 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:16.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.077 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-09-21T15:49:16.077 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:49:16.077 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.077 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:49:16.077 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6829", 2024-09-21T15:49:16.077 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 363733545 2024-09-21T15:49:16.078 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.078 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:49:16.078 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6831", 2024-09-21T15:49:16.078 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 363733545 2024-09-21T15:49:16.078 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:16.078 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:16.078 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.078 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-09-21T15:49:16.078 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:49:16.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.078 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:49:16.078 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6825", 2024-09-21T15:49:16.079 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 363733545 2024-09-21T15:49:16.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.079 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.079 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:49:16.079 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6827", 2024-09-21T15:49:16.079 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 363733545 2024-09-21T15:49:16.079 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:16.079 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:16.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.079 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.100:6818/363733545", 2024-09-21T15:49:16.079 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.100:6823/363733545", 2024-09-21T15:49:16.079 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.100:6831/363733545", 2024-09-21T15:49:16.079 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.100:6827/363733545", 2024-09-21T15:49:16.079 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-09-21T15:49:16.080 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-09-21T15:49:16.080 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-09-21T15:49:16.080 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:16.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.080 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 6, 2024-09-21T15:49:16.080 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "955b67e9-73bc-42c1-88e0-3356c105e1ea", 2024-09-21T15:49:16.080 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-09-21T15:49:16.080 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-09-21T15:49:16.080 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-09-21T15:49:16.080 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-09-21T15:49:16.080 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-09-21T15:49:16.080 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-09-21T15:49:16.080 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-09-21T15:49:16.080 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 22, 2024-09-21T15:49:16.080 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-09-21T15:49:16.081 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-09-21T15:49:16.081 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-09-21T15:49:16.081 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:49:16.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.081 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:49:16.081 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6808", 2024-09-21T15:49:16.081 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 205694879 2024-09-21T15:49:16.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.081 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:49:16.081 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6809", 2024-09-21T15:49:16.081 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 205694879 2024-09-21T15:49:16.081 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:16.081 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:16.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.081 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-09-21T15:49:16.082 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:49:16.082 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.082 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:49:16.082 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6810", 2024-09-21T15:49:16.082 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 205694879 2024-09-21T15:49:16.082 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.082 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.082 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:49:16.082 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6811", 2024-09-21T15:49:16.082 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 205694879 2024-09-21T15:49:16.082 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:16.082 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:16.082 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.082 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-09-21T15:49:16.082 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:49:16.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.083 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:49:16.083 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6814", 2024-09-21T15:49:16.083 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 205694879 2024-09-21T15:49:16.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.083 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:49:16.083 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6815", 2024-09-21T15:49:16.083 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 205694879 2024-09-21T15:49:16.083 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:16.083 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:16.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.083 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-09-21T15:49:16.083 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:49:16.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.084 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:49:16.084 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6812", 2024-09-21T15:49:16.084 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 205694879 2024-09-21T15:49:16.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.084 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:49:16.084 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6813", 2024-09-21T15:49:16.084 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 205694879 2024-09-21T15:49:16.084 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:16.084 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:16.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.084 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.100:6809/205694879", 2024-09-21T15:49:16.084 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.100:6811/205694879", 2024-09-21T15:49:16.084 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.100:6815/205694879", 2024-09-21T15:49:16.085 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.100:6813/205694879", 2024-09-21T15:49:16.085 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-09-21T15:49:16.085 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-09-21T15:49:16.085 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-09-21T15:49:16.085 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:16.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.085 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.085 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 7, 2024-09-21T15:49:16.085 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "414ea7f0-138d-404e-bce6-fe8dedcdc938", 2024-09-21T15:49:16.085 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-09-21T15:49:16.085 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-09-21T15:49:16.085 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-09-21T15:49:16.085 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-09-21T15:49:16.085 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-09-21T15:49:16.086 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-09-21T15:49:16.086 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-09-21T15:49:16.086 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 22, 2024-09-21T15:49:16.086 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-09-21T15:49:16.086 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-09-21T15:49:16.086 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-09-21T15:49:16.086 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:49:16.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.086 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:49:16.086 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6800", 2024-09-21T15:49:16.086 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3120011840 2024-09-21T15:49:16.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.086 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:49:16.087 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6801", 2024-09-21T15:49:16.087 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3120011840 2024-09-21T15:49:16.087 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:16.087 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:16.087 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.087 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-09-21T15:49:16.087 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:49:16.087 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.087 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:49:16.087 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6802", 2024-09-21T15:49:16.087 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3120011840 2024-09-21T15:49:16.087 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.087 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.087 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:49:16.088 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6803", 2024-09-21T15:49:16.088 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3120011840 2024-09-21T15:49:16.088 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:16.088 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:16.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.088 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-09-21T15:49:16.088 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:49:16.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.088 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:49:16.088 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6806", 2024-09-21T15:49:16.088 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3120011840 2024-09-21T15:49:16.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.088 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:49:16.088 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6807", 2024-09-21T15:49:16.088 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3120011840 2024-09-21T15:49:16.089 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:16.089 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:16.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.089 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-09-21T15:49:16.089 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:49:16.089 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.089 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:49:16.089 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6804", 2024-09-21T15:49:16.089 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3120011840 2024-09-21T15:49:16.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.089 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.089 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:49:16.089 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6805", 2024-09-21T15:49:16.089 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3120011840 2024-09-21T15:49:16.089 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:16.090 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:16.090 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.090 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.100:6801/3120011840", 2024-09-21T15:49:16.090 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.100:6803/3120011840", 2024-09-21T15:49:16.090 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.100:6807/3120011840", 2024-09-21T15:49:16.090 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.100:6805/3120011840", 2024-09-21T15:49:16.090 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-09-21T15:49:16.090 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-09-21T15:49:16.090 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-09-21T15:49:16.090 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:16.090 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:16.090 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-21T15:49:16.090 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_xinfo": [ 2024-09-21T15:49:16.090 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.090 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 0, 2024-09-21T15:49:16.091 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-09-21T15:49:16.091 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-09-21T15:49:16.091 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-09-21T15:49:16.091 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-09-21T15:49:16.091 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-09-21T15:49:16.091 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-09-21T15:48:45.230746+0000", 2024-09-21T15:49:16.091 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-09-21T15:49:16.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.091 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 1, 2024-09-21T15:49:16.091 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-09-21T15:49:16.091 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-09-21T15:49:16.091 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-09-21T15:49:16.091 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-09-21T15:49:16.091 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-09-21T15:49:16.092 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-09-21T15:48:45.303567+0000", 2024-09-21T15:49:16.092 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-09-21T15:49:16.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.092 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 2, 2024-09-21T15:49:16.092 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-09-21T15:49:16.092 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-09-21T15:49:16.092 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-09-21T15:49:16.092 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-09-21T15:49:16.092 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-09-21T15:49:16.092 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-09-21T15:48:45.210941+0000", 2024-09-21T15:49:16.092 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-09-21T15:49:16.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.092 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 3, 2024-09-21T15:49:16.093 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-09-21T15:49:16.093 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-09-21T15:49:16.093 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-09-21T15:49:16.093 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-09-21T15:49:16.093 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-09-21T15:49:16.093 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-09-21T15:48:45.218918+0000", 2024-09-21T15:49:16.093 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-09-21T15:49:16.093 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.093 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.093 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 4, 2024-09-21T15:49:16.093 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-09-21T15:49:16.093 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-09-21T15:49:16.093 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-09-21T15:49:16.093 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-09-21T15:49:16.093 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-09-21T15:49:16.094 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-09-21T15:48:45.301781+0000", 2024-09-21T15:49:16.094 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-09-21T15:49:16.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.094 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.094 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 5, 2024-09-21T15:49:16.094 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-09-21T15:49:16.094 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-09-21T15:49:16.094 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-09-21T15:49:16.094 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-09-21T15:49:16.094 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-09-21T15:49:16.094 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-09-21T15:48:45.248444+0000", 2024-09-21T15:49:16.094 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-09-21T15:49:16.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.094 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.094 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 6, 2024-09-21T15:49:16.095 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-09-21T15:49:16.095 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-09-21T15:49:16.095 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-09-21T15:49:16.095 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-09-21T15:49:16.095 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-09-21T15:49:16.095 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-09-21T15:48:45.275465+0000", 2024-09-21T15:49:16.095 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-09-21T15:49:16.095 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.095 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 7, 2024-09-21T15:49:16.095 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-09-21T15:49:16.095 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-09-21T15:49:16.095 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-09-21T15:49:16.095 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-09-21T15:49:16.096 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-09-21T15:49:16.096 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-09-21T15:48:45.166209+0000", 2024-09-21T15:49:16.096 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-09-21T15:49:16.096 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:16.096 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-21T15:49:16.096 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_upmap": [], 2024-09-21T15:49:16.096 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_upmap_items": [], 2024-09-21T15:49:16.096 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_upmap_primaries": [], 2024-09-21T15:49:16.096 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_temp": [], 2024-09-21T15:49:16.096 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_temp": [], 2024-09-21T15:49:16.096 INFO:teuthology.orchestra.run.smithi089.stdout: "blocklist": {}, 2024-09-21T15:49:16.096 INFO:teuthology.orchestra.run.smithi089.stdout: "range_blocklist": {}, 2024-09-21T15:49:16.096 INFO:teuthology.orchestra.run.smithi089.stdout: "erasure_code_profiles": { 2024-09-21T15:49:16.096 INFO:teuthology.orchestra.run.smithi089.stdout: "default": { 2024-09-21T15:49:16.096 INFO:teuthology.orchestra.run.smithi089.stdout: "crush-failure-domain": "osd", 2024-09-21T15:49:16.097 INFO:teuthology.orchestra.run.smithi089.stdout: "k": "2", 2024-09-21T15:49:16.097 INFO:teuthology.orchestra.run.smithi089.stdout: "m": "1", 2024-09-21T15:49:16.097 INFO:teuthology.orchestra.run.smithi089.stdout: "plugin": "jerasure", 2024-09-21T15:49:16.097 INFO:teuthology.orchestra.run.smithi089.stdout: "technique": "reed_sol_van" 2024-09-21T15:49:16.097 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:16.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:16.097 INFO:teuthology.orchestra.run.smithi089.stdout: "removed_snaps_queue": [ 2024-09-21T15:49:16.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.097 INFO:teuthology.orchestra.run.smithi089.stdout: "pool": 3, 2024-09-21T15:49:16.097 INFO:teuthology.orchestra.run.smithi089.stdout: "snaps": [ 2024-09-21T15:49:16.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:16.097 INFO:teuthology.orchestra.run.smithi089.stdout: "begin": 2, 2024-09-21T15:49:16.097 INFO:teuthology.orchestra.run.smithi089.stdout: "length": 1 2024-09-21T15:49:16.097 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:16.097 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:16.098 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:16.098 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-21T15:49:16.098 INFO:teuthology.orchestra.run.smithi089.stdout: "new_removed_snaps": [], 2024-09-21T15:49:16.098 INFO:teuthology.orchestra.run.smithi089.stdout: "new_purged_snaps": [], 2024-09-21T15:49:16.098 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_node_flags": {}, 2024-09-21T15:49:16.098 INFO:teuthology.orchestra.run.smithi089.stdout: "device_class_flags": {}, 2024-09-21T15:49:16.098 INFO:teuthology.orchestra.run.smithi089.stdout: "stretch_mode": { 2024-09-21T15:49:16.098 INFO:teuthology.orchestra.run.smithi089.stdout: "stretch_mode_enabled": false, 2024-09-21T15:49:16.098 INFO:teuthology.orchestra.run.smithi089.stdout: "stretch_bucket_count": 0, 2024-09-21T15:49:16.098 INFO:teuthology.orchestra.run.smithi089.stdout: "degraded_stretch_mode": 0, 2024-09-21T15:49:16.098 INFO:teuthology.orchestra.run.smithi089.stdout: "recovering_stretch_mode": 0, 2024-09-21T15:49:16.098 INFO:teuthology.orchestra.run.smithi089.stdout: "stretch_mode_bucket": 0 2024-09-21T15:49:16.098 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:16.099 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:16.099 INFO:teuthology.orchestra.run.smithi089.stdout:519 trunc from 0xa07dfa8 to 0x607a84d 2024-09-21T15:49:16.101 INFO:tasks.thrashosds.thrasher:j is {'epoch': 30, 'fsid': '4a9b64ea-3aaf-4a40-8ce8-02163a0f6e12', 'created': '2024-09-21T15:48:37.867501+0000', 'modified': '2024-09-21T15:49:15.715519+0000', 'last_up_change': '2024-09-21T15:48:49.160053+0000', 'last_in_change': '2024-09-21T15:48:40.108789+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 4, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T15:48:49.921079+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-21T15:48:55.812982+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 20, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.9900000095367432, 'score_stable': 1.9900000095367432, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 1.75, 'raw_score_stable': 1.75, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'pool_client.0', 'create_time': '2024-09-21T15:49:05.523888+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '29', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 5, 'snap_epoch': 29, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '7b3fabfa-3fe6-47c7-9d0c-a6036f738771', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 22, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6800', 'nonce': 3360073633}, {'type': 'v1', 'addr': '172.21.15.89:6801', 'nonce': 3360073633}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6803', 'nonce': 3360073633}, {'type': 'v1', 'addr': '172.21.15.89:6805', 'nonce': 3360073633}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6811', 'nonce': 3360073633}, {'type': 'v1', 'addr': '172.21.15.89:6813', 'nonce': 3360073633}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6807', 'nonce': 3360073633}, {'type': 'v1', 'addr': '172.21.15.89:6809', 'nonce': 3360073633}]}, 'public_addr': '172.21.15.89:6801/3360073633', 'cluster_addr': '172.21.15.89:6805/3360073633', 'heartbeat_back_addr': '172.21.15.89:6813/3360073633', 'heartbeat_front_addr': '172.21.15.89:6809/3360073633', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '767ad1bc-84ef-4102-83f5-aacda0917ddb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 22, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6816', 'nonce': 3187095367}, {'type': 'v1', 'addr': '172.21.15.89:6817', 'nonce': 3187095367}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6818', 'nonce': 3187095367}, {'type': 'v1', 'addr': '172.21.15.89:6819', 'nonce': 3187095367}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6822', 'nonce': 3187095367}, {'type': 'v1', 'addr': '172.21.15.89:6824', 'nonce': 3187095367}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6820', 'nonce': 3187095367}, {'type': 'v1', 'addr': '172.21.15.89:6821', 'nonce': 3187095367}]}, 'public_addr': '172.21.15.89:6817/3187095367', 'cluster_addr': '172.21.15.89:6819/3187095367', 'heartbeat_back_addr': '172.21.15.89:6824/3187095367', 'heartbeat_front_addr': '172.21.15.89:6821/3187095367', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '1dbd0d50-aae9-42a7-90a1-aa92697778ae', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 17, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6802', 'nonce': 4098625737}, {'type': 'v1', 'addr': '172.21.15.89:6804', 'nonce': 4098625737}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6806', 'nonce': 4098625737}, {'type': 'v1', 'addr': '172.21.15.89:6808', 'nonce': 4098625737}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6814', 'nonce': 4098625737}, {'type': 'v1', 'addr': '172.21.15.89:6815', 'nonce': 4098625737}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6810', 'nonce': 4098625737}, {'type': 'v1', 'addr': '172.21.15.89:6812', 'nonce': 4098625737}]}, 'public_addr': '172.21.15.89:6804/4098625737', 'cluster_addr': '172.21.15.89:6808/4098625737', 'heartbeat_back_addr': '172.21.15.89:6815/4098625737', 'heartbeat_front_addr': '172.21.15.89:6812/4098625737', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'f37cd295-2bac-4df4-a99b-abedc0978dce', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 22, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6823', 'nonce': 1886828410}, {'type': 'v1', 'addr': '172.21.15.89:6825', 'nonce': 1886828410}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6826', 'nonce': 1886828410}, {'type': 'v1', 'addr': '172.21.15.89:6827', 'nonce': 1886828410}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6830', 'nonce': 1886828410}, {'type': 'v1', 'addr': '172.21.15.89:6831', 'nonce': 1886828410}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6828', 'nonce': 1886828410}, {'type': 'v1', 'addr': '172.21.15.89:6829', 'nonce': 1886828410}]}, 'public_addr': '172.21.15.89:6825/1886828410', 'cluster_addr': '172.21.15.89:6827/1886828410', 'heartbeat_back_addr': '172.21.15.89:6831/1886828410', 'heartbeat_front_addr': '172.21.15.89:6829/1886828410', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '2eeb351d-d51c-4fc3-884e-debecd995aea', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 22, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6817', 'nonce': 22722842}, {'type': 'v1', 'addr': '172.21.15.100:6819', 'nonce': 22722842}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6820', 'nonce': 22722842}, {'type': 'v1', 'addr': '172.21.15.100:6822', 'nonce': 22722842}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6828', 'nonce': 22722842}, {'type': 'v1', 'addr': '172.21.15.100:6830', 'nonce': 22722842}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6824', 'nonce': 22722842}, {'type': 'v1', 'addr': '172.21.15.100:6826', 'nonce': 22722842}]}, 'public_addr': '172.21.15.100:6819/22722842', 'cluster_addr': '172.21.15.100:6822/22722842', 'heartbeat_back_addr': '172.21.15.100:6830/22722842', 'heartbeat_front_addr': '172.21.15.100:6826/22722842', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'b5396267-6823-42f1-bd04-bd8f144fa76c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 22, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6816', 'nonce': 363733545}, {'type': 'v1', 'addr': '172.21.15.100:6818', 'nonce': 363733545}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6821', 'nonce': 363733545}, {'type': 'v1', 'addr': '172.21.15.100:6823', 'nonce': 363733545}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6829', 'nonce': 363733545}, {'type': 'v1', 'addr': '172.21.15.100:6831', 'nonce': 363733545}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6825', 'nonce': 363733545}, {'type': 'v1', 'addr': '172.21.15.100:6827', 'nonce': 363733545}]}, 'public_addr': '172.21.15.100:6818/363733545', 'cluster_addr': '172.21.15.100:6823/363733545', 'heartbeat_back_addr': '172.21.15.100:6831/363733545', 'heartbeat_front_addr': '172.21.15.100:6827/363733545', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '955b67e9-73bc-42c1-88e0-3356c105e1ea', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 22, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6808', 'nonce': 205694879}, {'type': 'v1', 'addr': '172.21.15.100:6809', 'nonce': 205694879}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6810', 'nonce': 205694879}, {'type': 'v1', 'addr': '172.21.15.100:6811', 'nonce': 205694879}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6814', 'nonce': 205694879}, {'type': 'v1', 'addr': '172.21.15.100:6815', 'nonce': 205694879}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6812', 'nonce': 205694879}, {'type': 'v1', 'addr': '172.21.15.100:6813', 'nonce': 205694879}]}, 'public_addr': '172.21.15.100:6809/205694879', 'cluster_addr': '172.21.15.100:6811/205694879', 'heartbeat_back_addr': '172.21.15.100:6815/205694879', 'heartbeat_front_addr': '172.21.15.100:6813/205694879', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '414ea7f0-138d-404e-bce6-fe8dedcdc938', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 22, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6800', 'nonce': 3120011840}, {'type': 'v1', 'addr': '172.21.15.100:6801', 'nonce': 3120011840}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6802', 'nonce': 3120011840}, {'type': 'v1', 'addr': '172.21.15.100:6803', 'nonce': 3120011840}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6806', 'nonce': 3120011840}, {'type': 'v1', 'addr': '172.21.15.100:6807', 'nonce': 3120011840}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6804', 'nonce': 3120011840}, {'type': 'v1', 'addr': '172.21.15.100:6805', 'nonce': 3120011840}]}, 'public_addr': '172.21.15.100:6801/3120011840', 'cluster_addr': '172.21.15.100:6803/3120011840', 'heartbeat_back_addr': '172.21.15.100:6807/3120011840', 'heartbeat_front_addr': '172.21.15.100:6805/3120011840', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:48:45.230746+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:48:45.303567+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:48:45.210941+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:48:45.218918+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:48:45.301781+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:48:45.248444+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:48:45.275465+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:48:45.166209+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 3, 'snaps': [{'begin': 2, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-21T15:49:16.101 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T15:49:16.102 INFO:teuthology.orchestra.run.smithi089.stdout:520 read 0x408ae3 thru 0x40f547 (0x6a65 bytes) 2024-09-21T15:49:16.102 INFO:teuthology.orchestra.run.smithi089.stdout:522 read 0x25fb8be thru 0x2604ca8 (0x93eb bytes) 2024-09-21T15:49:16.102 INFO:teuthology.orchestra.run.smithi089.stdout:523 read 0x4d58dcc thru 0x4d58e09 (0x3e bytes) 2024-09-21T15:49:16.102 INFO:teuthology.orchestra.run.smithi089.stdout:525 punch from 0x402824a to 0x4028d0d, (0xac3 bytes) 2024-09-21T15:49:16.102 INFO:teuthology.orchestra.run.smithi089.stdout:526 write 0x60f16fe thru 0x60fd63c (0xbf3f bytes) 2024-09-21T15:49:16.102 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:16.102 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:16.103 INFO:teuthology.orchestra.run.smithi089.stdout:527 trunc from 0x60fd63d to 0x33c4c9b 2024-09-21T15:49:16.104 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:49:16.104 INFO:teuthology.orchestra.run.smithi089.stdout:530 read 0x1545924 thru 0x1549deb (0x44c8 bytes) 2024-09-21T15:49:16.104 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:16.105 INFO:teuthology.orchestra.run.smithi089.stdout:532 read 0x13d95fe thru 0x13e6de5 (0xd7e8 bytes) 2024-09-21T15:49:16.105 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:16.098+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:16.107 INFO:teuthology.orchestra.run.smithi089.stdout:535 trunc from 0x33c4c9b to 0x5f90700 2024-09-21T15:49:16.126 INFO:teuthology.orchestra.run.smithi089.stdout:536 writesame 0xc8a3785 thru 0xc8ae09f data_size 0x1(0xa91b bytes) 2024-09-21T15:49:16.126 INFO:teuthology.orchestra.run.smithi089.stdout:537 read 0x8fd7e0 thru 0x9014e1 (0x3d02 bytes) 2024-09-21T15:49:16.132 INFO:teuthology.orchestra.run.smithi089.stdout:538 write 0xd090cca thru 0xd09c29f (0xb5d6 bytes) 2024-09-21T15:49:16.149 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:16.149 INFO:teuthology.orchestra.run.smithi089.stdout:541 write 0x5db4548 thru 0x5dbf5a0 (0xb059 bytes) 2024-09-21T15:49:16.171 INFO:teuthology.orchestra.run.smithi089.stdout:542 write 0xe36634b thru 0xe369edb (0x3b91 bytes) 2024-09-21T15:49:16.178 INFO:teuthology.orchestra.run.smithi089.stdout:543 writesame 0x2795299 thru 0x279719f data_size 0x1(0x1f07 bytes) 2024-09-21T15:49:16.184 INFO:teuthology.orchestra.run.smithi089.stdout:545 writesame 0x4af20f2 thru 0x4b0175e data_size 0x37(0xf66d bytes) 2024-09-21T15:49:16.199 INFO:teuthology.orchestra.run.smithi089.stdout:547 read 0x3a59f0b thru 0x3a6991b (0xfa11 bytes) 2024-09-21T15:49:16.199 INFO:teuthology.orchestra.run.smithi089.stdout:548 punch from 0x9d84c70 to 0x9d89422, (0x47b2 bytes) 2024-09-21T15:49:16.199 INFO:teuthology.orchestra.run.smithi089.stdout:549 write 0xe05229c thru 0xe05fbf9 (0xd95e bytes) 2024-09-21T15:49:16.201 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:16.200+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:16.216 INFO:teuthology.orchestra.run.smithi089.stdout:551 read 0x88fee70 thru 0x8902a9c (0x3c2d bytes) 2024-09-21T15:49:16.216 INFO:teuthology.orchestra.run.smithi089.stdout:552 punch from 0x98eb117 to 0x98f7dd6, (0xccbf bytes) 2024-09-21T15:49:16.216 INFO:teuthology.orchestra.run.smithi089.stdout:553 write 0xafed79c thru 0xaffa6a7 (0xcf0c bytes) 2024-09-21T15:49:16.222 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:16.222 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:16.233 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:49:16.234 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:16.234 INFO:teuthology.orchestra.run.smithi089.stdout:555 write 0x86d51e6 thru 0x86dae51 (0x5c6c bytes) 2024-09-21T15:49:16.244 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:16.244 INFO:teuthology.orchestra.run.smithi089.stdout:557 read 0x3d3f79 thru 0x3daea7 (0x6f2f bytes) 2024-09-21T15:49:16.245 INFO:teuthology.orchestra.run.smithi089.stdout:558 read 0x12aac1e thru 0x12af798 (0x4b7b bytes) 2024-09-21T15:49:16.260 INFO:teuthology.orchestra.run.smithi089.stdout:560 trunc from 0xe369edc to 0x15191c1 2024-09-21T15:49:16.302 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:16.301+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:16.326 INFO:teuthology.orchestra.run.smithi089.stdout:561 punch from 0xadbb8d to 0xae5536, (0x99a9 bytes) 2024-09-21T15:49:16.326 INFO:teuthology.orchestra.run.smithi089.stdout:562 writesame 0x288db1 thru 0x28b319 data_size 0x1(0x2569 bytes) 2024-09-21T15:49:16.331 INFO:teuthology.orchestra.run.smithi089.stdout:564 punch from 0x1299816 to 0x12a2ecf, (0x96b9 bytes) 2024-09-21T15:49:16.331 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:16.331 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:16.331 INFO:teuthology.orchestra.run.smithi089.stdout:568 read 0xb1175b thru 0xb1d81c (0xc0c2 bytes) 2024-09-21T15:49:16.338 INFO:teuthology.orchestra.run.smithi089.stdout:570 trunc from 0x15191c1 to 0x62e4364 2024-09-21T15:49:16.344 INFO:teuthology.orchestra.run.smithi089.stdout:571 punch from 0x3b7d642 to 0x3b85d98, (0x8756 bytes) 2024-09-21T15:49:16.344 INFO:teuthology.orchestra.run.smithi089.stdout:572 write 0x54d4a97 thru 0x54d4d6f (0x2d9 bytes) 2024-09-21T15:49:16.359 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:16.359 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:16.361 INFO:teuthology.orchestra.run.smithi089.stdout:574 writesame 0x98ba7f9 thru 0x98c740b data_size 0x1(0xcc13 bytes) 2024-09-21T15:49:16.362 INFO:teuthology.orchestra.run.smithi089.stdout:575 write 0x14247be thru 0x14323eb (0xdc2e bytes) 2024-09-21T15:49:16.369 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:49:16.370 INFO:teuthology.orchestra.run.smithi089.stdout:576 write 0x553d248 thru 0x5547c5e (0xaa17 bytes) 2024-09-21T15:49:16.385 INFO:teuthology.orchestra.run.smithi089.stdout:577 punch from 0xf69879 to 0xf77f04, (0xe68b bytes) 2024-09-21T15:49:16.385 INFO:teuthology.orchestra.run.smithi089.stdout:578 punch from 0xf9de86 to 0xfa32a2, (0x541c bytes) 2024-09-21T15:49:16.385 INFO:teuthology.orchestra.run.smithi089.stdout:579 read 0x1cf82b0 thru 0x1cf84cc (0x21d bytes) 2024-09-21T15:49:16.385 INFO:teuthology.orchestra.run.smithi089.stdout:580 punch from 0x9391bfe to 0x93a02c8, (0xe6ca bytes) 2024-09-21T15:49:16.385 INFO:teuthology.orchestra.run.smithi089.stdout:581 write 0x82b9e2f thru 0x82bc67a (0x284c bytes) 2024-09-21T15:49:16.401 INFO:teuthology.orchestra.run.smithi089.stdout:585 write 0xc9edf5c thru 0xc9f3ff4 (0x6099 bytes) 2024-09-21T15:49:16.402 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:16.401+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:16.410 INFO:teuthology.orchestra.run.smithi089.stdout:586 punch from 0xda7c4d to 0xdac52d, (0x48e0 bytes) 2024-09-21T15:49:16.410 INFO:teuthology.orchestra.run.smithi089.stdout:588 read 0x18d9291 thru 0x18dd51b (0x428b bytes) 2024-09-21T15:49:16.410 INFO:teuthology.orchestra.run.smithi089.stdout:590 read 0x35e6e63 thru 0x35e7111 (0x2af bytes) 2024-09-21T15:49:16.424 INFO:teuthology.orchestra.run.smithi089.stdout:591 trunc from 0xc9f3ff5 to 0xcfd8d5 2024-09-21T15:49:16.426 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:49:16.426 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-09-21T15:49:16.441 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":36,"stamp":"2024-09-21T15:49:15.893728+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":115665835,"num_objects":77,"num_object_clones":10,"num_object_copies":154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77,"num_whiteouts":10,"num_read":8982,"num_read_kb":17044,"num_write":1023,"num_write_kb":8361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":791,"ondisk_log_size":791,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":230680,"kb_used_data":6304,"kb_used_omap":12,"kb_used_meta":224243,"kb_avail":754744040,"statfs":{"total":773094113280,"available":772857896960,"internally_reserved":0,"allocated":6455296,"data_stored":4575996,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12744,"internal_metadata":229625400},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,58],"upper_bound":8},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":115075383,"num_objects":68,"num_object_clones":10,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":10,"num_read":8832,"num_read_kb":16918,"num_write":856,"num_write_kb":5937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.193982"},"pg_stats":[{"pgid":"3.1f","version":"29'7","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-21T15:49:14.399641+0000","last_change":"2024-09-21T15:49:08.703259+0000","last_active":"2024-09-21T15:49:14.399641+0000","last_peered":"2024-09-21T15:49:14.399641+0000","last_clean":"2024-09-21T15:49:14.399641+0000","last_became_active":"2024-09-21T15:49:06.695160+0000","last_became_peered":"2024-09-21T15:49:06.695160+0000","last_unstale":"2024-09-21T15:49:14.399641+0000","last_undegraded":"2024-09-21T15:49:14.399641+0000","last_fullsized":"2024-09-21T15:49:14.399641+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:46:07.447383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089008499999999999,"stat_sum":{"num_bytes":16,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":2,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1e","version":"29'15","reported_seq":1063,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-21T15:49:14.800586+0000","last_change":"2024-09-21T15:49:08.713090+0000","last_active":"2024-09-21T15:49:14.800586+0000","last_peered":"2024-09-21T15:49:14.800586+0000","last_clean":"2024-09-21T15:49:14.800586+0000","last_became_active":"2024-09-21T15:49:06.692876+0000","last_became_peered":"2024-09-21T15:49:06.692876+0000","last_unstale":"2024-09-21T15:49:14.800586+0000","last_undegraded":"2024-09-21T15:49:14.800586+0000","last_fullsized":"2024-09-21T15:49:14.800586+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:28:06.024451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053093400000000005,"stat_sum":{"num_bytes":14551359,"num_objects":4,"num_object_clones":1,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":1042,"num_read_kb":4200,"num_write":16,"num_write_kb":187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1d","version":"29'35","reported_seq":572,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-21T15:49:14.510258+0000","last_change":"2024-09-21T15:49:08.700408+0000","last_active":"2024-09-21T15:49:14.510258+0000","last_peered":"2024-09-21T15:49:14.510258+0000","last_clean":"2024-09-21T15:49:14.510258+0000","last_became_active":"2024-09-21T15:49:06.689441+0000","last_became_peered":"2024-09-21T15:49:06.689441+0000","last_unstale":"2024-09-21T15:49:14.510258+0000","last_undegraded":"2024-09-21T15:49:14.510258+0000","last_fullsized":"2024-09-21T15:49:14.510258+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:16:05.218592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091034799999999995,"stat_sum":{"num_bytes":21254033,"num_objects":6,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":2,"num_read":533,"num_read_kb":2333,"num_write":35,"num_write_kb":670,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1c","version":"29'6","reported_seq":35,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-21T15:49:13.876576+0000","last_change":"2024-09-21T15:49:08.700201+0000","last_active":"2024-09-21T15:49:13.876576+0000","last_peered":"2024-09-21T15:49:13.876576+0000","last_clean":"2024-09-21T15:49:13.876576+0000","last_became_active":"2024-09-21T15:49:06.693732+0000","last_became_peered":"2024-09-21T15:49:06.693732+0000","last_unstale":"2024-09-21T15:49:13.876576+0000","last_undegraded":"2024-09-21T15:49:13.876576+0000","last_fullsized":"2024-09-21T15:49:13.876576+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:22:08.870053+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075225699999999999,"stat_sum":{"num_bytes":1015236,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":86,"num_write":6,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1b","version":"29'12","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-21T15:49:14.606052+0000","last_change":"2024-09-21T15:49:08.703451+0000","last_active":"2024-09-21T15:49:14.606052+0000","last_peered":"2024-09-21T15:49:14.606052+0000","last_clean":"2024-09-21T15:49:14.606052+0000","last_became_active":"2024-09-21T15:49:06.693637+0000","last_became_peered":"2024-09-21T15:49:06.693637+0000","last_unstale":"2024-09-21T15:49:14.606052+0000","last_undegraded":"2024-09-21T15:49:14.606052+0000","last_fullsized":"2024-09-21T15:49:14.606052+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:52:36.525362+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095325300000000004,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":13,"num_write_kb":115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.1a","version":"29'6","reported_seq":117,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-21T15:49:14.605870+0000","last_change":"2024-09-21T15:49:08.696309+0000","last_active":"2024-09-21T15:49:14.605870+0000","last_peered":"2024-09-21T15:49:14.605870+0000","last_clean":"2024-09-21T15:49:14.605870+0000","last_became_active":"2024-09-21T15:49:06.693903+0000","last_became_peered":"2024-09-21T15:49:06.693903+0000","last_unstale":"2024-09-21T15:49:14.605870+0000","last_undegraded":"2024-09-21T15:49:14.605870+0000","last_fullsized":"2024-09-21T15:49:14.605870+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:41:45.432893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00083131000000000004,"stat_sum":{"num_bytes":3100864,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":93,"num_read_kb":131,"num_write":6,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.19","version":"29'17","reported_seq":196,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-21T15:49:14.917347+0000","last_change":"2024-09-21T15:49:08.689456+0000","last_active":"2024-09-21T15:49:14.917347+0000","last_peered":"2024-09-21T15:49:14.917347+0000","last_clean":"2024-09-21T15:49:14.917347+0000","last_became_active":"2024-09-21T15:49:06.691222+0000","last_became_peered":"2024-09-21T15:49:06.691222+0000","last_unstale":"2024-09-21T15:49:14.917347+0000","last_undegraded":"2024-09-21T15:49:14.917347+0000","last_fullsized":"2024-09-21T15:49:14.917347+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:31:29.545866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041415400000000002,"stat_sum":{"num_bytes":10780959,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":172,"num_read_kb":669,"num_write":16,"num_write_kb":200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.18","version":"29'51","reported_seq":354,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-21T15:49:14.396277+0000","last_change":"2024-09-21T15:49:08.712853+0000","last_active":"2024-09-21T15:49:14.396277+0000","last_peered":"2024-09-21T15:49:14.396277+0000","last_clean":"2024-09-21T15:49:14.396277+0000","last_became_active":"2024-09-21T15:49:06.695141+0000","last_became_peered":"2024-09-21T15:49:06.695141+0000","last_unstale":"2024-09-21T15:49:14.396277+0000","last_undegraded":"2024-09-21T15:49:14.396277+0000","last_fullsized":"2024-09-21T15:49:14.396277+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:44:49.577992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000327239,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":808,"num_read_kb":698,"num_write":54,"num_write_kb":49,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.17","version":"29'136","reported_seq":163,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-21T15:49:14.593370+0000","last_change":"2024-09-21T15:49:08.703291+0000","last_active":"2024-09-21T15:49:14.593370+0000","last_peered":"2024-09-21T15:49:14.593370+0000","last_clean":"2024-09-21T15:49:14.593370+0000","last_became_active":"2024-09-21T15:49:06.694854+0000","last_became_peered":"2024-09-21T15:49:06.694854+0000","last_unstale":"2024-09-21T15:49:14.593370+0000","last_undegraded":"2024-09-21T15:49:14.593370+0000","last_fullsized":"2024-09-21T15:49:14.593370+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":136,"log_dups_size":0,"ondisk_log_size":136,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:46:36.039394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00085022299999999999,"stat_sum":{"num_bytes":114766,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":658,"num_read_kb":522,"num_write":224,"num_write_kb":308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.16","version":"26'3","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-21T15:49:13.710363+0000","last_change":"2024-09-21T15:49:08.700488+0000","last_active":"2024-09-21T15:49:13.710363+0000","last_peered":"2024-09-21T15:49:13.710363+0000","last_clean":"2024-09-21T15:49:13.710363+0000","last_became_active":"2024-09-21T15:49:06.688808+0000","last_became_peered":"2024-09-21T15:49:06.688808+0000","last_unstale":"2024-09-21T15:49:13.710363+0000","last_undegraded":"2024-09-21T15:49:13.710363+0000","last_fullsized":"2024-09-21T15:49:13.710363+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:43:02.012254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096357599999999995,"stat_sum":{"num_bytes":1649158,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.15","version":"29'8","reported_seq":28,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-21T15:49:13.877833+0000","last_change":"2024-09-21T15:49:08.707653+0000","last_active":"2024-09-21T15:49:13.877833+0000","last_peered":"2024-09-21T15:49:13.877833+0000","last_clean":"2024-09-21T15:49:13.877833+0000","last_became_active":"2024-09-21T15:49:06.691091+0000","last_became_peered":"2024-09-21T15:49:06.691091+0000","last_unstale":"2024-09-21T15:49:13.877833+0000","last_undegraded":"2024-09-21T15:49:13.877833+0000","last_fullsized":"2024-09-21T15:49:13.877833+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:13:12.258161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097539800000000004,"stat_sum":{"num_bytes":2976055,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":14,"num_write":8,"num_write_kb":116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.14","version":"29'6","reported_seq":25,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-21T15:49:14.608565+0000","last_change":"2024-09-21T15:49:08.696312+0000","last_active":"2024-09-21T15:49:14.608565+0000","last_peered":"2024-09-21T15:49:14.608565+0000","last_clean":"2024-09-21T15:49:14.608565+0000","last_became_active":"2024-09-21T15:49:06.688803+0000","last_became_peered":"2024-09-21T15:49:06.688803+0000","last_unstale":"2024-09-21T15:49:14.608565+0000","last_undegraded":"2024-09-21T15:49:14.608565+0000","last_fullsized":"2024-09-21T15:49:14.608565+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:18:47.662631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080494500000000005,"stat_sum":{"num_bytes":420258,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.7","version":"0'0","reported_seq":25,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-21T15:49:12.713233+0000","last_change":"2024-09-21T15:48:59.666431+0000","last_active":"2024-09-21T15:49:12.713233+0000","last_peered":"2024-09-21T15:49:12.713233+0000","last_clean":"2024-09-21T15:49:12.713233+0000","last_became_active":"2024-09-21T15:48:57.583665+0000","last_became_peered":"2024-09-21T15:48:57.583665+0000","last_unstale":"2024-09-21T15:49:12.713233+0000","last_undegraded":"2024-09-21T15:49:12.713233+0000","last_fullsized":"2024-09-21T15:49:12.713233+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:52:09.228964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031098399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"29'8","reported_seq":26,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-21T15:49:14.143134+0000","last_change":"2024-09-21T15:49:08.702919+0000","last_active":"2024-09-21T15:49:14.143134+0000","last_peered":"2024-09-21T15:49:14.143134+0000","last_clean":"2024-09-21T15:49:14.143134+0000","last_became_active":"2024-09-21T15:49:06.693665+0000","last_became_peered":"2024-09-21T15:49:06.693665+0000","last_unstale":"2024-09-21T15:49:14.143134+0000","last_undegraded":"2024-09-21T15:49:14.143134+0000","last_fullsized":"2024-09-21T15:49:14.143134+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:02:26.134353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000572081,"stat_sum":{"num_bytes":5053552,"num_objects":2,"num_object_clones":1,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":1,"num_read_kb":0,"num_write":6,"num_write_kb":119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":25,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-21T15:49:12.712669+0000","last_change":"2024-09-21T15:48:59.589790+0000","last_active":"2024-09-21T15:49:12.712669+0000","last_peered":"2024-09-21T15:49:12.712669+0000","last_clean":"2024-09-21T15:49:12.712669+0000","last_became_active":"2024-09-21T15:48:57.582121+0000","last_became_peered":"2024-09-21T15:48:57.582121+0000","last_unstale":"2024-09-21T15:49:12.712669+0000","last_undegraded":"2024-09-21T15:49:12.712669+0000","last_fullsized":"2024-09-21T15:49:12.712669+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:33:47.592624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039654700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"29'23","reported_seq":1684,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-21T15:49:14.554868+0000","last_change":"2024-09-21T15:49:08.713537+0000","last_active":"2024-09-21T15:49:14.554868+0000","last_peered":"2024-09-21T15:49:14.554868+0000","last_clean":"2024-09-21T15:49:14.554868+0000","last_became_active":"2024-09-21T15:49:06.690555+0000","last_became_peered":"2024-09-21T15:49:06.690555+0000","last_unstale":"2024-09-21T15:49:14.554868+0000","last_undegraded":"2024-09-21T15:49:14.554868+0000","last_fullsized":"2024-09-21T15:49:14.554868+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:43:15.712243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036370199999999998,"stat_sum":{"num_bytes":793710,"num_objects":5,"num_object_clones":1,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":1647,"num_read_kb":38,"num_write":24,"num_write_kb":756,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":25,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-21T15:49:12.737745+0000","last_change":"2024-09-21T15:49:00.114779+0000","last_active":"2024-09-21T15:49:12.737745+0000","last_peered":"2024-09-21T15:49:12.737745+0000","last_clean":"2024-09-21T15:49:12.737745+0000","last_became_active":"2024-09-21T15:48:57.937107+0000","last_became_peered":"2024-09-21T15:48:57.937107+0000","last_unstale":"2024-09-21T15:49:12.737745+0000","last_undegraded":"2024-09-21T15:49:12.737745+0000","last_fullsized":"2024-09-21T15:49:12.737745+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:38:43.937586+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036074600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-21T15:49:12.712435+0000","last_change":"2024-09-21T15:49:08.689574+0000","last_active":"2024-09-21T15:49:12.712435+0000","last_peered":"2024-09-21T15:49:12.712435+0000","last_clean":"2024-09-21T15:49:12.712435+0000","last_became_active":"2024-09-21T15:49:07.210449+0000","last_became_peered":"2024-09-21T15:49:07.210449+0000","last_unstale":"2024-09-21T15:49:12.712435+0000","last_undegraded":"2024-09-21T15:49:12.712435+0000","last_fullsized":"2024-09-21T15:49:12.712435+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:25:26.429326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000397286,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":25,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-21T15:49:12.712637+0000","last_change":"2024-09-21T15:48:59.589789+0000","last_active":"2024-09-21T15:49:12.712637+0000","last_peered":"2024-09-21T15:49:12.712637+0000","last_clean":"2024-09-21T15:49:12.712637+0000","last_became_active":"2024-09-21T15:48:57.936973+0000","last_became_peered":"2024-09-21T15:48:57.936973+0000","last_unstale":"2024-09-21T15:49:12.712637+0000","last_undegraded":"2024-09-21T15:49:12.712637+0000","last_fullsized":"2024-09-21T15:49:12.712637+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:41:58.458308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040525700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"26'1","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-21T15:49:13.710390+0000","last_change":"2024-09-21T15:49:08.700616+0000","last_active":"2024-09-21T15:49:13.710390+0000","last_peered":"2024-09-21T15:49:13.710390+0000","last_clean":"2024-09-21T15:49:13.710390+0000","last_became_active":"2024-09-21T15:49:06.690840+0000","last_became_peered":"2024-09-21T15:49:06.690840+0000","last_unstale":"2024-09-21T15:49:13.710390+0000","last_undegraded":"2024-09-21T15:49:13.710390+0000","last_fullsized":"2024-09-21T15:49:13.710390+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:58:25.780618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010520690000000001,"stat_sum":{"num_bytes":2949555,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":18,"num_write":1,"num_write_kb":18,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.3","version":"29'7","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-21T15:49:14.478851+0000","last_change":"2024-09-21T15:49:08.695765+0000","last_active":"2024-09-21T15:49:14.478851+0000","last_peered":"2024-09-21T15:49:14.478851+0000","last_clean":"2024-09-21T15:49:14.478851+0000","last_became_active":"2024-09-21T15:49:06.694961+0000","last_became_peered":"2024-09-21T15:49:06.694961+0000","last_unstale":"2024-09-21T15:49:14.478851+0000","last_undegraded":"2024-09-21T15:49:14.478851+0000","last_fullsized":"2024-09-21T15:49:14.478851+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:16:14.714906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00030818600000000001,"stat_sum":{"num_bytes":524288,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":4,"num_write":9,"num_write_kb":515,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"20'2","reported_seq":27,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-21T15:49:12.710152+0000","last_change":"2024-09-21T15:48:59.595513+0000","last_active":"2024-09-21T15:49:12.710152+0000","last_peered":"2024-09-21T15:49:12.710152+0000","last_clean":"2024-09-21T15:49:12.710152+0000","last_became_active":"2024-09-21T15:48:57.585496+0000","last_became_peered":"2024-09-21T15:48:57.585496+0000","last_unstale":"2024-09-21T15:49:12.710152+0000","last_undegraded":"2024-09-21T15:49:12.710152+0000","last_fullsized":"2024-09-21T15:49:12.710152+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:01:35.910466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000859595,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"29'29","reported_seq":54,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-21T15:49:14.350264+0000","last_change":"2024-09-21T15:49:08.689386+0000","last_active":"2024-09-21T15:49:14.350264+0000","last_peered":"2024-09-21T15:49:14.350264+0000","last_clean":"2024-09-21T15:49:14.350264+0000","last_became_active":"2024-09-21T15:49:07.210509+0000","last_became_peered":"2024-09-21T15:49:07.210509+0000","last_unstale":"2024-09-21T15:49:14.350264+0000","last_undegraded":"2024-09-21T15:49:14.350264+0000","last_fullsized":"2024-09-21T15:49:14.350264+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:54:03.900032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035709499999999998,"stat_sum":{"num_bytes":3062911,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":13,"num_read_kb":62,"num_write":28,"num_write_kb":313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":25,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-21T15:49:12.756858+0000","last_change":"2024-09-21T15:48:59.590261+0000","last_active":"2024-09-21T15:49:12.756858+0000","last_peered":"2024-09-21T15:49:12.756858+0000","last_clean":"2024-09-21T15:49:12.756858+0000","last_became_active":"2024-09-21T15:48:57.936108+0000","last_became_peered":"2024-09-21T15:48:57.936108+0000","last_unstale":"2024-09-21T15:49:12.756858+0000","last_undegraded":"2024-09-21T15:49:12.756858+0000","last_fullsized":"2024-09-21T15:49:12.756858+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:06:25.149550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043235099999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"29'39","reported_seq":299,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-21T15:49:13.878077+0000","last_change":"2024-09-21T15:49:08.702715+0000","last_active":"2024-09-21T15:49:13.878077+0000","last_peered":"2024-09-21T15:49:13.878077+0000","last_clean":"2024-09-21T15:49:13.878077+0000","last_became_active":"2024-09-21T15:49:06.693999+0000","last_became_peered":"2024-09-21T15:49:06.693999+0000","last_unstale":"2024-09-21T15:49:13.878077+0000","last_undegraded":"2024-09-21T15:49:13.878077+0000","last_fullsized":"2024-09-21T15:49:13.878077+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:41:49.056153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000414576,"stat_sum":{"num_bytes":59792,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":714,"num_read_kb":682,"num_write":34,"num_write_kb":151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":25,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-21T15:49:12.737495+0000","last_change":"2024-09-21T15:49:00.114730+0000","last_active":"2024-09-21T15:49:12.737495+0000","last_peered":"2024-09-21T15:49:12.737495+0000","last_clean":"2024-09-21T15:49:12.737495+0000","last_became_active":"2024-09-21T15:48:57.585525+0000","last_became_peered":"2024-09-21T15:48:57.585525+0000","last_unstale":"2024-09-21T15:49:12.737495+0000","last_undegraded":"2024-09-21T15:49:12.737495+0000","last_fullsized":"2024-09-21T15:49:12.737495+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:35:35.786526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031691100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"29'14","reported_seq":41,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-21T15:49:14.606219+0000","last_change":"2024-09-21T15:49:08.713296+0000","last_active":"2024-09-21T15:49:14.606219+0000","last_peered":"2024-09-21T15:49:14.606219+0000","last_clean":"2024-09-21T15:49:14.606219+0000","last_became_active":"2024-09-21T15:49:06.692410+0000","last_became_peered":"2024-09-21T15:49:06.692410+0000","last_unstale":"2024-09-21T15:49:14.606219+0000","last_undegraded":"2024-09-21T15:49:14.606219+0000","last_fullsized":"2024-09-21T15:49:14.606219+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:50:28.635836+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00099544400000000006,"stat_sum":{"num_bytes":6160013,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":15,"num_read_kb":180,"num_write":14,"num_write_kb":238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"18'1","reported_seq":26,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-21T15:49:12.710152+0000","last_change":"2024-09-21T15:48:59.595056+0000","last_active":"2024-09-21T15:49:12.710152+0000","last_peered":"2024-09-21T15:49:12.710152+0000","last_clean":"2024-09-21T15:49:12.710152+0000","last_became_active":"2024-09-21T15:48:57.585384+0000","last_became_peered":"2024-09-21T15:48:57.585384+0000","last_unstale":"2024-09-21T15:49:12.710152+0000","last_undegraded":"2024-09-21T15:49:12.710152+0000","last_fullsized":"2024-09-21T15:49:12.710152+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:17:00.878205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039559000000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":173,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-21T15:49:12.737522+0000","last_change":"2024-09-21T15:48:51.381021+0000","last_active":"2024-09-21T15:49:12.737522+0000","last_peered":"2024-09-21T15:49:12.737522+0000","last_clean":"2024-09-21T15:49:12.737522+0000","last_became_active":"2024-09-21T15:48:51.380092+0000","last_became_peered":"2024-09-21T15:48:51.380092+0000","last_unstale":"2024-09-21T15:49:12.737522+0000","last_undegraded":"2024-09-21T15:49:12.737522+0000","last_fullsized":"2024-09-21T15:49:12.737522+0000","mapping_epoch":14,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":15,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:50.161884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:50.161884+0000","last_clean_scrub_stamp":"2024-09-21T15:48:50.161884+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:32:51.001923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"29'9","reported_seq":30,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-21T15:49:14.783238+0000","last_change":"2024-09-21T15:49:08.713442+0000","last_active":"2024-09-21T15:49:14.783238+0000","last_peered":"2024-09-21T15:49:14.783238+0000","last_clean":"2024-09-21T15:49:14.783238+0000","last_became_active":"2024-09-21T15:49:06.690880+0000","last_became_peered":"2024-09-21T15:49:06.690880+0000","last_unstale":"2024-09-21T15:49:14.783238+0000","last_undegraded":"2024-09-21T15:49:14.783238+0000","last_fullsized":"2024-09-21T15:49:14.783238+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:54:30.915174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074364199999999998,"stat_sum":{"num_bytes":3700535,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":5,"num_read_kb":52,"num_write":9,"num_write_kb":31,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.9","version":"29'7","reported_seq":31,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-21T15:49:14.594220+0000","last_change":"2024-09-21T15:49:08.695717+0000","last_active":"2024-09-21T15:49:14.594220+0000","last_peered":"2024-09-21T15:49:14.594220+0000","last_clean":"2024-09-21T15:49:14.594220+0000","last_became_active":"2024-09-21T15:49:07.210506+0000","last_became_peered":"2024-09-21T15:49:07.210506+0000","last_unstale":"2024-09-21T15:49:14.594220+0000","last_undegraded":"2024-09-21T15:49:14.594220+0000","last_fullsized":"2024-09-21T15:49:14.594220+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:55:18.046738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000300663,"stat_sum":{"num_bytes":8466559,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":9,"num_read_kb":92,"num_write":11,"num_write_kb":1085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.a","version":"29'17","reported_seq":36,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-21T15:49:14.260711+0000","last_change":"2024-09-21T15:49:08.690050+0000","last_active":"2024-09-21T15:49:14.260711+0000","last_peered":"2024-09-21T15:49:14.260711+0000","last_clean":"2024-09-21T15:49:14.260711+0000","last_became_active":"2024-09-21T15:49:06.694202+0000","last_became_peered":"2024-09-21T15:49:06.694202+0000","last_unstale":"2024-09-21T15:49:14.260711+0000","last_undegraded":"2024-09-21T15:49:14.260711+0000","last_fullsized":"2024-09-21T15:49:14.260711+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:02:05.237705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00023511300000000001,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":16,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.b","version":"29'4","reported_seq":23,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-21T15:49:14.192591+0000","last_change":"2024-09-21T15:49:08.712531+0000","last_active":"2024-09-21T15:49:14.192591+0000","last_peered":"2024-09-21T15:49:14.192591+0000","last_clean":"2024-09-21T15:49:14.192591+0000","last_became_active":"2024-09-21T15:49:06.695206+0000","last_became_peered":"2024-09-21T15:49:06.695206+0000","last_unstale":"2024-09-21T15:49:14.192591+0000","last_undegraded":"2024-09-21T15:49:14.192591+0000","last_fullsized":"2024-09-21T15:49:14.192591+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:07:28.661007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039616400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.c","version":"29'3","reported_seq":27,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-21T15:49:13.877240+0000","last_change":"2024-09-21T15:49:08.700210+0000","last_active":"2024-09-21T15:49:13.877240+0000","last_peered":"2024-09-21T15:49:13.877240+0000","last_clean":"2024-09-21T15:49:13.877240+0000","last_became_active":"2024-09-21T15:49:06.695623+0000","last_became_peered":"2024-09-21T15:49:06.695623+0000","last_unstale":"2024-09-21T15:49:13.877240+0000","last_undegraded":"2024-09-21T15:49:13.877240+0000","last_fullsized":"2024-09-21T15:49:13.877240+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:26:28.910577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000746081,"stat_sum":{"num_bytes":4596410,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":184,"num_write":3,"num_write_kb":96,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.d","version":"29'3","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-21T15:49:14.192881+0000","last_change":"2024-09-21T15:49:08.707850+0000","last_active":"2024-09-21T15:49:14.192881+0000","last_peered":"2024-09-21T15:49:14.192881+0000","last_clean":"2024-09-21T15:49:14.192881+0000","last_became_active":"2024-09-21T15:49:06.689203+0000","last_became_peered":"2024-09-21T15:49:06.689203+0000","last_unstale":"2024-09-21T15:49:14.192881+0000","last_undegraded":"2024-09-21T15:49:14.192881+0000","last_fullsized":"2024-09-21T15:49:14.192881+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:22:17.578359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00115237,"stat_sum":{"num_bytes":307200,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.e","version":"29'39","reported_seq":1824,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-21T15:49:14.893686+0000","last_change":"2024-09-21T15:49:08.708055+0000","last_active":"2024-09-21T15:49:14.893686+0000","last_peered":"2024-09-21T15:49:14.893686+0000","last_clean":"2024-09-21T15:49:14.893686+0000","last_became_active":"2024-09-21T15:49:06.690101+0000","last_became_peered":"2024-09-21T15:49:06.690101+0000","last_unstale":"2024-09-21T15:49:14.893686+0000","last_undegraded":"2024-09-21T15:49:14.893686+0000","last_fullsized":"2024-09-21T15:49:14.893686+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:43:30.659104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001357219,"stat_sum":{"num_bytes":18626503,"num_objects":5,"num_object_clones":2,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":2,"num_read":1853,"num_read_kb":6364,"num_write":39,"num_write_kb":327,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.f","version":"0'0","reported_seq":19,"reported_epoch":28,"state":"active+clean","last_fresh":"2024-09-21T15:49:12.737663+0000","last_change":"2024-09-21T15:49:08.707770+0000","last_active":"2024-09-21T15:49:12.737663+0000","last_peered":"2024-09-21T15:49:12.737663+0000","last_clean":"2024-09-21T15:49:12.737663+0000","last_became_active":"2024-09-21T15:49:06.694250+0000","last_became_peered":"2024-09-21T15:49:06.694250+0000","last_unstale":"2024-09-21T15:49:12.737663+0000","last_undegraded":"2024-09-21T15:49:12.737663+0000","last_fullsized":"2024-09-21T15:49:12.737663+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:21:37.777769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010867019999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.10","version":"29'2","reported_seq":21,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-21T15:49:14.163027+0000","last_change":"2024-09-21T15:49:08.690051+0000","last_active":"2024-09-21T15:49:14.163027+0000","last_peered":"2024-09-21T15:49:14.163027+0000","last_clean":"2024-09-21T15:49:14.163027+0000","last_became_active":"2024-09-21T15:49:06.691482+0000","last_became_peered":"2024-09-21T15:49:06.691482+0000","last_unstale":"2024-09-21T15:49:14.163027+0000","last_undegraded":"2024-09-21T15:49:14.163027+0000","last_fullsized":"2024-09-21T15:49:14.163027+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:08:42.689645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00023347900000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.11","version":"29'10","reported_seq":540,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-21T15:49:14.606030+0000","last_change":"2024-09-21T15:49:08.707984+0000","last_active":"2024-09-21T15:49:14.606030+0000","last_peered":"2024-09-21T15:49:14.606030+0000","last_clean":"2024-09-21T15:49:14.606030+0000","last_became_active":"2024-09-21T15:49:06.688798+0000","last_became_peered":"2024-09-21T15:49:06.688798+0000","last_unstale":"2024-09-21T15:49:14.606030+0000","last_undegraded":"2024-09-21T15:49:14.606030+0000","last_fullsized":"2024-09-21T15:49:14.606030+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:48:33.679133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013395060000000001,"stat_sum":{"num_bytes":4604380,"num_objects":3,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":515,"num_read_kb":0,"num_write":9,"num_write_kb":44,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.12","version":"29'3","reported_seq":22,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-21T15:49:13.715854+0000","last_change":"2024-09-21T15:49:08.702721+0000","last_active":"2024-09-21T15:49:13.715854+0000","last_peered":"2024-09-21T15:49:13.715854+0000","last_clean":"2024-09-21T15:49:13.715854+0000","last_became_active":"2024-09-21T15:49:06.693555+0000","last_became_peered":"2024-09-21T15:49:06.693555+0000","last_unstale":"2024-09-21T15:49:13.715854+0000","last_undegraded":"2024-09-21T15:49:13.715854+0000","last_fullsized":"2024-09-21T15:49:13.715854+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:56:36.859994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037406800000000002,"stat_sum":{"num_bytes":307200,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"3.13","version":"29'151","reported_seq":178,"reported_epoch":29,"state":"active+clean","last_fresh":"2024-09-21T15:49:14.892268+0000","last_change":"2024-09-21T15:49:08.707572+0000","last_active":"2024-09-21T15:49:14.892268+0000","last_peered":"2024-09-21T15:49:14.892268+0000","last_clean":"2024-09-21T15:49:14.892268+0000","last_became_active":"2024-09-21T15:49:06.690128+0000","last_became_peered":"2024-09-21T15:49:06.690128+0000","last_unstale":"2024-09-21T15:49:14.892268+0000","last_undegraded":"2024-09-21T15:49:14.892268+0000","last_fullsized":"2024-09-21T15:49:14.892268+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":151,"log_dups_size":0,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:11.763959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00095961799999999997,"stat_sum":{"num_bytes":136,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":729,"num_read_kb":611,"num_write":254,"num_write_kb":424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2","length":"1"}]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":115075448,"num_objects":73,"num_object_clones":10,"num_object_copies":146,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":73,"num_whiteouts":10,"num_read":8866,"num_read_kb":16944,"num_write":862,"num_write_kb":5943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":6971392,"data_stored":6644502,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":671,"ondisk_log_size":671,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574856,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32180,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339660,"statfs":{"total":96636764160,"available":96603811840,"internally_reserved":0,"allocated":1093632,"data_stored":866501,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":31848907},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.313}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36699999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40300000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.435}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56999999999999995}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48099999999999998}]}]},{"osd":6,"up_from":13,"seq":55834574859,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27372,"kb_used_data":344,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344468,"statfs":{"total":96636764160,"available":96608735232,"internally_reserved":0,"allocated":352256,"data_stored":155634,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.438}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42899999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39300000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46200000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32300000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574858,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27728,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344112,"statfs":{"total":96636764160,"available":96608370688,"internally_reserved":0,"allocated":729088,"data_stored":466358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33700000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40799999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46800000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42799999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44700000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]}]},{"osd":4,"up_from":13,"seq":55834574857,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27764,"kb_used_data":748,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344076,"statfs":{"total":96636764160,"available":96608333824,"internally_reserved":0,"allocated":765952,"data_stored":495611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46300000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49099999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30599999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35299999999999998}]}]},{"osd":3,"up_from":13,"seq":55834574858,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27848,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94343992,"statfs":{"total":96636764160,"available":96608247808,"internally_reserved":0,"allocated":839680,"data_stored":586108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,58],"upper_bound":8},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.192}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.21099999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28499999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29799999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40300000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]}]},{"osd":2,"up_from":13,"seq":55834574858,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27456,"kb_used_data":428,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344384,"statfs":{"total":96636764160,"available":96608649216,"internally_reserved":0,"allocated":438272,"data_stored":223175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.314}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48899999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44400000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47399999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574857,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27676,"kb_used_data":648,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344164,"statfs":{"total":96636764160,"available":96608423936,"internally_reserved":0,"allocated":663552,"data_stored":422168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.17199999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33100000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.434}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.373}]}]},{"osd":0,"up_from":13,"seq":55834574858,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":32656,"kb_used_data":1536,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94339184,"statfs":{"total":96636764160,"available":96603324416,"internally_reserved":0,"allocated":1572864,"data_stored":1360441,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27100000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39300000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28799999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36499999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.379}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48299999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65000000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1172951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":348160,"data_stored":303432,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":1175552,"data_stored":1149294,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":1069056,"data_stored":1008063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":2027520,"data_stored":1947324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":409600,"data_stored":343526,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":32821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":716800,"data_stored":687091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T15:49:16.444 INFO:tasks.thrashosds.thrasher:Setting 3.8 to [1, 6, 4, 2] 2024-09-21T15:49:16.444 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '3.8', '1', '6', '4', '2'] 2024-09-21T15:49:16.444 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pg-upmap-items 3.8 1 6 4 2 2024-09-21T15:49:16.453 INFO:teuthology.orchestra.run.smithi089.stdout:592 punch from 0x53cb0f to 0x547801, (0xacf2 bytes) 2024-09-21T15:49:16.453 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:16.453 INFO:teuthology.orchestra.run.smithi089.stdout:594 punch from 0x8578d3 to 0x865497, (0xdbc4 bytes) 2024-09-21T15:49:16.453 INFO:teuthology.orchestra.run.smithi089.stdout:595 read 0x42d0b4 thru 0x42d9bc (0x909 bytes) 2024-09-21T15:49:16.453 INFO:teuthology.orchestra.run.smithi089.stdout:596 read 0x318b0b thru 0x31b6ea (0x2be0 bytes) 2024-09-21T15:49:16.454 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:16.458 INFO:teuthology.orchestra.run.smithi089.stdout:600 writesame 0x13793d3 thru 0x137f102 data_size 0x70(0x5d30 bytes) 2024-09-21T15:49:16.470 INFO:teuthology.orchestra.run.smithi089.stdout:601 writesame 0x55c1992 thru 0x55cfedc data_size 0x1(0xe54b bytes) 2024-09-21T15:49:16.470 INFO:teuthology.orchestra.run.smithi089.stdout:603 read 0x4e0fdb9 thru 0x4e1aca3 (0xaeeb bytes) 2024-09-21T15:49:16.470 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:16.470 INFO:teuthology.orchestra.run.smithi089.stdout:605 read 0xa68b69 thru 0xa785c4 (0xfa5c bytes) 2024-09-21T15:49:16.471 INFO:teuthology.orchestra.run.smithi089.stdout:606 read 0xbcfea6 thru 0xbd0c63 (0xdbe bytes) 2024-09-21T15:49:16.472 INFO:teuthology.orchestra.run.smithi089.stdout:609 punch from 0x3984473 to 0x398a40c, (0x5f99 bytes) 2024-09-21T15:49:16.472 INFO:teuthology.orchestra.run.smithi089.stdout:610 read 0x4d54e93 thru 0x4d63abd (0xec2b bytes) 2024-09-21T15:49:16.472 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:16.472 INFO:teuthology.orchestra.run.smithi089.stdout:612 read 0x3914105 thru 0x3917961 (0x385d bytes) 2024-09-21T15:49:16.472 INFO:teuthology.orchestra.run.smithi089.stdout:614 clone 3 order 20 su 4096 sc 12 2024-09-21T15:49:16.502 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:16.502 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:16.503 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:16.502+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:16.511 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:49:16.604 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:16.603+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:16.657 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:16.658 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:16.667 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:49:16.708 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:16.707+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:16.798 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:49:16.806 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:16.805+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:16.817 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:16.817 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:16.827 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:49:16.849 INFO:teuthology.orchestra.run.smithi089.stdout:checking clone #1, image image_client.0-clone1 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent2 2024-09-21T15:49:16.862 INFO:teuthology.orchestra.run.smithi089.stdout:615 writesame 0x7dae879 thru 0x7dafda9 data_size 0x1f(0x1531 bytes) 2024-09-21T15:49:16.862 INFO:teuthology.orchestra.run.smithi089.stdout:617 punch from 0x3b1854e to 0x3b227fb, (0xa2ad bytes) 2024-09-21T15:49:16.870 INFO:teuthology.orchestra.run.smithi089.stdout:618 write 0xa22ba39 thru 0xa235903 (0x9ecb bytes) 2024-09-21T15:49:16.891 INFO:teuthology.orchestra.run.smithi089.stdout:619 write 0x902f2db thru 0x9030aec (0x1812 bytes) 2024-09-21T15:49:16.897 INFO:teuthology.orchestra.run.smithi089.stdout:620 writesame 0x5cbcfcf thru 0x5cc61ca data_size 0x2(0x91fc bytes) 2024-09-21T15:49:16.906 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:16.905+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:16.914 INFO:teuthology.orchestra.run.smithi089.stdout:621 punch from 0x2f46ac6 to 0x2f4d9ab, (0x6ee5 bytes) 2024-09-21T15:49:16.914 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:16.921 INFO:teuthology.orchestra.run.smithi089.stdout:623 trunc from 0xa235904 to 0x502ac02 2024-09-21T15:49:16.938 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-09-21T15:49:16.938 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:49:16.956 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:49:16.956 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:49:16.965 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:16.965 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:16.966 INFO:teuthology.orchestra.run.smithi089.stdout:624 trunc from 0x502ac02 to 0x2dd0e9 2024-09-21T15:49:16.976 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:49:16.981 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:49:16.981 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:49:17.001 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:49:17.001 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:49:17.001 INFO:teuthology.orchestra.run.smithi089.stdout:625 read 0x547af thru 0x5d2ef (0x8b41 bytes) 2024-09-21T15:49:17.002 INFO:teuthology.orchestra.run.smithi089.stdout:627 read 0x2958bd thru 0x299080 (0x37c4 bytes) 2024-09-21T15:49:17.007 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:17.006+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:17.018 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:49:17.018 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:49:17.021 INFO:teuthology.orchestra.run.smithi089.stdout:628 write 0xb542a72 thru 0xb550cfd (0xe28c bytes) 2024-09-21T15:49:17.035 INFO:teuthology.orchestra.run.smithi089.stdout:629 writesame 0x9709da6 thru 0x9710dba data_size 0x1(0x7015 bytes) 2024-09-21T15:49:17.050 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:49:17.050 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:49:17.055 INFO:teuthology.orchestra.run.smithi089.stdout:631 write 0x9f57529 thru 0x9f5ca05 (0x54dd bytes) 2024-09-21T15:49:17.066 INFO:teuthology.orchestra.run.smithi089.stdout:633 read 0x954cf42 thru 0x9558918 (0xb9d7 bytes) 2024-09-21T15:49:17.067 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:49:17.067 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:49:17.073 INFO:teuthology.orchestra.run.smithi089.stdout:634 writesame 0xbdadd32 thru 0xbdbc64f data_size 0x35(0xe91e bytes) 2024-09-21T15:49:17.077 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:17.077 INFO:teuthology.orchestra.run.smithi089.stdout:636 read 0x94cb329 thru 0x94ce64a (0x3322 bytes) 2024-09-21T15:49:17.084 INFO:teuthology.orchestra.run.smithi089.stdout:637 trunc from 0xbdbc650 to 0x6519293 2024-09-21T15:49:17.087 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:49:17.087 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:49:17.106 INFO:teuthology.orchestra.run.smithi089.stdout:638 read 0x4e1919d thru 0x4e25f46 (0xcdaa bytes) 2024-09-21T15:49:17.106 INFO:teuthology.orchestra.run.smithi089.stdout:640 read 0x3ac3e52 thru 0x3ad3ac9 (0xfc78 bytes) 2024-09-21T15:49:17.107 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:17.106+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:17.113 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:17.113 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:49:17.113 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:49:17.113 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:49:17.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.113 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:887 3.0 e27/22 3:025e53e7:::rbd_data.1133f3dfbd6b.0000000000000008:head v 27'14, mlcod=27'12)", 2024-09-21T15:49:17.113 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.458555+0000", 2024-09-21T15:49:17.113 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.6518172419999999, 2024-09-21T15:49:17.113 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00080860600000000002, 2024-09-21T15:49:17.114 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:17.114 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:17.114 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:17.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.114 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:17.114 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.458555+0000", 2024-09-21T15:49:17.114 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.114 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.114 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:17.114 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.458555+0000", 2024-09-21T15:49:17.114 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.114 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:17.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.458553+0000", 2024-09-21T15:49:17.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:49:17.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:17.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.458583+0000", 2024-09-21T15:49:17.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0307999999999998e-05 2024-09-21T15:49:17.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:17.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.458603+0000", 2024-09-21T15:49:17.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.997e-05 2024-09-21T15:49:17.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:17.116 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.458614+0000", 2024-09-21T15:49:17.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0618999999999999e-05 2024-09-21T15:49:17.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.116 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:17.116 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.458667+0000", 2024-09-21T15:49:17.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3625999999999998e-05 2024-09-21T15:49:17.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.116 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:17.116 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.458689+0000", 2024-09-21T15:49:17.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1812000000000001e-05 2024-09-21T15:49:17.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.116 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:17.116 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.459327+0000", 2024-09-21T15:49:17.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00063789999999999995 2024-09-21T15:49:17.117 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.117 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.117 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:17.117 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.459363+0000", 2024-09-21T15:49:17.117 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6217000000000002e-05 2024-09-21T15:49:17.117 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.117 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:17.117 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.117 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.117 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.117 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:888 3.0 e27/22 3:025e53e7:::rbd_data.1133f3dfbd6b.0000000000000008:head v 27'15, mlcod=27'13)", 2024-09-21T15:49:17.117 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.459921+0000", 2024-09-21T15:49:17.117 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.650450749, 2024-09-21T15:49:17.117 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00084743299999999998, 2024-09-21T15:49:17.117 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:17.117 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:17.118 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:17.118 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.118 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:17.118 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.459921+0000", 2024-09-21T15:49:17.118 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.118 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.118 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.118 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:17.118 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.459921+0000", 2024-09-21T15:49:17.118 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.118 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.118 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.118 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:17.118 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.459921+0000", 2024-09-21T15:49:17.118 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:17.119 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.119 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.119 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:17.119 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.459956+0000", 2024-09-21T15:49:17.119 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5673999999999998e-05 2024-09-21T15:49:17.119 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.119 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.119 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:17.119 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.459970+0000", 2024-09-21T15:49:17.119 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3443e-05 2024-09-21T15:49:17.119 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.119 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.119 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:17.119 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.459976+0000", 2024-09-21T15:49:17.119 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5100000000000004e-06 2024-09-21T15:49:17.119 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.120 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.120 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:17.120 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.460020+0000", 2024-09-21T15:49:17.120 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4132999999999999e-05 2024-09-21T15:49:17.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.120 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.120 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:17.120 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.460040+0000", 2024-09-21T15:49:17.120 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9466e-05 2024-09-21T15:49:17.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.120 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.120 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:17.120 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.460742+0000", 2024-09-21T15:49:17.120 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00070200500000000001 2024-09-21T15:49:17.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.120 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.121 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:17.121 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.460769+0000", 2024-09-21T15:49:17.121 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6834000000000001e-05 2024-09-21T15:49:17.121 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.121 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:17.121 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.121 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:945 3.0 e27/22 3:05942dfc:::rbd_data.1133f3dfbd6b.000000000000000e:head v 27'18, mlcod=27'16)", 2024-09-21T15:49:17.121 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.546272+0000", 2024-09-21T15:49:17.121 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.564099981, 2024-09-21T15:49:17.121 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00096418899999999995, 2024-09-21T15:49:17.121 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:17.122 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:17.122 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:17.122 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.122 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:17.122 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.546272+0000", 2024-09-21T15:49:17.122 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.122 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.122 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.122 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:17.122 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.546272+0000", 2024-09-21T15:49:17.122 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.122 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.122 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.122 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:17.122 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.546270+0000", 2024-09-21T15:49:17.123 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:17.123 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.123 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:17.123 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.546320+0000", 2024-09-21T15:49:17.123 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0043999999999997e-05 2024-09-21T15:49:17.123 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.123 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:17.123 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.546338+0000", 2024-09-21T15:49:17.123 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7841000000000001e-05 2024-09-21T15:49:17.123 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.123 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:17.123 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.546348+0000", 2024-09-21T15:49:17.124 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4890000000000001e-06 2024-09-21T15:49:17.124 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.124 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.124 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:17.124 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.546392+0000", 2024-09-21T15:49:17.124 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4363000000000002e-05 2024-09-21T15:49:17.124 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.124 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.124 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:17.124 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.546414+0000", 2024-09-21T15:49:17.124 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2164999999999999e-05 2024-09-21T15:49:17.124 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.124 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.124 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:17.125 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.547211+0000", 2024-09-21T15:49:17.125 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00079653099999999995 2024-09-21T15:49:17.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.125 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.125 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:17.125 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.547236+0000", 2024-09-21T15:49:17.125 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5364e-05 2024-09-21T15:49:17.125 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.125 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:17.125 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.125 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.125 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:983 3.0 e27/22 3:025e53e7:::rbd_data.1133f3dfbd6b.0000000000000008:head v 27'20, mlcod=27'18)", 2024-09-21T15:49:17.125 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.615392+0000", 2024-09-21T15:49:17.125 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.4949801679999997, 2024-09-21T15:49:17.126 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032613970000000001, 2024-09-21T15:49:17.126 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:17.126 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:17.126 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:17.126 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.126 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:17.126 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.615392+0000", 2024-09-21T15:49:17.126 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.126 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.126 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.126 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:17.126 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.615392+0000", 2024-09-21T15:49:17.126 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.127 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.127 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.127 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:17.127 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.615390+0000", 2024-09-21T15:49:17.127 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:17.127 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.127 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.127 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:17.127 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.615431+0000", 2024-09-21T15:49:17.127 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0893999999999998e-05 2024-09-21T15:49:17.127 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.127 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.127 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:17.127 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.615450+0000", 2024-09-21T15:49:17.127 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8961e-05 2024-09-21T15:49:17.127 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.128 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.128 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:17.128 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.615458+0000", 2024-09-21T15:49:17.128 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1620000000000008e-06 2024-09-21T15:49:17.128 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.128 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.128 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:17.128 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.615501+0000", 2024-09-21T15:49:17.128 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2116999999999997e-05 2024-09-21T15:49:17.128 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.128 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.128 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:17.128 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.615518+0000", 2024-09-21T15:49:17.128 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7689e-05 2024-09-21T15:49:17.129 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.129 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.129 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:17.129 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.618621+0000", 2024-09-21T15:49:17.129 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0031022340000000002 2024-09-21T15:49:17.129 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.129 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.129 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:17.129 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.618653+0000", 2024-09-21T15:49:17.129 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2591e-05 2024-09-21T15:49:17.129 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.129 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:17.130 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.130 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.130 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.130 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:984 3.0 e27/22 3:025e53e7:::rbd_data.1133f3dfbd6b.0000000000000008:head v 27'21, mlcod=27'19)", 2024-09-21T15:49:17.130 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.619393+0000", 2024-09-21T15:49:17.130 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.4909793410000001, 2024-09-21T15:49:17.130 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003401837, 2024-09-21T15:49:17.130 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:17.130 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:17.130 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:17.130 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.130 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:17.131 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619393+0000", 2024-09-21T15:49:17.131 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.131 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.131 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.131 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:17.131 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619393+0000", 2024-09-21T15:49:17.131 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.131 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.131 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.131 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:17.131 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619392+0000", 2024-09-21T15:49:17.131 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:17.131 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.131 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.131 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:17.132 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619439+0000", 2024-09-21T15:49:17.132 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7170999999999999e-05 2024-09-21T15:49:17.132 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.132 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.132 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:17.132 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619452+0000", 2024-09-21T15:49:17.132 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2986999999999999e-05 2024-09-21T15:49:17.132 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.132 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.132 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:17.132 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619459+0000", 2024-09-21T15:49:17.132 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6109999999999999e-06 2024-09-21T15:49:17.132 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.132 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.132 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:17.133 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619518+0000", 2024-09-21T15:49:17.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9117999999999999e-05 2024-09-21T15:49:17.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.133 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.133 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:17.133 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619532+0000", 2024-09-21T15:49:17.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4622e-05 2024-09-21T15:49:17.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.133 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.133 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:17.133 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.622760+0000", 2024-09-21T15:49:17.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032278839999999999 2024-09-21T15:49:17.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.133 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.134 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:17.134 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.622794+0000", 2024-09-21T15:49:17.134 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4144999999999999e-05 2024-09-21T15:49:17.134 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.134 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:17.134 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.134 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.134 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.134 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:999 3.0 e27/22 3:025e53e7:::rbd_data.1133f3dfbd6b.0000000000000008:head v 27'22, mlcod=27'20)", 2024-09-21T15:49:17.134 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.638611+0000", 2024-09-21T15:49:17.134 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.4717609750000005, 2024-09-21T15:49:17.134 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00094990500000000002, 2024-09-21T15:49:17.134 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:17.134 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:17.135 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:17.135 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.135 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:17.135 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.638611+0000", 2024-09-21T15:49:17.135 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.135 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.135 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.135 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:17.135 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.638611+0000", 2024-09-21T15:49:17.135 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.135 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.135 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.135 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:17.135 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.638610+0000", 2024-09-21T15:49:17.136 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:17.136 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.136 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.136 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:17.136 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.638659+0000", 2024-09-21T15:49:17.136 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9434999999999997e-05 2024-09-21T15:49:17.136 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.136 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.136 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:17.136 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.638678+0000", 2024-09-21T15:49:17.136 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8992e-05 2024-09-21T15:49:17.136 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.136 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.136 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:17.137 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.638687+0000", 2024-09-21T15:49:17.137 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4379999999999995e-06 2024-09-21T15:49:17.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.137 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:17.137 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.638733+0000", 2024-09-21T15:49:17.137 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6338000000000003e-05 2024-09-21T15:49:17.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.137 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:17.137 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.638757+0000", 2024-09-21T15:49:17.137 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3825999999999999e-05 2024-09-21T15:49:17.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.137 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:17.138 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.639524+0000", 2024-09-21T15:49:17.138 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00076672000000000003 2024-09-21T15:49:17.138 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.138 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.138 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:17.138 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.639561+0000", 2024-09-21T15:49:17.138 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7317999999999997e-05 2024-09-21T15:49:17.138 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.138 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:17.138 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.138 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.138 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.138 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:1044 3.0 e27/22 3:025e53e7:::rbd_data.1133f3dfbd6b.0000000000000008:head v 27'24, mlcod=27'22)", 2024-09-21T15:49:17.139 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.709866+0000", 2024-09-21T15:49:17.139 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.4005055970000004, 2024-09-21T15:49:17.139 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0021649210000000002, 2024-09-21T15:49:17.139 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:17.139 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:17.139 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:17.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.139 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:17.139 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.709866+0000", 2024-09-21T15:49:17.139 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.139 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.139 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:17.139 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.709866+0000", 2024-09-21T15:49:17.139 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.140 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:17.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.709865+0000", 2024-09-21T15:49:17.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:17.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.140 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:17.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.709892+0000", 2024-09-21T15:49:17.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6990000000000001e-05 2024-09-21T15:49:17.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.140 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:17.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.709911+0000", 2024-09-21T15:49:17.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8386999999999999e-05 2024-09-21T15:49:17.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.141 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:17.141 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.709920+0000", 2024-09-21T15:49:17.141 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0059999999999998e-06 2024-09-21T15:49:17.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.141 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:17.141 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.709964+0000", 2024-09-21T15:49:17.141 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4132999999999999e-05 2024-09-21T15:49:17.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.141 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:17.141 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.709992+0000", 2024-09-21T15:49:17.141 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8204e-05 2024-09-21T15:49:17.142 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.142 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.142 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:17.142 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.711995+0000", 2024-09-21T15:49:17.142 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002002925 2024-09-21T15:49:17.142 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.142 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.142 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:17.142 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.712031+0000", 2024-09-21T15:49:17.142 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6434999999999999e-05 2024-09-21T15:49:17.142 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.142 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:17.142 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.142 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.143 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.143 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:1115 3.0 e29/22 3:04479396:::rbd_id.image_client.0-clone2:head v 29'25, mlcod=27'23)", 2024-09-21T15:49:17.143 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:13.747085+0000", 2024-09-21T15:49:17.143 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.3632872479999998, 2024-09-21T15:49:17.143 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00086631800000000004, 2024-09-21T15:49:17.143 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:17.143 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:17.143 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:17.143 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.143 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:17.143 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:13.747085+0000", 2024-09-21T15:49:17.143 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.143 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.143 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.144 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:17.144 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:13.747085+0000", 2024-09-21T15:49:17.144 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.144 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.144 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:17.144 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:13.747084+0000", 2024-09-21T15:49:17.144 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:17.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.144 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.144 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:17.144 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:13.747091+0000", 2024-09-21T15:49:17.144 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5089999999999999e-06 2024-09-21T15:49:17.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.144 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.145 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:17.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:13.747102+0000", 2024-09-21T15:49:17.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.119e-05 2024-09-21T15:49:17.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.145 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:17.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:13.747112+0000", 2024-09-21T15:49:17.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5850000000000006e-06 2024-09-21T15:49:17.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.145 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:17.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:13.747158+0000", 2024-09-21T15:49:17.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6804000000000001e-05 2024-09-21T15:49:17.146 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.146 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.146 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:17.146 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:13.747183+0000", 2024-09-21T15:49:17.146 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4736000000000002e-05 2024-09-21T15:49:17.146 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.146 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.146 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:17.146 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:13.747909+0000", 2024-09-21T15:49:17.146 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00072590899999999995 2024-09-21T15:49:17.146 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.146 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.146 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:17.147 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:13.747951+0000", 2024-09-21T15:49:17.147 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1888999999999998e-05 2024-09-21T15:49:17.147 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.147 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:17.147 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.147 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.147 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.147 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:1157 3.0 e29/22 3:025e53e7:::rbd_data.1133f3dfbd6b.0000000000000008:head v 29'27, mlcod=27'24)", 2024-09-21T15:49:17.147 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:13.781328+0000", 2024-09-21T15:49:17.147 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.3290438920000001, 2024-09-21T15:49:17.147 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001741561, 2024-09-21T15:49:17.147 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:17.147 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:17.147 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:17.147 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.147 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:17.148 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:13.781328+0000", 2024-09-21T15:49:17.148 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.148 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.148 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.148 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:17.148 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:13.781328+0000", 2024-09-21T15:49:17.148 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.148 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.148 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.148 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:17.148 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:13.781327+0000", 2024-09-21T15:49:17.148 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:17.148 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.148 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.148 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:17.149 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:13.781339+0000", 2024-09-21T15:49:17.149 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2284e-05 2024-09-21T15:49:17.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.149 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:17.149 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:13.781357+0000", 2024-09-21T15:49:17.149 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8224e-05 2024-09-21T15:49:17.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.149 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:17.149 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:13.781366+0000", 2024-09-21T15:49:17.149 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4610000000000008e-06 2024-09-21T15:49:17.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.150 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:17.150 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:13.781419+0000", 2024-09-21T15:49:17.150 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3610999999999999e-05 2024-09-21T15:49:17.150 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.150 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.150 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:17.150 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:13.781441+0000", 2024-09-21T15:49:17.150 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2036e-05 2024-09-21T15:49:17.150 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.150 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.150 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:17.150 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:13.783029+0000", 2024-09-21T15:49:17.150 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001587677 2024-09-21T15:49:17.150 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.151 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.151 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:17.151 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:13.783070+0000", 2024-09-21T15:49:17.151 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0558e-05 2024-09-21T15:49:17.151 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.151 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:17.151 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.151 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.151 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.151 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:4957 3.9 e29/22 3:95c5ed95:::rbd_data.11335fa2fe69.0000000000000018:head v 29'4, mlcod=26'2)", 2024-09-21T15:49:17.151 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:14.453951+0000", 2024-09-21T15:49:17.151 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 2.6564209089999999, 2024-09-21T15:49:17.151 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0043804559999999996, 2024-09-21T15:49:17.151 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:17.151 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:17.152 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:17.152 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.152 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:17.152 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.453951+0000", 2024-09-21T15:49:17.152 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.152 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.152 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.152 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:17.152 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.453951+0000", 2024-09-21T15:49:17.152 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.152 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.152 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.152 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:17.152 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.453950+0000", 2024-09-21T15:49:17.152 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:17.153 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.153 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.153 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:17.153 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454495+0000", 2024-09-21T15:49:17.153 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00054437799999999998 2024-09-21T15:49:17.153 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.153 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.153 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:17.153 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454508+0000", 2024-09-21T15:49:17.153 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3475000000000001e-05 2024-09-21T15:49:17.153 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.153 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.153 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:17.153 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454515+0000", 2024-09-21T15:49:17.153 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8990000000000004e-06 2024-09-21T15:49:17.154 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.154 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.154 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:17.154 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454581+0000", 2024-09-21T15:49:17.154 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5987000000000004e-05 2024-09-21T15:49:17.154 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.154 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.154 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:17.154 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454609+0000", 2024-09-21T15:49:17.155 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7466000000000001e-05 2024-09-21T15:49:17.155 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.155 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.155 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:17.155 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.458303+0000", 2024-09-21T15:49:17.155 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003693979 2024-09-21T15:49:17.155 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.155 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.155 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:17.155 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.458331+0000", 2024-09-21T15:49:17.155 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8932999999999999e-05 2024-09-21T15:49:17.155 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.155 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:17.155 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.156 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.156 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.156 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:5084 3.9 e29/22 3:96420cbd:::rbd_data.11335fa2fe69.0000000000000019:head v 29'5, mlcod=26'3)", 2024-09-21T15:49:17.156 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:14.463314+0000", 2024-09-21T15:49:17.156 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 2.6470581989999999, 2024-09-21T15:49:17.156 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0016920150000000001, 2024-09-21T15:49:17.156 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:17.156 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:17.156 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:17.156 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.156 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:17.156 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.463314+0000", 2024-09-21T15:49:17.156 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.156 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.156 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.156 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:17.157 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.463314+0000", 2024-09-21T15:49:17.157 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.157 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.157 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.157 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:17.157 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.463313+0000", 2024-09-21T15:49:17.157 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:17.157 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.157 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.157 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:17.157 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.463814+0000", 2024-09-21T15:49:17.157 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00050131500000000001 2024-09-21T15:49:17.157 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.157 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.157 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:17.158 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.463825+0000", 2024-09-21T15:49:17.158 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0917999999999999e-05 2024-09-21T15:49:17.158 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.158 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.158 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:17.158 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.463830+0000", 2024-09-21T15:49:17.158 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2599999999999996e-06 2024-09-21T15:49:17.158 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.158 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.158 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:17.158 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.463864+0000", 2024-09-21T15:49:17.158 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3729999999999997e-05 2024-09-21T15:49:17.158 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.158 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.158 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:17.159 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.463881+0000", 2024-09-21T15:49:17.159 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6682000000000001e-05 2024-09-21T15:49:17.159 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.159 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.159 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:17.159 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.464959+0000", 2024-09-21T15:49:17.159 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010779660000000001 2024-09-21T15:49:17.159 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.159 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.159 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:17.159 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.465006+0000", 2024-09-21T15:49:17.159 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6928999999999997e-05 2024-09-21T15:49:17.159 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.159 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:17.159 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.160 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.160 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.160 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:5232 3.9 e29/22 3:91d21607:::rbd_data.11335fa2fe69.000000000000001c:head v 29'6, mlcod=29'4)", 2024-09-21T15:49:17.160 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:14.500469+0000", 2024-09-21T15:49:17.160 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 2.6099032950000001, 2024-09-21T15:49:17.160 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00097083800000000002, 2024-09-21T15:49:17.160 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:17.160 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:17.160 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:17.160 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.160 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:17.160 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.500469+0000", 2024-09-21T15:49:17.160 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.160 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.160 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.160 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:17.160 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.500469+0000", 2024-09-21T15:49:17.161 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.161 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.161 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.161 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:17.161 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.500468+0000", 2024-09-21T15:49:17.161 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:17.161 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.161 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.161 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:17.161 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.500477+0000", 2024-09-21T15:49:17.161 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5319999999999995e-06 2024-09-21T15:49:17.161 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.161 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.161 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:17.161 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.500491+0000", 2024-09-21T15:49:17.161 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3405e-05 2024-09-21T15:49:17.162 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.162 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.162 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:17.162 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.500497+0000", 2024-09-21T15:49:17.162 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7499999999999997e-06 2024-09-21T15:49:17.162 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.162 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.162 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:17.162 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.500542+0000", 2024-09-21T15:49:17.162 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4628e-05 2024-09-21T15:49:17.162 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.162 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.162 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:17.162 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.500566+0000", 2024-09-21T15:49:17.162 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4144000000000001e-05 2024-09-21T15:49:17.163 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.163 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.163 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:17.163 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.501397+0000", 2024-09-21T15:49:17.163 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00083068400000000004 2024-09-21T15:49:17.163 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.163 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.163 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:17.163 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.501439+0000", 2024-09-21T15:49:17.163 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2724e-05 2024-09-21T15:49:17.163 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.163 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:17.163 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.163 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.163 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.163 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13127 3.0 e31/22 3:07440800:::rbd_data.1133dc5adfdf.0000000000000001:head v 31'46, mlcod=31'44)", 2024-09-21T15:49:17.164 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:16.984393+0000", 2024-09-21T15:49:17.164 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.125978958, 2024-09-21T15:49:17.164 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0019997330000000001, 2024-09-21T15:49:17.164 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:17.164 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:17.164 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:17.164 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.164 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:17.164 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:16.984393+0000", 2024-09-21T15:49:17.164 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.164 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.164 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.164 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:17.164 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:16.984393+0000", 2024-09-21T15:49:17.164 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.164 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.165 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.165 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:17.165 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:16.984392+0000", 2024-09-21T15:49:17.165 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:17.165 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.165 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.165 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:17.165 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:16.984510+0000", 2024-09-21T15:49:17.165 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011804200000000001 2024-09-21T15:49:17.165 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.165 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.165 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:17.165 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:16.984526+0000", 2024-09-21T15:49:17.165 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6158e-05 2024-09-21T15:49:17.165 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.166 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.166 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:17.166 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:16.984534+0000", 2024-09-21T15:49:17.166 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6580000000000002e-06 2024-09-21T15:49:17.166 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.166 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.166 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:17.166 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:16.984587+0000", 2024-09-21T15:49:17.166 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3278e-05 2024-09-21T15:49:17.166 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.166 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.166 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:17.166 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:16.984607+0000", 2024-09-21T15:49:17.166 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9338999999999999e-05 2024-09-21T15:49:17.166 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.167 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.167 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:17.167 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:16.986361+0000", 2024-09-21T15:49:17.167 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001754441 2024-09-21T15:49:17.167 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.167 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.167 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:17.167 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:16.986393+0000", 2024-09-21T15:49:17.167 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1717999999999997e-05 2024-09-21T15:49:17.167 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.167 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:17.167 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.167 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.167 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.167 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13189 3.4 e31/22 3:2704ec82:::rbd_data.1133dc5adfdf.00000000000000bc:head v 31'11, mlcod=30'9)", 2024-09-21T15:49:17.167 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.033320+0000", 2024-09-21T15:49:17.168 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.077052093000000002, 2024-09-21T15:49:17.168 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00086028400000000001, 2024-09-21T15:49:17.168 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:17.168 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:17.168 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:17.168 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.168 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:17.168 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.033320+0000", 2024-09-21T15:49:17.168 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.168 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.168 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.168 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:17.168 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.033320+0000", 2024-09-21T15:49:17.168 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.168 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.168 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.169 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:17.169 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.033319+0000", 2024-09-21T15:49:17.169 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:17.169 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.169 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.169 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:17.169 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.033331+0000", 2024-09-21T15:49:17.169 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2371000000000001e-05 2024-09-21T15:49:17.169 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.169 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.169 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:17.169 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.033351+0000", 2024-09-21T15:49:17.169 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9656e-05 2024-09-21T15:49:17.169 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.169 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.169 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:17.169 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.033359+0000", 2024-09-21T15:49:17.170 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3420000000000008e-06 2024-09-21T15:49:17.170 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.170 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.170 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:17.170 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.033409+0000", 2024-09-21T15:49:17.170 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9625e-05 2024-09-21T15:49:17.170 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.170 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.170 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:17.170 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.033429+0000", 2024-09-21T15:49:17.170 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.075e-05 2024-09-21T15:49:17.170 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.170 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.170 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:17.170 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.034138+0000", 2024-09-21T15:49:17.171 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00070835899999999999 2024-09-21T15:49:17.171 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.171 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.171 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:17.171 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.034180+0000", 2024-09-21T15:49:17.171 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2382999999999997e-05 2024-09-21T15:49:17.171 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.171 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:17.171 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.171 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.171 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.171 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13204 3.0 e31/22 3:074150b9:::rbd_data.1133dc5adfdf.0000000000000094:head v 31'47, mlcod=31'45)", 2024-09-21T15:49:17.171 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.047955+0000", 2024-09-21T15:49:17.171 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.062416884999999998, 2024-09-21T15:49:17.171 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00088784400000000005, 2024-09-21T15:49:17.171 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:17.172 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:17.172 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:17.172 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.172 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:17.172 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.047955+0000", 2024-09-21T15:49:17.172 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.172 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.172 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.172 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:17.172 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.047955+0000", 2024-09-21T15:49:17.172 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.172 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.172 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.172 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:17.172 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.047954+0000", 2024-09-21T15:49:17.172 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:17.172 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.173 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.173 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:17.173 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.047972+0000", 2024-09-21T15:49:17.173 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8102999999999999e-05 2024-09-21T15:49:17.173 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.173 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.173 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:17.173 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.047998+0000", 2024-09-21T15:49:17.173 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5783e-05 2024-09-21T15:49:17.173 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.173 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.173 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:17.173 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.048005+0000", 2024-09-21T15:49:17.173 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2869999999999999e-06 2024-09-21T15:49:17.173 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.173 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.174 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:17.174 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.048046+0000", 2024-09-21T15:49:17.174 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1359999999999997e-05 2024-09-21T15:49:17.174 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.174 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.174 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:17.174 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.048070+0000", 2024-09-21T15:49:17.174 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.34e-05 2024-09-21T15:49:17.174 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.174 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.174 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:17.174 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.048803+0000", 2024-09-21T15:49:17.174 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00073296799999999997 2024-09-21T15:49:17.174 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.174 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.174 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:17.175 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.048843+0000", 2024-09-21T15:49:17.175 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0117000000000002e-05 2024-09-21T15:49:17.175 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.175 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:17.175 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.175 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.175 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.175 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13217 3.9 e31/22 3:93003736:::rbd_data.1133dc5adfdf.00000000000000a1:head v 31'44, mlcod=31'42)", 2024-09-21T15:49:17.175 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.062847+0000", 2024-09-21T15:49:17.175 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.047524727000000003, 2024-09-21T15:49:17.175 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00086816499999999995, 2024-09-21T15:49:17.175 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:17.175 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:17.175 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:17.175 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.176 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:17.176 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.062847+0000", 2024-09-21T15:49:17.176 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.176 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.176 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.176 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:17.176 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.062847+0000", 2024-09-21T15:49:17.176 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.176 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.176 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.176 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:17.176 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.062846+0000", 2024-09-21T15:49:17.176 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:17.176 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.176 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.177 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:17.177 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.062860+0000", 2024-09-21T15:49:17.177 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4307e-05 2024-09-21T15:49:17.177 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.177 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.177 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:17.177 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.062881+0000", 2024-09-21T15:49:17.177 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1543e-05 2024-09-21T15:49:17.177 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.177 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.177 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:17.177 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.062893+0000", 2024-09-21T15:49:17.177 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1698e-05 2024-09-21T15:49:17.177 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.177 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.177 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:17.178 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.062943+0000", 2024-09-21T15:49:17.178 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0084999999999999e-05 2024-09-21T15:49:17.178 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.178 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.178 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:17.178 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.062967+0000", 2024-09-21T15:49:17.178 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4241000000000001e-05 2024-09-21T15:49:17.178 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.178 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.178 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:17.178 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.063684+0000", 2024-09-21T15:49:17.178 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00071619800000000003 2024-09-21T15:49:17.178 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.178 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.178 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:17.179 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.063715+0000", 2024-09-21T15:49:17.179 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1707e-05 2024-09-21T15:49:17.179 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.179 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:17.179 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.179 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.179 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.179 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "pg_update_log_missing(3.4 epoch 31/22 rep_tid 86 entries 31'12 (0'0) error 3:2704ec82:::rbd_data.1133dc5adfdf.00000000000000bc:head by client.4403.0:13246 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 30'10)", 2024-09-21T15:49:17.179 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.073886+0000", 2024-09-21T15:49:17.179 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.036485524999999998, 2024-09-21T15:49:17.179 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001030881, 2024-09-21T15:49:17.179 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:17.179 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-09-21T15:49:17.179 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:17.179 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.180 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:17.180 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.073886+0000", 2024-09-21T15:49:17.180 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.180 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.180 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.180 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:17.180 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.073886+0000", 2024-09-21T15:49:17.180 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.180 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.180 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.180 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:17.180 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.073885+0000", 2024-09-21T15:49:17.180 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:17.180 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.181 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.181 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:17.181 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.073897+0000", 2024-09-21T15:49:17.181 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1922e-05 2024-09-21T15:49:17.181 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.181 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.181 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:17.181 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.073922+0000", 2024-09-21T15:49:17.181 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4703999999999999e-05 2024-09-21T15:49:17.181 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.181 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.181 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:17.181 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.073935+0000", 2024-09-21T15:49:17.181 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3351e-05 2024-09-21T15:49:17.181 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.182 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.182 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:17.182 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.074003+0000", 2024-09-21T15:49:17.182 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7543e-05 2024-09-21T15:49:17.182 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.182 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.182 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:17.182 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.074917+0000", 2024-09-21T15:49:17.182 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00091447499999999999 2024-09-21T15:49:17.182 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.182 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:17.182 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.182 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.182 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.183 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13258 3.0 e31/22 3:074150b9:::rbd_data.1133dc5adfdf.0000000000000094:head v 31'48, mlcod=31'46)", 2024-09-21T15:49:17.183 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.086428+0000", 2024-09-21T15:49:17.183 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.023943677, 2024-09-21T15:49:17.183 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000856641, 2024-09-21T15:49:17.183 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:17.183 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:17.183 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:17.183 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.183 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:17.183 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.086428+0000", 2024-09-21T15:49:17.183 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.183 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.183 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.183 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:17.184 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.086428+0000", 2024-09-21T15:49:17.184 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.184 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.184 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.184 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:17.184 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.086427+0000", 2024-09-21T15:49:17.184 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:17.184 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.184 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.184 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:17.184 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.086439+0000", 2024-09-21T15:49:17.184 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1997000000000001e-05 2024-09-21T15:49:17.184 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.184 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.184 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:17.185 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.086459+0000", 2024-09-21T15:49:17.185 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0109999999999999e-05 2024-09-21T15:49:17.185 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.185 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.185 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:17.185 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.086468+0000", 2024-09-21T15:49:17.185 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2259999999999994e-06 2024-09-21T15:49:17.185 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.185 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.185 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:17.185 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.086517+0000", 2024-09-21T15:49:17.185 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8615000000000001e-05 2024-09-21T15:49:17.185 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.185 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.186 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:17.186 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.086539+0000", 2024-09-21T15:49:17.186 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1877000000000001e-05 2024-09-21T15:49:17.186 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.186 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.186 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:17.186 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.087247+0000", 2024-09-21T15:49:17.186 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00070812399999999997 2024-09-21T15:49:17.186 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.186 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.186 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:17.186 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.087285+0000", 2024-09-21T15:49:17.186 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7839999999999997e-05 2024-09-21T15:49:17.186 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.186 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:17.187 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.187 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.187 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.187 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13265 3.9 e31/22 3:93003736:::rbd_data.1133dc5adfdf.00000000000000a1:head v 31'45, mlcod=31'43)", 2024-09-21T15:49:17.187 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.088496+0000", 2024-09-21T15:49:17.187 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.021875853000000001, 2024-09-21T15:49:17.187 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0017981010000000001, 2024-09-21T15:49:17.187 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:17.187 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:17.187 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:17.187 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.187 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:17.187 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.088496+0000", 2024-09-21T15:49:17.187 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.187 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.187 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.188 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:17.188 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.088496+0000", 2024-09-21T15:49:17.188 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.188 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.188 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.188 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:17.188 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.088495+0000", 2024-09-21T15:49:17.188 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:17.188 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.188 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.188 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:17.188 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.088502+0000", 2024-09-21T15:49:17.188 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8970000000000002e-06 2024-09-21T15:49:17.188 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.188 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.188 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:17.188 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.088509+0000", 2024-09-21T15:49:17.189 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3339999999999998e-06 2024-09-21T15:49:17.189 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.189 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.189 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:17.189 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.088514+0000", 2024-09-21T15:49:17.189 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4149999999999998e-06 2024-09-21T15:49:17.189 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.189 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.189 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:17.189 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.088558+0000", 2024-09-21T15:49:17.189 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4391000000000002e-05 2024-09-21T15:49:17.189 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.189 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.189 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:17.189 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.088581+0000", 2024-09-21T15:49:17.189 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2478000000000001e-05 2024-09-21T15:49:17.190 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.190 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.190 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:17.190 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.090255+0000", 2024-09-21T15:49:17.190 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001673838 2024-09-21T15:49:17.190 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.190 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.190 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:17.190 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.090294+0000", 2024-09-21T15:49:17.190 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9477999999999997e-05 2024-09-21T15:49:17.190 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.190 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:17.190 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.190 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.190 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.190 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13277 3.4 e31/22 3:2704ec82:::rbd_data.1133dc5adfdf.00000000000000bc:head v 31'13, mlcod=31'11)", 2024-09-21T15:49:17.191 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.093651+0000", 2024-09-21T15:49:17.191 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.016720472, 2024-09-21T15:49:17.191 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0022583049999999999, 2024-09-21T15:49:17.191 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:17.191 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:17.191 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:17.191 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.191 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:17.191 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.093651+0000", 2024-09-21T15:49:17.191 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.191 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.191 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.191 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:17.191 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.093651+0000", 2024-09-21T15:49:17.191 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.192 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.192 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.192 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:17.192 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.093650+0000", 2024-09-21T15:49:17.192 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:17.192 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.192 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.192 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:17.192 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.093660+0000", 2024-09-21T15:49:17.192 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7070000000000004e-06 2024-09-21T15:49:17.192 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.192 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.192 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:17.192 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.093679+0000", 2024-09-21T15:49:17.192 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9154999999999999e-05 2024-09-21T15:49:17.192 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.192 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.193 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:17.193 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.093687+0000", 2024-09-21T15:49:17.193 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4179999999999997e-06 2024-09-21T15:49:17.193 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.193 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.193 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:17.193 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.093739+0000", 2024-09-21T15:49:17.193 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1638000000000003e-05 2024-09-21T15:49:17.193 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.193 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.193 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:17.193 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.093762+0000", 2024-09-21T15:49:17.193 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3076000000000001e-05 2024-09-21T15:49:17.193 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.193 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.193 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:17.194 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.095878+0000", 2024-09-21T15:49:17.194 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0021156339999999999 2024-09-21T15:49:17.194 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.194 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.194 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:17.194 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.095910+0000", 2024-09-21T15:49:17.194 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2023000000000002e-05 2024-09-21T15:49:17.194 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.194 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:17.194 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.194 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.194 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:17.194 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:17.195 INFO:teuthology.orchestra.run.smithi089.stdout:641 write 0xd230372 thru 0xd2336fd (0x338c bytes) 2024-09-21T15:49:17.196 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:49:17.197 INFO:teuthology.orchestra.run.smithi089.stdout:642 write 0x799b54e thru 0x79a4812 (0x92c5 bytes) 2024-09-21T15:49:17.197 INFO:teuthology.orchestra.run.smithi089.stdout:643 writesame 0xd2b659f thru 0xd2bd98e data_size 0x50(0x73f0 bytes) 2024-09-21T15:49:17.197 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:17.197 INFO:teuthology.orchestra.run.smithi089.stdout:645 read 0xae9573c thru 0xae9816d (0x2a32 bytes) 2024-09-21T15:49:17.197 INFO:teuthology.orchestra.run.smithi089.stdout:646 punch from 0xe71213 to 0xe78648, (0x7435 bytes) 2024-09-21T15:49:17.197 INFO:teuthology.orchestra.run.smithi089.stdout:648 trunc from 0xd2bd98f to 0x80e0d75 2024-09-21T15:49:17.197 INFO:teuthology.orchestra.run.smithi089.stdout:649 punch from 0x201b898 to 0x202abbc, (0xf324 bytes) 2024-09-21T15:49:17.197 INFO:teuthology.orchestra.run.smithi089.stdout:651 write 0x289ffa1 thru 0x28af79e (0xf7fe bytes) 2024-09-21T15:49:17.198 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:17.198 INFO:teuthology.orchestra.run.smithi089.stdout:656 read 0x122fa6f thru 0x1239b7c (0xa10e bytes) 2024-09-21T15:49:17.198 INFO:teuthology.orchestra.run.smithi089.stdout:657 punch from 0x5b43e48 to 0x5b4946d, (0x5625 bytes) 2024-09-21T15:49:17.198 INFO:teuthology.orchestra.run.smithi089.stdout:658 punch from 0x37c423f to 0x37d27fc, (0xe5bd bytes) 2024-09-21T15:49:17.198 INFO:teuthology.orchestra.run.smithi089.stdout:659 read 0x2dc9d30 thru 0x2dcbd3b (0x200c bytes) 2024-09-21T15:49:17.198 INFO:teuthology.orchestra.run.smithi089.stdout:660 punch from 0x6ffcdb3 to 0x700b781, (0xe9ce bytes) 2024-09-21T15:49:17.199 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:17.199 INFO:teuthology.orchestra.run.smithi089.stdout:663 read 0x10592f thru 0x1109ff (0xb0d1 bytes) 2024-09-21T15:49:17.199 INFO:teuthology.orchestra.run.smithi089.stdout:664 write 0x32cbe6e thru 0x32d71a5 (0xb338 bytes) 2024-09-21T15:49:17.217 INFO:teuthology.orchestra.run.smithi089.stdout:665 read 0x4e4a77c thru 0x4e53412 (0x8c97 bytes) 2024-09-21T15:49:17.217 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:17.216+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:17.225 INFO:teuthology.orchestra.run.smithi089.stdout:668 trunc from 0x80e0d75 to 0xf595c0 2024-09-21T15:49:17.269 INFO:teuthology.orchestra.run.smithi089.stdout:670 write 0xae1200c thru 0xae18d3f (0x6d34 bytes) 2024-09-21T15:49:17.280 INFO:teuthology.orchestra.run.smithi089.stdout:671 write 0x27579af thru 0x2761001 (0x9653 bytes) 2024-09-21T15:49:17.297 INFO:teuthology.orchestra.run.smithi089.stdout:672 read 0xaba2d13 thru 0xaba7e84 (0x5172 bytes) 2024-09-21T15:49:17.298 INFO:teuthology.orchestra.run.smithi089.stdout:673 punch from 0x2e18ae6 to 0x2e22e36, (0xa350 bytes) 2024-09-21T15:49:17.298 INFO:teuthology.orchestra.run.smithi089.stdout:674 write 0x42436f5 thru 0x42460b6 (0x29c2 bytes) 2024-09-21T15:49:17.309 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:17.308+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:17.311 INFO:teuthology.orchestra.run.smithi089.stdout:675 read 0x40fd19a thru 0x4102211 (0x5078 bytes) 2024-09-21T15:49:17.311 INFO:teuthology.orchestra.run.smithi089.stdout:676 write 0x82ca51d thru 0x82d270c (0x81f0 bytes) 2024-09-21T15:49:17.313 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:17.313 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:49:17.313 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:49:17.313 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:17.323 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:49:17.337 INFO:teuthology.orchestra.run.smithi089.stdout:677 trunc from 0xae18d40 to 0x1574af3 2024-09-21T15:49:17.363 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:17.363 INFO:teuthology.orchestra.run.smithi089.stdout:679 read 0xfeec7d thru 0xfeef08 (0x28c bytes) 2024-09-21T15:49:17.363 INFO:teuthology.orchestra.run.smithi089.stdout:680 read 0xaa1bfe thru 0xaa381e (0x1c21 bytes) 2024-09-21T15:49:17.363 INFO:teuthology.orchestra.run.smithi089.stdout:681 read 0x37787b thru 0x3837df (0xbf65 bytes) 2024-09-21T15:49:17.371 INFO:teuthology.orchestra.run.smithi089.stdout:683 write 0x25734fa thru 0x2581086 (0xdb8d bytes) 2024-09-21T15:49:17.385 INFO:teuthology.orchestra.run.smithi089.stdout:684 read 0x15a501a thru 0x15afcd2 (0xacb9 bytes) 2024-09-21T15:49:17.385 INFO:teuthology.orchestra.run.smithi089.stdout:685 punch from 0x11d2ba3 to 0x11d3534, (0x991 bytes) 2024-09-21T15:49:17.399 INFO:teuthology.orchestra.run.smithi089.stdout:687 write 0x91c1098 thru 0x91cbb0b (0xaa74 bytes) 2024-09-21T15:49:17.410 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:17.409+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:17.418 INFO:teuthology.orchestra.run.smithi089.stdout:688 writesame 0x892ff3 thru 0x897730 data_size 0xb(0x473e bytes) 2024-09-21T15:49:17.422 INFO:teuthology.orchestra.run.smithi089.stdout:689 read 0x166d667 thru 0x1670308 (0x2ca2 bytes) 2024-09-21T15:49:17.422 INFO:teuthology.orchestra.run.smithi089.stdout:690 read 0x7264b1f thru 0x72659cc (0xeae bytes) 2024-09-21T15:49:17.430 INFO:teuthology.orchestra.run.smithi089.stdout:691 writesame 0xbf05f46 thru 0xbf0cad0 data_size 0x45(0x6b8b bytes) 2024-09-21T15:49:17.430 INFO:teuthology.orchestra.run.smithi089.stdout:692 read 0xb126f45 thru 0xb13643f (0xf4fb bytes) 2024-09-21T15:49:17.430 INFO:teuthology.orchestra.run.smithi089.stdout:694 read 0x910a032 thru 0x9112708 (0x86d7 bytes) 2024-09-21T15:49:17.431 INFO:teuthology.orchestra.run.smithi089.stdout:700 read 0xbcf7db9 thru 0xbcfc051 (0x4299 bytes) 2024-09-21T15:49:17.431 INFO:teuthology.orchestra.run.smithi089.stdout:701 punch from 0x34251bf to 0x343454d, (0xf38e bytes) 2024-09-21T15:49:17.431 INFO:teuthology.orchestra.run.smithi089.stdout:702 read 0x2a7cc2 thru 0x2adee6 (0x6225 bytes) 2024-09-21T15:49:17.432 INFO:teuthology.orchestra.run.smithi089.stdout:703 write 0x3295cf8 thru 0x32a06be (0xa9c7 bytes) 2024-09-21T15:49:17.450 INFO:teuthology.orchestra.run.smithi089.stdout:704 write 0x9de133b thru 0x9df0451 (0xf117 bytes) 2024-09-21T15:49:17.452 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:17.452 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:49:17.452 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:49:17.452 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:49:17.452 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:17.462 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:49:17.468 INFO:teuthology.orchestra.run.smithi089.stdout:707 read 0xbc7393 thru 0xbd1f8b (0xabf9 bytes) 2024-09-21T15:49:17.473 INFO:teuthology.orchestra.run.smithi089.stdout:708 write 0xc4f63fe thru 0xc4fb7a0 (0x53a3 bytes) 2024-09-21T15:49:17.493 INFO:teuthology.orchestra.run.smithi089.stdout:709 write 0xedd2749 thru 0xedd5f9b (0x3853 bytes) 2024-09-21T15:49:17.502 INFO:teuthology.orchestra.run.smithi089.stdout:711 read 0x492d489 thru 0x493475b (0x72d3 bytes) 2024-09-21T15:49:17.502 INFO:teuthology.orchestra.run.smithi089.stdout:712 write 0x52e742d thru 0x52e94ba (0x208e bytes) 2024-09-21T15:49:17.509 INFO:teuthology.orchestra.run.smithi089.stdout:713 read 0xec3da25 thru 0xec403d9 (0x29b5 bytes) 2024-09-21T15:49:17.510 INFO:teuthology.orchestra.run.smithi089.stdout:714 read 0x4522064 thru 0x4524d70 (0x2d0d bytes) 2024-09-21T15:49:17.510 INFO:teuthology.orchestra.run.smithi089.stdout:715 read 0x99c6a0e thru 0x99cc9b9 (0x5fac bytes) 2024-09-21T15:49:17.510 INFO:teuthology.orchestra.run.smithi089.stdout:716 write 0xe396fca thru 0xe399eaf (0x2ee6 bytes) 2024-09-21T15:49:17.511 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:17.510+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:17.519 INFO:teuthology.orchestra.run.smithi089.stdout:718 writesame 0xa1c9fdb thru 0xa1d1863 data_size 0x1(0x7889 bytes) 2024-09-21T15:49:17.527 INFO:teuthology.orchestra.run.smithi089.stdout:722 write 0x9a3fe9f thru 0x9a48a7d (0x8bdf bytes) 2024-09-21T15:49:17.536 INFO:teuthology.orchestra.run.smithi089.stdout:723 punch from 0x7c8eec0 to 0x7c93df6, (0x4f36 bytes) 2024-09-21T15:49:17.537 INFO:teuthology.orchestra.run.smithi089.stdout:726 write 0xec5204d thru 0xec56dc3 (0x4d77 bytes) 2024-09-21T15:49:17.543 INFO:teuthology.orchestra.run.smithi089.stdout:727 read 0xd77a07b thru 0xd77c807 (0x278d bytes) 2024-09-21T15:49:17.543 INFO:teuthology.orchestra.run.smithi089.stdout:728 write 0xcf047db thru 0xcf12781 (0xdfa7 bytes) 2024-09-21T15:49:17.560 INFO:teuthology.orchestra.run.smithi089.stdout:730 read 0xcb9bf48 thru 0xcbaab85 (0xec3e bytes) 2024-09-21T15:49:17.560 INFO:teuthology.orchestra.run.smithi089.stdout:731 write 0xa7b2789 thru 0xa7c17da (0xf052 bytes) 2024-09-21T15:49:17.570 INFO:teuthology.orchestra.run.smithi089.stdout:732 write 0xe405707 thru 0xe4142f3 (0xebed bytes) 2024-09-21T15:49:17.582 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:17.582 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:49:17.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:49:17.582 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:49:17.582 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.582 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:5011 3.1e 3:7ceb6cf6:::rbd_data.1133f3dfbd6b.0000000000000001:5 [read 6299648~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:17.582 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:14.531179+0000", 2024-09-21T15:49:17.582 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.0486432350000001, 2024-09-21T15:49:17.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.041101397999999997, 2024-09-21T15:49:17.582 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:17.583 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:17.583 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:17.583 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:17.583 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:17.583 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 5011 2024-09-21T15:49:17.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.583 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:17.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:17.583 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.531179+0000", 2024-09-21T15:49:17.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:17.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.531179+0000", 2024-09-21T15:49:17.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:17.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.531179+0000", 2024-09-21T15:49:17.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:17.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:17.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.531181+0000", 2024-09-21T15:49:17.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.114e-06 2024-09-21T15:49:17.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:17.585 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.531187+0000", 2024-09-21T15:49:17.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7760000000000003e-06 2024-09-21T15:49:17.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.585 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:17.585 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.531193+0000", 2024-09-21T15:49:17.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3940000000000001e-06 2024-09-21T15:49:17.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.585 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:17.585 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.572182+0000", 2024-09-21T15:49:17.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.040989143999999998 2024-09-21T15:49:17.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.585 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:17.586 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.572228+0000", 2024-09-21T15:49:17.586 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5726999999999998e-05 2024-09-21T15:49:17.586 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.586 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:17.586 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.572281+0000", 2024-09-21T15:49:17.586 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2562000000000003e-05 2024-09-21T15:49:17.586 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.586 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:17.586 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.586 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.586 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:5023 3.1e 3:7ceb6cf6:::rbd_data.1133f3dfbd6b.0000000000000001:5 [read 6889472~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:17.586 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:14.533165+0000", 2024-09-21T15:49:17.586 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.0466576779999999, 2024-09-21T15:49:17.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.041105757, 2024-09-21T15:49:17.587 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:17.587 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:17.587 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:17.587 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:17.587 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:17.587 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 5023 2024-09-21T15:49:17.587 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.587 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:17.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.587 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:17.587 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.533165+0000", 2024-09-21T15:49:17.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.587 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.588 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:17.588 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.533165+0000", 2024-09-21T15:49:17.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.588 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.588 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:17.588 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.533164+0000", 2024-09-21T15:49:17.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:17.588 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.588 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:17.588 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.533167+0000", 2024-09-21T15:49:17.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2050000000000001e-06 2024-09-21T15:49:17.588 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.588 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:17.589 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.533176+0000", 2024-09-21T15:49:17.589 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4690000000000003e-06 2024-09-21T15:49:17.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.589 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.589 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:17.589 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.533181+0000", 2024-09-21T15:49:17.589 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5329999999999996e-06 2024-09-21T15:49:17.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.589 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.589 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:17.589 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.574170+0000", 2024-09-21T15:49:17.589 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.040989154 2024-09-21T15:49:17.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.589 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.589 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:17.590 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.574217+0000", 2024-09-21T15:49:17.590 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7261e-05 2024-09-21T15:49:17.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.590 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:17.590 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.574271+0000", 2024-09-21T15:49:17.590 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.346e-05 2024-09-21T15:49:17.590 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.590 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:17.590 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.590 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:5024 3.1e 3:7ceb6cf6:::rbd_data.1133f3dfbd6b.0000000000000001:5 [read 6938624~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:17.590 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:14.533185+0000", 2024-09-21T15:49:17.590 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.0466375440000002, 2024-09-21T15:49:17.591 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.041287839999999999, 2024-09-21T15:49:17.591 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:17.591 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:17.591 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:17.591 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:17.591 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:17.591 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 5024 2024-09-21T15:49:17.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.591 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:17.591 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.591 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:17.591 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.533185+0000", 2024-09-21T15:49:17.591 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.591 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.592 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:17.592 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.533185+0000", 2024-09-21T15:49:17.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.592 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:17.592 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.533185+0000", 2024-09-21T15:49:17.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:17.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.592 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:17.592 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.533187+0000", 2024-09-21T15:49:17.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.216e-06 2024-09-21T15:49:17.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.593 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:17.593 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.533196+0000", 2024-09-21T15:49:17.593 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2869999999999993e-06 2024-09-21T15:49:17.593 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.593 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.593 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:17.593 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.533204+0000", 2024-09-21T15:49:17.593 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9030000000000004e-06 2024-09-21T15:49:17.593 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.593 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.593 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:17.593 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.574302+0000", 2024-09-21T15:49:17.593 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.041097962000000002 2024-09-21T15:49:17.593 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.593 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.594 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:17.594 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.574384+0000", 2024-09-21T15:49:17.594 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1884000000000004e-05 2024-09-21T15:49:17.594 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.594 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.594 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:17.594 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.574473+0000", 2024-09-21T15:49:17.594 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8974000000000006e-05 2024-09-21T15:49:17.594 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.594 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:17.594 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.594 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.594 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.594 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:5025 3.1e 3:7ceb6cf6:::rbd_data.1133f3dfbd6b.0000000000000001:5 [read 6987776~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:17.594 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:14.533212+0000", 2024-09-21T15:49:17.595 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.0466100969999999, 2024-09-21T15:49:17.595 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.041390678, 2024-09-21T15:49:17.595 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:17.595 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:17.595 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:17.595 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:17.595 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:17.595 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 5025 2024-09-21T15:49:17.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.595 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:17.595 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.595 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:17.595 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.533212+0000", 2024-09-21T15:49:17.595 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:17.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.533212+0000", 2024-09-21T15:49:17.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:17.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.533212+0000", 2024-09-21T15:49:17.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:17.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:17.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.533214+0000", 2024-09-21T15:49:17.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1909999999999999e-06 2024-09-21T15:49:17.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.597 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.597 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:17.597 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.533222+0000", 2024-09-21T15:49:17.597 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9780000000000005e-06 2024-09-21T15:49:17.597 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.597 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.597 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:17.597 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.533228+0000", 2024-09-21T15:49:17.597 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6999999999999996e-06 2024-09-21T15:49:17.597 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.597 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.597 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:17.597 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.574502+0000", 2024-09-21T15:49:17.597 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.041273912000000003 2024-09-21T15:49:17.598 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.598 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.598 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:17.598 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.574550+0000", 2024-09-21T15:49:17.598 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7995999999999999e-05 2024-09-21T15:49:17.598 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.598 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.598 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:17.598 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.574603+0000", 2024-09-21T15:49:17.598 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3261999999999999e-05 2024-09-21T15:49:17.598 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.598 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:17.598 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.598 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.599 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.599 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:5035 3.1e 3:7ceb6cf6:::rbd_data.1133f3dfbd6b.0000000000000001:5 [read 7479296~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:17.599 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:14.535161+0000", 2024-09-21T15:49:17.599 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.0446613440000001, 2024-09-21T15:49:17.599 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.041116030999999997, 2024-09-21T15:49:17.599 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:17.599 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:17.599 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:17.599 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:17.599 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:17.599 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 5035 2024-09-21T15:49:17.599 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.599 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:17.599 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.599 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:17.600 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.535161+0000", 2024-09-21T15:49:17.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.600 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.600 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.600 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:17.600 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.535161+0000", 2024-09-21T15:49:17.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.600 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.600 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.600 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:17.600 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.535161+0000", 2024-09-21T15:49:17.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:49:17.600 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.600 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.600 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:17.601 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.535162+0000", 2024-09-21T15:49:17.601 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.144e-06 2024-09-21T15:49:17.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.601 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.601 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:17.601 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.535165+0000", 2024-09-21T15:49:17.601 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8629999999999999e-06 2024-09-21T15:49:17.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.601 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.601 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:17.601 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.535168+0000", 2024-09-21T15:49:17.601 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5969999999999999e-06 2024-09-21T15:49:17.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.601 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.601 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:17.602 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.576176+0000", 2024-09-21T15:49:17.602 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.041008398000000001 2024-09-21T15:49:17.602 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.602 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.602 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:17.602 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.576224+0000", 2024-09-21T15:49:17.602 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7831000000000003e-05 2024-09-21T15:49:17.602 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.602 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.602 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:17.602 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.576277+0000", 2024-09-21T15:49:17.602 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3400999999999999e-05 2024-09-21T15:49:17.602 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.602 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:17.602 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.603 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:5036 3.1e 3:7ceb6cf6:::rbd_data.1133f3dfbd6b.0000000000000001:5 [read 7528448~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:17.603 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:14.535170+0000", 2024-09-21T15:49:17.603 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.044652868, 2024-09-21T15:49:17.603 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.041308536999999999, 2024-09-21T15:49:17.603 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:17.603 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:17.603 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:17.603 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:17.603 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:17.603 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 5036 2024-09-21T15:49:17.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.603 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:17.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.604 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:17.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.535170+0000", 2024-09-21T15:49:17.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.604 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:17.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.535170+0000", 2024-09-21T15:49:17.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.604 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:17.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.535169+0000", 2024-09-21T15:49:17.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:49:17.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.605 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:17.605 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.535170+0000", 2024-09-21T15:49:17.605 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.094e-06 2024-09-21T15:49:17.605 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.605 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.605 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:17.605 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.535173+0000", 2024-09-21T15:49:17.605 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8059999999999999e-06 2024-09-21T15:49:17.605 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.605 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.605 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:17.605 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.535176+0000", 2024-09-21T15:49:17.605 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4320000000000002e-06 2024-09-21T15:49:17.605 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.605 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.606 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:17.606 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.576308+0000", 2024-09-21T15:49:17.606 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.041132113999999997 2024-09-21T15:49:17.606 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.606 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.606 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:17.606 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.576388+0000", 2024-09-21T15:49:17.606 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0507999999999996e-05 2024-09-21T15:49:17.606 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.606 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.606 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:17.606 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.576478+0000", 2024-09-21T15:49:17.606 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9789000000000004e-05 2024-09-21T15:49:17.606 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.606 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:17.607 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.607 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.607 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.607 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:5037 3.1e 3:7ceb6cf6:::rbd_data.1133f3dfbd6b.0000000000000001:5 [read 7577600~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:17.607 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:14.535178+0000", 2024-09-21T15:49:17.607 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.04464464, 2024-09-21T15:49:17.607 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.041428340000000001, 2024-09-21T15:49:17.607 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:17.607 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:17.607 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:17.607 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:17.607 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:17.607 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 5037 2024-09-21T15:49:17.607 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.607 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:17.608 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.608 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:17.608 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.535178+0000", 2024-09-21T15:49:17.608 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.608 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.608 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.608 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:17.608 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.535178+0000", 2024-09-21T15:49:17.608 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.608 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.608 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.608 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:17.608 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.535178+0000", 2024-09-21T15:49:17.608 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:49:17.608 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.609 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.609 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:17.609 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.535179+0000", 2024-09-21T15:49:17.609 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0809999999999999e-06 2024-09-21T15:49:17.609 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.609 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.609 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:17.609 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.535181+0000", 2024-09-21T15:49:17.609 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7630000000000001e-06 2024-09-21T15:49:17.609 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.609 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.609 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:17.609 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.535184+0000", 2024-09-21T15:49:17.609 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8490000000000002e-06 2024-09-21T15:49:17.609 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.609 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.610 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:17.610 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.576508+0000", 2024-09-21T15:49:17.610 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.041323341 2024-09-21T15:49:17.610 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.610 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.610 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:17.610 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.576552+0000", 2024-09-21T15:49:17.610 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4827000000000003e-05 2024-09-21T15:49:17.610 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.610 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.610 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:17.610 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.576606+0000", 2024-09-21T15:49:17.610 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3681999999999999e-05 2024-09-21T15:49:17.610 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.610 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:17.611 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.611 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.611 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.611 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:5048 3.1e 3:7ceb6cf6:::rbd_data.1133f3dfbd6b.0000000000000001:5 [read 8118272~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:17.611 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:14.537166+0000", 2024-09-21T15:49:17.611 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.0426566959999999, 2024-09-21T15:49:17.611 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.041115427000000003, 2024-09-21T15:49:17.611 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:17.611 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:17.611 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:17.611 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:17.611 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:17.611 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 5048 2024-09-21T15:49:17.611 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.611 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:17.612 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.612 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:17.612 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.537166+0000", 2024-09-21T15:49:17.612 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.612 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.612 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.612 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:17.612 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.537166+0000", 2024-09-21T15:49:17.612 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.612 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.612 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.612 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:17.612 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.537166+0000", 2024-09-21T15:49:17.612 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:49:17.612 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.612 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.613 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:17.613 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.537167+0000", 2024-09-21T15:49:17.613 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.229e-06 2024-09-21T15:49:17.613 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.613 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.613 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:17.613 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.537170+0000", 2024-09-21T15:49:17.613 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7630000000000001e-06 2024-09-21T15:49:17.613 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.613 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.613 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:17.613 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.537172+0000", 2024-09-21T15:49:17.613 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.418e-06 2024-09-21T15:49:17.613 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.613 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.614 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:17.614 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.578172+0000", 2024-09-21T15:49:17.614 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.041000270999999998 2024-09-21T15:49:17.614 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.614 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.614 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:17.614 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.578225+0000", 2024-09-21T15:49:17.614 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2475000000000002e-05 2024-09-21T15:49:17.614 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.614 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.614 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:17.614 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.578281+0000", 2024-09-21T15:49:17.614 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6471000000000002e-05 2024-09-21T15:49:17.614 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.614 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:17.615 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.615 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.615 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.615 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:5049 3.1e 3:7ceb6cf6:::rbd_data.1133f3dfbd6b.0000000000000001:5 [read 8167424~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:17.615 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:14.537174+0000", 2024-09-21T15:49:17.615 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.042648545, 2024-09-21T15:49:17.615 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.041238049999999998, 2024-09-21T15:49:17.615 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:17.615 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:17.615 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:17.615 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:17.615 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:17.615 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 5049 2024-09-21T15:49:17.615 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.616 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:17.616 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.616 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:17.616 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.537174+0000", 2024-09-21T15:49:17.616 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.616 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.616 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.616 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:17.616 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.537174+0000", 2024-09-21T15:49:17.616 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.616 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.616 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.616 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:17.616 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.537174+0000", 2024-09-21T15:49:17.616 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:49:17.616 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.617 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.617 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:17.617 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.537175+0000", 2024-09-21T15:49:17.617 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0139999999999999e-06 2024-09-21T15:49:17.617 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.617 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.617 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:17.617 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.537178+0000", 2024-09-21T15:49:17.617 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.791e-06 2024-09-21T15:49:17.617 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.617 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.617 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:17.617 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.537181+0000", 2024-09-21T15:49:17.617 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.489e-06 2024-09-21T15:49:17.617 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.618 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.618 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:17.618 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.578315+0000", 2024-09-21T15:49:17.618 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.041133540000000003 2024-09-21T15:49:17.618 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.618 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.618 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:17.618 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.578359+0000", 2024-09-21T15:49:17.618 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4561000000000002e-05 2024-09-21T15:49:17.618 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.618 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.618 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:17.618 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.578412+0000", 2024-09-21T15:49:17.618 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2856999999999999e-05 2024-09-21T15:49:17.618 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.619 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:17.619 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.619 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.619 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.619 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:5050 3.1e 3:7ceb6cf6:::rbd_data.1133f3dfbd6b.0000000000000001:5 [read 8216576~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:17.619 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:14.537183+0000", 2024-09-21T15:49:17.619 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.0426393150000002, 2024-09-21T15:49:17.619 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.041362471999999997, 2024-09-21T15:49:17.619 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:17.619 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:17.619 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:17.619 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:17.619 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:17.619 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 5050 2024-09-21T15:49:17.619 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.620 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:17.620 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.620 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:17.620 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.537183+0000", 2024-09-21T15:49:17.620 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.620 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.620 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.620 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:17.620 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.537183+0000", 2024-09-21T15:49:17.620 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.620 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.620 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.620 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:17.620 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.537183+0000", 2024-09-21T15:49:17.620 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:49:17.620 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.621 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.621 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:17.621 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.537184+0000", 2024-09-21T15:49:17.621 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1710000000000001e-06 2024-09-21T15:49:17.621 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.621 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.621 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:17.621 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.537187+0000", 2024-09-21T15:49:17.621 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7939999999999998e-06 2024-09-21T15:49:17.621 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.621 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.621 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:17.621 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.537189+0000", 2024-09-21T15:49:17.621 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.514e-06 2024-09-21T15:49:17.621 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.621 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.621 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:17.622 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.578438+0000", 2024-09-21T15:49:17.622 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.041248406000000001 2024-09-21T15:49:17.622 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.622 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.622 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:17.622 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.578489+0000", 2024-09-21T15:49:17.622 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1378000000000002e-05 2024-09-21T15:49:17.622 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.622 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.622 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:17.622 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.578546+0000", 2024-09-21T15:49:17.622 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6412000000000001e-05 2024-09-21T15:49:17.622 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.622 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:17.622 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.623 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.623 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.623 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:5051 3.1e 3:7ceb6cf6:::rbd_data.1133f3dfbd6b.0000000000000001:5 [read 8265728~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:17.623 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:14.537192+0000", 2024-09-21T15:49:17.623 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.0426304439999998, 2024-09-21T15:49:17.623 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.041486567000000002, 2024-09-21T15:49:17.623 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:17.623 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:17.623 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:17.623 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:17.623 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:17.623 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 5051 2024-09-21T15:49:17.623 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.623 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:17.623 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.623 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:17.624 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.537192+0000", 2024-09-21T15:49:17.624 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.624 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.624 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.624 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:17.624 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.537192+0000", 2024-09-21T15:49:17.624 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.624 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.624 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.624 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:17.624 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.537192+0000", 2024-09-21T15:49:17.624 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:49:17.624 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.624 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.624 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:17.624 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.537196+0000", 2024-09-21T15:49:17.625 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.726e-06 2024-09-21T15:49:17.625 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.625 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.625 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:17.625 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.537199+0000", 2024-09-21T15:49:17.625 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0599999999999999e-06 2024-09-21T15:49:17.625 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.625 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.625 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:17.625 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.537201+0000", 2024-09-21T15:49:17.625 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5009999999999999e-06 2024-09-21T15:49:17.625 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.625 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.625 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:17.625 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.578579+0000", 2024-09-21T15:49:17.626 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.041377619999999997 2024-09-21T15:49:17.626 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.626 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.626 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:17.626 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.578625+0000", 2024-09-21T15:49:17.626 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5806999999999997e-05 2024-09-21T15:49:17.626 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.626 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.626 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:17.626 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.578679+0000", 2024-09-21T15:49:17.626 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4054000000000001e-05 2024-09-21T15:49:17.626 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.626 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:17.626 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.627 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.627 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.627 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:5108 3.1e 3:7ceb6cf6:::rbd_data.1133f3dfbd6b.0000000000000001:5 [read 4681728~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:17.627 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:14.539142+0000", 2024-09-21T15:49:17.627 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.0406807320000002, 2024-09-21T15:49:17.627 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.041181875999999999, 2024-09-21T15:49:17.627 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:17.627 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:17.627 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:17.627 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:17.627 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:17.627 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 5108 2024-09-21T15:49:17.627 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.627 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:17.627 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.627 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:17.628 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.539142+0000", 2024-09-21T15:49:17.628 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.628 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.628 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.628 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:17.628 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.539142+0000", 2024-09-21T15:49:17.628 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.628 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.628 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.628 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:17.628 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.539141+0000", 2024-09-21T15:49:17.628 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:17.628 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.628 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.628 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:17.628 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.539150+0000", 2024-09-21T15:49:17.629 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1720000000000007e-06 2024-09-21T15:49:17.629 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.629 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.629 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:17.629 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.539155+0000", 2024-09-21T15:49:17.629 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7019999999999998e-06 2024-09-21T15:49:17.629 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.629 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.629 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:17.629 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.539160+0000", 2024-09-21T15:49:17.629 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9949999999999996e-06 2024-09-21T15:49:17.629 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.629 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.629 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:17.629 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.580221+0000", 2024-09-21T15:49:17.629 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.041061001 2024-09-21T15:49:17.630 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.630 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.630 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:17.630 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.580271+0000", 2024-09-21T15:49:17.630 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9747000000000003e-05 2024-09-21T15:49:17.630 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.630 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.630 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:17.630 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.580324+0000", 2024-09-21T15:49:17.630 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2608999999999997e-05 2024-09-21T15:49:17.630 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.630 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:17.630 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.630 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.630 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.631 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:5109 3.1e 3:7ceb6cf6:::rbd_data.1133f3dfbd6b.0000000000000001:5 [read 4730880~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:17.631 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:14.539165+0000", 2024-09-21T15:49:17.631 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.0406578770000001, 2024-09-21T15:49:17.631 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.041275898999999998, 2024-09-21T15:49:17.631 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:17.631 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:17.631 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:17.631 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:17.631 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:17.631 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 5109 2024-09-21T15:49:17.631 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.631 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:17.631 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.631 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:17.631 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.539165+0000", 2024-09-21T15:49:17.631 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.632 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.632 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.632 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:17.632 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.539165+0000", 2024-09-21T15:49:17.632 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.632 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.632 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.632 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:17.632 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.539164+0000", 2024-09-21T15:49:17.632 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:17.632 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.632 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.632 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:17.632 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.539166+0000", 2024-09-21T15:49:17.633 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.058e-06 2024-09-21T15:49:17.633 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.633 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.633 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:17.633 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.539172+0000", 2024-09-21T15:49:17.633 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5690000000000004e-06 2024-09-21T15:49:17.633 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.633 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.633 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:17.633 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.539181+0000", 2024-09-21T15:49:17.633 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4010000000000003e-06 2024-09-21T15:49:17.633 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.633 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.633 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:17.634 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.580344+0000", 2024-09-21T15:49:17.634 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.041162404999999999 2024-09-21T15:49:17.634 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.634 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.634 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:17.634 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.580388+0000", 2024-09-21T15:49:17.634 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4716999999999999e-05 2024-09-21T15:49:17.634 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.634 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.634 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:17.634 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.580440+0000", 2024-09-21T15:49:17.634 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2099999999999999e-05 2024-09-21T15:49:17.634 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.634 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:17.635 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.635 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.635 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.635 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:5120 3.1e 3:7ceb6cf6:::rbd_data.1133f3dfbd6b.0000000000000001:5 [read 5271552~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:17.635 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:14.541095+0000", 2024-09-21T15:49:17.635 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.0387272209999998, 2024-09-21T15:49:17.635 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.041235282999999998, 2024-09-21T15:49:17.635 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:17.635 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:17.635 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:17.635 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:17.635 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:17.635 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 5120 2024-09-21T15:49:17.635 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.636 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:17.636 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.636 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:17.636 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.541095+0000", 2024-09-21T15:49:17.636 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.636 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.636 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.636 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:17.636 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.541095+0000", 2024-09-21T15:49:17.636 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.636 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.636 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.636 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:17.636 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.541095+0000", 2024-09-21T15:49:17.636 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:49:17.637 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.637 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.637 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:17.637 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.541096+0000", 2024-09-21T15:49:17.637 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2190000000000001e-06 2024-09-21T15:49:17.637 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.637 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.637 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:17.637 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.541099+0000", 2024-09-21T15:49:17.637 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8669999999999998e-06 2024-09-21T15:49:17.637 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.637 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.637 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:17.637 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.541101+0000", 2024-09-21T15:49:17.637 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.26e-06 2024-09-21T15:49:17.638 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.638 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.638 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:17.638 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.582154+0000", 2024-09-21T15:49:17.638 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.041053061000000002 2024-09-21T15:49:17.638 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.638 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.638 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:17.638 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.582232+0000", 2024-09-21T15:49:17.638 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7723000000000001e-05 2024-09-21T15:49:17.638 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.638 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.638 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:17.638 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.582331+0000", 2024-09-21T15:49:17.639 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.8357999999999996e-05 2024-09-21T15:49:17.639 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.639 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:17.639 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.639 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.639 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.639 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:5121 3.1e 3:7ceb6cf6:::rbd_data.1133f3dfbd6b.0000000000000001:5 [read 5320704~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:17.639 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:14.541104+0000", 2024-09-21T15:49:17.639 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.0387188119999999, 2024-09-21T15:49:17.639 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.041465986000000003, 2024-09-21T15:49:17.639 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:17.639 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:17.639 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:17.639 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:17.639 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:17.640 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 5121 2024-09-21T15:49:17.640 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.640 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:17.640 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.640 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:17.640 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.541104+0000", 2024-09-21T15:49:17.640 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.640 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.640 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.640 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:17.640 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.541104+0000", 2024-09-21T15:49:17.640 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.640 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.640 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.640 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:17.641 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.541103+0000", 2024-09-21T15:49:17.641 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:49:17.641 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.641 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.641 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:17.641 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.541108+0000", 2024-09-21T15:49:17.641 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4309999999999996e-06 2024-09-21T15:49:17.641 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.641 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.641 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:17.641 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.541111+0000", 2024-09-21T15:49:17.641 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1209999999999998e-06 2024-09-21T15:49:17.641 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.641 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.641 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:17.642 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.541113+0000", 2024-09-21T15:49:17.642 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4789999999999999e-06 2024-09-21T15:49:17.642 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.642 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.642 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:17.642 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.582390+0000", 2024-09-21T15:49:17.642 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.041276582999999999 2024-09-21T15:49:17.642 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.642 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.642 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:17.642 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.582476+0000", 2024-09-21T15:49:17.642 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5649000000000006e-05 2024-09-21T15:49:17.642 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.642 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.642 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:17.643 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.582570+0000", 2024-09-21T15:49:17.643 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3927999999999999e-05 2024-09-21T15:49:17.643 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.643 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:17.643 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.643 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.643 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.643 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:5122 3.1e 3:7ceb6cf6:::rbd_data.1133f3dfbd6b.0000000000000001:5 [read 5369856~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:17.643 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:14.541116+0000", 2024-09-21T15:49:17.643 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.0387062130000002, 2024-09-21T15:49:17.643 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.041631385999999999, 2024-09-21T15:49:17.643 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:17.643 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:17.643 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:17.644 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:17.644 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:17.644 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 5122 2024-09-21T15:49:17.644 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.644 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:17.644 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.644 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:17.644 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.541116+0000", 2024-09-21T15:49:17.644 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.644 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.644 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.644 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:17.644 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.541116+0000", 2024-09-21T15:49:17.644 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.644 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.644 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.645 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:17.645 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.541116+0000", 2024-09-21T15:49:17.645 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:49:17.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.645 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.645 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:17.645 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.541117+0000", 2024-09-21T15:49:17.645 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1799999999999999e-06 2024-09-21T15:49:17.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.645 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.645 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:17.645 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.541121+0000", 2024-09-21T15:49:17.645 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3569999999999998e-06 2024-09-21T15:49:17.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.645 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.645 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:17.646 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.541123+0000", 2024-09-21T15:49:17.646 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7700000000000002e-06 2024-09-21T15:49:17.646 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.646 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.646 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:17.646 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.582608+0000", 2024-09-21T15:49:17.646 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.041484200999999998 2024-09-21T15:49:17.646 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.646 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.646 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:17.646 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.582673+0000", 2024-09-21T15:49:17.646 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5505000000000004e-05 2024-09-21T15:49:17.646 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.646 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.646 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:17.647 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.582748+0000", 2024-09-21T15:49:17.647 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4591999999999993e-05 2024-09-21T15:49:17.647 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.647 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:17.647 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.647 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.647 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.647 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:5134 3.1e 3:7ceb6cf6:::rbd_data.1133f3dfbd6b.0000000000000001:5 [read 5959680~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:17.647 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:14.543133+0000", 2024-09-21T15:49:17.647 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.036689821, 2024-09-21T15:49:17.647 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.041154027000000003, 2024-09-21T15:49:17.648 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:17.648 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:17.648 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:17.648 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:17.648 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:17.648 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 5134 2024-09-21T15:49:17.648 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.648 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:17.648 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.648 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:17.648 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.543133+0000", 2024-09-21T15:49:17.648 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.648 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.648 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.648 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:17.649 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.543133+0000", 2024-09-21T15:49:17.649 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.649 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.649 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.649 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:17.649 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.543132+0000", 2024-09-21T15:49:17.649 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:49:17.649 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.649 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.649 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:17.649 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.543134+0000", 2024-09-21T15:49:17.649 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.173e-06 2024-09-21T15:49:17.649 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.649 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.649 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:17.650 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.543136+0000", 2024-09-21T15:49:17.650 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.819e-06 2024-09-21T15:49:17.650 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.650 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.650 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:17.650 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.543139+0000", 2024-09-21T15:49:17.650 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5809999999999999e-06 2024-09-21T15:49:17.650 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.650 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.650 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:17.650 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.584192+0000", 2024-09-21T15:49:17.650 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.041052666000000002 2024-09-21T15:49:17.650 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.650 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.650 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:17.651 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.584236+0000", 2024-09-21T15:49:17.651 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4088000000000002e-05 2024-09-21T15:49:17.651 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.651 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.651 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:17.651 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.584287+0000", 2024-09-21T15:49:17.651 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0909999999999999e-05 2024-09-21T15:49:17.651 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.651 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:17.651 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.651 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.651 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.651 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:5135 3.1e 3:7ceb6cf6:::rbd_data.1133f3dfbd6b.0000000000000001:5 [read 6008832~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:17.651 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:14.543141+0000", 2024-09-21T15:49:17.652 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.0366814820000001, 2024-09-21T15:49:17.652 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.041270930999999997, 2024-09-21T15:49:17.652 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:17.652 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:17.652 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:17.652 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:17.652 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:17.652 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 5135 2024-09-21T15:49:17.652 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.652 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:17.652 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.652 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:17.652 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.543141+0000", 2024-09-21T15:49:17.652 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.652 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.652 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.653 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:17.653 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.543141+0000", 2024-09-21T15:49:17.653 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.653 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.653 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.653 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:17.653 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.543141+0000", 2024-09-21T15:49:17.653 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:49:17.653 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.653 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.653 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:17.653 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.543142+0000", 2024-09-21T15:49:17.653 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.158e-06 2024-09-21T15:49:17.653 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.654 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.654 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:17.654 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.543145+0000", 2024-09-21T15:49:17.654 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7669999999999999e-06 2024-09-21T15:49:17.654 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.654 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.654 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:17.654 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.543147+0000", 2024-09-21T15:49:17.654 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.384e-06 2024-09-21T15:49:17.654 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.654 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.654 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:17.654 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.584321+0000", 2024-09-21T15:49:17.654 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.041173500000000002 2024-09-21T15:49:17.655 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.655 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.655 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:17.655 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.584363+0000", 2024-09-21T15:49:17.655 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2911999999999998e-05 2024-09-21T15:49:17.655 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.655 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.655 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:17.655 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.584412+0000", 2024-09-21T15:49:17.655 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8415000000000003e-05 2024-09-21T15:49:17.655 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.655 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:17.655 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.655 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.656 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.656 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:5136 3.1e 3:7ceb6cf6:::rbd_data.1133f3dfbd6b.0000000000000001:5 [read 6057984~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:17.656 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:14.543149+0000", 2024-09-21T15:49:17.656 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.0366733560000001, 2024-09-21T15:49:17.656 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.041387446000000001, 2024-09-21T15:49:17.656 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:17.656 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:17.656 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:17.656 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:17.656 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:17.656 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 5136 2024-09-21T15:49:17.656 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.656 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:17.656 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.656 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:17.657 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.543149+0000", 2024-09-21T15:49:17.657 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.657 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.657 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.657 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:17.657 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.543149+0000", 2024-09-21T15:49:17.657 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.657 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.657 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.657 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:17.657 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.543149+0000", 2024-09-21T15:49:17.657 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:49:17.657 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.657 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.657 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:17.657 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.543150+0000", 2024-09-21T15:49:17.658 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3260000000000001e-06 2024-09-21T15:49:17.658 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.658 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.658 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:17.658 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.543153+0000", 2024-09-21T15:49:17.658 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7310000000000002e-06 2024-09-21T15:49:17.658 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.658 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.658 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:17.658 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.543155+0000", 2024-09-21T15:49:17.658 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.525e-06 2024-09-21T15:49:17.658 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.658 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.659 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:17.659 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.584436+0000", 2024-09-21T15:49:17.659 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.041280150000000002 2024-09-21T15:49:17.659 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.659 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.659 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:17.659 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.584479+0000", 2024-09-21T15:49:17.659 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3353999999999999e-05 2024-09-21T15:49:17.659 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.659 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.659 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:17.659 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.584537+0000", 2024-09-21T15:49:17.659 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7561e-05 2024-09-21T15:49:17.660 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.660 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:17.660 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.660 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.660 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.660 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:5137 3.1e 3:7ceb6cf6:::rbd_data.1133f3dfbd6b.0000000000000001:5 [read 6107136~4096] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:17.660 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:14.543158+0000", 2024-09-21T15:49:17.660 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.0366648120000002, 2024-09-21T15:49:17.660 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.041507038000000003, 2024-09-21T15:49:17.660 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:17.660 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:17.660 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:17.660 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:17.660 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:17.660 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 5137 2024-09-21T15:49:17.661 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.661 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:17.661 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.661 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:17.661 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.543158+0000", 2024-09-21T15:49:17.661 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.661 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.661 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.661 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:17.661 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.543158+0000", 2024-09-21T15:49:17.661 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:17.661 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.661 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.661 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:17.661 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.543157+0000", 2024-09-21T15:49:17.662 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:49:17.662 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.662 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.662 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:17.662 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.543159+0000", 2024-09-21T15:49:17.662 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.221e-06 2024-09-21T15:49:17.662 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.662 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.662 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:17.662 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.543161+0000", 2024-09-21T15:49:17.662 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.79e-06 2024-09-21T15:49:17.662 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.662 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.662 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:17.662 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.543164+0000", 2024-09-21T15:49:17.663 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4830000000000002e-06 2024-09-21T15:49:17.663 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.663 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.663 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:17.663 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.584566+0000", 2024-09-21T15:49:17.663 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.041402225000000001 2024-09-21T15:49:17.663 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.663 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.663 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:17.663 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.584610+0000", 2024-09-21T15:49:17.663 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4027999999999999e-05 2024-09-21T15:49:17.663 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:17.663 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:17.663 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:17.663 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.584665+0000", 2024-09-21T15:49:17.664 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4496000000000001e-05 2024-09-21T15:49:17.664 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.664 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:17.664 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.664 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:17.664 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:17.664 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:17.665 INFO:teuthology.orchestra.run.smithi089.stdout:734 read 0x1c84c37 thru 0x1c949ce (0xfd98 bytes) 2024-09-21T15:49:17.665 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:17.665 INFO:teuthology.orchestra.run.smithi089.stdout:736 write 0xb991211 thru 0xb99d714 (0xc504 bytes) 2024-09-21T15:49:17.666 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:49:17.667 INFO:teuthology.orchestra.run.smithi089.stdout:737 write 0x2b72d85 thru 0x2b80c86 (0xdf02 bytes) 2024-09-21T15:49:17.667 INFO:teuthology.orchestra.run.smithi089.stdout:738 trunc from 0xedd5f9c to 0xcf8af67 2024-09-21T15:49:17.667 INFO:teuthology.orchestra.run.smithi089.stdout:740 read 0x1e1beaa thru 0x1e2704c (0xb1a3 bytes) 2024-09-21T15:49:17.667 INFO:teuthology.orchestra.run.smithi089.stdout:742 read 0xb1f143f thru 0xb1fa2b4 (0x8e76 bytes) 2024-09-21T15:49:17.667 INFO:teuthology.orchestra.run.smithi089.stdout:743 trunc from 0xcf8af67 to 0x9633f44 2024-09-21T15:49:17.667 INFO:teuthology.orchestra.run.smithi089.stderr:set 3.8 pg_upmap_items mapping to [1->6,4->2] 2024-09-21T15:49:17.668 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:17.664+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:17.695 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:17.695 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:17.695 INFO:teuthology.orchestra.run.smithi089.stdout:747 write 0x383ca7e thru 0x383d31b (0x89e bytes) 2024-09-21T15:49:17.735 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T15:49:17.771 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:17.771 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:17.771 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:49:17.771 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:17.779 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:17.778+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:17.780 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:49:17.867 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:17.866+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:17.882 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:17.882 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:17.882 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:49:17.882 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:49:17.882 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:17.891 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:49:17.967 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:17.966+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:18.036 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:18.036 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:49:18.036 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:49:18.036 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:49:18.036 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.036 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:13177 3.14 3:2ac81419:::rbd_object_map.1133dc5adfdf:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:18.036 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.021676+0000", 2024-09-21T15:49:18.036 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 1.011843321, 2024-09-21T15:49:18.036 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.011122191999999999, 2024-09-21T15:49:18.036 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:18.037 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:18.037 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:18.037 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:18.037 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:18.037 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 13177 2024-09-21T15:49:18.037 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.037 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:18.037 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.037 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:18.037 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.021676+0000", 2024-09-21T15:49:18.037 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.037 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.037 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.037 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:18.037 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.021676+0000", 2024-09-21T15:49:18.038 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.038 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.038 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.038 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:18.038 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.021676+0000", 2024-09-21T15:49:18.038 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:18.038 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.038 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.038 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:18.038 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.021678+0000", 2024-09-21T15:49:18.038 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7049999999999999e-06 2024-09-21T15:49:18.038 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.038 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.039 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:18.039 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.021682+0000", 2024-09-21T15:49:18.039 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2940000000000001e-06 2024-09-21T15:49:18.039 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.039 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.039 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:18.039 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.021684+0000", 2024-09-21T15:49:18.039 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4700000000000001e-06 2024-09-21T15:49:18.039 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.039 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.039 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:18.039 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.026832+0000", 2024-09-21T15:49:18.039 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0051477049999999998 2024-09-21T15:49:18.039 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.040 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.040 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:18.040 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.026882+0000", 2024-09-21T15:49:18.040 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9744999999999999e-05 2024-09-21T15:49:18.040 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.040 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.040 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 7", 2024-09-21T15:49:18.040 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.027141+0000", 2024-09-21T15:49:18.040 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00025932700000000001 2024-09-21T15:49:18.040 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.040 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.040 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:18.040 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.032665+0000", 2024-09-21T15:49:18.040 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.005523788 2024-09-21T15:49:18.041 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.041 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.041 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:18.041 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.032744+0000", 2024-09-21T15:49:18.041 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9458999999999999e-05 2024-09-21T15:49:18.041 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.041 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.041 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:18.041 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.032777+0000", 2024-09-21T15:49:18.041 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2934e-05 2024-09-21T15:49:18.041 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.041 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.041 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:18.041 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.032798+0000", 2024-09-21T15:49:18.041 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1026000000000002e-05 2024-09-21T15:49:18.042 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.042 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:18.042 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.042 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.042 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.042 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:13178 3.14 3:2ac81419:::rbd_object_map.1133dc5adfdf:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:18.042 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.021686+0000", 2024-09-21T15:49:18.042 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 1.011832917, 2024-09-21T15:49:18.042 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.011158009999999999, 2024-09-21T15:49:18.042 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:18.042 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:18.042 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:18.042 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:18.043 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:18.043 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 13178 2024-09-21T15:49:18.043 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.043 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:18.043 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.043 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:18.043 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.021686+0000", 2024-09-21T15:49:18.043 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.043 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.043 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.043 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:18.043 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.021686+0000", 2024-09-21T15:49:18.043 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.043 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.044 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.044 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:18.044 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.021686+0000", 2024-09-21T15:49:18.044 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:18.044 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.044 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.044 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:18.044 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.021688+0000", 2024-09-21T15:49:18.044 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5379999999999999e-06 2024-09-21T15:49:18.044 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.044 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.044 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:18.044 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.021691+0000", 2024-09-21T15:49:18.044 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8589999999999998e-06 2024-09-21T15:49:18.044 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.044 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.044 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:18.045 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.021694+0000", 2024-09-21T15:49:18.045 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4650000000000001e-06 2024-09-21T15:49:18.045 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.045 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.045 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:18.045 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.027532+0000", 2024-09-21T15:49:18.045 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.005837732 2024-09-21T15:49:18.045 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.045 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.045 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:18.045 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.027617+0000", 2024-09-21T15:49:18.045 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5716000000000004e-05 2024-09-21T15:49:18.045 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.045 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.045 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 7", 2024-09-21T15:49:18.045 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.027847+0000", 2024-09-21T15:49:18.046 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00022928600000000001 2024-09-21T15:49:18.046 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.046 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.046 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:18.046 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.032696+0000", 2024-09-21T15:49:18.046 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0048494560000000003 2024-09-21T15:49:18.046 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.046 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.046 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:18.046 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.032805+0000", 2024-09-21T15:49:18.046 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010877100000000001 2024-09-21T15:49:18.046 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.046 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.046 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:18.046 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.032827+0000", 2024-09-21T15:49:18.047 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1676e-05 2024-09-21T15:49:18.047 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.047 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.047 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:18.047 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.032844+0000", 2024-09-21T15:49:18.047 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7744000000000002e-05 2024-09-21T15:49:18.047 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.047 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:18.047 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.047 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.047 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.047 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:13179 3.14 3:2ac81419:::rbd_object_map.1133dc5adfdf:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:18.047 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.021699+0000", 2024-09-21T15:49:18.047 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 1.0118199699999999, 2024-09-21T15:49:18.047 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.011187397, 2024-09-21T15:49:18.047 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:18.048 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:18.048 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:18.048 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:18.048 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:18.048 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 13179 2024-09-21T15:49:18.048 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.048 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:18.048 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.048 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:18.048 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.021699+0000", 2024-09-21T15:49:18.048 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.048 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.049 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.049 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:18.049 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.021699+0000", 2024-09-21T15:49:18.049 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.049 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.049 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.049 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:18.049 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.021699+0000", 2024-09-21T15:49:18.049 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:18.049 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.049 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.049 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:18.049 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.021701+0000", 2024-09-21T15:49:18.049 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8360000000000001e-06 2024-09-21T15:49:18.049 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.050 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.050 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:18.050 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.021704+0000", 2024-09-21T15:49:18.050 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2200000000000001e-06 2024-09-21T15:49:18.050 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.050 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.050 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:18.050 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.021706+0000", 2024-09-21T15:49:18.050 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4370000000000001e-06 2024-09-21T15:49:18.050 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.050 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.050 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:18.050 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.028135+0000", 2024-09-21T15:49:18.050 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0064284800000000003 2024-09-21T15:49:18.050 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.051 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.051 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:18.051 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.028183+0000", 2024-09-21T15:49:18.051 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8044999999999998e-05 2024-09-21T15:49:18.051 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.051 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.051 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 7", 2024-09-21T15:49:18.051 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.028383+0000", 2024-09-21T15:49:18.051 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000199693 2024-09-21T15:49:18.051 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.051 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.051 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:18.051 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.032729+0000", 2024-09-21T15:49:18.051 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0043464619999999997 2024-09-21T15:49:18.052 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.052 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.052 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:18.052 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.032849+0000", 2024-09-21T15:49:18.052 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000120314 2024-09-21T15:49:18.052 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.052 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.052 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:18.052 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.032871+0000", 2024-09-21T15:49:18.052 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1212e-05 2024-09-21T15:49:18.052 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.052 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.052 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:18.052 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.032887+0000", 2024-09-21T15:49:18.052 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5954999999999999e-05 2024-09-21T15:49:18.053 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.053 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:18.053 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.053 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.053 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.053 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:13180 3.14 3:2ac81419:::rbd_object_map.1133dc5adfdf:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:18.053 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.021729+0000", 2024-09-21T15:49:18.053 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 1.0117900440000001, 2024-09-21T15:49:18.053 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.011284733999999999, 2024-09-21T15:49:18.053 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:18.053 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:18.053 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:18.053 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:18.053 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:18.054 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 13180 2024-09-21T15:49:18.054 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.054 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:18.054 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.054 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:18.054 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.021729+0000", 2024-09-21T15:49:18.054 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.054 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.054 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.054 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:18.054 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.021729+0000", 2024-09-21T15:49:18.054 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.054 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.054 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.055 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:18.055 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.021729+0000", 2024-09-21T15:49:18.055 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:18.055 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.055 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.055 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:18.055 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.021731+0000", 2024-09-21T15:49:18.055 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6639999999999998e-06 2024-09-21T15:49:18.055 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.055 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.055 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:18.055 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.021739+0000", 2024-09-21T15:49:18.055 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1840000000000002e-06 2024-09-21T15:49:18.055 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.056 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.056 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:18.056 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.021741+0000", 2024-09-21T15:49:18.056 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6819999999999999e-06 2024-09-21T15:49:18.056 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.056 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.056 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:18.056 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.028628+0000", 2024-09-21T15:49:18.056 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0068870930000000004 2024-09-21T15:49:18.056 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.056 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.056 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:18.056 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.028676+0000", 2024-09-21T15:49:18.056 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7212999999999997e-05 2024-09-21T15:49:18.056 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.056 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.057 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 7", 2024-09-21T15:49:18.057 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.028872+0000", 2024-09-21T15:49:18.057 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00019620400000000001 2024-09-21T15:49:18.057 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.057 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.057 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:18.057 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.032893+0000", 2024-09-21T15:49:18.057 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0040208049999999997 2024-09-21T15:49:18.057 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.057 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.057 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:18.057 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.032925+0000", 2024-09-21T15:49:18.057 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.239e-05 2024-09-21T15:49:18.057 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.058 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.058 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:18.058 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.032969+0000", 2024-09-21T15:49:18.058 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4388999999999998e-05 2024-09-21T15:49:18.058 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.058 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.058 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:18.058 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.033014+0000", 2024-09-21T15:49:18.058 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4505000000000002e-05 2024-09-21T15:49:18.058 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.058 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:18.058 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.058 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.059 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.059 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:13200 3.14 3:2ac81419:::rbd_object_map.1133dc5adfdf:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:18.059 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.036020+0000", 2024-09-21T15:49:18.059 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.99749965900000004, 2024-09-21T15:49:18.059 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.011120329999999999, 2024-09-21T15:49:18.059 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:18.059 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:18.059 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:18.059 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:18.059 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:18.059 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 13200 2024-09-21T15:49:18.059 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.059 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:18.059 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.059 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:18.060 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.036020+0000", 2024-09-21T15:49:18.060 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.060 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.060 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.060 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:18.060 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.036020+0000", 2024-09-21T15:49:18.060 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.060 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.060 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.060 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:18.060 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.036019+0000", 2024-09-21T15:49:18.060 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:18.060 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.060 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.060 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:18.061 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.036021+0000", 2024-09-21T15:49:18.061 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6700000000000001e-06 2024-09-21T15:49:18.061 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.061 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.061 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:18.061 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.036024+0000", 2024-09-21T15:49:18.061 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.191e-06 2024-09-21T15:49:18.061 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.061 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.061 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:18.061 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.036027+0000", 2024-09-21T15:49:18.061 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9890000000000001e-06 2024-09-21T15:49:18.061 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.061 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.061 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:18.062 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.040058+0000", 2024-09-21T15:49:18.062 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.004030662 2024-09-21T15:49:18.062 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.062 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.062 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:18.062 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.040143+0000", 2024-09-21T15:49:18.062 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4790999999999995e-05 2024-09-21T15:49:18.062 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.062 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.062 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 7", 2024-09-21T15:49:18.062 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.040510+0000", 2024-09-21T15:49:18.062 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000367783 2024-09-21T15:49:18.062 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.062 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.063 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:18.063 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.042146+0000", 2024-09-21T15:49:18.063 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0016358519999999999 2024-09-21T15:49:18.063 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.063 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.063 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:18.063 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.047062+0000", 2024-09-21T15:49:18.063 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0049156249999999999 2024-09-21T15:49:18.063 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.063 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.063 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:18.063 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.047104+0000", 2024-09-21T15:49:18.063 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1938999999999999e-05 2024-09-21T15:49:18.063 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.063 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.064 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:18.064 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.047140+0000", 2024-09-21T15:49:18.064 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6047999999999998e-05 2024-09-21T15:49:18.064 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.064 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:18.064 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.064 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.064 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.064 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:13501 3.14 3:2ac81419:::rbd_object_map.1133dc5adfdf:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:18.064 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.280547+0000", 2024-09-21T15:49:18.064 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.75297262499999995, 2024-09-21T15:49:18.064 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.012090160000000001, 2024-09-21T15:49:18.064 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:18.064 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:18.064 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:18.064 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:18.065 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:18.065 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 13501 2024-09-21T15:49:18.065 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.065 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:18.065 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.065 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:18.065 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.280547+0000", 2024-09-21T15:49:18.065 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.065 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.065 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.065 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:18.065 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.280547+0000", 2024-09-21T15:49:18.065 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.065 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.065 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.066 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:18.066 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.280546+0000", 2024-09-21T15:49:18.066 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:18.066 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.066 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.066 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:18.066 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.280548+0000", 2024-09-21T15:49:18.066 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5269999999999999e-06 2024-09-21T15:49:18.066 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.066 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.066 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:18.066 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.280551+0000", 2024-09-21T15:49:18.066 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9340000000000002e-06 2024-09-21T15:49:18.066 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.066 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.067 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:18.067 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.280553+0000", 2024-09-21T15:49:18.067 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3609999999999999e-06 2024-09-21T15:49:18.067 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.067 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.067 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:18.067 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.284651+0000", 2024-09-21T15:49:18.067 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.004097458 2024-09-21T15:49:18.067 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.067 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.067 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:18.067 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.284739+0000", 2024-09-21T15:49:18.067 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8205e-05 2024-09-21T15:49:18.067 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.067 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.068 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 7", 2024-09-21T15:49:18.068 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.284952+0000", 2024-09-21T15:49:18.068 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00021316 2024-09-21T15:49:18.068 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.068 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.068 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:18.068 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.290034+0000", 2024-09-21T15:49:18.068 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0050816230000000004 2024-09-21T15:49:18.068 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.068 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.068 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:18.068 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.292550+0000", 2024-09-21T15:49:18.068 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0025168489999999998 2024-09-21T15:49:18.068 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.068 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.068 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:18.069 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.292599+0000", 2024-09-21T15:49:18.069 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8959000000000003e-05 2024-09-21T15:49:18.069 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.069 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.069 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:18.069 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.292637+0000", 2024-09-21T15:49:18.069 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7301000000000001e-05 2024-09-21T15:49:18.069 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.069 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:18.069 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.069 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.069 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.069 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:13502 3.14 3:2ac81419:::rbd_object_map.1133dc5adfdf:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:18.069 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.280555+0000", 2024-09-21T15:49:18.070 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.75296392099999998, 2024-09-21T15:49:18.070 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.012186236, 2024-09-21T15:49:18.070 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:18.070 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:18.070 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:18.070 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:18.070 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:18.070 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 13502 2024-09-21T15:49:18.070 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.070 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:18.070 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.070 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:18.070 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.280555+0000", 2024-09-21T15:49:18.070 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.070 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.070 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.071 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:18.071 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.280555+0000", 2024-09-21T15:49:18.071 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.071 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.071 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.071 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:18.071 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.280555+0000", 2024-09-21T15:49:18.071 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:18.071 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.071 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.071 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:18.071 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.280557+0000", 2024-09-21T15:49:18.071 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.53e-06 2024-09-21T15:49:18.071 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.071 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.072 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:18.072 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.280559+0000", 2024-09-21T15:49:18.072 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8200000000000001e-06 2024-09-21T15:49:18.072 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.072 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.072 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:18.072 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.280562+0000", 2024-09-21T15:49:18.072 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.311e-06 2024-09-21T15:49:18.072 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.072 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.072 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:18.072 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.285695+0000", 2024-09-21T15:49:18.072 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0051333569999999999 2024-09-21T15:49:18.072 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.072 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.072 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:18.073 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.285752+0000", 2024-09-21T15:49:18.073 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6548999999999997e-05 2024-09-21T15:49:18.073 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.073 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.073 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 7", 2024-09-21T15:49:18.073 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.285964+0000", 2024-09-21T15:49:18.073 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00021280999999999999 2024-09-21T15:49:18.073 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.073 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.073 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:18.073 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.290069+0000", 2024-09-21T15:49:18.073 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0041050069999999999 2024-09-21T15:49:18.073 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.073 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.073 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:18.073 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.292648+0000", 2024-09-21T15:49:18.073 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0025781229999999999 2024-09-21T15:49:18.074 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.074 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.074 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:18.074 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.292706+0000", 2024-09-21T15:49:18.074 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8795999999999997e-05 2024-09-21T15:49:18.074 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.074 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.074 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:18.074 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.292742+0000", 2024-09-21T15:49:18.074 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5163999999999997e-05 2024-09-21T15:49:18.074 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.074 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:18.074 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.074 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.075 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.075 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:13503 3.14 3:2ac81419:::rbd_object_map.1133dc5adfdf:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:18.075 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.280564+0000", 2024-09-21T15:49:18.075 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.75295542000000004, 2024-09-21T15:49:18.075 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.012569275, 2024-09-21T15:49:18.075 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:18.075 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:18.075 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:18.075 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:18.075 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:18.075 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 13503 2024-09-21T15:49:18.075 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.075 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:18.075 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.075 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:18.075 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.280564+0000", 2024-09-21T15:49:18.076 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.076 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.076 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.076 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:18.076 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.280564+0000", 2024-09-21T15:49:18.076 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.076 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.076 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.076 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:18.076 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.280564+0000", 2024-09-21T15:49:18.076 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:18.076 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.076 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.076 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:18.076 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.280565+0000", 2024-09-21T15:49:18.077 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5340000000000001e-06 2024-09-21T15:49:18.077 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.077 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.077 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:18.077 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.280569+0000", 2024-09-21T15:49:18.077 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.49e-06 2024-09-21T15:49:18.077 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.077 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.077 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:18.077 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.280571+0000", 2024-09-21T15:49:18.077 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.571e-06 2024-09-21T15:49:18.077 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.077 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.077 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:18.077 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.286371+0000", 2024-09-21T15:49:18.078 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0057999030000000003 2024-09-21T15:49:18.078 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.078 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.078 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:18.078 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.286418+0000", 2024-09-21T15:49:18.078 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7240000000000002e-05 2024-09-21T15:49:18.078 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.078 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.078 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 7", 2024-09-21T15:49:18.078 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.286624+0000", 2024-09-21T15:49:18.078 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000205909 2024-09-21T15:49:18.078 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.078 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.078 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:18.079 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.292751+0000", 2024-09-21T15:49:18.079 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0061272710000000001 2024-09-21T15:49:18.079 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.079 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.079 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:18.079 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.293034+0000", 2024-09-21T15:49:18.079 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00028223899999999999 2024-09-21T15:49:18.079 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.079 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.079 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:18.079 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.293097+0000", 2024-09-21T15:49:18.079 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3119000000000003e-05 2024-09-21T15:49:18.079 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.079 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.079 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:18.079 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.293133+0000", 2024-09-21T15:49:18.080 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6229000000000002e-05 2024-09-21T15:49:18.080 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.080 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:18.080 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.080 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.080 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.080 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:13504 3.14 3:2ac81419:::rbd_object_map.1133dc5adfdf:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:18.080 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.280573+0000", 2024-09-21T15:49:18.080 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.75294589300000003, 2024-09-21T15:49:18.080 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.012680702, 2024-09-21T15:49:18.080 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:18.080 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:18.080 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:18.080 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:18.080 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:18.081 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 13504 2024-09-21T15:49:18.081 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.081 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:18.081 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.081 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:18.081 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.280573+0000", 2024-09-21T15:49:18.081 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.081 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.081 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.081 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:18.081 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.280573+0000", 2024-09-21T15:49:18.081 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.081 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.081 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.081 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:18.082 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.280573+0000", 2024-09-21T15:49:18.082 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:18.082 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.082 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.082 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:18.082 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.280575+0000", 2024-09-21T15:49:18.082 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4300000000000001e-06 2024-09-21T15:49:18.082 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.082 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.082 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:18.082 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.280577+0000", 2024-09-21T15:49:18.082 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9160000000000001e-06 2024-09-21T15:49:18.082 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.082 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.083 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:18.083 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.280580+0000", 2024-09-21T15:49:18.083 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.486e-06 2024-09-21T15:49:18.083 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.083 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.083 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:18.083 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.286919+0000", 2024-09-21T15:49:18.083 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0063391539999999996 2024-09-21T15:49:18.083 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.083 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.083 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:18.083 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.287000+0000", 2024-09-21T15:49:18.083 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0859000000000006e-05 2024-09-21T15:49:18.083 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.083 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.084 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 7", 2024-09-21T15:49:18.084 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.287336+0000", 2024-09-21T15:49:18.084 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00033653699999999998 2024-09-21T15:49:18.084 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.084 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.084 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:18.084 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.292962+0000", 2024-09-21T15:49:18.084 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0056257640000000001 2024-09-21T15:49:18.084 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.084 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.084 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:18.084 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.293172+0000", 2024-09-21T15:49:18.084 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00020969000000000001 2024-09-21T15:49:18.084 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.085 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.085 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:18.085 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.293218+0000", 2024-09-21T15:49:18.085 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5834000000000002e-05 2024-09-21T15:49:18.085 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.085 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.085 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:18.085 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.293254+0000", 2024-09-21T15:49:18.085 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6266000000000001e-05 2024-09-21T15:49:18.085 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.085 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:18.085 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.085 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.085 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.085 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:13505 3.14 3:2ac81419:::rbd_object_map.1133dc5adfdf:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:18.086 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.280583+0000", 2024-09-21T15:49:18.086 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.75293577599999995, 2024-09-21T15:49:18.086 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.012765583, 2024-09-21T15:49:18.086 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:18.086 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:18.086 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:18.086 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:18.086 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:18.086 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 13505 2024-09-21T15:49:18.086 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.086 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:18.086 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.086 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:18.086 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.280583+0000", 2024-09-21T15:49:18.086 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.087 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.087 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.087 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:18.087 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.280583+0000", 2024-09-21T15:49:18.087 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.087 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.087 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.087 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:18.087 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.280583+0000", 2024-09-21T15:49:18.087 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:18.087 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.087 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.087 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:18.087 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.280585+0000", 2024-09-21T15:49:18.087 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.615e-06 2024-09-21T15:49:18.087 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.088 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.088 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:18.088 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.280588+0000", 2024-09-21T15:49:18.088 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1939999999999998e-06 2024-09-21T15:49:18.088 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.088 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.088 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:18.088 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.280590+0000", 2024-09-21T15:49:18.088 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3949999999999999e-06 2024-09-21T15:49:18.088 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.088 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.088 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:18.088 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.287618+0000", 2024-09-21T15:49:18.088 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0070271939999999996 2024-09-21T15:49:18.088 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.088 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.089 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:18.089 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.287704+0000", 2024-09-21T15:49:18.089 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6137999999999994e-05 2024-09-21T15:49:18.089 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.089 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.089 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 7", 2024-09-21T15:49:18.089 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.287943+0000", 2024-09-21T15:49:18.089 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00023945300000000001 2024-09-21T15:49:18.089 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.089 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.089 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:18.089 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.292993+0000", 2024-09-21T15:49:18.089 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0050501570000000004 2024-09-21T15:49:18.089 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.089 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.089 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:18.090 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.293296+0000", 2024-09-21T15:49:18.090 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00030243400000000003 2024-09-21T15:49:18.090 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.090 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.090 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:18.090 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.293319+0000", 2024-09-21T15:49:18.090 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2770000000000001e-05 2024-09-21T15:49:18.090 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.090 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.090 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:18.090 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.293349+0000", 2024-09-21T15:49:18.090 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0477e-05 2024-09-21T15:49:18.090 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.090 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:18.091 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.091 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.091 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.091 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:13737 3.14 3:2ac81419:::rbd_object_map.1133dc5adfdf:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:18.091 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.450386+0000", 2024-09-21T15:49:18.091 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.58313284899999995, 2024-09-21T15:49:18.091 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.012248167000000001, 2024-09-21T15:49:18.091 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:18.091 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:18.091 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:18.091 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:18.091 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:18.091 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 13737 2024-09-21T15:49:18.091 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.091 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:18.092 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.092 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:18.092 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.450386+0000", 2024-09-21T15:49:18.092 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.092 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.092 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.092 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:18.092 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.450386+0000", 2024-09-21T15:49:18.092 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.092 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.092 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.092 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:18.092 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.450386+0000", 2024-09-21T15:49:18.092 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:18.093 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.093 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.093 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:18.093 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.450388+0000", 2024-09-21T15:49:18.093 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.902e-06 2024-09-21T15:49:18.093 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.093 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.093 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:18.093 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.450392+0000", 2024-09-21T15:49:18.093 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6559999999999998e-06 2024-09-21T15:49:18.093 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.093 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.093 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:18.093 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.450394+0000", 2024-09-21T15:49:18.093 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6240000000000002e-06 2024-09-21T15:49:18.093 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.093 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.094 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:18.094 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.457261+0000", 2024-09-21T15:49:18.094 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0068665829999999999 2024-09-21T15:49:18.094 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.094 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.094 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:18.094 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.457352+0000", 2024-09-21T15:49:18.094 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1422999999999995e-05 2024-09-21T15:49:18.094 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.094 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.094 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 7", 2024-09-21T15:49:18.094 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.457677+0000", 2024-09-21T15:49:18.094 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00032440799999999999 2024-09-21T15:49:18.094 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.094 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.095 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:18.095 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.462503+0000", 2024-09-21T15:49:18.095 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0048263200000000003 2024-09-21T15:49:18.095 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.095 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.095 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:18.095 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.462574+0000", 2024-09-21T15:49:18.095 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0858000000000005e-05 2024-09-21T15:49:18.095 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.095 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.095 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:18.095 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.462615+0000", 2024-09-21T15:49:18.095 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0809999999999997e-05 2024-09-21T15:49:18.095 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.095 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.096 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:18.096 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.462635+0000", 2024-09-21T15:49:18.096 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.984e-05 2024-09-21T15:49:18.096 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.096 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:18.096 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.096 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.096 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.096 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:13738 3.14 3:2ac81419:::rbd_object_map.1133dc5adfdf:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:18.096 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.450397+0000", 2024-09-21T15:49:18.096 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.58312234299999999, 2024-09-21T15:49:18.096 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.012317148, 2024-09-21T15:49:18.096 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:18.096 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:18.096 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:18.096 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:18.097 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:18.097 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 13738 2024-09-21T15:49:18.097 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.097 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:18.097 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.097 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:18.097 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.450397+0000", 2024-09-21T15:49:18.097 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.097 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.097 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.097 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:18.097 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.450397+0000", 2024-09-21T15:49:18.097 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.097 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.098 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.098 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:18.098 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.450397+0000", 2024-09-21T15:49:18.098 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:18.098 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.098 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.098 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:18.098 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.450398+0000", 2024-09-21T15:49:18.098 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5659999999999999e-06 2024-09-21T15:49:18.098 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.098 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.098 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:18.098 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.450401+0000", 2024-09-21T15:49:18.098 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9359999999999999e-06 2024-09-21T15:49:18.098 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.099 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.099 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:18.099 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.450404+0000", 2024-09-21T15:49:18.099 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.396e-06 2024-09-21T15:49:18.099 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.099 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.099 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:18.099 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.458038+0000", 2024-09-21T15:49:18.099 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0076340440000000004 2024-09-21T15:49:18.099 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.099 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.099 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:18.099 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.458105+0000", 2024-09-21T15:49:18.100 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7125999999999994e-05 2024-09-21T15:49:18.100 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.100 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.100 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 7", 2024-09-21T15:49:18.100 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.458298+0000", 2024-09-21T15:49:18.100 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00019281300000000001 2024-09-21T15:49:18.100 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.100 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.100 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:18.100 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.462510+0000", 2024-09-21T15:49:18.100 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0042126749999999999 2024-09-21T15:49:18.100 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.100 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.100 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:18.100 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.462669+0000", 2024-09-21T15:49:18.100 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000158729 2024-09-21T15:49:18.101 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.101 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.101 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:18.101 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.462695+0000", 2024-09-21T15:49:18.101 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6103999999999999e-05 2024-09-21T15:49:18.101 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.101 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.101 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:18.101 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.462714+0000", 2024-09-21T15:49:18.101 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8961999999999999e-05 2024-09-21T15:49:18.101 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.101 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:18.101 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.101 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.101 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.101 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:13739 3.14 3:2ac81419:::rbd_object_map.1133dc5adfdf:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:18.102 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.450407+0000", 2024-09-21T15:49:18.102 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.58311240600000003, 2024-09-21T15:49:18.102 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.012409735999999999, 2024-09-21T15:49:18.102 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:18.102 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:18.102 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:18.102 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:18.102 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:18.102 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 13739 2024-09-21T15:49:18.102 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.102 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:18.102 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.102 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:18.102 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.450407+0000", 2024-09-21T15:49:18.102 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.103 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.103 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.103 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:18.103 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.450407+0000", 2024-09-21T15:49:18.103 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.103 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.103 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.103 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:18.103 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.450407+0000", 2024-09-21T15:49:18.103 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:18.103 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.103 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.103 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:18.103 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.450408+0000", 2024-09-21T15:49:18.103 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.739e-06 2024-09-21T15:49:18.104 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.104 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.104 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:18.104 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.450412+0000", 2024-09-21T15:49:18.104 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5039999999999998e-06 2024-09-21T15:49:18.104 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.104 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.104 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:18.104 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.450414+0000", 2024-09-21T15:49:18.104 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5059999999999998e-06 2024-09-21T15:49:18.104 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.104 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.104 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:18.104 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.458741+0000", 2024-09-21T15:49:18.104 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0083269829999999996 2024-09-21T15:49:18.104 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.105 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.105 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:18.105 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.458789+0000", 2024-09-21T15:49:18.105 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7988000000000002e-05 2024-09-21T15:49:18.105 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.105 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.105 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 7", 2024-09-21T15:49:18.105 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.458995+0000", 2024-09-21T15:49:18.105 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000205687 2024-09-21T15:49:18.105 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.105 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.105 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:18.105 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.462738+0000", 2024-09-21T15:49:18.105 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0037433750000000002 2024-09-21T15:49:18.106 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.106 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.106 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:18.106 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.462742+0000", 2024-09-21T15:49:18.106 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0210000000000001e-06 2024-09-21T15:49:18.106 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.106 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.106 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:18.106 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.462780+0000", 2024-09-21T15:49:18.106 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7549999999999998e-05 2024-09-21T15:49:18.106 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.106 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.106 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:18.106 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.462817+0000", 2024-09-21T15:49:18.107 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6593e-05 2024-09-21T15:49:18.107 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.107 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:18.107 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.107 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.107 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.107 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:13855 3.14 3:2ac81419:::rbd_object_map.1133dc5adfdf:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:18.107 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.543360+0000", 2024-09-21T15:49:18.107 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.49015881300000003, 2024-09-21T15:49:18.107 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.012484899000000001, 2024-09-21T15:49:18.107 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:18.107 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:18.107 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:18.107 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:18.108 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:18.108 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 13855 2024-09-21T15:49:18.108 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.108 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:18.108 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.108 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:18.108 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543360+0000", 2024-09-21T15:49:18.108 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.108 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.108 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.108 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:18.108 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543360+0000", 2024-09-21T15:49:18.108 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.108 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.108 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.108 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:18.109 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543360+0000", 2024-09-21T15:49:18.109 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:18.109 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.109 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.109 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:18.109 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543364+0000", 2024-09-21T15:49:18.109 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0350000000000003e-06 2024-09-21T15:49:18.109 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.109 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.109 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:18.109 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543368+0000", 2024-09-21T15:49:18.109 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9840000000000003e-06 2024-09-21T15:49:18.109 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.109 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.109 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:18.110 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543372+0000", 2024-09-21T15:49:18.110 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.579e-06 2024-09-21T15:49:18.110 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.110 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.110 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:18.110 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.547349+0000", 2024-09-21T15:49:18.110 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0039770220000000002 2024-09-21T15:49:18.110 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.110 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.110 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:18.110 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.547410+0000", 2024-09-21T15:49:18.110 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1128000000000002e-05 2024-09-21T15:49:18.110 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.110 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.111 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 7", 2024-09-21T15:49:18.111 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.547809+0000", 2024-09-21T15:49:18.111 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00039859699999999999 2024-09-21T15:49:18.111 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.111 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.111 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:18.111 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.553721+0000", 2024-09-21T15:49:18.111 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0059128440000000004 2024-09-21T15:49:18.111 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.111 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.111 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:18.111 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.555758+0000", 2024-09-21T15:49:18.111 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0020370980000000002 2024-09-21T15:49:18.111 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.111 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.112 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:18.112 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.555810+0000", 2024-09-21T15:49:18.112 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1570000000000003e-05 2024-09-21T15:49:18.112 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.112 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.112 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:18.112 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.555845+0000", 2024-09-21T15:49:18.112 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5250000000000003e-05 2024-09-21T15:49:18.112 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.112 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:18.112 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.112 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.112 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.112 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:13856 3.14 3:2ac81419:::rbd_object_map.1133dc5adfdf:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:18.112 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.543374+0000", 2024-09-21T15:49:18.113 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.490145004, 2024-09-21T15:49:18.113 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.012719601000000001, 2024-09-21T15:49:18.113 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:18.113 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:18.113 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:18.113 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:18.113 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:18.113 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 13856 2024-09-21T15:49:18.113 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.113 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:18.113 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.113 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:18.113 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543374+0000", 2024-09-21T15:49:18.113 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.113 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.114 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.114 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:18.114 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543374+0000", 2024-09-21T15:49:18.114 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.114 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.114 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.114 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:18.114 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543374+0000", 2024-09-21T15:49:18.114 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:18.114 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.114 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.114 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:18.114 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543376+0000", 2024-09-21T15:49:18.114 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7409999999999999e-06 2024-09-21T15:49:18.115 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.115 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.115 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:18.115 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543379+0000", 2024-09-21T15:49:18.115 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.914e-06 2024-09-21T15:49:18.115 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.115 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.115 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:18.115 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543381+0000", 2024-09-21T15:49:18.115 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4880000000000001e-06 2024-09-21T15:49:18.115 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.115 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.115 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:18.115 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.548128+0000", 2024-09-21T15:49:18.115 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0047472779999999997 2024-09-21T15:49:18.116 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.116 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.116 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:18.116 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.548178+0000", 2024-09-21T15:49:18.116 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9218999999999998e-05 2024-09-21T15:49:18.116 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.116 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.116 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 7", 2024-09-21T15:49:18.116 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.548375+0000", 2024-09-21T15:49:18.116 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00019771599999999999 2024-09-21T15:49:18.116 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.116 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.116 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:18.116 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.555855+0000", 2024-09-21T15:49:18.116 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0074795720000000003 2024-09-21T15:49:18.117 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.117 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.117 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:18.117 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.556015+0000", 2024-09-21T15:49:18.117 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016003799999999999 2024-09-21T15:49:18.117 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.117 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.117 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:18.117 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.556060+0000", 2024-09-21T15:49:18.117 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5305000000000001e-05 2024-09-21T15:49:18.117 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.117 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.117 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:18.117 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.556094+0000", 2024-09-21T15:49:18.117 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3531999999999997e-05 2024-09-21T15:49:18.118 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.118 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:18.118 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.118 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.118 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.118 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:13857 3.14 3:2ac81419:::rbd_object_map.1133dc5adfdf:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:18.118 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.543383+0000", 2024-09-21T15:49:18.118 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.49013604700000002, 2024-09-21T15:49:18.118 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.012793752, 2024-09-21T15:49:18.118 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:18.118 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:18.118 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:18.118 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:18.118 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:18.119 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 13857 2024-09-21T15:49:18.119 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.119 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:18.119 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.119 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:18.119 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543383+0000", 2024-09-21T15:49:18.119 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.119 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.119 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.119 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:18.119 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543383+0000", 2024-09-21T15:49:18.119 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.119 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.119 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.119 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:18.120 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543383+0000", 2024-09-21T15:49:18.120 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:18.120 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.120 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.120 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:18.120 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543385+0000", 2024-09-21T15:49:18.120 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7e-06 2024-09-21T15:49:18.120 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.120 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.120 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:18.120 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543388+0000", 2024-09-21T15:49:18.120 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3340000000000002e-06 2024-09-21T15:49:18.120 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.120 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.120 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:18.121 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543391+0000", 2024-09-21T15:49:18.121 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9110000000000002e-06 2024-09-21T15:49:18.121 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.121 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.121 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:18.121 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.548745+0000", 2024-09-21T15:49:18.121 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0053544200000000004 2024-09-21T15:49:18.121 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.121 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.121 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:18.121 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.548797+0000", 2024-09-21T15:49:18.121 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1875000000000001e-05 2024-09-21T15:49:18.121 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.121 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.121 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 7", 2024-09-21T15:49:18.122 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.549008+0000", 2024-09-21T15:49:18.122 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00021113999999999999 2024-09-21T15:49:18.122 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.122 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.122 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:18.122 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.555860+0000", 2024-09-21T15:49:18.122 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0068516269999999999 2024-09-21T15:49:18.122 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.122 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.122 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:18.122 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.556133+0000", 2024-09-21T15:49:18.122 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00027346800000000002 2024-09-21T15:49:18.122 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.122 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.123 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:18.123 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.556158+0000", 2024-09-21T15:49:18.123 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4901000000000001e-05 2024-09-21T15:49:18.123 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.123 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.123 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:18.123 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.556177+0000", 2024-09-21T15:49:18.123 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8576e-05 2024-09-21T15:49:18.123 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.123 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:18.123 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.123 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.123 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.123 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:13858 3.14 3:2ac81419:::rbd_object_map.1133dc5adfdf:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:18.123 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.543396+0000", 2024-09-21T15:49:18.124 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.490123015, 2024-09-21T15:49:18.124 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.012881375, 2024-09-21T15:49:18.124 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:18.124 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:18.124 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:18.124 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:18.124 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:18.124 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 13858 2024-09-21T15:49:18.124 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.124 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:18.124 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.124 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:18.124 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543396+0000", 2024-09-21T15:49:18.124 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.124 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.125 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.125 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:18.125 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543396+0000", 2024-09-21T15:49:18.125 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.125 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.125 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.125 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:18.125 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543396+0000", 2024-09-21T15:49:18.125 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:18.125 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.125 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.125 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:18.125 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543398+0000", 2024-09-21T15:49:18.125 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9190000000000002e-06 2024-09-21T15:49:18.126 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.126 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.126 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:18.126 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543402+0000", 2024-09-21T15:49:18.126 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6729999999999998e-06 2024-09-21T15:49:18.126 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.126 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.126 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:18.126 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543405+0000", 2024-09-21T15:49:18.126 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2030000000000001e-06 2024-09-21T15:49:18.126 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.126 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.126 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:18.127 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.549528+0000", 2024-09-21T15:49:18.127 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0061230379999999999 2024-09-21T15:49:18.127 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.127 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.127 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:18.127 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.549610+0000", 2024-09-21T15:49:18.127 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2602999999999995e-05 2024-09-21T15:49:18.127 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.127 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.127 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 7", 2024-09-21T15:49:18.127 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.549966+0000", 2024-09-21T15:49:18.127 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00035542700000000001 2024-09-21T15:49:18.127 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.127 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.128 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:18.128 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.555864+0000", 2024-09-21T15:49:18.128 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0058985460000000002 2024-09-21T15:49:18.128 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.128 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.128 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:18.128 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.556218+0000", 2024-09-21T15:49:18.128 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00035375899999999999 2024-09-21T15:49:18.128 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.128 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.128 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:18.128 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.556246+0000", 2024-09-21T15:49:18.128 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8096e-05 2024-09-21T15:49:18.128 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.128 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.128 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:18.129 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.556278+0000", 2024-09-21T15:49:18.129 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1359999999999998e-05 2024-09-21T15:49:18.129 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.129 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:18.129 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.129 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.129 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.129 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:13859 3.14 3:2ac81419:::rbd_object_map.1133dc5adfdf:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:18.129 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.543407+0000", 2024-09-21T15:49:18.129 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.49011223399999998, 2024-09-21T15:49:18.129 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.012991434, 2024-09-21T15:49:18.129 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:18.129 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:18.129 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:18.129 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:18.130 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:18.130 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 13859 2024-09-21T15:49:18.130 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.130 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:18.130 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.130 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:18.130 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543407+0000", 2024-09-21T15:49:18.130 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.130 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.130 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.130 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:18.130 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543407+0000", 2024-09-21T15:49:18.130 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.130 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.130 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.131 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:18.131 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543407+0000", 2024-09-21T15:49:18.131 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:18.131 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.131 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.131 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:18.131 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543408+0000", 2024-09-21T15:49:18.131 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.533e-06 2024-09-21T15:49:18.131 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.131 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.131 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:18.131 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543411+0000", 2024-09-21T15:49:18.131 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9239999999999999e-06 2024-09-21T15:49:18.131 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.131 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.132 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:18.132 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543414+0000", 2024-09-21T15:49:18.132 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0680000000000001e-06 2024-09-21T15:49:18.132 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.132 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.132 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:18.132 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.550404+0000", 2024-09-21T15:49:18.132 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0069900329999999997 2024-09-21T15:49:18.132 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.132 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.132 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:18.132 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.550466+0000", 2024-09-21T15:49:18.132 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1265000000000004e-05 2024-09-21T15:49:18.132 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.132 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.132 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 7", 2024-09-21T15:49:18.133 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.550752+0000", 2024-09-21T15:49:18.133 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00028663000000000001 2024-09-21T15:49:18.133 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.133 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.133 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:18.133 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.555868+0000", 2024-09-21T15:49:18.133 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.005116191 2024-09-21T15:49:18.133 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.133 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.133 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:18.133 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.556330+0000", 2024-09-21T15:49:18.133 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00046124799999999999 2024-09-21T15:49:18.133 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.133 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.133 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:18.133 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.556363+0000", 2024-09-21T15:49:18.134 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3592e-05 2024-09-21T15:49:18.134 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.134 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.134 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:18.134 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.556398+0000", 2024-09-21T15:49:18.134 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5148000000000003e-05 2024-09-21T15:49:18.134 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.134 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:18.134 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.134 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.134 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.134 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:13860 3.14 3:2ac81419:::rbd_object_map.1133dc5adfdf:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:18.134 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.543416+0000", 2024-09-21T15:49:18.134 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.49010283700000001, 2024-09-21T15:49:18.135 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.013060522999999999, 2024-09-21T15:49:18.135 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:18.135 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:18.135 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:18.135 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:18.135 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:18.135 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 13860 2024-09-21T15:49:18.135 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.135 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:18.135 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.135 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:18.135 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543416+0000", 2024-09-21T15:49:18.135 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.135 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.136 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.136 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:18.136 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543416+0000", 2024-09-21T15:49:18.136 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.136 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.136 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.136 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:18.136 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543416+0000", 2024-09-21T15:49:18.136 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:18.136 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.136 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.136 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:18.136 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543420+0000", 2024-09-21T15:49:18.136 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.63e-06 2024-09-21T15:49:18.136 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.137 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.137 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:18.137 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543423+0000", 2024-09-21T15:49:18.137 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9809999999999999e-06 2024-09-21T15:49:18.137 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.137 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.137 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:18.137 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543425+0000", 2024-09-21T15:49:18.137 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5390000000000002e-06 2024-09-21T15:49:18.137 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.137 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.137 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:18.137 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.551032+0000", 2024-09-21T15:49:18.137 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0076064879999999998 2024-09-21T15:49:18.137 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.138 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.138 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:18.138 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.551114+0000", 2024-09-21T15:49:18.138 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2639000000000006e-05 2024-09-21T15:49:18.138 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.138 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.138 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 7", 2024-09-21T15:49:18.138 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.551359+0000", 2024-09-21T15:49:18.138 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00024488199999999998 2024-09-21T15:49:18.138 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.138 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.138 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:18.138 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.555873+0000", 2024-09-21T15:49:18.138 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0045135669999999996 2024-09-21T15:49:18.139 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.139 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.139 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:18.139 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.556429+0000", 2024-09-21T15:49:18.139 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00055622799999999997 2024-09-21T15:49:18.139 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.139 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.139 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:18.139 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.556457+0000", 2024-09-21T15:49:18.139 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7931000000000001e-05 2024-09-21T15:49:18.139 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.139 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.139 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:18.139 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.556477+0000", 2024-09-21T15:49:18.139 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9845999999999999e-05 2024-09-21T15:49:18.140 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.140 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:18.140 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.140 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.140 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.140 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:13921 3.14 3:2ac81419:::rbd_object_map.1133dc5adfdf:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:18.140 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.586097+0000", 2024-09-21T15:49:18.140 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.44742172099999999, 2024-09-21T15:49:18.140 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016128601999999999, 2024-09-21T15:49:18.140 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:18.140 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:18.140 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:18.140 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:18.140 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:18.140 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 13921 2024-09-21T15:49:18.141 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.141 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:18.141 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.141 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:18.141 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.586097+0000", 2024-09-21T15:49:18.141 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.141 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.141 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.141 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:18.141 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.586097+0000", 2024-09-21T15:49:18.141 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.141 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.141 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.141 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:18.141 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.586097+0000", 2024-09-21T15:49:18.142 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:18.142 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.142 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.142 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:18.142 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.586099+0000", 2024-09-21T15:49:18.142 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.514e-06 2024-09-21T15:49:18.142 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.142 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.142 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:18.142 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.586102+0000", 2024-09-21T15:49:18.142 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.92e-06 2024-09-21T15:49:18.142 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.142 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.142 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:18.143 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.586104+0000", 2024-09-21T15:49:18.143 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5790000000000002e-06 2024-09-21T15:49:18.143 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.143 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.143 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:18.143 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.594361+0000", 2024-09-21T15:49:18.143 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0082570319999999992 2024-09-21T15:49:18.143 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.143 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.143 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:18.143 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.594448+0000", 2024-09-21T15:49:18.143 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6254000000000005e-05 2024-09-21T15:49:18.143 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.143 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.143 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 7", 2024-09-21T15:49:18.144 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.594765+0000", 2024-09-21T15:49:18.144 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00031769799999999999 2024-09-21T15:49:18.144 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.144 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.144 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:18.144 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.599427+0000", 2024-09-21T15:49:18.144 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0046615989999999998 2024-09-21T15:49:18.144 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.144 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.144 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:18.144 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.602156+0000", 2024-09-21T15:49:18.144 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0027293109999999999 2024-09-21T15:49:18.144 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.144 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.144 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:18.144 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.602197+0000", 2024-09-21T15:49:18.144 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0389000000000002e-05 2024-09-21T15:49:18.145 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.145 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.145 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:18.145 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.602226+0000", 2024-09-21T15:49:18.145 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9530999999999999e-05 2024-09-21T15:49:18.145 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.145 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:18.145 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.145 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.145 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:18.145 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:18.147 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:49:18.147 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:18.146+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:18.247 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:18.246+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:18.291 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:18.291 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:18.291 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:49:18.291 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:18.300 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:49:18.348 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:18.347+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:18.401 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:18.402 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:18.402 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:49:18.402 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:49:18.402 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:18.410 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:49:18.449 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:18.448+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:18.550 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:18.548+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:18.554 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:18.554 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:49:18.554 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:49:18.554 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:49:18.554 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.554 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:4938 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 659456~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:18.554 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.455162+0000", 2024-09-21T15:49:18.554 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.0972270030000004, 2024-09-21T15:49:18.554 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039399295000000001, 2024-09-21T15:49:18.555 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:18.555 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:18.555 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:18.555 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:18.555 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:18.555 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 4938 2024-09-21T15:49:18.555 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.555 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:18.555 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.555 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:18.555 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.455162+0000", 2024-09-21T15:49:18.555 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.555 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.555 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.556 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:18.556 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.455162+0000", 2024-09-21T15:49:18.556 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.556 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.556 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.556 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:18.556 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.455162+0000", 2024-09-21T15:49:18.556 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:18.556 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.556 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.556 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:18.556 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.455164+0000", 2024-09-21T15:49:18.556 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.074e-06 2024-09-21T15:49:18.556 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.556 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.556 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:18.557 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.455169+0000", 2024-09-21T15:49:18.557 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6760000000000004e-06 2024-09-21T15:49:18.557 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.557 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.557 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:18.557 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.455174+0000", 2024-09-21T15:49:18.557 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3680000000000004e-06 2024-09-21T15:49:18.557 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.557 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.557 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:18.557 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.494363+0000", 2024-09-21T15:49:18.557 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039189225000000001 2024-09-21T15:49:18.557 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.557 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.557 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:18.558 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.494456+0000", 2024-09-21T15:49:18.558 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2843999999999994e-05 2024-09-21T15:49:18.558 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.558 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.558 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:18.558 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.494561+0000", 2024-09-21T15:49:18.558 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000105627 2024-09-21T15:49:18.558 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.558 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:18.558 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.558 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.558 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.558 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:4939 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 708608~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:18.558 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.455178+0000", 2024-09-21T15:49:18.558 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.0972113119999998, 2024-09-21T15:49:18.559 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039493579000000001, 2024-09-21T15:49:18.559 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:18.559 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:18.559 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:18.559 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:18.559 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:18.559 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 4939 2024-09-21T15:49:18.559 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.559 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:18.559 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.559 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:18.559 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.455178+0000", 2024-09-21T15:49:18.559 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.559 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.560 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.560 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:18.560 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.455178+0000", 2024-09-21T15:49:18.560 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.560 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.560 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.560 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:18.560 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.455177+0000", 2024-09-21T15:49:18.560 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:18.560 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.560 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.560 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:18.560 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.455179+0000", 2024-09-21T15:49:18.560 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9489999999999999e-06 2024-09-21T15:49:18.561 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.561 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.561 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:18.561 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.455185+0000", 2024-09-21T15:49:18.561 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.49e-06 2024-09-21T15:49:18.561 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.561 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.561 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:18.561 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.455190+0000", 2024-09-21T15:49:18.561 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5910000000000003e-06 2024-09-21T15:49:18.561 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.561 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.561 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:18.561 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.494573+0000", 2024-09-21T15:49:18.561 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039382895000000001 2024-09-21T15:49:18.562 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.562 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.562 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:18.562 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.494622+0000", 2024-09-21T15:49:18.562 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8167000000000001e-05 2024-09-21T15:49:18.562 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.562 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.562 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:18.562 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.494671+0000", 2024-09-21T15:49:18.562 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9818000000000003e-05 2024-09-21T15:49:18.562 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.562 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:18.562 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.562 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.563 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.563 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:4949 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 1200128~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:18.563 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.457117+0000", 2024-09-21T15:49:18.563 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.0952720510000002, 2024-09-21T15:49:18.563 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039319875999999997, 2024-09-21T15:49:18.563 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:18.563 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:18.563 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:18.563 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:18.563 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:18.563 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 4949 2024-09-21T15:49:18.563 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.563 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:18.563 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.564 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:18.564 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.457117+0000", 2024-09-21T15:49:18.564 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.564 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.564 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.564 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:18.564 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.457117+0000", 2024-09-21T15:49:18.564 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.564 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.564 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.564 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:18.564 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.457117+0000", 2024-09-21T15:49:18.564 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:18.564 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.564 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.564 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:18.565 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.457118+0000", 2024-09-21T15:49:18.565 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0729999999999999e-06 2024-09-21T15:49:18.565 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.565 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.565 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:18.565 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.457121+0000", 2024-09-21T15:49:18.565 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.802e-06 2024-09-21T15:49:18.565 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.565 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.565 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:18.565 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.457123+0000", 2024-09-21T15:49:18.565 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1969999999999999e-06 2024-09-21T15:49:18.565 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.565 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.566 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:18.566 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.496340+0000", 2024-09-21T15:49:18.566 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039217120000000001 2024-09-21T15:49:18.566 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.566 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.566 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:18.566 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.496386+0000", 2024-09-21T15:49:18.566 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6360000000000003e-05 2024-09-21T15:49:18.566 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.566 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.566 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:18.566 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.496437+0000", 2024-09-21T15:49:18.566 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0555e-05 2024-09-21T15:49:18.566 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.567 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:18.567 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.567 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.567 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.567 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:4950 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 1249280~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:18.567 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.457125+0000", 2024-09-21T15:49:18.567 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.095264062, 2024-09-21T15:49:18.567 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039545631999999997, 2024-09-21T15:49:18.567 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:18.567 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:18.567 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:18.567 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:18.567 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:18.567 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 4950 2024-09-21T15:49:18.568 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.568 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:18.568 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.568 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:18.568 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.457125+0000", 2024-09-21T15:49:18.568 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.568 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.568 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.568 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:18.568 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.457125+0000", 2024-09-21T15:49:18.568 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.568 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.568 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.568 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:18.568 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.457125+0000", 2024-09-21T15:49:18.569 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:18.569 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.569 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.569 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:18.569 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.457126+0000", 2024-09-21T15:49:18.569 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5519999999999999e-06 2024-09-21T15:49:18.569 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.569 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.569 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:18.569 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.457129+0000", 2024-09-21T15:49:18.569 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7989999999999998e-06 2024-09-21T15:49:18.569 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.569 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.569 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:18.569 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.457131+0000", 2024-09-21T15:49:18.569 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3080000000000002e-06 2024-09-21T15:49:18.570 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.570 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.570 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:18.570 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.496473+0000", 2024-09-21T15:49:18.570 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039341222000000002 2024-09-21T15:49:18.570 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.570 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.570 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:18.570 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.496565+0000", 2024-09-21T15:49:18.570 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2330000000000005e-05 2024-09-21T15:49:18.570 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.570 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.570 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:18.570 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.496671+0000", 2024-09-21T15:49:18.570 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000105648 2024-09-21T15:49:18.571 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.571 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:18.571 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.571 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.571 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.571 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:4962 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 1789952~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:18.571 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.459143+0000", 2024-09-21T15:49:18.571 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.0932458089999999, 2024-09-21T15:49:18.571 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039478257000000003, 2024-09-21T15:49:18.571 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:18.571 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:18.571 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:18.571 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:18.571 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:18.571 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 4962 2024-09-21T15:49:18.571 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.572 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:18.572 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.572 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:18.572 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459143+0000", 2024-09-21T15:49:18.572 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.572 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.572 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.572 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:18.572 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459143+0000", 2024-09-21T15:49:18.572 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.572 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.572 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.572 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:18.572 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459143+0000", 2024-09-21T15:49:18.572 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:18.572 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.573 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.573 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:18.573 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459144+0000", 2024-09-21T15:49:18.573 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.142e-06 2024-09-21T15:49:18.573 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.573 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.573 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:18.573 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459147+0000", 2024-09-21T15:49:18.573 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8229999999999999e-06 2024-09-21T15:49:18.573 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.573 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.573 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:18.573 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459149+0000", 2024-09-21T15:49:18.573 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2859999999999998e-06 2024-09-21T15:49:18.573 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.574 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.574 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:18.574 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.498430+0000", 2024-09-21T15:49:18.574 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039280806000000001 2024-09-21T15:49:18.574 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.574 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.574 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:18.574 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.498520+0000", 2024-09-21T15:49:18.574 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0160000000000004e-05 2024-09-21T15:49:18.574 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.574 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.574 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:18.574 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.498622+0000", 2024-09-21T15:49:18.574 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000101275 2024-09-21T15:49:18.574 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.575 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:18.575 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.575 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.575 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.575 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:4963 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 1839104~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:18.575 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.459152+0000", 2024-09-21T15:49:18.575 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.0932374139999999, 2024-09-21T15:49:18.575 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039710886000000001, 2024-09-21T15:49:18.575 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:18.575 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:18.575 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:18.575 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:18.575 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:18.575 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 4963 2024-09-21T15:49:18.575 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.575 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:18.576 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.576 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:18.576 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459152+0000", 2024-09-21T15:49:18.576 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.576 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.576 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.576 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:18.576 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459152+0000", 2024-09-21T15:49:18.576 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.576 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.576 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.576 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:18.576 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459151+0000", 2024-09-21T15:49:18.576 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:18.576 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.576 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.577 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:18.577 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459153+0000", 2024-09-21T15:49:18.577 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1400000000000001e-06 2024-09-21T15:49:18.577 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.577 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.577 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:18.577 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459155+0000", 2024-09-21T15:49:18.577 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7130000000000001e-06 2024-09-21T15:49:18.577 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.577 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.577 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:18.577 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459158+0000", 2024-09-21T15:49:18.577 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.368e-06 2024-09-21T15:49:18.577 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.577 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.577 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:18.578 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.498664+0000", 2024-09-21T15:49:18.578 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039505928000000003 2024-09-21T15:49:18.578 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.578 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.578 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:18.578 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.498756+0000", 2024-09-21T15:49:18.578 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2505999999999998e-05 2024-09-21T15:49:18.578 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.578 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.578 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:18.578 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.498863+0000", 2024-09-21T15:49:18.578 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000106457 2024-09-21T15:49:18.578 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.578 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:18.578 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.579 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.579 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.579 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:4964 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 1888256~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:18.579 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.459160+0000", 2024-09-21T15:49:18.579 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.0932286969999998, 2024-09-21T15:49:18.579 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039933237000000003, 2024-09-21T15:49:18.579 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:18.579 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:18.579 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:18.579 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:18.579 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:18.579 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 4964 2024-09-21T15:49:18.579 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.579 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:18.579 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.579 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:18.580 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459160+0000", 2024-09-21T15:49:18.580 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.580 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.580 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.580 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:18.580 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459160+0000", 2024-09-21T15:49:18.580 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.580 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.580 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.580 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:18.580 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459160+0000", 2024-09-21T15:49:18.580 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:18.580 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.580 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.581 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:18.581 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459162+0000", 2024-09-21T15:49:18.581 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.42e-06 2024-09-21T15:49:18.581 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.581 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.581 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:18.581 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459164+0000", 2024-09-21T15:49:18.581 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8059999999999999e-06 2024-09-21T15:49:18.581 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.581 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.581 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:18.581 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459167+0000", 2024-09-21T15:49:18.581 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.835e-06 2024-09-21T15:49:18.581 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.581 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.581 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:18.582 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.498904+0000", 2024-09-21T15:49:18.582 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039736831 2024-09-21T15:49:18.582 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.582 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.582 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:18.582 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.498992+0000", 2024-09-21T15:49:18.582 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7886999999999994e-05 2024-09-21T15:49:18.582 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.582 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.582 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:18.582 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.499094+0000", 2024-09-21T15:49:18.582 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000101681 2024-09-21T15:49:18.582 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.582 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:18.583 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.583 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.583 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.583 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:4966 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 1986560~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:18.583 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.460005+0000", 2024-09-21T15:49:18.583 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.0923836710000003, 2024-09-21T15:49:18.583 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039390041000000001, 2024-09-21T15:49:18.583 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:18.583 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:18.583 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:18.583 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:18.583 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:18.583 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 4966 2024-09-21T15:49:18.583 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.583 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:18.584 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.584 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:18.584 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460005+0000", 2024-09-21T15:49:18.584 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.584 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.584 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.584 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:18.584 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460005+0000", 2024-09-21T15:49:18.584 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.584 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.584 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.584 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:18.584 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460005+0000", 2024-09-21T15:49:18.584 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:18.585 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.585 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.585 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:18.585 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460006+0000", 2024-09-21T15:49:18.585 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2359999999999999e-06 2024-09-21T15:49:18.585 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.585 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.585 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:18.585 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460013+0000", 2024-09-21T15:49:18.585 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1239999999999999e-06 2024-09-21T15:49:18.585 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.585 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.585 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:18.585 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460018+0000", 2024-09-21T15:49:18.585 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3449999999999999e-06 2024-09-21T15:49:18.585 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.586 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.586 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:18.586 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.499283+0000", 2024-09-21T15:49:18.586 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039264847999999998 2024-09-21T15:49:18.586 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.586 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.586 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:18.586 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.499334+0000", 2024-09-21T15:49:18.586 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1279000000000002e-05 2024-09-21T15:49:18.586 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.586 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.586 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:18.586 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.499395+0000", 2024-09-21T15:49:18.586 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1472999999999999e-05 2024-09-21T15:49:18.587 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.587 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:18.587 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.587 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.587 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.587 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:4967 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 2035712~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:18.587 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.460021+0000", 2024-09-21T15:49:18.587 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.0923677610000002, 2024-09-21T15:49:18.587 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039505047000000001, 2024-09-21T15:49:18.587 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:18.587 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:18.587 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:18.587 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:18.587 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:18.587 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 4967 2024-09-21T15:49:18.587 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.588 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:18.588 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.588 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:18.588 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460021+0000", 2024-09-21T15:49:18.588 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.588 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.588 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.588 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:18.588 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460021+0000", 2024-09-21T15:49:18.588 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.588 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.588 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.588 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:18.588 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460021+0000", 2024-09-21T15:49:18.588 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:18.588 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.589 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.589 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:18.589 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460022+0000", 2024-09-21T15:49:18.589 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3710000000000001e-06 2024-09-21T15:49:18.589 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.589 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.589 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:18.589 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460025+0000", 2024-09-21T15:49:18.589 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.067e-06 2024-09-21T15:49:18.589 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.589 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.589 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:18.589 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460028+0000", 2024-09-21T15:49:18.589 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5009999999999999e-06 2024-09-21T15:49:18.589 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.589 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.590 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:18.590 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.499420+0000", 2024-09-21T15:49:18.590 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039391977000000002 2024-09-21T15:49:18.590 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.590 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.590 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:18.590 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.499472+0000", 2024-09-21T15:49:18.590 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2057e-05 2024-09-21T15:49:18.590 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.590 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.590 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:18.590 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.499526+0000", 2024-09-21T15:49:18.590 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4335000000000001e-05 2024-09-21T15:49:18.590 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.591 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:18.591 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.591 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.591 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.591 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:4968 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 2084864~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:18.591 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.460030+0000", 2024-09-21T15:49:18.591 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.0923590130000003, 2024-09-21T15:49:18.591 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039629655, 2024-09-21T15:49:18.591 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:18.591 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:18.591 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:18.591 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:18.591 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:18.591 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 4968 2024-09-21T15:49:18.591 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.592 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:18.592 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.592 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:18.592 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460030+0000", 2024-09-21T15:49:18.592 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.592 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.592 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.592 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:18.592 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460030+0000", 2024-09-21T15:49:18.592 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.592 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.592 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.592 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:18.592 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460030+0000", 2024-09-21T15:49:18.592 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:18.593 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.593 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.593 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:18.593 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460037+0000", 2024-09-21T15:49:18.593 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8569999999999997e-06 2024-09-21T15:49:18.593 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.593 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.593 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:18.593 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460040+0000", 2024-09-21T15:49:18.593 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1729999999999999e-06 2024-09-21T15:49:18.593 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.593 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.593 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:18.593 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460042+0000", 2024-09-21T15:49:18.593 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4559999999999999e-06 2024-09-21T15:49:18.594 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.594 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.594 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:18.594 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.499547+0000", 2024-09-21T15:49:18.594 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039504723999999998 2024-09-21T15:49:18.594 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.594 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.594 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:18.594 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.499596+0000", 2024-09-21T15:49:18.594 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9230000000000001e-05 2024-09-21T15:49:18.594 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.594 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.594 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:18.594 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.499660+0000", 2024-09-21T15:49:18.595 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3473999999999996e-05 2024-09-21T15:49:18.595 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.595 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:18.595 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.595 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.595 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.595 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:5054 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 24576~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:18.595 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.460046+0000", 2024-09-21T15:49:18.595 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.0923428709999996, 2024-09-21T15:49:18.595 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039744092000000002, 2024-09-21T15:49:18.595 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:18.595 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:18.595 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:18.595 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:18.595 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:18.596 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 5054 2024-09-21T15:49:18.596 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.596 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:18.596 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.596 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:18.596 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460046+0000", 2024-09-21T15:49:18.596 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.596 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.596 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.596 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:18.596 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460046+0000", 2024-09-21T15:49:18.596 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.596 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.596 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.596 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:18.597 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460046+0000", 2024-09-21T15:49:18.597 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:18.597 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.597 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.597 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:18.597 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460051+0000", 2024-09-21T15:49:18.597 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4790000000000004e-06 2024-09-21T15:49:18.597 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.597 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.597 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:18.597 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460055+0000", 2024-09-21T15:49:18.597 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6830000000000001e-06 2024-09-21T15:49:18.597 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.597 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.597 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:18.598 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460057+0000", 2024-09-21T15:49:18.598 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5550000000000001e-06 2024-09-21T15:49:18.598 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.598 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.598 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:18.598 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.499687+0000", 2024-09-21T15:49:18.598 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039629736999999998 2024-09-21T15:49:18.598 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.598 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.598 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:18.598 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.499734+0000", 2024-09-21T15:49:18.598 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6412999999999998e-05 2024-09-21T15:49:18.598 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.598 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.599 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:18.599 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.499790+0000", 2024-09-21T15:49:18.599 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6666999999999998e-05 2024-09-21T15:49:18.599 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.599 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:18.599 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.599 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.599 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.599 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:5055 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 73728~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:18.599 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.460060+0000", 2024-09-21T15:49:18.599 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.0923292560000002, 2024-09-21T15:49:18.599 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039853507000000003, 2024-09-21T15:49:18.599 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:18.599 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:18.600 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:18.600 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:18.600 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:18.600 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 5055 2024-09-21T15:49:18.600 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.600 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:18.600 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.600 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:18.600 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460060+0000", 2024-09-21T15:49:18.600 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.600 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.600 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.600 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:18.600 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460060+0000", 2024-09-21T15:49:18.600 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.601 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.601 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.601 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:18.601 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460060+0000", 2024-09-21T15:49:18.601 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:18.601 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.601 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.601 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:18.601 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460061+0000", 2024-09-21T15:49:18.601 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2410000000000001e-06 2024-09-21T15:49:18.601 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.601 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.601 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:18.601 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460064+0000", 2024-09-21T15:49:18.601 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7539999999999998e-06 2024-09-21T15:49:18.601 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.602 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.602 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:18.602 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460069+0000", 2024-09-21T15:49:18.602 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.558e-06 2024-09-21T15:49:18.602 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.602 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.602 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:18.602 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.499816+0000", 2024-09-21T15:49:18.602 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039746385000000002 2024-09-21T15:49:18.602 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.602 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.602 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:18.602 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.499860+0000", 2024-09-21T15:49:18.602 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4740999999999997e-05 2024-09-21T15:49:18.602 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.603 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.603 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:18.603 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.499913+0000", 2024-09-21T15:49:18.603 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3057999999999999e-05 2024-09-21T15:49:18.603 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.603 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:18.603 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.603 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.603 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.603 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:5056 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 122880~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:18.603 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.460249+0000", 2024-09-21T15:49:18.603 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.0921403969999997, 2024-09-21T15:49:18.603 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039822748999999998, 2024-09-21T15:49:18.603 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:18.603 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:18.604 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:18.604 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:18.604 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:18.604 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 5056 2024-09-21T15:49:18.604 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.604 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:18.604 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.604 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:18.604 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460249+0000", 2024-09-21T15:49:18.604 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.604 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.604 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.604 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:18.604 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460249+0000", 2024-09-21T15:49:18.605 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.605 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.605 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.605 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:18.605 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460120+0000", 2024-09-21T15:49:18.605 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998713 2024-09-21T15:49:18.605 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.605 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.605 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:18.605 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460251+0000", 2024-09-21T15:49:18.605 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000130977 2024-09-21T15:49:18.605 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.605 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.605 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:18.605 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460261+0000", 2024-09-21T15:49:18.605 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.8879999999999992e-06 2024-09-21T15:49:18.606 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.606 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.606 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:18.606 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460267+0000", 2024-09-21T15:49:18.606 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7379999999999999e-06 2024-09-21T15:49:18.606 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.606 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.606 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:18.606 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.499940+0000", 2024-09-21T15:49:18.606 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039673228999999997 2024-09-21T15:49:18.606 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.606 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.606 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:18.606 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500008+0000", 2024-09-21T15:49:18.607 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7696999999999999e-05 2024-09-21T15:49:18.607 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.607 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.607 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:18.607 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500071+0000", 2024-09-21T15:49:18.607 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3856000000000007e-05 2024-09-21T15:49:18.607 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.607 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:18.607 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.607 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.607 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.607 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:5057 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 172032~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:18.607 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.460377+0000", 2024-09-21T15:49:18.607 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.0920116100000001, 2024-09-21T15:49:18.607 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039811802, 2024-09-21T15:49:18.608 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:18.608 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:18.608 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:18.608 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:18.608 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:18.608 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 5057 2024-09-21T15:49:18.608 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.608 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:18.608 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.608 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:18.608 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460377+0000", 2024-09-21T15:49:18.608 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.608 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.608 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.608 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:18.609 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460377+0000", 2024-09-21T15:49:18.609 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.609 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.609 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.609 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:18.609 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460277+0000", 2024-09-21T15:49:18.609 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998994 2024-09-21T15:49:18.609 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.609 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.609 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:18.609 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460380+0000", 2024-09-21T15:49:18.609 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010330700000000001 2024-09-21T15:49:18.609 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.609 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.610 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:18.610 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460387+0000", 2024-09-21T15:49:18.610 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8770000000000004e-06 2024-09-21T15:49:18.610 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.610 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.610 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:18.610 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460394+0000", 2024-09-21T15:49:18.610 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.489e-06 2024-09-21T15:49:18.610 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.610 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.610 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:18.610 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500097+0000", 2024-09-21T15:49:18.610 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039702863999999997 2024-09-21T15:49:18.610 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.610 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.611 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:18.611 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500139+0000", 2024-09-21T15:49:18.611 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2262999999999998e-05 2024-09-21T15:49:18.611 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.611 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.611 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:18.611 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500189+0000", 2024-09-21T15:49:18.611 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0377999999999998e-05 2024-09-21T15:49:18.611 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.611 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:18.611 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.611 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.611 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.611 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:5059 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 221184~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:18.611 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.460637+0000", 2024-09-21T15:49:18.612 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.0917524600000004, 2024-09-21T15:49:18.612 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039661409000000002, 2024-09-21T15:49:18.612 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:18.612 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:18.612 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:18.612 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:18.612 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:18.612 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 5059 2024-09-21T15:49:18.612 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.612 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:18.612 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.612 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:18.612 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460637+0000", 2024-09-21T15:49:18.612 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.612 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.612 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.613 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:18.613 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460637+0000", 2024-09-21T15:49:18.613 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.613 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.613 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.613 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:18.613 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460400+0000", 2024-09-21T15:49:18.613 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999763 2024-09-21T15:49:18.613 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.613 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.613 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:18.613 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460639+0000", 2024-09-21T15:49:18.613 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00023958900000000001 2024-09-21T15:49:18.613 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.613 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.614 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:18.614 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460645+0000", 2024-09-21T15:49:18.614 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.609e-06 2024-09-21T15:49:18.614 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.614 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.614 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:18.614 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460649+0000", 2024-09-21T15:49:18.614 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0239999999999999e-06 2024-09-21T15:49:18.614 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.614 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.614 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:18.614 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500211+0000", 2024-09-21T15:49:18.614 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039562011000000001 2024-09-21T15:49:18.614 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.615 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.615 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:18.615 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500251+0000", 2024-09-21T15:49:18.615 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0368999999999999e-05 2024-09-21T15:49:18.615 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.615 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.615 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:18.615 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500298+0000", 2024-09-21T15:49:18.615 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6826000000000002e-05 2024-09-21T15:49:18.615 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.615 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:18.615 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.615 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.615 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.615 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:5060 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 270336~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:18.615 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.460780+0000", 2024-09-21T15:49:18.615 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.091609483, 2024-09-21T15:49:18.616 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039627057, 2024-09-21T15:49:18.616 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:18.616 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:18.616 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:18.616 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:18.616 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:18.616 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 5060 2024-09-21T15:49:18.616 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.616 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:18.616 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.616 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:18.616 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460780+0000", 2024-09-21T15:49:18.616 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.616 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.616 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.616 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:18.617 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460780+0000", 2024-09-21T15:49:18.617 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.617 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.617 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.617 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:18.617 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460655+0000", 2024-09-21T15:49:18.617 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998755 2024-09-21T15:49:18.617 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.617 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.617 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:18.617 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460782+0000", 2024-09-21T15:49:18.617 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012715299999999999 2024-09-21T15:49:18.617 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.617 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.617 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:18.617 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460789+0000", 2024-09-21T15:49:18.618 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6959999999999999e-06 2024-09-21T15:49:18.618 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.618 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.618 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:18.618 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460794+0000", 2024-09-21T15:49:18.618 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1900000000000003e-06 2024-09-21T15:49:18.618 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.618 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.618 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:18.618 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500322+0000", 2024-09-21T15:49:18.618 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039527481000000003 2024-09-21T15:49:18.618 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.618 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.619 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:18.619 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500360+0000", 2024-09-21T15:49:18.619 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8674999999999998e-05 2024-09-21T15:49:18.619 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.619 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.619 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:18.619 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500407+0000", 2024-09-21T15:49:18.619 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6316999999999997e-05 2024-09-21T15:49:18.619 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.619 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:18.619 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.619 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.619 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.620 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:5061 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 319488~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:18.620 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.461015+0000", 2024-09-21T15:49:18.620 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.0913742710000003, 2024-09-21T15:49:18.620 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039499536000000002, 2024-09-21T15:49:18.620 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:18.620 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:18.620 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:18.620 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:18.620 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:18.620 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 5061 2024-09-21T15:49:18.620 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.620 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:18.620 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.620 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:18.620 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461015+0000", 2024-09-21T15:49:18.620 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.620 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.621 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.621 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:18.621 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461015+0000", 2024-09-21T15:49:18.621 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.621 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.621 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.621 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:18.621 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460799+0000", 2024-09-21T15:49:18.621 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9997845 2024-09-21T15:49:18.621 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.621 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.621 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:18.621 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461018+0000", 2024-09-21T15:49:18.621 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00021889000000000001 2024-09-21T15:49:18.622 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.622 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.622 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:18.622 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461026+0000", 2024-09-21T15:49:18.622 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1380000000000007e-06 2024-09-21T15:49:18.622 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.622 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.622 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:18.622 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461032+0000", 2024-09-21T15:49:18.622 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8799999999999996e-06 2024-09-21T15:49:18.622 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.622 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.622 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:18.622 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500429+0000", 2024-09-21T15:49:18.623 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039396704999999997 2024-09-21T15:49:18.623 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.623 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.623 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:18.623 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500467+0000", 2024-09-21T15:49:18.623 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7938999999999997e-05 2024-09-21T15:49:18.623 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.623 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.623 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:18.623 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500514+0000", 2024-09-21T15:49:18.623 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7562000000000003e-05 2024-09-21T15:49:18.623 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.623 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:18.623 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.623 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.624 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.624 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:5062 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 368640~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:18.624 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.461158+0000", 2024-09-21T15:49:18.624 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.0912308810000004, 2024-09-21T15:49:18.624 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039463221, 2024-09-21T15:49:18.624 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:18.624 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:18.624 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:18.624 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:18.624 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:18.624 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 5062 2024-09-21T15:49:18.624 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.624 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:18.624 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.624 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:18.625 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461158+0000", 2024-09-21T15:49:18.625 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.625 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.625 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.625 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:18.625 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461158+0000", 2024-09-21T15:49:18.625 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.625 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.625 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.625 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:18.625 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461037+0000", 2024-09-21T15:49:18.625 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998789 2024-09-21T15:49:18.625 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.625 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.626 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:18.626 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461162+0000", 2024-09-21T15:49:18.626 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000124461 2024-09-21T15:49:18.626 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.626 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.626 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:18.626 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461170+0000", 2024-09-21T15:49:18.626 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6789999999999994e-06 2024-09-21T15:49:18.626 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.626 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.626 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:18.626 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461176+0000", 2024-09-21T15:49:18.626 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0100000000000001e-06 2024-09-21T15:49:18.626 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.626 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.627 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:18.627 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500537+0000", 2024-09-21T15:49:18.627 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039360366000000001 2024-09-21T15:49:18.627 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.627 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.627 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:18.627 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500575+0000", 2024-09-21T15:49:18.627 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8126000000000001e-05 2024-09-21T15:49:18.627 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.627 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.627 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:18.627 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500621+0000", 2024-09-21T15:49:18.627 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6697999999999999e-05 2024-09-21T15:49:18.627 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.627 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:18.628 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.628 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.628 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.628 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:5063 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 417792~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:18.628 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.461421+0000", 2024-09-21T15:49:18.629 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.0909681859999996, 2024-09-21T15:49:18.629 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039308520999999999, 2024-09-21T15:49:18.629 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:18.629 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:18.629 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:18.629 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:18.630 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:18.630 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 5063 2024-09-21T15:49:18.630 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.630 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:18.630 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.630 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:18.630 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461421+0000", 2024-09-21T15:49:18.630 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.630 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.630 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.630 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:18.630 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461421+0000", 2024-09-21T15:49:18.630 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.630 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.630 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.631 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:18.631 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461181+0000", 2024-09-21T15:49:18.631 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9997602 2024-09-21T15:49:18.631 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.631 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.631 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:18.631 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461426+0000", 2024-09-21T15:49:18.631 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00024468599999999998 2024-09-21T15:49:18.631 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.631 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.631 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:18.631 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461438+0000", 2024-09-21T15:49:18.631 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2527000000000001e-05 2024-09-21T15:49:18.631 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.631 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.631 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:18.632 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461446+0000", 2024-09-21T15:49:18.632 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.199e-06 2024-09-21T15:49:18.632 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.632 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.632 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:18.632 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500645+0000", 2024-09-21T15:49:18.632 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039199680000000001 2024-09-21T15:49:18.632 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.632 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.632 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:18.632 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500684+0000", 2024-09-21T15:49:18.632 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9103999999999997e-05 2024-09-21T15:49:18.632 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.632 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.632 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:18.633 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500729+0000", 2024-09-21T15:49:18.633 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4941999999999998e-05 2024-09-21T15:49:18.633 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.633 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:18.633 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.633 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.633 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.633 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:5064 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 466944~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:18.633 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.461542+0000", 2024-09-21T15:49:18.633 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.0908466790000002, 2024-09-21T15:49:18.633 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.03938125, 2024-09-21T15:49:18.633 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:18.633 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:18.633 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:18.634 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:18.634 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:18.634 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 5064 2024-09-21T15:49:18.634 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.634 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:18.634 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.634 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:18.634 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461542+0000", 2024-09-21T15:49:18.634 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.634 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.634 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.634 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:18.634 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461542+0000", 2024-09-21T15:49:18.634 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:18.634 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.635 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.635 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:18.635 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461457+0000", 2024-09-21T15:49:18.635 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999146 2024-09-21T15:49:18.635 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.635 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.635 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:18.635 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461546+0000", 2024-09-21T15:49:18.635 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8718e-05 2024-09-21T15:49:18.635 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.635 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.635 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:18.636 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461555+0000", 2024-09-21T15:49:18.636 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8370000000000003e-06 2024-09-21T15:49:18.636 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.636 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.636 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:18.636 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461561+0000", 2024-09-21T15:49:18.636 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4150000000000004e-06 2024-09-21T15:49:18.636 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.636 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.636 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:18.636 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500753+0000", 2024-09-21T15:49:18.636 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039192183999999998 2024-09-21T15:49:18.636 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.636 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.636 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:18.637 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500826+0000", 2024-09-21T15:49:18.637 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2428000000000005e-05 2024-09-21T15:49:18.637 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:18.637 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:18.637 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:18.637 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500924+0000", 2024-09-21T15:49:18.637 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7848000000000002e-05 2024-09-21T15:49:18.637 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.637 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:18.637 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.637 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:18.637 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:18.637 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:18.638 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T15:49:18.665 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is set 2024-09-21T15:49:18.675 INFO:teuthology.orchestra.run.smithi089.stdout:750 read 0x2518073 thru 0x251c34a (0x42d8 bytes) 2024-09-21T15:49:18.678 INFO:teuthology.orchestra.run.smithi089.stdout:755 punch from 0x8ccfc09 to 0x8cd00df, (0x4d6 bytes) 2024-09-21T15:49:18.678 INFO:teuthology.orchestra.run.smithi089.stdout:757 write 0x8c0ce3c thru 0x8c106e1 (0x38a6 bytes) 2024-09-21T15:49:18.679 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:18.678+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:18.705 INFO:teuthology.orchestra.run.smithi089.stdout:758 write 0xc760d47 thru 0xc7698fd (0x8bb7 bytes) 2024-09-21T15:49:18.722 INFO:teuthology.orchestra.run.smithi089.stdout:760 trunc from 0xc7698fe to 0xad9e6af 2024-09-21T15:49:18.739 INFO:teuthology.orchestra.run.smithi089.stdout:761 read 0x197d490 thru 0x1986846 (0x93b7 bytes) 2024-09-21T15:49:18.739 INFO:teuthology.orchestra.run.smithi089.stdout:762 read 0x1d7d606 thru 0x1d8987a (0xc275 bytes) 2024-09-21T15:49:18.739 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:18.739 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:18.739 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:18.739 INFO:teuthology.orchestra.run.smithi089.stdout:766 write 0x5fb9289 thru 0x5fc45c0 (0xb338 bytes) 2024-09-21T15:49:18.752 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:18.751+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:18.756 INFO:teuthology.orchestra.run.smithi089.stdout:770 writesame 0x7b7b8f3 thru 0x7b89d4f data_size 0xd(0xe45d bytes) 2024-09-21T15:49:18.773 INFO:teuthology.orchestra.run.smithi089.stdout:771 read 0x1583511 thru 0x1589b19 (0x6609 bytes) 2024-09-21T15:49:18.773 INFO:teuthology.orchestra.run.smithi089.stdout:772 read 0x56a95d8 thru 0x56b4c1d (0xb646 bytes) 2024-09-21T15:49:18.775 INFO:teuthology.orchestra.run.smithi089.stdout:773 trunc from 0xad9e6af to 0xc44dc06 2024-09-21T15:49:18.780 INFO:teuthology.orchestra.run.smithi089.stdout:775 read 0x34ad67e thru 0x34af21b (0x1b9e bytes) 2024-09-21T15:49:18.781 INFO:teuthology.orchestra.run.smithi089.stdout:779 writesame 0xc315efd thru 0xc31d726 data_size 0x9(0x782a bytes) 2024-09-21T15:49:18.799 INFO:teuthology.orchestra.run.smithi089.stdout:780 write 0xe83bdb9 thru 0xe844be0 (0x8e28 bytes) 2024-09-21T15:49:18.802 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:18.802 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:18.802 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:49:18.802 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:18.813 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T15:49:18.819 INFO:teuthology.orchestra.run.smithi089.stdout:782 punch from 0xc2440a4 to 0xc24608d, (0x1fe9 bytes) 2024-09-21T15:49:18.829 INFO:teuthology.orchestra.run.smithi089.stdout:784 trunc from 0xe844be1 to 0x68aadc1 2024-09-21T15:49:18.853 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:18.852+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:18.866 INFO:teuthology.orchestra.run.smithi089.stdout:785 flatten 2024-09-21T15:49:18.888 INFO:teuthology.orchestra.run.smithi089.stdout:786 writesame 0xe37ddf1 thru 0xe38171e data_size 0xd(0x392e bytes) 2024-09-21T15:49:18.888 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:18.888 INFO:teuthology.orchestra.run.smithi089.stdout:788 read 0x7826ade thru 0x782bacc (0x4fef bytes) 2024-09-21T15:49:18.888 INFO:teuthology.orchestra.run.smithi089.stdout:789 read 0x97a35b8 thru 0x97aa18c (0x6bd5 bytes) 2024-09-21T15:49:18.888 INFO:teuthology.orchestra.run.smithi089.stdout:791 write 0x4c56362 thru 0x4c5f4c6 (0x9165 bytes) 2024-09-21T15:49:18.905 INFO:teuthology.orchestra.run.smithi089.stdout:793 trunc from 0xe38171f to 0xd1d0df9 2024-09-21T15:49:18.911 INFO:teuthology.orchestra.run.smithi089.stdout:794 writesame 0x3dfff01 thru 0x3e0085e data_size 0xb(0x95e bytes) 2024-09-21T15:49:18.915 INFO:teuthology.orchestra.run.smithi089.stdout:796 punch from 0xca56d39 to 0xca5e94b, (0x7c12 bytes) 2024-09-21T15:49:18.925 INFO:teuthology.orchestra.run.smithi089.stdout:797 trunc from 0xd1d0df9 to 0x4c4b35c 2024-09-21T15:49:18.949 INFO:teuthology.orchestra.run.smithi089.stdout:798 trunc from 0x4c4b35c to 0x77d9447 2024-09-21T15:49:18.953 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:18.952+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:18.954 INFO:teuthology.orchestra.run.smithi089.stdout:799 read 0x6cd40ec thru 0x6ce1dc6 (0xdcdb bytes) 2024-09-21T15:49:18.961 INFO:teuthology.orchestra.run.smithi089.stdout:800 trunc from 0x77d9447 to 0x1f5ae2b 2024-09-21T15:49:18.966 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:18.966 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:18.966 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:49:18.966 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:49:18.966 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:18.975 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T15:49:19.018 INFO:teuthology.orchestra.run.smithi089.stdout:802 writesame 0xe31d0aa thru 0xe325ac4 data_size 0x5(0x8a1b bytes) 2024-09-21T15:49:19.018 INFO:teuthology.orchestra.run.smithi089.stdout:804 write 0x3f36460 thru 0x3f39521 (0x30c2 bytes) 2024-09-21T15:49:19.032 INFO:teuthology.orchestra.run.smithi089.stdout:807 punch from 0x96c94ef to 0x96d01f4, (0x6d05 bytes) 2024-09-21T15:49:19.032 INFO:teuthology.orchestra.run.smithi089.stdout:808 read 0x8944fc5 thru 0x8954579 (0xf5b5 bytes) 2024-09-21T15:49:19.032 INFO:teuthology.orchestra.run.smithi089.stdout:809 punch from 0xd5e3cc1 to 0xd5e7395, (0x36d4 bytes) 2024-09-21T15:49:19.032 INFO:teuthology.orchestra.run.smithi089.stdout:810 punch from 0x7e7ba6f to 0x7e8813a, (0xc6cb bytes) 2024-09-21T15:49:19.032 INFO:teuthology.orchestra.run.smithi089.stdout:811 writesame 0xd5a3cc3 thru 0xd5a4091 data_size 0x5(0x3cf bytes) 2024-09-21T15:49:19.038 INFO:teuthology.orchestra.run.smithi089.stdout:813 punch from 0xaae9843 to 0xaaf0d5f, (0x751c bytes) 2024-09-21T15:49:19.038 INFO:teuthology.orchestra.run.smithi089.stdout:814 punch from 0x9201eb1 to 0x9202fdc, (0x112b bytes) 2024-09-21T15:49:19.038 INFO:teuthology.orchestra.run.smithi089.stdout:truncating to largest ever: 0xdccb1ad 2024-09-21T15:49:19.038 INFO:teuthology.orchestra.run.smithi089.stdout:817 trunc from 0xe325ac5 to 0xdccb1ad 2024-09-21T15:49:19.044 INFO:teuthology.orchestra.run.smithi089.stdout:818 writesame 0x571c552 thru 0x5722adb data_size 0x29(0x658a bytes) 2024-09-21T15:49:19.054 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:19.053+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:19.068 INFO:teuthology.orchestra.run.smithi089.stdout:822 trunc from 0xdccb1ad to 0x107080d 2024-09-21T15:49:19.092 INFO:teuthology.orchestra.run.smithi089.stdout:823 trunc from 0x107080d to 0x66c32b6 2024-09-21T15:49:19.101 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:49:19.107 INFO:teuthology.orchestra.run.smithi089.stdout:825 write 0xb206dbf thru 0xb2138bb (0xcafd bytes) 2024-09-21T15:49:19.129 INFO:teuthology.orchestra.run.smithi089.stdout:826 write 0x3944f55 thru 0x394c438 (0x74e4 bytes) 2024-09-21T15:49:19.132 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:19.132 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:49:19.132 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:49:19.132 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:49:19.132 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.132 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:2083 3.a 3:55b68c50:::rbd_data.11335fa2fe69.0000000000000191:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,truncate 77824] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:19.132 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.141682+0000", 2024-09-21T15:49:19.132 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.9878017410000002, 2024-09-21T15:49:19.132 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0053016579999999999, 2024-09-21T15:49:19.132 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:19.132 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:19.133 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:19.133 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:19.133 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:19.133 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2083 2024-09-21T15:49:19.133 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.133 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:19.133 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.133 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:19.133 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.141682+0000", 2024-09-21T15:49:19.133 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:19.133 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.133 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.133 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:19.133 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.141682+0000", 2024-09-21T15:49:19.133 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:19.134 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.134 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.134 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:19.134 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.141679+0000", 2024-09-21T15:49:19.134 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999976 2024-09-21T15:49:19.134 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.134 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.134 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:19.134 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.141691+0000", 2024-09-21T15:49:19.134 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1922e-05 2024-09-21T15:49:19.134 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.134 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.134 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:19.134 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.141712+0000", 2024-09-21T15:49:19.135 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0115e-05 2024-09-21T15:49:19.135 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.135 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.135 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:19.135 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.141723+0000", 2024-09-21T15:49:19.135 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1071e-05 2024-09-21T15:49:19.135 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.135 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.135 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:19.135 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.141776+0000", 2024-09-21T15:49:19.135 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3497e-05 2024-09-21T15:49:19.135 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.135 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.135 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:19.136 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.141918+0000", 2024-09-21T15:49:19.136 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000141433 2024-09-21T15:49:19.136 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.136 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.136 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-21T15:49:19.136 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.142129+0000", 2024-09-21T15:49:19.136 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00021179700000000001 2024-09-21T15:49:19.136 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.136 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.136 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:19.136 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.144018+0000", 2024-09-21T15:49:19.136 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0018886949999999999 2024-09-21T15:49:19.136 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.136 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.137 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:19.137 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.146889+0000", 2024-09-21T15:49:19.137 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002871017 2024-09-21T15:49:19.137 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.137 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.137 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:19.137 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.146947+0000", 2024-09-21T15:49:19.137 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7558e-05 2024-09-21T15:49:19.137 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.137 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.137 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:19.137 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.146984+0000", 2024-09-21T15:49:19.137 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7095999999999999e-05 2024-09-21T15:49:19.137 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:19.138 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:19.138 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:19.138 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.138 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.138 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:5263 3.a 3:573cf1c4:::rbd_data.11335fa2fe69.00000000000000b6:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:19.138 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.602268+0000", 2024-09-21T15:49:19.138 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.5272156969999999, 2024-09-21T15:49:19.138 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0039424129999999996, 2024-09-21T15:49:19.138 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:19.138 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:19.138 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:19.138 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:19.138 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:19.138 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 5263 2024-09-21T15:49:19.139 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.139 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:19.139 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.139 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:19.139 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.602268+0000", 2024-09-21T15:49:19.139 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:19.139 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.139 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.139 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:19.139 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.602268+0000", 2024-09-21T15:49:19.139 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:19.139 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.139 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.139 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:19.140 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.602266+0000", 2024-09-21T15:49:19.140 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999981 2024-09-21T15:49:19.140 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.140 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.140 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:19.140 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.602277+0000", 2024-09-21T15:49:19.140 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1090999999999999e-05 2024-09-21T15:49:19.140 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.140 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.140 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:19.140 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.602299+0000", 2024-09-21T15:49:19.140 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1532999999999999e-05 2024-09-21T15:49:19.140 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.140 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.140 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:19.141 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.602309+0000", 2024-09-21T15:49:19.141 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9650000000000003e-06 2024-09-21T15:49:19.141 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.141 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.141 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:19.141 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.602360+0000", 2024-09-21T15:49:19.141 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1195e-05 2024-09-21T15:49:19.141 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.141 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.141 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:19.141 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.602447+0000", 2024-09-21T15:49:19.141 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6561e-05 2024-09-21T15:49:19.141 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.141 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.141 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-21T15:49:19.142 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.602559+0000", 2024-09-21T15:49:19.142 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000112082 2024-09-21T15:49:19.142 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.142 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.142 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:19.142 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.603104+0000", 2024-09-21T15:49:19.142 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00054522499999999998 2024-09-21T15:49:19.142 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.142 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.142 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:19.142 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.606125+0000", 2024-09-21T15:49:19.142 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0030209859999999998 2024-09-21T15:49:19.142 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.143 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.143 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:19.143 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.606171+0000", 2024-09-21T15:49:19.143 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6529000000000001e-05 2024-09-21T15:49:19.143 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.143 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.143 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:19.143 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.606210+0000", 2024-09-21T15:49:19.143 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9109000000000001e-05 2024-09-21T15:49:19.143 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:19.143 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:19.143 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:19.143 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.143 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.144 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:13260 3.10 3:0d698a3f:::rbd_data.1133dc5adfdf.0000000000000096:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:19.144 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.086102+0000", 2024-09-21T15:49:19.144 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 2.04338189, 2024-09-21T15:49:19.144 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0050922229999999999, 2024-09-21T15:49:19.144 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:19.144 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:19.144 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:19.144 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:19.144 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:19.144 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 13260 2024-09-21T15:49:19.144 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.144 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:19.144 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.144 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:19.144 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.086102+0000", 2024-09-21T15:49:19.145 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:19.145 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.145 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.145 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:19.145 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.086102+0000", 2024-09-21T15:49:19.145 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:19.145 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.145 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.145 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:19.145 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.086101+0000", 2024-09-21T15:49:19.145 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:19.145 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.145 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.145 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:19.146 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.086104+0000", 2024-09-21T15:49:19.146 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1980000000000001e-06 2024-09-21T15:49:19.146 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.146 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.146 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:19.146 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.086112+0000", 2024-09-21T15:49:19.146 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5290000000000003e-06 2024-09-21T15:49:19.146 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.146 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.146 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:19.146 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.086120+0000", 2024-09-21T15:49:19.146 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9489999999999996e-06 2024-09-21T15:49:19.146 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.146 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.147 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:19.147 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.086526+0000", 2024-09-21T15:49:19.147 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000406103 2024-09-21T15:49:19.147 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.147 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.147 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:19.147 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.086596+0000", 2024-09-21T15:49:19.147 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9675e-05 2024-09-21T15:49:19.147 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.147 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.147 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0", 2024-09-21T15:49:19.147 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.086688+0000", 2024-09-21T15:49:19.147 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2576000000000003e-05 2024-09-21T15:49:19.147 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.148 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.148 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:19.148 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.089471+0000", 2024-09-21T15:49:19.148 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002782423 2024-09-21T15:49:19.148 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.148 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.148 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:19.148 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.091148+0000", 2024-09-21T15:49:19.148 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0016771379999999999 2024-09-21T15:49:19.148 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.148 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.148 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:19.148 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.091173+0000", 2024-09-21T15:49:19.148 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5347e-05 2024-09-21T15:49:19.148 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.149 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.149 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:19.149 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.091194+0000", 2024-09-21T15:49:19.149 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0888999999999999e-05 2024-09-21T15:49:19.149 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:19.149 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:19.149 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:19.149 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.149 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.149 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:13274 3.10 3:0b1f2304:::rbd_data.1133dc5adfdf.00000000000000b9:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:19.149 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.090744+0000", 2024-09-21T15:49:19.149 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 2.0387400800000002, 2024-09-21T15:49:19.149 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0082153750000000005, 2024-09-21T15:49:19.149 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:19.149 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:19.149 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:19.150 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:19.150 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:19.150 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 13274 2024-09-21T15:49:19.150 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.150 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:19.150 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.150 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:19.150 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.090744+0000", 2024-09-21T15:49:19.150 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:19.150 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.150 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.150 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:19.150 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.090744+0000", 2024-09-21T15:49:19.150 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:19.151 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.151 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.151 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:19.151 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.090743+0000", 2024-09-21T15:49:19.151 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-21T15:49:19.151 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.151 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.151 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:19.151 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.090747+0000", 2024-09-21T15:49:19.151 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.262e-06 2024-09-21T15:49:19.151 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.151 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.151 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:19.151 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.090758+0000", 2024-09-21T15:49:19.151 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0587999999999999e-05 2024-09-21T15:49:19.151 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.152 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.152 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:19.152 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.090762+0000", 2024-09-21T15:49:19.152 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7779999999999998e-06 2024-09-21T15:49:19.152 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.152 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.152 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:19.152 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.090799+0000", 2024-09-21T15:49:19.152 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6940999999999997e-05 2024-09-21T15:49:19.152 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.152 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.152 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:19.152 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.090847+0000", 2024-09-21T15:49:19.152 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7975000000000001e-05 2024-09-21T15:49:19.152 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.153 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.153 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0", 2024-09-21T15:49:19.153 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.090937+0000", 2024-09-21T15:49:19.153 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0148000000000005e-05 2024-09-21T15:49:19.153 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.153 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.153 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:19.153 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.094607+0000", 2024-09-21T15:49:19.153 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0036694029999999999 2024-09-21T15:49:19.153 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.153 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.153 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:19.153 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.098877+0000", 2024-09-21T15:49:19.153 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0042697239999999999 2024-09-21T15:49:19.153 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.154 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.154 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:19.154 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.098920+0000", 2024-09-21T15:49:19.154 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3547999999999997e-05 2024-09-21T15:49:19.154 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.154 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.154 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:19.154 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.098959+0000", 2024-09-21T15:49:19.154 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8890000000000002e-05 2024-09-21T15:49:19.154 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:19.154 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:19.154 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:19.154 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.154 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.155 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:13305 3.a 3:53172455:::rbd_data.1133dc5adfdf.00000000000000cd:head [set-alloc-hint object_size 1048576 write_size 1048576,write 540672~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:19.155 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.126122+0000", 2024-09-21T15:49:19.155 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 2.0033613699999999, 2024-09-21T15:49:19.155 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0051376010000000003, 2024-09-21T15:49:19.155 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:19.155 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:19.155 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:19.155 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:19.155 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:19.155 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 13305 2024-09-21T15:49:19.155 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.155 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:19.155 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.155 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:19.156 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.126122+0000", 2024-09-21T15:49:19.156 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:19.156 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.156 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.156 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:19.156 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.126122+0000", 2024-09-21T15:49:19.156 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:19.156 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.156 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.156 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:19.156 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.126121+0000", 2024-09-21T15:49:19.156 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:19.156 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.156 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.156 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:19.157 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.126131+0000", 2024-09-21T15:49:19.157 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4870000000000008e-06 2024-09-21T15:49:19.157 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.157 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.157 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:19.157 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.126142+0000", 2024-09-21T15:49:19.157 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1123e-05 2024-09-21T15:49:19.157 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.157 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.157 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:19.157 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.126148+0000", 2024-09-21T15:49:19.157 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4250000000000003e-06 2024-09-21T15:49:19.157 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.157 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.157 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:19.158 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.126190+0000", 2024-09-21T15:49:19.158 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2063e-05 2024-09-21T15:49:19.158 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.158 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.158 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:19.158 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.126272+0000", 2024-09-21T15:49:19.158 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1569000000000005e-05 2024-09-21T15:49:19.158 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.158 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.158 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-21T15:49:19.158 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.126356+0000", 2024-09-21T15:49:19.158 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3916000000000001e-05 2024-09-21T15:49:19.158 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.159 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.159 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:19.159 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.128614+0000", 2024-09-21T15:49:19.159 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0022582129999999998 2024-09-21T15:49:19.159 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.159 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.159 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:19.159 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.131210+0000", 2024-09-21T15:49:19.159 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0025964069999999998 2024-09-21T15:49:19.159 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.159 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.159 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:19.159 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.131239+0000", 2024-09-21T15:49:19.159 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8796999999999999e-05 2024-09-21T15:49:19.159 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.160 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.160 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:19.160 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.131260+0000", 2024-09-21T15:49:19.160 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0928999999999999e-05 2024-09-21T15:49:19.160 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:19.160 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:19.160 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:19.160 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.160 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.160 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4403.0:13426 3.1e e31/22 3:7bf656a2:::rbd_data.1133dc5adfdf.000000000000003b:head v 31'56, mlcod=31'54)", 2024-09-21T15:49:19.160 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.209924+0000", 2024-09-21T15:49:19.160 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 1.9195600690000001, 2024-09-21T15:49:19.160 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0043380529999999997, 2024-09-21T15:49:19.160 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:19.161 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:19.161 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:19.161 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.161 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:19.161 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.209924+0000", 2024-09-21T15:49:19.161 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:19.161 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.161 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.161 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:19.161 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.209924+0000", 2024-09-21T15:49:19.161 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:19.161 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.161 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.161 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:19.161 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.209923+0000", 2024-09-21T15:49:19.162 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:19.162 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.162 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.162 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:19.162 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.209930+0000", 2024-09-21T15:49:19.162 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.43e-06 2024-09-21T15:49:19.162 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.162 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.162 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:19.162 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.209940+0000", 2024-09-21T15:49:19.162 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0869999999999996e-06 2024-09-21T15:49:19.162 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.162 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.162 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:19.162 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.209946+0000", 2024-09-21T15:49:19.163 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9569999999999999e-06 2024-09-21T15:49:19.163 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.163 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.163 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:19.163 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.209974+0000", 2024-09-21T15:49:19.163 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8005999999999999e-05 2024-09-21T15:49:19.163 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.163 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.163 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:19.163 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.209987+0000", 2024-09-21T15:49:19.163 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2984e-05 2024-09-21T15:49:19.163 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.163 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.163 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:19.163 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.214182+0000", 2024-09-21T15:49:19.164 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0041956839999999999 2024-09-21T15:49:19.164 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.164 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.164 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:19.164 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.214262+0000", 2024-09-21T15:49:19.164 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9521e-05 2024-09-21T15:49:19.164 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:19.164 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:19.164 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:19.164 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.164 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.164 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:13550 3.a 3:5510f58f:::rbd_data.1133dc5adfdf.0000000000000080:head [set-alloc-hint object_size 1048576 write_size 1048576,write 942080~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:19.164 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.319272+0000", 2024-09-21T15:49:19.164 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 1.8102119189999999, 2024-09-21T15:49:19.164 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0068300210000000004, 2024-09-21T15:49:19.165 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:19.165 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:19.165 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:19.165 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:19.165 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:19.165 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 13550 2024-09-21T15:49:19.165 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.165 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:19.165 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.165 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:19.165 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.319272+0000", 2024-09-21T15:49:19.165 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:19.165 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.165 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.165 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:19.166 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.319272+0000", 2024-09-21T15:49:19.166 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:19.166 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.166 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.166 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:19.166 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.319270+0000", 2024-09-21T15:49:19.166 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:19.166 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.166 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.166 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:19.166 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.319282+0000", 2024-09-21T15:49:19.166 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2143999999999999e-05 2024-09-21T15:49:19.166 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.166 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.166 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:19.167 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.319297+0000", 2024-09-21T15:49:19.167 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5017999999999999e-05 2024-09-21T15:49:19.167 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.167 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.167 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:19.167 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.319303+0000", 2024-09-21T15:49:19.167 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8390000000000003e-06 2024-09-21T15:49:19.167 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.167 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.167 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:19.167 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.319355+0000", 2024-09-21T15:49:19.167 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1777999999999998e-05 2024-09-21T15:49:19.167 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.167 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.167 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:19.168 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.319441+0000", 2024-09-21T15:49:19.168 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.585e-05 2024-09-21T15:49:19.168 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.168 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.168 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-21T15:49:19.168 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.319585+0000", 2024-09-21T15:49:19.168 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000144003 2024-09-21T15:49:19.168 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.168 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.168 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:19.168 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.325771+0000", 2024-09-21T15:49:19.168 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0061861959999999997 2024-09-21T15:49:19.168 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.168 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.169 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:19.169 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.326013+0000", 2024-09-21T15:49:19.169 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00024232600000000001 2024-09-21T15:49:19.169 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.169 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.169 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:19.169 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.326063+0000", 2024-09-21T15:49:19.169 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9669000000000002e-05 2024-09-21T15:49:19.169 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.169 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.169 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:19.169 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.326102+0000", 2024-09-21T15:49:19.169 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8825999999999997e-05 2024-09-21T15:49:19.169 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:19.169 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:19.169 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:19.170 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.170 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.170 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:13575 3.a 3:5510f58f:::rbd_data.1133dc5adfdf.0000000000000080:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:19.170 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.345485+0000", 2024-09-21T15:49:19.170 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 1.7839991830000002, 2024-09-21T15:49:19.170 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0042117960000000003, 2024-09-21T15:49:19.170 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:19.170 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:19.170 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:19.170 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:19.170 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:19.170 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 13575 2024-09-21T15:49:19.170 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.170 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:19.171 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.171 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:19.171 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.345485+0000", 2024-09-21T15:49:19.171 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:19.171 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.171 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.171 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:19.171 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.345485+0000", 2024-09-21T15:49:19.171 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:19.171 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.171 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.171 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:19.171 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.345483+0000", 2024-09-21T15:49:19.171 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:19.172 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.172 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.172 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:19.172 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.345490+0000", 2024-09-21T15:49:19.172 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0539999999999997e-06 2024-09-21T15:49:19.172 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.172 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.172 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:19.172 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.345506+0000", 2024-09-21T15:49:19.172 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5792999999999999e-05 2024-09-21T15:49:19.172 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.172 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.172 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:19.172 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.345517+0000", 2024-09-21T15:49:19.173 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0825999999999999e-05 2024-09-21T15:49:19.173 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.173 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.173 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:19.173 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.345557+0000", 2024-09-21T15:49:19.173 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0028999999999999e-05 2024-09-21T15:49:19.173 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.173 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.173 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:19.173 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.345631+0000", 2024-09-21T15:49:19.173 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4071000000000003e-05 2024-09-21T15:49:19.173 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.173 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.173 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-21T15:49:19.173 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.345715+0000", 2024-09-21T15:49:19.174 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3978000000000001e-05 2024-09-21T15:49:19.174 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.174 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.174 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:19.174 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.346827+0000", 2024-09-21T15:49:19.174 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0011119229999999999 2024-09-21T15:49:19.174 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.174 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.174 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:19.174 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.349648+0000", 2024-09-21T15:49:19.174 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0028214099999999999 2024-09-21T15:49:19.174 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.174 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.174 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:19.174 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.349675+0000", 2024-09-21T15:49:19.175 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6801e-05 2024-09-21T15:49:19.175 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.175 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.175 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:19.175 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.349696+0000", 2024-09-21T15:49:19.175 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1477000000000001e-05 2024-09-21T15:49:19.175 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:19.175 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:19.175 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:19.175 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.175 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.175 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:13581 3.a 3:52d3dd80:::rbd_data.1133dc5adfdf.00000000000000b0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:19.175 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.347049+0000", 2024-09-21T15:49:19.175 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 1.7824342990000002, 2024-09-21T15:49:19.175 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0051259269999999997, 2024-09-21T15:49:19.176 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:19.176 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:19.176 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:19.176 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:19.176 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:19.176 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 13581 2024-09-21T15:49:19.176 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.176 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:19.176 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.176 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:19.176 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.347049+0000", 2024-09-21T15:49:19.176 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:19.176 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.176 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.177 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:19.177 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.347049+0000", 2024-09-21T15:49:19.177 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:19.177 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.177 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.177 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:19.177 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.347049+0000", 2024-09-21T15:49:19.177 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-21T15:49:19.177 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.177 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.177 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:19.177 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.347053+0000", 2024-09-21T15:49:19.177 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5229999999999997e-06 2024-09-21T15:49:19.177 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.177 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.178 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:19.178 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.347064+0000", 2024-09-21T15:49:19.178 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0336999999999999e-05 2024-09-21T15:49:19.178 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.178 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.178 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:19.178 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.347071+0000", 2024-09-21T15:49:19.178 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1629999999999999e-06 2024-09-21T15:49:19.178 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.178 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.178 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:19.178 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.347102+0000", 2024-09-21T15:49:19.178 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1647999999999999e-05 2024-09-21T15:49:19.178 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.178 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.179 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:19.179 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.347155+0000", 2024-09-21T15:49:19.179 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3093000000000001e-05 2024-09-21T15:49:19.179 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.179 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.179 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-21T15:49:19.179 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.347221+0000", 2024-09-21T15:49:19.179 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5922999999999999e-05 2024-09-21T15:49:19.179 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.179 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.179 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:19.179 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.349565+0000", 2024-09-21T15:49:19.179 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0023435909999999999 2024-09-21T15:49:19.179 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.180 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.180 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:19.180 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.352099+0000", 2024-09-21T15:49:19.180 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0025342089999999999 2024-09-21T15:49:19.180 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.180 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.180 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:19.180 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.352141+0000", 2024-09-21T15:49:19.180 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2259000000000003e-05 2024-09-21T15:49:19.180 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.180 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.180 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:19.180 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.352175+0000", 2024-09-21T15:49:19.180 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3973000000000002e-05 2024-09-21T15:49:19.180 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:19.180 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:19.181 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:19.181 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.181 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.181 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:13833 3.10 3:0b29d3b5:::rbd_data.1133dc5adfdf.0000000000000091:head [write 897024~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:19.181 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.527148+0000", 2024-09-21T15:49:19.181 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 1.60233571, 2024-09-21T15:49:19.181 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0044907690000000004, 2024-09-21T15:49:19.181 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:19.181 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:19.181 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:19.181 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:19.181 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:19.181 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 13833 2024-09-21T15:49:19.181 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.181 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:19.182 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.182 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:19.182 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.527148+0000", 2024-09-21T15:49:19.182 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:19.182 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.182 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.182 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:19.182 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.527148+0000", 2024-09-21T15:49:19.182 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:19.182 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.182 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.182 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:19.182 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.527147+0000", 2024-09-21T15:49:19.182 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:19.183 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.183 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.183 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:19.183 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.527157+0000", 2024-09-21T15:49:19.183 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7759999999999992e-06 2024-09-21T15:49:19.183 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.183 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.183 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:19.183 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.527167+0000", 2024-09-21T15:49:19.183 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0818999999999999e-05 2024-09-21T15:49:19.183 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.183 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.183 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:19.183 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.527173+0000", 2024-09-21T15:49:19.183 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.152e-06 2024-09-21T15:49:19.184 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.184 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.184 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:19.184 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.527208+0000", 2024-09-21T15:49:19.184 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5833000000000001e-05 2024-09-21T15:49:19.184 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.184 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.184 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:19.184 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.527304+0000", 2024-09-21T15:49:19.184 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5292999999999997e-05 2024-09-21T15:49:19.184 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.184 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.184 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0", 2024-09-21T15:49:19.184 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.527459+0000", 2024-09-21T15:49:19.184 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015507499999999999 2024-09-21T15:49:19.185 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.185 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.185 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:19.185 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.528200+0000", 2024-09-21T15:49:19.185 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00074178700000000005 2024-09-21T15:49:19.185 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.185 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.185 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:19.185 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.531575+0000", 2024-09-21T15:49:19.185 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0033740580000000001 2024-09-21T15:49:19.185 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.185 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.185 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:19.185 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.531605+0000", 2024-09-21T15:49:19.186 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0321999999999999e-05 2024-09-21T15:49:19.186 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.186 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.186 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:19.186 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.531639+0000", 2024-09-21T15:49:19.186 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3933000000000002e-05 2024-09-21T15:49:19.186 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:19.186 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:19.186 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:19.186 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.186 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.186 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:13862 3.a 3:53172455:::rbd_data.1133dc5adfdf.00000000000000cd:head [set-alloc-hint object_size 1048576 write_size 1048576,write 266240~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:19.186 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.549561+0000", 2024-09-21T15:49:19.186 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 1.579922754, 2024-09-21T15:49:19.186 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0065617749999999997, 2024-09-21T15:49:19.186 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:19.187 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:19.187 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:19.187 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:19.187 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:19.187 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 13862 2024-09-21T15:49:19.187 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.187 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:19.187 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.187 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:19.187 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.549561+0000", 2024-09-21T15:49:19.187 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:19.187 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.187 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.187 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:19.188 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.549561+0000", 2024-09-21T15:49:19.188 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:19.188 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.188 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.188 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:19.188 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.549560+0000", 2024-09-21T15:49:19.188 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:19.188 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.188 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.188 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:19.188 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.549569+0000", 2024-09-21T15:49:19.188 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2799999999999992e-06 2024-09-21T15:49:19.188 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.188 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.188 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:19.189 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.549582+0000", 2024-09-21T15:49:19.189 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2757e-05 2024-09-21T15:49:19.189 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.189 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.189 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:19.189 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.549590+0000", 2024-09-21T15:49:19.189 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4619999999999996e-06 2024-09-21T15:49:19.189 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.189 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.189 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:19.189 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.549642+0000", 2024-09-21T15:49:19.189 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1779e-05 2024-09-21T15:49:19.189 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.189 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.189 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:19.190 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.549736+0000", 2024-09-21T15:49:19.190 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4617000000000006e-05 2024-09-21T15:49:19.190 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.190 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.190 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-21T15:49:19.190 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.549886+0000", 2024-09-21T15:49:19.190 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014938199999999999 2024-09-21T15:49:19.190 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.190 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.190 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:19.190 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.553416+0000", 2024-09-21T15:49:19.190 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0035298339999999999 2024-09-21T15:49:19.190 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.190 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.190 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:19.191 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.556032+0000", 2024-09-21T15:49:19.191 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002615994 2024-09-21T15:49:19.191 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.191 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.191 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:19.191 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.556080+0000", 2024-09-21T15:49:19.191 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8442999999999997e-05 2024-09-21T15:49:19.191 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.191 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.191 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:19.191 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.556123+0000", 2024-09-21T15:49:19.191 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2673999999999998e-05 2024-09-21T15:49:19.191 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:19.191 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:19.192 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:19.192 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.192 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.192 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:13948 3.a 3:52b31f3c:::rbd_data.1133dc5adfdf.00000000000000db:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:19.192 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.616162+0000", 2024-09-21T15:49:19.192 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 1.513321439, 2024-09-21T15:49:19.192 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.004904594, 2024-09-21T15:49:19.192 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:19.192 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:19.192 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:19.192 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:19.192 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:19.192 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 13948 2024-09-21T15:49:19.192 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.192 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:19.192 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.193 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:19.193 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.616162+0000", 2024-09-21T15:49:19.193 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:19.193 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.193 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.193 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:19.193 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.616162+0000", 2024-09-21T15:49:19.193 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:19.193 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.193 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.193 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:19.193 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.616160+0000", 2024-09-21T15:49:19.193 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999981 2024-09-21T15:49:19.193 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.193 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.194 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:19.194 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.616168+0000", 2024-09-21T15:49:19.194 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3030000000000002e-06 2024-09-21T15:49:19.194 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.194 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.194 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:19.194 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.616183+0000", 2024-09-21T15:49:19.194 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5756e-05 2024-09-21T15:49:19.194 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.194 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.194 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:19.194 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.616191+0000", 2024-09-21T15:49:19.194 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6090000000000003e-06 2024-09-21T15:49:19.194 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.194 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.195 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:19.195 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.616249+0000", 2024-09-21T15:49:19.195 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8286999999999999e-05 2024-09-21T15:49:19.195 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.195 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.195 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:19.195 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.616333+0000", 2024-09-21T15:49:19.195 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3441000000000003e-05 2024-09-21T15:49:19.195 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.195 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.195 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-21T15:49:19.195 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.616441+0000", 2024-09-21T15:49:19.195 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000108117 2024-09-21T15:49:19.195 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.195 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.196 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:19.196 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.617968+0000", 2024-09-21T15:49:19.196 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001526662 2024-09-21T15:49:19.196 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.196 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.196 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:19.196 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.620963+0000", 2024-09-21T15:49:19.196 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0029951380000000001 2024-09-21T15:49:19.196 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.196 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.196 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:19.196 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.621008+0000", 2024-09-21T15:49:19.196 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5664000000000002e-05 2024-09-21T15:49:19.196 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.197 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.197 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:19.197 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.621067+0000", 2024-09-21T15:49:19.197 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8548000000000003e-05 2024-09-21T15:49:19.197 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:19.197 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:19.197 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:19.197 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.197 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.197 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:14003 3.10 3:0b1f2304:::rbd_data.1133dc5adfdf.00000000000000b9:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:19.197 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.658416+0000", 2024-09-21T15:49:19.197 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 1.4710673679999999, 2024-09-21T15:49:19.197 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0045796250000000004, 2024-09-21T15:49:19.197 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:19.197 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:19.198 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:19.198 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:19.198 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:19.198 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 14003 2024-09-21T15:49:19.198 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.198 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:19.198 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.198 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:19.198 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.658416+0000", 2024-09-21T15:49:19.198 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:19.198 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.198 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.198 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:19.198 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.658416+0000", 2024-09-21T15:49:19.198 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:19.199 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.199 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.199 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:19.199 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.658415+0000", 2024-09-21T15:49:19.199 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-21T15:49:19.199 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.199 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.199 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:19.199 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.658422+0000", 2024-09-21T15:49:19.199 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3910000000000003e-06 2024-09-21T15:49:19.199 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.199 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.199 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:19.199 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.658433+0000", 2024-09-21T15:49:19.200 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1435e-05 2024-09-21T15:49:19.200 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.200 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.200 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:19.200 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.658443+0000", 2024-09-21T15:49:19.200 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0012e-05 2024-09-21T15:49:19.200 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.200 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.200 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:19.200 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.658485+0000", 2024-09-21T15:49:19.200 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2079000000000001e-05 2024-09-21T15:49:19.200 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.200 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.200 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:19.200 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.658550+0000", 2024-09-21T15:49:19.201 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5028000000000001e-05 2024-09-21T15:49:19.201 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.201 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.201 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0", 2024-09-21T15:49:19.201 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.658622+0000", 2024-09-21T15:49:19.201 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1381e-05 2024-09-21T15:49:19.201 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.201 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.201 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:19.201 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.660779+0000", 2024-09-21T15:49:19.201 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0021570270000000002 2024-09-21T15:49:19.201 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.201 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.201 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:19.202 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.662939+0000", 2024-09-21T15:49:19.202 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0021602119999999999 2024-09-21T15:49:19.202 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.202 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.202 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:19.202 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.662968+0000", 2024-09-21T15:49:19.202 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9309999999999999e-05 2024-09-21T15:49:19.202 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.202 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.202 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:19.202 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.662996+0000", 2024-09-21T15:49:19.202 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.792e-05 2024-09-21T15:49:19.202 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:19.203 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:19.203 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:19.203 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.203 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.203 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:14017 3.a 3:53172455:::rbd_data.1133dc5adfdf.00000000000000cd:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-21T15:49:19.203 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.663573+0000", 2024-09-21T15:49:19.203 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 1.465910609, 2024-09-21T15:49:19.203 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.014018708, 2024-09-21T15:49:19.203 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:19.203 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:19.203 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:19.203 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:19.203 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:19.203 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 14017 2024-09-21T15:49:19.203 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.204 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:19.204 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.204 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:19.204 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.663573+0000", 2024-09-21T15:49:19.204 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:19.204 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.204 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.204 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:19.204 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.663573+0000", 2024-09-21T15:49:19.204 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:19.204 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.204 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.204 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:19.204 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.663572+0000", 2024-09-21T15:49:19.205 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:19.205 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.205 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.205 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:19.205 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.663579+0000", 2024-09-21T15:49:19.205 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4340000000000003e-06 2024-09-21T15:49:19.205 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.205 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.205 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:19.205 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.663594+0000", 2024-09-21T15:49:19.205 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4814000000000001e-05 2024-09-21T15:49:19.205 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.205 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.205 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:19.206 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.663602+0000", 2024-09-21T15:49:19.206 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7030000000000006e-06 2024-09-21T15:49:19.206 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.206 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.206 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:19.206 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.663652+0000", 2024-09-21T15:49:19.206 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0866999999999999e-05 2024-09-21T15:49:19.206 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.206 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.206 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-21T15:49:19.206 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.663659+0000", 2024-09-21T15:49:19.206 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7909999999999999e-06 2024-09-21T15:49:19.206 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.206 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.206 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:19.207 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.672664+0000", 2024-09-21T15:49:19.207 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0090051469999999998 2024-09-21T15:49:19.207 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.207 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.207 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:19.207 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.672740+0000", 2024-09-21T15:49:19.207 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5777000000000003e-05 2024-09-21T15:49:19.207 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.207 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.207 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-21T15:49:19.207 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.672840+0000", 2024-09-21T15:49:19.207 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000100108 2024-09-21T15:49:19.207 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.207 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.208 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:19.208 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.674807+0000", 2024-09-21T15:49:19.208 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0019661930000000002 2024-09-21T15:49:19.208 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.208 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.208 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:19.208 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.677520+0000", 2024-09-21T15:49:19.208 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0027133270000000002 2024-09-21T15:49:19.208 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.208 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.208 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:19.208 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.677567+0000", 2024-09-21T15:49:19.208 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7307e-05 2024-09-21T15:49:19.208 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.208 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.208 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:19.209 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.677592+0000", 2024-09-21T15:49:19.209 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4711000000000001e-05 2024-09-21T15:49:19.209 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:19.209 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:19.209 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:19.209 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.209 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.209 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:14161 3.a 3:5510f58f:::rbd_data.1133dc5adfdf.0000000000000080:head [set-alloc-hint object_size 1048576 write_size 1048576,write 303104~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-21T15:49:19.209 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:18.766261+0000", 2024-09-21T15:49:19.209 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.363222512, 2024-09-21T15:49:19.209 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0042368839999999998, 2024-09-21T15:49:19.209 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:19.209 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:19.209 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:19.209 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:19.210 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:19.210 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 14161 2024-09-21T15:49:19.210 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.210 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:19.210 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.210 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:19.210 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.766261+0000", 2024-09-21T15:49:19.210 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:19.210 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.210 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.210 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:19.210 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.766261+0000", 2024-09-21T15:49:19.210 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:19.210 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.210 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.211 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:19.211 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.766259+0000", 2024-09-21T15:49:19.211 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999976 2024-09-21T15:49:19.211 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.211 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.211 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:19.211 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.766273+0000", 2024-09-21T15:49:19.211 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3838e-05 2024-09-21T15:49:19.211 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.211 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.211 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:19.211 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.766289+0000", 2024-09-21T15:49:19.211 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6246e-05 2024-09-21T15:49:19.211 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.212 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.212 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:19.212 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.766296+0000", 2024-09-21T15:49:19.212 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6069999999999996e-06 2024-09-21T15:49:19.212 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.212 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.212 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:19.212 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.766353+0000", 2024-09-21T15:49:19.212 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6860000000000001e-05 2024-09-21T15:49:19.212 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.212 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.212 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:19.212 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.766429+0000", 2024-09-21T15:49:19.212 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6626999999999997e-05 2024-09-21T15:49:19.212 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.212 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.212 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-21T15:49:19.213 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.766596+0000", 2024-09-21T15:49:19.213 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016733500000000001 2024-09-21T15:49:19.213 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.213 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.213 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:19.213 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.769704+0000", 2024-09-21T15:49:19.213 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0031074560000000002 2024-09-21T15:49:19.213 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.213 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.213 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:19.213 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.770415+0000", 2024-09-21T15:49:19.213 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00071146499999999999 2024-09-21T15:49:19.213 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.213 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.214 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:19.214 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.770463+0000", 2024-09-21T15:49:19.214 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7330999999999998e-05 2024-09-21T15:49:19.214 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.214 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.214 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:19.214 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.770498+0000", 2024-09-21T15:49:19.214 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5380999999999999e-05 2024-09-21T15:49:19.214 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:19.214 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:19.214 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:19.214 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.214 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.214 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:14245 3.a 3:5510f58f:::rbd_data.1133dc5adfdf.0000000000000080:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-21T15:49:19.214 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:18.831104+0000", 2024-09-21T15:49:19.215 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.29838009300000001, 2024-09-21T15:49:19.215 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0047728409999999999, 2024-09-21T15:49:19.215 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:19.215 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:19.215 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:19.215 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:19.215 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:19.215 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 14245 2024-09-21T15:49:19.215 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.215 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:19.215 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.215 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:19.215 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.831104+0000", 2024-09-21T15:49:19.215 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:19.216 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.216 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.216 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:19.216 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.831104+0000", 2024-09-21T15:49:19.216 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:19.216 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.216 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.216 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:19.216 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.831102+0000", 2024-09-21T15:49:19.216 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999981 2024-09-21T15:49:19.216 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.216 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.216 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:19.216 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.831111+0000", 2024-09-21T15:49:19.217 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7899999999999994e-06 2024-09-21T15:49:19.217 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.217 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.217 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:19.217 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.831132+0000", 2024-09-21T15:49:19.217 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0693e-05 2024-09-21T15:49:19.217 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.217 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.217 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:19.217 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.831142+0000", 2024-09-21T15:49:19.217 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5400000000000001e-06 2024-09-21T15:49:19.217 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.217 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.217 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:19.218 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.831197+0000", 2024-09-21T15:49:19.218 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5078999999999999e-05 2024-09-21T15:49:19.218 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.218 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.218 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:19.218 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.831289+0000", 2024-09-21T15:49:19.218 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2614000000000005e-05 2024-09-21T15:49:19.218 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.218 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.218 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-21T15:49:19.218 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.831401+0000", 2024-09-21T15:49:19.218 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000111457 2024-09-21T15:49:19.218 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.218 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.219 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:19.219 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.833162+0000", 2024-09-21T15:49:19.219 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0017607009999999999 2024-09-21T15:49:19.219 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.219 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.219 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:19.219 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.835816+0000", 2024-09-21T15:49:19.219 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0026548510000000002 2024-09-21T15:49:19.219 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.219 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.219 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:19.219 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.835845+0000", 2024-09-21T15:49:19.219 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8538e-05 2024-09-21T15:49:19.219 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.220 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.220 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:19.220 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.835876+0000", 2024-09-21T15:49:19.220 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1565e-05 2024-09-21T15:49:19.220 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:19.220 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:19.220 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:19.220 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.220 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.220 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4403.0:14243 3.8 e33/32 3:154c00a4:::rbd_data.1133dc5adfdf.000000000000007e:head v 33'52, mlcod=33'50)", 2024-09-21T15:49:19.220 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:18.831614+0000", 2024-09-21T15:49:19.220 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.29786994900000002, 2024-09-21T15:49:19.220 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0039511540000000001, 2024-09-21T15:49:19.220 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:19.220 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:19.220 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:19.221 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.221 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:19.221 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.831614+0000", 2024-09-21T15:49:19.221 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:19.221 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.221 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.221 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:19.221 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.831614+0000", 2024-09-21T15:49:19.221 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:19.221 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.221 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.221 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:19.221 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.831613+0000", 2024-09-21T15:49:19.221 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-21T15:49:19.221 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.221 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.222 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:19.222 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.831621+0000", 2024-09-21T15:49:19.222 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0609999999999996e-06 2024-09-21T15:49:19.222 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.222 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.222 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:19.222 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.831632+0000", 2024-09-21T15:49:19.222 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1610000000000001e-05 2024-09-21T15:49:19.222 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.222 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.222 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:19.222 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.831643+0000", 2024-09-21T15:49:19.222 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0448999999999999e-05 2024-09-21T15:49:19.222 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.222 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.222 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:19.223 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.831710+0000", 2024-09-21T15:49:19.223 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7569000000000003e-05 2024-09-21T15:49:19.223 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.223 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.223 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:19.223 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.831731+0000", 2024-09-21T15:49:19.223 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0845000000000001e-05 2024-09-21T15:49:19.223 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.223 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.223 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:19.223 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.835530+0000", 2024-09-21T15:49:19.223 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0037991190000000001 2024-09-21T15:49:19.223 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.223 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.223 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:19.224 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.835565+0000", 2024-09-21T15:49:19.224 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.455e-05 2024-09-21T15:49:19.224 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:19.224 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:19.224 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:19.224 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.224 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.224 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4403.0:14408 3.8 e33/32 3:13437810:::rbd_data.1133dc5adfdf.0000000000000039:head v 33'55, mlcod=33'52)", 2024-09-21T15:49:19.224 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:18.971354+0000", 2024-09-21T15:49:19.224 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.15812984899999999, 2024-09-21T15:49:19.224 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0048492420000000001, 2024-09-21T15:49:19.224 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:19.224 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:19.224 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:19.224 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.224 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:19.225 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.971354+0000", 2024-09-21T15:49:19.225 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:19.225 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.225 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.225 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:19.225 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.971354+0000", 2024-09-21T15:49:19.225 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:19.225 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.225 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.225 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:19.225 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.971353+0000", 2024-09-21T15:49:19.225 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:19.225 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.225 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.225 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:19.226 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.971359+0000", 2024-09-21T15:49:19.226 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6829999999999996e-06 2024-09-21T15:49:19.226 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.226 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.226 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:19.226 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.971370+0000", 2024-09-21T15:49:19.226 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0776e-05 2024-09-21T15:49:19.226 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.226 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.226 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:19.226 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.971375+0000", 2024-09-21T15:49:19.226 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6670000000000001e-06 2024-09-21T15:49:19.226 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.226 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.227 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:19.227 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.971414+0000", 2024-09-21T15:49:19.227 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8376e-05 2024-09-21T15:49:19.227 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.227 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.227 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:19.227 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.971432+0000", 2024-09-21T15:49:19.227 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8603000000000001e-05 2024-09-21T15:49:19.227 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.227 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.227 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:19.227 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.976169+0000", 2024-09-21T15:49:19.227 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0047363099999999997 2024-09-21T15:49:19.227 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.227 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.228 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:19.228 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.976203+0000", 2024-09-21T15:49:19.228 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4473e-05 2024-09-21T15:49:19.228 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:19.228 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:19.228 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:19.228 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.228 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.228 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4403.0:14410 3.1e e33/22 3:7bf656a2:::rbd_data.1133dc5adfdf.000000000000003b:head v 33'95, mlcod=33'92)", 2024-09-21T15:49:19.228 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:18.971584+0000", 2024-09-21T15:49:19.228 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.15789968300000001, 2024-09-21T15:49:19.228 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0046296760000000001, 2024-09-21T15:49:19.228 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:19.228 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:19.228 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:19.228 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.228 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:19.229 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.971584+0000", 2024-09-21T15:49:19.229 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:19.229 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.229 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.229 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:19.229 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.971584+0000", 2024-09-21T15:49:19.229 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:19.229 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.229 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.229 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:19.229 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.971584+0000", 2024-09-21T15:49:19.229 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:19.229 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.229 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.229 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:19.230 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.971588+0000", 2024-09-21T15:49:19.230 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.34e-06 2024-09-21T15:49:19.230 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.230 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.230 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:19.230 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.971597+0000", 2024-09-21T15:49:19.230 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.85e-06 2024-09-21T15:49:19.230 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.230 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.230 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:19.230 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.971601+0000", 2024-09-21T15:49:19.230 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4780000000000002e-06 2024-09-21T15:49:19.230 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.230 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.230 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:19.231 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.971635+0000", 2024-09-21T15:49:19.231 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4190999999999998e-05 2024-09-21T15:49:19.231 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.231 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.231 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:19.231 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.971656+0000", 2024-09-21T15:49:19.231 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0608e-05 2024-09-21T15:49:19.231 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.231 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.231 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:19.231 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.976184+0000", 2024-09-21T15:49:19.231 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0045279730000000002 2024-09-21T15:49:19.231 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.231 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.231 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:19.231 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.976214+0000", 2024-09-21T15:49:19.232 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9709e-05 2024-09-21T15:49:19.232 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:19.232 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:19.232 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:19.232 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.232 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.232 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:14417 3.a 3:5715b5ca:::rbd_data.1133dc5adfdf.000000000000004e:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-21T15:49:19.232 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:18.975481+0000", 2024-09-21T15:49:19.232 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.154003062, 2024-09-21T15:49:19.232 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0087110940000000008, 2024-09-21T15:49:19.232 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:19.232 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:19.232 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:19.232 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:19.232 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:19.232 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 14417 2024-09-21T15:49:19.233 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.233 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:19.233 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.233 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:19.233 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.975481+0000", 2024-09-21T15:49:19.233 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:19.233 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.233 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.233 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:19.233 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.975481+0000", 2024-09-21T15:49:19.233 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:19.233 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.233 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.233 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:19.233 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.975479+0000", 2024-09-21T15:49:19.234 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:19.234 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.234 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.234 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:19.234 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.975485+0000", 2024-09-21T15:49:19.234 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9540000000000001e-06 2024-09-21T15:49:19.234 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.234 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.234 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:19.234 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.975497+0000", 2024-09-21T15:49:19.234 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1934e-05 2024-09-21T15:49:19.234 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.234 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.234 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:19.235 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.975505+0000", 2024-09-21T15:49:19.235 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0250000000000002e-06 2024-09-21T15:49:19.235 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.235 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.235 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:19.235 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.975552+0000", 2024-09-21T15:49:19.235 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6281e-05 2024-09-21T15:49:19.235 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.235 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.235 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:19.235 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.975638+0000", 2024-09-21T15:49:19.235 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6762999999999996e-05 2024-09-21T15:49:19.235 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.235 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.235 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-21T15:49:19.235 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.975734+0000", 2024-09-21T15:49:19.236 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5248999999999996e-05 2024-09-21T15:49:19.236 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.236 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.236 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:19.236 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.978856+0000", 2024-09-21T15:49:19.236 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0031219580000000002 2024-09-21T15:49:19.236 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.236 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.236 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:19.236 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.984106+0000", 2024-09-21T15:49:19.236 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0052503589999999996 2024-09-21T15:49:19.236 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.236 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.236 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:19.236 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.984156+0000", 2024-09-21T15:49:19.237 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0164000000000003e-05 2024-09-21T15:49:19.237 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:19.237 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:19.237 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:19.237 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.984192+0000", 2024-09-21T15:49:19.237 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5658999999999999e-05 2024-09-21T15:49:19.237 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:19.237 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:19.237 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:19.237 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:19.237 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:19.237 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:19.239 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:19.239 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:19.239 INFO:teuthology.orchestra.run.smithi089.stdout:829 read 0xa95d1ed thru 0xa96037c (0x3190 bytes) 2024-09-21T15:49:19.239 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:19.239 INFO:teuthology.orchestra.run.smithi089.stdout:831 write 0x7fc6777 thru 0x7fc8dc6 (0x2650 bytes) 2024-09-21T15:49:19.239 INFO:teuthology.orchestra.run.smithi089.stdout:832 clone 4 order 20 su 4096 sc 10 2024-09-21T15:49:19.240 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-09-21T15:49:19.240 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:49:19.240 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T15:49:19.240 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:19.237+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:19.241 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:49:19.241 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:49:19.255 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:49:19.255 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:49:19.270 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:49:19.270 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:49:19.286 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:49:19.286 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:49:19.302 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:49:19.303 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:49:19.319 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:49:19.320 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:49:19.335 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:49:19.336 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:49:19.339 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:19.338+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:19.384 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:19.384 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:19.393 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T15:49:19.440 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:19.439+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:19.494 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:19.494 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:19.502 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T15:49:19.546 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:19.545+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:19.605 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:19.605 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:19.613 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:49:19.657 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:19.656+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:19.716 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:19.716 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:19.725 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:49:19.742 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:19.741+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:19.844 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:19.843+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:19.872 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:19.872 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:19.882 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:49:19.913 INFO:teuthology.orchestra.run.smithi089.stdout:checking clone #2, image image_client.0-clone2 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent3 2024-09-21T15:49:19.945 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:19.943+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:20.000 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:20.000 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:20.009 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:49:20.023 INFO:teuthology.orchestra.run.smithi089.stdout:833 read 0x2e7cef2 thru 0x2e81ec3 (0x4fd2 bytes) 2024-09-21T15:49:20.024 INFO:teuthology.orchestra.run.smithi089.stdout:835 write 0x6c5f070 thru 0x6c5f4ad (0x43e bytes) 2024-09-21T15:49:20.031 INFO:teuthology.orchestra.run.smithi089.stdout:836 write 0xa405972 thru 0xa411c45 (0xc2d4 bytes) 2024-09-21T15:49:20.046 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:20.044+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:20.061 INFO:teuthology.orchestra.run.smithi089.stdout:837 trunc from 0xb2138bc to 0x766c09f 2024-09-21T15:49:20.092 INFO:teuthology.orchestra.run.smithi089.stdout:838 punch from 0x3c73609 to 0x3c8271a, (0xf111 bytes) 2024-09-21T15:49:20.092 INFO:teuthology.orchestra.run.smithi089.stdout:839 read 0x1d2f833 thru 0x1d32cab (0x3479 bytes) 2024-09-21T15:49:20.095 INFO:teuthology.orchestra.run.smithi089.stdout:840 trunc from 0x766c09f to 0x4fe3b1b 2024-09-21T15:49:20.130 INFO:teuthology.orchestra.run.smithi089.stdout:841 punch from 0x354c0b to 0x361613, (0xca08 bytes) 2024-09-21T15:49:20.145 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:20.144+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:20.146 INFO:teuthology.orchestra.run.smithi089.stdout:843 write 0xd42497c thru 0xd4276e0 (0x2d65 bytes) 2024-09-21T15:49:20.149 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:20.149 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:20.152 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:20.152 INFO:teuthology.orchestra.run.smithi089.stdout:845 read 0x8a4d43d thru 0x8a4db98 (0x75c bytes) 2024-09-21T15:49:20.152 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:20.153 INFO:teuthology.orchestra.run.smithi089.stdout:847 read 0x69c6c84 thru 0x69d1660 (0xa9dd bytes) 2024-09-21T15:49:20.153 INFO:teuthology.orchestra.run.smithi089.stdout:849 writesame 0xb2ec535 thru 0xb2f8828 data_size 0x1(0xc2f4 bytes) 2024-09-21T15:49:20.160 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:49:20.174 INFO:teuthology.orchestra.run.smithi089.stdout:851 read 0x748755a thru 0x7490433 (0x8eda bytes) 2024-09-21T15:49:20.174 INFO:teuthology.orchestra.run.smithi089.stdout:852 writesame 0xa1c3ea4 thru 0xa1c54c4 data_size 0xb(0x1621 bytes) 2024-09-21T15:49:20.179 INFO:teuthology.orchestra.run.smithi089.stdout:853 writesame 0xbe1bc2c thru 0xbe27e0e data_size 0xf(0xc1e3 bytes) 2024-09-21T15:49:20.196 INFO:teuthology.orchestra.run.smithi089.stdout:855 write 0x1c00f8e thru 0x1c102f3 (0xf366 bytes) 2024-09-21T15:49:20.213 INFO:teuthology.orchestra.run.smithi089.stdout:856 flatten 2024-09-21T15:49:20.246 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:20.245+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:20.328 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:20.328 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:20.342 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:49:20.386 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:20.385+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:20.448 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:20.447+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:20.519 INFO:teuthology.orchestra.run.smithi089.stdout:859 read 0x731840d thru 0x731c7fd (0x43f1 bytes) 2024-09-21T15:49:20.519 INFO:teuthology.orchestra.run.smithi089.stdout:860 read 0x64f7e59 thru 0x6500b49 (0x8cf1 bytes) 2024-09-21T15:49:20.520 INFO:teuthology.orchestra.run.smithi089.stdout:861 read 0x45b2db1 thru 0x45b5737 (0x2987 bytes) 2024-09-21T15:49:20.520 INFO:teuthology.orchestra.run.smithi089.stdout:862 read 0xcbe9f03 thru 0xcbf09fc (0x6afa bytes) 2024-09-21T15:49:20.520 INFO:teuthology.orchestra.run.smithi089.stdout:863 write 0x295efd6 thru 0x296d83e (0xe869 bytes) 2024-09-21T15:49:20.533 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:20.533 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:20.535 INFO:teuthology.orchestra.run.smithi089.stdout:865 write 0x4b1ebec thru 0x4b24cac (0x60c1 bytes) 2024-09-21T15:49:20.542 INFO:teuthology.orchestra.run.smithi089.stdout:867 write 0xb896798 thru 0xb898a2d (0x2296 bytes) 2024-09-21T15:49:20.543 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:49:20.547 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:20.547 INFO:teuthology.orchestra.run.smithi089.stdout:872 write 0x5f1af56 thru 0x5f2732c (0xc3d7 bytes) 2024-09-21T15:49:20.548 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:20.547+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:20.562 INFO:teuthology.orchestra.run.smithi089.stdout:873 punch from 0xbfe075c to 0xbfeb352, (0xabf6 bytes) 2024-09-21T15:49:20.562 INFO:teuthology.orchestra.run.smithi089.stdout:875 read 0xc523eba thru 0xc52d72e (0x9875 bytes) 2024-09-21T15:49:20.563 INFO:teuthology.orchestra.run.smithi089.stdout:877 read 0x6b21d73 thru 0x6b29af0 (0x7d7e bytes) 2024-09-21T15:49:20.563 INFO:teuthology.orchestra.run.smithi089.stdout:878 read 0xd2d0f0 thru 0xd3a9f6 (0xd907 bytes) 2024-09-21T15:49:20.563 INFO:teuthology.orchestra.run.smithi089.stdout:881 writesame 0x5546440 thru 0x554d5e1 data_size 0x5(0x71a2 bytes) 2024-09-21T15:49:20.584 INFO:teuthology.orchestra.run.smithi089.stdout:882 write 0xe0af013 thru 0xe0b5959 (0x6947 bytes) 2024-09-21T15:49:20.595 INFO:teuthology.orchestra.run.smithi089.stdout:883 read 0xd44144c thru 0xd44ec28 (0xd7dd bytes) 2024-09-21T15:49:20.595 INFO:teuthology.orchestra.run.smithi089.stdout:885 write 0x383d584 thru 0x384d067 (0xfae4 bytes) 2024-09-21T15:49:20.603 INFO:teuthology.orchestra.run.smithi089.stdout:886 write 0x7a9b760 thru 0x7aa8c9a (0xd53b bytes) 2024-09-21T15:49:20.619 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:20.619 INFO:teuthology.orchestra.run.smithi089.stdout:888 write 0x2a36826 thru 0x2a467c5 (0xffa0 bytes) 2024-09-21T15:49:20.628 INFO:teuthology.orchestra.run.smithi089.stdout:889 writesame 0xdd47b48 thru 0xdd49ce5 data_size 0xd(0x219e bytes) 2024-09-21T15:49:20.632 INFO:teuthology.orchestra.run.smithi089.stdout:890 read 0x1203d86 thru 0x120f875 (0xbaf0 bytes) 2024-09-21T15:49:20.632 INFO:teuthology.orchestra.run.smithi089.stdout:892 write 0x43fdc7a thru 0x440b482 (0xd809 bytes) 2024-09-21T15:49:20.647 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:20.647 INFO:teuthology.orchestra.run.smithi089.stdout:894 punch from 0xa2df752 to 0xa2e22a5, (0x2b53 bytes) 2024-09-21T15:49:20.648 INFO:teuthology.orchestra.run.smithi089.stdout:895 writesame 0x6a51a9 thru 0x6a654c data_size 0x6(0x13a4 bytes) 2024-09-21T15:49:20.650 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:20.649+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:20.652 INFO:teuthology.orchestra.run.smithi089.stdout:896 write 0x8f24220 thru 0x8f2fbe9 (0xb9ca bytes) 2024-09-21T15:49:20.668 INFO:teuthology.orchestra.run.smithi089.stdout:899 read 0x79ad722 thru 0x79b7e3d (0xa71c bytes) 2024-09-21T15:49:20.669 INFO:teuthology.orchestra.run.smithi089.stdout:902 write 0x7de79af thru 0x7dee6a9 (0x6cfb bytes) 2024-09-21T15:49:20.673 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:20.673 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:20.673 INFO:teuthology.orchestra.run.smithi089.stdout:903 write 0xc0ac4b thru 0xc112a0 (0x6656 bytes) 2024-09-21T15:49:20.682 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T15:49:20.685 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:49:20.685 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:20.685 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:20.685 INFO:teuthology.orchestra.run.smithi089.stdout:906 write 0x215cbcc thru 0x2160ef8 (0x432d bytes) 2024-09-21T15:49:20.694 INFO:teuthology.orchestra.run.smithi089.stdout:908 write 0xc8c5d73 thru 0xc8c6648 (0x8d6 bytes) 2024-09-21T15:49:20.701 INFO:teuthology.orchestra.run.smithi089.stdout:910 read 0x28f2260 thru 0x28f5869 (0x360a bytes) 2024-09-21T15:49:20.702 INFO:teuthology.orchestra.run.smithi089.stdout:911 punch from 0x223e2d3 to 0x2244810, (0x653d bytes) 2024-09-21T15:49:20.702 INFO:teuthology.orchestra.run.smithi089.stdout:912 writesame 0x51a4d9e thru 0x51add74 data_size 0x1(0x8fd7 bytes) 2024-09-21T15:49:20.712 INFO:teuthology.orchestra.run.smithi089.stdout:914 punch from 0xa02dead to 0xa03b199, (0xd2ec bytes) 2024-09-21T15:49:20.712 INFO:teuthology.orchestra.run.smithi089.stdout:915 writesame 0x70cd4a9 thru 0x70d0027 data_size 0x11(0x2b7f bytes) 2024-09-21T15:49:20.723 INFO:teuthology.orchestra.run.smithi089.stdout:917 write 0x3ea2cc9 thru 0x3eaa3f3 (0x772b bytes) 2024-09-21T15:49:20.728 INFO:teuthology.orchestra.run.smithi089.stdout:918 write 0x3dd12a thru 0x3e3863 (0x673a bytes) 2024-09-21T15:49:20.731 INFO:teuthology.orchestra.run.smithi089.stdout:921 write 0xc9122d9 thru 0xc919c6b (0x7993 bytes) 2024-09-21T15:49:20.740 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:20.740 INFO:teuthology.orchestra.run.smithi089.stdout:924 write 0xb726ccd thru 0xb734c50 (0xdf84 bytes) 2024-09-21T15:49:20.750 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:20.749+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:20.754 INFO:teuthology.orchestra.run.smithi089.stdout:926 writesame 0x79fe97 thru 0x7a2d87 data_size 0x1(0x2ef1 bytes) 2024-09-21T15:49:20.760 INFO:teuthology.orchestra.run.smithi089.stdout:927 write 0x980d898 thru 0x9814368 (0x6ad1 bytes) 2024-09-21T15:49:20.772 INFO:teuthology.orchestra.run.smithi089.stdout:931 writesame 0x362a587 thru 0x362cbe6 data_size 0x10(0x2660 bytes) 2024-09-21T15:49:20.774 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:20.775 INFO:teuthology.orchestra.run.smithi089.stdout:933 punch from 0x91a6dc2 to 0x91b3aca, (0xcd08 bytes) 2024-09-21T15:49:20.775 INFO:teuthology.orchestra.run.smithi089.stdout:935 write 0xd971b15 thru 0xd980a27 (0xef13 bytes) 2024-09-21T15:49:20.788 INFO:teuthology.orchestra.run.smithi089.stdout:937 writesame 0x82f20b2 thru 0x82faef2 data_size 0x3(0x8e41 bytes) 2024-09-21T15:49:20.815 INFO:teuthology.orchestra.run.smithi089.stdout:938 trunc from 0xe0b595a to 0x4c06b66 2024-09-21T15:49:20.830 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:20.830 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:20.844 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:49:20.851 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:20.850+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:20.912 INFO:teuthology.orchestra.run.smithi089.stdout:939 write 0x5aa1d91 thru 0x5aa66dc (0x494c bytes) 2024-09-21T15:49:20.924 INFO:teuthology.orchestra.run.smithi089.stdout:941 punch from 0x127cd9e to 0x127d85c, (0xabe bytes) 2024-09-21T15:49:20.937 INFO:teuthology.orchestra.run.smithi089.stdout:942 writesame 0xb76a0af thru 0xb77165f data_size 0xb(0x75b1 bytes) 2024-09-21T15:49:20.937 INFO:teuthology.orchestra.run.smithi089.stdout:943 write 0xa3a5e3d thru 0xa3a6676 (0x83a bytes) 2024-09-21T15:49:20.944 INFO:teuthology.orchestra.run.smithi089.stdout:945 read 0x7cc105 thru 0x7d60f9 (0x9ff5 bytes) 2024-09-21T15:49:20.945 INFO:teuthology.orchestra.run.smithi089.stdout:947 punch from 0x45498b3 to 0x455225c, (0x89a9 bytes) 2024-09-21T15:49:20.945 INFO:teuthology.orchestra.run.smithi089.stdout:949 punch from 0x35033f6 to 0x35044cd, (0x10d7 bytes) 2024-09-21T15:49:20.946 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:20.946 INFO:teuthology.orchestra.run.smithi089.stdout:953 read 0xa5450c8 thru 0xa54ba40 (0x6979 bytes) 2024-09-21T15:49:20.946 INFO:teuthology.orchestra.run.smithi089.stdout:954 read 0xa83d050 thru 0xa845eef (0x8ea0 bytes) 2024-09-21T15:49:20.946 INFO:teuthology.orchestra.run.smithi089.stdout:955 write 0x53f0a98 thru 0x53f1b99 (0x1102 bytes) 2024-09-21T15:49:20.952 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:20.951+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:20.960 INFO:teuthology.orchestra.run.smithi089.stdout:956 trunc from 0xb771660 to 0x2cb9504 2024-09-21T15:49:21.002 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:21.003 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:21.005 INFO:teuthology.orchestra.run.smithi089.stdout:958 write 0x256c88 thru 0x261baf (0xaf28 bytes) 2024-09-21T15:49:21.013 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:49:21.025 INFO:teuthology.orchestra.run.smithi089.stdout:959 write 0x9679fef thru 0x968550b (0xb51d bytes) 2024-09-21T15:49:21.039 INFO:teuthology.orchestra.run.smithi089.stdout:960 read 0x942c942 thru 0x942cf83 (0x642 bytes) 2024-09-21T15:49:21.039 INFO:teuthology.orchestra.run.smithi089.stdout:963 trunc from 0x968550c to 0x926881d 2024-09-21T15:49:21.051 INFO:teuthology.orchestra.run.smithi089.stdout:964 punch from 0x1a05560 to 0x1a0c75f, (0x71ff bytes) 2024-09-21T15:49:21.051 INFO:teuthology.orchestra.run.smithi089.stdout:965 read 0x45d9c7b thru 0x45ddc8c (0x4012 bytes) 2024-09-21T15:49:21.051 INFO:teuthology.orchestra.run.smithi089.stdout:967 write 0x1a15d0b thru 0x1a1c875 (0x6b6b bytes) 2024-09-21T15:49:21.052 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:21.051+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:21.058 INFO:teuthology.orchestra.run.smithi089.stdout:970 write 0x37c7022 thru 0x37d60bd (0xf09c bytes) 2024-09-21T15:49:21.075 INFO:teuthology.orchestra.run.smithi089.stdout:971 trunc from 0x926881d to 0xd6226a9 2024-09-21T15:49:21.080 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:21.083 INFO:teuthology.orchestra.run.smithi089.stdout:975 trunc from 0xd6226a9 to 0xaf78ada 2024-09-21T15:49:21.090 INFO:teuthology.orchestra.run.smithi089.stdout:976 read 0x1ad707f thru 0x1adcbf5 (0x5b77 bytes) 2024-09-21T15:49:21.090 INFO:teuthology.orchestra.run.smithi089.stdout:978 read 0x8e520ff thru 0x8e5c54f (0xa451 bytes) 2024-09-21T15:49:21.090 INFO:teuthology.orchestra.run.smithi089.stdout:979 write 0x7eb0d5b thru 0x7eb7366 (0x660c bytes) 2024-09-21T15:49:21.105 INFO:teuthology.orchestra.run.smithi089.stdout:980 punch from 0x3ae201f to 0x3af1168, (0xf149 bytes) 2024-09-21T15:49:21.105 INFO:teuthology.orchestra.run.smithi089.stdout:982 writesame 0x5276ea thru 0x52ab6b data_size 0x2f(0x3482 bytes) 2024-09-21T15:49:21.111 INFO:teuthology.orchestra.run.smithi089.stdout:983 read 0xa2c3474 thru 0xa2c5890 (0x241d bytes) 2024-09-21T15:49:21.111 INFO:teuthology.orchestra.run.smithi089.stdout:984 write 0x905e13a thru 0x9066f99 (0x8e60 bytes) 2024-09-21T15:49:21.126 INFO:teuthology.orchestra.run.smithi089.stdout:985 write 0xa97ef07 thru 0xa983a36 (0x4b30 bytes) 2024-09-21T15:49:21.137 INFO:teuthology.orchestra.run.smithi089.stdout:986 write 0xda84f2 thru 0xdaa01b (0x1b2a bytes) 2024-09-21T15:49:21.140 INFO:teuthology.orchestra.run.smithi089.stdout:989 punch from 0x314d04f to 0x3157cb9, (0xac6a bytes) 2024-09-21T15:49:21.140 INFO:teuthology.orchestra.run.smithi089.stdout:991 read 0x286ac5b thru 0x2879ef3 (0xf299 bytes) 2024-09-21T15:49:21.140 INFO:teuthology.orchestra.run.smithi089.stdout:994 read 0x2c5a97d thru 0x2c5f221 (0x48a5 bytes) 2024-09-21T15:49:21.141 INFO:teuthology.orchestra.run.smithi089.stdout:995 write 0x48e54f4 thru 0x48f0a00 (0xb50d bytes) 2024-09-21T15:49:21.150 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:21.150 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:21.153 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:21.152+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:21.159 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:49:21.161 INFO:teuthology.orchestra.run.smithi089.stdout:999 read 0x77e1733 thru 0x77e9085 (0x7953 bytes) 2024-09-21T15:49:21.161 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:21.161 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:21.161 INFO:teuthology.orchestra.run.smithi089.stdout:1003 write 0x2559c45 thru 0x2561f31 (0x82ed bytes) 2024-09-21T15:49:21.177 INFO:teuthology.orchestra.run.smithi089.stdout:1004 writesame 0x2644b28 thru 0x2645b94 data_size 0x5(0x106d bytes) 2024-09-21T15:49:21.189 INFO:teuthology.orchestra.run.smithi089.stdout:1005 trunc from 0xaf78ada to 0x4dc630d 2024-09-21T15:49:21.226 INFO:teuthology.orchestra.run.smithi089.stdout:1006 read 0x3604254 thru 0x3612ccf (0xea7c bytes) 2024-09-21T15:49:21.236 INFO:teuthology.orchestra.run.smithi089.stdout:1007 write 0x5f06187 thru 0x5f0706f (0xee9 bytes) 2024-09-21T15:49:21.241 INFO:teuthology.orchestra.run.smithi089.stdout:1008 punch from 0x53b3c42 to 0x53bb9af, (0x7d6d bytes) 2024-09-21T15:49:21.241 INFO:teuthology.orchestra.run.smithi089.stdout:1009 read 0x5cd5b69 thru 0x5cda509 (0x49a1 bytes) 2024-09-21T15:49:21.241 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:21.241 INFO:teuthology.orchestra.run.smithi089.stdout:1011 punch from 0x5b2dbdc to 0x5b3b732, (0xdb56 bytes) 2024-09-21T15:49:21.252 INFO:teuthology.orchestra.run.smithi089.stdout:1012 write 0xa748a70 thru 0xa74bba6 (0x3137 bytes) 2024-09-21T15:49:21.254 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:21.253+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:21.260 INFO:teuthology.orchestra.run.smithi089.stdout:1013 read 0x871b5ca thru 0x871b752 (0x189 bytes) 2024-09-21T15:49:21.260 INFO:teuthology.orchestra.run.smithi089.stdout:1014 write 0x3707bdc thru 0x370eaae (0x6ed3 bytes) 2024-09-21T15:49:21.271 INFO:teuthology.orchestra.run.smithi089.stdout:1016 writesame 0xabe2d08 thru 0xabee218 data_size 0x1(0xb511 bytes) 2024-09-21T15:49:21.271 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:21.271 INFO:teuthology.orchestra.run.smithi089.stdout:1020 write 0x4c748f2 thru 0x4c79e15 (0x5524 bytes) 2024-09-21T15:49:21.274 INFO:teuthology.orchestra.run.smithi089.stdout:1022 writesame 0x6693418 thru 0x66a03a4 data_size 0x3(0xcf8d bytes) 2024-09-21T15:49:21.277 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:21.277 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:21.289 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:49:21.290 INFO:teuthology.orchestra.run.smithi089.stdout:1023 punch from 0x8f66807 to 0x8f6fe63, (0x965c bytes) 2024-09-21T15:49:21.290 INFO:teuthology.orchestra.run.smithi089.stdout:1024 writesame 0x4167c96 thru 0x4170f96 data_size 0x1(0x9301 bytes) 2024-09-21T15:49:21.290 INFO:teuthology.orchestra.run.smithi089.stdout:1025 write 0x699840c thru 0x699a471 (0x2066 bytes) 2024-09-21T15:49:21.294 INFO:teuthology.orchestra.run.smithi089.stdout:1026 trunc from 0xabee219 to 0x929c687 2024-09-21T15:49:21.305 INFO:teuthology.orchestra.run.smithi089.stdout:1028 punch from 0xacb739 to 0xad23f7, (0x6cbe bytes) 2024-09-21T15:49:21.305 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:21.305 INFO:teuthology.orchestra.run.smithi089.stdout:1030 write 0x84c9cf5 thru 0x84d9b51 (0xfe5d bytes) 2024-09-21T15:49:21.322 INFO:teuthology.orchestra.run.smithi089.stdout:1032 write 0x147acd7 thru 0x1488f36 (0xe260 bytes) 2024-09-21T15:49:21.327 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:21.327 INFO:teuthology.orchestra.run.smithi089.stdout:1034 read 0x828552e thru 0x828def5 (0x89c8 bytes) 2024-09-21T15:49:21.334 INFO:teuthology.orchestra.run.smithi089.stdout:1035 write 0x9fbec0d thru 0x9fca9e6 (0xbdda bytes) 2024-09-21T15:49:21.347 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:49:21.349 INFO:teuthology.orchestra.run.smithi089.stdout:1037 read 0x40addf7 thru 0x40b2870 (0x4a7a bytes) 2024-09-21T15:49:21.356 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:21.355+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:21.360 INFO:teuthology.orchestra.run.smithi089.stdout:1038 trunc from 0x9fca9e7 to 0x173ab32 2024-09-21T15:49:21.414 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:21.414 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:21.425 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:49:21.434 INFO:teuthology.orchestra.run.smithi089.stdout:1040 write 0x93d15a7 thru 0x93d4fa3 (0x39fd bytes) 2024-09-21T15:49:21.457 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:21.456+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:21.458 INFO:teuthology.orchestra.run.smithi089.stdout:1041 write 0xb825d76 thru 0xb8301cc (0xa457 bytes) 2024-09-21T15:49:21.477 INFO:teuthology.orchestra.run.smithi089.stdout:1043 read 0x144cb41 thru 0x144ed7d (0x223d bytes) 2024-09-21T15:49:21.482 INFO:teuthology.orchestra.run.smithi089.stdout:1044 trunc from 0xb8301cd to 0x785ce07 2024-09-21T15:49:21.486 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-09-21T15:49:21.486 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:49:21.501 INFO:teuthology.orchestra.run.smithi089.stdout:1045 read 0x708b206 thru 0x708b496 (0x291 bytes) 2024-09-21T15:49:21.504 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:49:21.504 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:49:21.505 INFO:teuthology.orchestra.run.smithi089.stdout:1046 trunc from 0x785ce07 to 0xbd8ed55 2024-09-21T15:49:21.511 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:21.511 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:21.511 INFO:teuthology.orchestra.run.smithi089.stdout:1049 punch from 0x66fd720 to 0x670934a, (0xbc2a bytes) 2024-09-21T15:49:21.511 INFO:teuthology.orchestra.run.smithi089.stdout:1050 read 0x9c5b11d thru 0x9c6531b (0xa1ff bytes) 2024-09-21T15:49:21.520 INFO:teuthology.orchestra.run.smithi089.stdout:1051 trunc from 0xbd8ed55 to 0x42c6eb6 2024-09-21T15:49:21.521 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:49:21.521 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:49:21.532 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:21.533 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:21.536 INFO:teuthology.orchestra.run.smithi089.stdout:1053 write 0x7d36cae thru 0x7d444f8 (0xd84b bytes) 2024-09-21T15:49:21.539 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:49:21.539 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:49:21.542 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:49:21.558 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:21.557+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:21.559 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:49:21.559 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:49:21.560 INFO:teuthology.orchestra.run.smithi089.stdout:1056 writesame 0xca0433b thru 0xca061da data_size 0x10(0x1ea0 bytes) 2024-09-21T15:49:21.560 INFO:teuthology.orchestra.run.smithi089.stdout:1057 read 0xfcd330 thru 0xfce57a (0x124b bytes) 2024-09-21T15:49:21.572 INFO:teuthology.orchestra.run.smithi089.stdout:1058 trunc from 0xca061db to 0x32d90a2 2024-09-21T15:49:21.577 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:49:21.577 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:49:21.586 INFO:teuthology.orchestra.run.smithi089.stdout:1060 punch from 0x175eabf to 0x176aeb6, (0xc3f7 bytes) 2024-09-21T15:49:21.594 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:49:21.595 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:49:21.606 INFO:teuthology.orchestra.run.smithi089.stdout:1063 write 0xd5bd542 thru 0xd5bfcf8 (0x27b7 bytes) 2024-09-21T15:49:21.612 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:49:21.612 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:49:21.617 INFO:teuthology.orchestra.run.smithi089.stdout:1064 read 0x99b07c4 thru 0x99b6e08 (0x6645 bytes) 2024-09-21T15:49:21.617 INFO:teuthology.orchestra.run.smithi089.stdout:1065 write 0x1ed8218 thru 0x1ee6ea1 (0xec8a bytes) 2024-09-21T15:49:21.636 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:21.636 INFO:teuthology.orchestra.run.smithi089.stdout:1067 writesame 0x80ede97 thru 0x80f64d2 data_size 0x47(0x863c bytes) 2024-09-21T15:49:21.650 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:21.650 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:21.650 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:49:21.651 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:49:21.651 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:21.654 INFO:teuthology.orchestra.run.smithi089.stdout:1068 punch from 0xc6cb8c2 to 0xc6cc218, (0x956 bytes) 2024-09-21T15:49:21.659 INFO:teuthology.orchestra.run.smithi089.stdout:1070 trunc from 0xd5bfcf9 to 0x96da913 2024-09-21T15:49:21.659 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:49:21.660 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:21.658+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:21.679 INFO:teuthology.orchestra.run.smithi089.stdout:1071 trunc from 0x96da913 to 0x74c9eb9 2024-09-21T15:49:21.696 INFO:teuthology.orchestra.run.smithi089.stdout:1072 write 0x7365081 thru 0x7373133 (0xe0b3 bytes) 2024-09-21T15:49:21.708 INFO:teuthology.orchestra.run.smithi089.stdout:1073 read 0x677704d thru 0x677f65b (0x860f bytes) 2024-09-21T15:49:21.719 INFO:teuthology.orchestra.run.smithi089.stdout:1075 write 0xb92cd6d thru 0xb934aa5 (0x7d39 bytes) 2024-09-21T15:49:21.736 INFO:teuthology.orchestra.run.smithi089.stdout:1076 read 0x5529a51 thru 0x552a9f6 (0xfa6 bytes) 2024-09-21T15:49:21.747 INFO:teuthology.orchestra.run.smithi089.stdout:1077 writesame 0xd92895d thru 0xd92a318 data_size 0x36(0x19bc bytes) 2024-09-21T15:49:21.747 INFO:teuthology.orchestra.run.smithi089.stdout:1078 clone 5 order 24 su 8388608 sc 14 2024-09-21T15:49:21.751 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is set 2024-09-21T15:49:21.760 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:21.759+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:21.808 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:21.808 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:49:21.808 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:49:21.808 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:49:21.808 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.808 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:13505 3.14 3:2ac81419:::rbd_object_map.1133dc5adfdf:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:21.808 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.280583+0000", 2024-09-21T15:49:21.808 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.5248968449999998, 2024-09-21T15:49:21.808 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.012765583, 2024-09-21T15:49:21.808 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:21.808 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:21.808 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:21.808 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:21.808 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:21.809 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 13505 2024-09-21T15:49:21.809 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.809 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:21.809 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.809 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:21.809 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.280583+0000", 2024-09-21T15:49:21.809 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:21.809 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.809 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.809 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:21.809 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.280583+0000", 2024-09-21T15:49:21.809 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:21.809 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.809 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.809 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:21.809 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.280583+0000", 2024-09-21T15:49:21.810 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:21.810 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.810 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.810 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:21.810 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.280585+0000", 2024-09-21T15:49:21.810 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.615e-06 2024-09-21T15:49:21.810 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.810 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.810 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:21.810 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.280588+0000", 2024-09-21T15:49:21.810 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1939999999999998e-06 2024-09-21T15:49:21.810 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.810 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.810 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:21.810 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.280590+0000", 2024-09-21T15:49:21.811 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3949999999999999e-06 2024-09-21T15:49:21.811 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.811 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.811 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:21.811 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.287618+0000", 2024-09-21T15:49:21.811 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0070271939999999996 2024-09-21T15:49:21.811 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.811 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.811 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:21.811 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.287704+0000", 2024-09-21T15:49:21.811 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6137999999999994e-05 2024-09-21T15:49:21.811 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.811 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.812 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 7", 2024-09-21T15:49:21.812 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.287943+0000", 2024-09-21T15:49:21.812 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00023945300000000001 2024-09-21T15:49:21.812 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.812 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.812 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:21.812 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.292993+0000", 2024-09-21T15:49:21.812 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0050501570000000004 2024-09-21T15:49:21.812 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.812 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.812 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:21.812 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.293296+0000", 2024-09-21T15:49:21.812 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00030243400000000003 2024-09-21T15:49:21.812 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.812 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.813 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:21.813 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.293319+0000", 2024-09-21T15:49:21.813 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2770000000000001e-05 2024-09-21T15:49:21.813 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.813 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.813 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:21.813 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.293349+0000", 2024-09-21T15:49:21.813 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0477e-05 2024-09-21T15:49:21.813 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:21.813 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:21.813 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:21.813 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.813 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.813 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:13857 3.14 3:2ac81419:::rbd_object_map.1133dc5adfdf:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:21.813 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.543383+0000", 2024-09-21T15:49:21.814 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.2620971159999996, 2024-09-21T15:49:21.814 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.012793752, 2024-09-21T15:49:21.814 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:21.814 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:21.814 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:21.814 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:21.814 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:21.814 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 13857 2024-09-21T15:49:21.814 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.814 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:21.814 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.814 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:21.814 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543383+0000", 2024-09-21T15:49:21.814 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:21.814 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.814 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.815 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:21.815 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543383+0000", 2024-09-21T15:49:21.815 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:21.815 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.815 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.815 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:21.815 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543383+0000", 2024-09-21T15:49:21.815 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:21.815 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.815 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.815 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:21.815 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543385+0000", 2024-09-21T15:49:21.815 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7e-06 2024-09-21T15:49:21.816 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.816 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.816 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:21.816 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543388+0000", 2024-09-21T15:49:21.816 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3340000000000002e-06 2024-09-21T15:49:21.816 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.816 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.816 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:21.816 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543391+0000", 2024-09-21T15:49:21.816 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9110000000000002e-06 2024-09-21T15:49:21.816 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.816 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.816 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:21.816 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.548745+0000", 2024-09-21T15:49:21.816 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0053544200000000004 2024-09-21T15:49:21.817 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.817 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.817 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:21.817 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.548797+0000", 2024-09-21T15:49:21.817 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1875000000000001e-05 2024-09-21T15:49:21.817 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.817 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.817 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 7", 2024-09-21T15:49:21.817 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.549008+0000", 2024-09-21T15:49:21.817 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00021113999999999999 2024-09-21T15:49:21.817 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.817 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.817 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:21.817 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.555860+0000", 2024-09-21T15:49:21.817 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0068516269999999999 2024-09-21T15:49:21.818 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.818 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.818 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:21.818 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.556133+0000", 2024-09-21T15:49:21.818 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00027346800000000002 2024-09-21T15:49:21.818 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.818 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.818 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:21.818 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.556158+0000", 2024-09-21T15:49:21.818 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4901000000000001e-05 2024-09-21T15:49:21.818 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.818 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.818 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:21.818 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.556177+0000", 2024-09-21T15:49:21.819 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8576e-05 2024-09-21T15:49:21.819 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:21.819 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:21.819 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:21.819 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.819 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.819 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:13858 3.14 3:2ac81419:::rbd_object_map.1133dc5adfdf:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:21.819 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.543396+0000", 2024-09-21T15:49:21.819 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.2620840839999996, 2024-09-21T15:49:21.819 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.012881375, 2024-09-21T15:49:21.819 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:21.819 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:21.819 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:21.819 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:21.819 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:21.820 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 13858 2024-09-21T15:49:21.820 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.820 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:21.820 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.820 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:21.820 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543396+0000", 2024-09-21T15:49:21.820 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:21.820 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.820 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.820 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:21.820 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543396+0000", 2024-09-21T15:49:21.820 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:21.820 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.820 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.821 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:21.821 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543396+0000", 2024-09-21T15:49:21.821 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:21.821 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.821 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.821 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:21.821 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543398+0000", 2024-09-21T15:49:21.821 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9190000000000002e-06 2024-09-21T15:49:21.821 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.821 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.821 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:21.821 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543402+0000", 2024-09-21T15:49:21.821 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6729999999999998e-06 2024-09-21T15:49:21.821 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.821 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.822 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:21.822 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543405+0000", 2024-09-21T15:49:21.822 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2030000000000001e-06 2024-09-21T15:49:21.822 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.822 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.822 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:21.822 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.549528+0000", 2024-09-21T15:49:21.822 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0061230379999999999 2024-09-21T15:49:21.822 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.822 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.822 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:21.822 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.549610+0000", 2024-09-21T15:49:21.822 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2602999999999995e-05 2024-09-21T15:49:21.822 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.823 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.823 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 7", 2024-09-21T15:49:21.823 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.549966+0000", 2024-09-21T15:49:21.823 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00035542700000000001 2024-09-21T15:49:21.823 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.823 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.823 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:21.823 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.555864+0000", 2024-09-21T15:49:21.823 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0058985460000000002 2024-09-21T15:49:21.823 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.823 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.823 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:21.823 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.556218+0000", 2024-09-21T15:49:21.823 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00035375899999999999 2024-09-21T15:49:21.824 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.824 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.824 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:21.824 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.556246+0000", 2024-09-21T15:49:21.824 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8096e-05 2024-09-21T15:49:21.824 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.824 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.824 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:21.824 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.556278+0000", 2024-09-21T15:49:21.824 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1359999999999998e-05 2024-09-21T15:49:21.824 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:21.824 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:21.824 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:21.824 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.824 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.825 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:13859 3.14 3:2ac81419:::rbd_object_map.1133dc5adfdf:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:21.825 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.543407+0000", 2024-09-21T15:49:21.825 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.2620733030000002, 2024-09-21T15:49:21.825 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.012991434, 2024-09-21T15:49:21.825 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:21.825 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:21.825 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:21.825 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:21.825 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:21.825 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 13859 2024-09-21T15:49:21.825 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.825 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:21.825 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.825 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:21.825 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543407+0000", 2024-09-21T15:49:21.826 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:21.826 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.826 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.826 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:21.826 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543407+0000", 2024-09-21T15:49:21.826 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:21.826 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.826 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.826 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:21.826 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543407+0000", 2024-09-21T15:49:21.826 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:21.826 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.826 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.826 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:21.827 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543408+0000", 2024-09-21T15:49:21.827 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.533e-06 2024-09-21T15:49:21.827 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.827 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.827 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:21.827 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543411+0000", 2024-09-21T15:49:21.827 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9239999999999999e-06 2024-09-21T15:49:21.827 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.827 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.827 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:21.827 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543414+0000", 2024-09-21T15:49:21.827 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0680000000000001e-06 2024-09-21T15:49:21.827 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.827 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.828 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:21.828 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.550404+0000", 2024-09-21T15:49:21.828 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0069900329999999997 2024-09-21T15:49:21.828 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.828 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.828 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:21.828 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.550466+0000", 2024-09-21T15:49:21.828 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1265000000000004e-05 2024-09-21T15:49:21.828 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.828 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.828 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 7", 2024-09-21T15:49:21.828 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.550752+0000", 2024-09-21T15:49:21.828 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00028663000000000001 2024-09-21T15:49:21.828 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.828 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.829 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:21.829 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.555868+0000", 2024-09-21T15:49:21.829 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.005116191 2024-09-21T15:49:21.829 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.829 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.829 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:21.829 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.556330+0000", 2024-09-21T15:49:21.829 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00046124799999999999 2024-09-21T15:49:21.829 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.829 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.829 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:21.829 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.556363+0000", 2024-09-21T15:49:21.829 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3592e-05 2024-09-21T15:49:21.829 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.829 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.830 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:21.830 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.556398+0000", 2024-09-21T15:49:21.830 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5148000000000003e-05 2024-09-21T15:49:21.830 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:21.830 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:21.830 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:21.830 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.830 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.830 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:13860 3.14 3:2ac81419:::rbd_object_map.1133dc5adfdf:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:21.830 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.543416+0000", 2024-09-21T15:49:21.830 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.2620639059999998, 2024-09-21T15:49:21.830 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.013060522999999999, 2024-09-21T15:49:21.830 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:21.830 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:21.831 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:21.831 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:21.831 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:21.831 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 13860 2024-09-21T15:49:21.831 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.831 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:21.831 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.831 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:21.831 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543416+0000", 2024-09-21T15:49:21.831 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:21.831 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.831 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.831 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:21.831 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543416+0000", 2024-09-21T15:49:21.832 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:21.832 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.832 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.832 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:21.832 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543416+0000", 2024-09-21T15:49:21.832 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:21.832 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.832 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.832 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:21.832 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543420+0000", 2024-09-21T15:49:21.832 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.63e-06 2024-09-21T15:49:21.832 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.832 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.832 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:21.832 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543423+0000", 2024-09-21T15:49:21.832 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9809999999999999e-06 2024-09-21T15:49:21.833 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.833 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.833 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:21.833 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.543425+0000", 2024-09-21T15:49:21.833 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5390000000000002e-06 2024-09-21T15:49:21.833 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.833 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.833 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:21.833 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.551032+0000", 2024-09-21T15:49:21.833 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0076064879999999998 2024-09-21T15:49:21.833 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.833 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.833 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:21.833 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.551114+0000", 2024-09-21T15:49:21.834 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2639000000000006e-05 2024-09-21T15:49:21.834 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.834 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.834 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 7", 2024-09-21T15:49:21.834 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.551359+0000", 2024-09-21T15:49:21.834 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00024488199999999998 2024-09-21T15:49:21.834 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.834 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.834 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:21.834 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.555873+0000", 2024-09-21T15:49:21.834 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0045135669999999996 2024-09-21T15:49:21.834 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.834 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.834 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:21.834 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.556429+0000", 2024-09-21T15:49:21.835 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00055622799999999997 2024-09-21T15:49:21.835 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.835 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.835 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:21.835 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.556457+0000", 2024-09-21T15:49:21.835 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7931000000000001e-05 2024-09-21T15:49:21.835 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.835 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.835 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:21.835 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.556477+0000", 2024-09-21T15:49:21.835 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9845999999999999e-05 2024-09-21T15:49:21.835 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:21.835 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:21.835 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:21.835 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.836 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.836 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:13921 3.14 3:2ac81419:::rbd_object_map.1133dc5adfdf:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:21.836 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.586097+0000", 2024-09-21T15:49:21.836 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.21938279, 2024-09-21T15:49:21.836 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.016128601999999999, 2024-09-21T15:49:21.836 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:21.836 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:21.836 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:21.836 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:21.836 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:21.836 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 13921 2024-09-21T15:49:21.836 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.836 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:21.836 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.836 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:21.837 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.586097+0000", 2024-09-21T15:49:21.837 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:21.837 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.837 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.837 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:21.837 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.586097+0000", 2024-09-21T15:49:21.837 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:21.837 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.837 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.837 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:21.837 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.586097+0000", 2024-09-21T15:49:21.837 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:21.837 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.837 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.838 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:21.838 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.586099+0000", 2024-09-21T15:49:21.838 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.514e-06 2024-09-21T15:49:21.838 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.838 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.838 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:21.838 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.586102+0000", 2024-09-21T15:49:21.838 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.92e-06 2024-09-21T15:49:21.838 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.838 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.838 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:21.838 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.586104+0000", 2024-09-21T15:49:21.838 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5790000000000002e-06 2024-09-21T15:49:21.838 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.838 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.838 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:21.839 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.594361+0000", 2024-09-21T15:49:21.839 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0082570319999999992 2024-09-21T15:49:21.839 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.839 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.839 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:21.839 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.594448+0000", 2024-09-21T15:49:21.839 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6254000000000005e-05 2024-09-21T15:49:21.839 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.839 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.839 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 7", 2024-09-21T15:49:21.839 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.594765+0000", 2024-09-21T15:49:21.839 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00031769799999999999 2024-09-21T15:49:21.839 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.839 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.840 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:21.840 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.599427+0000", 2024-09-21T15:49:21.840 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0046615989999999998 2024-09-21T15:49:21.840 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.840 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.840 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:21.840 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.602156+0000", 2024-09-21T15:49:21.840 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0027293109999999999 2024-09-21T15:49:21.840 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.840 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.840 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:21.840 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.602197+0000", 2024-09-21T15:49:21.840 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0389000000000002e-05 2024-09-21T15:49:21.840 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.840 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.840 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:21.841 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.602226+0000", 2024-09-21T15:49:21.841 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9530999999999999e-05 2024-09-21T15:49:21.841 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:21.841 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:21.841 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:21.841 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.841 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.841 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:14123 3.14 3:2ac81419:::rbd_object_map.1133dc5adfdf:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-21T15:49:21.841 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:18.739345+0000", 2024-09-21T15:49:21.841 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 3.0661353010000001, 2024-09-21T15:49:21.841 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.012789897, 2024-09-21T15:49:21.841 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:21.841 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:21.841 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:21.842 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:21.842 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:21.842 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 14123 2024-09-21T15:49:21.842 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.842 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:21.842 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.842 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:21.842 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.739345+0000", 2024-09-21T15:49:21.842 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:21.842 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.842 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.842 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:21.842 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.739345+0000", 2024-09-21T15:49:21.842 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:21.843 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.843 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.843 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:21.843 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.739345+0000", 2024-09-21T15:49:21.843 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:21.843 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.843 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.843 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:21.843 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.739347+0000", 2024-09-21T15:49:21.843 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.751e-06 2024-09-21T15:49:21.843 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.843 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.843 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:21.843 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.739349+0000", 2024-09-21T15:49:21.844 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9330000000000001e-06 2024-09-21T15:49:21.844 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.844 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.844 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:21.844 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.739352+0000", 2024-09-21T15:49:21.844 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6560000000000001e-06 2024-09-21T15:49:21.844 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.844 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.844 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:21.844 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.744582+0000", 2024-09-21T15:49:21.844 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0052297949999999998 2024-09-21T15:49:21.844 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.844 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.844 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:21.844 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.744657+0000", 2024-09-21T15:49:21.844 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5512000000000005e-05 2024-09-21T15:49:21.845 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.845 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.845 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 7", 2024-09-21T15:49:21.845 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.744918+0000", 2024-09-21T15:49:21.845 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00026077499999999998 2024-09-21T15:49:21.845 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.845 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.845 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:21.845 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.749279+0000", 2024-09-21T15:49:21.845 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0043605110000000001 2024-09-21T15:49:21.845 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.845 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.845 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:21.845 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.752057+0000", 2024-09-21T15:49:21.846 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0027784509999999999 2024-09-21T15:49:21.846 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.846 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.846 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:21.846 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.752102+0000", 2024-09-21T15:49:21.846 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4428999999999998e-05 2024-09-21T15:49:21.846 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.846 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.846 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:21.846 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.752135+0000", 2024-09-21T15:49:21.846 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3300999999999998e-05 2024-09-21T15:49:21.846 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:21.846 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:21.846 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:21.846 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.847 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.847 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:14124 3.14 3:2ac81419:::rbd_object_map.1133dc5adfdf:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-21T15:49:21.847 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:18.739357+0000", 2024-09-21T15:49:21.847 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 3.0661233710000002, 2024-09-21T15:49:21.847 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.01288064, 2024-09-21T15:49:21.847 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:21.847 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:21.847 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:21.847 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:21.847 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:21.847 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 14124 2024-09-21T15:49:21.847 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.847 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:21.848 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.848 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:21.848 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.739357+0000", 2024-09-21T15:49:21.848 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:21.848 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.848 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.848 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:21.848 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.739357+0000", 2024-09-21T15:49:21.848 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:21.848 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.848 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.848 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:21.848 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.739357+0000", 2024-09-21T15:49:21.848 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:21.848 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.848 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.849 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:21.849 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.739359+0000", 2024-09-21T15:49:21.849 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8610000000000001e-06 2024-09-21T15:49:21.849 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.849 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.849 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:21.849 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.739365+0000", 2024-09-21T15:49:21.849 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5880000000000003e-06 2024-09-21T15:49:21.849 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.849 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.849 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:21.849 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.739369+0000", 2024-09-21T15:49:21.849 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6830000000000001e-06 2024-09-21T15:49:21.849 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.849 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.850 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:21.850 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.745330+0000", 2024-09-21T15:49:21.850 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0059609140000000003 2024-09-21T15:49:21.850 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.850 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.850 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:21.850 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.745407+0000", 2024-09-21T15:49:21.850 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7099000000000002e-05 2024-09-21T15:49:21.850 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.850 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.850 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 7", 2024-09-21T15:49:21.850 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.745630+0000", 2024-09-21T15:49:21.850 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00022355500000000001 2024-09-21T15:49:21.850 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.850 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.851 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:21.851 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.749285+0000", 2024-09-21T15:49:21.851 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0036548000000000001 2024-09-21T15:49:21.851 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.851 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.851 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:21.851 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.752175+0000", 2024-09-21T15:49:21.851 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0028896600000000001 2024-09-21T15:49:21.851 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.851 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.851 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:21.851 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.752204+0000", 2024-09-21T15:49:21.851 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8773999999999999e-05 2024-09-21T15:49:21.851 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.851 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.852 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:21.852 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.752238+0000", 2024-09-21T15:49:21.852 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3939000000000002e-05 2024-09-21T15:49:21.852 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:21.852 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:21.852 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:21.852 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.852 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.852 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:14125 3.14 3:2ac81419:::rbd_object_map.1133dc5adfdf:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-21T15:49:21.852 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:18.739372+0000", 2024-09-21T15:49:21.852 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 3.0661085080000001, 2024-09-21T15:49:21.852 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.012971388, 2024-09-21T15:49:21.852 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:21.852 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:21.853 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:21.853 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:21.853 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:21.853 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 14125 2024-09-21T15:49:21.853 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.853 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:21.853 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.853 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:21.853 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.739372+0000", 2024-09-21T15:49:21.853 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:21.853 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.854 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.854 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:21.854 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.739372+0000", 2024-09-21T15:49:21.854 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:21.854 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.854 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.854 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:21.854 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.739372+0000", 2024-09-21T15:49:21.854 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:21.854 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.854 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.854 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:21.854 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.739373+0000", 2024-09-21T15:49:21.855 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.815e-06 2024-09-21T15:49:21.855 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.855 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.855 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:21.855 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.739377+0000", 2024-09-21T15:49:21.855 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.287e-06 2024-09-21T15:49:21.855 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.855 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.855 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:21.855 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.739379+0000", 2024-09-21T15:49:21.855 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5349999999999999e-06 2024-09-21T15:49:21.855 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.855 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.855 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:21.855 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.745925+0000", 2024-09-21T15:49:21.856 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.006545754 2024-09-21T15:49:21.856 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.856 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.856 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:21.856 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.745972+0000", 2024-09-21T15:49:21.856 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7518999999999997e-05 2024-09-21T15:49:21.856 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.856 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.856 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 7", 2024-09-21T15:49:21.856 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.746164+0000", 2024-09-21T15:49:21.856 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00019189700000000001 2024-09-21T15:49:21.856 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.856 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.856 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:21.856 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.749290+0000", 2024-09-21T15:49:21.856 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003125942 2024-09-21T15:49:21.857 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.857 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.857 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:21.857 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.752282+0000", 2024-09-21T15:49:21.857 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0029913380000000001 2024-09-21T15:49:21.857 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.857 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.857 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:21.857 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.752316+0000", 2024-09-21T15:49:21.857 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4218999999999999e-05 2024-09-21T15:49:21.857 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.857 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.857 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:21.857 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.752343+0000", 2024-09-21T15:49:21.857 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7288e-05 2024-09-21T15:49:21.857 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:21.858 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:21.858 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:21.858 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.858 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.858 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:14126 3.14 3:2ac81419:::rbd_object_map.1133dc5adfdf:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-21T15:49:21.858 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:18.739383+0000", 2024-09-21T15:49:21.858 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 3.0660973139999999, 2024-09-21T15:49:21.858 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.013082982999999999, 2024-09-21T15:49:21.858 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:21.858 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:21.858 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:21.858 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:21.858 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:21.858 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 14126 2024-09-21T15:49:21.858 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.859 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:21.859 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.859 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:21.859 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.739383+0000", 2024-09-21T15:49:21.859 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:21.859 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.859 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.859 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:21.859 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.739383+0000", 2024-09-21T15:49:21.859 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:21.859 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.859 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.859 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:21.859 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.739383+0000", 2024-09-21T15:49:21.859 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:21.860 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.860 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.860 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:21.860 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.739385+0000", 2024-09-21T15:49:21.860 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9120000000000001e-06 2024-09-21T15:49:21.860 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.860 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.860 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:21.860 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.739388+0000", 2024-09-21T15:49:21.860 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0809999999999998e-06 2024-09-21T15:49:21.860 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.860 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.860 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:21.860 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.739392+0000", 2024-09-21T15:49:21.860 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0069999999999999e-06 2024-09-21T15:49:21.860 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.860 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.861 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:21.861 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.746466+0000", 2024-09-21T15:49:21.861 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0070742009999999996 2024-09-21T15:49:21.861 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.861 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.861 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:21.861 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.746512+0000", 2024-09-21T15:49:21.861 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6473e-05 2024-09-21T15:49:21.861 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.861 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.861 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 7", 2024-09-21T15:49:21.861 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.746793+0000", 2024-09-21T15:49:21.861 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00028090999999999999 2024-09-21T15:49:21.861 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.862 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.862 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:21.862 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.749493+0000", 2024-09-21T15:49:21.862 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0026998370000000001 2024-09-21T15:49:21.862 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.862 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.862 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:21.862 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.752392+0000", 2024-09-21T15:49:21.862 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0028988519999999999 2024-09-21T15:49:21.862 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.862 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.862 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:21.862 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.752419+0000", 2024-09-21T15:49:21.862 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6619999999999999e-05 2024-09-21T15:49:21.863 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.863 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.863 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:21.863 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.752466+0000", 2024-09-21T15:49:21.863 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7311999999999997e-05 2024-09-21T15:49:21.863 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:21.863 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:21.863 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:21.863 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.863 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.863 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:14127 3.14 3:2ac81419:::rbd_object_map.1133dc5adfdf:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-21T15:49:21.863 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:18.739397+0000", 2024-09-21T15:49:21.863 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 3.0660831719999999, 2024-09-21T15:49:21.863 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.013176426, 2024-09-21T15:49:21.864 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:21.864 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:21.864 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:21.864 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:21.864 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:21.864 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 14127 2024-09-21T15:49:21.864 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.864 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:21.864 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.864 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:21.864 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.739397+0000", 2024-09-21T15:49:21.864 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:21.864 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.864 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.865 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:21.865 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.739397+0000", 2024-09-21T15:49:21.865 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:21.865 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.865 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.865 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:21.865 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.739397+0000", 2024-09-21T15:49:21.865 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:21.865 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.865 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.865 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:21.865 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.739399+0000", 2024-09-21T15:49:21.865 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2680000000000001e-06 2024-09-21T15:49:21.865 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.866 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.866 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:21.866 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.739405+0000", 2024-09-21T15:49:21.866 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8259999999999998e-06 2024-09-21T15:49:21.866 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.866 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.866 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:21.866 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.739408+0000", 2024-09-21T15:49:21.866 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8459999999999999e-06 2024-09-21T15:49:21.866 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.866 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.866 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:21.866 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.747173+0000", 2024-09-21T15:49:21.866 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0077648370000000001 2024-09-21T15:49:21.866 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.867 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.867 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:21.867 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.747220+0000", 2024-09-21T15:49:21.867 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7658999999999999e-05 2024-09-21T15:49:21.867 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.867 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.867 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 7", 2024-09-21T15:49:21.867 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.747407+0000", 2024-09-21T15:49:21.867 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000187176 2024-09-21T15:49:21.867 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.867 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.867 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:21.867 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.749503+0000", 2024-09-21T15:49:21.867 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0020959440000000002 2024-09-21T15:49:21.868 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.868 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.868 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:21.868 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.752507+0000", 2024-09-21T15:49:21.868 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0030038450000000002 2024-09-21T15:49:21.868 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.868 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.868 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:21.868 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.752542+0000", 2024-09-21T15:49:21.868 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4381000000000002e-05 2024-09-21T15:49:21.868 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.868 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.868 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:21.868 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.752574+0000", 2024-09-21T15:49:21.868 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1918999999999997e-05 2024-09-21T15:49:21.869 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:21.869 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:21.869 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:21.869 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.869 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.869 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:14128 3.14 3:2ac81419:::rbd_object_map.1133dc5adfdf:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-21T15:49:21.869 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:18.739412+0000", 2024-09-21T15:49:21.869 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 3.0660683889999998, 2024-09-21T15:49:21.869 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.013301542, 2024-09-21T15:49:21.869 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:21.869 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:21.869 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:21.869 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:21.869 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:21.869 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 14128 2024-09-21T15:49:21.870 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.870 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:21.870 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.870 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:21.870 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.739412+0000", 2024-09-21T15:49:21.870 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:21.870 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.870 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.870 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:21.870 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.739412+0000", 2024-09-21T15:49:21.870 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:21.870 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.870 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.870 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:21.870 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.739412+0000", 2024-09-21T15:49:21.871 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:21.871 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.871 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.871 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:21.871 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.739414+0000", 2024-09-21T15:49:21.871 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9640000000000002e-06 2024-09-21T15:49:21.871 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.871 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.871 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:21.871 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.739418+0000", 2024-09-21T15:49:21.871 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4619999999999998e-06 2024-09-21T15:49:21.871 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.871 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.871 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:21.872 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.739421+0000", 2024-09-21T15:49:21.872 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5519999999999999e-06 2024-09-21T15:49:21.872 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.872 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.872 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:21.872 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.747680+0000", 2024-09-21T15:49:21.872 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.008258975 2024-09-21T15:49:21.872 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.872 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.872 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:21.872 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.747725+0000", 2024-09-21T15:49:21.872 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.549e-05 2024-09-21T15:49:21.872 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.873 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.873 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 7", 2024-09-21T15:49:21.873 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.747914+0000", 2024-09-21T15:49:21.873 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000188591 2024-09-21T15:49:21.873 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.873 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.873 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:21.873 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.749507+0000", 2024-09-21T15:49:21.873 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0015932019999999999 2024-09-21T15:49:21.873 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.873 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.873 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:21.873 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.752620+0000", 2024-09-21T15:49:21.873 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0031127490000000002 2024-09-21T15:49:21.873 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.873 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.874 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:21.874 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.752675+0000", 2024-09-21T15:49:21.874 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5528000000000001e-05 2024-09-21T15:49:21.874 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.874 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.874 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:21.874 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.752713+0000", 2024-09-21T15:49:21.874 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8263999999999998e-05 2024-09-21T15:49:21.874 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:21.874 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:21.874 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:21.874 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.874 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.874 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:14151 3.14 3:2ac81419:::rbd_object_map.1133dc5adfdf:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-21T15:49:21.875 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:18.756899+0000", 2024-09-21T15:49:21.875 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 3.048581671, 2024-09-21T15:49:21.875 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.013666866999999999, 2024-09-21T15:49:21.875 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:21.875 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:21.875 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:21.875 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:21.875 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:21.875 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 14151 2024-09-21T15:49:21.875 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.875 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:21.875 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.875 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:21.875 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.756899+0000", 2024-09-21T15:49:21.875 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:21.876 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.876 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.876 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:21.876 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.756899+0000", 2024-09-21T15:49:21.876 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:21.876 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.876 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.876 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:21.876 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.756898+0000", 2024-09-21T15:49:21.876 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:21.876 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.876 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.876 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:21.876 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.756901+0000", 2024-09-21T15:49:21.876 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1650000000000002e-06 2024-09-21T15:49:21.877 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.877 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.877 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:21.877 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.756910+0000", 2024-09-21T15:49:21.877 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0890000000000006e-06 2024-09-21T15:49:21.877 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.877 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.877 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:21.877 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.756921+0000", 2024-09-21T15:49:21.877 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0307e-05 2024-09-21T15:49:21.877 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.877 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.877 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:21.877 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.763660+0000", 2024-09-21T15:49:21.877 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0067391500000000002 2024-09-21T15:49:21.877 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.878 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.878 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:21.878 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.763721+0000", 2024-09-21T15:49:21.878 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1498e-05 2024-09-21T15:49:21.878 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.878 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.878 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 7", 2024-09-21T15:49:21.878 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.764062+0000", 2024-09-21T15:49:21.878 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00034055799999999997 2024-09-21T15:49:21.878 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.878 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.878 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:21.878 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.769579+0000", 2024-09-21T15:49:21.878 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0055169110000000002 2024-09-21T15:49:21.878 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.879 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.879 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:21.879 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.770486+0000", 2024-09-21T15:49:21.879 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00090701400000000004 2024-09-21T15:49:21.879 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.879 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.879 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:21.879 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.770531+0000", 2024-09-21T15:49:21.879 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5123000000000001e-05 2024-09-21T15:49:21.879 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.879 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.879 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:21.879 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.770565+0000", 2024-09-21T15:49:21.879 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4477999999999998e-05 2024-09-21T15:49:21.879 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:21.879 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:21.880 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:21.880 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.880 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.880 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:14152 3.14 3:2ac81419:::rbd_object_map.1133dc5adfdf:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-21T15:49:21.880 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:18.756929+0000", 2024-09-21T15:49:21.880 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 3.0485516860000001, 2024-09-21T15:49:21.880 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.013723299, 2024-09-21T15:49:21.880 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:21.880 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:21.880 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:21.880 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:21.880 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:21.880 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 14152 2024-09-21T15:49:21.880 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.880 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:21.880 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.881 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:21.881 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.756929+0000", 2024-09-21T15:49:21.881 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:21.881 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.881 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.881 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:21.881 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.756929+0000", 2024-09-21T15:49:21.881 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:21.881 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.881 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.881 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:21.881 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.756928+0000", 2024-09-21T15:49:21.881 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:21.881 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.881 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.881 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:21.881 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.756931+0000", 2024-09-21T15:49:21.882 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1200000000000002e-06 2024-09-21T15:49:21.882 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.882 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.882 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:21.882 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.756938+0000", 2024-09-21T15:49:21.882 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0999999999999998e-06 2024-09-21T15:49:21.882 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.882 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.882 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:21.882 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.756943+0000", 2024-09-21T15:49:21.882 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6369999999999998e-06 2024-09-21T15:49:21.882 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.882 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.882 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:21.882 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.764468+0000", 2024-09-21T15:49:21.883 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0075252950000000004 2024-09-21T15:49:21.883 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.883 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.883 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:21.883 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.764516+0000", 2024-09-21T15:49:21.883 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7327999999999998e-05 2024-09-21T15:49:21.883 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.883 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.883 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 7", 2024-09-21T15:49:21.883 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.764713+0000", 2024-09-21T15:49:21.883 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00019731300000000001 2024-09-21T15:49:21.883 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.883 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.883 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:21.883 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.769607+0000", 2024-09-21T15:49:21.883 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0048936279999999997 2024-09-21T15:49:21.884 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.884 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.884 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:21.884 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.770573+0000", 2024-09-21T15:49:21.884 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00096667699999999999 2024-09-21T15:49:21.884 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.884 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.884 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:21.884 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.770615+0000", 2024-09-21T15:49:21.884 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1967e-05 2024-09-21T15:49:21.884 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.884 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.884 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:21.884 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.770652+0000", 2024-09-21T15:49:21.884 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6674000000000002e-05 2024-09-21T15:49:21.884 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:21.885 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:21.885 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:21.885 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.885 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.885 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:14225 3.14 3:2ac81419:::rbd_object_map.1133dc5adfdf:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-21T15:49:21.885 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:18.799418+0000", 2024-09-21T15:49:21.885 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 3.0060622709999998, 2024-09-21T15:49:21.885 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.014726715, 2024-09-21T15:49:21.885 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:21.885 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:21.885 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:21.885 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:21.885 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:21.885 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 14225 2024-09-21T15:49:21.885 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.886 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:21.886 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.886 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:21.886 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.799418+0000", 2024-09-21T15:49:21.886 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:21.886 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.886 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.886 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:21.886 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.799418+0000", 2024-09-21T15:49:21.886 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:21.886 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.886 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.886 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:21.886 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.799418+0000", 2024-09-21T15:49:21.886 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:21.887 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.887 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.887 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:21.887 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.799420+0000", 2024-09-21T15:49:21.887 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.801e-06 2024-09-21T15:49:21.887 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.887 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.887 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:21.887 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.799426+0000", 2024-09-21T15:49:21.887 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.268e-06 2024-09-21T15:49:21.887 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.887 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.887 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:21.887 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.799430+0000", 2024-09-21T15:49:21.887 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0099999999999997e-06 2024-09-21T15:49:21.888 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.888 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.888 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:21.888 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.807648+0000", 2024-09-21T15:49:21.888 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0082185350000000008 2024-09-21T15:49:21.888 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.888 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.888 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:21.888 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.807701+0000", 2024-09-21T15:49:21.888 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2938e-05 2024-09-21T15:49:21.888 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.888 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.888 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 7", 2024-09-21T15:49:21.888 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.807927+0000", 2024-09-21T15:49:21.889 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00022552600000000001 2024-09-21T15:49:21.889 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.889 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.889 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:21.889 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.811380+0000", 2024-09-21T15:49:21.889 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0034529270000000002 2024-09-21T15:49:21.889 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.889 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.889 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:21.889 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.814072+0000", 2024-09-21T15:49:21.889 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0026921430000000001 2024-09-21T15:49:21.889 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.889 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.889 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:21.889 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.814116+0000", 2024-09-21T15:49:21.889 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3569000000000002e-05 2024-09-21T15:49:21.890 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.890 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.890 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:21.890 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.814145+0000", 2024-09-21T15:49:21.890 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9218e-05 2024-09-21T15:49:21.890 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:21.890 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:21.890 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:21.890 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.890 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.890 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:14526 3.14 3:2ac81419:::rbd_object_map.1133dc5adfdf:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-21T15:49:21.890 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:19.107882+0000", 2024-09-21T15:49:21.890 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 2.6975985850000002, 2024-09-21T15:49:21.890 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.012963977, 2024-09-21T15:49:21.890 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:21.891 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:21.891 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:21.891 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:21.891 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:21.891 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 14526 2024-09-21T15:49:21.891 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.891 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:21.891 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.891 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:21.891 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.107882+0000", 2024-09-21T15:49:21.891 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:21.891 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.891 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.891 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:21.891 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.107882+0000", 2024-09-21T15:49:21.891 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:21.892 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.892 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.892 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:21.892 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.107881+0000", 2024-09-21T15:49:21.892 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:21.892 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.892 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.892 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:21.892 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.107883+0000", 2024-09-21T15:49:21.892 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.545e-06 2024-09-21T15:49:21.892 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.892 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.892 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:21.892 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.107887+0000", 2024-09-21T15:49:21.892 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8029999999999998e-06 2024-09-21T15:49:21.893 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.893 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.893 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:21.893 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.107889+0000", 2024-09-21T15:49:21.893 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5409999999999999e-06 2024-09-21T15:49:21.893 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.893 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.893 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:21.893 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.114368+0000", 2024-09-21T15:49:21.893 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0064782920000000001 2024-09-21T15:49:21.893 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.893 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.893 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:21.893 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.114445+0000", 2024-09-21T15:49:21.893 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7693000000000003e-05 2024-09-21T15:49:21.893 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.893 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.894 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 7", 2024-09-21T15:49:21.894 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.114752+0000", 2024-09-21T15:49:21.894 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00030619099999999998 2024-09-21T15:49:21.894 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.894 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.894 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:21.894 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.118215+0000", 2024-09-21T15:49:21.894 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0034639359999999999 2024-09-21T15:49:21.894 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.894 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.894 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:21.894 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.120750+0000", 2024-09-21T15:49:21.894 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002534232 2024-09-21T15:49:21.894 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.894 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.894 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:21.895 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.120814+0000", 2024-09-21T15:49:21.895 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4098999999999997e-05 2024-09-21T15:49:21.895 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.895 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.895 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:21.895 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.120846+0000", 2024-09-21T15:49:21.895 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1869999999999998e-05 2024-09-21T15:49:21.895 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:21.895 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:21.895 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:21.895 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.895 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.895 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:14527 3.14 3:2ac81419:::rbd_object_map.1133dc5adfdf:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-21T15:49:21.895 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:19.107892+0000", 2024-09-21T15:49:21.895 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 2.6975887429999998, 2024-09-21T15:49:21.896 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.013029482, 2024-09-21T15:49:21.896 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:21.896 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:21.896 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:21.896 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:21.896 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:21.896 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 14527 2024-09-21T15:49:21.896 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.896 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:21.896 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.896 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:21.896 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.107892+0000", 2024-09-21T15:49:21.896 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:21.896 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.896 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.896 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:21.897 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.107892+0000", 2024-09-21T15:49:21.897 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:21.897 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.897 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.897 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:21.897 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.107891+0000", 2024-09-21T15:49:21.897 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:21.897 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.897 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.897 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:21.897 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.107893+0000", 2024-09-21T15:49:21.897 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.545e-06 2024-09-21T15:49:21.897 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.897 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.897 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:21.897 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.107896+0000", 2024-09-21T15:49:21.898 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9229999999999998e-06 2024-09-21T15:49:21.898 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.898 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.898 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:21.898 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.107898+0000", 2024-09-21T15:49:21.898 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4219999999999999e-06 2024-09-21T15:49:21.898 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.898 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.898 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:21.898 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.115210+0000", 2024-09-21T15:49:21.898 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0073116830000000002 2024-09-21T15:49:21.898 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.898 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.898 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:21.898 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.115262+0000", 2024-09-21T15:49:21.898 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1789999999999997e-05 2024-09-21T15:49:21.899 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.899 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.899 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 7", 2024-09-21T15:49:21.899 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.115445+0000", 2024-09-21T15:49:21.899 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00018346899999999999 2024-09-21T15:49:21.899 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.899 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.899 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:21.899 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.118264+0000", 2024-09-21T15:49:21.899 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0028191430000000001 2024-09-21T15:49:21.899 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.899 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.899 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:21.899 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.120853+0000", 2024-09-21T15:49:21.899 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0025887449999999999 2024-09-21T15:49:21.899 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.899 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.900 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:21.900 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.120891+0000", 2024-09-21T15:49:21.900 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8281999999999997e-05 2024-09-21T15:49:21.900 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.900 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.900 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:21.900 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.120921+0000", 2024-09-21T15:49:21.900 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9699000000000001e-05 2024-09-21T15:49:21.900 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:21.900 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:21.900 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:21.900 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.900 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.900 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:14528 3.14 3:2ac81419:::rbd_object_map.1133dc5adfdf:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-21T15:49:21.900 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:19.107900+0000", 2024-09-21T15:49:21.900 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 2.6975799710000001, 2024-09-21T15:49:21.901 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.013097161, 2024-09-21T15:49:21.901 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:21.901 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:21.901 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:21.901 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:21.901 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:21.901 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 14528 2024-09-21T15:49:21.901 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.901 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:21.901 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.901 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:21.901 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.107900+0000", 2024-09-21T15:49:21.901 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:21.901 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.901 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.901 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:21.902 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.107900+0000", 2024-09-21T15:49:21.902 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:21.902 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.902 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.902 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:21.902 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.107900+0000", 2024-09-21T15:49:21.902 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:21.902 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.902 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.902 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:21.902 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.107902+0000", 2024-09-21T15:49:21.902 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6190000000000001e-06 2024-09-21T15:49:21.902 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.902 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.902 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:21.902 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.107905+0000", 2024-09-21T15:49:21.903 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7629999999999998e-06 2024-09-21T15:49:21.903 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.903 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.903 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:21.903 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.107908+0000", 2024-09-21T15:49:21.903 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4899999999999999e-06 2024-09-21T15:49:21.903 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.903 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.903 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:21.903 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.115810+0000", 2024-09-21T15:49:21.903 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0079017919999999995 2024-09-21T15:49:21.903 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.903 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.903 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:21.903 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.115856+0000", 2024-09-21T15:49:21.903 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6690000000000002e-05 2024-09-21T15:49:21.903 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.904 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.904 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 7", 2024-09-21T15:49:21.904 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.116045+0000", 2024-09-21T15:49:21.904 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00018858699999999999 2024-09-21T15:49:21.904 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.904 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.904 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:21.904 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.118289+0000", 2024-09-21T15:49:21.904 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0022444779999999998 2024-09-21T15:49:21.904 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.904 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.904 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:21.904 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.120928+0000", 2024-09-21T15:49:21.904 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002638137 2024-09-21T15:49:21.905 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.905 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.905 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:21.905 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.120968+0000", 2024-09-21T15:49:21.905 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0172000000000001e-05 2024-09-21T15:49:21.905 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.905 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.905 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:21.905 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.120997+0000", 2024-09-21T15:49:21.905 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9658999999999998e-05 2024-09-21T15:49:21.905 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:21.905 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:21.905 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:21.905 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.905 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.905 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:14529 3.14 3:2ac81419:::rbd_object_map.1133dc5adfdf:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-21T15:49:21.906 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:19.107910+0000", 2024-09-21T15:49:21.906 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 2.6975701880000003, 2024-09-21T15:49:21.906 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.013431007, 2024-09-21T15:49:21.906 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:21.906 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:21.906 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:21.906 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:21.906 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:21.906 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 14529 2024-09-21T15:49:21.906 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.906 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:21.906 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.906 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:21.906 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.107910+0000", 2024-09-21T15:49:21.906 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:21.906 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.907 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.907 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:21.907 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.107910+0000", 2024-09-21T15:49:21.907 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:21.907 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.907 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.907 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:21.907 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.107910+0000", 2024-09-21T15:49:21.907 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:21.907 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.907 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.907 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:21.907 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.107911+0000", 2024-09-21T15:49:21.907 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.486e-06 2024-09-21T15:49:21.907 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.908 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.908 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:21.908 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.107914+0000", 2024-09-21T15:49:21.908 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9450000000000002e-06 2024-09-21T15:49:21.908 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.908 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.908 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:21.908 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.107917+0000", 2024-09-21T15:49:21.908 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3949999999999999e-06 2024-09-21T15:49:21.908 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.908 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.908 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:21.908 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.116411+0000", 2024-09-21T15:49:21.908 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0084939709999999995 2024-09-21T15:49:21.909 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.909 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.909 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:21.909 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.116482+0000", 2024-09-21T15:49:21.909 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1386999999999999e-05 2024-09-21T15:49:21.909 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.909 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.909 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 7", 2024-09-21T15:49:21.909 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.116662+0000", 2024-09-21T15:49:21.909 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000179569 2024-09-21T15:49:21.909 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.909 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.909 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:21.909 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.121005+0000", 2024-09-21T15:49:21.910 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0043430669999999999 2024-09-21T15:49:21.910 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.910 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.910 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:21.910 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.121220+0000", 2024-09-21T15:49:21.910 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00021515600000000001 2024-09-21T15:49:21.910 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.910 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.910 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:21.910 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.121292+0000", 2024-09-21T15:49:21.910 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2600000000000003e-05 2024-09-21T15:49:21.910 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.910 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.910 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:21.911 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.121341+0000", 2024-09-21T15:49:21.911 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8656000000000003e-05 2024-09-21T15:49:21.911 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:21.911 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:21.911 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:21.911 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.911 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.911 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:14530 3.14 3:2ac81419:::rbd_object_map.1133dc5adfdf:head [call lock.assert_locked in=17b,call rbd.object_map_update in=20b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-21T15:49:21.911 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:19.107919+0000", 2024-09-21T15:49:21.911 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 2.697560953, 2024-09-21T15:49:21.911 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.015994900999999999, 2024-09-21T15:49:21.911 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:21.911 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:21.911 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:21.912 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:21.912 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:21.912 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 14530 2024-09-21T15:49:21.912 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.912 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:21.912 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.912 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:21.912 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.107919+0000", 2024-09-21T15:49:21.912 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:21.912 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.912 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.912 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:21.912 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.107919+0000", 2024-09-21T15:49:21.912 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:21.912 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.912 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.912 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:21.913 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.107919+0000", 2024-09-21T15:49:21.913 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:21.913 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.913 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.913 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:21.913 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.107921+0000", 2024-09-21T15:49:21.913 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7379999999999999e-06 2024-09-21T15:49:21.913 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.913 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.913 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:21.913 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.107924+0000", 2024-09-21T15:49:21.913 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9249999999999999e-06 2024-09-21T15:49:21.913 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.913 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.913 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:21.913 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.107926+0000", 2024-09-21T15:49:21.913 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3460000000000001e-06 2024-09-21T15:49:21.914 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.914 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.914 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:21.914 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.117029+0000", 2024-09-21T15:49:21.914 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0091028979999999999 2024-09-21T15:49:21.914 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.914 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.914 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:21.914 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.117101+0000", 2024-09-21T15:49:21.914 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1713999999999998e-05 2024-09-21T15:49:21.914 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.914 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.914 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 7", 2024-09-21T15:49:21.914 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.117388+0000", 2024-09-21T15:49:21.914 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00028701300000000003 2024-09-21T15:49:21.915 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.915 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.915 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:21.915 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.121368+0000", 2024-09-21T15:49:21.915 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0039798510000000004 2024-09-21T15:49:21.915 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.915 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.915 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:21.915 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.123843+0000", 2024-09-21T15:49:21.915 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0024754600000000001 2024-09-21T15:49:21.915 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.915 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.915 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:21.915 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.123880+0000", 2024-09-21T15:49:21.915 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7045999999999997e-05 2024-09-21T15:49:21.915 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:21.916 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:21.916 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:21.916 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.123914+0000", 2024-09-21T15:49:21.916 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4133e-05 2024-09-21T15:49:21.916 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:21.916 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:21.916 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:21.916 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:21.916 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:21.916 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:21.918 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:49:21.918 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:21.916+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:22.018 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:22.017+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:22.020 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:22.021 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:22.021 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:49:22.021 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:22.029 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:49:22.119 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:22.117+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:22.171 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:22.171 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:22.171 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:49:22.171 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:49:22.172 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:22.180 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:49:22.220 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:22.219+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:22.321 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:22.320+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:22.324 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:22.324 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:49:22.324 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:49:22.324 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:49:22.324 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.324 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:4938 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 659456~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:22.324 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.455162+0000", 2024-09-21T15:49:22.324 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.8670283540000003, 2024-09-21T15:49:22.325 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039399295000000001, 2024-09-21T15:49:22.325 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:22.325 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:22.325 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:22.325 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:22.325 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:22.325 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 4938 2024-09-21T15:49:22.325 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.325 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:22.325 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.325 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:22.325 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.455162+0000", 2024-09-21T15:49:22.326 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.326 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.326 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.326 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:22.326 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.455162+0000", 2024-09-21T15:49:22.326 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.326 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.326 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.326 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:22.326 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.455162+0000", 2024-09-21T15:49:22.326 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:22.327 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.327 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.327 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:22.327 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.455164+0000", 2024-09-21T15:49:22.327 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.074e-06 2024-09-21T15:49:22.327 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.327 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.327 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:22.327 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.455169+0000", 2024-09-21T15:49:22.327 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6760000000000004e-06 2024-09-21T15:49:22.327 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.327 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.328 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:22.328 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.455174+0000", 2024-09-21T15:49:22.328 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3680000000000004e-06 2024-09-21T15:49:22.328 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.328 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.328 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:22.328 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.494363+0000", 2024-09-21T15:49:22.328 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039189225000000001 2024-09-21T15:49:22.328 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.328 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.328 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:22.328 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.494456+0000", 2024-09-21T15:49:22.329 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2843999999999994e-05 2024-09-21T15:49:22.329 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.329 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.329 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:22.329 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.494561+0000", 2024-09-21T15:49:22.329 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000105627 2024-09-21T15:49:22.329 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.329 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:22.329 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.329 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.329 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.329 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:4939 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 708608~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:22.329 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.455178+0000", 2024-09-21T15:49:22.330 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.8670126629999997, 2024-09-21T15:49:22.330 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039493579000000001, 2024-09-21T15:49:22.330 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:22.330 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:22.330 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:22.330 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:22.330 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:22.330 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 4939 2024-09-21T15:49:22.330 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.330 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:22.330 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.331 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:22.331 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.455178+0000", 2024-09-21T15:49:22.331 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.331 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.331 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.331 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:22.331 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.455178+0000", 2024-09-21T15:49:22.331 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.331 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.331 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.331 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:22.331 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.455177+0000", 2024-09-21T15:49:22.332 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:22.332 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.332 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.332 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:22.332 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.455179+0000", 2024-09-21T15:49:22.332 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9489999999999999e-06 2024-09-21T15:49:22.332 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.332 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.332 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:22.332 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.455185+0000", 2024-09-21T15:49:22.332 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.49e-06 2024-09-21T15:49:22.332 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.333 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.333 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:22.333 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.455190+0000", 2024-09-21T15:49:22.333 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5910000000000003e-06 2024-09-21T15:49:22.333 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.333 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.333 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:22.333 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.494573+0000", 2024-09-21T15:49:22.333 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039382895000000001 2024-09-21T15:49:22.333 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.333 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.333 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:22.333 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.494622+0000", 2024-09-21T15:49:22.334 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8167000000000001e-05 2024-09-21T15:49:22.334 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.334 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.334 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:22.334 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.494671+0000", 2024-09-21T15:49:22.334 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9818000000000003e-05 2024-09-21T15:49:22.334 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.334 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:22.334 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.334 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.334 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.334 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:4949 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 1200128~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:22.335 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.457117+0000", 2024-09-21T15:49:22.335 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.8650734020000002, 2024-09-21T15:49:22.335 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039319875999999997, 2024-09-21T15:49:22.335 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:22.335 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:22.335 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:22.335 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:22.335 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:22.335 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 4949 2024-09-21T15:49:22.335 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.335 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:22.336 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.336 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:22.336 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.457117+0000", 2024-09-21T15:49:22.336 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.336 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.336 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.336 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:22.336 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.457117+0000", 2024-09-21T15:49:22.336 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.336 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.336 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.336 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:22.337 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.457117+0000", 2024-09-21T15:49:22.337 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:22.337 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.337 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.337 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:22.337 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.457118+0000", 2024-09-21T15:49:22.337 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0729999999999999e-06 2024-09-21T15:49:22.337 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.337 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.337 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:22.337 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.457121+0000", 2024-09-21T15:49:22.337 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.802e-06 2024-09-21T15:49:22.338 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.338 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.338 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:22.338 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.457123+0000", 2024-09-21T15:49:22.338 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1969999999999999e-06 2024-09-21T15:49:22.338 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.338 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.338 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:22.338 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.496340+0000", 2024-09-21T15:49:22.338 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039217120000000001 2024-09-21T15:49:22.338 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.338 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.338 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:22.339 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.496386+0000", 2024-09-21T15:49:22.339 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6360000000000003e-05 2024-09-21T15:49:22.339 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.339 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.339 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:22.339 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.496437+0000", 2024-09-21T15:49:22.339 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0555e-05 2024-09-21T15:49:22.339 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.339 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:22.339 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.339 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.339 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.339 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:4950 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 1249280~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:22.340 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.457125+0000", 2024-09-21T15:49:22.340 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.8650654129999999, 2024-09-21T15:49:22.340 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039545631999999997, 2024-09-21T15:49:22.340 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:22.340 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:22.340 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:22.340 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:22.340 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:22.340 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 4950 2024-09-21T15:49:22.340 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.340 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:22.340 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.341 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:22.341 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.457125+0000", 2024-09-21T15:49:22.341 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.341 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.341 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.341 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:22.341 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.457125+0000", 2024-09-21T15:49:22.341 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.341 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.341 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.341 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:22.341 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.457125+0000", 2024-09-21T15:49:22.341 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:22.342 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.342 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.342 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:22.342 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.457126+0000", 2024-09-21T15:49:22.342 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5519999999999999e-06 2024-09-21T15:49:22.342 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.342 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.342 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:22.342 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.457129+0000", 2024-09-21T15:49:22.342 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7989999999999998e-06 2024-09-21T15:49:22.342 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.342 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.343 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:22.343 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.457131+0000", 2024-09-21T15:49:22.343 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3080000000000002e-06 2024-09-21T15:49:22.343 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.343 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.343 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:22.343 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.496473+0000", 2024-09-21T15:49:22.343 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039341222000000002 2024-09-21T15:49:22.343 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.343 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.343 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:22.344 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.496565+0000", 2024-09-21T15:49:22.344 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2330000000000005e-05 2024-09-21T15:49:22.344 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.344 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.344 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:22.344 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.496671+0000", 2024-09-21T15:49:22.344 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000105648 2024-09-21T15:49:22.344 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.344 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:22.344 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.344 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.344 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.345 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:4962 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 1789952~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:22.345 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.459143+0000", 2024-09-21T15:49:22.345 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.8630471599999998, 2024-09-21T15:49:22.345 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039478257000000003, 2024-09-21T15:49:22.345 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:22.345 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:22.345 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:22.345 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:22.345 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:22.345 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 4962 2024-09-21T15:49:22.345 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.345 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:22.345 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.346 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:22.346 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459143+0000", 2024-09-21T15:49:22.346 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.346 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.346 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.346 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:22.346 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459143+0000", 2024-09-21T15:49:22.346 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.346 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.346 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.346 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:22.347 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459143+0000", 2024-09-21T15:49:22.347 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:22.347 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.347 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.347 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:22.347 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459144+0000", 2024-09-21T15:49:22.347 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.142e-06 2024-09-21T15:49:22.347 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.347 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.347 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:22.347 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459147+0000", 2024-09-21T15:49:22.347 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8229999999999999e-06 2024-09-21T15:49:22.347 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.348 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.348 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:22.348 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459149+0000", 2024-09-21T15:49:22.348 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2859999999999998e-06 2024-09-21T15:49:22.348 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.348 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.348 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:22.348 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.498430+0000", 2024-09-21T15:49:22.348 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039280806000000001 2024-09-21T15:49:22.348 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.348 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.348 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:22.348 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.498520+0000", 2024-09-21T15:49:22.348 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0160000000000004e-05 2024-09-21T15:49:22.349 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.349 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.349 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:22.349 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.498622+0000", 2024-09-21T15:49:22.349 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000101275 2024-09-21T15:49:22.349 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.349 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:22.349 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.349 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.349 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.349 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:4963 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 1839104~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:22.349 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.459152+0000", 2024-09-21T15:49:22.349 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.8630387649999998, 2024-09-21T15:49:22.350 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039710886000000001, 2024-09-21T15:49:22.350 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:22.350 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:22.350 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:22.350 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:22.350 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:22.350 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 4963 2024-09-21T15:49:22.350 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.350 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:22.350 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.350 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:22.350 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459152+0000", 2024-09-21T15:49:22.351 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.351 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.351 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.351 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:22.351 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459152+0000", 2024-09-21T15:49:22.351 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.351 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.351 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.351 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:22.351 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459151+0000", 2024-09-21T15:49:22.351 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:22.351 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.351 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.352 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:22.352 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459153+0000", 2024-09-21T15:49:22.352 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1400000000000001e-06 2024-09-21T15:49:22.352 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.352 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.352 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:22.352 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459155+0000", 2024-09-21T15:49:22.352 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7130000000000001e-06 2024-09-21T15:49:22.352 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.352 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.352 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:22.352 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459158+0000", 2024-09-21T15:49:22.352 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.368e-06 2024-09-21T15:49:22.352 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.353 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.353 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:22.353 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.498664+0000", 2024-09-21T15:49:22.353 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039505928000000003 2024-09-21T15:49:22.353 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.353 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.353 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:22.353 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.498756+0000", 2024-09-21T15:49:22.353 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2505999999999998e-05 2024-09-21T15:49:22.353 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.353 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.353 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:22.353 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.498863+0000", 2024-09-21T15:49:22.353 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000106457 2024-09-21T15:49:22.354 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.354 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:22.354 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.354 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.354 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.354 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:4964 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 1888256~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:22.354 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.459160+0000", 2024-09-21T15:49:22.354 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.8630300479999997, 2024-09-21T15:49:22.354 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039933237000000003, 2024-09-21T15:49:22.354 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:22.354 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:22.354 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:22.354 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:22.355 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:22.355 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 4964 2024-09-21T15:49:22.355 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.355 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:22.355 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.355 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:22.355 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459160+0000", 2024-09-21T15:49:22.355 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.355 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.355 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.355 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:22.356 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459160+0000", 2024-09-21T15:49:22.356 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.356 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.356 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.356 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:22.356 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459160+0000", 2024-09-21T15:49:22.356 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:22.356 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.356 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.356 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:22.356 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459162+0000", 2024-09-21T15:49:22.356 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.42e-06 2024-09-21T15:49:22.356 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.357 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.357 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:22.357 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459164+0000", 2024-09-21T15:49:22.357 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8059999999999999e-06 2024-09-21T15:49:22.357 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.357 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.357 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:22.357 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459167+0000", 2024-09-21T15:49:22.357 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.835e-06 2024-09-21T15:49:22.357 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.357 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.357 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:22.358 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.498904+0000", 2024-09-21T15:49:22.358 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039736831 2024-09-21T15:49:22.358 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.358 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.358 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:22.358 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.498992+0000", 2024-09-21T15:49:22.358 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7886999999999994e-05 2024-09-21T15:49:22.358 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.358 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.358 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:22.358 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.499094+0000", 2024-09-21T15:49:22.358 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000101681 2024-09-21T15:49:22.358 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.359 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:22.359 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.359 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.359 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.359 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:4966 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 1986560~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:22.359 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.460005+0000", 2024-09-21T15:49:22.359 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.8621850220000002, 2024-09-21T15:49:22.359 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039390041000000001, 2024-09-21T15:49:22.359 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:22.359 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:22.359 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:22.359 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:22.359 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:22.360 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 4966 2024-09-21T15:49:22.360 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.360 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:22.360 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.360 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:22.360 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460005+0000", 2024-09-21T15:49:22.360 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.360 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.360 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.360 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:22.360 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460005+0000", 2024-09-21T15:49:22.360 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.360 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.361 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.361 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:22.361 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460005+0000", 2024-09-21T15:49:22.361 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:22.361 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.361 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.361 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:22.361 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460006+0000", 2024-09-21T15:49:22.361 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2359999999999999e-06 2024-09-21T15:49:22.361 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.361 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.361 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:22.362 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460013+0000", 2024-09-21T15:49:22.362 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1239999999999999e-06 2024-09-21T15:49:22.362 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.362 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.362 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:22.362 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460018+0000", 2024-09-21T15:49:22.362 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3449999999999999e-06 2024-09-21T15:49:22.362 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.362 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.362 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:22.362 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.499283+0000", 2024-09-21T15:49:22.362 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039264847999999998 2024-09-21T15:49:22.362 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.363 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.363 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:22.363 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.499334+0000", 2024-09-21T15:49:22.363 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1279000000000002e-05 2024-09-21T15:49:22.363 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.363 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.363 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:22.363 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.499395+0000", 2024-09-21T15:49:22.363 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1472999999999999e-05 2024-09-21T15:49:22.363 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.363 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:22.363 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.364 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.364 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.364 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:4967 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 2035712~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:22.364 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.460021+0000", 2024-09-21T15:49:22.364 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.8621691120000001, 2024-09-21T15:49:22.364 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039505047000000001, 2024-09-21T15:49:22.364 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:22.364 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:22.364 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:22.364 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:22.364 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:22.364 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 4967 2024-09-21T15:49:22.365 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.365 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:22.365 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.365 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:22.365 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460021+0000", 2024-09-21T15:49:22.365 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.365 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.365 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.365 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:22.365 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460021+0000", 2024-09-21T15:49:22.365 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.365 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.365 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.366 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:22.366 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460021+0000", 2024-09-21T15:49:22.366 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:22.366 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.366 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.366 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:22.366 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460022+0000", 2024-09-21T15:49:22.366 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3710000000000001e-06 2024-09-21T15:49:22.366 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.366 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.366 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:22.366 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460025+0000", 2024-09-21T15:49:22.367 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.067e-06 2024-09-21T15:49:22.367 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.367 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.367 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:22.367 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460028+0000", 2024-09-21T15:49:22.367 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5009999999999999e-06 2024-09-21T15:49:22.367 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.367 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.367 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:22.367 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.499420+0000", 2024-09-21T15:49:22.367 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039391977000000002 2024-09-21T15:49:22.367 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.367 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.367 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:22.368 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.499472+0000", 2024-09-21T15:49:22.368 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2057e-05 2024-09-21T15:49:22.368 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.368 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.368 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:22.368 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.499526+0000", 2024-09-21T15:49:22.368 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4335000000000001e-05 2024-09-21T15:49:22.368 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.368 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:22.368 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.368 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.368 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.368 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:4968 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 2084864~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:22.368 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.460030+0000", 2024-09-21T15:49:22.369 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.8621603640000002, 2024-09-21T15:49:22.369 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039629655, 2024-09-21T15:49:22.369 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:22.369 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:22.369 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:22.369 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:22.369 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:22.369 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 4968 2024-09-21T15:49:22.369 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.369 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:22.369 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.369 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:22.369 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460030+0000", 2024-09-21T15:49:22.370 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.370 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.370 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.370 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:22.370 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460030+0000", 2024-09-21T15:49:22.370 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.370 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.370 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.370 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:22.370 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460030+0000", 2024-09-21T15:49:22.370 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:22.371 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.371 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.371 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:22.371 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460037+0000", 2024-09-21T15:49:22.371 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8569999999999997e-06 2024-09-21T15:49:22.371 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.371 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.371 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:22.371 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460040+0000", 2024-09-21T15:49:22.371 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1729999999999999e-06 2024-09-21T15:49:22.371 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.371 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.371 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:22.372 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460042+0000", 2024-09-21T15:49:22.372 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4559999999999999e-06 2024-09-21T15:49:22.372 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.372 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.372 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:22.372 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.499547+0000", 2024-09-21T15:49:22.372 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039504723999999998 2024-09-21T15:49:22.372 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.372 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.372 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:22.372 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.499596+0000", 2024-09-21T15:49:22.372 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9230000000000001e-05 2024-09-21T15:49:22.373 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.373 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.373 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:22.373 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.499660+0000", 2024-09-21T15:49:22.373 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3473999999999996e-05 2024-09-21T15:49:22.373 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.373 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:22.373 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.373 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.373 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.373 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:5054 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 24576~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:22.373 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.460046+0000", 2024-09-21T15:49:22.373 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.8621442219999995, 2024-09-21T15:49:22.373 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039744092000000002, 2024-09-21T15:49:22.374 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:22.374 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:22.374 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:22.374 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:22.374 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:22.374 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 5054 2024-09-21T15:49:22.374 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.374 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:22.374 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.374 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:22.374 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460046+0000", 2024-09-21T15:49:22.374 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.375 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.375 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.375 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:22.375 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460046+0000", 2024-09-21T15:49:22.375 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.375 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.375 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.375 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:22.375 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460046+0000", 2024-09-21T15:49:22.375 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:22.375 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.375 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.375 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:22.376 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460051+0000", 2024-09-21T15:49:22.376 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4790000000000004e-06 2024-09-21T15:49:22.376 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.376 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.376 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:22.376 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460055+0000", 2024-09-21T15:49:22.376 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6830000000000001e-06 2024-09-21T15:49:22.376 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.376 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.376 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:22.376 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460057+0000", 2024-09-21T15:49:22.376 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5550000000000001e-06 2024-09-21T15:49:22.376 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.376 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.377 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:22.377 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.499687+0000", 2024-09-21T15:49:22.377 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039629736999999998 2024-09-21T15:49:22.377 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.377 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.377 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:22.377 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.499734+0000", 2024-09-21T15:49:22.377 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6412999999999998e-05 2024-09-21T15:49:22.377 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.377 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.377 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:22.377 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.499790+0000", 2024-09-21T15:49:22.377 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6666999999999998e-05 2024-09-21T15:49:22.377 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.378 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:22.378 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.378 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.378 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.378 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:5055 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 73728~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:22.378 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.460060+0000", 2024-09-21T15:49:22.378 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.8621306070000001, 2024-09-21T15:49:22.378 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039853507000000003, 2024-09-21T15:49:22.378 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:22.378 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:22.378 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:22.379 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:22.379 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:22.379 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 5055 2024-09-21T15:49:22.379 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.379 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:22.379 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.379 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:22.379 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460060+0000", 2024-09-21T15:49:22.379 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.379 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.379 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.379 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:22.379 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460060+0000", 2024-09-21T15:49:22.379 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.380 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.380 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.380 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:22.380 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460060+0000", 2024-09-21T15:49:22.380 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:22.380 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.380 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.380 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:22.380 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460061+0000", 2024-09-21T15:49:22.380 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2410000000000001e-06 2024-09-21T15:49:22.380 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.380 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.380 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:22.380 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460064+0000", 2024-09-21T15:49:22.381 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7539999999999998e-06 2024-09-21T15:49:22.381 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.381 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.381 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:22.381 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460069+0000", 2024-09-21T15:49:22.381 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.558e-06 2024-09-21T15:49:22.381 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.381 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.381 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:22.381 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.499816+0000", 2024-09-21T15:49:22.381 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039746385000000002 2024-09-21T15:49:22.381 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.381 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.382 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:22.382 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.499860+0000", 2024-09-21T15:49:22.382 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4740999999999997e-05 2024-09-21T15:49:22.382 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.382 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.382 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:22.382 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.499913+0000", 2024-09-21T15:49:22.382 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3057999999999999e-05 2024-09-21T15:49:22.382 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.382 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:22.382 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.382 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.382 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.383 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:5056 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 122880~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:22.383 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.460249+0000", 2024-09-21T15:49:22.383 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.8619417479999996, 2024-09-21T15:49:22.383 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039822748999999998, 2024-09-21T15:49:22.383 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:22.383 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:22.383 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:22.383 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:22.383 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:22.383 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 5056 2024-09-21T15:49:22.383 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.383 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:22.383 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.383 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:22.384 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460249+0000", 2024-09-21T15:49:22.384 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.384 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.384 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.384 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:22.384 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460249+0000", 2024-09-21T15:49:22.384 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.384 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.384 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.384 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:22.384 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460120+0000", 2024-09-21T15:49:22.384 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998713 2024-09-21T15:49:22.385 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.385 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.385 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:22.385 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460251+0000", 2024-09-21T15:49:22.385 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000130977 2024-09-21T15:49:22.385 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.385 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.385 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:22.385 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460261+0000", 2024-09-21T15:49:22.385 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.8879999999999992e-06 2024-09-21T15:49:22.385 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.385 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.385 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:22.386 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460267+0000", 2024-09-21T15:49:22.386 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7379999999999999e-06 2024-09-21T15:49:22.386 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.386 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.386 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:22.386 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.499940+0000", 2024-09-21T15:49:22.386 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039673228999999997 2024-09-21T15:49:22.386 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.386 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.386 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:22.386 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500008+0000", 2024-09-21T15:49:22.386 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7696999999999999e-05 2024-09-21T15:49:22.387 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.387 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.387 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:22.387 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500071+0000", 2024-09-21T15:49:22.387 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3856000000000007e-05 2024-09-21T15:49:22.387 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.387 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:22.387 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.387 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.387 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.387 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:5057 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 172032~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:22.388 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.460377+0000", 2024-09-21T15:49:22.388 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.861812961, 2024-09-21T15:49:22.388 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039811802, 2024-09-21T15:49:22.388 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:22.388 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:22.388 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:22.388 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:22.388 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:22.388 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 5057 2024-09-21T15:49:22.388 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.388 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:22.388 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.388 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:22.389 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460377+0000", 2024-09-21T15:49:22.389 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.389 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.389 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.389 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:22.389 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460377+0000", 2024-09-21T15:49:22.389 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.389 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.389 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.389 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:22.389 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460277+0000", 2024-09-21T15:49:22.389 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998994 2024-09-21T15:49:22.389 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.390 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.390 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:22.390 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460380+0000", 2024-09-21T15:49:22.390 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010330700000000001 2024-09-21T15:49:22.390 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.390 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.390 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:22.390 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460387+0000", 2024-09-21T15:49:22.390 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8770000000000004e-06 2024-09-21T15:49:22.390 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.390 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.390 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:22.391 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460394+0000", 2024-09-21T15:49:22.391 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.489e-06 2024-09-21T15:49:22.391 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.391 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.391 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:22.391 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500097+0000", 2024-09-21T15:49:22.391 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039702863999999997 2024-09-21T15:49:22.391 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.391 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.391 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:22.391 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500139+0000", 2024-09-21T15:49:22.391 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2262999999999998e-05 2024-09-21T15:49:22.392 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.392 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.392 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:22.392 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500189+0000", 2024-09-21T15:49:22.392 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0377999999999998e-05 2024-09-21T15:49:22.392 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.392 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:22.392 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.392 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.392 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.392 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:5059 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 221184~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:22.392 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.460637+0000", 2024-09-21T15:49:22.392 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.8615538110000003, 2024-09-21T15:49:22.393 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039661409000000002, 2024-09-21T15:49:22.393 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:22.393 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:22.393 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:22.393 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:22.393 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:22.393 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 5059 2024-09-21T15:49:22.393 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.393 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:22.393 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.393 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:22.393 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460637+0000", 2024-09-21T15:49:22.393 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.394 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.394 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.394 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:22.394 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460637+0000", 2024-09-21T15:49:22.394 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.394 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.394 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.394 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:22.394 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460400+0000", 2024-09-21T15:49:22.394 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999763 2024-09-21T15:49:22.394 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.395 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.395 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:22.395 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460639+0000", 2024-09-21T15:49:22.395 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00023958900000000001 2024-09-21T15:49:22.395 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.395 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.395 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:22.395 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460645+0000", 2024-09-21T15:49:22.395 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.609e-06 2024-09-21T15:49:22.395 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.395 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.395 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:22.395 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460649+0000", 2024-09-21T15:49:22.396 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0239999999999999e-06 2024-09-21T15:49:22.396 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.396 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.396 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:22.396 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500211+0000", 2024-09-21T15:49:22.396 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039562011000000001 2024-09-21T15:49:22.396 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.396 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.396 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:22.396 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500251+0000", 2024-09-21T15:49:22.396 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0368999999999999e-05 2024-09-21T15:49:22.396 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.396 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.397 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:22.397 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500298+0000", 2024-09-21T15:49:22.397 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6826000000000002e-05 2024-09-21T15:49:22.397 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.397 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:22.397 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.397 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.397 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.397 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:5060 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 270336~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:22.397 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.460780+0000", 2024-09-21T15:49:22.397 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.861410834, 2024-09-21T15:49:22.397 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039627057, 2024-09-21T15:49:22.397 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:22.398 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:22.398 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:22.398 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:22.398 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:22.398 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 5060 2024-09-21T15:49:22.398 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.398 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:22.398 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.398 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:22.398 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460780+0000", 2024-09-21T15:49:22.398 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.398 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.398 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.399 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:22.399 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460780+0000", 2024-09-21T15:49:22.399 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.399 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.399 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.399 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:22.399 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460655+0000", 2024-09-21T15:49:22.399 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998755 2024-09-21T15:49:22.399 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.399 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.399 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:22.399 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460782+0000", 2024-09-21T15:49:22.399 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012715299999999999 2024-09-21T15:49:22.400 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.400 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.400 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:22.400 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460789+0000", 2024-09-21T15:49:22.400 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6959999999999999e-06 2024-09-21T15:49:22.400 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.400 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.400 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:22.400 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460794+0000", 2024-09-21T15:49:22.400 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1900000000000003e-06 2024-09-21T15:49:22.400 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.400 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.400 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:22.400 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500322+0000", 2024-09-21T15:49:22.401 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039527481000000003 2024-09-21T15:49:22.401 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.401 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.401 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:22.401 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500360+0000", 2024-09-21T15:49:22.401 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8674999999999998e-05 2024-09-21T15:49:22.401 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.401 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.401 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:22.401 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500407+0000", 2024-09-21T15:49:22.401 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6316999999999997e-05 2024-09-21T15:49:22.401 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.401 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:22.402 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.402 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.402 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.402 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:5061 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 319488~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:22.402 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.461015+0000", 2024-09-21T15:49:22.402 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.8611756220000002, 2024-09-21T15:49:22.402 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039499536000000002, 2024-09-21T15:49:22.402 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:22.402 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:22.402 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:22.402 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:22.402 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:22.403 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 5061 2024-09-21T15:49:22.403 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.403 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:22.403 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.403 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:22.403 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461015+0000", 2024-09-21T15:49:22.403 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.403 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.403 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.403 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:22.403 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461015+0000", 2024-09-21T15:49:22.403 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.403 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.404 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.404 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:22.404 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460799+0000", 2024-09-21T15:49:22.404 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9997845 2024-09-21T15:49:22.404 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.404 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.404 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:22.404 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461018+0000", 2024-09-21T15:49:22.404 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00021889000000000001 2024-09-21T15:49:22.404 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.404 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.404 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:22.404 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461026+0000", 2024-09-21T15:49:22.405 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1380000000000007e-06 2024-09-21T15:49:22.405 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.405 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.405 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:22.405 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461032+0000", 2024-09-21T15:49:22.405 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8799999999999996e-06 2024-09-21T15:49:22.405 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.405 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.405 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:22.405 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500429+0000", 2024-09-21T15:49:22.405 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039396704999999997 2024-09-21T15:49:22.405 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.406 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.406 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:22.406 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500467+0000", 2024-09-21T15:49:22.406 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7938999999999997e-05 2024-09-21T15:49:22.406 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.406 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.406 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:22.406 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500514+0000", 2024-09-21T15:49:22.406 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7562000000000003e-05 2024-09-21T15:49:22.406 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.406 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:22.406 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.406 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.407 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.407 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:5062 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 368640~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:22.407 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.461158+0000", 2024-09-21T15:49:22.407 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.8610322320000003, 2024-09-21T15:49:22.407 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039463221, 2024-09-21T15:49:22.407 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:22.407 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:22.407 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:22.407 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:22.407 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:22.407 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 5062 2024-09-21T15:49:22.407 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.408 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:22.408 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.408 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:22.408 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461158+0000", 2024-09-21T15:49:22.408 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.408 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.408 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.408 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:22.408 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461158+0000", 2024-09-21T15:49:22.408 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.408 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.408 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.408 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:22.409 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461037+0000", 2024-09-21T15:49:22.409 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998789 2024-09-21T15:49:22.409 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.409 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.409 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:22.409 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461162+0000", 2024-09-21T15:49:22.409 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000124461 2024-09-21T15:49:22.409 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.409 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.409 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:22.409 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461170+0000", 2024-09-21T15:49:22.409 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6789999999999994e-06 2024-09-21T15:49:22.409 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.410 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.410 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:22.410 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461176+0000", 2024-09-21T15:49:22.410 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0100000000000001e-06 2024-09-21T15:49:22.410 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.410 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.410 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:22.410 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500537+0000", 2024-09-21T15:49:22.410 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039360366000000001 2024-09-21T15:49:22.410 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.410 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.410 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:22.411 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500575+0000", 2024-09-21T15:49:22.411 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8126000000000001e-05 2024-09-21T15:49:22.411 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.411 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.411 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:22.411 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500621+0000", 2024-09-21T15:49:22.411 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6697999999999999e-05 2024-09-21T15:49:22.411 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.411 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:22.411 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.411 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.411 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.412 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:5063 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 417792~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:22.412 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.461421+0000", 2024-09-21T15:49:22.412 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.8607695370000004, 2024-09-21T15:49:22.412 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039308520999999999, 2024-09-21T15:49:22.412 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:22.412 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:22.412 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:22.412 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:22.412 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:22.412 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 5063 2024-09-21T15:49:22.412 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.412 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:22.413 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.413 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:22.413 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461421+0000", 2024-09-21T15:49:22.413 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.413 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.413 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.413 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:22.413 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461421+0000", 2024-09-21T15:49:22.413 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.413 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.413 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.413 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:22.413 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461181+0000", 2024-09-21T15:49:22.414 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9997602 2024-09-21T15:49:22.414 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.414 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.414 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:22.414 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461426+0000", 2024-09-21T15:49:22.414 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00024468599999999998 2024-09-21T15:49:22.414 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.414 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.414 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:22.414 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461438+0000", 2024-09-21T15:49:22.414 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2527000000000001e-05 2024-09-21T15:49:22.414 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.415 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.415 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:22.415 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461446+0000", 2024-09-21T15:49:22.415 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.199e-06 2024-09-21T15:49:22.415 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.415 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.415 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:22.415 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500645+0000", 2024-09-21T15:49:22.415 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039199680000000001 2024-09-21T15:49:22.415 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.415 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.415 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:22.415 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500684+0000", 2024-09-21T15:49:22.416 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9103999999999997e-05 2024-09-21T15:49:22.416 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.416 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.416 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:22.416 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500729+0000", 2024-09-21T15:49:22.416 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4941999999999998e-05 2024-09-21T15:49:22.416 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.416 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:22.416 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.416 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.416 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.416 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:5064 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 466944~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:22.417 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.461542+0000", 2024-09-21T15:49:22.417 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.8606480300000001, 2024-09-21T15:49:22.417 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.03938125, 2024-09-21T15:49:22.417 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:22.417 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:22.417 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:22.417 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:22.417 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:22.417 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 5064 2024-09-21T15:49:22.417 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.417 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:22.417 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:22.418 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461542+0000", 2024-09-21T15:49:22.418 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.418 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:22.418 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461542+0000", 2024-09-21T15:49:22.418 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.418 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:22.418 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461457+0000", 2024-09-21T15:49:22.419 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999146 2024-09-21T15:49:22.419 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.419 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.419 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:22.419 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461546+0000", 2024-09-21T15:49:22.419 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8718e-05 2024-09-21T15:49:22.419 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.419 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.419 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:22.419 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461555+0000", 2024-09-21T15:49:22.419 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8370000000000003e-06 2024-09-21T15:49:22.420 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.420 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.420 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:22.420 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461561+0000", 2024-09-21T15:49:22.420 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4150000000000004e-06 2024-09-21T15:49:22.420 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.420 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.420 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:22.420 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500753+0000", 2024-09-21T15:49:22.420 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039192183999999998 2024-09-21T15:49:22.420 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.420 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.420 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:22.421 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500826+0000", 2024-09-21T15:49:22.421 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2428000000000005e-05 2024-09-21T15:49:22.421 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.421 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.421 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:22.421 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500924+0000", 2024-09-21T15:49:22.421 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7848000000000002e-05 2024-09-21T15:49:22.421 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.421 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:22.421 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.421 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.421 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:22.422 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:22.423 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T15:49:22.423 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:22.422+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:22.523 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:22.522+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:22.565 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:22.566 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:22.566 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:49:22.566 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:22.574 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T15:49:22.624 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:22.623+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:22.676 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:22.676 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:22.676 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:49:22.677 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:49:22.677 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:22.681 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T15:49:22.681 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T15:49:22.681 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-09-21T15:49:22.681 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T15:49:22.681 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T15:49:22.682 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3 2024-09-21T15:49:22.684 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T15:49:22.729 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:22.728+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:22.795 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:22.795 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:49:22.795 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:49:22.795 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:49:22.795 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.796 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:2083 3.a 3:55b68c50:::rbd_data.11335fa2fe69.0000000000000191:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,truncate 77824] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:22.796 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.141682+0000", 2024-09-21T15:49:22.796 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 8.6511809799999995, 2024-09-21T15:49:22.796 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0053016579999999999, 2024-09-21T15:49:22.796 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:22.796 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:22.796 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:22.796 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:22.796 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:22.796 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2083 2024-09-21T15:49:22.796 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.796 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:22.797 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.797 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:22.797 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.141682+0000", 2024-09-21T15:49:22.797 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.797 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.797 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.797 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:22.797 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.141682+0000", 2024-09-21T15:49:22.797 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.797 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.797 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.797 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:22.798 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.141679+0000", 2024-09-21T15:49:22.798 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999976 2024-09-21T15:49:22.798 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.798 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.798 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:22.798 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.141691+0000", 2024-09-21T15:49:22.798 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1922e-05 2024-09-21T15:49:22.798 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.798 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.798 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:22.798 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.141712+0000", 2024-09-21T15:49:22.799 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0115e-05 2024-09-21T15:49:22.799 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.799 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.799 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:22.799 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.141723+0000", 2024-09-21T15:49:22.799 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1071e-05 2024-09-21T15:49:22.799 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.799 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.799 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:22.799 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.141776+0000", 2024-09-21T15:49:22.799 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3497e-05 2024-09-21T15:49:22.800 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.800 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.800 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:22.800 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.141918+0000", 2024-09-21T15:49:22.800 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000141433 2024-09-21T15:49:22.800 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.800 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.800 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-21T15:49:22.800 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.142129+0000", 2024-09-21T15:49:22.800 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00021179700000000001 2024-09-21T15:49:22.800 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.800 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.801 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:22.801 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.144018+0000", 2024-09-21T15:49:22.801 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0018886949999999999 2024-09-21T15:49:22.801 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.801 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.801 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:22.801 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.146889+0000", 2024-09-21T15:49:22.801 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002871017 2024-09-21T15:49:22.801 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.801 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.801 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:22.802 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.146947+0000", 2024-09-21T15:49:22.802 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7558e-05 2024-09-21T15:49:22.802 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.802 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.802 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:22.802 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.146984+0000", 2024-09-21T15:49:22.802 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7095999999999999e-05 2024-09-21T15:49:22.802 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.802 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:22.802 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.802 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.803 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.803 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:13260 3.10 3:0d698a3f:::rbd_data.1133dc5adfdf.0000000000000096:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:22.803 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.086102+0000", 2024-09-21T15:49:22.803 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.7067611290000002, 2024-09-21T15:49:22.803 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0050922229999999999, 2024-09-21T15:49:22.803 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:22.803 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:22.803 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:22.803 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:22.803 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:22.804 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 13260 2024-09-21T15:49:22.804 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.804 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:22.804 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.804 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:22.804 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.086102+0000", 2024-09-21T15:49:22.804 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.804 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.804 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.804 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:22.804 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.086102+0000", 2024-09-21T15:49:22.805 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.805 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.805 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.805 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:22.805 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.086101+0000", 2024-09-21T15:49:22.805 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:22.805 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.805 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.805 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:22.805 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.086104+0000", 2024-09-21T15:49:22.805 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1980000000000001e-06 2024-09-21T15:49:22.805 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.805 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.806 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:22.806 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.086112+0000", 2024-09-21T15:49:22.806 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5290000000000003e-06 2024-09-21T15:49:22.806 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.806 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.806 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:22.806 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.086120+0000", 2024-09-21T15:49:22.806 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9489999999999996e-06 2024-09-21T15:49:22.806 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.806 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.806 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:22.806 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.086526+0000", 2024-09-21T15:49:22.807 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000406103 2024-09-21T15:49:22.807 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.807 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.807 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:22.807 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.086596+0000", 2024-09-21T15:49:22.807 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9675e-05 2024-09-21T15:49:22.807 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.807 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.807 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0", 2024-09-21T15:49:22.807 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.086688+0000", 2024-09-21T15:49:22.807 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2576000000000003e-05 2024-09-21T15:49:22.807 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.808 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.808 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:22.808 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.089471+0000", 2024-09-21T15:49:22.808 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002782423 2024-09-21T15:49:22.808 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.808 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.808 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:22.808 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.091148+0000", 2024-09-21T15:49:22.808 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0016771379999999999 2024-09-21T15:49:22.808 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.808 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.808 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:22.809 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.091173+0000", 2024-09-21T15:49:22.809 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5347e-05 2024-09-21T15:49:22.809 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.809 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.809 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:22.809 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.091194+0000", 2024-09-21T15:49:22.809 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0888999999999999e-05 2024-09-21T15:49:22.809 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.809 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:22.809 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.810 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.810 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.810 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:13274 3.10 3:0b1f2304:::rbd_data.1133dc5adfdf.00000000000000b9:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:22.810 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.090744+0000", 2024-09-21T15:49:22.810 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.7021193190000004, 2024-09-21T15:49:22.810 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0082153750000000005, 2024-09-21T15:49:22.810 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:22.810 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:22.810 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:22.810 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:22.810 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:22.810 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 13274 2024-09-21T15:49:22.811 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.811 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:22.811 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.811 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:22.811 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.090744+0000", 2024-09-21T15:49:22.811 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.811 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.811 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.811 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:22.811 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.090744+0000", 2024-09-21T15:49:22.811 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.812 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.812 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.812 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:22.812 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.090743+0000", 2024-09-21T15:49:22.812 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-21T15:49:22.812 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.812 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.812 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:22.812 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.090747+0000", 2024-09-21T15:49:22.812 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.262e-06 2024-09-21T15:49:22.812 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.813 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.813 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:22.813 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.090758+0000", 2024-09-21T15:49:22.813 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0587999999999999e-05 2024-09-21T15:49:22.813 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.813 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.813 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:22.813 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.090762+0000", 2024-09-21T15:49:22.813 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7779999999999998e-06 2024-09-21T15:49:22.813 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.813 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.813 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:22.814 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.090799+0000", 2024-09-21T15:49:22.814 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6940999999999997e-05 2024-09-21T15:49:22.814 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.814 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.814 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:22.814 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.090847+0000", 2024-09-21T15:49:22.814 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7975000000000001e-05 2024-09-21T15:49:22.814 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.814 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.814 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0", 2024-09-21T15:49:22.815 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.090937+0000", 2024-09-21T15:49:22.815 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0148000000000005e-05 2024-09-21T15:49:22.815 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.815 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.815 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:22.815 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.094607+0000", 2024-09-21T15:49:22.815 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0036694029999999999 2024-09-21T15:49:22.815 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.815 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.815 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:22.815 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.098877+0000", 2024-09-21T15:49:22.815 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0042697239999999999 2024-09-21T15:49:22.816 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.816 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.816 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:22.816 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.098920+0000", 2024-09-21T15:49:22.816 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3547999999999997e-05 2024-09-21T15:49:22.816 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.816 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.816 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:22.816 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.098959+0000", 2024-09-21T15:49:22.816 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8890000000000002e-05 2024-09-21T15:49:22.816 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.816 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:22.817 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.817 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.817 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.817 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:13305 3.a 3:53172455:::rbd_data.1133dc5adfdf.00000000000000cd:head [set-alloc-hint object_size 1048576 write_size 1048576,write 540672~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:22.817 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.126122+0000", 2024-09-21T15:49:22.817 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.6667406089999997, 2024-09-21T15:49:22.817 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0051376010000000003, 2024-09-21T15:49:22.817 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:22.817 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:22.817 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:22.818 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:22.818 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:22.818 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 13305 2024-09-21T15:49:22.818 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.818 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:22.818 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.818 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:22.818 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.126122+0000", 2024-09-21T15:49:22.818 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.818 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.818 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.819 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:22.819 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.126122+0000", 2024-09-21T15:49:22.819 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.819 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.819 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.819 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:22.819 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.126121+0000", 2024-09-21T15:49:22.819 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:22.819 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.819 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.820 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:22.820 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.126131+0000", 2024-09-21T15:49:22.820 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4870000000000008e-06 2024-09-21T15:49:22.820 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.820 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.820 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:22.820 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.126142+0000", 2024-09-21T15:49:22.820 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1123e-05 2024-09-21T15:49:22.820 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.820 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.821 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:22.821 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.126148+0000", 2024-09-21T15:49:22.821 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4250000000000003e-06 2024-09-21T15:49:22.821 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.821 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.821 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:22.821 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.126190+0000", 2024-09-21T15:49:22.821 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2063e-05 2024-09-21T15:49:22.821 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.821 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.821 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:22.822 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.126272+0000", 2024-09-21T15:49:22.822 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1569000000000005e-05 2024-09-21T15:49:22.822 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.822 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.822 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-21T15:49:22.822 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.126356+0000", 2024-09-21T15:49:22.822 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3916000000000001e-05 2024-09-21T15:49:22.822 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.822 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.822 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:22.822 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.128614+0000", 2024-09-21T15:49:22.822 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0022582129999999998 2024-09-21T15:49:22.823 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.823 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.823 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:22.823 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.131210+0000", 2024-09-21T15:49:22.823 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0025964069999999998 2024-09-21T15:49:22.823 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.823 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.823 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:22.823 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.131239+0000", 2024-09-21T15:49:22.823 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8796999999999999e-05 2024-09-21T15:49:22.823 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.824 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.824 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:22.824 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.131260+0000", 2024-09-21T15:49:22.824 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0928999999999999e-05 2024-09-21T15:49:22.824 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.824 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:22.824 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.824 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.824 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.824 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4403.0:13426 3.1e e31/22 3:7bf656a2:::rbd_data.1133dc5adfdf.000000000000003b:head v 31'56, mlcod=31'54)", 2024-09-21T15:49:22.825 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.209924+0000", 2024-09-21T15:49:22.825 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.5829393080000003, 2024-09-21T15:49:22.825 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0043380529999999997, 2024-09-21T15:49:22.825 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:22.825 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:22.825 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:22.825 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.825 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:22.825 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.209924+0000", 2024-09-21T15:49:22.825 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.825 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.825 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.826 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:22.826 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.209924+0000", 2024-09-21T15:49:22.826 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.826 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.826 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.826 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:22.826 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.209923+0000", 2024-09-21T15:49:22.826 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:22.826 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.826 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.826 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:22.827 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.209930+0000", 2024-09-21T15:49:22.827 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.43e-06 2024-09-21T15:49:22.827 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.827 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.827 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:22.827 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.209940+0000", 2024-09-21T15:49:22.827 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0869999999999996e-06 2024-09-21T15:49:22.827 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.827 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.827 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:22.827 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.209946+0000", 2024-09-21T15:49:22.828 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9569999999999999e-06 2024-09-21T15:49:22.828 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.828 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.828 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:22.828 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.209974+0000", 2024-09-21T15:49:22.828 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8005999999999999e-05 2024-09-21T15:49:22.828 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.828 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.828 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:22.828 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.209987+0000", 2024-09-21T15:49:22.829 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2984e-05 2024-09-21T15:49:22.829 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.829 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.829 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:22.829 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.214182+0000", 2024-09-21T15:49:22.829 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0041956839999999999 2024-09-21T15:49:22.829 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.829 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.829 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:22.829 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.214262+0000", 2024-09-21T15:49:22.829 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9521e-05 2024-09-21T15:49:22.830 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.830 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:22.830 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.830 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.830 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.830 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:13550 3.a 3:5510f58f:::rbd_data.1133dc5adfdf.0000000000000080:head [set-alloc-hint object_size 1048576 write_size 1048576,write 942080~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:22.830 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.319272+0000", 2024-09-21T15:49:22.830 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.4735911579999996, 2024-09-21T15:49:22.830 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0068300210000000004, 2024-09-21T15:49:22.830 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:22.831 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:22.831 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:22.831 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:22.831 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:22.831 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 13550 2024-09-21T15:49:22.831 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.831 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:22.831 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.831 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:22.831 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.319272+0000", 2024-09-21T15:49:22.831 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.831 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.831 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.832 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:22.832 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.319272+0000", 2024-09-21T15:49:22.832 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.832 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.832 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.832 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:22.832 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.319270+0000", 2024-09-21T15:49:22.832 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:22.832 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.832 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.832 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:22.832 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.319282+0000", 2024-09-21T15:49:22.832 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2143999999999999e-05 2024-09-21T15:49:22.833 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.833 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.833 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:22.833 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.319297+0000", 2024-09-21T15:49:22.833 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5017999999999999e-05 2024-09-21T15:49:22.833 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.833 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.833 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:22.833 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.319303+0000", 2024-09-21T15:49:22.833 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8390000000000003e-06 2024-09-21T15:49:22.833 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.833 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.834 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:22.834 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.319355+0000", 2024-09-21T15:49:22.834 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1777999999999998e-05 2024-09-21T15:49:22.834 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.834 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.834 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:22.834 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.319441+0000", 2024-09-21T15:49:22.834 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.585e-05 2024-09-21T15:49:22.834 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.834 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.834 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-21T15:49:22.834 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.319585+0000", 2024-09-21T15:49:22.835 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000144003 2024-09-21T15:49:22.835 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.835 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.835 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:22.835 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.325771+0000", 2024-09-21T15:49:22.835 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0061861959999999997 2024-09-21T15:49:22.835 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.835 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.835 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:22.835 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.326013+0000", 2024-09-21T15:49:22.835 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00024232600000000001 2024-09-21T15:49:22.835 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.836 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.836 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:22.836 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.326063+0000", 2024-09-21T15:49:22.836 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9669000000000002e-05 2024-09-21T15:49:22.836 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.836 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.836 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:22.836 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.326102+0000", 2024-09-21T15:49:22.836 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8825999999999997e-05 2024-09-21T15:49:22.836 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.836 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:22.837 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.837 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.837 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.837 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:13581 3.a 3:52d3dd80:::rbd_data.1133dc5adfdf.00000000000000b0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:22.837 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.347049+0000", 2024-09-21T15:49:22.837 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.4458135380000003, 2024-09-21T15:49:22.837 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0051259269999999997, 2024-09-21T15:49:22.837 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:22.837 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:22.837 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:22.837 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:22.837 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:22.837 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 13581 2024-09-21T15:49:22.838 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.838 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:22.838 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.838 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:22.838 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.347049+0000", 2024-09-21T15:49:22.838 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.838 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.838 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.838 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:22.838 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.347049+0000", 2024-09-21T15:49:22.838 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.838 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.839 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.839 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:22.839 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.347049+0000", 2024-09-21T15:49:22.839 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-21T15:49:22.839 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.839 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.839 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:22.839 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.347053+0000", 2024-09-21T15:49:22.839 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5229999999999997e-06 2024-09-21T15:49:22.839 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.839 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.839 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:22.840 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.347064+0000", 2024-09-21T15:49:22.840 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0336999999999999e-05 2024-09-21T15:49:22.840 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.840 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.840 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:22.840 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.347071+0000", 2024-09-21T15:49:22.840 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1629999999999999e-06 2024-09-21T15:49:22.840 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.840 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.840 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:22.840 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.347102+0000", 2024-09-21T15:49:22.840 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1647999999999999e-05 2024-09-21T15:49:22.840 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.841 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.841 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:22.841 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.347155+0000", 2024-09-21T15:49:22.841 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3093000000000001e-05 2024-09-21T15:49:22.841 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.841 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.841 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-21T15:49:22.841 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.347221+0000", 2024-09-21T15:49:22.841 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5922999999999999e-05 2024-09-21T15:49:22.841 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.841 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.841 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:22.842 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.349565+0000", 2024-09-21T15:49:22.842 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0023435909999999999 2024-09-21T15:49:22.842 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.842 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.842 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:22.842 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.352099+0000", 2024-09-21T15:49:22.842 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0025342089999999999 2024-09-21T15:49:22.842 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.842 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.842 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:22.842 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.352141+0000", 2024-09-21T15:49:22.842 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2259000000000003e-05 2024-09-21T15:49:22.843 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.843 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.843 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:22.843 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.352175+0000", 2024-09-21T15:49:22.843 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3973000000000002e-05 2024-09-21T15:49:22.843 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.843 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:22.843 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.843 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.843 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.843 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:13833 3.10 3:0b29d3b5:::rbd_data.1133dc5adfdf.0000000000000091:head [write 897024~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:22.843 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.527148+0000", 2024-09-21T15:49:22.844 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.2657149490000004, 2024-09-21T15:49:22.844 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0044907690000000004, 2024-09-21T15:49:22.844 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:22.844 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:22.844 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:22.844 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:22.844 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:22.844 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 13833 2024-09-21T15:49:22.844 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.844 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:22.844 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.844 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:22.845 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.527148+0000", 2024-09-21T15:49:22.845 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.845 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.845 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.845 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:22.845 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.527148+0000", 2024-09-21T15:49:22.845 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.845 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.845 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.845 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:22.845 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.527147+0000", 2024-09-21T15:49:22.846 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:22.846 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.846 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.846 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:22.846 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.527157+0000", 2024-09-21T15:49:22.846 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7759999999999992e-06 2024-09-21T15:49:22.846 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.846 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.846 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:22.846 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.527167+0000", 2024-09-21T15:49:22.846 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0818999999999999e-05 2024-09-21T15:49:22.847 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.847 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.847 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:22.847 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.527173+0000", 2024-09-21T15:49:22.847 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.152e-06 2024-09-21T15:49:22.847 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.847 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.847 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:22.847 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.527208+0000", 2024-09-21T15:49:22.847 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5833000000000001e-05 2024-09-21T15:49:22.847 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.847 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.847 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:22.848 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.527304+0000", 2024-09-21T15:49:22.848 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5292999999999997e-05 2024-09-21T15:49:22.848 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.848 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.848 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0", 2024-09-21T15:49:22.848 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.527459+0000", 2024-09-21T15:49:22.848 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015507499999999999 2024-09-21T15:49:22.848 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.848 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.848 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:22.848 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.528200+0000", 2024-09-21T15:49:22.848 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00074178700000000005 2024-09-21T15:49:22.849 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.849 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.849 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:22.849 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.531575+0000", 2024-09-21T15:49:22.849 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0033740580000000001 2024-09-21T15:49:22.849 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.849 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.849 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:22.849 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.531605+0000", 2024-09-21T15:49:22.849 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0321999999999999e-05 2024-09-21T15:49:22.849 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.849 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.850 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:22.850 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.531639+0000", 2024-09-21T15:49:22.850 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3933000000000002e-05 2024-09-21T15:49:22.850 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.850 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:22.850 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.850 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.850 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.850 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:13862 3.a 3:53172455:::rbd_data.1133dc5adfdf.00000000000000cd:head [set-alloc-hint object_size 1048576 write_size 1048576,write 266240~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:22.850 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.549561+0000", 2024-09-21T15:49:22.850 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.2433019930000002, 2024-09-21T15:49:22.850 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0065617749999999997, 2024-09-21T15:49:22.850 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:22.851 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:22.851 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:22.851 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:22.851 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:22.851 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 13862 2024-09-21T15:49:22.851 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.851 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:22.851 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.851 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:22.851 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.549561+0000", 2024-09-21T15:49:22.851 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.852 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.852 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.852 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:22.852 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.549561+0000", 2024-09-21T15:49:22.852 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.852 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.852 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.852 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:22.852 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.549560+0000", 2024-09-21T15:49:22.852 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:22.852 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.853 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.853 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:22.853 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.549569+0000", 2024-09-21T15:49:22.853 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2799999999999992e-06 2024-09-21T15:49:22.853 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.853 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.853 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:22.853 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.549582+0000", 2024-09-21T15:49:22.853 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2757e-05 2024-09-21T15:49:22.853 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.853 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.853 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:22.854 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.549590+0000", 2024-09-21T15:49:22.854 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4619999999999996e-06 2024-09-21T15:49:22.854 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.854 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.854 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:22.854 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.549642+0000", 2024-09-21T15:49:22.854 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1779e-05 2024-09-21T15:49:22.854 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.854 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.854 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:22.854 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.549736+0000", 2024-09-21T15:49:22.854 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4617000000000006e-05 2024-09-21T15:49:22.855 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.855 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.855 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-21T15:49:22.855 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.549886+0000", 2024-09-21T15:49:22.855 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014938199999999999 2024-09-21T15:49:22.855 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.855 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.855 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:22.855 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.553416+0000", 2024-09-21T15:49:22.855 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0035298339999999999 2024-09-21T15:49:22.855 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.855 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.856 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:22.856 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.556032+0000", 2024-09-21T15:49:22.856 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002615994 2024-09-21T15:49:22.856 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.856 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.856 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:22.856 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.556080+0000", 2024-09-21T15:49:22.856 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8442999999999997e-05 2024-09-21T15:49:22.856 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.856 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.856 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:22.856 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.556123+0000", 2024-09-21T15:49:22.856 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2673999999999998e-05 2024-09-21T15:49:22.857 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.857 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:22.857 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.857 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.857 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.857 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:13948 3.a 3:52b31f3c:::rbd_data.1133dc5adfdf.00000000000000db:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:22.857 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.616162+0000", 2024-09-21T15:49:22.857 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.1767006779999996, 2024-09-21T15:49:22.857 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.004904594, 2024-09-21T15:49:22.857 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:22.857 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:22.857 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:22.858 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:22.858 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:22.858 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 13948 2024-09-21T15:49:22.858 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.858 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:22.858 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.858 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:22.858 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.616162+0000", 2024-09-21T15:49:22.858 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.858 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.858 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.858 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:22.858 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.616162+0000", 2024-09-21T15:49:22.859 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.859 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.859 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.859 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:22.859 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.616160+0000", 2024-09-21T15:49:22.859 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999981 2024-09-21T15:49:22.859 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.859 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.859 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:22.859 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.616168+0000", 2024-09-21T15:49:22.859 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3030000000000002e-06 2024-09-21T15:49:22.859 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.859 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.860 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:22.860 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.616183+0000", 2024-09-21T15:49:22.860 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5756e-05 2024-09-21T15:49:22.860 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.860 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.860 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:22.860 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.616191+0000", 2024-09-21T15:49:22.860 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6090000000000003e-06 2024-09-21T15:49:22.860 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.860 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.860 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:22.860 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.616249+0000", 2024-09-21T15:49:22.861 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8286999999999999e-05 2024-09-21T15:49:22.861 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.861 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.861 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:22.861 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.616333+0000", 2024-09-21T15:49:22.861 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3441000000000003e-05 2024-09-21T15:49:22.861 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.861 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.861 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-21T15:49:22.861 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.616441+0000", 2024-09-21T15:49:22.861 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000108117 2024-09-21T15:49:22.861 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.861 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.862 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:22.862 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.617968+0000", 2024-09-21T15:49:22.862 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001526662 2024-09-21T15:49:22.862 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.862 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.862 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:22.862 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.620963+0000", 2024-09-21T15:49:22.862 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0029951380000000001 2024-09-21T15:49:22.862 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.862 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.862 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:22.862 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.621008+0000", 2024-09-21T15:49:22.863 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5664000000000002e-05 2024-09-21T15:49:22.863 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.863 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.863 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:22.863 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.621067+0000", 2024-09-21T15:49:22.863 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8548000000000003e-05 2024-09-21T15:49:22.863 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.863 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:22.863 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.863 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.863 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.863 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:14003 3.10 3:0b1f2304:::rbd_data.1133dc5adfdf.00000000000000b9:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:22.864 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.658416+0000", 2024-09-21T15:49:22.864 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.1344466070000001, 2024-09-21T15:49:22.864 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0045796250000000004, 2024-09-21T15:49:22.864 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:22.864 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:22.864 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:22.864 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:22.864 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:22.864 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 14003 2024-09-21T15:49:22.864 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.864 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:22.864 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.865 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:22.865 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.658416+0000", 2024-09-21T15:49:22.865 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.865 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.865 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.865 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:22.865 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.658416+0000", 2024-09-21T15:49:22.865 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.865 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.865 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.865 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:22.865 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.658415+0000", 2024-09-21T15:49:22.866 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-21T15:49:22.866 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.866 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.866 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:22.866 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.658422+0000", 2024-09-21T15:49:22.866 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3910000000000003e-06 2024-09-21T15:49:22.866 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.866 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.866 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:22.866 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.658433+0000", 2024-09-21T15:49:22.866 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1435e-05 2024-09-21T15:49:22.866 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.867 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.867 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:22.867 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.658443+0000", 2024-09-21T15:49:22.867 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0012e-05 2024-09-21T15:49:22.867 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.867 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.867 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:22.867 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.658485+0000", 2024-09-21T15:49:22.867 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2079000000000001e-05 2024-09-21T15:49:22.867 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.867 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.867 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:22.867 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.658550+0000", 2024-09-21T15:49:22.868 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5028000000000001e-05 2024-09-21T15:49:22.868 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.868 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.868 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0", 2024-09-21T15:49:22.868 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.658622+0000", 2024-09-21T15:49:22.868 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1381e-05 2024-09-21T15:49:22.868 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.868 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.868 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:22.868 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.660779+0000", 2024-09-21T15:49:22.868 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0021570270000000002 2024-09-21T15:49:22.868 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.868 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.869 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:22.869 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.662939+0000", 2024-09-21T15:49:22.869 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0021602119999999999 2024-09-21T15:49:22.869 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.869 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.869 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:22.869 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.662968+0000", 2024-09-21T15:49:22.869 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9309999999999999e-05 2024-09-21T15:49:22.869 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.869 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.869 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:22.869 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.662996+0000", 2024-09-21T15:49:22.869 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.792e-05 2024-09-21T15:49:22.870 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.870 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:22.870 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.870 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.870 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.870 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:14017 3.a 3:53172455:::rbd_data.1133dc5adfdf.00000000000000cd:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-21T15:49:22.870 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.663573+0000", 2024-09-21T15:49:22.870 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.129289848, 2024-09-21T15:49:22.870 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.014018708, 2024-09-21T15:49:22.870 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:22.870 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:22.870 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:22.871 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:22.871 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:22.871 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 14017 2024-09-21T15:49:22.871 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.871 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:22.871 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.871 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:22.871 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.663573+0000", 2024-09-21T15:49:22.871 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.871 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.871 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.871 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:22.872 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.663573+0000", 2024-09-21T15:49:22.872 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.872 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.872 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.872 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:22.872 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.663572+0000", 2024-09-21T15:49:22.872 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:22.872 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.872 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.872 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:22.872 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.663579+0000", 2024-09-21T15:49:22.872 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4340000000000003e-06 2024-09-21T15:49:22.872 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.873 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.873 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:22.873 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.663594+0000", 2024-09-21T15:49:22.873 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4814000000000001e-05 2024-09-21T15:49:22.873 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.873 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.873 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:22.873 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.663602+0000", 2024-09-21T15:49:22.873 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7030000000000006e-06 2024-09-21T15:49:22.873 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.873 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.873 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:22.874 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.663652+0000", 2024-09-21T15:49:22.874 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0866999999999999e-05 2024-09-21T15:49:22.874 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.874 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.874 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-21T15:49:22.874 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.663659+0000", 2024-09-21T15:49:22.874 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7909999999999999e-06 2024-09-21T15:49:22.874 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.874 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.874 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:22.874 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.672664+0000", 2024-09-21T15:49:22.874 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0090051469999999998 2024-09-21T15:49:22.875 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.875 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.875 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:22.875 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.672740+0000", 2024-09-21T15:49:22.875 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5777000000000003e-05 2024-09-21T15:49:22.875 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.875 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.875 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-21T15:49:22.875 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.672840+0000", 2024-09-21T15:49:22.875 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000100108 2024-09-21T15:49:22.875 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.876 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.876 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:22.876 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.674807+0000", 2024-09-21T15:49:22.876 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0019661930000000002 2024-09-21T15:49:22.876 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.876 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.876 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:22.876 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.677520+0000", 2024-09-21T15:49:22.876 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0027133270000000002 2024-09-21T15:49:22.876 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.876 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.876 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:22.876 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.677567+0000", 2024-09-21T15:49:22.877 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7307e-05 2024-09-21T15:49:22.877 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.877 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.877 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:22.877 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.677592+0000", 2024-09-21T15:49:22.877 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4711000000000001e-05 2024-09-21T15:49:22.877 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.877 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:22.877 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.877 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.877 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.878 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:14245 3.a 3:5510f58f:::rbd_data.1133dc5adfdf.0000000000000080:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-21T15:49:22.878 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:18.831104+0000", 2024-09-21T15:49:22.878 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 3.9617593319999997, 2024-09-21T15:49:22.878 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0047728409999999999, 2024-09-21T15:49:22.878 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:22.878 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:22.878 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:22.878 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:22.878 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:22.878 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 14245 2024-09-21T15:49:22.878 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.878 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:22.879 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.879 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:22.879 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.831104+0000", 2024-09-21T15:49:22.879 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.879 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.879 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.879 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:22.879 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.831104+0000", 2024-09-21T15:49:22.879 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.879 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.879 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.879 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:22.879 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.831102+0000", 2024-09-21T15:49:22.880 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999981 2024-09-21T15:49:22.880 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.880 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.880 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:22.880 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.831111+0000", 2024-09-21T15:49:22.880 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7899999999999994e-06 2024-09-21T15:49:22.880 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.880 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.880 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:22.880 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.831132+0000", 2024-09-21T15:49:22.880 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0693e-05 2024-09-21T15:49:22.881 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.881 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.881 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:22.881 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.831142+0000", 2024-09-21T15:49:22.881 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5400000000000001e-06 2024-09-21T15:49:22.881 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.881 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.881 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:22.881 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.831197+0000", 2024-09-21T15:49:22.881 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5078999999999999e-05 2024-09-21T15:49:22.881 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.881 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.882 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:22.882 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.831289+0000", 2024-09-21T15:49:22.882 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2614000000000005e-05 2024-09-21T15:49:22.882 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.882 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.882 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-21T15:49:22.882 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.831401+0000", 2024-09-21T15:49:22.882 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000111457 2024-09-21T15:49:22.882 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.882 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.882 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:22.883 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.833162+0000", 2024-09-21T15:49:22.883 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0017607009999999999 2024-09-21T15:49:22.883 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.883 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.883 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:22.883 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.835816+0000", 2024-09-21T15:49:22.883 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0026548510000000002 2024-09-21T15:49:22.883 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.883 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.883 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:22.883 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.835845+0000", 2024-09-21T15:49:22.884 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8538e-05 2024-09-21T15:49:22.884 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.884 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.884 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:22.884 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.835876+0000", 2024-09-21T15:49:22.884 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1565e-05 2024-09-21T15:49:22.884 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.884 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:22.884 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.884 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.884 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.884 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4403.0:14408 3.8 e33/32 3:13437810:::rbd_data.1133dc5adfdf.0000000000000039:head v 33'55, mlcod=33'52)", 2024-09-21T15:49:22.885 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:18.971354+0000", 2024-09-21T15:49:22.885 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 3.821509088, 2024-09-21T15:49:22.885 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0048492420000000001, 2024-09-21T15:49:22.885 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:22.885 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:22.885 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:22.885 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.885 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:22.885 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.971354+0000", 2024-09-21T15:49:22.885 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.885 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.885 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.886 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:22.886 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.971354+0000", 2024-09-21T15:49:22.886 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.886 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.886 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.886 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:22.886 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.971353+0000", 2024-09-21T15:49:22.886 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:22.886 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.886 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.886 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:22.887 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.971359+0000", 2024-09-21T15:49:22.887 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6829999999999996e-06 2024-09-21T15:49:22.887 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.887 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.887 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:22.887 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.971370+0000", 2024-09-21T15:49:22.887 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0776e-05 2024-09-21T15:49:22.887 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.887 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.887 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:22.887 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.971375+0000", 2024-09-21T15:49:22.887 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6670000000000001e-06 2024-09-21T15:49:22.888 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.888 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.888 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:22.888 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.971414+0000", 2024-09-21T15:49:22.888 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8376e-05 2024-09-21T15:49:22.888 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.888 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.888 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:22.888 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.971432+0000", 2024-09-21T15:49:22.888 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8603000000000001e-05 2024-09-21T15:49:22.888 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.888 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.889 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:22.889 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.976169+0000", 2024-09-21T15:49:22.889 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0047363099999999997 2024-09-21T15:49:22.889 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.889 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.889 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:22.889 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.976203+0000", 2024-09-21T15:49:22.889 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4473e-05 2024-09-21T15:49:22.889 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.889 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:22.889 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.889 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.890 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.890 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4403.0:14410 3.1e e33/22 3:7bf656a2:::rbd_data.1133dc5adfdf.000000000000003b:head v 33'95, mlcod=33'92)", 2024-09-21T15:49:22.890 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:18.971584+0000", 2024-09-21T15:49:22.890 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 3.8212789219999999, 2024-09-21T15:49:22.890 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0046296760000000001, 2024-09-21T15:49:22.890 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:22.890 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:22.890 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:22.890 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.890 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:22.890 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.971584+0000", 2024-09-21T15:49:22.891 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.891 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.891 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.891 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:22.891 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.971584+0000", 2024-09-21T15:49:22.891 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.891 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.891 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.891 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:22.891 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.971584+0000", 2024-09-21T15:49:22.891 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:22.891 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.891 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.892 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:22.892 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.971588+0000", 2024-09-21T15:49:22.892 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.34e-06 2024-09-21T15:49:22.892 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.892 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.892 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:22.892 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.971597+0000", 2024-09-21T15:49:22.892 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.85e-06 2024-09-21T15:49:22.892 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.892 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.892 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:22.893 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.971601+0000", 2024-09-21T15:49:22.893 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4780000000000002e-06 2024-09-21T15:49:22.893 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.893 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.893 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:22.893 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.971635+0000", 2024-09-21T15:49:22.893 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4190999999999998e-05 2024-09-21T15:49:22.893 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.893 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.893 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:22.893 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.971656+0000", 2024-09-21T15:49:22.894 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0608e-05 2024-09-21T15:49:22.894 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.894 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.894 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:22.894 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.976184+0000", 2024-09-21T15:49:22.894 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0045279730000000002 2024-09-21T15:49:22.894 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.894 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.894 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:22.894 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.976214+0000", 2024-09-21T15:49:22.894 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9709e-05 2024-09-21T15:49:22.894 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.895 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:22.895 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.895 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.895 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.895 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:14417 3.a 3:5715b5ca:::rbd_data.1133dc5adfdf.000000000000004e:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-21T15:49:22.895 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:18.975481+0000", 2024-09-21T15:49:22.895 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 3.8173823009999999, 2024-09-21T15:49:22.895 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0087110940000000008, 2024-09-21T15:49:22.895 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:22.895 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:22.895 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:22.895 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:22.896 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:22.896 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 14417 2024-09-21T15:49:22.896 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.896 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:22.896 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.896 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:22.896 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.975481+0000", 2024-09-21T15:49:22.896 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.896 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.896 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.896 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:22.896 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.975481+0000", 2024-09-21T15:49:22.896 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.897 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.897 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.897 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:22.897 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.975479+0000", 2024-09-21T15:49:22.897 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:22.897 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.897 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.897 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:22.897 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.975485+0000", 2024-09-21T15:49:22.897 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9540000000000001e-06 2024-09-21T15:49:22.897 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.897 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.898 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:22.898 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.975497+0000", 2024-09-21T15:49:22.898 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1934e-05 2024-09-21T15:49:22.898 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.898 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.898 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:22.898 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.975505+0000", 2024-09-21T15:49:22.898 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0250000000000002e-06 2024-09-21T15:49:22.898 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.898 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.898 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:22.898 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.975552+0000", 2024-09-21T15:49:22.899 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6281e-05 2024-09-21T15:49:22.899 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.899 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.899 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:22.899 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.975638+0000", 2024-09-21T15:49:22.899 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6762999999999996e-05 2024-09-21T15:49:22.899 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.899 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.899 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-21T15:49:22.899 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.975734+0000", 2024-09-21T15:49:22.899 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5248999999999996e-05 2024-09-21T15:49:22.899 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.900 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.900 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:22.900 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.978856+0000", 2024-09-21T15:49:22.900 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0031219580000000002 2024-09-21T15:49:22.900 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.900 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.900 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:22.900 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.984106+0000", 2024-09-21T15:49:22.900 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0052503589999999996 2024-09-21T15:49:22.900 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.900 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.900 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:22.901 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.984156+0000", 2024-09-21T15:49:22.901 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0164000000000003e-05 2024-09-21T15:49:22.901 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.901 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.901 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:22.901 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.984192+0000", 2024-09-21T15:49:22.901 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5658999999999999e-05 2024-09-21T15:49:22.901 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.901 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:22.901 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.901 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.901 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.902 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:14536 3.a 3:56ed26f3:::rbd_data.1133dc5adfdf.00000000000000ad:head [set-alloc-hint object_size 1048576 write_size 1048576,write 876544~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-21T15:49:22.902 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:19.118136+0000", 2024-09-21T15:49:22.902 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 3.6747272940000002, 2024-09-21T15:49:22.902 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.005992224, 2024-09-21T15:49:22.902 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:22.902 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:22.902 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:22.902 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:22.902 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:22.902 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 14536 2024-09-21T15:49:22.902 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.903 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:22.903 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.903 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:22.903 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.118136+0000", 2024-09-21T15:49:22.903 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.903 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.903 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.903 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:22.903 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.118136+0000", 2024-09-21T15:49:22.903 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.903 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.903 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.904 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:22.904 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.118134+0000", 2024-09-21T15:49:22.904 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999981 2024-09-21T15:49:22.904 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.904 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.904 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:22.904 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.118146+0000", 2024-09-21T15:49:22.904 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1565e-05 2024-09-21T15:49:22.904 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.904 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.904 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:22.904 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.118164+0000", 2024-09-21T15:49:22.905 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8850000000000001e-05 2024-09-21T15:49:22.905 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.905 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.905 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:22.905 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.118172+0000", 2024-09-21T15:49:22.905 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5329999999999997e-06 2024-09-21T15:49:22.905 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.905 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.905 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:22.905 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.118222+0000", 2024-09-21T15:49:22.905 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0077999999999997e-05 2024-09-21T15:49:22.906 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.906 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.906 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:22.906 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.118331+0000", 2024-09-21T15:49:22.906 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000109174 2024-09-21T15:49:22.906 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.906 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.906 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-21T15:49:22.906 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.118414+0000", 2024-09-21T15:49:22.906 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2701999999999995e-05 2024-09-21T15:49:22.906 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.907 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.907 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:22.907 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.120970+0000", 2024-09-21T15:49:22.907 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0025564250000000002 2024-09-21T15:49:22.907 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.907 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.907 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:22.907 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.124045+0000", 2024-09-21T15:49:22.907 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0030747309999999998 2024-09-21T15:49:22.907 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.907 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.907 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:22.907 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.124091+0000", 2024-09-21T15:49:22.908 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6276000000000002e-05 2024-09-21T15:49:22.908 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.908 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.908 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:22.908 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.124128+0000", 2024-09-21T15:49:22.908 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6566000000000002e-05 2024-09-21T15:49:22.908 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.908 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:22.908 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.908 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.908 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.909 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:14539 3.a 3:52d3dd80:::rbd_data.1133dc5adfdf.00000000000000b0:head [set-alloc-hint object_size 1048576 write_size 1048576,write 876544~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-21T15:49:22.909 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:19.121064+0000", 2024-09-21T15:49:22.909 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 3.671798769, 2024-09-21T15:49:22.909 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0080418529999999998, 2024-09-21T15:49:22.909 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:22.909 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:22.909 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:22.909 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:22.909 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:22.909 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 14539 2024-09-21T15:49:22.909 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.909 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:22.909 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.910 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:22.910 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.121064+0000", 2024-09-21T15:49:22.910 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.910 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.910 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.910 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:22.910 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.121064+0000", 2024-09-21T15:49:22.910 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.910 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.910 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.910 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:22.910 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.121063+0000", 2024-09-21T15:49:22.911 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-21T15:49:22.911 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.911 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.911 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:22.911 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.121074+0000", 2024-09-21T15:49:22.911 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1372000000000001e-05 2024-09-21T15:49:22.911 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.911 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.911 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:22.911 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.121087+0000", 2024-09-21T15:49:22.911 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2237e-05 2024-09-21T15:49:22.911 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.911 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.912 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:22.912 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.121098+0000", 2024-09-21T15:49:22.912 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0842000000000001e-05 2024-09-21T15:49:22.912 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.912 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.912 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:22.912 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.121137+0000", 2024-09-21T15:49:22.912 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9378000000000001e-05 2024-09-21T15:49:22.912 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.912 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.912 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:22.912 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.121211+0000", 2024-09-21T15:49:22.912 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3714999999999995e-05 2024-09-21T15:49:22.913 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.913 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.913 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-21T15:49:22.913 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.121333+0000", 2024-09-21T15:49:22.913 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000122745 2024-09-21T15:49:22.913 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.913 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.913 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:22.913 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.126370+0000", 2024-09-21T15:49:22.913 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0050369580000000002 2024-09-21T15:49:22.913 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.913 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.913 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:22.914 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.129048+0000", 2024-09-21T15:49:22.914 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002678153 2024-09-21T15:49:22.914 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.914 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.914 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:22.914 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.129078+0000", 2024-09-21T15:49:22.914 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9889000000000001e-05 2024-09-21T15:49:22.914 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.914 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.914 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:22.914 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.129106+0000", 2024-09-21T15:49:22.915 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7721000000000001e-05 2024-09-21T15:49:22.915 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.915 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:22.915 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.915 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.915 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.915 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4403.0:14541 3.1e e33/22 3:7e48baef:::rbd_data.1133dc5adfdf.00000000000000b2:head v 33'102, mlcod=33'100)", 2024-09-21T15:49:22.915 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:19.121771+0000", 2024-09-21T15:49:22.915 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 3.6710920639999998, 2024-09-21T15:49:22.915 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0072650869999999999, 2024-09-21T15:49:22.915 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:22.915 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:22.916 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:22.916 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.916 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:22.916 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.121771+0000", 2024-09-21T15:49:22.916 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.916 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.916 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.916 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:22.916 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.121771+0000", 2024-09-21T15:49:22.916 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.916 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.916 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.916 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:22.917 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.121770+0000", 2024-09-21T15:49:22.917 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-21T15:49:22.917 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.917 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.917 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:22.917 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.121778+0000", 2024-09-21T15:49:22.917 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5059999999999995e-06 2024-09-21T15:49:22.917 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.917 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.917 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:22.917 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.121788+0000", 2024-09-21T15:49:22.917 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2310000000000002e-06 2024-09-21T15:49:22.917 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.918 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.918 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:22.918 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.121792+0000", 2024-09-21T15:49:22.918 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4440000000000001e-06 2024-09-21T15:49:22.918 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.918 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.918 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:22.918 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.121823+0000", 2024-09-21T15:49:22.918 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1148000000000001e-05 2024-09-21T15:49:22.918 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.918 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.919 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:22.919 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.121836+0000", 2024-09-21T15:49:22.919 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3101e-05 2024-09-21T15:49:22.919 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.919 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.919 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:22.919 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.129003+0000", 2024-09-21T15:49:22.919 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0071666170000000001 2024-09-21T15:49:22.919 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.919 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.919 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:22.919 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.129036+0000", 2024-09-21T15:49:22.919 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3030000000000001e-05 2024-09-21T15:49:22.920 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.920 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:22.920 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.920 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.920 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.920 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4403.0:14558 3.8 e33/32 3:13437810:::rbd_data.1133dc5adfdf.0000000000000039:head v 33'57, mlcod=33'55)", 2024-09-21T15:49:22.920 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:19.139921+0000", 2024-09-21T15:49:22.920 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 3.6529423620000001, 2024-09-21T15:49:22.920 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0048845210000000002, 2024-09-21T15:49:22.920 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:22.920 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:22.921 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:22.921 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.921 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:22.921 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.139921+0000", 2024-09-21T15:49:22.921 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.921 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.921 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.921 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:22.921 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.139921+0000", 2024-09-21T15:49:22.921 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:22.921 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.921 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.921 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:22.921 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.139920+0000", 2024-09-21T15:49:22.922 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-21T15:49:22.922 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.922 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.922 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:22.922 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.139931+0000", 2024-09-21T15:49:22.922 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1138000000000001e-05 2024-09-21T15:49:22.922 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.922 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.922 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:22.922 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.139946+0000", 2024-09-21T15:49:22.922 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5622999999999999e-05 2024-09-21T15:49:22.922 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.923 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.923 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:22.923 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.139954+0000", 2024-09-21T15:49:22.923 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7559999999999999e-06 2024-09-21T15:49:22.923 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.923 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.923 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:22.923 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.139999+0000", 2024-09-21T15:49:22.923 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4481000000000003e-05 2024-09-21T15:49:22.923 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.923 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.923 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:22.924 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.140028+0000", 2024-09-21T15:49:22.924 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9045999999999999e-05 2024-09-21T15:49:22.924 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.924 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.924 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:22.924 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.144779+0000", 2024-09-21T15:49:22.924 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.004751529 2024-09-21T15:49:22.924 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:22.924 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:22.924 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:22.924 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.144805+0000", 2024-09-21T15:49:22.924 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5933000000000001e-05 2024-09-21T15:49:22.925 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.925 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:22.925 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.925 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:22.925 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:22.925 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:22.927 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:22.927 INFO:teuthology.orchestra.run.smithi089.stdout: "success": "" 2024-09-21T15:49:22.927 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:22.927 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T15:49:22.929 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:22.925+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:22.946 INFO:teuthology.orchestra.run.smithi089.stdout:checking clone #3, image image_client.0-clone3 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent4 2024-09-21T15:49:23.027 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:23.026+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:23.071 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:23.072 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:23.072 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:49:23.072 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:23.080 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T15:49:23.129 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:23.126+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:23.151 INFO:teuthology.orchestra.run.smithi089.stdout:1079 read 0x1aa7234 thru 0x1aa96f9 (0x24c6 bytes) 2024-09-21T15:49:23.153 INFO:teuthology.orchestra.run.smithi089.stdout:1080 punch from 0x876ecc3 to 0x8774b01, (0x5e3e bytes) 2024-09-21T15:49:23.156 INFO:teuthology.orchestra.run.smithi089.stdout:truncating to largest ever: 0xe0632a6 2024-09-21T15:49:23.156 INFO:teuthology.orchestra.run.smithi089.stdout:1082 trunc from 0xd92a319 to 0xe0632a6 2024-09-21T15:49:23.163 INFO:teuthology.orchestra.run.smithi089.stdout:1083 read 0x85d3859 thru 0x85d4d32 (0x14da bytes) 2024-09-21T15:49:23.163 INFO:teuthology.orchestra.run.smithi089.stdout:1085 read 0x311f361 thru 0x312340d (0x40ad bytes) 2024-09-21T15:49:23.163 INFO:teuthology.orchestra.run.smithi089.stdout:1086 read 0xb7007c0 thru 0xb70c176 (0xb9b7 bytes) 2024-09-21T15:49:23.163 INFO:teuthology.orchestra.run.smithi089.stdout:1087 write 0x7a9ce0c thru 0x7aa93c6 (0xc5bb bytes) 2024-09-21T15:49:23.170 INFO:teuthology.orchestra.run.smithi089.stdout:1088 punch from 0x834a868 to 0x8351525, (0x6cbd bytes) 2024-09-21T15:49:23.170 INFO:teuthology.orchestra.run.smithi089.stdout:1089 write 0xd7a70ad thru 0xd7aa0bb (0x300f bytes) 2024-09-21T15:49:23.174 INFO:teuthology.orchestra.run.smithi089.stdout:1090 read 0x56c561c thru 0x56d4cdf (0xf6c4 bytes) 2024-09-21T15:49:23.174 INFO:teuthology.orchestra.run.smithi089.stdout:1091 writesame 0x46acd02 thru 0x46adeb3 data_size 0xf(0x11b2 bytes) 2024-09-21T15:49:23.178 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:23.178 INFO:teuthology.orchestra.run.smithi089.stdout:1093 write 0x6934933 thru 0x693be16 (0x74e4 bytes) 2024-09-21T15:49:23.182 INFO:teuthology.orchestra.run.smithi089.stdout:1094 punch from 0xb781854 to 0xb78f798, (0xdf44 bytes) 2024-09-21T15:49:23.182 INFO:teuthology.orchestra.run.smithi089.stdout:1095 punch from 0xa0cb878 to 0xa0d6948, (0xb0d0 bytes) 2024-09-21T15:49:23.183 INFO:teuthology.orchestra.run.smithi089.stdout:1096 writesame 0x4f9d0f5 thru 0x4fa99fa data_size 0x2(0xc906 bytes) 2024-09-21T15:49:23.198 INFO:teuthology.orchestra.run.smithi089.stdout:truncating to largest ever: 0xe2d7125 2024-09-21T15:49:23.198 INFO:teuthology.orchestra.run.smithi089.stdout:1097 trunc from 0xe0632a6 to 0xe2d7125 2024-09-21T15:49:23.223 INFO:teuthology.orchestra.run.smithi089.stdout:1098 trunc from 0xe2d7125 to 0xf2455 2024-09-21T15:49:23.229 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:23.227+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:23.230 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:23.230 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:23.230 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:49:23.230 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:49:23.230 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:23.240 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T15:49:23.266 INFO:teuthology.orchestra.run.smithi089.stdout:1099 write 0x774a13d thru 0x77592dd (0xf1a1 bytes) 2024-09-21T15:49:23.268 INFO:teuthology.orchestra.run.smithi089.stdout:1100 read 0x1678192 thru 0x168613f (0xdfae bytes) 2024-09-21T15:49:23.273 INFO:teuthology.orchestra.run.smithi089.stdout:1101 trunc from 0x77592de to 0x38e25a0 2024-09-21T15:49:23.284 INFO:teuthology.orchestra.run.smithi089.stdout:1102 read 0xec0a2a thru 0xec2e51 (0x2428 bytes) 2024-09-21T15:49:23.284 INFO:teuthology.orchestra.run.smithi089.stdout:1103 punch from 0x7c0a63 to 0x7c54f3, (0x4a90 bytes) 2024-09-21T15:49:23.284 INFO:teuthology.orchestra.run.smithi089.stdout:1104 read 0x1858760 thru 0x18595a8 (0xe49 bytes) 2024-09-21T15:49:23.284 INFO:teuthology.orchestra.run.smithi089.stdout:1105 read 0x569650 thru 0x56a8a6 (0x1257 bytes) 2024-09-21T15:49:23.303 INFO:teuthology.orchestra.run.smithi089.stdout:1107 write 0xc92966c thru 0xc92c898 (0x322d bytes) 2024-09-21T15:49:23.311 INFO:teuthology.orchestra.run.smithi089.stdout:1108 write 0xac956eb thru 0xac9e298 (0x8bae bytes) 2024-09-21T15:49:23.314 INFO:teuthology.orchestra.run.smithi089.stdout:1109 writesame 0x5d4f5bf thru 0x5d50ffd data_size 0x1(0x1a3f bytes) 2024-09-21T15:49:23.319 INFO:teuthology.orchestra.run.smithi089.stdout:1110 read 0xc0ff2df thru 0xc102635 (0x3357 bytes) 2024-09-21T15:49:23.319 INFO:teuthology.orchestra.run.smithi089.stdout:1111 punch from 0x4fc5ba1 to 0x4fcbaa9, (0x5f08 bytes) 2024-09-21T15:49:23.319 INFO:teuthology.orchestra.run.smithi089.stdout:1114 write 0x149fb4 thru 0x151efe (0x7f4b bytes) 2024-09-21T15:49:23.320 INFO:teuthology.orchestra.run.smithi089.stdout:1116 read 0x77928d8 thru 0x7793f64 (0x168d bytes) 2024-09-21T15:49:23.320 INFO:teuthology.orchestra.run.smithi089.stdout:1117 writesame 0x7f1a63e thru 0x7f1bac3 data_size 0x25(0x1486 bytes) 2024-09-21T15:49:23.323 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:23.329 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:23.328+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:23.332 INFO:teuthology.orchestra.run.smithi089.stdout:1120 write 0xda00b7c thru 0xda0637d (0x5802 bytes) 2024-09-21T15:49:23.337 INFO:teuthology.orchestra.run.smithi089.stdout:1122 write 0xb549b6b thru 0xb55496d (0xae03 bytes) 2024-09-21T15:49:23.341 INFO:teuthology.orchestra.run.smithi089.stdout:1123 punch from 0x3d1b5e4 to 0x3d28adf, (0xd4fb bytes) 2024-09-21T15:49:23.341 INFO:teuthology.orchestra.run.smithi089.stdout:1125 writesame 0xb179e18 thru 0xb186fab data_size 0x44(0xd194 bytes) 2024-09-21T15:49:23.342 INFO:teuthology.orchestra.run.smithi089.stdout:1126 write 0x230899c thru 0x2317f82 (0xf5e7 bytes) 2024-09-21T15:49:23.347 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:23.347 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:49:23.347 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:49:23.347 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:49:23.347 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.347 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4209.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:49:23.347 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:49:23.347 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.153979231000001, 2024-09-21T15:49:23.347 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.19452909900000001, 2024-09-21T15:49:23.347 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:23.348 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:23.348 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:23.348 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4209", 2024-09-21T15:49:23.348 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/4045143050", 2024-09-21T15:49:23.348 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1 2024-09-21T15:49:23.348 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.348 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:23.348 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.348 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:23.348 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:49:23.348 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:23.348 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.348 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.348 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:23.348 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:49:23.348 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:23.349 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.349 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.349 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:23.349 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191161+0000", 2024-09-21T15:49:23.349 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:23.349 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.349 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.349 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:23.349 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191169+0000", 2024-09-21T15:49:23.349 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0090000000000007e-06 2024-09-21T15:49:23.349 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.349 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.349 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:23.349 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191180+0000", 2024-09-21T15:49:23.349 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0728e-05 2024-09-21T15:49:23.350 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.350 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.350 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:23.350 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191187+0000", 2024-09-21T15:49:23.350 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9009999999999994e-06 2024-09-21T15:49:23.350 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.350 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.350 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:23.350 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191238+0000", 2024-09-21T15:49:23.350 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0287000000000002e-05 2024-09-21T15:49:23.350 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.350 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.350 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-21T15:49:23.350 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191245+0000", 2024-09-21T15:49:23.350 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0709999999999997e-06 2024-09-21T15:49:23.350 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.351 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.351 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:23.351 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380309+0000", 2024-09-21T15:49:23.351 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.189064277 2024-09-21T15:49:23.351 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.351 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.351 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:23.351 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380454+0000", 2024-09-21T15:49:23.351 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014490799999999999 2024-09-21T15:49:23.351 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.351 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.351 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0", 2024-09-21T15:49:23.351 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380610+0000", 2024-09-21T15:49:23.352 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015650199999999999 2024-09-21T15:49:23.352 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.352 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.352 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:23.352 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.381160+0000", 2024-09-21T15:49:23.352 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00054903799999999996 2024-09-21T15:49:23.352 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.352 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.352 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:23.352 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385591+0000", 2024-09-21T15:49:23.352 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.004431272 2024-09-21T15:49:23.352 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.353 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.353 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:23.353 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385651+0000", 2024-09-21T15:49:23.353 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0455999999999999e-05 2024-09-21T15:49:23.353 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.353 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.353 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:23.353 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385691+0000", 2024-09-21T15:49:23.353 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0234999999999997e-05 2024-09-21T15:49:23.353 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:23.353 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:23.353 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:23.353 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.353 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.354 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:1838 3.11 3:8b85faea:::rbd_data.1133f3dfbd6b.000000000000000c:5 [read 7348224~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:23.354 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.093286+0000", 2024-09-21T15:49:23.354 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 9.251855484, 2024-09-21T15:49:23.354 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.042871797000000003, 2024-09-21T15:49:23.354 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:23.354 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:23.354 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:23.354 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:23.354 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:23.354 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1838 2024-09-21T15:49:23.354 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.354 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:23.354 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.355 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:23.355 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.093286+0000", 2024-09-21T15:49:23.355 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:23.355 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.355 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.355 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:23.355 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.093286+0000", 2024-09-21T15:49:23.355 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:23.355 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.355 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.355 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:23.355 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.093174+0000", 2024-09-21T15:49:23.355 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998879 2024-09-21T15:49:23.355 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.355 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.356 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:23.356 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.093288+0000", 2024-09-21T15:49:23.356 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00011387600000000001 2024-09-21T15:49:23.356 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.356 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.356 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:23.356 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.093293+0000", 2024-09-21T15:49:23.356 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7600000000000002e-06 2024-09-21T15:49:23.356 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.356 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.356 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:23.356 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.093302+0000", 2024-09-21T15:49:23.356 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3649999999999993e-06 2024-09-21T15:49:23.356 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.356 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.356 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:23.357 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.135394+0000", 2024-09-21T15:49:23.357 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.042091954000000001 2024-09-21T15:49:23.357 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.357 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.357 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:23.357 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.135538+0000", 2024-09-21T15:49:23.357 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014408199999999999 2024-09-21T15:49:23.357 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.357 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.357 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:23.357 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.136158+0000", 2024-09-21T15:49:23.357 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00061959500000000002 2024-09-21T15:49:23.357 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:23.357 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:23.357 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:23.357 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.358 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.358 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:1858 3.11 3:8b85faea:::rbd_data.1133f3dfbd6b.000000000000000c:5 [read 8331264~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:23.358 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.096865+0000", 2024-09-21T15:49:23.358 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 9.2482764839999998, 2024-09-21T15:49:23.358 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.042908221000000003, 2024-09-21T15:49:23.358 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:23.358 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:23.358 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:23.358 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:23.358 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:23.358 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1858 2024-09-21T15:49:23.358 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.358 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:23.358 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.358 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:23.359 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.096865+0000", 2024-09-21T15:49:23.359 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:23.359 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.359 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.359 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:23.359 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.096865+0000", 2024-09-21T15:49:23.359 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:23.359 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.359 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.359 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:23.359 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.096865+0000", 2024-09-21T15:49:23.359 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:23.359 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.359 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.359 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:23.359 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.096870+0000", 2024-09-21T15:49:23.360 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0710000000000002e-06 2024-09-21T15:49:23.360 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.360 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.360 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:23.360 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.096873+0000", 2024-09-21T15:49:23.360 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.005e-06 2024-09-21T15:49:23.360 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.360 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.360 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:23.360 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.096875+0000", 2024-09-21T15:49:23.360 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2740000000000002e-06 2024-09-21T15:49:23.360 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.360 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.360 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:23.360 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.139613+0000", 2024-09-21T15:49:23.361 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.042737334000000002 2024-09-21T15:49:23.361 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.361 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.361 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:23.361 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.139699+0000", 2024-09-21T15:49:23.361 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6687999999999994e-05 2024-09-21T15:49:23.361 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.361 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.361 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:23.361 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.139773+0000", 2024-09-21T15:49:23.361 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4053000000000004e-05 2024-09-21T15:49:23.361 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:23.361 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:23.361 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:23.362 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.362 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.362 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:1859 3.11 3:8b85faea:::rbd_data.1133f3dfbd6b.000000000000000c:5 [read 8380416~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:23.362 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.097017+0000", 2024-09-21T15:49:23.362 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 9.2481250359999994, 2024-09-21T15:49:23.362 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.042963952999999999, 2024-09-21T15:49:23.362 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:23.362 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:23.362 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:23.362 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:23.362 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:23.362 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1859 2024-09-21T15:49:23.362 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.362 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:23.362 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.363 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:23.363 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.097017+0000", 2024-09-21T15:49:23.363 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:23.363 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.363 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.363 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:23.363 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.097017+0000", 2024-09-21T15:49:23.363 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:23.363 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.363 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.363 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:23.363 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.096877+0000", 2024-09-21T15:49:23.363 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998608 2024-09-21T15:49:23.363 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.363 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.364 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:23.364 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.097020+0000", 2024-09-21T15:49:23.364 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014238 2024-09-21T15:49:23.364 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.364 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.364 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:23.364 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.097026+0000", 2024-09-21T15:49:23.364 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8310000000000003e-06 2024-09-21T15:49:23.364 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.364 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.364 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:23.364 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.097032+0000", 2024-09-21T15:49:23.364 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4249999999999997e-06 2024-09-21T15:49:23.364 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.364 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.364 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:23.365 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.139813+0000", 2024-09-21T15:49:23.365 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.042781053999999999 2024-09-21T15:49:23.365 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.365 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.365 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:23.365 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.139901+0000", 2024-09-21T15:49:23.365 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8472000000000003e-05 2024-09-21T15:49:23.365 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.365 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.365 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:23.365 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.139981+0000", 2024-09-21T15:49:23.365 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9128000000000005e-05 2024-09-21T15:49:23.365 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:23.365 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:23.365 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:23.366 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.366 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.366 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:1871 3.11 3:8b85faea:::rbd_data.1133f3dfbd6b.000000000000000c:5 [read 6860800~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:23.366 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.099208+0000", 2024-09-21T15:49:23.366 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 9.2459331680000005, 2024-09-21T15:49:23.366 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.042862763999999998, 2024-09-21T15:49:23.366 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:23.366 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:23.366 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:23.366 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:23.366 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:23.366 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1871 2024-09-21T15:49:23.366 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.366 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:23.366 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.366 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:23.367 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.099208+0000", 2024-09-21T15:49:23.367 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:23.367 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.367 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.367 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:23.367 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.099208+0000", 2024-09-21T15:49:23.367 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:23.367 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.367 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.367 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:23.367 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.099208+0000", 2024-09-21T15:49:23.367 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:23.367 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.367 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.367 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:23.368 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.099210+0000", 2024-09-21T15:49:23.368 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.12e-06 2024-09-21T15:49:23.368 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.368 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.368 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:23.368 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.099219+0000", 2024-09-21T15:49:23.368 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1370000000000008e-06 2024-09-21T15:49:23.368 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.368 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.368 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:23.368 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.099224+0000", 2024-09-21T15:49:23.368 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6110000000000004e-06 2024-09-21T15:49:23.368 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.368 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.368 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:23.368 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.141899+0000", 2024-09-21T15:49:23.369 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.042675376000000001 2024-09-21T15:49:23.369 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.369 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.369 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:23.369 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.141988+0000", 2024-09-21T15:49:23.369 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8565000000000003e-05 2024-09-21T15:49:23.369 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.369 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.369 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:23.369 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.142071+0000", 2024-09-21T15:49:23.369 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3325999999999995e-05 2024-09-21T15:49:23.369 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:23.369 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:23.369 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:23.369 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.369 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.370 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:6182 3.e 3:734145f0:::rbd_data.1133f3dfbd6b.0000000000000003:5 [read 618496~4096] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:23.370 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.736429+0000", 2024-09-21T15:49:23.370 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 8.6087126040000008, 2024-09-21T15:49:23.370 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.042734435000000001, 2024-09-21T15:49:23.370 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:23.370 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:23.370 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:23.370 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:23.370 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:23.370 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 6182 2024-09-21T15:49:23.370 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.370 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:23.370 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.370 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:23.370 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736429+0000", 2024-09-21T15:49:23.371 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:23.371 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.371 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.371 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:23.371 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736429+0000", 2024-09-21T15:49:23.371 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:23.371 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.371 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.371 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:23.371 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736429+0000", 2024-09-21T15:49:23.371 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:23.371 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.371 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.371 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:23.371 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736430+0000", 2024-09-21T15:49:23.371 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7900000000000007e-07 2024-09-21T15:49:23.372 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.372 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.372 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:23.372 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736433+0000", 2024-09-21T15:49:23.372 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7140000000000002e-06 2024-09-21T15:49:23.372 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.372 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.372 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:23.372 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736435+0000", 2024-09-21T15:49:23.372 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.357e-06 2024-09-21T15:49:23.372 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.372 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.372 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:23.372 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.779070+0000", 2024-09-21T15:49:23.372 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.042635550000000001 2024-09-21T15:49:23.372 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.373 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.373 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:23.373 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.779112+0000", 2024-09-21T15:49:23.373 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1124000000000001e-05 2024-09-21T15:49:23.373 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.373 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.373 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:23.373 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.779163+0000", 2024-09-21T15:49:23.373 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1913000000000003e-05 2024-09-21T15:49:23.373 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:23.373 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:23.373 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:23.373 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.373 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.373 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:6183 3.e 3:734145f0:::rbd_data.1133f3dfbd6b.0000000000000003:5 [read 667648~4096] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:23.373 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.736437+0000", 2024-09-21T15:49:23.374 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 8.6087047729999995, 2024-09-21T15:49:23.374 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.042929953, 2024-09-21T15:49:23.374 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:23.374 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:23.374 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:23.374 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:23.374 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:23.374 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 6183 2024-09-21T15:49:23.374 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.374 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:23.374 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.374 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:23.374 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736437+0000", 2024-09-21T15:49:23.375 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:23.375 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.375 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.375 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:23.375 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736437+0000", 2024-09-21T15:49:23.375 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:23.375 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.375 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.375 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:23.375 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736437+0000", 2024-09-21T15:49:23.375 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:23.375 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.375 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.375 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:23.375 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736441+0000", 2024-09-21T15:49:23.376 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8269999999999997e-06 2024-09-21T15:49:23.376 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.376 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.376 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:23.376 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736445+0000", 2024-09-21T15:49:23.376 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5329999999999999e-06 2024-09-21T15:49:23.376 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.376 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.376 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:23.376 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736447+0000", 2024-09-21T15:49:23.376 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3989999999999998e-06 2024-09-21T15:49:23.376 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.376 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.377 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:23.377 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.779194+0000", 2024-09-21T15:49:23.377 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.042746447999999999 2024-09-21T15:49:23.377 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.377 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.377 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:23.377 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.779280+0000", 2024-09-21T15:49:23.377 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6304999999999994e-05 2024-09-21T15:49:23.377 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.377 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.377 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:23.377 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.779367+0000", 2024-09-21T15:49:23.377 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6643000000000004e-05 2024-09-21T15:49:23.377 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:23.378 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:23.378 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:23.378 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.378 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.378 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:6184 3.e 3:734145f0:::rbd_data.1133f3dfbd6b.0000000000000003:5 [read 716800~4096] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:23.378 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.736450+0000", 2024-09-21T15:49:23.378 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 8.6086920899999999, 2024-09-21T15:49:23.378 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.043020819000000002, 2024-09-21T15:49:23.378 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:23.378 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:23.378 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:23.378 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:23.378 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:23.379 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 6184 2024-09-21T15:49:23.379 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.379 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:23.379 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.379 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:23.379 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736450+0000", 2024-09-21T15:49:23.379 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:23.379 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.379 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.379 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:23.379 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736450+0000", 2024-09-21T15:49:23.379 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:23.379 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.379 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.379 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:23.380 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736449+0000", 2024-09-21T15:49:23.380 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:23.380 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.380 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.380 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:23.380 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736450+0000", 2024-09-21T15:49:23.380 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0589999999999999e-06 2024-09-21T15:49:23.380 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.380 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.380 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:23.380 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736453+0000", 2024-09-21T15:49:23.380 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8080000000000001e-06 2024-09-21T15:49:23.380 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.380 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.380 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:23.381 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736455+0000", 2024-09-21T15:49:23.381 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2809999999999998e-06 2024-09-21T15:49:23.381 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.381 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.381 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:23.381 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.779378+0000", 2024-09-21T15:49:23.381 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.042922279000000001 2024-09-21T15:49:23.381 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.381 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.381 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:23.381 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.779419+0000", 2024-09-21T15:49:23.381 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1726e-05 2024-09-21T15:49:23.381 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.381 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.381 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:23.382 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.779470+0000", 2024-09-21T15:49:23.382 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0871999999999997e-05 2024-09-21T15:49:23.382 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:23.382 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:23.382 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:23.382 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.382 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.382 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:6185 3.e 3:734145f0:::rbd_data.1133f3dfbd6b.0000000000000003:5 [read 765952~4096] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:23.382 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.736458+0000", 2024-09-21T15:49:23.382 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 8.6086840979999995, 2024-09-21T15:49:23.382 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.043226015, 2024-09-21T15:49:23.382 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:23.382 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:23.382 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:23.382 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:23.383 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:23.383 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 6185 2024-09-21T15:49:23.383 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.383 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:23.383 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.383 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:23.383 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736458+0000", 2024-09-21T15:49:23.383 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:23.383 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.383 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.383 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:23.383 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736458+0000", 2024-09-21T15:49:23.383 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:23.383 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.383 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.383 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:23.384 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736457+0000", 2024-09-21T15:49:23.384 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:23.384 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.384 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.384 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:23.384 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736458+0000", 2024-09-21T15:49:23.384 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0279999999999999e-06 2024-09-21T15:49:23.384 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.384 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.384 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:23.384 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736461+0000", 2024-09-21T15:49:23.384 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0510000000000001e-06 2024-09-21T15:49:23.384 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.384 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.384 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:23.385 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736464+0000", 2024-09-21T15:49:23.385 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4159999999999998e-06 2024-09-21T15:49:23.385 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.385 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.385 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:23.385 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.779502+0000", 2024-09-21T15:49:23.385 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.043038218000000003 2024-09-21T15:49:23.385 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.385 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.385 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:23.385 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.779592+0000", 2024-09-21T15:49:23.385 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.9834999999999996e-05 2024-09-21T15:49:23.385 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.385 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.385 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:23.386 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.779684+0000", 2024-09-21T15:49:23.386 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1673000000000001e-05 2024-09-21T15:49:23.386 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:23.386 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:23.386 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:23.386 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.386 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.386 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:6186 3.e 3:734145f0:::rbd_data.1133f3dfbd6b.0000000000000003:5 [read 815104~4096] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:23.386 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.736466+0000", 2024-09-21T15:49:23.386 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 8.6086758240000005, 2024-09-21T15:49:23.386 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.043438982000000001, 2024-09-21T15:49:23.386 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:23.386 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:23.386 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:23.386 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:23.386 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:23.387 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 6186 2024-09-21T15:49:23.387 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.387 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:23.387 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.387 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:23.387 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736466+0000", 2024-09-21T15:49:23.387 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:23.387 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.387 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.387 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:23.387 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736466+0000", 2024-09-21T15:49:23.387 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:23.387 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.387 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.387 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:23.388 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736466+0000", 2024-09-21T15:49:23.388 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:23.388 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.388 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.388 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:23.388 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736467+0000", 2024-09-21T15:49:23.388 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.009e-06 2024-09-21T15:49:23.388 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.388 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.388 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:23.388 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736469+0000", 2024-09-21T15:49:23.388 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8449999999999999e-06 2024-09-21T15:49:23.388 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.388 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.388 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:23.389 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736472+0000", 2024-09-21T15:49:23.389 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3140000000000002e-06 2024-09-21T15:49:23.389 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.389 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.389 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:23.389 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.779724+0000", 2024-09-21T15:49:23.389 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.043252295000000003 2024-09-21T15:49:23.389 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.389 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.389 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:23.389 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.779808+0000", 2024-09-21T15:49:23.389 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3765999999999997e-05 2024-09-21T15:49:23.389 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.389 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.389 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:23.390 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.779905+0000", 2024-09-21T15:49:23.390 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.6953999999999994e-05 2024-09-21T15:49:23.390 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:23.390 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:23.390 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:23.390 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.390 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.390 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:6187 3.e 3:734145f0:::rbd_data.1133f3dfbd6b.0000000000000003:5 [read 864256~4096] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:23.390 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.736474+0000", 2024-09-21T15:49:23.390 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 8.6086678200000009, 2024-09-21T15:49:23.390 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.043635611999999997, 2024-09-21T15:49:23.390 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:23.390 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:23.391 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:23.391 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:23.391 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:23.391 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 6187 2024-09-21T15:49:23.391 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.391 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:23.391 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.391 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:23.391 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736474+0000", 2024-09-21T15:49:23.391 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:23.391 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.391 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.391 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:23.391 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736474+0000", 2024-09-21T15:49:23.391 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:23.391 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.392 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.392 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:23.392 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736474+0000", 2024-09-21T15:49:23.392 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:23.392 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.392 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.392 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:23.392 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736478+0000", 2024-09-21T15:49:23.392 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.843e-06 2024-09-21T15:49:23.392 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.392 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.392 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:23.392 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736481+0000", 2024-09-21T15:49:23.392 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.055e-06 2024-09-21T15:49:23.392 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.392 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.393 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:23.393 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736484+0000", 2024-09-21T15:49:23.393 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3649999999999998e-06 2024-09-21T15:49:23.393 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.393 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.393 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:23.393 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.779927+0000", 2024-09-21T15:49:23.393 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.043443038000000003 2024-09-21T15:49:23.393 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.393 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.393 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:23.393 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.780024+0000", 2024-09-21T15:49:23.393 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7087000000000001e-05 2024-09-21T15:49:23.393 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.393 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.393 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:23.394 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.780109+0000", 2024-09-21T15:49:23.394 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5422999999999998e-05 2024-09-21T15:49:23.394 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:23.394 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:23.394 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:23.394 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.394 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.394 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:6188 3.e 3:734145f0:::rbd_data.1133f3dfbd6b.0000000000000003:5 [read 913408~4096] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:23.394 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.736486+0000", 2024-09-21T15:49:23.394 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 8.6086555110000003, 2024-09-21T15:49:23.394 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.043726186, 2024-09-21T15:49:23.394 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:23.394 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:23.394 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:23.395 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:23.395 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:23.395 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 6188 2024-09-21T15:49:23.395 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.395 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:23.395 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.395 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:23.395 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736486+0000", 2024-09-21T15:49:23.395 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:23.395 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.395 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.395 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:23.395 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736486+0000", 2024-09-21T15:49:23.395 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:23.395 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.396 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.396 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:23.396 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736486+0000", 2024-09-21T15:49:23.396 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:23.396 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.396 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.396 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:23.396 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736487+0000", 2024-09-21T15:49:23.396 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.079e-06 2024-09-21T15:49:23.396 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.396 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.396 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:23.396 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736490+0000", 2024-09-21T15:49:23.396 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7829999999999999e-06 2024-09-21T15:49:23.396 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.396 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.397 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:23.397 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736492+0000", 2024-09-21T15:49:23.397 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3190000000000002e-06 2024-09-21T15:49:23.397 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.397 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.397 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:23.397 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.780122+0000", 2024-09-21T15:49:23.397 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.043630245999999998 2024-09-21T15:49:23.397 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.397 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.397 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:23.397 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.780165+0000", 2024-09-21T15:49:23.397 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2833999999999997e-05 2024-09-21T15:49:23.397 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.397 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.398 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:23.398 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.780212+0000", 2024-09-21T15:49:23.398 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7134e-05 2024-09-21T15:49:23.398 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:23.398 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:23.398 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:23.398 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.398 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.398 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:6189 3.e 3:734145f0:::rbd_data.1133f3dfbd6b.0000000000000003:5 [read 962560~4096] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:23.398 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.736494+0000", 2024-09-21T15:49:23.398 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 8.6086473179999992, 2024-09-21T15:49:23.398 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.043933139000000003, 2024-09-21T15:49:23.399 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:23.399 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:23.399 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:23.399 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:23.399 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:23.399 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 6189 2024-09-21T15:49:23.399 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.399 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:23.399 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.399 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:23.399 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736494+0000", 2024-09-21T15:49:23.399 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:23.399 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.399 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.399 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:23.400 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736494+0000", 2024-09-21T15:49:23.400 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:23.400 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.400 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.400 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:23.400 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736494+0000", 2024-09-21T15:49:23.400 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:23.400 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.400 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.400 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:23.400 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736495+0000", 2024-09-21T15:49:23.400 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.221e-06 2024-09-21T15:49:23.400 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.400 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.400 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:23.400 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736500+0000", 2024-09-21T15:49:23.401 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6029999999999998e-06 2024-09-21T15:49:23.401 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.401 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.401 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:23.401 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736502+0000", 2024-09-21T15:49:23.401 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4889999999999998e-06 2024-09-21T15:49:23.401 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.401 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.401 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:23.401 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.780241+0000", 2024-09-21T15:49:23.401 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.043738491999999997 2024-09-21T15:49:23.401 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.401 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.401 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:23.401 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.780335+0000", 2024-09-21T15:49:23.402 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3783000000000007e-05 2024-09-21T15:49:23.402 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.402 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.402 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:23.402 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.780427+0000", 2024-09-21T15:49:23.402 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2912999999999996e-05 2024-09-21T15:49:23.402 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:23.402 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:23.402 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:23.402 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.402 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.402 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:6192 3.e 3:734145f0:::rbd_data.1133f3dfbd6b.0000000000000003:5 [read 1110016~4096] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:23.402 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.738041+0000", 2024-09-21T15:49:23.402 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 8.6071006130000001, 2024-09-21T15:49:23.402 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.042836826000000001, 2024-09-21T15:49:23.402 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:23.403 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:23.403 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:23.403 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:23.403 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:23.403 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 6192 2024-09-21T15:49:23.403 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.403 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:23.403 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.403 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:23.403 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738041+0000", 2024-09-21T15:49:23.403 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:23.403 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.403 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.403 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:23.403 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738041+0000", 2024-09-21T15:49:23.403 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:23.404 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.404 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.404 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:23.404 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738041+0000", 2024-09-21T15:49:23.404 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:23.404 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.404 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.404 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:23.404 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738042+0000", 2024-09-21T15:49:23.404 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.144e-06 2024-09-21T15:49:23.404 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.404 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.404 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:23.404 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738046+0000", 2024-09-21T15:49:23.404 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.177e-06 2024-09-21T15:49:23.404 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.405 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.405 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:23.405 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738049+0000", 2024-09-21T15:49:23.405 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5270000000000001e-06 2024-09-21T15:49:23.405 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.405 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.405 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:23.405 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.780782+0000", 2024-09-21T15:49:23.405 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.042733110999999997 2024-09-21T15:49:23.405 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.405 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.405 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:23.405 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.780825+0000", 2024-09-21T15:49:23.405 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3578999999999997e-05 2024-09-21T15:49:23.405 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.406 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.406 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:23.406 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.780878+0000", 2024-09-21T15:49:23.406 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2493e-05 2024-09-21T15:49:23.406 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:23.406 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:23.406 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:23.406 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.406 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.406 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:6193 3.e 3:734145f0:::rbd_data.1133f3dfbd6b.0000000000000003:5 [read 1159168~4096] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:23.406 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.738051+0000", 2024-09-21T15:49:23.406 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 8.6070907430000005, 2024-09-21T15:49:23.406 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.043033658000000002, 2024-09-21T15:49:23.406 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:23.406 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:23.407 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:23.407 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:23.407 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:23.407 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 6193 2024-09-21T15:49:23.407 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.407 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:23.407 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.407 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:23.407 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738051+0000", 2024-09-21T15:49:23.407 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:23.407 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.407 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.407 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:23.407 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738051+0000", 2024-09-21T15:49:23.407 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:23.407 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.408 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.408 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:23.408 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738051+0000", 2024-09-21T15:49:23.408 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:23.408 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.408 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.408 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:23.408 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738052+0000", 2024-09-21T15:49:23.408 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1790000000000001e-06 2024-09-21T15:49:23.408 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.408 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.408 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:23.408 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738055+0000", 2024-09-21T15:49:23.408 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8880000000000001e-06 2024-09-21T15:49:23.408 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.409 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.409 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:23.409 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738057+0000", 2024-09-21T15:49:23.409 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4700000000000001e-06 2024-09-21T15:49:23.409 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.409 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.409 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:23.409 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.780911+0000", 2024-09-21T15:49:23.409 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.042854056000000001 2024-09-21T15:49:23.409 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.409 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.409 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:23.409 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.780987+0000", 2024-09-21T15:49:23.410 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5543999999999994e-05 2024-09-21T15:49:23.410 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.410 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.410 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:23.410 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.781085+0000", 2024-09-21T15:49:23.410 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7722999999999999e-05 2024-09-21T15:49:23.410 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:23.410 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:23.410 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:23.410 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.410 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.410 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:6194 3.e 3:734145f0:::rbd_data.1133f3dfbd6b.0000000000000003:5 [read 1208320~4096] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:23.410 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.738059+0000", 2024-09-21T15:49:23.410 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 8.6070823149999995, 2024-09-21T15:49:23.410 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.043212726, 2024-09-21T15:49:23.411 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:23.411 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:23.411 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:23.411 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:23.411 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:23.411 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 6194 2024-09-21T15:49:23.411 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.411 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:23.411 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.411 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:23.411 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738059+0000", 2024-09-21T15:49:23.411 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:23.411 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.411 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.412 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:23.412 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738059+0000", 2024-09-21T15:49:23.412 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:23.412 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.412 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.412 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:23.412 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738059+0000", 2024-09-21T15:49:23.412 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:23.412 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.412 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.412 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:23.412 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738060+0000", 2024-09-21T15:49:23.412 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.13e-06 2024-09-21T15:49:23.412 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.413 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.413 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:23.413 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738063+0000", 2024-09-21T15:49:23.413 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7130000000000001e-06 2024-09-21T15:49:23.413 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.413 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.413 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:23.413 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738066+0000", 2024-09-21T15:49:23.413 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3220000000000001e-06 2024-09-21T15:49:23.413 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.413 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.413 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:23.413 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.781126+0000", 2024-09-21T15:49:23.413 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.043059865000000003 2024-09-21T15:49:23.413 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.414 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.414 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:23.414 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.781205+0000", 2024-09-21T15:49:23.414 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8671999999999995e-05 2024-09-21T15:49:23.414 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.414 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.414 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:23.414 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.781272+0000", 2024-09-21T15:49:23.414 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7231999999999996e-05 2024-09-21T15:49:23.414 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:23.414 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:23.414 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:23.414 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.415 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.415 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:6195 3.e 3:734145f0:::rbd_data.1133f3dfbd6b.0000000000000003:5 [read 1257472~4096] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:23.415 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.738068+0000", 2024-09-21T15:49:23.415 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 8.6070731810000005, 2024-09-21T15:49:23.415 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.043388518000000001, 2024-09-21T15:49:23.415 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:23.415 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:23.415 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:23.415 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:23.415 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:23.415 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 6195 2024-09-21T15:49:23.415 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.415 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:23.415 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.416 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:23.416 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738068+0000", 2024-09-21T15:49:23.416 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:23.416 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.416 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.416 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:23.416 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738068+0000", 2024-09-21T15:49:23.416 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:23.416 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.416 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.416 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:23.416 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738068+0000", 2024-09-21T15:49:23.416 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:23.416 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.416 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.417 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:23.417 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738069+0000", 2024-09-21T15:49:23.417 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.187e-06 2024-09-21T15:49:23.417 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.417 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.417 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:23.417 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738072+0000", 2024-09-21T15:49:23.417 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8830000000000002e-06 2024-09-21T15:49:23.417 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.417 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.417 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:23.417 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738075+0000", 2024-09-21T15:49:23.417 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4990000000000001e-06 2024-09-21T15:49:23.417 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.417 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:23.418 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.781302+0000", 2024-09-21T15:49:23.418 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.043227527000000002 2024-09-21T15:49:23.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.418 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:23.418 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.781374+0000", 2024-09-21T15:49:23.418 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1307e-05 2024-09-21T15:49:23.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.418 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:23.418 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.781457+0000", 2024-09-21T15:49:23.418 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3349000000000004e-05 2024-09-21T15:49:23.418 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:23.419 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:23.419 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:23.419 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.419 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.419 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:6196 3.e 3:734145f0:::rbd_data.1133f3dfbd6b.0000000000000003:5 [read 1306624~4096] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:23.419 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.738089+0000", 2024-09-21T15:49:23.419 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 8.6070529269999998, 2024-09-21T15:49:23.419 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.043542355999999997, 2024-09-21T15:49:23.419 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:23.419 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:23.419 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:23.419 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:23.419 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:23.419 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 6196 2024-09-21T15:49:23.420 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.420 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:23.420 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.420 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:23.420 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738089+0000", 2024-09-21T15:49:23.420 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:23.420 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.420 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.420 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:23.420 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738089+0000", 2024-09-21T15:49:23.420 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:23.420 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.420 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.420 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:23.421 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738088+0000", 2024-09-21T15:49:23.421 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:23.421 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.421 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.421 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:23.421 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738090+0000", 2024-09-21T15:49:23.421 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.81e-06 2024-09-21T15:49:23.421 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.421 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.421 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:23.421 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738096+0000", 2024-09-21T15:49:23.421 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0140000000000004e-06 2024-09-21T15:49:23.421 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.421 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.422 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:23.422 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738099+0000", 2024-09-21T15:49:23.422 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.723e-06 2024-09-21T15:49:23.422 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.422 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.422 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:23.422 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.781482+0000", 2024-09-21T15:49:23.422 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.043383065999999998 2024-09-21T15:49:23.422 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.422 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.422 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:23.422 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.781526+0000", 2024-09-21T15:49:23.422 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4276000000000001e-05 2024-09-21T15:49:23.422 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.422 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.423 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:23.423 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.781631+0000", 2024-09-21T15:49:23.423 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000104894 2024-09-21T15:49:23.423 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:23.423 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:23.423 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:23.423 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.423 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.423 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:6197 3.e 3:734145f0:::rbd_data.1133f3dfbd6b.0000000000000003:5 [read 1355776~4096] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:23.423 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.738101+0000", 2024-09-21T15:49:23.423 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 8.6070407200000005, 2024-09-21T15:49:23.423 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.043764458999999999, 2024-09-21T15:49:23.423 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:23.423 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:23.423 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:23.424 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:23.424 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:23.424 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 6197 2024-09-21T15:49:23.424 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.424 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:23.424 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.424 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:23.424 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738101+0000", 2024-09-21T15:49:23.424 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:23.424 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.424 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.424 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:23.424 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738101+0000", 2024-09-21T15:49:23.424 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:23.424 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.425 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.425 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:23.425 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738101+0000", 2024-09-21T15:49:23.425 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:23.425 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.425 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.425 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:23.425 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738102+0000", 2024-09-21T15:49:23.425 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.186e-06 2024-09-21T15:49:23.425 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.425 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.425 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:23.425 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738105+0000", 2024-09-21T15:49:23.426 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.965e-06 2024-09-21T15:49:23.426 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.426 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.426 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:23.426 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738107+0000", 2024-09-21T15:49:23.426 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5289999999999998e-06 2024-09-21T15:49:23.426 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.426 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.426 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:23.426 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.781681+0000", 2024-09-21T15:49:23.426 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.043574031999999999 2024-09-21T15:49:23.426 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.426 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.427 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:23.427 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.781767+0000", 2024-09-21T15:49:23.427 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5995000000000006e-05 2024-09-21T15:49:23.427 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.427 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.427 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:23.427 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.781865+0000", 2024-09-21T15:49:23.427 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7962999999999997e-05 2024-09-21T15:49:23.427 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:23.427 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:23.427 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:23.427 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.427 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.427 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:6198 3.e 3:734145f0:::rbd_data.1133f3dfbd6b.0000000000000003:5 [read 1404928~4096] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:23.428 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.738109+0000", 2024-09-21T15:49:23.428 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 8.6070321399999994, 2024-09-21T15:49:23.428 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.043890720000000001, 2024-09-21T15:49:23.428 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:23.428 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:23.428 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:23.428 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:23.428 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:23.428 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 6198 2024-09-21T15:49:23.428 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.428 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:23.428 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.428 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:23.428 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738109+0000", 2024-09-21T15:49:23.428 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:23.429 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.429 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.429 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:23.429 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738109+0000", 2024-09-21T15:49:23.429 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:23.429 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.429 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.429 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:23.429 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738109+0000", 2024-09-21T15:49:23.429 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:23.429 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.429 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.429 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:23.429 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738110+0000", 2024-09-21T15:49:23.429 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1689999999999999e-06 2024-09-21T15:49:23.430 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.430 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.430 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:23.430 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738113+0000", 2024-09-21T15:49:23.430 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7879999999999998e-06 2024-09-21T15:49:23.430 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.430 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.430 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:23.430 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738116+0000", 2024-09-21T15:49:23.430 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4059999999999999e-06 2024-09-21T15:49:23.430 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.430 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.430 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:23.430 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.781879+0000", 2024-09-21T15:49:23.431 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.043763608000000002 2024-09-21T15:49:23.431 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.431 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.431 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:23.431 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.781946+0000", 2024-09-21T15:49:23.431 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6595999999999997e-05 2024-09-21T15:49:23.431 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:23.431 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:23.431 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:23.431 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.782000+0000", 2024-09-21T15:49:23.431 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4447999999999998e-05 2024-09-21T15:49:23.431 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:23.431 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:23.431 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:23.432 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:23.432 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:23.432 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:23.434 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:49:23.435 INFO:teuthology.orchestra.run.smithi089.stdout:1128 write 0x9279834 thru 0x927edf9 (0x55c6 bytes) 2024-09-21T15:49:23.435 INFO:teuthology.orchestra.run.smithi089.stdout:1129 read 0x38c0093 thru 0x38cb479 (0xb3e7 bytes) 2024-09-21T15:49:23.435 INFO:teuthology.orchestra.run.smithi089.stdout:1131 read 0x3482ac thru 0x352d2c (0xaa81 bytes) 2024-09-21T15:49:23.435 INFO:teuthology.orchestra.run.smithi089.stdout:1132 write 0xc5955d6 thru 0xc5a27e4 (0xd20f bytes) 2024-09-21T15:49:23.435 INFO:teuthology.orchestra.run.smithi089.stdout:1133 punch from 0x8ac72ed to 0x8ac9e1f, (0x2b32 bytes) 2024-09-21T15:49:23.435 INFO:teuthology.orchestra.run.smithi089.stdout:1134 trunc from 0xda0637e to 0x89b6c61 2024-09-21T15:49:23.435 INFO:teuthology.orchestra.run.smithi089.stdout:1135 read 0x31a6181 thru 0x31b2e83 (0xcd03 bytes) 2024-09-21T15:49:23.435 INFO:teuthology.orchestra.run.smithi089.stdout:1136 read 0x7810712 thru 0x781232e (0x1c1d bytes) 2024-09-21T15:49:23.435 INFO:teuthology.orchestra.run.smithi089.stdout:1137 punch from 0x2cc6cbd to 0x2cd5aee, (0xee31 bytes) 2024-09-21T15:49:23.436 INFO:teuthology.orchestra.run.smithi089.stdout:1138 writesame 0xba7005a thru 0xba705d4 data_size 0x1(0x57b bytes) 2024-09-21T15:49:23.436 INFO:teuthology.orchestra.run.smithi089.stdout:1139 punch from 0x8dc04c2 to 0x8dc2068, (0x1ba6 bytes) 2024-09-21T15:49:23.436 INFO:teuthology.orchestra.run.smithi089.stdout:1140 trunc from 0xba705d5 to 0xc586 2024-09-21T15:49:23.436 INFO:teuthology.orchestra.run.smithi089.stdout:1141 trunc from 0xc586 to 0x5dfb1d3 2024-09-21T15:49:23.436 INFO:teuthology.orchestra.run.smithi089.stdout:1142 read 0x399435c thru 0x3997344 (0x2fe9 bytes) 2024-09-21T15:49:23.436 INFO:teuthology.orchestra.run.smithi089.stdout:1143 write 0x4b07940 thru 0x4b0bba6 (0x4267 bytes) 2024-09-21T15:49:23.436 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:23.431+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:23.437 INFO:teuthology.orchestra.run.smithi089.stdout:1145 read 0x254dbbe thru 0x255c2bf (0xe702 bytes) 2024-09-21T15:49:23.442 INFO:teuthology.orchestra.run.smithi089.stdout:1146 write 0x95fe6b5 thru 0x96031f9 (0x4b45 bytes) 2024-09-21T15:49:23.444 INFO:teuthology.orchestra.run.smithi089.stdout:1147 write 0x126e726 thru 0x12736d1 (0x4fac bytes) 2024-09-21T15:49:23.447 INFO:teuthology.orchestra.run.smithi089.stdout:1148 write 0x8a77a81 thru 0x8a82579 (0xaaf9 bytes) 2024-09-21T15:49:23.454 INFO:teuthology.orchestra.run.smithi089.stdout:1150 punch from 0x5af770d to 0x5b01c2c, (0xa51f bytes) 2024-09-21T15:49:23.467 INFO:teuthology.orchestra.run.smithi089.stdout:1151 write 0xe393e71 thru 0xe394e82 (0x1012 bytes) 2024-09-21T15:49:23.470 INFO:teuthology.orchestra.run.smithi089.stdout:1152 write 0x8494b30 thru 0x849ffe1 (0xb4b2 bytes) 2024-09-21T15:49:23.471 INFO:teuthology.orchestra.run.smithi089.stdout:1153 write 0x9f1eff8 thru 0x9f2268a (0x3693 bytes) 2024-09-21T15:49:23.480 INFO:teuthology.orchestra.run.smithi089.stdout:1154 trunc from 0xe394e83 to 0x9cf1c69 2024-09-21T15:49:23.490 INFO:teuthology.orchestra.run.smithi089.stdout:1155 read 0x3bddbf2 thru 0x3bec938 (0xed47 bytes) 2024-09-21T15:49:23.490 INFO:teuthology.orchestra.run.smithi089.stdout:1156 writesame 0x94f9eb8 thru 0x950263a data_size 0x1(0x8783 bytes) 2024-09-21T15:49:23.507 INFO:teuthology.orchestra.run.smithi089.stdout:1157 read 0x248008d thru 0x248f65f (0xf5d3 bytes) 2024-09-21T15:49:23.509 INFO:teuthology.orchestra.run.smithi089.stdout:1158 trunc from 0x9cf1c69 to 0x8817f03 2024-09-21T15:49:23.519 INFO:teuthology.orchestra.run.smithi089.stdout:1159 read 0x760437e thru 0x760b6c1 (0x7344 bytes) 2024-09-21T15:49:23.525 INFO:teuthology.orchestra.run.smithi089.stdout:1160 write 0x978f10b thru 0x979d461 (0xe357 bytes) 2024-09-21T15:49:23.526 INFO:teuthology.orchestra.run.smithi089.stdout:1162 writesame 0x7749912 thru 0x774d3cd data_size 0x2a(0x3abc bytes) 2024-09-21T15:49:23.527 INFO:teuthology.orchestra.run.smithi089.stdout:1163 write 0x348d19d thru 0x348f24f (0x20b3 bytes) 2024-09-21T15:49:23.530 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:23.534 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:23.532+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:23.537 INFO:teuthology.orchestra.run.smithi089.stdout:1167 trunc from 0x979d462 to 0x388259b 2024-09-21T15:49:23.548 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:23.548 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:49:23.548 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:49:23.548 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:23.553 INFO:teuthology.orchestra.run.smithi089.stdout:1168 write 0x38f5831 thru 0x38fa9b5 (0x5185 bytes) 2024-09-21T15:49:23.557 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:49:23.568 INFO:teuthology.orchestra.run.smithi089.stdout:1169 write 0x88d1226 thru 0x88db9b7 (0xa792 bytes) 2024-09-21T15:49:23.570 INFO:teuthology.orchestra.run.smithi089.stdout:1170 punch from 0x49032c7 to 0x4912b79, (0xf8b2 bytes) 2024-09-21T15:49:23.570 INFO:teuthology.orchestra.run.smithi089.stdout:1171 punch from 0x220d10b to 0x221231a, (0x520f bytes) 2024-09-21T15:49:23.570 INFO:teuthology.orchestra.run.smithi089.stdout:1172 read 0x5f759f9 thru 0x5f843ac (0xe9b4 bytes) 2024-09-21T15:49:23.570 INFO:teuthology.orchestra.run.smithi089.stdout:1173 read 0x20bd3fe thru 0x20bf051 (0x1c54 bytes) 2024-09-21T15:49:23.570 INFO:teuthology.orchestra.run.smithi089.stdout:1175 read 0x22a1eeb thru 0x22ae0e4 (0xc1fa bytes) 2024-09-21T15:49:23.570 INFO:teuthology.orchestra.run.smithi089.stdout:1176 trunc from 0x88db9b8 to 0x8dac4b8 2024-09-21T15:49:23.575 INFO:teuthology.orchestra.run.smithi089.stdout:1177 write 0x3d09d54 thru 0x3d0a012 (0x2bf bytes) 2024-09-21T15:49:23.576 INFO:teuthology.orchestra.run.smithi089.stdout:1179 read 0x79d1af9 thru 0x79d67ef (0x4cf7 bytes) 2024-09-21T15:49:23.577 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:23.577 INFO:teuthology.orchestra.run.smithi089.stdout:1181 read 0x8461ba5 thru 0x846afcc (0x9428 bytes) 2024-09-21T15:49:23.577 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:23.577 INFO:teuthology.orchestra.run.smithi089.stdout:1183 punch from 0x2aff337 to 0x2b04799, (0x5462 bytes) 2024-09-21T15:49:23.577 INFO:teuthology.orchestra.run.smithi089.stdout:1184 read 0x683d3c3 thru 0x683d816 (0x454 bytes) 2024-09-21T15:49:23.584 INFO:teuthology.orchestra.run.smithi089.stdout:1185 trunc from 0x8dac4b8 to 0x2be99a8 2024-09-21T15:49:23.615 INFO:teuthology.orchestra.run.smithi089.stdout:1186 write 0x7e6b2b5 thru 0x7e787aa (0xd4f6 bytes) 2024-09-21T15:49:23.618 INFO:teuthology.orchestra.run.smithi089.stdout:1187 write 0x12e4698 thru 0x12e5726 (0x108f bytes) 2024-09-21T15:49:23.620 INFO:teuthology.orchestra.run.smithi089.stdout:1190 punch from 0x77cba99 to 0x77d21f3, (0x675a bytes) 2024-09-21T15:49:23.620 INFO:teuthology.orchestra.run.smithi089.stdout:1191 read 0x1e07605 thru 0x1e11961 (0xa35d bytes) 2024-09-21T15:49:23.620 INFO:teuthology.orchestra.run.smithi089.stdout:1192 read 0x663f8fd thru 0x66422ef (0x29f3 bytes) 2024-09-21T15:49:23.624 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:49:23.629 INFO:teuthology.orchestra.run.smithi089.stdout:1193 write 0xaadcd40 thru 0xaae2af8 (0x5db9 bytes) 2024-09-21T15:49:23.632 INFO:teuthology.orchestra.run.smithi089.stdout:1194 read 0xaac7740 thru 0xaacc9a3 (0x5264 bytes) 2024-09-21T15:49:23.632 INFO:teuthology.orchestra.run.smithi089.stdout:1195 punch from 0x5362643 to 0x536403c, (0x19f9 bytes) 2024-09-21T15:49:23.632 INFO:teuthology.orchestra.run.smithi089.stdout:1196 read 0x3fada20 thru 0x3fb464d (0x6c2e bytes) 2024-09-21T15:49:23.632 INFO:teuthology.orchestra.run.smithi089.stdout:1197 punch from 0x5953b44 to 0x5959eac, (0x6368 bytes) 2024-09-21T15:49:23.632 INFO:teuthology.orchestra.run.smithi089.stdout:1198 writesame 0x979d54d thru 0x97a910f data_size 0x1(0xbbc3 bytes) 2024-09-21T15:49:23.635 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:23.633+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:23.655 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:23.664 INFO:teuthology.orchestra.run.smithi089.stdout:1200 write 0xd655561 thru 0xd660ea1 (0xb941 bytes) 2024-09-21T15:49:23.667 INFO:teuthology.orchestra.run.smithi089.stdout:1201 write 0x3d84bbc thru 0x3d939ac (0xedf1 bytes) 2024-09-21T15:49:23.668 INFO:teuthology.orchestra.run.smithi089.stdout:1203 read 0x69f3ec9 thru 0x6a00a10 (0xcb48 bytes) 2024-09-21T15:49:23.669 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:23.669 INFO:teuthology.orchestra.run.smithi089.stdout:1206 punch from 0x66aebfd to 0x66bb30f, (0xc712 bytes) 2024-09-21T15:49:23.669 INFO:teuthology.orchestra.run.smithi089.stdout:1207 write 0x2794ce2 thru 0x27a3899 (0xebb8 bytes) 2024-09-21T15:49:23.670 INFO:teuthology.orchestra.run.smithi089.stdout:1208 write 0x330d4f2 thru 0x331c399 (0xeea8 bytes) 2024-09-21T15:49:23.673 INFO:teuthology.orchestra.run.smithi089.stdout:1210 write 0x77ac03e thru 0x77ae123 (0x20e6 bytes) 2024-09-21T15:49:23.675 INFO:teuthology.orchestra.run.smithi089.stdout:1211 writesame 0x1c8aa55 thru 0x1c961c7 data_size 0x1(0xb773 bytes) 2024-09-21T15:49:23.678 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:23.678 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:49:23.678 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:49:23.678 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:49:23.678 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:23.687 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:49:23.699 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:23.699 INFO:teuthology.orchestra.run.smithi089.stdout:1214 writesame 0xc79c1d0 thru 0xc7a1657 data_size 0x14(0x5488 bytes) 2024-09-21T15:49:23.702 INFO:teuthology.orchestra.run.smithi089.stdout:1216 read 0x68289b thru 0x68fc23 (0xd389 bytes) 2024-09-21T15:49:23.710 INFO:teuthology.orchestra.run.smithi089.stdout:1217 trunc from 0xd660ea2 to 0x76255a3 2024-09-21T15:49:23.724 INFO:teuthology.orchestra.run.smithi089.stdout:1218 trunc from 0x76255a3 to 0xbd7f229 2024-09-21T15:49:23.731 INFO:teuthology.orchestra.run.smithi089.stdout:1219 write 0x1ddd80d thru 0x1ded666 (0xfe5a bytes) 2024-09-21T15:49:23.735 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:23.734+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:23.736 INFO:teuthology.orchestra.run.smithi089.stdout:1221 trunc from 0xbd7f229 to 0x9632e56 2024-09-21T15:49:23.746 INFO:teuthology.orchestra.run.smithi089.stdout:1222 read 0x8feab5 thru 0x902c0d (0x4159 bytes) 2024-09-21T15:49:23.749 INFO:teuthology.orchestra.run.smithi089.stdout:1226 trunc from 0x9632e56 to 0x7b4d527 2024-09-21T15:49:23.758 INFO:teuthology.orchestra.run.smithi089.stdout:1227 writesame 0x79ccbd2 thru 0x79dbfe9 data_size 0x4(0xf418 bytes) 2024-09-21T15:49:23.761 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-09-21T15:49:23.761 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:49:23.764 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T15:49:23.773 INFO:teuthology.orchestra.run.smithi089.stdout:1228 write 0x9565a7c thru 0x9574cbc (0xf241 bytes) 2024-09-21T15:49:23.782 INFO:teuthology.orchestra.run.smithi089.stdout:1229 trunc from 0x9574cbd to 0xe1d03f8 2024-09-21T15:49:23.785 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:49:23.786 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:49:23.787 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:23.787 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:23.787 INFO:teuthology.orchestra.run.smithi089.stdout:1233 writesame 0xa4593e1 thru 0xa45db78 data_size 0x1d(0x4798 bytes) 2024-09-21T15:49:23.789 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:23.790 INFO:teuthology.orchestra.run.smithi089.stdout:1235 writesame 0xa42212 thru 0xa4fb77 data_size 0x1(0xd966 bytes) 2024-09-21T15:49:23.803 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:49:23.803 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:49:23.815 INFO:teuthology.orchestra.run.smithi089.stdout:1236 read 0x2c92c79 thru 0x2c97fcb (0x5353 bytes) 2024-09-21T15:49:23.815 INFO:teuthology.orchestra.run.smithi089.stdout:1238 write 0xbbeea13 thru 0xbbfb1de (0xc7cc bytes) 2024-09-21T15:49:23.818 INFO:teuthology.orchestra.run.smithi089.stdout:1239 writesame 0x3836ebc thru 0x383eec2 data_size 0x4b(0x8007 bytes) 2024-09-21T15:49:23.819 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:23.819 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:23.820 INFO:teuthology.orchestra.run.smithi089.stdout:1240 punch from 0x14a62db to 0x14aaf84, (0x4ca9 bytes) 2024-09-21T15:49:23.820 INFO:teuthology.orchestra.run.smithi089.stdout:1241 write 0xc18318 thru 0xc27826 (0xf50f bytes) 2024-09-21T15:49:23.824 INFO:teuthology.orchestra.run.smithi089.stdout:1244 punch from 0xc199775 to 0xc1a2da7, (0x9632 bytes) 2024-09-21T15:49:23.824 INFO:teuthology.orchestra.run.smithi089.stdout:1246 write 0x1c3cd19 thru 0x1c42bc7 (0x5eaf bytes) 2024-09-21T15:49:23.829 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:49:23.834 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:49:23.834 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:49:23.835 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:23.834+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:23.837 INFO:teuthology.orchestra.run.smithi089.stdout:1247 trunc from 0xe1d03f8 to 0x4d4d59d 2024-09-21T15:49:23.852 INFO:teuthology.orchestra.run.smithi089.stdout:1248 punch from 0x3718683 to 0x372616e, (0xdaeb bytes) 2024-09-21T15:49:23.852 INFO:teuthology.orchestra.run.smithi089.stdout:1249 read 0x2cc7239 thru 0x2ccd546 (0x630e bytes) 2024-09-21T15:49:23.852 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:23.855 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:49:23.856 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:49:23.863 INFO:teuthology.orchestra.run.smithi089.stdout:1251 write 0x9d9feca thru 0x9da2b62 (0x2c99 bytes) 2024-09-21T15:49:23.867 INFO:teuthology.orchestra.run.smithi089.stdout:1252 punch from 0x91a266d to 0x91aec82, (0xc615 bytes) 2024-09-21T15:49:23.867 INFO:teuthology.orchestra.run.smithi089.stdout:1253 read 0x60c78a4 thru 0x60d497f (0xd0dc bytes) 2024-09-21T15:49:23.869 INFO:teuthology.orchestra.run.smithi089.stdout:1254 trunc from 0x9da2b63 to 0x836a34e 2024-09-21T15:49:23.877 INFO:teuthology.orchestra.run.smithi089.stdout:1255 write 0x5b921d9 thru 0x5b9286f (0x697 bytes) 2024-09-21T15:49:23.880 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:23.887 INFO:teuthology.orchestra.run.smithi089.stdout:1257 trunc from 0x836a34e to 0x2c983ee 2024-09-21T15:49:23.887 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:49:23.887 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:49:23.913 INFO:teuthology.orchestra.run.smithi089.stdout:1258 trunc from 0x2c983ee to 0xb24fda3 2024-09-21T15:49:23.914 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:49:23.914 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:49:23.921 INFO:teuthology.orchestra.run.smithi089.stdout:1260 trunc from 0xb24fda3 to 0x85fbaa8 2024-09-21T15:49:23.930 INFO:teuthology.orchestra.run.smithi089.stdout:1261 writesame 0x1e2f2ea thru 0x1e2f69f data_size 0x13(0x3b6 bytes) 2024-09-21T15:49:23.931 INFO:teuthology.orchestra.run.smithi089.stdout:1262 read 0x714e6bc thru 0x715afbd (0xc902 bytes) 2024-09-21T15:49:23.936 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:23.935+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:23.945 INFO:teuthology.orchestra.run.smithi089.stdout:1263 write 0xde03a86 thru 0xde0f4c9 (0xba44 bytes) 2024-09-21T15:49:23.946 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:49:23.946 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:49:23.950 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:23.950 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:23.950 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:23.950 INFO:teuthology.orchestra.run.smithi089.stdout:1270 punch from 0x606e03b to 0x6076973, (0x8938 bytes) 2024-09-21T15:49:23.950 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:23.950 INFO:teuthology.orchestra.run.smithi089.stdout:1273 read 0x4d55b49 thru 0x4d5c335 (0x67ed bytes) 2024-09-21T15:49:23.950 INFO:teuthology.orchestra.run.smithi089.stdout:1274 read 0x6888fdc thru 0x688a6ac (0x16d1 bytes) 2024-09-21T15:49:23.950 INFO:teuthology.orchestra.run.smithi089.stdout:1275 read 0x25d34b5 thru 0x25d6f9b (0x3ae7 bytes) 2024-09-21T15:49:23.951 INFO:teuthology.orchestra.run.smithi089.stdout:1276 writesame 0x701c790 thru 0x702966b data_size 0x12(0xcedc bytes) 2024-09-21T15:49:23.958 INFO:teuthology.orchestra.run.smithi089.stdout:1278 writesame 0xe26e24b thru 0xe26eb08 data_size 0x3(0x8be bytes) 2024-09-21T15:49:23.958 INFO:teuthology.orchestra.run.smithi089.stdout:1279 read 0xce3b20a thru 0xce3fa50 (0x4847 bytes) 2024-09-21T15:49:23.958 INFO:teuthology.orchestra.run.smithi089.stdout:1280 writesame 0x5f7bcd8 thru 0x5f89317 data_size 0x20(0xd640 bytes) 2024-09-21T15:49:23.959 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:23.959 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:23.961 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:23.962 INFO:teuthology.orchestra.run.smithi089.stdout:1282 read 0xe177a38 thru 0xe184ca4 (0xd26d bytes) 2024-09-21T15:49:23.962 INFO:teuthology.orchestra.run.smithi089.stdout:1283 read 0x31beb59 thru 0x31c89bf (0x9e67 bytes) 2024-09-21T15:49:23.962 INFO:teuthology.orchestra.run.smithi089.stdout:1284 read 0x95f8bff thru 0x9603993 (0xad95 bytes) 2024-09-21T15:49:23.962 INFO:teuthology.orchestra.run.smithi089.stdout:1285 read 0x73f61e2 thru 0x73f87ad (0x25cc bytes) 2024-09-21T15:49:23.962 INFO:teuthology.orchestra.run.smithi089.stdout:1286 punch from 0x9acb9ad to 0x9acdacc, (0x211f bytes) 2024-09-21T15:49:23.962 INFO:teuthology.orchestra.run.smithi089.stdout:1287 read 0xa529c48 thru 0xa52a765 (0xb1e bytes) 2024-09-21T15:49:23.971 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:49:23.971 INFO:teuthology.orchestra.run.smithi089.stdout:1288 trunc from 0xe26eb09 to 0x71feb87 2024-09-21T15:49:23.982 INFO:teuthology.orchestra.run.smithi089.stdout:1289 flatten 2024-09-21T15:49:23.987 INFO:teuthology.orchestra.run.smithi089.stdout:1294 read 0x4ec08a0 thru 0x4ec0ad0 (0x231 bytes) 2024-09-21T15:49:23.987 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:23.990 INFO:teuthology.orchestra.run.smithi089.stdout:1297 trunc from 0x71feb87 to 0xa10311b 2024-09-21T15:49:23.995 INFO:teuthology.orchestra.run.smithi089.stdout:1298 read 0x269d317 thru 0x26a156b (0x4255 bytes) 2024-09-21T15:49:23.996 INFO:teuthology.orchestra.run.smithi089.stdout:1300 punch from 0x99dc2ff to 0x99e0e0e, (0x4b0f bytes) 2024-09-21T15:49:23.997 INFO:teuthology.orchestra.run.smithi089.stdout:1302 trunc from 0xa10311b to 0xb4ba675 2024-09-21T15:49:24.002 INFO:teuthology.orchestra.run.smithi089.stdout:1303 write 0x4ae17c7 thru 0x4aee5bc (0xcdf6 bytes) 2024-09-21T15:49:24.019 INFO:teuthology.orchestra.run.smithi089.stdout:1304 trunc from 0xb4ba675 to 0x154a323 2024-09-21T15:49:24.037 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:24.036+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:24.052 INFO:teuthology.orchestra.run.smithi089.stdout:1305 write 0x294ed43 thru 0x295c421 (0xd6df bytes) 2024-09-21T15:49:24.055 INFO:teuthology.orchestra.run.smithi089.stdout:1306 read 0xf2a4cc thru 0xf2e837 (0x436c bytes) 2024-09-21T15:49:24.055 INFO:teuthology.orchestra.run.smithi089.stdout:1307 read 0x1353ead thru 0x135dc26 (0x9d7a bytes) 2024-09-21T15:49:24.056 INFO:teuthology.orchestra.run.smithi089.stdout:1308 writesame 0x1cafd63 thru 0x1cba685 data_size 0x33(0xa923 bytes) 2024-09-21T15:49:24.062 INFO:teuthology.orchestra.run.smithi089.stdout:1309 read 0xa274aa thru 0xa339e6 (0xc53d bytes) 2024-09-21T15:49:24.062 INFO:teuthology.orchestra.run.smithi089.stdout:1310 punch from 0x606fec to 0x612a91, (0xbaa5 bytes) 2024-09-21T15:49:24.062 INFO:teuthology.orchestra.run.smithi089.stdout:1312 read 0x2165002 thru 0x216a7ce (0x57cd bytes) 2024-09-21T15:49:24.062 INFO:teuthology.orchestra.run.smithi089.stdout:1313 read 0x13f19b7 thru 0x13f8579 (0x6bc3 bytes) 2024-09-21T15:49:24.079 INFO:teuthology.orchestra.run.smithi089.stdout:1314 writesame 0xa6bdfa3 thru 0xa6c3ef4 data_size 0x62(0x5f52 bytes) 2024-09-21T15:49:24.079 INFO:teuthology.orchestra.run.smithi089.stdout:1316 writesame 0x8f5229b thru 0x8f5a3f6 data_size 0x22(0x815c bytes) 2024-09-21T15:49:24.080 INFO:teuthology.orchestra.run.smithi089.stdout:1317 read 0x864b8f thru 0x869b3b (0x4fad bytes) 2024-09-21T15:49:24.088 INFO:teuthology.orchestra.run.smithi089.stdout:1318 write 0xbf2e42a thru 0xbf333a2 (0x4f79 bytes) 2024-09-21T15:49:24.091 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:24.091 INFO:teuthology.orchestra.run.smithi089.stdout:1322 write 0x8c48f6d thru 0x8c57e0d (0xeea1 bytes) 2024-09-21T15:49:24.093 INFO:teuthology.orchestra.run.smithi089.stdout:1323 punch from 0x6a48751 to 0x6a4ccf8, (0x45a7 bytes) 2024-09-21T15:49:24.093 INFO:teuthology.orchestra.run.smithi089.stdout:1324 read 0x6d532c6 thru 0x6d59dad (0x6ae8 bytes) 2024-09-21T15:49:24.097 INFO:teuthology.orchestra.run.smithi089.stdout:1326 trunc from 0xbf333a3 to 0x7c59d19 2024-09-21T15:49:24.105 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:24.105 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:24.106 INFO:teuthology.orchestra.run.smithi089.stdout:1328 read 0x497fcd1 thru 0x4983468 (0x3798 bytes) 2024-09-21T15:49:24.106 INFO:teuthology.orchestra.run.smithi089.stdout:1330 write 0x601215f thru 0x6017b34 (0x59d6 bytes) 2024-09-21T15:49:24.109 INFO:teuthology.orchestra.run.smithi089.stdout:1331 read 0x4f1217f thru 0x4f184e9 (0x636b bytes) 2024-09-21T15:49:24.110 INFO:teuthology.orchestra.run.smithi089.stdout:1332 punch from 0x617ad20 to 0x61842d1, (0x95b1 bytes) 2024-09-21T15:49:24.110 INFO:teuthology.orchestra.run.smithi089.stdout:1333 read 0x5070a89 thru 0x507a32f (0x98a7 bytes) 2024-09-21T15:49:24.110 INFO:teuthology.orchestra.run.smithi089.stdout:1334 punch from 0x68d49ef to 0x68dd138, (0x8749 bytes) 2024-09-21T15:49:24.110 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:24.120 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:49:24.124 INFO:teuthology.orchestra.run.smithi089.stdout:1339 write 0xe5866bb thru 0xe58c7de (0x6124 bytes) 2024-09-21T15:49:24.126 INFO:teuthology.orchestra.run.smithi089.stdout:1340 punch from 0x77547d0 to 0x7757b0b, (0x333b bytes) 2024-09-21T15:49:24.126 INFO:teuthology.orchestra.run.smithi089.stdout:1341 read 0xb3c9457 thru 0xb3d7665 (0xe20f bytes) 2024-09-21T15:49:24.138 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:24.136+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:24.143 INFO:teuthology.orchestra.run.smithi089.stdout:1343 trunc from 0xe58c7df to 0xc07b11 2024-09-21T15:49:24.170 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:24.183 INFO:teuthology.orchestra.run.smithi089.stdout:1345 trunc from 0xc07b11 to 0xb1a91e7 2024-09-21T15:49:24.189 INFO:teuthology.orchestra.run.smithi089.stdout:1346 write 0xa2bc560 thru 0xa2c2965 (0x6406 bytes) 2024-09-21T15:49:24.195 INFO:teuthology.orchestra.run.smithi089.stdout:1347 writesame 0x1d7c947 thru 0x1d89e49 data_size 0x9(0xd503 bytes) 2024-09-21T15:49:24.200 INFO:teuthology.orchestra.run.smithi089.stdout:1348 read 0x1be68ce thru 0x1bf159d (0xacd0 bytes) 2024-09-21T15:49:24.205 INFO:teuthology.orchestra.run.smithi089.stdout:1351 writesame 0xb8b2bf7 thru 0xb8b489a data_size 0x34(0x1ca4 bytes) 2024-09-21T15:49:24.208 INFO:teuthology.orchestra.run.smithi089.stdout:1355 trunc from 0xb8b489b to 0x9cc6a04 2024-09-21T15:49:24.215 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:24.216 INFO:teuthology.orchestra.run.smithi089.stdout:1358 punch from 0x63a99ad to 0x63b0c83, (0x72d6 bytes) 2024-09-21T15:49:24.216 INFO:teuthology.orchestra.run.smithi089.stdout:1359 read 0x915009b thru 0x91546f7 (0x465d bytes) 2024-09-21T15:49:24.216 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:24.216 INFO:teuthology.orchestra.run.smithi089.stdout:1361 write 0x327265 thru 0x32be16 (0x4bb2 bytes) 2024-09-21T15:49:24.217 INFO:teuthology.orchestra.run.smithi089.stdout:1362 trunc from 0x9cc6a04 to 0x9ad0e95 2024-09-21T15:49:24.225 INFO:teuthology.orchestra.run.smithi089.stdout:1363 write 0x40c31be thru 0x40c736f (0x41b2 bytes) 2024-09-21T15:49:24.231 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:24.231 INFO:teuthology.orchestra.run.smithi089.stdout:1365 punch from 0x414101b to 0x414bdcc, (0xadb1 bytes) 2024-09-21T15:49:24.231 INFO:teuthology.orchestra.run.smithi089.stdout:1366 read 0x16d0bb3 thru 0x16d51be (0x460c bytes) 2024-09-21T15:49:24.231 INFO:teuthology.orchestra.run.smithi089.stdout:1368 read 0x57ca6f4 thru 0x57d2f02 (0x880f bytes) 2024-09-21T15:49:24.231 INFO:teuthology.orchestra.run.smithi089.stdout:1371 punch from 0x78bd26a to 0x78c7120, (0x9eb6 bytes) 2024-09-21T15:49:24.231 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:24.238 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:24.237+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:24.239 INFO:teuthology.orchestra.run.smithi089.stdout:1373 write 0xc7d722d thru 0xc7df471 (0x8245 bytes) 2024-09-21T15:49:24.239 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:24.239 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:24.241 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:24.241 INFO:teuthology.orchestra.run.smithi089.stdout:1375 writesame 0x59155a6 thru 0x5916745 data_size 0x20(0x11a0 bytes) 2024-09-21T15:49:24.246 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:24.247 INFO:teuthology.orchestra.run.smithi089.stdout:1378 write 0x4c4099 thru 0x4cad14 (0x6c7c bytes) 2024-09-21T15:49:24.249 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:49:24.249 INFO:teuthology.orchestra.run.smithi089.stdout:1379 read 0x1e41938 thru 0x1e48f75 (0x763e bytes) 2024-09-21T15:49:24.249 INFO:teuthology.orchestra.run.smithi089.stdout:1380 write 0xa864d0a thru 0xa86e251 (0x9548 bytes) 2024-09-21T15:49:24.251 INFO:teuthology.orchestra.run.smithi089.stdout:1381 read 0x7bbf5a8 thru 0x7bcd0bc (0xdb15 bytes) 2024-09-21T15:49:24.255 INFO:teuthology.orchestra.run.smithi089.stdout:1382 trunc from 0xc7df472 to 0x99cd306 2024-09-21T15:49:24.270 INFO:teuthology.orchestra.run.smithi089.stdout:1383 punch from 0x4b31ee3 to 0x4b3a6b0, (0x87cd bytes) 2024-09-21T15:49:24.277 INFO:teuthology.orchestra.run.smithi089.stdout:1385 write 0xa9ef736 thru 0xa9fc803 (0xd0ce bytes) 2024-09-21T15:49:24.279 INFO:teuthology.orchestra.run.smithi089.stdout:1388 read 0x6be7705 thru 0x6bed530 (0x5e2c bytes) 2024-09-21T15:49:24.279 INFO:teuthology.orchestra.run.smithi089.stdout:1389 write 0xa95808c thru 0xa9607e0 (0x8755 bytes) 2024-09-21T15:49:24.281 INFO:teuthology.orchestra.run.smithi089.stdout:1390 writesame 0x544b21e thru 0x5457b9d data_size 0x20(0xc980 bytes) 2024-09-21T15:49:24.283 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:24.283 INFO:teuthology.orchestra.run.smithi089.stdout:1392 write 0x20e8782 thru 0x20f1351 (0x8bd0 bytes) 2024-09-21T15:49:24.287 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:24.287 INFO:teuthology.orchestra.run.smithi089.stdout:1394 write 0x886382c thru 0x886862a (0x4dff bytes) 2024-09-21T15:49:24.289 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:24.289 INFO:teuthology.orchestra.run.smithi089.stdout:1397 punch from 0x9344ceb to 0x9349346, (0x465b bytes) 2024-09-21T15:49:24.289 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:24.298 INFO:teuthology.orchestra.run.smithi089.stdout:1401 writesame 0xc15bc06 thru 0xc1647d4 data_size 0x1(0x8bcf bytes) 2024-09-21T15:49:24.299 INFO:teuthology.orchestra.run.smithi089.stdout:1402 write 0x9592e7e thru 0x9596417 (0x359a bytes) 2024-09-21T15:49:24.300 INFO:teuthology.orchestra.run.smithi089.stdout:1404 read 0xf9a8f7 thru 0xfa2432 (0x7b3c bytes) 2024-09-21T15:49:24.300 INFO:teuthology.orchestra.run.smithi089.stdout:1406 read 0xa141adc thru 0xa14368d (0x1bb2 bytes) 2024-09-21T15:49:24.301 INFO:teuthology.orchestra.run.smithi089.stdout:1407 punch from 0xe35f78 to 0xe390bb, (0x3143 bytes) 2024-09-21T15:49:24.301 INFO:teuthology.orchestra.run.smithi089.stdout:1408 read 0x58e2801 thru 0x58eda53 (0xb253 bytes) 2024-09-21T15:49:24.301 INFO:teuthology.orchestra.run.smithi089.stdout:1409 writesame 0x7dd19c8 thru 0x7ddaaf8 data_size 0x1f(0x9131 bytes) 2024-09-21T15:49:24.303 INFO:teuthology.orchestra.run.smithi089.stdout:1411 write 0x59909ca thru 0x599165e (0xc95 bytes) 2024-09-21T15:49:24.304 INFO:teuthology.orchestra.run.smithi089.stdout:1412 punch from 0x6bc5913 to 0x6bcdf7b, (0x8668 bytes) 2024-09-21T15:49:24.316 INFO:teuthology.orchestra.run.smithi089.stdout:1413 trunc from 0xc1647d5 to 0x211ef4d 2024-09-21T15:49:24.336 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:24.339 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:24.338+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:24.342 INFO:teuthology.orchestra.run.smithi089.stdout:1415 trunc from 0x211ef4d to 0x70f5d94 2024-09-21T15:49:24.347 INFO:teuthology.orchestra.run.smithi089.stdout:1416 punch from 0x6f343ea to 0x6f3e8aa, (0xa4c0 bytes) 2024-09-21T15:49:24.353 INFO:teuthology.orchestra.run.smithi089.stdout:1419 trunc from 0x70f5d94 to 0xbf96f6d 2024-09-21T15:49:24.358 INFO:teuthology.orchestra.run.smithi089.stdout:1420 read 0x6a3a42f thru 0x6a3d645 (0x3217 bytes) 2024-09-21T15:49:24.366 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:24.366 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:24.366 INFO:teuthology.orchestra.run.smithi089.stdout:1421 write 0xe3c3431 thru 0xe3ccc59 (0x9829 bytes) 2024-09-21T15:49:24.368 INFO:teuthology.orchestra.run.smithi089.stdout:1423 read 0x301c208 thru 0x3023d95 (0x7b8e bytes) 2024-09-21T15:49:24.369 INFO:teuthology.orchestra.run.smithi089.stdout:1424 punch from 0x3c2e9e3 to 0x3c3bcb1, (0xd2ce bytes) 2024-09-21T15:49:24.369 INFO:teuthology.orchestra.run.smithi089.stdout:1426 punch from 0x46fb1fd to 0x46fdfb0, (0x2db3 bytes) 2024-09-21T15:49:24.369 INFO:teuthology.orchestra.run.smithi089.stdout:1427 read 0x61131aa thru 0x611e9ca (0xb821 bytes) 2024-09-21T15:49:24.369 INFO:teuthology.orchestra.run.smithi089.stdout:1428 write 0x1cb6435 thru 0x1cb8aa3 (0x266f bytes) 2024-09-21T15:49:24.370 INFO:teuthology.orchestra.run.smithi089.stdout:1429 writesame 0x3975c1f thru 0x398018f data_size 0x1(0xa571 bytes) 2024-09-21T15:49:24.374 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:49:24.393 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:24.409 INFO:teuthology.orchestra.run.smithi089.stdout:1433 trunc from 0xe3ccc5a to 0x73ce30 2024-09-21T15:49:24.428 INFO:teuthology.orchestra.run.smithi089.stdout:1434 punch from 0x206ab8 to 0x20c0b6, (0x55fe bytes) 2024-09-21T15:49:24.431 INFO:teuthology.orchestra.run.smithi089.stdout:1435 trunc from 0x73ce30 to 0x35ed8c2 2024-09-21T15:49:24.439 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:24.441 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:24.440+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:24.459 INFO:teuthology.orchestra.run.smithi089.stdout:1438 write 0xe13de6e thru 0xe13ebc0 (0xd53 bytes) 2024-09-21T15:49:24.465 INFO:teuthology.orchestra.run.smithi089.stdout:1439 write 0x12c9756 thru 0x12d3ba7 (0xa452 bytes) 2024-09-21T15:49:24.473 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:24.477 INFO:teuthology.orchestra.run.smithi089.stdout:1441 trunc from 0xe13ebc1 to 0xa90923a 2024-09-21T15:49:24.499 INFO:teuthology.orchestra.run.smithi089.stdout:1442 writesame 0xabd9641 thru 0xabdb46d data_size 0x5(0x1e2d bytes) 2024-09-21T15:49:24.508 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:24.508 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:24.510 INFO:teuthology.orchestra.run.smithi089.stdout:1443 trunc from 0xabdb46e to 0x1790b2f 2024-09-21T15:49:24.519 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:49:24.531 INFO:teuthology.orchestra.run.smithi089.stdout:1444 writesame 0x4f94f75 thru 0x4f9d0f0 data_size 0x2(0x817c bytes) 2024-09-21T15:49:24.539 INFO:teuthology.orchestra.run.smithi089.stdout:1445 trunc from 0x4f9d0f1 to 0xc2ad873 2024-09-21T15:49:24.542 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:24.540+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:24.547 INFO:teuthology.orchestra.run.smithi089.stdout:1446 write 0x2526211 thru 0x25352c9 (0xf0b9 bytes) 2024-09-21T15:49:24.552 INFO:teuthology.orchestra.run.smithi089.stdout:1448 read 0x57b5096 thru 0x57b68d3 (0x183e bytes) 2024-09-21T15:49:24.552 INFO:teuthology.orchestra.run.smithi089.stdout:1449 write 0x7c0c778 thru 0x7c15ba4 (0x942d bytes) 2024-09-21T15:49:24.558 INFO:teuthology.orchestra.run.smithi089.stdout:1450 trunc from 0xc2ad873 to 0xa254376 2024-09-21T15:49:24.564 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:24.564 INFO:teuthology.orchestra.run.smithi089.stdout:1452 write 0x9019e04 thru 0x90285d6 (0xe7d3 bytes) 2024-09-21T15:49:24.569 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:24.570 INFO:teuthology.orchestra.run.smithi089.stdout:1455 read 0x38cbe thru 0x3d739 (0x4a7c bytes) 2024-09-21T15:49:24.570 INFO:teuthology.orchestra.run.smithi089.stdout:1456 write 0x2420898 thru 0x2428521 (0x7c8a bytes) 2024-09-21T15:49:24.571 INFO:teuthology.orchestra.run.smithi089.stdout:1457 write 0xfd9ae1 thru 0xfe319a (0x96ba bytes) 2024-09-21T15:49:24.582 INFO:teuthology.orchestra.run.smithi089.stdout:1459 trunc from 0xa254376 to 0x2018752 2024-09-21T15:49:24.610 INFO:teuthology.orchestra.run.smithi089.stdout:1460 writesame 0xd47d51f thru 0xd4893aa data_size 0x5a(0xbe8c bytes) 2024-09-21T15:49:24.610 INFO:teuthology.orchestra.run.smithi089.stdout:1461 read 0x9009112 thru 0x900ea1c (0x590b bytes) 2024-09-21T15:49:24.610 INFO:teuthology.orchestra.run.smithi089.stdout:1462 read 0x80d165a thru 0x80e0905 (0xf2ac bytes) 2024-09-21T15:49:24.610 INFO:teuthology.orchestra.run.smithi089.stdout:1463 write 0x24524e2 thru 0x24563c2 (0x3ee1 bytes) 2024-09-21T15:49:24.612 INFO:teuthology.orchestra.run.smithi089.stdout:1464 write 0x7614335 thru 0x761ec51 (0xa91d bytes) 2024-09-21T15:49:24.614 INFO:teuthology.orchestra.run.smithi089.stdout:1465 punch from 0x1fad687 to 0x1fbb832, (0xe1ab bytes) 2024-09-21T15:49:24.614 INFO:teuthology.orchestra.run.smithi089.stdout:1466 read 0x87ea34 thru 0x880087 (0x1654 bytes) 2024-09-21T15:49:24.614 INFO:teuthology.orchestra.run.smithi089.stdout:1467 read 0x49cd8cc thru 0x49cef32 (0x1667 bytes) 2024-09-21T15:49:24.614 INFO:teuthology.orchestra.run.smithi089.stdout:1468 writesame 0x988ad94 thru 0x98952a9 data_size 0x16(0xa516 bytes) 2024-09-21T15:49:24.620 INFO:teuthology.orchestra.run.smithi089.stdout:1471 writesame 0x524132e thru 0x52456c0 data_size 0x1(0x4393 bytes) 2024-09-21T15:49:24.632 INFO:teuthology.orchestra.run.smithi089.stdout:1473 punch from 0x8850840 to 0x8859d00, (0x94c0 bytes) 2024-09-21T15:49:24.643 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:24.641+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:24.643 INFO:teuthology.orchestra.run.smithi089.stdout:1474 trunc from 0xd4893ab to 0x3f609af 2024-09-21T15:49:24.646 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:24.646 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:24.655 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:49:24.664 INFO:teuthology.orchestra.run.smithi089.stdout:1475 trunc from 0x3f609af to 0x4e47b7b 2024-09-21T15:49:24.681 INFO:teuthology.orchestra.run.smithi089.stdout:1476 trunc from 0x4e47b7b to 0xd9b7784 2024-09-21T15:49:24.692 INFO:teuthology.orchestra.run.smithi089.stdout:1477 write 0x625e9f3 thru 0x6268ec0 (0xa4ce bytes) 2024-09-21T15:49:24.698 INFO:teuthology.orchestra.run.smithi089.stdout:1478 write 0x8decae2 thru 0x8df696e (0x9e8d bytes) 2024-09-21T15:49:24.704 INFO:teuthology.orchestra.run.smithi089.stdout:1479 write 0x1158c53 thru 0x1161bdd (0x8f8b bytes) 2024-09-21T15:49:24.706 INFO:teuthology.orchestra.run.smithi089.stdout:1480 read 0x493cb32 thru 0x494a11f (0xd5ee bytes) 2024-09-21T15:49:24.706 INFO:teuthology.orchestra.run.smithi089.stdout:1481 read 0xa4673ec thru 0xa471935 (0xa54a bytes) 2024-09-21T15:49:24.706 INFO:teuthology.orchestra.run.smithi089.stdout:1482 writesame 0x5c20fa4 thru 0x5c2e633 data_size 0x8(0xd690 bytes) 2024-09-21T15:49:24.711 INFO:teuthology.orchestra.run.smithi089.stdout:1484 read 0x192cdb7 thru 0x193577f (0x89c9 bytes) 2024-09-21T15:49:24.711 INFO:teuthology.orchestra.run.smithi089.stdout:1485 read 0x10f3190 thru 0x10f50ad (0x1f1e bytes) 2024-09-21T15:49:24.712 INFO:teuthology.orchestra.run.smithi089.stdout:1488 trunc from 0xd9b7784 to 0xd0ecd53 2024-09-21T15:49:24.722 INFO:teuthology.orchestra.run.smithi089.stdout:1489 read 0x4c4a69d thru 0x4c530ea (0x8a4e bytes) 2024-09-21T15:49:24.732 INFO:teuthology.orchestra.run.smithi089.stdout:1491 trunc from 0xd0ecd53 to 0x4c23fd7 2024-09-21T15:49:24.743 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:24.742+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:24.755 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:24.760 INFO:teuthology.orchestra.run.smithi089.stdout:1494 write 0x4f3aa86 thru 0x4f49c8e (0xf209 bytes) 2024-09-21T15:49:24.769 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is unset 2024-09-21T15:49:24.769 INFO:teuthology.orchestra.run.smithi089.stdout:1495 trunc from 0x4f49c8f to 0x3f389f4 2024-09-21T15:49:24.781 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:24.781 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:24.786 INFO:teuthology.orchestra.run.smithi089.stdout:1496 read 0x3e131e7 thru 0x3e194be (0x62d8 bytes) 2024-09-21T15:49:24.786 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:24.786 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:24.789 INFO:teuthology.orchestra.run.smithi089.stdout:1499 trunc from 0x3f389f4 to 0x60f1906 2024-09-21T15:49:24.790 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:49:24.796 INFO:teuthology.orchestra.run.smithi089.stdout:1500 read 0x12a72bf thru 0x12aff74 (0x8cb6 bytes) 2024-09-21T15:49:24.803 INFO:teuthology.orchestra.run.smithi089.stdout:1501 write 0x68558f6 thru 0x685df37 (0x8642 bytes) 2024-09-21T15:49:24.809 INFO:teuthology.orchestra.run.smithi089.stdout:1503 trunc from 0x685df38 to 0x3ffd75c 2024-09-21T15:49:24.818 INFO:teuthology.orchestra.run.smithi089.stdout:1504 writesame 0x179bf4e thru 0x17a1c21 data_size 0x1a(0x5cd4 bytes) 2024-09-21T15:49:24.820 INFO:teuthology.orchestra.run.smithi089.stdout:truncating to largest ever: 0xe7928f6 2024-09-21T15:49:24.833 INFO:teuthology.orchestra.run.smithi089.stdout:1505 trunc from 0x3ffd75c to 0xe7928f6 2024-09-21T15:49:24.839 INFO:teuthology.orchestra.run.smithi089.stdout:1507 write 0x271a7c5 thru 0x2720849 (0x6085 bytes) 2024-09-21T15:49:24.840 INFO:teuthology.orchestra.run.smithi089.stdout:1508 read 0x773c28d thru 0x774479a (0x850e bytes) 2024-09-21T15:49:24.841 INFO:teuthology.orchestra.run.smithi089.stdout:1509 read 0x3151636 thru 0x315ccf7 (0xb6c2 bytes) 2024-09-21T15:49:24.841 INFO:teuthology.orchestra.run.smithi089.stdout:1510 punch from 0x40b5687 to 0x40c13a8, (0xbd21 bytes) 2024-09-21T15:49:24.844 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:24.843+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:24.844 INFO:teuthology.orchestra.run.smithi089.stdout:1511 trunc from 0xe7928f6 to 0xbb04b92 2024-09-21T15:49:24.857 INFO:teuthology.orchestra.run.smithi089.stdout:1512 write 0xe0c0d68 thru 0xe0c1722 (0x9bb bytes) 2024-09-21T15:49:24.859 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:24.860 INFO:teuthology.orchestra.run.smithi089.stdout:1514 read 0xca627db thru 0xca6dd4f (0xb575 bytes) 2024-09-21T15:49:24.873 INFO:teuthology.orchestra.run.smithi089.stdout:1515 trunc from 0xe0c1723 to 0x2e74150 2024-09-21T15:49:24.890 INFO:teuthology.orchestra.run.smithi089.stdout:1516 trunc from 0x2e74150 to 0x1825646 2024-09-21T15:49:24.901 INFO:teuthology.orchestra.run.smithi089.stdout:1519 punch from 0x135ea96 to 0x1366f54, (0x84be bytes) 2024-09-21T15:49:24.911 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:24.911 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:24.915 INFO:teuthology.orchestra.run.smithi089.stdout:1520 writesame 0x94c9008 thru 0x94d1b4a data_size 0xb(0x8b43 bytes) 2024-09-21T15:49:24.916 INFO:teuthology.orchestra.run.smithi089.stdout:1521 read 0x9042980 thru 0x905164b (0xeccc bytes) 2024-09-21T15:49:24.921 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:49:24.924 INFO:teuthology.orchestra.run.smithi089.stdout:1522 writesame 0xc5f6203 thru 0xc601b24 data_size 0x9(0xb922 bytes) 2024-09-21T15:49:24.924 INFO:teuthology.orchestra.run.smithi089.stdout:1523 read 0xb406696 thru 0xb409d95 (0x3700 bytes) 2024-09-21T15:49:24.924 INFO:teuthology.orchestra.run.smithi089.stdout:1524 write 0x36fb196 thru 0x37090a9 (0xdf14 bytes) 2024-09-21T15:49:24.941 INFO:teuthology.orchestra.run.smithi089.stdout:1525 trunc from 0xc601b25 to 0x49c22e 2024-09-21T15:49:24.945 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:24.943+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:24.954 INFO:teuthology.orchestra.run.smithi089.stdout:1526 read 0xe3ce6 thru 0xe6afe (0x2e19 bytes) 2024-09-21T15:49:24.977 INFO:teuthology.orchestra.run.smithi089.stdout:1527 write 0xe895494 thru 0xe8a0c0a (0xb777 bytes) 2024-09-21T15:49:24.980 INFO:teuthology.orchestra.run.smithi089.stdout:1529 writesame 0x7b8c35 thru 0x7bc290 data_size 0x31(0x365c bytes) 2024-09-21T15:49:24.981 INFO:teuthology.orchestra.run.smithi089.stdout:1531 read 0x5bbb3c3 thru 0x5bc87b4 (0xd3f2 bytes) 2024-09-21T15:49:24.982 INFO:teuthology.orchestra.run.smithi089.stdout:1532 read 0xd80e78f thru 0xd81d710 (0xef82 bytes) 2024-09-21T15:49:24.982 INFO:teuthology.orchestra.run.smithi089.stdout:1533 read 0x2cc9180 thru 0x2ccc915 (0x3796 bytes) 2024-09-21T15:49:24.982 INFO:teuthology.orchestra.run.smithi089.stdout:1534 read 0x5c7a613 thru 0x5c882b0 (0xdc9e bytes) 2024-09-21T15:49:24.982 INFO:teuthology.orchestra.run.smithi089.stdout:1535 write 0xaf5f8f2 thru 0xaf68888 (0x8f97 bytes) 2024-09-21T15:49:24.984 INFO:teuthology.orchestra.run.smithi089.stdout:1536 punch from 0x4e0f934 to 0x4e125e8, (0x2cb4 bytes) 2024-09-21T15:49:24.984 INFO:teuthology.orchestra.run.smithi089.stdout:1537 write 0x48090fb thru 0x481297f (0x9885 bytes) 2024-09-21T15:49:24.987 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:25.004 INFO:teuthology.orchestra.run.smithi089.stdout:1539 trunc from 0xe8a0c0b to 0x939621 2024-09-21T15:49:25.032 INFO:teuthology.orchestra.run.smithi089.stdout:1540 trunc from 0x939621 to 0x6979eb 2024-09-21T15:49:25.040 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:25.040 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:25.043 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:25.043 INFO:teuthology.orchestra.run.smithi089.stdout:1543 punch from 0x12da09 to 0x1350b2, (0x76a9 bytes) 2024-09-21T15:49:25.043 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:25.043 INFO:teuthology.orchestra.run.smithi089.stdout:1546 read 0x1c6df2 thru 0x1c91d0 (0x23df bytes) 2024-09-21T15:49:25.043 INFO:teuthology.orchestra.run.smithi089.stdout:1547 read 0x1aabbd thru 0x1b9fde (0xf422 bytes) 2024-09-21T15:49:25.046 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:25.044+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:25.050 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:49:25.051 INFO:teuthology.orchestra.run.smithi089.stdout:1548 write 0x1f75fc3 thru 0x1f7ca24 (0x6a62 bytes) 2024-09-21T15:49:25.053 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:25.059 INFO:teuthology.orchestra.run.smithi089.stdout:1551 trunc from 0x1f7ca25 to 0x73c4403 2024-09-21T15:49:25.070 INFO:teuthology.orchestra.run.smithi089.stdout:1552 trunc from 0x73c4403 to 0xbeb1cc7 2024-09-21T15:49:25.076 INFO:teuthology.orchestra.run.smithi089.stdout:1553 write 0x317c526 thru 0x318654b (0xa026 bytes) 2024-09-21T15:49:25.081 INFO:teuthology.orchestra.run.smithi089.stdout:1554 writesame 0xb0a8520 thru 0xb0aa825 data_size 0x1(0x2306 bytes) 2024-09-21T15:49:25.086 INFO:teuthology.orchestra.run.smithi089.stdout:1555 read 0xa56d4f3 thru 0xa57baa5 (0xe5b3 bytes) 2024-09-21T15:49:25.087 INFO:teuthology.orchestra.run.smithi089.stdout:1556 write 0x477301d thru 0x477cd50 (0x9d34 bytes) 2024-09-21T15:49:25.088 INFO:teuthology.orchestra.run.smithi089.stdout:1558 writesame 0x7cce911 thru 0x7cdb9f5 data_size 0x1(0xd0e5 bytes) 2024-09-21T15:49:25.121 INFO:teuthology.orchestra.run.smithi089.stdout:1559 write 0xc28973b thru 0xc298302 (0xebc8 bytes) 2024-09-21T15:49:25.124 INFO:teuthology.orchestra.run.smithi089.stdout:1560 read 0x5909666 thru 0x5914ae1 (0xb47c bytes) 2024-09-21T15:49:25.124 INFO:teuthology.orchestra.run.smithi089.stdout:1561 write 0xace5f thru 0xb373a (0x68dc bytes) 2024-09-21T15:49:25.139 INFO:teuthology.orchestra.run.smithi089.stdout:1563 trunc from 0xc298303 to 0x742d20 2024-09-21T15:49:25.146 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:25.145+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:25.158 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:25.158 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:25.167 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:49:25.172 INFO:teuthology.orchestra.run.smithi089.stdout:1564 writesame 0x7f549d0 thru 0x7f6411b data_size 0xc4(0xf74c bytes) 2024-09-21T15:49:25.183 INFO:teuthology.orchestra.run.smithi089.stdout:1565 write 0xda24ecc thru 0xda2f378 (0xa4ad bytes) 2024-09-21T15:49:25.186 INFO:teuthology.orchestra.run.smithi089.stdout:1568 write 0x97f4744 thru 0x97fc321 (0x7bde bytes) 2024-09-21T15:49:25.189 INFO:teuthology.orchestra.run.smithi089.stdout:1571 write 0x2caee9a thru 0x2cb46c3 (0x582a bytes) 2024-09-21T15:49:25.192 INFO:teuthology.orchestra.run.smithi089.stdout:1572 write 0x9d6f964 thru 0x9d77ca5 (0x8342 bytes) 2024-09-21T15:49:25.194 INFO:teuthology.orchestra.run.smithi089.stdout:1573 write 0x622feda thru 0x623861e (0x8745 bytes) 2024-09-21T15:49:25.197 INFO:teuthology.orchestra.run.smithi089.stdout:1574 write 0x8984e55 thru 0x8989379 (0x4525 bytes) 2024-09-21T15:49:25.201 INFO:teuthology.orchestra.run.smithi089.stdout:1575 write 0x30295ec thru 0x302e0de (0x4af3 bytes) 2024-09-21T15:49:25.206 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:25.221 INFO:teuthology.orchestra.run.smithi089.stdout:1577 trunc from 0xda2f379 to 0xdb0055 2024-09-21T15:49:25.251 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:25.250+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:25.257 INFO:teuthology.orchestra.run.smithi089.stdout:1580 write 0xb0d43a9 thru 0xb0dae4d (0x6aa5 bytes) 2024-09-21T15:49:25.261 INFO:teuthology.orchestra.run.smithi089.stdout:1581 writesame 0x7d46e32 thru 0x7d4845e data_size 0x1(0x162d bytes) 2024-09-21T15:49:25.267 INFO:teuthology.orchestra.run.smithi089.stdout:1582 read 0xa759952 thru 0xa76551c (0xbbcb bytes) 2024-09-21T15:49:25.267 INFO:teuthology.orchestra.run.smithi089.stdout:1583 write 0x4ef8fcb thru 0x4ef9cc6 (0xcfc bytes) 2024-09-21T15:49:25.270 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:25.279 INFO:teuthology.orchestra.run.smithi089.stdout:1585 write 0xcff7ac0 thru 0xcffc550 (0x4a91 bytes) 2024-09-21T15:49:25.283 INFO:teuthology.orchestra.run.smithi089.stdout:1586 writesame 0x102df2a thru 0x1038f95 data_size 0xe(0xb06c bytes) 2024-09-21T15:49:25.287 INFO:teuthology.orchestra.run.smithi089.stdout:1587 write 0x843a414 thru 0x843baf1 (0x16de bytes) 2024-09-21T15:49:25.288 INFO:teuthology.orchestra.run.smithi089.stdout:1589 writesame 0xba630bb thru 0xba6b146 data_size 0x26(0x808c bytes) 2024-09-21T15:49:25.295 INFO:teuthology.orchestra.run.smithi089.stdout:1590 write 0xd69e15c thru 0xd6aa23b (0xc0e0 bytes) 2024-09-21T15:49:25.298 INFO:teuthology.orchestra.run.smithi089.stdout:1591 punch from 0xbea6da0 to 0xbeaab71, (0x3dd1 bytes) 2024-09-21T15:49:25.304 INFO:teuthology.orchestra.run.smithi089.stdout:1592 trunc from 0xd6aa23c to 0x8df8076 2024-09-21T15:49:25.311 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:25.311 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:25.312 INFO:teuthology.orchestra.run.smithi089.stdout:1597 write 0x29389e3 thru 0x2941184 (0x87a2 bytes) 2024-09-21T15:49:25.315 INFO:teuthology.orchestra.run.smithi089.stdout:1599 punch from 0x2cb70cb to 0x2cbfdba, (0x8cef bytes) 2024-09-21T15:49:25.315 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:25.315 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:25.315 INFO:teuthology.orchestra.run.smithi089.stdout:1602 punch from 0x3c39efd to 0x3c3e3c0, (0x44c3 bytes) 2024-09-21T15:49:25.315 INFO:teuthology.orchestra.run.smithi089.stdout:1603 write 0x7e36b2e thru 0x7e453f1 (0xe8c4 bytes) 2024-09-21T15:49:25.316 INFO:teuthology.orchestra.run.smithi089.stdout:1605 write 0x8b753c thru 0x8bafaa (0x3a6f bytes) 2024-09-21T15:49:25.317 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:25.317 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:25.318 INFO:teuthology.orchestra.run.smithi089.stdout:1607 read 0x6da7320 thru 0x6db573f (0xe420 bytes) 2024-09-21T15:49:25.318 INFO:teuthology.orchestra.run.smithi089.stdout:1609 read 0x898e3bb thru 0x899b0f8 (0xcd3e bytes) 2024-09-21T15:49:25.318 INFO:teuthology.orchestra.run.smithi089.stdout:1610 read 0x8830195 thru 0x88351bc (0x5028 bytes) 2024-09-21T15:49:25.318 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:25.326 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:49:25.326 INFO:teuthology.orchestra.run.smithi089.stdout:1615 trunc from 0x8df8076 to 0x226955b 2024-09-21T15:49:25.345 INFO:teuthology.orchestra.run.smithi089.stdout:1616 punch from 0x9b84ff to 0x9c5183, (0xcc84 bytes) 2024-09-21T15:49:25.348 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:25.347+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:25.365 INFO:teuthology.orchestra.run.smithi089.stdout:1618 write 0xd6539b3 thru 0xd661179 (0xd7c7 bytes) 2024-09-21T15:49:25.371 INFO:teuthology.orchestra.run.smithi089.stdout:1621 punch from 0x49a69d7 to 0x49b3d68, (0xd391 bytes) 2024-09-21T15:49:25.371 INFO:teuthology.orchestra.run.smithi089.stdout:1622 punch from 0x77d9f26 to 0x77e4714, (0xa7ee bytes) 2024-09-21T15:49:25.371 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:25.371 INFO:teuthology.orchestra.run.smithi089.stdout:1625 read 0x64978c2 thru 0x649ea3c (0x717b bytes) 2024-09-21T15:49:25.372 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:25.372 INFO:teuthology.orchestra.run.smithi089.stdout:1627 read 0x9ab1ebb thru 0x9ab95df (0x7725 bytes) 2024-09-21T15:49:25.372 INFO:teuthology.orchestra.run.smithi089.stdout:1628 read 0x96e1dc7 thru 0x96e54da (0x3714 bytes) 2024-09-21T15:49:25.372 INFO:teuthology.orchestra.run.smithi089.stdout:1630 read 0xc4cfdc0 thru 0xc4d565b (0x589c bytes) 2024-09-21T15:49:25.372 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:25.372 INFO:teuthology.orchestra.run.smithi089.stdout:1633 read 0xa59bcd7 thru 0xa5a20f1 (0x641b bytes) 2024-09-21T15:49:25.372 INFO:teuthology.orchestra.run.smithi089.stdout:1634 punch from 0xc96c4e1 to 0xc971d30, (0x584f bytes) 2024-09-21T15:49:25.372 INFO:teuthology.orchestra.run.smithi089.stdout:1635 punch from 0x54b9a05 to 0x54c8ae6, (0xf0e1 bytes) 2024-09-21T15:49:25.372 INFO:teuthology.orchestra.run.smithi089.stdout:1636 write 0xaa67db8 thru 0xaa75885 (0xdace bytes) 2024-09-21T15:49:25.387 INFO:teuthology.orchestra.run.smithi089.stdout:1637 trunc from 0xd66117a to 0x499df14 2024-09-21T15:49:25.400 INFO:teuthology.orchestra.run.smithi089.stdout:1638 punch from 0x406de03 to 0x407207c, (0x4279 bytes) 2024-09-21T15:49:25.400 INFO:teuthology.orchestra.run.smithi089.stdout:1640 read 0x2cac677 thru 0x2cb7bd5 (0xb55f bytes) 2024-09-21T15:49:25.400 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:25.400 INFO:teuthology.orchestra.run.smithi089.stdout:1642 read 0x2dfcf1 thru 0x2e6582 (0x6892 bytes) 2024-09-21T15:49:25.401 INFO:teuthology.orchestra.run.smithi089.stdout:1643 clone 6 order 20 su 8192 sc 7 2024-09-21T15:49:25.449 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:25.448+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:25.468 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:25.469 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:25.477 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:49:25.561 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:25.561+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:25.619 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:25.620 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:25.629 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:49:25.650 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:25.649+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:25.751 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:25.751+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:25.774 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:25.775 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:25.784 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:49:25.868 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:25.867+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:25.877 INFO:teuthology.orchestra.run.smithi089.stdout:checking clone #4, image image_client.0-clone4 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent5 2024-09-21T15:49:25.929 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:25.929 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:25.940 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T15:49:25.959 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:49:25.981 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:25.980+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:26.053 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:26.052+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:26.083 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:26.083 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:26.084 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-09-21T15:49:26.084 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:49:26.092 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T15:49:26.101 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:49:26.101 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:49:26.117 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:49:26.117 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:49:26.132 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:26.133 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:49:26.133 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:49:26.151 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:49:26.151 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:49:26.154 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:26.153+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:26.155 INFO:teuthology.orchestra.run.smithi089.stdout:1645 writesame 0xe087eb7 thru 0xe08a0a4 data_size 0x2b(0x21ee bytes) 2024-09-21T15:49:26.155 INFO:teuthology.orchestra.run.smithi089.stdout:1646 writesame 0x6bb44b7 thru 0x6bc2726 data_size 0x8(0xe270 bytes) 2024-09-21T15:49:26.169 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:49:26.169 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:49:26.171 INFO:teuthology.orchestra.run.smithi089.stdout:1647 read 0x8596fa0 thru 0x85a54b1 (0xe512 bytes) 2024-09-21T15:49:26.171 INFO:teuthology.orchestra.run.smithi089.stdout:1651 read 0xd6e6e95 thru 0xd6f06e1 (0x984d bytes) 2024-09-21T15:49:26.171 INFO:teuthology.orchestra.run.smithi089.stdout:1654 write 0x6ae61b9 thru 0x6ae7cc4 (0x1b0c bytes) 2024-09-21T15:49:26.172 INFO:teuthology.orchestra.run.smithi089.stdout:1656 write 0xde4a3db thru 0xde56c65 (0xc88b bytes) 2024-09-21T15:49:26.186 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:49:26.186 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:49:26.189 INFO:teuthology.orchestra.run.smithi089.stdout:1657 trunc from 0xe08a0a5 to 0xc52f1a5 2024-09-21T15:49:26.203 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:49:26.203 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:49:26.206 INFO:teuthology.orchestra.run.smithi089.stdout:1658 punch from 0x444a002 to 0x4454938, (0xa936 bytes) 2024-09-21T15:49:26.206 INFO:teuthology.orchestra.run.smithi089.stdout:1659 writesame 0x6d2349 thru 0x6d7afa data_size 0x19(0x57b2 bytes) 2024-09-21T15:49:26.242 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:26.242 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:26.242 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:49:26.242 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:49:26.243 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:26.251 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T15:49:26.273 INFO:teuthology.orchestra.run.smithi089.stdout:1661 write 0x673b73b thru 0x673d3f5 (0x1cbb bytes) 2024-09-21T15:49:26.276 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:26.277 INFO:teuthology.orchestra.run.smithi089.stdout:1664 write 0x25a1ab6 thru 0x25a7dfe (0x6349 bytes) 2024-09-21T15:49:26.282 INFO:teuthology.orchestra.run.smithi089.stdout:1665 write 0xa1cbb19 thru 0xa1cf3c9 (0x38b1 bytes) 2024-09-21T15:49:26.288 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:26.288 INFO:teuthology.orchestra.run.smithi089.stdout:1667 read 0x1c34bf2 thru 0x1c359a0 (0xdaf bytes) 2024-09-21T15:49:26.288 INFO:teuthology.orchestra.run.smithi089.stdout:1668 write 0x552d773 thru 0x5535053 (0x78e1 bytes) 2024-09-21T15:49:26.294 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:26.293+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:26.301 INFO:teuthology.orchestra.run.smithi089.stdout:1669 read 0x972d576 thru 0x973a73c (0xd1c7 bytes) 2024-09-21T15:49:26.301 INFO:teuthology.orchestra.run.smithi089.stdout:1670 read 0x7f08ffb thru 0x7f101ac (0x71b2 bytes) 2024-09-21T15:49:26.301 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:26.301 INFO:teuthology.orchestra.run.smithi089.stdout:1672 read 0xa42dbb1 thru 0xa439e68 (0xc2b8 bytes) 2024-09-21T15:49:26.314 INFO:teuthology.orchestra.run.smithi089.stdout:1673 trunc from 0xc52f1a5 to 0x1630f57 2024-09-21T15:49:26.355 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:26.354+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:26.374 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:26.374 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:49:26.375 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:49:26.375 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:49:26.375 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.375 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:2083 3.a 3:55b68c50:::rbd_data.11335fa2fe69.0000000000000191:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 524288 write_size 524288,truncate 77824] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:26.375 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.141682+0000", 2024-09-21T15:49:26.375 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.230320102, 2024-09-21T15:49:26.375 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0053016579999999999, 2024-09-21T15:49:26.375 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:26.375 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:26.375 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:26.375 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:26.375 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:26.375 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 2083 2024-09-21T15:49:26.376 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.376 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:26.376 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.376 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:26.376 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.141682+0000", 2024-09-21T15:49:26.376 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.376 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.376 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.376 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:26.376 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.141682+0000", 2024-09-21T15:49:26.376 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.376 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.376 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.377 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:26.377 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.141679+0000", 2024-09-21T15:49:26.377 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999976 2024-09-21T15:49:26.377 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.377 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.377 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:26.377 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.141691+0000", 2024-09-21T15:49:26.377 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1922e-05 2024-09-21T15:49:26.377 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.377 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.377 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:26.377 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.141712+0000", 2024-09-21T15:49:26.377 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0115e-05 2024-09-21T15:49:26.378 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.378 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.378 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:26.378 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.141723+0000", 2024-09-21T15:49:26.378 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1071e-05 2024-09-21T15:49:26.378 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.378 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.378 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:26.378 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.141776+0000", 2024-09-21T15:49:26.378 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3497e-05 2024-09-21T15:49:26.378 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.378 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.378 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:26.379 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.141918+0000", 2024-09-21T15:49:26.379 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000141433 2024-09-21T15:49:26.379 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.379 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.379 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-21T15:49:26.379 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.142129+0000", 2024-09-21T15:49:26.379 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00021179700000000001 2024-09-21T15:49:26.379 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.379 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.379 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:26.379 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.144018+0000", 2024-09-21T15:49:26.379 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0018886949999999999 2024-09-21T15:49:26.379 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.379 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.380 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:26.380 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.146889+0000", 2024-09-21T15:49:26.380 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002871017 2024-09-21T15:49:26.380 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.380 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.380 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:26.380 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.146947+0000", 2024-09-21T15:49:26.380 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7558e-05 2024-09-21T15:49:26.380 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.380 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.380 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:26.380 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.146984+0000", 2024-09-21T15:49:26.380 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7095999999999999e-05 2024-09-21T15:49:26.380 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.381 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:26.381 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.381 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.381 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.381 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:13260 3.10 3:0d698a3f:::rbd_data.1133dc5adfdf.0000000000000096:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:26.381 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.086102+0000", 2024-09-21T15:49:26.381 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 9.2859002509999993, 2024-09-21T15:49:26.381 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0050922229999999999, 2024-09-21T15:49:26.381 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:26.381 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:26.381 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:26.381 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:26.381 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:26.382 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 13260 2024-09-21T15:49:26.382 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.382 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:26.382 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.382 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:26.382 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.086102+0000", 2024-09-21T15:49:26.382 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.382 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.382 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.382 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:26.382 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.086102+0000", 2024-09-21T15:49:26.382 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.382 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.382 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.383 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:26.383 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.086101+0000", 2024-09-21T15:49:26.383 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:26.383 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.383 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.383 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:26.383 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.086104+0000", 2024-09-21T15:49:26.383 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1980000000000001e-06 2024-09-21T15:49:26.383 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.383 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.383 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:26.383 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.086112+0000", 2024-09-21T15:49:26.383 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5290000000000003e-06 2024-09-21T15:49:26.384 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.384 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.384 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:26.384 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.086120+0000", 2024-09-21T15:49:26.384 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9489999999999996e-06 2024-09-21T15:49:26.384 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.384 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.384 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:26.384 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.086526+0000", 2024-09-21T15:49:26.384 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000406103 2024-09-21T15:49:26.384 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.384 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.384 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:26.384 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.086596+0000", 2024-09-21T15:49:26.385 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9675e-05 2024-09-21T15:49:26.385 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.385 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.385 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0", 2024-09-21T15:49:26.385 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.086688+0000", 2024-09-21T15:49:26.385 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2576000000000003e-05 2024-09-21T15:49:26.385 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.385 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.385 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:26.385 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.089471+0000", 2024-09-21T15:49:26.385 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002782423 2024-09-21T15:49:26.385 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.385 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.385 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:26.385 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.091148+0000", 2024-09-21T15:49:26.386 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0016771379999999999 2024-09-21T15:49:26.386 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.386 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.386 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:26.386 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.091173+0000", 2024-09-21T15:49:26.386 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5347e-05 2024-09-21T15:49:26.386 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.386 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.386 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:26.386 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.091194+0000", 2024-09-21T15:49:26.386 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0888999999999999e-05 2024-09-21T15:49:26.386 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.386 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:26.386 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.387 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.387 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.387 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:13274 3.10 3:0b1f2304:::rbd_data.1133dc5adfdf.00000000000000b9:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:26.387 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.090744+0000", 2024-09-21T15:49:26.387 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 9.2812584410000003, 2024-09-21T15:49:26.387 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0082153750000000005, 2024-09-21T15:49:26.387 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:26.387 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:26.387 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:26.387 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:26.387 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:26.387 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 13274 2024-09-21T15:49:26.387 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.387 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:26.387 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.387 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:26.388 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.090744+0000", 2024-09-21T15:49:26.388 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.388 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.388 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.388 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:26.388 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.090744+0000", 2024-09-21T15:49:26.388 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.388 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.388 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.388 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:26.388 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.090743+0000", 2024-09-21T15:49:26.388 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-21T15:49:26.388 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.388 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.388 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:26.389 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.090747+0000", 2024-09-21T15:49:26.389 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.262e-06 2024-09-21T15:49:26.389 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.389 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.389 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:26.389 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.090758+0000", 2024-09-21T15:49:26.389 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0587999999999999e-05 2024-09-21T15:49:26.389 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.389 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.389 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:26.389 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.090762+0000", 2024-09-21T15:49:26.389 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7779999999999998e-06 2024-09-21T15:49:26.389 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.389 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.389 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:26.389 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.090799+0000", 2024-09-21T15:49:26.390 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6940999999999997e-05 2024-09-21T15:49:26.390 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.390 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.390 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:26.390 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.090847+0000", 2024-09-21T15:49:26.390 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7975000000000001e-05 2024-09-21T15:49:26.390 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.390 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.390 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0", 2024-09-21T15:49:26.390 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.090937+0000", 2024-09-21T15:49:26.390 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0148000000000005e-05 2024-09-21T15:49:26.390 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.390 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.390 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:26.390 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.094607+0000", 2024-09-21T15:49:26.391 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0036694029999999999 2024-09-21T15:49:26.391 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.391 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.391 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:26.391 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.098877+0000", 2024-09-21T15:49:26.391 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0042697239999999999 2024-09-21T15:49:26.391 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.391 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.391 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:26.391 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.098920+0000", 2024-09-21T15:49:26.391 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3547999999999997e-05 2024-09-21T15:49:26.391 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.391 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.391 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:26.391 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.098959+0000", 2024-09-21T15:49:26.391 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8890000000000002e-05 2024-09-21T15:49:26.392 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.392 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:26.392 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.392 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.392 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.392 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:13305 3.a 3:53172455:::rbd_data.1133dc5adfdf.00000000000000cd:head [set-alloc-hint object_size 1048576 write_size 1048576,write 540672~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:26.392 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.126122+0000", 2024-09-21T15:49:26.392 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 9.2458797310000005, 2024-09-21T15:49:26.392 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0051376010000000003, 2024-09-21T15:49:26.392 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:26.392 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:26.392 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:26.392 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:26.392 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:26.393 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 13305 2024-09-21T15:49:26.393 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.393 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:26.393 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.393 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:26.393 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.126122+0000", 2024-09-21T15:49:26.393 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.393 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.393 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.393 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:26.393 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.126122+0000", 2024-09-21T15:49:26.393 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.393 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.393 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.393 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:26.393 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.126121+0000", 2024-09-21T15:49:26.394 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:26.394 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.394 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.394 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:26.394 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.126131+0000", 2024-09-21T15:49:26.394 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4870000000000008e-06 2024-09-21T15:49:26.394 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.394 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.394 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:26.394 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.126142+0000", 2024-09-21T15:49:26.394 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1123e-05 2024-09-21T15:49:26.394 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.394 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.394 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:26.395 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.126148+0000", 2024-09-21T15:49:26.395 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4250000000000003e-06 2024-09-21T15:49:26.395 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.395 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.395 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:26.395 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.126190+0000", 2024-09-21T15:49:26.395 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2063e-05 2024-09-21T15:49:26.395 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.395 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.395 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:26.395 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.126272+0000", 2024-09-21T15:49:26.395 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1569000000000005e-05 2024-09-21T15:49:26.395 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.395 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.395 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-21T15:49:26.395 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.126356+0000", 2024-09-21T15:49:26.396 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3916000000000001e-05 2024-09-21T15:49:26.396 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.396 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.396 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:26.396 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.128614+0000", 2024-09-21T15:49:26.396 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0022582129999999998 2024-09-21T15:49:26.396 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.396 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.396 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:26.396 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.131210+0000", 2024-09-21T15:49:26.396 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0025964069999999998 2024-09-21T15:49:26.396 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.396 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.396 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:26.396 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.131239+0000", 2024-09-21T15:49:26.396 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8796999999999999e-05 2024-09-21T15:49:26.397 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.397 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.397 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:26.397 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.131260+0000", 2024-09-21T15:49:26.397 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0928999999999999e-05 2024-09-21T15:49:26.397 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.397 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:26.397 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.397 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.397 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.397 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:13550 3.a 3:5510f58f:::rbd_data.1133dc5adfdf.0000000000000080:head [set-alloc-hint object_size 1048576 write_size 1048576,write 942080~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:26.397 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.319272+0000", 2024-09-21T15:49:26.397 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 9.0527302800000005, 2024-09-21T15:49:26.397 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0068300210000000004, 2024-09-21T15:49:26.397 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:26.398 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:26.398 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:26.398 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:26.398 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:26.398 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 13550 2024-09-21T15:49:26.398 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.398 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:26.398 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.398 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:26.398 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.319272+0000", 2024-09-21T15:49:26.398 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.398 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.398 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.398 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:26.399 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.319272+0000", 2024-09-21T15:49:26.399 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.399 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.399 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.399 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:26.399 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.319270+0000", 2024-09-21T15:49:26.399 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:26.399 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.399 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.399 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:26.399 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.319282+0000", 2024-09-21T15:49:26.399 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2143999999999999e-05 2024-09-21T15:49:26.399 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.399 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.399 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:26.399 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.319297+0000", 2024-09-21T15:49:26.400 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5017999999999999e-05 2024-09-21T15:49:26.400 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.400 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.400 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:26.400 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.319303+0000", 2024-09-21T15:49:26.400 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8390000000000003e-06 2024-09-21T15:49:26.400 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.400 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.400 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:26.400 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.319355+0000", 2024-09-21T15:49:26.400 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1777999999999998e-05 2024-09-21T15:49:26.400 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.400 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.400 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:26.400 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.319441+0000", 2024-09-21T15:49:26.400 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.585e-05 2024-09-21T15:49:26.401 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.401 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.401 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-21T15:49:26.401 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.319585+0000", 2024-09-21T15:49:26.401 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000144003 2024-09-21T15:49:26.401 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.401 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.401 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:26.401 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.325771+0000", 2024-09-21T15:49:26.401 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0061861959999999997 2024-09-21T15:49:26.401 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.401 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.401 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:26.401 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.326013+0000", 2024-09-21T15:49:26.401 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00024232600000000001 2024-09-21T15:49:26.402 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.402 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.402 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:26.402 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.326063+0000", 2024-09-21T15:49:26.402 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9669000000000002e-05 2024-09-21T15:49:26.402 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.402 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.402 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:26.402 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.326102+0000", 2024-09-21T15:49:26.402 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8825999999999997e-05 2024-09-21T15:49:26.402 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.402 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:26.402 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.403 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.403 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.403 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:13581 3.a 3:52d3dd80:::rbd_data.1133dc5adfdf.00000000000000b0:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:26.403 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.347049+0000", 2024-09-21T15:49:26.403 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 9.0249526600000003, 2024-09-21T15:49:26.403 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0051259269999999997, 2024-09-21T15:49:26.403 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:26.403 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:26.403 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:26.403 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:26.403 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:26.403 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 13581 2024-09-21T15:49:26.403 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.403 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:26.403 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.403 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:26.404 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.347049+0000", 2024-09-21T15:49:26.404 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.404 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.404 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.404 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:26.404 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.347049+0000", 2024-09-21T15:49:26.404 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.404 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.404 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.404 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:26.404 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.347049+0000", 2024-09-21T15:49:26.404 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-21T15:49:26.404 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.404 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.404 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:26.405 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.347053+0000", 2024-09-21T15:49:26.405 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5229999999999997e-06 2024-09-21T15:49:26.405 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.405 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.405 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:26.405 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.347064+0000", 2024-09-21T15:49:26.405 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0336999999999999e-05 2024-09-21T15:49:26.405 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.405 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.405 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:26.405 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.347071+0000", 2024-09-21T15:49:26.405 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1629999999999999e-06 2024-09-21T15:49:26.405 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.405 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.405 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:26.405 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.347102+0000", 2024-09-21T15:49:26.406 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1647999999999999e-05 2024-09-21T15:49:26.406 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.406 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.406 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:26.406 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.347155+0000", 2024-09-21T15:49:26.406 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3093000000000001e-05 2024-09-21T15:49:26.406 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.406 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.406 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-21T15:49:26.406 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.347221+0000", 2024-09-21T15:49:26.406 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5922999999999999e-05 2024-09-21T15:49:26.406 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.406 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.406 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:26.406 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.349565+0000", 2024-09-21T15:49:26.407 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0023435909999999999 2024-09-21T15:49:26.407 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.407 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.407 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:26.407 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.352099+0000", 2024-09-21T15:49:26.407 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0025342089999999999 2024-09-21T15:49:26.407 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.407 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.407 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:26.407 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.352141+0000", 2024-09-21T15:49:26.407 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2259000000000003e-05 2024-09-21T15:49:26.407 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.407 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.407 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:26.408 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.352175+0000", 2024-09-21T15:49:26.408 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3973000000000002e-05 2024-09-21T15:49:26.408 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.408 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:26.408 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.408 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.408 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.408 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:13862 3.a 3:53172455:::rbd_data.1133dc5adfdf.00000000000000cd:head [set-alloc-hint object_size 1048576 write_size 1048576,write 266240~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:26.408 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.549561+0000", 2024-09-21T15:49:26.408 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 8.8224411150000002, 2024-09-21T15:49:26.408 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0065617749999999997, 2024-09-21T15:49:26.408 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:26.408 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:26.408 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:26.408 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:26.409 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:26.409 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 13862 2024-09-21T15:49:26.409 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.409 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:26.409 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.409 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:26.409 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.549561+0000", 2024-09-21T15:49:26.409 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.409 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.409 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.409 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:26.409 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.549561+0000", 2024-09-21T15:49:26.409 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.409 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.409 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.410 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:26.410 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.549560+0000", 2024-09-21T15:49:26.410 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:26.410 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.410 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.410 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:26.410 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.549569+0000", 2024-09-21T15:49:26.410 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2799999999999992e-06 2024-09-21T15:49:26.410 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.410 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.410 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:26.410 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.549582+0000", 2024-09-21T15:49:26.410 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2757e-05 2024-09-21T15:49:26.410 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.410 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.411 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:26.411 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.549590+0000", 2024-09-21T15:49:26.411 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4619999999999996e-06 2024-09-21T15:49:26.411 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.411 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.411 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:26.411 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.549642+0000", 2024-09-21T15:49:26.411 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1779e-05 2024-09-21T15:49:26.411 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.411 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.411 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:26.411 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.549736+0000", 2024-09-21T15:49:26.411 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4617000000000006e-05 2024-09-21T15:49:26.411 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.411 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.412 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-21T15:49:26.412 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.549886+0000", 2024-09-21T15:49:26.412 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014938199999999999 2024-09-21T15:49:26.412 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.412 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.412 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:26.412 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.553416+0000", 2024-09-21T15:49:26.412 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0035298339999999999 2024-09-21T15:49:26.412 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.412 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.412 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:26.412 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.556032+0000", 2024-09-21T15:49:26.412 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002615994 2024-09-21T15:49:26.412 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.412 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.412 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:26.413 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.556080+0000", 2024-09-21T15:49:26.413 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8442999999999997e-05 2024-09-21T15:49:26.413 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.413 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.413 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:26.413 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.556123+0000", 2024-09-21T15:49:26.413 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2673999999999998e-05 2024-09-21T15:49:26.413 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.413 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:26.413 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.413 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.413 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.413 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:13948 3.a 3:52b31f3c:::rbd_data.1133dc5adfdf.00000000000000db:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:26.414 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.616162+0000", 2024-09-21T15:49:26.414 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 8.7558398000000004, 2024-09-21T15:49:26.414 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.004904594, 2024-09-21T15:49:26.414 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:26.414 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:26.414 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:26.414 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:26.414 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:26.414 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 13948 2024-09-21T15:49:26.414 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.414 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:26.414 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.414 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:26.414 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.616162+0000", 2024-09-21T15:49:26.414 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.415 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.415 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.415 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:26.415 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.616162+0000", 2024-09-21T15:49:26.415 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.415 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.415 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.415 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:26.415 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.616160+0000", 2024-09-21T15:49:26.415 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999981 2024-09-21T15:49:26.415 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.415 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.415 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:26.415 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.616168+0000", 2024-09-21T15:49:26.416 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3030000000000002e-06 2024-09-21T15:49:26.416 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.416 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.416 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:26.416 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.616183+0000", 2024-09-21T15:49:26.416 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5756e-05 2024-09-21T15:49:26.416 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.416 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.416 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:26.416 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.616191+0000", 2024-09-21T15:49:26.416 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6090000000000003e-06 2024-09-21T15:49:26.416 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.416 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.416 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:26.417 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.616249+0000", 2024-09-21T15:49:26.417 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8286999999999999e-05 2024-09-21T15:49:26.417 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.417 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.417 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:26.417 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.616333+0000", 2024-09-21T15:49:26.417 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3441000000000003e-05 2024-09-21T15:49:26.417 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.417 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.417 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-21T15:49:26.417 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.616441+0000", 2024-09-21T15:49:26.417 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000108117 2024-09-21T15:49:26.417 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.417 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.417 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:26.417 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.617968+0000", 2024-09-21T15:49:26.418 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.001526662 2024-09-21T15:49:26.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.418 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:26.418 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.620963+0000", 2024-09-21T15:49:26.418 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0029951380000000001 2024-09-21T15:49:26.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.418 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:26.418 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.621008+0000", 2024-09-21T15:49:26.418 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5664000000000002e-05 2024-09-21T15:49:26.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.418 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:26.419 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.621067+0000", 2024-09-21T15:49:26.419 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8548000000000003e-05 2024-09-21T15:49:26.419 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.419 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:26.419 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.419 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.419 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.419 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:14003 3.10 3:0b1f2304:::rbd_data.1133dc5adfdf.00000000000000b9:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:26.419 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.658416+0000", 2024-09-21T15:49:26.419 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 8.7135857290000001, 2024-09-21T15:49:26.419 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0045796250000000004, 2024-09-21T15:49:26.419 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:26.419 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:26.419 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:26.419 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:26.420 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:26.420 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 14003 2024-09-21T15:49:26.420 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.420 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:26.420 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.420 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:26.420 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.658416+0000", 2024-09-21T15:49:26.420 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.420 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.420 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.420 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:26.420 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.658416+0000", 2024-09-21T15:49:26.420 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.420 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.420 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.420 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:26.421 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.658415+0000", 2024-09-21T15:49:26.421 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-21T15:49:26.421 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.421 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.421 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:26.421 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.658422+0000", 2024-09-21T15:49:26.421 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3910000000000003e-06 2024-09-21T15:49:26.421 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.421 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.421 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:26.421 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.658433+0000", 2024-09-21T15:49:26.421 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1435e-05 2024-09-21T15:49:26.421 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.421 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.421 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:26.421 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.658443+0000", 2024-09-21T15:49:26.422 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0012e-05 2024-09-21T15:49:26.422 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.422 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.422 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:26.422 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.658485+0000", 2024-09-21T15:49:26.422 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2079000000000001e-05 2024-09-21T15:49:26.422 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.422 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.422 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:26.422 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.658550+0000", 2024-09-21T15:49:26.422 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5028000000000001e-05 2024-09-21T15:49:26.422 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.422 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.422 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0", 2024-09-21T15:49:26.422 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.658622+0000", 2024-09-21T15:49:26.423 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1381e-05 2024-09-21T15:49:26.423 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.423 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.423 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:26.423 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.660779+0000", 2024-09-21T15:49:26.423 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0021570270000000002 2024-09-21T15:49:26.423 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.423 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.423 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:26.423 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.662939+0000", 2024-09-21T15:49:26.423 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0021602119999999999 2024-09-21T15:49:26.423 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.423 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.423 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:26.423 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.662968+0000", 2024-09-21T15:49:26.423 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9309999999999999e-05 2024-09-21T15:49:26.424 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.424 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.424 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:26.424 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.662996+0000", 2024-09-21T15:49:26.424 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.792e-05 2024-09-21T15:49:26.424 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.424 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:26.424 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.424 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.424 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.424 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:14017 3.a 3:53172455:::rbd_data.1133dc5adfdf.00000000000000cd:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-21T15:49:26.424 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:17.663573+0000", 2024-09-21T15:49:26.424 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 8.7084289699999999, 2024-09-21T15:49:26.424 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.014018708, 2024-09-21T15:49:26.424 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:26.424 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:26.425 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:26.425 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:26.425 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:26.425 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 14017 2024-09-21T15:49:26.425 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.425 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:26.425 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.425 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:26.425 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.663573+0000", 2024-09-21T15:49:26.425 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.425 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.425 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.425 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:26.425 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.663573+0000", 2024-09-21T15:49:26.425 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.425 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.426 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.426 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:26.426 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.663572+0000", 2024-09-21T15:49:26.426 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:26.426 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.426 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.426 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:26.426 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.663579+0000", 2024-09-21T15:49:26.426 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4340000000000003e-06 2024-09-21T15:49:26.426 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.426 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.426 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:26.426 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.663594+0000", 2024-09-21T15:49:26.426 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4814000000000001e-05 2024-09-21T15:49:26.427 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.427 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.427 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:26.427 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.663602+0000", 2024-09-21T15:49:26.427 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7030000000000006e-06 2024-09-21T15:49:26.427 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.427 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.427 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:26.427 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.663652+0000", 2024-09-21T15:49:26.427 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0866999999999999e-05 2024-09-21T15:49:26.427 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.427 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.427 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op must wait for map", 2024-09-21T15:49:26.427 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.663659+0000", 2024-09-21T15:49:26.427 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7909999999999999e-06 2024-09-21T15:49:26.428 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.428 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.428 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:26.428 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.672664+0000", 2024-09-21T15:49:26.428 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0090051469999999998 2024-09-21T15:49:26.428 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.428 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.428 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:26.428 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.672740+0000", 2024-09-21T15:49:26.428 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5777000000000003e-05 2024-09-21T15:49:26.428 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.428 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.428 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-21T15:49:26.428 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.672840+0000", 2024-09-21T15:49:26.428 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000100108 2024-09-21T15:49:26.429 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.429 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.429 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:26.429 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.674807+0000", 2024-09-21T15:49:26.429 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0019661930000000002 2024-09-21T15:49:26.429 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.429 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.429 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:26.429 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.677520+0000", 2024-09-21T15:49:26.429 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0027133270000000002 2024-09-21T15:49:26.429 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.429 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.429 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:26.429 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.677567+0000", 2024-09-21T15:49:26.429 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7307e-05 2024-09-21T15:49:26.429 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.430 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.430 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:26.430 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:17.677592+0000", 2024-09-21T15:49:26.430 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4711000000000001e-05 2024-09-21T15:49:26.430 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.430 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:26.430 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.430 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.430 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.430 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:14245 3.a 3:5510f58f:::rbd_data.1133dc5adfdf.0000000000000080:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-21T15:49:26.430 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:18.831104+0000", 2024-09-21T15:49:26.430 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.5408984539999997, 2024-09-21T15:49:26.430 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0047728409999999999, 2024-09-21T15:49:26.430 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:26.430 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:26.431 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:26.431 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:26.431 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:26.431 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 14245 2024-09-21T15:49:26.431 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.431 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:26.431 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.431 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:26.431 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.831104+0000", 2024-09-21T15:49:26.431 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.431 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.431 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.431 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:26.431 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.831104+0000", 2024-09-21T15:49:26.431 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.431 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.432 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.432 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:26.432 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.831102+0000", 2024-09-21T15:49:26.432 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999981 2024-09-21T15:49:26.432 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.432 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.432 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:26.432 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.831111+0000", 2024-09-21T15:49:26.432 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7899999999999994e-06 2024-09-21T15:49:26.432 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.432 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.432 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:26.432 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.831132+0000", 2024-09-21T15:49:26.432 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0693e-05 2024-09-21T15:49:26.432 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.432 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.433 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:26.433 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.831142+0000", 2024-09-21T15:49:26.433 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5400000000000001e-06 2024-09-21T15:49:26.433 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.433 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.433 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:26.433 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.831197+0000", 2024-09-21T15:49:26.433 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5078999999999999e-05 2024-09-21T15:49:26.433 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.433 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.433 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:26.433 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.831289+0000", 2024-09-21T15:49:26.433 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2614000000000005e-05 2024-09-21T15:49:26.433 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.433 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.433 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-21T15:49:26.434 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.831401+0000", 2024-09-21T15:49:26.434 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000111457 2024-09-21T15:49:26.434 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.434 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.434 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:26.434 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.833162+0000", 2024-09-21T15:49:26.434 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0017607009999999999 2024-09-21T15:49:26.434 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.434 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.434 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:26.434 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.835816+0000", 2024-09-21T15:49:26.434 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0026548510000000002 2024-09-21T15:49:26.434 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.434 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.434 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:26.435 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.835845+0000", 2024-09-21T15:49:26.435 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8538e-05 2024-09-21T15:49:26.435 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.435 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.435 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:26.435 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.835876+0000", 2024-09-21T15:49:26.435 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1565e-05 2024-09-21T15:49:26.435 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.435 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:26.435 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.435 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.435 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.435 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4403.0:14408 3.8 e33/32 3:13437810:::rbd_data.1133dc5adfdf.0000000000000039:head v 33'55, mlcod=33'52)", 2024-09-21T15:49:26.435 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:18.971354+0000", 2024-09-21T15:49:26.435 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.4006482099999999, 2024-09-21T15:49:26.435 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0048492420000000001, 2024-09-21T15:49:26.435 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:26.436 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:26.436 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:26.436 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.436 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:26.436 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.971354+0000", 2024-09-21T15:49:26.436 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.436 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.436 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.436 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:26.436 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.971354+0000", 2024-09-21T15:49:26.436 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.436 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.436 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.436 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:26.436 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.971353+0000", 2024-09-21T15:49:26.436 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:26.437 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.437 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.437 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:26.437 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.971359+0000", 2024-09-21T15:49:26.437 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6829999999999996e-06 2024-09-21T15:49:26.437 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.437 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.437 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:26.437 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.971370+0000", 2024-09-21T15:49:26.437 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0776e-05 2024-09-21T15:49:26.437 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.437 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.437 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:26.437 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.971375+0000", 2024-09-21T15:49:26.437 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6670000000000001e-06 2024-09-21T15:49:26.437 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.438 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.438 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:26.438 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.971414+0000", 2024-09-21T15:49:26.438 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8376e-05 2024-09-21T15:49:26.438 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.438 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.438 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:26.438 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.971432+0000", 2024-09-21T15:49:26.438 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8603000000000001e-05 2024-09-21T15:49:26.438 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.438 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.438 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:26.438 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.976169+0000", 2024-09-21T15:49:26.438 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0047363099999999997 2024-09-21T15:49:26.438 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.439 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.439 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:26.439 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.976203+0000", 2024-09-21T15:49:26.439 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4473e-05 2024-09-21T15:49:26.439 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.439 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:26.439 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.439 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.439 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.439 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4403.0:14410 3.1e e33/22 3:7bf656a2:::rbd_data.1133dc5adfdf.000000000000003b:head v 33'95, mlcod=33'92)", 2024-09-21T15:49:26.439 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:18.971584+0000", 2024-09-21T15:49:26.439 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.4004180440000003, 2024-09-21T15:49:26.439 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0046296760000000001, 2024-09-21T15:49:26.439 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:26.439 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:26.439 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:26.440 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.440 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:26.440 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.971584+0000", 2024-09-21T15:49:26.440 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.440 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.440 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.440 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:26.440 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.971584+0000", 2024-09-21T15:49:26.440 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.440 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.440 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.440 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:26.440 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.971584+0000", 2024-09-21T15:49:26.440 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:26.440 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.440 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.441 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:26.441 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.971588+0000", 2024-09-21T15:49:26.441 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.34e-06 2024-09-21T15:49:26.441 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.441 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.441 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:26.441 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.971597+0000", 2024-09-21T15:49:26.441 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.85e-06 2024-09-21T15:49:26.441 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.441 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.441 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:26.441 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.971601+0000", 2024-09-21T15:49:26.441 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4780000000000002e-06 2024-09-21T15:49:26.441 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.442 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.442 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:26.442 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.971635+0000", 2024-09-21T15:49:26.442 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4190999999999998e-05 2024-09-21T15:49:26.442 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.442 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.442 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:26.442 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.971656+0000", 2024-09-21T15:49:26.442 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0608e-05 2024-09-21T15:49:26.442 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.442 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.442 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:26.442 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.976184+0000", 2024-09-21T15:49:26.442 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0045279730000000002 2024-09-21T15:49:26.442 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.443 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.443 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:26.443 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.976214+0000", 2024-09-21T15:49:26.443 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9709e-05 2024-09-21T15:49:26.443 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.443 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:26.443 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.443 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.443 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.443 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:14417 3.a 3:5715b5ca:::rbd_data.1133dc5adfdf.000000000000004e:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-21T15:49:26.443 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:18.975481+0000", 2024-09-21T15:49:26.443 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.3965214230000003, 2024-09-21T15:49:26.443 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0087110940000000008, 2024-09-21T15:49:26.443 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:26.443 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:26.444 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:26.444 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:26.444 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:26.444 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 14417 2024-09-21T15:49:26.444 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.444 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:26.444 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.444 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:26.444 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.975481+0000", 2024-09-21T15:49:26.444 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.444 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.444 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.444 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:26.444 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.975481+0000", 2024-09-21T15:49:26.444 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.444 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.445 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.445 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:26.445 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.975479+0000", 2024-09-21T15:49:26.445 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:26.445 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.445 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.445 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:26.445 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.975485+0000", 2024-09-21T15:49:26.445 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9540000000000001e-06 2024-09-21T15:49:26.445 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.445 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.445 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:26.445 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.975497+0000", 2024-09-21T15:49:26.445 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1934e-05 2024-09-21T15:49:26.445 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.445 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.446 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:26.446 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.975505+0000", 2024-09-21T15:49:26.446 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0250000000000002e-06 2024-09-21T15:49:26.446 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.446 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.446 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:26.446 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.975552+0000", 2024-09-21T15:49:26.446 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6281e-05 2024-09-21T15:49:26.446 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.446 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.446 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:26.446 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.975638+0000", 2024-09-21T15:49:26.446 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6762999999999996e-05 2024-09-21T15:49:26.446 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.446 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.447 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-21T15:49:26.447 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.975734+0000", 2024-09-21T15:49:26.447 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5248999999999996e-05 2024-09-21T15:49:26.447 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.447 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.447 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:26.447 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.978856+0000", 2024-09-21T15:49:26.447 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0031219580000000002 2024-09-21T15:49:26.447 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.447 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.447 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:26.447 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.984106+0000", 2024-09-21T15:49:26.447 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0052503589999999996 2024-09-21T15:49:26.447 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.447 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.447 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:26.448 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.984156+0000", 2024-09-21T15:49:26.448 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0164000000000003e-05 2024-09-21T15:49:26.448 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.448 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.448 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:26.448 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:18.984192+0000", 2024-09-21T15:49:26.448 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5658999999999999e-05 2024-09-21T15:49:26.448 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.448 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:26.448 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.448 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.448 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.448 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:14536 3.a 3:56ed26f3:::rbd_data.1133dc5adfdf.00000000000000ad:head [set-alloc-hint object_size 1048576 write_size 1048576,write 876544~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-21T15:49:26.448 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:19.118136+0000", 2024-09-21T15:49:26.448 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.2538664160000002, 2024-09-21T15:49:26.448 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.005992224, 2024-09-21T15:49:26.449 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:26.449 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:26.449 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:26.449 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:26.449 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:26.449 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 14536 2024-09-21T15:49:26.449 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.449 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:26.449 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.449 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:26.449 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.118136+0000", 2024-09-21T15:49:26.449 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.449 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.449 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.449 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:26.449 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.118136+0000", 2024-09-21T15:49:26.450 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.450 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.450 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.450 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:26.450 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.118134+0000", 2024-09-21T15:49:26.450 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999981 2024-09-21T15:49:26.450 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.450 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.450 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:26.450 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.118146+0000", 2024-09-21T15:49:26.450 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1565e-05 2024-09-21T15:49:26.450 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.450 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.450 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:26.450 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.118164+0000", 2024-09-21T15:49:26.451 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8850000000000001e-05 2024-09-21T15:49:26.451 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.451 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.451 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:26.451 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.118172+0000", 2024-09-21T15:49:26.451 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5329999999999997e-06 2024-09-21T15:49:26.451 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.451 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.451 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:26.451 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.118222+0000", 2024-09-21T15:49:26.451 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0077999999999997e-05 2024-09-21T15:49:26.451 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.451 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.451 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:26.451 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.118331+0000", 2024-09-21T15:49:26.452 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000109174 2024-09-21T15:49:26.452 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.452 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.452 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-21T15:49:26.452 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.118414+0000", 2024-09-21T15:49:26.452 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2701999999999995e-05 2024-09-21T15:49:26.452 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.452 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.452 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:26.452 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.120970+0000", 2024-09-21T15:49:26.452 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0025564250000000002 2024-09-21T15:49:26.452 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.452 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.452 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:26.453 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.124045+0000", 2024-09-21T15:49:26.453 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0030747309999999998 2024-09-21T15:49:26.453 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.453 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.453 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:26.453 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.124091+0000", 2024-09-21T15:49:26.453 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6276000000000002e-05 2024-09-21T15:49:26.453 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.453 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.453 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:26.453 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.124128+0000", 2024-09-21T15:49:26.453 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6566000000000002e-05 2024-09-21T15:49:26.453 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.453 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:26.454 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.454 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.454 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.454 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:14539 3.a 3:52d3dd80:::rbd_data.1133dc5adfdf.00000000000000b0:head [set-alloc-hint object_size 1048576 write_size 1048576,write 876544~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-21T15:49:26.454 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:19.121064+0000", 2024-09-21T15:49:26.454 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.2509378910000004, 2024-09-21T15:49:26.454 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0080418529999999998, 2024-09-21T15:49:26.454 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:26.454 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:26.454 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:26.454 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:26.454 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:26.454 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 14539 2024-09-21T15:49:26.454 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.454 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:26.455 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.455 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:26.455 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.121064+0000", 2024-09-21T15:49:26.455 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.455 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.455 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.455 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:26.455 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.121064+0000", 2024-09-21T15:49:26.455 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.455 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.455 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.455 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:26.455 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.121063+0000", 2024-09-21T15:49:26.455 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-21T15:49:26.455 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.455 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.456 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:26.456 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.121074+0000", 2024-09-21T15:49:26.456 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1372000000000001e-05 2024-09-21T15:49:26.456 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.456 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.456 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:26.456 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.121087+0000", 2024-09-21T15:49:26.456 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2237e-05 2024-09-21T15:49:26.456 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.456 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.456 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:26.456 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.121098+0000", 2024-09-21T15:49:26.456 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0842000000000001e-05 2024-09-21T15:49:26.456 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.456 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.456 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:26.457 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.121137+0000", 2024-09-21T15:49:26.457 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9378000000000001e-05 2024-09-21T15:49:26.457 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.457 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.457 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:26.457 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.121211+0000", 2024-09-21T15:49:26.457 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3714999999999995e-05 2024-09-21T15:49:26.457 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.457 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.457 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-21T15:49:26.457 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.121333+0000", 2024-09-21T15:49:26.457 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000122745 2024-09-21T15:49:26.457 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.457 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.457 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:26.457 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.126370+0000", 2024-09-21T15:49:26.458 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0050369580000000002 2024-09-21T15:49:26.458 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.458 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.458 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:26.458 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.129048+0000", 2024-09-21T15:49:26.458 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002678153 2024-09-21T15:49:26.458 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.458 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.458 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:26.458 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.129078+0000", 2024-09-21T15:49:26.458 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9889000000000001e-05 2024-09-21T15:49:26.458 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.458 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.458 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:26.458 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.129106+0000", 2024-09-21T15:49:26.459 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7721000000000001e-05 2024-09-21T15:49:26.459 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.459 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:26.459 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.459 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.459 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.459 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4403.0:14541 3.1e e33/22 3:7e48baef:::rbd_data.1133dc5adfdf.00000000000000b2:head v 33'102, mlcod=33'100)", 2024-09-21T15:49:26.459 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:19.121771+0000", 2024-09-21T15:49:26.459 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.2502311859999997, 2024-09-21T15:49:26.459 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0072650869999999999, 2024-09-21T15:49:26.459 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:26.459 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:26.459 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:26.459 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.459 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:26.460 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.121771+0000", 2024-09-21T15:49:26.460 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.460 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.460 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.460 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:26.460 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.121771+0000", 2024-09-21T15:49:26.460 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.460 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.460 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.460 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:26.460 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.121770+0000", 2024-09-21T15:49:26.460 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-21T15:49:26.460 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.460 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.460 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:26.460 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.121778+0000", 2024-09-21T15:49:26.461 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5059999999999995e-06 2024-09-21T15:49:26.461 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.461 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.461 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:26.461 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.121788+0000", 2024-09-21T15:49:26.461 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2310000000000002e-06 2024-09-21T15:49:26.461 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.461 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.461 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:26.461 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.121792+0000", 2024-09-21T15:49:26.461 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4440000000000001e-06 2024-09-21T15:49:26.461 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.461 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.461 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:26.461 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.121823+0000", 2024-09-21T15:49:26.461 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1148000000000001e-05 2024-09-21T15:49:26.461 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.462 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.462 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:26.462 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.121836+0000", 2024-09-21T15:49:26.462 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3101e-05 2024-09-21T15:49:26.462 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.462 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.462 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:26.462 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.129003+0000", 2024-09-21T15:49:26.462 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0071666170000000001 2024-09-21T15:49:26.462 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.462 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.462 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:26.462 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.129036+0000", 2024-09-21T15:49:26.462 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3030000000000001e-05 2024-09-21T15:49:26.463 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.463 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:26.463 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.463 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.463 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.463 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4403.0:14558 3.8 e33/32 3:13437810:::rbd_data.1133dc5adfdf.0000000000000039:head v 33'57, mlcod=33'55)", 2024-09-21T15:49:26.463 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:19.139921+0000", 2024-09-21T15:49:26.463 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.2320814840000001, 2024-09-21T15:49:26.463 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0048845210000000002, 2024-09-21T15:49:26.463 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:26.463 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:26.463 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:26.463 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.463 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:26.463 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.139921+0000", 2024-09-21T15:49:26.463 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.464 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.464 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.464 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:26.464 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.139921+0000", 2024-09-21T15:49:26.464 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.464 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.464 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.464 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:26.464 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.139920+0000", 2024-09-21T15:49:26.464 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-21T15:49:26.464 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.464 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.464 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:26.464 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.139931+0000", 2024-09-21T15:49:26.464 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1138000000000001e-05 2024-09-21T15:49:26.465 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.465 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.465 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:26.465 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.139946+0000", 2024-09-21T15:49:26.465 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5622999999999999e-05 2024-09-21T15:49:26.465 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.465 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.465 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:26.465 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.139954+0000", 2024-09-21T15:49:26.465 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7559999999999999e-06 2024-09-21T15:49:26.465 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.465 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.465 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:26.465 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.139999+0000", 2024-09-21T15:49:26.465 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4481000000000003e-05 2024-09-21T15:49:26.465 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.466 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.466 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:26.466 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.140028+0000", 2024-09-21T15:49:26.466 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9045999999999999e-05 2024-09-21T15:49:26.466 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.466 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.466 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:26.466 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.144779+0000", 2024-09-21T15:49:26.466 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.004751529 2024-09-21T15:49:26.466 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.466 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.466 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:26.466 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:19.144805+0000", 2024-09-21T15:49:26.466 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5933000000000001e-05 2024-09-21T15:49:26.466 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.467 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:26.467 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.467 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.467 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.467 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:24843 3.a 3:5401b18d:::rbd_data.1133ebd6d3ba.00000000000000de:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:26.467 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.190719+0000", 2024-09-21T15:49:26.467 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.181282835, 2024-09-21T15:49:26.467 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0053147089999999999, 2024-09-21T15:49:26.467 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:26.467 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:26.467 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:26.467 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:26.467 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:26.467 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 24843 2024-09-21T15:49:26.468 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.468 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:26.468 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.468 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:26.468 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.190719+0000", 2024-09-21T15:49:26.468 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.468 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.468 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.468 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:26.468 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.190719+0000", 2024-09-21T15:49:26.468 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.468 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.468 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.468 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:26.468 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.190719+0000", 2024-09-21T15:49:26.468 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:26.469 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.469 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.469 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:26.469 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.190722+0000", 2024-09-21T15:49:26.469 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2739999999999999e-06 2024-09-21T15:49:26.469 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.469 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.469 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:26.469 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.190733+0000", 2024-09-21T15:49:26.469 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1401e-05 2024-09-21T15:49:26.469 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.469 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.469 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:26.469 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.190740+0000", 2024-09-21T15:49:26.469 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1970000000000001e-06 2024-09-21T15:49:26.470 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.470 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.470 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:26.470 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.191317+0000", 2024-09-21T15:49:26.470 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00057769500000000003 2024-09-21T15:49:26.470 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.470 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.470 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:26.470 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.191386+0000", 2024-09-21T15:49:26.470 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8578000000000006e-05 2024-09-21T15:49:26.470 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.470 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.470 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-21T15:49:26.470 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.191475+0000", 2024-09-21T15:49:26.470 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.9650999999999999e-05 2024-09-21T15:49:26.471 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.471 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.471 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:26.471 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.195687+0000", 2024-09-21T15:49:26.471 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0042116100000000002 2024-09-21T15:49:26.471 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.471 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.471 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:26.471 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.195981+0000", 2024-09-21T15:49:26.471 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000293567 2024-09-21T15:49:26.471 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.471 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.471 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:26.471 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.196008+0000", 2024-09-21T15:49:26.471 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7277999999999999e-05 2024-09-21T15:49:26.471 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.472 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.472 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:26.472 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.196034+0000", 2024-09-21T15:49:26.472 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6037000000000002e-05 2024-09-21T15:49:26.472 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.472 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:26.472 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.472 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.472 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.472 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:25242 3.a 3:541615ed:::rbd_data.1133ebd6d3ba.0000000000000005:head [call rbd.sparse_copyup in=16b,set-alloc-hint object_size 1048576 write_size 1048576,write 1016649~7351] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:26.472 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.254105+0000", 2024-09-21T15:49:26.472 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.117896759, 2024-09-21T15:49:26.472 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0059636489999999997, 2024-09-21T15:49:26.472 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:26.472 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:26.472 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:26.473 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:26.473 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:26.473 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 25242 2024-09-21T15:49:26.473 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.473 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:26.473 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.473 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:26.473 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.254105+0000", 2024-09-21T15:49:26.473 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.473 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.473 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.473 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:26.473 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.254105+0000", 2024-09-21T15:49:26.473 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.473 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.473 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.474 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:26.474 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.254103+0000", 2024-09-21T15:49:26.474 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999981 2024-09-21T15:49:26.474 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.474 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.474 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:26.474 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.256049+0000", 2024-09-21T15:49:26.474 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0019453910000000001 2024-09-21T15:49:26.474 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.474 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.474 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:26.474 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.256064+0000", 2024-09-21T15:49:26.474 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5273000000000001e-05 2024-09-21T15:49:26.474 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.474 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.475 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:26.475 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.256071+0000", 2024-09-21T15:49:26.475 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9750000000000001e-06 2024-09-21T15:49:26.475 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.475 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.475 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:26.475 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.256125+0000", 2024-09-21T15:49:26.475 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3813000000000001e-05 2024-09-21T15:49:26.475 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.475 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.475 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:26.475 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.256285+0000", 2024-09-21T15:49:26.475 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000160185 2024-09-21T15:49:26.475 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.475 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.475 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 4", 2024-09-21T15:49:26.476 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.256557+0000", 2024-09-21T15:49:26.476 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00027169299999999999 2024-09-21T15:49:26.476 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.476 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.476 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:26.476 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.259108+0000", 2024-09-21T15:49:26.476 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.002551396 2024-09-21T15:49:26.476 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.476 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.476 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:26.476 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.259999+0000", 2024-09-21T15:49:26.476 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00089094900000000004 2024-09-21T15:49:26.476 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.476 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.476 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:26.476 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.260042+0000", 2024-09-21T15:49:26.477 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2540999999999998e-05 2024-09-21T15:49:26.477 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.477 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.477 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:26.477 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.260069+0000", 2024-09-21T15:49:26.477 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7285e-05 2024-09-21T15:49:26.477 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.477 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:26.477 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.477 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.477 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:26.477 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:26.479 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T15:49:26.479 INFO:teuthology.orchestra.run.smithi089.stdout:1674 write 0x6d2bfb4 thru 0x6d37a45 (0xba92 bytes) 2024-09-21T15:49:26.480 INFO:teuthology.orchestra.run.smithi089.stdout:1676 punch from 0x668bde5 to 0x66985a9, (0xc7c4 bytes) 2024-09-21T15:49:26.480 INFO:teuthology.orchestra.run.smithi089.stdout:1677 read 0x97e4fc thru 0x986084 (0x7b89 bytes) 2024-09-21T15:49:26.480 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:26.480 INFO:teuthology.orchestra.run.smithi089.stdout:1682 write 0x51076d thru 0x515533 (0x4dc7 bytes) 2024-09-21T15:49:26.480 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:26.480 INFO:teuthology.orchestra.run.smithi089.stdout:1684 write 0x6c54f89 thru 0x6c5f5bc (0xa634 bytes) 2024-09-21T15:49:26.480 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:26.480 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:26.480 INFO:teuthology.orchestra.run.smithi089.stdout:1688 writesame 0xcb4b0c6 thru 0xcb5a1c6 data_size 0x1(0xf101 bytes) 2024-09-21T15:49:26.480 INFO:teuthology.orchestra.run.smithi089.stdout:1689 read 0x31da545 thru 0x31e5a93 (0xb54f bytes) 2024-09-21T15:49:26.480 INFO:teuthology.orchestra.run.smithi089.stdout:1691 writesame 0x84acdfd thru 0x84bcc5e data_size 0x1(0xfe62 bytes) 2024-09-21T15:49:26.480 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:26.477+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:26.486 INFO:teuthology.orchestra.run.smithi089.stdout:1692 punch from 0x25b8a20 to 0x25c25b2, (0x9b92 bytes) 2024-09-21T15:49:26.486 INFO:teuthology.orchestra.run.smithi089.stdout:1693 read 0x3fbcf57 thru 0x3fc8cc5 (0xbd6f bytes) 2024-09-21T15:49:26.486 INFO:teuthology.orchestra.run.smithi089.stdout:1694 read 0x7bc61d0 thru 0x7bcba10 (0x5841 bytes) 2024-09-21T15:49:26.486 INFO:teuthology.orchestra.run.smithi089.stdout:1695 writesame 0x7cc441c thru 0x7cc45d8 data_size 0x1(0x1bd bytes) 2024-09-21T15:49:26.488 INFO:teuthology.orchestra.run.smithi089.stdout:1696 read 0x874d0fd thru 0x8757ba4 (0xaaa8 bytes) 2024-09-21T15:49:26.489 INFO:teuthology.orchestra.run.smithi089.stdout:1697 read 0x787364 thru 0x7921ea (0xae87 bytes) 2024-09-21T15:49:26.490 INFO:teuthology.orchestra.run.smithi089.stdout:1698 punch from 0x164fbdb to 0x165851c, (0x8941 bytes) 2024-09-21T15:49:26.490 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:26.490 INFO:teuthology.orchestra.run.smithi089.stdout:1700 punch from 0x61496b6 to 0x614ae84, (0x17ce bytes) 2024-09-21T15:49:26.490 INFO:teuthology.orchestra.run.smithi089.stdout:1701 write 0xaa56a39 thru 0xaa59564 (0x2b2c bytes) 2024-09-21T15:49:26.497 INFO:teuthology.orchestra.run.smithi089.stdout:1702 read 0x5579d41 thru 0x557dd25 (0x3fe5 bytes) 2024-09-21T15:49:26.497 INFO:teuthology.orchestra.run.smithi089.stdout:1704 read 0xa02101 thru 0xa05aba (0x39ba bytes) 2024-09-21T15:49:26.497 INFO:teuthology.orchestra.run.smithi089.stdout:1707 write 0x9d4f06 thru 0x9d8d45 (0x3e40 bytes) 2024-09-21T15:49:26.548 INFO:teuthology.orchestra.run.smithi089.stdout:1710 read 0xbe91570 thru 0xbe9f76c (0xe1fd bytes) 2024-09-21T15:49:26.548 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:26.549 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:26.549 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:26.549 INFO:teuthology.orchestra.run.smithi089.stdout:1714 punch from 0x270310e to 0x271244d, (0xf33f bytes) 2024-09-21T15:49:26.549 INFO:teuthology.orchestra.run.smithi089.stdout:1716 read 0x4f8bd4a thru 0x4f98d93 (0xd04a bytes) 2024-09-21T15:49:26.549 INFO:teuthology.orchestra.run.smithi089.stdout:1719 read 0x11d82ca thru 0x11e3a73 (0xb7aa bytes) 2024-09-21T15:49:26.554 INFO:teuthology.orchestra.run.smithi089.stdout:1720 trunc from 0xcb5a1c7 to 0x908cc1e 2024-09-21T15:49:26.572 INFO:teuthology.orchestra.run.smithi089.stdout:1722 writesame 0xadf1796 thru 0xadff25b data_size 0x29(0xdac6 bytes) 2024-09-21T15:49:26.572 INFO:teuthology.orchestra.run.smithi089.stdout:1723 read 0x7cf8304 thru 0x7d03ec8 (0xbbc5 bytes) 2024-09-21T15:49:26.572 INFO:teuthology.orchestra.run.smithi089.stdout:1724 write 0xa8f9361 thru 0xa8fd9aa (0x464a bytes) 2024-09-21T15:49:26.577 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:26.577 INFO:teuthology.orchestra.run.smithi089.stdout:1727 clone 7 order 22 su 65536 sc 12 2024-09-21T15:49:26.578 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:26.577+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:26.627 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:26.627 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:26.627 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:49:26.627 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:26.635 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T15:49:26.679 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:26.678+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:26.737 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:26.737 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:26.737 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:49:26.737 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:49:26.737 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:26.746 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T15:49:26.781 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:26.780+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:26.782 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T15:49:26.825 INFO:teuthology.orchestra.run.smithi089.stdout:leaving image image_client.0-clone6 intact 2024-09-21T15:49:26.882 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:26.881+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:26.890 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:26.890 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:49:26.890 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:49:26.890 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:49:26.891 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.891 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4209.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:49:26.891 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:49:26.891 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 35.697720932999999, 2024-09-21T15:49:26.891 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.19452909900000001, 2024-09-21T15:49:26.891 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:26.891 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:26.891 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:26.891 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4209", 2024-09-21T15:49:26.891 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/4045143050", 2024-09-21T15:49:26.891 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1 2024-09-21T15:49:26.891 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.891 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:26.891 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.892 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:26.892 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:49:26.892 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.892 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.892 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.892 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:26.892 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:49:26.892 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.892 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.892 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.892 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:26.892 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191161+0000", 2024-09-21T15:49:26.892 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:26.892 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.892 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.892 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:26.893 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191169+0000", 2024-09-21T15:49:26.893 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0090000000000007e-06 2024-09-21T15:49:26.893 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.893 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.893 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:26.893 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191180+0000", 2024-09-21T15:49:26.893 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0728e-05 2024-09-21T15:49:26.893 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.893 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.893 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:26.893 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191187+0000", 2024-09-21T15:49:26.893 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9009999999999994e-06 2024-09-21T15:49:26.893 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.893 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.893 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:26.893 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191238+0000", 2024-09-21T15:49:26.894 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0287000000000002e-05 2024-09-21T15:49:26.894 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.894 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.894 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-21T15:49:26.894 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191245+0000", 2024-09-21T15:49:26.894 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0709999999999997e-06 2024-09-21T15:49:26.894 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.894 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.894 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:26.894 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380309+0000", 2024-09-21T15:49:26.894 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.189064277 2024-09-21T15:49:26.894 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.894 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.894 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:26.895 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380454+0000", 2024-09-21T15:49:26.895 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014490799999999999 2024-09-21T15:49:26.895 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.895 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.895 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0", 2024-09-21T15:49:26.895 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380610+0000", 2024-09-21T15:49:26.895 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015650199999999999 2024-09-21T15:49:26.895 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.895 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.895 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:26.895 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.381160+0000", 2024-09-21T15:49:26.895 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00054903799999999996 2024-09-21T15:49:26.895 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.895 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.896 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:26.896 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385591+0000", 2024-09-21T15:49:26.896 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.004431272 2024-09-21T15:49:26.896 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.896 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.896 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:26.896 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385651+0000", 2024-09-21T15:49:26.896 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0455999999999999e-05 2024-09-21T15:49:26.896 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.896 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.896 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:26.896 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385691+0000", 2024-09-21T15:49:26.896 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0234999999999997e-05 2024-09-21T15:49:26.896 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.897 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:26.897 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.897 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.897 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.897 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:1838 3.11 3:8b85faea:::rbd_data.1133f3dfbd6b.000000000000000c:5 [read 7348224~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:26.897 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.093286+0000", 2024-09-21T15:49:26.897 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.795597186, 2024-09-21T15:49:26.897 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.042871797000000003, 2024-09-21T15:49:26.897 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:26.897 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:26.897 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:26.897 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:26.897 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:26.897 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1838 2024-09-21T15:49:26.897 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.898 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:26.898 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.898 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:26.898 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.093286+0000", 2024-09-21T15:49:26.898 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.898 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.898 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.898 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:26.898 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.093286+0000", 2024-09-21T15:49:26.898 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.898 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.898 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.898 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:26.898 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.093174+0000", 2024-09-21T15:49:26.898 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998879 2024-09-21T15:49:26.899 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.899 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.899 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:26.899 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.093288+0000", 2024-09-21T15:49:26.899 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00011387600000000001 2024-09-21T15:49:26.899 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.899 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.899 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:26.899 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.093293+0000", 2024-09-21T15:49:26.899 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7600000000000002e-06 2024-09-21T15:49:26.899 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.899 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.899 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:26.899 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.093302+0000", 2024-09-21T15:49:26.899 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3649999999999993e-06 2024-09-21T15:49:26.899 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.900 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.900 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:26.900 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.135394+0000", 2024-09-21T15:49:26.900 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.042091954000000001 2024-09-21T15:49:26.900 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.900 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.900 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:26.900 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.135538+0000", 2024-09-21T15:49:26.900 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014408199999999999 2024-09-21T15:49:26.900 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.900 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.900 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:26.900 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.136158+0000", 2024-09-21T15:49:26.900 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00061959500000000002 2024-09-21T15:49:26.900 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.900 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:26.901 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.901 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.901 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.901 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:1858 3.11 3:8b85faea:::rbd_data.1133f3dfbd6b.000000000000000c:5 [read 8331264~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:26.901 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.096865+0000", 2024-09-21T15:49:26.901 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.792018186, 2024-09-21T15:49:26.901 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.042908221000000003, 2024-09-21T15:49:26.901 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:26.901 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:26.901 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:26.901 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:26.901 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:26.901 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1858 2024-09-21T15:49:26.901 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.901 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:26.902 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.902 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:26.902 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.096865+0000", 2024-09-21T15:49:26.902 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.902 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.902 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.902 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:26.902 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.096865+0000", 2024-09-21T15:49:26.902 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.902 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.902 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.902 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:26.902 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.096865+0000", 2024-09-21T15:49:26.902 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:26.902 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.902 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.903 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:26.903 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.096870+0000", 2024-09-21T15:49:26.903 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0710000000000002e-06 2024-09-21T15:49:26.903 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.903 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.903 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:26.903 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.096873+0000", 2024-09-21T15:49:26.903 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.005e-06 2024-09-21T15:49:26.903 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.903 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.903 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:26.903 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.096875+0000", 2024-09-21T15:49:26.903 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2740000000000002e-06 2024-09-21T15:49:26.903 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.904 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.904 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:26.904 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.139613+0000", 2024-09-21T15:49:26.904 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.042737334000000002 2024-09-21T15:49:26.904 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.904 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.904 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:26.904 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.139699+0000", 2024-09-21T15:49:26.904 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6687999999999994e-05 2024-09-21T15:49:26.904 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.904 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.904 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:26.904 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.139773+0000", 2024-09-21T15:49:26.904 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4053000000000004e-05 2024-09-21T15:49:26.904 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.904 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:26.905 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.905 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.905 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.905 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:1859 3.11 3:8b85faea:::rbd_data.1133f3dfbd6b.000000000000000c:5 [read 8380416~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:26.905 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.097017+0000", 2024-09-21T15:49:26.905 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.791866738, 2024-09-21T15:49:26.905 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.042963952999999999, 2024-09-21T15:49:26.905 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:26.905 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:26.905 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:26.905 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:26.905 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:26.905 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1859 2024-09-21T15:49:26.905 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.905 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:26.905 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.906 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:26.906 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.097017+0000", 2024-09-21T15:49:26.906 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.906 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.906 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.906 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:26.906 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.097017+0000", 2024-09-21T15:49:26.906 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.906 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.906 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.906 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:26.906 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.096877+0000", 2024-09-21T15:49:26.906 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998608 2024-09-21T15:49:26.906 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.906 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.907 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:26.907 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.097020+0000", 2024-09-21T15:49:26.907 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014238 2024-09-21T15:49:26.907 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.907 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.907 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:26.907 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.097026+0000", 2024-09-21T15:49:26.907 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8310000000000003e-06 2024-09-21T15:49:26.907 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.907 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.907 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:26.907 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.097032+0000", 2024-09-21T15:49:26.907 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4249999999999997e-06 2024-09-21T15:49:26.907 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.907 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.907 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:26.908 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.139813+0000", 2024-09-21T15:49:26.908 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.042781053999999999 2024-09-21T15:49:26.908 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.908 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.908 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:26.908 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.139901+0000", 2024-09-21T15:49:26.908 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8472000000000003e-05 2024-09-21T15:49:26.908 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.908 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.908 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:26.908 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.139981+0000", 2024-09-21T15:49:26.908 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9128000000000005e-05 2024-09-21T15:49:26.908 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.908 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:26.908 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.909 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.909 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.909 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:1871 3.11 3:8b85faea:::rbd_data.1133f3dfbd6b.000000000000000c:5 [read 6860800~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:26.909 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.099208+0000", 2024-09-21T15:49:26.909 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.789674870000001, 2024-09-21T15:49:26.909 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.042862763999999998, 2024-09-21T15:49:26.909 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:26.909 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:26.909 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:26.909 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:26.909 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:26.909 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1871 2024-09-21T15:49:26.909 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.909 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:26.910 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.910 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:26.910 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.099208+0000", 2024-09-21T15:49:26.910 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.910 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.910 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.910 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:26.910 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.099208+0000", 2024-09-21T15:49:26.910 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.910 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.910 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.910 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:26.910 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.099208+0000", 2024-09-21T15:49:26.910 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:26.911 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.911 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.911 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:26.911 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.099210+0000", 2024-09-21T15:49:26.911 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.12e-06 2024-09-21T15:49:26.911 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.911 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.911 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:26.911 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.099219+0000", 2024-09-21T15:49:26.911 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1370000000000008e-06 2024-09-21T15:49:26.911 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.911 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.911 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:26.911 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.099224+0000", 2024-09-21T15:49:26.912 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6110000000000004e-06 2024-09-21T15:49:26.912 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.912 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.912 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:26.912 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.141899+0000", 2024-09-21T15:49:26.912 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.042675376000000001 2024-09-21T15:49:26.912 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.912 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.912 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:26.912 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.141988+0000", 2024-09-21T15:49:26.912 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8565000000000003e-05 2024-09-21T15:49:26.912 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.912 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.912 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:26.912 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.142071+0000", 2024-09-21T15:49:26.912 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3325999999999995e-05 2024-09-21T15:49:26.913 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.913 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:26.913 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.913 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.913 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.913 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:6182 3.e 3:734145f0:::rbd_data.1133f3dfbd6b.0000000000000003:5 [read 618496~4096] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:26.913 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.736429+0000", 2024-09-21T15:49:26.913 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.152454305999999, 2024-09-21T15:49:26.913 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.042734435000000001, 2024-09-21T15:49:26.913 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:26.913 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:26.913 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:26.913 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:26.913 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:26.913 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 6182 2024-09-21T15:49:26.914 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.914 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:26.914 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.914 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:26.914 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736429+0000", 2024-09-21T15:49:26.914 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.914 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.914 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.914 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:26.914 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736429+0000", 2024-09-21T15:49:26.914 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.914 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.914 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.914 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:26.914 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736429+0000", 2024-09-21T15:49:26.915 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:26.915 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.915 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.915 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:26.915 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736430+0000", 2024-09-21T15:49:26.915 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7900000000000007e-07 2024-09-21T15:49:26.915 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.915 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.915 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:26.915 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736433+0000", 2024-09-21T15:49:26.915 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7140000000000002e-06 2024-09-21T15:49:26.915 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.915 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.915 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:26.915 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736435+0000", 2024-09-21T15:49:26.916 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.357e-06 2024-09-21T15:49:26.916 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.916 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.916 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:26.916 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.779070+0000", 2024-09-21T15:49:26.916 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.042635550000000001 2024-09-21T15:49:26.916 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.916 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.916 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:26.916 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.779112+0000", 2024-09-21T15:49:26.916 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1124000000000001e-05 2024-09-21T15:49:26.916 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.916 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.916 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:26.916 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.779163+0000", 2024-09-21T15:49:26.917 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1913000000000003e-05 2024-09-21T15:49:26.917 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.917 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:26.917 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.917 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.917 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.917 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:6183 3.e 3:734145f0:::rbd_data.1133f3dfbd6b.0000000000000003:5 [read 667648~4096] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:26.917 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.736437+0000", 2024-09-21T15:49:26.917 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.152446475, 2024-09-21T15:49:26.917 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.042929953, 2024-09-21T15:49:26.917 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:26.917 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:26.917 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:26.917 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:26.917 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:26.917 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 6183 2024-09-21T15:49:26.918 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.918 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:26.918 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.918 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:26.918 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736437+0000", 2024-09-21T15:49:26.918 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.918 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.918 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.918 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:26.918 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736437+0000", 2024-09-21T15:49:26.918 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.918 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.918 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.919 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:26.919 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736437+0000", 2024-09-21T15:49:26.919 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:26.919 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.919 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.919 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:26.919 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736441+0000", 2024-09-21T15:49:26.919 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8269999999999997e-06 2024-09-21T15:49:26.919 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.919 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.919 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:26.919 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736445+0000", 2024-09-21T15:49:26.919 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5329999999999999e-06 2024-09-21T15:49:26.919 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.919 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.920 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:26.920 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736447+0000", 2024-09-21T15:49:26.920 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3989999999999998e-06 2024-09-21T15:49:26.920 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.920 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.920 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:26.920 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.779194+0000", 2024-09-21T15:49:26.920 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.042746447999999999 2024-09-21T15:49:26.920 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.920 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.920 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:26.920 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.779280+0000", 2024-09-21T15:49:26.920 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6304999999999994e-05 2024-09-21T15:49:26.920 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.920 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.921 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:26.921 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.779367+0000", 2024-09-21T15:49:26.921 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6643000000000004e-05 2024-09-21T15:49:26.921 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.921 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:26.921 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.921 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.921 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.921 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:6184 3.e 3:734145f0:::rbd_data.1133f3dfbd6b.0000000000000003:5 [read 716800~4096] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:26.921 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.736450+0000", 2024-09-21T15:49:26.921 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.152433792, 2024-09-21T15:49:26.921 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.043020819000000002, 2024-09-21T15:49:26.921 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:26.921 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:26.922 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:26.922 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:26.922 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:26.922 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 6184 2024-09-21T15:49:26.922 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.922 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:26.922 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.922 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:26.922 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736450+0000", 2024-09-21T15:49:26.922 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.922 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.922 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.922 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:26.922 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736450+0000", 2024-09-21T15:49:26.922 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.923 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.923 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.923 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:26.923 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736449+0000", 2024-09-21T15:49:26.923 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:26.923 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.923 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.923 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:26.923 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736450+0000", 2024-09-21T15:49:26.923 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0589999999999999e-06 2024-09-21T15:49:26.923 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.923 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.923 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:26.923 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736453+0000", 2024-09-21T15:49:26.924 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8080000000000001e-06 2024-09-21T15:49:26.924 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.924 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.924 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:26.924 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736455+0000", 2024-09-21T15:49:26.924 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2809999999999998e-06 2024-09-21T15:49:26.924 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.924 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.924 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:26.924 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.779378+0000", 2024-09-21T15:49:26.924 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.042922279000000001 2024-09-21T15:49:26.924 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.924 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.924 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:26.924 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.779419+0000", 2024-09-21T15:49:26.925 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1726e-05 2024-09-21T15:49:26.925 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.925 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.925 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:26.925 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.779470+0000", 2024-09-21T15:49:26.925 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0871999999999997e-05 2024-09-21T15:49:26.925 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.925 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:26.925 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.925 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.925 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.925 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:6185 3.e 3:734145f0:::rbd_data.1133f3dfbd6b.0000000000000003:5 [read 765952~4096] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:26.925 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.736458+0000", 2024-09-21T15:49:26.925 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.1524258, 2024-09-21T15:49:26.926 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.043226015, 2024-09-21T15:49:26.926 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:26.926 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:26.926 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:26.926 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:26.926 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:26.926 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 6185 2024-09-21T15:49:26.926 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.926 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:26.926 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.926 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:26.926 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736458+0000", 2024-09-21T15:49:26.926 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.926 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.927 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.927 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:26.927 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736458+0000", 2024-09-21T15:49:26.927 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.927 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.927 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.927 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:26.927 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736457+0000", 2024-09-21T15:49:26.927 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:26.927 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.927 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.927 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:26.927 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736458+0000", 2024-09-21T15:49:26.927 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0279999999999999e-06 2024-09-21T15:49:26.927 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.928 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.928 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:26.928 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736461+0000", 2024-09-21T15:49:26.928 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0510000000000001e-06 2024-09-21T15:49:26.928 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.928 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.928 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:26.928 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736464+0000", 2024-09-21T15:49:26.928 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4159999999999998e-06 2024-09-21T15:49:26.928 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.928 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.928 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:26.928 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.779502+0000", 2024-09-21T15:49:26.928 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.043038218000000003 2024-09-21T15:49:26.928 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.928 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.929 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:26.929 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.779592+0000", 2024-09-21T15:49:26.929 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.9834999999999996e-05 2024-09-21T15:49:26.929 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.929 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.929 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:26.929 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.779684+0000", 2024-09-21T15:49:26.929 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1673000000000001e-05 2024-09-21T15:49:26.929 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.929 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:26.929 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.929 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.929 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.929 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:6186 3.e 3:734145f0:::rbd_data.1133f3dfbd6b.0000000000000003:5 [read 815104~4096] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:26.929 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.736466+0000", 2024-09-21T15:49:26.930 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.152417526000001, 2024-09-21T15:49:26.930 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.043438982000000001, 2024-09-21T15:49:26.930 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:26.930 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:26.930 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:26.930 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:26.930 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:26.930 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 6186 2024-09-21T15:49:26.930 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.930 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:26.930 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.930 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:26.930 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736466+0000", 2024-09-21T15:49:26.930 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.931 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.931 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.931 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:26.931 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736466+0000", 2024-09-21T15:49:26.931 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.931 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.931 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.931 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:26.931 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736466+0000", 2024-09-21T15:49:26.931 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:26.931 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.931 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.931 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:26.931 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736467+0000", 2024-09-21T15:49:26.931 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.009e-06 2024-09-21T15:49:26.932 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.932 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.932 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:26.932 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736469+0000", 2024-09-21T15:49:26.932 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8449999999999999e-06 2024-09-21T15:49:26.932 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.932 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.932 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:26.932 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736472+0000", 2024-09-21T15:49:26.932 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3140000000000002e-06 2024-09-21T15:49:26.932 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.932 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.932 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:26.932 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.779724+0000", 2024-09-21T15:49:26.932 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.043252295000000003 2024-09-21T15:49:26.932 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.932 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.933 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:26.933 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.779808+0000", 2024-09-21T15:49:26.933 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3765999999999997e-05 2024-09-21T15:49:26.933 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.933 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.933 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:26.933 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.779905+0000", 2024-09-21T15:49:26.933 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.6953999999999994e-05 2024-09-21T15:49:26.933 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.933 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:26.933 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.933 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.933 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.933 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:6187 3.e 3:734145f0:::rbd_data.1133f3dfbd6b.0000000000000003:5 [read 864256~4096] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:26.933 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.736474+0000", 2024-09-21T15:49:26.934 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.152409521999999, 2024-09-21T15:49:26.934 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.043635611999999997, 2024-09-21T15:49:26.934 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:26.934 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:26.934 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:26.934 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:26.934 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:26.934 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 6187 2024-09-21T15:49:26.934 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.934 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:26.934 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.934 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:26.934 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736474+0000", 2024-09-21T15:49:26.934 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.934 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.935 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.935 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:26.935 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736474+0000", 2024-09-21T15:49:26.935 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.935 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.935 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.935 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:26.935 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736474+0000", 2024-09-21T15:49:26.935 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:26.935 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.935 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.935 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:26.935 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736478+0000", 2024-09-21T15:49:26.935 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.843e-06 2024-09-21T15:49:26.935 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.936 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.936 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:26.936 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736481+0000", 2024-09-21T15:49:26.936 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.055e-06 2024-09-21T15:49:26.936 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.936 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.936 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:26.936 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736484+0000", 2024-09-21T15:49:26.936 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3649999999999998e-06 2024-09-21T15:49:26.936 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.936 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.936 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:26.936 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.779927+0000", 2024-09-21T15:49:26.936 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.043443038000000003 2024-09-21T15:49:26.936 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.937 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.937 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:26.937 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.780024+0000", 2024-09-21T15:49:26.937 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7087000000000001e-05 2024-09-21T15:49:26.937 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.937 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.937 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:26.937 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.780109+0000", 2024-09-21T15:49:26.937 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5422999999999998e-05 2024-09-21T15:49:26.937 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.937 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:26.937 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.937 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.937 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.937 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:6188 3.e 3:734145f0:::rbd_data.1133f3dfbd6b.0000000000000003:5 [read 913408~4096] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:26.937 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.736486+0000", 2024-09-21T15:49:26.938 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.152397213, 2024-09-21T15:49:26.938 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.043726186, 2024-09-21T15:49:26.938 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:26.938 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:26.938 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:26.938 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:26.938 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:26.938 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 6188 2024-09-21T15:49:26.938 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.938 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:26.938 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.938 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:26.938 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736486+0000", 2024-09-21T15:49:26.938 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.938 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.939 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.939 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:26.939 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736486+0000", 2024-09-21T15:49:26.939 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.939 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.939 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.939 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:26.939 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736486+0000", 2024-09-21T15:49:26.939 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:26.939 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.939 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.939 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:26.939 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736487+0000", 2024-09-21T15:49:26.939 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.079e-06 2024-09-21T15:49:26.940 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.940 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.940 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:26.940 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736490+0000", 2024-09-21T15:49:26.940 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7829999999999999e-06 2024-09-21T15:49:26.940 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.940 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.940 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:26.940 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736492+0000", 2024-09-21T15:49:26.940 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3190000000000002e-06 2024-09-21T15:49:26.940 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.940 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.940 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:26.940 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.780122+0000", 2024-09-21T15:49:26.941 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.043630245999999998 2024-09-21T15:49:26.941 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.941 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.941 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:26.941 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.780165+0000", 2024-09-21T15:49:26.941 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2833999999999997e-05 2024-09-21T15:49:26.941 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.941 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.941 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:26.941 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.780212+0000", 2024-09-21T15:49:26.941 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7134e-05 2024-09-21T15:49:26.941 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.941 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:26.942 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.942 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.942 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.942 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:6189 3.e 3:734145f0:::rbd_data.1133f3dfbd6b.0000000000000003:5 [read 962560~4096] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:26.942 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.736494+0000", 2024-09-21T15:49:26.942 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.152389019999999, 2024-09-21T15:49:26.942 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.043933139000000003, 2024-09-21T15:49:26.942 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:26.942 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:26.942 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:26.942 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:26.942 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:26.943 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 6189 2024-09-21T15:49:26.943 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.943 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:26.943 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.943 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:26.943 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736494+0000", 2024-09-21T15:49:26.943 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.943 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.943 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.943 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:26.943 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736494+0000", 2024-09-21T15:49:26.943 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.943 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.943 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.943 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:26.944 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736494+0000", 2024-09-21T15:49:26.944 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:26.944 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.944 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.944 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:26.944 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736495+0000", 2024-09-21T15:49:26.944 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.221e-06 2024-09-21T15:49:26.944 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.944 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.944 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:26.944 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736500+0000", 2024-09-21T15:49:26.944 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6029999999999998e-06 2024-09-21T15:49:26.944 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.944 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.944 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:26.944 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.736502+0000", 2024-09-21T15:49:26.945 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4889999999999998e-06 2024-09-21T15:49:26.945 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.945 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.945 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:26.945 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.780241+0000", 2024-09-21T15:49:26.945 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.043738491999999997 2024-09-21T15:49:26.945 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.945 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.945 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:26.945 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.780335+0000", 2024-09-21T15:49:26.945 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3783000000000007e-05 2024-09-21T15:49:26.945 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.945 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.945 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:26.945 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.780427+0000", 2024-09-21T15:49:26.945 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2912999999999996e-05 2024-09-21T15:49:26.945 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.946 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:26.946 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.946 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.946 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.946 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:6192 3.e 3:734145f0:::rbd_data.1133f3dfbd6b.0000000000000003:5 [read 1110016~4096] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:26.946 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.738041+0000", 2024-09-21T15:49:26.946 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.150842315, 2024-09-21T15:49:26.946 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.042836826000000001, 2024-09-21T15:49:26.946 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:26.946 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:26.946 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:26.946 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:26.946 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:26.946 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 6192 2024-09-21T15:49:26.946 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.947 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:26.947 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.947 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:26.947 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738041+0000", 2024-09-21T15:49:26.947 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.947 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.947 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.947 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:26.947 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738041+0000", 2024-09-21T15:49:26.947 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.947 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.947 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.948 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:26.948 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738041+0000", 2024-09-21T15:49:26.948 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:26.948 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.948 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.948 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:26.948 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738042+0000", 2024-09-21T15:49:26.948 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.144e-06 2024-09-21T15:49:26.948 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.948 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.948 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:26.948 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738046+0000", 2024-09-21T15:49:26.948 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.177e-06 2024-09-21T15:49:26.948 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.948 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.949 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:26.949 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738049+0000", 2024-09-21T15:49:26.949 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5270000000000001e-06 2024-09-21T15:49:26.949 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.949 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.949 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:26.949 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.780782+0000", 2024-09-21T15:49:26.949 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.042733110999999997 2024-09-21T15:49:26.949 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.949 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.949 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:26.949 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.780825+0000", 2024-09-21T15:49:26.949 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3578999999999997e-05 2024-09-21T15:49:26.949 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.950 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.950 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:26.950 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.780878+0000", 2024-09-21T15:49:26.950 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2493e-05 2024-09-21T15:49:26.950 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.950 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:26.950 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.950 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.950 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.950 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:6193 3.e 3:734145f0:::rbd_data.1133f3dfbd6b.0000000000000003:5 [read 1159168~4096] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:26.950 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.738051+0000", 2024-09-21T15:49:26.950 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.150832445000001, 2024-09-21T15:49:26.950 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.043033658000000002, 2024-09-21T15:49:26.950 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:26.951 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:26.951 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:26.951 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:26.951 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:26.951 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 6193 2024-09-21T15:49:26.951 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.951 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:26.951 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.951 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:26.951 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738051+0000", 2024-09-21T15:49:26.951 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.951 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.951 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.951 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:26.952 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738051+0000", 2024-09-21T15:49:26.952 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.952 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.952 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.952 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:26.952 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738051+0000", 2024-09-21T15:49:26.952 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:26.952 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.952 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.952 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:26.952 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738052+0000", 2024-09-21T15:49:26.952 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1790000000000001e-06 2024-09-21T15:49:26.952 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.952 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.952 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:26.953 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738055+0000", 2024-09-21T15:49:26.953 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8880000000000001e-06 2024-09-21T15:49:26.953 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.953 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.953 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:26.953 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738057+0000", 2024-09-21T15:49:26.953 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4700000000000001e-06 2024-09-21T15:49:26.953 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.953 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.953 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:26.953 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.780911+0000", 2024-09-21T15:49:26.953 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.042854056000000001 2024-09-21T15:49:26.953 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.953 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.953 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:26.954 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.780987+0000", 2024-09-21T15:49:26.954 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5543999999999994e-05 2024-09-21T15:49:26.954 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.954 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.954 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:26.954 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.781085+0000", 2024-09-21T15:49:26.954 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7722999999999999e-05 2024-09-21T15:49:26.954 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.954 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:26.954 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.954 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.954 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.954 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:6194 3.e 3:734145f0:::rbd_data.1133f3dfbd6b.0000000000000003:5 [read 1208320~4096] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:26.954 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.738059+0000", 2024-09-21T15:49:26.955 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.150824017, 2024-09-21T15:49:26.955 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.043212726, 2024-09-21T15:49:26.955 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:26.955 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:26.955 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:26.955 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:26.955 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:26.955 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 6194 2024-09-21T15:49:26.955 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.955 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:26.955 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.955 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:26.955 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738059+0000", 2024-09-21T15:49:26.955 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.956 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.956 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.956 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:26.956 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738059+0000", 2024-09-21T15:49:26.956 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.956 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.956 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.956 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:26.956 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738059+0000", 2024-09-21T15:49:26.956 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:26.956 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.956 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.956 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:26.956 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738060+0000", 2024-09-21T15:49:26.956 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.13e-06 2024-09-21T15:49:26.957 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.957 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.957 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:26.957 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738063+0000", 2024-09-21T15:49:26.957 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7130000000000001e-06 2024-09-21T15:49:26.957 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.957 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.957 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:26.957 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738066+0000", 2024-09-21T15:49:26.957 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3220000000000001e-06 2024-09-21T15:49:26.957 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.957 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.957 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:26.957 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.781126+0000", 2024-09-21T15:49:26.957 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.043059865000000003 2024-09-21T15:49:26.957 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.957 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.958 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:26.958 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.781205+0000", 2024-09-21T15:49:26.958 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8671999999999995e-05 2024-09-21T15:49:26.958 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.958 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.958 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:26.958 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.781272+0000", 2024-09-21T15:49:26.958 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7231999999999996e-05 2024-09-21T15:49:26.958 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.958 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:26.958 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.958 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.958 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.958 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:6195 3.e 3:734145f0:::rbd_data.1133f3dfbd6b.0000000000000003:5 [read 1257472~4096] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:26.958 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.738068+0000", 2024-09-21T15:49:26.959 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.150814883000001, 2024-09-21T15:49:26.959 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.043388518000000001, 2024-09-21T15:49:26.959 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:26.959 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:26.959 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:26.959 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:26.959 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:26.959 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 6195 2024-09-21T15:49:26.959 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.959 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:26.959 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.959 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:26.959 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738068+0000", 2024-09-21T15:49:26.959 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.959 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.959 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.960 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:26.960 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738068+0000", 2024-09-21T15:49:26.960 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.960 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.960 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.960 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:26.960 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738068+0000", 2024-09-21T15:49:26.960 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:26.960 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.960 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.960 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:26.960 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738069+0000", 2024-09-21T15:49:26.960 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.187e-06 2024-09-21T15:49:26.960 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.960 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.960 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:26.961 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738072+0000", 2024-09-21T15:49:26.961 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8830000000000002e-06 2024-09-21T15:49:26.961 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.961 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.961 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:26.961 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738075+0000", 2024-09-21T15:49:26.961 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4990000000000001e-06 2024-09-21T15:49:26.961 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.961 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.961 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:26.961 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.781302+0000", 2024-09-21T15:49:26.961 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.043227527000000002 2024-09-21T15:49:26.961 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.961 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.961 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:26.961 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.781374+0000", 2024-09-21T15:49:26.962 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1307e-05 2024-09-21T15:49:26.962 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.962 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.962 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:26.962 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.781457+0000", 2024-09-21T15:49:26.962 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3349000000000004e-05 2024-09-21T15:49:26.962 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.962 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:26.962 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.962 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.962 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.962 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:6196 3.e 3:734145f0:::rbd_data.1133f3dfbd6b.0000000000000003:5 [read 1306624~4096] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:26.962 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.738089+0000", 2024-09-21T15:49:26.962 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.150794629, 2024-09-21T15:49:26.962 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.043542355999999997, 2024-09-21T15:49:26.962 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:26.963 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:26.963 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:26.963 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:26.963 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:26.963 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 6196 2024-09-21T15:49:26.963 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.963 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:26.963 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.963 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:26.963 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738089+0000", 2024-09-21T15:49:26.963 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.963 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.963 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.963 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:26.964 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738089+0000", 2024-09-21T15:49:26.964 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.964 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.964 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.964 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:26.964 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738088+0000", 2024-09-21T15:49:26.964 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:26.964 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.964 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.964 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:26.964 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738090+0000", 2024-09-21T15:49:26.964 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.81e-06 2024-09-21T15:49:26.964 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.964 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.965 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:26.965 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738096+0000", 2024-09-21T15:49:26.965 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0140000000000004e-06 2024-09-21T15:49:26.965 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.965 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.965 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:26.965 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738099+0000", 2024-09-21T15:49:26.965 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.723e-06 2024-09-21T15:49:26.965 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.965 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.965 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:26.965 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.781482+0000", 2024-09-21T15:49:26.965 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.043383065999999998 2024-09-21T15:49:26.965 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.966 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.966 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:26.966 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.781526+0000", 2024-09-21T15:49:26.966 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4276000000000001e-05 2024-09-21T15:49:26.966 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.966 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.966 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:26.966 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.781631+0000", 2024-09-21T15:49:26.966 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000104894 2024-09-21T15:49:26.966 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.966 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:26.966 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.966 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.967 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.967 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:6197 3.e 3:734145f0:::rbd_data.1133f3dfbd6b.0000000000000003:5 [read 1355776~4096] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:26.967 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.738101+0000", 2024-09-21T15:49:26.967 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.150782422000001, 2024-09-21T15:49:26.967 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.043764458999999999, 2024-09-21T15:49:26.967 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:26.967 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:26.967 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:26.967 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:26.967 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:26.967 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 6197 2024-09-21T15:49:26.967 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.967 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:26.967 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.967 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:26.968 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738101+0000", 2024-09-21T15:49:26.968 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.968 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.968 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.968 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:26.968 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738101+0000", 2024-09-21T15:49:26.968 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.968 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.968 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.968 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:26.968 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738101+0000", 2024-09-21T15:49:26.968 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:26.968 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.968 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.968 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:26.969 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738102+0000", 2024-09-21T15:49:26.969 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.186e-06 2024-09-21T15:49:26.969 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.969 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.969 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:26.969 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738105+0000", 2024-09-21T15:49:26.969 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.965e-06 2024-09-21T15:49:26.969 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.969 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.969 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:26.969 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738107+0000", 2024-09-21T15:49:26.969 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5289999999999998e-06 2024-09-21T15:49:26.969 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.969 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.969 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:26.970 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.781681+0000", 2024-09-21T15:49:26.970 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.043574031999999999 2024-09-21T15:49:26.970 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.970 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.970 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:26.970 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.781767+0000", 2024-09-21T15:49:26.970 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5995000000000006e-05 2024-09-21T15:49:26.970 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.970 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.970 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:26.970 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.781865+0000", 2024-09-21T15:49:26.970 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7962999999999997e-05 2024-09-21T15:49:26.970 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.971 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:26.971 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.971 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.971 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.971 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:6198 3.e 3:734145f0:::rbd_data.1133f3dfbd6b.0000000000000003:5 [read 1404928~4096] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:26.971 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.738109+0000", 2024-09-21T15:49:26.971 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.150773842, 2024-09-21T15:49:26.971 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.043890720000000001, 2024-09-21T15:49:26.971 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:26.971 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:26.971 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:26.971 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:26.971 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:26.971 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 6198 2024-09-21T15:49:26.972 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.972 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:26.972 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.972 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:26.972 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738109+0000", 2024-09-21T15:49:26.972 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.972 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.972 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.972 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:26.972 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738109+0000", 2024-09-21T15:49:26.972 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:26.972 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.972 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.972 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:26.972 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738109+0000", 2024-09-21T15:49:26.973 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:26.973 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.973 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.973 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:26.973 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738110+0000", 2024-09-21T15:49:26.973 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1689999999999999e-06 2024-09-21T15:49:26.973 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.973 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.973 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:26.973 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738113+0000", 2024-09-21T15:49:26.973 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7879999999999998e-06 2024-09-21T15:49:26.973 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.973 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.973 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:26.973 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.738116+0000", 2024-09-21T15:49:26.974 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4059999999999999e-06 2024-09-21T15:49:26.974 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.974 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.974 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:26.974 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.781879+0000", 2024-09-21T15:49:26.974 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.043763608000000002 2024-09-21T15:49:26.974 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.974 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.974 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:26.974 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.781946+0000", 2024-09-21T15:49:26.974 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6595999999999997e-05 2024-09-21T15:49:26.974 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:26.974 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:26.974 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:26.975 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.782000+0000", 2024-09-21T15:49:26.975 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4447999999999998e-05 2024-09-21T15:49:26.975 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.975 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:26.975 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.975 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:26.975 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:26.975 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:26.976 INFO:teuthology.orchestra.run.smithi089.stdout:checking clone #5, image image_client.0-clone5 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent6 2024-09-21T15:49:26.976 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:49:27.020 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:27.019+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:27.068 INFO:teuthology.orchestra.run.smithi089.stdout:1728 trunc from 0xadff25c to 0xe2ccc96 2024-09-21T15:49:27.077 INFO:teuthology.orchestra.run.smithi089.stdout:1730 trunc from 0xe2ccc96 to 0xdfc6f7a 2024-09-21T15:49:27.083 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:27.083 INFO:teuthology.orchestra.run.smithi089.stdout:1732 read 0xadd6441 thru 0xadd8274 (0x1e34 bytes) 2024-09-21T15:49:27.083 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:27.082+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:27.088 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:27.088 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:49:27.088 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:49:27.088 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:27.089 INFO:teuthology.orchestra.run.smithi089.stdout:1733 trunc from 0xdfc6f7a to 0x902ce9f 2024-09-21T15:49:27.100 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:49:27.111 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:27.111 INFO:teuthology.orchestra.run.smithi089.stdout:1736 punch from 0x28babc5 to 0x28c1f0d, (0x7348 bytes) 2024-09-21T15:49:27.111 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:27.125 INFO:teuthology.orchestra.run.smithi089.stdout:1738 write 0xe379e79 thru 0xe379e9e (0x26 bytes) 2024-09-21T15:49:27.133 INFO:teuthology.orchestra.run.smithi089.stdout:1739 write 0x808ad5d thru 0x808f85c (0x4b00 bytes) 2024-09-21T15:49:27.149 INFO:teuthology.orchestra.run.smithi089.stdout:1740 read 0x6f08db6 thru 0x6f17585 (0xe7d0 bytes) 2024-09-21T15:49:27.150 INFO:teuthology.orchestra.run.smithi089.stdout:1741 write 0xcca2b5c thru 0xcca8241 (0x56e6 bytes) 2024-09-21T15:49:27.156 INFO:teuthology.orchestra.run.smithi089.stdout:1742 read 0x1a91607 thru 0x1a99eb8 (0x88b2 bytes) 2024-09-21T15:49:27.172 INFO:teuthology.orchestra.run.smithi089.stdout:1743 trunc from 0xe379e9f to 0xbe8a1e 2024-09-21T15:49:27.185 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:27.184+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:27.252 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:27.252 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:49:27.252 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:49:27.252 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:49:27.253 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:27.264 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:49:27.286 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:27.284+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:27.387 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:27.386+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:27.407 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:27.407 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:49:27.407 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:49:27.407 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:49:27.407 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.407 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28204 3.1f 3:f84cae28:::rbd_data.1133ebd6d3ba.0000000000000006:a [read 491520~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:27.407 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.205303+0000", 2024-09-21T15:49:27.408 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.200203987, 2024-09-21T15:49:27.408 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.027260120999999998, 2024-09-21T15:49:27.408 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:27.408 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:27.408 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:27.408 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:27.408 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:27.408 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28204 2024-09-21T15:49:27.408 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.408 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:27.408 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.408 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:27.408 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205303+0000", 2024-09-21T15:49:27.408 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.408 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.409 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.409 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:27.409 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205303+0000", 2024-09-21T15:49:27.409 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.409 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.409 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.409 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:27.409 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205302+0000", 2024-09-21T15:49:27.409 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:27.409 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.409 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.409 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:27.409 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205305+0000", 2024-09-21T15:49:27.409 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4480000000000001e-06 2024-09-21T15:49:27.409 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.409 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.410 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:27.410 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205312+0000", 2024-09-21T15:49:27.410 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6789999999999999e-06 2024-09-21T15:49:27.410 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.410 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.410 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:27.410 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205316+0000", 2024-09-21T15:49:27.410 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8189999999999999e-06 2024-09-21T15:49:27.410 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.410 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.410 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:27.410 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.232290+0000", 2024-09-21T15:49:27.410 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.02697397 2024-09-21T15:49:27.411 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.411 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.411 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:27.411 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.232353+0000", 2024-09-21T15:49:27.411 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2990000000000005e-05 2024-09-21T15:49:27.411 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.411 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.411 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:27.411 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.232563+0000", 2024-09-21T15:49:27.411 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000209712 2024-09-21T15:49:27.411 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.411 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:27.411 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.411 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.411 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.412 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28205 3.1f 3:f84cae28:::rbd_data.1133ebd6d3ba.0000000000000006:a [read 606208~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:27.412 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.205321+0000", 2024-09-21T15:49:27.412 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.20018639599999999, 2024-09-21T15:49:27.412 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.027700763999999999, 2024-09-21T15:49:27.412 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:27.412 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:27.412 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:27.412 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:27.412 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:27.412 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28205 2024-09-21T15:49:27.412 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.412 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:27.412 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.412 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:27.412 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205321+0000", 2024-09-21T15:49:27.412 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.413 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.413 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.413 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:27.413 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205321+0000", 2024-09-21T15:49:27.413 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.413 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.413 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.413 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:27.413 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205320+0000", 2024-09-21T15:49:27.413 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:27.413 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.413 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.413 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:27.413 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205322+0000", 2024-09-21T15:49:27.413 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.34e-06 2024-09-21T15:49:27.414 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.414 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.414 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:27.414 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205328+0000", 2024-09-21T15:49:27.414 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6250000000000004e-06 2024-09-21T15:49:27.414 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.414 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.414 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:27.414 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205347+0000", 2024-09-21T15:49:27.414 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8890999999999999e-05 2024-09-21T15:49:27.414 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.414 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.414 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:27.414 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.232590+0000", 2024-09-21T15:49:27.414 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.027242971000000001 2024-09-21T15:49:27.414 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.415 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.415 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:27.415 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.232725+0000", 2024-09-21T15:49:27.415 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013536099999999999 2024-09-21T15:49:27.415 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.415 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.415 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:27.415 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.233021+0000", 2024-09-21T15:49:27.415 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000296025 2024-09-21T15:49:27.415 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.415 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:27.415 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.415 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.415 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.415 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28206 3.1f 3:f84cae28:::rbd_data.1133ebd6d3ba.0000000000000006:a [read 720896~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:27.416 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.205353+0000", 2024-09-21T15:49:27.416 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.20015412699999999, 2024-09-21T15:49:27.416 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.028073124000000001, 2024-09-21T15:49:27.416 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:27.416 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:27.416 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:27.416 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:27.416 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:27.416 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28206 2024-09-21T15:49:27.416 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.416 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:27.416 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.416 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:27.416 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205353+0000", 2024-09-21T15:49:27.416 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.416 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.417 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.417 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:27.417 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205353+0000", 2024-09-21T15:49:27.417 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.417 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.417 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.417 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:27.417 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205352+0000", 2024-09-21T15:49:27.417 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:27.417 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.417 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.417 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:27.417 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205358+0000", 2024-09-21T15:49:27.417 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.784e-06 2024-09-21T15:49:27.417 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.418 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.418 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:27.418 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205365+0000", 2024-09-21T15:49:27.418 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5339999999999996e-06 2024-09-21T15:49:27.418 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.418 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.418 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:27.418 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205370+0000", 2024-09-21T15:49:27.418 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0170000000000004e-06 2024-09-21T15:49:27.418 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.418 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.418 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:27.418 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.233048+0000", 2024-09-21T15:49:27.418 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.027678514000000001 2024-09-21T15:49:27.418 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.418 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.419 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:27.419 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.233117+0000", 2024-09-21T15:49:27.419 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8720000000000006e-05 2024-09-21T15:49:27.419 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.419 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.419 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:27.419 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.233426+0000", 2024-09-21T15:49:27.419 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00030903300000000001 2024-09-21T15:49:27.419 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.419 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:27.419 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.419 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.419 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.419 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28209 3.1f 3:f84cae28:::rbd_data.1133ebd6d3ba.0000000000000006:a [read 827392~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:27.419 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.205374+0000", 2024-09-21T15:49:27.419 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.20013291, 2024-09-21T15:49:27.419 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.028371325999999999, 2024-09-21T15:49:27.420 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:27.420 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:27.420 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:27.420 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:27.420 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:27.420 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28209 2024-09-21T15:49:27.420 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.420 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:27.420 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.420 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:27.420 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205374+0000", 2024-09-21T15:49:27.420 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.420 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.420 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.420 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:27.421 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205374+0000", 2024-09-21T15:49:27.421 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.421 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.421 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.421 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:27.421 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205374+0000", 2024-09-21T15:49:27.421 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:27.421 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.421 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.421 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:27.421 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205376+0000", 2024-09-21T15:49:27.421 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3700000000000002e-06 2024-09-21T15:49:27.421 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.421 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.421 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:27.422 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205382+0000", 2024-09-21T15:49:27.422 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7679999999999997e-06 2024-09-21T15:49:27.422 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.422 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.422 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:27.422 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205386+0000", 2024-09-21T15:49:27.422 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2089999999999999e-06 2024-09-21T15:49:27.422 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.422 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.422 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:27.422 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.233448+0000", 2024-09-21T15:49:27.422 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.028062133999999999 2024-09-21T15:49:27.422 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.422 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.422 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:27.422 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.233511+0000", 2024-09-21T15:49:27.423 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3189999999999996e-05 2024-09-21T15:49:27.423 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.423 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.423 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:27.423 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.233745+0000", 2024-09-21T15:49:27.423 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000234052 2024-09-21T15:49:27.423 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.423 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:27.423 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.423 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.423 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.425 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28210 3.1f 3:f84cae28:::rbd_data.1133ebd6d3ba.0000000000000006:a [read 942080~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:27.425 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.205390+0000", 2024-09-21T15:49:27.425 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.20011699699999999, 2024-09-21T15:49:27.425 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.028697378999999999, 2024-09-21T15:49:27.425 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:27.426 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:27.426 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:27.426 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:27.426 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:27.426 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28210 2024-09-21T15:49:27.426 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.426 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:27.426 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.426 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:27.426 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205390+0000", 2024-09-21T15:49:27.426 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.426 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.426 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.426 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:27.426 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205390+0000", 2024-09-21T15:49:27.427 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.427 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.427 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.427 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:27.427 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205390+0000", 2024-09-21T15:49:27.427 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:27.427 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.427 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.427 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:27.427 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205392+0000", 2024-09-21T15:49:27.427 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1950000000000002e-06 2024-09-21T15:49:27.427 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.427 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.427 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:27.427 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205399+0000", 2024-09-21T15:49:27.427 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8589999999999999e-06 2024-09-21T15:49:27.428 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.428 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.428 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:27.428 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205403+0000", 2024-09-21T15:49:27.428 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6650000000000002e-06 2024-09-21T15:49:27.428 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.428 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.428 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:27.428 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.233785+0000", 2024-09-21T15:49:27.428 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.028381516999999998 2024-09-21T15:49:27.428 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.428 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.428 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:27.428 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.233843+0000", 2024-09-21T15:49:27.428 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8186000000000002e-05 2024-09-21T15:49:27.428 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.429 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.429 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:27.429 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.234087+0000", 2024-09-21T15:49:27.429 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00024431700000000003 2024-09-21T15:49:27.429 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.429 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:27.429 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.429 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.429 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.429 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28275 3.17 3:ea41abc7:::rbd_data.1133ebd6d3ba.0000000000000016:a [read 163840~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:27.429 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.207071+0000", 2024-09-21T15:49:27.429 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.198436007, 2024-09-21T15:49:27.429 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.027031952000000001, 2024-09-21T15:49:27.429 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:27.429 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:27.429 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:27.430 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:27.430 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:27.430 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28275 2024-09-21T15:49:27.430 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.430 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:27.430 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.430 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:27.430 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207071+0000", 2024-09-21T15:49:27.430 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.430 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.430 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.430 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:27.430 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207071+0000", 2024-09-21T15:49:27.430 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.430 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.430 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.431 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:27.431 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207070+0000", 2024-09-21T15:49:27.431 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:27.431 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.431 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.431 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:27.431 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207073+0000", 2024-09-21T15:49:27.431 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5799999999999999e-06 2024-09-21T15:49:27.431 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.431 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.431 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:27.431 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207079+0000", 2024-09-21T15:49:27.431 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9920000000000004e-06 2024-09-21T15:49:27.431 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.431 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.431 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:27.432 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207084+0000", 2024-09-21T15:49:27.432 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9799999999999998e-06 2024-09-21T15:49:27.432 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.432 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.432 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:27.432 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.233949+0000", 2024-09-21T15:49:27.432 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.026864974999999999 2024-09-21T15:49:27.432 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.432 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.432 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:27.432 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.234023+0000", 2024-09-21T15:49:27.432 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3857999999999997e-05 2024-09-21T15:49:27.432 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.432 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.432 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:27.432 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.234103+0000", 2024-09-21T15:49:27.433 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0055000000000005e-05 2024-09-21T15:49:27.433 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.433 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:27.433 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.433 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.433 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.433 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28276 3.17 3:ea41abc7:::rbd_data.1133ebd6d3ba.0000000000000016:a [read 270336~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:27.433 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.207088+0000", 2024-09-21T15:49:27.433 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.19841873400000001, 2024-09-21T15:49:27.433 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.027154052000000001, 2024-09-21T15:49:27.433 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:27.433 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:27.433 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:27.433 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:27.433 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:27.433 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28276 2024-09-21T15:49:27.434 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.434 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:27.434 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.434 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:27.434 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207088+0000", 2024-09-21T15:49:27.434 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.434 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.434 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.434 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:27.434 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207088+0000", 2024-09-21T15:49:27.434 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.434 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.434 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.434 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:27.434 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207088+0000", 2024-09-21T15:49:27.435 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:27.435 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.435 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.435 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:27.435 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207090+0000", 2024-09-21T15:49:27.435 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2479999999999999e-06 2024-09-21T15:49:27.435 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.435 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.435 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:27.435 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207096+0000", 2024-09-21T15:49:27.435 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9329999999999998e-06 2024-09-21T15:49:27.435 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.435 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.435 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:27.435 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207101+0000", 2024-09-21T15:49:27.435 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8060000000000002e-06 2024-09-21T15:49:27.436 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.436 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.436 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:27.436 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.234128+0000", 2024-09-21T15:49:27.436 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.027026853 2024-09-21T15:49:27.436 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.436 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.436 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:27.436 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.234188+0000", 2024-09-21T15:49:27.436 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0105000000000001e-05 2024-09-21T15:49:27.436 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.436 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.436 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:27.436 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.234242+0000", 2024-09-21T15:49:27.436 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4468000000000001e-05 2024-09-21T15:49:27.436 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.437 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:27.437 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.437 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.437 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.437 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28277 3.17 3:ea41abc7:::rbd_data.1133ebd6d3ba.0000000000000016:a [read 385024~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:27.437 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.207105+0000", 2024-09-21T15:49:27.437 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.19840206099999999, 2024-09-21T15:49:27.437 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.027310174999999999, 2024-09-21T15:49:27.437 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:27.437 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:27.437 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:27.437 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:27.437 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:27.437 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28277 2024-09-21T15:49:27.437 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.437 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:27.438 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.438 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:27.438 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207105+0000", 2024-09-21T15:49:27.438 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.438 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.438 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.438 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:27.438 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207105+0000", 2024-09-21T15:49:27.438 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.438 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.438 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.438 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:27.438 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207105+0000", 2024-09-21T15:49:27.438 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:27.438 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.438 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.439 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:27.439 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207107+0000", 2024-09-21T15:49:27.439 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0849999999999999e-06 2024-09-21T15:49:27.439 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.439 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.439 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:27.439 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207112+0000", 2024-09-21T15:49:27.439 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6010000000000002e-06 2024-09-21T15:49:27.439 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.439 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.439 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:27.439 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207117+0000", 2024-09-21T15:49:27.439 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.831e-06 2024-09-21T15:49:27.439 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.439 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.439 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:27.440 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.234286+0000", 2024-09-21T15:49:27.440 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.027168512999999998 2024-09-21T15:49:27.440 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.440 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.440 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:27.440 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.234350+0000", 2024-09-21T15:49:27.440 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4505999999999995e-05 2024-09-21T15:49:27.440 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.440 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.440 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:27.440 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.234415+0000", 2024-09-21T15:49:27.440 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4999000000000006e-05 2024-09-21T15:49:27.440 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.440 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:27.440 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.441 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.441 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.441 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28278 3.17 3:ea41abc7:::rbd_data.1133ebd6d3ba.0000000000000016:a [read 499712~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:27.441 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.207121+0000", 2024-09-21T15:49:27.441 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.19838582900000001, 2024-09-21T15:49:27.441 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.027445132000000001, 2024-09-21T15:49:27.441 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:27.441 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:27.441 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:27.441 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:27.441 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:27.441 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28278 2024-09-21T15:49:27.441 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.441 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:27.441 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.441 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:27.442 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207121+0000", 2024-09-21T15:49:27.442 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.442 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.442 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.442 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:27.442 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207121+0000", 2024-09-21T15:49:27.442 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.442 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.442 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.442 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:27.442 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207121+0000", 2024-09-21T15:49:27.442 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:27.442 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.442 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.442 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:27.443 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207123+0000", 2024-09-21T15:49:27.443 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6170000000000001e-06 2024-09-21T15:49:27.443 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.443 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.443 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:27.443 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207129+0000", 2024-09-21T15:49:27.443 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8420000000000001e-06 2024-09-21T15:49:27.443 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.443 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.443 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:27.443 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207133+0000", 2024-09-21T15:49:27.443 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1470000000000003e-06 2024-09-21T15:49:27.443 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.443 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.443 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:27.444 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.234439+0000", 2024-09-21T15:49:27.444 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.027306119 2024-09-21T15:49:27.444 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.444 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.444 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:27.444 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.234505+0000", 2024-09-21T15:49:27.444 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5493999999999993e-05 2024-09-21T15:49:27.444 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.444 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.444 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:27.444 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.234566+0000", 2024-09-21T15:49:27.444 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1280999999999998e-05 2024-09-21T15:49:27.444 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.444 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:27.444 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.445 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.445 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.445 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28279 3.17 3:ea41abc7:::rbd_data.1133ebd6d3ba.0000000000000016:a [read 614400~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:27.445 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.207137+0000", 2024-09-21T15:49:27.445 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.19836946699999999, 2024-09-21T15:49:27.445 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.027589632999999999, 2024-09-21T15:49:27.445 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:27.445 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:27.445 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:27.445 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:27.445 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:27.445 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28279 2024-09-21T15:49:27.445 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.445 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:27.445 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.445 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:27.446 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207137+0000", 2024-09-21T15:49:27.446 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.446 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.446 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.446 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:27.446 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207137+0000", 2024-09-21T15:49:27.446 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.446 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.446 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.446 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:27.446 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207137+0000", 2024-09-21T15:49:27.446 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:27.446 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.446 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.446 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:27.446 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207139+0000", 2024-09-21T15:49:27.447 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2639999999999998e-06 2024-09-21T15:49:27.447 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.447 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.447 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:27.447 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207145+0000", 2024-09-21T15:49:27.447 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6899999999999997e-06 2024-09-21T15:49:27.447 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.447 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.447 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:27.447 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207150+0000", 2024-09-21T15:49:27.447 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6959999999999996e-06 2024-09-21T15:49:27.447 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.447 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.447 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:27.447 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.234590+0000", 2024-09-21T15:49:27.447 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.027440042000000001 2024-09-21T15:49:27.448 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.448 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.448 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:27.448 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.234666+0000", 2024-09-21T15:49:27.448 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5867999999999999e-05 2024-09-21T15:49:27.448 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.448 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.448 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:27.448 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.234727+0000", 2024-09-21T15:49:27.448 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1447999999999998e-05 2024-09-21T15:49:27.448 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.448 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:27.448 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.448 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.448 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.448 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28280 3.17 3:ea41abc7:::rbd_data.1133ebd6d3ba.0000000000000016:a [read 720896~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:27.449 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.207154+0000", 2024-09-21T15:49:27.449 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.198353162, 2024-09-21T15:49:27.449 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.027708854000000002, 2024-09-21T15:49:27.449 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:27.449 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:27.449 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:27.449 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:27.449 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:27.449 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28280 2024-09-21T15:49:27.449 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.449 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:27.449 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.449 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:27.449 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207154+0000", 2024-09-21T15:49:27.449 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.449 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.450 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.450 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:27.450 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207154+0000", 2024-09-21T15:49:27.450 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.450 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.450 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.450 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:27.450 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207153+0000", 2024-09-21T15:49:27.450 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:27.450 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.450 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.450 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:27.450 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207156+0000", 2024-09-21T15:49:27.450 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.057e-06 2024-09-21T15:49:27.450 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.451 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.451 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:27.451 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207161+0000", 2024-09-21T15:49:27.451 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5910000000000003e-06 2024-09-21T15:49:27.451 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.451 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.451 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:27.451 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207166+0000", 2024-09-21T15:49:27.451 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6020000000000002e-06 2024-09-21T15:49:27.451 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.451 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.451 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:27.451 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.234750+0000", 2024-09-21T15:49:27.451 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.027584132000000001 2024-09-21T15:49:27.451 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.451 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.451 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:27.452 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.234807+0000", 2024-09-21T15:49:27.452 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7482000000000003e-05 2024-09-21T15:49:27.452 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.452 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.452 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:27.452 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.234863+0000", 2024-09-21T15:49:27.452 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5284999999999997e-05 2024-09-21T15:49:27.452 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.452 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:27.452 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.452 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.452 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.452 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28281 3.17 3:ea41abc7:::rbd_data.1133ebd6d3ba.0000000000000016:a [read 835584~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:27.452 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.207170+0000", 2024-09-21T15:49:27.452 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.198337029, 2024-09-21T15:49:27.452 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.027858447000000001, 2024-09-21T15:49:27.453 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:27.453 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:27.453 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:27.453 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:27.453 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:27.453 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28281 2024-09-21T15:49:27.453 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.453 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:27.453 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.453 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:27.453 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207170+0000", 2024-09-21T15:49:27.453 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.453 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.453 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.453 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:27.454 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207170+0000", 2024-09-21T15:49:27.454 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.454 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.454 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.454 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:27.454 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207170+0000", 2024-09-21T15:49:27.454 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:27.454 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.454 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.454 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:27.454 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207172+0000", 2024-09-21T15:49:27.454 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1610000000000001e-06 2024-09-21T15:49:27.454 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.454 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.455 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:27.455 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207177+0000", 2024-09-21T15:49:27.455 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.609e-06 2024-09-21T15:49:27.455 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.455 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.455 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:27.455 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207182+0000", 2024-09-21T15:49:27.455 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5360000000000003e-06 2024-09-21T15:49:27.455 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.455 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.455 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:27.455 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.234885+0000", 2024-09-21T15:49:27.455 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.027702926999999999 2024-09-21T15:49:27.455 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.455 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.455 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:27.456 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.234944+0000", 2024-09-21T15:49:27.456 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9604000000000001e-05 2024-09-21T15:49:27.456 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.456 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.456 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:27.456 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.235028+0000", 2024-09-21T15:49:27.456 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3982999999999999e-05 2024-09-21T15:49:27.456 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.456 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:27.456 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.456 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.456 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.456 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28282 3.17 3:ea41abc7:::rbd_data.1133ebd6d3ba.0000000000000016:a [read 950272~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:27.456 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.207186+0000", 2024-09-21T15:49:27.456 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.19832121, 2024-09-21T15:49:27.457 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.028448035999999999, 2024-09-21T15:49:27.457 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:27.457 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:27.457 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:27.457 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:27.457 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:27.457 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28282 2024-09-21T15:49:27.457 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.457 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:27.457 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.457 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:27.457 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207186+0000", 2024-09-21T15:49:27.457 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.457 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.457 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.457 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:27.458 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207186+0000", 2024-09-21T15:49:27.458 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.458 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.458 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.458 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:27.458 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207185+0000", 2024-09-21T15:49:27.458 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:27.458 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.458 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.458 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:27.458 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207187+0000", 2024-09-21T15:49:27.458 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9659999999999999e-06 2024-09-21T15:49:27.458 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.458 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.458 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:27.459 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207193+0000", 2024-09-21T15:49:27.459 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5539999999999997e-06 2024-09-21T15:49:27.459 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.459 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.459 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:27.459 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207197+0000", 2024-09-21T15:49:27.459 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4889999999999997e-06 2024-09-21T15:49:27.459 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.459 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.459 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:27.459 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.235491+0000", 2024-09-21T15:49:27.459 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.028293490000000001 2024-09-21T15:49:27.459 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.459 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.459 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:27.460 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.235576+0000", 2024-09-21T15:49:27.460 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5609e-05 2024-09-21T15:49:27.460 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.460 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.460 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:27.460 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.235634+0000", 2024-09-21T15:49:27.460 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7306000000000003e-05 2024-09-21T15:49:27.460 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.460 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:27.460 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.460 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.460 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.460 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28355 3.1f 3:fea953f4:::rbd_data.1133ebd6d3ba.000000000000001b:a [read 942080~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:27.460 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.209224+0000", 2024-09-21T15:49:27.460 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.19628330299999999, 2024-09-21T15:49:27.460 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.027110213000000001, 2024-09-21T15:49:27.461 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:27.461 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:27.461 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:27.461 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:27.461 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:27.461 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28355 2024-09-21T15:49:27.461 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.461 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:27.461 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.461 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:27.461 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.209224+0000", 2024-09-21T15:49:27.461 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.461 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.461 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.461 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:27.461 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.209224+0000", 2024-09-21T15:49:27.462 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.462 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.462 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.462 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:27.462 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.209223+0000", 2024-09-21T15:49:27.462 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:27.462 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.462 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.462 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:27.462 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.209225+0000", 2024-09-21T15:49:27.462 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2469999999999998e-06 2024-09-21T15:49:27.462 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.462 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.462 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:27.462 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.209231+0000", 2024-09-21T15:49:27.463 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8919999999999997e-06 2024-09-21T15:49:27.463 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.463 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.463 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:27.463 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.209236+0000", 2024-09-21T15:49:27.463 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4789999999999998e-06 2024-09-21T15:49:27.463 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.463 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.463 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:27.463 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.236213+0000", 2024-09-21T15:49:27.463 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.026976834000000002 2024-09-21T15:49:27.463 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.463 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.463 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:27.463 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.236272+0000", 2024-09-21T15:49:27.463 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9284999999999999e-05 2024-09-21T15:49:27.463 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.464 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.464 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:27.464 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.236334+0000", 2024-09-21T15:49:27.464 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1878000000000006e-05 2024-09-21T15:49:27.464 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.464 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:27.464 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.464 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.464 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.464 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28472 3.1f 3:f84cae28:::rbd_data.1133ebd6d3ba.0000000000000006:a [read 393216~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:27.464 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.210514+0000", 2024-09-21T15:49:27.464 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.194993257, 2024-09-21T15:49:27.464 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.027497165, 2024-09-21T15:49:27.464 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:27.464 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:27.464 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:27.465 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:27.465 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:27.465 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28472 2024-09-21T15:49:27.465 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.465 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:27.465 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.465 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:27.465 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.210514+0000", 2024-09-21T15:49:27.465 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.465 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.465 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.465 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:27.465 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.210514+0000", 2024-09-21T15:49:27.465 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.466 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.466 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.466 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:27.466 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.210513+0000", 2024-09-21T15:49:27.466 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:27.466 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.466 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.466 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:27.466 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.210518+0000", 2024-09-21T15:49:27.466 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8690000000000003e-06 2024-09-21T15:49:27.466 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.466 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.466 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:27.466 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.210525+0000", 2024-09-21T15:49:27.467 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.126e-06 2024-09-21T15:49:27.467 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.467 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.467 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:27.467 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.210531+0000", 2024-09-21T15:49:27.467 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6389999999999997e-06 2024-09-21T15:49:27.467 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.467 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.467 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:27.467 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.237372+0000", 2024-09-21T15:49:27.467 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.026840955 2024-09-21T15:49:27.467 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.467 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.467 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:27.467 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.237435+0000", 2024-09-21T15:49:27.468 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2942999999999997e-05 2024-09-21T15:49:27.468 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.468 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.468 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:27.468 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.238011+0000", 2024-09-21T15:49:27.468 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00057616700000000004 2024-09-21T15:49:27.468 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.468 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:27.468 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.468 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.468 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.468 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28473 3.1f 3:f84cae28:::rbd_data.1133ebd6d3ba.0000000000000006:a [read 499712~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:27.468 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.210536+0000", 2024-09-21T15:49:27.468 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.19497111, 2024-09-21T15:49:27.468 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.02778216, 2024-09-21T15:49:27.468 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:27.469 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:27.469 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:27.469 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:27.469 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:27.469 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28473 2024-09-21T15:49:27.469 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.469 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:27.469 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.469 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:27.469 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.210536+0000", 2024-09-21T15:49:27.469 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.469 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.469 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.469 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:27.469 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.210536+0000", 2024-09-21T15:49:27.469 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.470 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.470 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.470 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:27.470 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.210535+0000", 2024-09-21T15:49:27.470 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:27.470 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.470 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.470 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:27.470 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.210538+0000", 2024-09-21T15:49:27.470 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4980000000000001e-06 2024-09-21T15:49:27.470 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.470 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.470 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:27.470 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.210544+0000", 2024-09-21T15:49:27.470 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0719999999999996e-06 2024-09-21T15:49:27.470 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.471 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.471 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:27.471 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.210549+0000", 2024-09-21T15:49:27.471 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.977e-06 2024-09-21T15:49:27.471 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.471 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.471 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:27.471 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.238037+0000", 2024-09-21T15:49:27.471 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.027487864000000001 2024-09-21T15:49:27.471 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.471 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.471 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:27.471 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.238127+0000", 2024-09-21T15:49:27.471 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0555000000000003e-05 2024-09-21T15:49:27.471 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.472 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.472 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:27.472 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.238318+0000", 2024-09-21T15:49:27.472 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00019061800000000001 2024-09-21T15:49:27.472 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.472 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:27.472 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.472 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.472 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.472 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28475 3.1f 3:f84cae28:::rbd_data.1133ebd6d3ba.0000000000000006:a [read 614400~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:27.472 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.210918+0000", 2024-09-21T15:49:27.472 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.194588751, 2024-09-21T15:49:27.472 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.027783961999999999, 2024-09-21T15:49:27.472 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:27.472 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:27.472 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:27.473 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:27.473 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:27.473 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28475 2024-09-21T15:49:27.473 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.473 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:27.473 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.473 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:27.473 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.210918+0000", 2024-09-21T15:49:27.473 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.473 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.473 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.473 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:27.473 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.210918+0000", 2024-09-21T15:49:27.473 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.473 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.473 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.474 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:27.474 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.210917+0000", 2024-09-21T15:49:27.474 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:27.474 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.474 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.474 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:27.474 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.210923+0000", 2024-09-21T15:49:27.474 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2680000000000002e-06 2024-09-21T15:49:27.474 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.474 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.474 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:27.474 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.210934+0000", 2024-09-21T15:49:27.474 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1246e-05 2024-09-21T15:49:27.474 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.475 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.475 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:27.475 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.210941+0000", 2024-09-21T15:49:27.475 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3660000000000003e-06 2024-09-21T15:49:27.475 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.475 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.475 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:27.475 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.238344+0000", 2024-09-21T15:49:27.475 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.027402603000000001 2024-09-21T15:49:27.475 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.475 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.475 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:27.475 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.238401+0000", 2024-09-21T15:49:27.475 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7077000000000003e-05 2024-09-21T15:49:27.475 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.476 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.476 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:27.476 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.238702+0000", 2024-09-21T15:49:27.476 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000301265 2024-09-21T15:49:27.476 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.476 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:27.476 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.476 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.476 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.476 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28476 3.1f 3:f84cae28:::rbd_data.1133ebd6d3ba.0000000000000006:a [read 729088~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:27.476 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.210947+0000", 2024-09-21T15:49:27.476 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.194559446, 2024-09-21T15:49:27.476 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.028242356999999999, 2024-09-21T15:49:27.476 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:27.476 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:27.476 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:27.477 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:27.477 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:27.477 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28476 2024-09-21T15:49:27.477 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.477 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:27.477 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.477 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:27.477 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.210947+0000", 2024-09-21T15:49:27.477 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.477 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.477 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.477 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:27.477 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.210947+0000", 2024-09-21T15:49:27.477 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.477 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.478 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.478 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:27.478 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.210947+0000", 2024-09-21T15:49:27.478 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:27.478 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.478 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.478 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:27.478 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.210950+0000", 2024-09-21T15:49:27.478 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6180000000000002e-06 2024-09-21T15:49:27.478 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.478 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.478 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:27.478 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.210956+0000", 2024-09-21T15:49:27.479 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3810000000000004e-06 2024-09-21T15:49:27.479 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.479 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.479 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:27.479 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.210961+0000", 2024-09-21T15:49:27.479 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1320000000000002e-06 2024-09-21T15:49:27.479 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.479 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.479 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:27.479 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.238730+0000", 2024-09-21T15:49:27.479 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.027768352 2024-09-21T15:49:27.479 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.479 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.479 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:27.479 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.238800+0000", 2024-09-21T15:49:27.479 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0837e-05 2024-09-21T15:49:27.480 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.480 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.480 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:27.480 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.239190+0000", 2024-09-21T15:49:27.480 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038950399999999999 2024-09-21T15:49:27.480 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.480 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:27.480 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.480 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.480 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.480 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28478 3.1f 3:f84cae28:::rbd_data.1133ebd6d3ba.0000000000000006:a [read 835584~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:27.480 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.211131+0000", 2024-09-21T15:49:27.480 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.194376259, 2024-09-21T15:49:27.480 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.028450188000000001, 2024-09-21T15:49:27.480 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:27.481 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:27.481 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:27.481 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:27.481 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:27.481 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28478 2024-09-21T15:49:27.481 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.481 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:27.481 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.481 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:27.481 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.211131+0000", 2024-09-21T15:49:27.481 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.481 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.481 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.481 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:27.481 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.211131+0000", 2024-09-21T15:49:27.481 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.482 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.482 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.482 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:27.482 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.211130+0000", 2024-09-21T15:49:27.482 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:27.482 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.482 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.482 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:27.482 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.211134+0000", 2024-09-21T15:49:27.482 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.369e-06 2024-09-21T15:49:27.482 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.482 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.482 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:27.482 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.211145+0000", 2024-09-21T15:49:27.482 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0552e-05 2024-09-21T15:49:27.483 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.483 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.483 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:27.483 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.211152+0000", 2024-09-21T15:49:27.483 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9859999999999997e-06 2024-09-21T15:49:27.483 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.483 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.483 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:27.483 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.239220+0000", 2024-09-21T15:49:27.483 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.028068629000000001 2024-09-21T15:49:27.483 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.483 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.483 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:27.483 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.239298+0000", 2024-09-21T15:49:27.483 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7890000000000001e-05 2024-09-21T15:49:27.484 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.484 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.484 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:27.484 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.239581+0000", 2024-09-21T15:49:27.484 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000282501 2024-09-21T15:49:27.484 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.484 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:27.484 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.484 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.484 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.484 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28480 3.1f 3:f84cae28:::rbd_data.1133ebd6d3ba.0000000000000006:a [read 950272~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:27.484 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.211158+0000", 2024-09-21T15:49:27.484 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.194349207, 2024-09-21T15:49:27.484 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.028974907000000001, 2024-09-21T15:49:27.484 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:27.485 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:27.485 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:27.485 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:27.485 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:27.485 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28480 2024-09-21T15:49:27.485 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.485 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:27.485 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.485 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:27.485 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.211158+0000", 2024-09-21T15:49:27.485 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.485 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.485 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.485 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:27.485 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.211158+0000", 2024-09-21T15:49:27.486 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.486 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.486 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.486 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:27.486 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.211157+0000", 2024-09-21T15:49:27.486 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:27.486 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.486 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.486 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:27.486 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.211160+0000", 2024-09-21T15:49:27.486 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5100000000000001e-06 2024-09-21T15:49:27.486 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.486 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.486 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:27.486 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.211167+0000", 2024-09-21T15:49:27.486 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7399999999999998e-06 2024-09-21T15:49:27.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.487 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.487 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:27.487 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.211172+0000", 2024-09-21T15:49:27.487 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1839999999999998e-06 2024-09-21T15:49:27.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.487 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.487 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:27.487 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.239782+0000", 2024-09-21T15:49:27.487 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.028610588999999999 2024-09-21T15:49:27.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.487 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.487 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:27.487 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.239858+0000", 2024-09-21T15:49:27.487 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5753000000000005e-05 2024-09-21T15:49:27.488 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.488 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.488 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:27.488 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.240133+0000", 2024-09-21T15:49:27.488 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00027458300000000001 2024-09-21T15:49:27.488 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.488 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:27.488 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.488 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.488 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:27.488 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:27.490 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:49:27.490 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:27.488+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:27.554 INFO:teuthology.orchestra.run.smithi089.stdout:1744 read 0x7dbe3f thru 0x7eb1b1 (0xf373 bytes) 2024-09-21T15:49:27.555 INFO:teuthology.orchestra.run.smithi089.stdout:1745 read 0x1f5a93 thru 0x2039da (0xdf48 bytes) 2024-09-21T15:49:27.566 INFO:teuthology.orchestra.run.smithi089.stdout:1747 write 0x56f2b9a thru 0x56fd735 (0xab9c bytes) 2024-09-21T15:49:27.574 INFO:teuthology.orchestra.run.smithi089.stdout:1749 write 0x59a9c47 thru 0x59aa492 (0x84c bytes) 2024-09-21T15:49:27.581 INFO:teuthology.orchestra.run.smithi089.stdout:1750 trunc from 0x59aa493 to 0x9071f93 2024-09-21T15:49:27.586 INFO:teuthology.orchestra.run.smithi089.stdout:1751 punch from 0x32522b5 to 0x32578e8, (0x5633 bytes) 2024-09-21T15:49:27.586 INFO:teuthology.orchestra.run.smithi089.stdout:1753 read 0x369717b thru 0x369d6f5 (0x657b bytes) 2024-09-21T15:49:27.586 INFO:teuthology.orchestra.run.smithi089.stdout:1754 writesame 0x380b9b7 thru 0x38130ca data_size 0x2(0x7714 bytes) 2024-09-21T15:49:27.590 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:27.589+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:27.594 INFO:teuthology.orchestra.run.smithi089.stdout:1755 write 0x48af3f5 thru 0x48b4abf (0x56cb bytes) 2024-09-21T15:49:27.597 INFO:teuthology.orchestra.run.smithi089.stdout:1756 writesame 0x5e68231 thru 0x5e750e9 data_size 0x11(0xceb9 bytes) 2024-09-21T15:49:27.600 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:27.600 INFO:teuthology.orchestra.run.smithi089.stdout:1759 read 0x5aed5d9 thru 0x5af36f9 (0x6121 bytes) 2024-09-21T15:49:27.600 INFO:teuthology.orchestra.run.smithi089.stdout:1760 read 0x6c29b0a thru 0x6c34f95 (0xb48c bytes) 2024-09-21T15:49:27.600 INFO:teuthology.orchestra.run.smithi089.stdout:1762 write 0x8a92799 thru 0x8aa162e (0xee96 bytes) 2024-09-21T15:49:27.615 INFO:teuthology.orchestra.run.smithi089.stdout:1763 writesame 0xdb8dae6 thru 0xdb9d4eb data_size 0x1(0xfa06 bytes) 2024-09-21T15:49:27.615 INFO:teuthology.orchestra.run.smithi089.stdout:1765 write 0x1790508 thru 0x1795a09 (0x5502 bytes) 2024-09-21T15:49:27.616 INFO:teuthology.orchestra.run.smithi089.stdout:1769 writesame 0x7177131 thru 0x7178e00 data_size 0x8(0x1cd0 bytes) 2024-09-21T15:49:27.619 INFO:teuthology.orchestra.run.smithi089.stdout:1770 read 0x7e8f39d thru 0x7e9f017 (0xfc7b bytes) 2024-09-21T15:49:27.620 INFO:teuthology.orchestra.run.smithi089.stdout:1771 write 0x1b5ab92 thru 0x1b6a7c3 (0xfc32 bytes) 2024-09-21T15:49:27.622 INFO:teuthology.orchestra.run.smithi089.stdout:1772 punch from 0x37c821a to 0x37d3e26, (0xbc0c bytes) 2024-09-21T15:49:27.622 INFO:teuthology.orchestra.run.smithi089.stdout:1773 read 0x3158171 thru 0x3166507 (0xe397 bytes) 2024-09-21T15:49:27.622 INFO:teuthology.orchestra.run.smithi089.stdout:1774 write 0x80719c4 thru 0x807fc1f (0xe25c bytes) 2024-09-21T15:49:27.624 INFO:teuthology.orchestra.run.smithi089.stdout:1775 write 0x78c8f0f thru 0x78d521a (0xc30c bytes) 2024-09-21T15:49:27.630 INFO:teuthology.orchestra.run.smithi089.stdout:1776 read 0x9bc3f20 thru 0x9bcff01 (0xbfe2 bytes) 2024-09-21T15:49:27.636 INFO:teuthology.orchestra.run.smithi089.stdout:1778 trunc from 0xdb9d4ec to 0x8d1292d 2024-09-21T15:49:27.647 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:27.647 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:49:27.647 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:49:27.647 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:27.648 INFO:teuthology.orchestra.run.smithi089.stdout:1779 trunc from 0x8d1292d to 0xb3c6a27 2024-09-21T15:49:27.653 INFO:teuthology.orchestra.run.smithi089.stdout:1780 read 0x6f18bff thru 0x6f1d880 (0x4c82 bytes) 2024-09-21T15:49:27.653 INFO:teuthology.orchestra.run.smithi089.stdout:1781 writesame 0x2e9461c thru 0x2e9521b data_size 0x20(0xc00 bytes) 2024-09-21T15:49:27.655 INFO:teuthology.orchestra.run.smithi089.stdout:1782 read 0x43d5856 thru 0x43de7a7 (0x8f52 bytes) 2024-09-21T15:49:27.655 INFO:teuthology.orchestra.run.smithi089.stdout:1783 writesame 0x5db7ee9 thru 0x5db8334 data_size 0x16(0x44c bytes) 2024-09-21T15:49:27.656 INFO:teuthology.orchestra.run.smithi089.stdout:1784 write 0x7f79d3f thru 0x7f7aae2 (0xda4 bytes) 2024-09-21T15:49:27.657 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:49:27.661 INFO:teuthology.orchestra.run.smithi089.stdout:1786 punch from 0xb13eb53 to 0xb1452c9, (0x6776 bytes) 2024-09-21T15:49:27.663 INFO:teuthology.orchestra.run.smithi089.stdout:1787 trunc from 0xb3c6a27 to 0x9c93ad7 2024-09-21T15:49:27.667 INFO:teuthology.orchestra.run.smithi089.stdout:1788 writesame 0x9696aa1 thru 0x9699700 data_size 0x20(0x2c60 bytes) 2024-09-21T15:49:27.684 INFO:teuthology.orchestra.run.smithi089.stdout:1789 writesame 0xee5c5d8 thru 0xee64a2e data_size 0x17(0x8457 bytes) 2024-09-21T15:49:27.684 INFO:teuthology.orchestra.run.smithi089.stdout:1790 read 0x6ce6739 thru 0x6cebe93 (0x575b bytes) 2024-09-21T15:49:27.684 INFO:teuthology.orchestra.run.smithi089.stdout:1792 read 0x7ba4ee2 thru 0x7bb1fa5 (0xd0c4 bytes) 2024-09-21T15:49:27.684 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:27.684 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:27.684 INFO:teuthology.orchestra.run.smithi089.stdout:1796 read 0x43ee42a thru 0x43fc201 (0xddd8 bytes) 2024-09-21T15:49:27.685 INFO:teuthology.orchestra.run.smithi089.stdout:1797 read 0xb990426 thru 0xb99c9f0 (0xc5cb bytes) 2024-09-21T15:49:27.685 INFO:teuthology.orchestra.run.smithi089.stdout:1798 write 0x2b24da6 thru 0x2b31cdb (0xcf36 bytes) 2024-09-21T15:49:27.686 INFO:teuthology.orchestra.run.smithi089.stdout:1799 write 0x9214f04 thru 0x92222e6 (0xd3e3 bytes) 2024-09-21T15:49:27.690 INFO:teuthology.orchestra.run.smithi089.stdout:1800 punch from 0x52c9f5c to 0x52d9ee8, (0xff8c bytes) 2024-09-21T15:49:27.690 INFO:teuthology.orchestra.run.smithi089.stdout:1801 write 0x7ef8c48 thru 0x7f04f25 (0xc2de bytes) 2024-09-21T15:49:27.691 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:27.689+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:27.707 INFO:teuthology.orchestra.run.smithi089.stdout:1802 trunc from 0xee64a2f to 0x5dfab80 2024-09-21T15:49:27.723 INFO:teuthology.orchestra.run.smithi089.stdout:1804 punch from 0x376c690 to 0x376f682, (0x2ff2 bytes) 2024-09-21T15:49:27.723 INFO:teuthology.orchestra.run.smithi089.stdout:1805 read 0x55b1fa1 thru 0x55b2d5f (0xdbf bytes) 2024-09-21T15:49:27.723 INFO:teuthology.orchestra.run.smithi089.stdout:1808 punch from 0x42c9c3b to 0x42d3431, (0x97f6 bytes) 2024-09-21T15:49:27.723 INFO:teuthology.orchestra.run.smithi089.stdout:1810 read 0x1d07874 thru 0x1d0989c (0x2029 bytes) 2024-09-21T15:49:27.723 INFO:teuthology.orchestra.run.smithi089.stdout:1811 write 0x3ce2ada thru 0x3cebb4a (0x9071 bytes) 2024-09-21T15:49:27.742 INFO:teuthology.orchestra.run.smithi089.stdout:1812 write 0xe034e96 thru 0xe035788 (0x8f3 bytes) 2024-09-21T15:49:27.750 INFO:teuthology.orchestra.run.smithi089.stdout:1813 trunc from 0xe035789 to 0x95d8a05 2024-09-21T15:49:27.761 INFO:teuthology.orchestra.run.smithi089.stdout:1814 punch from 0x37821bd to 0x378891f, (0x6762 bytes) 2024-09-21T15:49:27.761 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:27.762 INFO:teuthology.orchestra.run.smithi089.stdout:1816 write 0x2ae8fdc thru 0x2aed903 (0x4928 bytes) 2024-09-21T15:49:27.763 INFO:teuthology.orchestra.run.smithi089.stdout:1817 write 0x7da40c thru 0x7e60bc (0xbcb1 bytes) 2024-09-21T15:49:27.764 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:27.765 INFO:teuthology.orchestra.run.smithi089.stdout:1820 punch from 0x8626639 to 0x8636540, (0xff07 bytes) 2024-09-21T15:49:27.765 INFO:teuthology.orchestra.run.smithi089.stdout:1822 read 0x59b407a thru 0x59b6157 (0x20de bytes) 2024-09-21T15:49:27.771 INFO:teuthology.orchestra.run.smithi089.stdout:1823 writesame 0xb36cb38 thru 0xb36e04f data_size 0x36(0x1518 bytes) 2024-09-21T15:49:27.771 INFO:teuthology.orchestra.run.smithi089.stdout:1825 write 0x6badd93 thru 0x6baedd9 (0x1047 bytes) 2024-09-21T15:49:27.774 INFO:teuthology.orchestra.run.smithi089.stdout:1826 writesame 0x9671f66 thru 0x967ffae data_size 0x1(0xe049 bytes) 2024-09-21T15:49:27.777 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:27.777 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:49:27.777 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:49:27.777 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:49:27.777 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:27.779 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is unset 2024-09-21T15:49:27.789 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:49:27.793 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:27.792+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:27.806 INFO:teuthology.orchestra.run.smithi089.stdout:1827 read 0x561d233 thru 0x56236ea (0x64b8 bytes) 2024-09-21T15:49:27.808 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:27.809 INFO:teuthology.orchestra.run.smithi089.stdout:1830 trunc from 0xb36e050 to 0xc65ca8d 2024-09-21T15:49:27.816 INFO:teuthology.orchestra.run.smithi089.stdout:1831 read 0xc13e6c4 thru 0xc13efa4 (0x8e1 bytes) 2024-09-21T15:49:27.816 INFO:teuthology.orchestra.run.smithi089.stdout:1832 read 0x5239f84 thru 0x523c80e (0x288b bytes) 2024-09-21T15:49:27.823 INFO:teuthology.orchestra.run.smithi089.stdout:1834 write 0xde7156d thru 0xde7af50 (0x99e4 bytes) 2024-09-21T15:49:27.839 INFO:teuthology.orchestra.run.smithi089.stdout:1835 trunc from 0xde7af51 to 0x3292141 2024-09-21T15:49:27.852 INFO:teuthology.orchestra.run.smithi089.stdout:1838 read 0xe2b1c2 thru 0xe33adb (0x891a bytes) 2024-09-21T15:49:27.864 INFO:teuthology.orchestra.run.smithi089.stdout:1839 write 0x4443590 thru 0x4447060 (0x3ad1 bytes) 2024-09-21T15:49:27.869 INFO:teuthology.orchestra.run.smithi089.stdout:1840 read 0x2392479 thru 0x239e4d2 (0xc05a bytes) 2024-09-21T15:49:27.887 INFO:teuthology.orchestra.run.smithi089.stdout:1842 writesame 0xdf4f655 thru 0xdf583cc data_size 0x24(0x8d78 bytes) 2024-09-21T15:49:27.887 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:27.892 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:27.891+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:27.895 INFO:teuthology.orchestra.run.smithi089.stdout:1846 trunc from 0xdf583cd to 0x6a2d0d5 2024-09-21T15:49:27.906 INFO:teuthology.orchestra.run.smithi089.stdout:1847 read 0x1ba3f17 thru 0x1baafe0 (0x70ca bytes) 2024-09-21T15:49:27.906 INFO:teuthology.orchestra.run.smithi089.stdout:1848 read 0x1c32eac thru 0x1c3a198 (0x72ed bytes) 2024-09-21T15:49:27.912 INFO:teuthology.orchestra.run.smithi089.stdout:1849 trunc from 0x6a2d0d5 to 0x1f3d4a8 2024-09-21T15:49:27.927 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:27.927 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:49:27.927 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:49:27.927 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:49:27.927 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.927 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:13420 3.0 3:038b8840:::rbd_data.1133dc5adfdf.0000000000000035:head [set-alloc-hint object_size 1048576 write_size 1048576,write 245760~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:27.928 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.208804+0000", 2024-09-21T15:49:27.928 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.716433706, 2024-09-21T15:49:27.928 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0049290649999999998, 2024-09-21T15:49:27.928 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:27.928 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:27.928 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:27.928 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:27.928 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:27.928 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 13420 2024-09-21T15:49:27.928 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.928 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:27.928 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.928 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:27.928 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.208804+0000", 2024-09-21T15:49:27.928 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.928 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.929 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.929 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:27.929 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.208804+0000", 2024-09-21T15:49:27.929 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.929 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.929 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.929 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:27.929 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.208803+0000", 2024-09-21T15:49:27.929 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:27.929 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.929 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.929 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:27.929 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.208813+0000", 2024-09-21T15:49:27.929 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9639999999999998e-06 2024-09-21T15:49:27.929 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.929 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.929 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:27.930 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.208827+0000", 2024-09-21T15:49:27.930 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3952e-05 2024-09-21T15:49:27.930 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.930 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.930 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:27.930 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.208835+0000", 2024-09-21T15:49:27.930 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3680000000000002e-06 2024-09-21T15:49:27.930 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.930 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.930 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:27.930 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.208885+0000", 2024-09-21T15:49:27.930 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9721e-05 2024-09-21T15:49:27.930 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.930 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.931 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:27.931 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209017+0000", 2024-09-21T15:49:27.931 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013215500000000001 2024-09-21T15:49:27.931 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.931 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.931 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-09-21T15:49:27.931 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209152+0000", 2024-09-21T15:49:27.931 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013543100000000001 2024-09-21T15:49:27.931 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.931 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.931 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:27.931 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.213629+0000", 2024-09-21T15:49:27.931 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0044762259999999998 2024-09-21T15:49:27.931 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.931 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.932 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:27.932 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.213649+0000", 2024-09-21T15:49:27.932 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0415e-05 2024-09-21T15:49:27.932 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.932 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.932 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:27.932 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.213698+0000", 2024-09-21T15:49:27.932 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8619000000000003e-05 2024-09-21T15:49:27.932 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.932 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.932 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:27.932 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.213733+0000", 2024-09-21T15:49:27.932 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5312999999999999e-05 2024-09-21T15:49:27.932 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.932 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:27.932 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.932 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.933 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.933 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:13421 3.19 3:9a44caf3:::rbd_data.1133dc5adfdf.0000000000000036:head [set-alloc-hint object_size 1048576 write_size 1048576,write 245760~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:27.933 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.208998+0000", 2024-09-21T15:49:27.933 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.716240034, 2024-09-21T15:49:27.933 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0048896720000000003, 2024-09-21T15:49:27.933 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:27.933 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:27.933 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:27.933 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:27.933 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:27.933 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 13421 2024-09-21T15:49:27.933 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.933 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:27.933 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.933 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:27.933 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.208998+0000", 2024-09-21T15:49:27.934 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.934 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.934 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.934 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:27.934 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.208998+0000", 2024-09-21T15:49:27.934 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.934 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.934 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.934 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:27.934 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.208997+0000", 2024-09-21T15:49:27.934 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:27.934 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.934 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.934 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:27.934 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209006+0000", 2024-09-21T15:49:27.934 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4329999999999993e-06 2024-09-21T15:49:27.935 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.935 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.935 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:27.935 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209014+0000", 2024-09-21T15:49:27.935 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0339999999999997e-06 2024-09-21T15:49:27.935 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.935 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.935 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:27.935 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209019+0000", 2024-09-21T15:49:27.935 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3839999999999999e-06 2024-09-21T15:49:27.935 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.935 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.935 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:27.935 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209058+0000", 2024-09-21T15:49:27.935 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8798999999999999e-05 2024-09-21T15:49:27.935 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.935 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.936 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:27.936 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209143+0000", 2024-09-21T15:49:27.936 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5075999999999997e-05 2024-09-21T15:49:27.936 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.936 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.936 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:49:27.936 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209254+0000", 2024-09-21T15:49:27.936 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000111816 2024-09-21T15:49:27.936 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.936 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.936 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:27.936 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.210789+0000", 2024-09-21T15:49:27.936 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0015348969999999999 2024-09-21T15:49:27.936 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.936 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.937 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:27.937 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.213809+0000", 2024-09-21T15:49:27.937 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0030192069999999999 2024-09-21T15:49:27.937 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.937 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.937 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:27.937 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.213853+0000", 2024-09-21T15:49:27.937 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4023999999999997e-05 2024-09-21T15:49:27.937 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.937 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.937 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:27.937 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.213887+0000", 2024-09-21T15:49:27.937 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4650000000000002e-05 2024-09-21T15:49:27.937 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.937 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:27.937 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.938 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.938 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.938 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:13444 3.19 3:9dbcd9a3:::rbd_data.1133dc5adfdf.0000000000000034:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:27.938 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.232125+0000", 2024-09-21T15:49:27.938 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.693113074999999, 2024-09-21T15:49:27.938 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.004701607, 2024-09-21T15:49:27.938 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:27.938 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:27.938 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:27.938 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:27.938 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:27.938 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 13444 2024-09-21T15:49:27.938 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.938 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:27.938 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.938 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:27.939 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232125+0000", 2024-09-21T15:49:27.939 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.939 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.939 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.939 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:27.939 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232125+0000", 2024-09-21T15:49:27.939 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.939 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.939 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.939 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:27.939 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232123+0000", 2024-09-21T15:49:27.939 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:27.939 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.939 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.939 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:27.940 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232129+0000", 2024-09-21T15:49:27.940 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1059999999999996e-06 2024-09-21T15:49:27.940 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.940 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.940 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:27.940 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232143+0000", 2024-09-21T15:49:27.940 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3611000000000001e-05 2024-09-21T15:49:27.940 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.940 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.940 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:27.940 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232150+0000", 2024-09-21T15:49:27.940 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2699999999999999e-06 2024-09-21T15:49:27.940 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.940 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.940 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:27.940 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232205+0000", 2024-09-21T15:49:27.941 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5024e-05 2024-09-21T15:49:27.941 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.941 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.941 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:27.941 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232313+0000", 2024-09-21T15:49:27.941 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000107742 2024-09-21T15:49:27.941 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.941 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.941 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:49:27.941 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232413+0000", 2024-09-21T15:49:27.941 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010011 2024-09-21T15:49:27.941 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.941 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.941 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:27.941 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.233569+0000", 2024-09-21T15:49:27.942 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0011554639999999999 2024-09-21T15:49:27.942 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.942 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.942 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:27.942 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.236744+0000", 2024-09-21T15:49:27.942 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0031754330000000001 2024-09-21T15:49:27.942 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.942 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.942 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:27.942 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.236791+0000", 2024-09-21T15:49:27.942 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7199e-05 2024-09-21T15:49:27.942 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.942 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.942 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:27.942 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.236826+0000", 2024-09-21T15:49:27.943 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4870000000000003e-05 2024-09-21T15:49:27.943 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.943 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:27.943 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.943 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.943 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.943 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:13445 3.0 3:038b8840:::rbd_data.1133dc5adfdf.0000000000000035:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:27.943 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.232171+0000", 2024-09-21T15:49:27.943 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.693066184999999, 2024-09-21T15:49:27.943 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.005214364, 2024-09-21T15:49:27.943 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:27.943 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:27.943 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:27.943 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:27.944 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:27.944 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 13445 2024-09-21T15:49:27.944 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.944 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:27.944 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.944 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:27.944 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232171+0000", 2024-09-21T15:49:27.944 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.944 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.944 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.944 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:27.944 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232171+0000", 2024-09-21T15:49:27.944 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.944 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.944 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.945 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:27.945 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232171+0000", 2024-09-21T15:49:27.945 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:27.945 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.945 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.945 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:27.945 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232175+0000", 2024-09-21T15:49:27.945 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7689999999999998e-06 2024-09-21T15:49:27.945 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.945 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.945 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:27.945 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232182+0000", 2024-09-21T15:49:27.945 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.712e-06 2024-09-21T15:49:27.945 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.946 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.946 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:27.946 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232188+0000", 2024-09-21T15:49:27.946 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0800000000000002e-06 2024-09-21T15:49:27.946 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.946 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.946 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:27.946 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232257+0000", 2024-09-21T15:49:27.946 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8267000000000002e-05 2024-09-21T15:49:27.946 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.946 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.946 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:27.946 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232337+0000", 2024-09-21T15:49:27.946 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0422999999999999e-05 2024-09-21T15:49:27.946 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.947 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.947 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-09-21T15:49:27.947 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232421+0000", 2024-09-21T15:49:27.947 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3492000000000006e-05 2024-09-21T15:49:27.947 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.947 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.947 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:27.947 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.233588+0000", 2024-09-21T15:49:27.947 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001167707 2024-09-21T15:49:27.947 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.947 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.947 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:27.947 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237298+0000", 2024-09-21T15:49:27.947 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0037098309999999998 2024-09-21T15:49:27.947 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.947 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.948 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:27.948 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237349+0000", 2024-09-21T15:49:27.948 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1102e-05 2024-09-21T15:49:27.948 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.948 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.948 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:27.948 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237386+0000", 2024-09-21T15:49:27.948 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6551999999999998e-05 2024-09-21T15:49:27.948 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.948 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:27.948 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.948 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.948 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.948 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:13446 3.19 3:9a44caf3:::rbd_data.1133dc5adfdf.0000000000000036:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:27.948 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.232312+0000", 2024-09-21T15:49:27.948 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.692925161, 2024-09-21T15:49:27.949 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046255230000000003, 2024-09-21T15:49:27.949 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:27.949 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:27.949 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:27.949 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:27.949 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:27.949 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 13446 2024-09-21T15:49:27.949 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.949 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:27.949 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.949 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:27.949 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232312+0000", 2024-09-21T15:49:27.949 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.949 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.949 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.950 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:27.950 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232312+0000", 2024-09-21T15:49:27.950 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.950 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.950 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.950 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:27.950 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232312+0000", 2024-09-21T15:49:27.950 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:27.950 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.950 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.950 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:27.950 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232329+0000", 2024-09-21T15:49:27.950 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7787000000000001e-05 2024-09-21T15:49:27.950 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.950 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.951 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:27.951 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232340+0000", 2024-09-21T15:49:27.951 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0193e-05 2024-09-21T15:49:27.951 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.951 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.951 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:27.951 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232346+0000", 2024-09-21T15:49:27.951 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3380000000000001e-06 2024-09-21T15:49:27.951 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.951 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.951 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:27.951 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232655+0000", 2024-09-21T15:49:27.951 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00030870099999999998 2024-09-21T15:49:27.951 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.951 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.951 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:27.952 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232729+0000", 2024-09-21T15:49:27.952 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4089000000000002e-05 2024-09-21T15:49:27.952 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.952 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.952 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:49:27.952 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232839+0000", 2024-09-21T15:49:27.952 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000109799 2024-09-21T15:49:27.952 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.952 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.952 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:27.952 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.236550+0000", 2024-09-21T15:49:27.952 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0037115059999999998 2024-09-21T15:49:27.952 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.952 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.952 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:27.953 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.236870+0000", 2024-09-21T15:49:27.953 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00032010699999999998 2024-09-21T15:49:27.953 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.953 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.953 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:27.953 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.236908+0000", 2024-09-21T15:49:27.953 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8087999999999999e-05 2024-09-21T15:49:27.953 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.953 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.953 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:27.953 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.236938+0000", 2024-09-21T15:49:27.953 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9734e-05 2024-09-21T15:49:27.953 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.953 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:27.953 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.954 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.954 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.954 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13447 3.8 e31/22 3:12a4f7ad:::rbd_data.1133dc5adfdf.0000000000000037:head v 31'34, mlcod=31'32)", 2024-09-21T15:49:27.954 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.234176+0000", 2024-09-21T15:49:27.954 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.691061666, 2024-09-21T15:49:27.954 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0091398529999999999, 2024-09-21T15:49:27.954 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:27.954 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:27.954 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:27.954 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.954 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:27.954 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.234176+0000", 2024-09-21T15:49:27.954 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.954 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.954 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.955 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:27.955 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.234176+0000", 2024-09-21T15:49:27.955 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.955 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.955 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.955 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:27.955 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.234175+0000", 2024-09-21T15:49:27.955 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:27.955 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.955 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.955 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:27.955 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.234181+0000", 2024-09-21T15:49:27.955 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2929999999999997e-06 2024-09-21T15:49:27.955 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.956 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.956 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:27.956 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.234195+0000", 2024-09-21T15:49:27.956 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4034e-05 2024-09-21T15:49:27.956 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.956 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.956 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:27.956 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.234202+0000", 2024-09-21T15:49:27.956 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7730000000000002e-06 2024-09-21T15:49:27.956 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.956 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.956 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:27.956 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.234241+0000", 2024-09-21T15:49:27.956 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8590999999999997e-05 2024-09-21T15:49:27.956 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.956 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.956 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:27.957 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.234262+0000", 2024-09-21T15:49:27.957 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0716999999999998e-05 2024-09-21T15:49:27.957 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.957 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.957 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:27.957 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.243281+0000", 2024-09-21T15:49:27.957 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0090193510000000001 2024-09-21T15:49:27.957 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.957 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.957 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:27.957 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.243316+0000", 2024-09-21T15:49:27.957 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4901999999999999e-05 2024-09-21T15:49:27.957 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.957 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:27.957 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.958 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.958 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.958 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13449 3.8 e31/22 3:13437810:::rbd_data.1133dc5adfdf.0000000000000039:head v 31'35, mlcod=31'32)", 2024-09-21T15:49:27.958 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.234889+0000", 2024-09-21T15:49:27.958 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.690348845000001, 2024-09-21T15:49:27.958 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.008441558, 2024-09-21T15:49:27.958 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:27.958 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:27.958 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:27.958 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.958 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:27.958 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.234889+0000", 2024-09-21T15:49:27.958 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.958 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.958 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.959 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:27.959 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.234889+0000", 2024-09-21T15:49:27.959 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.959 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.959 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.959 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:27.959 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.234888+0000", 2024-09-21T15:49:27.959 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:27.959 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.959 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.959 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:27.959 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.234894+0000", 2024-09-21T15:49:27.959 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1680000000000001e-06 2024-09-21T15:49:27.959 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.959 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.959 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:27.959 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.234907+0000", 2024-09-21T15:49:27.960 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3241000000000001e-05 2024-09-21T15:49:27.960 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.960 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.960 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:27.960 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.234914+0000", 2024-09-21T15:49:27.960 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6699999999999997e-06 2024-09-21T15:49:27.960 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.960 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.960 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:27.960 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.234948+0000", 2024-09-21T15:49:27.960 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4359000000000001e-05 2024-09-21T15:49:27.960 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.960 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.960 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:27.960 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.234966+0000", 2024-09-21T15:49:27.961 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8025000000000001e-05 2024-09-21T15:49:27.961 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.961 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.961 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:27.961 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.243319+0000", 2024-09-21T15:49:27.961 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0083525260000000007 2024-09-21T15:49:27.961 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.961 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.961 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:27.961 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.243330+0000", 2024-09-21T15:49:27.961 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1319e-05 2024-09-21T15:49:27.961 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.961 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:27.961 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.961 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.961 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.962 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13456 3.8 e31/22 3:154c00a4:::rbd_data.1133dc5adfdf.000000000000007e:head v 31'36, mlcod=31'32)", 2024-09-21T15:49:27.962 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.237960+0000", 2024-09-21T15:49:27.962 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.6872779, 2024-09-21T15:49:27.962 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0078875109999999998, 2024-09-21T15:49:27.962 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:27.962 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:27.962 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:27.962 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.962 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:27.962 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237960+0000", 2024-09-21T15:49:27.962 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.962 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.962 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.962 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:27.962 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237960+0000", 2024-09-21T15:49:27.963 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.963 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.963 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.963 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:27.963 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237959+0000", 2024-09-21T15:49:27.963 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:27.963 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.963 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.963 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:27.963 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237965+0000", 2024-09-21T15:49:27.963 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3539999999999996e-06 2024-09-21T15:49:27.963 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.963 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.963 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:27.963 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237980+0000", 2024-09-21T15:49:27.963 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.449e-05 2024-09-21T15:49:27.964 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.964 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.964 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:27.964 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237994+0000", 2024-09-21T15:49:27.964 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3910000000000001e-05 2024-09-21T15:49:27.964 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.964 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.964 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:27.964 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.238167+0000", 2024-09-21T15:49:27.964 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000172986 2024-09-21T15:49:27.964 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.964 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.964 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:27.964 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.238189+0000", 2024-09-21T15:49:27.964 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2240000000000001e-05 2024-09-21T15:49:27.964 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.965 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.965 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:27.965 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.245821+0000", 2024-09-21T15:49:27.965 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0076315799999999998 2024-09-21T15:49:27.965 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.965 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.965 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:27.965 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.245847+0000", 2024-09-21T15:49:27.965 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6614999999999998e-05 2024-09-21T15:49:27.965 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.965 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:27.965 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.965 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.965 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.965 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:13839 3.0 3:03c4fb32:::rbd_data.1133dc5adfdf.0000000000000099:head [write 892928~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:27.965 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.527235+0000", 2024-09-21T15:49:27.966 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.398002292999999, 2024-09-21T15:49:27.966 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0067577130000000003, 2024-09-21T15:49:27.966 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:27.966 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:27.966 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:27.966 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:27.966 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:27.966 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 13839 2024-09-21T15:49:27.966 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.966 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:27.966 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.966 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:27.966 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527235+0000", 2024-09-21T15:49:27.966 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.966 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.967 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.967 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:27.967 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527235+0000", 2024-09-21T15:49:27.967 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.967 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.967 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.967 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:27.967 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527234+0000", 2024-09-21T15:49:27.967 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:27.967 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.967 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.967 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:27.967 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527245+0000", 2024-09-21T15:49:27.967 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0597e-05 2024-09-21T15:49:27.967 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.967 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.968 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:27.968 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527266+0000", 2024-09-21T15:49:27.968 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1095e-05 2024-09-21T15:49:27.968 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.968 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.968 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:27.968 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527275+0000", 2024-09-21T15:49:27.968 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9080000000000001e-06 2024-09-21T15:49:27.968 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.968 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.968 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:27.968 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527816+0000", 2024-09-21T15:49:27.968 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00054145400000000002 2024-09-21T15:49:27.968 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.968 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.968 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:27.969 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527891+0000", 2024-09-21T15:49:27.969 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4898e-05 2024-09-21T15:49:27.969 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.969 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.969 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-09-21T15:49:27.969 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528024+0000", 2024-09-21T15:49:27.969 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000133242 2024-09-21T15:49:27.969 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.969 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.969 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:27.969 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.531270+0000", 2024-09-21T15:49:27.969 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003245469 2024-09-21T15:49:27.969 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.969 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.969 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:27.969 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.533926+0000", 2024-09-21T15:49:27.970 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0026558559999999998 2024-09-21T15:49:27.970 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.970 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.970 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:27.970 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.533953+0000", 2024-09-21T15:49:27.970 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7551000000000001e-05 2024-09-21T15:49:27.970 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.970 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.970 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:27.970 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.533993+0000", 2024-09-21T15:49:27.970 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9782e-05 2024-09-21T15:49:27.970 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.970 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:27.970 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.970 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.970 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.971 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:14257 3.6 e33/22 3:628c5ea8:::rbd_data.1133dc5adfdf.0000000000000093:head v 33'45, mlcod=31'43)", 2024-09-21T15:49:27.971 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:18.838030+0000", 2024-09-21T15:49:27.971 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.087207867, 2024-09-21T15:49:27.971 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0048671549999999997, 2024-09-21T15:49:27.971 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:27.971 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:27.971 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:27.971 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.971 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:27.971 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.838030+0000", 2024-09-21T15:49:27.971 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.971 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.971 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.971 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:27.971 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.838030+0000", 2024-09-21T15:49:27.971 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.971 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.972 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.972 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:27.972 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.838029+0000", 2024-09-21T15:49:27.972 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:27.972 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.972 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.972 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:27.972 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.838034+0000", 2024-09-21T15:49:27.972 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3340000000000003e-06 2024-09-21T15:49:27.972 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.972 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.972 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:27.972 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.838044+0000", 2024-09-21T15:49:27.972 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7720000000000006e-06 2024-09-21T15:49:27.972 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.973 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.973 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:27.973 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.838049+0000", 2024-09-21T15:49:27.973 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8490000000000002e-06 2024-09-21T15:49:27.973 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.973 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.973 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:27.973 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.838091+0000", 2024-09-21T15:49:27.973 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1455000000000002e-05 2024-09-21T15:49:27.973 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.973 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.973 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:27.973 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.838109+0000", 2024-09-21T15:49:27.973 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7663e-05 2024-09-21T15:49:27.973 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.973 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.974 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:27.974 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.842869+0000", 2024-09-21T15:49:27.974 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0047603439999999997 2024-09-21T15:49:27.974 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.974 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.974 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:27.974 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.842897+0000", 2024-09-21T15:49:27.974 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8033000000000001e-05 2024-09-21T15:49:27.974 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.974 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:27.974 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.974 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.974 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.974 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:14263 3.0 3:03c4fb32:::rbd_data.1133dc5adfdf.0000000000000099:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-21T15:49:27.975 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:18.838485+0000", 2024-09-21T15:49:27.975 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.0867530040000002, 2024-09-21T15:49:27.975 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0044523089999999998, 2024-09-21T15:49:27.975 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:27.975 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:27.975 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:27.975 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:27.975 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:27.975 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 14263 2024-09-21T15:49:27.975 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.975 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:27.975 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.975 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:27.975 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.838485+0000", 2024-09-21T15:49:27.975 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.975 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.975 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.976 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:27.976 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.838485+0000", 2024-09-21T15:49:27.976 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.976 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.976 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.976 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:27.976 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.838484+0000", 2024-09-21T15:49:27.976 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:27.976 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.976 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.976 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:27.976 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.838489+0000", 2024-09-21T15:49:27.976 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.339e-06 2024-09-21T15:49:27.976 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.976 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.977 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:27.977 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.838504+0000", 2024-09-21T15:49:27.977 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4603000000000001e-05 2024-09-21T15:49:27.977 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.977 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.977 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:27.977 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.838511+0000", 2024-09-21T15:49:27.977 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2559999999999996e-06 2024-09-21T15:49:27.977 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.977 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.977 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:27.977 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.838565+0000", 2024-09-21T15:49:27.977 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3668000000000002e-05 2024-09-21T15:49:27.977 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.977 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.978 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:27.978 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.838646+0000", 2024-09-21T15:49:27.978 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1643000000000004e-05 2024-09-21T15:49:27.978 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.978 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.978 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-09-21T15:49:27.978 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.838753+0000", 2024-09-21T15:49:27.978 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000107057 2024-09-21T15:49:27.978 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.978 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.978 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:27.978 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.840378+0000", 2024-09-21T15:49:27.978 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0016248020000000001 2024-09-21T15:49:27.978 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.978 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.978 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:27.979 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.842889+0000", 2024-09-21T15:49:27.979 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0025106519999999999 2024-09-21T15:49:27.979 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.979 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.979 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:27.979 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.842915+0000", 2024-09-21T15:49:27.979 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6614e-05 2024-09-21T15:49:27.979 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.979 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.979 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:27.979 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.842937+0000", 2024-09-21T15:49:27.979 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1625e-05 2024-09-21T15:49:27.979 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.979 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:27.979 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.979 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.979 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.980 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:14393 3.6 e33/22 3:640f2ea6:::rbd_data.1133dc5adfdf.000000000000002a:head v 33'48, mlcod=33'46)", 2024-09-21T15:49:27.980 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:18.963289+0000", 2024-09-21T15:49:27.980 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.9619484190000005, 2024-09-21T15:49:27.980 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0057897479999999999, 2024-09-21T15:49:27.980 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:27.980 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:27.980 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:27.980 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.980 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:27.980 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.963289+0000", 2024-09-21T15:49:27.980 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.980 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.980 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.980 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:27.980 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.963289+0000", 2024-09-21T15:49:27.980 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.981 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.981 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.981 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:27.981 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.963288+0000", 2024-09-21T15:49:27.981 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:27.981 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.981 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.981 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:27.981 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.963297+0000", 2024-09-21T15:49:27.981 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2180000000000005e-06 2024-09-21T15:49:27.981 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.981 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.981 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:27.981 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.963312+0000", 2024-09-21T15:49:27.981 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4414e-05 2024-09-21T15:49:27.981 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.981 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.982 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:27.982 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.963318+0000", 2024-09-21T15:49:27.982 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5590000000000002e-06 2024-09-21T15:49:27.982 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.982 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.982 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:27.982 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.963361+0000", 2024-09-21T15:49:27.982 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3108000000000001e-05 2024-09-21T15:49:27.982 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.982 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.982 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:27.982 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.963382+0000", 2024-09-21T15:49:27.982 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0601000000000001e-05 2024-09-21T15:49:27.982 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.983 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.983 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:27.983 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969043+0000", 2024-09-21T15:49:27.983 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0056611889999999996 2024-09-21T15:49:27.983 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.983 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.983 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:27.983 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969079+0000", 2024-09-21T15:49:27.983 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5731000000000001e-05 2024-09-21T15:49:27.983 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.983 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:27.983 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.983 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.983 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.983 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:14403 3.19 3:9dbcd9a3:::rbd_data.1133dc5adfdf.0000000000000034:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-21T15:49:27.984 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:18.969363+0000", 2024-09-21T15:49:27.984 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.9558745670000004, 2024-09-21T15:49:27.984 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0059848089999999998, 2024-09-21T15:49:27.984 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:27.984 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:27.984 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:27.984 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:27.984 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:27.984 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 14403 2024-09-21T15:49:27.984 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.984 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:27.984 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.984 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:27.984 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969363+0000", 2024-09-21T15:49:27.984 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.985 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.985 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.985 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:27.985 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969363+0000", 2024-09-21T15:49:27.985 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.985 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.985 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.985 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:27.985 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969362+0000", 2024-09-21T15:49:27.985 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:27.985 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.985 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.985 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:27.985 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969367+0000", 2024-09-21T15:49:27.986 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9400000000000001e-06 2024-09-21T15:49:27.986 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.986 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.986 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:27.986 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969378+0000", 2024-09-21T15:49:27.986 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1126e-05 2024-09-21T15:49:27.986 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.986 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.986 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:27.986 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969383+0000", 2024-09-21T15:49:27.986 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1290000000000004e-06 2024-09-21T15:49:27.986 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.986 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.986 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:27.987 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969420+0000", 2024-09-21T15:49:27.987 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.68e-05 2024-09-21T15:49:27.987 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.987 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.987 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:27.987 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969487+0000", 2024-09-21T15:49:27.987 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7211000000000004e-05 2024-09-21T15:49:27.987 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.987 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.987 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:49:27.987 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969543+0000", 2024-09-21T15:49:27.987 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5890999999999997e-05 2024-09-21T15:49:27.987 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.988 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.988 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:27.988 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.972060+0000", 2024-09-21T15:49:27.988 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002516379 2024-09-21T15:49:27.988 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.988 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.988 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:27.988 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.975273+0000", 2024-09-21T15:49:27.988 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032138560000000002 2024-09-21T15:49:27.988 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.988 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.988 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:27.988 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.975315+0000", 2024-09-21T15:49:27.988 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1795999999999998e-05 2024-09-21T15:49:27.988 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.989 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.989 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:27.989 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.975348+0000", 2024-09-21T15:49:27.989 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2666999999999997e-05 2024-09-21T15:49:27.989 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.989 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:27.989 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.989 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.989 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.989 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:14404 3.0 3:038b8840:::rbd_data.1133dc5adfdf.0000000000000035:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-21T15:49:27.989 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:18.969391+0000", 2024-09-21T15:49:27.989 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.9558470040000007, 2024-09-21T15:49:27.989 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0057367420000000004, 2024-09-21T15:49:27.989 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:27.990 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:27.990 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:27.990 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:27.990 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:27.990 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 14404 2024-09-21T15:49:27.990 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.990 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:27.990 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.990 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:27.990 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969391+0000", 2024-09-21T15:49:27.990 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.990 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.990 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.990 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:27.991 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969391+0000", 2024-09-21T15:49:27.991 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.991 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.991 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.991 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:27.991 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969390+0000", 2024-09-21T15:49:27.991 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:27.991 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.991 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.991 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:27.991 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969392+0000", 2024-09-21T15:49:27.991 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.827e-06 2024-09-21T15:49:27.991 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.991 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.991 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:27.992 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969397+0000", 2024-09-21T15:49:27.992 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8799999999999999e-06 2024-09-21T15:49:27.992 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.992 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.992 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:27.992 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969400+0000", 2024-09-21T15:49:27.992 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.004e-06 2024-09-21T15:49:27.992 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.992 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.992 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:27.992 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969446+0000", 2024-09-21T15:49:27.992 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6446999999999998e-05 2024-09-21T15:49:27.992 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.992 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.992 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:27.993 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969522+0000", 2024-09-21T15:49:27.993 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5419000000000004e-05 2024-09-21T15:49:27.993 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.993 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.993 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-09-21T15:49:27.993 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969626+0000", 2024-09-21T15:49:27.993 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000103898 2024-09-21T15:49:27.993 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.993 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.993 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:27.993 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.972220+0000", 2024-09-21T15:49:27.993 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0025942169999999998 2024-09-21T15:49:27.993 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.993 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.993 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:27.994 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.975030+0000", 2024-09-21T15:49:27.994 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0028103469999999999 2024-09-21T15:49:27.994 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.994 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.994 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:27.994 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.975093+0000", 2024-09-21T15:49:27.994 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2744999999999996e-05 2024-09-21T15:49:27.994 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.994 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.994 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:27.994 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.975127+0000", 2024-09-21T15:49:27.994 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4248000000000001e-05 2024-09-21T15:49:27.994 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.994 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:27.995 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:27.995 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.995 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.995 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:14405 3.19 3:9a44caf3:::rbd_data.1133dc5adfdf.0000000000000036:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-21T15:49:27.995 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:18.969509+0000", 2024-09-21T15:49:27.995 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.9557284589999995, 2024-09-21T15:49:27.995 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0059130320000000004, 2024-09-21T15:49:27.995 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:27.995 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:27.995 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:27.995 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:27.995 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:27.995 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 14405 2024-09-21T15:49:27.995 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.996 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:27.996 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.996 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:27.996 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969509+0000", 2024-09-21T15:49:27.996 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.996 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.996 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.996 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:27.996 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969509+0000", 2024-09-21T15:49:27.996 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:27.996 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.996 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.996 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:27.996 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969509+0000", 2024-09-21T15:49:27.996 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:27.996 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.997 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.997 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:27.997 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969512+0000", 2024-09-21T15:49:27.997 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1429999999999998e-06 2024-09-21T15:49:27.997 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.997 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.997 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:27.997 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969518+0000", 2024-09-21T15:49:27.997 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.268e-06 2024-09-21T15:49:27.997 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.997 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.997 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:27.997 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969522+0000", 2024-09-21T15:49:27.997 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4499999999999997e-06 2024-09-21T15:49:27.997 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.997 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.998 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:27.998 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969816+0000", 2024-09-21T15:49:27.998 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00029325400000000001 2024-09-21T15:49:27.998 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.998 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.998 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:27.998 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969871+0000", 2024-09-21T15:49:27.998 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5337000000000002e-05 2024-09-21T15:49:27.998 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.998 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.998 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:49:27.998 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969927+0000", 2024-09-21T15:49:27.998 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5878000000000003e-05 2024-09-21T15:49:27.998 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.998 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.998 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:27.999 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.975067+0000", 2024-09-21T15:49:27.999 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0051400980000000001 2024-09-21T15:49:27.999 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.999 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.999 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:27.999 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.975380+0000", 2024-09-21T15:49:27.999 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00031274100000000001 2024-09-21T15:49:27.999 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.999 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.999 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:27.999 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.975404+0000", 2024-09-21T15:49:27.999 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3839999999999999e-05 2024-09-21T15:49:27.999 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:27.999 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:27.999 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:28.000 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.975422+0000", 2024-09-21T15:49:28.000 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8593e-05 2024-09-21T15:49:28.000 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:28.000 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:28.000 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:28.000 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.000 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.000 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:14449 3.6 e33/22 3:62956c53:::rbd_data.1133dc5adfdf.0000000000000045:head v 33'50, mlcod=33'48)", 2024-09-21T15:49:28.000 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:19.025593+0000", 2024-09-21T15:49:28.000 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.8996448739999998, 2024-09-21T15:49:28.000 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.006225146, 2024-09-21T15:49:28.000 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:28.000 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:28.000 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:28.000 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.000 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:28.000 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:19.025593+0000", 2024-09-21T15:49:28.001 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:28.001 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.001 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.001 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:28.001 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:19.025593+0000", 2024-09-21T15:49:28.001 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:28.001 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.001 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.001 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:28.001 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:19.025588+0000", 2024-09-21T15:49:28.001 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999948 2024-09-21T15:49:28.001 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.001 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.001 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:28.001 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:19.025601+0000", 2024-09-21T15:49:28.001 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3833e-05 2024-09-21T15:49:28.001 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.002 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.002 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:28.002 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:19.025627+0000", 2024-09-21T15:49:28.002 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5979999999999999e-05 2024-09-21T15:49:28.002 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.002 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.002 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:28.002 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:19.025636+0000", 2024-09-21T15:49:28.002 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6659999999999997e-06 2024-09-21T15:49:28.002 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.002 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.002 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:28.002 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:19.025692+0000", 2024-09-21T15:49:28.002 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6431999999999998e-05 2024-09-21T15:49:28.003 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.003 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.003 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:28.003 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:19.025713+0000", 2024-09-21T15:49:28.003 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0971999999999998e-05 2024-09-21T15:49:28.003 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.003 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.003 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:28.003 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:19.031788+0000", 2024-09-21T15:49:28.003 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0060743580000000002 2024-09-21T15:49:28.003 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.003 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.003 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:28.003 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:19.031818+0000", 2024-09-21T15:49:28.003 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0054000000000001e-05 2024-09-21T15:49:28.003 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:28.004 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:28.004 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:28.004 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.004 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.004 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:22445 3.6 e34/22 3:66402617:::rbd_data.1133896adcfd.00000000000000bc:head v 34'72, mlcod=34'70)", 2024-09-21T15:49:28.004 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:21.730937+0000", 2024-09-21T15:49:28.004 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 6.1943003330000002, 2024-09-21T15:49:28.004 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0049811309999999998, 2024-09-21T15:49:28.004 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:28.004 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:28.004 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:28.004 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.004 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:28.004 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.730937+0000", 2024-09-21T15:49:28.004 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:28.004 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.005 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.005 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:28.005 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.730937+0000", 2024-09-21T15:49:28.005 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:28.005 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.005 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.005 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:28.005 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.730936+0000", 2024-09-21T15:49:28.005 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:28.005 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.005 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.005 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:28.005 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.730950+0000", 2024-09-21T15:49:28.005 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4028e-05 2024-09-21T15:49:28.005 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.005 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.005 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:28.006 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.730968+0000", 2024-09-21T15:49:28.006 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8289999999999999e-05 2024-09-21T15:49:28.006 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.006 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.006 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:28.006 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.730977+0000", 2024-09-21T15:49:28.006 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5150000000000006e-06 2024-09-21T15:49:28.006 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.006 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.006 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:28.006 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.731026+0000", 2024-09-21T15:49:28.006 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8770999999999998e-05 2024-09-21T15:49:28.006 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.006 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.006 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:28.007 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.731047+0000", 2024-09-21T15:49:28.007 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1407999999999999e-05 2024-09-21T15:49:28.007 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.007 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.007 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:28.007 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.735888+0000", 2024-09-21T15:49:28.007 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0048415660000000003 2024-09-21T15:49:28.007 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.007 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.007 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:28.007 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.735918+0000", 2024-09-21T15:49:28.007 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9899999999999998e-05 2024-09-21T15:49:28.007 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:28.007 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:28.007 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:28.007 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.008 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.008 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:25278 3.4 3:2251cc1c:::rbd_data.1133ebd6d3ba.0000000000000024:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:28.008 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.316512+0000", 2024-09-21T15:49:28.008 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.6087253289999999, 2024-09-21T15:49:28.008 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0064005590000000001, 2024-09-21T15:49:28.008 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:28.008 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:28.008 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:28.008 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:28.008 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:28.008 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 25278 2024-09-21T15:49:28.008 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.008 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:28.008 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.008 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:28.008 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.316512+0000", 2024-09-21T15:49:28.008 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:28.009 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.009 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.009 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:28.009 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.316512+0000", 2024-09-21T15:49:28.009 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:28.009 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.009 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.009 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:28.009 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.316510+0000", 2024-09-21T15:49:28.009 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:49:28.009 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.009 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.009 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:28.009 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.316527+0000", 2024-09-21T15:49:28.009 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6643e-05 2024-09-21T15:49:28.009 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.010 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.010 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:28.010 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.316547+0000", 2024-09-21T15:49:28.010 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9925999999999999e-05 2024-09-21T15:49:28.010 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.010 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.010 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:28.010 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.316559+0000", 2024-09-21T15:49:28.010 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2018000000000001e-05 2024-09-21T15:49:28.010 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.010 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.010 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:28.010 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.316618+0000", 2024-09-21T15:49:28.010 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9456000000000001e-05 2024-09-21T15:49:28.011 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.011 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.011 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:28.011 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.316719+0000", 2024-09-21T15:49:28.011 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010033400000000001 2024-09-21T15:49:28.011 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.011 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.011 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-09-21T15:49:28.011 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.316837+0000", 2024-09-21T15:49:28.011 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000118898 2024-09-21T15:49:28.011 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.011 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.011 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:28.011 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317562+0000", 2024-09-21T15:49:28.011 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00072486399999999998 2024-09-21T15:49:28.011 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.012 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.012 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:28.012 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.322838+0000", 2024-09-21T15:49:28.012 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052753139999999997 2024-09-21T15:49:28.012 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.012 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.012 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:28.012 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.322879+0000", 2024-09-21T15:49:28.012 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1559e-05 2024-09-21T15:49:28.012 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.012 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.012 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:28.012 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.322913+0000", 2024-09-21T15:49:28.012 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3618000000000002e-05 2024-09-21T15:49:28.012 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:28.012 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:28.012 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:28.013 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.013 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.013 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:25280 3.4 3:27fca38c:::rbd_data.1133ebd6d3ba.0000000000000026:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:28.013 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.316577+0000", 2024-09-21T15:49:28.013 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.608660413, 2024-09-21T15:49:28.013 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0066444329999999999, 2024-09-21T15:49:28.013 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:28.013 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:28.013 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:28.013 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:28.013 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:28.013 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 25280 2024-09-21T15:49:28.013 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.013 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:28.013 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.014 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:28.014 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.316577+0000", 2024-09-21T15:49:28.014 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:28.014 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.014 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.014 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:28.014 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.316577+0000", 2024-09-21T15:49:28.014 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:28.014 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.014 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.014 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:28.014 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.316577+0000", 2024-09-21T15:49:28.014 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:28.014 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.015 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.015 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:28.015 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.316580+0000", 2024-09-21T15:49:28.015 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1839999999999999e-06 2024-09-21T15:49:28.015 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.015 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.015 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:28.015 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.316588+0000", 2024-09-21T15:49:28.015 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6109999999999996e-06 2024-09-21T15:49:28.015 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.015 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.015 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:28.015 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.316593+0000", 2024-09-21T15:49:28.015 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0540000000000002e-06 2024-09-21T15:49:28.015 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.016 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.016 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:28.016 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317176+0000", 2024-09-21T15:49:28.016 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00058346600000000004 2024-09-21T15:49:28.016 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.016 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.016 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:28.016 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317245+0000", 2024-09-21T15:49:28.016 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9165000000000006e-05 2024-09-21T15:49:28.016 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.016 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.016 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-09-21T15:49:28.016 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317348+0000", 2024-09-21T15:49:28.016 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000102897 2024-09-21T15:49:28.016 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.016 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.016 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:28.017 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323070+0000", 2024-09-21T15:49:28.017 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0057218360000000001 2024-09-21T15:49:28.017 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.017 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.017 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:28.017 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323151+0000", 2024-09-21T15:49:28.017 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0920000000000005e-05 2024-09-21T15:49:28.017 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.017 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.017 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:28.017 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323190+0000", 2024-09-21T15:49:28.017 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8914e-05 2024-09-21T15:49:28.017 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.017 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.017 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:28.017 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323222+0000", 2024-09-21T15:49:28.018 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1819999999999997e-05 2024-09-21T15:49:28.018 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:28.018 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:28.018 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:28.018 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.018 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.018 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:25286 3.0 3:0248fb60:::rbd_data.1133ebd6d3ba.0000000000000066:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:28.018 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.316617+0000", 2024-09-21T15:49:28.018 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.608620771, 2024-09-21T15:49:28.018 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0066683009999999997, 2024-09-21T15:49:28.018 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:28.018 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:28.018 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:28.018 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:28.018 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:28.019 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 25286 2024-09-21T15:49:28.019 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.019 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:28.019 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.019 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:28.019 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.316617+0000", 2024-09-21T15:49:28.019 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:28.019 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.019 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.019 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:28.019 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.316617+0000", 2024-09-21T15:49:28.019 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:28.019 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.019 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.019 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:28.019 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.316616+0000", 2024-09-21T15:49:28.020 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:28.020 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.020 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.020 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:28.020 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.316620+0000", 2024-09-21T15:49:28.020 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2169999999999999e-06 2024-09-21T15:49:28.020 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.020 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.020 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:28.020 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.316627+0000", 2024-09-21T15:49:28.020 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.435e-06 2024-09-21T15:49:28.020 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.020 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.020 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:28.020 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.316632+0000", 2024-09-21T15:49:28.021 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3709999999999999e-06 2024-09-21T15:49:28.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.021 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.021 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:28.021 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.316675+0000", 2024-09-21T15:49:28.021 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2280999999999997e-05 2024-09-21T15:49:28.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.021 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.021 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:28.021 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.316750+0000", 2024-09-21T15:49:28.021 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5799999999999999e-05 2024-09-21T15:49:28.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.021 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.021 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-09-21T15:49:28.021 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.316841+0000", 2024-09-21T15:49:28.021 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0375000000000001e-05 2024-09-21T15:49:28.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.022 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.022 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:28.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.322697+0000", 2024-09-21T15:49:28.022 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0058559409999999999 2024-09-21T15:49:28.022 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.022 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.022 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:28.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323194+0000", 2024-09-21T15:49:28.022 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000496997 2024-09-21T15:49:28.022 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.022 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.022 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:28.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323252+0000", 2024-09-21T15:49:28.022 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7868999999999998e-05 2024-09-21T15:49:28.022 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:28.022 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:28.023 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:28.023 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323285+0000", 2024-09-21T15:49:28.023 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3488000000000002e-05 2024-09-21T15:49:28.023 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:28.023 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:28.023 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:28.023 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:28.023 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:28.023 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:28.023 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T15:49:28.023 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T15:49:28.023 INFO:tasks.thrashosds.thrasher:inject_pause on osd.0 2024-09-21T15:49:28.023 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T15:49:28.024 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T15:49:28.024 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok config set filestore_inject_stall 3 2024-09-21T15:49:28.025 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:49:28.027 INFO:teuthology.orchestra.run.smithi089.stdout:1851 trunc from 0x1f3d4a8 to 0x9d547a9 2024-09-21T15:49:28.027 INFO:teuthology.orchestra.run.smithi089.stdout:1852 read 0x584ef12 thru 0x585d7a1 (0xe890 bytes) 2024-09-21T15:49:28.027 INFO:teuthology.orchestra.run.smithi089.stdout:1853 read 0x1246ace thru 0x1246faa (0x4dd bytes) 2024-09-21T15:49:28.027 INFO:teuthology.orchestra.run.smithi089.stdout:1854 read 0x6294c66 thru 0x6296bed (0x1f88 bytes) 2024-09-21T15:49:28.027 INFO:teuthology.orchestra.run.smithi089.stdout:1855 write 0x8b35d1a thru 0x8b3afd0 (0x52b7 bytes) 2024-09-21T15:49:28.027 INFO:teuthology.orchestra.run.smithi089.stdout:1856 writesame 0x50aa8ce thru 0x50b2587 data_size 0x67(0x7cba bytes) 2024-09-21T15:49:28.027 INFO:teuthology.orchestra.run.smithi089.stdout:1857 read 0x568465d thru 0x568559b (0xf3f bytes) 2024-09-21T15:49:28.027 INFO:teuthology.orchestra.run.smithi089.stdout:1860 punch from 0x76ff1b2 to 0x7708a6a, (0x98b8 bytes) 2024-09-21T15:49:28.027 INFO:teuthology.orchestra.run.smithi089.stdout:1861 read 0x340ab5d thru 0x340ce6d (0x2311 bytes) 2024-09-21T15:49:28.027 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:28.028 INFO:teuthology.orchestra.run.smithi089.stdout:1863 write 0xd6ed234 thru 0xd6fbe7c (0xec49 bytes) 2024-09-21T15:49:28.028 INFO:teuthology.orchestra.run.smithi089.stdout:1864 write 0x5fb4f0e thru 0x5fbfc9a (0xad8d bytes) 2024-09-21T15:49:28.028 INFO:teuthology.orchestra.run.smithi089.stdout:1866 trunc from 0xd6fbe7d to 0x10c8acf 2024-09-21T15:49:28.028 INFO:teuthology.orchestra.run.smithi089.stdout:1867 read 0xc67940 thru 0xc6dfb9 (0x667a bytes) 2024-09-21T15:49:28.028 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:28.028 INFO:teuthology.orchestra.run.smithi089.stdout:1872 trunc from 0x10c8acf to 0x38df552 2024-09-21T15:49:28.028 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:28.024+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:28.032 INFO:teuthology.orchestra.run.smithi089.stdout:1873 write 0x5c131e6 thru 0x5c18e29 (0x5c44 bytes) 2024-09-21T15:49:28.037 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:28.038 INFO:teuthology.orchestra.run.smithi089.stdout:1875 write 0x17f884e thru 0x17fb207 (0x29ba bytes) 2024-09-21T15:49:28.040 INFO:teuthology.orchestra.run.smithi089.stdout:1876 read 0x4ed6c97 thru 0x4ed7744 (0xaae bytes) 2024-09-21T15:49:28.040 INFO:teuthology.orchestra.run.smithi089.stdout:1877 write 0x14d03c2 thru 0x14d1b35 (0x1774 bytes) 2024-09-21T15:49:28.041 INFO:teuthology.orchestra.run.smithi089.stdout:1880 write 0x26f89d2 thru 0x26fd8d2 (0x4f01 bytes) 2024-09-21T15:49:28.043 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:28.043 INFO:teuthology.orchestra.run.smithi089.stdout:1882 read 0x2546ebe thru 0x254d882 (0x69c5 bytes) 2024-09-21T15:49:28.045 INFO:teuthology.orchestra.run.smithi089.stdout:1883 trunc from 0x5c18e2a to 0x7434ac2 2024-09-21T15:49:28.058 INFO:teuthology.orchestra.run.smithi089.stdout:1885 trunc from 0x7434ac2 to 0xc4c832 2024-09-21T15:49:28.084 INFO:teuthology.orchestra.run.smithi089.stdout:1886 write 0x1ae1992 thru 0x1ae4adb (0x314a bytes) 2024-09-21T15:49:28.087 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:28.096 INFO:teuthology.orchestra.run.smithi089.stdout:1888 writesame 0x53e77ae thru 0x53eaf3c data_size 0xb(0x378f bytes) 2024-09-21T15:49:28.104 INFO:teuthology.orchestra.run.smithi089.stdout:1889 trunc from 0x53eaf3d to 0xbd0a943 2024-09-21T15:49:28.109 INFO:teuthology.orchestra.run.smithi089.stdout:1890 punch from 0xbc8430a to 0xbc8b239, (0x6f2f bytes) 2024-09-21T15:49:28.109 INFO:teuthology.orchestra.run.smithi089.stdout:1891 write 0x4ad6971 thru 0x4ae383b (0xcecb bytes) 2024-09-21T15:49:28.115 INFO:teuthology.orchestra.run.smithi089.stdout:1892 write 0x7caf59 thru 0x7d9414 (0xe4bc bytes) 2024-09-21T15:49:28.120 INFO:teuthology.orchestra.run.smithi089.stdout:1893 trunc from 0xbd0a943 to 0xc4576c3 2024-09-21T15:49:28.126 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:28.125+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:28.130 INFO:teuthology.orchestra.run.smithi089.stdout:1894 trunc from 0xc4576c3 to 0x8088951 2024-09-21T15:49:28.135 INFO:teuthology.orchestra.run.smithi089.stdout:1895 writesame 0xa4a5 thru 0x120c1 data_size 0x59(0x7c1d bytes) 2024-09-21T15:49:28.138 INFO:teuthology.orchestra.run.smithi089.stdout:1896 punch from 0x7c7c67d to 0x7c8c12c, (0xfaaf bytes) 2024-09-21T15:49:28.146 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:28.146 INFO:teuthology.orchestra.run.smithi089.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T15:49:28.147 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:28.149 INFO:teuthology.orchestra.run.smithi089.stdout:1897 write 0xd91339b thru 0xd9195e0 (0x6246 bytes) 2024-09-21T15:49:28.154 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:28.154 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:49:28.154 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:49:28.154 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:28.156 INFO:teuthology.orchestra.run.smithi089.stdout:1899 read 0xd4229f1 thru 0xd42bd2d (0x933d bytes) 2024-09-21T15:49:28.156 INFO:teuthology.orchestra.run.smithi089.stdout:1900 writesame 0x6c2c68a thru 0x6c34bfc data_size 0x1(0x8573 bytes) 2024-09-21T15:49:28.163 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:49:28.167 INFO:teuthology.orchestra.run.smithi089.stdout:1901 read 0x82cea0d thru 0x82db18a (0xc77e bytes) 2024-09-21T15:49:28.168 INFO:teuthology.orchestra.run.smithi089.stdout:1902 read 0xd0f12e3 thru 0xd0f5cca (0x49e8 bytes) 2024-09-21T15:49:28.168 INFO:teuthology.orchestra.run.smithi089.stdout:1903 read 0xba4f937 thru 0xba57c2a (0x82f4 bytes) 2024-09-21T15:49:28.178 INFO:teuthology.orchestra.run.smithi089.stdout:1904 trunc from 0xd9195e1 to 0x503e7f2 2024-09-21T15:49:28.188 INFO:teuthology.orchestra.run.smithi089.stdout:1907 read 0x24556e8 thru 0x245ed34 (0x964d bytes) 2024-09-21T15:49:28.188 INFO:teuthology.orchestra.run.smithi089.stdout:1908 read 0x3ab39db thru 0x3ab4931 (0xf57 bytes) 2024-09-21T15:49:28.191 INFO:teuthology.orchestra.run.smithi089.stdout:1909 trunc from 0x503e7f2 to 0x7380a84 2024-09-21T15:49:28.197 INFO:teuthology.orchestra.run.smithi089.stdout:1912 punch from 0x1069e86 to 0x1071a1c, (0x7b96 bytes) 2024-09-21T15:49:28.197 INFO:teuthology.orchestra.run.smithi089.stdout:1913 punch from 0xab0b8f to 0xabb8af, (0xad20 bytes) 2024-09-21T15:49:28.197 INFO:teuthology.orchestra.run.smithi089.stdout:1914 punch from 0x14dce90 to 0x14e33de, (0x654e bytes) 2024-09-21T15:49:28.197 INFO:teuthology.orchestra.run.smithi089.stdout:1915 punch from 0x28236f2 to 0x282867f, (0x4f8d bytes) 2024-09-21T15:49:28.197 INFO:teuthology.orchestra.run.smithi089.stdout:1916 read 0x4c696f thru 0x4cde8d (0x751f bytes) 2024-09-21T15:49:28.197 INFO:teuthology.orchestra.run.smithi089.stdout:1917 write 0x623ff92 thru 0x624418a (0x41f9 bytes) 2024-09-21T15:49:28.201 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:28.201 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:28.201 INFO:teuthology.orchestra.run.smithi089.stdout:1921 punch from 0x364e259 to 0x36593c5, (0xb16c bytes) 2024-09-21T15:49:28.201 INFO:teuthology.orchestra.run.smithi089.stdout:1922 read 0x62b2728 thru 0x62c1392 (0xec6b bytes) 2024-09-21T15:49:28.201 INFO:teuthology.orchestra.run.smithi089.stdout:1923 read 0xa7f502 thru 0xa8c354 (0xce53 bytes) 2024-09-21T15:49:28.201 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:28.202 INFO:teuthology.orchestra.run.smithi089.stdout:1926 read 0x6d7dd06 thru 0x6d83640 (0x593b bytes) 2024-09-21T15:49:28.202 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:28.202 INFO:teuthology.orchestra.run.smithi089.stdout:1928 punch from 0x39e4586 to 0x39e5f85, (0x19ff bytes) 2024-09-21T15:49:28.202 INFO:teuthology.orchestra.run.smithi089.stdout:1929 writesame 0x18470f0 thru 0x184a743 data_size 0x39(0x3654 bytes) 2024-09-21T15:49:28.203 INFO:teuthology.orchestra.run.smithi089.stdout:1932 punch from 0x354fe95 to 0x3558500, (0x866b bytes) 2024-09-21T15:49:28.203 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:28.203 INFO:teuthology.orchestra.run.smithi089.stdout:1934 punch from 0x3b0e403 to 0x3b10258, (0x1e55 bytes) 2024-09-21T15:49:28.203 INFO:teuthology.orchestra.run.smithi089.stdout:1935 punch from 0xba1d74 to 0xbad397, (0xb623 bytes) 2024-09-21T15:49:28.203 INFO:teuthology.orchestra.run.smithi089.stdout:1940 read 0x361030e thru 0x36166a8 (0x639b bytes) 2024-09-21T15:49:28.206 INFO:teuthology.orchestra.run.smithi089.stdout:1941 trunc from 0x7380a84 to 0x9b2de9d 2024-09-21T15:49:28.211 INFO:teuthology.orchestra.run.smithi089.stdout:1942 write 0x294b473 thru 0x294dedd (0x2a6b bytes) 2024-09-21T15:49:28.212 INFO:teuthology.orchestra.run.smithi089.stdout:1944 read 0x8c90095 thru 0x8c9dd54 (0xdcc0 bytes) 2024-09-21T15:49:28.215 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:49:28.215 INFO:teuthology.orchestra.run.smithi089.stdout:1945 trunc from 0x9b2de9d to 0xc051355 2024-09-21T15:49:28.220 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:28.226 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:28.225+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:28.233 INFO:teuthology.orchestra.run.smithi089.stdout:1949 trunc from 0xc051355 to 0x150555b 2024-09-21T15:49:28.271 INFO:teuthology.orchestra.run.smithi089.stdout:1950 writesame 0xe5c4962 thru 0xe5ca68d data_size 0x4(0x5d2c bytes) 2024-09-21T15:49:28.271 INFO:teuthology.orchestra.run.smithi089.stdout:1951 read 0xc746e69 thru 0xc74a2a6 (0x343e bytes) 2024-09-21T15:49:28.271 INFO:teuthology.orchestra.run.smithi089.stdout:1952 read 0x15f6e71 thru 0x1602d60 (0xbef0 bytes) 2024-09-21T15:49:28.271 INFO:teuthology.orchestra.run.smithi089.stdout:1953 punch from 0xcd4305a to 0xcd52829, (0xf7cf bytes) 2024-09-21T15:49:28.283 INFO:teuthology.orchestra.run.smithi089.stdout:1955 trunc from 0xe5ca68e to 0x4340414 2024-09-21T15:49:28.289 INFO:teuthology.orchestra.run.smithi089.stdout:1956 read 0x6a0571 thru 0x6a0bbb (0x64b bytes) 2024-09-21T15:49:28.289 INFO:teuthology.orchestra.run.smithi089.stdout:1957 read 0x7acd08 thru 0x7ad8f1 (0xbea bytes) 2024-09-21T15:49:28.290 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:28.290 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:49:28.290 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:49:28.290 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:49:28.290 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:28.291 INFO:teuthology.orchestra.run.smithi089.stdout:1958 read 0x2900ec5 thru 0x2901b38 (0xc74 bytes) 2024-09-21T15:49:28.291 INFO:teuthology.orchestra.run.smithi089.stdout:1959 writesame 0x3a7a6fe thru 0x3a89369 data_size 0x2(0xec6c bytes) 2024-09-21T15:49:28.301 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:49:28.304 INFO:teuthology.orchestra.run.smithi089.stdout:1960 punch from 0xa93a98 to 0xa9f805, (0xbd6d bytes) 2024-09-21T15:49:28.307 INFO:teuthology.orchestra.run.smithi089.stdout:1961 trunc from 0x4340414 to 0x6e4596d 2024-09-21T15:49:28.313 INFO:teuthology.orchestra.run.smithi089.stdout:1962 punch from 0x24a559a to 0x24b3cbd, (0xe723 bytes) 2024-09-21T15:49:28.313 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:28.313 INFO:teuthology.orchestra.run.smithi089.stdout:1966 punch from 0x491356a to 0x49199d2, (0x6468 bytes) 2024-09-21T15:49:28.322 INFO:teuthology.orchestra.run.smithi089.stdout:1968 writesame 0x9be1caa thru 0x9be491c data_size 0x1(0x2c73 bytes) 2024-09-21T15:49:28.322 INFO:teuthology.orchestra.run.smithi089.stdout:1969 write 0x3f11210 thru 0x3f19230 (0x8021 bytes) 2024-09-21T15:49:28.325 INFO:teuthology.orchestra.run.smithi089.stdout:1970 write 0x1a8c654 thru 0x1a92dd7 (0x6784 bytes) 2024-09-21T15:49:28.327 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:28.326+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:28.327 INFO:teuthology.orchestra.run.smithi089.stdout:1971 punch from 0x36e243b to 0x36e9ddf, (0x79a4 bytes) 2024-09-21T15:49:28.328 INFO:teuthology.orchestra.run.smithi089.stdout:1972 trunc from 0x9be491d to 0x933dd8b 2024-09-21T15:49:28.341 INFO:teuthology.orchestra.run.smithi089.stdout:1973 trunc from 0x933dd8b to 0x3c98884 2024-09-21T15:49:28.349 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:28.349 INFO:teuthology.orchestra.run.smithi089.stdout:1975 punch from 0x9d7f8a to 0x9e6658, (0xe6ce bytes) 2024-09-21T15:49:28.349 INFO:teuthology.orchestra.run.smithi089.stdout:1976 write 0x318cfcb thru 0x319a53a (0xd570 bytes) 2024-09-21T15:49:28.354 INFO:teuthology.orchestra.run.smithi089.stdout:1977 trunc from 0x3c98884 to 0x13172a4 2024-09-21T15:49:28.366 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-09-21T15:49:28.366 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:49:28.370 INFO:teuthology.orchestra.run.smithi089.stdout:1978 read 0x529006 thru 0x53170c (0x8707 bytes) 2024-09-21T15:49:28.384 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:49:28.384 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:49:28.388 INFO:teuthology.orchestra.run.smithi089.stdout:1979 write 0xb02cd89 thru 0xb039ac3 (0xcd3b bytes) 2024-09-21T15:49:28.401 INFO:teuthology.orchestra.run.smithi089.stdout:1980 trunc from 0xb039ac4 to 0x52a8904 2024-09-21T15:49:28.401 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:49:28.402 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:49:28.413 INFO:teuthology.orchestra.run.smithi089.stdout:1981 punch from 0xc96d36 to 0xca08e1, (0x9bab bytes) 2024-09-21T15:49:28.413 INFO:teuthology.orchestra.run.smithi089.stdout:1982 punch from 0x14de1a8 to 0x14de2b0, (0x108 bytes) 2024-09-21T15:49:28.417 INFO:teuthology.orchestra.run.smithi089.stdout:1983 trunc from 0x52a8904 to 0xbf8f7c 2024-09-21T15:49:28.420 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:49:28.420 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:49:28.428 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:28.427+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:28.430 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:28.430 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:28.437 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:49:28.437 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:49:28.442 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:49:28.445 INFO:teuthology.orchestra.run.smithi089.stdout:1984 write 0x613cffb thru 0x61480af (0xb0b5 bytes) 2024-09-21T15:49:28.454 INFO:teuthology.orchestra.run.smithi089.stdout:1986 read 0x2255cde thru 0x2262398 (0xc6bb bytes) 2024-09-21T15:49:28.455 INFO:teuthology.orchestra.run.smithi089.stdout:1988 punch from 0x6aed54 to 0x6b6b8b, (0x7e37 bytes) 2024-09-21T15:49:28.455 INFO:teuthology.orchestra.run.smithi089.stdout:1990 read 0x54d6a5c thru 0x54deb40 (0x80e5 bytes) 2024-09-21T15:49:28.455 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:49:28.455 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:49:28.458 INFO:teuthology.orchestra.run.smithi089.stdout:1993 trunc from 0x61480b0 to 0x27e0a6d 2024-09-21T15:49:28.474 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:49:28.475 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:49:28.483 INFO:teuthology.orchestra.run.smithi089.stdout:1994 read 0x330481 thru 0x33e1c9 (0xdd49 bytes) 2024-09-21T15:49:28.490 INFO:teuthology.orchestra.run.smithi089.stdout:1995 trunc from 0x27e0a6d to 0x87715f8 2024-09-21T15:49:28.492 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:49:28.492 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:49:28.495 INFO:teuthology.orchestra.run.smithi089.stdout:1996 read 0x3081800 thru 0x308ccb8 (0xb4b9 bytes) 2024-09-21T15:49:28.496 INFO:teuthology.orchestra.run.smithi089.stdout:1997 read 0x3f9d1f8 thru 0x3f9d465 (0x26e bytes) 2024-09-21T15:49:28.505 INFO:teuthology.orchestra.run.smithi089.stdout:1998 writesame 0xa247c83 thru 0xa251175 data_size 0x1(0x94f3 bytes) 2024-09-21T15:49:28.506 INFO:teuthology.orchestra.run.smithi089.stdout:1999 read 0x5614b76 thru 0x5619fc5 (0x5450 bytes) 2024-09-21T15:49:28.506 INFO:teuthology.orchestra.run.smithi089.stdout:2000 read 0x531d553 thru 0x532380e (0x62bc bytes) 2024-09-21T15:49:28.506 INFO:teuthology.orchestra.run.smithi089.stdout:2001 punch from 0x6ce7cf6 to 0x6cec8d9, (0x4be3 bytes) 2024-09-21T15:49:28.506 INFO:teuthology.orchestra.run.smithi089.stdout:2002 read 0x2ad315e thru 0x2ad8072 (0x4f15 bytes) 2024-09-21T15:49:28.506 INFO:teuthology.orchestra.run.smithi089.stdout:2003 write 0x6f23ddc thru 0x6f29487 (0x56ac bytes) 2024-09-21T15:49:28.508 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:28.508 INFO:teuthology.orchestra.run.smithi089.stdout:2006 writesame 0x46da341 thru 0x46e0bf2 data_size 0x9(0x68b2 bytes) 2024-09-21T15:49:28.512 INFO:teuthology.orchestra.run.smithi089.stdout:2007 read 0x2abf267 thru 0x2acea47 (0xf7e1 bytes) 2024-09-21T15:49:28.512 INFO:teuthology.orchestra.run.smithi089.stdout:2008 write 0x36774f2 thru 0x3683f30 (0xca3f bytes) 2024-09-21T15:49:28.522 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:28.522 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:28.526 INFO:teuthology.orchestra.run.smithi089.stdout:2012 trunc from 0xa251176 to 0x6638f82 2024-09-21T15:49:28.528 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:28.527+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:28.536 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:28.536 INFO:teuthology.orchestra.run.smithi089.stdout:2015 read 0x5464265 thru 0x5468c62 (0x49fe bytes) 2024-09-21T15:49:28.552 INFO:teuthology.orchestra.run.smithi089.stdout:2016 write 0xd2a8e63 thru 0xd2b4fc8 (0xc166 bytes) 2024-09-21T15:49:28.561 INFO:teuthology.orchestra.run.smithi089.stdout:2018 punch from 0x25666c4 to 0x2575284, (0xebc0 bytes) 2024-09-21T15:49:28.561 INFO:teuthology.orchestra.run.smithi089.stdout:2021 read 0xcafbe thru 0xcc681 (0x16c4 bytes) 2024-09-21T15:49:28.561 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:28.561 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:28.562 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:28.562 INFO:teuthology.orchestra.run.smithi089.stdout:2024 read 0x2100ad5 thru 0x210f5c3 (0xeaef bytes) 2024-09-21T15:49:28.570 INFO:teuthology.orchestra.run.smithi089.stdout:2025 writesame 0xdf0786e thru 0xdf0e1d4 data_size 0xb(0x6967 bytes) 2024-09-21T15:49:28.570 INFO:teuthology.orchestra.run.smithi089.stdout:2030 read 0xc245ed1 thru 0xc2501bd (0xa2ed bytes) 2024-09-21T15:49:28.570 INFO:teuthology.orchestra.run.smithi089.stdout:2033 write 0xe86904 thru 0xe91e65 (0xb562 bytes) 2024-09-21T15:49:28.571 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:49:28.572 INFO:teuthology.orchestra.run.smithi089.stdout:2034 read 0x2229924 thru 0x222c6e9 (0x2dc6 bytes) 2024-09-21T15:49:28.572 INFO:teuthology.orchestra.run.smithi089.stdout:2035 clone 8 order 23 su 1048576 sc 7 2024-09-21T15:49:28.630 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:28.629+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:28.673 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:28.673 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:28.682 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:49:28.730 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:28.729+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:28.786 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:28.786 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:28.796 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:49:28.837 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:28.836+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:28.843 INFO:teuthology.orchestra.run.smithi089.stdout:truncating image image_client.0-clone7 from 0xdf0e1d5 (overlap 0xbe8a1e) to 0x4c373f 2024-09-21T15:49:28.908 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:28.908 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:28.917 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:49:28.927 INFO:teuthology.orchestra.run.smithi089.stdout:checking clone #6, image image_client.0-clone6 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent7 2024-09-21T15:49:28.961 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:28.960+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:29.033 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:29.031+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:29.041 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:29.042 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:29.053 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:49:29.137 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:29.136+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:29.195 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:29.195 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:29.204 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:49:29.237 INFO:teuthology.orchestra.run.smithi089.stdout:2036 trunc from 0xdf0e1d5 to 0x71b6fca 2024-09-21T15:49:29.245 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:29.244+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:29.257 INFO:teuthology.orchestra.run.smithi089.stdout:2037 punch from 0x344a670 to 0x3450c9d, (0x662d bytes) 2024-09-21T15:49:29.258 INFO:teuthology.orchestra.run.smithi089.stdout:2038 punch from 0x16e7118 to 0x16f18aa, (0xa792 bytes) 2024-09-21T15:49:29.266 INFO:teuthology.orchestra.run.smithi089.stdout:2039 writesame 0x9d30bd2 thru 0x9d40a18 data_size 0x2f(0xfe47 bytes) 2024-09-21T15:49:29.266 INFO:teuthology.orchestra.run.smithi089.stdout:2040 read 0x163df6c thru 0x163f096 (0x112b bytes) 2024-09-21T15:49:29.273 INFO:teuthology.orchestra.run.smithi089.stdout:2041 write 0xaa7e04a thru 0xaa85830 (0x77e7 bytes) 2024-09-21T15:49:29.278 INFO:teuthology.orchestra.run.smithi089.stdout:2042 trunc from 0xaa85831 to 0xe084067 2024-09-21T15:49:29.282 INFO:teuthology.orchestra.run.smithi089.stdout:2043 write 0x9ea5ab2 thru 0x9ea6360 (0x8af bytes) 2024-09-21T15:49:29.285 INFO:teuthology.orchestra.run.smithi089.stdout:2044 writesame 0x1a822ab thru 0x1a82588 data_size 0x1(0x2de bytes) 2024-09-21T15:49:29.297 INFO:teuthology.orchestra.run.smithi089.stdout:2046 write 0xd29990d thru 0xd2a4cfe (0xb3f2 bytes) 2024-09-21T15:49:29.300 INFO:teuthology.orchestra.run.smithi089.stdout:2047 read 0x502e7e thru 0x50b774 (0x88f7 bytes) 2024-09-21T15:49:29.301 INFO:teuthology.orchestra.run.smithi089.stdout:2049 punch from 0x9050372 to 0x9053f9f, (0x3c2d bytes) 2024-09-21T15:49:29.301 INFO:teuthology.orchestra.run.smithi089.stdout:2050 write 0x95b370 thru 0x960789 (0x541a bytes) 2024-09-21T15:49:29.312 INFO:teuthology.orchestra.run.smithi089.stdout:2051 clone 9 order 19 su 131072 sc 5 2024-09-21T15:49:29.335 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:29.334+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:29.352 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:29.352 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:29.360 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:49:29.444 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:29.443+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:29.501 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:29.502 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:29.510 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:49:29.551 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:29.550+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:29.637 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:29.636+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:29.653 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:29.653 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:29.661 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T15:49:29.738 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:29.737+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:29.794 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T15:49:29.806 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:29.806 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:29.815 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T15:49:29.839 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:29.838+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:29.867 INFO:teuthology.orchestra.run.smithi089.stdout:truncating image image_client.0-clone8 from 0xe084067 (overlap 0x71b6fca) to 0xb5f194 2024-09-21T15:49:29.934 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:29.935 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:29.940 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:29.938+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:29.943 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T15:49:29.997 INFO:teuthology.orchestra.run.smithi089.stdout:checking clone #7, image image_client.0-clone7 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent8 2024-09-21T15:49:30.007 INFO:teuthology.orchestra.run.smithi089.stdout:2052 read 0x72f9838 thru 0x72ffe87 (0x6650 bytes) 2024-09-21T15:49:30.008 INFO:teuthology.orchestra.run.smithi089.stdout:2053 read 0xc2175f5 thru 0xc221605 (0xa011 bytes) 2024-09-21T15:49:30.008 INFO:teuthology.orchestra.run.smithi089.stdout:2054 punch from 0xb96e0ca to 0xb97d510, (0xf446 bytes) 2024-09-21T15:49:30.014 INFO:teuthology.orchestra.run.smithi089.stdout:2055 write 0x807fb4 thru 0x80815b (0x1a8 bytes) 2024-09-21T15:49:30.019 INFO:teuthology.orchestra.run.smithi089.stdout:2058 read 0x5db5209 thru 0x5dc1e48 (0xcc40 bytes) 2024-09-21T15:49:30.019 INFO:teuthology.orchestra.run.smithi089.stdout:2060 read 0x8dfea01 thru 0x8e0e0b3 (0xf6b3 bytes) 2024-09-21T15:49:30.020 INFO:teuthology.orchestra.run.smithi089.stdout:2061 writesame 0xa84a692 thru 0xa856ca5 data_size 0xe(0xc614 bytes) 2024-09-21T15:49:30.027 INFO:teuthology.orchestra.run.smithi089.stdout:2062 punch from 0xb3ebad6 to 0xb3f120c, (0x5736 bytes) 2024-09-21T15:49:30.027 INFO:teuthology.orchestra.run.smithi089.stdout:2063 write 0xb5f46fe thru 0xb5fbb2a (0x742d bytes) 2024-09-21T15:49:30.030 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:30.030 INFO:teuthology.orchestra.run.smithi089.stdout:2065 writesame 0x596e1d6 thru 0x5974923 data_size 0x7(0x674e bytes) 2024-09-21T15:49:30.035 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:30.035 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:30.035 INFO:teuthology.orchestra.run.smithi089.stdout:2069 writesame 0xacb3dff thru 0xacb8d78 data_size 0x3(0x4f7a bytes) 2024-09-21T15:49:30.041 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:30.040+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:30.051 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:30.051 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:30.055 INFO:teuthology.orchestra.run.smithi089.stdout:2070 write 0x2a4e04c thru 0x2a512f1 (0x32a6 bytes) 2024-09-21T15:49:30.060 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T15:49:30.060 INFO:teuthology.orchestra.run.smithi089.stdout:2071 punch from 0x3bdd6bd to 0x3bebbb7, (0xe4fa bytes) 2024-09-21T15:49:30.061 INFO:teuthology.orchestra.run.smithi089.stdout:2072 writesame 0x16b5fd thru 0x178d63 data_size 0x21(0xd767 bytes) 2024-09-21T15:49:30.068 INFO:teuthology.orchestra.run.smithi089.stdout:2074 writesame 0x2a38fe9 thru 0x2a470c2 data_size 0x11(0xe0da bytes) 2024-09-21T15:49:30.072 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:30.072 INFO:teuthology.orchestra.run.smithi089.stdout:2076 write 0x248e7c9 thru 0x2495940 (0x7178 bytes) 2024-09-21T15:49:30.086 INFO:teuthology.orchestra.run.smithi089.stdout:2077 trunc from 0xe084067 to 0x523820c 2024-09-21T15:49:30.102 INFO:teuthology.orchestra.run.smithi089.stdout:2078 punch from 0x14c0e53 to 0x14cce1e, (0xbfcb bytes) 2024-09-21T15:49:30.102 INFO:teuthology.orchestra.run.smithi089.stdout:2079 read 0x4cd3a1 thru 0x4cebc8 (0x1828 bytes) 2024-09-21T15:49:30.102 INFO:teuthology.orchestra.run.smithi089.stdout:2082 writesame 0x41d36b4 thru 0x41d64f2 data_size 0x1(0x2e3f bytes) 2024-09-21T15:49:30.110 INFO:teuthology.orchestra.run.smithi089.stdout:2083 read 0xf93d0c thru 0xf98ce0 (0x4fd5 bytes) 2024-09-21T15:49:30.110 INFO:teuthology.orchestra.run.smithi089.stdout:2084 read 0x1a802e8 thru 0x1a892fa (0x9013 bytes) 2024-09-21T15:49:30.111 INFO:teuthology.orchestra.run.smithi089.stdout:2085 punch from 0x19377fc to 0x194731f, (0xfb23 bytes) 2024-09-21T15:49:30.111 INFO:teuthology.orchestra.run.smithi089.stdout:2086 read 0x318d64c thru 0x319cd36 (0xf6eb bytes) 2024-09-21T15:49:30.111 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:30.111 INFO:teuthology.orchestra.run.smithi089.stdout:2088 read 0x486e31a thru 0x487bc87 (0xd96e bytes) 2024-09-21T15:49:30.112 INFO:teuthology.orchestra.run.smithi089.stdout:2089 read 0x1d06def thru 0x1d15ce1 (0xeef3 bytes) 2024-09-21T15:49:30.112 INFO:teuthology.orchestra.run.smithi089.stdout:2090 punch from 0xcaed06 to 0xcba305, (0xb5ff bytes) 2024-09-21T15:49:30.121 INFO:teuthology.orchestra.run.smithi089.stdout:2091 write 0x8a2ac4a thru 0x8a3a8dd (0xfc94 bytes) 2024-09-21T15:49:30.124 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:30.131 INFO:teuthology.orchestra.run.smithi089.stdout:2095 write 0xa71d721 thru 0xa7255f9 (0x7ed9 bytes) 2024-09-21T15:49:30.139 INFO:teuthology.orchestra.run.smithi089.stdout:2096 punch from 0x7746b35 to 0x774beb9, (0x5384 bytes) 2024-09-21T15:49:30.139 INFO:teuthology.orchestra.run.smithi089.stdout:2098 clone 10 order 19 su 65536 sc 11 2024-09-21T15:49:30.142 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:30.141+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:30.203 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:30.203 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:30.212 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T15:49:30.255 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:30.254+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:30.312 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:30.312 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:30.320 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T15:49:30.362 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:30.361+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:30.444 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:30.443+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:30.464 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:30.464 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:30.473 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:49:30.505 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:49:30.560 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:30.559+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:30.580 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:30.580 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:30.589 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:49:30.636 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-09-21T15:49:30.636 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:49:30.647 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:30.645+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:30.654 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:49:30.655 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:49:30.685 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:49:30.685 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:49:30.701 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:49:30.701 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:49:30.718 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:49:30.718 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:49:30.735 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:49:30.735 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:49:30.743 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:30.744 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:49:30.744 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:49:30.744 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:49:30.744 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:30.748 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:30.747+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:30.751 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:49:30.752 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:49:30.752 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:49:30.768 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:49:30.768 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:49:30.794 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is set 2024-09-21T15:49:30.849 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:30.848+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:30.879 INFO:teuthology.orchestra.run.smithi089.stdout:truncating image image_client.0-clone9 from 0xa7255fa (overlap 0x523820c) to 0x398db08 2024-09-21T15:49:30.880 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:30.880 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:49:30.880 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:49:30.880 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:49:30.880 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.880 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28204 3.1f 3:f84cae28:::rbd_data.1133ebd6d3ba.0000000000000006:a [read 491520~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:30.881 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.205303+0000", 2024-09-21T15:49:30.881 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.6730246520000001, 2024-09-21T15:49:30.881 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.027260120999999998, 2024-09-21T15:49:30.881 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:30.881 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:30.881 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:30.881 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:30.881 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:30.881 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28204 2024-09-21T15:49:30.881 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.881 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:30.881 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.881 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:30.881 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205303+0000", 2024-09-21T15:49:30.881 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:30.881 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.882 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.882 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:30.882 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205303+0000", 2024-09-21T15:49:30.882 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:30.882 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.882 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.882 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:30.882 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205302+0000", 2024-09-21T15:49:30.882 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:30.882 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.882 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.882 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:30.882 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205305+0000", 2024-09-21T15:49:30.882 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4480000000000001e-06 2024-09-21T15:49:30.882 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.883 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.883 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:30.883 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205312+0000", 2024-09-21T15:49:30.883 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6789999999999999e-06 2024-09-21T15:49:30.883 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.883 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.883 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:30.883 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205316+0000", 2024-09-21T15:49:30.883 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8189999999999999e-06 2024-09-21T15:49:30.883 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.883 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.883 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:30.883 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.232290+0000", 2024-09-21T15:49:30.883 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.02697397 2024-09-21T15:49:30.883 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.884 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.884 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:30.884 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.232353+0000", 2024-09-21T15:49:30.884 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2990000000000005e-05 2024-09-21T15:49:30.884 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.884 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.884 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:30.884 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.232563+0000", 2024-09-21T15:49:30.884 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000209712 2024-09-21T15:49:30.884 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:30.884 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:30.884 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:30.884 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.884 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.885 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28205 3.1f 3:f84cae28:::rbd_data.1133ebd6d3ba.0000000000000006:a [read 606208~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:30.885 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.205321+0000", 2024-09-21T15:49:30.885 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.6730070609999999, 2024-09-21T15:49:30.885 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.027700763999999999, 2024-09-21T15:49:30.885 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:30.885 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:30.885 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:30.885 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:30.885 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:30.885 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28205 2024-09-21T15:49:30.885 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.885 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:30.885 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.885 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:30.885 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205321+0000", 2024-09-21T15:49:30.885 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:30.885 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.886 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.886 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:30.886 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205321+0000", 2024-09-21T15:49:30.886 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:30.886 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.886 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.886 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:30.886 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205320+0000", 2024-09-21T15:49:30.886 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:30.886 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.886 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.886 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:30.886 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205322+0000", 2024-09-21T15:49:30.887 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.34e-06 2024-09-21T15:49:30.887 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.887 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.887 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:30.887 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205328+0000", 2024-09-21T15:49:30.887 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6250000000000004e-06 2024-09-21T15:49:30.887 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.887 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.887 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:30.887 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205347+0000", 2024-09-21T15:49:30.887 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8890999999999999e-05 2024-09-21T15:49:30.887 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.887 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.887 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:30.888 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.232590+0000", 2024-09-21T15:49:30.888 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.027242971000000001 2024-09-21T15:49:30.888 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.888 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.888 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:30.888 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.232725+0000", 2024-09-21T15:49:30.888 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013536099999999999 2024-09-21T15:49:30.888 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.888 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.888 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:30.888 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.233021+0000", 2024-09-21T15:49:30.888 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000296025 2024-09-21T15:49:30.888 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:30.888 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:30.889 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:30.889 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.889 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.889 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28206 3.1f 3:f84cae28:::rbd_data.1133ebd6d3ba.0000000000000006:a [read 720896~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:30.889 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.205353+0000", 2024-09-21T15:49:30.889 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.6729747919999998, 2024-09-21T15:49:30.889 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.028073124000000001, 2024-09-21T15:49:30.889 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:30.889 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:30.889 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:30.889 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:30.889 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:30.889 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28206 2024-09-21T15:49:30.889 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.890 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:30.890 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.890 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:30.890 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205353+0000", 2024-09-21T15:49:30.890 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:30.890 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.890 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.890 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:30.890 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205353+0000", 2024-09-21T15:49:30.890 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:30.890 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.890 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.890 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:30.890 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205352+0000", 2024-09-21T15:49:30.891 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:30.891 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.891 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.891 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:30.891 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205358+0000", 2024-09-21T15:49:30.891 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.784e-06 2024-09-21T15:49:30.891 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.891 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.891 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:30.891 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205365+0000", 2024-09-21T15:49:30.891 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5339999999999996e-06 2024-09-21T15:49:30.891 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.891 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.892 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:30.892 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205370+0000", 2024-09-21T15:49:30.892 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0170000000000004e-06 2024-09-21T15:49:30.892 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.892 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.892 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:30.892 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.233048+0000", 2024-09-21T15:49:30.892 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.027678514000000001 2024-09-21T15:49:30.892 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.892 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.892 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:30.892 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.233117+0000", 2024-09-21T15:49:30.892 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8720000000000006e-05 2024-09-21T15:49:30.892 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.892 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.892 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:30.893 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.233426+0000", 2024-09-21T15:49:30.893 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00030903300000000001 2024-09-21T15:49:30.893 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:30.893 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:30.893 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:30.893 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.893 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.893 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28209 3.1f 3:f84cae28:::rbd_data.1133ebd6d3ba.0000000000000006:a [read 827392~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:30.893 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.205374+0000", 2024-09-21T15:49:30.893 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.6729535750000002, 2024-09-21T15:49:30.893 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.028371325999999999, 2024-09-21T15:49:30.893 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:30.893 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:30.893 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:30.893 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:30.893 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:30.894 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28209 2024-09-21T15:49:30.894 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.894 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:30.894 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.894 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:30.894 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205374+0000", 2024-09-21T15:49:30.894 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:30.894 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.894 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.894 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:30.894 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205374+0000", 2024-09-21T15:49:30.894 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:30.894 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.894 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.894 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:30.894 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205374+0000", 2024-09-21T15:49:30.895 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:30.895 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.895 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.895 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:30.895 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205376+0000", 2024-09-21T15:49:30.895 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3700000000000002e-06 2024-09-21T15:49:30.895 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.895 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.895 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:30.895 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205382+0000", 2024-09-21T15:49:30.895 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7679999999999997e-06 2024-09-21T15:49:30.895 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.895 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.895 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:30.895 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205386+0000", 2024-09-21T15:49:30.895 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2089999999999999e-06 2024-09-21T15:49:30.896 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.896 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.896 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:30.896 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.233448+0000", 2024-09-21T15:49:30.896 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.028062133999999999 2024-09-21T15:49:30.896 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.896 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.896 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:30.896 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.233511+0000", 2024-09-21T15:49:30.896 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3189999999999996e-05 2024-09-21T15:49:30.896 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.896 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.896 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:30.896 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.233745+0000", 2024-09-21T15:49:30.896 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000234052 2024-09-21T15:49:30.896 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:30.897 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:30.897 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:30.897 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.897 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.897 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28210 3.1f 3:f84cae28:::rbd_data.1133ebd6d3ba.0000000000000006:a [read 942080~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:30.897 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.205390+0000", 2024-09-21T15:49:30.897 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.6729376619999998, 2024-09-21T15:49:30.897 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.028697378999999999, 2024-09-21T15:49:30.897 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:30.897 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:30.897 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:30.897 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:30.897 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:30.897 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28210 2024-09-21T15:49:30.897 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.897 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:30.898 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.898 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:30.898 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205390+0000", 2024-09-21T15:49:30.898 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:30.898 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.898 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.898 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:30.898 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205390+0000", 2024-09-21T15:49:30.898 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:30.898 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.898 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.898 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:30.898 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205390+0000", 2024-09-21T15:49:30.898 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:30.898 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.899 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.899 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:30.899 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205392+0000", 2024-09-21T15:49:30.899 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1950000000000002e-06 2024-09-21T15:49:30.899 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.899 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.899 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:30.899 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205399+0000", 2024-09-21T15:49:30.899 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8589999999999999e-06 2024-09-21T15:49:30.899 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.899 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.899 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:30.899 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.205403+0000", 2024-09-21T15:49:30.899 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6650000000000002e-06 2024-09-21T15:49:30.899 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.900 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.900 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:30.900 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.233785+0000", 2024-09-21T15:49:30.900 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.028381516999999998 2024-09-21T15:49:30.900 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.900 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.900 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:30.900 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.233843+0000", 2024-09-21T15:49:30.900 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8186000000000002e-05 2024-09-21T15:49:30.900 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.900 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.900 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:30.900 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.234087+0000", 2024-09-21T15:49:30.901 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00024431700000000003 2024-09-21T15:49:30.901 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:30.901 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:30.901 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:30.901 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.901 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.901 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28276 3.17 3:ea41abc7:::rbd_data.1133ebd6d3ba.0000000000000016:a [read 270336~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:30.901 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.207088+0000", 2024-09-21T15:49:30.901 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.6712393990000001, 2024-09-21T15:49:30.901 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.027154052000000001, 2024-09-21T15:49:30.901 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:30.901 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:30.901 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:30.901 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:30.901 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:30.901 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28276 2024-09-21T15:49:30.902 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.902 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:30.902 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.902 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:30.902 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207088+0000", 2024-09-21T15:49:30.902 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:30.902 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.902 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.902 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:30.902 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207088+0000", 2024-09-21T15:49:30.902 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:30.902 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.902 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.902 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:30.902 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207088+0000", 2024-09-21T15:49:30.903 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:30.903 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.903 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.903 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:30.903 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207090+0000", 2024-09-21T15:49:30.903 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2479999999999999e-06 2024-09-21T15:49:30.903 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.903 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.903 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:30.903 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207096+0000", 2024-09-21T15:49:30.903 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9329999999999998e-06 2024-09-21T15:49:30.903 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.903 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.903 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:30.903 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207101+0000", 2024-09-21T15:49:30.904 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8060000000000002e-06 2024-09-21T15:49:30.904 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.904 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.905 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:30.905 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.234128+0000", 2024-09-21T15:49:30.905 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.027026853 2024-09-21T15:49:30.906 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.906 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.906 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:30.906 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.234188+0000", 2024-09-21T15:49:30.906 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0105000000000001e-05 2024-09-21T15:49:30.906 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.906 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.906 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:30.906 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.234242+0000", 2024-09-21T15:49:30.906 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4468000000000001e-05 2024-09-21T15:49:30.906 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:30.906 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:30.906 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:30.907 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.907 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.907 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28277 3.17 3:ea41abc7:::rbd_data.1133ebd6d3ba.0000000000000016:a [read 385024~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:30.907 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.207105+0000", 2024-09-21T15:49:30.907 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.6712227259999999, 2024-09-21T15:49:30.907 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.027310174999999999, 2024-09-21T15:49:30.907 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:30.907 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:30.907 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:30.907 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:30.907 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:30.907 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28277 2024-09-21T15:49:30.907 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.907 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:30.908 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.908 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:30.908 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207105+0000", 2024-09-21T15:49:30.908 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:30.908 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.908 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.908 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:30.908 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207105+0000", 2024-09-21T15:49:30.908 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:30.908 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.908 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.908 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:30.908 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207105+0000", 2024-09-21T15:49:30.908 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:30.909 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.909 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.909 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:30.909 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207107+0000", 2024-09-21T15:49:30.909 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0849999999999999e-06 2024-09-21T15:49:30.909 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.909 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.909 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:30.909 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207112+0000", 2024-09-21T15:49:30.909 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6010000000000002e-06 2024-09-21T15:49:30.909 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.909 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.909 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:30.909 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207117+0000", 2024-09-21T15:49:30.909 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.831e-06 2024-09-21T15:49:30.910 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.910 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.910 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:30.910 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.234286+0000", 2024-09-21T15:49:30.910 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.027168512999999998 2024-09-21T15:49:30.910 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.910 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.910 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:30.910 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.234350+0000", 2024-09-21T15:49:30.910 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4505999999999995e-05 2024-09-21T15:49:30.910 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.910 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.910 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:30.910 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.234415+0000", 2024-09-21T15:49:30.910 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4999000000000006e-05 2024-09-21T15:49:30.911 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:30.911 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:30.911 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:30.911 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.911 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.911 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28278 3.17 3:ea41abc7:::rbd_data.1133ebd6d3ba.0000000000000016:a [read 499712~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:30.911 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.207121+0000", 2024-09-21T15:49:30.911 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.6712064939999998, 2024-09-21T15:49:30.911 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.027445132000000001, 2024-09-21T15:49:30.911 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:30.911 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:30.911 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:30.911 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:30.911 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:30.912 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28278 2024-09-21T15:49:30.912 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.912 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:30.912 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.912 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:30.912 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207121+0000", 2024-09-21T15:49:30.912 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:30.912 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.912 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.912 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:30.912 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207121+0000", 2024-09-21T15:49:30.912 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:30.912 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.912 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.912 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:30.912 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207121+0000", 2024-09-21T15:49:30.913 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:30.913 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.913 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.913 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:30.913 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207123+0000", 2024-09-21T15:49:30.913 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6170000000000001e-06 2024-09-21T15:49:30.913 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.913 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.913 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:30.913 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207129+0000", 2024-09-21T15:49:30.913 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8420000000000001e-06 2024-09-21T15:49:30.913 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.913 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.913 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:30.913 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207133+0000", 2024-09-21T15:49:30.913 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1470000000000003e-06 2024-09-21T15:49:30.914 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.914 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.914 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:30.914 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.234439+0000", 2024-09-21T15:49:30.914 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.027306119 2024-09-21T15:49:30.914 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.914 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.914 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:30.914 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.234505+0000", 2024-09-21T15:49:30.914 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5493999999999993e-05 2024-09-21T15:49:30.914 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.914 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.914 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:30.914 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.234566+0000", 2024-09-21T15:49:30.915 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1280999999999998e-05 2024-09-21T15:49:30.915 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:30.915 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:30.915 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:30.915 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.915 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.915 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28279 3.17 3:ea41abc7:::rbd_data.1133ebd6d3ba.0000000000000016:a [read 614400~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:30.915 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.207137+0000", 2024-09-21T15:49:30.915 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.671190132, 2024-09-21T15:49:30.915 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.027589632999999999, 2024-09-21T15:49:30.915 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:30.915 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:30.915 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:30.916 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:30.916 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:30.916 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28279 2024-09-21T15:49:30.916 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.916 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:30.916 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.916 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:30.916 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207137+0000", 2024-09-21T15:49:30.916 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:30.916 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.916 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.916 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:30.916 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207137+0000", 2024-09-21T15:49:30.916 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:30.917 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.917 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.917 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:30.917 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207137+0000", 2024-09-21T15:49:30.917 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:30.917 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.917 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.917 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:30.917 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207139+0000", 2024-09-21T15:49:30.917 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2639999999999998e-06 2024-09-21T15:49:30.917 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.917 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.917 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:30.917 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207145+0000", 2024-09-21T15:49:30.917 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6899999999999997e-06 2024-09-21T15:49:30.918 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.918 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.918 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:30.918 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207150+0000", 2024-09-21T15:49:30.918 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6959999999999996e-06 2024-09-21T15:49:30.918 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.918 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.918 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:30.918 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.234590+0000", 2024-09-21T15:49:30.918 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.027440042000000001 2024-09-21T15:49:30.918 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.918 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.918 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:30.918 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.234666+0000", 2024-09-21T15:49:30.919 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5867999999999999e-05 2024-09-21T15:49:30.919 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.919 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.919 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:30.919 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.234727+0000", 2024-09-21T15:49:30.919 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1447999999999998e-05 2024-09-21T15:49:30.919 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:30.919 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:30.919 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:30.919 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.919 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.919 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28280 3.17 3:ea41abc7:::rbd_data.1133ebd6d3ba.0000000000000016:a [read 720896~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:30.919 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.207154+0000", 2024-09-21T15:49:30.919 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.6711738270000001, 2024-09-21T15:49:30.920 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.027708854000000002, 2024-09-21T15:49:30.920 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:30.920 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:30.920 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:30.920 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:30.920 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:30.920 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28280 2024-09-21T15:49:30.920 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.920 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:30.920 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.920 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:30.920 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207154+0000", 2024-09-21T15:49:30.920 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:30.920 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.921 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.921 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:30.921 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207154+0000", 2024-09-21T15:49:30.921 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:30.921 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.921 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.921 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:30.921 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207153+0000", 2024-09-21T15:49:30.921 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:30.921 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.921 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.921 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:30.921 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207156+0000", 2024-09-21T15:49:30.921 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.057e-06 2024-09-21T15:49:30.922 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.922 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.922 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:30.922 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207161+0000", 2024-09-21T15:49:30.922 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5910000000000003e-06 2024-09-21T15:49:30.922 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.922 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.922 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:30.922 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207166+0000", 2024-09-21T15:49:30.922 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6020000000000002e-06 2024-09-21T15:49:30.922 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.922 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.922 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:30.923 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.234750+0000", 2024-09-21T15:49:30.923 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.027584132000000001 2024-09-21T15:49:30.923 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.923 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.923 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:30.923 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.234807+0000", 2024-09-21T15:49:30.923 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7482000000000003e-05 2024-09-21T15:49:30.923 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.923 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.923 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:30.923 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.234863+0000", 2024-09-21T15:49:30.923 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5284999999999997e-05 2024-09-21T15:49:30.923 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:30.923 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:30.924 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:30.924 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.924 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.924 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28281 3.17 3:ea41abc7:::rbd_data.1133ebd6d3ba.0000000000000016:a [read 835584~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:30.924 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.207170+0000", 2024-09-21T15:49:30.924 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.6711576940000001, 2024-09-21T15:49:30.924 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.027858447000000001, 2024-09-21T15:49:30.924 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:30.924 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:30.924 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:30.924 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:30.924 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:30.924 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28281 2024-09-21T15:49:30.924 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.925 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:30.925 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.925 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:30.925 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207170+0000", 2024-09-21T15:49:30.925 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:30.925 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.925 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.925 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:30.925 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207170+0000", 2024-09-21T15:49:30.925 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:30.925 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.925 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.925 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:30.926 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207170+0000", 2024-09-21T15:49:30.926 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:30.926 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.926 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.926 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:30.926 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207172+0000", 2024-09-21T15:49:30.926 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1610000000000001e-06 2024-09-21T15:49:30.926 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.926 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.926 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:30.926 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207177+0000", 2024-09-21T15:49:30.926 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.609e-06 2024-09-21T15:49:30.926 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.926 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.926 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:30.927 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207182+0000", 2024-09-21T15:49:30.927 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5360000000000003e-06 2024-09-21T15:49:30.927 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.927 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.927 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:30.927 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.234885+0000", 2024-09-21T15:49:30.927 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.027702926999999999 2024-09-21T15:49:30.927 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.927 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.927 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:30.927 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.234944+0000", 2024-09-21T15:49:30.927 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9604000000000001e-05 2024-09-21T15:49:30.927 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.927 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.928 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:30.928 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.235028+0000", 2024-09-21T15:49:30.928 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3982999999999999e-05 2024-09-21T15:49:30.928 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:30.928 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:30.928 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:30.928 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.928 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.928 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28282 3.17 3:ea41abc7:::rbd_data.1133ebd6d3ba.0000000000000016:a [read 950272~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:30.928 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.207186+0000", 2024-09-21T15:49:30.928 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.671141875, 2024-09-21T15:49:30.928 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.028448035999999999, 2024-09-21T15:49:30.928 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:30.928 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:30.929 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:30.929 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:30.929 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:30.929 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28282 2024-09-21T15:49:30.929 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.929 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:30.929 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.929 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:30.929 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207186+0000", 2024-09-21T15:49:30.929 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:30.929 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.929 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.929 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:30.929 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207186+0000", 2024-09-21T15:49:30.930 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:30.930 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.930 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.930 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:30.930 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207185+0000", 2024-09-21T15:49:30.930 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:30.930 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.930 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.930 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:30.930 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207187+0000", 2024-09-21T15:49:30.930 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9659999999999999e-06 2024-09-21T15:49:30.930 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.930 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.930 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:30.930 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207193+0000", 2024-09-21T15:49:30.930 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5539999999999997e-06 2024-09-21T15:49:30.931 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.931 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.931 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:30.931 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.207197+0000", 2024-09-21T15:49:30.931 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4889999999999997e-06 2024-09-21T15:49:30.931 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.931 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.931 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:30.931 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.235491+0000", 2024-09-21T15:49:30.931 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.028293490000000001 2024-09-21T15:49:30.931 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.931 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.931 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:30.931 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.235576+0000", 2024-09-21T15:49:30.931 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5609e-05 2024-09-21T15:49:30.932 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.932 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.932 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:30.932 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.235634+0000", 2024-09-21T15:49:30.932 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7306000000000003e-05 2024-09-21T15:49:30.932 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:30.932 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:30.932 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:30.932 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.932 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.932 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28355 3.1f 3:fea953f4:::rbd_data.1133ebd6d3ba.000000000000001b:a [read 942080~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:30.932 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.209224+0000", 2024-09-21T15:49:30.932 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.6691039679999999, 2024-09-21T15:49:30.932 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.027110213000000001, 2024-09-21T15:49:30.932 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:30.932 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:30.933 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:30.933 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:30.933 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:30.933 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28355 2024-09-21T15:49:30.933 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.933 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:30.933 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.933 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:30.933 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.209224+0000", 2024-09-21T15:49:30.933 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:30.933 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.933 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.933 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:30.933 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.209224+0000", 2024-09-21T15:49:30.933 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:30.933 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.933 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.934 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:30.934 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.209223+0000", 2024-09-21T15:49:30.934 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:30.934 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.934 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.934 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:30.934 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.209225+0000", 2024-09-21T15:49:30.934 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2469999999999998e-06 2024-09-21T15:49:30.934 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.934 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.934 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:30.934 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.209231+0000", 2024-09-21T15:49:30.934 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8919999999999997e-06 2024-09-21T15:49:30.934 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.934 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.935 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:30.935 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.209236+0000", 2024-09-21T15:49:30.935 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4789999999999998e-06 2024-09-21T15:49:30.935 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.935 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.935 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:30.935 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.236213+0000", 2024-09-21T15:49:30.935 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.026976834000000002 2024-09-21T15:49:30.935 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.935 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.935 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:30.935 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.236272+0000", 2024-09-21T15:49:30.935 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9284999999999999e-05 2024-09-21T15:49:30.935 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.935 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.936 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:30.936 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.236334+0000", 2024-09-21T15:49:30.936 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1878000000000006e-05 2024-09-21T15:49:30.936 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:30.936 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:30.936 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:30.936 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.936 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.936 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28472 3.1f 3:f84cae28:::rbd_data.1133ebd6d3ba.0000000000000006:a [read 393216~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:30.936 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.210514+0000", 2024-09-21T15:49:30.936 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.6678139220000001, 2024-09-21T15:49:30.936 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.027497165, 2024-09-21T15:49:30.936 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:30.936 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:30.936 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:30.936 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:30.937 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:30.937 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28472 2024-09-21T15:49:30.937 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.937 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:30.937 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.937 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:30.937 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.210514+0000", 2024-09-21T15:49:30.937 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:30.937 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.937 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.937 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:30.937 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.210514+0000", 2024-09-21T15:49:30.937 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:30.937 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.937 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.937 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:30.938 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.210513+0000", 2024-09-21T15:49:30.938 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:30.938 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.938 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.938 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:30.938 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.210518+0000", 2024-09-21T15:49:30.938 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8690000000000003e-06 2024-09-21T15:49:30.938 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.938 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.938 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:30.938 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.210525+0000", 2024-09-21T15:49:30.938 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.126e-06 2024-09-21T15:49:30.938 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.938 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.938 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:30.939 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.210531+0000", 2024-09-21T15:49:30.939 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6389999999999997e-06 2024-09-21T15:49:30.939 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.939 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.939 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:30.939 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.237372+0000", 2024-09-21T15:49:30.939 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.026840955 2024-09-21T15:49:30.939 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.939 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.939 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:30.939 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.237435+0000", 2024-09-21T15:49:30.939 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2942999999999997e-05 2024-09-21T15:49:30.939 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.939 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.939 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:30.939 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.238011+0000", 2024-09-21T15:49:30.940 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00057616700000000004 2024-09-21T15:49:30.940 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:30.940 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:30.940 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:30.940 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.940 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.940 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28473 3.1f 3:f84cae28:::rbd_data.1133ebd6d3ba.0000000000000006:a [read 499712~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:30.940 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.210536+0000", 2024-09-21T15:49:30.940 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.667791775, 2024-09-21T15:49:30.940 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.02778216, 2024-09-21T15:49:30.940 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:30.940 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:30.940 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:30.940 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:30.940 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:30.940 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28473 2024-09-21T15:49:30.941 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.941 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:30.941 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.941 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:30.941 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.210536+0000", 2024-09-21T15:49:30.941 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:30.941 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.941 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.941 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:30.941 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.210536+0000", 2024-09-21T15:49:30.941 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:30.941 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.941 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.941 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:30.941 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.210535+0000", 2024-09-21T15:49:30.941 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:30.941 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.942 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.942 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:30.942 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.210538+0000", 2024-09-21T15:49:30.942 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4980000000000001e-06 2024-09-21T15:49:30.942 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.942 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.942 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:30.942 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.210544+0000", 2024-09-21T15:49:30.942 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0719999999999996e-06 2024-09-21T15:49:30.942 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.942 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.942 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:30.942 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.210549+0000", 2024-09-21T15:49:30.942 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.977e-06 2024-09-21T15:49:30.942 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.943 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.943 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:30.943 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.238037+0000", 2024-09-21T15:49:30.943 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.027487864000000001 2024-09-21T15:49:30.943 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.943 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.943 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:30.943 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.238127+0000", 2024-09-21T15:49:30.943 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0555000000000003e-05 2024-09-21T15:49:30.943 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.943 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.943 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:30.943 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.238318+0000", 2024-09-21T15:49:30.943 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00019061800000000001 2024-09-21T15:49:30.943 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:30.943 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:30.944 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:30.944 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.944 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.944 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28475 3.1f 3:f84cae28:::rbd_data.1133ebd6d3ba.0000000000000006:a [read 614400~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:30.944 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.210918+0000", 2024-09-21T15:49:30.944 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.6674094159999999, 2024-09-21T15:49:30.944 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.027783961999999999, 2024-09-21T15:49:30.944 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:30.944 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:30.944 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:30.944 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:30.944 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:30.944 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28475 2024-09-21T15:49:30.944 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.944 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:30.944 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.945 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:30.945 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.210918+0000", 2024-09-21T15:49:30.945 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:30.945 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.945 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.945 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:30.945 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.210918+0000", 2024-09-21T15:49:30.945 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:30.945 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.945 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.945 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:30.945 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.210917+0000", 2024-09-21T15:49:30.945 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:30.945 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.945 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.945 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:30.945 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.210923+0000", 2024-09-21T15:49:30.946 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2680000000000002e-06 2024-09-21T15:49:30.946 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.946 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.946 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:30.946 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.210934+0000", 2024-09-21T15:49:30.946 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1246e-05 2024-09-21T15:49:30.946 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.946 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.946 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:30.946 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.210941+0000", 2024-09-21T15:49:30.946 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3660000000000003e-06 2024-09-21T15:49:30.946 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.946 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.946 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:30.946 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.238344+0000", 2024-09-21T15:49:30.947 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.027402603000000001 2024-09-21T15:49:30.947 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.947 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.947 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:30.947 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.238401+0000", 2024-09-21T15:49:30.947 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7077000000000003e-05 2024-09-21T15:49:30.947 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.947 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.947 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:30.947 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.238702+0000", 2024-09-21T15:49:30.947 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000301265 2024-09-21T15:49:30.947 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:30.947 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:30.947 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:30.947 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.948 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.948 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28476 3.1f 3:f84cae28:::rbd_data.1133ebd6d3ba.0000000000000006:a [read 729088~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:30.948 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.210947+0000", 2024-09-21T15:49:30.948 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.6673801109999999, 2024-09-21T15:49:30.948 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.028242356999999999, 2024-09-21T15:49:30.948 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:30.948 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:30.948 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:30.948 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:30.948 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:30.948 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28476 2024-09-21T15:49:30.948 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.948 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:30.948 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.948 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:30.948 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.210947+0000", 2024-09-21T15:49:30.948 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:30.949 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.949 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.949 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:30.949 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.210947+0000", 2024-09-21T15:49:30.949 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:30.949 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.949 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.949 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:30.949 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.210947+0000", 2024-09-21T15:49:30.949 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:30.949 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.949 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.949 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:30.949 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.210950+0000", 2024-09-21T15:49:30.949 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6180000000000002e-06 2024-09-21T15:49:30.949 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.949 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.950 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:30.950 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.210956+0000", 2024-09-21T15:49:30.950 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3810000000000004e-06 2024-09-21T15:49:30.950 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.950 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.950 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:30.950 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.210961+0000", 2024-09-21T15:49:30.950 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1320000000000002e-06 2024-09-21T15:49:30.950 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.950 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.950 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:30.950 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.238730+0000", 2024-09-21T15:49:30.950 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.027768352 2024-09-21T15:49:30.950 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.950 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.951 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:30.951 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.238800+0000", 2024-09-21T15:49:30.951 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0837e-05 2024-09-21T15:49:30.951 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.951 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.951 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:30.951 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.239190+0000", 2024-09-21T15:49:30.951 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038950399999999999 2024-09-21T15:49:30.951 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:30.951 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:30.951 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:30.951 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.951 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.951 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28478 3.1f 3:f84cae28:::rbd_data.1133ebd6d3ba.0000000000000006:a [read 835584~16384] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:30.951 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.211131+0000", 2024-09-21T15:49:30.952 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.6671969239999997, 2024-09-21T15:49:30.952 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.028450188000000001, 2024-09-21T15:49:30.952 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:30.952 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:30.952 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:30.952 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:30.952 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:30.952 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28478 2024-09-21T15:49:30.952 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.952 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:30.952 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.952 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:30.952 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.211131+0000", 2024-09-21T15:49:30.952 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:30.952 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.952 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.952 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:30.953 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.211131+0000", 2024-09-21T15:49:30.953 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:30.953 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.953 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.953 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:30.953 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.211130+0000", 2024-09-21T15:49:30.953 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:30.953 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.953 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.953 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:30.953 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.211134+0000", 2024-09-21T15:49:30.953 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.369e-06 2024-09-21T15:49:30.953 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.953 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.953 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:30.953 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.211145+0000", 2024-09-21T15:49:30.954 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0552e-05 2024-09-21T15:49:30.954 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.954 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.954 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:30.954 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.211152+0000", 2024-09-21T15:49:30.954 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9859999999999997e-06 2024-09-21T15:49:30.954 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.954 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.954 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:30.954 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.239220+0000", 2024-09-21T15:49:30.954 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.028068629000000001 2024-09-21T15:49:30.954 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.954 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.955 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:30.955 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.239298+0000", 2024-09-21T15:49:30.955 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7890000000000001e-05 2024-09-21T15:49:30.955 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.955 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.955 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:30.955 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.239581+0000", 2024-09-21T15:49:30.955 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000282501 2024-09-21T15:49:30.955 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:30.955 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:30.955 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:30.955 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.955 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.955 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28480 3.1f 3:f84cae28:::rbd_data.1133ebd6d3ba.0000000000000006:a [read 950272~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:30.955 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.211158+0000", 2024-09-21T15:49:30.956 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.6671698720000001, 2024-09-21T15:49:30.956 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.028974907000000001, 2024-09-21T15:49:30.956 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:30.956 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:30.956 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:30.956 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:30.956 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:30.956 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28480 2024-09-21T15:49:30.956 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.956 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:30.956 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.956 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:30.956 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.211158+0000", 2024-09-21T15:49:30.956 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:30.956 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.957 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.957 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:30.957 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.211158+0000", 2024-09-21T15:49:30.957 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:30.957 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.957 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.957 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:30.957 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.211157+0000", 2024-09-21T15:49:30.957 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:30.957 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.957 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.957 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:30.957 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.211160+0000", 2024-09-21T15:49:30.957 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5100000000000001e-06 2024-09-21T15:49:30.957 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.958 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.958 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:30.958 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.211167+0000", 2024-09-21T15:49:30.958 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7399999999999998e-06 2024-09-21T15:49:30.958 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.958 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.958 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:30.958 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.211172+0000", 2024-09-21T15:49:30.958 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1839999999999998e-06 2024-09-21T15:49:30.958 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.958 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.958 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:30.958 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.239782+0000", 2024-09-21T15:49:30.958 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.028610588999999999 2024-09-21T15:49:30.959 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.959 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.959 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:30.959 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.239858+0000", 2024-09-21T15:49:30.959 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5753000000000005e-05 2024-09-21T15:49:30.959 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.959 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.959 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:30.959 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.240133+0000", 2024-09-21T15:49:30.959 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00027458300000000001 2024-09-21T15:49:30.959 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:30.959 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:30.959 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:30.959 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.959 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.960 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:30371 3.1 3:83065594:::rbd_data.1133957d62b2.000000000000002b:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:30.960 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.776483+0000", 2024-09-21T15:49:30.960 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.1018447409999998, 2024-09-21T15:49:30.960 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.029859434000000001, 2024-09-21T15:49:30.960 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:30.960 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:30.960 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:30.960 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:30.960 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:30.960 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 30371 2024-09-21T15:49:30.960 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.960 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:30.960 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.960 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:30.961 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.776483+0000", 2024-09-21T15:49:30.961 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:30.961 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.961 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.961 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:30.961 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.776483+0000", 2024-09-21T15:49:30.961 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:30.961 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.961 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.961 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:30.961 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.776481+0000", 2024-09-21T15:49:30.961 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:49:30.961 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.961 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.961 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:30.962 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.776491+0000", 2024-09-21T15:49:30.962 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7559999999999994e-06 2024-09-21T15:49:30.962 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.962 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.962 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:30.962 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.776508+0000", 2024-09-21T15:49:30.962 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7592999999999999e-05 2024-09-21T15:49:30.962 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.962 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.962 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:30.962 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.776518+0000", 2024-09-21T15:49:30.962 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0093e-05 2024-09-21T15:49:30.962 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.963 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.963 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:30.963 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.776570+0000", 2024-09-21T15:49:30.963 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1140000000000002e-05 2024-09-21T15:49:30.963 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.963 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.963 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:30.963 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.776662+0000", 2024-09-21T15:49:30.963 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2278e-05 2024-09-21T15:49:30.963 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.963 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.963 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-09-21T15:49:30.963 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.782965+0000", 2024-09-21T15:49:30.963 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0063035060000000004 2024-09-21T15:49:30.963 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.963 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.964 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:30.964 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.806269+0000", 2024-09-21T15:49:30.964 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.023303733 2024-09-21T15:49:30.964 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.964 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.964 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:30.964 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.806278+0000", 2024-09-21T15:49:30.964 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8179999999999993e-06 2024-09-21T15:49:30.964 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.964 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.964 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:30.964 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.806314+0000", 2024-09-21T15:49:30.964 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6155000000000002e-05 2024-09-21T15:49:30.964 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:30.964 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:30.964 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:30.965 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.806342+0000", 2024-09-21T15:49:30.965 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8252999999999999e-05 2024-09-21T15:49:30.965 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:30.965 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:30.965 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:30.965 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:30.965 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:30.965 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:30.966 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:49:30.967 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:30.965+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:31.059 INFO:teuthology.orchestra.run.smithi089.stdout:checking clone #8, image image_client.0-clone8 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent9 2024-09-21T15:49:31.067 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:31.066+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:31.082 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:31.082 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:49:31.082 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:49:31.082 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:31.082 INFO:teuthology.orchestra.run.smithi089.stdout:2099 trunc from 0xa7255fa to 0x887a523 2024-09-21T15:49:31.091 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:49:31.105 INFO:teuthology.orchestra.run.smithi089.stdout:2100 read 0x45c9524 thru 0x45d49d4 (0xb4b1 bytes) 2024-09-21T15:49:31.105 INFO:teuthology.orchestra.run.smithi089.stdout:2101 read 0x1851688 thru 0x18578a9 (0x6222 bytes) 2024-09-21T15:49:31.111 INFO:teuthology.orchestra.run.smithi089.stdout:2102 trunc from 0x887a523 to 0x4144af2 2024-09-21T15:49:31.146 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:31.146 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:31.146 INFO:teuthology.orchestra.run.smithi089.stdout:2105 punch from 0x144e936 to 0x145b2b5, (0xc97f bytes) 2024-09-21T15:49:31.146 INFO:teuthology.orchestra.run.smithi089.stdout:2106 read 0x1694fef thru 0x169d345 (0x8357 bytes) 2024-09-21T15:49:31.151 INFO:teuthology.orchestra.run.smithi089.stdout:2107 trunc from 0x4144af2 to 0x8b0a7c2 2024-09-21T15:49:31.156 INFO:teuthology.orchestra.run.smithi089.stdout:2108 write 0x1db1c82 thru 0x1db3ca6 (0x2025 bytes) 2024-09-21T15:49:31.160 INFO:teuthology.orchestra.run.smithi089.stdout:2110 write 0x203c1ff thru 0x2048016 (0xbe18 bytes) 2024-09-21T15:49:31.168 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:31.167+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:31.169 INFO:teuthology.orchestra.run.smithi089.stdout:2111 clone 11 order 18 su 2048 sc 4 2024-09-21T15:49:31.214 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:31.214 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:49:31.214 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:49:31.214 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:49:31.214 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:31.222 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:49:31.306 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:31.305+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:31.365 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:31.366 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:49:31.366 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:49:31.366 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:49:31.366 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.366 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:13420 3.0 3:038b8840:::rbd_data.1133dc5adfdf.0000000000000035:head [set-alloc-hint object_size 1048576 write_size 1048576,write 245760~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:31.366 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.208804+0000", 2024-09-21T15:49:31.366 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.154786175, 2024-09-21T15:49:31.366 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0049290649999999998, 2024-09-21T15:49:31.366 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:31.366 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:31.366 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:31.366 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:31.366 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:31.367 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 13420 2024-09-21T15:49:31.367 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.367 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:31.367 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.367 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:31.367 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.208804+0000", 2024-09-21T15:49:31.367 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.367 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.367 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.367 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:31.367 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.208804+0000", 2024-09-21T15:49:31.367 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.367 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.367 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.367 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:31.368 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.208803+0000", 2024-09-21T15:49:31.368 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:31.368 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.368 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.368 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:31.368 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.208813+0000", 2024-09-21T15:49:31.368 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9639999999999998e-06 2024-09-21T15:49:31.368 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.368 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.368 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:31.368 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.208827+0000", 2024-09-21T15:49:31.368 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3952e-05 2024-09-21T15:49:31.368 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.368 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.368 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:31.368 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.208835+0000", 2024-09-21T15:49:31.368 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3680000000000002e-06 2024-09-21T15:49:31.369 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.369 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.369 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:31.369 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.208885+0000", 2024-09-21T15:49:31.369 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9721e-05 2024-09-21T15:49:31.369 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.369 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.369 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:31.369 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209017+0000", 2024-09-21T15:49:31.369 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013215500000000001 2024-09-21T15:49:31.369 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.369 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.369 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-09-21T15:49:31.369 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209152+0000", 2024-09-21T15:49:31.369 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013543100000000001 2024-09-21T15:49:31.370 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.370 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.370 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:31.370 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.213629+0000", 2024-09-21T15:49:31.370 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0044762259999999998 2024-09-21T15:49:31.370 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.370 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.370 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:31.370 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.213649+0000", 2024-09-21T15:49:31.370 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0415e-05 2024-09-21T15:49:31.370 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.370 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.370 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:31.370 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.213698+0000", 2024-09-21T15:49:31.370 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8619000000000003e-05 2024-09-21T15:49:31.370 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.371 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.371 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:31.371 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.213733+0000", 2024-09-21T15:49:31.371 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5312999999999999e-05 2024-09-21T15:49:31.371 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.371 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:31.371 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.371 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.371 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.371 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:13421 3.19 3:9a44caf3:::rbd_data.1133dc5adfdf.0000000000000036:head [set-alloc-hint object_size 1048576 write_size 1048576,write 245760~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:31.371 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.208998+0000", 2024-09-21T15:49:31.371 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.154592503, 2024-09-21T15:49:31.371 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0048896720000000003, 2024-09-21T15:49:31.371 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:31.371 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:31.371 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:31.372 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:31.372 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:31.372 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 13421 2024-09-21T15:49:31.372 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.372 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:31.372 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.372 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:31.372 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.208998+0000", 2024-09-21T15:49:31.372 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.372 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.372 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.372 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:31.372 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.208998+0000", 2024-09-21T15:49:31.372 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.372 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.372 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.373 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:31.373 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.208997+0000", 2024-09-21T15:49:31.373 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:31.373 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.373 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.373 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:31.373 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209006+0000", 2024-09-21T15:49:31.373 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4329999999999993e-06 2024-09-21T15:49:31.373 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.373 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.373 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:31.373 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209014+0000", 2024-09-21T15:49:31.373 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0339999999999997e-06 2024-09-21T15:49:31.373 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.373 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.373 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:31.374 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209019+0000", 2024-09-21T15:49:31.374 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3839999999999999e-06 2024-09-21T15:49:31.374 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.374 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.374 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:31.374 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209058+0000", 2024-09-21T15:49:31.374 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8798999999999999e-05 2024-09-21T15:49:31.374 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.374 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.374 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:31.374 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209143+0000", 2024-09-21T15:49:31.374 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5075999999999997e-05 2024-09-21T15:49:31.374 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.374 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.374 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:49:31.374 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209254+0000", 2024-09-21T15:49:31.375 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000111816 2024-09-21T15:49:31.375 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.375 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.375 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:31.375 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.210789+0000", 2024-09-21T15:49:31.375 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0015348969999999999 2024-09-21T15:49:31.375 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.375 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.375 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:31.375 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.213809+0000", 2024-09-21T15:49:31.375 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0030192069999999999 2024-09-21T15:49:31.375 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.375 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.375 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:31.375 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.213853+0000", 2024-09-21T15:49:31.375 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4023999999999997e-05 2024-09-21T15:49:31.376 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.376 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.376 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:31.376 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.213887+0000", 2024-09-21T15:49:31.376 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4650000000000002e-05 2024-09-21T15:49:31.376 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.376 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:31.376 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.376 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.376 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.376 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:13445 3.0 3:038b8840:::rbd_data.1133dc5adfdf.0000000000000035:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:31.376 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.232171+0000", 2024-09-21T15:49:31.376 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.131418654000001, 2024-09-21T15:49:31.376 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.005214364, 2024-09-21T15:49:31.376 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:31.376 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:31.377 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:31.377 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:31.377 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:31.377 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 13445 2024-09-21T15:49:31.377 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.377 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:31.377 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.377 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:31.377 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232171+0000", 2024-09-21T15:49:31.377 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.377 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.377 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.377 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:31.377 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232171+0000", 2024-09-21T15:49:31.377 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.377 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.378 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.378 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:31.378 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232171+0000", 2024-09-21T15:49:31.378 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:31.378 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.378 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.378 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:31.378 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232175+0000", 2024-09-21T15:49:31.378 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7689999999999998e-06 2024-09-21T15:49:31.378 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.378 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.378 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:31.378 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232182+0000", 2024-09-21T15:49:31.378 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.712e-06 2024-09-21T15:49:31.378 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.378 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.379 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:31.379 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232188+0000", 2024-09-21T15:49:31.379 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0800000000000002e-06 2024-09-21T15:49:31.379 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.379 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.379 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:31.379 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232257+0000", 2024-09-21T15:49:31.379 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8267000000000002e-05 2024-09-21T15:49:31.379 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.379 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.379 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:31.379 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232337+0000", 2024-09-21T15:49:31.379 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0422999999999999e-05 2024-09-21T15:49:31.379 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.380 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.380 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-09-21T15:49:31.380 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232421+0000", 2024-09-21T15:49:31.380 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3492000000000006e-05 2024-09-21T15:49:31.380 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.380 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.380 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:31.380 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.233588+0000", 2024-09-21T15:49:31.380 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001167707 2024-09-21T15:49:31.380 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.380 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.380 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:31.380 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237298+0000", 2024-09-21T15:49:31.380 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0037098309999999998 2024-09-21T15:49:31.380 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.380 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.381 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:31.381 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237349+0000", 2024-09-21T15:49:31.381 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1102e-05 2024-09-21T15:49:31.381 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.381 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.381 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:31.381 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237386+0000", 2024-09-21T15:49:31.381 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6551999999999998e-05 2024-09-21T15:49:31.381 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.381 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:31.381 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.381 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.381 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.381 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13447 3.8 e31/22 3:12a4f7ad:::rbd_data.1133dc5adfdf.0000000000000037:head v 31'34, mlcod=31'32)", 2024-09-21T15:49:31.381 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.234176+0000", 2024-09-21T15:49:31.382 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.129414134999999, 2024-09-21T15:49:31.382 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0091398529999999999, 2024-09-21T15:49:31.382 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:31.382 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:31.382 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:31.382 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.382 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:31.382 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.234176+0000", 2024-09-21T15:49:31.382 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.382 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.382 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.382 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:31.382 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.234176+0000", 2024-09-21T15:49:31.382 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.382 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.382 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.383 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:31.383 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.234175+0000", 2024-09-21T15:49:31.383 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:31.383 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.383 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.383 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:31.383 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.234181+0000", 2024-09-21T15:49:31.383 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2929999999999997e-06 2024-09-21T15:49:31.383 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.383 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.383 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:31.383 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.234195+0000", 2024-09-21T15:49:31.383 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4034e-05 2024-09-21T15:49:31.383 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.383 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.383 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:31.383 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.234202+0000", 2024-09-21T15:49:31.384 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7730000000000002e-06 2024-09-21T15:49:31.384 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.384 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.384 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:31.384 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.234241+0000", 2024-09-21T15:49:31.384 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8590999999999997e-05 2024-09-21T15:49:31.384 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.384 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.384 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:31.384 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.234262+0000", 2024-09-21T15:49:31.384 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0716999999999998e-05 2024-09-21T15:49:31.384 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.384 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.384 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:31.384 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.243281+0000", 2024-09-21T15:49:31.384 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0090193510000000001 2024-09-21T15:49:31.385 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.385 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.385 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:31.385 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.243316+0000", 2024-09-21T15:49:31.385 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4901999999999999e-05 2024-09-21T15:49:31.385 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.385 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:31.385 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.385 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.385 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.385 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13449 3.8 e31/22 3:13437810:::rbd_data.1133dc5adfdf.0000000000000039:head v 31'35, mlcod=31'32)", 2024-09-21T15:49:31.385 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.234889+0000", 2024-09-21T15:49:31.385 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.128701314000001, 2024-09-21T15:49:31.385 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.008441558, 2024-09-21T15:49:31.385 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:31.385 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:31.386 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:31.386 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.386 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:31.386 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.234889+0000", 2024-09-21T15:49:31.386 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.386 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.386 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.386 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:31.386 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.234889+0000", 2024-09-21T15:49:31.386 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.386 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.386 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.386 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:31.386 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.234888+0000", 2024-09-21T15:49:31.386 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:31.387 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.387 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.387 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:31.387 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.234894+0000", 2024-09-21T15:49:31.387 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1680000000000001e-06 2024-09-21T15:49:31.387 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.387 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.387 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:31.387 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.234907+0000", 2024-09-21T15:49:31.387 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3241000000000001e-05 2024-09-21T15:49:31.387 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.387 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.387 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:31.387 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.234914+0000", 2024-09-21T15:49:31.387 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6699999999999997e-06 2024-09-21T15:49:31.387 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.388 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.388 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:31.388 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.234948+0000", 2024-09-21T15:49:31.388 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4359000000000001e-05 2024-09-21T15:49:31.388 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.388 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.388 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:31.388 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.234966+0000", 2024-09-21T15:49:31.388 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8025000000000001e-05 2024-09-21T15:49:31.388 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.388 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.388 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:31.388 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.243319+0000", 2024-09-21T15:49:31.388 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0083525260000000007 2024-09-21T15:49:31.388 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.388 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.389 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:31.389 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.243330+0000", 2024-09-21T15:49:31.389 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1319e-05 2024-09-21T15:49:31.389 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.389 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:31.389 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.389 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.389 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.389 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13456 3.8 e31/22 3:154c00a4:::rbd_data.1133dc5adfdf.000000000000007e:head v 31'36, mlcod=31'32)", 2024-09-21T15:49:31.389 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.237960+0000", 2024-09-21T15:49:31.389 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.125630369, 2024-09-21T15:49:31.389 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0078875109999999998, 2024-09-21T15:49:31.389 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:31.389 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:31.389 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:31.389 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.389 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:31.390 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237960+0000", 2024-09-21T15:49:31.390 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.390 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.390 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.390 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:31.390 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237960+0000", 2024-09-21T15:49:31.390 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.390 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.390 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.390 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:31.390 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237959+0000", 2024-09-21T15:49:31.390 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:31.390 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.390 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.390 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:31.391 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237965+0000", 2024-09-21T15:49:31.391 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3539999999999996e-06 2024-09-21T15:49:31.391 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.391 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.391 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:31.391 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237980+0000", 2024-09-21T15:49:31.391 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.449e-05 2024-09-21T15:49:31.391 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.391 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.391 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:31.391 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237994+0000", 2024-09-21T15:49:31.391 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3910000000000001e-05 2024-09-21T15:49:31.391 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.391 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.391 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:31.392 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.238167+0000", 2024-09-21T15:49:31.392 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000172986 2024-09-21T15:49:31.392 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.392 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.392 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:31.392 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.238189+0000", 2024-09-21T15:49:31.392 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2240000000000001e-05 2024-09-21T15:49:31.392 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.392 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.392 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:31.392 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.245821+0000", 2024-09-21T15:49:31.392 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0076315799999999998 2024-09-21T15:49:31.392 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.392 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.393 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:31.393 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.245847+0000", 2024-09-21T15:49:31.393 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6614999999999998e-05 2024-09-21T15:49:31.393 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.393 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:31.393 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.393 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.393 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.393 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:13839 3.0 3:03c4fb32:::rbd_data.1133dc5adfdf.0000000000000099:head [write 892928~4096] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e31)", 2024-09-21T15:49:31.393 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.527235+0000", 2024-09-21T15:49:31.393 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.836354761999999, 2024-09-21T15:49:31.393 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0067577130000000003, 2024-09-21T15:49:31.393 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:31.393 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:31.393 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:31.393 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:31.393 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:31.394 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 13839 2024-09-21T15:49:31.394 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.394 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:31.394 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.394 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:31.394 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527235+0000", 2024-09-21T15:49:31.394 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.394 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.394 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.394 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:31.394 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527235+0000", 2024-09-21T15:49:31.394 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.394 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.394 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.394 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:31.395 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527234+0000", 2024-09-21T15:49:31.395 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:31.395 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.395 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.395 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:31.395 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527245+0000", 2024-09-21T15:49:31.395 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0597e-05 2024-09-21T15:49:31.395 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.395 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.395 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:31.395 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527266+0000", 2024-09-21T15:49:31.395 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1095e-05 2024-09-21T15:49:31.395 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.395 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.395 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:31.395 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527275+0000", 2024-09-21T15:49:31.395 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9080000000000001e-06 2024-09-21T15:49:31.396 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.396 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.396 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:31.396 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527816+0000", 2024-09-21T15:49:31.396 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00054145400000000002 2024-09-21T15:49:31.396 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.396 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.396 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:31.396 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527891+0000", 2024-09-21T15:49:31.396 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4898e-05 2024-09-21T15:49:31.396 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.396 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.396 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-09-21T15:49:31.396 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528024+0000", 2024-09-21T15:49:31.396 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000133242 2024-09-21T15:49:31.397 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.397 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.397 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:31.397 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.531270+0000", 2024-09-21T15:49:31.397 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003245469 2024-09-21T15:49:31.397 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.397 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.397 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:31.397 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.533926+0000", 2024-09-21T15:49:31.397 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0026558559999999998 2024-09-21T15:49:31.397 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.397 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.397 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:31.397 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.533953+0000", 2024-09-21T15:49:31.397 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7551000000000001e-05 2024-09-21T15:49:31.398 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.398 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.398 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:31.398 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.533993+0000", 2024-09-21T15:49:31.398 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9782e-05 2024-09-21T15:49:31.398 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.398 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:31.398 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.398 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.398 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.398 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:14393 3.6 e33/22 3:640f2ea6:::rbd_data.1133dc5adfdf.000000000000002a:head v 33'48, mlcod=33'46)", 2024-09-21T15:49:31.398 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:18.963289+0000", 2024-09-21T15:49:31.398 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.400300888, 2024-09-21T15:49:31.398 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0057897479999999999, 2024-09-21T15:49:31.398 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:31.398 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:31.399 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:31.399 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.399 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:31.399 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.963289+0000", 2024-09-21T15:49:31.399 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.399 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.399 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.399 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:31.399 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.963289+0000", 2024-09-21T15:49:31.399 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.399 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.399 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.399 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:31.399 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.963288+0000", 2024-09-21T15:49:31.399 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:31.400 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.400 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.400 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:31.400 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.963297+0000", 2024-09-21T15:49:31.400 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2180000000000005e-06 2024-09-21T15:49:31.400 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.400 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.400 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:31.400 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.963312+0000", 2024-09-21T15:49:31.400 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4414e-05 2024-09-21T15:49:31.400 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.400 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.400 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:31.400 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.963318+0000", 2024-09-21T15:49:31.400 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5590000000000002e-06 2024-09-21T15:49:31.400 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.401 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.401 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:31.401 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.963361+0000", 2024-09-21T15:49:31.401 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3108000000000001e-05 2024-09-21T15:49:31.401 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.401 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.401 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:31.401 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.963382+0000", 2024-09-21T15:49:31.401 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0601000000000001e-05 2024-09-21T15:49:31.401 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.401 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.401 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:31.401 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969043+0000", 2024-09-21T15:49:31.401 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0056611889999999996 2024-09-21T15:49:31.401 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.402 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.402 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:31.402 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969079+0000", 2024-09-21T15:49:31.402 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5731000000000001e-05 2024-09-21T15:49:31.402 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.402 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:31.402 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.402 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.402 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.402 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:14403 3.19 3:9dbcd9a3:::rbd_data.1133dc5adfdf.0000000000000034:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-21T15:49:31.402 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:18.969363+0000", 2024-09-21T15:49:31.402 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.394227036, 2024-09-21T15:49:31.402 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0059848089999999998, 2024-09-21T15:49:31.402 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:31.402 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:31.402 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:31.403 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:31.403 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:31.403 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 14403 2024-09-21T15:49:31.403 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.403 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:31.403 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.403 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:31.403 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969363+0000", 2024-09-21T15:49:31.403 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.403 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.403 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.403 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:31.403 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969363+0000", 2024-09-21T15:49:31.403 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.403 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.403 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.404 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:31.404 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969362+0000", 2024-09-21T15:49:31.404 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:31.404 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.404 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.404 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:31.404 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969367+0000", 2024-09-21T15:49:31.404 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9400000000000001e-06 2024-09-21T15:49:31.404 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.404 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.404 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:31.404 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969378+0000", 2024-09-21T15:49:31.404 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1126e-05 2024-09-21T15:49:31.404 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.404 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.404 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:31.405 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969383+0000", 2024-09-21T15:49:31.405 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1290000000000004e-06 2024-09-21T15:49:31.405 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.405 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.405 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:31.405 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969420+0000", 2024-09-21T15:49:31.405 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.68e-05 2024-09-21T15:49:31.405 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.405 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.405 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:31.405 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969487+0000", 2024-09-21T15:49:31.405 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7211000000000004e-05 2024-09-21T15:49:31.405 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.405 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.405 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:49:31.405 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969543+0000", 2024-09-21T15:49:31.406 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5890999999999997e-05 2024-09-21T15:49:31.406 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.406 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.406 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:31.406 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.972060+0000", 2024-09-21T15:49:31.406 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002516379 2024-09-21T15:49:31.406 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.406 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.406 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:31.406 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.975273+0000", 2024-09-21T15:49:31.406 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032138560000000002 2024-09-21T15:49:31.406 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.406 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.406 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:31.406 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.975315+0000", 2024-09-21T15:49:31.407 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1795999999999998e-05 2024-09-21T15:49:31.407 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.407 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.407 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:31.407 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.975348+0000", 2024-09-21T15:49:31.407 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2666999999999997e-05 2024-09-21T15:49:31.407 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.407 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:31.407 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.407 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.407 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.407 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:14404 3.0 3:038b8840:::rbd_data.1133dc5adfdf.0000000000000035:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-21T15:49:31.407 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:18.969391+0000", 2024-09-21T15:49:31.407 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.394199473, 2024-09-21T15:49:31.407 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0057367420000000004, 2024-09-21T15:49:31.408 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:31.408 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:31.408 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:31.408 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:31.408 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:31.408 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 14404 2024-09-21T15:49:31.408 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.408 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:31.408 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.408 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:31.408 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969391+0000", 2024-09-21T15:49:31.408 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.408 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.408 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.409 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:31.409 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969391+0000", 2024-09-21T15:49:31.409 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.409 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.409 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.409 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:31.409 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969390+0000", 2024-09-21T15:49:31.409 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:31.409 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.409 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.409 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:31.409 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969392+0000", 2024-09-21T15:49:31.409 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.827e-06 2024-09-21T15:49:31.409 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.409 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.410 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:31.410 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969397+0000", 2024-09-21T15:49:31.410 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8799999999999999e-06 2024-09-21T15:49:31.410 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.410 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.410 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:31.410 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969400+0000", 2024-09-21T15:49:31.410 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.004e-06 2024-09-21T15:49:31.410 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.410 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.410 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:31.410 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969446+0000", 2024-09-21T15:49:31.410 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6446999999999998e-05 2024-09-21T15:49:31.410 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.410 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.411 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:31.411 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969522+0000", 2024-09-21T15:49:31.411 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5419000000000004e-05 2024-09-21T15:49:31.411 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.411 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.411 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-09-21T15:49:31.411 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969626+0000", 2024-09-21T15:49:31.411 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000103898 2024-09-21T15:49:31.411 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.411 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.411 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:31.411 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.972220+0000", 2024-09-21T15:49:31.411 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0025942169999999998 2024-09-21T15:49:31.411 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.411 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.411 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:31.412 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.975030+0000", 2024-09-21T15:49:31.412 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0028103469999999999 2024-09-21T15:49:31.412 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.412 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.412 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:31.412 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.975093+0000", 2024-09-21T15:49:31.412 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2744999999999996e-05 2024-09-21T15:49:31.412 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.412 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.412 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:31.412 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.975127+0000", 2024-09-21T15:49:31.412 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4248000000000001e-05 2024-09-21T15:49:31.412 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.412 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:31.412 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.412 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.413 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.413 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:14405 3.19 3:9a44caf3:::rbd_data.1133dc5adfdf.0000000000000036:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e33)", 2024-09-21T15:49:31.413 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:18.969509+0000", 2024-09-21T15:49:31.413 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.394080927999999, 2024-09-21T15:49:31.413 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0059130320000000004, 2024-09-21T15:49:31.413 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:31.413 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:31.413 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:31.413 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:31.413 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:31.413 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 14405 2024-09-21T15:49:31.413 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.413 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:31.413 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.413 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:31.414 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969509+0000", 2024-09-21T15:49:31.414 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.414 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.414 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.414 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:31.414 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969509+0000", 2024-09-21T15:49:31.414 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.414 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.414 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.414 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:31.414 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969509+0000", 2024-09-21T15:49:31.414 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:31.414 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.414 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.414 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:31.415 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969512+0000", 2024-09-21T15:49:31.415 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1429999999999998e-06 2024-09-21T15:49:31.415 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.415 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.415 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:31.415 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969518+0000", 2024-09-21T15:49:31.415 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.268e-06 2024-09-21T15:49:31.415 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.415 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.415 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:31.415 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969522+0000", 2024-09-21T15:49:31.415 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4499999999999997e-06 2024-09-21T15:49:31.415 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.415 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.415 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:31.416 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969816+0000", 2024-09-21T15:49:31.416 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00029325400000000001 2024-09-21T15:49:31.416 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.416 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.416 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:31.416 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969871+0000", 2024-09-21T15:49:31.416 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5337000000000002e-05 2024-09-21T15:49:31.416 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.416 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.416 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:49:31.416 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.969927+0000", 2024-09-21T15:49:31.416 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5878000000000003e-05 2024-09-21T15:49:31.416 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.416 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.416 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:31.417 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.975067+0000", 2024-09-21T15:49:31.417 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0051400980000000001 2024-09-21T15:49:31.417 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.417 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.417 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:31.417 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.975380+0000", 2024-09-21T15:49:31.417 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00031274100000000001 2024-09-21T15:49:31.417 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.417 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.417 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:31.417 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.975404+0000", 2024-09-21T15:49:31.417 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3839999999999999e-05 2024-09-21T15:49:31.417 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.417 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.417 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:31.417 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.975422+0000", 2024-09-21T15:49:31.418 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8593e-05 2024-09-21T15:49:31.418 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.418 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:31.418 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.418 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.418 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.418 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:14449 3.6 e33/22 3:62956c53:::rbd_data.1133dc5adfdf.0000000000000045:head v 33'50, mlcod=33'48)", 2024-09-21T15:49:31.418 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:19.025593+0000", 2024-09-21T15:49:31.418 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.337997343, 2024-09-21T15:49:31.418 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.006225146, 2024-09-21T15:49:31.418 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:31.418 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:31.418 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:31.418 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.418 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:31.419 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:19.025593+0000", 2024-09-21T15:49:31.419 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.419 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.419 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.419 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:31.419 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:19.025593+0000", 2024-09-21T15:49:31.419 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.419 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.419 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.419 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:31.419 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:19.025588+0000", 2024-09-21T15:49:31.419 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999948 2024-09-21T15:49:31.419 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.419 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.419 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:31.419 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:19.025601+0000", 2024-09-21T15:49:31.420 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3833e-05 2024-09-21T15:49:31.420 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.420 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.420 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:31.420 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:19.025627+0000", 2024-09-21T15:49:31.420 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5979999999999999e-05 2024-09-21T15:49:31.420 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.420 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.420 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:31.420 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:19.025636+0000", 2024-09-21T15:49:31.420 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6659999999999997e-06 2024-09-21T15:49:31.420 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.420 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.420 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:31.420 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:19.025692+0000", 2024-09-21T15:49:31.420 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6431999999999998e-05 2024-09-21T15:49:31.421 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.421 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.421 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:31.421 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:19.025713+0000", 2024-09-21T15:49:31.421 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0971999999999998e-05 2024-09-21T15:49:31.421 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.421 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.421 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:31.421 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:19.031788+0000", 2024-09-21T15:49:31.421 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0060743580000000002 2024-09-21T15:49:31.421 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.421 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.421 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:31.421 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:19.031818+0000", 2024-09-21T15:49:31.421 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0054000000000001e-05 2024-09-21T15:49:31.422 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.422 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:31.422 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.422 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.422 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.422 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:22445 3.6 e34/22 3:66402617:::rbd_data.1133896adcfd.00000000000000bc:head v 34'72, mlcod=34'70)", 2024-09-21T15:49:31.422 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:21.730937+0000", 2024-09-21T15:49:31.422 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.6326528019999991, 2024-09-21T15:49:31.422 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0049811309999999998, 2024-09-21T15:49:31.422 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:31.422 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:31.422 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:31.422 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.422 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:31.423 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.730937+0000", 2024-09-21T15:49:31.423 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.423 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.423 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.423 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:31.423 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.730937+0000", 2024-09-21T15:49:31.423 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.423 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.423 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.423 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:31.423 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.730936+0000", 2024-09-21T15:49:31.423 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:31.423 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.423 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.423 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:31.423 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.730950+0000", 2024-09-21T15:49:31.424 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4028e-05 2024-09-21T15:49:31.424 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.424 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.424 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:31.424 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.730968+0000", 2024-09-21T15:49:31.424 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8289999999999999e-05 2024-09-21T15:49:31.424 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.424 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.424 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:31.424 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.730977+0000", 2024-09-21T15:49:31.424 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5150000000000006e-06 2024-09-21T15:49:31.424 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.424 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.424 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:31.424 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.731026+0000", 2024-09-21T15:49:31.424 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8770999999999998e-05 2024-09-21T15:49:31.425 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.425 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.425 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:31.425 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.731047+0000", 2024-09-21T15:49:31.425 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1407999999999999e-05 2024-09-21T15:49:31.425 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.425 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.425 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:31.425 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.735888+0000", 2024-09-21T15:49:31.425 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0048415660000000003 2024-09-21T15:49:31.425 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.425 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.425 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:31.425 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.735918+0000", 2024-09-21T15:49:31.425 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9899999999999998e-05 2024-09-21T15:49:31.425 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.426 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:31.426 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.426 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.426 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.426 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:25278 3.4 3:2251cc1c:::rbd_data.1133ebd6d3ba.0000000000000024:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:31.426 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.316512+0000", 2024-09-21T15:49:31.426 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.0470777980000001, 2024-09-21T15:49:31.426 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0064005590000000001, 2024-09-21T15:49:31.426 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:31.426 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:31.426 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:31.426 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:31.426 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:31.426 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 25278 2024-09-21T15:49:31.426 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.426 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:31.427 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.427 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:31.427 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.316512+0000", 2024-09-21T15:49:31.427 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.427 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.427 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.427 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:31.427 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.316512+0000", 2024-09-21T15:49:31.427 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.427 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.427 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.427 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:31.427 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.316510+0000", 2024-09-21T15:49:31.427 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:49:31.428 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.428 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.428 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:31.428 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.316527+0000", 2024-09-21T15:49:31.428 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6643e-05 2024-09-21T15:49:31.428 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.428 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.428 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:31.428 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.316547+0000", 2024-09-21T15:49:31.428 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9925999999999999e-05 2024-09-21T15:49:31.428 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.428 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.428 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:31.428 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.316559+0000", 2024-09-21T15:49:31.428 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2018000000000001e-05 2024-09-21T15:49:31.429 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.429 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.429 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:31.429 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.316618+0000", 2024-09-21T15:49:31.429 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9456000000000001e-05 2024-09-21T15:49:31.429 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.429 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.429 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:31.429 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.316719+0000", 2024-09-21T15:49:31.429 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010033400000000001 2024-09-21T15:49:31.429 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.429 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.429 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-09-21T15:49:31.429 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.316837+0000", 2024-09-21T15:49:31.429 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000118898 2024-09-21T15:49:31.430 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.430 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.430 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:31.430 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317562+0000", 2024-09-21T15:49:31.430 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00072486399999999998 2024-09-21T15:49:31.430 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.430 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.430 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:31.430 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.322838+0000", 2024-09-21T15:49:31.430 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052753139999999997 2024-09-21T15:49:31.430 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.430 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.430 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:31.430 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.322879+0000", 2024-09-21T15:49:31.431 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1559e-05 2024-09-21T15:49:31.431 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.431 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.431 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:31.431 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.322913+0000", 2024-09-21T15:49:31.431 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3618000000000002e-05 2024-09-21T15:49:31.431 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.431 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:31.431 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.431 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.431 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.431 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:25280 3.4 3:27fca38c:::rbd_data.1133ebd6d3ba.0000000000000026:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:31.431 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.316577+0000", 2024-09-21T15:49:31.431 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.0470128819999998, 2024-09-21T15:49:31.432 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0066444329999999999, 2024-09-21T15:49:31.432 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:31.432 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:31.432 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:31.432 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:31.432 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:31.432 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 25280 2024-09-21T15:49:31.432 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.432 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:31.432 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.432 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:31.432 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.316577+0000", 2024-09-21T15:49:31.432 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.432 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.433 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.433 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:31.433 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.316577+0000", 2024-09-21T15:49:31.433 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.433 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.433 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.433 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:31.433 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.316577+0000", 2024-09-21T15:49:31.433 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:31.433 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.433 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.433 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:31.433 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.316580+0000", 2024-09-21T15:49:31.433 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1839999999999999e-06 2024-09-21T15:49:31.433 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.433 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.434 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:31.434 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.316588+0000", 2024-09-21T15:49:31.434 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6109999999999996e-06 2024-09-21T15:49:31.434 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.434 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.434 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:31.434 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.316593+0000", 2024-09-21T15:49:31.434 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0540000000000002e-06 2024-09-21T15:49:31.434 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.434 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.434 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:31.434 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317176+0000", 2024-09-21T15:49:31.434 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00058346600000000004 2024-09-21T15:49:31.434 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.434 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.434 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:31.435 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317245+0000", 2024-09-21T15:49:31.435 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9165000000000006e-05 2024-09-21T15:49:31.435 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.435 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.435 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-09-21T15:49:31.435 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317348+0000", 2024-09-21T15:49:31.435 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000102897 2024-09-21T15:49:31.435 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.435 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.435 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:31.435 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323070+0000", 2024-09-21T15:49:31.435 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0057218360000000001 2024-09-21T15:49:31.435 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.435 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.435 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:31.435 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323151+0000", 2024-09-21T15:49:31.436 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0920000000000005e-05 2024-09-21T15:49:31.436 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.436 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.436 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:31.436 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323190+0000", 2024-09-21T15:49:31.436 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8914e-05 2024-09-21T15:49:31.436 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.436 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.436 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:31.436 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323222+0000", 2024-09-21T15:49:31.436 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1819999999999997e-05 2024-09-21T15:49:31.436 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.436 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:31.436 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.436 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.436 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.437 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:25286 3.0 3:0248fb60:::rbd_data.1133ebd6d3ba.0000000000000066:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:31.437 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.316617+0000", 2024-09-21T15:49:31.437 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.0469732399999998, 2024-09-21T15:49:31.437 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0066683009999999997, 2024-09-21T15:49:31.437 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:31.437 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:31.437 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:31.437 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:31.437 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:31.437 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 25286 2024-09-21T15:49:31.437 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.437 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:31.437 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.437 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:31.437 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.316617+0000", 2024-09-21T15:49:31.437 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.438 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.438 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.438 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:31.438 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.316617+0000", 2024-09-21T15:49:31.438 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.438 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.438 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.438 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:31.438 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.316616+0000", 2024-09-21T15:49:31.438 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:31.438 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.438 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.438 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:31.438 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.316620+0000", 2024-09-21T15:49:31.438 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2169999999999999e-06 2024-09-21T15:49:31.439 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.439 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.439 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:31.439 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.316627+0000", 2024-09-21T15:49:31.439 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.435e-06 2024-09-21T15:49:31.439 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.439 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.439 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:31.439 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.316632+0000", 2024-09-21T15:49:31.439 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3709999999999999e-06 2024-09-21T15:49:31.439 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.439 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.439 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:31.439 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.316675+0000", 2024-09-21T15:49:31.439 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2280999999999997e-05 2024-09-21T15:49:31.439 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.439 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.440 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:31.440 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.316750+0000", 2024-09-21T15:49:31.440 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5799999999999999e-05 2024-09-21T15:49:31.440 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.440 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.440 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-09-21T15:49:31.440 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.316841+0000", 2024-09-21T15:49:31.440 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0375000000000001e-05 2024-09-21T15:49:31.440 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.440 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.440 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:31.440 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.322697+0000", 2024-09-21T15:49:31.440 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0058559409999999999 2024-09-21T15:49:31.440 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.440 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.440 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:31.441 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323194+0000", 2024-09-21T15:49:31.441 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000496997 2024-09-21T15:49:31.441 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.441 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.441 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:31.441 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323252+0000", 2024-09-21T15:49:31.441 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7868999999999998e-05 2024-09-21T15:49:31.441 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.441 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.441 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:31.441 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323285+0000", 2024-09-21T15:49:31.441 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3488000000000002e-05 2024-09-21T15:49:31.441 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.441 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:31.441 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.441 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.442 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.442 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:30466 3.19 3:99f4d532:::rbd_data.1133957d62b2.0000000000000009:head [stat,truncate 2686976] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-21T15:49:31.442 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.920192+0000", 2024-09-21T15:49:31.442 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.4433979460000002, 2024-09-21T15:49:31.442 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.006475504, 2024-09-21T15:49:31.442 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:31.442 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:31.442 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:31.442 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:31.442 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:31.442 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 30466 2024-09-21T15:49:31.442 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.442 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:31.442 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.442 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:31.443 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.920192+0000", 2024-09-21T15:49:31.443 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.443 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.443 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.443 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:31.443 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.920192+0000", 2024-09-21T15:49:31.443 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.443 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.443 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.443 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:31.443 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.920191+0000", 2024-09-21T15:49:31.443 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:31.443 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.443 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.443 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:31.443 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.920197+0000", 2024-09-21T15:49:31.443 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3890000000000004e-06 2024-09-21T15:49:31.444 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.444 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.444 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:31.444 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.920207+0000", 2024-09-21T15:49:31.444 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0392e-05 2024-09-21T15:49:31.444 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.444 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.444 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:31.444 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.920214+0000", 2024-09-21T15:49:31.444 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6370000000000002e-06 2024-09-21T15:49:31.444 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.444 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.444 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:31.444 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.920276+0000", 2024-09-21T15:49:31.444 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2470000000000003e-05 2024-09-21T15:49:31.444 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.445 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.445 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:31.445 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.920396+0000", 2024-09-21T15:49:31.445 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000119766 2024-09-21T15:49:31.445 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.445 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.445 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:49:31.445 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.920571+0000", 2024-09-21T15:49:31.445 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000175281 2024-09-21T15:49:31.445 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.445 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.445 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:31.445 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.923541+0000", 2024-09-21T15:49:31.445 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002969733 2024-09-21T15:49:31.445 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.445 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.446 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:31.446 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.926617+0000", 2024-09-21T15:49:31.446 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003076049 2024-09-21T15:49:31.446 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.446 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.446 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:31.446 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.926648+0000", 2024-09-21T15:49:31.446 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0728000000000001e-05 2024-09-21T15:49:31.446 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.446 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.446 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:31.446 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.926668+0000", 2024-09-21T15:49:31.446 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9928e-05 2024-09-21T15:49:31.446 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.446 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:31.447 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.447 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.447 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.447 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:30770 3.4 3:20bef686:::rbd_data.1133957d62b2.000000000000000c:head [set-alloc-hint object_size 4194304 write_size 4194304,write-same] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e40)", 2024-09-21T15:49:31.447 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:28.292950+0000", 2024-09-21T15:49:31.447 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.0706396740000002, 2024-09-21T15:49:31.447 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.010856783, 2024-09-21T15:49:31.447 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:31.447 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:31.447 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:31.447 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:31.447 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:31.447 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 30770 2024-09-21T15:49:31.447 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.447 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:31.447 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.448 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:31.448 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:28.292950+0000", 2024-09-21T15:49:31.448 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.448 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.448 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.448 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:31.448 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:28.292950+0000", 2024-09-21T15:49:31.448 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.448 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.448 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.448 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:31.448 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:28.292949+0000", 2024-09-21T15:49:31.448 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:31.448 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.448 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.448 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:31.449 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:28.292959+0000", 2024-09-21T15:49:31.449 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0003000000000001e-05 2024-09-21T15:49:31.449 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.449 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.449 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:31.449 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:28.292975+0000", 2024-09-21T15:49:31.449 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6266e-05 2024-09-21T15:49:31.449 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.449 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.449 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:31.449 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:28.292990+0000", 2024-09-21T15:49:31.449 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5204e-05 2024-09-21T15:49:31.449 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.449 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.449 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:31.449 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:28.293039+0000", 2024-09-21T15:49:31.450 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8868e-05 2024-09-21T15:49:31.450 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.450 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.450 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:31.450 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:28.293095+0000", 2024-09-21T15:49:31.450 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5801000000000003e-05 2024-09-21T15:49:31.450 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.450 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.450 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-09-21T15:49:31.450 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:28.294549+0000", 2024-09-21T15:49:31.450 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001453635 2024-09-21T15:49:31.450 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.450 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.451 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:31.451 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:28.303712+0000", 2024-09-21T15:49:31.451 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.009163226 2024-09-21T15:49:31.451 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.451 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.451 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:31.451 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:28.303744+0000", 2024-09-21T15:49:31.451 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2317999999999998e-05 2024-09-21T15:49:31.451 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.451 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.451 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:31.451 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:28.303779+0000", 2024-09-21T15:49:31.451 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4929999999999999e-05 2024-09-21T15:49:31.451 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.451 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.451 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:31.452 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:28.303807+0000", 2024-09-21T15:49:31.452 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8068999999999998e-05 2024-09-21T15:49:31.452 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.452 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:31.452 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.452 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.452 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.452 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:34837 3.0 3:038527c8:::rbd_header.1133fed1a840:head [watch unwatch cookie 140381477069328] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e43)", 2024-09-21T15:49:31.452 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:30.983559+0000", 2024-09-21T15:49:31.452 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.38003134799999999, 2024-09-21T15:49:31.452 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0050826719999999999, 2024-09-21T15:49:31.452 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:31.452 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:31.452 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:31.452 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:31.452 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:31.453 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 34837 2024-09-21T15:49:31.453 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.453 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:31.453 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.453 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:31.453 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983559+0000", 2024-09-21T15:49:31.453 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.453 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.453 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.453 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:31.453 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983559+0000", 2024-09-21T15:49:31.453 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.453 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.453 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.453 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:31.453 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983558+0000", 2024-09-21T15:49:31.454 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:31.454 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.454 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.454 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:31.454 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983563+0000", 2024-09-21T15:49:31.454 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6700000000000002e-06 2024-09-21T15:49:31.454 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.454 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.454 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:31.454 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983567+0000", 2024-09-21T15:49:31.454 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2119999999999997e-06 2024-09-21T15:49:31.454 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.454 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.454 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:31.454 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983571+0000", 2024-09-21T15:49:31.455 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7160000000000001e-06 2024-09-21T15:49:31.455 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.455 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.455 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:31.455 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983694+0000", 2024-09-21T15:49:31.455 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012292800000000001 2024-09-21T15:49:31.455 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.455 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.455 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:31.455 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983758+0000", 2024-09-21T15:49:31.455 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4614000000000002e-05 2024-09-21T15:49:31.455 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.455 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.455 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-09-21T15:49:31.455 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983832+0000", 2024-09-21T15:49:31.455 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3473000000000006e-05 2024-09-21T15:49:31.456 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.456 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.456 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:31.456 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.988417+0000", 2024-09-21T15:49:31.456 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045845829999999997 2024-09-21T15:49:31.456 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.456 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.456 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:31.456 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.988558+0000", 2024-09-21T15:49:31.456 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014133200000000001 2024-09-21T15:49:31.456 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.456 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.456 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:31.456 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.988587+0000", 2024-09-21T15:49:31.456 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9363e-05 2024-09-21T15:49:31.457 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.457 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.457 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:31.457 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.988641+0000", 2024-09-21T15:49:31.457 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4150000000000002e-05 2024-09-21T15:49:31.457 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.457 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:31.457 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.457 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.457 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.457 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:34838 3.0 3:038527c8:::rbd_header.1133fed1a840:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e43)", 2024-09-21T15:49:31.457 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:30.983718+0000", 2024-09-21T15:49:31.457 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.37987157300000002, 2024-09-21T15:49:31.457 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0050601320000000002, 2024-09-21T15:49:31.458 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:31.458 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:31.458 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:31.458 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:31.458 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:31.458 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 34838 2024-09-21T15:49:31.458 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.458 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:31.458 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.458 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:31.458 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983718+0000", 2024-09-21T15:49:31.458 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.458 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.458 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.458 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:31.459 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983718+0000", 2024-09-21T15:49:31.459 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.459 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.459 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.459 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:31.459 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983718+0000", 2024-09-21T15:49:31.459 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:31.459 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.459 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.459 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:31.459 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983722+0000", 2024-09-21T15:49:31.459 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3669999999999999e-06 2024-09-21T15:49:31.459 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.459 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.460 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:31.460 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983729+0000", 2024-09-21T15:49:31.460 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3780000000000003e-06 2024-09-21T15:49:31.460 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.460 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.460 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:31.460 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983740+0000", 2024-09-21T15:49:31.460 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0638999999999999e-05 2024-09-21T15:49:31.460 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.460 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.460 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:31.460 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.984100+0000", 2024-09-21T15:49:31.460 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00036012700000000002 2024-09-21T15:49:31.460 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.460 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.461 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-09-21T15:49:31.461 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.984160+0000", 2024-09-21T15:49:31.461 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9859999999999999e-05 2024-09-21T15:49:31.461 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.461 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.461 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:31.461 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.988673+0000", 2024-09-21T15:49:31.461 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045125249999999999 2024-09-21T15:49:31.461 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.461 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.461 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:31.461 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.988731+0000", 2024-09-21T15:49:31.461 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8576999999999998e-05 2024-09-21T15:49:31.461 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.461 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.462 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:31.462 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.988779+0000", 2024-09-21T15:49:31.462 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7423000000000003e-05 2024-09-21T15:49:31.462 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.462 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:31.462 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.462 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.462 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:31.462 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:31.463 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:49:31.464 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:31.462+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:31.564 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:31.562+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:31.605 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:31.605 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:49:31.605 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:49:31.605 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:31.614 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:49:31.665 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:31.664+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:31.756 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:31.756 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:49:31.756 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:49:31.756 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:49:31.756 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:31.764 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:49:31.765 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:31.764+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:31.866 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:31.864+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:31.877 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:31.877 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:49:31.877 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:49:31.877 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:49:31.877 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.877 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:983 3.0 e27/22 3:025e53e7:::rbd_data.1133f3dfbd6b.0000000000000008:head v 27'20, mlcod=27'18)", 2024-09-21T15:49:31.877 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.615392+0000", 2024-09-21T15:49:31.877 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 19.259689645000002, 2024-09-21T15:49:31.877 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032613970000000001, 2024-09-21T15:49:31.877 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:31.877 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:31.877 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:31.877 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.878 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:31.878 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.615392+0000", 2024-09-21T15:49:31.878 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.878 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.878 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.878 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:31.878 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.615392+0000", 2024-09-21T15:49:31.878 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.878 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.878 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.878 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:31.878 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.615390+0000", 2024-09-21T15:49:31.878 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:31.879 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.879 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.879 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:31.879 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.615431+0000", 2024-09-21T15:49:31.879 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0893999999999998e-05 2024-09-21T15:49:31.879 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.879 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.879 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:31.879 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.615450+0000", 2024-09-21T15:49:31.879 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8961e-05 2024-09-21T15:49:31.879 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.879 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.879 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:31.879 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.615458+0000", 2024-09-21T15:49:31.879 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1620000000000008e-06 2024-09-21T15:49:31.880 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.880 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.880 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:31.880 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.615501+0000", 2024-09-21T15:49:31.880 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2116999999999997e-05 2024-09-21T15:49:31.880 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.880 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.880 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:31.880 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.615518+0000", 2024-09-21T15:49:31.880 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7689e-05 2024-09-21T15:49:31.880 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.880 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.880 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:31.880 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.618621+0000", 2024-09-21T15:49:31.881 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0031022340000000002 2024-09-21T15:49:31.881 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.881 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.881 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:31.881 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.618653+0000", 2024-09-21T15:49:31.881 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2591e-05 2024-09-21T15:49:31.881 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.881 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:31.881 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.881 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.881 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.881 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:984 3.0 e27/22 3:025e53e7:::rbd_data.1133f3dfbd6b.0000000000000008:head v 27'21, mlcod=27'19)", 2024-09-21T15:49:31.881 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.619393+0000", 2024-09-21T15:49:31.881 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 19.255688817999999, 2024-09-21T15:49:31.881 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003401837, 2024-09-21T15:49:31.882 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:31.882 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:31.882 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:31.882 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.882 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:31.882 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619393+0000", 2024-09-21T15:49:31.882 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.882 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.882 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.882 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:31.882 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619393+0000", 2024-09-21T15:49:31.882 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.882 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.882 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.883 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:31.883 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619392+0000", 2024-09-21T15:49:31.883 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:31.883 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.883 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.883 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:31.883 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619439+0000", 2024-09-21T15:49:31.883 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7170999999999999e-05 2024-09-21T15:49:31.883 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.883 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.883 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:31.883 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619452+0000", 2024-09-21T15:49:31.883 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2986999999999999e-05 2024-09-21T15:49:31.883 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.883 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.883 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:31.884 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619459+0000", 2024-09-21T15:49:31.884 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6109999999999999e-06 2024-09-21T15:49:31.884 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.884 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.884 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:31.884 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619518+0000", 2024-09-21T15:49:31.884 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9117999999999999e-05 2024-09-21T15:49:31.884 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.884 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.884 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:31.884 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619532+0000", 2024-09-21T15:49:31.884 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4622e-05 2024-09-21T15:49:31.884 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.884 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.884 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:31.884 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.622760+0000", 2024-09-21T15:49:31.885 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032278839999999999 2024-09-21T15:49:31.885 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.885 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.885 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:31.885 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.622794+0000", 2024-09-21T15:49:31.885 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4144999999999999e-05 2024-09-21T15:49:31.885 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.885 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:31.885 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.885 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.885 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.885 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:4957 3.9 e29/22 3:95c5ed95:::rbd_data.11335fa2fe69.0000000000000018:head v 29'4, mlcod=26'2)", 2024-09-21T15:49:31.885 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:14.453951+0000", 2024-09-21T15:49:31.885 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.421130386000002, 2024-09-21T15:49:31.885 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0043804559999999996, 2024-09-21T15:49:31.885 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:31.886 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:31.886 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:31.886 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.886 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:31.886 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.453951+0000", 2024-09-21T15:49:31.886 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.886 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.886 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.886 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:31.886 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.453951+0000", 2024-09-21T15:49:31.886 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.886 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.886 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.886 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:31.886 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.453950+0000", 2024-09-21T15:49:31.887 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:31.887 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.887 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.887 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:31.887 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454495+0000", 2024-09-21T15:49:31.887 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00054437799999999998 2024-09-21T15:49:31.887 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.887 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.887 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:31.887 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454508+0000", 2024-09-21T15:49:31.887 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3475000000000001e-05 2024-09-21T15:49:31.887 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.887 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.887 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:31.887 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454515+0000", 2024-09-21T15:49:31.888 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8990000000000004e-06 2024-09-21T15:49:31.888 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.888 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.888 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:31.888 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454581+0000", 2024-09-21T15:49:31.888 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5987000000000004e-05 2024-09-21T15:49:31.888 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.888 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.888 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:31.888 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454609+0000", 2024-09-21T15:49:31.888 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7466000000000001e-05 2024-09-21T15:49:31.888 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.888 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.888 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:31.888 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.458303+0000", 2024-09-21T15:49:31.888 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003693979 2024-09-21T15:49:31.889 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.889 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.889 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:31.889 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.458331+0000", 2024-09-21T15:49:31.889 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8932999999999999e-05 2024-09-21T15:49:31.889 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.889 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:31.889 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.889 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.889 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.889 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13420 3.0 e31/22 3:038b8840:::rbd_data.1133dc5adfdf.0000000000000035:head v 31'49, mlcod=31'47)", 2024-09-21T15:49:31.889 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.209233+0000", 2024-09-21T15:49:31.889 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.66584877, 2024-09-21T15:49:31.889 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0042624960000000002, 2024-09-21T15:49:31.889 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:31.889 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:31.890 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:31.890 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.890 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:31.890 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209233+0000", 2024-09-21T15:49:31.890 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.890 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.890 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.890 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:31.890 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209233+0000", 2024-09-21T15:49:31.890 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.890 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.890 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.890 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:31.890 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209232+0000", 2024-09-21T15:49:31.891 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:31.891 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.891 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.891 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:31.891 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209251+0000", 2024-09-21T15:49:31.891 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9304e-05 2024-09-21T15:49:31.891 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.891 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.891 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:31.891 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209265+0000", 2024-09-21T15:49:31.891 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3532e-05 2024-09-21T15:49:31.891 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.891 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.891 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:31.892 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209272+0000", 2024-09-21T15:49:31.892 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.289e-06 2024-09-21T15:49:31.892 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.892 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.892 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:31.892 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209321+0000", 2024-09-21T15:49:31.892 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8773999999999998e-05 2024-09-21T15:49:31.892 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.892 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.892 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:31.892 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209341+0000", 2024-09-21T15:49:31.892 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0387999999999999e-05 2024-09-21T15:49:31.892 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.892 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.893 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:31.893 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.213464+0000", 2024-09-21T15:49:31.893 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0041232450000000002 2024-09-21T15:49:31.893 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.893 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.893 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:31.893 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.213495+0000", 2024-09-21T15:49:31.893 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0793999999999997e-05 2024-09-21T15:49:31.893 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.893 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:31.893 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.893 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.893 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.893 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13437 3.4 e31/22 3:23f0da5d:::rbd_data.1133dc5adfdf.000000000000002d:head v 31'15, mlcod=31'13)", 2024-09-21T15:49:31.894 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.227522+0000", 2024-09-21T15:49:31.894 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.647558935999999, 2024-09-21T15:49:31.894 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033127679999999998, 2024-09-21T15:49:31.894 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:31.894 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:31.894 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:31.894 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.894 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:31.894 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.227522+0000", 2024-09-21T15:49:31.894 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.894 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.894 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.894 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:31.894 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.227522+0000", 2024-09-21T15:49:31.894 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.895 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.895 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.895 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:31.895 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.227521+0000", 2024-09-21T15:49:31.895 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:31.895 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.895 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.895 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:31.895 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.227531+0000", 2024-09-21T15:49:31.895 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.8379999999999997e-06 2024-09-21T15:49:31.895 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.895 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.895 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:31.895 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.227552+0000", 2024-09-21T15:49:31.896 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1341000000000001e-05 2024-09-21T15:49:31.896 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.896 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.896 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:31.896 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.227561+0000", 2024-09-21T15:49:31.896 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9400000000000008e-06 2024-09-21T15:49:31.896 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.896 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.896 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:31.896 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.227612+0000", 2024-09-21T15:49:31.896 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0701000000000001e-05 2024-09-21T15:49:31.896 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.896 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.896 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:31.897 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.227633+0000", 2024-09-21T15:49:31.897 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1209999999999999e-05 2024-09-21T15:49:31.897 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.897 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.897 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:31.897 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.230799+0000", 2024-09-21T15:49:31.897 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003165616 2024-09-21T15:49:31.897 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.897 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.897 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:31.897 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.230835+0000", 2024-09-21T15:49:31.897 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6321e-05 2024-09-21T15:49:31.897 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.897 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:31.898 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.898 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.898 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.898 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13445 3.0 e31/22 3:038b8840:::rbd_data.1133dc5adfdf.0000000000000035:head v 31'50, mlcod=31'48)", 2024-09-21T15:49:31.898 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.232497+0000", 2024-09-21T15:49:31.898 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.642584758, 2024-09-21T15:49:31.898 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0040493559999999996, 2024-09-21T15:49:31.898 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:31.898 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:31.898 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:31.898 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.898 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:31.898 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232497+0000", 2024-09-21T15:49:31.899 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.899 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.899 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.899 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:31.899 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232497+0000", 2024-09-21T15:49:31.899 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.899 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.899 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.899 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:31.899 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232496+0000", 2024-09-21T15:49:31.899 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:31.899 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.899 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.899 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:31.900 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232503+0000", 2024-09-21T15:49:31.900 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7320000000000001e-06 2024-09-21T15:49:31.900 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.900 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.900 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:31.900 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232512+0000", 2024-09-21T15:49:31.900 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9259999999999995e-06 2024-09-21T15:49:31.900 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.900 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.900 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:31.900 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232520+0000", 2024-09-21T15:49:31.900 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3699999999999997e-06 2024-09-21T15:49:31.900 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.901 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.901 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:31.901 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232564+0000", 2024-09-21T15:49:31.901 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4128999999999997e-05 2024-09-21T15:49:31.901 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.901 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.901 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:31.901 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232585+0000", 2024-09-21T15:49:31.901 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0704999999999999e-05 2024-09-21T15:49:31.901 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.901 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.901 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:31.901 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.236507+0000", 2024-09-21T15:49:31.901 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0039218730000000002 2024-09-21T15:49:31.901 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.902 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.902 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:31.902 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.236546+0000", 2024-09-21T15:49:31.902 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9399e-05 2024-09-21T15:49:31.902 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.902 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:31.902 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.902 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.902 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.902 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13452 3.9 e31/22 3:93f5396a:::rbd_data.1133dc5adfdf.0000000000000078:head v 31'50, mlcod=31'47)", 2024-09-21T15:49:31.902 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.237263+0000", 2024-09-21T15:49:31.902 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.637818052, 2024-09-21T15:49:31.902 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033768180000000002, 2024-09-21T15:49:31.902 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:31.902 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:31.903 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:31.903 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.903 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:31.903 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237263+0000", 2024-09-21T15:49:31.903 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.903 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.903 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.903 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:31.903 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237263+0000", 2024-09-21T15:49:31.903 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.903 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.903 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.903 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:31.903 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237263+0000", 2024-09-21T15:49:31.904 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:31.904 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.904 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.904 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:31.904 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237269+0000", 2024-09-21T15:49:31.904 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.472e-06 2024-09-21T15:49:31.904 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.904 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.904 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:31.904 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237282+0000", 2024-09-21T15:49:31.904 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2788e-05 2024-09-21T15:49:31.904 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.904 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.904 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:31.904 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237288+0000", 2024-09-21T15:49:31.905 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4899999999999997e-06 2024-09-21T15:49:31.905 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.905 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.905 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:31.905 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237331+0000", 2024-09-21T15:49:31.905 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2425000000000001e-05 2024-09-21T15:49:31.905 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.905 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.905 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:31.905 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237349+0000", 2024-09-21T15:49:31.905 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8405000000000001e-05 2024-09-21T15:49:31.905 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.905 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.905 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:31.905 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.240610+0000", 2024-09-21T15:49:31.906 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032605379999999999 2024-09-21T15:49:31.906 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.906 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.906 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:31.906 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.240640+0000", 2024-09-21T15:49:31.906 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0448000000000001e-05 2024-09-21T15:49:31.906 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.906 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:31.906 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.906 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.906 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.906 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13832 3.0 e31/22 3:0792efd2:::rbd_data.1133dc5adfdf.0000000000000090:head v 31'58, mlcod=31'56)", 2024-09-21T15:49:31.906 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.527530+0000", 2024-09-21T15:49:31.907 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.347551232000001, 2024-09-21T15:49:31.907 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035256530000000001, 2024-09-21T15:49:31.907 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:31.907 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:31.907 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:31.907 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.907 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:31.907 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527530+0000", 2024-09-21T15:49:31.907 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.907 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.907 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.907 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:31.907 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527530+0000", 2024-09-21T15:49:31.907 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.908 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.908 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.908 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:31.908 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527529+0000", 2024-09-21T15:49:31.908 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:31.908 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.908 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.908 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:31.908 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527550+0000", 2024-09-21T15:49:31.908 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1387999999999999e-05 2024-09-21T15:49:31.908 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.908 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.908 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:31.909 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527568+0000", 2024-09-21T15:49:31.909 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8068e-05 2024-09-21T15:49:31.909 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.909 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.909 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:31.909 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527577+0000", 2024-09-21T15:49:31.909 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9919999999999996e-06 2024-09-21T15:49:31.909 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.909 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.909 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:31.909 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527626+0000", 2024-09-21T15:49:31.909 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8597000000000002e-05 2024-09-21T15:49:31.909 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.909 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.909 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:31.910 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527652+0000", 2024-09-21T15:49:31.910 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5993e-05 2024-09-21T15:49:31.910 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.910 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.910 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:31.910 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.531024+0000", 2024-09-21T15:49:31.910 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033721910000000001 2024-09-21T15:49:31.910 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.910 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.910 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:31.910 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.531056+0000", 2024-09-21T15:49:31.910 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1628000000000003e-05 2024-09-21T15:49:31.910 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.911 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:31.911 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.911 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.911 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.911 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13839 3.0 e31/22 3:03c4fb32:::rbd_data.1133dc5adfdf.0000000000000099:head v 31'59, mlcod=31'56)", 2024-09-21T15:49:31.911 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.528097+0000", 2024-09-21T15:49:31.911 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.346984634, 2024-09-21T15:49:31.911 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0057228840000000001, 2024-09-21T15:49:31.911 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:31.911 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:31.911 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:31.911 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.911 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:31.912 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528097+0000", 2024-09-21T15:49:31.912 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.912 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.912 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.912 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:31.912 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528097+0000", 2024-09-21T15:49:31.912 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.912 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.912 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.912 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:31.912 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528096+0000", 2024-09-21T15:49:31.912 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:31.912 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.912 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.913 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:31.913 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528105+0000", 2024-09-21T15:49:31.913 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7350000000000002e-06 2024-09-21T15:49:31.913 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.913 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.913 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:31.913 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528117+0000", 2024-09-21T15:49:31.913 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1909000000000001e-05 2024-09-21T15:49:31.913 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.913 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.913 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:31.913 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528124+0000", 2024-09-21T15:49:31.913 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8419999999999999e-06 2024-09-21T15:49:31.913 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.914 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.914 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:31.914 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528168+0000", 2024-09-21T15:49:31.914 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4301999999999997e-05 2024-09-21T15:49:31.914 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.914 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.914 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:31.914 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528191+0000", 2024-09-21T15:49:31.914 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2622000000000001e-05 2024-09-21T15:49:31.914 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.914 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.914 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:31.914 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.533780+0000", 2024-09-21T15:49:31.914 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0055897519999999999 2024-09-21T15:49:31.915 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.915 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.915 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:31.915 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.533820+0000", 2024-09-21T15:49:31.915 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9366999999999998e-05 2024-09-21T15:49:31.915 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.915 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:31.915 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.915 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.915 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.915 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13884 3.9 e31/22 3:93003736:::rbd_data.1133dc5adfdf.00000000000000a1:head v 31'54, mlcod=31'52)", 2024-09-21T15:49:31.915 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.562433+0000", 2024-09-21T15:49:31.915 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.312647886000001, 2024-09-21T15:49:31.915 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.005162313, 2024-09-21T15:49:31.916 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:31.916 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:31.916 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:31.916 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.916 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:31.916 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562433+0000", 2024-09-21T15:49:31.916 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.916 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.916 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.916 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:31.916 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562433+0000", 2024-09-21T15:49:31.916 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.916 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.916 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.916 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:31.916 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562432+0000", 2024-09-21T15:49:31.917 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:31.917 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.917 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.917 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:31.917 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562467+0000", 2024-09-21T15:49:31.917 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4885999999999998e-05 2024-09-21T15:49:31.917 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.917 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.917 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:31.917 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562481+0000", 2024-09-21T15:49:31.917 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3604000000000001e-05 2024-09-21T15:49:31.917 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.917 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.917 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:31.917 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562489+0000", 2024-09-21T15:49:31.917 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1829999999999994e-06 2024-09-21T15:49:31.918 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.918 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.918 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:31.918 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562536+0000", 2024-09-21T15:49:31.918 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6830999999999999e-05 2024-09-21T15:49:31.918 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.918 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.918 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:31.918 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562559+0000", 2024-09-21T15:49:31.918 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2765999999999999e-05 2024-09-21T15:49:31.918 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.918 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.918 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:31.918 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.567566+0000", 2024-09-21T15:49:31.918 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0050073219999999998 2024-09-21T15:49:31.919 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.919 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.919 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:31.919 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.567596+0000", 2024-09-21T15:49:31.919 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9802999999999999e-05 2024-09-21T15:49:31.919 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.919 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:31.919 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.919 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.919 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.919 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13908 3.9 e31/22 3:922f1ce0:::rbd_data.1133dc5adfdf.00000000000000e9:head v 31'55, mlcod=31'53)", 2024-09-21T15:49:31.919 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.581150+0000", 2024-09-21T15:49:31.919 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.293931714999999, 2024-09-21T15:49:31.919 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033342329999999998, 2024-09-21T15:49:31.919 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:31.919 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:31.920 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:31.920 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.920 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:31.920 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.581150+0000", 2024-09-21T15:49:31.920 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.920 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.920 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.920 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:31.920 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.581150+0000", 2024-09-21T15:49:31.920 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.920 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.920 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.920 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:31.920 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.581149+0000", 2024-09-21T15:49:31.920 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:31.921 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.921 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.921 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:31.921 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.581167+0000", 2024-09-21T15:49:31.921 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7948000000000002e-05 2024-09-21T15:49:31.921 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.921 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.921 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:31.921 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.581181+0000", 2024-09-21T15:49:31.921 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4392e-05 2024-09-21T15:49:31.921 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.921 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.921 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:31.921 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.581189+0000", 2024-09-21T15:49:31.921 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7640000000000005e-06 2024-09-21T15:49:31.922 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.922 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.922 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:31.922 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.581232+0000", 2024-09-21T15:49:31.922 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2639999999999998e-05 2024-09-21T15:49:31.922 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.922 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.922 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:31.922 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.581255+0000", 2024-09-21T15:49:31.922 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3802999999999999e-05 2024-09-21T15:49:31.922 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.922 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.922 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:31.922 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.584445+0000", 2024-09-21T15:49:31.922 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0031900380000000001 2024-09-21T15:49:31.923 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.923 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.923 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:31.923 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.584484+0000", 2024-09-21T15:49:31.923 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8472e-05 2024-09-21T15:49:31.923 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.923 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:31.923 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.923 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.923 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.923 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13991 3.9 e31/22 3:93003736:::rbd_data.1133dc5adfdf.00000000000000a1:head v 31'57, mlcod=31'55)", 2024-09-21T15:49:31.923 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.652441+0000", 2024-09-21T15:49:31.923 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.222640518, 2024-09-21T15:49:31.923 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0031299409999999998, 2024-09-21T15:49:31.923 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:31.923 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:31.924 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:31.924 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.924 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:31.924 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.652441+0000", 2024-09-21T15:49:31.924 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.924 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.924 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.924 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:31.924 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.652441+0000", 2024-09-21T15:49:31.924 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.924 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.924 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.924 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:31.924 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.652440+0000", 2024-09-21T15:49:31.924 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:31.924 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.925 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.925 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:31.925 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.652449+0000", 2024-09-21T15:49:31.925 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4190000000000008e-06 2024-09-21T15:49:31.925 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.925 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.925 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:31.925 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.652467+0000", 2024-09-21T15:49:31.925 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7946000000000001e-05 2024-09-21T15:49:31.925 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.925 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.925 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:31.925 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.652476+0000", 2024-09-21T15:49:31.925 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.789e-06 2024-09-21T15:49:31.925 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.926 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.926 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:31.926 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.652534+0000", 2024-09-21T15:49:31.926 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7986999999999999e-05 2024-09-21T15:49:31.926 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.926 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.926 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:31.926 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.652555+0000", 2024-09-21T15:49:31.926 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1501999999999998e-05 2024-09-21T15:49:31.926 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.926 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.926 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:31.926 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.655534+0000", 2024-09-21T15:49:31.926 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029786869999999998 2024-09-21T15:49:31.926 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.926 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.927 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:31.927 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.655571+0000", 2024-09-21T15:49:31.927 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6736999999999997e-05 2024-09-21T15:49:31.927 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.927 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:31.927 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.927 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.927 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.927 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:22335 3.9 e34/22 3:9490b97f:::rbd_data.1133896adcfd.000000000000001e:head v 34'91, mlcod=34'89)", 2024-09-21T15:49:31.927 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:21.624226+0000", 2024-09-21T15:49:31.927 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.250855097000001, 2024-09-21T15:49:31.927 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003106017, 2024-09-21T15:49:31.927 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:31.927 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:31.927 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:31.928 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.928 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:31.928 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.624226+0000", 2024-09-21T15:49:31.928 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.928 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.928 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.928 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:31.928 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.624226+0000", 2024-09-21T15:49:31.928 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.928 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.928 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.928 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:31.928 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.624225+0000", 2024-09-21T15:49:31.928 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:31.928 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.928 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.929 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:31.929 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.624254+0000", 2024-09-21T15:49:31.929 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9261e-05 2024-09-21T15:49:31.929 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.929 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.929 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:31.929 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.624272+0000", 2024-09-21T15:49:31.929 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8189000000000002e-05 2024-09-21T15:49:31.929 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.929 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.929 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:31.929 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.624281+0000", 2024-09-21T15:49:31.929 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3760000000000008e-06 2024-09-21T15:49:31.929 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.929 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.929 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:31.930 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.624339+0000", 2024-09-21T15:49:31.930 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8631000000000002e-05 2024-09-21T15:49:31.930 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.930 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.930 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:31.930 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.624362+0000", 2024-09-21T15:49:31.930 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2688e-05 2024-09-21T15:49:31.930 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.930 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.930 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:31.930 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.627296+0000", 2024-09-21T15:49:31.930 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002933742 2024-09-21T15:49:31.930 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.930 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.931 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:31.931 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.627332+0000", 2024-09-21T15:49:31.931 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6609999999999997e-05 2024-09-21T15:49:31.931 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.931 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:31.931 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.931 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.931 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.931 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:22340 3.9 e34/22 3:910a5abe:::rbd_data.1133896adcfd.0000000000000023:head v 34'92, mlcod=34'90)", 2024-09-21T15:49:31.931 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:21.628025+0000", 2024-09-21T15:49:31.931 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.247056506, 2024-09-21T15:49:31.931 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046196010000000001, 2024-09-21T15:49:31.931 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:31.931 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:31.931 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:31.931 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.932 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:31.932 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628025+0000", 2024-09-21T15:49:31.932 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.932 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.932 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.932 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:31.932 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628025+0000", 2024-09-21T15:49:31.932 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.932 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.932 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.932 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:31.932 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628024+0000", 2024-09-21T15:49:31.932 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:31.932 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.932 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.933 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:31.933 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628034+0000", 2024-09-21T15:49:31.933 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3470000000000005e-06 2024-09-21T15:49:31.933 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.933 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.933 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:31.933 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628043+0000", 2024-09-21T15:49:31.933 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9450000000000005e-06 2024-09-21T15:49:31.933 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.933 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.933 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:31.933 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628048+0000", 2024-09-21T15:49:31.933 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8080000000000004e-06 2024-09-21T15:49:31.933 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.934 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.934 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:31.934 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628081+0000", 2024-09-21T15:49:31.934 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2738999999999999e-05 2024-09-21T15:49:31.934 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.934 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.934 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:31.934 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628094+0000", 2024-09-21T15:49:31.934 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3273e-05 2024-09-21T15:49:31.934 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.934 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.934 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:31.934 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.632616+0000", 2024-09-21T15:49:31.935 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045217269999999997 2024-09-21T15:49:31.935 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.935 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.935 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:31.935 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.632644+0000", 2024-09-21T15:49:31.935 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8469000000000001e-05 2024-09-21T15:49:31.935 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.935 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:31.935 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.935 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.935 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.935 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:22417 3.4 e34/22 3:25310a10:::rbd_data.1133896adcfd.0000000000000075:head v 34'48, mlcod=34'46)", 2024-09-21T15:49:31.935 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:21.704297+0000", 2024-09-21T15:49:31.935 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 10.170784407999999, 2024-09-21T15:49:31.936 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0038101799999999998, 2024-09-21T15:49:31.936 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:31.936 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:31.936 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:31.936 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.936 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:31.936 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704297+0000", 2024-09-21T15:49:31.936 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.936 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.936 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.936 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:31.936 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704297+0000", 2024-09-21T15:49:31.936 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.936 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.936 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.937 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:31.937 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704296+0000", 2024-09-21T15:49:31.937 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:31.937 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.937 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.937 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:31.937 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704308+0000", 2024-09-21T15:49:31.937 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2559e-05 2024-09-21T15:49:31.937 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.937 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.937 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:31.937 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704337+0000", 2024-09-21T15:49:31.937 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8960000000000001e-05 2024-09-21T15:49:31.938 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.938 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.938 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:31.938 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704345+0000", 2024-09-21T15:49:31.938 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3059999999999998e-06 2024-09-21T15:49:31.938 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.938 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.938 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:31.938 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704408+0000", 2024-09-21T15:49:31.938 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2156000000000006e-05 2024-09-21T15:49:31.938 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.938 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.938 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:31.938 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704430+0000", 2024-09-21T15:49:31.938 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1931000000000001e-05 2024-09-21T15:49:31.939 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.939 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.939 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:31.939 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.708073+0000", 2024-09-21T15:49:31.939 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003643028 2024-09-21T15:49:31.939 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.939 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.939 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:31.939 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.708107+0000", 2024-09-21T15:49:31.939 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4613999999999998e-05 2024-09-21T15:49:31.939 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.939 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:31.939 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.939 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.940 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.940 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:25278 3.4 e38/22 3:2251cc1c:::rbd_data.1133ebd6d3ba.0000000000000024:head v 38'51, mlcod=38'49)", 2024-09-21T15:49:31.940 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.317337+0000", 2024-09-21T15:49:31.940 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.5577446479999999, 2024-09-21T15:49:31.940 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053904110000000003, 2024-09-21T15:49:31.940 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:31.940 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:31.940 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:31.940 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.940 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:31.940 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317337+0000", 2024-09-21T15:49:31.940 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.940 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.940 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.941 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:31.941 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317337+0000", 2024-09-21T15:49:31.941 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.941 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.941 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.941 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:31.941 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317335+0000", 2024-09-21T15:49:31.941 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:49:31.941 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.941 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.941 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:31.941 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317347+0000", 2024-09-21T15:49:31.941 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.19e-05 2024-09-21T15:49:31.941 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.942 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.942 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:31.942 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317369+0000", 2024-09-21T15:49:31.942 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2232e-05 2024-09-21T15:49:31.942 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.942 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.942 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:31.942 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317381+0000", 2024-09-21T15:49:31.942 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2231e-05 2024-09-21T15:49:31.942 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.942 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.942 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:31.942 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317425+0000", 2024-09-21T15:49:31.942 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3625000000000003e-05 2024-09-21T15:49:31.942 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.943 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.943 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:31.943 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317449+0000", 2024-09-21T15:49:31.943 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4301999999999999e-05 2024-09-21T15:49:31.943 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.943 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.943 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:31.943 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.322688+0000", 2024-09-21T15:49:31.943 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.005238609 2024-09-21T15:49:31.943 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.943 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.943 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:31.943 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.322727+0000", 2024-09-21T15:49:31.943 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9201e-05 2024-09-21T15:49:31.944 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.944 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:31.944 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.944 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.944 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.944 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:25286 3.0 e38/22 3:0248fb60:::rbd_data.1133ebd6d3ba.0000000000000066:head v 38'109, mlcod=34'107)", 2024-09-21T15:49:31.944 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.317638+0000", 2024-09-21T15:49:31.944 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.5574437689999998, 2024-09-21T15:49:31.944 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0054074370000000002, 2024-09-21T15:49:31.944 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:31.944 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:31.944 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:31.944 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.945 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:31.945 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317638+0000", 2024-09-21T15:49:31.945 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.945 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.945 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.945 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:31.945 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317638+0000", 2024-09-21T15:49:31.945 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.945 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.945 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.945 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:31.945 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317637+0000", 2024-09-21T15:49:31.945 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:31.945 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.945 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.945 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:31.946 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317643+0000", 2024-09-21T15:49:31.946 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5609999999999998e-06 2024-09-21T15:49:31.946 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.946 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.946 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:31.946 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317653+0000", 2024-09-21T15:49:31.946 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0705e-05 2024-09-21T15:49:31.946 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.946 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.946 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:31.946 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317660+0000", 2024-09-21T15:49:31.946 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2099999999999998e-06 2024-09-21T15:49:31.946 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.946 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.947 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:31.947 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317704+0000", 2024-09-21T15:49:31.947 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4659000000000001e-05 2024-09-21T15:49:31.947 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.947 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.947 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:31.947 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317725+0000", 2024-09-21T15:49:31.947 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0537e-05 2024-09-21T15:49:31.947 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.947 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.947 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:31.947 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323010+0000", 2024-09-21T15:49:31.947 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052853420000000002 2024-09-21T15:49:31.947 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.947 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.948 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:31.948 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323045+0000", 2024-09-21T15:49:31.948 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4860000000000002e-05 2024-09-21T15:49:31.948 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.948 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:31.948 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.948 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.948 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.948 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:25280 3.4 e38/22 3:27fca38c:::rbd_data.1133ebd6d3ba.0000000000000026:head v 38'52, mlcod=38'49)", 2024-09-21T15:49:31.948 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.317684+0000", 2024-09-21T15:49:31.948 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.5573978730000002, 2024-09-21T15:49:31.948 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053570809999999996, 2024-09-21T15:49:31.948 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:31.948 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:31.948 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:31.949 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.949 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:31.949 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317684+0000", 2024-09-21T15:49:31.949 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.949 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.949 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.949 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:31.949 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317684+0000", 2024-09-21T15:49:31.949 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.949 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.949 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.949 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:31.949 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317683+0000", 2024-09-21T15:49:31.949 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:31.949 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.950 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.950 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:31.950 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317688+0000", 2024-09-21T15:49:31.950 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5009999999999998e-06 2024-09-21T15:49:31.950 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.950 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.950 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:31.950 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317694+0000", 2024-09-21T15:49:31.950 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3579999999999999e-06 2024-09-21T15:49:31.950 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.950 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.950 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:31.950 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317698+0000", 2024-09-21T15:49:31.950 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4150000000000001e-06 2024-09-21T15:49:31.950 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.951 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.951 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:31.951 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317738+0000", 2024-09-21T15:49:31.951 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9575e-05 2024-09-21T15:49:31.951 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.951 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.951 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:31.951 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317762+0000", 2024-09-21T15:49:31.951 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4173999999999999e-05 2024-09-21T15:49:31.951 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.951 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.951 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:31.951 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323005+0000", 2024-09-21T15:49:31.951 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00524333 2024-09-21T15:49:31.951 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.951 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.952 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:31.952 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323041+0000", 2024-09-21T15:49:31.952 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5139999999999999e-05 2024-09-21T15:49:31.952 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.952 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:31.952 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.952 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.952 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.952 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:30578 3.4 e40/22 3:27afb24d:::rbd_data.1133957d62b2.0000000000000005:head v 40'99, mlcod=40'97)", 2024-09-21T15:49:31.952 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:28.063816+0000", 2024-09-21T15:49:31.952 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.8112657329999999, 2024-09-21T15:49:31.952 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032266920000000002, 2024-09-21T15:49:31.952 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:31.952 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:31.952 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:31.953 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.953 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:31.953 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:28.063816+0000", 2024-09-21T15:49:31.953 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.953 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.953 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.953 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:31.953 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:28.063816+0000", 2024-09-21T15:49:31.953 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.953 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.953 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.953 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:31.953 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:28.063814+0000", 2024-09-21T15:49:31.953 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:31.953 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.954 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.954 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:31.954 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:28.063824+0000", 2024-09-21T15:49:31.954 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.8900000000000002e-06 2024-09-21T15:49:31.954 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.954 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.954 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:31.954 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:28.063844+0000", 2024-09-21T15:49:31.954 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9709999999999999e-05 2024-09-21T15:49:31.954 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.954 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.954 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:31.954 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:28.063853+0000", 2024-09-21T15:49:31.954 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7479999999999999e-06 2024-09-21T15:49:31.955 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.955 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.955 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:31.955 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:28.063895+0000", 2024-09-21T15:49:31.955 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2191000000000003e-05 2024-09-21T15:49:31.955 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.955 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.955 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:31.955 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:28.063915+0000", 2024-09-21T15:49:31.955 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0075999999999999e-05 2024-09-21T15:49:31.955 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.955 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.955 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:31.955 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:28.067006+0000", 2024-09-21T15:49:31.955 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0030907479999999999 2024-09-21T15:49:31.955 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.956 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.956 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:31.956 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:28.067042+0000", 2024-09-21T15:49:31.956 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6803e-05 2024-09-21T15:49:31.956 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.956 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:31.956 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.956 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.956 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.956 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:34837 3.0 e43/22 3:038527c8:::rbd_header.1133fed1a840:head v 43'130, mlcod=43'128)", 2024-09-21T15:49:31.956 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:49:31.956 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.89119317499999995, 2024-09-21T15:49:31.956 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045776519999999998, 2024-09-21T15:49:31.956 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:31.956 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:31.957 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:31.957 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.957 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:31.957 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:49:31.957 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.957 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.957 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.957 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:31.957 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:49:31.957 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:31.957 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.957 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.957 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:31.957 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:49:31.957 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:31.958 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.958 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.958 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:31.958 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983892+0000", 2024-09-21T15:49:31.958 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8450000000000002e-06 2024-09-21T15:49:31.958 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.958 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.958 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:31.958 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983902+0000", 2024-09-21T15:49:31.958 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0299999999999999e-06 2024-09-21T15:49:31.958 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.958 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.958 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:31.958 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983906+0000", 2024-09-21T15:49:31.958 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2300000000000002e-06 2024-09-21T15:49:31.959 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.959 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.959 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:31.959 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983940+0000", 2024-09-21T15:49:31.959 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4563000000000001e-05 2024-09-21T15:49:31.959 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.959 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.959 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:31.959 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983960+0000", 2024-09-21T15:49:31.959 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9199999999999999e-05 2024-09-21T15:49:31.959 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.959 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.959 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:31.959 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.988430+0000", 2024-09-21T15:49:31.959 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0044709149999999998 2024-09-21T15:49:31.960 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:31.960 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:31.960 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:31.960 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.988466+0000", 2024-09-21T15:49:31.960 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5420000000000003e-05 2024-09-21T15:49:31.960 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.960 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:31.960 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.960 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:31.960 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:31.960 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:31.961 INFO:teuthology.orchestra.run.smithi089.stdout:truncating image image_client.0-clone10 from 0x8b0a7c2 (overlap 0x4144af2) to 0x2db0143 2024-09-21T15:49:31.962 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:49:31.967 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:31.966+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:32.036 INFO:teuthology.orchestra.run.smithi089.stdout:checking clone #9, image image_client.0-clone9 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent10 2024-09-21T15:49:32.068 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:32.067+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:32.094 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:32.094 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:49:32.094 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:49:32.094 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:32.103 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:49:32.153 INFO:teuthology.orchestra.run.smithi089.stdout:2112 writesame 0xed8580a thru 0xed8c5ce data_size 0x13(0x6dc5 bytes) 2024-09-21T15:49:32.153 INFO:teuthology.orchestra.run.smithi089.stdout:2113 punch from 0xde6db4d to 0xde7bf56, (0xe409 bytes) 2024-09-21T15:49:32.157 INFO:teuthology.orchestra.run.smithi089.stdout:2115 trunc from 0xed8c5cf to 0xbac6784 2024-09-21T15:49:32.162 INFO:teuthology.orchestra.run.smithi089.stdout:2116 writesame 0x67d09c3 thru 0x67de7b4 data_size 0x1(0xddf2 bytes) 2024-09-21T15:49:32.169 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:32.168+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:32.181 INFO:teuthology.orchestra.run.smithi089.stdout:2117 write 0xb6062dd thru 0xb609ce4 (0x3a08 bytes) 2024-09-21T15:49:32.192 INFO:teuthology.orchestra.run.smithi089.stdout:2118 write 0x23e4bce thru 0x23ecbb1 (0x7fe4 bytes) 2024-09-21T15:49:32.248 INFO:teuthology.orchestra.run.smithi089.stdout:2119 write 0x2536f45 thru 0x2544bf0 (0xdcac bytes) 2024-09-21T15:49:32.260 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:32.260 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:49:32.260 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:49:32.260 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:49:32.261 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:32.270 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:32.269+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:32.272 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:49:32.320 INFO:teuthology.orchestra.run.smithi089.stdout:2120 clone 12 order 20 su 262144 sc 8 2024-09-21T15:49:32.371 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:32.369+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:32.416 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:32.416 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:49:32.416 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:49:32.416 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:49:32.416 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.416 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:14051 3.8 3:13437810:::rbd_data.1133dc5adfdf.0000000000000039:head [write 716800~796] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-21T15:49:32.416 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:49:32.416 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.71877587, 2024-09-21T15:49:32.416 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.98064002699999997, 2024-09-21T15:49:32.417 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:32.417 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:32.417 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:32.417 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:32.417 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:32.417 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 14051 2024-09-21T15:49:32.417 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.417 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:32.417 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.417 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:32.417 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:49:32.417 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:32.417 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.417 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.418 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:32.418 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:49:32.418 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:32.418 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.418 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.418 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:32.418 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:49:32.418 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:32.418 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.418 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.419 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:32.419 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695131+0000", 2024-09-21T15:49:32.419 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4680000000000003e-06 2024-09-21T15:49:32.419 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.419 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.419 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:32.419 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695139+0000", 2024-09-21T15:49:32.419 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9400000000000002e-06 2024-09-21T15:49:32.419 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.419 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.419 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:32.419 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695143+0000", 2024-09-21T15:49:32.419 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9899999999999999e-06 2024-09-21T15:49:32.419 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.420 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.420 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:32.420 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695190+0000", 2024-09-21T15:49:32.420 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6458000000000001e-05 2024-09-21T15:49:32.420 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.420 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.420 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:49:32.420 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695201+0000", 2024-09-21T15:49:32.420 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0964999999999999e-05 2024-09-21T15:49:32.420 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.420 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.420 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:32.420 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.669490+0000", 2024-09-21T15:49:32.420 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.97428980499999995 2024-09-21T15:49:32.420 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.421 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.421 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-09-21T15:49:32.421 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.670033+0000", 2024-09-21T15:49:32.421 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00054252599999999999 2024-09-21T15:49:32.421 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.421 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.421 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:32.421 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.671095+0000", 2024-09-21T15:49:32.421 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001062183 2024-09-21T15:49:32.421 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.421 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.421 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:32.421 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.671186+0000", 2024-09-21T15:49:32.421 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1191000000000002e-05 2024-09-21T15:49:32.421 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.422 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.422 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-09-21T15:49:32.422 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.671346+0000", 2024-09-21T15:49:32.422 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015980399999999999 2024-09-21T15:49:32.422 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.422 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.422 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:32.422 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.672221+0000", 2024-09-21T15:49:32.422 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00087471999999999995 2024-09-21T15:49:32.422 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.422 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.422 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:32.422 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.675619+0000", 2024-09-21T15:49:32.422 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00339858 2024-09-21T15:49:32.423 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.423 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.423 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:32.423 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.675670+0000", 2024-09-21T15:49:32.423 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0664000000000001e-05 2024-09-21T15:49:32.423 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.423 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.423 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:32.423 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.675768+0000", 2024-09-21T15:49:32.423 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7415000000000002e-05 2024-09-21T15:49:32.423 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:32.423 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:32.423 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:32.423 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.423 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.424 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28537 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 409600~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:32.424 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.227783+0000", 2024-09-21T15:49:32.424 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.1861207580000004, 2024-09-21T15:49:32.424 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.046702573999999997, 2024-09-21T15:49:32.424 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:32.424 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:32.424 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:32.424 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:32.424 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:32.424 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28537 2024-09-21T15:49:32.424 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.424 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:32.424 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.424 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:32.424 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227783+0000", 2024-09-21T15:49:32.425 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:32.425 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.425 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.425 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:32.425 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227783+0000", 2024-09-21T15:49:32.425 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:32.425 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.425 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.425 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:32.425 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227782+0000", 2024-09-21T15:49:32.425 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:32.425 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.425 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.425 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:32.426 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227785+0000", 2024-09-21T15:49:32.426 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.34e-06 2024-09-21T15:49:32.426 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.426 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.426 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:32.426 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227791+0000", 2024-09-21T15:49:32.426 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2020000000000001e-06 2024-09-21T15:49:32.426 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.426 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.426 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:32.426 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227796+0000", 2024-09-21T15:49:32.426 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8289999999999998e-06 2024-09-21T15:49:32.426 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.426 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.426 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:32.427 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.274387+0000", 2024-09-21T15:49:32.427 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.046591024000000002 2024-09-21T15:49:32.427 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.427 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.427 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:32.427 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.274427+0000", 2024-09-21T15:49:32.427 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0825000000000003e-05 2024-09-21T15:49:32.427 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.427 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.427 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:32.427 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.274485+0000", 2024-09-21T15:49:32.427 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7713000000000001e-05 2024-09-21T15:49:32.427 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:32.427 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:32.428 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:32.428 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.428 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.428 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28538 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 1212416~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:32.428 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.227800+0000", 2024-09-21T15:49:32.428 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.1861033770000002, 2024-09-21T15:49:32.428 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.046874213999999997, 2024-09-21T15:49:32.428 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:32.428 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:32.428 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:32.428 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:32.428 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:32.428 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28538 2024-09-21T15:49:32.428 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.428 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:32.428 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.429 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:32.429 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227800+0000", 2024-09-21T15:49:32.429 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:32.429 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.429 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.429 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:32.429 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227800+0000", 2024-09-21T15:49:32.429 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:32.429 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.429 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.429 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:32.429 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227800+0000", 2024-09-21T15:49:32.429 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:32.429 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.429 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.430 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:32.430 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227809+0000", 2024-09-21T15:49:32.430 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4709999999999996e-06 2024-09-21T15:49:32.430 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.430 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.430 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:32.430 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227816+0000", 2024-09-21T15:49:32.430 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0809999999999996e-06 2024-09-21T15:49:32.430 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.430 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.430 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:32.430 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227821+0000", 2024-09-21T15:49:32.430 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9420000000000003e-06 2024-09-21T15:49:32.430 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.430 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.431 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:32.431 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.274515+0000", 2024-09-21T15:49:32.431 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.046694201999999997 2024-09-21T15:49:32.431 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.431 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.431 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:32.431 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.274585+0000", 2024-09-21T15:49:32.431 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9250000000000003e-05 2024-09-21T15:49:32.431 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.431 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.431 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:32.431 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.274674+0000", 2024-09-21T15:49:32.431 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9621000000000001e-05 2024-09-21T15:49:32.431 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:32.431 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:32.432 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:32.432 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.432 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.432 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28539 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 2015232~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:32.432 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.227826+0000", 2024-09-21T15:49:32.432 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.186077375, 2024-09-21T15:49:32.432 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.046969221999999998, 2024-09-21T15:49:32.432 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:32.432 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:32.432 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:32.432 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:32.432 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:32.432 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28539 2024-09-21T15:49:32.432 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.432 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:32.432 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.433 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:32.433 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227826+0000", 2024-09-21T15:49:32.433 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:32.433 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.433 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.433 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:32.433 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227826+0000", 2024-09-21T15:49:32.433 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:32.433 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.433 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.433 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:32.433 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227826+0000", 2024-09-21T15:49:32.433 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:32.433 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.433 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.434 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:32.434 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227828+0000", 2024-09-21T15:49:32.434 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4040000000000002e-06 2024-09-21T15:49:32.434 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.434 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.434 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:32.434 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227835+0000", 2024-09-21T15:49:32.434 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6100000000000002e-06 2024-09-21T15:49:32.434 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.434 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.434 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:32.434 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227839+0000", 2024-09-21T15:49:32.434 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7589999999999997e-06 2024-09-21T15:49:32.434 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.434 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.434 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:32.435 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.274701+0000", 2024-09-21T15:49:32.435 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.046861828000000001 2024-09-21T15:49:32.435 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.435 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.435 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:32.435 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.274742+0000", 2024-09-21T15:49:32.435 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1074e-05 2024-09-21T15:49:32.435 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.435 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.435 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:32.435 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.274795+0000", 2024-09-21T15:49:32.435 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2945999999999998e-05 2024-09-21T15:49:32.435 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:32.435 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:32.436 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:32.436 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.436 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.436 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28540 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 2760704~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:32.436 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.227844+0000", 2024-09-21T15:49:32.436 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.1860593660000003, 2024-09-21T15:49:32.436 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047146018999999997, 2024-09-21T15:49:32.436 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:32.436 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:32.436 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:32.436 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:32.436 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:32.436 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28540 2024-09-21T15:49:32.436 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.437 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:32.437 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.437 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:32.437 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227844+0000", 2024-09-21T15:49:32.437 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:32.437 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.437 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.437 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:32.437 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227844+0000", 2024-09-21T15:49:32.437 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:32.437 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.437 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.437 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:32.437 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227844+0000", 2024-09-21T15:49:32.438 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:32.438 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.438 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.438 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:32.438 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227846+0000", 2024-09-21T15:49:32.438 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1959999999999998e-06 2024-09-21T15:49:32.438 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.438 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.438 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:32.438 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227852+0000", 2024-09-21T15:49:32.438 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1659999999999999e-06 2024-09-21T15:49:32.438 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.438 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.438 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:32.439 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227856+0000", 2024-09-21T15:49:32.439 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4789999999999998e-06 2024-09-21T15:49:32.439 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.439 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.439 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:32.439 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.274825+0000", 2024-09-21T15:49:32.439 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.046968325999999998 2024-09-21T15:49:32.439 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.439 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.439 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:32.439 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.274896+0000", 2024-09-21T15:49:32.439 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0859999999999996e-05 2024-09-21T15:49:32.439 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.439 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.440 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:32.440 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.274990+0000", 2024-09-21T15:49:32.440 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4345999999999994e-05 2024-09-21T15:49:32.440 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:32.440 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:32.440 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:32.440 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.440 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.440 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28541 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 3563520~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:32.440 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.227861+0000", 2024-09-21T15:49:32.440 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.1860423769999997, 2024-09-21T15:49:32.440 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047277444000000002, 2024-09-21T15:49:32.440 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:32.440 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:32.440 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:32.441 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:32.441 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:32.441 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28541 2024-09-21T15:49:32.441 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.441 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:32.441 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.441 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:32.441 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227861+0000", 2024-09-21T15:49:32.441 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:32.441 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.441 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.441 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:32.441 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227861+0000", 2024-09-21T15:49:32.441 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:32.441 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.442 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.442 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:32.442 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227861+0000", 2024-09-21T15:49:32.442 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:32.442 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.442 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.442 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:32.442 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227863+0000", 2024-09-21T15:49:32.442 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0789999999999999e-06 2024-09-21T15:49:32.442 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.442 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.442 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:32.442 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227870+0000", 2024-09-21T15:49:32.442 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8070000000000002e-06 2024-09-21T15:49:32.443 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.443 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.443 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:32.443 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227874+0000", 2024-09-21T15:49:32.443 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9599999999999999e-06 2024-09-21T15:49:32.443 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.443 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.443 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:32.443 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275002+0000", 2024-09-21T15:49:32.443 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047127251000000002 2024-09-21T15:49:32.444 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.444 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.444 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:32.444 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275049+0000", 2024-09-21T15:49:32.444 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7592000000000001e-05 2024-09-21T15:49:32.444 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.444 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.444 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:32.444 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275138+0000", 2024-09-21T15:49:32.444 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9104000000000006e-05 2024-09-21T15:49:32.444 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:32.444 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:32.444 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:32.444 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.445 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.445 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28542 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 4366336~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:32.445 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.227879+0000", 2024-09-21T15:49:32.445 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.1860244169999996, 2024-09-21T15:49:32.445 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047394002999999997, 2024-09-21T15:49:32.445 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:32.445 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:32.445 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:32.445 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:32.445 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:32.445 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28542 2024-09-21T15:49:32.445 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.445 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:32.445 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.445 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:32.446 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227879+0000", 2024-09-21T15:49:32.446 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:32.446 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.446 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.446 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:32.446 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227879+0000", 2024-09-21T15:49:32.446 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:32.446 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.446 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.446 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:32.446 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227878+0000", 2024-09-21T15:49:32.446 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:32.446 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.447 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.447 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:32.447 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227881+0000", 2024-09-21T15:49:32.447 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2340000000000001e-06 2024-09-21T15:49:32.447 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.447 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.447 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:32.447 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227887+0000", 2024-09-21T15:49:32.447 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.049e-06 2024-09-21T15:49:32.447 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.447 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.447 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:32.447 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227891+0000", 2024-09-21T15:49:32.447 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5480000000000003e-06 2024-09-21T15:49:32.447 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.447 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.448 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:32.448 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275168+0000", 2024-09-21T15:49:32.448 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047276986999999999 2024-09-21T15:49:32.448 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.448 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.448 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:32.448 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275219+0000", 2024-09-21T15:49:32.448 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1010000000000001e-05 2024-09-21T15:49:32.448 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.448 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.448 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:32.448 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275273+0000", 2024-09-21T15:49:32.448 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3649000000000001e-05 2024-09-21T15:49:32.448 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:32.449 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:32.449 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:32.449 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.449 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.449 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28544 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 5111808~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:32.449 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.227895+0000", 2024-09-21T15:49:32.449 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.186008159, 2024-09-21T15:49:32.449 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047502682999999997, 2024-09-21T15:49:32.449 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:32.449 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:32.449 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:32.449 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:32.449 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:32.449 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28544 2024-09-21T15:49:32.450 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.450 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:32.450 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.450 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:32.450 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227895+0000", 2024-09-21T15:49:32.450 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:32.450 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.450 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.450 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:32.450 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227895+0000", 2024-09-21T15:49:32.450 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:32.450 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.450 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.450 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:32.450 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227895+0000", 2024-09-21T15:49:32.451 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:32.451 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.451 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.451 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:32.451 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227897+0000", 2024-09-21T15:49:32.451 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.153e-06 2024-09-21T15:49:32.451 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.451 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.451 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:32.451 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227903+0000", 2024-09-21T15:49:32.451 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1940000000000003e-06 2024-09-21T15:49:32.451 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.451 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.451 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:32.452 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227908+0000", 2024-09-21T15:49:32.452 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6110000000000004e-06 2024-09-21T15:49:32.452 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.452 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.452 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:32.452 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275290+0000", 2024-09-21T15:49:32.452 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047381805999999999 2024-09-21T15:49:32.452 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.452 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.452 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:32.452 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275346+0000", 2024-09-21T15:49:32.452 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6313000000000001e-05 2024-09-21T15:49:32.452 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.452 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.452 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:32.453 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275398+0000", 2024-09-21T15:49:32.453 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2027000000000002e-05 2024-09-21T15:49:32.453 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:32.453 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:32.453 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:32.453 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.453 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.453 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28545 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 5169152~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:32.453 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.227912+0000", 2024-09-21T15:49:32.453 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.1859915379999997, 2024-09-21T15:49:32.453 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047684807000000003, 2024-09-21T15:49:32.453 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:32.453 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:32.453 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:32.454 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:32.454 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:32.454 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28545 2024-09-21T15:49:32.454 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.454 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:32.454 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.454 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:32.454 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227912+0000", 2024-09-21T15:49:32.454 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:32.454 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.454 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.454 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:32.454 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227912+0000", 2024-09-21T15:49:32.454 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:32.455 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.455 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.455 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:32.455 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227911+0000", 2024-09-21T15:49:32.455 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:32.455 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.455 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.455 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:32.455 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227913+0000", 2024-09-21T15:49:32.455 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9970000000000001e-06 2024-09-21T15:49:32.455 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.455 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.455 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:32.455 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227920+0000", 2024-09-21T15:49:32.456 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0499999999999997e-06 2024-09-21T15:49:32.456 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.456 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.456 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:32.456 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227924+0000", 2024-09-21T15:49:32.456 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7219999999999999e-06 2024-09-21T15:49:32.456 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.456 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.456 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:32.456 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275424+0000", 2024-09-21T15:49:32.456 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047499574000000003 2024-09-21T15:49:32.456 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.456 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.456 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:32.457 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275495+0000", 2024-09-21T15:49:32.457 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1347999999999995e-05 2024-09-21T15:49:32.457 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.457 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.457 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:32.457 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275597+0000", 2024-09-21T15:49:32.457 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101446 2024-09-21T15:49:32.457 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:32.457 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:32.457 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:32.457 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.457 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.457 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28546 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 5914624~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:32.457 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.227929+0000", 2024-09-21T15:49:32.458 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.185974292, 2024-09-21T15:49:32.458 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047783297000000002, 2024-09-21T15:49:32.458 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:32.458 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:32.458 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:32.458 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:32.458 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:32.458 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28546 2024-09-21T15:49:32.458 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.458 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:32.458 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.458 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:32.458 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227929+0000", 2024-09-21T15:49:32.458 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:32.459 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.459 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.459 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:32.459 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227929+0000", 2024-09-21T15:49:32.459 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:32.459 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.459 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.459 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:32.459 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227929+0000", 2024-09-21T15:49:32.459 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:32.459 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.459 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.459 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:32.459 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227931+0000", 2024-09-21T15:49:32.459 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2529999999999999e-06 2024-09-21T15:49:32.460 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.460 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.460 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:32.460 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227937+0000", 2024-09-21T15:49:32.460 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9699999999999996e-06 2024-09-21T15:49:32.460 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.460 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.460 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:32.460 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227942+0000", 2024-09-21T15:49:32.460 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7650000000000001e-06 2024-09-21T15:49:32.460 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.460 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.460 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:32.460 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275621+0000", 2024-09-21T15:49:32.460 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047679694000000002 2024-09-21T15:49:32.461 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.461 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.461 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:32.461 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275661+0000", 2024-09-21T15:49:32.461 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9363999999999998e-05 2024-09-21T15:49:32.461 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.461 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.461 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:32.461 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275712+0000", 2024-09-21T15:49:32.461 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1598999999999998e-05 2024-09-21T15:49:32.461 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:32.461 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:32.461 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:32.461 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.462 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.462 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28547 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 6717440~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:32.462 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.228080+0000", 2024-09-21T15:49:32.462 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.1858230150000004, 2024-09-21T15:49:32.462 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047744609, 2024-09-21T15:49:32.462 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:32.462 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:32.462 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:32.462 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:32.462 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:32.462 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28547 2024-09-21T15:49:32.462 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.463 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:32.463 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.463 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:32.463 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228080+0000", 2024-09-21T15:49:32.463 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:32.463 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.463 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.463 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:32.463 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228080+0000", 2024-09-21T15:49:32.463 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:32.463 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.463 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.463 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:32.463 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227946+0000", 2024-09-21T15:49:32.464 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999866 2024-09-21T15:49:32.464 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.464 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.464 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:32.464 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228084+0000", 2024-09-21T15:49:32.464 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000137517 2024-09-21T15:49:32.464 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.464 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.464 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:32.464 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228093+0000", 2024-09-21T15:49:32.464 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.645e-06 2024-09-21T15:49:32.464 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.464 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.464 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:32.464 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228101+0000", 2024-09-21T15:49:32.465 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9559999999999997e-06 2024-09-21T15:49:32.465 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.465 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.465 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:32.465 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275735+0000", 2024-09-21T15:49:32.465 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047633928999999998 2024-09-21T15:49:32.465 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.465 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.465 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:32.465 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275774+0000", 2024-09-21T15:49:32.465 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8699999999999999e-05 2024-09-21T15:49:32.465 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.465 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.465 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:32.465 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275825+0000", 2024-09-21T15:49:32.466 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0926e-05 2024-09-21T15:49:32.466 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:32.466 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:32.466 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:32.466 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.466 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.466 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28549 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 417792~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:32.466 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.228847+0000", 2024-09-21T15:49:32.466 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.18505591, 2024-09-21T15:49:32.466 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047086515000000002, 2024-09-21T15:49:32.466 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:32.466 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:32.466 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:32.467 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:32.467 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:32.467 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28549 2024-09-21T15:49:32.467 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.467 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:32.467 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.467 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:32.467 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228847+0000", 2024-09-21T15:49:32.467 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:32.467 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.467 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.467 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:32.467 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228847+0000", 2024-09-21T15:49:32.467 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:32.468 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.468 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.468 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:32.468 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228107+0000", 2024-09-21T15:49:32.468 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9992595 2024-09-21T15:49:32.468 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.468 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.468 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:32.468 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228852+0000", 2024-09-21T15:49:32.468 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00074505000000000003 2024-09-21T15:49:32.468 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.468 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.468 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:32.468 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228871+0000", 2024-09-21T15:49:32.469 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8635e-05 2024-09-21T15:49:32.469 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.469 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.469 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:32.469 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228880+0000", 2024-09-21T15:49:32.469 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9849999999999995e-06 2024-09-21T15:49:32.469 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.469 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.469 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:32.469 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275847+0000", 2024-09-21T15:49:32.470 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.046967239000000001 2024-09-21T15:49:32.470 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.470 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.470 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:32.470 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275885+0000", 2024-09-21T15:49:32.470 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8274e-05 2024-09-21T15:49:32.470 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.470 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.470 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:32.470 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275934+0000", 2024-09-21T15:49:32.470 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8764999999999998e-05 2024-09-21T15:49:32.471 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:32.471 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:32.471 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:32.471 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.471 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.471 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28551 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 1220608~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:32.471 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.228886+0000", 2024-09-21T15:49:32.471 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.1850168639999996, 2024-09-21T15:49:32.471 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047166819999999998, 2024-09-21T15:49:32.471 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:32.471 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:32.471 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:32.472 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:32.472 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:32.472 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28551 2024-09-21T15:49:32.472 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.472 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:32.472 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.472 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:32.472 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228886+0000", 2024-09-21T15:49:32.472 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:32.472 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.472 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.472 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:32.472 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228886+0000", 2024-09-21T15:49:32.473 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:32.473 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.473 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.473 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:32.473 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228886+0000", 2024-09-21T15:49:32.473 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:32.473 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.473 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.473 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:32.473 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228889+0000", 2024-09-21T15:49:32.473 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8650000000000001e-06 2024-09-21T15:49:32.473 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.473 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.473 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:32.474 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228896+0000", 2024-09-21T15:49:32.474 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0790000000000003e-06 2024-09-21T15:49:32.474 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.474 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.474 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:32.474 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228903+0000", 2024-09-21T15:49:32.474 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9770000000000003e-06 2024-09-21T15:49:32.474 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.474 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.474 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:32.474 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275954+0000", 2024-09-21T15:49:32.474 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047050986000000003 2024-09-21T15:49:32.474 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.475 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.475 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:32.475 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276001+0000", 2024-09-21T15:49:32.475 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7527000000000001e-05 2024-09-21T15:49:32.475 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.475 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.475 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:32.475 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276053+0000", 2024-09-21T15:49:32.475 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1854999999999997e-05 2024-09-21T15:49:32.475 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:32.475 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:32.475 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:32.475 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.475 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.476 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28552 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 1966080~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:32.476 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.228908+0000", 2024-09-21T15:49:32.476 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.1849955789999997, 2024-09-21T15:49:32.476 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047256193000000002, 2024-09-21T15:49:32.476 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:32.476 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:32.476 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:32.476 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:32.476 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:32.476 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28552 2024-09-21T15:49:32.476 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.476 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:32.476 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.477 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:32.477 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228908+0000", 2024-09-21T15:49:32.477 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:32.477 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.477 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.477 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:32.477 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228908+0000", 2024-09-21T15:49:32.477 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:32.477 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.477 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.477 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:32.477 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228907+0000", 2024-09-21T15:49:32.477 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:32.477 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.477 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.478 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:32.478 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228910+0000", 2024-09-21T15:49:32.478 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3819999999999998e-06 2024-09-21T15:49:32.478 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.478 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.478 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:32.478 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228916+0000", 2024-09-21T15:49:32.478 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6390000000000003e-06 2024-09-21T15:49:32.478 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.478 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.478 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:32.478 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228921+0000", 2024-09-21T15:49:32.478 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0350000000000001e-06 2024-09-21T15:49:32.479 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.479 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.479 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:32.479 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276077+0000", 2024-09-21T15:49:32.479 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047155326999999997 2024-09-21T15:49:32.479 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.479 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.479 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:32.479 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276114+0000", 2024-09-21T15:49:32.479 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6746999999999999e-05 2024-09-21T15:49:32.479 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.479 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.479 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:32.480 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276164+0000", 2024-09-21T15:49:32.480 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0438000000000001e-05 2024-09-21T15:49:32.480 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:32.480 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:32.480 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:32.480 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.480 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.480 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28553 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 2023424~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:32.480 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.228926+0000", 2024-09-21T15:49:32.480 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.1849773729999997, 2024-09-21T15:49:32.480 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047351920999999998, 2024-09-21T15:49:32.480 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:32.480 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:32.480 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:32.481 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:32.481 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:32.481 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28553 2024-09-21T15:49:32.481 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.481 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:32.481 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.481 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:32.481 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228926+0000", 2024-09-21T15:49:32.481 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:32.481 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.481 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.481 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:32.481 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228926+0000", 2024-09-21T15:49:32.481 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:32.482 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.482 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.482 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:32.482 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228926+0000", 2024-09-21T15:49:32.482 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:32.482 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.482 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.482 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:32.482 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228932+0000", 2024-09-21T15:49:32.482 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6930000000000001e-06 2024-09-21T15:49:32.482 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.482 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.482 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:32.482 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228939+0000", 2024-09-21T15:49:32.483 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8199999999999999e-06 2024-09-21T15:49:32.483 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.483 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.483 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:32.483 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228953+0000", 2024-09-21T15:49:32.483 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.347e-05 2024-09-21T15:49:32.483 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.483 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.483 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:32.483 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276187+0000", 2024-09-21T15:49:32.483 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047234405 2024-09-21T15:49:32.483 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.483 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.483 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:32.484 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276228+0000", 2024-09-21T15:49:32.484 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1010000000000002e-05 2024-09-21T15:49:32.484 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.484 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.484 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:32.484 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276278+0000", 2024-09-21T15:49:32.484 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9874999999999999e-05 2024-09-21T15:49:32.484 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:32.484 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:32.484 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:32.484 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.484 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.484 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28554 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 2768896~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:32.485 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.228991+0000", 2024-09-21T15:49:32.485 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.184911896, 2024-09-21T15:49:32.485 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047394177000000003, 2024-09-21T15:49:32.485 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:32.485 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:32.485 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:32.485 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:32.485 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:32.485 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28554 2024-09-21T15:49:32.485 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.485 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:32.485 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.485 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:32.485 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228991+0000", 2024-09-21T15:49:32.486 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:32.486 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.486 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.486 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:32.486 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228991+0000", 2024-09-21T15:49:32.486 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:32.486 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.486 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.486 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:32.486 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228991+0000", 2024-09-21T15:49:32.486 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:32.486 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.486 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.486 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:32.487 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228994+0000", 2024-09-21T15:49:32.487 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.422e-06 2024-09-21T15:49:32.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.487 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.487 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:32.487 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229003+0000", 2024-09-21T15:49:32.487 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8699999999999998e-06 2024-09-21T15:49:32.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.487 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.487 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:32.488 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229009+0000", 2024-09-21T15:49:32.488 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3099999999999997e-06 2024-09-21T15:49:32.488 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.488 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.488 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:32.488 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276301+0000", 2024-09-21T15:49:32.488 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047291305999999998 2024-09-21T15:49:32.488 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.488 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.488 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:32.489 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276337+0000", 2024-09-21T15:49:32.489 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6378999999999998e-05 2024-09-21T15:49:32.489 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.489 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.489 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:32.489 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276386+0000", 2024-09-21T15:49:32.489 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8535999999999997e-05 2024-09-21T15:49:32.489 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:32.489 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:32.489 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:32.489 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.489 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.489 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28555 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 3571712~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:32.489 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.229015+0000", 2024-09-21T15:49:32.490 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.1848888019999997, 2024-09-21T15:49:32.490 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047478713999999998, 2024-09-21T15:49:32.490 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:32.490 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:32.490 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:32.490 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:32.490 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:32.490 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28555 2024-09-21T15:49:32.490 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.490 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:32.490 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.490 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:32.490 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229015+0000", 2024-09-21T15:49:32.491 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:32.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.491 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:32.491 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229015+0000", 2024-09-21T15:49:32.491 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:32.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.491 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:32.491 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229014+0000", 2024-09-21T15:49:32.491 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:32.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.491 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:32.492 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229029+0000", 2024-09-21T15:49:32.492 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4503e-05 2024-09-21T15:49:32.492 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.492 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.492 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:32.492 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229036+0000", 2024-09-21T15:49:32.492 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5989999999999996e-06 2024-09-21T15:49:32.492 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.492 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.492 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:32.492 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229042+0000", 2024-09-21T15:49:32.492 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6699999999999999e-06 2024-09-21T15:49:32.492 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.492 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.493 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:32.493 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276407+0000", 2024-09-21T15:49:32.493 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047365646999999997 2024-09-21T15:49:32.493 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.493 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.493 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:32.493 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276445+0000", 2024-09-21T15:49:32.493 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7064000000000003e-05 2024-09-21T15:49:32.493 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.493 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.493 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:32.493 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276493+0000", 2024-09-21T15:49:32.493 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8704e-05 2024-09-21T15:49:32.493 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:32.494 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:32.494 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:32.494 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.494 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.494 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28556 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 4374528~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:32.494 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.229183+0000", 2024-09-21T15:49:32.494 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.1847200109999996, 2024-09-21T15:49:32.494 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047417709000000002, 2024-09-21T15:49:32.494 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:32.494 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:32.494 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:32.494 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:32.494 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:32.494 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28556 2024-09-21T15:49:32.495 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.495 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:32.495 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.495 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:32.495 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229183+0000", 2024-09-21T15:49:32.495 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:32.495 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.495 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.495 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:32.495 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229183+0000", 2024-09-21T15:49:32.495 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:32.495 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.495 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.495 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:32.496 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229046+0000", 2024-09-21T15:49:32.496 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9998631 2024-09-21T15:49:32.496 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.496 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.496 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:32.496 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229186+0000", 2024-09-21T15:49:32.496 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014003900000000001 2024-09-21T15:49:32.496 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.496 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.496 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:32.496 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229195+0000", 2024-09-21T15:49:32.496 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.422e-06 2024-09-21T15:49:32.496 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.496 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.496 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:32.496 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229199+0000", 2024-09-21T15:49:32.497 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5379999999999996e-06 2024-09-21T15:49:32.497 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.497 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.497 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:32.497 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276516+0000", 2024-09-21T15:49:32.497 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047316432999999998 2024-09-21T15:49:32.497 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.497 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.497 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:32.497 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276552+0000", 2024-09-21T15:49:32.497 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6019000000000002e-05 2024-09-21T15:49:32.497 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.497 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.497 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:32.498 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276601+0000", 2024-09-21T15:49:32.498 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9172999999999998e-05 2024-09-21T15:49:32.498 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:32.498 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:32.498 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:32.498 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.498 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.498 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28558 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 5120000~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:32.498 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.229531+0000", 2024-09-21T15:49:32.498 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.1843721279999997, 2024-09-21T15:49:32.498 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047179666000000002, 2024-09-21T15:49:32.498 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:32.498 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:32.499 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:32.499 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:32.499 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:32.499 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28558 2024-09-21T15:49:32.499 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.499 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:32.499 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.499 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:32.499 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229531+0000", 2024-09-21T15:49:32.499 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:32.499 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.499 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.499 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:32.500 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229531+0000", 2024-09-21T15:49:32.500 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:32.500 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.500 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.500 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:32.500 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229203+0000", 2024-09-21T15:49:32.500 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9996719 2024-09-21T15:49:32.500 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.500 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.500 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:32.500 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229536+0000", 2024-09-21T15:49:32.500 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000332885 2024-09-21T15:49:32.500 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.500 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.500 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:32.501 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229548+0000", 2024-09-21T15:49:32.501 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1861999999999999e-05 2024-09-21T15:49:32.501 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.501 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.501 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:32.501 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229555+0000", 2024-09-21T15:49:32.501 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6009999999999997e-06 2024-09-21T15:49:32.501 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.501 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.501 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:32.501 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276624+0000", 2024-09-21T15:49:32.501 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047068250999999998 2024-09-21T15:49:32.501 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.501 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.502 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:32.502 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276662+0000", 2024-09-21T15:49:32.502 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8705999999999998e-05 2024-09-21T15:49:32.502 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.502 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.502 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:32.502 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276711+0000", 2024-09-21T15:49:32.502 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8443999999999999e-05 2024-09-21T15:49:32.502 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:32.502 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:32.502 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:32.502 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.502 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.503 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28560 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 5922816~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:32.503 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.229562+0000", 2024-09-21T15:49:32.503 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.1843409229999997, 2024-09-21T15:49:32.503 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047255377000000001, 2024-09-21T15:49:32.503 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:32.503 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:32.503 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:32.503 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:32.503 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:32.503 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28560 2024-09-21T15:49:32.503 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.503 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:32.503 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.503 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:32.504 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229562+0000", 2024-09-21T15:49:32.504 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:32.504 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.504 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.504 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:32.504 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229562+0000", 2024-09-21T15:49:32.504 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:32.504 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.504 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.504 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:32.504 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229562+0000", 2024-09-21T15:49:32.504 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:32.504 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.505 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.505 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:32.505 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229565+0000", 2024-09-21T15:49:32.505 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9450000000000002e-06 2024-09-21T15:49:32.505 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.505 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.505 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:32.505 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229572+0000", 2024-09-21T15:49:32.505 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4240000000000004e-06 2024-09-21T15:49:32.505 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.505 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.505 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:32.505 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229577+0000", 2024-09-21T15:49:32.506 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2000000000000002e-06 2024-09-21T15:49:32.506 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.506 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.506 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:32.506 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276733+0000", 2024-09-21T15:49:32.506 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047155202 2024-09-21T15:49:32.506 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.506 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.506 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:32.506 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276769+0000", 2024-09-21T15:49:32.506 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6013000000000002e-05 2024-09-21T15:49:32.506 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:32.506 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:32.506 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:32.507 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276818+0000", 2024-09-21T15:49:32.507 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9178000000000002e-05 2024-09-21T15:49:32.507 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:32.507 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:32.507 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:32.507 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:32.507 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:32.507 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:32.509 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:49:32.509 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:32.507+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:32.609 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:32.608+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:32.611 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:32.611 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:32.611 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:49:32.611 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:32.620 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:49:32.710 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:32.709+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:32.721 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:32.721 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:32.721 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:49:32.721 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:49:32.721 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:32.729 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:49:32.779 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:49:32.808 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T15:49:32.813 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:32.812+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:32.879 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:32.879 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:49:32.879 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:49:32.879 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:49:32.879 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.879 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26224 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 2326528~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:32.879 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.504487+0000", 2024-09-21T15:49:32.879 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 6.3725206610000003, 2024-09-21T15:49:32.879 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032819441999999997, 2024-09-21T15:49:32.879 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:32.879 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:32.879 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:32.880 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:32.880 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:32.880 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26224 2024-09-21T15:49:32.880 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.880 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:32.880 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.880 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:32.880 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504487+0000", 2024-09-21T15:49:32.880 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:32.880 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.880 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.880 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:32.880 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504487+0000", 2024-09-21T15:49:32.880 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:32.881 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.881 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.881 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:32.881 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504487+0000", 2024-09-21T15:49:32.881 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:32.881 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.881 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.881 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:32.881 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504490+0000", 2024-09-21T15:49:32.881 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6049999999999998e-06 2024-09-21T15:49:32.881 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.881 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.881 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:32.881 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504493+0000", 2024-09-21T15:49:32.881 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1200000000000002e-06 2024-09-21T15:49:32.882 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.882 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.882 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:32.882 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504496+0000", 2024-09-21T15:49:32.882 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.977e-06 2024-09-21T15:49:32.882 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.882 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.882 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:32.882 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537221+0000", 2024-09-21T15:49:32.882 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032724529000000002 2024-09-21T15:49:32.882 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.882 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.882 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:32.883 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537259+0000", 2024-09-21T15:49:32.883 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7923999999999998e-05 2024-09-21T15:49:32.883 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.883 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.883 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:32.883 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537306+0000", 2024-09-21T15:49:32.883 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7512999999999997e-05 2024-09-21T15:49:32.883 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:32.883 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:32.883 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:32.883 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.883 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.883 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26225 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 2383872~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:32.883 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.504498+0000", 2024-09-21T15:49:32.884 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 6.3725090919999996, 2024-09-21T15:49:32.884 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.03291703, 2024-09-21T15:49:32.884 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:32.884 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:32.884 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:32.884 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:32.884 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:32.884 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26225 2024-09-21T15:49:32.884 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.884 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:32.884 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.884 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:32.884 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504498+0000", 2024-09-21T15:49:32.885 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:32.885 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.885 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.885 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:32.885 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504498+0000", 2024-09-21T15:49:32.885 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:32.885 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.885 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.885 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:32.885 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504498+0000", 2024-09-21T15:49:32.885 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:32.885 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.885 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.885 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:32.886 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504499+0000", 2024-09-21T15:49:32.886 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0070000000000001e-06 2024-09-21T15:49:32.886 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.886 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.886 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:32.886 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504502+0000", 2024-09-21T15:49:32.886 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8119999999999999e-06 2024-09-21T15:49:32.886 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.886 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.886 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:32.886 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504504+0000", 2024-09-21T15:49:32.886 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4380000000000002e-06 2024-09-21T15:49:32.886 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.886 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.887 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:32.887 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537328+0000", 2024-09-21T15:49:32.887 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032823659999999998 2024-09-21T15:49:32.887 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.887 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.887 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:32.887 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537366+0000", 2024-09-21T15:49:32.887 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8083000000000001e-05 2024-09-21T15:49:32.887 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.887 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.887 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:32.887 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537415+0000", 2024-09-21T15:49:32.887 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9233000000000001e-05 2024-09-21T15:49:32.888 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:32.888 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:32.888 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:32.888 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.888 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.888 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26226 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 2441216~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:32.888 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.504506+0000", 2024-09-21T15:49:32.888 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 6.3725009769999996, 2024-09-21T15:49:32.888 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033016726000000003, 2024-09-21T15:49:32.888 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:32.888 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:32.888 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:32.888 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:32.888 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:32.888 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26226 2024-09-21T15:49:32.889 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.889 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:32.889 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.889 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:32.889 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504506+0000", 2024-09-21T15:49:32.889 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:32.889 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.889 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.889 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:32.889 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504506+0000", 2024-09-21T15:49:32.889 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:32.889 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.889 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.889 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:32.889 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504506+0000", 2024-09-21T15:49:32.890 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:32.890 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.890 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.890 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:32.890 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504507+0000", 2024-09-21T15:49:32.890 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9699999999999994e-07 2024-09-21T15:49:32.890 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.890 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.890 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:32.890 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504510+0000", 2024-09-21T15:49:32.890 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.773e-06 2024-09-21T15:49:32.890 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.890 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.891 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:32.891 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504514+0000", 2024-09-21T15:49:32.891 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1150000000000004e-06 2024-09-21T15:49:32.891 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.891 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.891 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:32.891 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537437+0000", 2024-09-21T15:49:32.891 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032923249000000002 2024-09-21T15:49:32.891 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.891 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.891 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:32.891 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537476+0000", 2024-09-21T15:49:32.891 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8679e-05 2024-09-21T15:49:32.891 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.892 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.892 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:32.892 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537523+0000", 2024-09-21T15:49:32.892 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7119000000000001e-05 2024-09-21T15:49:32.892 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:32.892 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:32.892 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:32.892 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.892 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.892 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26227 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 2498560~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:32.892 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.504516+0000", 2024-09-21T15:49:32.892 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 6.3724913760000002, 2024-09-21T15:49:32.892 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033117681000000003, 2024-09-21T15:49:32.892 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:32.892 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:32.893 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:32.893 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:32.893 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:32.893 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26227 2024-09-21T15:49:32.893 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.893 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:32.893 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.893 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:32.893 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504516+0000", 2024-09-21T15:49:32.893 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:32.893 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.893 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.893 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:32.893 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504516+0000", 2024-09-21T15:49:32.894 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:32.894 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.894 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.894 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:32.894 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504516+0000", 2024-09-21T15:49:32.894 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:32.894 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.894 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.894 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:32.894 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504517+0000", 2024-09-21T15:49:32.894 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0610000000000001e-06 2024-09-21T15:49:32.894 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.894 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.894 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:32.895 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504520+0000", 2024-09-21T15:49:32.895 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.83e-06 2024-09-21T15:49:32.895 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.895 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.895 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:32.895 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504522+0000", 2024-09-21T15:49:32.895 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.497e-06 2024-09-21T15:49:32.895 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.895 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.895 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:32.895 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537544+0000", 2024-09-21T15:49:32.895 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033021948000000002 2024-09-21T15:49:32.895 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.896 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.896 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:32.896 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537585+0000", 2024-09-21T15:49:32.896 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1094999999999999e-05 2024-09-21T15:49:32.896 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.896 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.896 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:32.896 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537634+0000", 2024-09-21T15:49:32.896 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8449000000000003e-05 2024-09-21T15:49:32.896 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:32.896 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:32.896 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:32.896 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.896 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.897 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26228 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 2555904~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:32.897 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.504524+0000", 2024-09-21T15:49:32.897 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 6.3724832779999998, 2024-09-21T15:49:32.897 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.03321735, 2024-09-21T15:49:32.897 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:32.897 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:32.897 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:32.897 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:32.897 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:32.897 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26228 2024-09-21T15:49:32.897 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.897 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:32.897 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.898 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:32.898 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504524+0000", 2024-09-21T15:49:32.898 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:32.898 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.898 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.898 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:32.898 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504524+0000", 2024-09-21T15:49:32.898 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:32.898 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.898 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.898 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:32.898 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504524+0000", 2024-09-21T15:49:32.898 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:32.898 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.899 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.899 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:32.899 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504525+0000", 2024-09-21T15:49:32.899 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0419999999999999e-06 2024-09-21T15:49:32.899 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.899 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.899 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:32.899 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504528+0000", 2024-09-21T15:49:32.899 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7530000000000002e-06 2024-09-21T15:49:32.899 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.899 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.899 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:32.899 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504530+0000", 2024-09-21T15:49:32.899 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4530000000000001e-06 2024-09-21T15:49:32.900 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.900 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.900 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:32.900 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537656+0000", 2024-09-21T15:49:32.900 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033126023999999997 2024-09-21T15:49:32.900 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.900 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.900 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:32.900 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537694+0000", 2024-09-21T15:49:32.900 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7799000000000002e-05 2024-09-21T15:49:32.900 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.900 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.900 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:32.900 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537741+0000", 2024-09-21T15:49:32.901 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7484999999999997e-05 2024-09-21T15:49:32.901 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:32.901 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:32.901 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:32.901 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.901 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.901 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26229 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 2613248~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:32.901 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.504532+0000", 2024-09-21T15:49:32.901 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 6.3724753989999998, 2024-09-21T15:49:32.901 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033315897999999997, 2024-09-21T15:49:32.901 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:32.901 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:32.901 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:32.902 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:32.902 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:32.902 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26229 2024-09-21T15:49:32.902 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.902 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:32.902 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.902 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:32.902 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504532+0000", 2024-09-21T15:49:32.902 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:32.902 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.902 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.902 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:32.902 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504532+0000", 2024-09-21T15:49:32.903 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:32.903 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.903 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.903 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:32.903 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504532+0000", 2024-09-21T15:49:32.903 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:32.903 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.903 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.903 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:32.903 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504533+0000", 2024-09-21T15:49:32.903 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0610000000000001e-06 2024-09-21T15:49:32.903 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.903 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.904 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:32.904 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504536+0000", 2024-09-21T15:49:32.904 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.309e-06 2024-09-21T15:49:32.904 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.904 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.904 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:32.904 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504539+0000", 2024-09-21T15:49:32.904 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.435e-06 2024-09-21T15:49:32.904 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.904 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.904 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:32.904 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537762+0000", 2024-09-21T15:49:32.904 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033223813999999997 2024-09-21T15:49:32.904 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.905 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.905 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:32.905 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537801+0000", 2024-09-21T15:49:32.905 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8136999999999998e-05 2024-09-21T15:49:32.905 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.905 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.905 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:32.905 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537848+0000", 2024-09-21T15:49:32.905 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7345999999999997e-05 2024-09-21T15:49:32.905 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:32.905 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:32.905 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:32.905 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.905 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.906 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26230 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 2670592~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:32.906 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.504606+0000", 2024-09-21T15:49:32.906 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 6.3724012410000004, 2024-09-21T15:49:32.906 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033348801999999997, 2024-09-21T15:49:32.906 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:32.906 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:32.906 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:32.906 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:32.906 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:32.906 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26230 2024-09-21T15:49:32.906 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.906 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:32.906 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.907 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:32.907 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504606+0000", 2024-09-21T15:49:32.907 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:32.907 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.907 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.907 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:32.907 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504606+0000", 2024-09-21T15:49:32.907 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:32.907 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.907 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.907 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:32.907 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504541+0000", 2024-09-21T15:49:32.907 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999352 2024-09-21T15:49:32.908 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.908 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.908 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:32.908 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504608+0000", 2024-09-21T15:49:32.908 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6427000000000006e-05 2024-09-21T15:49:32.908 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.908 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.908 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:32.908 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504612+0000", 2024-09-21T15:49:32.908 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1799999999999998e-06 2024-09-21T15:49:32.908 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.908 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.908 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:32.908 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504615+0000", 2024-09-21T15:49:32.909 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0400000000000001e-06 2024-09-21T15:49:32.909 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.909 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.909 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:32.909 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537870+0000", 2024-09-21T15:49:32.909 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033254745000000002 2024-09-21T15:49:32.909 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.909 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.909 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:32.909 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537908+0000", 2024-09-21T15:49:32.909 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7882000000000001e-05 2024-09-21T15:49:32.909 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.909 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.909 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:32.910 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537955+0000", 2024-09-21T15:49:32.910 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7361999999999998e-05 2024-09-21T15:49:32.910 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:32.910 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:32.910 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:32.910 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.910 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.910 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26239 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 3186688~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:32.910 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.506097+0000", 2024-09-21T15:49:32.910 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 6.3709105350000002, 2024-09-21T15:49:32.910 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032843331000000003, 2024-09-21T15:49:32.910 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:32.910 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:32.910 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:32.911 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:32.911 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:32.911 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26239 2024-09-21T15:49:32.911 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.911 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:32.911 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.911 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:32.911 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506097+0000", 2024-09-21T15:49:32.911 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:32.911 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.911 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.911 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:32.911 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506097+0000", 2024-09-21T15:49:32.911 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:32.911 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.912 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.912 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:32.912 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506097+0000", 2024-09-21T15:49:32.912 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:32.912 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.912 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.912 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:32.912 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506098+0000", 2024-09-21T15:49:32.912 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.079e-06 2024-09-21T15:49:32.912 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.912 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.912 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:32.912 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506101+0000", 2024-09-21T15:49:32.913 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8499999999999998e-06 2024-09-21T15:49:32.913 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.913 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.913 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:32.913 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506103+0000", 2024-09-21T15:49:32.913 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5770000000000001e-06 2024-09-21T15:49:32.913 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.913 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.913 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:32.913 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.538854+0000", 2024-09-21T15:49:32.913 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032750835999999998 2024-09-21T15:49:32.913 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.913 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.913 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:32.914 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.538892+0000", 2024-09-21T15:49:32.914 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7904000000000002e-05 2024-09-21T15:49:32.914 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.914 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.914 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:32.914 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.538940+0000", 2024-09-21T15:49:32.914 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8303999999999997e-05 2024-09-21T15:49:32.914 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:32.914 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:32.914 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:32.914 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.914 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.914 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26240 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 3244032~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:32.915 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.506106+0000", 2024-09-21T15:49:32.915 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 6.3709014030000004, 2024-09-21T15:49:32.915 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032955696, 2024-09-21T15:49:32.915 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:32.915 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:32.915 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:32.915 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:32.915 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:32.915 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26240 2024-09-21T15:49:32.915 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.915 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:32.915 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.915 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:32.915 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506106+0000", 2024-09-21T15:49:32.916 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:32.916 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.916 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.916 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:32.916 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506106+0000", 2024-09-21T15:49:32.916 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:32.916 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.916 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.916 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:32.916 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506106+0000", 2024-09-21T15:49:32.916 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:32.916 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.916 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.916 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:32.916 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506107+0000", 2024-09-21T15:49:32.917 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1179999999999999e-06 2024-09-21T15:49:32.917 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.917 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.917 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:32.917 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506110+0000", 2024-09-21T15:49:32.917 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0259999999999999e-06 2024-09-21T15:49:32.917 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.917 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.917 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:32.917 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506112+0000", 2024-09-21T15:49:32.917 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5890000000000001e-06 2024-09-21T15:49:32.917 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.917 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.918 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:32.918 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.538962+0000", 2024-09-21T15:49:32.919 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032849861000000001 2024-09-21T15:49:32.919 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.919 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.919 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:32.919 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539003+0000", 2024-09-21T15:49:32.919 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0756000000000001e-05 2024-09-21T15:49:32.919 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.919 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.919 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:32.919 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539062+0000", 2024-09-21T15:49:32.919 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8597999999999997e-05 2024-09-21T15:49:32.919 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:32.919 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:32.919 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:32.920 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.920 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.920 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26241 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 3301376~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:32.920 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.506128+0000", 2024-09-21T15:49:32.920 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 6.3708792369999996, 2024-09-21T15:49:32.920 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033043453, 2024-09-21T15:49:32.920 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:32.920 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:32.920 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:32.920 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:32.920 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:32.920 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26241 2024-09-21T15:49:32.920 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.920 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:32.920 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.921 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:32.921 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506128+0000", 2024-09-21T15:49:32.921 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:32.921 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.921 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.921 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:32.921 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506128+0000", 2024-09-21T15:49:32.921 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:32.921 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.921 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.921 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:32.921 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506128+0000", 2024-09-21T15:49:32.921 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:32.921 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.922 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.922 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:32.922 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506129+0000", 2024-09-21T15:49:32.922 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7290000000000001e-06 2024-09-21T15:49:32.922 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.922 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.922 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:32.922 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506132+0000", 2024-09-21T15:49:32.922 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9919999999999999e-06 2024-09-21T15:49:32.922 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.923 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.923 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:32.923 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506135+0000", 2024-09-21T15:49:32.923 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4090000000000001e-06 2024-09-21T15:49:32.923 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.923 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.923 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:32.923 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539084+0000", 2024-09-21T15:49:32.923 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032949231000000002 2024-09-21T15:49:32.923 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.923 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.923 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:32.923 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539124+0000", 2024-09-21T15:49:32.923 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9421000000000001e-05 2024-09-21T15:49:32.924 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.924 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.924 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:32.924 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539172+0000", 2024-09-21T15:49:32.924 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8068000000000001e-05 2024-09-21T15:49:32.924 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:32.924 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:32.924 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:32.924 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.924 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.924 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26242 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 3358720~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:32.924 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.506137+0000", 2024-09-21T15:49:32.924 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 6.3708703030000002, 2024-09-21T15:49:32.924 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033142360000000003, 2024-09-21T15:49:32.925 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:32.925 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:32.925 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:32.925 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:32.925 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:32.925 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26242 2024-09-21T15:49:32.925 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.925 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:32.925 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.925 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:32.925 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506137+0000", 2024-09-21T15:49:32.925 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:32.925 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.925 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.925 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:32.926 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506137+0000", 2024-09-21T15:49:32.926 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:32.926 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.926 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.926 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:32.926 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506137+0000", 2024-09-21T15:49:32.926 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:32.926 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.926 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.926 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:32.926 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506138+0000", 2024-09-21T15:49:32.926 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2330000000000001e-06 2024-09-21T15:49:32.926 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.927 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.927 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:32.927 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506141+0000", 2024-09-21T15:49:32.927 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7999999999999999e-06 2024-09-21T15:49:32.927 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.927 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.927 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:32.927 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506143+0000", 2024-09-21T15:49:32.927 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5919999999999999e-06 2024-09-21T15:49:32.927 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.927 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.927 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:32.927 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539193+0000", 2024-09-21T15:49:32.927 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033049229999999999 2024-09-21T15:49:32.927 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.928 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.928 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:32.928 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539232+0000", 2024-09-21T15:49:32.928 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9743999999999998e-05 2024-09-21T15:49:32.928 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.928 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.928 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:32.928 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539279+0000", 2024-09-21T15:49:32.928 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7039999999999997e-05 2024-09-21T15:49:32.928 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:32.928 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:32.928 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:32.928 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.928 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.929 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26243 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 3416064~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:32.929 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.506145+0000", 2024-09-21T15:49:32.929 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 6.3708620180000004, 2024-09-21T15:49:32.929 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033240933, 2024-09-21T15:49:32.929 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:32.929 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:32.929 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:32.929 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:32.929 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:32.929 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26243 2024-09-21T15:49:32.929 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.929 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:32.929 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.929 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:32.929 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506145+0000", 2024-09-21T15:49:32.930 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:32.930 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.930 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.930 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:32.930 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506145+0000", 2024-09-21T15:49:32.930 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:32.930 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.930 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.930 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:32.930 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506145+0000", 2024-09-21T15:49:32.930 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:32.930 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.930 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.930 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:32.930 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506146+0000", 2024-09-21T15:49:32.931 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0279999999999999e-06 2024-09-21T15:49:32.931 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.931 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.931 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:32.931 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506149+0000", 2024-09-21T15:49:32.931 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2449999999999998e-06 2024-09-21T15:49:32.931 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.931 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.931 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:32.931 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506152+0000", 2024-09-21T15:49:32.931 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5050000000000002e-06 2024-09-21T15:49:32.931 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.931 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.931 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:32.931 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539301+0000", 2024-09-21T15:49:32.932 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033149093999999997 2024-09-21T15:49:32.932 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.932 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.932 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:32.932 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539339+0000", 2024-09-21T15:49:32.932 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7685999999999998e-05 2024-09-21T15:49:32.932 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.932 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.932 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:32.932 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539386+0000", 2024-09-21T15:49:32.932 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7602000000000003e-05 2024-09-21T15:49:32.932 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:32.932 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:32.932 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:32.933 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.933 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.933 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26258 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 4276224~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:32.933 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.508167+0000", 2024-09-21T15:49:32.933 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 6.3688400999999999, 2024-09-21T15:49:32.933 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032934857999999997, 2024-09-21T15:49:32.933 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:32.933 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:32.933 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:32.933 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:32.933 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:32.933 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26258 2024-09-21T15:49:32.933 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.933 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:32.934 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.934 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:32.934 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508167+0000", 2024-09-21T15:49:32.934 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:32.934 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.934 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.934 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:32.934 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508167+0000", 2024-09-21T15:49:32.934 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:32.934 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.934 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.934 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:32.934 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508167+0000", 2024-09-21T15:49:32.935 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:32.935 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.935 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.935 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:32.935 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508168+0000", 2024-09-21T15:49:32.935 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.088e-06 2024-09-21T15:49:32.935 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.935 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.935 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:32.935 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508171+0000", 2024-09-21T15:49:32.935 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7080000000000002e-06 2024-09-21T15:49:32.935 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.935 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.935 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:32.936 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508173+0000", 2024-09-21T15:49:32.936 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4030000000000001e-06 2024-09-21T15:49:32.936 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.936 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.936 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:32.936 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.540936+0000", 2024-09-21T15:49:32.936 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032762279999999998 2024-09-21T15:49:32.936 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.936 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.936 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:32.936 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.541007+0000", 2024-09-21T15:49:32.936 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1335999999999996e-05 2024-09-21T15:49:32.936 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.936 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.937 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:32.937 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.541102+0000", 2024-09-21T15:49:32.937 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5259000000000004e-05 2024-09-21T15:49:32.937 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:32.937 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:32.937 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:32.937 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.937 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.937 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26259 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 4333568~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:32.937 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.508175+0000", 2024-09-21T15:49:32.937 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 6.368832158, 2024-09-21T15:49:32.937 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033044845000000003, 2024-09-21T15:49:32.937 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:32.937 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:32.938 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:32.938 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:32.938 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:32.938 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26259 2024-09-21T15:49:32.938 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.938 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:32.938 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.938 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:32.938 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508175+0000", 2024-09-21T15:49:32.938 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:32.938 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.938 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.938 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:32.939 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508175+0000", 2024-09-21T15:49:32.939 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:32.939 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.939 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.939 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:32.939 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508175+0000", 2024-09-21T15:49:32.939 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:32.939 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.939 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.939 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:32.939 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508176+0000", 2024-09-21T15:49:32.939 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1039999999999999e-06 2024-09-21T15:49:32.939 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.939 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.940 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:32.940 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508179+0000", 2024-09-21T15:49:32.940 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.689e-06 2024-09-21T15:49:32.940 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.940 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.940 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:32.940 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508181+0000", 2024-09-21T15:49:32.940 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5610000000000001e-06 2024-09-21T15:49:32.940 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.940 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.940 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:32.940 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.541126+0000", 2024-09-21T15:49:32.940 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032944284999999997 2024-09-21T15:49:32.940 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.940 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.941 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:32.941 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.541168+0000", 2024-09-21T15:49:32.941 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2565000000000003e-05 2024-09-21T15:49:32.941 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.941 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.941 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:32.941 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.541220+0000", 2024-09-21T15:49:32.941 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1863000000000001e-05 2024-09-21T15:49:32.941 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:32.941 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:32.941 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:32.941 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.941 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.941 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26260 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 4390912~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:32.942 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.508183+0000", 2024-09-21T15:49:32.942 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 6.368824043, 2024-09-21T15:49:32.942 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033150304999999998, 2024-09-21T15:49:32.942 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:32.942 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:32.942 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:32.942 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:32.942 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:32.942 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26260 2024-09-21T15:49:32.942 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.942 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:32.942 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.942 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:32.943 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508183+0000", 2024-09-21T15:49:32.943 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:32.943 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.943 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.943 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:32.943 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508183+0000", 2024-09-21T15:49:32.943 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:32.943 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.943 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.943 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:32.943 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508183+0000", 2024-09-21T15:49:32.943 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:32.943 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.943 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.944 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:32.944 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508184+0000", 2024-09-21T15:49:32.944 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0580000000000001e-06 2024-09-21T15:49:32.944 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.944 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.944 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:32.944 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508187+0000", 2024-09-21T15:49:32.944 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7209999999999999e-06 2024-09-21T15:49:32.944 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.944 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.944 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:32.944 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508189+0000", 2024-09-21T15:49:32.944 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5239999999999999e-06 2024-09-21T15:49:32.944 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.945 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.945 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:32.945 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.541243+0000", 2024-09-21T15:49:32.945 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033053658 2024-09-21T15:49:32.945 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.945 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.945 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:32.945 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.541284+0000", 2024-09-21T15:49:32.945 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0834000000000002e-05 2024-09-21T15:49:32.945 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.945 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.945 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:32.945 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.541334+0000", 2024-09-21T15:49:32.945 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9724e-05 2024-09-21T15:49:32.946 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:32.946 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:32.946 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:32.946 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.946 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.946 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26275 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 5251072~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:32.946 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.510151+0000", 2024-09-21T15:49:32.946 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 6.3668562639999999, 2024-09-21T15:49:32.946 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032836153999999999, 2024-09-21T15:49:32.946 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:32.946 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:32.946 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:32.946 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:32.946 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:32.947 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26275 2024-09-21T15:49:32.947 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.947 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:32.947 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.947 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:32.947 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510151+0000", 2024-09-21T15:49:32.947 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:32.947 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.947 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.947 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:32.947 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510151+0000", 2024-09-21T15:49:32.947 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:32.947 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.948 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.948 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:32.948 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510151+0000", 2024-09-21T15:49:32.948 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:32.948 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.948 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.948 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:32.948 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510152+0000", 2024-09-21T15:49:32.948 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1990000000000001e-06 2024-09-21T15:49:32.948 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.948 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.948 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:32.948 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510155+0000", 2024-09-21T15:49:32.948 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7089999999999998e-06 2024-09-21T15:49:32.948 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.949 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.949 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:32.949 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510157+0000", 2024-09-21T15:49:32.949 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5189999999999999e-06 2024-09-21T15:49:32.949 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.949 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.949 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:32.949 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.542902+0000", 2024-09-21T15:49:32.949 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032744599999999999 2024-09-21T15:49:32.949 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.949 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.949 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:32.949 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.542941+0000", 2024-09-21T15:49:32.949 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8761999999999999e-05 2024-09-21T15:49:32.949 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.950 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.950 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:32.950 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.542987+0000", 2024-09-21T15:49:32.950 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6665999999999997e-05 2024-09-21T15:49:32.950 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:32.950 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:32.950 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:32.950 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.950 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.950 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26276 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 5308416~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:32.950 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.510159+0000", 2024-09-21T15:49:32.950 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 6.3668483, 2024-09-21T15:49:32.950 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032943014999999999, 2024-09-21T15:49:32.951 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:32.951 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:32.951 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:32.951 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:32.951 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:32.951 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26276 2024-09-21T15:49:32.951 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.951 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:32.951 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.951 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:32.951 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510159+0000", 2024-09-21T15:49:32.951 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:32.951 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.952 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.952 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:32.953 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510159+0000", 2024-09-21T15:49:32.953 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:32.953 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.953 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.954 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:32.954 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510159+0000", 2024-09-21T15:49:32.954 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:32.954 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.954 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.954 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:32.954 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510160+0000", 2024-09-21T15:49:32.954 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.13e-06 2024-09-21T15:49:32.954 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.954 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.954 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:32.954 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510163+0000", 2024-09-21T15:49:32.954 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7489999999999999e-06 2024-09-21T15:49:32.955 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.955 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.955 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:32.955 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510165+0000", 2024-09-21T15:49:32.955 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4200000000000001e-06 2024-09-21T15:49:32.955 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.955 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.955 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:32.955 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.543013+0000", 2024-09-21T15:49:32.955 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032847736000000002 2024-09-21T15:49:32.955 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.955 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.955 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:32.955 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.543054+0000", 2024-09-21T15:49:32.955 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0725000000000001e-05 2024-09-21T15:49:32.956 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.956 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.956 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:32.956 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.543102+0000", 2024-09-21T15:49:32.956 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8471999999999999e-05 2024-09-21T15:49:32.956 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:32.956 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:32.956 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:32.956 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.956 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.956 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26277 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 5365760~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:32.956 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.510167+0000", 2024-09-21T15:49:32.956 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 6.3668401689999996, 2024-09-21T15:49:32.956 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033113111000000001, 2024-09-21T15:49:32.957 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:32.957 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:32.957 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:32.957 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:32.957 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:32.957 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26277 2024-09-21T15:49:32.957 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.957 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:32.957 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.957 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:32.957 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510167+0000", 2024-09-21T15:49:32.957 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:32.957 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.957 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.957 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:32.958 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510167+0000", 2024-09-21T15:49:32.958 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:32.958 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.958 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.958 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:32.958 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510167+0000", 2024-09-21T15:49:32.958 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:32.958 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.958 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.958 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:32.958 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510168+0000", 2024-09-21T15:49:32.958 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.6899999999999996e-07 2024-09-21T15:49:32.958 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.958 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.959 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:32.959 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510171+0000", 2024-09-21T15:49:32.959 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7080000000000002e-06 2024-09-21T15:49:32.959 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.959 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.959 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:32.959 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510173+0000", 2024-09-21T15:49:32.959 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5050000000000002e-06 2024-09-21T15:49:32.959 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.959 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.959 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:32.959 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.543125+0000", 2024-09-21T15:49:32.959 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032951504 2024-09-21T15:49:32.959 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.960 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.960 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:32.960 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.543192+0000", 2024-09-21T15:49:32.960 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6938000000000002e-05 2024-09-21T15:49:32.960 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.960 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.960 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:32.960 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.543280+0000", 2024-09-21T15:49:32.960 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8702999999999994e-05 2024-09-21T15:49:32.960 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:32.960 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:32.960 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:32.961 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.961 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.961 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26292 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 6225920~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:32.961 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.512160+0000", 2024-09-21T15:49:32.961 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 6.3648473550000002, 2024-09-21T15:49:32.961 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032838748000000001, 2024-09-21T15:49:32.961 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:32.961 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:32.961 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:32.961 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:32.961 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:32.961 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26292 2024-09-21T15:49:32.961 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.961 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:32.961 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.962 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:32.962 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512160+0000", 2024-09-21T15:49:32.962 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:32.962 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.962 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.962 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:32.962 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512160+0000", 2024-09-21T15:49:32.962 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:32.962 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.962 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.962 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:32.962 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512160+0000", 2024-09-21T15:49:32.962 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:32.963 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.963 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.963 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:32.963 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512161+0000", 2024-09-21T15:49:32.963 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.158e-06 2024-09-21T15:49:32.963 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.963 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.963 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:32.963 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512164+0000", 2024-09-21T15:49:32.963 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7429999999999998e-06 2024-09-21T15:49:32.963 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.963 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.963 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:32.964 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512166+0000", 2024-09-21T15:49:32.964 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5280000000000002e-06 2024-09-21T15:49:32.964 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.964 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.964 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:32.964 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.544912+0000", 2024-09-21T15:49:32.964 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032745962000000003 2024-09-21T15:49:32.964 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.964 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.964 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:32.964 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.544950+0000", 2024-09-21T15:49:32.964 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7657000000000002e-05 2024-09-21T15:49:32.964 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.964 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.964 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:32.965 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.544999+0000", 2024-09-21T15:49:32.965 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8912999999999997e-05 2024-09-21T15:49:32.965 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:32.965 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:32.965 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:32.965 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.965 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.965 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26293 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 6283264~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:32.965 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.512169+0000", 2024-09-21T15:49:32.965 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 6.3648382059999999, 2024-09-21T15:49:32.965 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032937552000000002, 2024-09-21T15:49:32.965 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:32.965 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:32.966 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:32.966 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:32.966 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:32.966 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26293 2024-09-21T15:49:32.966 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.966 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:32.966 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.966 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:32.966 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512169+0000", 2024-09-21T15:49:32.966 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:32.966 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.966 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.966 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:32.966 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512169+0000", 2024-09-21T15:49:32.966 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:32.967 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.967 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.967 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:32.967 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512169+0000", 2024-09-21T15:49:32.967 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:32.967 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.967 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.967 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:32.967 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512170+0000", 2024-09-21T15:49:32.967 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1650000000000001e-06 2024-09-21T15:49:32.967 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.967 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.967 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:32.967 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512173+0000", 2024-09-21T15:49:32.968 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.773e-06 2024-09-21T15:49:32.968 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.968 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.968 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:32.968 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512175+0000", 2024-09-21T15:49:32.968 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4700000000000001e-06 2024-09-21T15:49:32.968 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.968 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.968 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:32.968 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.545018+0000", 2024-09-21T15:49:32.968 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032843043000000002 2024-09-21T15:49:32.968 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.968 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.968 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:32.968 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.545057+0000", 2024-09-21T15:49:32.969 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8862999999999997e-05 2024-09-21T15:49:32.969 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:32.969 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:32.969 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:32.969 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.545107+0000", 2024-09-21T15:49:32.969 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9455e-05 2024-09-21T15:49:32.969 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:32.969 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:32.969 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:32.969 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:32.969 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:32.969 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:32.970 INFO:teuthology.orchestra.run.smithi089.stdout:truncating image image_client.0-clone11 from 0xbac6784 (overlap 0x8b0a7c2) to 0x29b6587 2024-09-21T15:49:32.971 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-09-21T15:49:32.971 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:49:32.971 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:49:32.972 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:49:32.972 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:32.969+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:32.972 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:49:32.982 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:49:32.982 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:49:33.013 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:49:33.014 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:49:33.036 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:49:33.036 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:49:33.057 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:49:33.057 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:49:33.071 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:33.070+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:33.073 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:49:33.073 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:49:33.089 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:49:33.089 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:49:33.118 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:33.118 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:33.126 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:49:33.157 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T15:49:33.157 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T15:49:33.157 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-09-21T15:49:33.157 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T15:49:33.158 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T15:49:33.158 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok config set filestore_inject_stall 3 2024-09-21T15:49:33.172 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:33.171+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:33.273 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:33.272+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:33.334 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:33.371 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:33.371 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:33.371 INFO:teuthology.orchestra.run.smithi100.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T15:49:33.372 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:33.372 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:49:33.375 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:33.374+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:33.475 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:33.474+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:33.569 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:33.570 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:33.577 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:33.575+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:33.582 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T15:49:33.676 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:33.675+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:33.749 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:33.749 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:33.760 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T15:49:33.777 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:33.776+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:33.806 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is set 2024-09-21T15:49:33.878 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:33.877+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:33.926 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:33.927 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:33.935 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T15:49:33.980 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:33.978+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:34.042 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:34.042 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:34.051 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T15:49:34.080 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:34.079+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:34.086 INFO:teuthology.orchestra.run.smithi089.stdout:checking clone #10, image image_client.0-clone10 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent11 2024-09-21T15:49:34.164 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:34.164 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:34.173 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T15:49:34.179 INFO:teuthology.orchestra.run.smithi089.stdout:2121 write 0xc449336 thru 0xc457013 (0xdcde bytes) 2024-09-21T15:49:34.181 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:34.180+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:34.183 INFO:teuthology.orchestra.run.smithi089.stdout:2122 read 0xa42ef8a thru 0xa4367ed (0x7864 bytes) 2024-09-21T15:49:34.183 INFO:teuthology.orchestra.run.smithi089.stdout:2126 read 0x8172397 thru 0x817fc78 (0xd8e2 bytes) 2024-09-21T15:49:34.183 INFO:teuthology.orchestra.run.smithi089.stdout:2127 read 0x2bae391 thru 0x2bb6421 (0x8091 bytes) 2024-09-21T15:49:34.185 INFO:teuthology.orchestra.run.smithi089.stdout:2128 writesame 0x4552857 thru 0x455bd61 data_size 0xd(0x950b bytes) 2024-09-21T15:49:34.198 INFO:teuthology.orchestra.run.smithi089.stdout:2129 writesame 0x88cb4f2 thru 0x88cbc04 data_size 0x1(0x713 bytes) 2024-09-21T15:49:34.205 INFO:teuthology.orchestra.run.smithi089.stdout:2132 read 0x6f59de3 thru 0x6f665a9 (0xc7c7 bytes) 2024-09-21T15:49:34.205 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:34.205 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:34.205 INFO:teuthology.orchestra.run.smithi089.stdout:2136 write 0x64c142b thru 0x64c23f3 (0xfc9 bytes) 2024-09-21T15:49:34.210 INFO:teuthology.orchestra.run.smithi089.stdout:2137 read 0x3f8714d thru 0x3f8f8c1 (0x8775 bytes) 2024-09-21T15:49:34.213 INFO:teuthology.orchestra.run.smithi089.stdout:2138 writesame 0x8d54392 thru 0x8d58127 data_size 0x1(0x3d96 bytes) 2024-09-21T15:49:34.225 INFO:teuthology.orchestra.run.smithi089.stdout:2139 punch from 0x2a80c3f to 0x2a8c536, (0xb8f7 bytes) 2024-09-21T15:49:34.225 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:34.225 INFO:teuthology.orchestra.run.smithi089.stdout:2143 write 0x679ff42 thru 0x67a16fe (0x17bd bytes) 2024-09-21T15:49:34.231 INFO:teuthology.orchestra.run.smithi089.stdout:2144 writesame 0x5345640 thru 0x534a13e data_size 0x1(0x4aff bytes) 2024-09-21T15:49:34.248 INFO:teuthology.orchestra.run.smithi089.stdout:2146 read 0x3241706 thru 0x3246404 (0x4cff bytes) 2024-09-21T15:49:34.250 INFO:teuthology.orchestra.run.smithi089.stdout:2148 read 0x80c51e4 thru 0x80cb34e (0x616b bytes) 2024-09-21T15:49:34.250 INFO:teuthology.orchestra.run.smithi089.stdout:2149 write 0x2ef46e4 thru 0x2efb841 (0x715e bytes) 2024-09-21T15:49:34.277 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:34.282 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:34.281+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:34.287 INFO:teuthology.orchestra.run.smithi089.stdout:2152 trunc from 0xc457014 to 0x3eea53d 2024-09-21T15:49:34.287 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:34.287 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:34.297 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T15:49:34.383 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:34.382+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:34.397 INFO:teuthology.orchestra.run.smithi089.stdout:2153 trunc from 0x3eea53d to 0x7e6433b 2024-09-21T15:49:34.409 INFO:teuthology.orchestra.run.smithi089.stdout:2154 trunc from 0x7e6433b to 0xb298518 2024-09-21T15:49:34.418 INFO:teuthology.orchestra.run.smithi089.stdout:2155 write 0x92119dd thru 0x921838c (0x69b0 bytes) 2024-09-21T15:49:34.423 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:34.423 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:34.426 INFO:teuthology.orchestra.run.smithi089.stdout:2157 punch from 0x620197d to 0x62068b5, (0x4f38 bytes) 2024-09-21T15:49:34.427 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:34.427 INFO:teuthology.orchestra.run.smithi089.stdout:2160 read 0x552766c thru 0x5533b1f (0xc4b4 bytes) 2024-09-21T15:49:34.432 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:49:34.439 INFO:teuthology.orchestra.run.smithi089.stdout:2161 trunc from 0xb298518 to 0x6dca64 2024-09-21T15:49:34.485 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:34.483+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:34.578 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:34.579 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:34.586 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:34.585+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:34.587 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:49:34.606 INFO:teuthology.orchestra.run.smithi089.stdout:2162 punch from 0x62baf3 to 0x638e0c, (0xd319 bytes) 2024-09-21T15:49:34.606 INFO:teuthology.orchestra.run.smithi089.stdout:2164 read 0xfd588 thru 0x109610 (0xc089 bytes) 2024-09-21T15:49:34.608 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:34.614 INFO:teuthology.orchestra.run.smithi089.stdout:2166 writesame 0x1dce2e3 thru 0x1dd9356 data_size 0x2e(0xb074 bytes) 2024-09-21T15:49:34.620 INFO:teuthology.orchestra.run.smithi089.stdout:2167 write 0x2929ab0 thru 0x29392e1 (0xf832 bytes) 2024-09-21T15:49:34.630 INFO:teuthology.orchestra.run.smithi089.stdout:2169 trunc from 0x29392e2 to 0x8453181 2024-09-21T15:49:34.635 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:34.635 INFO:teuthology.orchestra.run.smithi089.stdout:2174 read 0x5fdaa19 thru 0x5fe6b6c (0xc154 bytes) 2024-09-21T15:49:34.635 INFO:teuthology.orchestra.run.smithi089.stdout:2176 write 0x4c75a75 thru 0x4c79495 (0x3a21 bytes) 2024-09-21T15:49:34.639 INFO:teuthology.orchestra.run.smithi089.stdout:2177 write 0x29410ac thru 0x294e7ee (0xd743 bytes) 2024-09-21T15:49:34.656 INFO:teuthology.orchestra.run.smithi089.stdout:2178 writesame 0xe47d730 thru 0xe48351b data_size 0x2(0x5dec bytes) 2024-09-21T15:49:34.666 INFO:teuthology.orchestra.run.smithi089.stdout:2179 trunc from 0xe48351c to 0x61eafa8 2024-09-21T15:49:34.673 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:34.685 INFO:teuthology.orchestra.run.smithi089.stdout:2182 write 0xb97d251 thru 0xb986b7a (0x992a bytes) 2024-09-21T15:49:34.687 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:34.686+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:34.695 INFO:teuthology.orchestra.run.smithi089.stdout:2183 punch from 0xa7b257e to 0xa7c2566, (0xffe8 bytes) 2024-09-21T15:49:34.695 INFO:teuthology.orchestra.run.smithi089.stdout:2184 read 0x717ba24 thru 0x717ce7a (0x1457 bytes) 2024-09-21T15:49:34.696 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:34.696 INFO:teuthology.orchestra.run.smithi089.stdout:2186 read 0x92b3c6f thru 0x92c1984 (0xdd16 bytes) 2024-09-21T15:49:34.705 INFO:teuthology.orchestra.run.smithi089.stdout:2187 trunc from 0xb986b7b to 0x384955c 2024-09-21T15:49:34.721 INFO:teuthology.orchestra.run.smithi089.stdout:2188 read 0xd25146 thru 0xd2e4da (0x9395 bytes) 2024-09-21T15:49:34.721 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:34.721 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:34.721 INFO:teuthology.orchestra.run.smithi089.stdout:2193 read 0x17327f7 thru 0x1734aaa (0x22b4 bytes) 2024-09-21T15:49:34.722 INFO:teuthology.orchestra.run.smithi089.stdout:2197 read 0x306787b thru 0x3076b67 (0xf2ed bytes) 2024-09-21T15:49:34.722 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:34.722 INFO:teuthology.orchestra.run.smithi089.stdout:2202 write 0x26cedd3 thru 0x26d490d (0x5b3b bytes) 2024-09-21T15:49:34.722 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:34.722 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:34.734 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:49:34.745 INFO:teuthology.orchestra.run.smithi089.stdout:2203 write 0xecf8b1f thru 0xed00f17 (0x83f9 bytes) 2024-09-21T15:49:34.749 INFO:teuthology.orchestra.run.smithi089.stdout:2204 read 0xca16cf6 thru 0xca25175 (0xe480 bytes) 2024-09-21T15:49:34.749 INFO:teuthology.orchestra.run.smithi089.stdout:2207 writesame 0xdeb04b6 thru 0xdeb8698 data_size 0x1(0x81e3 bytes) 2024-09-21T15:49:34.768 INFO:teuthology.orchestra.run.smithi089.stdout:2209 write 0x2bbb3c4 thru 0x2bbde7a (0x2ab7 bytes) 2024-09-21T15:49:34.774 INFO:teuthology.orchestra.run.smithi089.stdout:2210 write 0x678ae7d thru 0x679ad7f (0xff03 bytes) 2024-09-21T15:49:34.777 INFO:teuthology.orchestra.run.smithi089.stdout:2211 writesame 0xc9fbcaf thru 0xca000e7 data_size 0x7(0x4439 bytes) 2024-09-21T15:49:34.787 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:34.786+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:34.789 INFO:teuthology.orchestra.run.smithi089.stdout:2212 trunc from 0xed00f18 to 0xd071a77 2024-09-21T15:49:34.811 INFO:teuthology.orchestra.run.smithi089.stdout:2213 read 0x8febc19 thru 0x8ff52c7 (0x96af bytes) 2024-09-21T15:49:34.811 INFO:teuthology.orchestra.run.smithi089.stdout:2214 write 0xb9040a2 thru 0xb913e70 (0xfdcf bytes) 2024-09-21T15:49:34.817 INFO:teuthology.orchestra.run.smithi089.stdout:2215 write 0x5782acb thru 0x5784aae (0x1fe4 bytes) 2024-09-21T15:49:34.825 INFO:teuthology.orchestra.run.smithi089.stdout:2216 trunc from 0xd071a77 to 0x91bd55e 2024-09-21T15:49:34.836 INFO:teuthology.orchestra.run.smithi089.stdout:2217 write 0xc589ef thru 0xc64bbf (0xc1d1 bytes) 2024-09-21T15:49:34.851 INFO:teuthology.orchestra.run.smithi089.stdout:2219 trunc from 0x91bd55e to 0x1a66dd5 2024-09-21T15:49:34.853 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:34.853 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:34.862 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:49:34.865 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:34.865 INFO:teuthology.orchestra.run.smithi089.stdout:2223 punch from 0x168b9ba to 0x169433f, (0x8985 bytes) 2024-09-21T15:49:34.866 INFO:teuthology.orchestra.run.smithi089.stdout:2225 trunc from 0x1a66dd5 to 0x255017c 2024-09-21T15:49:34.879 INFO:teuthology.orchestra.run.smithi089.stdout:2226 write 0x560162b thru 0x5603d40 (0x2716 bytes) 2024-09-21T15:49:34.882 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:34.882 INFO:teuthology.orchestra.run.smithi089.stdout:2228 punch from 0x2df5415 to 0x2e00ec4, (0xbaaf bytes) 2024-09-21T15:49:34.882 INFO:teuthology.orchestra.run.smithi089.stdout:2229 punch from 0x1b22c03 to 0x1b2607d, (0x347a bytes) 2024-09-21T15:49:34.883 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:34.883 INFO:teuthology.orchestra.run.smithi089.stdout:2231 read 0xd940c4 thru 0xda3bea (0xfb27 bytes) 2024-09-21T15:49:34.883 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:34.883 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:34.888 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:34.887+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:34.892 INFO:teuthology.orchestra.run.smithi089.stdout:2234 write 0x6f80557 thru 0x6f869e4 (0x648e bytes) 2024-09-21T15:49:34.905 INFO:teuthology.orchestra.run.smithi089.stdout:2235 trunc from 0x6f869e5 to 0xd3834f6 2024-09-21T15:49:34.910 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:34.910 INFO:teuthology.orchestra.run.smithi089.stdout:2239 read 0x9275dc4 thru 0x927e91f (0x8b5c bytes) 2024-09-21T15:49:34.910 INFO:teuthology.orchestra.run.smithi089.stdout:2240 write 0x6694ba8 thru 0x669a02b (0x5484 bytes) 2024-09-21T15:49:34.916 INFO:teuthology.orchestra.run.smithi089.stdout:2241 read 0x10b9f25 thru 0x10ba308 (0x3e4 bytes) 2024-09-21T15:49:34.916 INFO:teuthology.orchestra.run.smithi089.stdout:2242 writesame 0x5478c96 thru 0x5481e45 data_size 0x3f(0x91b0 bytes) 2024-09-21T15:49:34.923 INFO:teuthology.orchestra.run.smithi089.stdout:2243 read 0x1530896 thru 0x153688e (0x5ff9 bytes) 2024-09-21T15:49:34.923 INFO:teuthology.orchestra.run.smithi089.stdout:2245 write 0x26dbc45 thru 0x26e1096 (0x5452 bytes) 2024-09-21T15:49:34.940 INFO:teuthology.orchestra.run.smithi089.stdout:2246 trunc from 0xd3834f6 to 0x3707eab 2024-09-21T15:49:34.949 INFO:teuthology.orchestra.run.smithi089.stdout:2247 read 0x1900325 thru 0x1904e03 (0x4adf bytes) 2024-09-21T15:49:34.961 INFO:teuthology.orchestra.run.smithi089.stdout:2248 writesame 0x9a13265 thru 0x9a17aad data_size 0x1(0x4849 bytes) 2024-09-21T15:49:34.973 INFO:teuthology.orchestra.run.smithi089.stdout:2250 writesame 0xec9f006 thru 0xecad397 data_size 0x1(0xe392 bytes) 2024-09-21T15:49:34.973 INFO:teuthology.orchestra.run.smithi089.stdout:2251 read 0x82f3244 thru 0x82f8618 (0x53d5 bytes) 2024-09-21T15:49:34.973 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:34.973 INFO:teuthology.orchestra.run.smithi089.stdout:2253 read 0x1a3e5aa thru 0x1a45368 (0x6dbf bytes) 2024-09-21T15:49:34.973 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:34.973 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:34.973 INFO:teuthology.orchestra.run.smithi089.stdout:2256 punch from 0x2c504cf to 0x2c5e530, (0xe061 bytes) 2024-09-21T15:49:34.974 INFO:teuthology.orchestra.run.smithi089.stdout:2258 trunc from 0xecad398 to 0xde45976 2024-09-21T15:49:34.977 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:34.977 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:34.980 INFO:teuthology.orchestra.run.smithi089.stdout:2259 writesame 0x1171a9d thru 0x117a731 data_size 0x1d(0x8c95 bytes) 2024-09-21T15:49:34.982 INFO:teuthology.orchestra.run.smithi089.stdout:2261 write 0x827d4fe thru 0x827fa15 (0x2518 bytes) 2024-09-21T15:49:34.986 INFO:teuthology.orchestra.run.smithi089.stdout:2262 read 0x2b1bba2 thru 0x2b1e101 (0x2560 bytes) 2024-09-21T15:49:34.986 INFO:teuthology.orchestra.run.smithi089.stdout:2263 read 0xa07f421 thru 0xa08da21 (0xe601 bytes) 2024-09-21T15:49:34.987 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:49:34.989 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:34.988+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:34.992 INFO:teuthology.orchestra.run.smithi089.stdout:2264 trunc from 0xde45976 to 0x893086f 2024-09-21T15:49:34.997 INFO:teuthology.orchestra.run.smithi089.stdout:2265 punch from 0x103a031 to 0x1049ac6, (0xfa95 bytes) 2024-09-21T15:49:35.010 INFO:teuthology.orchestra.run.smithi089.stdout:2266 writesame 0xeae31cf thru 0xeaeac5c data_size 0x53(0x7a8e bytes) 2024-09-21T15:49:35.010 INFO:teuthology.orchestra.run.smithi089.stdout:2267 punch from 0x621446 to 0x623b9a, (0x2754 bytes) 2024-09-21T15:49:35.010 INFO:teuthology.orchestra.run.smithi089.stdout:2268 write 0x66f1fcd thru 0x66fe1ba (0xc1ee bytes) 2024-09-21T15:49:35.012 INFO:teuthology.orchestra.run.smithi089.stdout:2269 write 0x543fc17 thru 0x544b894 (0xbc7e bytes) 2024-09-21T15:49:35.016 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:35.016 INFO:teuthology.orchestra.run.smithi089.stdout:2271 read 0x2f5a2e thru 0x30415a (0xe72d bytes) 2024-09-21T15:49:35.017 INFO:teuthology.orchestra.run.smithi089.stdout:2274 read 0x11b9506 thru 0x11bf3cb (0x5ec6 bytes) 2024-09-21T15:49:35.017 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:35.017 INFO:teuthology.orchestra.run.smithi089.stdout:2276 writesame 0x557f883 thru 0x55885e2 data_size 0x34(0x8d60 bytes) 2024-09-21T15:49:35.026 INFO:teuthology.orchestra.run.smithi089.stdout:2277 write 0xafc5f9c thru 0xafc6f6c (0xfd1 bytes) 2024-09-21T15:49:35.032 INFO:teuthology.orchestra.run.smithi089.stdout:2278 read 0x6ae1c10 thru 0x6ae3c98 (0x2089 bytes) 2024-09-21T15:49:35.032 INFO:teuthology.orchestra.run.smithi089.stdout:2279 writesame 0x1c829d0 thru 0x1c9231d data_size 0x21(0xf94e bytes) 2024-09-21T15:49:35.042 INFO:teuthology.orchestra.run.smithi089.stdout:2280 flatten 2024-09-21T15:49:35.090 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:35.088+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:35.101 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:49:35.121 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:35.121 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:35.130 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:49:35.133 INFO:teuthology.orchestra.run.smithi089.stdout:2281 read 0x448394e thru 0x448c296 (0x8949 bytes) 2024-09-21T15:49:35.133 INFO:teuthology.orchestra.run.smithi089.stdout:2282 read 0x384e4d7 thru 0x38592dd (0xae07 bytes) 2024-09-21T15:49:35.134 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:35.134 INFO:teuthology.orchestra.run.smithi089.stdout:2284 read 0x537b278 thru 0x5381dee (0x6b77 bytes) 2024-09-21T15:49:35.134 INFO:teuthology.orchestra.run.smithi089.stdout:2285 punch from 0x80dde2b to 0x80e9d0b, (0xbee0 bytes) 2024-09-21T15:49:35.134 INFO:teuthology.orchestra.run.smithi089.stdout:2287 writesame 0x78c738e thru 0x78d6c46 data_size 0x1(0xf8b9 bytes) 2024-09-21T15:49:35.175 INFO:teuthology.orchestra.run.smithi089.stdout:2289 write 0xa7914b3 thru 0xa7982b1 (0x6dff bytes) 2024-09-21T15:49:35.179 INFO:teuthology.orchestra.run.smithi089.stdout:2290 read 0x564c527 thru 0x565885a (0xc334 bytes) 2024-09-21T15:49:35.179 INFO:teuthology.orchestra.run.smithi089.stdout:2291 write 0x744eb40 thru 0x744f237 (0x6f8 bytes) 2024-09-21T15:49:35.183 INFO:teuthology.orchestra.run.smithi089.stdout:2293 write 0xcaedd3d thru 0xcafd227 (0xf4eb bytes) 2024-09-21T15:49:35.188 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:35.190 INFO:teuthology.orchestra.run.smithi089.stdout:2296 trunc from 0xeaeac5d to 0xccc1896 2024-09-21T15:49:35.191 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:35.189+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:35.198 INFO:teuthology.orchestra.run.smithi089.stdout:2298 trunc from 0xccc1896 to 0xdf08b59 2024-09-21T15:49:35.211 INFO:teuthology.orchestra.run.smithi089.stdout:2299 trunc from 0xdf08b59 to 0x7cce039 2024-09-21T15:49:35.224 INFO:teuthology.orchestra.run.smithi089.stdout:2300 read 0x5eac7c8 thru 0x5eb7cbf (0xb4f8 bytes) 2024-09-21T15:49:35.231 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-09-21T15:49:35.231 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:49:35.237 INFO:teuthology.orchestra.run.smithi089.stdout:2301 write 0xe265ae4 thru 0xe26ce07 (0x7324 bytes) 2024-09-21T15:49:35.240 INFO:teuthology.orchestra.run.smithi089.stdout:2302 write 0xbd8d6b4 thru 0xbd8eb2e (0x147b bytes) 2024-09-21T15:49:35.243 INFO:teuthology.orchestra.run.smithi089.stdout:2304 writesame 0x5122bd9 thru 0x5130aae data_size 0xb(0xded6 bytes) 2024-09-21T15:49:35.244 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:35.244 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:35.248 INFO:teuthology.orchestra.run.smithi089.stdout:2306 write 0xde057f9 thru 0xde0bc4d (0x6455 bytes) 2024-09-21T15:49:35.248 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:49:35.248 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:49:35.251 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:35.251 INFO:teuthology.orchestra.run.smithi089.stdout:2309 write 0x17c1dfd thru 0x17c31a8 (0x13ac bytes) 2024-09-21T15:49:35.253 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:35.254 INFO:teuthology.orchestra.run.smithi089.stdout:2315 read 0x2346e1a thru 0x2348d49 (0x1f30 bytes) 2024-09-21T15:49:35.254 INFO:teuthology.orchestra.run.smithi089.stdout:2317 read 0x56048c9 thru 0x56106e6 (0xbe1e bytes) 2024-09-21T15:49:35.255 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:49:35.255 INFO:teuthology.orchestra.run.smithi089.stdout:2318 read 0x46016ea thru 0x4610c31 (0xf548 bytes) 2024-09-21T15:49:35.256 INFO:teuthology.orchestra.run.smithi089.stdout:2319 trunc from 0xe26ce08 to 0xc78972b 2024-09-21T15:49:35.267 INFO:teuthology.orchestra.run.smithi089.stdout:2320 write 0x12edf59 thru 0x12f592d (0x79d5 bytes) 2024-09-21T15:49:35.268 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:49:35.268 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:49:35.271 INFO:teuthology.orchestra.run.smithi089.stdout:2321 punch from 0x45f11d to 0x467036, (0x7f19 bytes) 2024-09-21T15:49:35.271 INFO:teuthology.orchestra.run.smithi089.stdout:2322 punch from 0xaea6f8b to 0xaeb5909, (0xe97e bytes) 2024-09-21T15:49:35.271 INFO:teuthology.orchestra.run.smithi089.stdout:2323 read 0x65dbdbc thru 0x65ddb16 (0x1d5b bytes) 2024-09-21T15:49:35.271 INFO:teuthology.orchestra.run.smithi089.stdout:2324 read 0x5f9f0d thru 0x5fa195 (0x289 bytes) 2024-09-21T15:49:35.271 INFO:teuthology.orchestra.run.smithi089.stdout:2325 write 0x3b4543a thru 0x3b5285f (0xd426 bytes) 2024-09-21T15:49:35.274 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:35.274 INFO:teuthology.orchestra.run.smithi089.stdout:2327 write 0x424633 thru 0x429379 (0x4d47 bytes) 2024-09-21T15:49:35.277 INFO:teuthology.orchestra.run.smithi089.stdout:2328 write 0x9dc2254 thru 0x9dca0e3 (0x7e90 bytes) 2024-09-21T15:49:35.280 INFO:teuthology.orchestra.run.smithi089.stdout:2329 write 0x9eb62e0 thru 0x9ec0924 (0xa645 bytes) 2024-09-21T15:49:35.284 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:35.284 INFO:teuthology.orchestra.run.smithi089.stdout:2331 read 0x1e8bc27 thru 0x1e99053 (0xd42d bytes) 2024-09-21T15:49:35.285 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:35.285 INFO:teuthology.orchestra.run.smithi089.stdout:2334 write 0x978e427 thru 0x97978b5 (0x948f bytes) 2024-09-21T15:49:35.285 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:49:35.285 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:49:35.289 INFO:teuthology.orchestra.run.smithi089.stdout:2335 punch from 0x7b333d0 to 0x7b3ba1b, (0x864b bytes) 2024-09-21T15:49:35.289 INFO:teuthology.orchestra.run.smithi089.stdout:2336 read 0x7a4afa4 thru 0x7a4bf2d (0xf8a bytes) 2024-09-21T15:49:35.289 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:35.289 INFO:teuthology.orchestra.run.smithi089.stdout:2338 write 0xaf842f0 thru 0xaf86605 (0x2316 bytes) 2024-09-21T15:49:35.291 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:35.290+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:35.295 INFO:teuthology.orchestra.run.smithi089.stdout:2339 read 0x4915ecb thru 0x49248ff (0xea35 bytes) 2024-09-21T15:49:35.295 INFO:teuthology.orchestra.run.smithi089.stdout:2340 write 0x99df73a thru 0x99ede6d (0xe734 bytes) 2024-09-21T15:49:35.297 INFO:teuthology.orchestra.run.smithi089.stdout:2341 writesame 0x8d32621 thru 0x8d354a6 data_size 0xf(0x2e86 bytes) 2024-09-21T15:49:35.300 INFO:teuthology.orchestra.run.smithi089.stdout:2342 read 0xaec3540 thru 0xaed1b16 (0xe5d7 bytes) 2024-09-21T15:49:35.300 INFO:teuthology.orchestra.run.smithi089.stdout:2343 write 0x977479d thru 0x9777573 (0x2dd7 bytes) 2024-09-21T15:49:35.302 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:49:35.302 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:49:35.304 INFO:teuthology.orchestra.run.smithi089.stdout:2344 punch from 0x2069096 to 0x2069469, (0x3d3 bytes) 2024-09-21T15:49:35.304 INFO:teuthology.orchestra.run.smithi089.stdout:2345 write 0x39d3c2e thru 0x39e36df (0xfab2 bytes) 2024-09-21T15:49:35.307 INFO:teuthology.orchestra.run.smithi089.stdout:2346 write 0x5ee46bb thru 0x5eeda5b (0x93a1 bytes) 2024-09-21T15:49:35.315 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:35.316 INFO:teuthology.orchestra.run.smithi089.stdout:2348 trunc from 0xc78972b to 0xd4ec508 2024-09-21T15:49:35.323 INFO:teuthology.orchestra.run.smithi089.stdout:2350 writesame 0xacbffec thru 0xacc2239 data_size 0x1(0x224e bytes) 2024-09-21T15:49:35.333 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:49:35.333 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:49:35.342 INFO:teuthology.orchestra.run.smithi089.stdout:2351 trunc from 0xd4ec508 to 0x6260914 2024-09-21T15:49:35.364 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:49:35.365 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:49:35.371 INFO:teuthology.orchestra.run.smithi089.stdout:2352 write 0xa58f1da thru 0xa59301f (0x3e46 bytes) 2024-09-21T15:49:35.375 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:35.375 INFO:teuthology.orchestra.run.smithi089.stdout:2354 read 0x315ddbf thru 0x316038e (0x25d0 bytes) 2024-09-21T15:49:35.375 INFO:teuthology.orchestra.run.smithi089.stdout:2355 read 0x76bd2e3 thru 0x76ca8e9 (0xd607 bytes) 2024-09-21T15:49:35.375 INFO:teuthology.orchestra.run.smithi089.stdout:2357 write 0x763c708 thru 0x7648219 (0xbb12 bytes) 2024-09-21T15:49:35.383 INFO:teuthology.orchestra.run.smithi089.stdout:2359 trunc from 0xa593020 to 0xdd15734 2024-09-21T15:49:35.387 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:49:35.387 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:49:35.388 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:35.388 INFO:teuthology.orchestra.run.smithi089.stdout:2364 read 0x1c63a7a thru 0x1c71733 (0xdcba bytes) 2024-09-21T15:49:35.388 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:35.392 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:35.391+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:35.393 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:35.393 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:49:35.393 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:49:35.393 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:35.393 INFO:teuthology.orchestra.run.smithi089.stdout:2366 write 0xe5502ca thru 0xe554698 (0x43cf bytes) 2024-09-21T15:49:35.396 INFO:teuthology.orchestra.run.smithi089.stdout:2367 read 0x18ae1cd thru 0x18b056f (0x23a3 bytes) 2024-09-21T15:49:35.397 INFO:teuthology.orchestra.run.smithi089.stdout:2368 read 0x7824cf3 thru 0x7827c3d (0x2f4b bytes) 2024-09-21T15:49:35.397 INFO:teuthology.orchestra.run.smithi089.stdout:2369 read 0xba71f42 thru 0xba76023 (0x40e2 bytes) 2024-09-21T15:49:35.397 INFO:teuthology.orchestra.run.smithi089.stdout:2370 writesame 0x7159468 thru 0x7164deb data_size 0x4(0xb984 bytes) 2024-09-21T15:49:35.403 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:49:35.405 INFO:teuthology.orchestra.run.smithi089.stdout:2372 writesame 0xc20b6c8 thru 0xc2182f6 data_size 0x1(0xcc2f bytes) 2024-09-21T15:49:35.434 INFO:teuthology.orchestra.run.smithi089.stdout:2373 trunc from 0xe554699 to 0xbff465a 2024-09-21T15:49:35.446 INFO:teuthology.orchestra.run.smithi089.stdout:2374 write 0xb319092 thru 0xb31c633 (0x35a2 bytes) 2024-09-21T15:49:35.465 INFO:teuthology.orchestra.run.smithi089.stdout:2376 trunc from 0xbff465a to 0x1cb4497 2024-09-21T15:49:35.485 INFO:teuthology.orchestra.run.smithi089.stdout:2377 read 0xd97a0f thru 0xd9ffd8 (0x85ca bytes) 2024-09-21T15:49:35.485 INFO:teuthology.orchestra.run.smithi089.stdout:2379 punch from 0xd3bf5b to 0xd475e4, (0xb689 bytes) 2024-09-21T15:49:35.485 INFO:teuthology.orchestra.run.smithi089.stdout:2380 writesame 0xaefc9 thru 0xbbf6c data_size 0x61(0xcfa4 bytes) 2024-09-21T15:49:35.493 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:35.492+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:35.494 INFO:teuthology.orchestra.run.smithi089.stdout:2381 write 0x3a5ee29 thru 0x3a61423 (0x25fb bytes) 2024-09-21T15:49:35.509 INFO:teuthology.orchestra.run.smithi089.stdout:2382 write 0x849a071 thru 0x849c192 (0x2122 bytes) 2024-09-21T15:49:35.512 INFO:teuthology.orchestra.run.smithi089.stdout:2383 read 0x49da74 thru 0x4ac636 (0xebc3 bytes) 2024-09-21T15:49:35.513 INFO:teuthology.orchestra.run.smithi089.stdout:2385 punch from 0x4de82e9 to 0x4de8488, (0x19f bytes) 2024-09-21T15:49:35.513 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:35.513 INFO:teuthology.orchestra.run.smithi089.stdout:2387 punch from 0x21ac65b to 0x21ae949, (0x22ee bytes) 2024-09-21T15:49:35.515 INFO:teuthology.orchestra.run.smithi089.stdout:2388 trunc from 0x849c193 to 0xa2f2bbf 2024-09-21T15:49:35.517 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:35.517 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:49:35.517 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:49:35.517 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:49:35.517 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:35.520 INFO:teuthology.orchestra.run.smithi089.stdout:2389 punch from 0x5898f86 to 0x589b22f, (0x22a9 bytes) 2024-09-21T15:49:35.520 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:35.528 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:49:35.530 INFO:teuthology.orchestra.run.smithi089.stdout:2391 write 0xdf41205 thru 0xdf497da (0x85d6 bytes) 2024-09-21T15:49:35.534 INFO:teuthology.orchestra.run.smithi089.stdout:2392 trunc from 0xdf497db to 0xd990a80 2024-09-21T15:49:35.540 INFO:teuthology.orchestra.run.smithi089.stdout:2393 write 0x1c4fe12 thru 0x1c556a2 (0x5891 bytes) 2024-09-21T15:49:35.544 INFO:teuthology.orchestra.run.smithi089.stdout:2394 punch from 0xc1ff6ac to 0xc2076a3, (0x7ff7 bytes) 2024-09-21T15:49:35.544 INFO:teuthology.orchestra.run.smithi089.stdout:2395 read 0x76042f0 thru 0x7607d68 (0x3a79 bytes) 2024-09-21T15:49:35.544 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:35.544 INFO:teuthology.orchestra.run.smithi089.stdout:2398 write 0x73e4fb thru 0x74a2db (0xbde1 bytes) 2024-09-21T15:49:35.545 INFO:teuthology.orchestra.run.smithi089.stdout:2400 trunc from 0xd990a80 to 0xd9fe6e7 2024-09-21T15:49:35.552 INFO:teuthology.orchestra.run.smithi089.stdout:2401 trunc from 0xd9fe6e7 to 0xddfe2e5 2024-09-21T15:49:35.558 INFO:teuthology.orchestra.run.smithi089.stdout:2402 write 0x50cf221 thru 0x50d761c (0x83fc bytes) 2024-09-21T15:49:35.561 INFO:teuthology.orchestra.run.smithi089.stdout:2403 punch from 0xb5ca985 to 0xb5cfa12, (0x508d bytes) 2024-09-21T15:49:35.562 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:35.562 INFO:teuthology.orchestra.run.smithi089.stdout:2406 write 0x97f6ab1 thru 0x97fd9ef (0x6f3f bytes) 2024-09-21T15:49:35.565 INFO:teuthology.orchestra.run.smithi089.stdout:2410 writesame 0x43fbebf thru 0x440513b data_size 0x1(0x927d bytes) 2024-09-21T15:49:35.581 INFO:teuthology.orchestra.run.smithi089.stdout:2411 write 0x71e57df thru 0x71f2609 (0xce2b bytes) 2024-09-21T15:49:35.585 INFO:teuthology.orchestra.run.smithi089.stdout:2412 read 0x38f1bae thru 0x38fca78 (0xaecb bytes) 2024-09-21T15:49:35.585 INFO:teuthology.orchestra.run.smithi089.stdout:2414 write 0x726e762 thru 0x72707d0 (0x206f bytes) 2024-09-21T15:49:35.592 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:35.592 INFO:teuthology.orchestra.run.smithi089.stdout:2416 writesame 0x489d513 thru 0x48aa82a data_size 0x46(0xd318 bytes) 2024-09-21T15:49:35.594 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:35.593+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:35.596 INFO:teuthology.orchestra.run.smithi089.stdout:2418 read 0x8b04483 thru 0x8b071e4 (0x2d62 bytes) 2024-09-21T15:49:35.596 INFO:teuthology.orchestra.run.smithi089.stdout:2420 writesame 0x2cf3a8d thru 0x2cfbde4 data_size 0xc(0x8358 bytes) 2024-09-21T15:49:35.601 INFO:teuthology.orchestra.run.smithi089.stdout:2421 writesame 0x7212f4a thru 0x721b7f7 data_size 0x5(0x88ae bytes) 2024-09-21T15:49:35.607 INFO:teuthology.orchestra.run.smithi089.stdout:2422 write 0x3bc8663 thru 0x3bcdba3 (0x5541 bytes) 2024-09-21T15:49:35.612 INFO:teuthology.orchestra.run.smithi089.stdout:2424 writesame 0xc2e8bec thru 0xc2f7e76 data_size 0x3(0xf28b bytes) 2024-09-21T15:49:35.626 INFO:teuthology.orchestra.run.smithi089.stdout:2425 writesame 0x14e6368 thru 0x14e9957 data_size 0x8(0x35f0 bytes) 2024-09-21T15:49:35.627 INFO:teuthology.orchestra.run.smithi089.stdout:2426 write 0x1e9f8d1 thru 0x1eaed64 (0xf494 bytes) 2024-09-21T15:49:35.639 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:35.639 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:49:35.639 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:49:35.639 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:49:35.639 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.639 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:983 3.0 e27/22 3:025e53e7:::rbd_data.1133f3dfbd6b.0000000000000008:head v 27'20, mlcod=27'18)", 2024-09-21T15:49:35.639 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.615392+0000", 2024-09-21T15:49:35.639 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.021626450999999, 2024-09-21T15:49:35.639 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032613970000000001, 2024-09-21T15:49:35.639 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:35.639 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:35.640 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:35.640 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.640 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:35.640 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.615392+0000", 2024-09-21T15:49:35.640 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:35.640 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.640 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.640 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:35.640 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.615392+0000", 2024-09-21T15:49:35.640 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:35.640 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.640 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.640 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:35.640 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.615390+0000", 2024-09-21T15:49:35.641 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:35.641 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.641 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.641 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:35.641 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.615431+0000", 2024-09-21T15:49:35.641 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0893999999999998e-05 2024-09-21T15:49:35.641 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.641 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.641 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:35.641 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.615450+0000", 2024-09-21T15:49:35.641 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8961e-05 2024-09-21T15:49:35.641 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.641 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.641 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:35.642 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.615458+0000", 2024-09-21T15:49:35.642 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1620000000000008e-06 2024-09-21T15:49:35.642 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.642 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.642 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:35.642 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.615501+0000", 2024-09-21T15:49:35.642 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2116999999999997e-05 2024-09-21T15:49:35.642 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.642 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.642 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:35.642 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.615518+0000", 2024-09-21T15:49:35.642 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7689e-05 2024-09-21T15:49:35.642 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.643 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.643 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:35.643 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.618621+0000", 2024-09-21T15:49:35.643 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0031022340000000002 2024-09-21T15:49:35.643 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.643 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.643 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:35.643 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.618653+0000", 2024-09-21T15:49:35.643 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2591e-05 2024-09-21T15:49:35.643 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:35.643 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:35.643 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:35.643 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.644 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.644 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:984 3.0 e27/22 3:025e53e7:::rbd_data.1133f3dfbd6b.0000000000000008:head v 27'21, mlcod=27'19)", 2024-09-21T15:49:35.644 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.619393+0000", 2024-09-21T15:49:35.644 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.017625624000001, 2024-09-21T15:49:35.644 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003401837, 2024-09-21T15:49:35.644 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:35.644 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:35.644 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:35.644 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.644 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:35.644 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619393+0000", 2024-09-21T15:49:35.644 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:35.644 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.644 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.644 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:35.645 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619393+0000", 2024-09-21T15:49:35.645 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:35.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.645 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.645 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:35.645 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619392+0000", 2024-09-21T15:49:35.645 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:35.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.645 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.645 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:35.645 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619439+0000", 2024-09-21T15:49:35.645 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7170999999999999e-05 2024-09-21T15:49:35.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.645 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.646 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:35.646 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619452+0000", 2024-09-21T15:49:35.646 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2986999999999999e-05 2024-09-21T15:49:35.646 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.646 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.646 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:35.646 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619459+0000", 2024-09-21T15:49:35.646 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6109999999999999e-06 2024-09-21T15:49:35.646 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.646 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.646 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:35.646 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619518+0000", 2024-09-21T15:49:35.646 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9117999999999999e-05 2024-09-21T15:49:35.646 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.647 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.647 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:35.647 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619532+0000", 2024-09-21T15:49:35.647 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4622e-05 2024-09-21T15:49:35.647 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.647 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.647 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:35.647 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.622760+0000", 2024-09-21T15:49:35.647 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032278839999999999 2024-09-21T15:49:35.647 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.647 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.647 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:35.647 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.622794+0000", 2024-09-21T15:49:35.647 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4144999999999999e-05 2024-09-21T15:49:35.648 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:35.648 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:35.648 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:35.648 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.648 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.648 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:4957 3.9 e29/22 3:95c5ed95:::rbd_data.11335fa2fe69.0000000000000018:head v 29'4, mlcod=26'2)", 2024-09-21T15:49:35.648 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:14.453951+0000", 2024-09-21T15:49:35.648 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 21.183067191999999, 2024-09-21T15:49:35.648 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0043804559999999996, 2024-09-21T15:49:35.648 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:35.648 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:35.648 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:35.648 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.648 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:35.648 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.453951+0000", 2024-09-21T15:49:35.649 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:35.649 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.649 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.649 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:35.649 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.453951+0000", 2024-09-21T15:49:35.649 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:35.649 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.649 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.649 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:35.649 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.453950+0000", 2024-09-21T15:49:35.649 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:35.649 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.649 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.649 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:35.649 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454495+0000", 2024-09-21T15:49:35.650 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00054437799999999998 2024-09-21T15:49:35.650 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.650 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.650 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:35.650 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454508+0000", 2024-09-21T15:49:35.650 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3475000000000001e-05 2024-09-21T15:49:35.650 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.650 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.650 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:35.650 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454515+0000", 2024-09-21T15:49:35.650 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8990000000000004e-06 2024-09-21T15:49:35.650 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.650 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.651 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:35.651 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454581+0000", 2024-09-21T15:49:35.651 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5987000000000004e-05 2024-09-21T15:49:35.651 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.651 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.651 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:35.651 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454609+0000", 2024-09-21T15:49:35.651 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7466000000000001e-05 2024-09-21T15:49:35.651 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.651 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.651 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:35.651 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.458303+0000", 2024-09-21T15:49:35.651 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003693979 2024-09-21T15:49:35.651 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.651 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.652 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:35.652 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.458331+0000", 2024-09-21T15:49:35.652 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8932999999999999e-05 2024-09-21T15:49:35.652 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:35.652 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:35.652 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:35.652 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.652 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.652 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13420 3.0 e31/22 3:038b8840:::rbd_data.1133dc5adfdf.0000000000000035:head v 31'49, mlcod=31'47)", 2024-09-21T15:49:35.652 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.209233+0000", 2024-09-21T15:49:35.652 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.427785576000002, 2024-09-21T15:49:35.652 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0042624960000000002, 2024-09-21T15:49:35.652 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:35.652 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:35.653 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:35.653 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.653 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:35.653 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209233+0000", 2024-09-21T15:49:35.653 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:35.653 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.653 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.653 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:35.653 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209233+0000", 2024-09-21T15:49:35.653 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:35.653 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.653 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.653 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:35.653 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209232+0000", 2024-09-21T15:49:35.653 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:35.654 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.654 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.654 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:35.654 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209251+0000", 2024-09-21T15:49:35.654 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9304e-05 2024-09-21T15:49:35.654 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.654 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.654 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:35.654 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209265+0000", 2024-09-21T15:49:35.654 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3532e-05 2024-09-21T15:49:35.654 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.654 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.655 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:35.655 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209272+0000", 2024-09-21T15:49:35.655 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.289e-06 2024-09-21T15:49:35.655 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.655 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.655 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:35.655 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209321+0000", 2024-09-21T15:49:35.655 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8773999999999998e-05 2024-09-21T15:49:35.655 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.655 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.655 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:35.655 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209341+0000", 2024-09-21T15:49:35.655 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0387999999999999e-05 2024-09-21T15:49:35.655 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.655 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.656 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:35.656 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.213464+0000", 2024-09-21T15:49:35.656 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0041232450000000002 2024-09-21T15:49:35.656 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.656 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.656 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:35.656 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.213495+0000", 2024-09-21T15:49:35.656 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0793999999999997e-05 2024-09-21T15:49:35.656 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:35.656 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:35.656 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:35.656 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.656 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.656 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13437 3.4 e31/22 3:23f0da5d:::rbd_data.1133dc5adfdf.000000000000002d:head v 31'15, mlcod=31'13)", 2024-09-21T15:49:35.657 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.227522+0000", 2024-09-21T15:49:35.657 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.409495742000001, 2024-09-21T15:49:35.657 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033127679999999998, 2024-09-21T15:49:35.657 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:35.657 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:35.657 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:35.657 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.657 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:35.657 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.227522+0000", 2024-09-21T15:49:35.657 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:35.657 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.657 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.657 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:35.657 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.227522+0000", 2024-09-21T15:49:35.658 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:35.658 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.658 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.658 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:35.658 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.227521+0000", 2024-09-21T15:49:35.658 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:35.658 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.658 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.658 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:35.658 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.227531+0000", 2024-09-21T15:49:35.658 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.8379999999999997e-06 2024-09-21T15:49:35.658 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.658 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.658 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:35.659 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.227552+0000", 2024-09-21T15:49:35.659 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1341000000000001e-05 2024-09-21T15:49:35.659 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.659 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.659 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:35.659 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.227561+0000", 2024-09-21T15:49:35.659 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9400000000000008e-06 2024-09-21T15:49:35.659 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.659 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.659 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:35.659 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.227612+0000", 2024-09-21T15:49:35.659 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0701000000000001e-05 2024-09-21T15:49:35.659 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.659 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.659 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:35.660 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.227633+0000", 2024-09-21T15:49:35.660 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1209999999999999e-05 2024-09-21T15:49:35.660 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.660 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.660 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:35.660 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.230799+0000", 2024-09-21T15:49:35.660 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003165616 2024-09-21T15:49:35.660 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.660 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.660 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:35.660 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.230835+0000", 2024-09-21T15:49:35.660 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6321e-05 2024-09-21T15:49:35.660 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:35.660 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:35.660 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:35.661 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.661 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.661 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13445 3.0 e31/22 3:038b8840:::rbd_data.1133dc5adfdf.0000000000000035:head v 31'50, mlcod=31'48)", 2024-09-21T15:49:35.661 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.232497+0000", 2024-09-21T15:49:35.661 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.404521564, 2024-09-21T15:49:35.661 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0040493559999999996, 2024-09-21T15:49:35.661 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:35.661 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:35.661 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:35.661 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.661 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:35.661 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232497+0000", 2024-09-21T15:49:35.661 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:35.661 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.661 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.661 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:35.662 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232497+0000", 2024-09-21T15:49:35.662 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:35.662 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.662 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.662 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:35.662 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232496+0000", 2024-09-21T15:49:35.662 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:35.662 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.662 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.662 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:35.662 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232503+0000", 2024-09-21T15:49:35.662 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7320000000000001e-06 2024-09-21T15:49:35.662 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.662 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.663 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:35.663 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232512+0000", 2024-09-21T15:49:35.663 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9259999999999995e-06 2024-09-21T15:49:35.663 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.663 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.663 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:35.663 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232520+0000", 2024-09-21T15:49:35.663 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3699999999999997e-06 2024-09-21T15:49:35.663 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.663 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.663 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:35.663 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232564+0000", 2024-09-21T15:49:35.663 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4128999999999997e-05 2024-09-21T15:49:35.663 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.664 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.664 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:35.664 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232585+0000", 2024-09-21T15:49:35.664 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0704999999999999e-05 2024-09-21T15:49:35.664 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.664 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.664 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:35.664 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.236507+0000", 2024-09-21T15:49:35.664 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0039218730000000002 2024-09-21T15:49:35.664 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.664 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.664 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:35.664 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.236546+0000", 2024-09-21T15:49:35.664 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9399e-05 2024-09-21T15:49:35.665 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:35.665 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:35.665 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:35.665 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.665 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.665 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13452 3.9 e31/22 3:93f5396a:::rbd_data.1133dc5adfdf.0000000000000078:head v 31'50, mlcod=31'47)", 2024-09-21T15:49:35.665 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.237263+0000", 2024-09-21T15:49:35.665 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.399754858000001, 2024-09-21T15:49:35.665 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033768180000000002, 2024-09-21T15:49:35.665 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:35.665 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:35.665 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:35.665 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.665 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:35.666 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237263+0000", 2024-09-21T15:49:35.666 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:35.666 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.666 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.666 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:35.666 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237263+0000", 2024-09-21T15:49:35.666 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:35.666 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.666 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.666 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:35.666 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237263+0000", 2024-09-21T15:49:35.666 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:35.666 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.667 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.667 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:35.667 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237269+0000", 2024-09-21T15:49:35.667 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.472e-06 2024-09-21T15:49:35.667 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.667 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.667 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:35.667 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237282+0000", 2024-09-21T15:49:35.667 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2788e-05 2024-09-21T15:49:35.667 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.667 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.667 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:35.667 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237288+0000", 2024-09-21T15:49:35.667 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4899999999999997e-06 2024-09-21T15:49:35.668 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.668 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.668 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:35.668 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237331+0000", 2024-09-21T15:49:35.668 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2425000000000001e-05 2024-09-21T15:49:35.668 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.668 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.668 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:35.668 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237349+0000", 2024-09-21T15:49:35.668 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8405000000000001e-05 2024-09-21T15:49:35.668 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.668 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.668 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:35.668 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.240610+0000", 2024-09-21T15:49:35.669 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032605379999999999 2024-09-21T15:49:35.669 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.669 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.669 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:35.669 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.240640+0000", 2024-09-21T15:49:35.669 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0448000000000001e-05 2024-09-21T15:49:35.669 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:35.669 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:35.669 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:35.669 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.669 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.669 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13832 3.0 e31/22 3:0792efd2:::rbd_data.1133dc5adfdf.0000000000000090:head v 31'58, mlcod=31'56)", 2024-09-21T15:49:35.669 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.527530+0000", 2024-09-21T15:49:35.669 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.109488037999999, 2024-09-21T15:49:35.670 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035256530000000001, 2024-09-21T15:49:35.670 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:35.670 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:35.670 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:35.670 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.670 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:35.670 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527530+0000", 2024-09-21T15:49:35.670 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:35.670 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.672 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.672 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:35.672 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527530+0000", 2024-09-21T15:49:35.672 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:35.672 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.672 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.672 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:35.672 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527529+0000", 2024-09-21T15:49:35.672 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:35.672 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.673 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.673 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:35.673 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527550+0000", 2024-09-21T15:49:35.673 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1387999999999999e-05 2024-09-21T15:49:35.673 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.673 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.673 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:35.673 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527568+0000", 2024-09-21T15:49:35.673 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8068e-05 2024-09-21T15:49:35.673 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.673 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.673 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:35.673 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527577+0000", 2024-09-21T15:49:35.673 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9919999999999996e-06 2024-09-21T15:49:35.674 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.674 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.674 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:35.674 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527626+0000", 2024-09-21T15:49:35.674 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8597000000000002e-05 2024-09-21T15:49:35.674 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.674 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.674 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:35.674 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527652+0000", 2024-09-21T15:49:35.674 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5993e-05 2024-09-21T15:49:35.674 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.674 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.674 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:35.674 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.531024+0000", 2024-09-21T15:49:35.675 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033721910000000001 2024-09-21T15:49:35.675 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.675 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.675 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:35.675 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.531056+0000", 2024-09-21T15:49:35.675 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1628000000000003e-05 2024-09-21T15:49:35.675 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:35.675 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:35.675 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:35.675 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.675 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.675 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13839 3.0 e31/22 3:03c4fb32:::rbd_data.1133dc5adfdf.0000000000000099:head v 31'59, mlcod=31'56)", 2024-09-21T15:49:35.675 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.528097+0000", 2024-09-21T15:49:35.675 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.10892144, 2024-09-21T15:49:35.675 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0057228840000000001, 2024-09-21T15:49:35.676 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:35.676 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:35.676 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:35.676 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.676 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:35.676 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528097+0000", 2024-09-21T15:49:35.676 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:35.676 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.676 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.676 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:35.676 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528097+0000", 2024-09-21T15:49:35.676 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:35.676 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.676 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.676 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:35.677 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528096+0000", 2024-09-21T15:49:35.677 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:35.677 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.677 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.677 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:35.677 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528105+0000", 2024-09-21T15:49:35.677 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7350000000000002e-06 2024-09-21T15:49:35.677 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.677 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.677 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:35.677 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528117+0000", 2024-09-21T15:49:35.677 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1909000000000001e-05 2024-09-21T15:49:35.677 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.677 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.677 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:35.678 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528124+0000", 2024-09-21T15:49:35.678 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8419999999999999e-06 2024-09-21T15:49:35.678 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.678 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.678 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:35.678 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528168+0000", 2024-09-21T15:49:35.678 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4301999999999997e-05 2024-09-21T15:49:35.678 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.678 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.678 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:35.678 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528191+0000", 2024-09-21T15:49:35.678 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2622000000000001e-05 2024-09-21T15:49:35.678 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.678 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.679 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:35.679 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.533780+0000", 2024-09-21T15:49:35.679 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0055897519999999999 2024-09-21T15:49:35.679 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.679 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.679 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:35.679 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.533820+0000", 2024-09-21T15:49:35.679 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9366999999999998e-05 2024-09-21T15:49:35.679 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:35.679 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:35.679 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:35.679 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.679 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.679 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13884 3.9 e31/22 3:93003736:::rbd_data.1133dc5adfdf.00000000000000a1:head v 31'54, mlcod=31'52)", 2024-09-21T15:49:35.679 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.562433+0000", 2024-09-21T15:49:35.680 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.074584691999998, 2024-09-21T15:49:35.680 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.005162313, 2024-09-21T15:49:35.680 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:35.680 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:35.680 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:35.680 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.680 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:35.680 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562433+0000", 2024-09-21T15:49:35.680 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:35.680 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.680 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.680 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:35.680 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562433+0000", 2024-09-21T15:49:35.680 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:35.680 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.680 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.681 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:35.681 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562432+0000", 2024-09-21T15:49:35.681 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:35.681 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.681 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.681 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:35.681 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562467+0000", 2024-09-21T15:49:35.681 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4885999999999998e-05 2024-09-21T15:49:35.681 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.681 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.681 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:35.681 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562481+0000", 2024-09-21T15:49:35.681 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3604000000000001e-05 2024-09-21T15:49:35.681 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.681 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.682 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:35.682 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562489+0000", 2024-09-21T15:49:35.682 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1829999999999994e-06 2024-09-21T15:49:35.682 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.682 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.682 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:35.682 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562536+0000", 2024-09-21T15:49:35.682 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6830999999999999e-05 2024-09-21T15:49:35.682 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.682 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.682 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:35.682 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562559+0000", 2024-09-21T15:49:35.682 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2765999999999999e-05 2024-09-21T15:49:35.682 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.682 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.683 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:35.683 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.567566+0000", 2024-09-21T15:49:35.683 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0050073219999999998 2024-09-21T15:49:35.683 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.683 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.683 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:35.683 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.567596+0000", 2024-09-21T15:49:35.683 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9802999999999999e-05 2024-09-21T15:49:35.683 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:35.683 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:35.683 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:35.683 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.683 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.683 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13908 3.9 e31/22 3:922f1ce0:::rbd_data.1133dc5adfdf.00000000000000e9:head v 31'55, mlcod=31'53)", 2024-09-21T15:49:35.684 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.581150+0000", 2024-09-21T15:49:35.684 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.055868521000001, 2024-09-21T15:49:35.684 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033342329999999998, 2024-09-21T15:49:35.684 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:35.684 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:35.684 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:35.684 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.684 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:35.684 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.581150+0000", 2024-09-21T15:49:35.684 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:35.684 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.684 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.684 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:35.684 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.581150+0000", 2024-09-21T15:49:35.685 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:35.685 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.685 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.685 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:35.685 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.581149+0000", 2024-09-21T15:49:35.685 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:35.685 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.685 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.685 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:35.685 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.581167+0000", 2024-09-21T15:49:35.685 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7948000000000002e-05 2024-09-21T15:49:35.685 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.685 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.685 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:35.686 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.581181+0000", 2024-09-21T15:49:35.686 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4392e-05 2024-09-21T15:49:35.686 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.686 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.686 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:35.686 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.581189+0000", 2024-09-21T15:49:35.686 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7640000000000005e-06 2024-09-21T15:49:35.686 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.686 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.686 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:35.686 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.581232+0000", 2024-09-21T15:49:35.686 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2639999999999998e-05 2024-09-21T15:49:35.686 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.686 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.687 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:35.687 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.581255+0000", 2024-09-21T15:49:35.687 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3802999999999999e-05 2024-09-21T15:49:35.687 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.687 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.687 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:35.687 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.584445+0000", 2024-09-21T15:49:35.687 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0031900380000000001 2024-09-21T15:49:35.687 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.687 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.687 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:35.687 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.584484+0000", 2024-09-21T15:49:35.687 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8472e-05 2024-09-21T15:49:35.687 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:35.687 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:35.688 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:35.688 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.688 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.688 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:22340 3.9 e34/22 3:910a5abe:::rbd_data.1133896adcfd.0000000000000023:head v 34'92, mlcod=34'90)", 2024-09-21T15:49:35.688 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:21.628025+0000", 2024-09-21T15:49:35.688 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.008993311999999, 2024-09-21T15:49:35.688 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046196010000000001, 2024-09-21T15:49:35.688 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:35.688 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:35.688 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:35.688 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.688 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:35.688 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628025+0000", 2024-09-21T15:49:35.688 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:35.688 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.688 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.689 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:35.689 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628025+0000", 2024-09-21T15:49:35.689 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:35.689 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.689 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.689 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:35.689 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628024+0000", 2024-09-21T15:49:35.689 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:35.689 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.689 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.689 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:35.689 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628034+0000", 2024-09-21T15:49:35.689 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3470000000000005e-06 2024-09-21T15:49:35.689 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.690 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.690 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:35.690 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628043+0000", 2024-09-21T15:49:35.690 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9450000000000005e-06 2024-09-21T15:49:35.690 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.690 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.690 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:35.690 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628048+0000", 2024-09-21T15:49:35.690 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8080000000000004e-06 2024-09-21T15:49:35.690 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.690 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.690 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:35.690 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628081+0000", 2024-09-21T15:49:35.690 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2738999999999999e-05 2024-09-21T15:49:35.691 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.691 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.691 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:35.691 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628094+0000", 2024-09-21T15:49:35.691 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3273e-05 2024-09-21T15:49:35.691 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.691 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.691 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:35.691 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.632616+0000", 2024-09-21T15:49:35.691 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045217269999999997 2024-09-21T15:49:35.691 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.691 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.691 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:35.691 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.632644+0000", 2024-09-21T15:49:35.692 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8469000000000001e-05 2024-09-21T15:49:35.692 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:35.692 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:35.692 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:35.692 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.692 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.692 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:22417 3.4 e34/22 3:25310a10:::rbd_data.1133896adcfd.0000000000000075:head v 34'48, mlcod=34'46)", 2024-09-21T15:49:35.692 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:21.704297+0000", 2024-09-21T15:49:35.692 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.932721214000001, 2024-09-21T15:49:35.692 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0038101799999999998, 2024-09-21T15:49:35.692 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:35.692 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:35.692 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:35.692 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.692 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:35.693 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704297+0000", 2024-09-21T15:49:35.693 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:35.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.693 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.693 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:35.693 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704297+0000", 2024-09-21T15:49:35.693 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:35.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.693 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.693 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:35.693 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704296+0000", 2024-09-21T15:49:35.693 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:35.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.693 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.694 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:35.694 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704308+0000", 2024-09-21T15:49:35.694 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2559e-05 2024-09-21T15:49:35.694 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.694 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.694 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:35.694 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704337+0000", 2024-09-21T15:49:35.694 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8960000000000001e-05 2024-09-21T15:49:35.694 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.694 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.694 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:35.694 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704345+0000", 2024-09-21T15:49:35.694 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3059999999999998e-06 2024-09-21T15:49:35.694 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.695 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.695 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:35.695 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704408+0000", 2024-09-21T15:49:35.695 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2156000000000006e-05 2024-09-21T15:49:35.695 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.695 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.695 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:35.695 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704430+0000", 2024-09-21T15:49:35.695 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1931000000000001e-05 2024-09-21T15:49:35.695 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.695 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.695 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:35.695 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.708073+0000", 2024-09-21T15:49:35.695 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003643028 2024-09-21T15:49:35.696 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.696 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.696 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:35.696 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.708107+0000", 2024-09-21T15:49:35.696 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4613999999999998e-05 2024-09-21T15:49:35.696 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:35.696 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:35.696 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:35.696 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.696 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.696 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:25278 3.4 e38/22 3:2251cc1c:::rbd_data.1133ebd6d3ba.0000000000000024:head v 38'51, mlcod=38'49)", 2024-09-21T15:49:35.696 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.317337+0000", 2024-09-21T15:49:35.696 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.3196814539999995, 2024-09-21T15:49:35.696 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053904110000000003, 2024-09-21T15:49:35.697 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:35.697 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:35.697 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:35.697 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.697 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:35.697 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317337+0000", 2024-09-21T15:49:35.697 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:35.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.697 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.697 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:35.697 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317337+0000", 2024-09-21T15:49:35.697 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:35.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.697 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.698 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:35.698 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317335+0000", 2024-09-21T15:49:35.698 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:49:35.698 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.698 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.698 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:35.698 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317347+0000", 2024-09-21T15:49:35.698 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.19e-05 2024-09-21T15:49:35.698 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.698 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.698 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:35.698 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317369+0000", 2024-09-21T15:49:35.698 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2232e-05 2024-09-21T15:49:35.698 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.698 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.699 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:35.699 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317381+0000", 2024-09-21T15:49:35.699 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2231e-05 2024-09-21T15:49:35.699 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.699 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.699 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:35.699 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317425+0000", 2024-09-21T15:49:35.699 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3625000000000003e-05 2024-09-21T15:49:35.699 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.699 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.699 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:35.699 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317449+0000", 2024-09-21T15:49:35.699 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4301999999999999e-05 2024-09-21T15:49:35.699 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.700 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.700 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:35.700 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.322688+0000", 2024-09-21T15:49:35.700 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.005238609 2024-09-21T15:49:35.700 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.700 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.700 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:35.700 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.322727+0000", 2024-09-21T15:49:35.700 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9201e-05 2024-09-21T15:49:35.700 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:35.700 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:35.700 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:35.700 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.700 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.701 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:25286 3.0 e38/22 3:0248fb60:::rbd_data.1133ebd6d3ba.0000000000000066:head v 38'109, mlcod=34'107)", 2024-09-21T15:49:35.701 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.317638+0000", 2024-09-21T15:49:35.701 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.3193805750000003, 2024-09-21T15:49:35.701 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0054074370000000002, 2024-09-21T15:49:35.701 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:35.701 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:35.701 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:35.701 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.701 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:35.701 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317638+0000", 2024-09-21T15:49:35.701 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:35.701 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.701 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.701 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:35.701 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317638+0000", 2024-09-21T15:49:35.702 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:35.702 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.702 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.702 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:35.702 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317637+0000", 2024-09-21T15:49:35.702 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:35.702 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.702 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.702 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:35.702 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317643+0000", 2024-09-21T15:49:35.702 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5609999999999998e-06 2024-09-21T15:49:35.702 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.702 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.702 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:35.703 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317653+0000", 2024-09-21T15:49:35.703 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0705e-05 2024-09-21T15:49:35.703 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.703 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.703 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:35.703 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317660+0000", 2024-09-21T15:49:35.703 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2099999999999998e-06 2024-09-21T15:49:35.703 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.703 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.703 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:35.703 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317704+0000", 2024-09-21T15:49:35.703 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4659000000000001e-05 2024-09-21T15:49:35.703 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.703 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.703 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:35.704 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317725+0000", 2024-09-21T15:49:35.704 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0537e-05 2024-09-21T15:49:35.704 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.704 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.704 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:35.704 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323010+0000", 2024-09-21T15:49:35.704 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052853420000000002 2024-09-21T15:49:35.704 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.704 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.704 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:35.704 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323045+0000", 2024-09-21T15:49:35.704 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4860000000000002e-05 2024-09-21T15:49:35.704 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:35.704 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:35.705 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:35.705 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.705 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.705 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:25280 3.4 e38/22 3:27fca38c:::rbd_data.1133ebd6d3ba.0000000000000026:head v 38'52, mlcod=38'49)", 2024-09-21T15:49:35.705 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.317684+0000", 2024-09-21T15:49:35.705 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.3193346790000007, 2024-09-21T15:49:35.705 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053570809999999996, 2024-09-21T15:49:35.705 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:35.705 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:35.705 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:35.705 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.705 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:35.705 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317684+0000", 2024-09-21T15:49:35.706 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:35.706 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.706 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.706 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:35.706 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317684+0000", 2024-09-21T15:49:35.706 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:35.706 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.706 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.706 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:35.706 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317683+0000", 2024-09-21T15:49:35.706 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:35.706 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.706 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.706 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:35.707 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317688+0000", 2024-09-21T15:49:35.707 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5009999999999998e-06 2024-09-21T15:49:35.707 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.707 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.707 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:35.707 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317694+0000", 2024-09-21T15:49:35.707 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3579999999999999e-06 2024-09-21T15:49:35.707 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.707 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.707 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:35.707 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317698+0000", 2024-09-21T15:49:35.707 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4150000000000001e-06 2024-09-21T15:49:35.707 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.707 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.708 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:35.708 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317738+0000", 2024-09-21T15:49:35.708 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9575e-05 2024-09-21T15:49:35.708 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.708 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.708 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:35.708 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317762+0000", 2024-09-21T15:49:35.708 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4173999999999999e-05 2024-09-21T15:49:35.708 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.708 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.708 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:35.708 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323005+0000", 2024-09-21T15:49:35.708 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00524333 2024-09-21T15:49:35.708 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.708 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.709 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:35.709 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323041+0000", 2024-09-21T15:49:35.709 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5139999999999999e-05 2024-09-21T15:49:35.709 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:35.709 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:35.709 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:35.709 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.709 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.709 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:34837 3.0 e43/22 3:038527c8:::rbd_header.1133fed1a840:head v 43'130, mlcod=43'128)", 2024-09-21T15:49:35.709 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:49:35.709 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.6531299810000002, 2024-09-21T15:49:35.709 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045776519999999998, 2024-09-21T15:49:35.709 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:35.709 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:35.710 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:35.710 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.710 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:35.710 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:49:35.710 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:35.710 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.710 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.710 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:35.710 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:49:35.710 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:35.710 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.710 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.710 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:35.710 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:49:35.711 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:35.711 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.711 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.711 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:35.711 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983892+0000", 2024-09-21T15:49:35.711 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8450000000000002e-06 2024-09-21T15:49:35.711 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.711 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.711 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:35.711 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983902+0000", 2024-09-21T15:49:35.711 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0299999999999999e-06 2024-09-21T15:49:35.711 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.711 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.711 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:35.712 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983906+0000", 2024-09-21T15:49:35.712 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2300000000000002e-06 2024-09-21T15:49:35.712 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.712 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.712 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:35.712 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983940+0000", 2024-09-21T15:49:35.712 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4563000000000001e-05 2024-09-21T15:49:35.712 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.712 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.712 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:35.712 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983960+0000", 2024-09-21T15:49:35.712 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9199999999999999e-05 2024-09-21T15:49:35.712 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.712 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.713 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:35.713 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.988430+0000", 2024-09-21T15:49:35.713 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0044709149999999998 2024-09-21T15:49:35.713 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.713 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.713 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:35.713 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.988466+0000", 2024-09-21T15:49:35.713 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5420000000000003e-05 2024-09-21T15:49:35.713 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:35.713 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:35.713 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:35.713 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.713 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.713 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:57718 3.9 e46/22 3:909db603:::rbd_data.1133a7d294ca.00000000000000ae:head v 46'207, mlcod=46'205)", 2024-09-21T15:49:35.714 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:35.291927+0000", 2024-09-21T15:49:35.714 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.34509140999999999, 2024-09-21T15:49:35.714 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003293514, 2024-09-21T15:49:35.714 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:35.714 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:35.714 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:35.714 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.714 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:35.714 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:35.291927+0000", 2024-09-21T15:49:35.714 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:35.714 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.714 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.714 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:35.714 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:35.291927+0000", 2024-09-21T15:49:35.715 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:35.715 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.715 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.715 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:35.715 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:35.291926+0000", 2024-09-21T15:49:35.715 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:35.715 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.715 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.715 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:35.715 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:35.291951+0000", 2024-09-21T15:49:35.715 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5570000000000001e-05 2024-09-21T15:49:35.715 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.715 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.715 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:35.715 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:35.291965+0000", 2024-09-21T15:49:35.716 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3781000000000001e-05 2024-09-21T15:49:35.716 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.716 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.716 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:35.716 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:35.291971+0000", 2024-09-21T15:49:35.716 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1330000000000004e-06 2024-09-21T15:49:35.716 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.716 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.716 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:35.716 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:35.292013+0000", 2024-09-21T15:49:35.716 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1409000000000003e-05 2024-09-21T15:49:35.716 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.716 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.716 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:35.716 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:35.292028+0000", 2024-09-21T15:49:35.717 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5885000000000001e-05 2024-09-21T15:49:35.717 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.717 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.717 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:35.717 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:35.295191+0000", 2024-09-21T15:49:35.717 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0031626779999999999 2024-09-21T15:49:35.717 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.717 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.717 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:35.717 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:35.295220+0000", 2024-09-21T15:49:35.717 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9138000000000001e-05 2024-09-21T15:49:35.717 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:35.717 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:35.717 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:35.717 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.718 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.718 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:57819 3.4 e46/22 3:205b7228:::rbd_data.1133a7d294ca.00000000000000b4:head v 46'181, mlcod=46'179)", 2024-09-21T15:49:35.718 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:35.449459+0000", 2024-09-21T15:49:35.718 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.187559263, 2024-09-21T15:49:35.718 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033066530000000001, 2024-09-21T15:49:35.718 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:35.718 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:35.718 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:35.718 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.718 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:35.718 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:35.449459+0000", 2024-09-21T15:49:35.718 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:35.718 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.718 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.718 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:35.719 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:35.449459+0000", 2024-09-21T15:49:35.719 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:35.719 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.719 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.719 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:35.719 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:35.449458+0000", 2024-09-21T15:49:35.719 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:35.719 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.719 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.719 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:35.719 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:35.449481+0000", 2024-09-21T15:49:35.719 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3702000000000001e-05 2024-09-21T15:49:35.719 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.719 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.719 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:35.719 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:35.449498+0000", 2024-09-21T15:49:35.720 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6070999999999999e-05 2024-09-21T15:49:35.720 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.720 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.720 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:35.720 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:35.449504+0000", 2024-09-21T15:49:35.720 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4069999999999998e-06 2024-09-21T15:49:35.720 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.720 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.720 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:35.720 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:35.449557+0000", 2024-09-21T15:49:35.720 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3402999999999997e-05 2024-09-21T15:49:35.720 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.720 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.720 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:35.720 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:35.449583+0000", 2024-09-21T15:49:35.720 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5644999999999999e-05 2024-09-21T15:49:35.721 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.721 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.721 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:35.721 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:35.452738+0000", 2024-09-21T15:49:35.721 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0031548600000000002 2024-09-21T15:49:35.721 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.721 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.721 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:35.721 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:35.452766+0000", 2024-09-21T15:49:35.721 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7750999999999999e-05 2024-09-21T15:49:35.721 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:35.721 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:35.721 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:35.721 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.721 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.722 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:57835 3.4 e46/22 3:205b7228:::rbd_data.1133a7d294ca.00000000000000b4:head v 46'183, mlcod=46'180)", 2024-09-21T15:49:35.722 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:35.470418+0000", 2024-09-21T15:49:35.722 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.166599685, 2024-09-21T15:49:35.722 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032325510000000002, 2024-09-21T15:49:35.722 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:35.722 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:35.722 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:35.722 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.722 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:35.722 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:35.470418+0000", 2024-09-21T15:49:35.722 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:35.722 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.722 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.722 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:35.722 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:35.470418+0000", 2024-09-21T15:49:35.723 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:35.723 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.723 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.723 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:35.723 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:35.470418+0000", 2024-09-21T15:49:35.723 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:35.723 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.723 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.723 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:35.723 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:35.470423+0000", 2024-09-21T15:49:35.723 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5210000000000002e-06 2024-09-21T15:49:35.723 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.723 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.723 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:35.724 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:35.470432+0000", 2024-09-21T15:49:35.724 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9940000000000006e-06 2024-09-21T15:49:35.724 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.724 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.724 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:35.724 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:35.470438+0000", 2024-09-21T15:49:35.724 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0260000000000004e-06 2024-09-21T15:49:35.724 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.724 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.724 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:35.724 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:35.470480+0000", 2024-09-21T15:49:35.724 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1093000000000001e-05 2024-09-21T15:49:35.724 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.724 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.724 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:35.725 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:35.470498+0000", 2024-09-21T15:49:35.725 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.808e-05 2024-09-21T15:49:35.725 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.725 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.725 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:35.725 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:35.473617+0000", 2024-09-21T15:49:35.725 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0031198440000000001 2024-09-21T15:49:35.725 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:35.725 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:35.725 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:35.725 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:35.473651+0000", 2024-09-21T15:49:35.725 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3599000000000001e-05 2024-09-21T15:49:35.725 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:35.725 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:35.725 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:35.725 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:35.726 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:35.726 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:35.726 INFO:teuthology.orchestra.run.smithi089.stdout:2428 trunc from 0xddfe2e5 to 0x3874aaf 2024-09-21T15:49:35.727 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:49:35.729 INFO:teuthology.orchestra.run.smithi089.stdout:2431 punch from 0x3860157 to 0x386e20c, (0xe0b5 bytes) 2024-09-21T15:49:35.729 INFO:teuthology.orchestra.run.smithi089.stdout:2432 write 0x9adb06d thru 0x9ae2b8e (0x7b22 bytes) 2024-09-21T15:49:35.729 INFO:teuthology.orchestra.run.smithi089.stdout:2435 punch from 0x4ef59a to 0x4f5c41, (0x66a7 bytes) 2024-09-21T15:49:35.729 INFO:teuthology.orchestra.run.smithi089.stdout:2436 read 0x14985d6 thru 0x14a0d0a (0x8735 bytes) 2024-09-21T15:49:35.729 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:35.729 INFO:teuthology.orchestra.run.smithi089.stdout:2439 punch from 0x25db3bd to 0x25e4d5f, (0x99a2 bytes) 2024-09-21T15:49:35.729 INFO:teuthology.orchestra.run.smithi089.stdout:2441 read 0x2a7633 thru 0x2b5507 (0xded5 bytes) 2024-09-21T15:49:35.729 INFO:teuthology.orchestra.run.smithi089.stdout:2443 write 0xc903a82 thru 0xc90fd9a (0xc319 bytes) 2024-09-21T15:49:35.729 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:35.729 INFO:teuthology.orchestra.run.smithi089.stdout:2445 writesame 0x32b0cf thru 0x3324ae data_size 0x30(0x73e0 bytes) 2024-09-21T15:49:35.729 INFO:teuthology.orchestra.run.smithi089.stdout:2448 writesame 0xec4094b thru 0xec4140b data_size 0x1(0xac1 bytes) 2024-09-21T15:49:35.729 INFO:teuthology.orchestra.run.smithi089.stdout:2449 punch from 0x829f3d1 to 0x82ad7fd, (0xe42c bytes) 2024-09-21T15:49:35.730 INFO:teuthology.orchestra.run.smithi089.stdout:2450 write 0xb52584f thru 0xb5302f9 (0xaaab bytes) 2024-09-21T15:49:35.730 INFO:teuthology.orchestra.run.smithi089.stdout:2451 writesame 0xd5c53b8 thru 0xd5d1373 data_size 0xe(0xbfbc bytes) 2024-09-21T15:49:35.730 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:35.725+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:35.734 INFO:teuthology.orchestra.run.smithi089.stdout:2452 trunc from 0xec4140c to 0x7f63828 2024-09-21T15:49:35.745 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:35.745 INFO:teuthology.orchestra.run.smithi089.stdout:2454 read 0x5883f91 thru 0x589320b (0xf27b bytes) 2024-09-21T15:49:35.745 INFO:teuthology.orchestra.run.smithi089.stdout:2457 read 0x6b7494f thru 0x6b75e51 (0x1503 bytes) 2024-09-21T15:49:35.752 INFO:teuthology.orchestra.run.smithi089.stdout:2459 trunc from 0x7f63828 to 0x1cac219 2024-09-21T15:49:35.767 INFO:teuthology.orchestra.run.smithi089.stdout:2460 punch from 0x13d3b96 to 0x13df464, (0xb8ce bytes) 2024-09-21T15:49:35.767 INFO:teuthology.orchestra.run.smithi089.stdout:2461 read 0x137aab1 thru 0x13867f2 (0xbd42 bytes) 2024-09-21T15:49:35.775 INFO:teuthology.orchestra.run.smithi089.stdout:2463 write 0x45d8d1c thru 0x45e6448 (0xd72d bytes) 2024-09-21T15:49:35.779 INFO:teuthology.orchestra.run.smithi089.stdout:2464 read 0x411c9e0 thru 0x412474c (0x7d6d bytes) 2024-09-21T15:49:35.794 INFO:teuthology.orchestra.run.smithi089.stdout:2465 writesame 0xc190c1a thru 0xc19ad5f data_size 0x15(0xa146 bytes) 2024-09-21T15:49:35.794 INFO:teuthology.orchestra.run.smithi089.stdout:2466 read 0x18ed0b7 thru 0x18f7a29 (0xa973 bytes) 2024-09-21T15:49:35.794 INFO:teuthology.orchestra.run.smithi089.stdout:2468 write 0x8ca9920 thru 0x8cb9082 (0xf763 bytes) 2024-09-21T15:49:35.809 INFO:teuthology.orchestra.run.smithi089.stdout:2469 trunc from 0xc19ad60 to 0x32b284a 2024-09-21T15:49:35.819 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:35.819 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:35.823 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T15:49:35.829 INFO:teuthology.orchestra.run.smithi089.stdout:2472 trunc from 0x32b284a to 0xb9498fb 2024-09-21T15:49:35.830 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:35.829+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:35.837 INFO:teuthology.orchestra.run.smithi089.stdout:2473 trunc from 0xb9498fb to 0x8eb6087 2024-09-21T15:49:35.842 INFO:teuthology.orchestra.run.smithi089.stdout:2474 punch from 0x7b4b707 to 0x7b56b4e, (0xb447 bytes) 2024-09-21T15:49:35.851 INFO:teuthology.orchestra.run.smithi089.stdout:2475 trunc from 0x8eb6087 to 0x1f21f8e 2024-09-21T15:49:35.853 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:35.853 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:49:35.853 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:49:35.853 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:35.855 INFO:teuthology.orchestra.run.smithi089.stdout:2477 punch from 0x1a6dec8 to 0x1a7c6fe, (0xe836 bytes) 2024-09-21T15:49:35.864 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:49:35.865 INFO:teuthology.orchestra.run.smithi089.stdout:2478 write 0x5a1175e thru 0x5a139a0 (0x2243 bytes) 2024-09-21T15:49:35.868 INFO:teuthology.orchestra.run.smithi089.stdout:2479 read 0x1489782 thru 0x1489840 (0xbf bytes) 2024-09-21T15:49:35.868 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:35.882 INFO:teuthology.orchestra.run.smithi089.stdout:2481 write 0xb43b1ac thru 0xb43ff00 (0x4d55 bytes) 2024-09-21T15:49:35.885 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:35.885 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:35.886 INFO:teuthology.orchestra.run.smithi089.stdout:2485 read 0x520e15e thru 0x521d77c (0xf61f bytes) 2024-09-21T15:49:35.886 INFO:teuthology.orchestra.run.smithi089.stdout:2486 trunc from 0xb43ff01 to 0xb020d32 2024-09-21T15:49:35.894 INFO:teuthology.orchestra.run.smithi089.stdout:2487 punch from 0x655edd0 to 0x656405e, (0x528e bytes) 2024-09-21T15:49:35.894 INFO:teuthology.orchestra.run.smithi089.stdout:2488 write 0x5849d65 thru 0x584f626 (0x58c2 bytes) 2024-09-21T15:49:35.897 INFO:teuthology.orchestra.run.smithi089.stdout:2489 read 0x794038f thru 0x7945b9e (0x5810 bytes) 2024-09-21T15:49:35.897 INFO:teuthology.orchestra.run.smithi089.stdout:2490 write 0x329ee5f thru 0x32ae832 (0xf9d4 bytes) 2024-09-21T15:49:35.909 INFO:teuthology.orchestra.run.smithi089.stdout:2491 trunc from 0xb020d32 to 0x56e5d03 2024-09-21T15:49:35.923 INFO:teuthology.orchestra.run.smithi089.stdout:2493 read 0xa3ccd0 thru 0xa4846f (0xb7a0 bytes) 2024-09-21T15:49:35.931 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:35.930+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:35.932 INFO:teuthology.orchestra.run.smithi089.stdout:2494 write 0x6fc4e33 thru 0x6fd36e8 (0xe8b6 bytes) 2024-09-21T15:49:35.937 INFO:teuthology.orchestra.run.smithi089.stdout:2496 trunc from 0x6fd36e9 to 0x81bc503 2024-09-21T15:49:35.942 INFO:teuthology.orchestra.run.smithi089.stdout:2499 write 0x8136422 thru 0x813b3d2 (0x4fb1 bytes) 2024-09-21T15:49:35.945 INFO:teuthology.orchestra.run.smithi089.stdout:2500 read 0xe8e48c thru 0xe8f6fa (0x126f bytes) 2024-09-21T15:49:35.946 INFO:teuthology.orchestra.run.smithi089.stdout:2501 writesame 0x25a7fd3 thru 0x25b7a83 data_size 0x1(0xfab1 bytes) 2024-09-21T15:49:35.981 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:35.981 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:49:35.981 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:49:35.981 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:49:35.981 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:35.993 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:49:35.994 INFO:teuthology.orchestra.run.smithi089.stdout:2502 write 0xe06996f thru 0xe06b958 (0x1fea bytes) 2024-09-21T15:49:35.997 INFO:teuthology.orchestra.run.smithi089.stdout:2504 read 0x1e0b566 thru 0x1e19be5 (0xe680 bytes) 2024-09-21T15:49:35.998 INFO:teuthology.orchestra.run.smithi089.stdout:2505 punch from 0x1e23ef0 to 0x1e2ae22, (0x6f32 bytes) 2024-09-21T15:49:35.998 INFO:teuthology.orchestra.run.smithi089.stdout:2506 read 0x12e1594 thru 0x12e766b (0x60d8 bytes) 2024-09-21T15:49:35.998 INFO:teuthology.orchestra.run.smithi089.stdout:2507 write 0x3285005 thru 0x328f02c (0xa028 bytes) 2024-09-21T15:49:36.000 INFO:teuthology.orchestra.run.smithi089.stdout:2509 read 0xc56ec05 thru 0xc57bcb3 (0xd0af bytes) 2024-09-21T15:49:36.014 INFO:teuthology.orchestra.run.smithi089.stdout:2510 trunc from 0xe06b959 to 0x29543bc 2024-09-21T15:49:36.026 INFO:teuthology.orchestra.run.smithi089.stdout:2511 read 0x265c77f thru 0x2661148 (0x49ca bytes) 2024-09-21T15:49:36.028 INFO:teuthology.orchestra.run.smithi089.stdout:2512 trunc from 0x29543bc to 0x173da63 2024-09-21T15:49:36.032 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:36.031+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:36.044 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:36.066 INFO:teuthology.orchestra.run.smithi089.stdout:2515 write 0xe6e8bcc thru 0xe6f7c47 (0xf07c bytes) 2024-09-21T15:49:36.070 INFO:teuthology.orchestra.run.smithi089.stdout:2516 punch from 0x3e08bc8 to 0x3e0d787, (0x4bbf bytes) 2024-09-21T15:49:36.071 INFO:teuthology.orchestra.run.smithi089.stdout:2517 read 0x5fbfb2d thru 0x5fc5a55 (0x5f29 bytes) 2024-09-21T15:49:36.071 INFO:teuthology.orchestra.run.smithi089.stdout:2518 punch from 0x322778b to 0x3230dfa, (0x966f bytes) 2024-09-21T15:49:36.071 INFO:teuthology.orchestra.run.smithi089.stdout:2519 read 0x4688261 thru 0x4695d72 (0xdb12 bytes) 2024-09-21T15:49:36.071 INFO:teuthology.orchestra.run.smithi089.stdout:2520 writesame 0xc2ec897 thru 0xc2f1bca data_size 0x64(0x5334 bytes) 2024-09-21T15:49:36.074 INFO:teuthology.orchestra.run.smithi089.stdout:2522 punch from 0x6d10a7 to 0x6d1d57, (0xcb0 bytes) 2024-09-21T15:49:36.075 INFO:teuthology.orchestra.run.smithi089.stdout:2523 write 0x9b619fb thru 0x9b679ac (0x5fb2 bytes) 2024-09-21T15:49:36.080 INFO:teuthology.orchestra.run.smithi089.stdout:2524 read 0x9046778 thru 0x9050a7b (0xa304 bytes) 2024-09-21T15:49:36.080 INFO:teuthology.orchestra.run.smithi089.stdout:2525 writesame 0xd3f68ef thru 0xd4013ee data_size 0x30(0xab00 bytes) 2024-09-21T15:49:36.084 INFO:teuthology.orchestra.run.smithi089.stdout:2526 punch from 0x8cb5ff5 to 0x8cc2efe, (0xcf09 bytes) 2024-09-21T15:49:36.084 INFO:teuthology.orchestra.run.smithi089.stdout:2528 writesame 0x5b246b thru 0x5bc3cf data_size 0x1(0x9f65 bytes) 2024-09-21T15:49:36.104 INFO:teuthology.orchestra.run.smithi089.stdout:2529 write 0x72b5e7f thru 0x72be437 (0x85b9 bytes) 2024-09-21T15:49:36.107 INFO:teuthology.orchestra.run.smithi089.stdout:2530 writesame 0x3fd197c thru 0x3fde5df data_size 0x4(0xcc64 bytes) 2024-09-21T15:49:36.108 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:36.108 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:49:36.108 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:49:36.108 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:49:36.108 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.108 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:14051 3.8 3:13437810:::rbd_data.1133dc5adfdf.0000000000000039:head [write 716800~796] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-21T15:49:36.108 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:49:36.108 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.411357503000001, 2024-09-21T15:49:36.109 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.98064002699999997, 2024-09-21T15:49:36.109 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:36.109 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:36.109 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:36.109 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:36.109 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:36.109 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 14051 2024-09-21T15:49:36.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.109 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:36.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.109 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:36.109 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:49:36.109 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:36.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.110 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.110 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:36.110 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:49:36.110 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:36.110 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.110 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.110 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:36.110 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:49:36.110 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:36.110 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.110 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.110 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:36.110 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695131+0000", 2024-09-21T15:49:36.111 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4680000000000003e-06 2024-09-21T15:49:36.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.111 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:36.111 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695139+0000", 2024-09-21T15:49:36.111 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9400000000000002e-06 2024-09-21T15:49:36.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.111 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:36.111 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695143+0000", 2024-09-21T15:49:36.111 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9899999999999999e-06 2024-09-21T15:49:36.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.111 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:36.112 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695190+0000", 2024-09-21T15:49:36.112 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6458000000000001e-05 2024-09-21T15:49:36.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.112 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:49:36.112 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695201+0000", 2024-09-21T15:49:36.112 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0964999999999999e-05 2024-09-21T15:49:36.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.112 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:36.112 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.669490+0000", 2024-09-21T15:49:36.112 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.97428980499999995 2024-09-21T15:49:36.113 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.113 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-09-21T15:49:36.113 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.670033+0000", 2024-09-21T15:49:36.113 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00054252599999999999 2024-09-21T15:49:36.113 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.113 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:36.113 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.671095+0000", 2024-09-21T15:49:36.113 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001062183 2024-09-21T15:49:36.113 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.113 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:36.113 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.671186+0000", 2024-09-21T15:49:36.114 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1191000000000002e-05 2024-09-21T15:49:36.114 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.114 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-09-21T15:49:36.114 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.671346+0000", 2024-09-21T15:49:36.114 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015980399999999999 2024-09-21T15:49:36.114 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.114 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:36.114 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.672221+0000", 2024-09-21T15:49:36.114 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00087471999999999995 2024-09-21T15:49:36.114 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:36.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.675619+0000", 2024-09-21T15:49:36.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00339858 2024-09-21T15:49:36.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:36.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.675670+0000", 2024-09-21T15:49:36.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0664000000000001e-05 2024-09-21T15:49:36.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:36.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.675768+0000", 2024-09-21T15:49:36.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7415000000000002e-05 2024-09-21T15:49:36.115 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:36.116 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:36.116 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:36.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.116 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28537 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 409600~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:36.116 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.227783+0000", 2024-09-21T15:49:36.116 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.8787023910000009, 2024-09-21T15:49:36.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.046702573999999997, 2024-09-21T15:49:36.116 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:36.116 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:36.116 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:36.117 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:36.117 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:36.117 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28537 2024-09-21T15:49:36.117 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.117 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:36.117 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.117 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:36.117 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227783+0000", 2024-09-21T15:49:36.117 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:36.117 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.117 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.117 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:36.117 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227783+0000", 2024-09-21T15:49:36.117 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:36.118 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.118 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.118 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:36.118 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227782+0000", 2024-09-21T15:49:36.118 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:36.118 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.118 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.118 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:36.118 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227785+0000", 2024-09-21T15:49:36.118 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.34e-06 2024-09-21T15:49:36.118 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.118 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.118 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:36.118 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227791+0000", 2024-09-21T15:49:36.119 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2020000000000001e-06 2024-09-21T15:49:36.119 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.119 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.119 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:36.119 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227796+0000", 2024-09-21T15:49:36.119 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8289999999999998e-06 2024-09-21T15:49:36.119 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.119 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.119 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:36.119 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.274387+0000", 2024-09-21T15:49:36.119 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.046591024000000002 2024-09-21T15:49:36.119 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.119 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.119 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:36.120 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.274427+0000", 2024-09-21T15:49:36.120 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0825000000000003e-05 2024-09-21T15:49:36.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.120 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.120 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:36.120 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.274485+0000", 2024-09-21T15:49:36.120 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7713000000000001e-05 2024-09-21T15:49:36.120 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:36.120 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:36.120 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:36.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.120 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.120 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28538 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 1212416~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:36.120 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.227800+0000", 2024-09-21T15:49:36.121 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.8786850099999999, 2024-09-21T15:49:36.121 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.046874213999999997, 2024-09-21T15:49:36.121 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:36.121 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:36.121 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:36.121 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:36.121 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:36.121 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28538 2024-09-21T15:49:36.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.121 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:36.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.121 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:36.121 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227800+0000", 2024-09-21T15:49:36.122 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:36.122 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.122 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.122 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:36.122 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227800+0000", 2024-09-21T15:49:36.122 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:36.122 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.122 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.122 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:36.122 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227800+0000", 2024-09-21T15:49:36.122 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:36.122 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.123 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:36.123 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227809+0000", 2024-09-21T15:49:36.123 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4709999999999996e-06 2024-09-21T15:49:36.123 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.123 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:36.123 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227816+0000", 2024-09-21T15:49:36.123 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0809999999999996e-06 2024-09-21T15:49:36.123 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.123 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:36.124 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227821+0000", 2024-09-21T15:49:36.124 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9420000000000003e-06 2024-09-21T15:49:36.124 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.124 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.124 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:36.124 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.274515+0000", 2024-09-21T15:49:36.124 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.046694201999999997 2024-09-21T15:49:36.124 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.124 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.124 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:36.124 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.274585+0000", 2024-09-21T15:49:36.124 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9250000000000003e-05 2024-09-21T15:49:36.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.125 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.125 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:36.125 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.274674+0000", 2024-09-21T15:49:36.125 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9621000000000001e-05 2024-09-21T15:49:36.125 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:36.125 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:36.125 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:36.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.125 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.125 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28539 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 2015232~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:36.125 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.227826+0000", 2024-09-21T15:49:36.125 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.8786590079999996, 2024-09-21T15:49:36.125 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.046969221999999998, 2024-09-21T15:49:36.126 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:36.126 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:36.126 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:36.126 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:36.126 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:36.126 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28539 2024-09-21T15:49:36.126 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.126 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:36.126 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.126 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:36.126 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227826+0000", 2024-09-21T15:49:36.126 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:36.127 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.127 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.127 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:36.127 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227826+0000", 2024-09-21T15:49:36.127 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:36.127 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.127 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.127 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:36.127 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227826+0000", 2024-09-21T15:49:36.127 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:36.127 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.127 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.127 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:36.127 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227828+0000", 2024-09-21T15:49:36.127 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4040000000000002e-06 2024-09-21T15:49:36.128 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.128 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.128 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:36.128 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227835+0000", 2024-09-21T15:49:36.128 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6100000000000002e-06 2024-09-21T15:49:36.128 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.128 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.128 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:36.128 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227839+0000", 2024-09-21T15:49:36.128 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7589999999999997e-06 2024-09-21T15:49:36.128 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.128 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.128 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:36.129 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.274701+0000", 2024-09-21T15:49:36.129 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.046861828000000001 2024-09-21T15:49:36.129 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.129 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.129 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:36.129 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.274742+0000", 2024-09-21T15:49:36.129 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1074e-05 2024-09-21T15:49:36.129 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.129 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.129 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:36.129 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.274795+0000", 2024-09-21T15:49:36.129 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2945999999999998e-05 2024-09-21T15:49:36.129 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:36.129 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:36.129 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:36.130 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.130 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.130 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28540 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 2760704~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:36.130 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.227844+0000", 2024-09-21T15:49:36.130 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.8786409989999999, 2024-09-21T15:49:36.130 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047146018999999997, 2024-09-21T15:49:36.130 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:36.130 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:36.130 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:36.130 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:36.130 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:36.130 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28540 2024-09-21T15:49:36.131 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.131 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:36.131 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.131 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:36.131 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227844+0000", 2024-09-21T15:49:36.131 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:36.131 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.131 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.131 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:36.131 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227844+0000", 2024-09-21T15:49:36.131 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:36.131 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.131 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.131 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:36.132 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227844+0000", 2024-09-21T15:49:36.132 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:36.132 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.132 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.132 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:36.132 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227846+0000", 2024-09-21T15:49:36.132 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1959999999999998e-06 2024-09-21T15:49:36.132 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.132 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.132 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:36.132 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227852+0000", 2024-09-21T15:49:36.132 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1659999999999999e-06 2024-09-21T15:49:36.132 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.132 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.132 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:36.133 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227856+0000", 2024-09-21T15:49:36.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4789999999999998e-06 2024-09-21T15:49:36.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.133 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.133 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:36.133 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.274825+0000", 2024-09-21T15:49:36.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.046968325999999998 2024-09-21T15:49:36.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.133 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.133 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:36.133 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.274896+0000", 2024-09-21T15:49:36.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0859999999999996e-05 2024-09-21T15:49:36.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.133 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.134 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:36.134 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.274990+0000", 2024-09-21T15:49:36.134 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4345999999999994e-05 2024-09-21T15:49:36.134 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:36.134 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:36.134 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:36.134 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.134 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.134 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28541 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 3563520~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:36.134 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.227861+0000", 2024-09-21T15:49:36.134 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.8786240099999993, 2024-09-21T15:49:36.134 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047277444000000002, 2024-09-21T15:49:36.134 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:36.135 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:36.135 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:36.135 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:36.135 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:36.135 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28541 2024-09-21T15:49:36.135 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.135 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:36.135 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.135 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:36.135 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227861+0000", 2024-09-21T15:49:36.135 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:36.135 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.135 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.135 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:36.136 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227861+0000", 2024-09-21T15:49:36.136 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:36.136 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.136 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.136 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:36.136 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227861+0000", 2024-09-21T15:49:36.136 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:36.136 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.136 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.136 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:36.136 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227863+0000", 2024-09-21T15:49:36.136 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0789999999999999e-06 2024-09-21T15:49:36.136 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.136 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.137 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:36.137 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227870+0000", 2024-09-21T15:49:36.137 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8070000000000002e-06 2024-09-21T15:49:36.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.137 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:36.137 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227874+0000", 2024-09-21T15:49:36.137 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9599999999999999e-06 2024-09-21T15:49:36.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.137 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:36.137 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275002+0000", 2024-09-21T15:49:36.137 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047127251000000002 2024-09-21T15:49:36.138 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.138 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.138 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:36.138 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275049+0000", 2024-09-21T15:49:36.138 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7592000000000001e-05 2024-09-21T15:49:36.138 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.138 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.138 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:36.138 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275138+0000", 2024-09-21T15:49:36.138 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9104000000000006e-05 2024-09-21T15:49:36.138 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:36.138 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:36.138 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:36.138 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.139 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28542 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 4366336~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:36.139 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.227879+0000", 2024-09-21T15:49:36.139 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.8786060500000001, 2024-09-21T15:49:36.139 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047394002999999997, 2024-09-21T15:49:36.139 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:36.139 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:36.139 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:36.139 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:36.139 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:36.139 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28542 2024-09-21T15:49:36.139 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.139 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:36.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.140 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:36.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227879+0000", 2024-09-21T15:49:36.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:36.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.140 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:36.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227879+0000", 2024-09-21T15:49:36.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:36.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.140 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:36.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227878+0000", 2024-09-21T15:49:36.141 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:36.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.141 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:36.141 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227881+0000", 2024-09-21T15:49:36.141 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2340000000000001e-06 2024-09-21T15:49:36.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.141 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:36.141 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227887+0000", 2024-09-21T15:49:36.141 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.049e-06 2024-09-21T15:49:36.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.141 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:36.142 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227891+0000", 2024-09-21T15:49:36.142 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5480000000000003e-06 2024-09-21T15:49:36.142 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.142 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.142 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:36.142 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275168+0000", 2024-09-21T15:49:36.142 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047276986999999999 2024-09-21T15:49:36.142 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.142 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.142 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:36.142 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275219+0000", 2024-09-21T15:49:36.142 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1010000000000001e-05 2024-09-21T15:49:36.143 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.143 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.143 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:36.143 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275273+0000", 2024-09-21T15:49:36.143 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3649000000000001e-05 2024-09-21T15:49:36.143 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:36.143 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:36.143 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:36.143 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.143 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.143 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28544 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 5111808~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:36.143 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.227895+0000", 2024-09-21T15:49:36.143 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.8785897919999996, 2024-09-21T15:49:36.144 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047502682999999997, 2024-09-21T15:49:36.144 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:36.144 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:36.144 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:36.144 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:36.144 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:36.144 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28544 2024-09-21T15:49:36.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.144 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:36.144 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.144 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:36.144 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227895+0000", 2024-09-21T15:49:36.144 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:36.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.145 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:36.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227895+0000", 2024-09-21T15:49:36.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:36.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.145 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:36.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227895+0000", 2024-09-21T15:49:36.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:36.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.145 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:36.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227897+0000", 2024-09-21T15:49:36.146 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.153e-06 2024-09-21T15:49:36.146 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.146 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.146 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:36.146 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227903+0000", 2024-09-21T15:49:36.146 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1940000000000003e-06 2024-09-21T15:49:36.146 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.146 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.146 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:36.146 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227908+0000", 2024-09-21T15:49:36.146 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6110000000000004e-06 2024-09-21T15:49:36.146 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.146 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.146 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:36.147 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275290+0000", 2024-09-21T15:49:36.147 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047381805999999999 2024-09-21T15:49:36.147 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.147 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.147 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:36.147 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275346+0000", 2024-09-21T15:49:36.147 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6313000000000001e-05 2024-09-21T15:49:36.147 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.147 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.147 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:36.147 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275398+0000", 2024-09-21T15:49:36.147 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2027000000000002e-05 2024-09-21T15:49:36.147 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:36.147 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:36.147 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:36.148 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.148 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.148 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28545 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 5169152~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:36.148 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.227912+0000", 2024-09-21T15:49:36.148 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.8785731709999993, 2024-09-21T15:49:36.148 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047684807000000003, 2024-09-21T15:49:36.148 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:36.148 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:36.148 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:36.148 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:36.148 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:36.148 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28545 2024-09-21T15:49:36.148 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.148 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:36.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.149 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:36.149 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227912+0000", 2024-09-21T15:49:36.149 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:36.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.149 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:36.149 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227912+0000", 2024-09-21T15:49:36.149 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:36.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.149 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:36.149 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227911+0000", 2024-09-21T15:49:36.149 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:36.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.150 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.150 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:36.150 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227913+0000", 2024-09-21T15:49:36.150 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9970000000000001e-06 2024-09-21T15:49:36.150 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.150 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.150 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:36.150 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227920+0000", 2024-09-21T15:49:36.150 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0499999999999997e-06 2024-09-21T15:49:36.150 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.150 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.150 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:36.150 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227924+0000", 2024-09-21T15:49:36.150 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7219999999999999e-06 2024-09-21T15:49:36.151 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.151 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.151 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:36.151 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275424+0000", 2024-09-21T15:49:36.151 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047499574000000003 2024-09-21T15:49:36.151 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.151 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.151 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:36.151 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275495+0000", 2024-09-21T15:49:36.151 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1347999999999995e-05 2024-09-21T15:49:36.151 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.151 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.151 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:36.151 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275597+0000", 2024-09-21T15:49:36.152 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101446 2024-09-21T15:49:36.152 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:36.152 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:36.152 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:36.152 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.152 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.152 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28546 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 5914624~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:36.152 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.227929+0000", 2024-09-21T15:49:36.152 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.8785559250000006, 2024-09-21T15:49:36.152 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047783297000000002, 2024-09-21T15:49:36.152 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:36.152 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:36.152 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:36.152 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:36.152 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:36.153 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28546 2024-09-21T15:49:36.153 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.153 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:36.153 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.153 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:36.153 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227929+0000", 2024-09-21T15:49:36.153 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:36.153 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.153 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.153 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:36.153 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227929+0000", 2024-09-21T15:49:36.153 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:36.153 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.153 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.153 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:36.154 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227929+0000", 2024-09-21T15:49:36.154 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:36.154 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.154 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.154 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:36.154 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227931+0000", 2024-09-21T15:49:36.154 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2529999999999999e-06 2024-09-21T15:49:36.154 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.154 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.154 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:36.154 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227937+0000", 2024-09-21T15:49:36.154 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9699999999999996e-06 2024-09-21T15:49:36.154 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.154 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.154 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:36.155 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227942+0000", 2024-09-21T15:49:36.155 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7650000000000001e-06 2024-09-21T15:49:36.155 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.155 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.155 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:36.155 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275621+0000", 2024-09-21T15:49:36.155 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047679694000000002 2024-09-21T15:49:36.155 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.155 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.155 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:36.155 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275661+0000", 2024-09-21T15:49:36.155 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9363999999999998e-05 2024-09-21T15:49:36.155 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.155 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.155 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:36.155 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275712+0000", 2024-09-21T15:49:36.156 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1598999999999998e-05 2024-09-21T15:49:36.156 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:36.156 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:36.156 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:36.156 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.156 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.156 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28547 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 6717440~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:36.156 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.228080+0000", 2024-09-21T15:49:36.156 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.8784046480000001, 2024-09-21T15:49:36.156 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047744609, 2024-09-21T15:49:36.156 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:36.156 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:36.156 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:36.156 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:36.156 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:36.157 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28547 2024-09-21T15:49:36.157 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.157 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:36.157 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.157 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:36.157 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228080+0000", 2024-09-21T15:49:36.157 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:36.157 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.157 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.157 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:36.157 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228080+0000", 2024-09-21T15:49:36.157 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:36.157 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.157 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.157 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:36.158 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227946+0000", 2024-09-21T15:49:36.158 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999866 2024-09-21T15:49:36.158 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.158 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.158 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:36.158 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228084+0000", 2024-09-21T15:49:36.158 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000137517 2024-09-21T15:49:36.158 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.158 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.158 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:36.158 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228093+0000", 2024-09-21T15:49:36.158 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.645e-06 2024-09-21T15:49:36.158 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.158 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.159 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:36.159 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228101+0000", 2024-09-21T15:49:36.159 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9559999999999997e-06 2024-09-21T15:49:36.159 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.159 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.159 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:36.159 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275735+0000", 2024-09-21T15:49:36.159 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047633928999999998 2024-09-21T15:49:36.159 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.159 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.159 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:36.159 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275774+0000", 2024-09-21T15:49:36.159 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8699999999999999e-05 2024-09-21T15:49:36.159 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.160 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.160 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:36.160 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275825+0000", 2024-09-21T15:49:36.160 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0926e-05 2024-09-21T15:49:36.160 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:36.160 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:36.160 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:36.160 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.160 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.160 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28549 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 417792~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:36.160 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.228847+0000", 2024-09-21T15:49:36.160 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.8776375430000005, 2024-09-21T15:49:36.160 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047086515000000002, 2024-09-21T15:49:36.160 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:36.161 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:36.161 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:36.161 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:36.161 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:36.161 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28549 2024-09-21T15:49:36.161 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.161 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:36.161 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.161 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:36.161 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228847+0000", 2024-09-21T15:49:36.161 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:36.161 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.161 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.161 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:36.162 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228847+0000", 2024-09-21T15:49:36.162 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:36.162 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.162 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.162 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:36.162 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228107+0000", 2024-09-21T15:49:36.162 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9992595 2024-09-21T15:49:36.162 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.162 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.162 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:36.162 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228852+0000", 2024-09-21T15:49:36.162 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00074505000000000003 2024-09-21T15:49:36.162 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.162 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.163 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:36.163 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228871+0000", 2024-09-21T15:49:36.163 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8635e-05 2024-09-21T15:49:36.163 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.163 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.163 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:36.163 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228880+0000", 2024-09-21T15:49:36.163 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9849999999999995e-06 2024-09-21T15:49:36.163 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.163 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.163 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:36.163 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275847+0000", 2024-09-21T15:49:36.163 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.046967239000000001 2024-09-21T15:49:36.163 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.164 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.164 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:36.164 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275885+0000", 2024-09-21T15:49:36.164 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8274e-05 2024-09-21T15:49:36.164 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.164 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.164 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:36.164 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275934+0000", 2024-09-21T15:49:36.164 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8764999999999998e-05 2024-09-21T15:49:36.164 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:36.164 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:36.164 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:36.164 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.164 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.165 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28551 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 1220608~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:36.165 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.228886+0000", 2024-09-21T15:49:36.165 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.8775984969999993, 2024-09-21T15:49:36.165 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047166819999999998, 2024-09-21T15:49:36.165 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:36.165 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:36.165 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:36.165 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:36.165 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:36.165 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28551 2024-09-21T15:49:36.165 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.165 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:36.166 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.166 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:36.166 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228886+0000", 2024-09-21T15:49:36.166 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:36.166 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.166 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.166 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:36.166 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228886+0000", 2024-09-21T15:49:36.166 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:36.166 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.166 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.166 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:36.166 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228886+0000", 2024-09-21T15:49:36.167 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:36.167 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.167 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.167 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:36.167 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228889+0000", 2024-09-21T15:49:36.167 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8650000000000001e-06 2024-09-21T15:49:36.167 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.167 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.167 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:36.167 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228896+0000", 2024-09-21T15:49:36.167 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0790000000000003e-06 2024-09-21T15:49:36.167 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.168 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.168 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:36.168 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228903+0000", 2024-09-21T15:49:36.168 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9770000000000003e-06 2024-09-21T15:49:36.168 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.168 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.168 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:36.168 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275954+0000", 2024-09-21T15:49:36.168 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047050986000000003 2024-09-21T15:49:36.168 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.168 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.168 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:36.169 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276001+0000", 2024-09-21T15:49:36.169 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7527000000000001e-05 2024-09-21T15:49:36.169 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.169 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.169 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:36.169 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276053+0000", 2024-09-21T15:49:36.169 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1854999999999997e-05 2024-09-21T15:49:36.169 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:36.169 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:36.169 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:36.169 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.169 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.169 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28552 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 1966080~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:36.169 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.228908+0000", 2024-09-21T15:49:36.170 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.8775772120000003, 2024-09-21T15:49:36.170 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047256193000000002, 2024-09-21T15:49:36.170 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:36.170 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:36.170 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:36.170 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:36.170 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:36.170 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28552 2024-09-21T15:49:36.170 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.170 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:36.170 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.170 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:36.170 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228908+0000", 2024-09-21T15:49:36.171 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:36.171 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.171 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.171 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:36.171 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228908+0000", 2024-09-21T15:49:36.171 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:36.171 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.171 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.171 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:36.171 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228907+0000", 2024-09-21T15:49:36.171 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:36.171 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.172 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.172 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:36.172 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228910+0000", 2024-09-21T15:49:36.172 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3819999999999998e-06 2024-09-21T15:49:36.172 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.172 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.172 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:36.172 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228916+0000", 2024-09-21T15:49:36.172 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6390000000000003e-06 2024-09-21T15:49:36.172 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.172 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.172 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:36.172 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228921+0000", 2024-09-21T15:49:36.172 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0350000000000001e-06 2024-09-21T15:49:36.173 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.173 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.173 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:36.173 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276077+0000", 2024-09-21T15:49:36.173 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047155326999999997 2024-09-21T15:49:36.173 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.173 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.173 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:36.173 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276114+0000", 2024-09-21T15:49:36.173 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6746999999999999e-05 2024-09-21T15:49:36.173 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.173 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.173 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:36.174 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276164+0000", 2024-09-21T15:49:36.174 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0438000000000001e-05 2024-09-21T15:49:36.174 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:36.174 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:36.174 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:36.174 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.174 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.174 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28553 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 2023424~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:36.174 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.228926+0000", 2024-09-21T15:49:36.174 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.8775590060000003, 2024-09-21T15:49:36.174 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047351920999999998, 2024-09-21T15:49:36.174 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:36.175 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:36.175 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:36.175 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:36.175 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:36.175 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28553 2024-09-21T15:49:36.175 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.175 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:36.175 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.175 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:36.175 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228926+0000", 2024-09-21T15:49:36.175 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:36.175 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.175 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.175 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:36.176 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228926+0000", 2024-09-21T15:49:36.176 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:36.176 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.176 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.176 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:36.176 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228926+0000", 2024-09-21T15:49:36.176 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:36.176 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.176 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.176 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:36.176 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228932+0000", 2024-09-21T15:49:36.176 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6930000000000001e-06 2024-09-21T15:49:36.176 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.176 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.177 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:36.177 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228939+0000", 2024-09-21T15:49:36.177 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8199999999999999e-06 2024-09-21T15:49:36.177 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.177 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.177 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:36.177 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228953+0000", 2024-09-21T15:49:36.177 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.347e-05 2024-09-21T15:49:36.177 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.177 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.177 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:36.177 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276187+0000", 2024-09-21T15:49:36.177 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047234405 2024-09-21T15:49:36.177 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.178 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.178 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:36.178 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276228+0000", 2024-09-21T15:49:36.178 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1010000000000002e-05 2024-09-21T15:49:36.178 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.178 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.178 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:36.178 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276278+0000", 2024-09-21T15:49:36.178 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9874999999999999e-05 2024-09-21T15:49:36.178 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:36.178 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:36.178 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:36.178 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.178 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.179 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28554 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 2768896~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:36.179 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.228991+0000", 2024-09-21T15:49:36.179 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.8774935290000005, 2024-09-21T15:49:36.179 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047394177000000003, 2024-09-21T15:49:36.179 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:36.179 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:36.179 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:36.179 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:36.179 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:36.179 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28554 2024-09-21T15:49:36.179 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.179 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:36.179 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.179 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:36.180 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228991+0000", 2024-09-21T15:49:36.180 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:36.180 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.180 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.180 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:36.180 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228991+0000", 2024-09-21T15:49:36.180 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:36.180 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.180 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.180 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:36.180 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228991+0000", 2024-09-21T15:49:36.180 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:36.180 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.180 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.180 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:36.181 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228994+0000", 2024-09-21T15:49:36.181 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.422e-06 2024-09-21T15:49:36.181 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.181 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.181 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:36.181 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229003+0000", 2024-09-21T15:49:36.181 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8699999999999998e-06 2024-09-21T15:49:36.181 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.181 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.181 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:36.181 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229009+0000", 2024-09-21T15:49:36.181 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3099999999999997e-06 2024-09-21T15:49:36.181 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.181 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.182 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:36.182 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276301+0000", 2024-09-21T15:49:36.182 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047291305999999998 2024-09-21T15:49:36.182 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.182 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.182 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:36.182 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276337+0000", 2024-09-21T15:49:36.182 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6378999999999998e-05 2024-09-21T15:49:36.182 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.182 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.182 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:36.182 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276386+0000", 2024-09-21T15:49:36.182 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8535999999999997e-05 2024-09-21T15:49:36.182 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:36.183 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:36.183 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:36.183 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.183 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.183 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28555 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 3571712~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:36.183 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.229015+0000", 2024-09-21T15:49:36.183 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.8774704349999993, 2024-09-21T15:49:36.183 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047478713999999998, 2024-09-21T15:49:36.183 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:36.183 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:36.183 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:36.183 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:36.183 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:36.184 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28555 2024-09-21T15:49:36.184 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.184 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:36.184 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.184 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:36.184 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229015+0000", 2024-09-21T15:49:36.184 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:36.184 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.184 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.184 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:36.184 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229015+0000", 2024-09-21T15:49:36.184 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:36.184 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.184 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.185 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:36.185 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229014+0000", 2024-09-21T15:49:36.185 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:36.185 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.185 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.185 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:36.185 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229029+0000", 2024-09-21T15:49:36.185 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4503e-05 2024-09-21T15:49:36.185 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.185 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.185 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:36.185 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229036+0000", 2024-09-21T15:49:36.185 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5989999999999996e-06 2024-09-21T15:49:36.185 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.186 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.186 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:36.186 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229042+0000", 2024-09-21T15:49:36.186 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6699999999999999e-06 2024-09-21T15:49:36.186 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.186 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.186 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:36.186 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276407+0000", 2024-09-21T15:49:36.186 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047365646999999997 2024-09-21T15:49:36.186 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.186 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.186 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:36.186 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276445+0000", 2024-09-21T15:49:36.187 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7064000000000003e-05 2024-09-21T15:49:36.187 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.187 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.187 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:36.187 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276493+0000", 2024-09-21T15:49:36.187 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8704e-05 2024-09-21T15:49:36.187 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:36.187 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:36.187 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:36.187 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.187 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.187 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28556 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 4374528~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:36.187 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.229183+0000", 2024-09-21T15:49:36.187 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.8773016439999992, 2024-09-21T15:49:36.188 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047417709000000002, 2024-09-21T15:49:36.188 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:36.188 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:36.188 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:36.188 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:36.188 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:36.188 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28556 2024-09-21T15:49:36.188 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.188 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:36.188 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.188 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:36.188 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229183+0000", 2024-09-21T15:49:36.188 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:36.188 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.189 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.189 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:36.189 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229183+0000", 2024-09-21T15:49:36.189 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:36.189 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.189 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.189 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:36.189 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229046+0000", 2024-09-21T15:49:36.189 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9998631 2024-09-21T15:49:36.189 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.189 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.189 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:36.189 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229186+0000", 2024-09-21T15:49:36.189 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014003900000000001 2024-09-21T15:49:36.190 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.190 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.190 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:36.190 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229195+0000", 2024-09-21T15:49:36.190 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.422e-06 2024-09-21T15:49:36.190 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.190 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.190 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:36.190 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229199+0000", 2024-09-21T15:49:36.190 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5379999999999996e-06 2024-09-21T15:49:36.190 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.190 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.190 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:36.190 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276516+0000", 2024-09-21T15:49:36.191 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047316432999999998 2024-09-21T15:49:36.191 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.191 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.191 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:36.191 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276552+0000", 2024-09-21T15:49:36.191 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6019000000000002e-05 2024-09-21T15:49:36.191 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.191 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.191 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:36.191 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276601+0000", 2024-09-21T15:49:36.191 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9172999999999998e-05 2024-09-21T15:49:36.191 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:36.191 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:36.191 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:36.191 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.192 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.192 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28558 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 5120000~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:36.192 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.229531+0000", 2024-09-21T15:49:36.192 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.8769537609999993, 2024-09-21T15:49:36.192 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047179666000000002, 2024-09-21T15:49:36.192 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:36.192 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:36.192 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:36.192 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:36.192 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:36.192 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28558 2024-09-21T15:49:36.192 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.192 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:36.192 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.193 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:36.193 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229531+0000", 2024-09-21T15:49:36.193 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:36.193 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.193 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.193 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:36.193 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229531+0000", 2024-09-21T15:49:36.193 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:36.193 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.193 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.193 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:36.193 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229203+0000", 2024-09-21T15:49:36.193 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9996719 2024-09-21T15:49:36.194 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.194 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.194 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:36.194 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229536+0000", 2024-09-21T15:49:36.194 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000332885 2024-09-21T15:49:36.194 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.194 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.194 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:36.194 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229548+0000", 2024-09-21T15:49:36.194 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1861999999999999e-05 2024-09-21T15:49:36.194 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.194 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.194 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:36.194 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229555+0000", 2024-09-21T15:49:36.195 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6009999999999997e-06 2024-09-21T15:49:36.195 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.195 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.195 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:36.195 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276624+0000", 2024-09-21T15:49:36.195 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047068250999999998 2024-09-21T15:49:36.195 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.195 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.195 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:36.195 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276662+0000", 2024-09-21T15:49:36.195 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8705999999999998e-05 2024-09-21T15:49:36.195 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.195 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.195 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:36.196 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276711+0000", 2024-09-21T15:49:36.196 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8443999999999999e-05 2024-09-21T15:49:36.196 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:36.196 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:36.196 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:36.196 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.196 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.196 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28560 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 5922816~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:36.196 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.229562+0000", 2024-09-21T15:49:36.196 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.8769225560000002, 2024-09-21T15:49:36.196 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047255377000000001, 2024-09-21T15:49:36.196 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:36.196 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:36.196 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:36.197 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:36.197 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:36.197 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28560 2024-09-21T15:49:36.197 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.197 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:36.197 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.197 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:36.197 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229562+0000", 2024-09-21T15:49:36.197 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:36.197 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.197 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.197 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:36.197 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229562+0000", 2024-09-21T15:49:36.197 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:36.197 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:36.198 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229562+0000", 2024-09-21T15:49:36.198 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:36.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:36.198 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229565+0000", 2024-09-21T15:49:36.198 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9450000000000002e-06 2024-09-21T15:49:36.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:36.198 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229572+0000", 2024-09-21T15:49:36.198 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4240000000000004e-06 2024-09-21T15:49:36.199 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.199 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.199 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:36.199 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229577+0000", 2024-09-21T15:49:36.199 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2000000000000002e-06 2024-09-21T15:49:36.199 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.199 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.199 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:36.199 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276733+0000", 2024-09-21T15:49:36.199 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047155202 2024-09-21T15:49:36.199 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.199 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.199 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:36.199 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276769+0000", 2024-09-21T15:49:36.200 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6013000000000002e-05 2024-09-21T15:49:36.200 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:36.200 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:36.200 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:36.200 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276818+0000", 2024-09-21T15:49:36.200 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9178000000000002e-05 2024-09-21T15:49:36.200 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:36.200 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:36.200 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:36.200 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:36.200 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:36.200 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:36.201 INFO:teuthology.orchestra.run.smithi089.stdout:2531 read 0xb19a626 thru 0xb1a00b2 (0x5a8d bytes) 2024-09-21T15:49:36.201 INFO:teuthology.orchestra.run.smithi089.stdout:2532 read 0x8485b36 thru 0x84881b3 (0x267e bytes) 2024-09-21T15:49:36.202 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:49:36.204 INFO:teuthology.orchestra.run.smithi089.stdout:2533 trunc from 0xe6f7c48 to 0x28efbdb 2024-09-21T15:49:36.204 INFO:teuthology.orchestra.run.smithi089.stdout:2534 write 0xc4a036f thru 0xc4a3e16 (0x3aa8 bytes) 2024-09-21T15:49:36.204 INFO:teuthology.orchestra.run.smithi089.stdout:2535 writesame 0x804c119 thru 0x805bab1 data_size 0x3b(0xf999 bytes) 2024-09-21T15:49:36.204 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:36.204 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:36.204 INFO:teuthology.orchestra.run.smithi089.stdout:2538 write 0x2a0f410 thru 0x2a11180 (0x1d71 bytes) 2024-09-21T15:49:36.204 INFO:teuthology.orchestra.run.smithi089.stdout:truncating to largest ever: 0xea0e0f2 2024-09-21T15:49:36.204 INFO:teuthology.orchestra.run.smithi089.stdout:2539 trunc from 0xc4a3e17 to 0xea0e0f2 2024-09-21T15:49:36.204 INFO:teuthology.orchestra.run.smithi089.stdout:2540 write 0x66f3ae3 thru 0x66ffef3 (0xc411 bytes) 2024-09-21T15:49:36.205 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:36.205 INFO:teuthology.orchestra.run.smithi089.stdout:2543 read 0x51764df thru 0x517af6f (0x4a91 bytes) 2024-09-21T15:49:36.205 INFO:teuthology.orchestra.run.smithi089.stdout:2544 write 0xe9d772 thru 0xea8814 (0xb0a3 bytes) 2024-09-21T15:49:36.205 INFO:teuthology.orchestra.run.smithi089.stdout:2545 read 0x2f9aed thru 0x305b12 (0xc026 bytes) 2024-09-21T15:49:36.205 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:36.205 INFO:teuthology.orchestra.run.smithi089.stdout:2548 trunc from 0xea0e0f2 to 0x1f09689 2024-09-21T15:49:36.205 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:36.200+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:36.222 INFO:teuthology.orchestra.run.smithi089.stdout:2550 trunc from 0x1f09689 to 0xb94cce4 2024-09-21T15:49:36.227 INFO:teuthology.orchestra.run.smithi089.stdout:2552 write 0x705804f thru 0x705b809 (0x37bb bytes) 2024-09-21T15:49:36.232 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:36.232 INFO:teuthology.orchestra.run.smithi089.stdout:2554 punch from 0x56cb1ea to 0x56d3198, (0x7fae bytes) 2024-09-21T15:49:36.232 INFO:teuthology.orchestra.run.smithi089.stdout:2558 punch from 0xb0a737d to 0xb0b6631, (0xf2b4 bytes) 2024-09-21T15:49:36.232 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:36.232 INFO:teuthology.orchestra.run.smithi089.stdout:2560 read 0xb807c0f thru 0xb80fa45 (0x7e37 bytes) 2024-09-21T15:49:36.239 INFO:teuthology.orchestra.run.smithi089.stdout:2562 trunc from 0xb94cce4 to 0x5a161af 2024-09-21T15:49:36.250 INFO:teuthology.orchestra.run.smithi089.stdout:2563 punch from 0x28a0f14 to 0x28a711e, (0x620a bytes) 2024-09-21T15:49:36.265 INFO:teuthology.orchestra.run.smithi089.stdout:2564 write 0xc52560f thru 0xc5353d3 (0xfdc5 bytes) 2024-09-21T15:49:36.268 INFO:teuthology.orchestra.run.smithi089.stdout:2565 write 0x8c968fb thru 0x8c98edb (0x25e1 bytes) 2024-09-21T15:49:36.272 INFO:teuthology.orchestra.run.smithi089.stdout:2566 read 0x659bd2b thru 0x65a8bd6 (0xceac bytes) 2024-09-21T15:49:36.272 INFO:teuthology.orchestra.run.smithi089.stdout:2567 writesame 0x447edf2 thru 0x448a679 data_size 0x14(0xb888 bytes) 2024-09-21T15:49:36.284 INFO:teuthology.orchestra.run.smithi089.stdout:2568 trunc from 0xc5353d4 to 0x5f916ef 2024-09-21T15:49:36.302 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:36.301+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:36.315 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:36.315 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:36.315 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:49:36.315 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:36.319 INFO:teuthology.orchestra.run.smithi089.stdout:2569 write 0x7ae7339 thru 0x7ae9e64 (0x2b2c bytes) 2024-09-21T15:49:36.325 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:49:36.337 INFO:teuthology.orchestra.run.smithi089.stdout:2570 write 0xd2e2b4c thru 0xd2ec255 (0x970a bytes) 2024-09-21T15:49:36.340 INFO:teuthology.orchestra.run.smithi089.stdout:2571 writesame 0x9ae3b17 thru 0x9ae43d8 data_size 0x2(0x8c2 bytes) 2024-09-21T15:49:36.344 INFO:teuthology.orchestra.run.smithi089.stdout:2572 write 0xafe7549 thru 0xafef33c (0x7df4 bytes) 2024-09-21T15:49:36.347 INFO:teuthology.orchestra.run.smithi089.stdout:2573 write 0x330f8ee thru 0x331b2b4 (0xb9c7 bytes) 2024-09-21T15:49:36.351 INFO:teuthology.orchestra.run.smithi089.stdout:2575 punch from 0xc1472a0 to 0xc1533de, (0xc13e bytes) 2024-09-21T15:49:36.358 INFO:teuthology.orchestra.run.smithi089.stdout:2578 trunc from 0xd2ec256 to 0x7323108 2024-09-21T15:49:36.378 INFO:teuthology.orchestra.run.smithi089.stdout:2579 write 0x433699e thru 0x4339ec3 (0x3526 bytes) 2024-09-21T15:49:36.384 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:36.384 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:36.402 INFO:teuthology.orchestra.run.smithi089.stdout:2583 writesame 0xc946bc9 thru 0xc950f68 data_size 0x77(0xa3a0 bytes) 2024-09-21T15:49:36.402 INFO:teuthology.orchestra.run.smithi089.stdout:2585 write 0x25cfcd1 thru 0x25d745c (0x778c bytes) 2024-09-21T15:49:36.410 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:36.409+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:36.412 INFO:teuthology.orchestra.run.smithi089.stdout:2586 trunc from 0xc950f69 to 0x4b06200 2024-09-21T15:49:36.418 INFO:teuthology.orchestra.run.smithi089.stdout:2587 write 0x46bcc98 thru 0x46c9c58 (0xcfc1 bytes) 2024-09-21T15:49:36.424 INFO:teuthology.orchestra.run.smithi089.stdout:2588 write 0x14ca701 thru 0x14d79a8 (0xd2a8 bytes) 2024-09-21T15:49:36.426 INFO:teuthology.orchestra.run.smithi089.stdout:2589 write 0x1f0c088 thru 0x1f155dd (0x9556 bytes) 2024-09-21T15:49:36.430 INFO:teuthology.orchestra.run.smithi089.stdout:2591 read 0x1caf27 thru 0x1d3b23 (0x8bfd bytes) 2024-09-21T15:49:36.431 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:36.431 INFO:teuthology.orchestra.run.smithi089.stdout:2594 read 0x415b8c3 thru 0x415cfb9 (0x16f7 bytes) 2024-09-21T15:49:36.431 INFO:teuthology.orchestra.run.smithi089.stdout:2596 punch from 0xd859f1 to 0xd86ba3, (0x11b2 bytes) 2024-09-21T15:49:36.431 INFO:teuthology.orchestra.run.smithi089.stdout:2597 punch from 0x447883a to 0x4487e32, (0xf5f8 bytes) 2024-09-21T15:49:36.431 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:36.445 INFO:teuthology.orchestra.run.smithi089.stdout:2599 writesame 0xb43c499 thru 0xb43cd8c data_size 0x6(0x8f4 bytes) 2024-09-21T15:49:36.445 INFO:teuthology.orchestra.run.smithi089.stdout:2600 writesame 0x6827f7b thru 0x68343bd data_size 0x1(0xc443 bytes) 2024-09-21T15:49:36.469 INFO:teuthology.orchestra.run.smithi089.stdout:2601 writesame 0x3488ea thru 0x34a9dc data_size 0x5(0x20f3 bytes) 2024-09-21T15:49:36.471 INFO:teuthology.orchestra.run.smithi089.stdout:2603 read 0x6254c5d thru 0x625da02 (0x8da6 bytes) 2024-09-21T15:49:36.471 INFO:teuthology.orchestra.run.smithi089.stdout:2605 punch from 0xa8a59fe to 0xa8a5fe7, (0x5e9 bytes) 2024-09-21T15:49:36.471 INFO:teuthology.orchestra.run.smithi089.stdout:2606 read 0x9f25ba6 thru 0x9f26aaa (0xf05 bytes) 2024-09-21T15:49:36.471 INFO:teuthology.orchestra.run.smithi089.stdout:2607 writesame 0x17cebca thru 0x17d6c4a data_size 0x1(0x8081 bytes) 2024-09-21T15:49:36.475 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:36.475 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:36.476 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:49:36.476 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:49:36.476 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:36.484 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:49:36.487 INFO:teuthology.orchestra.run.smithi089.stdout:2608 read 0x685dff4 thru 0x686ad97 (0xcda4 bytes) 2024-09-21T15:49:36.487 INFO:teuthology.orchestra.run.smithi089.stdout:2609 punch from 0x6e7afe0 to 0x6e80d88, (0x5da8 bytes) 2024-09-21T15:49:36.493 INFO:teuthology.orchestra.run.smithi089.stdout:2610 write 0xb6b92cd thru 0xb6bfda4 (0x6ad8 bytes) 2024-09-21T15:49:36.501 INFO:teuthology.orchestra.run.smithi089.stdout:2611 write 0xb83c80b thru 0xb84101d (0x4813 bytes) 2024-09-21T15:49:36.504 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:36.503+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:36.514 INFO:teuthology.orchestra.run.smithi089.stdout:2612 punch from 0x1c61f1b to 0x1c71b9d, (0xfc82 bytes) 2024-09-21T15:49:36.514 INFO:teuthology.orchestra.run.smithi089.stdout:2614 write 0x61008b thru 0x6157a1 (0x5717 bytes) 2024-09-21T15:49:36.518 INFO:teuthology.orchestra.run.smithi089.stdout:2615 write 0x12d8c57 thru 0x12e4904 (0xbcae bytes) 2024-09-21T15:49:36.522 INFO:teuthology.orchestra.run.smithi089.stdout:2616 write 0x261a0f0 thru 0x2623190 (0x90a1 bytes) 2024-09-21T15:49:36.528 INFO:teuthology.orchestra.run.smithi089.stdout:2617 writesame 0xe466ef thru 0xe4a7c4 data_size 0x2b(0x40d6 bytes) 2024-09-21T15:49:36.530 INFO:teuthology.orchestra.run.smithi089.stdout:2618 read 0x1aa1681 thru 0x1aa929c (0x7c1c bytes) 2024-09-21T15:49:36.530 INFO:teuthology.orchestra.run.smithi089.stdout:2619 trunc from 0xb84101e to 0xbfe2320 2024-09-21T15:49:36.548 INFO:teuthology.orchestra.run.smithi089.stdout:2621 write 0xd01c424 thru 0xd027648 (0xb225 bytes) 2024-09-21T15:49:36.552 INFO:teuthology.orchestra.run.smithi089.stdout:2622 write 0xc16ed2a thru 0xc17b99d (0xcc74 bytes) 2024-09-21T15:49:36.555 INFO:teuthology.orchestra.run.smithi089.stdout:2625 write 0xabc704 thru 0xac7ddb (0xb6d8 bytes) 2024-09-21T15:49:36.559 INFO:teuthology.orchestra.run.smithi089.stdout:2626 writesame 0xb5c2726 thru 0xb5d01b9 data_size 0x6(0xda94 bytes) 2024-09-21T15:49:36.566 INFO:teuthology.orchestra.run.smithi089.stdout:2628 writesame 0x4cbd141 thru 0x4cc470e data_size 0x11(0x75ce bytes) 2024-09-21T15:49:36.579 INFO:teuthology.orchestra.run.smithi089.stdout:2629 write 0xd5970c0 thru 0xd5a6680 (0xf5c1 bytes) 2024-09-21T15:49:36.585 INFO:teuthology.orchestra.run.smithi089.stdout:2630 read 0x9c6a73d thru 0x9c6b696 (0xf5a bytes) 2024-09-21T15:49:36.585 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:36.585 INFO:teuthology.orchestra.run.smithi089.stdout:2634 read 0xb4f9032 thru 0xb5004d3 (0x74a2 bytes) 2024-09-21T15:49:36.585 INFO:teuthology.orchestra.run.smithi089.stdout:2635 write 0x9f6a929 thru 0x9f71868 (0x6f40 bytes) 2024-09-21T15:49:36.589 INFO:teuthology.orchestra.run.smithi089.stdout:2636 read 0xb6757de thru 0xb676d36 (0x1559 bytes) 2024-09-21T15:49:36.589 INFO:teuthology.orchestra.run.smithi089.stdout:2637 read 0x5afafe3 thru 0x5b0aa50 (0xfa6e bytes) 2024-09-21T15:49:36.598 INFO:teuthology.orchestra.run.smithi089.stdout:2638 trunc from 0xd5a6681 to 0x5a8068f 2024-09-21T15:49:36.605 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:36.604+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:36.613 INFO:teuthology.orchestra.run.smithi089.stdout:2639 read 0x3d62ae9 thru 0x3d62da8 (0x2c0 bytes) 2024-09-21T15:49:36.613 INFO:teuthology.orchestra.run.smithi089.stdout:2640 clone 13 order 25 su 1048576 sc 15 2024-09-21T15:49:36.637 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:36.637 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:49:36.637 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:49:36.637 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:49:36.637 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.637 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26224 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 2326528~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:36.637 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.504487+0000", 2024-09-21T15:49:36.638 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 10.130812797000001, 2024-09-21T15:49:36.638 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032819441999999997, 2024-09-21T15:49:36.638 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:36.638 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:36.638 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:36.638 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:36.638 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:36.638 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26224 2024-09-21T15:49:36.638 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.638 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:36.639 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.639 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:36.639 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504487+0000", 2024-09-21T15:49:36.639 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:36.639 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.639 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.639 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:36.639 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504487+0000", 2024-09-21T15:49:36.639 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:36.639 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.639 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.639 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:36.639 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504487+0000", 2024-09-21T15:49:36.639 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:36.640 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.640 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.640 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:36.640 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504490+0000", 2024-09-21T15:49:36.640 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6049999999999998e-06 2024-09-21T15:49:36.640 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.640 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.640 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:36.640 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504493+0000", 2024-09-21T15:49:36.640 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1200000000000002e-06 2024-09-21T15:49:36.640 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.640 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.640 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:36.640 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504496+0000", 2024-09-21T15:49:36.641 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.977e-06 2024-09-21T15:49:36.641 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.641 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.641 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:36.641 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537221+0000", 2024-09-21T15:49:36.641 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032724529000000002 2024-09-21T15:49:36.641 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.641 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.641 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:36.641 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537259+0000", 2024-09-21T15:49:36.641 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7923999999999998e-05 2024-09-21T15:49:36.641 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.641 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.641 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:36.641 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537306+0000", 2024-09-21T15:49:36.642 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7512999999999997e-05 2024-09-21T15:49:36.642 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:36.642 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:36.642 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:36.642 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.642 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.642 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26225 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 2383872~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:36.642 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.504498+0000", 2024-09-21T15:49:36.642 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 10.130801227999999, 2024-09-21T15:49:36.642 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.03291703, 2024-09-21T15:49:36.642 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:36.642 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:36.642 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:36.643 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:36.643 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:36.643 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26225 2024-09-21T15:49:36.643 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.643 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:36.643 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.643 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:36.645 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504498+0000", 2024-09-21T15:49:36.646 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:36.646 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.646 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.646 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:36.646 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504498+0000", 2024-09-21T15:49:36.646 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:36.646 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.646 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.646 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:36.646 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504498+0000", 2024-09-21T15:49:36.646 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:36.646 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.647 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.647 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:36.647 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504499+0000", 2024-09-21T15:49:36.647 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0070000000000001e-06 2024-09-21T15:49:36.647 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.647 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.647 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:36.647 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504502+0000", 2024-09-21T15:49:36.647 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8119999999999999e-06 2024-09-21T15:49:36.647 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.647 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.647 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:36.647 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504504+0000", 2024-09-21T15:49:36.647 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4380000000000002e-06 2024-09-21T15:49:36.648 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.648 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.648 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:36.648 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537328+0000", 2024-09-21T15:49:36.648 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032823659999999998 2024-09-21T15:49:36.648 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.648 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.648 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:36.648 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537366+0000", 2024-09-21T15:49:36.648 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8083000000000001e-05 2024-09-21T15:49:36.648 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.648 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.648 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:36.648 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537415+0000", 2024-09-21T15:49:36.649 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9233000000000001e-05 2024-09-21T15:49:36.649 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:36.649 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:36.649 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:36.649 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.649 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.649 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26226 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 2441216~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:36.649 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.504506+0000", 2024-09-21T15:49:36.649 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 10.130793112999999, 2024-09-21T15:49:36.649 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033016726000000003, 2024-09-21T15:49:36.649 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:36.649 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:36.650 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:36.650 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:36.650 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:36.650 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26226 2024-09-21T15:49:36.650 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.650 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:36.650 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.650 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:36.650 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504506+0000", 2024-09-21T15:49:36.650 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:36.650 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.650 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.650 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:36.650 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504506+0000", 2024-09-21T15:49:36.651 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:36.651 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.651 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.651 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:36.651 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504506+0000", 2024-09-21T15:49:36.651 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:36.651 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.651 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.651 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:36.651 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504507+0000", 2024-09-21T15:49:36.651 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9699999999999994e-07 2024-09-21T15:49:36.651 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.651 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.652 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:36.652 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504510+0000", 2024-09-21T15:49:36.652 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.773e-06 2024-09-21T15:49:36.652 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.652 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.652 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:36.652 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504514+0000", 2024-09-21T15:49:36.652 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1150000000000004e-06 2024-09-21T15:49:36.652 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.652 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.652 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:36.652 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537437+0000", 2024-09-21T15:49:36.652 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032923249000000002 2024-09-21T15:49:36.652 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.653 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.653 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:36.653 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537476+0000", 2024-09-21T15:49:36.653 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8679e-05 2024-09-21T15:49:36.653 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.653 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.653 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:36.653 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537523+0000", 2024-09-21T15:49:36.653 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7119000000000001e-05 2024-09-21T15:49:36.653 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:36.653 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:36.653 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:36.653 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.653 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.653 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26227 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 2498560~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:36.654 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.504516+0000", 2024-09-21T15:49:36.654 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 10.130783512000001, 2024-09-21T15:49:36.654 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033117681000000003, 2024-09-21T15:49:36.654 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:36.654 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:36.654 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:36.654 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:36.654 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:36.654 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26227 2024-09-21T15:49:36.655 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.655 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:36.655 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.655 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:36.655 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504516+0000", 2024-09-21T15:49:36.655 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:36.655 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.655 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.655 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:36.655 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504516+0000", 2024-09-21T15:49:36.655 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:36.655 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.655 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.655 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:36.655 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504516+0000", 2024-09-21T15:49:36.656 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:36.656 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.656 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.656 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:36.656 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504517+0000", 2024-09-21T15:49:36.656 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0610000000000001e-06 2024-09-21T15:49:36.656 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.656 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.656 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:36.656 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504520+0000", 2024-09-21T15:49:36.656 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.83e-06 2024-09-21T15:49:36.656 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.656 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.656 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:36.656 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504522+0000", 2024-09-21T15:49:36.657 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.497e-06 2024-09-21T15:49:36.657 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.657 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.657 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:36.657 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537544+0000", 2024-09-21T15:49:36.657 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033021948000000002 2024-09-21T15:49:36.657 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.657 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.657 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:36.657 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537585+0000", 2024-09-21T15:49:36.657 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1094999999999999e-05 2024-09-21T15:49:36.657 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.657 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.657 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:36.658 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537634+0000", 2024-09-21T15:49:36.658 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8449000000000003e-05 2024-09-21T15:49:36.658 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:36.658 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:36.658 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:36.658 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.658 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.658 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26228 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 2555904~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:36.658 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.504524+0000", 2024-09-21T15:49:36.658 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 10.130775414, 2024-09-21T15:49:36.658 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.03321735, 2024-09-21T15:49:36.658 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:36.658 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:36.659 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:36.659 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:36.659 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:36.659 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26228 2024-09-21T15:49:36.659 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.659 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:36.659 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.659 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:36.659 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504524+0000", 2024-09-21T15:49:36.659 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:36.659 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.660 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.660 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:36.660 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504524+0000", 2024-09-21T15:49:36.660 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:36.660 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.660 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.660 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:36.660 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504524+0000", 2024-09-21T15:49:36.660 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:36.660 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.660 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.660 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:36.660 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504525+0000", 2024-09-21T15:49:36.660 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0419999999999999e-06 2024-09-21T15:49:36.660 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.661 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.661 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:36.661 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504528+0000", 2024-09-21T15:49:36.661 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7530000000000002e-06 2024-09-21T15:49:36.661 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.661 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.661 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:36.661 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504530+0000", 2024-09-21T15:49:36.661 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4530000000000001e-06 2024-09-21T15:49:36.661 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.661 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.661 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:36.661 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537656+0000", 2024-09-21T15:49:36.661 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033126023999999997 2024-09-21T15:49:36.661 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.662 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.662 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:36.662 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537694+0000", 2024-09-21T15:49:36.662 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7799000000000002e-05 2024-09-21T15:49:36.662 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.662 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.662 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:36.662 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537741+0000", 2024-09-21T15:49:36.662 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7484999999999997e-05 2024-09-21T15:49:36.662 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:36.662 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:36.662 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:36.662 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.662 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.662 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26229 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 2613248~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:36.663 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.504532+0000", 2024-09-21T15:49:36.663 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 10.130767535, 2024-09-21T15:49:36.663 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033315897999999997, 2024-09-21T15:49:36.663 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:36.663 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:36.663 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:36.663 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:36.663 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:36.663 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26229 2024-09-21T15:49:36.663 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.663 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:36.663 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.663 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:36.663 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504532+0000", 2024-09-21T15:49:36.663 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:36.664 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.664 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.664 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:36.664 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504532+0000", 2024-09-21T15:49:36.664 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:36.664 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.664 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.664 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:36.664 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504532+0000", 2024-09-21T15:49:36.664 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:36.664 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.665 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.665 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:36.665 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504533+0000", 2024-09-21T15:49:36.665 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0610000000000001e-06 2024-09-21T15:49:36.665 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.665 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.665 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:36.665 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504536+0000", 2024-09-21T15:49:36.665 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.309e-06 2024-09-21T15:49:36.665 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.665 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.665 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:36.665 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504539+0000", 2024-09-21T15:49:36.665 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.435e-06 2024-09-21T15:49:36.665 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.666 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.666 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:36.666 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537762+0000", 2024-09-21T15:49:36.666 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033223813999999997 2024-09-21T15:49:36.666 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.666 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.666 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:36.666 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537801+0000", 2024-09-21T15:49:36.666 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8136999999999998e-05 2024-09-21T15:49:36.666 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.666 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.666 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:36.666 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537848+0000", 2024-09-21T15:49:36.666 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7345999999999997e-05 2024-09-21T15:49:36.667 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:36.667 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:36.667 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:36.667 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.667 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.667 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26230 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 2670592~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:36.667 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.504606+0000", 2024-09-21T15:49:36.667 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 10.130693377, 2024-09-21T15:49:36.667 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033348801999999997, 2024-09-21T15:49:36.667 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:36.667 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:36.667 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:36.667 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:36.668 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:36.668 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26230 2024-09-21T15:49:36.668 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.668 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:36.668 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.668 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:36.668 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504606+0000", 2024-09-21T15:49:36.668 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:36.668 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.668 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.668 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:36.668 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504606+0000", 2024-09-21T15:49:36.669 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:36.669 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.669 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.669 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:36.669 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504541+0000", 2024-09-21T15:49:36.669 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999352 2024-09-21T15:49:36.669 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.669 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.669 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:36.670 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504608+0000", 2024-09-21T15:49:36.670 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6427000000000006e-05 2024-09-21T15:49:36.670 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.670 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.670 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:36.670 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504612+0000", 2024-09-21T15:49:36.670 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1799999999999998e-06 2024-09-21T15:49:36.670 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.670 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.671 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:36.671 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504615+0000", 2024-09-21T15:49:36.671 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0400000000000001e-06 2024-09-21T15:49:36.671 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.671 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.671 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:36.671 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537870+0000", 2024-09-21T15:49:36.671 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033254745000000002 2024-09-21T15:49:36.671 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.672 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.672 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:36.672 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537908+0000", 2024-09-21T15:49:36.672 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7882000000000001e-05 2024-09-21T15:49:36.672 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.672 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.672 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:36.672 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537955+0000", 2024-09-21T15:49:36.672 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7361999999999998e-05 2024-09-21T15:49:36.673 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:36.673 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:36.673 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:36.673 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.673 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.673 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26239 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 3186688~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:36.673 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.506097+0000", 2024-09-21T15:49:36.673 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 10.129202671, 2024-09-21T15:49:36.673 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032843331000000003, 2024-09-21T15:49:36.674 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:36.674 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:36.674 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:36.674 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:36.674 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:36.674 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26239 2024-09-21T15:49:36.674 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.674 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:36.675 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.675 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:36.675 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506097+0000", 2024-09-21T15:49:36.675 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:36.675 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.675 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.675 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:36.675 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506097+0000", 2024-09-21T15:49:36.676 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:36.676 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.676 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.676 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:36.676 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506097+0000", 2024-09-21T15:49:36.676 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:36.676 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.676 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.676 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:36.677 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506098+0000", 2024-09-21T15:49:36.677 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.079e-06 2024-09-21T15:49:36.677 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.677 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.677 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:36.677 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506101+0000", 2024-09-21T15:49:36.677 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8499999999999998e-06 2024-09-21T15:49:36.677 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.677 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.678 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:36.678 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506103+0000", 2024-09-21T15:49:36.678 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5770000000000001e-06 2024-09-21T15:49:36.678 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.678 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.678 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:36.678 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.538854+0000", 2024-09-21T15:49:36.678 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032750835999999998 2024-09-21T15:49:36.678 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.679 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.679 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:36.679 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.538892+0000", 2024-09-21T15:49:36.679 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7904000000000002e-05 2024-09-21T15:49:36.679 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.679 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.679 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:36.679 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.538940+0000", 2024-09-21T15:49:36.679 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8303999999999997e-05 2024-09-21T15:49:36.680 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:36.680 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:36.680 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:36.680 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.680 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.680 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26240 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 3244032~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:36.680 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.506106+0000", 2024-09-21T15:49:36.680 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 10.129193538999999, 2024-09-21T15:49:36.680 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032955696, 2024-09-21T15:49:36.681 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:36.681 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:36.681 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:36.681 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:36.681 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:36.681 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26240 2024-09-21T15:49:36.681 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.681 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:36.681 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.682 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:36.682 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506106+0000", 2024-09-21T15:49:36.682 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:36.682 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.682 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.682 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:36.682 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506106+0000", 2024-09-21T15:49:36.682 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:36.682 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.683 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.683 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:36.683 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506106+0000", 2024-09-21T15:49:36.683 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:36.683 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.683 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.683 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:36.683 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506107+0000", 2024-09-21T15:49:36.683 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1179999999999999e-06 2024-09-21T15:49:36.684 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.684 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.684 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:36.684 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506110+0000", 2024-09-21T15:49:36.684 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0259999999999999e-06 2024-09-21T15:49:36.684 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.684 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.684 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:36.684 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506112+0000", 2024-09-21T15:49:36.684 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5890000000000001e-06 2024-09-21T15:49:36.684 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.685 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.685 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:36.685 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.538962+0000", 2024-09-21T15:49:36.685 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032849861000000001 2024-09-21T15:49:36.685 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.685 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.685 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:36.685 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539003+0000", 2024-09-21T15:49:36.685 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0756000000000001e-05 2024-09-21T15:49:36.685 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.685 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.686 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:36.686 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539062+0000", 2024-09-21T15:49:36.686 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8597999999999997e-05 2024-09-21T15:49:36.686 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:36.686 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:36.686 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:36.686 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.686 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.686 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26241 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 3301376~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:36.686 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.506128+0000", 2024-09-21T15:49:36.686 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 10.129171373, 2024-09-21T15:49:36.686 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033043453, 2024-09-21T15:49:36.686 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:36.686 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:36.687 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:36.687 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:36.687 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:36.687 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26241 2024-09-21T15:49:36.687 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.687 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:36.687 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.687 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:36.687 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506128+0000", 2024-09-21T15:49:36.687 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:36.687 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.687 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.687 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:36.687 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506128+0000", 2024-09-21T15:49:36.688 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:36.688 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.688 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.688 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:36.688 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506128+0000", 2024-09-21T15:49:36.688 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:36.688 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.688 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.688 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:36.688 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506129+0000", 2024-09-21T15:49:36.688 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7290000000000001e-06 2024-09-21T15:49:36.688 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.688 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.688 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:36.689 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506132+0000", 2024-09-21T15:49:36.689 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9919999999999999e-06 2024-09-21T15:49:36.689 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.689 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.689 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:36.689 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506135+0000", 2024-09-21T15:49:36.689 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4090000000000001e-06 2024-09-21T15:49:36.689 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.689 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.689 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:36.689 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539084+0000", 2024-09-21T15:49:36.689 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032949231000000002 2024-09-21T15:49:36.689 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.689 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.690 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:36.690 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539124+0000", 2024-09-21T15:49:36.690 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9421000000000001e-05 2024-09-21T15:49:36.690 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.690 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.690 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:36.690 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539172+0000", 2024-09-21T15:49:36.690 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8068000000000001e-05 2024-09-21T15:49:36.690 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:36.690 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:36.690 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:36.690 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.690 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.691 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26242 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 3358720~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:36.691 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.506137+0000", 2024-09-21T15:49:36.691 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 10.129162439, 2024-09-21T15:49:36.691 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033142360000000003, 2024-09-21T15:49:36.691 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:36.691 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:36.691 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:36.691 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:36.691 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:36.691 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26242 2024-09-21T15:49:36.691 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.691 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:36.692 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.692 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:36.692 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506137+0000", 2024-09-21T15:49:36.692 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:36.692 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.692 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.692 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:36.692 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506137+0000", 2024-09-21T15:49:36.692 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:36.692 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.692 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.692 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:36.692 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506137+0000", 2024-09-21T15:49:36.692 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:36.693 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.693 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.693 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:36.693 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506138+0000", 2024-09-21T15:49:36.693 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2330000000000001e-06 2024-09-21T15:49:36.693 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.693 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.693 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:36.693 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506141+0000", 2024-09-21T15:49:36.693 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7999999999999999e-06 2024-09-21T15:49:36.693 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.693 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.693 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:36.693 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506143+0000", 2024-09-21T15:49:36.693 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5919999999999999e-06 2024-09-21T15:49:36.694 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.694 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.694 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:36.694 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539193+0000", 2024-09-21T15:49:36.694 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033049229999999999 2024-09-21T15:49:36.694 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.694 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.694 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:36.694 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539232+0000", 2024-09-21T15:49:36.694 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9743999999999998e-05 2024-09-21T15:49:36.694 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.694 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.694 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:36.694 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539279+0000", 2024-09-21T15:49:36.695 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7039999999999997e-05 2024-09-21T15:49:36.695 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:36.695 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:36.695 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:36.695 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.695 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.695 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26243 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 3416064~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:36.695 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.506145+0000", 2024-09-21T15:49:36.695 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 10.129154154, 2024-09-21T15:49:36.695 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033240933, 2024-09-21T15:49:36.695 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:36.695 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:36.695 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:36.696 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:36.696 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:36.696 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26243 2024-09-21T15:49:36.696 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.696 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:36.696 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.696 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:36.696 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506145+0000", 2024-09-21T15:49:36.696 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:36.696 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.696 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.696 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:36.696 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506145+0000", 2024-09-21T15:49:36.697 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:36.697 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.697 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.697 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:36.697 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506145+0000", 2024-09-21T15:49:36.697 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:36.697 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.697 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.697 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:36.697 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506146+0000", 2024-09-21T15:49:36.697 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0279999999999999e-06 2024-09-21T15:49:36.697 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.697 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.697 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:36.697 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506149+0000", 2024-09-21T15:49:36.698 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2449999999999998e-06 2024-09-21T15:49:36.698 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.698 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.698 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:36.698 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506152+0000", 2024-09-21T15:49:36.698 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5050000000000002e-06 2024-09-21T15:49:36.698 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.698 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.698 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:36.698 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539301+0000", 2024-09-21T15:49:36.698 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033149093999999997 2024-09-21T15:49:36.698 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.698 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.699 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:36.699 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539339+0000", 2024-09-21T15:49:36.699 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7685999999999998e-05 2024-09-21T15:49:36.699 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.699 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.699 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:36.699 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539386+0000", 2024-09-21T15:49:36.699 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7602000000000003e-05 2024-09-21T15:49:36.699 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:36.699 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:36.699 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:36.699 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.699 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.700 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26258 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 4276224~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:36.700 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.508167+0000", 2024-09-21T15:49:36.700 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 10.127132236, 2024-09-21T15:49:36.700 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032934857999999997, 2024-09-21T15:49:36.700 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:36.700 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:36.700 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:36.700 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:36.700 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:36.700 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26258 2024-09-21T15:49:36.700 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.700 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:36.700 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.700 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:36.701 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508167+0000", 2024-09-21T15:49:36.701 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:36.701 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.701 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.701 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:36.701 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508167+0000", 2024-09-21T15:49:36.701 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:36.701 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.701 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.701 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:36.701 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508167+0000", 2024-09-21T15:49:36.701 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:36.701 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.701 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.701 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:36.702 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508168+0000", 2024-09-21T15:49:36.702 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.088e-06 2024-09-21T15:49:36.702 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.702 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.702 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:36.702 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508171+0000", 2024-09-21T15:49:36.702 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7080000000000002e-06 2024-09-21T15:49:36.702 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.702 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.702 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:36.702 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508173+0000", 2024-09-21T15:49:36.702 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4030000000000001e-06 2024-09-21T15:49:36.702 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.703 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.703 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:36.703 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.540936+0000", 2024-09-21T15:49:36.703 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032762279999999998 2024-09-21T15:49:36.703 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.703 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.703 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:36.703 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.541007+0000", 2024-09-21T15:49:36.703 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1335999999999996e-05 2024-09-21T15:49:36.703 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.703 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.703 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:36.703 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.541102+0000", 2024-09-21T15:49:36.703 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5259000000000004e-05 2024-09-21T15:49:36.704 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:36.704 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:36.704 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:36.704 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.704 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.704 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26259 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 4333568~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:36.704 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.508175+0000", 2024-09-21T15:49:36.704 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 10.127124294, 2024-09-21T15:49:36.704 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033044845000000003, 2024-09-21T15:49:36.704 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:36.704 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:36.704 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:36.704 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:36.705 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:36.705 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26259 2024-09-21T15:49:36.705 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.705 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:36.705 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.705 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:36.705 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508175+0000", 2024-09-21T15:49:36.705 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:36.705 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.705 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.705 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:36.705 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508175+0000", 2024-09-21T15:49:36.705 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:36.706 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.706 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.706 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:36.706 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508175+0000", 2024-09-21T15:49:36.706 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:36.706 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.706 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.706 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:36.706 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508176+0000", 2024-09-21T15:49:36.706 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1039999999999999e-06 2024-09-21T15:49:36.707 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.707 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.707 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:36.707 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508179+0000", 2024-09-21T15:49:36.707 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.689e-06 2024-09-21T15:49:36.707 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.707 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.707 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:36.707 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508181+0000", 2024-09-21T15:49:36.707 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5610000000000001e-06 2024-09-21T15:49:36.707 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.708 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.708 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:36.708 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.541126+0000", 2024-09-21T15:49:36.708 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032944284999999997 2024-09-21T15:49:36.708 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.708 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.708 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:36.708 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.541168+0000", 2024-09-21T15:49:36.708 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2565000000000003e-05 2024-09-21T15:49:36.708 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.708 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.708 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:36.708 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.541220+0000", 2024-09-21T15:49:36.708 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1863000000000001e-05 2024-09-21T15:49:36.709 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:36.709 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:36.709 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:36.709 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.709 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.709 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26260 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 4390912~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:36.709 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.508183+0000", 2024-09-21T15:49:36.709 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 10.127116179, 2024-09-21T15:49:36.709 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033150304999999998, 2024-09-21T15:49:36.709 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:36.709 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:36.709 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:36.709 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:36.709 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:36.710 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26260 2024-09-21T15:49:36.710 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.710 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:36.710 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.710 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:36.710 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508183+0000", 2024-09-21T15:49:36.710 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:36.710 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.710 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.710 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:36.710 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508183+0000", 2024-09-21T15:49:36.711 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:36.711 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.711 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.711 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:36.711 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508183+0000", 2024-09-21T15:49:36.711 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:36.711 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.711 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.711 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:36.711 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508184+0000", 2024-09-21T15:49:36.711 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0580000000000001e-06 2024-09-21T15:49:36.711 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.711 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.711 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:36.711 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508187+0000", 2024-09-21T15:49:36.712 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7209999999999999e-06 2024-09-21T15:49:36.712 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.712 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.712 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:36.712 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508189+0000", 2024-09-21T15:49:36.712 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5239999999999999e-06 2024-09-21T15:49:36.712 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.712 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.712 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:36.712 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.541243+0000", 2024-09-21T15:49:36.712 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033053658 2024-09-21T15:49:36.712 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.712 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.712 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:36.713 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.541284+0000", 2024-09-21T15:49:36.713 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0834000000000002e-05 2024-09-21T15:49:36.713 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.713 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.713 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:36.713 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.541334+0000", 2024-09-21T15:49:36.713 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9724e-05 2024-09-21T15:49:36.713 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:36.713 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:36.713 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:36.713 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.713 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.713 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26275 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 5251072~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:36.714 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.510151+0000", 2024-09-21T15:49:36.714 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 10.1251484, 2024-09-21T15:49:36.714 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032836153999999999, 2024-09-21T15:49:36.714 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:36.714 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:36.714 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:36.714 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:36.714 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:36.714 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26275 2024-09-21T15:49:36.714 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.714 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:36.714 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.714 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:36.714 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510151+0000", 2024-09-21T15:49:36.714 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:36.715 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.715 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.715 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:36.715 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510151+0000", 2024-09-21T15:49:36.715 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:36.715 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.715 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.715 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:36.715 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510151+0000", 2024-09-21T15:49:36.715 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:36.715 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.715 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.715 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:36.716 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510152+0000", 2024-09-21T15:49:36.716 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1990000000000001e-06 2024-09-21T15:49:36.716 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.716 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.716 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:36.716 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510155+0000", 2024-09-21T15:49:36.716 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7089999999999998e-06 2024-09-21T15:49:36.716 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.716 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.716 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:36.716 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510157+0000", 2024-09-21T15:49:36.717 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5189999999999999e-06 2024-09-21T15:49:36.717 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.717 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.717 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:36.717 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.542902+0000", 2024-09-21T15:49:36.717 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032744599999999999 2024-09-21T15:49:36.717 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.717 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.717 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:36.717 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.542941+0000", 2024-09-21T15:49:36.717 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8761999999999999e-05 2024-09-21T15:49:36.717 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.717 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.718 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:36.718 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.542987+0000", 2024-09-21T15:49:36.718 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6665999999999997e-05 2024-09-21T15:49:36.718 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:36.718 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:36.718 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:36.718 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.718 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.718 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26276 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 5308416~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:36.718 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.510159+0000", 2024-09-21T15:49:36.718 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 10.125140436000001, 2024-09-21T15:49:36.718 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032943014999999999, 2024-09-21T15:49:36.719 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:36.719 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:36.719 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:36.719 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:36.719 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:36.719 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26276 2024-09-21T15:49:36.719 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.719 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:36.719 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.719 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:36.719 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510159+0000", 2024-09-21T15:49:36.719 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:36.719 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.719 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.720 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:36.720 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510159+0000", 2024-09-21T15:49:36.720 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:36.720 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.720 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.720 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:36.720 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510159+0000", 2024-09-21T15:49:36.720 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:36.720 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.720 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.720 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:36.720 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510160+0000", 2024-09-21T15:49:36.720 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.13e-06 2024-09-21T15:49:36.721 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.721 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.721 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:36.721 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510163+0000", 2024-09-21T15:49:36.721 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7489999999999999e-06 2024-09-21T15:49:36.721 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.721 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.721 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:36.721 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510165+0000", 2024-09-21T15:49:36.721 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4200000000000001e-06 2024-09-21T15:49:36.721 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.721 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.722 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:36.722 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.543013+0000", 2024-09-21T15:49:36.722 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032847736000000002 2024-09-21T15:49:36.722 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.722 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.722 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:36.722 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.543054+0000", 2024-09-21T15:49:36.722 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0725000000000001e-05 2024-09-21T15:49:36.722 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.722 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.722 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:36.722 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.543102+0000", 2024-09-21T15:49:36.722 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8471999999999999e-05 2024-09-21T15:49:36.723 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:36.723 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:36.723 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:36.723 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.723 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.723 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26277 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 5365760~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:36.723 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.510167+0000", 2024-09-21T15:49:36.723 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 10.125132304999999, 2024-09-21T15:49:36.723 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033113111000000001, 2024-09-21T15:49:36.723 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:36.724 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:36.724 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:36.724 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:36.724 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:36.724 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26277 2024-09-21T15:49:36.724 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.724 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:36.724 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.724 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:36.724 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510167+0000", 2024-09-21T15:49:36.724 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:36.724 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.724 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.724 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:36.724 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510167+0000", 2024-09-21T15:49:36.725 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:36.725 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.725 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.725 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:36.725 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510167+0000", 2024-09-21T15:49:36.725 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:36.725 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.725 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.725 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:36.725 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510168+0000", 2024-09-21T15:49:36.725 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.6899999999999996e-07 2024-09-21T15:49:36.725 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.725 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.725 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:36.726 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510171+0000", 2024-09-21T15:49:36.726 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7080000000000002e-06 2024-09-21T15:49:36.726 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.726 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.726 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:36.726 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510173+0000", 2024-09-21T15:49:36.726 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5050000000000002e-06 2024-09-21T15:49:36.726 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.726 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.726 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:36.726 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.543125+0000", 2024-09-21T15:49:36.726 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032951504 2024-09-21T15:49:36.726 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.726 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.727 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:36.727 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.543192+0000", 2024-09-21T15:49:36.727 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6938000000000002e-05 2024-09-21T15:49:36.727 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.727 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.727 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:36.727 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.543280+0000", 2024-09-21T15:49:36.727 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8702999999999994e-05 2024-09-21T15:49:36.727 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:36.727 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:36.728 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:36.728 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.728 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.728 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26292 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 6225920~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:36.728 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.512160+0000", 2024-09-21T15:49:36.728 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 10.123139491, 2024-09-21T15:49:36.728 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032838748000000001, 2024-09-21T15:49:36.728 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:36.728 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:36.728 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:36.728 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:36.728 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:36.728 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26292 2024-09-21T15:49:36.728 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.728 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:36.729 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.729 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:36.729 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512160+0000", 2024-09-21T15:49:36.729 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:36.729 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.729 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.729 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:36.729 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512160+0000", 2024-09-21T15:49:36.729 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:36.729 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.729 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.729 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:36.729 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512160+0000", 2024-09-21T15:49:36.729 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:36.729 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.730 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.730 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:36.730 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512161+0000", 2024-09-21T15:49:36.730 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.158e-06 2024-09-21T15:49:36.730 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.730 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.730 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:36.730 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512164+0000", 2024-09-21T15:49:36.730 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7429999999999998e-06 2024-09-21T15:49:36.730 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.730 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.730 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:36.730 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512166+0000", 2024-09-21T15:49:36.731 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5280000000000002e-06 2024-09-21T15:49:36.731 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.731 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.731 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:36.731 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.544912+0000", 2024-09-21T15:49:36.731 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032745962000000003 2024-09-21T15:49:36.731 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.731 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.731 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:36.731 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.544950+0000", 2024-09-21T15:49:36.731 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7657000000000002e-05 2024-09-21T15:49:36.731 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.732 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.732 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:36.732 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.544999+0000", 2024-09-21T15:49:36.732 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8912999999999997e-05 2024-09-21T15:49:36.732 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:36.732 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:36.732 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:36.732 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.732 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.732 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26293 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 6283264~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:36.732 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.512169+0000", 2024-09-21T15:49:36.732 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 10.123130342, 2024-09-21T15:49:36.732 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032937552000000002, 2024-09-21T15:49:36.733 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:36.733 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:36.733 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:36.733 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:36.733 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:36.733 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26293 2024-09-21T15:49:36.733 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.733 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:36.733 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.733 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:36.733 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512169+0000", 2024-09-21T15:49:36.733 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:36.733 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.733 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.734 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:36.734 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512169+0000", 2024-09-21T15:49:36.734 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:36.734 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.734 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.734 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:36.734 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512169+0000", 2024-09-21T15:49:36.734 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:36.734 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.734 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.734 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:36.734 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512170+0000", 2024-09-21T15:49:36.734 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1650000000000001e-06 2024-09-21T15:49:36.734 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.735 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.735 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:36.735 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512173+0000", 2024-09-21T15:49:36.735 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.773e-06 2024-09-21T15:49:36.735 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.735 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.735 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:36.735 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512175+0000", 2024-09-21T15:49:36.735 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4700000000000001e-06 2024-09-21T15:49:36.735 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.735 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.735 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:36.735 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.545018+0000", 2024-09-21T15:49:36.735 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032843043000000002 2024-09-21T15:49:36.735 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.736 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.736 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:36.736 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.545057+0000", 2024-09-21T15:49:36.736 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8862999999999997e-05 2024-09-21T15:49:36.736 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:36.736 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:36.736 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:36.736 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.545107+0000", 2024-09-21T15:49:36.736 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9455e-05 2024-09-21T15:49:36.736 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:36.736 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:36.736 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:36.736 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:36.736 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:36.736 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:36.738 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:49:36.739 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:36.737+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:36.830 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is unset 2024-09-21T15:49:36.839 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:36.838+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:36.893 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:36.893 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:36.893 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:49:36.893 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:36.903 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:49:36.940 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:36.938+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:37.000 INFO:teuthology.orchestra.run.smithi089.stdout:checking clone #11, image image_client.0-clone11 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent12 2024-09-21T15:49:37.009 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:37.009 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:37.009 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:49:37.009 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:49:37.009 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:37.020 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:49:37.042 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:37.040+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:37.142 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:37.141+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:37.173 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:37.173 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:49:37.173 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:49:37.174 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:49:37.174 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.174 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:4938 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 659456~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:37.174 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.455162+0000", 2024-09-21T15:49:37.174 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.714810396000001, 2024-09-21T15:49:37.174 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039399295000000001, 2024-09-21T15:49:37.175 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:37.175 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:37.175 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:37.175 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:37.175 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:37.175 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 4938 2024-09-21T15:49:37.176 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.176 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:37.176 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.176 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:37.176 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.455162+0000", 2024-09-21T15:49:37.176 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.177 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.177 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.177 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:37.177 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.455162+0000", 2024-09-21T15:49:37.177 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.178 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.178 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.178 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:37.178 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.455162+0000", 2024-09-21T15:49:37.178 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:37.178 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.179 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.179 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:37.179 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.455164+0000", 2024-09-21T15:49:37.179 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.074e-06 2024-09-21T15:49:37.179 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.179 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.180 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:37.180 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.455169+0000", 2024-09-21T15:49:37.180 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6760000000000004e-06 2024-09-21T15:49:37.180 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.180 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.180 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:37.181 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.455174+0000", 2024-09-21T15:49:37.181 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3680000000000004e-06 2024-09-21T15:49:37.181 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.181 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.181 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:37.182 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.494363+0000", 2024-09-21T15:49:37.182 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039189225000000001 2024-09-21T15:49:37.182 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.182 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.182 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:37.182 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.494456+0000", 2024-09-21T15:49:37.183 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2843999999999994e-05 2024-09-21T15:49:37.183 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.183 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.183 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:37.183 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.494561+0000", 2024-09-21T15:49:37.183 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000105627 2024-09-21T15:49:37.184 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.184 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:37.184 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.184 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.184 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.184 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:4939 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 708608~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:37.185 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.455178+0000", 2024-09-21T15:49:37.185 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.714794704999999, 2024-09-21T15:49:37.185 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039493579000000001, 2024-09-21T15:49:37.185 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:37.185 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:37.185 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:37.186 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:37.186 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:37.186 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 4939 2024-09-21T15:49:37.186 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.186 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:37.186 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.187 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:37.187 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.455178+0000", 2024-09-21T15:49:37.187 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.187 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.187 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.187 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:37.187 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.455178+0000", 2024-09-21T15:49:37.188 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.188 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.188 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.188 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:37.188 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.455177+0000", 2024-09-21T15:49:37.188 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:37.189 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.189 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.189 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:37.189 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.455179+0000", 2024-09-21T15:49:37.189 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9489999999999999e-06 2024-09-21T15:49:37.189 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.190 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.190 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:37.190 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.455185+0000", 2024-09-21T15:49:37.190 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.49e-06 2024-09-21T15:49:37.190 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.190 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.190 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:37.190 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.455190+0000", 2024-09-21T15:49:37.190 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5910000000000003e-06 2024-09-21T15:49:37.191 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.191 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.191 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:37.191 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.494573+0000", 2024-09-21T15:49:37.191 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039382895000000001 2024-09-21T15:49:37.191 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.191 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.191 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:37.191 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.494622+0000", 2024-09-21T15:49:37.191 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8167000000000001e-05 2024-09-21T15:49:37.191 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.191 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.192 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:37.192 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.494671+0000", 2024-09-21T15:49:37.192 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9818000000000003e-05 2024-09-21T15:49:37.192 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.192 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:37.192 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.192 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.192 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.192 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:4949 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 1200128~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:37.192 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.457117+0000", 2024-09-21T15:49:37.193 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.712855443999999, 2024-09-21T15:49:37.193 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039319875999999997, 2024-09-21T15:49:37.193 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:37.193 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:37.193 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:37.193 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:37.193 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:37.193 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 4949 2024-09-21T15:49:37.193 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.193 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:37.193 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.194 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:37.194 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.457117+0000", 2024-09-21T15:49:37.194 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.194 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.194 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.194 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:37.194 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.457117+0000", 2024-09-21T15:49:37.194 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.194 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.194 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.194 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:37.194 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.457117+0000", 2024-09-21T15:49:37.195 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:37.195 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.195 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.195 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:37.195 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.457118+0000", 2024-09-21T15:49:37.195 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0729999999999999e-06 2024-09-21T15:49:37.196 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.196 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.196 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:37.196 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.457121+0000", 2024-09-21T15:49:37.196 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.802e-06 2024-09-21T15:49:37.196 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.196 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.196 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:37.196 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.457123+0000", 2024-09-21T15:49:37.196 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1969999999999999e-06 2024-09-21T15:49:37.196 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.197 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.197 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:37.197 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.496340+0000", 2024-09-21T15:49:37.197 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039217120000000001 2024-09-21T15:49:37.197 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.197 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.197 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:37.197 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.496386+0000", 2024-09-21T15:49:37.197 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6360000000000003e-05 2024-09-21T15:49:37.197 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.197 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.197 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:37.198 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.496437+0000", 2024-09-21T15:49:37.198 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0555e-05 2024-09-21T15:49:37.198 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.198 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:37.198 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.198 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.198 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.198 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:4950 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 1249280~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:37.198 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.457125+0000", 2024-09-21T15:49:37.198 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.712847454999999, 2024-09-21T15:49:37.198 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039545631999999997, 2024-09-21T15:49:37.198 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:37.199 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:37.199 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:37.199 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:37.199 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:37.199 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 4950 2024-09-21T15:49:37.199 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.199 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:37.199 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.199 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:37.199 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.457125+0000", 2024-09-21T15:49:37.199 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.199 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.200 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.200 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:37.200 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.457125+0000", 2024-09-21T15:49:37.200 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.200 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.200 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.200 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:37.200 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.457125+0000", 2024-09-21T15:49:37.200 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:37.200 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.200 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.201 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:37.201 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.457126+0000", 2024-09-21T15:49:37.201 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5519999999999999e-06 2024-09-21T15:49:37.201 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.201 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.201 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:37.201 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.457129+0000", 2024-09-21T15:49:37.201 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7989999999999998e-06 2024-09-21T15:49:37.201 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.201 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.202 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:37.202 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.457131+0000", 2024-09-21T15:49:37.202 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3080000000000002e-06 2024-09-21T15:49:37.202 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.202 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.202 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:37.202 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.496473+0000", 2024-09-21T15:49:37.202 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039341222000000002 2024-09-21T15:49:37.202 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.202 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.202 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:37.203 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.496565+0000", 2024-09-21T15:49:37.203 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2330000000000005e-05 2024-09-21T15:49:37.203 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.203 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.203 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:37.203 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.496671+0000", 2024-09-21T15:49:37.203 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000105648 2024-09-21T15:49:37.204 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.204 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:37.204 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.204 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.204 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.204 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:4962 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 1789952~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:37.205 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.459143+0000", 2024-09-21T15:49:37.205 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.710829201999999, 2024-09-21T15:49:37.205 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039478257000000003, 2024-09-21T15:49:37.205 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:37.205 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:37.205 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:37.205 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:37.206 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:37.206 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 4962 2024-09-21T15:49:37.206 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.206 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:37.206 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.206 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:37.207 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459143+0000", 2024-09-21T15:49:37.207 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.207 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.207 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.207 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:37.207 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459143+0000", 2024-09-21T15:49:37.208 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.208 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.208 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.208 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:37.208 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459143+0000", 2024-09-21T15:49:37.208 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:37.209 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.209 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.209 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:37.209 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459144+0000", 2024-09-21T15:49:37.209 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.142e-06 2024-09-21T15:49:37.209 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.209 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.210 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:37.210 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459147+0000", 2024-09-21T15:49:37.210 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8229999999999999e-06 2024-09-21T15:49:37.210 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.210 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.210 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:37.210 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459149+0000", 2024-09-21T15:49:37.211 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2859999999999998e-06 2024-09-21T15:49:37.211 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.211 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.211 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:37.211 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.498430+0000", 2024-09-21T15:49:37.211 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039280806000000001 2024-09-21T15:49:37.212 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.212 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.212 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:37.212 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.498520+0000", 2024-09-21T15:49:37.212 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0160000000000004e-05 2024-09-21T15:49:37.212 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.212 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.213 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:37.213 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.498622+0000", 2024-09-21T15:49:37.213 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000101275 2024-09-21T15:49:37.213 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.213 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:37.213 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.214 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.214 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.214 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:4963 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 1839104~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:37.214 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.459152+0000", 2024-09-21T15:49:37.214 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.710820807000001, 2024-09-21T15:49:37.214 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039710886000000001, 2024-09-21T15:49:37.214 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:37.215 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:37.215 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:37.215 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:37.215 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:37.215 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 4963 2024-09-21T15:49:37.215 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.216 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:37.216 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.216 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:37.216 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459152+0000", 2024-09-21T15:49:37.216 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.216 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.217 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.217 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:37.217 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459152+0000", 2024-09-21T15:49:37.217 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.217 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.217 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.217 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:37.218 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459151+0000", 2024-09-21T15:49:37.218 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:37.218 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.218 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.218 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:37.218 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459153+0000", 2024-09-21T15:49:37.219 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1400000000000001e-06 2024-09-21T15:49:37.219 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.219 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.219 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:37.219 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459155+0000", 2024-09-21T15:49:37.219 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7130000000000001e-06 2024-09-21T15:49:37.220 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.220 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.220 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:37.220 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459158+0000", 2024-09-21T15:49:37.220 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.368e-06 2024-09-21T15:49:37.220 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.220 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.221 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:37.221 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.498664+0000", 2024-09-21T15:49:37.221 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039505928000000003 2024-09-21T15:49:37.221 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.221 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.221 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:37.222 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.498756+0000", 2024-09-21T15:49:37.222 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2505999999999998e-05 2024-09-21T15:49:37.222 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.222 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.222 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:37.222 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.498863+0000", 2024-09-21T15:49:37.222 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000106457 2024-09-21T15:49:37.223 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.223 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:37.223 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.223 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.223 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.223 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:4964 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 1888256~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:37.223 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.459160+0000", 2024-09-21T15:49:37.224 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.710812090000001, 2024-09-21T15:49:37.224 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039933237000000003, 2024-09-21T15:49:37.224 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:37.224 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:37.224 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:37.224 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:37.224 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:37.225 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 4964 2024-09-21T15:49:37.225 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.225 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:37.225 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.225 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:37.225 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459160+0000", 2024-09-21T15:49:37.226 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.226 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.226 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.226 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:37.226 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459160+0000", 2024-09-21T15:49:37.226 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.227 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.227 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.227 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:37.227 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459160+0000", 2024-09-21T15:49:37.227 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:37.227 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.228 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.228 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:37.228 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459162+0000", 2024-09-21T15:49:37.228 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.42e-06 2024-09-21T15:49:37.228 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.228 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.228 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:37.229 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459164+0000", 2024-09-21T15:49:37.229 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8059999999999999e-06 2024-09-21T15:49:37.229 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.229 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.229 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:37.229 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.459167+0000", 2024-09-21T15:49:37.230 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.835e-06 2024-09-21T15:49:37.230 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.230 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.230 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:37.230 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.498904+0000", 2024-09-21T15:49:37.230 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039736831 2024-09-21T15:49:37.231 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.231 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.231 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:37.231 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.498992+0000", 2024-09-21T15:49:37.231 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7886999999999994e-05 2024-09-21T15:49:37.231 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.231 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.232 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:37.232 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.499094+0000", 2024-09-21T15:49:37.232 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000101681 2024-09-21T15:49:37.232 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.232 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:37.232 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.233 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.233 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.233 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:4966 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 1986560~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:37.233 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.460005+0000", 2024-09-21T15:49:37.233 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.709967064000001, 2024-09-21T15:49:37.233 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039390041000000001, 2024-09-21T15:49:37.233 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:37.234 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:37.234 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:37.234 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:37.234 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:37.234 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 4966 2024-09-21T15:49:37.234 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.234 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:37.235 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.235 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:37.235 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460005+0000", 2024-09-21T15:49:37.235 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.235 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.235 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.235 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:37.236 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460005+0000", 2024-09-21T15:49:37.236 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.236 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.236 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.236 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:37.236 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460005+0000", 2024-09-21T15:49:37.237 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:37.237 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.237 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.237 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:37.237 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460006+0000", 2024-09-21T15:49:37.237 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2359999999999999e-06 2024-09-21T15:49:37.238 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.238 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.238 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:37.238 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460013+0000", 2024-09-21T15:49:37.238 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1239999999999999e-06 2024-09-21T15:49:37.238 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.239 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.239 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:37.239 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460018+0000", 2024-09-21T15:49:37.239 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3449999999999999e-06 2024-09-21T15:49:37.239 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.239 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.239 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:37.240 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.499283+0000", 2024-09-21T15:49:37.240 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039264847999999998 2024-09-21T15:49:37.240 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.240 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.240 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:37.240 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.499334+0000", 2024-09-21T15:49:37.240 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1279000000000002e-05 2024-09-21T15:49:37.240 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.241 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.241 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:37.241 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.499395+0000", 2024-09-21T15:49:37.241 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1472999999999999e-05 2024-09-21T15:49:37.241 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.241 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:37.241 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.242 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.242 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.242 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:4967 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 2035712~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:37.242 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.460021+0000", 2024-09-21T15:49:37.242 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.709951153999999, 2024-09-21T15:49:37.242 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039505047000000001, 2024-09-21T15:49:37.242 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:37.243 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:37.243 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:37.243 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:37.243 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:37.243 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 4967 2024-09-21T15:49:37.243 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.243 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:37.244 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.244 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:37.244 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460021+0000", 2024-09-21T15:49:37.244 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.244 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.244 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.244 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:37.245 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460021+0000", 2024-09-21T15:49:37.245 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.245 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.245 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.245 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:37.245 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460021+0000", 2024-09-21T15:49:37.246 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:37.246 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.246 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.246 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:37.246 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460022+0000", 2024-09-21T15:49:37.246 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3710000000000001e-06 2024-09-21T15:49:37.246 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.247 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.247 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:37.247 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460025+0000", 2024-09-21T15:49:37.247 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.067e-06 2024-09-21T15:49:37.247 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.247 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.247 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:37.248 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460028+0000", 2024-09-21T15:49:37.248 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5009999999999999e-06 2024-09-21T15:49:37.248 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.248 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.248 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:37.248 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.499420+0000", 2024-09-21T15:49:37.248 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039391977000000002 2024-09-21T15:49:37.248 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.249 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.249 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:37.249 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.499472+0000", 2024-09-21T15:49:37.249 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2057e-05 2024-09-21T15:49:37.249 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.249 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.250 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:37.250 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.499526+0000", 2024-09-21T15:49:37.250 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4335000000000001e-05 2024-09-21T15:49:37.250 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.250 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:37.250 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.250 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.251 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.251 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:4968 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 2084864~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:37.251 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.460030+0000", 2024-09-21T15:49:37.251 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.709942406, 2024-09-21T15:49:37.251 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039629655, 2024-09-21T15:49:37.251 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:37.251 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:37.252 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:37.252 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:37.252 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:37.252 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 4968 2024-09-21T15:49:37.252 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.252 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:37.252 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.253 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:37.253 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460030+0000", 2024-09-21T15:49:37.253 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.253 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.253 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.253 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:37.253 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460030+0000", 2024-09-21T15:49:37.253 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.254 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.254 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.254 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:37.254 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460030+0000", 2024-09-21T15:49:37.254 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:37.254 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.255 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.255 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:37.255 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460037+0000", 2024-09-21T15:49:37.255 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8569999999999997e-06 2024-09-21T15:49:37.255 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.255 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.255 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:37.255 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460040+0000", 2024-09-21T15:49:37.255 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1729999999999999e-06 2024-09-21T15:49:37.256 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.256 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.256 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:37.256 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460042+0000", 2024-09-21T15:49:37.256 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4559999999999999e-06 2024-09-21T15:49:37.256 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.256 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.256 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:37.256 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.499547+0000", 2024-09-21T15:49:37.256 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039504723999999998 2024-09-21T15:49:37.256 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.257 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.257 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:37.257 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.499596+0000", 2024-09-21T15:49:37.257 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9230000000000001e-05 2024-09-21T15:49:37.257 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.257 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.258 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:37.258 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.499660+0000", 2024-09-21T15:49:37.258 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3473999999999996e-05 2024-09-21T15:49:37.258 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.258 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:37.258 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.258 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.259 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.259 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:5054 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 24576~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:37.259 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.460046+0000", 2024-09-21T15:49:37.259 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.709926264, 2024-09-21T15:49:37.259 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039744092000000002, 2024-09-21T15:49:37.259 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:37.259 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:37.260 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:37.260 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:37.260 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:37.260 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 5054 2024-09-21T15:49:37.260 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.260 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:37.260 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.261 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:37.261 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460046+0000", 2024-09-21T15:49:37.261 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.261 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.261 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.261 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:37.261 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460046+0000", 2024-09-21T15:49:37.262 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.262 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.262 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.262 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:37.262 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460046+0000", 2024-09-21T15:49:37.262 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:37.262 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.263 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.263 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:37.263 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460051+0000", 2024-09-21T15:49:37.263 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4790000000000004e-06 2024-09-21T15:49:37.263 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.263 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.264 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:37.264 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460055+0000", 2024-09-21T15:49:37.264 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6830000000000001e-06 2024-09-21T15:49:37.264 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.264 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.264 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:37.265 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460057+0000", 2024-09-21T15:49:37.265 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5550000000000001e-06 2024-09-21T15:49:37.265 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.265 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.265 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:37.265 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.499687+0000", 2024-09-21T15:49:37.265 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039629736999999998 2024-09-21T15:49:37.265 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.265 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.265 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:37.265 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.499734+0000", 2024-09-21T15:49:37.266 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6412999999999998e-05 2024-09-21T15:49:37.266 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.266 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.266 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:37.266 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.499790+0000", 2024-09-21T15:49:37.266 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6666999999999998e-05 2024-09-21T15:49:37.266 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.266 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:37.266 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.266 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.266 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.266 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:5055 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 73728~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:37.266 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.460060+0000", 2024-09-21T15:49:37.267 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.709912649, 2024-09-21T15:49:37.267 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039853507000000003, 2024-09-21T15:49:37.267 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:37.267 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:37.267 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:37.267 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:37.267 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:37.267 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 5055 2024-09-21T15:49:37.267 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.267 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:37.267 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.267 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:37.267 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460060+0000", 2024-09-21T15:49:37.268 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.268 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.268 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.268 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:37.268 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460060+0000", 2024-09-21T15:49:37.268 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.268 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.268 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.268 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:37.268 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460060+0000", 2024-09-21T15:49:37.268 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:37.268 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.268 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.269 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:37.269 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460061+0000", 2024-09-21T15:49:37.269 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2410000000000001e-06 2024-09-21T15:49:37.269 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.269 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.269 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:37.269 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460064+0000", 2024-09-21T15:49:37.269 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7539999999999998e-06 2024-09-21T15:49:37.269 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.269 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.269 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:37.269 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460069+0000", 2024-09-21T15:49:37.269 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.558e-06 2024-09-21T15:49:37.269 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.270 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.270 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:37.270 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.499816+0000", 2024-09-21T15:49:37.270 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039746385000000002 2024-09-21T15:49:37.270 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.270 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.270 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:37.270 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.499860+0000", 2024-09-21T15:49:37.270 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4740999999999997e-05 2024-09-21T15:49:37.270 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.270 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.270 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:37.270 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.499913+0000", 2024-09-21T15:49:37.270 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3057999999999999e-05 2024-09-21T15:49:37.271 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.271 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:37.271 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.271 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.271 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.271 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:5056 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 122880~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:37.271 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.460249+0000", 2024-09-21T15:49:37.271 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.709723789999998, 2024-09-21T15:49:37.271 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039822748999999998, 2024-09-21T15:49:37.271 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:37.271 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:37.271 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:37.271 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:37.271 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:37.272 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 5056 2024-09-21T15:49:37.272 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.272 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:37.272 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.272 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:37.272 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460249+0000", 2024-09-21T15:49:37.272 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.272 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.272 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.272 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:37.272 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460249+0000", 2024-09-21T15:49:37.272 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.272 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.273 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.273 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:37.273 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460120+0000", 2024-09-21T15:49:37.273 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998713 2024-09-21T15:49:37.273 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.273 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.273 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:37.273 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460251+0000", 2024-09-21T15:49:37.273 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000130977 2024-09-21T15:49:37.273 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.273 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.273 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:37.273 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460261+0000", 2024-09-21T15:49:37.273 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.8879999999999992e-06 2024-09-21T15:49:37.273 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.274 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.274 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:37.274 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460267+0000", 2024-09-21T15:49:37.274 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7379999999999999e-06 2024-09-21T15:49:37.274 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.274 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.274 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:37.274 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.499940+0000", 2024-09-21T15:49:37.274 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039673228999999997 2024-09-21T15:49:37.274 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.274 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.274 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:37.274 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500008+0000", 2024-09-21T15:49:37.275 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7696999999999999e-05 2024-09-21T15:49:37.275 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.275 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.275 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:37.275 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500071+0000", 2024-09-21T15:49:37.275 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3856000000000007e-05 2024-09-21T15:49:37.275 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.275 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:37.275 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.275 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.275 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.275 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:5057 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 172032~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:37.275 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.460377+0000", 2024-09-21T15:49:37.275 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.709595003, 2024-09-21T15:49:37.276 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039811802, 2024-09-21T15:49:37.276 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:37.276 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:37.276 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:37.276 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:37.276 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:37.276 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 5057 2024-09-21T15:49:37.276 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.276 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:37.276 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.276 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:37.276 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460377+0000", 2024-09-21T15:49:37.276 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.276 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.276 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.277 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:37.277 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460377+0000", 2024-09-21T15:49:37.277 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.277 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.277 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.277 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:37.277 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460277+0000", 2024-09-21T15:49:37.277 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998994 2024-09-21T15:49:37.277 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.277 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.277 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:37.277 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460380+0000", 2024-09-21T15:49:37.277 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010330700000000001 2024-09-21T15:49:37.277 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.278 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.278 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:37.278 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460387+0000", 2024-09-21T15:49:37.278 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8770000000000004e-06 2024-09-21T15:49:37.278 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.278 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.278 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:37.278 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460394+0000", 2024-09-21T15:49:37.278 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.489e-06 2024-09-21T15:49:37.278 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.278 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.278 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:37.278 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500097+0000", 2024-09-21T15:49:37.278 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039702863999999997 2024-09-21T15:49:37.278 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.279 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.279 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:37.279 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500139+0000", 2024-09-21T15:49:37.279 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2262999999999998e-05 2024-09-21T15:49:37.279 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.279 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.279 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:37.279 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500189+0000", 2024-09-21T15:49:37.279 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0377999999999998e-05 2024-09-21T15:49:37.279 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.279 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:37.279 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.279 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.279 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.280 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:5059 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 221184~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:37.280 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.460637+0000", 2024-09-21T15:49:37.280 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.709335852999999, 2024-09-21T15:49:37.280 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039661409000000002, 2024-09-21T15:49:37.280 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:37.280 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:37.280 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:37.280 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:37.280 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:37.280 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 5059 2024-09-21T15:49:37.280 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.280 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:37.280 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.280 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:37.280 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460637+0000", 2024-09-21T15:49:37.281 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.281 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.281 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.281 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:37.281 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460637+0000", 2024-09-21T15:49:37.281 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.281 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.281 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.281 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:37.281 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460400+0000", 2024-09-21T15:49:37.281 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999763 2024-09-21T15:49:37.281 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.281 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.281 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:37.281 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460639+0000", 2024-09-21T15:49:37.282 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00023958900000000001 2024-09-21T15:49:37.282 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.282 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.282 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:37.282 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460645+0000", 2024-09-21T15:49:37.282 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.609e-06 2024-09-21T15:49:37.282 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.282 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.282 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:37.282 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460649+0000", 2024-09-21T15:49:37.282 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0239999999999999e-06 2024-09-21T15:49:37.282 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.282 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.282 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:37.283 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500211+0000", 2024-09-21T15:49:37.283 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039562011000000001 2024-09-21T15:49:37.283 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.283 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.283 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:37.283 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500251+0000", 2024-09-21T15:49:37.283 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0368999999999999e-05 2024-09-21T15:49:37.283 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.283 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.283 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:37.283 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500298+0000", 2024-09-21T15:49:37.283 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6826000000000002e-05 2024-09-21T15:49:37.283 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.283 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:37.283 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.284 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.284 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.284 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:5060 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 270336~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:37.284 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.460780+0000", 2024-09-21T15:49:37.284 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.709192875999999, 2024-09-21T15:49:37.284 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039627057, 2024-09-21T15:49:37.284 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:37.284 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:37.284 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:37.284 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:37.284 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:37.284 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 5060 2024-09-21T15:49:37.284 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.284 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:37.285 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.285 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:37.285 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460780+0000", 2024-09-21T15:49:37.285 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.285 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.285 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.285 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:37.285 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460780+0000", 2024-09-21T15:49:37.285 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.285 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.285 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.285 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:37.285 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460655+0000", 2024-09-21T15:49:37.285 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998755 2024-09-21T15:49:37.285 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.286 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.286 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:37.286 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460782+0000", 2024-09-21T15:49:37.286 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012715299999999999 2024-09-21T15:49:37.286 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.286 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.286 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:37.286 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460789+0000", 2024-09-21T15:49:37.286 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6959999999999999e-06 2024-09-21T15:49:37.286 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.286 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.286 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:37.286 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460794+0000", 2024-09-21T15:49:37.287 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1900000000000003e-06 2024-09-21T15:49:37.287 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.287 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.287 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:37.287 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500322+0000", 2024-09-21T15:49:37.287 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039527481000000003 2024-09-21T15:49:37.287 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.287 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.287 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:37.287 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500360+0000", 2024-09-21T15:49:37.287 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8674999999999998e-05 2024-09-21T15:49:37.287 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.287 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.287 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:37.288 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500407+0000", 2024-09-21T15:49:37.288 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6316999999999997e-05 2024-09-21T15:49:37.288 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.288 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:37.288 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.288 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.288 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.288 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:5061 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 319488~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:37.288 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.461015+0000", 2024-09-21T15:49:37.288 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.708957664, 2024-09-21T15:49:37.288 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039499536000000002, 2024-09-21T15:49:37.288 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:37.288 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:37.289 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:37.289 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:37.289 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:37.289 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 5061 2024-09-21T15:49:37.289 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.289 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:37.289 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.289 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:37.289 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461015+0000", 2024-09-21T15:49:37.289 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.289 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.289 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.289 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:37.289 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461015+0000", 2024-09-21T15:49:37.290 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.290 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.290 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.290 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:37.290 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.460799+0000", 2024-09-21T15:49:37.290 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9997845 2024-09-21T15:49:37.290 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.290 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.290 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:37.290 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461018+0000", 2024-09-21T15:49:37.290 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00021889000000000001 2024-09-21T15:49:37.290 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.290 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.290 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:37.291 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461026+0000", 2024-09-21T15:49:37.291 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1380000000000007e-06 2024-09-21T15:49:37.291 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.291 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.291 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:37.291 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461032+0000", 2024-09-21T15:49:37.291 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8799999999999996e-06 2024-09-21T15:49:37.291 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.291 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.291 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:37.291 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500429+0000", 2024-09-21T15:49:37.291 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039396704999999997 2024-09-21T15:49:37.291 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.291 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.291 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:37.292 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500467+0000", 2024-09-21T15:49:37.292 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7938999999999997e-05 2024-09-21T15:49:37.292 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.292 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.292 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:37.292 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500514+0000", 2024-09-21T15:49:37.292 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7562000000000003e-05 2024-09-21T15:49:37.292 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.292 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:37.292 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.292 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.292 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.292 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:5062 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 368640~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:37.292 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.461158+0000", 2024-09-21T15:49:37.293 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.708814274000002, 2024-09-21T15:49:37.293 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039463221, 2024-09-21T15:49:37.293 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:37.293 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:37.293 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:37.293 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:37.293 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:37.293 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 5062 2024-09-21T15:49:37.293 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.293 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:37.293 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.293 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:37.293 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461158+0000", 2024-09-21T15:49:37.293 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.294 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.294 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.294 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:37.294 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461158+0000", 2024-09-21T15:49:37.294 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.294 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.294 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.294 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:37.294 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461037+0000", 2024-09-21T15:49:37.294 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998789 2024-09-21T15:49:37.294 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.294 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.294 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:37.295 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461162+0000", 2024-09-21T15:49:37.295 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000124461 2024-09-21T15:49:37.295 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.295 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.295 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:37.295 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461170+0000", 2024-09-21T15:49:37.295 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6789999999999994e-06 2024-09-21T15:49:37.295 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.295 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.295 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:37.295 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461176+0000", 2024-09-21T15:49:37.295 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0100000000000001e-06 2024-09-21T15:49:37.295 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.295 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.295 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:37.296 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500537+0000", 2024-09-21T15:49:37.296 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039360366000000001 2024-09-21T15:49:37.296 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.296 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.296 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:37.296 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500575+0000", 2024-09-21T15:49:37.296 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8126000000000001e-05 2024-09-21T15:49:37.296 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.296 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.296 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:37.296 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500621+0000", 2024-09-21T15:49:37.296 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6697999999999999e-05 2024-09-21T15:49:37.296 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.296 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:37.297 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.297 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.297 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.297 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:5063 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 417792~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:37.297 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.461421+0000", 2024-09-21T15:49:37.297 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.708551579000002, 2024-09-21T15:49:37.297 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039308520999999999, 2024-09-21T15:49:37.297 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:37.297 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:37.297 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:37.297 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:37.297 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:37.297 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 5063 2024-09-21T15:49:37.297 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.298 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:37.298 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.298 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:37.298 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461421+0000", 2024-09-21T15:49:37.298 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.298 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.298 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.298 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:37.298 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461421+0000", 2024-09-21T15:49:37.298 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.298 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.298 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.298 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:37.298 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461181+0000", 2024-09-21T15:49:37.299 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9997602 2024-09-21T15:49:37.299 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.299 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.299 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:37.299 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461426+0000", 2024-09-21T15:49:37.299 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00024468599999999998 2024-09-21T15:49:37.299 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.299 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.299 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:37.299 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461438+0000", 2024-09-21T15:49:37.299 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2527000000000001e-05 2024-09-21T15:49:37.299 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.299 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.299 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:37.300 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461446+0000", 2024-09-21T15:49:37.300 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.199e-06 2024-09-21T15:49:37.300 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.300 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.300 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:37.300 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500645+0000", 2024-09-21T15:49:37.300 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039199680000000001 2024-09-21T15:49:37.300 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.300 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.300 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:37.300 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500684+0000", 2024-09-21T15:49:37.300 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9103999999999997e-05 2024-09-21T15:49:37.300 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.300 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.301 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:37.301 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500729+0000", 2024-09-21T15:49:37.301 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4941999999999998e-05 2024-09-21T15:49:37.301 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.301 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:37.301 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.301 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.301 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.301 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:5064 3.1d 3:b9853cc4:::rbd_data.1133f3dfbd6b.0000000000000002:5 [read 466944~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e29)", 2024-09-21T15:49:37.301 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:14.461542+0000", 2024-09-21T15:49:37.301 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.708430071999999, 2024-09-21T15:49:37.301 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.03938125, 2024-09-21T15:49:37.301 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:37.302 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:37.302 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:37.302 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:37.302 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:37.302 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 5064 2024-09-21T15:49:37.302 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.302 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:37.302 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.302 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:37.302 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461542+0000", 2024-09-21T15:49:37.302 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.302 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.302 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.302 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:37.303 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461542+0000", 2024-09-21T15:49:37.303 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.303 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.303 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.303 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:37.303 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461457+0000", 2024-09-21T15:49:37.303 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999146 2024-09-21T15:49:37.303 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.303 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.303 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:37.303 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461546+0000", 2024-09-21T15:49:37.303 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8718e-05 2024-09-21T15:49:37.303 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.303 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.304 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:37.304 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461555+0000", 2024-09-21T15:49:37.304 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8370000000000003e-06 2024-09-21T15:49:37.304 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.304 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.304 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:37.304 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.461561+0000", 2024-09-21T15:49:37.304 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4150000000000004e-06 2024-09-21T15:49:37.304 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.304 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.304 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:37.304 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500753+0000", 2024-09-21T15:49:37.304 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.039192183999999998 2024-09-21T15:49:37.304 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.305 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.305 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:37.305 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500826+0000", 2024-09-21T15:49:37.305 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2428000000000005e-05 2024-09-21T15:49:37.305 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.305 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.305 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:37.305 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:14.500924+0000", 2024-09-21T15:49:37.305 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7848000000000002e-05 2024-09-21T15:49:37.305 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.305 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:37.305 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.305 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.305 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:37.306 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:37.307 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T15:49:37.307 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:37.306+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:37.399 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:49:37.443 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:37.442+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:37.508 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:37.507+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:37.525 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:37.525 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:49:37.526 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.526 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:71527 3.a 3.814f916a (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:37.526 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.500635+0000", 2024-09-21T15:49:37.526 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.023155041000000001, 2024-09-21T15:49:37.526 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.023188311999999999, 2024-09-21T15:49:37.526 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:37.526 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "queued for pg", 2024-09-21T15:49:37.526 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:37.526 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:37.526 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:37.526 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 71527 2024-09-21T15:49:37.527 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.527 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:37.527 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.527 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:37.527 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.500635+0000", 2024-09-21T15:49:37.527 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.527 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.527 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.527 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:37.527 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.500635+0000", 2024-09-21T15:49:37.527 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.528 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.528 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.528 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:37.528 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.500635+0000", 2024-09-21T15:49:37.528 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:37.528 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.528 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.528 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:37.528 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.500637+0000", 2024-09-21T15:49:37.528 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.295e-06 2024-09-21T15:49:37.528 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.528 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.529 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:37.529 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.500643+0000", 2024-09-21T15:49:37.529 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8409999999999996e-06 2024-09-21T15:49:37.529 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.529 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.529 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:37.529 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.500648+0000", 2024-09-21T15:49:37.529 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.707e-06 2024-09-21T15:49:37.529 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.529 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:37.529 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.530 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.530 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.530 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:71528 3.a 3.814f916a (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:37.530 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.500678+0000", 2024-09-21T15:49:37.530 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.023112685000000001, 2024-09-21T15:49:37.530 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.023211688000000001, 2024-09-21T15:49:37.530 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:37.530 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "queued for pg", 2024-09-21T15:49:37.530 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:37.530 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:37.530 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:37.531 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 71528 2024-09-21T15:49:37.531 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.531 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:37.531 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.531 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:37.531 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.500678+0000", 2024-09-21T15:49:37.531 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.531 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.531 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.531 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:37.531 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.500678+0000", 2024-09-21T15:49:37.532 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.532 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.532 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.532 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:37.532 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.500677+0000", 2024-09-21T15:49:37.532 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:37.532 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.532 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.532 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:37.532 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.500680+0000", 2024-09-21T15:49:37.532 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5409999999999999e-06 2024-09-21T15:49:37.533 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.533 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.533 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:37.533 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.500686+0000", 2024-09-21T15:49:37.533 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5490000000000003e-06 2024-09-21T15:49:37.533 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.533 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.533 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:37.533 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.500691+0000", 2024-09-21T15:49:37.533 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.826e-06 2024-09-21T15:49:37.533 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.533 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:37.534 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.534 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.534 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.534 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:71529 3.a 3.814f916a (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:37.534 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.500695+0000", 2024-09-21T15:49:37.534 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.023095107, 2024-09-21T15:49:37.534 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.023249525, 2024-09-21T15:49:37.534 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:37.534 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "queued for pg", 2024-09-21T15:49:37.534 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:37.534 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:37.534 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:37.535 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 71529 2024-09-21T15:49:37.535 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.535 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:37.535 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.535 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:37.535 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.500695+0000", 2024-09-21T15:49:37.535 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.535 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.535 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.535 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:37.535 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.500695+0000", 2024-09-21T15:49:37.535 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.536 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.536 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.536 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:37.536 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.500695+0000", 2024-09-21T15:49:37.536 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:37.536 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.536 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.536 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:37.536 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.500704+0000", 2024-09-21T15:49:37.536 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.9479999999999997e-06 2024-09-21T15:49:37.537 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.537 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.537 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:37.537 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.500710+0000", 2024-09-21T15:49:37.537 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6239999999999996e-06 2024-09-21T15:49:37.537 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.537 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.537 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:37.537 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.500715+0000", 2024-09-21T15:49:37.537 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.729e-06 2024-09-21T15:49:37.537 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.537 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:37.537 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.538 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.538 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.538 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:71530 3.a 3.814f916a (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:37.538 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.500720+0000", 2024-09-21T15:49:37.538 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.023070876000000001, 2024-09-21T15:49:37.538 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.023297941999999999, 2024-09-21T15:49:37.538 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:37.538 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "queued for pg", 2024-09-21T15:49:37.538 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:37.538 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:37.538 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:37.538 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 71530 2024-09-21T15:49:37.539 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.539 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:37.539 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.539 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:37.539 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.500720+0000", 2024-09-21T15:49:37.539 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.539 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.539 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.539 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:37.539 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.500720+0000", 2024-09-21T15:49:37.539 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.539 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.540 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.540 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:37.540 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.500719+0000", 2024-09-21T15:49:37.540 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:37.540 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.540 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.540 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:37.540 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.500721+0000", 2024-09-21T15:49:37.540 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3120000000000001e-06 2024-09-21T15:49:37.540 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.540 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.541 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:37.541 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.500727+0000", 2024-09-21T15:49:37.541 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7679999999999997e-06 2024-09-21T15:49:37.541 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.541 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.541 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:37.541 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.500732+0000", 2024-09-21T15:49:37.541 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4039999999999997e-06 2024-09-21T15:49:37.541 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.541 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:37.541 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.541 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.542 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.542 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:71531 3.a 3.814f916a (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:37.542 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.500736+0000", 2024-09-21T15:49:37.542 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.023054710999999999, 2024-09-21T15:49:37.542 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.023338558999999998, 2024-09-21T15:49:37.542 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:37.542 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "queued for pg", 2024-09-21T15:49:37.542 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:37.542 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:37.542 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:37.542 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 71531 2024-09-21T15:49:37.543 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.543 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:37.543 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.543 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:37.543 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.500736+0000", 2024-09-21T15:49:37.543 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.543 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.543 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.543 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:37.543 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.500736+0000", 2024-09-21T15:49:37.543 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.544 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.544 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.544 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:37.544 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.500735+0000", 2024-09-21T15:49:37.544 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:37.544 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.544 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.544 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:37.544 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.500737+0000", 2024-09-21T15:49:37.544 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0499999999999999e-06 2024-09-21T15:49:37.544 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.545 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.545 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:37.545 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.500744+0000", 2024-09-21T15:49:37.545 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9120000000000001e-06 2024-09-21T15:49:37.545 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.545 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.545 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:37.545 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.500749+0000", 2024-09-21T15:49:37.545 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7319999999999998e-06 2024-09-21T15:49:37.545 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.545 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:37.545 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.546 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.546 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.546 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:71532 3.a 3.814f916a (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:37.546 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.500753+0000", 2024-09-21T15:49:37.546 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.023037216999999999, 2024-09-21T15:49:37.546 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.023377555000000001, 2024-09-21T15:49:37.546 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:37.546 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "queued for pg", 2024-09-21T15:49:37.546 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:37.546 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:37.547 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:37.547 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 71532 2024-09-21T15:49:37.547 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.547 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:37.547 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.547 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:37.547 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.500753+0000", 2024-09-21T15:49:37.547 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.547 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.547 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.547 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:37.547 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.500753+0000", 2024-09-21T15:49:37.548 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.548 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.548 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.548 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:37.548 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.500753+0000", 2024-09-21T15:49:37.548 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:37.548 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.548 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.548 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:37.548 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.500755+0000", 2024-09-21T15:49:37.548 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1260000000000001e-06 2024-09-21T15:49:37.549 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.549 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.549 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:37.549 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.500761+0000", 2024-09-21T15:49:37.549 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7869999999999998e-06 2024-09-21T15:49:37.549 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.549 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.549 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:37.549 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.500765+0000", 2024-09-21T15:49:37.549 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6720000000000003e-06 2024-09-21T15:49:37.549 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.550 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:37.550 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.550 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.550 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.550 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:71533 3.a 3.814f916a (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:37.550 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.500770+0000", 2024-09-21T15:49:37.551 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.023020797999999999, 2024-09-21T15:49:37.551 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.023425217000000002, 2024-09-21T15:49:37.551 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:37.551 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "queued for pg", 2024-09-21T15:49:37.551 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:37.551 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:37.551 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:37.551 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 71533 2024-09-21T15:49:37.551 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.551 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:37.551 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.551 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:37.552 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.500770+0000", 2024-09-21T15:49:37.552 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.552 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.552 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.552 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:37.552 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.500770+0000", 2024-09-21T15:49:37.552 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.552 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.552 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.552 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:37.552 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.500769+0000", 2024-09-21T15:49:37.552 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:37.552 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.553 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.553 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:37.553 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.500771+0000", 2024-09-21T15:49:37.553 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1610000000000001e-06 2024-09-21T15:49:37.553 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.553 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.553 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:37.553 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.500777+0000", 2024-09-21T15:49:37.553 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6620000000000002e-06 2024-09-21T15:49:37.553 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.553 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.553 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:37.553 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.500782+0000", 2024-09-21T15:49:37.553 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5820000000000003e-06 2024-09-21T15:49:37.553 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.553 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:37.554 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.554 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.554 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.554 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:71536 3.a 3.814f916a (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:37.554 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.501035+0000", 2024-09-21T15:49:37.554 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.022755315000000002, 2024-09-21T15:49:37.554 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.02321664, 2024-09-21T15:49:37.554 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:37.554 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "queued for pg", 2024-09-21T15:49:37.554 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:37.554 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:37.554 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:37.554 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 71536 2024-09-21T15:49:37.554 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.555 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:37.555 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.555 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:37.555 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.501035+0000", 2024-09-21T15:49:37.555 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.555 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.555 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.555 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:37.555 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.501035+0000", 2024-09-21T15:49:37.555 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.555 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.555 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.555 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:37.555 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.501034+0000", 2024-09-21T15:49:37.556 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:37.556 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.556 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.556 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:37.556 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.501038+0000", 2024-09-21T15:49:37.556 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1019999999999999e-06 2024-09-21T15:49:37.556 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.556 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.556 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:37.556 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.501048+0000", 2024-09-21T15:49:37.556 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3079999999999997e-06 2024-09-21T15:49:37.556 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.556 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.556 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:37.557 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.501054+0000", 2024-09-21T15:49:37.557 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2169999999999999e-06 2024-09-21T15:49:37.557 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.557 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:37.557 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.557 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.557 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.557 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:71537 3.a 3.814f916a (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:37.557 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.501060+0000", 2024-09-21T15:49:37.557 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.022730865999999999, 2024-09-21T15:49:37.557 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.023246816, 2024-09-21T15:49:37.557 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:37.557 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "queued for pg", 2024-09-21T15:49:37.557 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:37.557 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:37.558 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:37.558 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 71537 2024-09-21T15:49:37.558 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.558 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:37.558 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.558 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:37.558 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.501060+0000", 2024-09-21T15:49:37.558 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.558 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.558 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.558 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:37.558 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.501060+0000", 2024-09-21T15:49:37.558 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.559 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.559 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.559 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:37.559 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.501059+0000", 2024-09-21T15:49:37.559 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:37.559 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.559 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.559 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:37.559 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.501062+0000", 2024-09-21T15:49:37.559 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7829999999999999e-06 2024-09-21T15:49:37.559 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.559 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.559 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:37.559 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.501069+0000", 2024-09-21T15:49:37.560 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.939e-06 2024-09-21T15:49:37.560 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.560 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.560 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:37.560 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.501074+0000", 2024-09-21T15:49:37.560 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0490000000000003e-06 2024-09-21T15:49:37.560 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.560 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:37.560 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.560 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.560 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.560 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:71539 3.a 3.814f916a (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:37.560 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.501088+0000", 2024-09-21T15:49:37.560 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.022702731, 2024-09-21T15:49:37.561 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.023279597999999999, 2024-09-21T15:49:37.561 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:37.561 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "queued for pg", 2024-09-21T15:49:37.561 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:37.561 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:37.561 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:37.561 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 71539 2024-09-21T15:49:37.561 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.561 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:37.561 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.561 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:37.561 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.501088+0000", 2024-09-21T15:49:37.561 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.561 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.562 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.562 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:37.562 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.501088+0000", 2024-09-21T15:49:37.562 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.562 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.562 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.562 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:37.562 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.501087+0000", 2024-09-21T15:49:37.562 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:37.562 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.562 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.562 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:37.562 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.501090+0000", 2024-09-21T15:49:37.562 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5600000000000001e-06 2024-09-21T15:49:37.563 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.563 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.563 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:37.563 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.501096+0000", 2024-09-21T15:49:37.563 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.489e-06 2024-09-21T15:49:37.563 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.563 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.563 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:37.563 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.501101+0000", 2024-09-21T15:49:37.563 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.994e-06 2024-09-21T15:49:37.563 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.563 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:37.563 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.563 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.564 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.564 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:71540 3.a 3.814f916a (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:37.564 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.501105+0000", 2024-09-21T15:49:37.564 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.022684879000000002, 2024-09-21T15:49:37.564 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.023324779, 2024-09-21T15:49:37.564 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:37.564 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "queued for pg", 2024-09-21T15:49:37.564 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:37.564 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:37.564 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:37.564 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 71540 2024-09-21T15:49:37.564 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.564 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:37.564 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.564 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:37.565 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.501105+0000", 2024-09-21T15:49:37.565 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.565 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.565 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.565 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:37.565 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.501105+0000", 2024-09-21T15:49:37.565 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.565 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.565 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.565 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:37.565 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.501105+0000", 2024-09-21T15:49:37.565 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:37.565 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.565 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.565 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:37.566 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.501108+0000", 2024-09-21T15:49:37.566 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5019999999999999e-06 2024-09-21T15:49:37.566 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.566 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.566 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:37.566 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.501114+0000", 2024-09-21T15:49:37.566 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9599999999999997e-06 2024-09-21T15:49:37.566 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.566 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.566 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:37.566 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.501118+0000", 2024-09-21T15:49:37.566 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6999999999999999e-06 2024-09-21T15:49:37.566 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.567 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:37.567 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.567 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.567 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.567 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:71543 3.a 3.814f916a (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:37.567 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.501123+0000", 2024-09-21T15:49:37.567 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.022667809000000001, 2024-09-21T15:49:37.567 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.023362394000000002, 2024-09-21T15:49:37.567 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:37.567 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "queued for pg", 2024-09-21T15:49:37.567 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:37.567 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:37.567 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:37.567 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 71543 2024-09-21T15:49:37.568 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.568 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:37.568 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.568 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:37.568 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.501123+0000", 2024-09-21T15:49:37.568 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.568 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.568 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.568 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:37.568 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.501123+0000", 2024-09-21T15:49:37.568 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.568 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.568 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.568 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:37.569 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.501122+0000", 2024-09-21T15:49:37.569 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:37.569 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.569 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.569 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:37.569 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.501124+0000", 2024-09-21T15:49:37.569 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2249999999999999e-06 2024-09-21T15:49:37.569 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.569 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.569 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:37.569 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.501130+0000", 2024-09-21T15:49:37.569 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6849999999999998e-06 2024-09-21T15:49:37.569 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.569 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.569 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:37.570 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.501134+0000", 2024-09-21T15:49:37.570 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2780000000000004e-06 2024-09-21T15:49:37.570 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.570 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:37.570 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.570 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.570 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.570 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:71551 3.a 3.814f916a (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:37.570 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.501244+0000", 2024-09-21T15:49:37.570 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.022545961999999999, 2024-09-21T15:49:37.570 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.023298135000000001, 2024-09-21T15:49:37.570 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:37.570 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "queued for pg", 2024-09-21T15:49:37.570 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:37.571 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:37.571 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:37.571 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 71551 2024-09-21T15:49:37.571 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.571 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:37.571 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.571 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:37.571 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.501244+0000", 2024-09-21T15:49:37.571 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.571 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.571 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.571 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:37.571 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.501244+0000", 2024-09-21T15:49:37.571 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:37.572 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.572 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.572 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:37.572 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.501244+0000", 2024-09-21T15:49:37.572 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:37.572 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.572 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.572 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:37.572 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.501247+0000", 2024-09-21T15:49:37.572 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5260000000000002e-06 2024-09-21T15:49:37.572 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.572 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.573 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:37.573 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.501255+0000", 2024-09-21T15:49:37.573 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5700000000000004e-06 2024-09-21T15:49:37.573 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:37.573 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:37.573 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:37.573 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.501260+0000", 2024-09-21T15:49:37.573 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3020000000000002e-06 2024-09-21T15:49:37.573 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.573 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:37.573 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.573 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:37.573 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-21T15:49:37.573 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 13 2024-09-21T15:49:37.574 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:37.574 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-09-21T15:49:37.575 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:49:37.575 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:49:37.576 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:49:37.576 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T15:49:37.576 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:49:37.577 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:49:37.591 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:49:37.592 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:49:37.611 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:49:37.612 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:49:37.616 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:37.615+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:37.710 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:37.709+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:37.745 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:49:37.745 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:49:37.751 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:37.752 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:37.752 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:49:37.752 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:49:37.752 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:37.763 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T15:49:37.773 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:49:37.773 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:49:37.803 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:49:37.803 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:49:37.811 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:37.809+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:37.901 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:37.901 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:37.910 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T15:49:37.911 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:37.910+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:37.965 INFO:teuthology.orchestra.run.smithi089.stdout:2642 punch from 0x2cd2334 to 0x2cd5610, (0x32dc bytes) 2024-09-21T15:49:37.978 INFO:teuthology.orchestra.run.smithi089.stdout:2645 write 0x6d08b47 thru 0x6d1166f (0x8b29 bytes) 2024-09-21T15:49:37.989 INFO:teuthology.orchestra.run.smithi089.stdout:2646 write 0x440cca5 thru 0x441ada0 (0xe0fc bytes) 2024-09-21T15:49:37.996 INFO:teuthology.orchestra.run.smithi089.stdout:2647 writesame 0x6c16475 thru 0x6c1ebf0 data_size 0x1d(0x877c bytes) 2024-09-21T15:49:38.005 INFO:teuthology.orchestra.run.smithi089.stdout:2649 read 0x4f18a40 thru 0x4f2795e (0xef1f bytes) 2024-09-21T15:49:38.005 INFO:teuthology.orchestra.run.smithi089.stdout:2650 read 0x5680e9b thru 0x5688737 (0x789d bytes) 2024-09-21T15:49:38.005 INFO:teuthology.orchestra.run.smithi089.stdout:2651 punch from 0x300a164 to 0x300e8e0, (0x477c bytes) 2024-09-21T15:49:38.006 INFO:teuthology.orchestra.run.smithi089.stdout:2652 writesame 0x284c36d thru 0x284d97b data_size 0x1(0x160f bytes) 2024-09-21T15:49:38.012 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:38.011+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:38.017 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:38.017 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:38.022 INFO:teuthology.orchestra.run.smithi089.stdout:2653 write 0xb2e0594 thru 0xb2e5f5e (0x59cb bytes) 2024-09-21T15:49:38.026 INFO:teuthology.orchestra.run.smithi089.stdout:2655 write 0x2ab459 thru 0x2b526c (0x9e14 bytes) 2024-09-21T15:49:38.027 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T15:49:38.035 INFO:teuthology.orchestra.run.smithi089.stdout:2656 punch from 0x5f33770 to 0x5f3e273, (0xab03 bytes) 2024-09-21T15:49:38.043 INFO:teuthology.orchestra.run.smithi089.stdout:2657 write 0xcf05de0 thru 0xcf113b3 (0xb5d4 bytes) 2024-09-21T15:49:38.047 INFO:teuthology.orchestra.run.smithi089.stdout:2659 read 0x728afa4 thru 0x7295ca0 (0xacfd bytes) 2024-09-21T15:49:38.047 INFO:teuthology.orchestra.run.smithi089.stdout:2660 read 0x6be4c36 thru 0x6bf08c8 (0xbc93 bytes) 2024-09-21T15:49:38.048 INFO:teuthology.orchestra.run.smithi089.stdout:2661 read 0xbb76d24 thru 0xbb80c48 (0x9f25 bytes) 2024-09-21T15:49:38.048 INFO:teuthology.orchestra.run.smithi089.stdout:2662 write 0x5a9fd20 thru 0x5aa2881 (0x2b62 bytes) 2024-09-21T15:49:38.055 INFO:teuthology.orchestra.run.smithi089.stdout:2663 punch from 0x95f9e31 to 0x95ff1b3, (0x5382 bytes) 2024-09-21T15:49:38.055 INFO:teuthology.orchestra.run.smithi089.stdout:2664 read 0xc0e64b0 thru 0xc0ecfbf (0x6b10 bytes) 2024-09-21T15:49:38.055 INFO:teuthology.orchestra.run.smithi089.stdout:2666 write 0xbb4c647 thru 0xbb536bb (0x7075 bytes) 2024-09-21T15:49:38.069 INFO:teuthology.orchestra.run.smithi089.stdout:2667 trunc from 0xcf113b4 to 0x4b61699 2024-09-21T15:49:38.108 INFO:teuthology.orchestra.run.smithi089.stdout:2668 writesame 0xb7f4f00 thru 0xb7feac3 data_size 0x6(0x9bc4 bytes) 2024-09-21T15:49:38.109 INFO:teuthology.orchestra.run.smithi089.stdout:2670 read 0x8f33f2d thru 0x8f42f26 (0xeffa bytes) 2024-09-21T15:49:38.109 INFO:teuthology.orchestra.run.smithi089.stdout:2671 write 0x1397a4c thru 0x13a3de4 (0xc399 bytes) 2024-09-21T15:49:38.111 INFO:teuthology.orchestra.run.smithi089.stdout:2672 write 0x1b3e3a8 thru 0x1b40d3e (0x2997 bytes) 2024-09-21T15:49:38.112 INFO:teuthology.orchestra.run.smithi089.stdout:2673 punch from 0x92260f to 0x92adcb, (0x87bc bytes) 2024-09-21T15:49:38.112 INFO:teuthology.orchestra.run.smithi089.stdout:2675 read 0xa987cd2 thru 0xa989bb7 (0x1ee6 bytes) 2024-09-21T15:49:38.113 INFO:teuthology.orchestra.run.smithi089.stdout:2678 writesame 0x2648372 thru 0x264db30 data_size 0x1(0x57bf bytes) 2024-09-21T15:49:38.113 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:38.111+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:38.123 INFO:teuthology.orchestra.run.smithi089.stdout:2680 read 0x9b10a1 thru 0x9b312a (0x208a bytes) 2024-09-21T15:49:38.123 INFO:teuthology.orchestra.run.smithi089.stdout:2681 read 0xd44793 thru 0xd544d8 (0xfd46 bytes) 2024-09-21T15:49:38.126 INFO:teuthology.orchestra.run.smithi089.stdout:2682 trunc from 0xb7feac4 to 0xd213994 2024-09-21T15:49:38.131 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:38.131 INFO:teuthology.orchestra.run.smithi089.stdout:2684 punch from 0xc09ab24 to 0xc0a8f51, (0xe42d bytes) 2024-09-21T15:49:38.132 INFO:teuthology.orchestra.run.smithi089.stdout:2685 read 0x41b69f7 thru 0x41bcfe7 (0x65f1 bytes) 2024-09-21T15:49:38.132 INFO:teuthology.orchestra.run.smithi089.stdout:2686 read 0xc569849 thru 0xc57832c (0xeae4 bytes) 2024-09-21T15:49:38.132 INFO:teuthology.orchestra.run.smithi089.stdout:2689 read 0x7c68120 thru 0x7c74a29 (0xc90a bytes) 2024-09-21T15:49:38.132 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:38.132 INFO:teuthology.orchestra.run.smithi089.stdout:2691 write 0x5890893 thru 0x589f51a (0xec88 bytes) 2024-09-21T15:49:38.134 INFO:teuthology.orchestra.run.smithi089.stdout:2692 punch from 0xc1bbc90 to 0xc1be3aa, (0x271a bytes) 2024-09-21T15:49:38.134 INFO:teuthology.orchestra.run.smithi089.stdout:2693 read 0x2314f34 thru 0x23193bb (0x4488 bytes) 2024-09-21T15:49:38.134 INFO:teuthology.orchestra.run.smithi089.stdout:2695 write 0xc5ee799 thru 0xc5f3bca (0x5432 bytes) 2024-09-21T15:49:38.136 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:38.136 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:38.136 INFO:teuthology.orchestra.run.smithi089.stdout:2696 read 0xb2e0571 thru 0xb2efddc (0xf86c bytes) 2024-09-21T15:49:38.137 INFO:teuthology.orchestra.run.smithi089.stdout:2697 read 0x193f9c6 thru 0x194d616 (0xdc51 bytes) 2024-09-21T15:49:38.137 INFO:teuthology.orchestra.run.smithi089.stdout:2698 write 0x27518f4 thru 0x275a303 (0x8a10 bytes) 2024-09-21T15:49:38.139 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:38.139 INFO:teuthology.orchestra.run.smithi089.stdout:2702 read 0x4a90c73 thru 0x4a93be8 (0x2f76 bytes) 2024-09-21T15:49:38.139 INFO:teuthology.orchestra.run.smithi089.stdout:2704 write 0xd182b3b thru 0xd1859cc (0x2e92 bytes) 2024-09-21T15:49:38.143 INFO:teuthology.orchestra.run.smithi089.stdout:2705 punch from 0xa8c5e21 to 0xa8d242b, (0xc60a bytes) 2024-09-21T15:49:38.143 INFO:teuthology.orchestra.run.smithi089.stdout:2706 read 0x3176f04 thru 0x3178d57 (0x1e54 bytes) 2024-09-21T15:49:38.143 INFO:teuthology.orchestra.run.smithi089.stdout:2707 read 0x2694228 thru 0x269ea1d (0xa7f6 bytes) 2024-09-21T15:49:38.144 INFO:teuthology.orchestra.run.smithi089.stdout:2708 read 0x5776bcc thru 0x5786327 (0xf75c bytes) 2024-09-21T15:49:38.144 INFO:teuthology.orchestra.run.smithi089.stdout:2709 write 0xc8b038d thru 0xc8bf171 (0xede5 bytes) 2024-09-21T15:49:38.145 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T15:49:38.146 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:38.146 INFO:teuthology.orchestra.run.smithi089.stdout:2711 read 0xad5f090 thru 0xad6592b (0x689c bytes) 2024-09-21T15:49:38.146 INFO:teuthology.orchestra.run.smithi089.stdout:2712 write 0x33d1469 thru 0x33dd517 (0xc0af bytes) 2024-09-21T15:49:38.148 INFO:teuthology.orchestra.run.smithi089.stdout:2714 writesame 0xcb7f1a1 thru 0xcb8e320 data_size 0x20(0xf180 bytes) 2024-09-21T15:49:38.150 INFO:teuthology.orchestra.run.smithi089.stdout:2715 write 0x7061e9f thru 0x7062ca5 (0xe07 bytes) 2024-09-21T15:49:38.151 INFO:teuthology.orchestra.run.smithi089.stdout:2718 writesame 0x3d1d2d5 thru 0x3d21477 data_size 0x3(0x41a3 bytes) 2024-09-21T15:49:38.154 INFO:teuthology.orchestra.run.smithi089.stdout:2719 write 0x16afbcd thru 0x16b1983 (0x1db7 bytes) 2024-09-21T15:49:38.155 INFO:teuthology.orchestra.run.smithi089.stdout:2720 write 0x88b3ea8 thru 0x88b86b8 (0x4811 bytes) 2024-09-21T15:49:38.156 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:38.156 INFO:teuthology.orchestra.run.smithi089.stdout:2722 write 0xa1b1f76 thru 0xa1b3d71 (0x1dfc bytes) 2024-09-21T15:49:38.159 INFO:teuthology.orchestra.run.smithi089.stdout:2723 punch from 0x67a14d9 to 0x67a7b07, (0x662e bytes) 2024-09-21T15:49:38.159 INFO:teuthology.orchestra.run.smithi089.stdout:2725 punch from 0x1412a40 to 0x141e08a, (0xb64a bytes) 2024-09-21T15:49:38.159 INFO:teuthology.orchestra.run.smithi089.stdout:2726 punch from 0xb407365 to 0xb40aa3d, (0x36d8 bytes) 2024-09-21T15:49:38.159 INFO:teuthology.orchestra.run.smithi089.stdout:2727 write 0x977a5c0 thru 0x9782c77 (0x86b8 bytes) 2024-09-21T15:49:38.160 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:38.172 INFO:teuthology.orchestra.run.smithi089.stdout:2729 trunc from 0xd213994 to 0x2b9efc8 2024-09-21T15:49:38.196 INFO:teuthology.orchestra.run.smithi089.stdout:2730 write 0x596c46f thru 0x59718d6 (0x5468 bytes) 2024-09-21T15:49:38.198 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:38.198 INFO:teuthology.orchestra.run.smithi089.stdout:2732 read 0x34a5d1e thru 0x34a6973 (0xc56 bytes) 2024-09-21T15:49:38.199 INFO:teuthology.orchestra.run.smithi089.stdout:2733 write 0xf72bfe thru 0xf778ab (0x4cae bytes) 2024-09-21T15:49:38.200 INFO:teuthology.orchestra.run.smithi089.stdout:2734 read 0x3e93a4 thru 0x3f3a6e (0xa6cb bytes) 2024-09-21T15:49:38.201 INFO:teuthology.orchestra.run.smithi089.stdout:2735 read 0x525521c thru 0x52629a1 (0xd786 bytes) 2024-09-21T15:49:38.201 INFO:teuthology.orchestra.run.smithi089.stdout:2736 write 0x2d467f0 thru 0x2d495c4 (0x2dd5 bytes) 2024-09-21T15:49:38.210 INFO:teuthology.orchestra.run.smithi089.stdout:2737 write 0x7663a72 thru 0x766660c (0x2b9b bytes) 2024-09-21T15:49:38.212 INFO:teuthology.orchestra.run.smithi089.stdout:2738 read 0x5c6a061 thru 0x5c6c7d4 (0x2774 bytes) 2024-09-21T15:49:38.212 INFO:teuthology.orchestra.run.smithi089.stdout:2740 read 0x3ab4f5e thru 0x3ac01e2 (0xb285 bytes) 2024-09-21T15:49:38.213 INFO:teuthology.orchestra.run.smithi089.stdout:2741 read 0x353b47d thru 0x3545c88 (0xa80c bytes) 2024-09-21T15:49:38.213 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:38.212+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:38.214 INFO:teuthology.orchestra.run.smithi089.stdout:2742 read 0x1c537f7 thru 0x1c5c586 (0x8d90 bytes) 2024-09-21T15:49:38.214 INFO:teuthology.orchestra.run.smithi089.stdout:2743 read 0x384d3f thru 0x38e677 (0x9939 bytes) 2024-09-21T15:49:38.214 INFO:teuthology.orchestra.run.smithi089.stdout:2744 write 0x2251a5b thru 0x2252832 (0xdd8 bytes) 2024-09-21T15:49:38.229 INFO:teuthology.orchestra.run.smithi089.stdout:2745 writesame 0xcd45103 thru 0xcd4ca8b data_size 0x3(0x7989 bytes) 2024-09-21T15:49:38.231 INFO:teuthology.orchestra.run.smithi089.stdout:2746 punch from 0x492b505 to 0x4932927, (0x7422 bytes) 2024-09-21T15:49:38.231 INFO:teuthology.orchestra.run.smithi089.stdout:2747 read 0xadaa22 thru 0xadd242 (0x2821 bytes) 2024-09-21T15:49:38.231 INFO:teuthology.orchestra.run.smithi089.stdout:2748 write 0xa398686 thru 0xa39c739 (0x40b4 bytes) 2024-09-21T15:49:38.233 INFO:teuthology.orchestra.run.smithi089.stdout:2749 read 0x3475e81 thru 0x3481ada (0xbc5a bytes) 2024-09-21T15:49:38.236 INFO:teuthology.orchestra.run.smithi089.stdout:2750 trunc from 0xcd4ca8c to 0xa4af720 2024-09-21T15:49:38.247 INFO:teuthology.orchestra.run.smithi089.stdout:2751 read 0x1b3a089 thru 0x1b3f02d (0x4fa5 bytes) 2024-09-21T15:49:38.248 INFO:teuthology.orchestra.run.smithi089.stdout:2752 trunc from 0xa4af720 to 0x9ec7ece 2024-09-21T15:49:38.258 INFO:teuthology.orchestra.run.smithi089.stdout:2753 writesame 0x2901de8 thru 0x2903a7d data_size 0x1(0x1c96 bytes) 2024-09-21T15:49:38.262 INFO:teuthology.orchestra.run.smithi089.stdout:2754 write 0x5cad3cc thru 0x5cb1967 (0x459c bytes) 2024-09-21T15:49:38.263 INFO:teuthology.orchestra.run.smithi089.stdout:2755 writesame 0x3b9e65c thru 0x3b9f383 data_size 0x4(0xd28 bytes) 2024-09-21T15:49:38.274 INFO:teuthology.orchestra.run.smithi089.stdout:2756 trunc from 0x9ec7ece to 0x1f6abd8 2024-09-21T15:49:38.290 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:38.294 INFO:teuthology.orchestra.run.smithi089.stdout:2759 trunc from 0x1f6abd8 to 0x4b4d413 2024-09-21T15:49:38.296 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:38.296 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:38.303 INFO:teuthology.orchestra.run.smithi089.stdout:2760 read 0x3a7dc9 thru 0x3b7244 (0xf47c bytes) 2024-09-21T15:49:38.304 INFO:teuthology.orchestra.run.smithi089.stdout:2762 write 0x2fc0224 thru 0x2fc7202 (0x6fdf bytes) 2024-09-21T15:49:38.305 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:49:38.306 INFO:teuthology.orchestra.run.smithi089.stdout:2763 writesame 0x2d0ca88 thru 0x2d0d634 data_size 0x7(0xbad bytes) 2024-09-21T15:49:38.308 INFO:teuthology.orchestra.run.smithi089.stdout:2764 punch from 0x2bf0c11 to 0x2bfd3d4, (0xc7c3 bytes) 2024-09-21T15:49:38.314 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:38.313+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:38.319 INFO:teuthology.orchestra.run.smithi089.stdout:2766 writesame 0x585d222 thru 0x585f0e3 data_size 0x1f(0x1ec2 bytes) 2024-09-21T15:49:38.320 INFO:teuthology.orchestra.run.smithi089.stdout:2767 read 0x4f74184 thru 0x4f7e02e (0x9eab bytes) 2024-09-21T15:49:38.321 INFO:teuthology.orchestra.run.smithi089.stdout:2768 write 0xe7e983 thru 0xe80664 (0x1ce2 bytes) 2024-09-21T15:49:38.322 INFO:teuthology.orchestra.run.smithi089.stdout:2769 punch from 0x14d654f to 0x14e60ac, (0xfb5d bytes) 2024-09-21T15:49:38.322 INFO:teuthology.orchestra.run.smithi089.stdout:2770 read 0x15a2fdc thru 0x15acf88 (0x9fad bytes) 2024-09-21T15:49:38.322 INFO:teuthology.orchestra.run.smithi089.stdout:2771 clone 14 order 19 su 16384 sc 5 2024-09-21T15:49:38.375 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T15:49:38.375 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T15:49:38.375 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-21T15:49:38.375 INFO:tasks.thrashosds.thrasher:Setting osd 5 primary_affinity to 0.292618 2024-09-21T15:49:38.375 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 0.29261796272996876 2024-09-21T15:49:38.410 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:38.410 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:38.415 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:38.414+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:38.419 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:49:38.516 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:38.515+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:38.562 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:38.562 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:38.571 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:49:38.617 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:38.615+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:38.714 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:38.714 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:38.718 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:38.716+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:38.722 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:49:38.819 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:38.818+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:38.825 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:38.825 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:38.833 INFO:teuthology.orchestra.run.smithi089.stderr:set osd.5 primary-affinity to 0.292618 (8191772) 2024-09-21T15:49:38.836 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:49:38.843 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T15:49:38.929 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:38.928+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:38.940 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:38.940 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:38.949 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:49:39.033 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:39.032+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:39.092 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:39.093 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:39.102 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:49:39.143 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:39.142+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:39.223 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:39.221+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:39.245 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:39.245 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:39.254 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:49:39.339 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:39.338+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:39.397 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:39.398 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:39.407 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:49:39.447 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:39.446+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:39.526 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:39.525+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:39.548 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:39.548 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:39.556 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:49:39.627 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:39.626+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:39.659 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:39.659 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:39.667 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:49:39.728 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:39.727+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:39.809 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:39.809 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:39.818 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:49:39.820 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:49:39.831 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is unset 2024-09-21T15:49:39.831 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:39.830+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:39.892 INFO:teuthology.orchestra.run.smithi089.stdout:truncating image image_client.0-clone13 from 0x585f0e4 (overlap 0x1f6abd8) to 0x0 2024-09-21T15:49:39.929 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:39.928+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:39.936 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:39.936 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:39.946 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:49:39.947 INFO:teuthology.orchestra.run.smithi089.stdout:checking clone #12, image image_client.0-clone12 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent13 2024-09-21T15:49:39.951 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-09-21T15:49:39.951 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:49:39.971 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:49:39.971 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:49:39.988 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:49:39.988 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:49:40.004 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:49:40.004 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:49:40.021 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:49:40.021 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:49:40.029 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:40.028+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:40.039 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:49:40.039 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:49:40.053 INFO:teuthology.orchestra.run.smithi089.stdout:2772 read 0x1d750ed thru 0x1d83b40 (0xea54 bytes) 2024-09-21T15:49:40.054 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:40.054 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:40.054 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:49:40.054 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:40.054 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:49:40.055 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:49:40.055 INFO:teuthology.orchestra.run.smithi089.stdout:2773 writesame 0x17364ac thru 0x1738608 data_size 0x3(0x215d bytes) 2024-09-21T15:49:40.065 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:49:40.073 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:49:40.073 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:49:40.084 INFO:teuthology.orchestra.run.smithi089.stdout:2774 write 0xcdff7ba thru 0xce06b13 (0x735a bytes) 2024-09-21T15:49:40.099 INFO:teuthology.orchestra.run.smithi089.stdout:2775 trunc from 0xce06b14 to 0x66553c0 2024-09-21T15:49:40.119 INFO:teuthology.orchestra.run.smithi089.stdout:2777 write 0x9b15dd9 thru 0x9b246bf (0xe8e7 bytes) 2024-09-21T15:49:40.129 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:40.129 INFO:teuthology.orchestra.run.smithi089.stdout:2779 clone 15 order 18 su 8192 sc 10 2024-09-21T15:49:40.149 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:40.148+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:40.210 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:40.210 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:40.210 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:49:40.210 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:49:40.210 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:40.218 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:49:40.231 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:40.230+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:40.331 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:40.331+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:40.361 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:40.361 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:49:40.361 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:49:40.361 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:49:40.362 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.362 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26224 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 2326528~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:40.362 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.504487+0000", 2024-09-21T15:49:40.362 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 13.855225852, 2024-09-21T15:49:40.362 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032819441999999997, 2024-09-21T15:49:40.362 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:40.362 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:40.362 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:40.362 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:40.362 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:40.362 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26224 2024-09-21T15:49:40.362 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.363 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:40.363 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.363 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:40.363 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504487+0000", 2024-09-21T15:49:40.363 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.363 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.363 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.363 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:40.363 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504487+0000", 2024-09-21T15:49:40.363 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.363 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.363 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.363 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:40.363 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504487+0000", 2024-09-21T15:49:40.364 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:40.364 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.364 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.364 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:40.364 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504490+0000", 2024-09-21T15:49:40.364 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6049999999999998e-06 2024-09-21T15:49:40.364 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.364 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.364 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:40.364 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504493+0000", 2024-09-21T15:49:40.364 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1200000000000002e-06 2024-09-21T15:49:40.364 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.364 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.365 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:40.365 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504496+0000", 2024-09-21T15:49:40.365 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.977e-06 2024-09-21T15:49:40.365 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.365 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.365 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:40.365 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537221+0000", 2024-09-21T15:49:40.365 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032724529000000002 2024-09-21T15:49:40.365 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.365 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.365 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:40.365 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537259+0000", 2024-09-21T15:49:40.365 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7923999999999998e-05 2024-09-21T15:49:40.365 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.365 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.366 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:40.366 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537306+0000", 2024-09-21T15:49:40.366 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7512999999999997e-05 2024-09-21T15:49:40.366 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.366 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:40.366 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.366 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.366 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.366 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26225 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 2383872~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:40.366 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.504498+0000", 2024-09-21T15:49:40.366 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 13.855214283, 2024-09-21T15:49:40.366 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.03291703, 2024-09-21T15:49:40.366 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:40.366 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:40.366 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:40.367 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:40.367 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:40.367 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26225 2024-09-21T15:49:40.367 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.367 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:40.367 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.367 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:40.367 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504498+0000", 2024-09-21T15:49:40.367 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.367 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.367 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.367 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:40.367 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504498+0000", 2024-09-21T15:49:40.367 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.367 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.368 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.368 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:40.368 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504498+0000", 2024-09-21T15:49:40.368 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:40.368 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.368 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.368 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:40.368 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504499+0000", 2024-09-21T15:49:40.368 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0070000000000001e-06 2024-09-21T15:49:40.368 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.368 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.368 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:40.368 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504502+0000", 2024-09-21T15:49:40.368 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8119999999999999e-06 2024-09-21T15:49:40.368 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.369 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.369 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:40.369 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504504+0000", 2024-09-21T15:49:40.369 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4380000000000002e-06 2024-09-21T15:49:40.369 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.369 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.369 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:40.369 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537328+0000", 2024-09-21T15:49:40.369 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032823659999999998 2024-09-21T15:49:40.369 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.369 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.369 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:40.369 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537366+0000", 2024-09-21T15:49:40.369 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8083000000000001e-05 2024-09-21T15:49:40.369 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.369 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.370 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:40.370 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537415+0000", 2024-09-21T15:49:40.370 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9233000000000001e-05 2024-09-21T15:49:40.370 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.370 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:40.370 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.370 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.370 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.370 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26226 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 2441216~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:40.370 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.504506+0000", 2024-09-21T15:49:40.370 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 13.855206168, 2024-09-21T15:49:40.370 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033016726000000003, 2024-09-21T15:49:40.370 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:40.370 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:40.370 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:40.371 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:40.371 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:40.371 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26226 2024-09-21T15:49:40.371 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.371 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:40.371 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.371 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:40.371 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504506+0000", 2024-09-21T15:49:40.371 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.371 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.371 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.371 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:40.371 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504506+0000", 2024-09-21T15:49:40.371 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.372 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.372 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.372 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:40.372 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504506+0000", 2024-09-21T15:49:40.372 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:40.372 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.372 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.372 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:40.372 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504507+0000", 2024-09-21T15:49:40.372 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9699999999999994e-07 2024-09-21T15:49:40.372 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.372 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.372 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:40.372 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504510+0000", 2024-09-21T15:49:40.372 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.773e-06 2024-09-21T15:49:40.373 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.373 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.373 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:40.373 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504514+0000", 2024-09-21T15:49:40.373 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1150000000000004e-06 2024-09-21T15:49:40.373 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.373 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.373 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:40.373 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537437+0000", 2024-09-21T15:49:40.373 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032923249000000002 2024-09-21T15:49:40.373 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.373 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.373 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:40.373 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537476+0000", 2024-09-21T15:49:40.373 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8679e-05 2024-09-21T15:49:40.373 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.374 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.374 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:40.374 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537523+0000", 2024-09-21T15:49:40.374 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7119000000000001e-05 2024-09-21T15:49:40.374 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.374 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:40.374 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.374 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.374 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.374 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26227 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 2498560~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:40.374 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.504516+0000", 2024-09-21T15:49:40.374 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 13.855196567, 2024-09-21T15:49:40.374 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033117681000000003, 2024-09-21T15:49:40.374 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:40.375 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:40.375 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:40.375 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:40.375 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:40.375 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26227 2024-09-21T15:49:40.375 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.375 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:40.375 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.375 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:40.375 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504516+0000", 2024-09-21T15:49:40.375 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.375 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.375 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.376 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:40.376 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504516+0000", 2024-09-21T15:49:40.376 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.376 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.376 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.376 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:40.376 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504516+0000", 2024-09-21T15:49:40.376 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:40.376 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.376 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.376 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:40.376 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504517+0000", 2024-09-21T15:49:40.376 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0610000000000001e-06 2024-09-21T15:49:40.376 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.376 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.376 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:40.376 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504520+0000", 2024-09-21T15:49:40.377 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.83e-06 2024-09-21T15:49:40.377 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.377 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.377 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:40.377 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504522+0000", 2024-09-21T15:49:40.377 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.497e-06 2024-09-21T15:49:40.377 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.377 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.377 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:40.377 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537544+0000", 2024-09-21T15:49:40.377 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033021948000000002 2024-09-21T15:49:40.377 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.377 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.377 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:40.377 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537585+0000", 2024-09-21T15:49:40.377 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1094999999999999e-05 2024-09-21T15:49:40.378 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.378 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.378 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:40.378 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537634+0000", 2024-09-21T15:49:40.378 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8449000000000003e-05 2024-09-21T15:49:40.378 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.378 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:40.378 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.378 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.378 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.378 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26228 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 2555904~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:40.378 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.504524+0000", 2024-09-21T15:49:40.378 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 13.855188469, 2024-09-21T15:49:40.378 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.03321735, 2024-09-21T15:49:40.379 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:40.379 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:40.379 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:40.379 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:40.379 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:40.379 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26228 2024-09-21T15:49:40.379 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.379 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:40.379 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.379 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:40.379 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504524+0000", 2024-09-21T15:49:40.379 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.379 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.379 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.380 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:40.380 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504524+0000", 2024-09-21T15:49:40.380 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.380 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.380 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.380 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:40.380 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504524+0000", 2024-09-21T15:49:40.380 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:40.380 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.380 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.380 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:40.380 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504525+0000", 2024-09-21T15:49:40.380 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0419999999999999e-06 2024-09-21T15:49:40.380 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.380 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.380 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:40.381 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504528+0000", 2024-09-21T15:49:40.381 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7530000000000002e-06 2024-09-21T15:49:40.381 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.381 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.381 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:40.381 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504530+0000", 2024-09-21T15:49:40.381 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4530000000000001e-06 2024-09-21T15:49:40.381 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.381 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.381 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:40.381 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537656+0000", 2024-09-21T15:49:40.381 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033126023999999997 2024-09-21T15:49:40.381 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.381 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.381 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:40.381 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537694+0000", 2024-09-21T15:49:40.382 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7799000000000002e-05 2024-09-21T15:49:40.382 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.382 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.382 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:40.382 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537741+0000", 2024-09-21T15:49:40.382 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7484999999999997e-05 2024-09-21T15:49:40.382 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.382 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:40.382 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.382 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.382 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.382 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26229 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 2613248~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:40.382 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.504532+0000", 2024-09-21T15:49:40.382 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 13.85518059, 2024-09-21T15:49:40.382 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033315897999999997, 2024-09-21T15:49:40.382 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:40.383 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:40.383 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:40.383 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:40.383 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:40.383 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26229 2024-09-21T15:49:40.383 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.383 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:40.383 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.383 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:40.383 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504532+0000", 2024-09-21T15:49:40.383 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.383 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.383 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.383 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:40.383 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504532+0000", 2024-09-21T15:49:40.383 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.384 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.384 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.384 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:40.384 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504532+0000", 2024-09-21T15:49:40.384 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:40.384 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.384 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.384 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:40.384 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504533+0000", 2024-09-21T15:49:40.384 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0610000000000001e-06 2024-09-21T15:49:40.384 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.384 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.384 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:40.385 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504536+0000", 2024-09-21T15:49:40.385 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.309e-06 2024-09-21T15:49:40.385 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.385 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.385 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:40.385 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504539+0000", 2024-09-21T15:49:40.385 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.435e-06 2024-09-21T15:49:40.385 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.385 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.385 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:40.385 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537762+0000", 2024-09-21T15:49:40.385 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033223813999999997 2024-09-21T15:49:40.385 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.385 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.386 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:40.386 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537801+0000", 2024-09-21T15:49:40.386 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8136999999999998e-05 2024-09-21T15:49:40.386 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.386 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.386 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:40.386 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537848+0000", 2024-09-21T15:49:40.386 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7345999999999997e-05 2024-09-21T15:49:40.386 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.386 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:40.386 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.386 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.386 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.386 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26230 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 2670592~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:40.386 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.504606+0000", 2024-09-21T15:49:40.387 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 13.855106431999999, 2024-09-21T15:49:40.387 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033348801999999997, 2024-09-21T15:49:40.387 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:40.387 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:40.387 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:40.387 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:40.387 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:40.387 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26230 2024-09-21T15:49:40.387 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.387 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:40.387 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.387 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:40.387 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504606+0000", 2024-09-21T15:49:40.387 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.387 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.388 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.388 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:40.388 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504606+0000", 2024-09-21T15:49:40.388 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.388 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.388 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.388 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:40.388 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504541+0000", 2024-09-21T15:49:40.388 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999352 2024-09-21T15:49:40.388 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.388 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.388 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:40.388 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504608+0000", 2024-09-21T15:49:40.388 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6427000000000006e-05 2024-09-21T15:49:40.389 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.389 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.389 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:40.389 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504612+0000", 2024-09-21T15:49:40.389 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1799999999999998e-06 2024-09-21T15:49:40.389 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.389 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.389 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:40.389 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504615+0000", 2024-09-21T15:49:40.389 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0400000000000001e-06 2024-09-21T15:49:40.389 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.389 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.389 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:40.389 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537870+0000", 2024-09-21T15:49:40.389 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033254745000000002 2024-09-21T15:49:40.390 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.390 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.390 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:40.390 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537908+0000", 2024-09-21T15:49:40.390 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7882000000000001e-05 2024-09-21T15:49:40.390 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.390 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.390 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:40.390 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537955+0000", 2024-09-21T15:49:40.390 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7361999999999998e-05 2024-09-21T15:49:40.390 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.390 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:40.390 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.390 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.391 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.391 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26239 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 3186688~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:40.391 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.506097+0000", 2024-09-21T15:49:40.391 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 13.853615725999999, 2024-09-21T15:49:40.391 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032843331000000003, 2024-09-21T15:49:40.391 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:40.391 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:40.391 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:40.391 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:40.391 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:40.391 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26239 2024-09-21T15:49:40.391 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.391 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:40.391 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.391 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:40.392 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506097+0000", 2024-09-21T15:49:40.392 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.392 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.392 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.392 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:40.392 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506097+0000", 2024-09-21T15:49:40.392 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.392 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.392 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.392 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:40.392 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506097+0000", 2024-09-21T15:49:40.392 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:40.392 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.392 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.392 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:40.392 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506098+0000", 2024-09-21T15:49:40.393 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.079e-06 2024-09-21T15:49:40.393 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.393 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.393 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:40.393 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506101+0000", 2024-09-21T15:49:40.393 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8499999999999998e-06 2024-09-21T15:49:40.393 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.393 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.393 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:40.393 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506103+0000", 2024-09-21T15:49:40.393 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5770000000000001e-06 2024-09-21T15:49:40.393 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.393 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.393 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:40.393 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.538854+0000", 2024-09-21T15:49:40.393 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032750835999999998 2024-09-21T15:49:40.393 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.394 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.394 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:40.394 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.538892+0000", 2024-09-21T15:49:40.394 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7904000000000002e-05 2024-09-21T15:49:40.394 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.394 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.394 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:40.394 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.538940+0000", 2024-09-21T15:49:40.394 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8303999999999997e-05 2024-09-21T15:49:40.394 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.394 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:40.394 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.394 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.394 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.395 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26240 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 3244032~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:40.395 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.506106+0000", 2024-09-21T15:49:40.395 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 13.853606594, 2024-09-21T15:49:40.395 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032955696, 2024-09-21T15:49:40.395 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:40.395 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:40.395 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:40.395 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:40.395 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:40.395 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26240 2024-09-21T15:49:40.395 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.395 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:40.395 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.395 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:40.395 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506106+0000", 2024-09-21T15:49:40.395 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.396 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.396 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.396 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:40.396 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506106+0000", 2024-09-21T15:49:40.396 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.396 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.396 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.396 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:40.396 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506106+0000", 2024-09-21T15:49:40.396 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:40.396 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.396 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.396 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:40.396 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506107+0000", 2024-09-21T15:49:40.396 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1179999999999999e-06 2024-09-21T15:49:40.397 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.397 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.397 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:40.397 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506110+0000", 2024-09-21T15:49:40.397 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0259999999999999e-06 2024-09-21T15:49:40.397 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.397 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.397 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:40.397 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506112+0000", 2024-09-21T15:49:40.397 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5890000000000001e-06 2024-09-21T15:49:40.397 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.397 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.397 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:40.397 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.538962+0000", 2024-09-21T15:49:40.397 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032849861000000001 2024-09-21T15:49:40.398 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.398 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.398 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:40.398 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539003+0000", 2024-09-21T15:49:40.398 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0756000000000001e-05 2024-09-21T15:49:40.398 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.398 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.398 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:40.398 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539062+0000", 2024-09-21T15:49:40.398 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8597999999999997e-05 2024-09-21T15:49:40.398 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.398 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:40.398 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.398 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.399 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.399 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26241 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 3301376~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:40.399 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.506128+0000", 2024-09-21T15:49:40.399 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 13.853584428, 2024-09-21T15:49:40.399 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033043453, 2024-09-21T15:49:40.399 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:40.399 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:40.399 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:40.399 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:40.399 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:40.399 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26241 2024-09-21T15:49:40.399 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.399 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:40.399 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.399 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:40.400 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506128+0000", 2024-09-21T15:49:40.400 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.400 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.400 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.400 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:40.400 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506128+0000", 2024-09-21T15:49:40.400 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.400 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.400 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.400 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:40.400 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506128+0000", 2024-09-21T15:49:40.400 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:40.400 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.400 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.400 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:40.401 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506129+0000", 2024-09-21T15:49:40.401 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7290000000000001e-06 2024-09-21T15:49:40.401 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.401 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.401 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:40.401 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506132+0000", 2024-09-21T15:49:40.401 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9919999999999999e-06 2024-09-21T15:49:40.401 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.401 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.401 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:40.401 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506135+0000", 2024-09-21T15:49:40.401 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4090000000000001e-06 2024-09-21T15:49:40.401 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.401 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.402 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:40.402 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539084+0000", 2024-09-21T15:49:40.402 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032949231000000002 2024-09-21T15:49:40.402 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.402 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.402 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:40.402 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539124+0000", 2024-09-21T15:49:40.402 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9421000000000001e-05 2024-09-21T15:49:40.402 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.402 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.402 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:40.402 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539172+0000", 2024-09-21T15:49:40.402 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8068000000000001e-05 2024-09-21T15:49:40.403 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.403 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:40.403 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.403 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.403 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.403 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26242 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 3358720~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:40.403 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.506137+0000", 2024-09-21T15:49:40.403 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 13.853575493999999, 2024-09-21T15:49:40.403 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033142360000000003, 2024-09-21T15:49:40.403 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:40.403 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:40.403 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:40.403 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:40.403 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:40.404 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26242 2024-09-21T15:49:40.404 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.404 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:40.404 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.404 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:40.404 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506137+0000", 2024-09-21T15:49:40.404 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.404 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.404 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.404 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:40.404 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506137+0000", 2024-09-21T15:49:40.404 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.404 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.404 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.405 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:40.405 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506137+0000", 2024-09-21T15:49:40.405 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:40.405 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.405 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.405 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:40.405 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506138+0000", 2024-09-21T15:49:40.405 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2330000000000001e-06 2024-09-21T15:49:40.405 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.405 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.405 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:40.405 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506141+0000", 2024-09-21T15:49:40.405 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7999999999999999e-06 2024-09-21T15:49:40.406 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.406 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.406 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:40.406 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506143+0000", 2024-09-21T15:49:40.406 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5919999999999999e-06 2024-09-21T15:49:40.406 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.406 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.406 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:40.406 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539193+0000", 2024-09-21T15:49:40.406 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033049229999999999 2024-09-21T15:49:40.406 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.406 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.406 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:40.407 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539232+0000", 2024-09-21T15:49:40.407 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9743999999999998e-05 2024-09-21T15:49:40.407 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.407 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.407 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:40.407 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539279+0000", 2024-09-21T15:49:40.407 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7039999999999997e-05 2024-09-21T15:49:40.407 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.407 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:40.407 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.407 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.407 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.407 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26243 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 3416064~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:40.407 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.506145+0000", 2024-09-21T15:49:40.408 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 13.853567208999999, 2024-09-21T15:49:40.408 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033240933, 2024-09-21T15:49:40.408 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:40.408 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:40.408 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:40.408 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:40.408 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:40.408 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26243 2024-09-21T15:49:40.408 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.408 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:40.408 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.408 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:40.408 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506145+0000", 2024-09-21T15:49:40.408 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.408 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.409 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.409 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:40.409 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506145+0000", 2024-09-21T15:49:40.409 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.409 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.409 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.409 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:40.409 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506145+0000", 2024-09-21T15:49:40.409 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:40.409 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.409 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.409 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:40.409 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506146+0000", 2024-09-21T15:49:40.409 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0279999999999999e-06 2024-09-21T15:49:40.409 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.410 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.410 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:40.410 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506149+0000", 2024-09-21T15:49:40.410 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2449999999999998e-06 2024-09-21T15:49:40.410 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.410 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.410 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:40.410 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506152+0000", 2024-09-21T15:49:40.410 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5050000000000002e-06 2024-09-21T15:49:40.410 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.410 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.410 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:40.410 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539301+0000", 2024-09-21T15:49:40.410 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033149093999999997 2024-09-21T15:49:40.411 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.411 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.411 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:40.411 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539339+0000", 2024-09-21T15:49:40.411 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7685999999999998e-05 2024-09-21T15:49:40.411 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.411 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.411 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:40.411 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539386+0000", 2024-09-21T15:49:40.411 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7602000000000003e-05 2024-09-21T15:49:40.411 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.411 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:40.411 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.411 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.411 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.411 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26258 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 4276224~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:40.412 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.508167+0000", 2024-09-21T15:49:40.412 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 13.851545291000001, 2024-09-21T15:49:40.412 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032934857999999997, 2024-09-21T15:49:40.412 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:40.412 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:40.412 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:40.412 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:40.412 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:40.412 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26258 2024-09-21T15:49:40.412 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.412 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:40.412 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.412 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:40.412 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508167+0000", 2024-09-21T15:49:40.412 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.412 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.413 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.413 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:40.413 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508167+0000", 2024-09-21T15:49:40.413 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.413 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.413 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.413 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:40.413 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508167+0000", 2024-09-21T15:49:40.413 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:40.413 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.413 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.413 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:40.413 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508168+0000", 2024-09-21T15:49:40.413 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.088e-06 2024-09-21T15:49:40.413 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.413 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.414 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:40.414 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508171+0000", 2024-09-21T15:49:40.414 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7080000000000002e-06 2024-09-21T15:49:40.414 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.414 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.414 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:40.414 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508173+0000", 2024-09-21T15:49:40.414 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4030000000000001e-06 2024-09-21T15:49:40.414 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.414 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.414 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:40.414 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.540936+0000", 2024-09-21T15:49:40.414 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032762279999999998 2024-09-21T15:49:40.414 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.415 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.415 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:40.415 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.541007+0000", 2024-09-21T15:49:40.415 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1335999999999996e-05 2024-09-21T15:49:40.415 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.415 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.415 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:40.415 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.541102+0000", 2024-09-21T15:49:40.415 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5259000000000004e-05 2024-09-21T15:49:40.415 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.415 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:40.415 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.415 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.415 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.415 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26259 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 4333568~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:40.415 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.508175+0000", 2024-09-21T15:49:40.416 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 13.851537349000001, 2024-09-21T15:49:40.416 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033044845000000003, 2024-09-21T15:49:40.416 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:40.416 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:40.416 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:40.416 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:40.416 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:40.416 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26259 2024-09-21T15:49:40.416 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.416 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:40.416 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.416 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:40.416 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508175+0000", 2024-09-21T15:49:40.416 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.416 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.416 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.417 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:40.417 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508175+0000", 2024-09-21T15:49:40.417 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.417 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.417 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.417 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:40.417 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508175+0000", 2024-09-21T15:49:40.417 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:40.417 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.417 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.417 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:40.417 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508176+0000", 2024-09-21T15:49:40.417 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1039999999999999e-06 2024-09-21T15:49:40.417 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.417 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.417 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:40.417 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508179+0000", 2024-09-21T15:49:40.418 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.689e-06 2024-09-21T15:49:40.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.418 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:40.418 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508181+0000", 2024-09-21T15:49:40.418 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5610000000000001e-06 2024-09-21T15:49:40.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.418 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:40.418 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.541126+0000", 2024-09-21T15:49:40.418 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032944284999999997 2024-09-21T15:49:40.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.418 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:40.418 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.541168+0000", 2024-09-21T15:49:40.419 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2565000000000003e-05 2024-09-21T15:49:40.419 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.419 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.419 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:40.419 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.541220+0000", 2024-09-21T15:49:40.419 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1863000000000001e-05 2024-09-21T15:49:40.419 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.419 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:40.419 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.419 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.419 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.419 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26260 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 4390912~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:40.419 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.508183+0000", 2024-09-21T15:49:40.419 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 13.851529234000001, 2024-09-21T15:49:40.419 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033150304999999998, 2024-09-21T15:49:40.419 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:40.420 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:40.420 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:40.420 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:40.420 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:40.420 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26260 2024-09-21T15:49:40.420 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.420 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:40.420 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.420 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:40.420 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508183+0000", 2024-09-21T15:49:40.420 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.420 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.420 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.420 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:40.420 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508183+0000", 2024-09-21T15:49:40.420 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.421 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.421 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.421 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:40.421 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508183+0000", 2024-09-21T15:49:40.421 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:40.421 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.421 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.421 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:40.421 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508184+0000", 2024-09-21T15:49:40.421 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0580000000000001e-06 2024-09-21T15:49:40.421 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.421 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.421 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:40.421 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508187+0000", 2024-09-21T15:49:40.421 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7209999999999999e-06 2024-09-21T15:49:40.421 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.421 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.422 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:40.422 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508189+0000", 2024-09-21T15:49:40.422 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5239999999999999e-06 2024-09-21T15:49:40.422 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.422 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.422 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:40.422 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.541243+0000", 2024-09-21T15:49:40.422 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033053658 2024-09-21T15:49:40.422 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.422 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.422 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:40.422 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.541284+0000", 2024-09-21T15:49:40.422 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0834000000000002e-05 2024-09-21T15:49:40.422 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.422 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.423 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:40.423 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.541334+0000", 2024-09-21T15:49:40.423 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9724e-05 2024-09-21T15:49:40.423 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.423 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:40.423 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.423 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.423 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.423 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26275 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 5251072~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:40.423 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.510151+0000", 2024-09-21T15:49:40.423 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 13.849561455, 2024-09-21T15:49:40.423 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032836153999999999, 2024-09-21T15:49:40.423 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:40.423 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:40.423 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:40.423 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:40.424 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:40.424 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26275 2024-09-21T15:49:40.424 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.424 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:40.424 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.424 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:40.424 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510151+0000", 2024-09-21T15:49:40.424 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.424 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.424 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.424 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:40.424 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510151+0000", 2024-09-21T15:49:40.424 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.424 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.424 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.424 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:40.425 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510151+0000", 2024-09-21T15:49:40.425 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:40.425 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.425 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.425 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:40.425 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510152+0000", 2024-09-21T15:49:40.425 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1990000000000001e-06 2024-09-21T15:49:40.425 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.425 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.425 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:40.425 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510155+0000", 2024-09-21T15:49:40.425 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7089999999999998e-06 2024-09-21T15:49:40.425 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.425 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.425 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:40.425 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510157+0000", 2024-09-21T15:49:40.426 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5189999999999999e-06 2024-09-21T15:49:40.426 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.426 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.426 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:40.426 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.542902+0000", 2024-09-21T15:49:40.426 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032744599999999999 2024-09-21T15:49:40.426 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.426 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.426 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:40.426 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.542941+0000", 2024-09-21T15:49:40.426 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8761999999999999e-05 2024-09-21T15:49:40.426 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.426 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.426 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:40.426 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.542987+0000", 2024-09-21T15:49:40.427 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6665999999999997e-05 2024-09-21T15:49:40.427 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.427 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:40.427 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.427 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.427 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.427 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26276 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 5308416~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:40.427 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.510159+0000", 2024-09-21T15:49:40.427 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 13.849553491, 2024-09-21T15:49:40.427 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032943014999999999, 2024-09-21T15:49:40.427 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:40.427 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:40.427 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:40.427 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:40.428 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:40.428 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26276 2024-09-21T15:49:40.428 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.428 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:40.428 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.428 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:40.428 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510159+0000", 2024-09-21T15:49:40.428 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.428 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.428 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.428 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:40.428 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510159+0000", 2024-09-21T15:49:40.428 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.428 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.428 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.429 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:40.429 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510159+0000", 2024-09-21T15:49:40.429 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:40.429 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.429 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.429 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:40.429 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510160+0000", 2024-09-21T15:49:40.429 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.13e-06 2024-09-21T15:49:40.429 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.429 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.429 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:40.429 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510163+0000", 2024-09-21T15:49:40.429 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7489999999999999e-06 2024-09-21T15:49:40.429 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.429 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.429 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:40.430 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510165+0000", 2024-09-21T15:49:40.430 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4200000000000001e-06 2024-09-21T15:49:40.430 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.430 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.430 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:40.430 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.543013+0000", 2024-09-21T15:49:40.430 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032847736000000002 2024-09-21T15:49:40.430 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.430 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.430 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:40.430 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.543054+0000", 2024-09-21T15:49:40.430 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0725000000000001e-05 2024-09-21T15:49:40.430 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.430 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.431 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:40.431 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.543102+0000", 2024-09-21T15:49:40.431 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8471999999999999e-05 2024-09-21T15:49:40.431 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.431 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:40.431 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.431 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.431 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.431 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26277 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 5365760~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:40.431 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.510167+0000", 2024-09-21T15:49:40.431 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 13.84954536, 2024-09-21T15:49:40.431 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033113111000000001, 2024-09-21T15:49:40.431 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:40.431 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:40.432 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:40.432 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:40.432 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:40.432 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26277 2024-09-21T15:49:40.432 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.432 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:40.432 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.432 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:40.432 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510167+0000", 2024-09-21T15:49:40.432 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.432 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.432 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.432 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:40.432 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510167+0000", 2024-09-21T15:49:40.432 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.433 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.433 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.433 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:40.433 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510167+0000", 2024-09-21T15:49:40.433 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:40.433 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.433 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.433 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:40.433 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510168+0000", 2024-09-21T15:49:40.433 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.6899999999999996e-07 2024-09-21T15:49:40.433 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.433 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.433 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:40.433 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510171+0000", 2024-09-21T15:49:40.433 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7080000000000002e-06 2024-09-21T15:49:40.434 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.434 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.434 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:40.434 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510173+0000", 2024-09-21T15:49:40.434 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5050000000000002e-06 2024-09-21T15:49:40.434 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.434 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.434 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:40.434 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.543125+0000", 2024-09-21T15:49:40.434 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032951504 2024-09-21T15:49:40.434 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.434 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.434 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:40.434 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.543192+0000", 2024-09-21T15:49:40.435 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6938000000000002e-05 2024-09-21T15:49:40.435 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.435 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.435 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:40.435 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.543280+0000", 2024-09-21T15:49:40.435 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8702999999999994e-05 2024-09-21T15:49:40.435 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.435 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:40.435 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.435 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.435 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.435 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26292 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 6225920~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:40.435 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.512160+0000", 2024-09-21T15:49:40.435 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 13.847552545999999, 2024-09-21T15:49:40.436 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032838748000000001, 2024-09-21T15:49:40.436 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:40.436 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:40.436 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:40.436 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:40.436 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:40.436 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26292 2024-09-21T15:49:40.436 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.436 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:40.436 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.436 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:40.436 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512160+0000", 2024-09-21T15:49:40.436 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.436 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.436 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.437 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:40.437 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512160+0000", 2024-09-21T15:49:40.437 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.437 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.437 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.437 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:40.437 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512160+0000", 2024-09-21T15:49:40.437 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:40.437 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.437 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.437 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:40.437 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512161+0000", 2024-09-21T15:49:40.437 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.158e-06 2024-09-21T15:49:40.437 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.437 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.438 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:40.438 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512164+0000", 2024-09-21T15:49:40.438 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7429999999999998e-06 2024-09-21T15:49:40.438 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.438 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.438 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:40.438 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512166+0000", 2024-09-21T15:49:40.438 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5280000000000002e-06 2024-09-21T15:49:40.438 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.438 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.438 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:40.438 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.544912+0000", 2024-09-21T15:49:40.438 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032745962000000003 2024-09-21T15:49:40.439 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.439 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.439 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:40.439 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.544950+0000", 2024-09-21T15:49:40.439 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7657000000000002e-05 2024-09-21T15:49:40.439 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.439 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.439 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:40.439 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.544999+0000", 2024-09-21T15:49:40.439 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8912999999999997e-05 2024-09-21T15:49:40.439 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.439 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:40.439 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.439 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.440 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.440 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26293 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 6283264~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:40.440 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.512169+0000", 2024-09-21T15:49:40.440 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 13.847543397000001, 2024-09-21T15:49:40.440 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032937552000000002, 2024-09-21T15:49:40.440 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:40.440 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:40.440 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:40.440 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:40.440 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:40.440 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26293 2024-09-21T15:49:40.440 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.440 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:40.440 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.440 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:40.440 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512169+0000", 2024-09-21T15:49:40.441 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.441 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.441 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.441 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:40.441 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512169+0000", 2024-09-21T15:49:40.441 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.441 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.441 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.441 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:40.441 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512169+0000", 2024-09-21T15:49:40.441 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:40.441 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.441 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.442 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:40.442 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512170+0000", 2024-09-21T15:49:40.442 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1650000000000001e-06 2024-09-21T15:49:40.442 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.442 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.442 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:40.442 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512173+0000", 2024-09-21T15:49:40.442 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.773e-06 2024-09-21T15:49:40.442 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.442 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.442 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:40.442 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512175+0000", 2024-09-21T15:49:40.442 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4700000000000001e-06 2024-09-21T15:49:40.442 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.442 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.442 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:40.443 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.545018+0000", 2024-09-21T15:49:40.443 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032843043000000002 2024-09-21T15:49:40.443 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.443 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.443 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:40.443 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.545057+0000", 2024-09-21T15:49:40.443 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8862999999999997e-05 2024-09-21T15:49:40.443 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.443 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.443 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:40.443 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.545107+0000", 2024-09-21T15:49:40.443 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9455e-05 2024-09-21T15:49:40.443 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.444 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:40.444 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.444 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.444 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:40.444 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:40.445 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:49:40.446 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:40.444+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:40.546 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:40.544+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:40.588 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:40.588 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:40.588 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:49:40.588 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:40.596 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:49:40.647 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:40.646+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:40.738 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:40.738 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:40.738 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:49:40.738 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:49:40.738 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:40.746 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:49:40.747 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:40.747+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:40.848 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:40.847+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:40.850 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:40.850 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:49:40.850 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:49:40.850 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:49:40.851 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.851 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:65402 3.1d 3:bda7fcfc:::rbd_data.11336b2fdf4a.0000000000000000:c [read 4915200~0] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:40.851 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.279722+0000", 2024-09-21T15:49:40.851 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 3.5684983580000003, 2024-09-21T15:49:40.851 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045558766000000001, 2024-09-21T15:49:40.851 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:40.851 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:40.851 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:40.851 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:40.851 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:40.851 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 65402 2024-09-21T15:49:40.851 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.851 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:40.851 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.852 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:40.852 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279722+0000", 2024-09-21T15:49:40.852 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.852 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.852 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.852 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:40.852 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279722+0000", 2024-09-21T15:49:40.852 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.852 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.852 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.852 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:40.852 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279722+0000", 2024-09-21T15:49:40.852 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:40.852 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.852 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.852 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:40.853 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279724+0000", 2024-09-21T15:49:40.853 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3929999999999998e-06 2024-09-21T15:49:40.853 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.853 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.853 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:40.853 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279731+0000", 2024-09-21T15:49:40.853 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3269999999999997e-06 2024-09-21T15:49:40.853 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.853 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.853 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:40.853 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279736+0000", 2024-09-21T15:49:40.853 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.113e-06 2024-09-21T15:49:40.853 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.853 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.853 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:40.854 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.325136+0000", 2024-09-21T15:49:40.854 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045399912000000001 2024-09-21T15:49:40.854 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.854 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.854 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:40.854 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.325213+0000", 2024-09-21T15:49:40.854 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7546999999999995e-05 2024-09-21T15:49:40.854 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.854 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.854 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:40.854 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.325281+0000", 2024-09-21T15:49:40.854 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7931999999999999e-05 2024-09-21T15:49:40.854 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.854 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:40.855 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.855 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.855 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.855 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:65403 3.1d 3:bda7fcfc:::rbd_data.11336b2fdf4a.0000000000000000:c [read 4923392~0] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:40.855 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.279740+0000", 2024-09-21T15:49:40.855 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 3.5684804460000001, 2024-09-21T15:49:40.855 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045722756000000003, 2024-09-21T15:49:40.855 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:40.855 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:40.855 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:40.855 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:40.855 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:40.855 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 65403 2024-09-21T15:49:40.855 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.855 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:40.856 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.856 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:40.856 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279740+0000", 2024-09-21T15:49:40.856 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.856 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.856 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.856 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:40.856 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279740+0000", 2024-09-21T15:49:40.856 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.856 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.856 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.856 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:40.856 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279740+0000", 2024-09-21T15:49:40.856 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:40.856 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.856 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.857 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:40.857 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279742+0000", 2024-09-21T15:49:40.857 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1849999999999998e-06 2024-09-21T15:49:40.857 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.857 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.857 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:40.857 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279749+0000", 2024-09-21T15:49:40.857 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2450000000000001e-06 2024-09-21T15:49:40.857 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.857 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.857 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:40.857 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279754+0000", 2024-09-21T15:49:40.857 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8940000000000001e-06 2024-09-21T15:49:40.857 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.857 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.857 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:40.857 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.325316+0000", 2024-09-21T15:49:40.858 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045561938000000003 2024-09-21T15:49:40.858 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.858 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.858 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:40.858 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.325399+0000", 2024-09-21T15:49:40.858 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2647999999999999e-05 2024-09-21T15:49:40.858 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.858 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.858 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:40.858 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.325463+0000", 2024-09-21T15:49:40.858 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4247999999999999e-05 2024-09-21T15:49:40.858 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.858 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:40.858 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.859 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.859 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.859 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:65404 3.1d 3:bda7fcfc:::rbd_data.11336b2fdf4a.0000000000000000:c [read 4931584~0] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:40.859 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.279757+0000", 2024-09-21T15:49:40.859 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 3.5684636159999998, 2024-09-21T15:49:40.859 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045883348999999997, 2024-09-21T15:49:40.859 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:40.859 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:40.859 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:40.859 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:40.859 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:40.859 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 65404 2024-09-21T15:49:40.859 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.859 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:40.860 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.860 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:40.860 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279757+0000", 2024-09-21T15:49:40.860 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.860 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.860 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.860 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:40.860 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279757+0000", 2024-09-21T15:49:40.860 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.860 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.860 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.860 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:40.860 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279757+0000", 2024-09-21T15:49:40.861 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:40.861 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.861 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.861 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:40.861 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279759+0000", 2024-09-21T15:49:40.861 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.64e-06 2024-09-21T15:49:40.861 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.861 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.861 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:40.861 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279764+0000", 2024-09-21T15:49:40.861 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0030000000000002e-06 2024-09-21T15:49:40.861 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.861 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.861 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:40.861 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279766+0000", 2024-09-21T15:49:40.862 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6620000000000001e-06 2024-09-21T15:49:40.862 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.862 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.862 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:40.862 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.325489+0000", 2024-09-21T15:49:40.862 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045722721000000001 2024-09-21T15:49:40.862 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.862 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.862 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:40.862 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.325574+0000", 2024-09-21T15:49:40.862 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5109000000000001e-05 2024-09-21T15:49:40.862 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.862 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.862 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:40.863 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.325641+0000", 2024-09-21T15:49:40.863 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6475000000000003e-05 2024-09-21T15:49:40.863 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.863 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:40.863 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.863 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.863 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.863 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:65405 3.1d 3:bda7fcfc:::rbd_data.11336b2fdf4a.0000000000000000:c [read 4939776~0] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:40.863 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.279768+0000", 2024-09-21T15:49:40.863 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 3.5684523530000001, 2024-09-21T15:49:40.863 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046038737000000003, 2024-09-21T15:49:40.863 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:40.863 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:40.863 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:40.863 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:40.864 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:40.864 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 65405 2024-09-21T15:49:40.864 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.864 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:40.864 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.864 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:40.864 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279768+0000", 2024-09-21T15:49:40.864 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.864 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.864 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.864 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:40.864 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279768+0000", 2024-09-21T15:49:40.864 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.864 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.864 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.864 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:40.865 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279768+0000", 2024-09-21T15:49:40.865 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:40.865 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.865 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.865 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:40.865 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279769+0000", 2024-09-21T15:49:40.865 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.22e-06 2024-09-21T15:49:40.865 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.865 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.865 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:40.865 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279772+0000", 2024-09-21T15:49:40.865 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9349999999999999e-06 2024-09-21T15:49:40.865 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.865 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.865 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:40.865 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279775+0000", 2024-09-21T15:49:40.866 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4959999999999999e-06 2024-09-21T15:49:40.866 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.866 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.866 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:40.866 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.325668+0000", 2024-09-21T15:49:40.866 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045892779000000002 2024-09-21T15:49:40.866 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.866 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.866 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:40.866 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.325742+0000", 2024-09-21T15:49:40.866 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4092999999999997e-05 2024-09-21T15:49:40.866 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.866 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.866 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:40.866 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.325807+0000", 2024-09-21T15:49:40.867 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5419999999999994e-05 2024-09-21T15:49:40.867 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.867 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:40.867 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.867 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.867 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.867 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:65406 3.1d 3:bda7fcfc:::rbd_data.11336b2fdf4a.0000000000000000:c [read 4947968~0] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:40.867 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.279777+0000", 2024-09-21T15:49:40.867 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 3.5684437940000002, 2024-09-21T15:49:40.867 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046198388999999999, 2024-09-21T15:49:40.867 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:40.867 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:40.867 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:40.867 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:40.867 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:40.867 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 65406 2024-09-21T15:49:40.868 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.868 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:40.868 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.868 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:40.868 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279777+0000", 2024-09-21T15:49:40.868 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.868 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.868 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.868 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:40.868 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279777+0000", 2024-09-21T15:49:40.868 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.868 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.868 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.868 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:40.868 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279777+0000", 2024-09-21T15:49:40.868 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:40.869 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.869 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.869 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:40.869 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279778+0000", 2024-09-21T15:49:40.869 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.238e-06 2024-09-21T15:49:40.869 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.869 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.869 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:40.869 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279781+0000", 2024-09-21T15:49:40.869 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8679999999999999e-06 2024-09-21T15:49:40.869 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.869 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.869 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:40.869 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279783+0000", 2024-09-21T15:49:40.869 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4729999999999999e-06 2024-09-21T15:49:40.869 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.869 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.870 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:40.870 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.325834+0000", 2024-09-21T15:49:40.870 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046050328000000001 2024-09-21T15:49:40.870 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.870 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.870 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:40.870 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.325909+0000", 2024-09-21T15:49:40.870 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5463999999999994e-05 2024-09-21T15:49:40.870 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.870 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.870 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:40.870 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.325975+0000", 2024-09-21T15:49:40.870 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6229999999999994e-05 2024-09-21T15:49:40.870 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.870 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:40.870 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.871 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.871 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.871 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:65407 3.1d 3:bda7fcfc:::rbd_data.11336b2fdf4a.0000000000000000:c [read 4956160~0] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:40.871 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.279786+0000", 2024-09-21T15:49:40.871 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 3.5684352399999999, 2024-09-21T15:49:40.871 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046366821000000003, 2024-09-21T15:49:40.871 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:40.871 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:40.871 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:40.871 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:40.871 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:40.871 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 65407 2024-09-21T15:49:40.871 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.871 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:40.872 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.872 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:40.872 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279786+0000", 2024-09-21T15:49:40.872 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.872 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.872 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.872 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:40.872 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279786+0000", 2024-09-21T15:49:40.872 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.872 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.872 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.872 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:40.872 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279785+0000", 2024-09-21T15:49:40.873 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:40.873 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.873 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.873 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:40.873 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279786+0000", 2024-09-21T15:49:40.873 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1379999999999999e-06 2024-09-21T15:49:40.873 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.873 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.873 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:40.873 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279789+0000", 2024-09-21T15:49:40.873 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8629999999999999e-06 2024-09-21T15:49:40.873 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.873 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.873 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:40.873 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279792+0000", 2024-09-21T15:49:40.874 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.469e-06 2024-09-21T15:49:40.874 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.874 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.874 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:40.874 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.326002+0000", 2024-09-21T15:49:40.874 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046210481999999997 2024-09-21T15:49:40.874 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.874 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.874 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:40.874 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.326084+0000", 2024-09-21T15:49:40.874 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1570999999999995e-05 2024-09-21T15:49:40.874 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.874 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.874 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:40.874 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.326152+0000", 2024-09-21T15:49:40.875 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8523999999999996e-05 2024-09-21T15:49:40.875 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.875 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:40.875 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.875 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.875 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.875 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:65408 3.1d 3:bda7fcfc:::rbd_data.11336b2fdf4a.0000000000000000:c [read 4964352~0] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:40.875 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.279794+0000", 2024-09-21T15:49:40.875 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 3.5684269390000001, 2024-09-21T15:49:40.875 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046551309999999999, 2024-09-21T15:49:40.875 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:40.875 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:40.875 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:40.875 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:40.875 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:40.876 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 65408 2024-09-21T15:49:40.876 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.876 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:40.876 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.876 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:40.876 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279794+0000", 2024-09-21T15:49:40.876 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.876 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.876 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.876 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:40.876 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279794+0000", 2024-09-21T15:49:40.876 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.876 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.877 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.877 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:40.877 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279794+0000", 2024-09-21T15:49:40.877 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:40.877 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.877 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.877 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:40.877 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279795+0000", 2024-09-21T15:49:40.877 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.105e-06 2024-09-21T15:49:40.877 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.877 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.877 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:40.877 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279797+0000", 2024-09-21T15:49:40.877 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7590000000000002e-06 2024-09-21T15:49:40.877 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.877 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.878 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:40.878 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279800+0000", 2024-09-21T15:49:40.878 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4030000000000001e-06 2024-09-21T15:49:40.878 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.878 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.878 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:40.878 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.326193+0000", 2024-09-21T15:49:40.878 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046392845000000002 2024-09-21T15:49:40.878 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.878 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.878 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:40.878 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.326272+0000", 2024-09-21T15:49:40.879 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8788999999999994e-05 2024-09-21T15:49:40.879 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.879 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.879 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:40.879 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.326345+0000", 2024-09-21T15:49:40.879 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3629999999999998e-05 2024-09-21T15:49:40.879 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.879 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:40.879 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.879 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.879 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.879 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:65409 3.1d 3:bda7fcfc:::rbd_data.11336b2fdf4a.0000000000000000:c [read 4972544~0] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:40.879 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.279802+0000", 2024-09-21T15:49:40.879 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 3.5684186489999998, 2024-09-21T15:49:40.880 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046723775000000002, 2024-09-21T15:49:40.880 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:40.880 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:40.880 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:40.880 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:40.880 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:40.880 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 65409 2024-09-21T15:49:40.880 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.880 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:40.880 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.880 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:40.880 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279802+0000", 2024-09-21T15:49:40.880 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.880 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.880 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.881 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:40.881 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279802+0000", 2024-09-21T15:49:40.881 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.881 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.881 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.881 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:40.881 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279802+0000", 2024-09-21T15:49:40.881 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:40.881 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.881 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.881 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:40.881 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279803+0000", 2024-09-21T15:49:40.881 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.155e-06 2024-09-21T15:49:40.881 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.881 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.881 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:40.882 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279806+0000", 2024-09-21T15:49:40.882 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7949999999999999e-06 2024-09-21T15:49:40.882 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.882 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.882 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:40.882 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279808+0000", 2024-09-21T15:49:40.882 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.491e-06 2024-09-21T15:49:40.882 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.882 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.882 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:40.882 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.326375+0000", 2024-09-21T15:49:40.882 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046566824999999999 2024-09-21T15:49:40.882 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.882 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.882 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:40.883 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.326455+0000", 2024-09-21T15:49:40.883 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9680999999999998e-05 2024-09-21T15:49:40.883 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.883 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.883 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:40.883 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.326526+0000", 2024-09-21T15:49:40.883 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1057999999999996e-05 2024-09-21T15:49:40.883 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.883 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:40.883 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.883 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.883 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.883 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:70737 3.c 3:32ea75c5:::rbd_data.11336b2fdf4a.0000000000000001:c [read 1021952~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:40.883 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.495543+0000", 2024-09-21T15:49:40.884 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 3.3526774609999999, 2024-09-21T15:49:40.884 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045418501, 2024-09-21T15:49:40.884 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:40.884 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:40.884 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:40.884 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:40.884 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:40.884 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 70737 2024-09-21T15:49:40.884 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.884 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:40.884 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.884 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:40.884 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495543+0000", 2024-09-21T15:49:40.884 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.885 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.885 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.885 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:40.885 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495543+0000", 2024-09-21T15:49:40.885 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.885 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.885 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.885 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:40.885 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495543+0000", 2024-09-21T15:49:40.885 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:40.885 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.885 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.885 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:40.885 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495545+0000", 2024-09-21T15:49:40.885 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5380000000000001e-06 2024-09-21T15:49:40.886 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.886 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.886 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:40.886 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495552+0000", 2024-09-21T15:49:40.886 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4740000000000002e-06 2024-09-21T15:49:40.886 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.886 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.886 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:40.886 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495557+0000", 2024-09-21T15:49:40.886 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.792e-06 2024-09-21T15:49:40.886 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.886 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.886 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:40.886 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.540761+0000", 2024-09-21T15:49:40.887 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045203967999999997 2024-09-21T15:49:40.887 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.887 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.887 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:40.887 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.540871+0000", 2024-09-21T15:49:40.887 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00011034799999999999 2024-09-21T15:49:40.887 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.887 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.887 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:40.887 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.540962+0000", 2024-09-21T15:49:40.887 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0931000000000001e-05 2024-09-21T15:49:40.887 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.887 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:40.887 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.888 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.888 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.888 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:70738 3.c 3:32ea75c5:::rbd_data.11336b2fdf4a.0000000000000001:c [read 1030144~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:40.888 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.495576+0000", 2024-09-21T15:49:40.888 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 3.352645093, 2024-09-21T15:49:40.888 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045596921999999998, 2024-09-21T15:49:40.888 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:40.888 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:40.888 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:40.888 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:40.888 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:40.888 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 70738 2024-09-21T15:49:40.888 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.888 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:40.889 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.889 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:40.889 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495576+0000", 2024-09-21T15:49:40.889 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.889 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.889 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.889 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:40.889 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495576+0000", 2024-09-21T15:49:40.889 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.889 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.889 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.889 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:40.889 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495575+0000", 2024-09-21T15:49:40.889 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:40.890 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.890 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.890 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:40.890 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495578+0000", 2024-09-21T15:49:40.890 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4779999999999998e-06 2024-09-21T15:49:40.890 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.890 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.890 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:40.890 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495584+0000", 2024-09-21T15:49:40.890 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1149999999999999e-06 2024-09-21T15:49:40.890 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.890 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.890 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:40.891 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495589+0000", 2024-09-21T15:49:40.891 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8300000000000003e-06 2024-09-21T15:49:40.891 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.891 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.891 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:40.891 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.540995+0000", 2024-09-21T15:49:40.891 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045406390999999997 2024-09-21T15:49:40.891 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.891 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.891 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:40.891 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.541083+0000", 2024-09-21T15:49:40.891 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7579999999999998e-05 2024-09-21T15:49:40.891 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.892 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.892 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:40.892 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.541173+0000", 2024-09-21T15:49:40.892 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0017000000000002e-05 2024-09-21T15:49:40.892 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.892 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:40.892 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.892 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.892 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.892 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:70739 3.c 3:32ea75c5:::rbd_data.11336b2fdf4a.0000000000000001:c [read 1038336~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:40.892 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.495608+0000", 2024-09-21T15:49:40.892 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 3.352613034, 2024-09-21T15:49:40.892 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045757673999999998, 2024-09-21T15:49:40.892 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:40.893 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:40.893 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:40.893 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:40.893 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:40.893 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 70739 2024-09-21T15:49:40.893 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.893 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:40.893 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.893 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:40.893 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495608+0000", 2024-09-21T15:49:40.893 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.893 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.893 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.893 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:40.894 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495608+0000", 2024-09-21T15:49:40.894 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.894 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.894 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.894 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:40.894 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495607+0000", 2024-09-21T15:49:40.894 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:40.894 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.894 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.894 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:40.894 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495610+0000", 2024-09-21T15:49:40.894 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5610000000000001e-06 2024-09-21T15:49:40.894 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.894 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.895 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:40.895 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495617+0000", 2024-09-21T15:49:40.895 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8290000000000002e-06 2024-09-21T15:49:40.895 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.895 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.895 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:40.895 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495622+0000", 2024-09-21T15:49:40.895 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0730000000000004e-06 2024-09-21T15:49:40.895 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.895 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.895 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:40.895 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.541203+0000", 2024-09-21T15:49:40.895 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045581165 2024-09-21T15:49:40.895 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.896 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.896 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:40.896 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.541279+0000", 2024-09-21T15:49:40.896 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6637000000000005e-05 2024-09-21T15:49:40.896 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.896 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.896 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:40.896 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.541365+0000", 2024-09-21T15:49:40.896 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5965999999999997e-05 2024-09-21T15:49:40.896 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.896 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:40.896 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.896 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.896 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.896 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:70742 3.c 3:32ea75c5:::rbd_data.11336b2fdf4a.0000000000000001:c [read 1046528~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:40.897 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.495639+0000", 2024-09-21T15:49:40.897 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 3.352581995, 2024-09-21T15:49:40.897 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045872673000000003, 2024-09-21T15:49:40.897 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:40.897 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:40.897 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:40.897 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:40.897 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:40.897 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 70742 2024-09-21T15:49:40.897 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.897 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:40.897 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.897 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:40.897 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495639+0000", 2024-09-21T15:49:40.897 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.897 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.897 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.898 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:40.898 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495639+0000", 2024-09-21T15:49:40.898 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.898 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.898 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.898 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:40.898 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495638+0000", 2024-09-21T15:49:40.898 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:40.898 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.898 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.898 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:40.898 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495641+0000", 2024-09-21T15:49:40.898 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5849999999999998e-06 2024-09-21T15:49:40.898 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.899 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.899 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:40.899 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495647+0000", 2024-09-21T15:49:40.899 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3069999999999999e-06 2024-09-21T15:49:40.899 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.899 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.899 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:40.899 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495652+0000", 2024-09-21T15:49:40.899 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2229999999999998e-06 2024-09-21T15:49:40.899 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.899 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.899 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:40.899 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.541400+0000", 2024-09-21T15:49:40.899 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045748024999999998 2024-09-21T15:49:40.899 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.900 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.900 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:40.900 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.541451+0000", 2024-09-21T15:49:40.900 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0093999999999998e-05 2024-09-21T15:49:40.900 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.900 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.900 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:40.900 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.541511+0000", 2024-09-21T15:49:40.900 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0928999999999999e-05 2024-09-21T15:49:40.900 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.900 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:40.900 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.900 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.900 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.900 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:70743 3.c 3:32ea75c5:::rbd_data.11336b2fdf4a.0000000000000001:c [read 333824~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:40.901 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.495672+0000", 2024-09-21T15:49:40.901 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 3.3525492219999999, 2024-09-21T15:49:40.901 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045980785000000003, 2024-09-21T15:49:40.901 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:40.901 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:40.901 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:40.901 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:40.901 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:40.901 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 70743 2024-09-21T15:49:40.901 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.901 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:40.901 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.901 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:40.901 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495672+0000", 2024-09-21T15:49:40.901 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.901 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.901 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.902 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:40.902 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495672+0000", 2024-09-21T15:49:40.902 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.902 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.902 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.902 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:40.902 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495671+0000", 2024-09-21T15:49:40.902 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:40.902 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.902 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.902 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:40.902 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495674+0000", 2024-09-21T15:49:40.902 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5790000000000002e-06 2024-09-21T15:49:40.903 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.903 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.903 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:40.903 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495686+0000", 2024-09-21T15:49:40.903 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2276000000000001e-05 2024-09-21T15:49:40.903 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.903 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.903 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:40.903 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495692+0000", 2024-09-21T15:49:40.903 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8939999999999998e-06 2024-09-21T15:49:40.903 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.904 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.904 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:40.904 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.541538+0000", 2024-09-21T15:49:40.904 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045845761999999998 2024-09-21T15:49:40.904 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.904 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.904 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:40.904 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.541583+0000", 2024-09-21T15:49:40.904 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5343000000000003e-05 2024-09-21T15:49:40.904 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.904 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.904 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:40.904 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.541652+0000", 2024-09-21T15:49:40.904 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9468999999999995e-05 2024-09-21T15:49:40.905 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.905 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:40.905 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.905 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.905 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.905 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:70744 3.c 3:32ea75c5:::rbd_data.11336b2fdf4a.0000000000000001:c [read 342016~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:40.905 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.495705+0000", 2024-09-21T15:49:40.905 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 3.35251549, 2024-09-21T15:49:40.905 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046156794000000001, 2024-09-21T15:49:40.905 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:40.905 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:40.905 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:40.905 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:40.905 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:40.905 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 70744 2024-09-21T15:49:40.906 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.906 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:40.906 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.906 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:40.906 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495705+0000", 2024-09-21T15:49:40.906 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.906 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.906 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.906 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:40.906 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495705+0000", 2024-09-21T15:49:40.906 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.906 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.906 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.906 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:40.906 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495705+0000", 2024-09-21T15:49:40.907 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:40.907 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.907 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.907 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:40.907 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495707+0000", 2024-09-21T15:49:40.907 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4260000000000002e-06 2024-09-21T15:49:40.907 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.907 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.907 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:40.907 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495713+0000", 2024-09-21T15:49:40.907 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1029999999999998e-06 2024-09-21T15:49:40.907 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.907 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.908 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:40.908 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495720+0000", 2024-09-21T15:49:40.908 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.178e-06 2024-09-21T15:49:40.908 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.908 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.908 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:40.908 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.541690+0000", 2024-09-21T15:49:40.908 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045970069000000002 2024-09-21T15:49:40.908 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.908 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.908 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:40.908 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.541770+0000", 2024-09-21T15:49:40.908 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9976999999999997e-05 2024-09-21T15:49:40.908 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.908 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.909 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:40.909 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.541862+0000", 2024-09-21T15:49:40.909 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2532000000000001e-05 2024-09-21T15:49:40.909 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.909 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:40.909 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.909 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.909 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.909 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:70746 3.c 3:32ea75c5:::rbd_data.11336b2fdf4a.0000000000000001:c [read 350208~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:40.909 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.495741+0000", 2024-09-21T15:49:40.909 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 3.3524798809999998, 2024-09-21T15:49:40.909 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046299229999999997, 2024-09-21T15:49:40.909 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:40.909 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:40.909 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:40.910 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:40.910 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:40.910 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 70746 2024-09-21T15:49:40.910 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.910 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:40.910 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.910 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:40.910 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495741+0000", 2024-09-21T15:49:40.910 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.910 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.910 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.910 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:40.910 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495741+0000", 2024-09-21T15:49:40.910 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.910 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.911 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.911 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:40.911 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495740+0000", 2024-09-21T15:49:40.911 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:40.911 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.911 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.911 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:40.911 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495743+0000", 2024-09-21T15:49:40.911 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5189999999999999e-06 2024-09-21T15:49:40.911 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.911 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.911 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:40.911 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495749+0000", 2024-09-21T15:49:40.911 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5470000000000002e-06 2024-09-21T15:49:40.911 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.911 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.912 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:40.912 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495755+0000", 2024-09-21T15:49:40.912 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2399999999999998e-06 2024-09-21T15:49:40.912 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.912 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.912 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:40.912 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.541877+0000", 2024-09-21T15:49:40.912 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046122375 2024-09-21T15:49:40.912 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.912 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.912 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:40.912 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.541945+0000", 2024-09-21T15:49:40.912 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7825999999999997e-05 2024-09-21T15:49:40.912 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.912 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.912 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:40.913 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.542040+0000", 2024-09-21T15:49:40.913 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5303999999999994e-05 2024-09-21T15:49:40.913 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.913 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:40.913 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.913 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.913 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.913 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:70748 3.c 3:32ea75c5:::rbd_data.11336b2fdf4a.0000000000000001:c [read 358400~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:40.913 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.495759+0000", 2024-09-21T15:49:40.913 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 3.3524617270000001, 2024-09-21T15:49:40.913 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046462341999999997, 2024-09-21T15:49:40.913 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:40.913 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:40.913 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:40.913 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:40.914 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:40.914 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 70748 2024-09-21T15:49:40.914 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.914 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:40.914 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.914 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:40.914 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495759+0000", 2024-09-21T15:49:40.914 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.914 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.914 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.914 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:40.914 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495759+0000", 2024-09-21T15:49:40.914 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.914 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.915 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.915 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:40.915 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495759+0000", 2024-09-21T15:49:40.915 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:40.915 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.915 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.915 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:40.915 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495761+0000", 2024-09-21T15:49:40.915 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1900000000000002e-06 2024-09-21T15:49:40.915 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.915 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.915 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:40.915 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495767+0000", 2024-09-21T15:49:40.915 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0800000000000002e-06 2024-09-21T15:49:40.915 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.916 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.916 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:40.916 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495773+0000", 2024-09-21T15:49:40.916 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0000000000000002e-06 2024-09-21T15:49:40.916 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.916 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.916 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:40.916 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.542066+0000", 2024-09-21T15:49:40.916 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046293197000000001 2024-09-21T15:49:40.916 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.916 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.916 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:40.916 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.542139+0000", 2024-09-21T15:49:40.916 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2770999999999998e-05 2024-09-21T15:49:40.916 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.917 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.917 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:40.917 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.542221+0000", 2024-09-21T15:49:40.917 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2620000000000005e-05 2024-09-21T15:49:40.917 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.917 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:40.917 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.917 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.917 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.917 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:70750 3.c 3:32ea75c5:::rbd_data.11336b2fdf4a.0000000000000001:c [read 366592~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:40.917 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.495781+0000", 2024-09-21T15:49:40.917 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 3.3524393830000001, 2024-09-21T15:49:40.917 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046624691000000003, 2024-09-21T15:49:40.917 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:40.917 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:40.918 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:40.918 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:40.918 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:40.918 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 70750 2024-09-21T15:49:40.918 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.918 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:40.918 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.918 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:40.918 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495781+0000", 2024-09-21T15:49:40.918 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.918 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.918 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.918 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:40.918 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495781+0000", 2024-09-21T15:49:40.919 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.919 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.919 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.919 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:40.919 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495781+0000", 2024-09-21T15:49:40.919 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:40.919 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.919 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.919 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:40.919 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495783+0000", 2024-09-21T15:49:40.919 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4729999999999999e-06 2024-09-21T15:49:40.919 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.919 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.919 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:40.919 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495790+0000", 2024-09-21T15:49:40.920 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3500000000000002e-06 2024-09-21T15:49:40.920 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.920 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.920 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:40.920 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495795+0000", 2024-09-21T15:49:40.920 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9679999999999997e-06 2024-09-21T15:49:40.920 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.920 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.920 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:40.920 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.542248+0000", 2024-09-21T15:49:40.920 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046453117000000002 2024-09-21T15:49:40.920 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.920 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.920 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:40.920 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.542312+0000", 2024-09-21T15:49:40.920 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4215999999999996e-05 2024-09-21T15:49:40.921 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.921 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.921 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:40.921 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.542406+0000", 2024-09-21T15:49:40.921 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3980000000000005e-05 2024-09-21T15:49:40.921 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.921 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:40.921 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.921 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.921 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.921 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:70751 3.c 3:32ea75c5:::rbd_data.11336b2fdf4a.0000000000000001:c [read 374784~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:40.921 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.495799+0000", 2024-09-21T15:49:40.921 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 3.352421589, 2024-09-21T15:49:40.921 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046807169000000003, 2024-09-21T15:49:40.921 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:40.922 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:40.922 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:40.922 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:40.922 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:40.922 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 70751 2024-09-21T15:49:40.922 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.922 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:40.922 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.922 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:40.922 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495799+0000", 2024-09-21T15:49:40.922 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.922 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.922 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.922 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:40.923 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495799+0000", 2024-09-21T15:49:40.923 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.923 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.923 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.923 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:40.923 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495799+0000", 2024-09-21T15:49:40.923 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:40.923 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.923 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.923 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:40.923 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495801+0000", 2024-09-21T15:49:40.923 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2560000000000001e-06 2024-09-21T15:49:40.923 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.923 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.923 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:40.923 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495807+0000", 2024-09-21T15:49:40.924 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8209999999999998e-06 2024-09-21T15:49:40.924 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.924 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.924 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:40.924 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495812+0000", 2024-09-21T15:49:40.924 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0289999999999996e-06 2024-09-21T15:49:40.924 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.924 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.924 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:40.924 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.542433+0000", 2024-09-21T15:49:40.924 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046621392999999997 2024-09-21T15:49:40.924 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.924 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.924 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:40.924 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.542510+0000", 2024-09-21T15:49:40.925 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7069000000000004e-05 2024-09-21T15:49:40.925 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.925 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.925 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:40.925 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.542606+0000", 2024-09-21T15:49:40.925 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.6036e-05 2024-09-21T15:49:40.925 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.925 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:40.925 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.925 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.925 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.925 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:70753 3.c 3:32ea75c5:::rbd_data.11336b2fdf4a.0000000000000001:c [read 382976~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:40.925 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.495816+0000", 2024-09-21T15:49:40.925 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 3.3524045839999999, 2024-09-21T15:49:40.925 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046993252999999999, 2024-09-21T15:49:40.925 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:40.926 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:40.926 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:40.926 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:40.926 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:40.926 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 70753 2024-09-21T15:49:40.926 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.926 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:40.926 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.926 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:40.926 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495816+0000", 2024-09-21T15:49:40.926 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.926 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.926 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.926 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:40.926 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495816+0000", 2024-09-21T15:49:40.926 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.927 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.927 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.927 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:40.927 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495816+0000", 2024-09-21T15:49:40.927 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:40.927 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.927 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.927 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:40.927 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495818+0000", 2024-09-21T15:49:40.927 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1160000000000002e-06 2024-09-21T15:49:40.927 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.927 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.927 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:40.927 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495826+0000", 2024-09-21T15:49:40.927 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7800000000000001e-06 2024-09-21T15:49:40.927 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.928 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.928 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:40.928 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495831+0000", 2024-09-21T15:49:40.928 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9010000000000002e-06 2024-09-21T15:49:40.928 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.928 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.928 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:40.928 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.542665+0000", 2024-09-21T15:49:40.928 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046834037000000002 2024-09-21T15:49:40.928 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.928 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.928 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:40.928 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.542729+0000", 2024-09-21T15:49:40.928 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4199e-05 2024-09-21T15:49:40.928 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.928 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.929 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:40.929 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.542809+0000", 2024-09-21T15:49:40.929 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0592000000000004e-05 2024-09-21T15:49:40.929 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.929 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:40.929 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.929 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.929 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.929 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:70755 3.c 3:32ea75c5:::rbd_data.11336b2fdf4a.0000000000000001:c [read 391168~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:40.929 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.495835+0000", 2024-09-21T15:49:40.929 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 3.3523860669999999, 2024-09-21T15:49:40.929 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047136708999999999, 2024-09-21T15:49:40.929 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:40.929 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:40.929 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:40.930 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:40.930 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:40.930 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 70755 2024-09-21T15:49:40.930 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.930 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:40.930 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.930 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:40.930 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495835+0000", 2024-09-21T15:49:40.930 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.930 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.930 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.930 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:40.930 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495835+0000", 2024-09-21T15:49:40.930 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:40.930 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.931 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.931 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:40.931 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495834+0000", 2024-09-21T15:49:40.931 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:40.931 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.931 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.931 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:40.931 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495836+0000", 2024-09-21T15:49:40.931 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1280000000000002e-06 2024-09-21T15:49:40.931 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.931 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.931 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:40.931 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495842+0000", 2024-09-21T15:49:40.931 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8819999999999998e-06 2024-09-21T15:49:40.931 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.931 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.932 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:40.932 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495848+0000", 2024-09-21T15:49:40.932 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1850000000000003e-06 2024-09-21T15:49:40.932 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.932 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.932 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:40.932 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.542836+0000", 2024-09-21T15:49:40.932 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046988156000000003 2024-09-21T15:49:40.932 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.932 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.932 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:40.932 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.542897+0000", 2024-09-21T15:49:40.932 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1505000000000002e-05 2024-09-21T15:49:40.932 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:40.932 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:40.932 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:40.932 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.542971+0000", 2024-09-21T15:49:40.933 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4221000000000006e-05 2024-09-21T15:49:40.933 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.933 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:40.933 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.933 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:40.933 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:40.933 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:40.933 INFO:teuthology.orchestra.run.smithi089.stdout:truncating image image_client.0-clone14 from 0x9b246c0 (overlap 0x585f0e4) to 0x3ddc239 2024-09-21T15:49:40.934 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T15:49:40.978 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:40.977+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:41.049 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:41.048+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:41.062 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:41.062 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:41.063 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:49:41.063 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:41.071 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T15:49:41.108 INFO:teuthology.orchestra.run.smithi089.stdout:checking clone #13, image image_client.0-clone13 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent14 2024-09-21T15:49:41.114 INFO:teuthology.orchestra.run.smithi089.stdout:2783 trunc from 0x9b246c0 to 0x6ee0edb 2024-09-21T15:49:41.142 INFO:teuthology.orchestra.run.smithi089.stdout:2784 writesame 0x1931b6f thru 0x193cad2 data_size 0x19(0xaf64 bytes) 2024-09-21T15:49:41.155 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:41.154+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:41.171 INFO:teuthology.orchestra.run.smithi089.stdout:2786 read 0x34b5626 thru 0x34c2f0c (0xd8e7 bytes) 2024-09-21T15:49:41.172 INFO:teuthology.orchestra.run.smithi089.stdout:2787 write 0x3e2377f thru 0x3e30314 (0xcb96 bytes) 2024-09-21T15:49:41.192 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:41.207 INFO:teuthology.orchestra.run.smithi089.stdout:2790 write 0xda98174 thru 0xda9ac37 (0x2ac4 bytes) 2024-09-21T15:49:41.218 INFO:teuthology.orchestra.run.smithi089.stdout:2791 trunc from 0xda9ac38 to 0x9ca682e 2024-09-21T15:49:41.230 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:41.230 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:41.230 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:49:41.230 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:49:41.230 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:41.239 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T15:49:41.244 INFO:teuthology.orchestra.run.smithi089.stdout:2792 write 0xcaef039 thru 0xcafe112 (0xf0da bytes) 2024-09-21T15:49:41.264 INFO:teuthology.orchestra.run.smithi089.stdout:2793 write 0xeae016f thru 0xeaea322 (0xa1b4 bytes) 2024-09-21T15:49:41.280 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:41.279+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:41.284 INFO:teuthology.orchestra.run.smithi089.stdout:2794 trunc from 0xeaea323 to 0x5589078 2024-09-21T15:49:41.344 INFO:teuthology.orchestra.run.smithi089.stdout:2795 writesame 0x6c2a433 thru 0x6c34c8d data_size 0x7(0xa85b bytes) 2024-09-21T15:49:41.344 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:41.344 INFO:teuthology.orchestra.run.smithi089.stdout:2798 trunc from 0x6c34c8e to 0x69e0307 2024-09-21T15:49:41.350 INFO:teuthology.orchestra.run.smithi089.stdout:2799 punch from 0x26eccd5 to 0x26ed232, (0x55d bytes) 2024-09-21T15:49:41.350 INFO:teuthology.orchestra.run.smithi089.stdout:2800 read 0x636ef75 thru 0x6371db5 (0x2e41 bytes) 2024-09-21T15:49:41.351 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:41.350+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:41.363 INFO:teuthology.orchestra.run.smithi089.stdout:2801 write 0xc160461 thru 0xc1647be (0x435e bytes) 2024-09-21T15:49:41.370 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:41.371 INFO:teuthology.orchestra.run.smithi089.stdout:2803 read 0x9f7ee94 thru 0x9f81505 (0x2672 bytes) 2024-09-21T15:49:41.379 INFO:teuthology.orchestra.run.smithi089.stdout:2804 writesame 0xde29906 thru 0xde32ce7 data_size 0x17(0x93e2 bytes) 2024-09-21T15:49:41.379 INFO:teuthology.orchestra.run.smithi089.stdout:2807 write 0xdbf18a0 thru 0xdbf9882 (0x7fe3 bytes) 2024-09-21T15:49:41.395 INFO:teuthology.orchestra.run.smithi089.stdout:2809 punch from 0x70d921a to 0x70e3600, (0xa3e6 bytes) 2024-09-21T15:49:41.395 INFO:teuthology.orchestra.run.smithi089.stdout:2810 write 0x5e25413 thru 0x5e2b6d7 (0x62c5 bytes) 2024-09-21T15:49:41.400 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:41.400 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:49:41.401 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:49:41.401 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:49:41.401 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.401 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63140 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 876544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:41.401 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.133649+0000", 2024-09-21T15:49:41.401 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.2648420370000002, 2024-09-21T15:49:41.401 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053950201000000003, 2024-09-21T15:49:41.401 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:41.401 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:41.401 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:41.401 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:41.401 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:41.401 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63140 2024-09-21T15:49:41.402 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.402 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:41.402 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.402 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:41.402 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133649+0000", 2024-09-21T15:49:41.402 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.402 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.402 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.402 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:41.402 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133649+0000", 2024-09-21T15:49:41.402 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.402 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.402 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.402 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:41.403 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133648+0000", 2024-09-21T15:49:41.403 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:41.403 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.403 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.403 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:41.403 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133651+0000", 2024-09-21T15:49:41.403 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1549999999999999e-06 2024-09-21T15:49:41.403 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.403 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.403 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:41.403 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133660+0000", 2024-09-21T15:49:41.403 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3899999999999993e-06 2024-09-21T15:49:41.403 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.403 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.403 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:41.404 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133669+0000", 2024-09-21T15:49:41.404 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3219999999999999e-06 2024-09-21T15:49:41.404 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.404 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.404 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:41.404 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187409+0000", 2024-09-21T15:49:41.404 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053740257 2024-09-21T15:49:41.404 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.404 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.404 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:41.404 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187492+0000", 2024-09-21T15:49:41.404 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2409000000000003e-05 2024-09-21T15:49:41.404 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.405 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.405 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:41.405 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187599+0000", 2024-09-21T15:49:41.405 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000107334 2024-09-21T15:49:41.405 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.405 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:41.405 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.405 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.405 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.405 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63141 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 884736~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:41.405 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.133843+0000", 2024-09-21T15:49:41.405 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.2646479179999996, 2024-09-21T15:49:41.405 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053954015000000001, 2024-09-21T15:49:41.405 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:41.405 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:41.406 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:41.406 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:41.406 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:41.406 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63141 2024-09-21T15:49:41.406 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.406 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:41.406 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.406 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:41.406 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133843+0000", 2024-09-21T15:49:41.406 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.406 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.406 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.406 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:41.406 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133843+0000", 2024-09-21T15:49:41.407 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.407 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.407 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.407 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:41.407 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133673+0000", 2024-09-21T15:49:41.407 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998302 2024-09-21T15:49:41.407 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.407 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.407 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:41.407 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133846+0000", 2024-09-21T15:49:41.407 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000172762 2024-09-21T15:49:41.407 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.407 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.407 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:41.407 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133855+0000", 2024-09-21T15:49:41.408 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2940000000000006e-06 2024-09-21T15:49:41.408 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.408 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.408 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:41.408 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133863+0000", 2024-09-21T15:49:41.408 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9740000000000002e-06 2024-09-21T15:49:41.408 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.408 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.408 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:41.408 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187630+0000", 2024-09-21T15:49:41.408 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053767211000000002 2024-09-21T15:49:41.408 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.408 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.408 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:41.408 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187707+0000", 2024-09-21T15:49:41.409 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7446000000000004e-05 2024-09-21T15:49:41.409 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.409 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.409 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:41.409 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187797+0000", 2024-09-21T15:49:41.409 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.9858000000000006e-05 2024-09-21T15:49:41.409 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.409 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:41.409 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.409 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.409 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.409 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63142 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 892928~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:41.409 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134046+0000", 2024-09-21T15:49:41.409 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.2644449760000001, 2024-09-21T15:49:41.410 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054017849999999999, 2024-09-21T15:49:41.410 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:41.410 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:41.410 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:41.410 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:41.410 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:41.410 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63142 2024-09-21T15:49:41.410 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.410 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:41.410 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.410 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:41.410 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134046+0000", 2024-09-21T15:49:41.410 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.410 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.411 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.411 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:41.411 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134046+0000", 2024-09-21T15:49:41.411 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.411 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.411 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.411 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:41.411 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133867+0000", 2024-09-21T15:49:41.411 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998217 2024-09-21T15:49:41.411 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.411 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.411 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:41.411 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134049+0000", 2024-09-21T15:49:41.412 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000181695 2024-09-21T15:49:41.412 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.412 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.412 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:41.412 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134057+0000", 2024-09-21T15:49:41.412 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.303e-06 2024-09-21T15:49:41.412 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.412 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.412 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:41.412 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134063+0000", 2024-09-21T15:49:41.412 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9959999999999999e-06 2024-09-21T15:49:41.412 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.412 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.412 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:41.413 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187849+0000", 2024-09-21T15:49:41.413 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053785571999999997 2024-09-21T15:49:41.413 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.413 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.413 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:41.413 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187954+0000", 2024-09-21T15:49:41.413 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010524899999999999 2024-09-21T15:49:41.413 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.413 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.413 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:41.413 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188064+0000", 2024-09-21T15:49:41.413 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000109549 2024-09-21T15:49:41.413 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.413 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:41.414 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.414 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.414 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.414 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63143 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 901120~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:41.414 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134239+0000", 2024-09-21T15:49:41.414 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.2642523829999996, 2024-09-21T15:49:41.414 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054026058000000002, 2024-09-21T15:49:41.414 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:41.414 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:41.414 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:41.414 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:41.414 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:41.414 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63143 2024-09-21T15:49:41.414 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.415 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:41.415 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.415 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:41.415 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134239+0000", 2024-09-21T15:49:41.415 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.415 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.415 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.415 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:41.415 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134239+0000", 2024-09-21T15:49:41.415 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.415 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.415 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.415 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:41.415 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134068+0000", 2024-09-21T15:49:41.416 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998298 2024-09-21T15:49:41.416 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.416 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.416 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:41.416 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134242+0000", 2024-09-21T15:49:41.416 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00017414199999999999 2024-09-21T15:49:41.416 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.417 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.417 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:41.417 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134264+0000", 2024-09-21T15:49:41.417 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1552e-05 2024-09-21T15:49:41.417 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.417 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.417 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:41.417 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134270+0000", 2024-09-21T15:49:41.417 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6939999999999998e-06 2024-09-21T15:49:41.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.418 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:41.418 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188108+0000", 2024-09-21T15:49:41.418 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.05383719 2024-09-21T15:49:41.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.418 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:41.418 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188186+0000", 2024-09-21T15:49:41.418 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8318000000000005e-05 2024-09-21T15:49:41.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.418 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:41.418 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188265+0000", 2024-09-21T15:49:41.419 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8597000000000007e-05 2024-09-21T15:49:41.419 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.419 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:41.419 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.419 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.419 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.419 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63144 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 909312~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:41.419 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134432+0000", 2024-09-21T15:49:41.419 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.2640588270000004, 2024-09-21T15:49:41.419 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053968953, 2024-09-21T15:49:41.419 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:41.419 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:41.419 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:41.419 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:41.419 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:41.420 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63144 2024-09-21T15:49:41.420 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.420 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:41.420 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.420 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:41.420 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134432+0000", 2024-09-21T15:49:41.420 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.420 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.420 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.420 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:41.420 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134432+0000", 2024-09-21T15:49:41.420 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.420 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.420 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.420 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:41.421 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134276+0000", 2024-09-21T15:49:41.421 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998441 2024-09-21T15:49:41.421 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.421 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.421 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:41.421 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134436+0000", 2024-09-21T15:49:41.421 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000159591 2024-09-21T15:49:41.421 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.421 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.421 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:41.421 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134445+0000", 2024-09-21T15:49:41.421 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6780000000000006e-06 2024-09-21T15:49:41.421 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.422 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.422 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:41.422 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134452+0000", 2024-09-21T15:49:41.422 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6830000000000008e-06 2024-09-21T15:49:41.422 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.422 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.422 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:41.422 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188282+0000", 2024-09-21T15:49:41.422 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053829252000000001 2024-09-21T15:49:41.422 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.422 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.422 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:41.422 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188328+0000", 2024-09-21T15:49:41.422 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6734999999999999e-05 2024-09-21T15:49:41.422 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.423 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.423 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:41.423 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188401+0000", 2024-09-21T15:49:41.423 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2731000000000005e-05 2024-09-21T15:49:41.423 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.423 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:41.423 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.423 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.423 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.423 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63145 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 860160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:41.423 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134736+0000", 2024-09-21T15:49:41.423 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.2637550419999997, 2024-09-21T15:49:41.423 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053865960999999997, 2024-09-21T15:49:41.423 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:41.424 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:41.424 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:41.424 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:41.424 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:41.424 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63145 2024-09-21T15:49:41.424 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.424 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:41.424 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.424 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:41.424 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134736+0000", 2024-09-21T15:49:41.424 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.424 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.424 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.424 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:41.424 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134736+0000", 2024-09-21T15:49:41.425 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.425 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.425 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.425 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:41.425 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134458+0000", 2024-09-21T15:49:41.425 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999722 2024-09-21T15:49:41.425 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.425 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.425 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:41.425 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134741+0000", 2024-09-21T15:49:41.425 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00028245599999999999 2024-09-21T15:49:41.425 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.425 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.425 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:41.426 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134753+0000", 2024-09-21T15:49:41.426 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2548000000000001e-05 2024-09-21T15:49:41.426 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.426 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.426 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:41.426 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134760+0000", 2024-09-21T15:49:41.426 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4189999999999996e-06 2024-09-21T15:49:41.426 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.426 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.426 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:41.426 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188426+0000", 2024-09-21T15:49:41.426 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053665894999999998 2024-09-21T15:49:41.426 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.426 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.427 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:41.427 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188502+0000", 2024-09-21T15:49:41.427 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5791000000000007e-05 2024-09-21T15:49:41.427 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.427 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.427 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:41.427 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188602+0000", 2024-09-21T15:49:41.427 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9667999999999995e-05 2024-09-21T15:49:41.427 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.427 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:41.427 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.427 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.427 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.427 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63148 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 933888~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:41.427 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.135289+0000", 2024-09-21T15:49:41.428 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.2632020869999998, 2024-09-21T15:49:41.428 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053948488000000003, 2024-09-21T15:49:41.428 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:41.428 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:41.428 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:41.428 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:41.428 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:41.428 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63148 2024-09-21T15:49:41.428 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.428 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:41.428 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.428 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:41.428 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135289+0000", 2024-09-21T15:49:41.428 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.428 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.429 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.429 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:41.429 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135289+0000", 2024-09-21T15:49:41.429 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.429 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.429 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.429 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:41.429 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135288+0000", 2024-09-21T15:49:41.429 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:41.429 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.429 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.429 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:41.429 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135296+0000", 2024-09-21T15:49:41.429 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5399999999999998e-06 2024-09-21T15:49:41.430 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.430 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.430 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:41.430 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135303+0000", 2024-09-21T15:49:41.430 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0500000000000003e-06 2024-09-21T15:49:41.430 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.430 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.430 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:41.430 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135308+0000", 2024-09-21T15:49:41.430 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9749999999999998e-06 2024-09-21T15:49:41.430 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.430 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.430 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:41.431 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189066+0000", 2024-09-21T15:49:41.431 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053758501 2024-09-21T15:49:41.431 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.431 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.431 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:41.431 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189145+0000", 2024-09-21T15:49:41.431 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9017000000000006e-05 2024-09-21T15:49:41.431 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.431 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.431 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:41.431 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189237+0000", 2024-09-21T15:49:41.431 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1906999999999999e-05 2024-09-21T15:49:41.431 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.431 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:41.432 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.432 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.432 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.432 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63149 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 942080~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:41.432 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.135430+0000", 2024-09-21T15:49:41.432 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.263060758, 2024-09-21T15:49:41.432 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054005434999999997, 2024-09-21T15:49:41.432 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:41.432 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:41.432 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:41.432 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:41.432 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:41.432 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63149 2024-09-21T15:49:41.432 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.433 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:41.433 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.433 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:41.433 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135430+0000", 2024-09-21T15:49:41.433 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.433 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.433 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.433 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:41.433 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135430+0000", 2024-09-21T15:49:41.433 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.433 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.433 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.433 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:41.433 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135313+0000", 2024-09-21T15:49:41.434 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998827 2024-09-21T15:49:41.434 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.434 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.434 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:41.434 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135434+0000", 2024-09-21T15:49:41.434 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000120876 2024-09-21T15:49:41.434 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.434 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.434 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:41.434 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135443+0000", 2024-09-21T15:49:41.434 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3340000000000008e-06 2024-09-21T15:49:41.434 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.434 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.435 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:41.435 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135449+0000", 2024-09-21T15:49:41.435 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1580000000000002e-06 2024-09-21T15:49:41.435 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.435 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.435 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:41.435 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189267+0000", 2024-09-21T15:49:41.435 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053817775999999998 2024-09-21T15:49:41.435 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.435 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.435 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:41.435 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189336+0000", 2024-09-21T15:49:41.435 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9485000000000003e-05 2024-09-21T15:49:41.436 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.436 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.436 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:41.436 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189436+0000", 2024-09-21T15:49:41.436 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9277000000000005e-05 2024-09-21T15:49:41.436 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.436 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:41.436 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.436 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.436 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.436 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63150 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 950272~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:41.436 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.135618+0000", 2024-09-21T15:49:41.436 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.2628730130000001, 2024-09-21T15:49:41.436 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054027393, 2024-09-21T15:49:41.437 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:41.437 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:41.437 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:41.437 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:41.437 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:41.437 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63150 2024-09-21T15:49:41.437 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.437 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:41.437 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.437 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:41.437 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135618+0000", 2024-09-21T15:49:41.437 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.437 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.437 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.438 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:41.438 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135618+0000", 2024-09-21T15:49:41.438 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.438 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.438 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.438 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:41.438 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135454+0000", 2024-09-21T15:49:41.438 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998364 2024-09-21T15:49:41.438 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.438 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.438 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:41.438 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135621+0000", 2024-09-21T15:49:41.438 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016714099999999999 2024-09-21T15:49:41.438 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.439 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.439 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:41.439 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135631+0000", 2024-09-21T15:49:41.439 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5340000000000005e-06 2024-09-21T15:49:41.439 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.439 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.439 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:41.439 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135638+0000", 2024-09-21T15:49:41.439 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9280000000000004e-06 2024-09-21T15:49:41.439 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.439 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.439 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:41.439 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189463+0000", 2024-09-21T15:49:41.439 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053825681 2024-09-21T15:49:41.440 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.440 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.440 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:41.440 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189546+0000", 2024-09-21T15:49:41.440 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2278000000000001e-05 2024-09-21T15:49:41.440 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.440 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.440 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:41.440 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189645+0000", 2024-09-21T15:49:41.440 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9543999999999994e-05 2024-09-21T15:49:41.440 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.440 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:41.440 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.440 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.441 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.441 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63152 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 966656~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:41.441 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.136029+0000", 2024-09-21T15:49:41.441 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.2624624210000004, 2024-09-21T15:49:41.441 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054024823999999999, 2024-09-21T15:49:41.441 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:41.441 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:41.441 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:41.441 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:41.441 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:41.441 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63152 2024-09-21T15:49:41.441 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.441 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:41.441 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.442 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:41.442 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136029+0000", 2024-09-21T15:49:41.442 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.442 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.442 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.442 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:41.442 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136029+0000", 2024-09-21T15:49:41.442 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.442 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.442 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.442 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:41.442 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136028+0000", 2024-09-21T15:49:41.442 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:41.442 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.442 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.443 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:41.443 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136031+0000", 2024-09-21T15:49:41.443 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.033e-06 2024-09-21T15:49:41.443 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.443 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.443 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:41.443 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136039+0000", 2024-09-21T15:49:41.443 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7200000000000006e-06 2024-09-21T15:49:41.443 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.443 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.443 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:41.443 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136051+0000", 2024-09-21T15:49:41.443 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1987000000000001e-05 2024-09-21T15:49:41.443 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.444 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.444 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:41.444 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189876+0000", 2024-09-21T15:49:41.444 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053825264999999997 2024-09-21T15:49:41.444 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.444 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.444 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:41.444 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189955+0000", 2024-09-21T15:49:41.444 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9296999999999997e-05 2024-09-21T15:49:41.444 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.444 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.444 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:41.444 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190053+0000", 2024-09-21T15:49:41.444 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.8067999999999997e-05 2024-09-21T15:49:41.444 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.445 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:41.445 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.445 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.445 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.445 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63153 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 974848~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:41.445 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.136311+0000", 2024-09-21T15:49:41.445 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.2621799080000002, 2024-09-21T15:49:41.445 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053933825999999997, 2024-09-21T15:49:41.445 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:41.445 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:41.445 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:41.445 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:41.445 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:41.445 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63153 2024-09-21T15:49:41.445 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.446 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:41.446 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.446 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:41.446 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136311+0000", 2024-09-21T15:49:41.446 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.446 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.446 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.446 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:41.446 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136311+0000", 2024-09-21T15:49:41.446 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.446 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.446 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.446 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:41.446 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136057+0000", 2024-09-21T15:49:41.447 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9997463 2024-09-21T15:49:41.447 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.447 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.447 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:41.447 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136316+0000", 2024-09-21T15:49:41.447 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00025872299999999998 2024-09-21T15:49:41.447 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.447 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.447 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:41.447 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136326+0000", 2024-09-21T15:49:41.447 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0552e-05 2024-09-21T15:49:41.447 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.447 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.447 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:41.447 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136334+0000", 2024-09-21T15:49:41.448 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1409999999999999e-06 2024-09-21T15:49:41.448 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.448 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.448 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:41.448 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190080+0000", 2024-09-21T15:49:41.448 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053745933000000003 2024-09-21T15:49:41.448 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.448 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.448 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:41.448 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190157+0000", 2024-09-21T15:49:41.448 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7412999999999999e-05 2024-09-21T15:49:41.448 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.448 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.448 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:41.449 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190245+0000", 2024-09-21T15:49:41.449 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7904000000000004e-05 2024-09-21T15:49:41.449 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.449 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:41.449 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.449 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.449 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.449 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63154 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 983040~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:41.449 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.136472+0000", 2024-09-21T15:49:41.449 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.2620185570000002, 2024-09-21T15:49:41.449 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053968448000000002, 2024-09-21T15:49:41.449 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:41.449 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:41.449 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:41.450 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:41.450 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:41.450 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63154 2024-09-21T15:49:41.450 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.450 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:41.450 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.450 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:41.450 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136472+0000", 2024-09-21T15:49:41.450 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.450 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.450 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.450 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:41.450 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136472+0000", 2024-09-21T15:49:41.450 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.451 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.451 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.451 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:41.451 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136341+0000", 2024-09-21T15:49:41.451 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998684 2024-09-21T15:49:41.451 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.451 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.451 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:41.451 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136476+0000", 2024-09-21T15:49:41.451 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013528100000000001 2024-09-21T15:49:41.451 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.451 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.451 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:41.451 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136484+0000", 2024-09-21T15:49:41.451 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4760000000000001e-06 2024-09-21T15:49:41.452 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.452 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.452 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:41.452 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136490+0000", 2024-09-21T15:49:41.452 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8640000000000001e-06 2024-09-21T15:49:41.452 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.452 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.452 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:41.452 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190271+0000", 2024-09-21T15:49:41.452 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053781223000000003 2024-09-21T15:49:41.452 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.452 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.452 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:41.452 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190348+0000", 2024-09-21T15:49:41.452 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7058999999999995e-05 2024-09-21T15:49:41.452 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.453 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.453 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:41.453 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190441+0000", 2024-09-21T15:49:41.453 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2937999999999997e-05 2024-09-21T15:49:41.453 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.453 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:41.453 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.453 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.453 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.453 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63157 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1015808~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:41.453 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137112+0000", 2024-09-21T15:49:41.453 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.2613792699999999, 2024-09-21T15:49:41.453 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053932345, 2024-09-21T15:49:41.453 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:41.453 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:41.453 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:41.454 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:41.454 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:41.454 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63157 2024-09-21T15:49:41.454 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.454 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:41.454 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.454 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:41.454 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137112+0000", 2024-09-21T15:49:41.454 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.454 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.454 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.454 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:41.454 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137112+0000", 2024-09-21T15:49:41.454 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.454 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.454 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.455 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:41.455 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137072+0000", 2024-09-21T15:49:41.455 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999604 2024-09-21T15:49:41.455 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.455 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.455 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:41.455 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137115+0000", 2024-09-21T15:49:41.455 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2379999999999997e-05 2024-09-21T15:49:41.455 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.455 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.455 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:41.455 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137121+0000", 2024-09-21T15:49:41.455 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6499999999999999e-06 2024-09-21T15:49:41.455 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.455 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.455 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:41.456 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137126+0000", 2024-09-21T15:49:41.456 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0769999999999998e-06 2024-09-21T15:49:41.456 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.456 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.456 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:41.456 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190868+0000", 2024-09-21T15:49:41.456 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053741555000000003 2024-09-21T15:49:41.456 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.456 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.456 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:41.456 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190945+0000", 2024-09-21T15:49:41.456 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7171999999999999e-05 2024-09-21T15:49:41.456 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.456 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.456 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:41.456 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191044+0000", 2024-09-21T15:49:41.456 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9031999999999996e-05 2024-09-21T15:49:41.457 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.457 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:41.457 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.457 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.457 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.457 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63158 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1007616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:41.457 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137293+0000", 2024-09-21T15:49:41.457 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.2611979829999997, 2024-09-21T15:49:41.457 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054005642, 2024-09-21T15:49:41.457 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:41.457 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:41.457 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:41.457 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:41.457 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:41.457 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63158 2024-09-21T15:49:41.457 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.458 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:41.458 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.458 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:41.458 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137293+0000", 2024-09-21T15:49:41.458 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.458 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.458 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.458 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:41.458 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137293+0000", 2024-09-21T15:49:41.458 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.458 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.458 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.458 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:41.458 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137131+0000", 2024-09-21T15:49:41.458 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998379 2024-09-21T15:49:41.458 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.459 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.459 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:41.459 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137296+0000", 2024-09-21T15:49:41.459 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016568999999999999 2024-09-21T15:49:41.459 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.459 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.459 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:41.459 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137306+0000", 2024-09-21T15:49:41.459 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.6029999999999994e-06 2024-09-21T15:49:41.459 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.459 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.459 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:41.459 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137312+0000", 2024-09-21T15:49:41.459 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3210000000000001e-06 2024-09-21T15:49:41.459 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.459 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.460 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:41.460 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191087+0000", 2024-09-21T15:49:41.460 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053774686000000002 2024-09-21T15:49:41.460 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.460 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.460 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:41.460 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191175+0000", 2024-09-21T15:49:41.460 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8106e-05 2024-09-21T15:49:41.460 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.460 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.460 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:41.460 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191299+0000", 2024-09-21T15:49:41.460 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012343599999999999 2024-09-21T15:49:41.460 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.460 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:41.461 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.461 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.461 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.461 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63159 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1024000~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:41.461 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137568+0000", 2024-09-21T15:49:41.461 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.2609227110000001, 2024-09-21T15:49:41.461 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053962999999999997, 2024-09-21T15:49:41.461 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:41.461 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:41.461 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:41.461 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:41.461 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:41.461 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63159 2024-09-21T15:49:41.461 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.462 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:41.462 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.462 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:41.462 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137568+0000", 2024-09-21T15:49:41.462 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.462 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.462 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.462 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:41.462 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137568+0000", 2024-09-21T15:49:41.462 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.462 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.462 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.462 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:41.462 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137317+0000", 2024-09-21T15:49:41.463 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9997492 2024-09-21T15:49:41.463 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.463 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.463 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:41.463 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137572+0000", 2024-09-21T15:49:41.463 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00025491499999999998 2024-09-21T15:49:41.463 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.463 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.463 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:41.463 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137582+0000", 2024-09-21T15:49:41.463 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4490000000000005e-06 2024-09-21T15:49:41.463 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.463 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.463 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:41.463 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137589+0000", 2024-09-21T15:49:41.463 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5580000000000003e-06 2024-09-21T15:49:41.464 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.464 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.464 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:41.464 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191344+0000", 2024-09-21T15:49:41.464 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053754272999999998 2024-09-21T15:49:41.464 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.464 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.464 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:41.464 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191429+0000", 2024-09-21T15:49:41.464 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.509e-05 2024-09-21T15:49:41.464 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.464 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.464 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:41.464 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191531+0000", 2024-09-21T15:49:41.464 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000102519 2024-09-21T15:49:41.464 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.465 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:41.465 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.465 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.465 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.465 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63160 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1032192~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:41.465 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137771+0000", 2024-09-21T15:49:41.465 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.2607201769999996, 2024-09-21T15:49:41.465 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053957818999999997, 2024-09-21T15:49:41.465 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:41.465 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:41.465 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:41.465 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:41.465 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:41.465 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63160 2024-09-21T15:49:41.465 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.466 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:41.466 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.466 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:41.466 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137771+0000", 2024-09-21T15:49:41.466 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.466 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.466 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.466 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:41.466 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137771+0000", 2024-09-21T15:49:41.466 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.466 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.466 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.466 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:41.467 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137596+0000", 2024-09-21T15:49:41.467 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998255 2024-09-21T15:49:41.467 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.467 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.467 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:41.467 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137780+0000", 2024-09-21T15:49:41.467 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00018363900000000001 2024-09-21T15:49:41.467 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.467 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.467 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:41.467 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137791+0000", 2024-09-21T15:49:41.467 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.096e-05 2024-09-21T15:49:41.467 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.467 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.467 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:41.468 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137798+0000", 2024-09-21T15:49:41.468 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1169999999999998e-06 2024-09-21T15:49:41.468 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.468 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.468 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:41.468 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191560+0000", 2024-09-21T15:49:41.468 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053762163000000002 2024-09-21T15:49:41.468 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.468 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.468 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:41.468 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191634+0000", 2024-09-21T15:49:41.468 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4381999999999994e-05 2024-09-21T15:49:41.468 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.468 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.469 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:41.469 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191729+0000", 2024-09-21T15:49:41.469 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4098000000000006e-05 2024-09-21T15:49:41.469 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.469 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:41.469 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.469 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.469 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.469 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63161 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1040384~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:41.469 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.138018+0000", 2024-09-21T15:49:41.469 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.2604734229999996, 2024-09-21T15:49:41.469 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053905657000000003, 2024-09-21T15:49:41.469 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:41.469 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:41.469 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:41.469 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:41.470 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:41.470 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63161 2024-09-21T15:49:41.470 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.470 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:41.470 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.470 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:41.470 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138018+0000", 2024-09-21T15:49:41.470 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.470 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.470 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.470 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:41.470 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138018+0000", 2024-09-21T15:49:41.470 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.470 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.471 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.471 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:41.471 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137804+0000", 2024-09-21T15:49:41.471 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9997859 2024-09-21T15:49:41.471 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.471 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.471 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:41.471 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138021+0000", 2024-09-21T15:49:41.471 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00021769899999999999 2024-09-21T15:49:41.471 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.471 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.471 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:41.471 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138043+0000", 2024-09-21T15:49:41.471 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1627000000000001e-05 2024-09-21T15:49:41.471 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.472 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.472 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:41.472 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138054+0000", 2024-09-21T15:49:41.472 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1321999999999999e-05 2024-09-21T15:49:41.472 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.472 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.472 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:41.472 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191755+0000", 2024-09-21T15:49:41.472 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053700354999999998 2024-09-21T15:49:41.472 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.472 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.472 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:41.472 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191830+0000", 2024-09-21T15:49:41.472 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5320000000000004e-05 2024-09-21T15:49:41.472 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.472 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.473 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:41.473 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191923+0000", 2024-09-21T15:49:41.473 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3282999999999994e-05 2024-09-21T15:49:41.473 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.473 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:41.473 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.473 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.473 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.473 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63176 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 108544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:41.473 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.140861+0000", 2024-09-21T15:49:41.473 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.2576301040000004, 2024-09-21T15:49:41.473 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053935394999999997, 2024-09-21T15:49:41.473 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:41.473 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:41.474 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:41.474 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:41.474 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:41.474 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63176 2024-09-21T15:49:41.474 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.474 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:41.474 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.474 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:41.474 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140861+0000", 2024-09-21T15:49:41.474 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.474 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.474 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.474 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:41.474 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140861+0000", 2024-09-21T15:49:41.474 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.475 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.475 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.475 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:41.475 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140860+0000", 2024-09-21T15:49:41.475 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:41.475 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.475 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.475 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:41.475 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140863+0000", 2024-09-21T15:49:41.475 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8710000000000001e-06 2024-09-21T15:49:41.475 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.475 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.475 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:41.475 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140871+0000", 2024-09-21T15:49:41.475 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8089999999999993e-06 2024-09-21T15:49:41.476 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.476 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.476 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:41.476 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140877+0000", 2024-09-21T15:49:41.476 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5969999999999999e-06 2024-09-21T15:49:41.476 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.476 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.476 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:41.476 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.194608+0000", 2024-09-21T15:49:41.476 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053731749000000002 2024-09-21T15:49:41.476 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.476 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.476 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:41.476 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.194697+0000", 2024-09-21T15:49:41.476 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8504000000000005e-05 2024-09-21T15:49:41.477 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.477 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.477 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:41.477 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.194796+0000", 2024-09-21T15:49:41.477 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9414999999999996e-05 2024-09-21T15:49:41.477 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.477 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:41.477 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.477 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.477 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.477 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63182 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 157696~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:41.477 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:49:41.477 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.256357704, 2024-09-21T15:49:41.477 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053958825000000002, 2024-09-21T15:49:41.477 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:41.478 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:41.478 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:41.478 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:41.478 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:41.478 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63182 2024-09-21T15:49:41.478 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.478 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:41.478 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.478 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:41.478 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:49:41.478 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.478 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.478 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.478 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:41.479 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:49:41.479 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.479 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.479 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.479 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:41.479 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:49:41.479 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:41.479 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.479 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.479 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:41.479 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142136+0000", 2024-09-21T15:49:41.479 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8169999999999999e-06 2024-09-21T15:49:41.479 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.479 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.479 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:41.480 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:49:41.480 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3752e-05 2024-09-21T15:49:41.480 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.480 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.480 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:41.480 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142155+0000", 2024-09-21T15:49:41.480 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1859999999999997e-06 2024-09-21T15:49:41.480 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.480 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.480 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:41.480 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.195914+0000", 2024-09-21T15:49:41.480 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053758528999999999 2024-09-21T15:49:41.480 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.480 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.480 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:41.481 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.195995+0000", 2024-09-21T15:49:41.481 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0983999999999996e-05 2024-09-21T15:49:41.481 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.481 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.481 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:41.481 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.196092+0000", 2024-09-21T15:49:41.481 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7069000000000002e-05 2024-09-21T15:49:41.481 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.481 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:41.481 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.481 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.481 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.481 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63193 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 239616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:41.481 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:49:41.482 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.2542701190000001, 2024-09-21T15:49:41.482 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053956577999999998, 2024-09-21T15:49:41.482 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:41.482 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:41.482 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:41.482 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:41.482 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:41.482 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63193 2024-09-21T15:49:41.482 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.482 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:41.482 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.482 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:41.482 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:49:41.482 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.483 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.483 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.483 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:41.483 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:49:41.483 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.483 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.483 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.483 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:41.483 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:49:41.483 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:41.483 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.483 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.483 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:41.483 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144223+0000", 2024-09-21T15:49:41.484 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0990000000000001e-06 2024-09-21T15:49:41.484 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.484 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.484 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:41.484 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144230+0000", 2024-09-21T15:49:41.484 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9070000000000001e-06 2024-09-21T15:49:41.484 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.484 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.484 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:41.484 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144234+0000", 2024-09-21T15:49:41.484 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9010000000000002e-06 2024-09-21T15:49:41.484 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.484 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.484 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:41.484 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.198004+0000", 2024-09-21T15:49:41.485 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053769539999999998 2024-09-21T15:49:41.485 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.485 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.485 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:41.485 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.198083+0000", 2024-09-21T15:49:41.485 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9531999999999997e-05 2024-09-21T15:49:41.485 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.485 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.485 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:41.485 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.198177+0000", 2024-09-21T15:49:41.485 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3906000000000005e-05 2024-09-21T15:49:41.485 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.485 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:41.485 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.486 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.486 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:41.486 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:41.488 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T15:49:41.488 INFO:teuthology.orchestra.run.smithi089.stdout:2812 punch from 0x566027d to 0x566f3ac, (0xf12f bytes) 2024-09-21T15:49:41.488 INFO:teuthology.orchestra.run.smithi089.stdout:2815 read 0xc932b02 thru 0xc93ad03 (0x8202 bytes) 2024-09-21T15:49:41.488 INFO:teuthology.orchestra.run.smithi089.stdout:2816 punch from 0x273a53d to 0x27419ed, (0x74b0 bytes) 2024-09-21T15:49:41.488 INFO:teuthology.orchestra.run.smithi089.stdout:2817 writesame 0x9032bd0 thru 0x9037d12 data_size 0x1(0x5143 bytes) 2024-09-21T15:49:41.489 INFO:teuthology.orchestra.run.smithi089.stdout:2819 writesame 0xbfd796e thru 0xbfd7b46 data_size 0x1(0x1d9 bytes) 2024-09-21T15:49:41.489 INFO:teuthology.orchestra.run.smithi089.stdout:2820 trunc from 0xde32ce8 to 0xd6f3b34 2024-09-21T15:49:41.489 INFO:teuthology.orchestra.run.smithi089.stdout:2821 read 0xa99d6ab thru 0xa9ab4fc (0xde52 bytes) 2024-09-21T15:49:41.489 INFO:teuthology.orchestra.run.smithi089.stdout:2822 trunc from 0xd6f3b34 to 0x52b3117 2024-09-21T15:49:41.489 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:41.486+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:41.492 INFO:teuthology.orchestra.run.smithi089.stdout:2823 write 0x797cb91 thru 0x7988370 (0xb7e0 bytes) 2024-09-21T15:49:41.505 INFO:teuthology.orchestra.run.smithi089.stdout:2824 write 0x240c64 thru 0x24819e (0x753b bytes) 2024-09-21T15:49:41.529 INFO:teuthology.orchestra.run.smithi089.stdout:2825 clone 16 order 18 su 32768 sc 12 2024-09-21T15:49:41.588 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:41.587+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:41.597 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:41.597 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:41.597 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:49:41.597 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:41.605 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T15:49:41.689 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:41.688+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:41.747 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:41.747 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:41.747 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:49:41.748 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:49:41.748 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:41.756 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T15:49:41.789 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:41.789+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:41.849 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T15:49:41.892 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:41.891+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:41.906 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:41.906 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:49:41.906 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:49:41.906 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:49:41.906 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.906 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4209.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:49:41.906 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:49:41.906 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 50.713054554000003, 2024-09-21T15:49:41.906 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.19452909900000001, 2024-09-21T15:49:41.907 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:41.907 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:41.907 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:41.907 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4209", 2024-09-21T15:49:41.907 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/4045143050", 2024-09-21T15:49:41.907 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1 2024-09-21T15:49:41.907 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.907 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:41.907 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.907 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:41.907 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:49:41.907 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.907 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.907 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.908 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:41.908 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:49:41.908 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.908 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.908 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.908 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:41.908 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191161+0000", 2024-09-21T15:49:41.908 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:41.908 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.908 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.908 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:41.908 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191169+0000", 2024-09-21T15:49:41.908 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0090000000000007e-06 2024-09-21T15:49:41.908 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.909 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.909 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:41.909 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191180+0000", 2024-09-21T15:49:41.909 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0728e-05 2024-09-21T15:49:41.909 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.909 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.909 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:41.909 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191187+0000", 2024-09-21T15:49:41.909 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9009999999999994e-06 2024-09-21T15:49:41.909 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.909 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.909 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:41.909 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191238+0000", 2024-09-21T15:49:41.909 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0287000000000002e-05 2024-09-21T15:49:41.909 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.909 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.910 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-21T15:49:41.910 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191245+0000", 2024-09-21T15:49:41.910 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0709999999999997e-06 2024-09-21T15:49:41.910 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.910 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.910 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:41.910 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380309+0000", 2024-09-21T15:49:41.910 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.189064277 2024-09-21T15:49:41.910 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.910 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.910 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:41.910 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380454+0000", 2024-09-21T15:49:41.910 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014490799999999999 2024-09-21T15:49:41.911 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.911 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.911 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0", 2024-09-21T15:49:41.911 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380610+0000", 2024-09-21T15:49:41.911 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015650199999999999 2024-09-21T15:49:41.911 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.911 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.911 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:41.911 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.381160+0000", 2024-09-21T15:49:41.911 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00054903799999999996 2024-09-21T15:49:41.911 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.911 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.911 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:41.911 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385591+0000", 2024-09-21T15:49:41.912 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.004431272 2024-09-21T15:49:41.912 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.912 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.912 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:41.912 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385651+0000", 2024-09-21T15:49:41.912 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0455999999999999e-05 2024-09-21T15:49:41.912 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.912 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.912 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:41.912 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385691+0000", 2024-09-21T15:49:41.912 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0234999999999997e-05 2024-09-21T15:49:41.912 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.912 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:41.912 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.913 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.913 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.913 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64028 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1763328~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:41.913 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167504+0000", 2024-09-21T15:49:41.913 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.7367129979999998, 2024-09-21T15:49:41.913 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047494153999999997, 2024-09-21T15:49:41.913 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:41.913 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:41.913 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:41.913 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:41.913 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:41.913 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64028 2024-09-21T15:49:41.913 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.913 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:41.913 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.914 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:41.914 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167504+0000", 2024-09-21T15:49:41.914 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.914 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.914 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.914 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:41.914 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167504+0000", 2024-09-21T15:49:41.914 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.914 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.914 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.914 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:41.914 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167504+0000", 2024-09-21T15:49:41.914 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:41.914 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.915 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.915 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:41.915 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167506+0000", 2024-09-21T15:49:41.915 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0600000000000002e-06 2024-09-21T15:49:41.915 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.915 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.915 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:41.915 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167512+0000", 2024-09-21T15:49:41.915 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.083e-06 2024-09-21T15:49:41.915 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.915 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.915 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:41.915 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167517+0000", 2024-09-21T15:49:41.915 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7650000000000001e-06 2024-09-21T15:49:41.915 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.916 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.916 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:41.916 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.214835+0000", 2024-09-21T15:49:41.916 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047318067999999998 2024-09-21T15:49:41.916 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.916 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.916 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:41.916 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.214916+0000", 2024-09-21T15:49:41.916 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1815000000000002e-05 2024-09-21T15:49:41.916 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.916 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.916 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:41.916 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.214998+0000", 2024-09-21T15:49:41.916 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1642000000000002e-05 2024-09-21T15:49:41.916 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.917 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:41.917 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.917 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.917 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.917 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64029 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1050624~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:41.917 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167521+0000", 2024-09-21T15:49:41.917 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.7366960870000003, 2024-09-21T15:49:41.917 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047649915000000001, 2024-09-21T15:49:41.917 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:41.917 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:41.917 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:41.917 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:41.917 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:41.917 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64029 2024-09-21T15:49:41.917 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.918 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:41.918 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.918 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:41.918 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167521+0000", 2024-09-21T15:49:41.918 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.918 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.918 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.918 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:41.918 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167521+0000", 2024-09-21T15:49:41.918 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.918 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.918 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.918 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:41.918 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167521+0000", 2024-09-21T15:49:41.919 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:41.919 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.919 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.919 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:41.919 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167523+0000", 2024-09-21T15:49:41.919 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9810000000000002e-06 2024-09-21T15:49:41.919 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.919 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.919 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:41.919 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167529+0000", 2024-09-21T15:49:41.919 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0900000000000001e-06 2024-09-21T15:49:41.919 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.919 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.919 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:41.919 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167533+0000", 2024-09-21T15:49:41.920 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5730000000000001e-06 2024-09-21T15:49:41.920 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.920 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.920 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:41.920 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215031+0000", 2024-09-21T15:49:41.920 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047497893999999999 2024-09-21T15:49:41.920 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.920 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.920 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:41.920 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215099+0000", 2024-09-21T15:49:41.920 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8416000000000004e-05 2024-09-21T15:49:41.920 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.920 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.920 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:41.921 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215171+0000", 2024-09-21T15:49:41.921 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1279000000000007e-05 2024-09-21T15:49:41.921 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.921 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:41.921 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.921 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.921 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.921 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64030 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1058816~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:41.921 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167537+0000", 2024-09-21T15:49:41.921 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.7366795780000004, 2024-09-21T15:49:41.921 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047783816999999999, 2024-09-21T15:49:41.921 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:41.921 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:41.921 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:41.921 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:41.922 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:41.922 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64030 2024-09-21T15:49:41.922 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.922 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:41.922 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.922 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:41.922 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167537+0000", 2024-09-21T15:49:41.922 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.922 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.922 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.922 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:41.922 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167537+0000", 2024-09-21T15:49:41.922 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.922 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.923 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.923 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:41.923 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167537+0000", 2024-09-21T15:49:41.923 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:41.923 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.923 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.923 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:41.923 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167539+0000", 2024-09-21T15:49:41.923 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9700000000000002e-06 2024-09-21T15:49:41.923 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.923 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.923 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:41.923 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167545+0000", 2024-09-21T15:49:41.923 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8629999999999996e-06 2024-09-21T15:49:41.924 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.924 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.924 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:41.924 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167549+0000", 2024-09-21T15:49:41.924 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.386e-06 2024-09-21T15:49:41.924 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.924 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.924 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:41.924 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215197+0000", 2024-09-21T15:49:41.924 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047647241 2024-09-21T15:49:41.924 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.924 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.924 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:41.924 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215263+0000", 2024-09-21T15:49:41.924 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6295000000000001e-05 2024-09-21T15:49:41.925 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.925 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.925 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:41.925 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215321+0000", 2024-09-21T15:49:41.925 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8362000000000001e-05 2024-09-21T15:49:41.925 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.925 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:41.925 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.925 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.925 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.925 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64031 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1738752~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:41.925 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167554+0000", 2024-09-21T15:49:41.925 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.7366630750000001, 2024-09-21T15:49:41.925 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047917855000000002, 2024-09-21T15:49:41.925 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:41.926 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:41.926 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:41.926 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:41.926 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:41.926 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64031 2024-09-21T15:49:41.926 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.926 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:41.926 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.926 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:41.926 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167554+0000", 2024-09-21T15:49:41.926 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.926 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.926 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.926 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:41.926 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167554+0000", 2024-09-21T15:49:41.927 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.927 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.927 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.927 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:41.927 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167554+0000", 2024-09-21T15:49:41.927 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:41.927 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.927 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.927 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:41.927 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167556+0000", 2024-09-21T15:49:41.927 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3870000000000002e-06 2024-09-21T15:49:41.927 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.927 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.927 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:41.928 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167562+0000", 2024-09-21T15:49:41.928 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9100000000000002e-06 2024-09-21T15:49:41.928 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.928 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.928 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:41.928 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167566+0000", 2024-09-21T15:49:41.928 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6029999999999998e-06 2024-09-21T15:49:41.928 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.928 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.928 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:41.928 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215346+0000", 2024-09-21T15:49:41.928 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.04777998 2024-09-21T15:49:41.928 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.928 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.928 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:41.929 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215412+0000", 2024-09-21T15:49:41.929 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5598000000000005e-05 2024-09-21T15:49:41.929 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.929 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.929 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:41.929 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215472+0000", 2024-09-21T15:49:41.929 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9728000000000001e-05 2024-09-21T15:49:41.929 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.929 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:41.929 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.929 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.929 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.929 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64032 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1067008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:41.929 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167673+0000", 2024-09-21T15:49:41.929 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.7365443620000001, 2024-09-21T15:49:41.930 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047947667999999999, 2024-09-21T15:49:41.930 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:41.930 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:41.930 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:41.930 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:41.930 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:41.930 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64032 2024-09-21T15:49:41.930 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.930 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:41.930 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.930 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:41.930 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167673+0000", 2024-09-21T15:49:41.930 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.930 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.931 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.931 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:41.931 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167673+0000", 2024-09-21T15:49:41.931 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.931 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.931 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.931 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:41.931 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167672+0000", 2024-09-21T15:49:41.931 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:41.931 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.931 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.931 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:41.931 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167675+0000", 2024-09-21T15:49:41.931 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3780000000000001e-06 2024-09-21T15:49:41.931 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.932 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.932 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:41.932 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167685+0000", 2024-09-21T15:49:41.932 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3039999999999994e-06 2024-09-21T15:49:41.932 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.932 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.932 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:41.932 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167692+0000", 2024-09-21T15:49:41.932 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2559999999999996e-06 2024-09-21T15:49:41.932 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.932 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.932 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:41.932 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215495+0000", 2024-09-21T15:49:41.932 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047803267000000003 2024-09-21T15:49:41.933 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.933 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.933 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:41.933 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215560+0000", 2024-09-21T15:49:41.933 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4703999999999996e-05 2024-09-21T15:49:41.933 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.933 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.933 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:41.933 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215620+0000", 2024-09-21T15:49:41.933 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0328000000000002e-05 2024-09-21T15:49:41.933 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.933 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:41.933 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.933 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.934 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.934 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64033 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1075200~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:41.934 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167697+0000", 2024-09-21T15:49:41.934 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.73651979, 2024-09-21T15:49:41.934 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048082863000000003, 2024-09-21T15:49:41.934 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:41.934 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:41.934 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:41.934 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:41.934 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:41.934 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64033 2024-09-21T15:49:41.934 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.934 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:41.934 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.935 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:41.935 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167697+0000", 2024-09-21T15:49:41.935 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.935 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.935 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.935 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:41.935 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167697+0000", 2024-09-21T15:49:41.935 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.935 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.935 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.935 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:41.935 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167697+0000", 2024-09-21T15:49:41.935 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:41.935 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.935 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.936 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:41.936 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167699+0000", 2024-09-21T15:49:41.936 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5299999999999999e-06 2024-09-21T15:49:41.936 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.936 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.936 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:41.936 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167706+0000", 2024-09-21T15:49:41.936 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7490000000000001e-06 2024-09-21T15:49:41.936 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.936 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.936 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:41.936 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167712+0000", 2024-09-21T15:49:41.936 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1940000000000003e-06 2024-09-21T15:49:41.936 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.936 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.937 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:41.937 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215643+0000", 2024-09-21T15:49:41.937 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047930647999999999 2024-09-21T15:49:41.937 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.937 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.937 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:41.937 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215713+0000", 2024-09-21T15:49:41.937 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0488999999999995e-05 2024-09-21T15:49:41.937 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.937 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.937 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:41.937 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215780+0000", 2024-09-21T15:49:41.937 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6656999999999996e-05 2024-09-21T15:49:41.937 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.937 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:41.938 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.938 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.938 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.938 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64034 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1083392~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:41.938 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167717+0000", 2024-09-21T15:49:41.938 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.7365002240000003, 2024-09-21T15:49:41.938 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048233316999999998, 2024-09-21T15:49:41.938 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:41.938 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:41.938 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:41.938 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:41.938 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:41.938 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64034 2024-09-21T15:49:41.938 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.939 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:41.939 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.939 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:41.939 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167717+0000", 2024-09-21T15:49:41.939 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.939 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.939 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.939 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:41.939 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167717+0000", 2024-09-21T15:49:41.939 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.939 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.939 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.939 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:41.939 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167716+0000", 2024-09-21T15:49:41.939 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:41.940 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.940 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.940 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:41.940 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167718+0000", 2024-09-21T15:49:41.940 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.165e-06 2024-09-21T15:49:41.940 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.940 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.940 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:41.940 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167725+0000", 2024-09-21T15:49:41.940 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3380000000000001e-06 2024-09-21T15:49:41.940 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.940 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.940 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:41.940 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167730+0000", 2024-09-21T15:49:41.940 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8010000000000003e-06 2024-09-21T15:49:41.941 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.941 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.941 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:41.941 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215807+0000", 2024-09-21T15:49:41.941 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048077823999999998 2024-09-21T15:49:41.941 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.941 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.941 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:41.941 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215884+0000", 2024-09-21T15:49:41.941 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6283000000000001e-05 2024-09-21T15:49:41.941 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.941 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.941 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:41.941 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215950+0000", 2024-09-21T15:49:41.941 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6290000000000004e-05 2024-09-21T15:49:41.941 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.942 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:41.942 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.942 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.942 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.942 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64035 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1091584~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:41.942 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:49:41.942 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.7364830109999998, 2024-09-21T15:49:41.942 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048412117999999997, 2024-09-21T15:49:41.942 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:41.942 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:41.942 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:41.942 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:41.942 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:41.942 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64035 2024-09-21T15:49:41.943 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.943 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:41.943 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.943 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:41.943 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:49:41.943 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.943 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.943 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.943 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:41.943 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:49:41.943 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.943 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.943 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.943 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:41.943 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:49:41.944 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:41.944 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.944 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.944 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:41.944 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167746+0000", 2024-09-21T15:49:41.944 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2031000000000001e-05 2024-09-21T15:49:41.944 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.944 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.944 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:41.944 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167753+0000", 2024-09-21T15:49:41.944 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0620000000000003e-06 2024-09-21T15:49:41.944 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.944 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.944 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:41.944 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167759+0000", 2024-09-21T15:49:41.945 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.596e-06 2024-09-21T15:49:41.945 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.945 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.945 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:41.945 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215977+0000", 2024-09-21T15:49:41.945 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048217318000000002 2024-09-21T15:49:41.945 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.945 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.945 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:41.945 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216076+0000", 2024-09-21T15:49:41.945 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9846000000000006e-05 2024-09-21T15:49:41.945 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.945 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.945 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:41.946 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216146+0000", 2024-09-21T15:49:41.946 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9578000000000004e-05 2024-09-21T15:49:41.946 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.946 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:41.946 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.946 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.946 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.946 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64036 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1099776~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:41.946 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167853+0000", 2024-09-21T15:49:41.946 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.7363643980000001, 2024-09-21T15:49:41.946 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048450675999999998, 2024-09-21T15:49:41.946 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:41.946 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:41.946 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:41.947 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:41.947 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:41.947 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64036 2024-09-21T15:49:41.947 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.947 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:41.947 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.947 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:41.947 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167853+0000", 2024-09-21T15:49:41.947 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.947 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.947 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.947 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:41.947 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167853+0000", 2024-09-21T15:49:41.947 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.948 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.948 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.948 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:41.948 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167852+0000", 2024-09-21T15:49:41.948 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:41.948 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.948 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.948 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:41.948 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167855+0000", 2024-09-21T15:49:41.948 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5640000000000001e-06 2024-09-21T15:49:41.948 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.948 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.948 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:41.948 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167864+0000", 2024-09-21T15:49:41.949 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8140000000000007e-06 2024-09-21T15:49:41.949 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.949 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.949 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:41.949 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167871+0000", 2024-09-21T15:49:41.949 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5729999999999996e-06 2024-09-21T15:49:41.949 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.949 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.949 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:41.949 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216174+0000", 2024-09-21T15:49:41.949 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048303437999999997 2024-09-21T15:49:41.949 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.949 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.949 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:41.949 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216250+0000", 2024-09-21T15:49:41.950 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5661999999999995e-05 2024-09-21T15:49:41.950 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.950 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.950 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:41.950 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216303+0000", 2024-09-21T15:49:41.950 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3273000000000003e-05 2024-09-21T15:49:41.950 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.950 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:41.950 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.950 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.950 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.950 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64037 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1771520~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:41.950 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:49:41.950 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.7363408549999999, 2024-09-21T15:49:41.951 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048588771000000003, 2024-09-21T15:49:41.951 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:41.951 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:41.951 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:41.951 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:41.951 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:41.951 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64037 2024-09-21T15:49:41.951 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.951 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:41.951 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.951 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:41.951 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:49:41.951 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.951 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.951 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.952 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:41.952 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:49:41.952 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.952 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.952 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.952 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:41.952 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:49:41.952 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:41.952 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.952 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.952 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:41.952 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167878+0000", 2024-09-21T15:49:41.952 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3e-06 2024-09-21T15:49:41.952 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.952 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.953 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:41.953 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167884+0000", 2024-09-21T15:49:41.953 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3269999999999997e-06 2024-09-21T15:49:41.953 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.953 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.953 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:41.953 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167889+0000", 2024-09-21T15:49:41.953 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9269999999999996e-06 2024-09-21T15:49:41.953 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.953 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.953 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:41.953 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216331+0000", 2024-09-21T15:49:41.953 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048442102000000001 2024-09-21T15:49:41.953 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.954 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.954 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:41.954 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216393+0000", 2024-09-21T15:49:41.954 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1132999999999999e-05 2024-09-21T15:49:41.954 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.954 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.954 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:41.954 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216465+0000", 2024-09-21T15:49:41.954 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2339e-05 2024-09-21T15:49:41.954 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.954 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:41.954 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.954 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.954 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.955 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64038 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1779712~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:41.955 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:49:41.955 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.7363230239999998, 2024-09-21T15:49:41.955 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048749795999999998, 2024-09-21T15:49:41.955 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:41.955 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:41.955 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:41.955 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:41.955 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:41.955 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64038 2024-09-21T15:49:41.955 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.955 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:41.955 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.955 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:41.956 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:49:41.956 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.956 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.956 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.956 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:41.956 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:49:41.956 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.956 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.956 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.956 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:41.956 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:49:41.956 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:41.956 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.956 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.956 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:41.957 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167896+0000", 2024-09-21T15:49:41.957 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2419999999999999e-06 2024-09-21T15:49:41.957 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.957 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.957 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:41.957 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167902+0000", 2024-09-21T15:49:41.957 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0859999999999998e-06 2024-09-21T15:49:41.957 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.957 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.957 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:41.957 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167907+0000", 2024-09-21T15:49:41.957 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9679999999999997e-06 2024-09-21T15:49:41.957 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.957 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.957 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:41.957 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216506+0000", 2024-09-21T15:49:41.958 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048599374000000001 2024-09-21T15:49:41.958 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.958 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.958 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:41.958 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216579+0000", 2024-09-21T15:49:41.958 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2801999999999999e-05 2024-09-21T15:49:41.958 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.958 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.958 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:41.958 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216644+0000", 2024-09-21T15:49:41.958 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4715000000000006e-05 2024-09-21T15:49:41.958 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.958 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:41.958 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.958 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.959 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.959 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64039 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1107968~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:41.959 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:49:41.959 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.7363056690000001, 2024-09-21T15:49:41.959 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048911370000000003, 2024-09-21T15:49:41.959 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:41.959 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:41.959 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:41.959 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:41.959 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:41.959 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64039 2024-09-21T15:49:41.959 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.959 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:41.959 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.959 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:41.960 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:49:41.960 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.960 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.960 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.961 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:41.961 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:49:41.961 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.961 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.961 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.961 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:41.962 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:49:41.962 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:41.962 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.962 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.962 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:41.962 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167913+0000", 2024-09-21T15:49:41.962 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0839999999999999e-06 2024-09-21T15:49:41.962 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.962 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.962 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:41.962 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167921+0000", 2024-09-21T15:49:41.962 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8310000000000001e-06 2024-09-21T15:49:41.962 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.962 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.962 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:41.963 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167926+0000", 2024-09-21T15:49:41.963 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0270000000000003e-06 2024-09-21T15:49:41.963 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.963 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.963 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:41.963 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216680+0000", 2024-09-21T15:49:41.963 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048753890000000001 2024-09-21T15:49:41.963 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.963 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.963 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:41.963 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216757+0000", 2024-09-21T15:49:41.963 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6848999999999996e-05 2024-09-21T15:49:41.963 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.963 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.963 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:41.964 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216823+0000", 2024-09-21T15:49:41.964 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6068000000000005e-05 2024-09-21T15:49:41.964 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.964 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:41.964 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.964 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.964 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.964 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64040 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1787904~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:41.964 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167931+0000", 2024-09-21T15:49:41.964 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.7362864370000004, 2024-09-21T15:49:41.964 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049010561000000001, 2024-09-21T15:49:41.964 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:41.964 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:41.964 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:41.965 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:41.965 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:41.965 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64040 2024-09-21T15:49:41.965 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.965 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:41.965 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.965 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:41.965 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167931+0000", 2024-09-21T15:49:41.965 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.965 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.965 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.965 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:41.965 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167931+0000", 2024-09-21T15:49:41.965 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.966 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.966 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.966 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:41.966 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167930+0000", 2024-09-21T15:49:41.966 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:41.966 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.966 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.966 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:41.966 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167932+0000", 2024-09-21T15:49:41.966 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.198e-06 2024-09-21T15:49:41.966 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.966 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.966 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:41.966 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167938+0000", 2024-09-21T15:49:41.967 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.105e-06 2024-09-21T15:49:41.967 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.967 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.967 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:41.967 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167943+0000", 2024-09-21T15:49:41.967 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9429999999999999e-06 2024-09-21T15:49:41.967 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.967 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.967 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:41.967 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216848+0000", 2024-09-21T15:49:41.967 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048905200000000003 2024-09-21T15:49:41.967 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.967 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.967 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:41.967 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216903+0000", 2024-09-21T15:49:41.968 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4793999999999997e-05 2024-09-21T15:49:41.968 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.968 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.968 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:41.968 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216941+0000", 2024-09-21T15:49:41.968 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7790999999999998e-05 2024-09-21T15:49:41.968 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.968 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:41.968 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.968 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.968 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.968 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64041 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1796096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:41.968 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:49:41.968 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.7362694659999995, 2024-09-21T15:49:41.968 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049181457999999997, 2024-09-21T15:49:41.969 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:41.969 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:41.969 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:41.969 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:41.969 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:41.969 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64041 2024-09-21T15:49:41.969 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.969 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:41.969 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.969 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:41.969 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:49:41.969 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.969 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.969 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.969 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:41.970 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:49:41.970 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.970 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.970 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.970 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:41.970 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:49:41.970 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:41.970 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.970 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.970 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:41.970 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167949+0000", 2024-09-21T15:49:41.970 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0200000000000001e-06 2024-09-21T15:49:41.970 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.971 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.971 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:41.971 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167955+0000", 2024-09-21T15:49:41.971 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5990000000000001e-06 2024-09-21T15:49:41.971 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.971 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.971 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:41.971 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167959+0000", 2024-09-21T15:49:41.971 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5589999999999999e-06 2024-09-21T15:49:41.971 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.971 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.971 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:41.971 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216972+0000", 2024-09-21T15:49:41.971 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049012574000000003 2024-09-21T15:49:41.971 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.972 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.972 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:41.972 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217057+0000", 2024-09-21T15:49:41.972 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4820000000000004e-05 2024-09-21T15:49:41.972 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.972 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.972 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:41.972 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217129+0000", 2024-09-21T15:49:41.972 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2186000000000003e-05 2024-09-21T15:49:41.972 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.972 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:41.972 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.972 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.972 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.972 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64042 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1804288~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:41.973 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167964+0000", 2024-09-21T15:49:41.973 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.7362531749999999, 2024-09-21T15:49:41.973 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049333372, 2024-09-21T15:49:41.973 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:41.973 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:41.973 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:41.973 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:41.973 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:41.973 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64042 2024-09-21T15:49:41.973 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.973 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:41.973 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.973 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:41.973 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167964+0000", 2024-09-21T15:49:41.974 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.974 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.974 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.974 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:41.974 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167964+0000", 2024-09-21T15:49:41.974 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.974 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.974 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.974 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:41.974 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167963+0000", 2024-09-21T15:49:41.974 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:41.974 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.974 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.974 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:41.975 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167966+0000", 2024-09-21T15:49:41.975 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1119999999999999e-06 2024-09-21T15:49:41.975 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.975 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.975 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:41.975 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167975+0000", 2024-09-21T15:49:41.975 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1130000000000007e-06 2024-09-21T15:49:41.975 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.975 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.975 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:41.975 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167980+0000", 2024-09-21T15:49:41.975 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9599999999999999e-06 2024-09-21T15:49:41.975 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.975 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.975 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:41.976 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217158+0000", 2024-09-21T15:49:41.976 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049178569999999998 2024-09-21T15:49:41.976 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.976 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.976 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:41.976 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217232+0000", 2024-09-21T15:49:41.976 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3589000000000003e-05 2024-09-21T15:49:41.976 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.976 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.976 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:41.976 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217297+0000", 2024-09-21T15:49:41.976 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5339999999999994e-05 2024-09-21T15:49:41.976 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.976 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:41.977 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.977 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.977 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.977 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64043 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1812480~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:41.977 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.168085+0000", 2024-09-21T15:49:41.977 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.7361321079999996, 2024-09-21T15:49:41.977 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049365157999999999, 2024-09-21T15:49:41.977 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:41.977 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:41.977 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:41.977 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:41.977 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:41.977 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64043 2024-09-21T15:49:41.977 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.978 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:41.978 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.978 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:41.978 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168085+0000", 2024-09-21T15:49:41.978 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.978 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.978 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.978 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:41.978 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168085+0000", 2024-09-21T15:49:41.978 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.978 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.978 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.978 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:41.978 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168084+0000", 2024-09-21T15:49:41.979 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:41.979 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.979 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.979 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:41.979 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168088+0000", 2024-09-21T15:49:41.979 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5420000000000001e-06 2024-09-21T15:49:41.979 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.979 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.979 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:41.979 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168097+0000", 2024-09-21T15:49:41.979 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5410000000000006e-06 2024-09-21T15:49:41.979 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.979 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.979 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:41.979 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168104+0000", 2024-09-21T15:49:41.979 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6719999999999998e-06 2024-09-21T15:49:41.980 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.980 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.980 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:41.980 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217323+0000", 2024-09-21T15:49:41.980 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049218857999999997 2024-09-21T15:49:41.980 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.980 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.980 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:41.980 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217397+0000", 2024-09-21T15:49:41.980 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3932999999999998e-05 2024-09-21T15:49:41.980 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.980 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.980 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:41.980 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217450+0000", 2024-09-21T15:49:41.980 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3168000000000003e-05 2024-09-21T15:49:41.981 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.981 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:41.981 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.981 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.981 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.981 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64044 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1820672~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:41.981 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.170036+0000", 2024-09-21T15:49:41.981 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.7341807889999998, 2024-09-21T15:49:41.981 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047578069000000001, 2024-09-21T15:49:41.981 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:41.981 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:41.981 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:41.981 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:41.981 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:41.981 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64044 2024-09-21T15:49:41.981 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.982 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:41.982 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.982 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:41.982 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170036+0000", 2024-09-21T15:49:41.982 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.982 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.982 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.982 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:41.982 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170036+0000", 2024-09-21T15:49:41.982 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.982 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.982 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.982 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:41.983 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168109+0000", 2024-09-21T15:49:41.983 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9980721 2024-09-21T15:49:41.983 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.983 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.983 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:41.983 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170040+0000", 2024-09-21T15:49:41.983 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0019318079999999999 2024-09-21T15:49:41.983 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.983 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.983 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:41.983 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170051+0000", 2024-09-21T15:49:41.983 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0220999999999999e-05 2024-09-21T15:49:41.983 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.983 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.983 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:41.984 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170058+0000", 2024-09-21T15:49:41.984 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0110000000000003e-06 2024-09-21T15:49:41.984 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.984 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.984 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:41.984 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217478+0000", 2024-09-21T15:49:41.984 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047420334000000001 2024-09-21T15:49:41.984 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.984 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.984 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:41.984 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217547+0000", 2024-09-21T15:49:41.984 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9416000000000001e-05 2024-09-21T15:49:41.984 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.984 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.985 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:41.985 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217614+0000", 2024-09-21T15:49:41.985 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6909000000000006e-05 2024-09-21T15:49:41.985 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.985 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:41.985 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.985 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.985 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.985 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64045 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1828864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:41.985 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.170064+0000", 2024-09-21T15:49:41.985 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.7341527499999998, 2024-09-21T15:49:41.985 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047709252000000001, 2024-09-21T15:49:41.985 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:41.985 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:41.986 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:41.986 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:41.986 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:41.986 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64045 2024-09-21T15:49:41.986 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.986 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:41.986 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.986 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:41.986 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170064+0000", 2024-09-21T15:49:41.986 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.986 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.986 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.986 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:41.986 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170064+0000", 2024-09-21T15:49:41.987 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.987 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.987 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.987 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:41.987 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170064+0000", 2024-09-21T15:49:41.987 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:41.987 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.987 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.987 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:41.987 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170066+0000", 2024-09-21T15:49:41.987 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8109999999999999e-06 2024-09-21T15:49:41.987 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.987 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.987 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:41.988 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170073+0000", 2024-09-21T15:49:41.988 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5570000000000001e-06 2024-09-21T15:49:41.988 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.988 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.988 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:41.988 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170078+0000", 2024-09-21T15:49:41.988 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9910000000000002e-06 2024-09-21T15:49:41.988 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.988 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.988 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:41.988 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217640+0000", 2024-09-21T15:49:41.988 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047561840000000001 2024-09-21T15:49:41.988 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.988 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.988 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:41.989 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217710+0000", 2024-09-21T15:49:41.989 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9967000000000003e-05 2024-09-21T15:49:41.989 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.989 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.989 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:41.989 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217773+0000", 2024-09-21T15:49:41.989 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3638000000000003e-05 2024-09-21T15:49:41.989 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.989 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:41.989 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.989 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.989 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.989 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64356 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 3076096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:41.989 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.183065+0000", 2024-09-21T15:49:41.989 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.7211516190000005, 2024-09-21T15:49:41.990 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047484923999999998, 2024-09-21T15:49:41.990 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:41.990 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:41.990 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:41.990 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:41.990 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:41.990 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64356 2024-09-21T15:49:41.990 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.990 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:41.990 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.990 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:41.990 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.183065+0000", 2024-09-21T15:49:41.990 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.990 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.991 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.991 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:41.991 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.183065+0000", 2024-09-21T15:49:41.991 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:41.991 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.991 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.991 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:41.991 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.182922+0000", 2024-09-21T15:49:41.991 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999856 2024-09-21T15:49:41.991 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.991 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.991 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:41.991 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.183069+0000", 2024-09-21T15:49:41.991 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000147097 2024-09-21T15:49:41.991 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.992 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.992 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:41.992 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.183078+0000", 2024-09-21T15:49:41.992 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.9099999999999994e-06 2024-09-21T15:49:41.992 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.992 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.992 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:41.992 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.183085+0000", 2024-09-21T15:49:41.992 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0670000000000003e-06 2024-09-21T15:49:41.992 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.992 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.992 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:41.992 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.230377+0000", 2024-09-21T15:49:41.992 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047292698000000001 2024-09-21T15:49:41.993 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.993 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.993 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:41.993 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.230458+0000", 2024-09-21T15:49:41.993 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0980000000000001e-05 2024-09-21T15:49:41.993 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:41.993 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:41.993 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:41.993 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.230550+0000", 2024-09-21T15:49:41.993 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1954999999999996e-05 2024-09-21T15:49:41.993 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.993 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:41.993 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.993 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:41.993 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:41.994 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:41.994 INFO:teuthology.orchestra.run.smithi089.stdout:leaving image image_client.0-clone15 intact 2024-09-21T15:49:41.995 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:41.993+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:41.995 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:49:42.061 INFO:teuthology.orchestra.run.smithi089.stdout:checking clone #14, image image_client.0-clone14 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent15 2024-09-21T15:49:42.087 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:49:42.138 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:42.138 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:49:42.138 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:42.138 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:86050 3.12 3:49e547e3:::rbd_data.11331c75f7c4.0000000000000008:11 [read 4046848~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-21T15:49:42.138 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:42.131340+0000", 2024-09-21T15:49:42.138 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.00091345700000000001, 2024-09-21T15:49:42.139 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00094953799999999999, 2024-09-21T15:49:42.139 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:42.139 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:42.139 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:42.139 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:42.139 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:42.139 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 86050 2024-09-21T15:49:42.139 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:42.139 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:42.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:42.139 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:42.139 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:42.131340+0000", 2024-09-21T15:49:42.139 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:42.139 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:42.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:42.140 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:42.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:42.131340+0000", 2024-09-21T15:49:42.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:42.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:42.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:42.140 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:42.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:42.131339+0000", 2024-09-21T15:49:42.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:42.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:42.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:42.140 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:42.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:42.131342+0000", 2024-09-21T15:49:42.141 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.153e-06 2024-09-21T15:49:42.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:42.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:42.141 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:42.141 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:42.131348+0000", 2024-09-21T15:49:42.141 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2400000000000004e-06 2024-09-21T15:49:42.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:42.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:42.141 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:42.141 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:42.131353+0000", 2024-09-21T15:49:42.141 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.797e-06 2024-09-21T15:49:42.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:42.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:42.141 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:42.141 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:42.132124+0000", 2024-09-21T15:49:42.142 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00077141700000000004 2024-09-21T15:49:42.142 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:42.142 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:42.142 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:42.142 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:42.132193+0000", 2024-09-21T15:49:42.142 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9466000000000002e-05 2024-09-21T15:49:42.142 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:42.142 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:42.142 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:42.142 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:42.142 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:42.142 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:86051 3.12 3.c7e2a792 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e51)", 2024-09-21T15:49:42.142 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:42.131357+0000", 2024-09-21T15:49:42.142 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.00089639199999999998, 2024-09-21T15:49:42.143 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001008611, 2024-09-21T15:49:42.143 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:42.143 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-09-21T15:49:42.143 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:42.143 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:42.143 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:42.143 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 86051 2024-09-21T15:49:42.143 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:42.143 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:42.143 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:42.143 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:42.143 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:42.131357+0000", 2024-09-21T15:49:42.143 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:42.143 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:42.144 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:42.144 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:42.144 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:42.131357+0000", 2024-09-21T15:49:42.144 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:42.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:42.144 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:42.144 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:42.144 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:42.131356+0000", 2024-09-21T15:49:42.144 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:42.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:42.144 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:42.144 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:42.144 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:42.131359+0000", 2024-09-21T15:49:42.144 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1019999999999999e-06 2024-09-21T15:49:42.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:42.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:42.145 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:42.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:42.131365+0000", 2024-09-21T15:49:42.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1999999999999999e-06 2024-09-21T15:49:42.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:42.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:42.145 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:42.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:42.131376+0000", 2024-09-21T15:49:42.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1708e-05 2024-09-21T15:49:42.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:42.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:42.145 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:42.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:42.132350+0000", 2024-09-21T15:49:42.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00097390300000000003 2024-09-21T15:49:42.146 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:42.146 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:42.146 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:42.146 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:42.146 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-21T15:49:42.146 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 2 2024-09-21T15:49:42.146 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:42.146 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:42.137+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:42.151 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:49:42.196 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:42.195+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:42.296 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:42.295+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:42.299 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-09-21T15:49:42.299 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:49:42.317 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:49:42.318 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:49:42.333 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:49:42.333 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:49:42.338 INFO:teuthology.orchestra.run.smithi089.stdout:2826 write 0x5c2da9a thru 0x5c35585 (0x7aec bytes) 2024-09-21T15:49:42.341 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:42.341 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:42.351 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:49:42.351 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:49:42.352 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:49:42.352 INFO:teuthology.orchestra.run.smithi089.stdout:2827 read 0x3dfdbaf thru 0x3e07ca7 (0xa0f9 bytes) 2024-09-21T15:49:42.365 INFO:teuthology.orchestra.run.smithi089.stdout:2828 write 0xc52946a thru 0xc537d14 (0xe8ab bytes) 2024-09-21T15:49:42.370 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:49:42.370 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:49:42.383 INFO:teuthology.orchestra.run.smithi089.stdout:2830 write 0xebf8fa9 thru 0xec02e77 (0x9ecf bytes) 2024-09-21T15:49:42.388 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:49:42.388 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:49:42.389 INFO:teuthology.orchestra.run.smithi089.stdout:2831 write 0xccd4537 thru 0xccd52df (0xda9 bytes) 2024-09-21T15:49:42.394 INFO:teuthology.orchestra.run.smithi089.stdout:2832 read 0x2cb39d8 thru 0x2cbf48d (0xbab6 bytes) 2024-09-21T15:49:42.395 INFO:teuthology.orchestra.run.smithi089.stdout:2834 writesame 0x2850c15 thru 0x28544c4 data_size 0x8(0x38b0 bytes) 2024-09-21T15:49:42.397 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:42.396+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:42.401 INFO:teuthology.orchestra.run.smithi089.stdout:2835 writesame 0x53ed26a thru 0x53f4c29 data_size 0x20(0x79c0 bytes) 2024-09-21T15:49:42.407 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:49:42.408 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:49:42.409 INFO:teuthology.orchestra.run.smithi089.stdout:2841 punch from 0x88c42d3 to 0x88d2376, (0xe0a3 bytes) 2024-09-21T15:49:42.409 INFO:teuthology.orchestra.run.smithi089.stdout:2842 read 0xaaa4ce thru 0xaac54b (0x207e bytes) 2024-09-21T15:49:42.410 INFO:teuthology.orchestra.run.smithi089.stdout:2844 read 0x6829257 thru 0x6834bf9 (0xb9a3 bytes) 2024-09-21T15:49:42.410 INFO:teuthology.orchestra.run.smithi089.stdout:2845 read 0x73b29c thru 0x74006c (0x4dd1 bytes) 2024-09-21T15:49:42.411 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:42.411 INFO:teuthology.orchestra.run.smithi089.stdout:2847 read 0x8d4e92d thru 0x8d5af6a (0xc63e bytes) 2024-09-21T15:49:42.411 INFO:teuthology.orchestra.run.smithi089.stdout:2849 read 0xc95da4b thru 0xc963bed (0x61a3 bytes) 2024-09-21T15:49:42.411 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:42.411 INFO:teuthology.orchestra.run.smithi089.stdout:2851 read 0xe0b8721 thru 0xe0c30d1 (0xa9b1 bytes) 2024-09-21T15:49:42.411 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:42.411 INFO:teuthology.orchestra.run.smithi089.stdout:2853 read 0xafe228 thru 0xb0d25b (0xf034 bytes) 2024-09-21T15:49:42.412 INFO:teuthology.orchestra.run.smithi089.stdout:2855 punch from 0xead0eb7 to 0xead2323, (0x146c bytes) 2024-09-21T15:49:42.412 INFO:teuthology.orchestra.run.smithi089.stdout:2858 write 0x660d4bc thru 0x661cf1c (0xfa61 bytes) 2024-09-21T15:49:42.420 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:42.420 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:42.420 INFO:teuthology.orchestra.run.smithi089.stdout:2861 write 0x68b3a2 thru 0x69a301 (0xef60 bytes) 2024-09-21T15:49:42.426 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:49:42.427 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:49:42.434 INFO:teuthology.orchestra.run.smithi089.stdout:2863 write 0xbd37812 thru 0xbd40221 (0x8a10 bytes) 2024-09-21T15:49:42.439 INFO:teuthology.orchestra.run.smithi089.stdout:2866 writesame 0x36c6c3c thru 0x36d1398 data_size 0x5f(0xa75d bytes) 2024-09-21T15:49:42.449 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:42.449 INFO:teuthology.orchestra.run.smithi089.stdout:2869 read 0x2654942 thru 0x265f63a (0xacf9 bytes) 2024-09-21T15:49:42.451 INFO:teuthology.orchestra.run.smithi089.stdout:2870 punch from 0xe96e1c9 to 0xe97520d, (0x7044 bytes) 2024-09-21T15:49:42.451 INFO:teuthology.orchestra.run.smithi089.stdout:2871 read 0x6d3013b thru 0x6d39884 (0x974a bytes) 2024-09-21T15:49:42.464 INFO:teuthology.orchestra.run.smithi089.stdout:2872 trunc from 0xec02e78 to 0x3104bf0 2024-09-21T15:49:42.477 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:42.477 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:42.489 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:49:42.499 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:42.497+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:42.510 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:42.510 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:42.527 INFO:teuthology.orchestra.run.smithi089.stdout:2878 write 0xd8c2835 thru 0xd8d1eb6 (0xf682 bytes) 2024-09-21T15:49:42.537 INFO:teuthology.orchestra.run.smithi089.stdout:2879 punch from 0xc2c8607 to 0xc2d357c, (0xaf75 bytes) 2024-09-21T15:49:42.538 INFO:teuthology.orchestra.run.smithi089.stdout:2881 trunc from 0xd8d1eb7 to 0xcb64a02 2024-09-21T15:49:42.555 INFO:teuthology.orchestra.run.smithi089.stdout:2882 trunc from 0xcb64a02 to 0xc18309e 2024-09-21T15:49:42.561 INFO:teuthology.orchestra.run.smithi089.stdout:2883 write 0x52cb7f0 thru 0x52d8cb9 (0xd4ca bytes) 2024-09-21T15:49:42.575 INFO:teuthology.orchestra.run.smithi089.stdout:2884 writesame 0xd29c42c thru 0xd29d02d data_size 0x1d(0xc02 bytes) 2024-09-21T15:49:42.575 INFO:teuthology.orchestra.run.smithi089.stdout:2885 punch from 0x217b85d to 0x2182476, (0x6c19 bytes) 2024-09-21T15:49:42.575 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:42.592 INFO:teuthology.orchestra.run.smithi089.stdout:2887 trunc from 0xd29d02e to 0x975141 2024-09-21T15:49:42.600 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:42.598+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:42.645 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:42.645 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:42.651 INFO:teuthology.orchestra.run.smithi089.stdout:2888 write 0x3c7b623 thru 0x3c7fc98 (0x4676 bytes) 2024-09-21T15:49:42.654 INFO:teuthology.orchestra.run.smithi089.stdout:2889 read 0x3b2b677 thru 0x3b33b44 (0x84ce bytes) 2024-09-21T15:49:42.654 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:42.655 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:49:42.669 INFO:teuthology.orchestra.run.smithi089.stdout:2891 writesame 0xc14169c thru 0xc14ef4e data_size 0x23(0xd8b3 bytes) 2024-09-21T15:49:42.669 INFO:teuthology.orchestra.run.smithi089.stdout:2892 flatten 2024-09-21T15:49:42.714 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:42.713+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:42.735 INFO:teuthology.orchestra.run.smithi089.stdout:2893 punch from 0x5807abb to 0x5808dcb, (0x1310 bytes) 2024-09-21T15:49:42.735 INFO:teuthology.orchestra.run.smithi089.stdout:2894 read 0x1f0de8d thru 0x1f0f719 (0x188d bytes) 2024-09-21T15:49:42.742 INFO:teuthology.orchestra.run.smithi089.stdout:2895 writesame 0xc8639af thru 0xc8658af data_size 0x1(0x1f01 bytes) 2024-09-21T15:49:42.742 INFO:teuthology.orchestra.run.smithi089.stdout:2896 read 0x6601459 thru 0x660d1b6 (0xbd5e bytes) 2024-09-21T15:49:42.742 INFO:teuthology.orchestra.run.smithi089.stdout:2897 punch from 0xbc2f252 to 0xbc3309f, (0x3e4d bytes) 2024-09-21T15:49:42.743 INFO:teuthology.orchestra.run.smithi089.stdout:2898 read 0x3cd2246 thru 0x3cda987 (0x8742 bytes) 2024-09-21T15:49:42.743 INFO:teuthology.orchestra.run.smithi089.stdout:2899 read 0x39daff6 thru 0x39e6624 (0xb62f bytes) 2024-09-21T15:49:42.743 INFO:teuthology.orchestra.run.smithi089.stdout:2900 punch from 0x740196c to 0x740f25b, (0xd8ef bytes) 2024-09-21T15:49:42.743 INFO:teuthology.orchestra.run.smithi089.stdout:2901 read 0x65d257c thru 0x65def84 (0xca09 bytes) 2024-09-21T15:49:42.743 INFO:teuthology.orchestra.run.smithi089.stdout:2902 read 0x688690f thru 0x68947ae (0xdea0 bytes) 2024-09-21T15:49:42.743 INFO:teuthology.orchestra.run.smithi089.stdout:2903 punch from 0x80be406 to 0x80c8391, (0x9f8b bytes) 2024-09-21T15:49:42.751 INFO:teuthology.orchestra.run.smithi089.stdout:2904 writesame 0xddbce02 thru 0xddc1e93 data_size 0x1(0x5092 bytes) 2024-09-21T15:49:42.751 INFO:teuthology.orchestra.run.smithi089.stdout:2905 read 0x739ab95 thru 0x73a5a71 (0xaedd bytes) 2024-09-21T15:49:42.751 INFO:teuthology.orchestra.run.smithi089.stdout:2906 write 0xb758e2b thru 0xb764086 (0xb25c bytes) 2024-09-21T15:49:42.757 INFO:teuthology.orchestra.run.smithi089.stdout:2907 read 0x77cac74 thru 0x77d9ad1 (0xee5e bytes) 2024-09-21T15:49:42.757 INFO:teuthology.orchestra.run.smithi089.stdout:2909 read 0x332ee70 thru 0x3339df5 (0xaf86 bytes) 2024-09-21T15:49:42.767 INFO:teuthology.orchestra.run.smithi089.stdout:2910 trunc from 0xddc1e94 to 0x5277ab8 2024-09-21T15:49:42.794 INFO:teuthology.orchestra.run.smithi089.stdout:2911 write 0xdc70ea2 thru 0xdc7e6de (0xd83d bytes) 2024-09-21T15:49:42.802 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:42.800+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:42.802 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:42.802 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:42.804 INFO:teuthology.orchestra.run.smithi089.stdout:2912 writesame 0x13d84ad thru 0x13dda87 data_size 0x1(0x55db bytes) 2024-09-21T15:49:42.811 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:49:42.818 INFO:teuthology.orchestra.run.smithi089.stdout:2914 read 0x1e15326 thru 0x1e21218 (0xbef3 bytes) 2024-09-21T15:49:42.818 INFO:teuthology.orchestra.run.smithi089.stdout:2916 writesame 0x4d1cb47 thru 0x4d285c8 data_size 0x1(0xba82 bytes) 2024-09-21T15:49:42.838 INFO:teuthology.orchestra.run.smithi089.stdout:2917 read 0x22d882e thru 0x22dfba2 (0x7375 bytes) 2024-09-21T15:49:42.838 INFO:teuthology.orchestra.run.smithi089.stdout:2918 read 0x7ff2dfc thru 0x7ff5eb0 (0x30b5 bytes) 2024-09-21T15:49:42.838 INFO:teuthology.orchestra.run.smithi089.stdout:2919 punch from 0xd3d1f43 to 0xd3e1ea3, (0xff60 bytes) 2024-09-21T15:49:42.838 INFO:teuthology.orchestra.run.smithi089.stdout:2920 write 0xb8d2014 thru 0xb8d95b2 (0x759f bytes) 2024-09-21T15:49:42.842 INFO:teuthology.orchestra.run.smithi089.stdout:2921 punch from 0x7b608fa to 0x7b6149b, (0xba1 bytes) 2024-09-21T15:49:42.842 INFO:teuthology.orchestra.run.smithi089.stdout:2924 read 0x3abc927 thru 0x3ac874d (0xbe27 bytes) 2024-09-21T15:49:42.842 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:42.842 INFO:teuthology.orchestra.run.smithi089.stdout:2926 write 0x921f261 thru 0x92289d8 (0x9778 bytes) 2024-09-21T15:49:42.848 INFO:teuthology.orchestra.run.smithi089.stdout:2927 read 0x28f60aa thru 0x29027ce (0xc725 bytes) 2024-09-21T15:49:42.848 INFO:teuthology.orchestra.run.smithi089.stdout:2930 read 0x69a9709 thru 0x69b0283 (0x6b7b bytes) 2024-09-21T15:49:42.849 INFO:teuthology.orchestra.run.smithi089.stdout:2931 write 0xc6fbdf9 thru 0xc70b4bc (0xf6c4 bytes) 2024-09-21T15:49:42.853 INFO:teuthology.orchestra.run.smithi089.stdout:2932 punch from 0x6c809c0 to 0x6c8346a, (0x2aaa bytes) 2024-09-21T15:49:42.853 INFO:teuthology.orchestra.run.smithi089.stdout:2934 write 0x8f90b71 thru 0x8f9c5a9 (0xba39 bytes) 2024-09-21T15:49:42.868 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is set 2024-09-21T15:49:42.879 INFO:teuthology.orchestra.run.smithi089.stdout:2935 write 0xe7196ae thru 0xe71fdb8 (0x670b bytes) 2024-09-21T15:49:42.885 INFO:teuthology.orchestra.run.smithi089.stdout:2936 writesame 0x407ff3f thru 0x408d0fe data_size 0x20(0xd1c0 bytes) 2024-09-21T15:49:42.894 INFO:teuthology.orchestra.run.smithi089.stdout:2937 read 0x7cc5df7 thru 0x7ccb504 (0x570e bytes) 2024-09-21T15:49:42.894 INFO:teuthology.orchestra.run.smithi089.stdout:2940 punch from 0x14439a2 to 0x1448df6, (0x5454 bytes) 2024-09-21T15:49:42.894 INFO:teuthology.orchestra.run.smithi089.stdout:2941 writesame 0x6b8222d thru 0x6b8ac97 data_size 0x13(0x8a6b bytes) 2024-09-21T15:49:42.902 INFO:teuthology.orchestra.run.smithi089.stdout:2942 read 0xca6cd4 thru 0xcb4043 (0xd370 bytes) 2024-09-21T15:49:42.902 INFO:teuthology.orchestra.run.smithi089.stdout:2943 read 0x6c289c9 thru 0x6c28d56 (0x38e bytes) 2024-09-21T15:49:42.902 INFO:teuthology.orchestra.run.smithi089.stdout:2946 writesame 0xae04f17 thru 0xae10899 data_size 0x1(0xb983 bytes) 2024-09-21T15:49:42.903 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:42.901+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:42.928 INFO:teuthology.orchestra.run.smithi089.stdout:2947 write 0x163f125 thru 0x16483df (0x92bb bytes) 2024-09-21T15:49:42.930 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:42.930 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:42.942 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:49:42.952 INFO:teuthology.orchestra.run.smithi089.stdout:2948 trunc from 0xe71fdb9 to 0xa36326 2024-09-21T15:49:42.984 INFO:teuthology.orchestra.run.smithi089.stdout:2949 trunc from 0xa36326 to 0x7417aa7 2024-09-21T15:49:42.989 INFO:teuthology.orchestra.run.smithi089.stdout:2952 read 0x722bee9 thru 0x7233e92 (0x7faa bytes) 2024-09-21T15:49:42.990 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:42.990 INFO:teuthology.orchestra.run.smithi089.stdout:2954 trunc from 0x7417aa7 to 0x82e01c0 2024-09-21T15:49:42.995 INFO:teuthology.orchestra.run.smithi089.stdout:2955 read 0x2a1d585 thru 0x2a2b198 (0xdc14 bytes) 2024-09-21T15:49:42.995 INFO:teuthology.orchestra.run.smithi089.stdout:2957 read 0x3c809d4 thru 0x3c8cc2d (0xc25a bytes) 2024-09-21T15:49:43.000 INFO:teuthology.orchestra.run.smithi089.stdout:2958 write 0x85d2982 thru 0x85d4d75 (0x23f4 bytes) 2024-09-21T15:49:43.003 INFO:teuthology.orchestra.run.smithi089.stdout:2962 writesame 0x4a05b06 thru 0x4a1405f data_size 0x2(0xe55a bytes) 2024-09-21T15:49:43.003 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:43.002+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:43.013 INFO:teuthology.orchestra.run.smithi089.stdout:2963 write 0x426700b thru 0x426c1f8 (0x51ee bytes) 2024-09-21T15:49:43.018 INFO:teuthology.orchestra.run.smithi089.stdout:2965 read 0x3b599e8 thru 0x3b5d7fe (0x3e17 bytes) 2024-09-21T15:49:43.018 INFO:teuthology.orchestra.run.smithi089.stdout:2968 punch from 0x1c7ac0f to 0x1c86520, (0xb911 bytes) 2024-09-21T15:49:43.018 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:43.026 INFO:teuthology.orchestra.run.smithi089.stdout:2972 write 0xa2e3f5a thru 0xa2e9e31 (0x5ed8 bytes) 2024-09-21T15:49:43.032 INFO:teuthology.orchestra.run.smithi089.stdout:2973 clone 17 order 25 su 8388608 sc 11 2024-09-21T15:49:43.067 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:43.067 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:43.076 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:49:43.117 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:43.116+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:43.205 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:43.204+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:43.218 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:43.218 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:43.227 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:49:43.306 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:43.305+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:43.370 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:43.370 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:43.379 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:49:43.423 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:43.422+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:43.482 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:43.483 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:43.491 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:49:43.506 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:43.505+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:43.607 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:43.606+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:43.633 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:43.633 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:43.642 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:49:43.726 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:43.725+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:43.784 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:43.784 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:43.793 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:49:43.837 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:43.836+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:43.847 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T15:49:43.847 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T15:49:43.848 INFO:tasks.thrashosds.thrasher:inject_pause on osd.7 2024-09-21T15:49:43.848 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T15:49:43.848 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T15:49:43.848 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok config set filestore_inject_stall 3 2024-09-21T15:49:43.910 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:43.909+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:43.911 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:43.911 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:43.921 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:49:43.959 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:43.959 INFO:teuthology.orchestra.run.smithi100.stdout: "success": "" 2024-09-21T15:49:43.959 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:44.011 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:44.010+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:44.025 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:44.025 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:44.033 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:49:44.046 INFO:teuthology.orchestra.run.smithi089.stdout:checking clone #15, image image_client.0-clone15 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent16 2024-09-21T15:49:44.112 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:44.111+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:44.180 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:44.181 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:44.190 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:49:44.214 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:44.212+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:44.315 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:44.312+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:44.370 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:44.371 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:44.379 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:49:44.422 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:44.421+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:44.442 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:49:44.480 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:44.480 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:44.489 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:49:44.530 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:44.529+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:44.542 INFO:teuthology.orchestra.run.smithi089.stdout:2975 write 0xc729da8 thru 0xc731c4e (0x7ea7 bytes) 2024-09-21T15:49:44.548 INFO:teuthology.orchestra.run.smithi089.stdout:2976 trunc from 0xc731c4f to 0xd6e9ecc 2024-09-21T15:49:44.554 INFO:teuthology.orchestra.run.smithi089.stdout:2978 punch from 0x6992df6 to 0x699f69d, (0xc8a7 bytes) 2024-09-21T15:49:44.554 INFO:teuthology.orchestra.run.smithi089.stdout:2979 punch from 0x151ee32 to 0x152a84a, (0xba18 bytes) 2024-09-21T15:49:44.556 INFO:teuthology.orchestra.run.smithi089.stdout:2981 trunc from 0xd6e9ecc to 0xbb78423 2024-09-21T15:49:44.568 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-09-21T15:49:44.568 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:49:44.586 INFO:teuthology.orchestra.run.smithi089.stdout:2982 write 0xe9def3f thru 0xe9df99b (0xa5d bytes) 2024-09-21T15:49:44.586 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:49:44.586 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:49:44.592 INFO:teuthology.orchestra.run.smithi089.stdout:2983 write 0xae1fa79 thru 0xae2d78d (0xdd15 bytes) 2024-09-21T15:49:44.598 INFO:teuthology.orchestra.run.smithi089.stdout:2984 punch from 0xb0a6fd7 to 0xb0b1508, (0xa531 bytes) 2024-09-21T15:49:44.598 INFO:teuthology.orchestra.run.smithi089.stdout:2985 read 0x2369685 thru 0x236d127 (0x3aa3 bytes) 2024-09-21T15:49:44.598 INFO:teuthology.orchestra.run.smithi089.stdout:2986 writesame 0x3c91e6e thru 0x3c9f5fb data_size 0x33(0xd78e bytes) 2024-09-21T15:49:44.600 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:44.600 INFO:teuthology.orchestra.run.smithi089.stdout:2989 write 0x1b39a5f thru 0x1b3a55e (0xb00 bytes) 2024-09-21T15:49:44.604 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:49:44.604 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:49:44.615 INFO:teuthology.orchestra.run.smithi089.stdout:2991 trunc from 0xe9df99c to 0x31426ca 2024-09-21T15:49:44.616 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:44.614+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:44.623 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:49:44.624 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:49:44.638 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:44.638 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:44.641 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:49:44.641 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:49:44.648 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T15:49:44.655 INFO:teuthology.orchestra.run.smithi089.stdout:2993 write 0x17cd7e2 thru 0x17d168b (0x3eaa bytes) 2024-09-21T15:49:44.657 INFO:teuthology.orchestra.run.smithi089.stdout:2994 punch from 0x1eb5f80 to 0x1ebca8c, (0x6b0c bytes) 2024-09-21T15:49:44.657 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:44.658 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:49:44.658 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:49:44.674 INFO:teuthology.orchestra.run.smithi089.stdout:2997 writesame 0xbb754e5 thru 0xbb842f0 data_size 0x6e(0xee0c bytes) 2024-09-21T15:49:44.675 INFO:teuthology.orchestra.run.smithi089.stdout:2998 read 0x932cacc thru 0x93365c7 (0x9afc bytes) 2024-09-21T15:49:44.676 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:49:44.676 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:49:44.681 INFO:teuthology.orchestra.run.smithi089.stdout:2999 write 0xcdcb793 thru 0xcdd3de4 (0x8652 bytes) 2024-09-21T15:49:44.683 INFO:teuthology.orchestra.run.smithi089.stdout:3000 write 0x8826622 thru 0x882a9b0 (0x438f bytes) 2024-09-21T15:49:44.684 INFO:teuthology.orchestra.run.smithi089.stdout:3001 writesame 0x9c21be8 thru 0x9c30a86 data_size 0x25(0xee9f bytes) 2024-09-21T15:49:44.688 INFO:teuthology.orchestra.run.smithi089.stdout:3004 punch from 0x64f556c to 0x64fc795, (0x7229 bytes) 2024-09-21T15:49:44.694 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:49:44.694 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:49:44.697 INFO:teuthology.orchestra.run.smithi089.stdout:3005 trunc from 0xcdd3de5 to 0x534921e 2024-09-21T15:49:44.705 INFO:teuthology.orchestra.run.smithi089.stdout:3006 read 0x92c597 thru 0x93a0cc (0xdb36 bytes) 2024-09-21T15:49:44.706 INFO:teuthology.orchestra.run.smithi089.stdout:3007 write 0x3f66ae6 thru 0x3f7105a (0xa575 bytes) 2024-09-21T15:49:44.709 INFO:teuthology.orchestra.run.smithi089.stdout:3008 punch from 0x4113c30 to 0x411a30d, (0x66dd bytes) 2024-09-21T15:49:44.709 INFO:teuthology.orchestra.run.smithi089.stdout:3009 punch from 0x313edd2 to 0x3148dad, (0x9fdb bytes) 2024-09-21T15:49:44.709 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:44.725 INFO:teuthology.orchestra.run.smithi089.stdout:3011 writesame 0xd5daa7f thru 0xd5e66c5 data_size 0x1(0xbc47 bytes) 2024-09-21T15:49:44.726 INFO:teuthology.orchestra.run.smithi089.stdout:3014 write 0x51bc321 thru 0x51c5b4a (0x982a bytes) 2024-09-21T15:49:44.733 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:44.732+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:44.735 INFO:teuthology.orchestra.run.smithi089.stdout:3016 writesame 0x52be983 thru 0x52c9566 data_size 0x72(0xabe4 bytes) 2024-09-21T15:49:44.739 INFO:teuthology.orchestra.run.smithi089.stdout:3017 read 0xaa97ff0 thru 0xaaa6672 (0xe683 bytes) 2024-09-21T15:49:44.739 INFO:teuthology.orchestra.run.smithi089.stdout:3018 writesame 0x4c5a8bd thru 0x4c5e3d8 data_size 0x4(0x3b1c bytes) 2024-09-21T15:49:44.744 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:44.744 INFO:teuthology.orchestra.run.smithi089.stdout:3021 read 0x8943a3 thru 0x8991dc (0x4e3a bytes) 2024-09-21T15:49:44.744 INFO:teuthology.orchestra.run.smithi089.stdout:3022 write 0xc97b4d3 thru 0xc97c08c (0xbba bytes) 2024-09-21T15:49:44.746 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:44.746 INFO:teuthology.orchestra.run.smithi089.stdout:3024 read 0x911c655 thru 0x9127c7d (0xb629 bytes) 2024-09-21T15:49:44.752 INFO:teuthology.orchestra.run.smithi089.stdout:3030 writesame 0xe82a6a7 thru 0xe839899 data_size 0x1(0xf1f3 bytes) 2024-09-21T15:49:44.753 INFO:teuthology.orchestra.run.smithi089.stdout:3031 write 0xa5c2cc1 thru 0xa5c7569 (0x48a9 bytes) 2024-09-21T15:49:44.755 INFO:teuthology.orchestra.run.smithi089.stdout:3032 write 0x73ddbae thru 0x73e3783 (0x5bd6 bytes) 2024-09-21T15:49:44.766 INFO:teuthology.orchestra.run.smithi089.stdout:3033 trunc from 0xe83989a to 0x6bc7b22 2024-09-21T15:49:44.778 INFO:teuthology.orchestra.run.smithi089.stdout:3034 read 0x3a8e72e thru 0x3a97a36 (0x9309 bytes) 2024-09-21T15:49:44.778 INFO:teuthology.orchestra.run.smithi089.stdout:3035 read 0x279bc5c thru 0x27a2bb0 (0x6f55 bytes) 2024-09-21T15:49:44.779 INFO:teuthology.orchestra.run.smithi089.stdout:3036 punch from 0x5655fd1 to 0x565cc29, (0x6c58 bytes) 2024-09-21T15:49:44.779 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:44.794 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:44.794 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:44.794 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:49:44.795 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:44.798 INFO:teuthology.orchestra.run.smithi089.stdout:3039 write 0xdee2b40 thru 0xdeee708 (0xbbc9 bytes) 2024-09-21T15:49:44.802 INFO:teuthology.orchestra.run.smithi089.stdout:3040 write 0x1fa17fd thru 0x1fa4c64 (0x3468 bytes) 2024-09-21T15:49:44.803 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T15:49:44.804 INFO:teuthology.orchestra.run.smithi089.stdout:3041 read 0x7dd6b4a thru 0x7de3ded (0xd2a4 bytes) 2024-09-21T15:49:44.804 INFO:teuthology.orchestra.run.smithi089.stdout:3042 read 0x8c5241c thru 0x8c5ea37 (0xc61c bytes) 2024-09-21T15:49:44.805 INFO:teuthology.orchestra.run.smithi089.stdout:3043 punch from 0xdbe93e7 to 0xdbf7379, (0xdf92 bytes) 2024-09-21T15:49:44.805 INFO:teuthology.orchestra.run.smithi089.stdout:3045 punch from 0xb02e158 to 0xb037cab, (0x9b53 bytes) 2024-09-21T15:49:44.805 INFO:teuthology.orchestra.run.smithi089.stdout:3046 read 0x7d2eb6c thru 0x7d38a31 (0x9ec6 bytes) 2024-09-21T15:49:44.812 INFO:teuthology.orchestra.run.smithi089.stdout:3047 trunc from 0xdeee709 to 0x7a8bdb1 2024-09-21T15:49:44.825 INFO:teuthology.orchestra.run.smithi089.stdout:3048 write 0x70c2861 thru 0x70d1bd4 (0xf374 bytes) 2024-09-21T15:49:44.831 INFO:teuthology.orchestra.run.smithi089.stdout:3049 trunc from 0x7a8bdb1 to 0x443c128 2024-09-21T15:49:44.844 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:44.843+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:44.853 INFO:teuthology.orchestra.run.smithi089.stdout:3050 read 0x1040bdf thru 0x104d408 (0xc82a bytes) 2024-09-21T15:49:44.873 INFO:teuthology.orchestra.run.smithi089.stdout:3051 write 0xd98b21d thru 0xd9994a7 (0xe28b bytes) 2024-09-21T15:49:44.877 INFO:teuthology.orchestra.run.smithi089.stdout:3052 read 0x2874e6b thru 0x287e6b2 (0x9848 bytes) 2024-09-21T15:49:44.877 INFO:teuthology.orchestra.run.smithi089.stdout:3053 writesame 0x49d8ba7 thru 0x49dae49 data_size 0x1(0x22a3 bytes) 2024-09-21T15:49:44.883 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T15:49:44.884 INFO:teuthology.orchestra.run.smithi089.stdout:3054 read 0x7d951f5 thru 0x7d96d08 (0x1b14 bytes) 2024-09-21T15:49:44.884 INFO:teuthology.orchestra.run.smithi089.stdout:3055 write 0x9e97fc2 thru 0x9e98080 (0xbf bytes) 2024-09-21T15:49:44.888 INFO:teuthology.orchestra.run.smithi089.stdout:3056 read 0xa38c518 thru 0xa39c06c (0xfb55 bytes) 2024-09-21T15:49:44.889 INFO:teuthology.orchestra.run.smithi089.stdout:3057 write 0x616eb7a thru 0x6177911 (0x8d98 bytes) 2024-09-21T15:49:44.890 INFO:teuthology.orchestra.run.smithi089.stdout:3058 write 0xd6cb1ca thru 0xd6cff4e (0x4d85 bytes) 2024-09-21T15:49:44.894 INFO:teuthology.orchestra.run.smithi089.stdout:3060 read 0x2c086b8 thru 0x2c0a7bc (0x2105 bytes) 2024-09-21T15:49:44.895 INFO:teuthology.orchestra.run.smithi089.stdout:3061 writesame 0x970fdcb thru 0x9711226 data_size 0x2(0x145c bytes) 2024-09-21T15:49:44.897 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:44.905 INFO:teuthology.orchestra.run.smithi089.stdout:3063 writesame 0xe09a5aa thru 0xe09d9e5 data_size 0x2(0x343c bytes) 2024-09-21T15:49:44.907 INFO:teuthology.orchestra.run.smithi089.stdout:3064 write 0xd547143 thru 0xd54ddf7 (0x6cb5 bytes) 2024-09-21T15:49:44.918 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:44.917+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:44.920 INFO:teuthology.orchestra.run.smithi089.stdout:3065 trunc from 0xe09d9e6 to 0x5a9c09e 2024-09-21T15:49:44.930 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:44.931 INFO:teuthology.orchestra.run.smithi089.stdout:3067 read 0x39389d9 thru 0x39484e3 (0xfb0b bytes) 2024-09-21T15:49:44.931 INFO:teuthology.orchestra.run.smithi089.stdout:3068 writesame 0x41b23ea thru 0x41b7e24 data_size 0x1(0x5a3b bytes) 2024-09-21T15:49:44.943 INFO:teuthology.orchestra.run.smithi089.stdout:3069 trunc from 0x5a9c09e to 0x57bced8 2024-09-21T15:49:44.948 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:44.948 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:44.948 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:49:44.948 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:49:44.948 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:44.950 INFO:teuthology.orchestra.run.smithi089.stdout:3070 read 0x4a4e027 thru 0x4a5b1c5 (0xd19f bytes) 2024-09-21T15:49:44.950 INFO:teuthology.orchestra.run.smithi089.stdout:3071 punch from 0x19f3da0 to 0x19fd3a2, (0x9602 bytes) 2024-09-21T15:49:44.957 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T15:49:44.960 INFO:teuthology.orchestra.run.smithi089.stdout:3072 write 0x97e54b5 thru 0x97e6f3c (0x1a88 bytes) 2024-09-21T15:49:44.961 INFO:teuthology.orchestra.run.smithi089.stdout:3074 write 0x4c85733 thru 0x4c9413c (0xea0a bytes) 2024-09-21T15:49:44.963 INFO:teuthology.orchestra.run.smithi089.stdout:3076 punch from 0x77acffa to 0x77b3970, (0x6976 bytes) 2024-09-21T15:49:44.963 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:44.963 INFO:teuthology.orchestra.run.smithi089.stdout:3078 write 0x95e1be7 thru 0x95e5dff (0x4219 bytes) 2024-09-21T15:49:44.964 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:44.964 INFO:teuthology.orchestra.run.smithi089.stdout:3083 punch from 0x1b09b65 to 0x1b15862, (0xbcfd bytes) 2024-09-21T15:49:44.964 INFO:teuthology.orchestra.run.smithi089.stdout:3084 write 0x7cd14d thru 0x7da471 (0xd325 bytes) 2024-09-21T15:49:44.968 INFO:teuthology.orchestra.run.smithi089.stdout:3087 writesame 0x597740d thru 0x597e3a1 data_size 0x5(0x6f95 bytes) 2024-09-21T15:49:44.971 INFO:teuthology.orchestra.run.smithi089.stdout:3088 punch from 0x2d69aec to 0x2d712ad, (0x77c1 bytes) 2024-09-21T15:49:44.971 INFO:teuthology.orchestra.run.smithi089.stdout:3089 write 0x8900223 thru 0x890eb25 (0xe903 bytes) 2024-09-21T15:49:44.976 INFO:teuthology.orchestra.run.smithi089.stdout:3090 punch from 0x90d7822 to 0x90e1354, (0x9b32 bytes) 2024-09-21T15:49:44.989 INFO:teuthology.orchestra.run.smithi089.stdout:3091 writesame 0xdf7d3a8 thru 0xdf7f832 data_size 0x1(0x248b bytes) 2024-09-21T15:49:44.991 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:44.998 INFO:teuthology.orchestra.run.smithi089.stdout:3093 trunc from 0xdf7f833 to 0x7c1ae94 2024-09-21T15:49:45.015 INFO:teuthology.orchestra.run.smithi089.stdout:3094 trunc from 0x7c1ae94 to 0xb1168d7 2024-09-21T15:49:45.018 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:45.017+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:45.025 INFO:teuthology.orchestra.run.smithi089.stdout:3095 trunc from 0xb1168d7 to 0xcd57b38 2024-09-21T15:49:45.033 INFO:teuthology.orchestra.run.smithi089.stdout:3096 write 0x9668d6b thru 0x96786a9 (0xf93f bytes) 2024-09-21T15:49:45.036 INFO:teuthology.orchestra.run.smithi089.stdout:3097 write 0x2921842 thru 0x2928b96 (0x7355 bytes) 2024-09-21T15:49:45.041 INFO:teuthology.orchestra.run.smithi089.stdout:3099 punch from 0x3cf05c9 to 0x3cf81b5, (0x7bec bytes) 2024-09-21T15:49:45.041 INFO:teuthology.orchestra.run.smithi089.stdout:3100 punch from 0x9f3c288 to 0x9f49c2d, (0xd9a5 bytes) 2024-09-21T15:49:45.041 INFO:teuthology.orchestra.run.smithi089.stdout:3101 read 0x653e389 thru 0x6546129 (0x7da1 bytes) 2024-09-21T15:49:45.041 INFO:teuthology.orchestra.run.smithi089.stdout:3102 write 0x150700e thru 0x15098cf (0x28c2 bytes) 2024-09-21T15:49:45.042 INFO:teuthology.orchestra.run.smithi089.stdout:3103 read 0xb237157 thru 0xb2442d9 (0xd183 bytes) 2024-09-21T15:49:45.043 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:45.043 INFO:teuthology.orchestra.run.smithi089.stdout:3105 punch from 0x26ae67 to 0x26ed90, (0x3f29 bytes) 2024-09-21T15:49:45.043 INFO:teuthology.orchestra.run.smithi089.stdout:3107 read 0x4b69051 thru 0x4b7412e (0xb0de bytes) 2024-09-21T15:49:45.045 INFO:teuthology.orchestra.run.smithi089.stdout:3108 trunc from 0xcd57b38 to 0xbbaead4 2024-09-21T15:49:45.051 INFO:teuthology.orchestra.run.smithi089.stdout:3110 writesame 0x54d0e3 thru 0x5591cd data_size 0x1d(0xc0eb bytes) 2024-09-21T15:49:45.056 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:45.056 INFO:teuthology.orchestra.run.smithi089.stdout:3113 writesame 0x6fdeb26 thru 0x6fdf6cb data_size 0x15(0xba6 bytes) 2024-09-21T15:49:45.057 INFO:teuthology.orchestra.run.smithi089.stdout:3115 read 0x3b37cba thru 0x3b3ae3a (0x3181 bytes) 2024-09-21T15:49:45.058 INFO:teuthology.orchestra.run.smithi089.stdout:3117 punch from 0x816aad1 to 0x8178264, (0xd793 bytes) 2024-09-21T15:49:45.058 INFO:teuthology.orchestra.run.smithi089.stdout:3118 read 0x8eca9e1 thru 0x8ecb650 (0xc70 bytes) 2024-09-21T15:49:45.058 INFO:teuthology.orchestra.run.smithi089.stdout:3119 write 0x951c0e5 thru 0x9528605 (0xc521 bytes) 2024-09-21T15:49:45.060 INFO:teuthology.orchestra.run.smithi089.stdout:3120 write 0x2596000 thru 0x259677d (0x77e bytes) 2024-09-21T15:49:45.063 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:45.063 INFO:teuthology.orchestra.run.smithi089.stdout:3124 read 0x3e033e6 thru 0x3e09ee7 (0x6b02 bytes) 2024-09-21T15:49:45.064 INFO:teuthology.orchestra.run.smithi089.stdout:3125 write 0xbee3c5 thru 0xbf0986 (0x25c2 bytes) 2024-09-21T15:49:45.076 INFO:teuthology.orchestra.run.smithi089.stdout:3126 writesame 0xdab4b7c thru 0xdac2b65 data_size 0x1(0xdfea bytes) 2024-09-21T15:49:45.077 INFO:teuthology.orchestra.run.smithi089.stdout:3127 read 0x316e97a thru 0x3179485 (0xab0c bytes) 2024-09-21T15:49:45.090 INFO:teuthology.orchestra.run.smithi089.stdout:3128 trunc from 0xdac2b66 to 0x53ff90e 2024-09-21T15:49:45.101 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:45.105 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:45.105 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:49:45.105 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:49:45.105 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:49:45.105 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.105 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63140 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 876544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:45.105 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.133649+0000", 2024-09-21T15:49:45.105 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.9694450620000001, 2024-09-21T15:49:45.105 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053950201000000003, 2024-09-21T15:49:45.105 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:45.105 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:45.105 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:45.106 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:45.106 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:45.106 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63140 2024-09-21T15:49:45.106 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.106 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:45.106 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.106 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:45.106 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133649+0000", 2024-09-21T15:49:45.106 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.106 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.106 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.106 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:45.106 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133649+0000", 2024-09-21T15:49:45.106 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.107 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.107 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.107 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:45.107 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133648+0000", 2024-09-21T15:49:45.107 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:45.107 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.107 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.107 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:45.107 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133651+0000", 2024-09-21T15:49:45.107 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1549999999999999e-06 2024-09-21T15:49:45.107 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.107 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.107 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:45.107 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133660+0000", 2024-09-21T15:49:45.108 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3899999999999993e-06 2024-09-21T15:49:45.108 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.108 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.108 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:45.108 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133669+0000", 2024-09-21T15:49:45.108 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3219999999999999e-06 2024-09-21T15:49:45.108 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.108 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.108 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:45.108 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187409+0000", 2024-09-21T15:49:45.108 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053740257 2024-09-21T15:49:45.108 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.108 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.108 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:45.109 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187492+0000", 2024-09-21T15:49:45.109 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2409000000000003e-05 2024-09-21T15:49:45.109 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.109 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.109 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:45.109 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187599+0000", 2024-09-21T15:49:45.109 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000107334 2024-09-21T15:49:45.109 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.109 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:45.109 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.109 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.109 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.109 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63141 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 884736~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:45.109 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.133843+0000", 2024-09-21T15:49:45.110 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.9692509430000005, 2024-09-21T15:49:45.110 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053954015000000001, 2024-09-21T15:49:45.110 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:45.110 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:45.110 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:45.110 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:45.110 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:45.110 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63141 2024-09-21T15:49:45.110 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.110 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:45.110 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.110 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:45.110 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133843+0000", 2024-09-21T15:49:45.110 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.111 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.111 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.111 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:45.111 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133843+0000", 2024-09-21T15:49:45.111 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.111 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.111 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.111 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:45.111 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133673+0000", 2024-09-21T15:49:45.111 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998302 2024-09-21T15:49:45.111 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.111 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.111 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:45.111 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133846+0000", 2024-09-21T15:49:45.112 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000172762 2024-09-21T15:49:45.112 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.112 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.112 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:45.112 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133855+0000", 2024-09-21T15:49:45.112 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2940000000000006e-06 2024-09-21T15:49:45.112 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.112 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.112 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:45.112 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133863+0000", 2024-09-21T15:49:45.112 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9740000000000002e-06 2024-09-21T15:49:45.113 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.113 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.113 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:45.113 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187630+0000", 2024-09-21T15:49:45.113 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053767211000000002 2024-09-21T15:49:45.113 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.113 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.113 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:45.113 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187707+0000", 2024-09-21T15:49:45.113 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7446000000000004e-05 2024-09-21T15:49:45.113 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.113 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.113 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:45.113 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187797+0000", 2024-09-21T15:49:45.114 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.9858000000000006e-05 2024-09-21T15:49:45.114 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.114 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:45.114 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.114 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.114 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.114 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63142 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 892928~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:45.114 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134046+0000", 2024-09-21T15:49:45.114 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.969048001, 2024-09-21T15:49:45.114 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054017849999999999, 2024-09-21T15:49:45.114 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:45.114 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:45.114 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:45.114 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:45.115 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:45.115 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63142 2024-09-21T15:49:45.115 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.115 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:45.115 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.115 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:45.115 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134046+0000", 2024-09-21T15:49:45.115 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.115 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.115 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.115 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:45.115 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134046+0000", 2024-09-21T15:49:45.115 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.115 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.116 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.116 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:45.116 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133867+0000", 2024-09-21T15:49:45.116 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998217 2024-09-21T15:49:45.116 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.116 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.116 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:45.116 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134049+0000", 2024-09-21T15:49:45.116 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000181695 2024-09-21T15:49:45.116 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.116 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.116 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:45.116 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134057+0000", 2024-09-21T15:49:45.116 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.303e-06 2024-09-21T15:49:45.116 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.117 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.117 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:45.117 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134063+0000", 2024-09-21T15:49:45.117 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9959999999999999e-06 2024-09-21T15:49:45.117 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.117 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.117 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:45.117 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187849+0000", 2024-09-21T15:49:45.117 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053785571999999997 2024-09-21T15:49:45.117 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.117 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.117 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:45.117 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187954+0000", 2024-09-21T15:49:45.117 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010524899999999999 2024-09-21T15:49:45.117 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.118 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.118 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:45.118 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188064+0000", 2024-09-21T15:49:45.118 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000109549 2024-09-21T15:49:45.118 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.118 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:45.118 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.118 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.118 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.118 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63143 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 901120~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:45.118 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134239+0000", 2024-09-21T15:49:45.118 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.9688554079999996, 2024-09-21T15:49:45.118 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054026058000000002, 2024-09-21T15:49:45.118 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:45.119 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:45.119 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:45.119 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:45.119 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:45.119 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63143 2024-09-21T15:49:45.119 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.119 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:45.119 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.119 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:45.119 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134239+0000", 2024-09-21T15:49:45.119 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.119 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.119 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.119 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:45.120 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134239+0000", 2024-09-21T15:49:45.120 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.120 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.120 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.120 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:45.120 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134068+0000", 2024-09-21T15:49:45.120 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998298 2024-09-21T15:49:45.120 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.120 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.120 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:45.120 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134242+0000", 2024-09-21T15:49:45.120 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00017414199999999999 2024-09-21T15:49:45.120 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.120 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.121 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:45.121 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134264+0000", 2024-09-21T15:49:45.121 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1552e-05 2024-09-21T15:49:45.121 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.121 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.121 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:45.121 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134270+0000", 2024-09-21T15:49:45.121 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6939999999999998e-06 2024-09-21T15:49:45.121 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.121 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.121 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:45.121 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188108+0000", 2024-09-21T15:49:45.121 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.05383719 2024-09-21T15:49:45.121 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.121 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.122 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:45.122 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188186+0000", 2024-09-21T15:49:45.122 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8318000000000005e-05 2024-09-21T15:49:45.122 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.122 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.122 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:45.122 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188265+0000", 2024-09-21T15:49:45.122 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8597000000000007e-05 2024-09-21T15:49:45.122 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.122 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:45.123 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.123 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.123 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.123 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63144 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 909312~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:45.123 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134432+0000", 2024-09-21T15:49:45.123 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.9686618520000003, 2024-09-21T15:49:45.123 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053968953, 2024-09-21T15:49:45.123 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:45.123 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:45.123 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:45.123 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:45.123 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:45.123 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63144 2024-09-21T15:49:45.123 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.123 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:45.124 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.124 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:45.124 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134432+0000", 2024-09-21T15:49:45.124 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.124 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.124 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.124 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:45.124 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134432+0000", 2024-09-21T15:49:45.124 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.124 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.124 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.124 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:45.124 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134276+0000", 2024-09-21T15:49:45.125 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998441 2024-09-21T15:49:45.125 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.125 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.125 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:45.125 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134436+0000", 2024-09-21T15:49:45.125 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000159591 2024-09-21T15:49:45.125 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.125 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.125 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:45.125 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134445+0000", 2024-09-21T15:49:45.125 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6780000000000006e-06 2024-09-21T15:49:45.125 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.125 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.125 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:45.126 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134452+0000", 2024-09-21T15:49:45.126 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6830000000000008e-06 2024-09-21T15:49:45.126 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.126 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.126 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:45.126 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188282+0000", 2024-09-21T15:49:45.126 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053829252000000001 2024-09-21T15:49:45.126 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.126 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.126 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:45.126 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188328+0000", 2024-09-21T15:49:45.126 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6734999999999999e-05 2024-09-21T15:49:45.126 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.126 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.126 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:45.127 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188401+0000", 2024-09-21T15:49:45.127 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2731000000000005e-05 2024-09-21T15:49:45.127 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.127 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:45.127 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.127 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.127 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.127 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63145 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 860160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:45.127 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134736+0000", 2024-09-21T15:49:45.127 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.9683580670000005, 2024-09-21T15:49:45.127 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053865960999999997, 2024-09-21T15:49:45.127 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:45.127 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:45.127 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:45.128 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:45.128 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:45.128 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63145 2024-09-21T15:49:45.128 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.128 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:45.128 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.128 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:45.128 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134736+0000", 2024-09-21T15:49:45.128 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.128 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.128 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.128 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:45.128 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134736+0000", 2024-09-21T15:49:45.128 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.129 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.129 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.129 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:45.129 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134458+0000", 2024-09-21T15:49:45.129 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999722 2024-09-21T15:49:45.129 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.129 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.129 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:45.129 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134741+0000", 2024-09-21T15:49:45.129 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00028245599999999999 2024-09-21T15:49:45.129 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.129 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.129 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:45.129 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134753+0000", 2024-09-21T15:49:45.129 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2548000000000001e-05 2024-09-21T15:49:45.130 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.130 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.130 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:45.130 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134760+0000", 2024-09-21T15:49:45.130 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4189999999999996e-06 2024-09-21T15:49:45.130 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.130 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.130 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:45.130 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188426+0000", 2024-09-21T15:49:45.130 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053665894999999998 2024-09-21T15:49:45.130 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.130 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.130 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:45.130 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188502+0000", 2024-09-21T15:49:45.130 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5791000000000007e-05 2024-09-21T15:49:45.131 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.131 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.131 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:45.131 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188602+0000", 2024-09-21T15:49:45.131 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9667999999999995e-05 2024-09-21T15:49:45.131 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.131 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:45.131 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.131 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.131 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.131 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63148 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 933888~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:45.131 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.135289+0000", 2024-09-21T15:49:45.131 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.9678051119999997, 2024-09-21T15:49:45.131 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053948488000000003, 2024-09-21T15:49:45.132 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:45.132 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:45.132 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:45.132 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:45.132 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:45.132 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63148 2024-09-21T15:49:45.132 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.132 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:45.132 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.132 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:45.132 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135289+0000", 2024-09-21T15:49:45.132 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.132 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.132 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.133 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:45.133 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135289+0000", 2024-09-21T15:49:45.133 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.133 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.133 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.133 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:45.133 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135288+0000", 2024-09-21T15:49:45.133 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:45.133 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.133 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.133 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:45.133 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135296+0000", 2024-09-21T15:49:45.133 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5399999999999998e-06 2024-09-21T15:49:45.133 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.133 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.134 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:45.134 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135303+0000", 2024-09-21T15:49:45.134 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0500000000000003e-06 2024-09-21T15:49:45.134 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.134 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.134 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:45.134 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135308+0000", 2024-09-21T15:49:45.134 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9749999999999998e-06 2024-09-21T15:49:45.134 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.134 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.134 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:45.134 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189066+0000", 2024-09-21T15:49:45.134 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053758501 2024-09-21T15:49:45.134 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.135 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.135 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:45.135 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189145+0000", 2024-09-21T15:49:45.135 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9017000000000006e-05 2024-09-21T15:49:45.135 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.135 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.135 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:45.135 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189237+0000", 2024-09-21T15:49:45.135 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1906999999999999e-05 2024-09-21T15:49:45.135 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.135 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:45.135 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.135 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.135 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.135 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63149 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 942080~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:45.136 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.135430+0000", 2024-09-21T15:49:45.136 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.9676637829999999, 2024-09-21T15:49:45.136 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054005434999999997, 2024-09-21T15:49:45.136 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:45.136 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:45.136 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:45.136 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:45.136 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:45.136 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63149 2024-09-21T15:49:45.136 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.136 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:45.136 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.136 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:45.136 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135430+0000", 2024-09-21T15:49:45.137 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.137 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.137 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.137 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:45.137 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135430+0000", 2024-09-21T15:49:45.137 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.137 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.137 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.137 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:45.137 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135313+0000", 2024-09-21T15:49:45.137 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998827 2024-09-21T15:49:45.137 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.137 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.137 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:45.138 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135434+0000", 2024-09-21T15:49:45.138 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000120876 2024-09-21T15:49:45.138 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.138 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.138 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:45.138 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135443+0000", 2024-09-21T15:49:45.138 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3340000000000008e-06 2024-09-21T15:49:45.138 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.138 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.138 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:45.138 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135449+0000", 2024-09-21T15:49:45.138 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1580000000000002e-06 2024-09-21T15:49:45.138 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.138 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.139 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:45.139 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189267+0000", 2024-09-21T15:49:45.139 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053817775999999998 2024-09-21T15:49:45.139 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.139 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.139 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:45.139 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189336+0000", 2024-09-21T15:49:45.139 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9485000000000003e-05 2024-09-21T15:49:45.139 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.139 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.139 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:45.139 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189436+0000", 2024-09-21T15:49:45.139 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9277000000000005e-05 2024-09-21T15:49:45.139 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.140 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:45.140 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.140 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.140 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.140 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63150 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 950272~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:45.140 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.135618+0000", 2024-09-21T15:49:45.140 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.967476038, 2024-09-21T15:49:45.140 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054027393, 2024-09-21T15:49:45.140 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:45.140 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:45.140 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:45.140 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:45.140 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:45.140 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63150 2024-09-21T15:49:45.141 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.141 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:45.141 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.141 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:45.141 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135618+0000", 2024-09-21T15:49:45.141 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.141 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.141 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.141 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:45.141 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135618+0000", 2024-09-21T15:49:45.141 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.141 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.141 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.141 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:45.142 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135454+0000", 2024-09-21T15:49:45.142 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998364 2024-09-21T15:49:45.142 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.142 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.142 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:45.142 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135621+0000", 2024-09-21T15:49:45.142 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016714099999999999 2024-09-21T15:49:45.142 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.142 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.142 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:45.142 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135631+0000", 2024-09-21T15:49:45.142 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5340000000000005e-06 2024-09-21T15:49:45.142 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.142 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.143 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:45.143 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135638+0000", 2024-09-21T15:49:45.143 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9280000000000004e-06 2024-09-21T15:49:45.143 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.143 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.143 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:45.143 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189463+0000", 2024-09-21T15:49:45.143 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053825681 2024-09-21T15:49:45.143 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.143 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.143 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:45.143 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189546+0000", 2024-09-21T15:49:45.143 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2278000000000001e-05 2024-09-21T15:49:45.143 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.144 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.144 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:45.144 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189645+0000", 2024-09-21T15:49:45.144 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9543999999999994e-05 2024-09-21T15:49:45.144 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.144 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:45.144 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.144 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.144 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.144 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63152 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 966656~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:45.144 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.136029+0000", 2024-09-21T15:49:45.144 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.9670654460000003, 2024-09-21T15:49:45.144 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054024823999999999, 2024-09-21T15:49:45.144 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:45.144 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:45.145 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:45.145 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:45.145 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:45.145 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63152 2024-09-21T15:49:45.145 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.145 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:45.145 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.145 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:45.145 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136029+0000", 2024-09-21T15:49:45.145 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.145 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.145 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.145 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:45.145 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136029+0000", 2024-09-21T15:49:45.145 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.145 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.146 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.146 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:45.146 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136028+0000", 2024-09-21T15:49:45.146 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:45.146 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.146 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.146 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:45.146 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136031+0000", 2024-09-21T15:49:45.146 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.033e-06 2024-09-21T15:49:45.146 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.146 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.146 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:45.147 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136039+0000", 2024-09-21T15:49:45.147 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7200000000000006e-06 2024-09-21T15:49:45.147 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.147 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.147 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:45.147 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136051+0000", 2024-09-21T15:49:45.147 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1987000000000001e-05 2024-09-21T15:49:45.147 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.147 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.147 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:45.147 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189876+0000", 2024-09-21T15:49:45.147 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053825264999999997 2024-09-21T15:49:45.147 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.147 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.148 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:45.148 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189955+0000", 2024-09-21T15:49:45.148 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9296999999999997e-05 2024-09-21T15:49:45.148 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.148 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.148 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:45.148 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190053+0000", 2024-09-21T15:49:45.148 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.8067999999999997e-05 2024-09-21T15:49:45.148 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.148 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:45.148 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.148 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.148 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.148 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63153 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 974848~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:45.149 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.136311+0000", 2024-09-21T15:49:45.149 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.9667829330000002, 2024-09-21T15:49:45.149 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053933825999999997, 2024-09-21T15:49:45.149 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:45.149 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:45.149 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:45.149 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:45.149 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:45.149 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63153 2024-09-21T15:49:45.149 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.149 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:45.149 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.149 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:45.149 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136311+0000", 2024-09-21T15:49:45.150 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.150 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.150 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.150 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:45.150 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136311+0000", 2024-09-21T15:49:45.150 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.150 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.150 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.150 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:45.150 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136057+0000", 2024-09-21T15:49:45.150 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9997463 2024-09-21T15:49:45.150 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.150 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.150 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:45.151 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136316+0000", 2024-09-21T15:49:45.151 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00025872299999999998 2024-09-21T15:49:45.151 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.151 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.151 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:45.151 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136326+0000", 2024-09-21T15:49:45.151 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0552e-05 2024-09-21T15:49:45.151 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.151 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.151 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:45.151 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136334+0000", 2024-09-21T15:49:45.151 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1409999999999999e-06 2024-09-21T15:49:45.151 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.151 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.151 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:45.152 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190080+0000", 2024-09-21T15:49:45.152 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053745933000000003 2024-09-21T15:49:45.152 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.152 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.152 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:45.152 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190157+0000", 2024-09-21T15:49:45.152 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7412999999999999e-05 2024-09-21T15:49:45.152 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.152 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.152 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:45.152 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190245+0000", 2024-09-21T15:49:45.152 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7904000000000004e-05 2024-09-21T15:49:45.152 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.152 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:45.153 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.153 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.153 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.153 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63154 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 983040~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:45.153 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.136472+0000", 2024-09-21T15:49:45.153 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.9666215820000001, 2024-09-21T15:49:45.153 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053968448000000002, 2024-09-21T15:49:45.153 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:45.153 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:45.153 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:45.153 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:45.153 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:45.153 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63154 2024-09-21T15:49:45.153 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.153 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:45.154 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.154 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:45.154 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136472+0000", 2024-09-21T15:49:45.154 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.154 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.154 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.154 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:45.154 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136472+0000", 2024-09-21T15:49:45.154 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.154 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.154 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.154 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:45.154 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136341+0000", 2024-09-21T15:49:45.154 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998684 2024-09-21T15:49:45.155 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.155 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.155 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:45.155 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136476+0000", 2024-09-21T15:49:45.155 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013528100000000001 2024-09-21T15:49:45.155 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.155 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.155 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:45.155 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136484+0000", 2024-09-21T15:49:45.155 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4760000000000001e-06 2024-09-21T15:49:45.155 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.155 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.155 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:45.155 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136490+0000", 2024-09-21T15:49:45.155 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8640000000000001e-06 2024-09-21T15:49:45.156 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.156 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.156 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:45.156 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190271+0000", 2024-09-21T15:49:45.156 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053781223000000003 2024-09-21T15:49:45.156 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.156 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.156 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:45.156 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190348+0000", 2024-09-21T15:49:45.156 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7058999999999995e-05 2024-09-21T15:49:45.156 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.156 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.156 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:45.156 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190441+0000", 2024-09-21T15:49:45.156 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2937999999999997e-05 2024-09-21T15:49:45.157 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.157 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:45.157 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.157 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.157 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.157 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63157 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1015808~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:45.157 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137112+0000", 2024-09-21T15:49:45.157 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.9659822949999999, 2024-09-21T15:49:45.157 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053932345, 2024-09-21T15:49:45.157 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:45.157 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:45.157 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:45.157 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:45.157 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:45.157 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63157 2024-09-21T15:49:45.158 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.158 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:45.158 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.158 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:45.158 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137112+0000", 2024-09-21T15:49:45.158 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.158 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.158 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.158 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:45.158 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137112+0000", 2024-09-21T15:49:45.158 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.158 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.158 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.158 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:45.159 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137072+0000", 2024-09-21T15:49:45.159 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999604 2024-09-21T15:49:45.159 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.159 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.159 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:45.159 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137115+0000", 2024-09-21T15:49:45.159 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2379999999999997e-05 2024-09-21T15:49:45.159 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.159 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.159 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:45.159 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137121+0000", 2024-09-21T15:49:45.159 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6499999999999999e-06 2024-09-21T15:49:45.159 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.159 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.160 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:45.160 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137126+0000", 2024-09-21T15:49:45.160 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0769999999999998e-06 2024-09-21T15:49:45.160 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.160 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.160 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:45.160 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190868+0000", 2024-09-21T15:49:45.160 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053741555000000003 2024-09-21T15:49:45.160 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.160 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.160 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:45.160 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190945+0000", 2024-09-21T15:49:45.160 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7171999999999999e-05 2024-09-21T15:49:45.161 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.161 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.161 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:45.161 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191044+0000", 2024-09-21T15:49:45.161 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9031999999999996e-05 2024-09-21T15:49:45.161 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.161 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:45.161 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.161 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.161 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.161 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63158 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1007616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:45.161 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137293+0000", 2024-09-21T15:49:45.161 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.9658010079999997, 2024-09-21T15:49:45.161 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054005642, 2024-09-21T15:49:45.162 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:45.162 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:45.162 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:45.162 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:45.162 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:45.162 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63158 2024-09-21T15:49:45.162 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.162 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:45.162 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.162 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:45.162 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137293+0000", 2024-09-21T15:49:45.162 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.162 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.162 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.163 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:45.163 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137293+0000", 2024-09-21T15:49:45.163 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.163 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.163 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.163 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:45.163 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137131+0000", 2024-09-21T15:49:45.163 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998379 2024-09-21T15:49:45.163 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.163 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.163 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:45.163 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137296+0000", 2024-09-21T15:49:45.163 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016568999999999999 2024-09-21T15:49:45.164 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.164 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.164 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:45.164 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137306+0000", 2024-09-21T15:49:45.164 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.6029999999999994e-06 2024-09-21T15:49:45.164 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.164 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.164 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:45.164 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137312+0000", 2024-09-21T15:49:45.164 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3210000000000001e-06 2024-09-21T15:49:45.164 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.164 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.164 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:45.165 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191087+0000", 2024-09-21T15:49:45.165 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053774686000000002 2024-09-21T15:49:45.165 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.165 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.165 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:45.165 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191175+0000", 2024-09-21T15:49:45.165 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8106e-05 2024-09-21T15:49:45.165 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.165 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.165 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:45.165 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191299+0000", 2024-09-21T15:49:45.165 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012343599999999999 2024-09-21T15:49:45.165 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.166 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:45.166 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.166 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.166 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.166 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63159 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1024000~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:45.166 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137568+0000", 2024-09-21T15:49:45.166 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.965525736, 2024-09-21T15:49:45.166 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053962999999999997, 2024-09-21T15:49:45.166 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:45.166 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:45.166 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:45.166 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:45.166 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:45.166 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63159 2024-09-21T15:49:45.167 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.167 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:45.167 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.167 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:45.167 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137568+0000", 2024-09-21T15:49:45.167 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.167 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.167 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.167 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:45.167 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137568+0000", 2024-09-21T15:49:45.167 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.167 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.167 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.167 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:45.167 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137317+0000", 2024-09-21T15:49:45.168 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9997492 2024-09-21T15:49:45.168 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.168 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.168 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:45.168 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137572+0000", 2024-09-21T15:49:45.168 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00025491499999999998 2024-09-21T15:49:45.168 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.168 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.168 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:45.168 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137582+0000", 2024-09-21T15:49:45.168 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4490000000000005e-06 2024-09-21T15:49:45.168 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.168 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.168 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:45.169 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137589+0000", 2024-09-21T15:49:45.169 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5580000000000003e-06 2024-09-21T15:49:45.169 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.169 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.169 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:45.169 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191344+0000", 2024-09-21T15:49:45.169 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053754272999999998 2024-09-21T15:49:45.169 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.169 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.169 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:45.169 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191429+0000", 2024-09-21T15:49:45.169 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.509e-05 2024-09-21T15:49:45.169 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.169 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.170 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:45.170 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191531+0000", 2024-09-21T15:49:45.170 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000102519 2024-09-21T15:49:45.170 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.170 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:45.170 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.170 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.170 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.170 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63160 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1032192~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:45.170 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137771+0000", 2024-09-21T15:49:45.170 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.9653232020000004, 2024-09-21T15:49:45.170 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053957818999999997, 2024-09-21T15:49:45.170 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:45.170 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:45.171 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:45.171 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:45.171 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:45.171 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63160 2024-09-21T15:49:45.171 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.171 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:45.171 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.171 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:45.171 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137771+0000", 2024-09-21T15:49:45.171 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.171 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.171 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.171 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:45.171 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137771+0000", 2024-09-21T15:49:45.171 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.172 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.172 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.172 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:45.172 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137596+0000", 2024-09-21T15:49:45.172 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998255 2024-09-21T15:49:45.172 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.172 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.172 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:45.172 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137780+0000", 2024-09-21T15:49:45.172 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00018363900000000001 2024-09-21T15:49:45.172 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.172 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.172 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:45.172 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137791+0000", 2024-09-21T15:49:45.173 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.096e-05 2024-09-21T15:49:45.173 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.173 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.173 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:45.173 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137798+0000", 2024-09-21T15:49:45.173 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1169999999999998e-06 2024-09-21T15:49:45.173 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.173 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.173 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:45.173 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191560+0000", 2024-09-21T15:49:45.173 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053762163000000002 2024-09-21T15:49:45.173 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.173 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.173 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:45.173 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191634+0000", 2024-09-21T15:49:45.174 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4381999999999994e-05 2024-09-21T15:49:45.174 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.174 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.174 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:45.174 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191729+0000", 2024-09-21T15:49:45.174 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4098000000000006e-05 2024-09-21T15:49:45.174 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.174 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:45.174 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.174 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.174 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.174 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63161 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1040384~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:45.174 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.138018+0000", 2024-09-21T15:49:45.175 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.9650764479999996, 2024-09-21T15:49:45.175 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053905657000000003, 2024-09-21T15:49:45.175 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:45.175 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:45.175 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:45.175 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:45.175 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:45.175 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63161 2024-09-21T15:49:45.175 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.175 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:45.175 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.175 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:45.175 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138018+0000", 2024-09-21T15:49:45.175 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.176 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.176 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.176 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:45.176 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138018+0000", 2024-09-21T15:49:45.176 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.176 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.176 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.176 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:45.176 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137804+0000", 2024-09-21T15:49:45.176 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9997859 2024-09-21T15:49:45.176 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.176 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.176 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:45.176 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138021+0000", 2024-09-21T15:49:45.177 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00021769899999999999 2024-09-21T15:49:45.177 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.177 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.177 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:45.177 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138043+0000", 2024-09-21T15:49:45.177 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1627000000000001e-05 2024-09-21T15:49:45.177 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.177 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.177 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:45.177 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138054+0000", 2024-09-21T15:49:45.177 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1321999999999999e-05 2024-09-21T15:49:45.177 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.177 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.177 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:45.178 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191755+0000", 2024-09-21T15:49:45.178 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053700354999999998 2024-09-21T15:49:45.178 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.178 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.178 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:45.178 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191830+0000", 2024-09-21T15:49:45.178 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5320000000000004e-05 2024-09-21T15:49:45.178 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.178 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.178 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:45.178 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191923+0000", 2024-09-21T15:49:45.178 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3282999999999994e-05 2024-09-21T15:49:45.178 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.178 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:45.179 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.179 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.179 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.179 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63176 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 108544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:45.179 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.140861+0000", 2024-09-21T15:49:45.179 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.9622331290000004, 2024-09-21T15:49:45.179 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053935394999999997, 2024-09-21T15:49:45.179 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:45.179 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:45.179 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:45.179 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:45.179 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:45.179 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63176 2024-09-21T15:49:45.179 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.179 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:45.180 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.180 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:45.180 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140861+0000", 2024-09-21T15:49:45.180 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.180 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.180 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.180 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:45.180 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140861+0000", 2024-09-21T15:49:45.180 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.180 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.180 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.180 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:45.180 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140860+0000", 2024-09-21T15:49:45.180 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:45.180 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.181 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.181 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:45.181 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140863+0000", 2024-09-21T15:49:45.181 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8710000000000001e-06 2024-09-21T15:49:45.181 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.181 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.181 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:45.181 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140871+0000", 2024-09-21T15:49:45.181 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8089999999999993e-06 2024-09-21T15:49:45.181 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.181 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.181 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:45.181 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140877+0000", 2024-09-21T15:49:45.181 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5969999999999999e-06 2024-09-21T15:49:45.182 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.182 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.182 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:45.182 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.194608+0000", 2024-09-21T15:49:45.182 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053731749000000002 2024-09-21T15:49:45.182 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.182 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.182 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:45.182 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.194697+0000", 2024-09-21T15:49:45.182 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8504000000000005e-05 2024-09-21T15:49:45.182 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.182 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.182 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:45.183 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.194796+0000", 2024-09-21T15:49:45.183 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9414999999999996e-05 2024-09-21T15:49:45.183 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.183 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:45.183 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.183 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.183 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.183 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63182 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 157696~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:45.183 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:49:45.183 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.960960729, 2024-09-21T15:49:45.183 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053958825000000002, 2024-09-21T15:49:45.183 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:45.183 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:45.183 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:45.184 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:45.184 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:45.184 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63182 2024-09-21T15:49:45.184 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.184 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:45.184 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.184 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:45.184 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:49:45.184 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.184 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.184 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.184 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:45.184 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:49:45.184 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.185 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.185 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.185 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:45.185 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:49:45.185 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:45.185 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.185 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.185 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:45.185 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142136+0000", 2024-09-21T15:49:45.185 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8169999999999999e-06 2024-09-21T15:49:45.185 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.185 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.185 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:45.185 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:49:45.185 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3752e-05 2024-09-21T15:49:45.186 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.186 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.186 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:45.186 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142155+0000", 2024-09-21T15:49:45.186 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1859999999999997e-06 2024-09-21T15:49:45.186 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.186 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.186 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:45.186 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.195914+0000", 2024-09-21T15:49:45.186 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053758528999999999 2024-09-21T15:49:45.186 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.186 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.186 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:45.186 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.195995+0000", 2024-09-21T15:49:45.186 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0983999999999996e-05 2024-09-21T15:49:45.187 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.187 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.187 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:45.187 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.196092+0000", 2024-09-21T15:49:45.187 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7069000000000002e-05 2024-09-21T15:49:45.187 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.187 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:45.187 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.187 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.187 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.187 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63193 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 239616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:45.187 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:49:45.187 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 7.958873144, 2024-09-21T15:49:45.187 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053956577999999998, 2024-09-21T15:49:45.188 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:45.188 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:45.188 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:45.188 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:45.188 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:45.188 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63193 2024-09-21T15:49:45.188 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.188 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:45.188 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.188 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:45.188 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:49:45.188 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.188 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.188 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.189 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:45.189 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:49:45.189 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.189 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.189 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.189 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:45.189 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:49:45.189 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:45.189 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.189 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.189 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:45.189 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144223+0000", 2024-09-21T15:49:45.189 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0990000000000001e-06 2024-09-21T15:49:45.190 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.190 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.190 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:45.190 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144230+0000", 2024-09-21T15:49:45.190 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9070000000000001e-06 2024-09-21T15:49:45.190 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.190 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.190 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:45.190 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144234+0000", 2024-09-21T15:49:45.190 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9010000000000002e-06 2024-09-21T15:49:45.190 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.190 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.190 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:45.190 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.198004+0000", 2024-09-21T15:49:45.191 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053769539999999998 2024-09-21T15:49:45.191 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.191 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.191 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:45.191 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.198083+0000", 2024-09-21T15:49:45.191 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9531999999999997e-05 2024-09-21T15:49:45.191 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.191 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.191 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:45.191 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.198177+0000", 2024-09-21T15:49:45.191 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3906000000000005e-05 2024-09-21T15:49:45.191 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.191 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:45.191 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.192 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.192 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:45.192 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:45.194 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T15:49:45.194 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:45.192+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:45.194 INFO:teuthology.orchestra.run.smithi089.stdout:3131 trunc from 0x53ff90e to 0x338b36a 2024-09-21T15:49:45.195 INFO:teuthology.orchestra.run.smithi089.stdout:3133 punch from 0x174cde9 to 0x1752bba, (0x5dd1 bytes) 2024-09-21T15:49:45.195 INFO:teuthology.orchestra.run.smithi089.stdout:3135 punch from 0x213c685 to 0x2142337, (0x5cb2 bytes) 2024-09-21T15:49:45.195 INFO:teuthology.orchestra.run.smithi089.stdout:3136 write 0xcd67bab thru 0xcd67d3f (0x195 bytes) 2024-09-21T15:49:45.195 INFO:teuthology.orchestra.run.smithi089.stdout:3137 trunc from 0xcd67d40 to 0x1260ad 2024-09-21T15:49:45.195 INFO:teuthology.orchestra.run.smithi089.stdout:3138 punch from 0xdd9e8 to 0xe158b, (0x3ba3 bytes) 2024-09-21T15:49:45.195 INFO:teuthology.orchestra.run.smithi089.stdout:3139 read 0x8474f thru 0x8ed48 (0xa5fa bytes) 2024-09-21T15:49:45.195 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:45.204 INFO:teuthology.orchestra.run.smithi089.stdout:3142 write 0x6ee37ac thru 0x6ee9390 (0x5be5 bytes) 2024-09-21T15:49:45.208 INFO:teuthology.orchestra.run.smithi089.stdout:3143 write 0x3bf12fa thru 0x3bf83f2 (0x70f9 bytes) 2024-09-21T15:49:45.214 INFO:teuthology.orchestra.run.smithi089.stdout:3144 punch from 0x1aa4e5e to 0x1ab27bc, (0xd95e bytes) 2024-09-21T15:49:45.216 INFO:teuthology.orchestra.run.smithi089.stdout:3145 trunc from 0x6ee9391 to 0x8ca896c 2024-09-21T15:49:45.233 INFO:teuthology.orchestra.run.smithi089.stdout:3146 writesame 0xc2d8b9a thru 0xc2e2293 data_size 0x2(0x96fa bytes) 2024-09-21T15:49:45.234 INFO:teuthology.orchestra.run.smithi089.stdout:3147 read 0x430fb28 thru 0x43124ce (0x29a7 bytes) 2024-09-21T15:49:45.235 INFO:teuthology.orchestra.run.smithi089.stdout:3148 write 0x1dacfce thru 0x1db3628 (0x665b bytes) 2024-09-21T15:49:45.239 INFO:teuthology.orchestra.run.smithi089.stdout:3149 trunc from 0xc2e2294 to 0xccc4222 2024-09-21T15:49:45.246 INFO:teuthology.orchestra.run.smithi089.stdout:3150 punch from 0x12bd9b3 to 0x12ccfb1, (0xf5fe bytes) 2024-09-21T15:49:45.246 INFO:teuthology.orchestra.run.smithi089.stdout:3151 writesame 0x3ed02ed thru 0x3ee01c4 data_size 0x38(0xfed8 bytes) 2024-09-21T15:49:45.247 INFO:teuthology.orchestra.run.smithi089.stdout:3152 writesame 0x4a02bbe thru 0x4a03f2f data_size 0x13(0x1372 bytes) 2024-09-21T15:49:45.252 INFO:teuthology.orchestra.run.smithi089.stdout:3155 writesame 0xbce5d6a thru 0xbcee1c9 data_size 0x30(0x8460 bytes) 2024-09-21T15:49:45.256 INFO:teuthology.orchestra.run.smithi089.stdout:3156 read 0x82ed998 thru 0x82fb798 (0xde01 bytes) 2024-09-21T15:49:45.256 INFO:teuthology.orchestra.run.smithi089.stdout:3159 read 0x9a1d0f1 thru 0x9a2c4bb (0xf3cb bytes) 2024-09-21T15:49:45.256 INFO:teuthology.orchestra.run.smithi089.stdout:3160 write 0x803be9b thru 0x8046a1a (0xab80 bytes) 2024-09-21T15:49:45.259 INFO:teuthology.orchestra.run.smithi089.stdout:truncating to largest ever: 0xeafffa2 2024-09-21T15:49:45.261 INFO:teuthology.orchestra.run.smithi089.stdout:3165 trunc from 0xccc4222 to 0xeafffa2 2024-09-21T15:49:45.267 INFO:teuthology.orchestra.run.smithi089.stdout:3166 read 0xe923947 thru 0xe9264e5 (0x2b9f bytes) 2024-09-21T15:49:45.267 INFO:teuthology.orchestra.run.smithi089.stdout:3167 write 0xb6d82df thru 0xb6e6e48 (0xeb6a bytes) 2024-09-21T15:49:45.270 INFO:teuthology.orchestra.run.smithi089.stdout:3168 read 0x2e8f33 thru 0x2f87b6 (0xf884 bytes) 2024-09-21T15:49:45.270 INFO:teuthology.orchestra.run.smithi089.stdout:3169 write 0xb5d6ef3 thru 0xb5dc803 (0x5911 bytes) 2024-09-21T15:49:45.272 INFO:teuthology.orchestra.run.smithi089.stdout:3171 read 0x73731a8 thru 0x7380117 (0xcf70 bytes) 2024-09-21T15:49:45.272 INFO:teuthology.orchestra.run.smithi089.stdout:3172 read 0xcc8a4c4 thru 0xcc8ec78 (0x47b5 bytes) 2024-09-21T15:49:45.273 INFO:teuthology.orchestra.run.smithi089.stdout:3174 punch from 0x3d53fbb to 0x3d5c1b0, (0x81f5 bytes) 2024-09-21T15:49:45.274 INFO:teuthology.orchestra.run.smithi089.stdout:3176 write 0x7cece92 thru 0x7cf2a22 (0x5b91 bytes) 2024-09-21T15:49:45.276 INFO:teuthology.orchestra.run.smithi089.stdout:3177 punch from 0xa4cbff1 to 0xa4d3144, (0x7153 bytes) 2024-09-21T15:49:45.276 INFO:teuthology.orchestra.run.smithi089.stdout:3178 read 0x30ac900 thru 0x30b0127 (0x3828 bytes) 2024-09-21T15:49:45.284 INFO:teuthology.orchestra.run.smithi089.stdout:3179 trunc from 0xeafffa2 to 0x7e0e124 2024-09-21T15:49:45.293 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:45.292+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:45.305 INFO:teuthology.orchestra.run.smithi089.stdout:3180 write 0xaaa60ca thru 0xaab4669 (0xe5a0 bytes) 2024-09-21T15:49:45.311 INFO:teuthology.orchestra.run.smithi089.stdout:3181 writesame 0xa77bbac thru 0xa77d4b8 data_size 0xb(0x190d bytes) 2024-09-21T15:49:45.327 INFO:teuthology.orchestra.run.smithi089.stdout:3182 write 0xe990ee0 thru 0xe99223d (0x135e bytes) 2024-09-21T15:49:45.328 INFO:teuthology.orchestra.run.smithi089.stdout:3185 write 0x3e9b595 thru 0x3ea4568 (0x8fd4 bytes) 2024-09-21T15:49:45.339 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:45.339 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:45.339 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:49:45.339 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:45.340 INFO:teuthology.orchestra.run.smithi089.stdout:3187 trunc from 0xe99223e to 0x674e556 2024-09-21T15:49:45.348 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T15:49:45.364 INFO:teuthology.orchestra.run.smithi089.stdout:3188 write 0x77b905d thru 0x77c58c1 (0xc865 bytes) 2024-09-21T15:49:45.365 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:45.373 INFO:teuthology.orchestra.run.smithi089.stdout:3192 write 0x9c70e56 thru 0x9c74680 (0x382b bytes) 2024-09-21T15:49:45.378 INFO:teuthology.orchestra.run.smithi089.stdout:3193 read 0xeba7a2 thru 0xec4f2e (0xa78d bytes) 2024-09-21T15:49:45.379 INFO:teuthology.orchestra.run.smithi089.stdout:3194 write 0x18ce55c thru 0x18d5a53 (0x74f8 bytes) 2024-09-21T15:49:45.384 INFO:teuthology.orchestra.run.smithi089.stdout:3195 trunc from 0x9c74681 to 0x6e0d1fc 2024-09-21T15:49:45.394 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:45.393+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:45.409 INFO:teuthology.orchestra.run.smithi089.stdout:3196 write 0xcef3a81 thru 0xcf00de5 (0xd365 bytes) 2024-09-21T15:49:45.410 INFO:teuthology.orchestra.run.smithi089.stdout:3197 read 0xb06eeb7 thru 0xb076ec4 (0x800e bytes) 2024-09-21T15:49:45.411 INFO:teuthology.orchestra.run.smithi089.stdout:3198 read 0x5913c1a thru 0x5917557 (0x393e bytes) 2024-09-21T15:49:45.421 INFO:teuthology.orchestra.run.smithi089.stdout:3201 writesame 0xea14af1 thru 0xea18ac7 data_size 0x1(0x3fd7 bytes) 2024-09-21T15:49:45.422 INFO:teuthology.orchestra.run.smithi089.stdout:3202 writesame 0x776c03a thru 0x7770b8f data_size 0x1(0x4b56 bytes) 2024-09-21T15:49:45.432 INFO:teuthology.orchestra.run.smithi089.stdout:3203 read 0x95de4e5 thru 0x95dff6e (0x1a8a bytes) 2024-09-21T15:49:45.432 INFO:teuthology.orchestra.run.smithi089.stdout:3205 write 0x2965488 thru 0x2966220 (0xd99 bytes) 2024-09-21T15:49:45.433 INFO:teuthology.orchestra.run.smithi089.stdout:3207 read 0xa505dea thru 0xa50e9c9 (0x8be0 bytes) 2024-09-21T15:49:45.434 INFO:teuthology.orchestra.run.smithi089.stdout:3208 flatten 2024-09-21T15:49:45.441 INFO:teuthology.orchestra.run.smithi089.stdout:3210 write 0x9720960 thru 0x972c029 (0xb6ca bytes) 2024-09-21T15:49:45.443 INFO:teuthology.orchestra.run.smithi089.stdout:3211 read 0xe5e0068 thru 0xe5e63fd (0x6396 bytes) 2024-09-21T15:49:45.443 INFO:teuthology.orchestra.run.smithi089.stdout:3212 read 0x9a02c5b thru 0x9a03986 (0xd2c bytes) 2024-09-21T15:49:45.444 INFO:teuthology.orchestra.run.smithi089.stdout:3213 write 0xe1f4868 thru 0xe2044c9 (0xfc62 bytes) 2024-09-21T15:49:45.447 INFO:teuthology.orchestra.run.smithi089.stdout:3215 read 0x88f2b3 thru 0x89d51a (0xe268 bytes) 2024-09-21T15:49:45.447 INFO:teuthology.orchestra.run.smithi089.stdout:3216 read 0xe49d036 thru 0xe4a61a0 (0x916b bytes) 2024-09-21T15:49:45.447 INFO:teuthology.orchestra.run.smithi089.stdout:3217 write 0x759e1e thru 0x75c0cc (0x22af bytes) 2024-09-21T15:49:45.449 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:45.449 INFO:teuthology.orchestra.run.smithi089.stdout:3219 writesame 0x6b23bba thru 0x6b29910 data_size 0x2d(0x5d57 bytes) 2024-09-21T15:49:45.454 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:45.455 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:45.455 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:49:45.455 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:49:45.455 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:45.456 INFO:teuthology.orchestra.run.smithi089.stdout:3220 trunc from 0xea18ac8 to 0xa2d4777 2024-09-21T15:49:45.464 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T15:49:45.472 INFO:teuthology.orchestra.run.smithi089.stdout:3221 write 0xbb9369c thru 0xbb943cb (0xd30 bytes) 2024-09-21T15:49:45.473 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:45.480 INFO:teuthology.orchestra.run.smithi089.stdout:3223 write 0xd63a350 thru 0xd646ea9 (0xcb5a bytes) 2024-09-21T15:49:45.483 INFO:teuthology.orchestra.run.smithi089.stdout:3224 write 0x1a65956 thru 0x1a6bd35 (0x63e0 bytes) 2024-09-21T15:49:45.484 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:45.484 INFO:teuthology.orchestra.run.smithi089.stdout:3226 punch from 0x29311a8 to 0x293260d, (0x1465 bytes) 2024-09-21T15:49:45.484 INFO:teuthology.orchestra.run.smithi089.stdout:3228 write 0x24dc98b thru 0x24e5ba1 (0x9217 bytes) 2024-09-21T15:49:45.486 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:45.486 INFO:teuthology.orchestra.run.smithi089.stdout:3230 write 0xd0521a2 thru 0xd05bdac (0x9c0b bytes) 2024-09-21T15:49:45.488 INFO:teuthology.orchestra.run.smithi089.stdout:3231 write 0xba84c1f thru 0xba9473c (0xfb1e bytes) 2024-09-21T15:49:45.489 INFO:teuthology.orchestra.run.smithi089.stdout:3233 writesame 0xaea9242 thru 0xaeb6b7f data_size 0x5d(0xd93e bytes) 2024-09-21T15:49:45.491 INFO:teuthology.orchestra.run.smithi089.stdout:3234 read 0x2a1c2db thru 0x2a2bfcc (0xfcf2 bytes) 2024-09-21T15:49:45.492 INFO:teuthology.orchestra.run.smithi089.stdout:3235 punch from 0x4ac4dfa to 0x4ad39df, (0xebe5 bytes) 2024-09-21T15:49:45.492 INFO:teuthology.orchestra.run.smithi089.stdout:3238 read 0xc0cfe29 thru 0xc0dd05f (0xd237 bytes) 2024-09-21T15:49:45.492 INFO:teuthology.orchestra.run.smithi089.stdout:3240 read 0x800ef6d thru 0x801c419 (0xd4ad bytes) 2024-09-21T15:49:45.492 INFO:teuthology.orchestra.run.smithi089.stdout:3241 read 0x8699efe thru 0x869c6a6 (0x27a9 bytes) 2024-09-21T15:49:45.495 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:45.494+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:45.499 INFO:teuthology.orchestra.run.smithi089.stdout:3242 write 0xedc8cd8 thru 0xedd3607 (0xa930 bytes) 2024-09-21T15:49:45.503 INFO:teuthology.orchestra.run.smithi089.stdout:3244 write 0xb6256a5 thru 0xb6351c4 (0xfb20 bytes) 2024-09-21T15:49:45.505 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:45.505 INFO:teuthology.orchestra.run.smithi089.stdout:3246 punch from 0x6a49915 to 0x6a4ff90, (0x667b bytes) 2024-09-21T15:49:45.505 INFO:teuthology.orchestra.run.smithi089.stdout:3247 write 0x67e9805 thru 0x67edd43 (0x453f bytes) 2024-09-21T15:49:45.506 INFO:teuthology.orchestra.run.smithi089.stdout:3248 write 0x97fc16d thru 0x9808c68 (0xcafc bytes) 2024-09-21T15:49:45.508 INFO:teuthology.orchestra.run.smithi089.stdout:3250 writesame 0xcabf932 thru 0xcac5999 data_size 0x14(0x6068 bytes) 2024-09-21T15:49:45.510 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:45.510 INFO:teuthology.orchestra.run.smithi089.stdout:3253 read 0x39f9f9a thru 0x39fc9b7 (0x2a1e bytes) 2024-09-21T15:49:45.526 INFO:teuthology.orchestra.run.smithi089.stdout:3254 trunc from 0xedd3608 to 0x18030e5 2024-09-21T15:49:45.549 INFO:teuthology.orchestra.run.smithi089.stdout:3257 clone 18 order 21 su 16384 sc 13 2024-09-21T15:49:45.570 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:45.570 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:49:45.570 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:49:45.570 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:49:45.571 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.571 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4209.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:49:45.571 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:49:45.571 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 54.377402310000001, 2024-09-21T15:49:45.571 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.19452909900000001, 2024-09-21T15:49:45.571 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:45.571 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:45.571 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:45.571 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4209", 2024-09-21T15:49:45.571 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/4045143050", 2024-09-21T15:49:45.571 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1 2024-09-21T15:49:45.571 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.571 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:45.571 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.572 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:45.572 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:49:45.572 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.572 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.572 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.572 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:45.572 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:49:45.572 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.572 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.572 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.572 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:45.572 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191161+0000", 2024-09-21T15:49:45.572 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:45.572 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.573 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.573 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:45.573 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191169+0000", 2024-09-21T15:49:45.573 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0090000000000007e-06 2024-09-21T15:49:45.573 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.573 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.573 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:45.573 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191180+0000", 2024-09-21T15:49:45.573 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0728e-05 2024-09-21T15:49:45.573 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.573 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.573 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:45.573 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191187+0000", 2024-09-21T15:49:45.573 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9009999999999994e-06 2024-09-21T15:49:45.574 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.574 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.574 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:45.574 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191238+0000", 2024-09-21T15:49:45.574 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0287000000000002e-05 2024-09-21T15:49:45.574 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.574 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.574 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-21T15:49:45.574 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191245+0000", 2024-09-21T15:49:45.574 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0709999999999997e-06 2024-09-21T15:49:45.574 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.574 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.574 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:45.574 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380309+0000", 2024-09-21T15:49:45.575 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.189064277 2024-09-21T15:49:45.575 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.575 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.575 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:45.575 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380454+0000", 2024-09-21T15:49:45.575 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014490799999999999 2024-09-21T15:49:45.575 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.575 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.575 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0", 2024-09-21T15:49:45.575 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380610+0000", 2024-09-21T15:49:45.575 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015650199999999999 2024-09-21T15:49:45.575 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.575 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.576 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:45.576 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.381160+0000", 2024-09-21T15:49:45.576 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00054903799999999996 2024-09-21T15:49:45.576 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.576 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.576 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:45.576 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385591+0000", 2024-09-21T15:49:45.576 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.004431272 2024-09-21T15:49:45.576 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.576 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.576 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:45.576 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385651+0000", 2024-09-21T15:49:45.576 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0455999999999999e-05 2024-09-21T15:49:45.576 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.576 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.576 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:45.577 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385691+0000", 2024-09-21T15:49:45.577 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0234999999999997e-05 2024-09-21T15:49:45.577 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.577 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:45.577 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.577 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.577 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.577 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64028 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1763328~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:45.577 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167504+0000", 2024-09-21T15:49:45.577 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 8.4010607539999995, 2024-09-21T15:49:45.577 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047494153999999997, 2024-09-21T15:49:45.577 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:45.577 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:45.578 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:45.578 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:45.578 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:45.578 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64028 2024-09-21T15:49:45.578 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.578 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:45.578 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.578 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:45.578 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167504+0000", 2024-09-21T15:49:45.578 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.578 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.578 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.578 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:45.578 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167504+0000", 2024-09-21T15:49:45.579 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.579 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.579 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.579 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:45.579 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167504+0000", 2024-09-21T15:49:45.579 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:45.579 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.579 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.579 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:45.579 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167506+0000", 2024-09-21T15:49:45.579 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0600000000000002e-06 2024-09-21T15:49:45.579 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.579 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.579 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:45.580 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167512+0000", 2024-09-21T15:49:45.580 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.083e-06 2024-09-21T15:49:45.580 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.580 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.580 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:45.580 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167517+0000", 2024-09-21T15:49:45.580 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7650000000000001e-06 2024-09-21T15:49:45.580 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.580 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.580 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:45.580 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.214835+0000", 2024-09-21T15:49:45.580 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047318067999999998 2024-09-21T15:49:45.580 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.580 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.581 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:45.581 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.214916+0000", 2024-09-21T15:49:45.581 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1815000000000002e-05 2024-09-21T15:49:45.581 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.581 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.581 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:45.581 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.214998+0000", 2024-09-21T15:49:45.581 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1642000000000002e-05 2024-09-21T15:49:45.581 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.581 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:45.581 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.581 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.581 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.581 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64029 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1050624~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:45.582 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167521+0000", 2024-09-21T15:49:45.582 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 8.4010438430000001, 2024-09-21T15:49:45.582 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047649915000000001, 2024-09-21T15:49:45.582 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:45.582 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:45.582 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:45.582 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:45.582 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:45.582 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64029 2024-09-21T15:49:45.582 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.582 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:45.582 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.582 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:45.582 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167521+0000", 2024-09-21T15:49:45.582 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.583 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.583 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.583 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:45.583 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167521+0000", 2024-09-21T15:49:45.583 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.583 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.583 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.583 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:45.583 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167521+0000", 2024-09-21T15:49:45.583 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:45.583 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.583 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.583 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:45.583 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167523+0000", 2024-09-21T15:49:45.584 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9810000000000002e-06 2024-09-21T15:49:45.584 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.584 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.584 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:45.584 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167529+0000", 2024-09-21T15:49:45.584 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0900000000000001e-06 2024-09-21T15:49:45.584 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.584 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.584 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:45.584 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167533+0000", 2024-09-21T15:49:45.584 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5730000000000001e-06 2024-09-21T15:49:45.584 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.584 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.584 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:45.584 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215031+0000", 2024-09-21T15:49:45.585 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047497893999999999 2024-09-21T15:49:45.585 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.585 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.585 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:45.585 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215099+0000", 2024-09-21T15:49:45.585 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8416000000000004e-05 2024-09-21T15:49:45.585 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.585 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.585 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:45.585 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215171+0000", 2024-09-21T15:49:45.585 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1279000000000007e-05 2024-09-21T15:49:45.585 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.585 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:45.585 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.585 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.586 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.586 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64030 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1058816~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:45.586 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167537+0000", 2024-09-21T15:49:45.586 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 8.4010273340000001, 2024-09-21T15:49:45.586 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047783816999999999, 2024-09-21T15:49:45.586 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:45.586 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:45.586 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:45.586 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:45.586 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:45.586 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64030 2024-09-21T15:49:45.586 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.586 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:45.587 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.587 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:45.587 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167537+0000", 2024-09-21T15:49:45.587 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.587 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.587 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.587 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:45.587 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167537+0000", 2024-09-21T15:49:45.587 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.587 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.587 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.587 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:45.587 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167537+0000", 2024-09-21T15:49:45.587 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:45.588 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.588 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.588 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:45.588 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167539+0000", 2024-09-21T15:49:45.588 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9700000000000002e-06 2024-09-21T15:49:45.588 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.588 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.588 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:45.588 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167545+0000", 2024-09-21T15:49:45.588 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8629999999999996e-06 2024-09-21T15:49:45.588 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.588 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.588 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:45.588 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167549+0000", 2024-09-21T15:49:45.588 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.386e-06 2024-09-21T15:49:45.589 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.589 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.589 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:45.589 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215197+0000", 2024-09-21T15:49:45.589 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047647241 2024-09-21T15:49:45.589 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.589 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.589 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:45.589 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215263+0000", 2024-09-21T15:49:45.589 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6295000000000001e-05 2024-09-21T15:49:45.589 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.589 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.589 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:45.589 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215321+0000", 2024-09-21T15:49:45.590 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8362000000000001e-05 2024-09-21T15:49:45.590 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.590 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:45.590 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.590 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.590 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.590 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64031 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1738752~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:45.590 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167554+0000", 2024-09-21T15:49:45.590 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 8.4010108310000007, 2024-09-21T15:49:45.590 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047917855000000002, 2024-09-21T15:49:45.590 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:45.590 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:45.590 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:45.590 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:45.590 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:45.591 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64031 2024-09-21T15:49:45.591 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.591 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:45.591 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.591 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:45.591 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167554+0000", 2024-09-21T15:49:45.591 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.591 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.591 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.591 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:45.591 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167554+0000", 2024-09-21T15:49:45.591 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.591 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.591 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.592 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:45.592 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167554+0000", 2024-09-21T15:49:45.592 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:45.592 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.592 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.592 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:45.592 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167556+0000", 2024-09-21T15:49:45.592 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3870000000000002e-06 2024-09-21T15:49:45.592 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.592 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.592 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:45.592 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167562+0000", 2024-09-21T15:49:45.592 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9100000000000002e-06 2024-09-21T15:49:45.592 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.592 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.593 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:45.593 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167566+0000", 2024-09-21T15:49:45.593 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6029999999999998e-06 2024-09-21T15:49:45.593 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.593 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.593 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:45.593 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215346+0000", 2024-09-21T15:49:45.593 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.04777998 2024-09-21T15:49:45.593 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.593 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.593 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:45.593 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215412+0000", 2024-09-21T15:49:45.593 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5598000000000005e-05 2024-09-21T15:49:45.593 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.593 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.594 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:45.594 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215472+0000", 2024-09-21T15:49:45.594 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9728000000000001e-05 2024-09-21T15:49:45.594 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.594 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:45.594 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.594 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.594 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.594 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64032 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1067008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:45.594 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167673+0000", 2024-09-21T15:49:45.594 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 8.4008921179999998, 2024-09-21T15:49:45.594 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047947667999999999, 2024-09-21T15:49:45.594 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:45.594 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:45.595 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:45.595 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:45.595 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:45.595 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64032 2024-09-21T15:49:45.595 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.595 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:45.595 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.595 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:45.595 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167673+0000", 2024-09-21T15:49:45.595 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.595 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.595 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.595 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:45.595 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167673+0000", 2024-09-21T15:49:45.595 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.596 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.596 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.596 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:45.596 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167672+0000", 2024-09-21T15:49:45.596 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:45.596 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.596 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.596 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:45.596 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167675+0000", 2024-09-21T15:49:45.596 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3780000000000001e-06 2024-09-21T15:49:45.596 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.596 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.596 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:45.596 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167685+0000", 2024-09-21T15:49:45.596 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3039999999999994e-06 2024-09-21T15:49:45.597 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.597 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.597 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:45.597 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167692+0000", 2024-09-21T15:49:45.597 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2559999999999996e-06 2024-09-21T15:49:45.597 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.597 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.597 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:45.597 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215495+0000", 2024-09-21T15:49:45.597 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047803267000000003 2024-09-21T15:49:45.597 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.597 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.597 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:45.597 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215560+0000", 2024-09-21T15:49:45.597 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4703999999999996e-05 2024-09-21T15:49:45.598 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.598 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.598 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:45.598 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215620+0000", 2024-09-21T15:49:45.598 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0328000000000002e-05 2024-09-21T15:49:45.598 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.598 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:45.598 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.598 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.598 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.598 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64033 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1075200~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:45.598 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167697+0000", 2024-09-21T15:49:45.598 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 8.4008675460000006, 2024-09-21T15:49:45.598 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048082863000000003, 2024-09-21T15:49:45.599 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:45.599 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:45.599 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:45.599 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:45.599 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:45.599 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64033 2024-09-21T15:49:45.599 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.599 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:45.599 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.599 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:45.599 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167697+0000", 2024-09-21T15:49:45.599 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.599 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.599 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.599 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:45.600 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167697+0000", 2024-09-21T15:49:45.600 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.600 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.600 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.600 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:45.600 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167697+0000", 2024-09-21T15:49:45.600 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:45.600 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.600 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.600 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:45.600 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167699+0000", 2024-09-21T15:49:45.600 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5299999999999999e-06 2024-09-21T15:49:45.600 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.600 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.601 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:45.601 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167706+0000", 2024-09-21T15:49:45.601 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7490000000000001e-06 2024-09-21T15:49:45.601 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.601 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.601 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:45.601 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167712+0000", 2024-09-21T15:49:45.601 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1940000000000003e-06 2024-09-21T15:49:45.601 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.601 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.601 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:45.601 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215643+0000", 2024-09-21T15:49:45.601 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047930647999999999 2024-09-21T15:49:45.601 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.601 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.601 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:45.602 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215713+0000", 2024-09-21T15:49:45.602 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0488999999999995e-05 2024-09-21T15:49:45.602 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.602 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.602 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:45.602 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215780+0000", 2024-09-21T15:49:45.602 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6656999999999996e-05 2024-09-21T15:49:45.602 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.602 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:45.602 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.602 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.602 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.602 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64034 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1083392~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:45.602 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167717+0000", 2024-09-21T15:49:45.603 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 8.40084798, 2024-09-21T15:49:45.603 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048233316999999998, 2024-09-21T15:49:45.603 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:45.603 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:45.603 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:45.603 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:45.603 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:45.603 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64034 2024-09-21T15:49:45.603 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.603 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:45.603 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.603 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:45.603 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167717+0000", 2024-09-21T15:49:45.603 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.603 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.604 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.604 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:45.604 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167717+0000", 2024-09-21T15:49:45.604 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.604 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.604 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.604 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:45.604 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167716+0000", 2024-09-21T15:49:45.604 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:45.604 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.604 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.604 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:45.604 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167718+0000", 2024-09-21T15:49:45.604 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.165e-06 2024-09-21T15:49:45.604 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.605 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.605 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:45.605 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167725+0000", 2024-09-21T15:49:45.605 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3380000000000001e-06 2024-09-21T15:49:45.605 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.605 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.605 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:45.605 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167730+0000", 2024-09-21T15:49:45.605 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8010000000000003e-06 2024-09-21T15:49:45.605 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.605 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.605 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:45.605 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215807+0000", 2024-09-21T15:49:45.605 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048077823999999998 2024-09-21T15:49:45.605 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.606 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.606 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:45.606 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215884+0000", 2024-09-21T15:49:45.606 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6283000000000001e-05 2024-09-21T15:49:45.606 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.606 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.606 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:45.606 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215950+0000", 2024-09-21T15:49:45.606 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6290000000000004e-05 2024-09-21T15:49:45.606 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.606 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:45.606 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.606 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.606 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.606 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64035 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1091584~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:45.607 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:49:45.607 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 8.4008307670000004, 2024-09-21T15:49:45.607 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048412117999999997, 2024-09-21T15:49:45.607 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:45.607 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:45.607 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:45.607 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:45.607 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:45.607 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64035 2024-09-21T15:49:45.607 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.607 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:45.607 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.607 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:45.607 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:49:45.608 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.608 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.608 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.608 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:45.608 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:49:45.608 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.608 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.608 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.608 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:45.608 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:49:45.608 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:45.608 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.608 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.608 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:45.608 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167746+0000", 2024-09-21T15:49:45.609 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2031000000000001e-05 2024-09-21T15:49:45.609 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.609 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.609 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:45.609 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167753+0000", 2024-09-21T15:49:45.609 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0620000000000003e-06 2024-09-21T15:49:45.609 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.609 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.609 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:45.609 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167759+0000", 2024-09-21T15:49:45.609 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.596e-06 2024-09-21T15:49:45.609 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.609 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.609 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:45.609 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215977+0000", 2024-09-21T15:49:45.609 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048217318000000002 2024-09-21T15:49:45.610 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.610 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.610 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:45.610 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216076+0000", 2024-09-21T15:49:45.610 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9846000000000006e-05 2024-09-21T15:49:45.610 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.610 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.610 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:45.610 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216146+0000", 2024-09-21T15:49:45.610 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9578000000000004e-05 2024-09-21T15:49:45.610 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.610 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:45.610 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.610 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.611 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.611 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64036 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1099776~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:45.611 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167853+0000", 2024-09-21T15:49:45.611 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 8.4007121540000007, 2024-09-21T15:49:45.611 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048450675999999998, 2024-09-21T15:49:45.611 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:45.611 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:45.611 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:45.611 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:45.611 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:45.611 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64036 2024-09-21T15:49:45.611 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.611 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:45.611 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.611 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:45.612 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167853+0000", 2024-09-21T15:49:45.612 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.612 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.612 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.612 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:45.612 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167853+0000", 2024-09-21T15:49:45.612 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.612 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.612 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.612 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:45.612 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167852+0000", 2024-09-21T15:49:45.612 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:45.612 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.612 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.612 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:45.613 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167855+0000", 2024-09-21T15:49:45.613 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5640000000000001e-06 2024-09-21T15:49:45.613 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.613 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.613 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:45.613 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167864+0000", 2024-09-21T15:49:45.613 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8140000000000007e-06 2024-09-21T15:49:45.613 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.613 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.613 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:45.613 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167871+0000", 2024-09-21T15:49:45.613 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5729999999999996e-06 2024-09-21T15:49:45.613 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.613 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.613 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:45.613 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216174+0000", 2024-09-21T15:49:45.614 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048303437999999997 2024-09-21T15:49:45.614 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.614 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.614 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:45.614 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216250+0000", 2024-09-21T15:49:45.614 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5661999999999995e-05 2024-09-21T15:49:45.614 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.614 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.614 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:45.614 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216303+0000", 2024-09-21T15:49:45.614 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3273000000000003e-05 2024-09-21T15:49:45.614 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.614 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:45.614 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.615 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.615 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.615 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64037 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1771520~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:45.615 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:49:45.615 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 8.4006886109999996, 2024-09-21T15:49:45.615 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048588771000000003, 2024-09-21T15:49:45.615 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:45.615 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:45.615 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:45.615 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:45.615 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:45.615 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64037 2024-09-21T15:49:45.615 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.615 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:45.615 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.616 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:45.616 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:49:45.616 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.616 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.616 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.616 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:45.616 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:49:45.616 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.616 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.616 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.616 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:45.616 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:49:45.616 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:45.617 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.617 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.617 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:45.617 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167878+0000", 2024-09-21T15:49:45.617 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3e-06 2024-09-21T15:49:45.617 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.617 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.617 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:45.617 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167884+0000", 2024-09-21T15:49:45.617 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3269999999999997e-06 2024-09-21T15:49:45.617 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.617 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.617 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:45.617 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167889+0000", 2024-09-21T15:49:45.617 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9269999999999996e-06 2024-09-21T15:49:45.618 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.618 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.618 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:45.618 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216331+0000", 2024-09-21T15:49:45.618 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048442102000000001 2024-09-21T15:49:45.618 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.618 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.618 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:45.618 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216393+0000", 2024-09-21T15:49:45.618 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1132999999999999e-05 2024-09-21T15:49:45.618 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.618 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.618 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:45.618 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216465+0000", 2024-09-21T15:49:45.619 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2339e-05 2024-09-21T15:49:45.619 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.619 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:45.619 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.619 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.619 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.619 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64038 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1779712~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:45.619 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:49:45.619 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 8.4006707800000004, 2024-09-21T15:49:45.619 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048749795999999998, 2024-09-21T15:49:45.619 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:45.619 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:45.619 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:45.619 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:45.619 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:45.620 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64038 2024-09-21T15:49:45.620 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.620 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:45.620 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.620 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:45.620 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:49:45.620 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.620 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.620 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.620 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:45.620 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:49:45.620 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.620 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.620 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.621 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:45.621 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:49:45.621 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:45.621 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.621 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.621 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:45.621 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167896+0000", 2024-09-21T15:49:45.621 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2419999999999999e-06 2024-09-21T15:49:45.621 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.621 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.621 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:45.621 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167902+0000", 2024-09-21T15:49:45.621 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0859999999999998e-06 2024-09-21T15:49:45.621 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.621 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.621 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:45.622 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167907+0000", 2024-09-21T15:49:45.622 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9679999999999997e-06 2024-09-21T15:49:45.622 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.622 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.622 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:45.622 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216506+0000", 2024-09-21T15:49:45.622 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048599374000000001 2024-09-21T15:49:45.622 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.622 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.622 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:45.622 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216579+0000", 2024-09-21T15:49:45.622 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2801999999999999e-05 2024-09-21T15:49:45.622 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.623 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.623 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:45.623 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216644+0000", 2024-09-21T15:49:45.623 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4715000000000006e-05 2024-09-21T15:49:45.623 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.623 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:45.623 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.623 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.623 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.623 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64039 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1107968~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:45.623 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:49:45.623 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 8.4006534249999998, 2024-09-21T15:49:45.623 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048911370000000003, 2024-09-21T15:49:45.623 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:45.624 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:45.624 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:45.624 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:45.624 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:45.624 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64039 2024-09-21T15:49:45.624 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.624 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:45.624 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.624 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:45.624 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:49:45.624 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.624 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.624 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.624 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:45.624 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:49:45.625 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.625 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.625 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.625 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:45.625 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:49:45.625 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:45.625 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.625 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.625 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:45.625 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167913+0000", 2024-09-21T15:49:45.625 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0839999999999999e-06 2024-09-21T15:49:45.625 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.625 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.625 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:45.626 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167921+0000", 2024-09-21T15:49:45.626 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8310000000000001e-06 2024-09-21T15:49:45.626 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.626 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.626 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:45.626 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167926+0000", 2024-09-21T15:49:45.626 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0270000000000003e-06 2024-09-21T15:49:45.626 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.626 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.626 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:45.626 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216680+0000", 2024-09-21T15:49:45.626 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048753890000000001 2024-09-21T15:49:45.626 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.626 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.627 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:45.627 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216757+0000", 2024-09-21T15:49:45.627 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6848999999999996e-05 2024-09-21T15:49:45.627 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.627 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.627 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:45.627 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216823+0000", 2024-09-21T15:49:45.627 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6068000000000005e-05 2024-09-21T15:49:45.627 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.627 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:45.627 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.627 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.627 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.627 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64040 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1787904~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:45.628 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167931+0000", 2024-09-21T15:49:45.628 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 8.4006341930000001, 2024-09-21T15:49:45.628 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049010561000000001, 2024-09-21T15:49:45.628 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:45.628 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:45.628 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:45.628 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:45.628 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:45.628 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64040 2024-09-21T15:49:45.628 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.628 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:45.628 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.628 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:45.628 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167931+0000", 2024-09-21T15:49:45.629 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.629 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.629 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.629 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:45.629 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167931+0000", 2024-09-21T15:49:45.629 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.629 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.629 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.629 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:45.629 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167930+0000", 2024-09-21T15:49:45.629 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:45.629 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.629 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.630 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:45.630 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167932+0000", 2024-09-21T15:49:45.630 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.198e-06 2024-09-21T15:49:45.630 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.630 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.630 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:45.630 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167938+0000", 2024-09-21T15:49:45.630 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.105e-06 2024-09-21T15:49:45.630 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.630 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.630 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:45.631 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167943+0000", 2024-09-21T15:49:45.631 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9429999999999999e-06 2024-09-21T15:49:45.631 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.631 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.631 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:45.631 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216848+0000", 2024-09-21T15:49:45.631 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048905200000000003 2024-09-21T15:49:45.631 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.631 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.631 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:45.631 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216903+0000", 2024-09-21T15:49:45.631 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4793999999999997e-05 2024-09-21T15:49:45.631 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.631 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.632 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:45.632 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216941+0000", 2024-09-21T15:49:45.632 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7790999999999998e-05 2024-09-21T15:49:45.632 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.632 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:45.632 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.632 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.632 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.632 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64041 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1796096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:45.632 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:49:45.632 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 8.4006172219999993, 2024-09-21T15:49:45.632 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049181457999999997, 2024-09-21T15:49:45.632 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:45.632 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:45.633 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:45.633 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:45.633 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:45.633 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64041 2024-09-21T15:49:45.633 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.633 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:45.633 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.633 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:45.633 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:49:45.633 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.633 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.633 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.633 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:45.633 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:49:45.634 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.634 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.634 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.634 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:45.634 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:49:45.634 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:45.634 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.634 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.634 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:45.634 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167949+0000", 2024-09-21T15:49:45.634 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0200000000000001e-06 2024-09-21T15:49:45.634 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.634 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.634 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:45.635 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167955+0000", 2024-09-21T15:49:45.635 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5990000000000001e-06 2024-09-21T15:49:45.635 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.635 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.635 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:45.635 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167959+0000", 2024-09-21T15:49:45.635 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5589999999999999e-06 2024-09-21T15:49:45.635 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.635 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.635 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:45.635 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216972+0000", 2024-09-21T15:49:45.635 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049012574000000003 2024-09-21T15:49:45.635 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.636 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.636 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:45.636 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217057+0000", 2024-09-21T15:49:45.636 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4820000000000004e-05 2024-09-21T15:49:45.636 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.636 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.636 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:45.636 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217129+0000", 2024-09-21T15:49:45.636 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2186000000000003e-05 2024-09-21T15:49:45.636 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.636 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:45.636 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.636 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.636 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.636 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64042 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1804288~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:45.637 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167964+0000", 2024-09-21T15:49:45.637 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 8.4006009309999996, 2024-09-21T15:49:45.637 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049333372, 2024-09-21T15:49:45.637 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:45.637 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:45.637 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:45.637 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:45.637 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:45.637 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64042 2024-09-21T15:49:45.637 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.637 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:45.637 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.637 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:45.637 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167964+0000", 2024-09-21T15:49:45.638 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.638 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.638 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.638 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:45.638 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167964+0000", 2024-09-21T15:49:45.638 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.638 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.638 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.638 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:45.638 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167963+0000", 2024-09-21T15:49:45.638 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:45.638 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.638 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.638 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:45.639 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167966+0000", 2024-09-21T15:49:45.639 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1119999999999999e-06 2024-09-21T15:49:45.639 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.639 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.639 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:45.639 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167975+0000", 2024-09-21T15:49:45.639 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1130000000000007e-06 2024-09-21T15:49:45.639 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.639 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.639 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:45.639 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167980+0000", 2024-09-21T15:49:45.639 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9599999999999999e-06 2024-09-21T15:49:45.639 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.640 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.640 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:45.640 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217158+0000", 2024-09-21T15:49:45.640 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049178569999999998 2024-09-21T15:49:45.640 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.640 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.640 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:45.640 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217232+0000", 2024-09-21T15:49:45.640 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3589000000000003e-05 2024-09-21T15:49:45.640 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.640 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.640 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:45.641 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217297+0000", 2024-09-21T15:49:45.641 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5339999999999994e-05 2024-09-21T15:49:45.641 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.641 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:45.641 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.641 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.641 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.641 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64043 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1812480~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:45.641 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.168085+0000", 2024-09-21T15:49:45.641 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 8.4004798639999994, 2024-09-21T15:49:45.641 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049365157999999999, 2024-09-21T15:49:45.641 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:45.641 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:45.642 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:45.642 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:45.642 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:45.642 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64043 2024-09-21T15:49:45.642 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.642 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:45.642 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.642 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:45.642 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168085+0000", 2024-09-21T15:49:45.642 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.642 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.642 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.643 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:45.643 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168085+0000", 2024-09-21T15:49:45.643 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.643 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.643 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.643 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:45.643 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168084+0000", 2024-09-21T15:49:45.643 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:45.643 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.643 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.643 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:45.643 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168088+0000", 2024-09-21T15:49:45.643 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5420000000000001e-06 2024-09-21T15:49:45.644 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.644 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.644 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:45.644 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168097+0000", 2024-09-21T15:49:45.645 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5410000000000006e-06 2024-09-21T15:49:45.645 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.645 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.645 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:45.646 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168104+0000", 2024-09-21T15:49:45.646 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6719999999999998e-06 2024-09-21T15:49:45.646 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.646 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.646 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:45.646 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217323+0000", 2024-09-21T15:49:45.646 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049218857999999997 2024-09-21T15:49:45.646 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.646 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.646 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:45.646 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217397+0000", 2024-09-21T15:49:45.646 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3932999999999998e-05 2024-09-21T15:49:45.646 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.646 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.647 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:45.647 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217450+0000", 2024-09-21T15:49:45.647 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3168000000000003e-05 2024-09-21T15:49:45.647 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.647 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:45.647 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.647 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.647 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.647 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64044 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1820672~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:45.647 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.170036+0000", 2024-09-21T15:49:45.647 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 8.3985285449999996, 2024-09-21T15:49:45.647 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047578069000000001, 2024-09-21T15:49:45.647 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:45.647 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:45.647 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:45.648 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:45.648 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:45.648 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64044 2024-09-21T15:49:45.648 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.648 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:45.648 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.648 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:45.648 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170036+0000", 2024-09-21T15:49:45.648 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.648 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.648 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.648 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:45.648 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170036+0000", 2024-09-21T15:49:45.649 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.649 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.649 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.649 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:45.649 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168109+0000", 2024-09-21T15:49:45.649 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9980721 2024-09-21T15:49:45.649 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.649 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.649 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:45.649 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170040+0000", 2024-09-21T15:49:45.649 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0019318079999999999 2024-09-21T15:49:45.649 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.649 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.649 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:45.649 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170051+0000", 2024-09-21T15:49:45.650 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0220999999999999e-05 2024-09-21T15:49:45.650 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.650 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.650 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:45.650 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170058+0000", 2024-09-21T15:49:45.650 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0110000000000003e-06 2024-09-21T15:49:45.650 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.650 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.650 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:45.650 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217478+0000", 2024-09-21T15:49:45.650 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047420334000000001 2024-09-21T15:49:45.650 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.650 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.650 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:45.651 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217547+0000", 2024-09-21T15:49:45.651 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9416000000000001e-05 2024-09-21T15:49:45.651 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.651 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.651 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:45.651 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217614+0000", 2024-09-21T15:49:45.651 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6909000000000006e-05 2024-09-21T15:49:45.651 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.651 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:45.651 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.651 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.651 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.651 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64045 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1828864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:45.651 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.170064+0000", 2024-09-21T15:49:45.652 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 8.3985005059999995, 2024-09-21T15:49:45.652 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047709252000000001, 2024-09-21T15:49:45.652 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:45.652 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:45.652 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:45.652 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:45.652 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:45.652 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64045 2024-09-21T15:49:45.652 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.652 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:45.652 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.652 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:45.652 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170064+0000", 2024-09-21T15:49:45.652 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.653 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.653 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.653 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:45.653 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170064+0000", 2024-09-21T15:49:45.653 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.653 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.653 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.653 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:45.653 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170064+0000", 2024-09-21T15:49:45.653 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:45.653 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.653 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.653 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:45.653 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170066+0000", 2024-09-21T15:49:45.653 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8109999999999999e-06 2024-09-21T15:49:45.654 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.654 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.654 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:45.654 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170073+0000", 2024-09-21T15:49:45.654 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5570000000000001e-06 2024-09-21T15:49:45.654 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.654 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.654 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:45.654 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170078+0000", 2024-09-21T15:49:45.654 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9910000000000002e-06 2024-09-21T15:49:45.654 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.654 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.654 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:45.654 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217640+0000", 2024-09-21T15:49:45.655 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047561840000000001 2024-09-21T15:49:45.655 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.655 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.655 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:45.655 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217710+0000", 2024-09-21T15:49:45.655 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9967000000000003e-05 2024-09-21T15:49:45.655 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.655 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.655 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:45.655 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217773+0000", 2024-09-21T15:49:45.655 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3638000000000003e-05 2024-09-21T15:49:45.655 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.655 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:45.655 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.656 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.656 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.656 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64356 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 3076096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:45.656 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.183065+0000", 2024-09-21T15:49:45.656 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 8.3854993750000002, 2024-09-21T15:49:45.656 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047484923999999998, 2024-09-21T15:49:45.656 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:45.656 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:45.656 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:45.656 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:45.656 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:45.656 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64356 2024-09-21T15:49:45.657 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.657 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:45.657 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.657 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:45.657 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.183065+0000", 2024-09-21T15:49:45.657 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.657 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.657 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.657 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:45.657 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.183065+0000", 2024-09-21T15:49:45.657 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:45.657 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.657 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.657 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:45.658 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.182922+0000", 2024-09-21T15:49:45.658 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999856 2024-09-21T15:49:45.658 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.658 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.658 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:45.658 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.183069+0000", 2024-09-21T15:49:45.658 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000147097 2024-09-21T15:49:45.658 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.658 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.658 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:45.658 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.183078+0000", 2024-09-21T15:49:45.658 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.9099999999999994e-06 2024-09-21T15:49:45.658 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.659 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.659 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:45.659 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.183085+0000", 2024-09-21T15:49:45.659 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0670000000000003e-06 2024-09-21T15:49:45.659 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.659 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.659 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:45.659 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.230377+0000", 2024-09-21T15:49:45.659 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047292698000000001 2024-09-21T15:49:45.659 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.659 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.659 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:45.659 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.230458+0000", 2024-09-21T15:49:45.659 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0980000000000001e-05 2024-09-21T15:49:45.660 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:45.660 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:45.660 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:45.660 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.230550+0000", 2024-09-21T15:49:45.660 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1954999999999996e-05 2024-09-21T15:49:45.660 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.660 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:45.660 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.660 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:45.660 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:45.660 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:45.662 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:45.661+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:45.662 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:49:45.763 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:45.761+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:45.764 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:45.764 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:49:45.764 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:49:45.764 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:45.772 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:49:45.863 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:45.862+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:45.874 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:45.874 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:49:45.874 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:49:45.874 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:49:45.874 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:45.880 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is set 2024-09-21T15:49:45.885 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:49:45.976 INFO:teuthology.orchestra.run.smithi089.stdout:checking clone #16, image image_client.0-clone16 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent17 2024-09-21T15:49:45.977 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:45.975+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:46.054 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:46.055 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:49:46.055 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:49:46.055 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:49:46.055 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.055 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65283 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2836480~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:46.055 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.218024+0000", 2024-09-21T15:49:46.055 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.832582596, 2024-09-21T15:49:46.055 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043727861, 2024-09-21T15:49:46.055 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:46.055 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:46.055 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:46.055 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:46.056 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:46.056 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65283 2024-09-21T15:49:46.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.056 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:46.056 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.056 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:46.056 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218024+0000", 2024-09-21T15:49:46.056 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.056 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.056 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:46.056 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218024+0000", 2024-09-21T15:49:46.056 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.057 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:46.057 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218024+0000", 2024-09-21T15:49:46.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:46.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.057 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:46.057 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218026+0000", 2024-09-21T15:49:46.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.717e-06 2024-09-21T15:49:46.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.057 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:46.057 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218030+0000", 2024-09-21T15:49:46.058 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8630000000000001e-06 2024-09-21T15:49:46.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.058 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:46.058 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218033+0000", 2024-09-21T15:49:46.058 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9660000000000001e-06 2024-09-21T15:49:46.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.058 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:46.059 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.261601+0000", 2024-09-21T15:49:46.059 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043567862999999998 2024-09-21T15:49:46.059 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.059 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.059 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:46.059 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.261679+0000", 2024-09-21T15:49:46.059 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7807999999999998e-05 2024-09-21T15:49:46.059 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.059 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.059 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:46.059 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.261752+0000", 2024-09-21T15:49:46.059 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2977000000000003e-05 2024-09-21T15:49:46.059 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.059 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:46.059 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.060 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.060 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.060 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65284 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2844672~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:46.060 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.218037+0000", 2024-09-21T15:49:46.060 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.8325696730000001, 2024-09-21T15:49:46.060 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043873034999999998, 2024-09-21T15:49:46.060 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:46.060 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:46.060 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:46.060 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:46.060 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:46.060 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65284 2024-09-21T15:49:46.060 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.060 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:46.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.061 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:46.061 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218037+0000", 2024-09-21T15:49:46.061 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.061 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:46.061 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218037+0000", 2024-09-21T15:49:46.061 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.061 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:46.061 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218037+0000", 2024-09-21T15:49:46.061 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:49:46.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.062 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:46.062 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218038+0000", 2024-09-21T15:49:46.062 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.533e-06 2024-09-21T15:49:46.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.062 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:46.062 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218041+0000", 2024-09-21T15:49:46.062 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9799999999999998e-06 2024-09-21T15:49:46.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.062 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:46.062 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218044+0000", 2024-09-21T15:49:46.063 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.458e-06 2024-09-21T15:49:46.063 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.063 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.063 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:46.063 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.261781+0000", 2024-09-21T15:49:46.063 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043736935999999997 2024-09-21T15:49:46.063 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.063 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.063 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:46.063 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.261846+0000", 2024-09-21T15:49:46.063 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5233000000000004e-05 2024-09-21T15:49:46.063 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.063 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.063 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:46.063 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.261910+0000", 2024-09-21T15:49:46.064 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4103000000000006e-05 2024-09-21T15:49:46.064 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.064 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:46.064 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.064 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.064 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.064 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65285 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2861056~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:46.064 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.218171+0000", 2024-09-21T15:49:46.064 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.8324355259999994, 2024-09-21T15:49:46.064 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043909057000000001, 2024-09-21T15:49:46.064 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:46.064 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:46.064 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:46.064 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:46.064 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:46.065 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65285 2024-09-21T15:49:46.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.065 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:46.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.065 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:46.065 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218171+0000", 2024-09-21T15:49:46.065 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.065 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:46.065 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218171+0000", 2024-09-21T15:49:46.065 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.065 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:46.066 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218171+0000", 2024-09-21T15:49:46.066 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:46.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.066 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.066 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:46.066 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218173+0000", 2024-09-21T15:49:46.066 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.588e-06 2024-09-21T15:49:46.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.066 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.066 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:46.066 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218178+0000", 2024-09-21T15:49:46.066 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4719999999999997e-06 2024-09-21T15:49:46.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.067 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:46.067 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218181+0000", 2024-09-21T15:49:46.067 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0979999999999998e-06 2024-09-21T15:49:46.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.067 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:46.067 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.261943+0000", 2024-09-21T15:49:46.067 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043762652999999999 2024-09-21T15:49:46.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.067 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:46.067 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262018+0000", 2024-09-21T15:49:46.067 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4352999999999998e-05 2024-09-21T15:49:46.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.068 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.068 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:46.068 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262080+0000", 2024-09-21T15:49:46.068 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2477999999999994e-05 2024-09-21T15:49:46.068 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.068 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:46.068 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.068 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.068 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.068 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65286 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2869248~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:46.068 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.218284+0000", 2024-09-21T15:49:46.068 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.8323230430000006, 2024-09-21T15:49:46.068 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043921241999999999, 2024-09-21T15:49:46.068 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:46.068 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:46.069 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:46.069 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:46.069 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:46.069 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65286 2024-09-21T15:49:46.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.069 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:46.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.069 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:46.069 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218284+0000", 2024-09-21T15:49:46.069 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.069 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:46.069 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218284+0000", 2024-09-21T15:49:46.070 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.070 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.070 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:46.070 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218283+0000", 2024-09-21T15:49:46.070 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:46.070 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.070 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:46.070 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218286+0000", 2024-09-21T15:49:46.070 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4159999999999998e-06 2024-09-21T15:49:46.070 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.070 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:46.070 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218290+0000", 2024-09-21T15:49:46.071 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3370000000000001e-06 2024-09-21T15:49:46.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.071 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:46.071 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218293+0000", 2024-09-21T15:49:46.071 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0960000000000001e-06 2024-09-21T15:49:46.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.071 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:46.071 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262105+0000", 2024-09-21T15:49:46.071 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043811914 2024-09-21T15:49:46.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.071 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:46.072 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262152+0000", 2024-09-21T15:49:46.072 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7341e-05 2024-09-21T15:49:46.072 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.072 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.072 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:46.072 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262205+0000", 2024-09-21T15:49:46.072 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2667000000000002e-05 2024-09-21T15:49:46.072 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.072 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:46.072 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.072 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.072 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.072 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65287 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2877440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:46.072 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.218326+0000", 2024-09-21T15:49:46.072 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.8322807799999996, 2024-09-21T15:49:46.073 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.044001874000000003, 2024-09-21T15:49:46.073 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:46.073 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:46.073 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:46.073 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:46.073 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:46.073 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65287 2024-09-21T15:49:46.073 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.073 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:46.073 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.073 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:46.073 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218326+0000", 2024-09-21T15:49:46.073 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.073 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.073 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.074 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:46.074 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218326+0000", 2024-09-21T15:49:46.074 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.074 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.074 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.074 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:46.074 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218326+0000", 2024-09-21T15:49:46.074 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:46.074 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.074 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.074 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:46.074 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218328+0000", 2024-09-21T15:49:46.074 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.029e-06 2024-09-21T15:49:46.074 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:46.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218334+0000", 2024-09-21T15:49:46.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9519999999999999e-06 2024-09-21T15:49:46.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:46.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218338+0000", 2024-09-21T15:49:46.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7500000000000003e-06 2024-09-21T15:49:46.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:46.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262228+0000", 2024-09-21T15:49:46.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043890058000000003 2024-09-21T15:49:46.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.076 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:46.076 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262285+0000", 2024-09-21T15:49:46.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6779e-05 2024-09-21T15:49:46.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.076 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:46.076 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262328+0000", 2024-09-21T15:49:46.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2599999999999999e-05 2024-09-21T15:49:46.076 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.076 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:46.076 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.077 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.077 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65288 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2885632~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:46.077 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.218345+0000", 2024-09-21T15:49:46.077 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.8322620480000005, 2024-09-21T15:49:46.077 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.044143245999999997, 2024-09-21T15:49:46.077 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:46.077 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:46.077 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:46.077 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:46.077 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:46.077 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65288 2024-09-21T15:49:46.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.077 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:46.077 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.077 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:46.078 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218345+0000", 2024-09-21T15:49:46.078 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.078 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.078 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:46.078 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218345+0000", 2024-09-21T15:49:46.078 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.078 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.078 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:46.078 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218344+0000", 2024-09-21T15:49:46.078 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:46.078 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.079 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:46.079 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218347+0000", 2024-09-21T15:49:46.079 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3920000000000001e-06 2024-09-21T15:49:46.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.079 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.079 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:46.079 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218353+0000", 2024-09-21T15:49:46.079 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9610000000000002e-06 2024-09-21T15:49:46.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.079 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.079 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:46.079 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218357+0000", 2024-09-21T15:49:46.079 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4700000000000004e-06 2024-09-21T15:49:46.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.079 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.080 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:46.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262405+0000", 2024-09-21T15:49:46.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.044047500000000003 2024-09-21T15:49:46.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.080 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:46.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262446+0000", 2024-09-21T15:49:46.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1151e-05 2024-09-21T15:49:46.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.080 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:46.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262488+0000", 2024-09-21T15:49:46.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2182999999999999e-05 2024-09-21T15:49:46.080 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.080 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:46.081 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.081 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65289 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2893824~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:46.081 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.218394+0000", 2024-09-21T15:49:46.081 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.8322128450000008, 2024-09-21T15:49:46.081 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.044235021999999999, 2024-09-21T15:49:46.081 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:46.081 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:46.081 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:46.081 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:46.081 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:46.081 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65289 2024-09-21T15:49:46.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.081 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:46.082 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.082 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:46.082 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218394+0000", 2024-09-21T15:49:46.082 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.082 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.082 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.082 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:46.082 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218394+0000", 2024-09-21T15:49:46.082 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.082 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.082 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.082 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:46.082 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218393+0000", 2024-09-21T15:49:46.082 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:46.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.083 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:46.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218397+0000", 2024-09-21T15:49:46.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.27e-06 2024-09-21T15:49:46.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.083 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:46.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218404+0000", 2024-09-21T15:49:46.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6329999999999996e-06 2024-09-21T15:49:46.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.083 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:46.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218415+0000", 2024-09-21T15:49:46.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0373999999999999e-05 2024-09-21T15:49:46.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.084 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:46.084 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262511+0000", 2024-09-21T15:49:46.084 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.044096630999999997 2024-09-21T15:49:46.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.084 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:46.084 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262562+0000", 2024-09-21T15:49:46.084 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1266e-05 2024-09-21T15:49:46.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.084 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:46.084 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262629+0000", 2024-09-21T15:49:46.085 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6388000000000002e-05 2024-09-21T15:49:46.085 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.085 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:46.085 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.085 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.085 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65290 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2852864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:46.085 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.218451+0000", 2024-09-21T15:49:46.085 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.8321552380000004, 2024-09-21T15:49:46.085 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.044280721000000002, 2024-09-21T15:49:46.085 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:46.085 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:46.085 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:46.085 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:46.085 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:46.086 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65290 2024-09-21T15:49:46.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.086 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:46.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.086 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:46.086 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218451+0000", 2024-09-21T15:49:46.086 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.086 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:46.086 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218451+0000", 2024-09-21T15:49:46.086 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.087 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:46.087 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218451+0000", 2024-09-21T15:49:46.087 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:46.087 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.087 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.087 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:46.087 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218454+0000", 2024-09-21T15:49:46.087 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7489999999999999e-06 2024-09-21T15:49:46.087 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.087 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.087 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:46.087 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218462+0000", 2024-09-21T15:49:46.087 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5329999999999994e-06 2024-09-21T15:49:46.087 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.087 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.088 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:46.088 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218465+0000", 2024-09-21T15:49:46.088 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1990000000000002e-06 2024-09-21T15:49:46.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.088 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:46.088 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262655+0000", 2024-09-21T15:49:46.088 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.044189472 2024-09-21T15:49:46.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.088 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:46.088 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262695+0000", 2024-09-21T15:49:46.088 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0476999999999999e-05 2024-09-21T15:49:46.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.089 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:46.089 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262732+0000", 2024-09-21T15:49:46.089 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6795000000000003e-05 2024-09-21T15:49:46.089 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.089 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:46.089 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.089 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.089 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65291 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2902016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:46.089 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.218644+0000", 2024-09-21T15:49:46.089 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.8319627220000001, 2024-09-21T15:49:46.089 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.044258391000000001, 2024-09-21T15:49:46.089 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:46.090 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:46.090 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:46.090 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:46.090 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:46.090 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65291 2024-09-21T15:49:46.090 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.090 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:46.090 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.090 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:46.090 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218644+0000", 2024-09-21T15:49:46.090 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.090 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.090 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.090 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:46.090 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218644+0000", 2024-09-21T15:49:46.091 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.091 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:46.091 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218468+0000", 2024-09-21T15:49:46.091 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999824 2024-09-21T15:49:46.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.091 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:46.091 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218649+0000", 2024-09-21T15:49:46.091 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00018061999999999999 2024-09-21T15:49:46.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.091 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:46.092 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218657+0000", 2024-09-21T15:49:46.092 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6430000000000001e-06 2024-09-21T15:49:46.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.092 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:46.092 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218662+0000", 2024-09-21T15:49:46.092 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.814e-06 2024-09-21T15:49:46.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.092 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:46.092 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262766+0000", 2024-09-21T15:49:46.092 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.044103705999999999 2024-09-21T15:49:46.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.093 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:46.093 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262838+0000", 2024-09-21T15:49:46.093 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2312999999999997e-05 2024-09-21T15:49:46.093 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.093 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.093 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:46.093 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262902+0000", 2024-09-21T15:49:46.093 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4282999999999994e-05 2024-09-21T15:49:46.093 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.093 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:46.093 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.093 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.093 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.093 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65292 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2910208~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:46.093 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.218810+0000", 2024-09-21T15:49:46.094 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.8317965100000002, 2024-09-21T15:49:46.094 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.044257131999999998, 2024-09-21T15:49:46.094 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:46.094 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:46.094 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:46.094 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:46.094 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:46.094 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65292 2024-09-21T15:49:46.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.094 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:46.094 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.094 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:46.094 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218810+0000", 2024-09-21T15:49:46.094 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.095 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.095 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:46.095 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218810+0000", 2024-09-21T15:49:46.095 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.095 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.095 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:46.095 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218666+0000", 2024-09-21T15:49:46.095 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999856 2024-09-21T15:49:46.095 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.095 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:46.095 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218814+0000", 2024-09-21T15:49:46.095 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014768299999999999 2024-09-21T15:49:46.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:46.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218823+0000", 2024-09-21T15:49:46.096 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2399999999999996e-06 2024-09-21T15:49:46.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:46.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218829+0000", 2024-09-21T15:49:46.096 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8279999999999999e-06 2024-09-21T15:49:46.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:46.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262933+0000", 2024-09-21T15:49:46.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.044104065999999997 2024-09-21T15:49:46.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.097 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:46.097 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263014+0000", 2024-09-21T15:49:46.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0717999999999995e-05 2024-09-21T15:49:46.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.097 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:46.097 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263067+0000", 2024-09-21T15:49:46.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3653999999999998e-05 2024-09-21T15:49:46.097 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.097 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:46.097 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.098 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65293 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2918400~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:46.098 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.218930+0000", 2024-09-21T15:49:46.098 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.8316768289999992, 2024-09-21T15:49:46.098 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.044253536000000003, 2024-09-21T15:49:46.098 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:46.098 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:46.098 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:46.098 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:46.098 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:46.098 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65293 2024-09-21T15:49:46.098 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.098 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:46.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:46.099 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218930+0000", 2024-09-21T15:49:46.099 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.099 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:46.099 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218930+0000", 2024-09-21T15:49:46.099 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.099 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:46.099 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218834+0000", 2024-09-21T15:49:46.099 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999042 2024-09-21T15:49:46.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.100 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:46.100 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218933+0000", 2024-09-21T15:49:46.100 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9047000000000002e-05 2024-09-21T15:49:46.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.100 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:46.100 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218941+0000", 2024-09-21T15:49:46.100 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0849999999999997e-06 2024-09-21T15:49:46.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.100 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:46.100 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218945+0000", 2024-09-21T15:49:46.100 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9179999999999996e-06 2024-09-21T15:49:46.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.101 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:46.101 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263093+0000", 2024-09-21T15:49:46.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.044147906000000001 2024-09-21T15:49:46.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.101 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:46.101 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263144+0000", 2024-09-21T15:49:46.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0470999999999998e-05 2024-09-21T15:49:46.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.101 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:46.101 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263183+0000", 2024-09-21T15:49:46.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9839999999999998e-05 2024-09-21T15:49:46.101 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.102 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:46.102 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.102 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65294 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2926592~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:46.102 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.219137+0000", 2024-09-21T15:49:46.102 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.8314696529999992, 2024-09-21T15:49:46.102 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.044168171999999999, 2024-09-21T15:49:46.102 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:46.102 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:46.102 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:46.102 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:46.102 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:46.102 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65294 2024-09-21T15:49:46.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.103 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:46.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.103 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:46.103 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.219137+0000", 2024-09-21T15:49:46.103 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.103 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:46.103 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.219137+0000", 2024-09-21T15:49:46.103 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.103 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:46.103 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218952+0000", 2024-09-21T15:49:46.103 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999815 2024-09-21T15:49:46.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.104 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:46.104 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.219140+0000", 2024-09-21T15:49:46.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00018848899999999999 2024-09-21T15:49:46.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.104 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:46.104 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.219147+0000", 2024-09-21T15:49:46.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0010000000000004e-06 2024-09-21T15:49:46.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.104 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:46.104 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.219153+0000", 2024-09-21T15:49:46.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4060000000000004e-06 2024-09-21T15:49:46.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.105 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:46.105 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263206+0000", 2024-09-21T15:49:46.105 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.044053467999999998 2024-09-21T15:49:46.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.105 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:46.105 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263248+0000", 2024-09-21T15:49:46.105 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1282000000000002e-05 2024-09-21T15:49:46.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.105 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:46.105 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263305+0000", 2024-09-21T15:49:46.106 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7645000000000001e-05 2024-09-21T15:49:46.106 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.106 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:46.106 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.106 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65295 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2934784~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:46.106 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.219274+0000", 2024-09-21T15:49:46.106 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.8313328939999991, 2024-09-21T15:49:46.106 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.044207710999999997, 2024-09-21T15:49:46.106 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:46.106 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:46.106 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:46.106 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:46.106 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:46.107 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65295 2024-09-21T15:49:46.107 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.107 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:46.107 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.107 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:46.107 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.219274+0000", 2024-09-21T15:49:46.107 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.107 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.107 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.107 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:46.107 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.219274+0000", 2024-09-21T15:49:46.107 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.107 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.107 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.107 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:46.108 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.219158+0000", 2024-09-21T15:49:46.108 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9998837 2024-09-21T15:49:46.108 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.108 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.108 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:46.108 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.219276+0000", 2024-09-21T15:49:46.108 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000118656 2024-09-21T15:49:46.108 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.108 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.108 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:46.108 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.219281+0000", 2024-09-21T15:49:46.108 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7360000000000001e-06 2024-09-21T15:49:46.108 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.108 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.108 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:46.109 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.219285+0000", 2024-09-21T15:49:46.109 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6349999999999999e-06 2024-09-21T15:49:46.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.109 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:46.109 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263343+0000", 2024-09-21T15:49:46.109 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.044058264 2024-09-21T15:49:46.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.109 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:46.109 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263418+0000", 2024-09-21T15:49:46.109 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5001999999999998e-05 2024-09-21T15:49:46.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.109 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:46.110 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263482+0000", 2024-09-21T15:49:46.110 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3639999999999994e-05 2024-09-21T15:49:46.110 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.110 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:46.110 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.110 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.110 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.110 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65297 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2951168~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:46.110 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.220022+0000", 2024-09-21T15:49:46.110 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.8305850009999993, 2024-09-21T15:49:46.110 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043682718000000002, 2024-09-21T15:49:46.110 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:46.110 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:46.110 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:46.111 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:46.111 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:46.111 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65297 2024-09-21T15:49:46.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.111 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:46.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.111 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:46.111 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220022+0000", 2024-09-21T15:49:46.111 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.111 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:46.111 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220022+0000", 2024-09-21T15:49:46.111 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.112 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:46.112 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220021+0000", 2024-09-21T15:49:46.112 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:46.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.112 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:46.112 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220024+0000", 2024-09-21T15:49:46.112 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3870000000000002e-06 2024-09-21T15:49:46.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.112 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:46.112 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220029+0000", 2024-09-21T15:49:46.112 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4439999999999999e-06 2024-09-21T15:49:46.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.113 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:46.113 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220033+0000", 2024-09-21T15:49:46.113 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7359999999999999e-06 2024-09-21T15:49:46.113 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.113 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:46.113 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263625+0000", 2024-09-21T15:49:46.113 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043592190000000003 2024-09-21T15:49:46.113 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.113 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:46.113 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263667+0000", 2024-09-21T15:49:46.113 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2048000000000001e-05 2024-09-21T15:49:46.113 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.114 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:46.114 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263704+0000", 2024-09-21T15:49:46.114 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7295000000000001e-05 2024-09-21T15:49:46.114 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.114 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:46.114 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.114 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.114 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65298 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2959360~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:46.114 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.220036+0000", 2024-09-21T15:49:46.114 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.8305709730000004, 2024-09-21T15:49:46.114 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043821723999999999, 2024-09-21T15:49:46.114 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:46.114 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:46.115 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:46.115 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:46.115 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:46.115 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65298 2024-09-21T15:49:46.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.115 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:46.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:46.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220036+0000", 2024-09-21T15:49:46.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:46.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220036+0000", 2024-09-21T15:49:46.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.116 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:46.116 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220035+0000", 2024-09-21T15:49:46.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:46.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.116 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:46.116 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220037+0000", 2024-09-21T15:49:46.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.66e-06 2024-09-21T15:49:46.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.116 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:46.116 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220040+0000", 2024-09-21T15:49:46.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9699999999999999e-06 2024-09-21T15:49:46.117 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.117 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.117 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:46.117 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220042+0000", 2024-09-21T15:49:46.117 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4940000000000002e-06 2024-09-21T15:49:46.117 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.117 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.117 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:46.117 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263727+0000", 2024-09-21T15:49:46.117 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043684850999999997 2024-09-21T15:49:46.117 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.117 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.117 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:46.117 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263789+0000", 2024-09-21T15:49:46.117 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2147e-05 2024-09-21T15:49:46.118 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.118 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.118 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:46.118 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263857+0000", 2024-09-21T15:49:46.118 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7970000000000001e-05 2024-09-21T15:49:46.118 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.118 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:46.118 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.118 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.118 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.118 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65299 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2975744~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:46.118 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.220045+0000", 2024-09-21T15:49:46.118 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.8305621730000006, 2024-09-21T15:49:46.118 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043920945000000003, 2024-09-21T15:49:46.118 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:46.119 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:46.119 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:46.119 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:46.119 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:46.119 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65299 2024-09-21T15:49:46.119 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.119 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:46.119 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.119 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:46.119 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220045+0000", 2024-09-21T15:49:46.119 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.119 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.119 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.119 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:46.119 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220045+0000", 2024-09-21T15:49:46.120 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.120 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.120 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:46.120 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220044+0000", 2024-09-21T15:49:46.120 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:49:46.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.120 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.120 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:46.120 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220046+0000", 2024-09-21T15:49:46.120 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.207e-06 2024-09-21T15:49:46.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.120 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.120 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:46.120 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220048+0000", 2024-09-21T15:49:46.121 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7939999999999998e-06 2024-09-21T15:49:46.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.121 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:46.121 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220051+0000", 2024-09-21T15:49:46.121 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4770000000000002e-06 2024-09-21T15:49:46.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.121 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:46.121 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263885+0000", 2024-09-21T15:49:46.121 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043833719 2024-09-21T15:49:46.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.121 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:46.121 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263927+0000", 2024-09-21T15:49:46.122 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2409e-05 2024-09-21T15:49:46.122 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.122 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.122 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:46.122 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263965+0000", 2024-09-21T15:49:46.122 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8550999999999997e-05 2024-09-21T15:49:46.122 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.122 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:46.122 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.122 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.122 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.122 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65773 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2961408~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:46.122 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.241336+0000", 2024-09-21T15:49:46.122 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.8092706980000006, 2024-09-21T15:49:46.123 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043734599999999998, 2024-09-21T15:49:46.123 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:46.123 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:46.123 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:46.123 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:46.123 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:46.123 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65773 2024-09-21T15:49:46.123 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.123 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:46.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.123 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:46.123 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241336+0000", 2024-09-21T15:49:46.123 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.123 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.124 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.124 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:46.124 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241336+0000", 2024-09-21T15:49:46.124 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.124 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.124 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.124 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:46.124 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241336+0000", 2024-09-21T15:49:46.124 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:46.124 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.124 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.124 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:46.124 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241337+0000", 2024-09-21T15:49:46.124 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.505e-06 2024-09-21T15:49:46.124 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.125 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.125 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:46.125 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241340+0000", 2024-09-21T15:49:46.125 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8169999999999999e-06 2024-09-21T15:49:46.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.125 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.125 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:46.125 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241342+0000", 2024-09-21T15:49:46.125 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4820000000000001e-06 2024-09-21T15:49:46.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.125 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.125 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:46.125 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.284936+0000", 2024-09-21T15:49:46.125 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043593728999999998 2024-09-21T15:49:46.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.126 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.126 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:46.126 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.285002+0000", 2024-09-21T15:49:46.126 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5581999999999997e-05 2024-09-21T15:49:46.126 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.126 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.126 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:46.126 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.285071+0000", 2024-09-21T15:49:46.126 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8818000000000004e-05 2024-09-21T15:49:46.126 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.126 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:46.126 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.126 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.126 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.127 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65774 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2985984~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:46.127 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.241555+0000", 2024-09-21T15:49:46.127 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.8090518840000005, 2024-09-21T15:49:46.127 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043704701999999998, 2024-09-21T15:49:46.127 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:46.127 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:46.127 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:46.127 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:46.127 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:46.127 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65774 2024-09-21T15:49:46.127 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.127 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:46.127 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.127 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:46.127 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241555+0000", 2024-09-21T15:49:46.128 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.128 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.128 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.128 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:46.128 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241555+0000", 2024-09-21T15:49:46.128 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.128 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.128 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.128 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:46.128 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241344+0000", 2024-09-21T15:49:46.129 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9997897 2024-09-21T15:49:46.129 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.129 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.129 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:46.129 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241560+0000", 2024-09-21T15:49:46.129 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00021524700000000001 2024-09-21T15:49:46.129 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.129 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.129 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:46.129 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241571+0000", 2024-09-21T15:49:46.129 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1627000000000001e-05 2024-09-21T15:49:46.129 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.129 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.129 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:46.129 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241578+0000", 2024-09-21T15:49:46.130 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6889999999999998e-06 2024-09-21T15:49:46.130 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.130 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.130 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:46.130 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.285108+0000", 2024-09-21T15:49:46.130 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043529644999999999 2024-09-21T15:49:46.130 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.130 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.130 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:46.130 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.285188+0000", 2024-09-21T15:49:46.130 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0774999999999998e-05 2024-09-21T15:49:46.130 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.130 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.130 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:46.131 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.285260+0000", 2024-09-21T15:49:46.131 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1173000000000004e-05 2024-09-21T15:49:46.131 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.131 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:46.131 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.131 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.131 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.131 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65775 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2977792~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:46.131 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.241672+0000", 2024-09-21T15:49:46.131 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.8089346830000004, 2024-09-21T15:49:46.131 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043751534000000002, 2024-09-21T15:49:46.131 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:46.131 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:46.131 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:46.131 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:46.132 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:46.132 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65775 2024-09-21T15:49:46.132 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.132 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:46.132 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.132 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:46.132 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241672+0000", 2024-09-21T15:49:46.132 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.132 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.132 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.132 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:46.132 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241672+0000", 2024-09-21T15:49:46.132 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.132 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.132 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.133 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:46.133 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241584+0000", 2024-09-21T15:49:46.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999118 2024-09-21T15:49:46.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.133 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.133 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:46.133 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241676+0000", 2024-09-21T15:49:46.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2115999999999997e-05 2024-09-21T15:49:46.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.133 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.133 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:46.133 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241684+0000", 2024-09-21T15:49:46.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3149999999999992e-06 2024-09-21T15:49:46.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.133 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.134 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:46.134 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241691+0000", 2024-09-21T15:49:46.134 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2999999999999998e-06 2024-09-21T15:49:46.134 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.134 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.134 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:46.134 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.285289+0000", 2024-09-21T15:49:46.134 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043598494000000002 2024-09-21T15:49:46.134 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.134 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.134 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:46.134 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.285361+0000", 2024-09-21T15:49:46.134 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1731999999999997e-05 2024-09-21T15:49:46.134 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.134 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.135 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:46.135 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.285424+0000", 2024-09-21T15:49:46.135 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2577999999999996e-05 2024-09-21T15:49:46.135 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.135 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:46.135 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.135 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.135 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.135 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65776 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2994176~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:46.135 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.241867+0000", 2024-09-21T15:49:46.135 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.8087401019999998, 2024-09-21T15:49:46.135 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043720638999999999, 2024-09-21T15:49:46.135 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:46.135 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:46.136 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:46.136 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:46.136 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:46.136 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65776 2024-09-21T15:49:46.136 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.136 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:46.136 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.136 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:46.136 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241867+0000", 2024-09-21T15:49:46.136 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.136 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.136 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.136 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:46.136 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241867+0000", 2024-09-21T15:49:46.136 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.137 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:46.137 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241696+0000", 2024-09-21T15:49:46.137 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9998293 2024-09-21T15:49:46.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.137 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:46.137 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241871+0000", 2024-09-21T15:49:46.137 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00017491300000000001 2024-09-21T15:49:46.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.137 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:46.137 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241889+0000", 2024-09-21T15:49:46.137 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8065000000000001e-05 2024-09-21T15:49:46.138 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.138 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.138 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:46.138 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241897+0000", 2024-09-21T15:49:46.138 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9310000000000008e-06 2024-09-21T15:49:46.138 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.138 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.138 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:46.138 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.285446+0000", 2024-09-21T15:49:46.138 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043548735999999998 2024-09-21T15:49:46.138 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.138 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.138 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:46.138 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.285523+0000", 2024-09-21T15:49:46.138 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7014000000000005e-05 2024-09-21T15:49:46.139 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.139 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:46.139 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.285587+0000", 2024-09-21T15:49:46.139 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4455000000000005e-05 2024-09-21T15:49:46.139 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.139 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:46.139 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.139 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.139 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:46.139 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:46.141 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:49:46.141 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:46.139+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:46.186 INFO:teuthology.orchestra.run.smithi089.stdout:3258 write 0x72afd2d thru 0x72b778b (0x7a5f bytes) 2024-09-21T15:49:46.193 INFO:teuthology.orchestra.run.smithi089.stdout:3259 read 0x1ba402a thru 0x1badfab (0x9f82 bytes) 2024-09-21T15:49:46.193 INFO:teuthology.orchestra.run.smithi089.stdout:3260 writesame 0xa31615 thru 0xa380c1 data_size 0x1(0x6aad bytes) 2024-09-21T15:49:46.235 INFO:teuthology.orchestra.run.smithi089.stdout:3263 read 0x27915c6 thru 0x27943d0 (0x2e0b bytes) 2024-09-21T15:49:46.241 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:46.240+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:46.248 INFO:teuthology.orchestra.run.smithi089.stdout:3264 writesame 0xd68e5d4 thru 0xd699038 data_size 0x1(0xaa65 bytes) 2024-09-21T15:49:46.248 INFO:teuthology.orchestra.run.smithi089.stdout:3265 write 0x566b96d thru 0x56730aa (0x773e bytes) 2024-09-21T15:49:46.258 INFO:teuthology.orchestra.run.smithi089.stdout:3267 writesame 0xb3742bb thru 0xb380e2a data_size 0x7c(0xcb70 bytes) 2024-09-21T15:49:46.271 INFO:teuthology.orchestra.run.smithi089.stdout:3268 writesame 0xd7ec51c thru 0xd7f4e1a data_size 0x1(0x88ff bytes) 2024-09-21T15:49:46.277 INFO:teuthology.orchestra.run.smithi089.stdout:3269 trunc from 0xd7f4e1b to 0x8bca13f 2024-09-21T15:49:46.301 INFO:teuthology.orchestra.run.smithi089.stdout:3270 write 0xcdd1164 thru 0xcdd926b (0x8108 bytes) 2024-09-21T15:49:46.305 INFO:teuthology.orchestra.run.smithi089.stdout:3271 read 0x7e9be5c thru 0x7ea9b05 (0xdcaa bytes) 2024-09-21T15:49:46.310 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:46.310 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:49:46.310 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:49:46.310 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:46.313 INFO:teuthology.orchestra.run.smithi089.stdout:3272 write 0xe128bda thru 0xe131a2d (0x8e54 bytes) 2024-09-21T15:49:46.317 INFO:teuthology.orchestra.run.smithi089.stdout:3273 read 0xbf49ef2 thru 0xbf50a90 (0x6b9f bytes) 2024-09-21T15:49:46.317 INFO:teuthology.orchestra.run.smithi089.stdout:3274 read 0x81015b7 thru 0x8106e6d (0x58b7 bytes) 2024-09-21T15:49:46.318 INFO:teuthology.orchestra.run.smithi089.stdout:3275 write 0x1a31a67 thru 0x1a333c2 (0x195c bytes) 2024-09-21T15:49:46.321 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:49:46.321 INFO:teuthology.orchestra.run.smithi089.stdout:3276 writesame 0xb01d473 thru 0xb02260c data_size 0x5(0x519a bytes) 2024-09-21T15:49:46.326 INFO:teuthology.orchestra.run.smithi089.stdout:3277 write 0x7c733bb thru 0x7c76fec (0x3c32 bytes) 2024-09-21T15:49:46.328 INFO:teuthology.orchestra.run.smithi089.stdout:3278 read 0x450193e thru 0x4504ef6 (0x35b9 bytes) 2024-09-21T15:49:46.328 INFO:teuthology.orchestra.run.smithi089.stdout:3280 writesame 0xbe0b84a thru 0xbe1312a data_size 0x5(0x78e1 bytes) 2024-09-21T15:49:46.333 INFO:teuthology.orchestra.run.smithi089.stdout:3281 writesame 0x42e85bf thru 0x42f54f4 data_size 0x3f(0xcf36 bytes) 2024-09-21T15:49:46.339 INFO:teuthology.orchestra.run.smithi089.stdout:3282 writesame 0xd8c543e thru 0xd8c9ed5 data_size 0x58(0x4a98 bytes) 2024-09-21T15:49:46.341 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:46.340+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:46.343 INFO:teuthology.orchestra.run.smithi089.stdout:3284 write 0x1a6acac thru 0x1a733ee (0x8743 bytes) 2024-09-21T15:49:46.347 INFO:teuthology.orchestra.run.smithi089.stdout:3286 read 0xc6a0901 thru 0xc6a91c0 (0x88c0 bytes) 2024-09-21T15:49:46.348 INFO:teuthology.orchestra.run.smithi089.stdout:3287 punch from 0x673326a to 0x673a49b, (0x7231 bytes) 2024-09-21T15:49:46.349 INFO:teuthology.orchestra.run.smithi089.stdout:3288 trunc from 0xe131a2e to 0xd0e0bdc 2024-09-21T15:49:46.356 INFO:teuthology.orchestra.run.smithi089.stdout:3289 punch from 0xccbb464 to 0xccc242c, (0x6fc8 bytes) 2024-09-21T15:49:46.356 INFO:teuthology.orchestra.run.smithi089.stdout:3290 writesame 0x4943db thru 0x49c604 data_size 0x1(0x822a bytes) 2024-09-21T15:49:46.386 INFO:teuthology.orchestra.run.smithi089.stdout:3291 writesame 0xa4f7bfb thru 0xa503f5c data_size 0x2(0xc362 bytes) 2024-09-21T15:49:46.398 INFO:teuthology.orchestra.run.smithi089.stdout:3292 read 0x59e428 thru 0x5a2050 (0x3c29 bytes) 2024-09-21T15:49:46.399 INFO:teuthology.orchestra.run.smithi089.stdout:3293 clone 19 order 24 su 262144 sc 8 2024-09-21T15:49:46.442 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:46.441+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:46.446 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:46.446 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:49:46.446 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:49:46.446 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:49:46.446 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:46.455 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:49:46.544 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:46.542+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:46.599 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:46.599 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:49:46.599 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:49:46.599 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:49:46.599 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.599 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62833 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 192512~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:46.600 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:49:46.600 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.4552730329999992, 2024-09-21T15:49:46.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050387007999999997, 2024-09-21T15:49:46.600 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:46.600 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:46.600 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:46.600 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:46.600 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:46.600 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62833 2024-09-21T15:49:46.600 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.600 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:46.600 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.600 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:46.600 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:49:46.601 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.601 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.601 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:46.601 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:49:46.601 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.601 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.601 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:46.601 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:49:46.601 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:49:46.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.601 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.601 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:46.601 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142150+0000", 2024-09-21T15:49:46.602 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1999999999999999e-06 2024-09-21T15:49:46.602 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.602 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.602 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:46.602 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142153+0000", 2024-09-21T15:49:46.602 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7410000000000001e-06 2024-09-21T15:49:46.602 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.602 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.602 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:46.602 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142156+0000", 2024-09-21T15:49:46.602 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5739999999999998e-06 2024-09-21T15:49:46.602 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.602 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.602 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:46.603 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.192358+0000", 2024-09-21T15:49:46.603 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050202452000000002 2024-09-21T15:49:46.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.603 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:46.603 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.192435+0000", 2024-09-21T15:49:46.603 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6738999999999998e-05 2024-09-21T15:49:46.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.603 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:46.603 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.192536+0000", 2024-09-21T15:49:46.603 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101508 2024-09-21T15:49:46.603 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.603 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:46.603 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.604 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62847 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 200704~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:46.604 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.144163+0000", 2024-09-21T15:49:46.604 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.453259053, 2024-09-21T15:49:46.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050248293999999999, 2024-09-21T15:49:46.604 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:46.604 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:46.604 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:46.604 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:46.604 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:46.604 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62847 2024-09-21T15:49:46.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.604 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:46.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.605 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:46.605 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144163+0000", 2024-09-21T15:49:46.605 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.605 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.605 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.605 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:46.605 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144163+0000", 2024-09-21T15:49:46.605 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.605 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.605 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.605 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:46.605 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144163+0000", 2024-09-21T15:49:46.605 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:46.606 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.606 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.606 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:46.606 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144165+0000", 2024-09-21T15:49:46.606 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4439999999999998e-06 2024-09-21T15:49:46.606 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.606 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.607 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:46.607 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144173+0000", 2024-09-21T15:49:46.607 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7100000000000007e-06 2024-09-21T15:49:46.607 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.607 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.607 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:46.607 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144177+0000", 2024-09-21T15:49:46.608 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3000000000000003e-06 2024-09-21T15:49:46.608 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.608 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.608 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:46.608 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.194237+0000", 2024-09-21T15:49:46.608 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050060064000000001 2024-09-21T15:49:46.608 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.609 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.609 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:46.609 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.194317+0000", 2024-09-21T15:49:46.609 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9683000000000003e-05 2024-09-21T15:49:46.609 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.609 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.609 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:46.610 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.194411+0000", 2024-09-21T15:49:46.610 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4734000000000005e-05 2024-09-21T15:49:46.610 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.610 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:46.610 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.610 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.611 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.611 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62849 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 208896~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:46.611 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.144181+0000", 2024-09-21T15:49:46.611 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.4532409610000006, 2024-09-21T15:49:46.611 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050460390000000001, 2024-09-21T15:49:46.611 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:46.612 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:46.612 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:46.612 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:46.612 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:46.612 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62849 2024-09-21T15:49:46.612 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.612 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:46.613 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.613 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:46.613 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144181+0000", 2024-09-21T15:49:46.613 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.613 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.613 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.614 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:46.614 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144181+0000", 2024-09-21T15:49:46.614 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.614 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.614 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.614 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:46.614 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144181+0000", 2024-09-21T15:49:46.615 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:46.615 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.615 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.615 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:46.615 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144183+0000", 2024-09-21T15:49:46.615 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3860000000000001e-06 2024-09-21T15:49:46.616 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.616 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.616 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:46.616 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144189+0000", 2024-09-21T15:49:46.616 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0599999999999996e-06 2024-09-21T15:49:46.616 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.616 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.617 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:46.617 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144194+0000", 2024-09-21T15:49:46.617 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4379999999999997e-06 2024-09-21T15:49:46.617 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.617 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.617 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:46.618 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.194457+0000", 2024-09-21T15:49:46.618 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050263317000000002 2024-09-21T15:49:46.618 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.618 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.618 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:46.618 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.194548+0000", 2024-09-21T15:49:46.619 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0785000000000006e-05 2024-09-21T15:49:46.619 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.619 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.619 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:46.619 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.194642+0000", 2024-09-21T15:49:46.619 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3837000000000003e-05 2024-09-21T15:49:46.619 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.620 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:46.620 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.620 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.620 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.620 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62861 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 946176~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:46.620 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.146156+0000", 2024-09-21T15:49:46.621 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.4512661369999993, 2024-09-21T15:49:46.621 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050260860999999997, 2024-09-21T15:49:46.621 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:46.621 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:46.621 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:46.621 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:46.622 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:46.622 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62861 2024-09-21T15:49:46.622 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.622 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:46.622 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.622 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:46.622 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146156+0000", 2024-09-21T15:49:46.623 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.623 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.623 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.623 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:46.623 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146156+0000", 2024-09-21T15:49:46.623 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.624 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.624 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.624 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:46.624 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146156+0000", 2024-09-21T15:49:46.624 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:46.624 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.624 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.625 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:46.625 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146159+0000", 2024-09-21T15:49:46.625 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.14e-06 2024-09-21T15:49:46.625 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.625 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.625 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:46.626 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146167+0000", 2024-09-21T15:49:46.626 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9500000000000001e-06 2024-09-21T15:49:46.626 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.626 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.626 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:46.626 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146172+0000", 2024-09-21T15:49:46.626 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2900000000000002e-06 2024-09-21T15:49:46.627 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.627 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.627 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:46.627 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196247+0000", 2024-09-21T15:49:46.627 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050075195000000003 2024-09-21T15:49:46.627 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.628 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.628 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:46.628 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196324+0000", 2024-09-21T15:49:46.628 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6621999999999999e-05 2024-09-21T15:49:46.628 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.628 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.629 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:46.629 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196417+0000", 2024-09-21T15:49:46.629 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3251000000000006e-05 2024-09-21T15:49:46.629 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.629 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:46.629 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.629 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.629 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.630 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62862 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 954368~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:46.630 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.146176+0000", 2024-09-21T15:49:46.630 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.4512458979999998, 2024-09-21T15:49:46.630 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050442901999999998, 2024-09-21T15:49:46.630 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:46.630 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:46.630 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:46.631 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:46.631 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:46.631 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62862 2024-09-21T15:49:46.631 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.631 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:46.631 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.631 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:46.631 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146176+0000", 2024-09-21T15:49:46.632 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.632 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.632 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.632 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:46.632 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146176+0000", 2024-09-21T15:49:46.632 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.632 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.632 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.632 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:46.632 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146176+0000", 2024-09-21T15:49:46.632 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:46.632 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.632 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.632 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:46.632 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146178+0000", 2024-09-21T15:49:46.633 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4739999999999999e-06 2024-09-21T15:49:46.633 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.633 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.633 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:46.633 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146186+0000", 2024-09-21T15:49:46.633 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4989999999999997e-06 2024-09-21T15:49:46.633 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.633 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.633 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:46.633 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146191+0000", 2024-09-21T15:49:46.633 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7999999999999998e-06 2024-09-21T15:49:46.633 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.633 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.633 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:46.633 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196442+0000", 2024-09-21T15:49:46.633 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050251665000000001 2024-09-21T15:49:46.634 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.634 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.634 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:46.634 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196518+0000", 2024-09-21T15:49:46.634 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5677000000000001e-05 2024-09-21T15:49:46.634 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.634 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.634 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:46.634 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196619+0000", 2024-09-21T15:49:46.634 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101247 2024-09-21T15:49:46.634 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.634 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:46.634 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.635 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.635 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.635 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62863 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 937984~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:46.635 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.146195+0000", 2024-09-21T15:49:46.635 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.4512277069999993, 2024-09-21T15:49:46.635 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050642305999999998, 2024-09-21T15:49:46.635 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:46.635 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:46.635 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:46.635 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:46.635 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:46.635 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62863 2024-09-21T15:49:46.635 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.635 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:46.635 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.636 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:46.636 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146195+0000", 2024-09-21T15:49:46.636 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.636 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.636 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.636 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:46.636 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146195+0000", 2024-09-21T15:49:46.636 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.636 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.636 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.636 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:46.636 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146194+0000", 2024-09-21T15:49:46.636 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:46.636 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.637 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.637 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:46.637 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146196+0000", 2024-09-21T15:49:46.637 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1900000000000002e-06 2024-09-21T15:49:46.637 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.637 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.637 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:46.637 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146202+0000", 2024-09-21T15:49:46.637 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.57e-06 2024-09-21T15:49:46.637 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.637 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.637 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:46.637 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146206+0000", 2024-09-21T15:49:46.637 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5079999999999999e-06 2024-09-21T15:49:46.638 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.638 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.638 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:46.638 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196655+0000", 2024-09-21T15:49:46.638 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050448973000000001 2024-09-21T15:49:46.638 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.638 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.638 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:46.638 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196730+0000", 2024-09-21T15:49:46.638 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4728999999999996e-05 2024-09-21T15:49:46.638 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.638 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.639 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:46.639 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196837+0000", 2024-09-21T15:49:46.639 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000106735 2024-09-21T15:49:46.639 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.639 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:46.639 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.639 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.639 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.639 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62871 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 307200~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:46.639 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148134+0000", 2024-09-21T15:49:46.639 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.4492884979999996, 2024-09-21T15:49:46.639 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050310001, 2024-09-21T15:49:46.639 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:46.639 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:46.639 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:46.640 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:46.640 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:46.640 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62871 2024-09-21T15:49:46.640 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.640 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:46.640 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.640 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:46.640 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148134+0000", 2024-09-21T15:49:46.640 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.640 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.640 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.640 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:46.640 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148134+0000", 2024-09-21T15:49:46.640 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.640 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.641 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.641 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:46.641 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148133+0000", 2024-09-21T15:49:46.641 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:46.641 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.641 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.641 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:46.641 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148135+0000", 2024-09-21T15:49:46.641 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9829999999999999e-06 2024-09-21T15:49:46.641 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.641 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.641 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:46.641 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148141+0000", 2024-09-21T15:49:46.641 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5439999999999998e-06 2024-09-21T15:49:46.642 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.642 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.642 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:46.642 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148145+0000", 2024-09-21T15:49:46.642 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4229999999999998e-06 2024-09-21T15:49:46.642 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.642 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.642 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:46.642 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198294+0000", 2024-09-21T15:49:46.642 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050148558000000003 2024-09-21T15:49:46.642 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.643 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.643 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:46.643 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198372+0000", 2024-09-21T15:49:46.643 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8163000000000004e-05 2024-09-21T15:49:46.643 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.643 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.643 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:46.643 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198444+0000", 2024-09-21T15:49:46.643 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1620999999999997e-05 2024-09-21T15:49:46.643 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.643 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:46.643 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.643 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.643 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.644 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62872 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 290816~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:46.644 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148149+0000", 2024-09-21T15:49:46.644 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.4492730310000006, 2024-09-21T15:49:46.644 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.05046991, 2024-09-21T15:49:46.644 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:46.644 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:46.644 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:46.644 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:46.644 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:46.644 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62872 2024-09-21T15:49:46.644 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.644 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:46.644 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.644 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:46.644 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148149+0000", 2024-09-21T15:49:46.645 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.645 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.645 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:46.645 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148149+0000", 2024-09-21T15:49:46.645 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.645 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.645 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:46.645 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148149+0000", 2024-09-21T15:49:46.645 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:46.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.645 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.645 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:46.645 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148157+0000", 2024-09-21T15:49:46.646 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4749999999999993e-06 2024-09-21T15:49:46.646 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.646 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.646 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:46.646 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148163+0000", 2024-09-21T15:49:46.646 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8250000000000001e-06 2024-09-21T15:49:46.646 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.646 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.646 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:46.646 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148168+0000", 2024-09-21T15:49:46.646 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4730000000000002e-06 2024-09-21T15:49:46.646 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.646 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.646 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:46.647 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198471+0000", 2024-09-21T15:49:46.647 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050302974 2024-09-21T15:49:46.647 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.647 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.647 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:46.647 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198555+0000", 2024-09-21T15:49:46.647 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4783999999999993e-05 2024-09-21T15:49:46.647 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.647 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.647 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:46.647 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198619+0000", 2024-09-21T15:49:46.647 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3751000000000007e-05 2024-09-21T15:49:46.647 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.647 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:46.647 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.648 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.648 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.648 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62873 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 315392~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:46.648 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148172+0000", 2024-09-21T15:49:46.648 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.4492507400000001, 2024-09-21T15:49:46.648 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050707998999999997, 2024-09-21T15:49:46.648 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:46.648 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:46.648 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:46.648 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:46.649 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:46.649 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62873 2024-09-21T15:49:46.649 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.649 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:46.649 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.649 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:46.649 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148172+0000", 2024-09-21T15:49:46.649 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.649 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.649 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.649 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:46.649 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148172+0000", 2024-09-21T15:49:46.649 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.649 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.650 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.650 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:46.650 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148171+0000", 2024-09-21T15:49:46.650 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:46.650 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.650 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.650 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:46.650 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148173+0000", 2024-09-21T15:49:46.650 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2340000000000001e-06 2024-09-21T15:49:46.650 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.650 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.650 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:46.650 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148179+0000", 2024-09-21T15:49:46.650 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9340000000000003e-06 2024-09-21T15:49:46.651 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.651 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.651 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:46.651 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148186+0000", 2024-09-21T15:49:46.651 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2349999999999996e-06 2024-09-21T15:49:46.651 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.651 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.651 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:46.651 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198684+0000", 2024-09-21T15:49:46.651 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050498890999999997 2024-09-21T15:49:46.651 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.651 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.651 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:46.652 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198778+0000", 2024-09-21T15:49:46.652 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3134999999999995e-05 2024-09-21T15:49:46.652 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.652 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.652 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:46.652 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198880+0000", 2024-09-21T15:49:46.652 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101906 2024-09-21T15:49:46.652 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.652 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:46.652 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.652 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.652 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.652 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62874 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 323584~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:46.652 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148211+0000", 2024-09-21T15:49:46.652 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.4492109800000001, 2024-09-21T15:49:46.652 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050887766000000001, 2024-09-21T15:49:46.653 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:46.653 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:46.653 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:46.653 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:46.653 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:46.653 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62874 2024-09-21T15:49:46.653 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.653 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:46.653 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.653 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:46.653 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148211+0000", 2024-09-21T15:49:46.653 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.653 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.653 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.653 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:46.653 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148211+0000", 2024-09-21T15:49:46.654 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.654 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.654 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.654 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:46.654 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148211+0000", 2024-09-21T15:49:46.654 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:46.654 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.654 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.654 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:46.654 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148213+0000", 2024-09-21T15:49:46.654 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6860000000000002e-06 2024-09-21T15:49:46.654 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.654 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.654 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:46.654 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148220+0000", 2024-09-21T15:49:46.655 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4740000000000002e-06 2024-09-21T15:49:46.655 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.655 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.655 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:46.655 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148226+0000", 2024-09-21T15:49:46.655 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0120000000000002e-06 2024-09-21T15:49:46.655 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.655 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.655 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:46.655 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198940+0000", 2024-09-21T15:49:46.655 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050714412 2024-09-21T15:49:46.655 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.655 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.655 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:46.655 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198995+0000", 2024-09-21T15:49:46.655 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4863e-05 2024-09-21T15:49:46.656 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.656 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.656 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:46.656 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.199099+0000", 2024-09-21T15:49:46.656 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010383299999999999 2024-09-21T15:49:46.656 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.656 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:46.656 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.656 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.656 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.656 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62877 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 995328~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:46.656 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148230+0000", 2024-09-21T15:49:46.656 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.4491921199999993, 2024-09-21T15:49:46.656 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.051108949000000001, 2024-09-21T15:49:46.656 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:46.656 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:46.657 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:46.657 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:46.657 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:46.657 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62877 2024-09-21T15:49:46.657 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.657 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:46.657 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.657 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:46.657 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148230+0000", 2024-09-21T15:49:46.657 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.657 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.657 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.657 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:46.657 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148230+0000", 2024-09-21T15:49:46.657 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.658 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.658 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.658 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:46.658 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148230+0000", 2024-09-21T15:49:46.658 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:46.658 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.658 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.658 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:46.658 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148232+0000", 2024-09-21T15:49:46.658 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1220000000000002e-06 2024-09-21T15:49:46.658 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.658 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.658 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:46.658 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148238+0000", 2024-09-21T15:49:46.659 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.789e-06 2024-09-21T15:49:46.659 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.659 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.659 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:46.659 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148242+0000", 2024-09-21T15:49:46.659 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7369999999999997e-06 2024-09-21T15:49:46.659 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.659 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.659 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:46.659 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.199201+0000", 2024-09-21T15:49:46.659 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050959088 2024-09-21T15:49:46.659 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.659 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.659 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:46.659 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.199285+0000", 2024-09-21T15:49:46.660 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3147999999999997e-05 2024-09-21T15:49:46.660 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.660 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.660 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:46.660 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.199339+0000", 2024-09-21T15:49:46.660 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4484999999999998e-05 2024-09-21T15:49:46.660 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.660 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:46.660 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.660 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.660 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.660 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62886 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 1044480~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:46.660 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150138+0000", 2024-09-21T15:49:46.660 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.4472844519999999, 2024-09-21T15:49:46.661 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050345527000000001, 2024-09-21T15:49:46.661 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:46.661 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:46.661 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:46.661 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:46.661 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:46.661 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62886 2024-09-21T15:49:46.661 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.661 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:46.661 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.661 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:46.661 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150138+0000", 2024-09-21T15:49:46.661 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.662 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.662 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.662 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:46.662 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150138+0000", 2024-09-21T15:49:46.662 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.662 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.662 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.662 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:46.662 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150137+0000", 2024-09-21T15:49:46.662 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:46.662 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.662 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.663 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:46.663 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150140+0000", 2024-09-21T15:49:46.663 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2460000000000002e-06 2024-09-21T15:49:46.663 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.663 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.666 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:46.666 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150145+0000", 2024-09-21T15:49:46.666 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6520000000000003e-06 2024-09-21T15:49:46.666 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.666 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.666 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:46.666 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150150+0000", 2024-09-21T15:49:46.666 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5310000000000003e-06 2024-09-21T15:49:46.666 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.666 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.666 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:46.667 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200356+0000", 2024-09-21T15:49:46.667 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050205759000000003 2024-09-21T15:49:46.667 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.667 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.667 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:46.667 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200415+0000", 2024-09-21T15:49:46.667 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9719999999999997e-05 2024-09-21T15:49:46.667 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.667 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.667 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:46.667 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200483+0000", 2024-09-21T15:49:46.667 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7959000000000004e-05 2024-09-21T15:49:46.667 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.668 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:46.668 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.668 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.668 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.668 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62888 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 331776~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:46.668 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150154+0000", 2024-09-21T15:49:46.668 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.4472685260000002, 2024-09-21T15:49:46.668 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050483169000000001, 2024-09-21T15:49:46.668 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:46.668 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:46.668 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:46.669 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:46.669 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:46.669 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62888 2024-09-21T15:49:46.669 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.669 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:46.669 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.669 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:46.669 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150154+0000", 2024-09-21T15:49:46.669 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.669 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.669 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.669 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:46.669 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150154+0000", 2024-09-21T15:49:46.669 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.669 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.670 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.670 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:46.670 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150153+0000", 2024-09-21T15:49:46.670 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:46.670 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.670 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.670 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:46.670 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150155+0000", 2024-09-21T15:49:46.670 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9920000000000002e-06 2024-09-21T15:49:46.670 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.670 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.670 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:46.670 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150161+0000", 2024-09-21T15:49:46.671 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5439999999999998e-06 2024-09-21T15:49:46.671 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.671 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.671 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:46.671 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150165+0000", 2024-09-21T15:49:46.671 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3460000000000004e-06 2024-09-21T15:49:46.671 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.671 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.671 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:46.671 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200502+0000", 2024-09-21T15:49:46.671 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050337150999999997 2024-09-21T15:49:46.671 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.671 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.671 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:46.671 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200575+0000", 2024-09-21T15:49:46.672 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2250999999999997e-05 2024-09-21T15:49:46.672 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.672 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.672 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:46.672 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200637+0000", 2024-09-21T15:49:46.672 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2195999999999999e-05 2024-09-21T15:49:46.672 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.672 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:46.672 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.672 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.672 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.673 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62889 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 339968~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:46.673 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150169+0000", 2024-09-21T15:49:46.673 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.4472530270000004, 2024-09-21T15:49:46.673 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050629678999999997, 2024-09-21T15:49:46.673 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:46.673 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:46.673 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:46.673 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:46.673 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:46.673 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62889 2024-09-21T15:49:46.673 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.673 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:46.673 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.673 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:46.674 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150169+0000", 2024-09-21T15:49:46.674 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.674 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.674 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.674 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:46.674 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150169+0000", 2024-09-21T15:49:46.674 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.674 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.674 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.674 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:46.674 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150169+0000", 2024-09-21T15:49:46.674 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:46.674 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.674 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.675 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:46.675 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150171+0000", 2024-09-21T15:49:46.675 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3910000000000001e-06 2024-09-21T15:49:46.675 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.675 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.675 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:46.675 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150177+0000", 2024-09-21T15:49:46.675 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4079999999999997e-06 2024-09-21T15:49:46.675 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.675 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.675 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:46.675 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150182+0000", 2024-09-21T15:49:46.675 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3890000000000004e-06 2024-09-21T15:49:46.675 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.676 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.676 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:46.676 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200664+0000", 2024-09-21T15:49:46.676 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050482212999999998 2024-09-21T15:49:46.676 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.676 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.676 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:46.676 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200738+0000", 2024-09-21T15:49:46.676 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4114000000000002e-05 2024-09-21T15:49:46.676 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.676 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.676 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:46.676 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200799+0000", 2024-09-21T15:49:46.676 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0503e-05 2024-09-21T15:49:46.676 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.677 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:46.677 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.677 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.677 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.677 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62891 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 348160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:46.677 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150186+0000", 2024-09-21T15:49:46.677 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.4472358849999996, 2024-09-21T15:49:46.677 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050794552, 2024-09-21T15:49:46.677 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:46.678 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:46.678 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:46.678 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:46.678 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:46.678 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62891 2024-09-21T15:49:46.678 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.678 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:46.678 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.678 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:46.678 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150186+0000", 2024-09-21T15:49:46.678 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.678 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.678 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.679 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:46.679 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150186+0000", 2024-09-21T15:49:46.679 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.679 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.679 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.679 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:46.679 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150186+0000", 2024-09-21T15:49:46.679 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:46.679 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.679 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.679 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:46.679 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150193+0000", 2024-09-21T15:49:46.679 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6830000000000002e-06 2024-09-21T15:49:46.680 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.680 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.680 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:46.680 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150199+0000", 2024-09-21T15:49:46.680 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9859999999999999e-06 2024-09-21T15:49:46.680 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.680 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.680 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:46.680 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150203+0000", 2024-09-21T15:49:46.680 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5920000000000002e-06 2024-09-21T15:49:46.680 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.680 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.680 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:46.680 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200835+0000", 2024-09-21T15:49:46.680 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050631783 2024-09-21T15:49:46.680 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.681 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.681 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:46.681 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200894+0000", 2024-09-21T15:49:46.681 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8560999999999997e-05 2024-09-21T15:49:46.681 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.681 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.681 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:46.681 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200981+0000", 2024-09-21T15:49:46.681 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7305999999999994e-05 2024-09-21T15:49:46.681 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.681 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:46.681 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.681 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.681 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.681 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62892 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 356352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:46.682 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150208+0000", 2024-09-21T15:49:46.682 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.4472144969999992, 2024-09-21T15:49:46.682 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050951769000000001, 2024-09-21T15:49:46.682 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:46.682 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:46.682 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:46.682 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:46.682 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:46.682 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62892 2024-09-21T15:49:46.682 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.682 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:46.682 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.682 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:46.682 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150208+0000", 2024-09-21T15:49:46.683 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.683 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.683 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.683 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:46.683 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150208+0000", 2024-09-21T15:49:46.683 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.683 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.684 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.684 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:46.684 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150207+0000", 2024-09-21T15:49:46.684 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:46.684 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.684 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.684 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:46.684 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150210+0000", 2024-09-21T15:49:46.684 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.238e-06 2024-09-21T15:49:46.684 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.684 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.684 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:46.685 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150215+0000", 2024-09-21T15:49:46.685 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6350000000000003e-06 2024-09-21T15:49:46.685 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.685 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.685 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:46.685 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150220+0000", 2024-09-21T15:49:46.685 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.544e-06 2024-09-21T15:49:46.685 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.685 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.685 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:46.685 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201014+0000", 2024-09-21T15:49:46.685 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050793932999999999 2024-09-21T15:49:46.685 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.685 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.686 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:46.686 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201092+0000", 2024-09-21T15:49:46.686 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7961999999999997e-05 2024-09-21T15:49:46.686 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.686 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.686 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:46.686 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201160+0000", 2024-09-21T15:49:46.686 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7836000000000006e-05 2024-09-21T15:49:46.686 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.686 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:46.686 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.686 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.686 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.687 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62894 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 364544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:46.687 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150820+0000", 2024-09-21T15:49:46.687 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.4466022970000001, 2024-09-21T15:49:46.687 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050489196, 2024-09-21T15:49:46.687 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:46.687 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:46.687 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:46.687 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:46.687 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:46.687 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62894 2024-09-21T15:49:46.687 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.687 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:46.687 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.687 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:46.687 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150820+0000", 2024-09-21T15:49:46.688 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.688 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.688 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.688 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:46.688 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150820+0000", 2024-09-21T15:49:46.688 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.688 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.688 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.688 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:46.688 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150223+0000", 2024-09-21T15:49:46.688 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9994035 2024-09-21T15:49:46.688 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.688 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.688 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:46.689 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150825+0000", 2024-09-21T15:49:46.689 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00060137099999999996 2024-09-21T15:49:46.689 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.689 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.689 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:46.689 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150837+0000", 2024-09-21T15:49:46.689 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2503999999999999e-05 2024-09-21T15:49:46.689 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.689 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.689 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:46.689 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150844+0000", 2024-09-21T15:49:46.689 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6900000000000003e-06 2024-09-21T15:49:46.689 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.689 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.689 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:46.690 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201187+0000", 2024-09-21T15:49:46.690 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050342973999999999 2024-09-21T15:49:46.690 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.690 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.690 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:46.690 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201250+0000", 2024-09-21T15:49:46.690 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3274000000000004e-05 2024-09-21T15:49:46.690 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.690 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.690 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:46.691 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201309+0000", 2024-09-21T15:49:46.691 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9058000000000003e-05 2024-09-21T15:49:46.691 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.691 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:46.691 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.691 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.691 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.691 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62895 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 380928~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:46.691 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150851+0000", 2024-09-21T15:49:46.691 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.4465717009999999, 2024-09-21T15:49:46.691 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050611097000000001, 2024-09-21T15:49:46.691 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:46.691 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:46.692 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:46.692 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:46.692 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:46.692 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62895 2024-09-21T15:49:46.692 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.692 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:46.692 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.692 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:46.692 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150851+0000", 2024-09-21T15:49:46.692 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.692 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.692 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.692 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:46.692 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150851+0000", 2024-09-21T15:49:46.692 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.693 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.693 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:46.693 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150850+0000", 2024-09-21T15:49:46.693 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:46.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.693 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.693 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:46.693 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150853+0000", 2024-09-21T15:49:46.693 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1039999999999998e-06 2024-09-21T15:49:46.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.693 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.693 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:46.693 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150860+0000", 2024-09-21T15:49:46.694 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3180000000000001e-06 2024-09-21T15:49:46.694 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.694 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.694 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:46.694 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150866+0000", 2024-09-21T15:49:46.694 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1599999999999997e-06 2024-09-21T15:49:46.694 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.694 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.694 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:46.694 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201332+0000", 2024-09-21T15:49:46.694 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050466100999999999 2024-09-21T15:49:46.694 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.694 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.694 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:46.695 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201392+0000", 2024-09-21T15:49:46.695 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0053999999999998e-05 2024-09-21T15:49:46.695 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.695 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.695 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:46.695 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201462+0000", 2024-09-21T15:49:46.695 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9861000000000001e-05 2024-09-21T15:49:46.695 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.695 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:46.695 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.695 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.695 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.695 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62896 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 372736~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:46.695 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150870+0000", 2024-09-21T15:49:46.696 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.4465519209999993, 2024-09-21T15:49:46.696 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050753292999999998, 2024-09-21T15:49:46.696 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:46.696 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:46.696 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:46.696 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:46.696 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:46.696 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62896 2024-09-21T15:49:46.696 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.696 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:46.696 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.696 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:46.696 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150870+0000", 2024-09-21T15:49:46.696 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.696 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.697 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.697 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:46.697 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150870+0000", 2024-09-21T15:49:46.697 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.697 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.697 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:46.697 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150870+0000", 2024-09-21T15:49:46.697 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:46.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.697 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.697 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:46.697 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150872+0000", 2024-09-21T15:49:46.697 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.644e-06 2024-09-21T15:49:46.698 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.698 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.698 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:46.698 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150879+0000", 2024-09-21T15:49:46.698 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3509999999999998e-06 2024-09-21T15:49:46.698 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.698 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.698 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:46.698 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150884+0000", 2024-09-21T15:49:46.698 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9150000000000004e-06 2024-09-21T15:49:46.698 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.698 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.698 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:46.698 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201487+0000", 2024-09-21T15:49:46.699 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050603276000000003 2024-09-21T15:49:46.699 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.699 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.699 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:46.699 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201557+0000", 2024-09-21T15:49:46.699 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9847999999999999e-05 2024-09-21T15:49:46.699 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.699 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.699 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:46.699 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201624+0000", 2024-09-21T15:49:46.699 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6730999999999995e-05 2024-09-21T15:49:46.699 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.699 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:46.699 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.699 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.700 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.700 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62897 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 389120~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:46.700 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.151044+0000", 2024-09-21T15:49:46.700 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.4463778810000001, 2024-09-21T15:49:46.700 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050720531999999999, 2024-09-21T15:49:46.700 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:46.700 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:46.700 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:46.700 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:46.700 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:46.700 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62897 2024-09-21T15:49:46.700 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.700 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:46.700 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.701 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:46.701 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151044+0000", 2024-09-21T15:49:46.701 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.701 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.701 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.701 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:46.701 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151044+0000", 2024-09-21T15:49:46.701 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:46.701 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.701 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.701 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:46.701 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150888+0000", 2024-09-21T15:49:46.701 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9998431 2024-09-21T15:49:46.701 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.701 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.702 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:46.702 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151048+0000", 2024-09-21T15:49:46.702 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00016077299999999999 2024-09-21T15:49:46.702 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.702 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.702 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:46.702 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151058+0000", 2024-09-21T15:49:46.702 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.465e-06 2024-09-21T15:49:46.702 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.702 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.702 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:46.702 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151064+0000", 2024-09-21T15:49:46.702 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3829999999999997e-06 2024-09-21T15:49:46.702 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.702 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.703 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:46.703 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201648+0000", 2024-09-21T15:49:46.703 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050583597000000001 2024-09-21T15:49:46.703 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.703 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.703 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:46.703 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201711+0000", 2024-09-21T15:49:46.703 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3616999999999998e-05 2024-09-21T15:49:46.703 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:46.703 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:46.703 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:46.703 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201765+0000", 2024-09-21T15:49:46.703 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3464000000000002e-05 2024-09-21T15:49:46.703 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.703 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:46.704 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.704 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:46.704 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:46.704 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:46.705 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:49:46.706 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:46.704+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:46.707 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:49:46.805 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:46.804+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:46.852 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:46.852 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:49:46.852 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:49:46.852 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:46.861 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:49:46.867 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-09-21T15:49:46.867 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:49:46.886 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:49:46.886 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:49:46.905 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:49:46.906 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:49:46.906 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:46.904+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:46.922 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:49:46.922 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:49:46.940 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:49:46.940 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:49:46.954 INFO:teuthology.orchestra.run.smithi089.stdout:truncating image image_client.0-clone18 from 0xd0e0bdc (overlap 0x18030e5) to 0xe683bc 2024-09-21T15:49:46.968 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:49:46.968 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:49:46.993 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:46.993 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:46.996 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:49:46.997 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:49:47.007 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:47.006+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:47.008 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:49:47.025 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:49:47.025 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:49:47.084 INFO:teuthology.orchestra.run.smithi089.stdout:checking clone #17, image image_client.0-clone17 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent18 2024-09-21T15:49:47.108 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:47.107+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:47.114 INFO:teuthology.orchestra.run.smithi089.stdout:3294 punch from 0x70f2fbb to 0x70f3a18, (0xa5d bytes) 2024-09-21T15:49:47.121 INFO:teuthology.orchestra.run.smithi089.stdout:3295 punch from 0x3760c2d to 0x376e246, (0xd619 bytes) 2024-09-21T15:49:47.121 INFO:teuthology.orchestra.run.smithi089.stdout:3296 write 0x4698ebb thru 0x46a24ae (0x95f4 bytes) 2024-09-21T15:49:47.142 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:47.142 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:47.151 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:49:47.159 INFO:teuthology.orchestra.run.smithi089.stdout:3297 clone 20 order 24 su 65536 sc 11 2024-09-21T15:49:47.208 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:47.207+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:47.253 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:47.254 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:47.262 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:49:47.346 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:47.344+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:47.404 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:47.404 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:47.410 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:47.409+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:47.413 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:49:47.511 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:47.509+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:47.555 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:47.555 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:47.563 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:49:47.612 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:47.611+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:47.668 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:47.668 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:47.677 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:49:47.718 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:47.717+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:47.814 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:47.813+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:47.819 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:47.819 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:47.828 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:49:47.895 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T15:49:47.939 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:47.938+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:47.964 INFO:teuthology.orchestra.run.smithi089.stdout:flattening image image_client.0-clone19 2024-09-21T15:49:47.983 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:47.983 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:47.997 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:49:48.040 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:48.039+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:48.115 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:48.114+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:48.148 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:48.148 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:48.157 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:49:48.241 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:48.240+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:48.286 INFO:teuthology.orchestra.run.smithi089.stdout:checking clone #18, image image_client.0-clone18 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent19 2024-09-21T15:49:48.302 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:48.302 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:48.308 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:48.309 INFO:teuthology.orchestra.run.smithi089.stdout:3300 trunc from 0xd0e0bdc to 0xdd9a987 2024-09-21T15:49:48.311 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:49:48.320 INFO:teuthology.orchestra.run.smithi089.stdout:3301 write 0xba75ad0 thru 0xba7d5aa (0x7adb bytes) 2024-09-21T15:49:48.328 INFO:teuthology.orchestra.run.smithi089.stdout:3302 write 0x9b8f4dd thru 0x9b9bd2b (0xc84f bytes) 2024-09-21T15:49:48.352 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:48.351+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:48.357 INFO:teuthology.orchestra.run.smithi089.stdout:3306 read 0x89e78e2 thru 0x89f2b56 (0xb275 bytes) 2024-09-21T15:49:48.358 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:48.358 INFO:teuthology.orchestra.run.smithi089.stdout:3312 read 0x8e0c5f1 thru 0x8e0d5c5 (0xfd5 bytes) 2024-09-21T15:49:48.359 INFO:teuthology.orchestra.run.smithi089.stdout:3313 read 0x5bd4841 thru 0x5bde0cc (0x988c bytes) 2024-09-21T15:49:48.359 INFO:teuthology.orchestra.run.smithi089.stdout:3314 read 0xc84f1c4 thru 0xc857f8c (0x8dc9 bytes) 2024-09-21T15:49:48.359 INFO:teuthology.orchestra.run.smithi089.stdout:3316 writesame 0x3aa95e0 thru 0x3ab3f89 data_size 0x7f(0xa9aa bytes) 2024-09-21T15:49:48.385 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:48.385 INFO:teuthology.orchestra.run.smithi089.stdout:3318 write 0x3c8ec48 thru 0x3c97149 (0x8502 bytes) 2024-09-21T15:49:48.418 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:48.417+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:48.429 INFO:teuthology.orchestra.run.smithi089.stdout:3319 trunc from 0xdd9a987 to 0x73a96b 2024-09-21T15:49:48.478 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:48.478 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:48.488 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T15:49:48.514 INFO:teuthology.orchestra.run.smithi089.stdout:3320 trunc from 0x73a96b to 0x2a65a37 2024-09-21T15:49:48.519 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:48.518+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:48.522 INFO:teuthology.orchestra.run.smithi089.stdout:3322 trunc from 0x2a65a37 to 0x3e13f14 2024-09-21T15:49:48.533 INFO:teuthology.orchestra.run.smithi089.stdout:3323 writesame 0x511aa6f thru 0x512512d data_size 0x3(0xa6bf bytes) 2024-09-21T15:49:48.537 INFO:teuthology.orchestra.run.smithi089.stdout:3325 trunc from 0x512512e to 0x3ae5342 2024-09-21T15:49:48.547 INFO:teuthology.orchestra.run.smithi089.stdout:3326 read 0x2ea11bd thru 0x2ea3f4c (0x2d90 bytes) 2024-09-21T15:49:48.547 INFO:teuthology.orchestra.run.smithi089.stdout:3327 punch from 0x9d73db to 0x9e4f7e, (0xdba3 bytes) 2024-09-21T15:49:48.547 INFO:teuthology.orchestra.run.smithi089.stdout:3328 punch from 0x1098aaf to 0x109f8d4, (0x6e25 bytes) 2024-09-21T15:49:48.548 INFO:teuthology.orchestra.run.smithi089.stdout:3329 read 0xc76cd5 thru 0xc832d6 (0xc602 bytes) 2024-09-21T15:49:48.565 INFO:teuthology.orchestra.run.smithi089.stdout:3330 writesame 0xe01ee7c thru 0xe02a33d data_size 0x11(0xb4c2 bytes) 2024-09-21T15:49:48.566 INFO:teuthology.orchestra.run.smithi089.stdout:3331 writesame 0x808640b thru 0x8087e2f data_size 0x3(0x1a25 bytes) 2024-09-21T15:49:48.567 INFO:teuthology.orchestra.run.smithi089.stdout:3332 read 0x6d75ad1 thru 0x6d77b9a (0x20ca bytes) 2024-09-21T15:49:48.568 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:48.568 INFO:teuthology.orchestra.run.smithi089.stdout:3336 read 0xd08f371 thru 0xd09d8d6 (0xe566 bytes) 2024-09-21T15:49:48.568 INFO:teuthology.orchestra.run.smithi089.stdout:3339 read 0x97d25d6 thru 0x97d6478 (0x3ea3 bytes) 2024-09-21T15:49:48.568 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:48.568 INFO:teuthology.orchestra.run.smithi089.stdout:3341 write 0xa499ba8 thru 0xa4a1665 (0x7abe bytes) 2024-09-21T15:49:48.570 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:48.575 INFO:teuthology.orchestra.run.smithi089.stdout:3343 write 0xe849fb8 thru 0xe84d506 (0x354f bytes) 2024-09-21T15:49:48.579 INFO:teuthology.orchestra.run.smithi089.stdout:3346 write 0xba81d68 thru 0xba8b1cd (0x9466 bytes) 2024-09-21T15:49:48.598 INFO:teuthology.orchestra.run.smithi089.stdout:3348 trunc from 0xe84d507 to 0x6f31fb 2024-09-21T15:49:48.616 INFO:teuthology.orchestra.run.smithi089.stdout:3350 trunc from 0x6f31fb to 0x445ccd7 2024-09-21T15:49:48.619 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:48.618+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:48.621 INFO:teuthology.orchestra.run.smithi089.stdout:3351 read 0x3b77675 thru 0x3b803ac (0x8d38 bytes) 2024-09-21T15:49:48.637 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:48.637 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:48.639 INFO:teuthology.orchestra.run.smithi089.stdout:3353 writesame 0xd7f72e8 thru 0xd802386 data_size 0x1(0xb09f bytes) 2024-09-21T15:49:48.639 INFO:teuthology.orchestra.run.smithi089.stdout:3354 write 0x36e47db thru 0x36ea415 (0x5c3b bytes) 2024-09-21T15:49:48.641 INFO:teuthology.orchestra.run.smithi089.stdout:3355 punch from 0x6700590 to 0x6702b07, (0x2577 bytes) 2024-09-21T15:49:48.641 INFO:teuthology.orchestra.run.smithi089.stdout:3356 read 0x23a8542 thru 0x23aebcd (0x668c bytes) 2024-09-21T15:49:48.641 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:48.641 INFO:teuthology.orchestra.run.smithi089.stdout:3358 punch from 0x24312d5 to 0x243e482, (0xd1ad bytes) 2024-09-21T15:49:48.641 INFO:teuthology.orchestra.run.smithi089.stdout:3359 write 0x7bcda18 thru 0x7bd75b5 (0x9b9e bytes) 2024-09-21T15:49:48.643 INFO:teuthology.orchestra.run.smithi089.stdout:3360 write 0x17d526f thru 0x17d9cfa (0x4a8c bytes) 2024-09-21T15:49:48.645 INFO:teuthology.orchestra.run.smithi089.stdout:3362 read 0x6169fcd thru 0x6172f8b (0x8fbf bytes) 2024-09-21T15:49:48.645 INFO:teuthology.orchestra.run.smithi089.stdout:3363 read 0x53bf626 thru 0x53cae0a (0xb7e5 bytes) 2024-09-21T15:49:48.647 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T15:49:48.657 INFO:teuthology.orchestra.run.smithi089.stdout:3365 write 0xe2ec438 thru 0xe2fc26d (0xfe36 bytes) 2024-09-21T15:49:48.666 INFO:teuthology.orchestra.run.smithi089.stdout:3366 read 0x7001cc7 thru 0x7009baf (0x7ee9 bytes) 2024-09-21T15:49:48.667 INFO:teuthology.orchestra.run.smithi089.stdout:3367 read 0xaf1def thru 0xaf9e0b (0x801d bytes) 2024-09-21T15:49:48.667 INFO:teuthology.orchestra.run.smithi089.stdout:3368 punch from 0x10652a3 to 0x1071274, (0xbfd1 bytes) 2024-09-21T15:49:48.667 INFO:teuthology.orchestra.run.smithi089.stdout:3369 punch from 0x9cadb10 to 0x9cba724, (0xcc14 bytes) 2024-09-21T15:49:48.667 INFO:teuthology.orchestra.run.smithi089.stdout:3370 read 0xdc8e0dc thru 0xdc92cc6 (0x4beb bytes) 2024-09-21T15:49:48.667 INFO:teuthology.orchestra.run.smithi089.stdout:3371 read 0x635b0e1 thru 0x6366850 (0xb770 bytes) 2024-09-21T15:49:48.669 INFO:teuthology.orchestra.run.smithi089.stdout:3373 trunc from 0xe2fc26e to 0xcf8c624 2024-09-21T15:49:48.674 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:48.674 INFO:teuthology.orchestra.run.smithi089.stdout:3375 write 0xa2cb020 thru 0xa2d1c8f (0x6c70 bytes) 2024-09-21T15:49:48.681 INFO:teuthology.orchestra.run.smithi089.stdout:3380 trunc from 0xcf8c624 to 0x856a5d5 2024-09-21T15:49:48.697 INFO:teuthology.orchestra.run.smithi089.stdout:3381 punch from 0x503ccdb to 0x5043434, (0x6759 bytes) 2024-09-21T15:49:48.703 INFO:teuthology.orchestra.run.smithi089.stdout:3382 write 0x974e6f3 thru 0x974f147 (0xa55 bytes) 2024-09-21T15:49:48.705 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:48.705 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:48.715 INFO:teuthology.orchestra.run.smithi089.stdout:3385 writesame 0xdfc908b thru 0xdfccf99 data_size 0x1(0x3f0f bytes) 2024-09-21T15:49:48.715 INFO:teuthology.orchestra.run.smithi089.stdout:3387 write 0x7d32aac thru 0x7d36e46 (0x439b bytes) 2024-09-21T15:49:48.716 INFO:teuthology.orchestra.run.smithi089.stdout:3389 write 0x131bc09 thru 0x1328ddd (0xd1d5 bytes) 2024-09-21T15:49:48.718 INFO:teuthology.orchestra.run.smithi089.stdout:3391 read 0xddea91a thru 0xddef8b3 (0x4f9a bytes) 2024-09-21T15:49:48.718 INFO:teuthology.orchestra.run.smithi089.stdout:3393 punch from 0x9c1edee to 0x9c2417b, (0x538d bytes) 2024-09-21T15:49:48.718 INFO:teuthology.orchestra.run.smithi089.stdout:3394 writesame 0xc0f17a9 thru 0xc0fa439 data_size 0x5(0x8c91 bytes) 2024-09-21T15:49:48.720 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:48.719+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:48.724 INFO:teuthology.orchestra.run.smithi089.stdout:3397 write 0x9e2db46 thru 0x9e31c19 (0x40d4 bytes) 2024-09-21T15:49:48.739 INFO:teuthology.orchestra.run.smithi089.stdout:3398 trunc from 0xdfccf9a to 0x3406cc3 2024-09-21T15:49:48.767 INFO:teuthology.orchestra.run.smithi089.stdout:3399 write 0x7b6f1bd thru 0x7b7a074 (0xaeb8 bytes) 2024-09-21T15:49:48.774 INFO:teuthology.orchestra.run.smithi089.stdout:3402 trunc from 0x7b7a075 to 0x3168470 2024-09-21T15:49:48.795 INFO:teuthology.orchestra.run.smithi089.stdout:3403 trunc from 0x3168470 to 0xc2e13f9 2024-09-21T15:49:48.795 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:48.795 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:48.799 INFO:teuthology.orchestra.run.smithi089.stdout:3404 read 0x84fb683 thru 0x84fe3e0 (0x2d5e bytes) 2024-09-21T15:49:48.800 INFO:teuthology.orchestra.run.smithi089.stdout:3405 write 0x9320a90 thru 0x932df89 (0xd4fa bytes) 2024-09-21T15:49:48.801 INFO:teuthology.orchestra.run.smithi089.stdout:3406 read 0x11fe5ab thru 0x120b25e (0xccb4 bytes) 2024-09-21T15:49:48.803 INFO:teuthology.orchestra.run.smithi089.stdout:3407 trunc from 0xc2e13f9 to 0xcb279cd 2024-09-21T15:49:48.804 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T15:49:48.822 INFO:teuthology.orchestra.run.smithi089.stdout:3409 trunc from 0xcb279cd to 0x1f9e1ad 2024-09-21T15:49:48.822 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:48.821+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:48.847 INFO:teuthology.orchestra.run.smithi089.stdout:3410 write 0x967b78c thru 0x9684db7 (0x962c bytes) 2024-09-21T15:49:48.848 INFO:teuthology.orchestra.run.smithi089.stdout:3411 write 0x2a6e22f thru 0x2a77679 (0x944b bytes) 2024-09-21T15:49:48.851 INFO:teuthology.orchestra.run.smithi089.stdout:3412 writesame 0x3bd83b3 thru 0x3be597b data_size 0x9(0xd5c9 bytes) 2024-09-21T15:49:48.863 INFO:teuthology.orchestra.run.smithi089.stdout:3413 writesame 0xa9dc031 thru 0xa9e141e data_size 0x3(0x53ee bytes) 2024-09-21T15:49:48.865 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:48.865 INFO:teuthology.orchestra.run.smithi089.stdout:3417 read 0x6a40d91 thru 0x6a4e151 (0xd3c1 bytes) 2024-09-21T15:49:48.869 INFO:teuthology.orchestra.run.smithi089.stdout:3418 trunc from 0xa9e141f to 0x74ed56e 2024-09-21T15:49:48.881 INFO:teuthology.orchestra.run.smithi089.stdout:3419 read 0x5b975d4 thru 0x5ba0033 (0x8a60 bytes) 2024-09-21T15:49:48.881 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:48.881 INFO:teuthology.orchestra.run.smithi089.stdout:3421 read 0x4750055 thru 0x475c1f4 (0xc1a0 bytes) 2024-09-21T15:49:48.893 INFO:teuthology.orchestra.run.smithi089.stdout:3422 writesame 0xc920f21 thru 0xc92beb0 data_size 0x6a(0xaf90 bytes) 2024-09-21T15:49:48.893 INFO:teuthology.orchestra.run.smithi089.stdout:3423 write 0x114e2c1 thru 0x115cfb5 (0xecf5 bytes) 2024-09-21T15:49:48.896 INFO:teuthology.orchestra.run.smithi089.stdout:3426 punch from 0x50ffd0e to 0x50ffff0, (0x2e2 bytes) 2024-09-21T15:49:48.897 INFO:teuthology.orchestra.run.smithi089.stdout:3427 write 0x8c03c13 thru 0x8c0aef1 (0x72df bytes) 2024-09-21T15:49:48.898 INFO:teuthology.orchestra.run.smithi089.stdout:3429 read 0x2095930 thru 0x209ca62 (0x7133 bytes) 2024-09-21T15:49:48.898 INFO:teuthology.orchestra.run.smithi089.stdout:3430 write 0x1d89fe1 thru 0x1d997ac (0xf7cc bytes) 2024-09-21T15:49:48.905 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is unset 2024-09-21T15:49:48.910 INFO:teuthology.orchestra.run.smithi089.stdout:3431 write 0xcec9403 thru 0xcecf872 (0x6470 bytes) 2024-09-21T15:49:48.915 INFO:teuthology.orchestra.run.smithi089.stdout:3432 write 0xae82254 thru 0xae86c91 (0x4a3e bytes) 2024-09-21T15:49:48.915 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:48.915 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:48.917 INFO:teuthology.orchestra.run.smithi089.stdout:3433 clone 21 order 18 su 131072 sc 13 2024-09-21T15:49:48.924 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T15:49:48.960 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:48.959+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:48.969 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T15:49:48.969 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T15:49:48.969 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-21T15:49:48.969 INFO:tasks.thrashosds.thrasher:Setting osd 0 primary_affinity to 1.000000 2024-09-21T15:49:48.970 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-09-21T15:49:49.023 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:49.022+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:49.025 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:49.025 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:49.034 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T15:49:49.041 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:49:49.124 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:49.123+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:49.135 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:49.135 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:49.144 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T15:49:49.162 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-09-21T15:49:49.162 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:49:49.178 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:49:49.179 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:49:49.194 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:49:49.194 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:49:49.209 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:49:49.210 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:49:49.225 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:49:49.226 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:49:49.228 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:49.227+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:49.241 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:49:49.242 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:49:49.258 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:49:49.258 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:49:49.275 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:49:49.275 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:49:49.290 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:49.290 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:49:49.290 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:49:49.290 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:49:49.290 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.291 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4209.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:49:49.291 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:49:49.291 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 58.097428450000002, 2024-09-21T15:49:49.291 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.19452909900000001, 2024-09-21T15:49:49.291 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:49.291 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:49.291 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:49.291 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4209", 2024-09-21T15:49:49.291 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/4045143050", 2024-09-21T15:49:49.291 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1 2024-09-21T15:49:49.291 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.291 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:49.292 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.292 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:49.292 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:49:49.292 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:49.292 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.292 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.292 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:49.292 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:49:49.292 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:49.292 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.292 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.293 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:49.293 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191161+0000", 2024-09-21T15:49:49.293 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:49.293 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.293 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.293 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:49.293 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191169+0000", 2024-09-21T15:49:49.293 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0090000000000007e-06 2024-09-21T15:49:49.293 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.293 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.293 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:49.293 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191180+0000", 2024-09-21T15:49:49.293 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0728e-05 2024-09-21T15:49:49.293 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.294 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.294 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:49.294 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191187+0000", 2024-09-21T15:49:49.294 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9009999999999994e-06 2024-09-21T15:49:49.294 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.294 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.294 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:49.294 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191238+0000", 2024-09-21T15:49:49.294 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0287000000000002e-05 2024-09-21T15:49:49.294 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.294 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.294 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-21T15:49:49.294 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191245+0000", 2024-09-21T15:49:49.294 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0709999999999997e-06 2024-09-21T15:49:49.295 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.295 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.295 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:49.295 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380309+0000", 2024-09-21T15:49:49.295 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.189064277 2024-09-21T15:49:49.295 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.295 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.295 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:49.295 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380454+0000", 2024-09-21T15:49:49.295 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014490799999999999 2024-09-21T15:49:49.295 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.295 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.295 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0", 2024-09-21T15:49:49.295 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380610+0000", 2024-09-21T15:49:49.296 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015650199999999999 2024-09-21T15:49:49.296 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.296 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.296 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:49:49.296 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.381160+0000", 2024-09-21T15:49:49.296 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00054903799999999996 2024-09-21T15:49:49.296 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.296 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.296 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:49.296 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385591+0000", 2024-09-21T15:49:49.296 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.004431272 2024-09-21T15:49:49.296 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.296 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.296 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:49:49.296 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385651+0000", 2024-09-21T15:49:49.297 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0455999999999999e-05 2024-09-21T15:49:49.297 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.297 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.297 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:49.297 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385691+0000", 2024-09-21T15:49:49.297 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0234999999999997e-05 2024-09-21T15:49:49.297 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:49.297 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:49.297 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:49.297 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.297 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.297 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64028 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1763328~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:49.297 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167504+0000", 2024-09-21T15:49:49.297 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.121086893999999, 2024-09-21T15:49:49.298 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047494153999999997, 2024-09-21T15:49:49.298 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:49.298 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:49.298 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:49.298 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:49.298 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:49.298 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64028 2024-09-21T15:49:49.298 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.298 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:49.298 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.298 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:49.298 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167504+0000", 2024-09-21T15:49:49.298 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:49.298 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.298 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.299 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:49.299 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167504+0000", 2024-09-21T15:49:49.299 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:49.299 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.299 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.299 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:49.299 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167504+0000", 2024-09-21T15:49:49.299 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:49.299 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.299 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.299 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:49.299 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167506+0000", 2024-09-21T15:49:49.299 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0600000000000002e-06 2024-09-21T15:49:49.299 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.299 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.300 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:49.300 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167512+0000", 2024-09-21T15:49:49.300 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.083e-06 2024-09-21T15:49:49.300 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.300 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.300 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:49.300 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167517+0000", 2024-09-21T15:49:49.300 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7650000000000001e-06 2024-09-21T15:49:49.300 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.300 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.300 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:49.300 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.214835+0000", 2024-09-21T15:49:49.300 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047318067999999998 2024-09-21T15:49:49.300 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.301 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.301 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:49.301 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.214916+0000", 2024-09-21T15:49:49.301 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1815000000000002e-05 2024-09-21T15:49:49.301 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.301 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.301 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:49.301 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.214998+0000", 2024-09-21T15:49:49.301 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1642000000000002e-05 2024-09-21T15:49:49.301 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:49.301 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:49.301 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:49.301 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.301 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.301 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64029 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1050624~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:49.302 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167521+0000", 2024-09-21T15:49:49.302 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.121069983, 2024-09-21T15:49:49.302 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047649915000000001, 2024-09-21T15:49:49.302 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:49.302 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:49.302 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:49.302 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:49.302 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:49.302 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64029 2024-09-21T15:49:49.302 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.302 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:49.302 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.302 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:49.302 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167521+0000", 2024-09-21T15:49:49.303 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:49.303 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.303 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.303 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:49.303 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167521+0000", 2024-09-21T15:49:49.303 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:49.303 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.303 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.303 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:49.303 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167521+0000", 2024-09-21T15:49:49.303 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:49.303 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.303 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.303 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:49.304 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167523+0000", 2024-09-21T15:49:49.304 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9810000000000002e-06 2024-09-21T15:49:49.304 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.304 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.304 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:49.304 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167529+0000", 2024-09-21T15:49:49.304 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0900000000000001e-06 2024-09-21T15:49:49.304 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.304 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.304 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:49.304 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167533+0000", 2024-09-21T15:49:49.304 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5730000000000001e-06 2024-09-21T15:49:49.304 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.304 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.305 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:49.305 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215031+0000", 2024-09-21T15:49:49.305 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047497893999999999 2024-09-21T15:49:49.305 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.305 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.305 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:49.305 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215099+0000", 2024-09-21T15:49:49.305 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8416000000000004e-05 2024-09-21T15:49:49.305 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.305 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.305 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:49.305 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215171+0000", 2024-09-21T15:49:49.305 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1279000000000007e-05 2024-09-21T15:49:49.305 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:49.306 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:49.306 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:49.306 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.306 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.306 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64030 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1058816~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:49.306 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167537+0000", 2024-09-21T15:49:49.306 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.121053474, 2024-09-21T15:49:49.306 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047783816999999999, 2024-09-21T15:49:49.306 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:49.306 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:49.306 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:49.306 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:49.306 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:49.307 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64030 2024-09-21T15:49:49.307 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.307 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:49.307 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.307 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:49.307 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167537+0000", 2024-09-21T15:49:49.307 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:49.307 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.307 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.307 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:49.307 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167537+0000", 2024-09-21T15:49:49.307 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:49.307 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.307 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.307 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:49.308 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167537+0000", 2024-09-21T15:49:49.308 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:49.308 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.308 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.308 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:49.308 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167539+0000", 2024-09-21T15:49:49.308 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9700000000000002e-06 2024-09-21T15:49:49.308 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.308 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.308 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:49.308 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167545+0000", 2024-09-21T15:49:49.308 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8629999999999996e-06 2024-09-21T15:49:49.308 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.308 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.308 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:49.309 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167549+0000", 2024-09-21T15:49:49.309 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.386e-06 2024-09-21T15:49:49.309 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.309 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.309 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:49.309 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215197+0000", 2024-09-21T15:49:49.309 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047647241 2024-09-21T15:49:49.309 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.309 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.309 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:49.309 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215263+0000", 2024-09-21T15:49:49.309 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6295000000000001e-05 2024-09-21T15:49:49.309 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.309 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.309 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:49.309 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215321+0000", 2024-09-21T15:49:49.310 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8362000000000001e-05 2024-09-21T15:49:49.310 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:49.310 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:49.310 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:49.310 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.310 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.310 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64031 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1738752~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:49.310 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167554+0000", 2024-09-21T15:49:49.310 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.121036971000001, 2024-09-21T15:49:49.310 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047917855000000002, 2024-09-21T15:49:49.310 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:49.310 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:49.310 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:49.311 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:49.311 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:49.311 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64031 2024-09-21T15:49:49.311 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.311 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:49.311 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.311 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:49.311 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167554+0000", 2024-09-21T15:49:49.311 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:49.311 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.311 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.311 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:49.311 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167554+0000", 2024-09-21T15:49:49.311 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:49.312 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.312 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.312 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:49.312 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167554+0000", 2024-09-21T15:49:49.312 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:49.312 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.312 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.312 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:49.312 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167556+0000", 2024-09-21T15:49:49.312 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3870000000000002e-06 2024-09-21T15:49:49.312 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.312 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.312 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:49.312 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167562+0000", 2024-09-21T15:49:49.312 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9100000000000002e-06 2024-09-21T15:49:49.312 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.313 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.313 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:49.313 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167566+0000", 2024-09-21T15:49:49.313 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6029999999999998e-06 2024-09-21T15:49:49.313 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.313 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.313 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:49.313 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215346+0000", 2024-09-21T15:49:49.313 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.04777998 2024-09-21T15:49:49.313 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.313 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.313 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:49.313 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215412+0000", 2024-09-21T15:49:49.313 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5598000000000005e-05 2024-09-21T15:49:49.313 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.313 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.314 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:49.314 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215472+0000", 2024-09-21T15:49:49.314 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9728000000000001e-05 2024-09-21T15:49:49.314 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:49.314 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:49.314 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:49.314 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.314 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.314 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64032 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1067008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:49.314 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167673+0000", 2024-09-21T15:49:49.314 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.120918258, 2024-09-21T15:49:49.314 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047947667999999999, 2024-09-21T15:49:49.314 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:49.314 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:49.315 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:49.315 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:49.315 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:49.315 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64032 2024-09-21T15:49:49.315 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.315 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:49.315 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.315 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:49.315 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167673+0000", 2024-09-21T15:49:49.315 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:49.315 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.315 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.315 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:49.315 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167673+0000", 2024-09-21T15:49:49.315 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:49.316 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.316 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.316 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:49.316 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167672+0000", 2024-09-21T15:49:49.316 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:49.316 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.316 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.316 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:49.316 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167675+0000", 2024-09-21T15:49:49.316 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3780000000000001e-06 2024-09-21T15:49:49.316 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.317 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.317 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:49.317 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167685+0000", 2024-09-21T15:49:49.317 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3039999999999994e-06 2024-09-21T15:49:49.317 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.317 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.317 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:49.317 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167692+0000", 2024-09-21T15:49:49.317 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2559999999999996e-06 2024-09-21T15:49:49.317 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.317 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.317 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:49.317 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215495+0000", 2024-09-21T15:49:49.318 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047803267000000003 2024-09-21T15:49:49.318 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.318 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.318 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:49.318 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215560+0000", 2024-09-21T15:49:49.318 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4703999999999996e-05 2024-09-21T15:49:49.318 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.318 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.318 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:49.318 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215620+0000", 2024-09-21T15:49:49.318 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0328000000000002e-05 2024-09-21T15:49:49.318 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:49.318 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:49.319 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:49.322 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.322 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.322 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64033 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1075200~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:49.322 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167697+0000", 2024-09-21T15:49:49.322 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.120893686, 2024-09-21T15:49:49.322 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048082863000000003, 2024-09-21T15:49:49.322 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:49.322 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:49.322 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:49.322 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:49.323 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:49.323 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64033 2024-09-21T15:49:49.323 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.323 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:49.323 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.323 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:49.323 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167697+0000", 2024-09-21T15:49:49.323 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:49.323 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.323 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.323 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:49.323 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167697+0000", 2024-09-21T15:49:49.323 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:49.323 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.324 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.324 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:49.324 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167697+0000", 2024-09-21T15:49:49.324 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:49.324 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.324 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.324 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:49.324 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167699+0000", 2024-09-21T15:49:49.324 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5299999999999999e-06 2024-09-21T15:49:49.324 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.324 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.324 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:49.324 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167706+0000", 2024-09-21T15:49:49.324 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7490000000000001e-06 2024-09-21T15:49:49.325 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.325 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.325 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:49.325 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167712+0000", 2024-09-21T15:49:49.325 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1940000000000003e-06 2024-09-21T15:49:49.325 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.325 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.325 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:49.325 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215643+0000", 2024-09-21T15:49:49.325 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047930647999999999 2024-09-21T15:49:49.325 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.325 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.325 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:49.325 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215713+0000", 2024-09-21T15:49:49.325 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0488999999999995e-05 2024-09-21T15:49:49.326 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.326 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.326 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:49.326 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215780+0000", 2024-09-21T15:49:49.326 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6656999999999996e-05 2024-09-21T15:49:49.326 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:49.326 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:49.326 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:49.326 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.326 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.326 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64034 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1083392~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:49.326 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167717+0000", 2024-09-21T15:49:49.326 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.12087412, 2024-09-21T15:49:49.327 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048233316999999998, 2024-09-21T15:49:49.327 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:49.327 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:49.327 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:49.327 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:49.327 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:49.327 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64034 2024-09-21T15:49:49.327 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.327 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:49.327 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.327 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:49.327 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167717+0000", 2024-09-21T15:49:49.327 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:49.327 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.328 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.328 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:49.328 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167717+0000", 2024-09-21T15:49:49.328 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:49.328 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.328 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.328 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:49.328 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167716+0000", 2024-09-21T15:49:49.328 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:49.328 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.328 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.328 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:49.328 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167718+0000", 2024-09-21T15:49:49.328 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.165e-06 2024-09-21T15:49:49.328 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.329 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.329 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:49.329 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167725+0000", 2024-09-21T15:49:49.329 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3380000000000001e-06 2024-09-21T15:49:49.329 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.329 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.329 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:49.329 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167730+0000", 2024-09-21T15:49:49.329 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8010000000000003e-06 2024-09-21T15:49:49.329 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.329 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.329 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:49.329 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215807+0000", 2024-09-21T15:49:49.329 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048077823999999998 2024-09-21T15:49:49.330 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.330 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.330 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:49.330 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215884+0000", 2024-09-21T15:49:49.330 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6283000000000001e-05 2024-09-21T15:49:49.330 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.330 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.330 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:49.330 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215950+0000", 2024-09-21T15:49:49.330 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6290000000000004e-05 2024-09-21T15:49:49.330 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:49.330 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:49.330 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:49.330 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.331 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.331 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64035 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1091584~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:49.331 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:49:49.331 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.120856907, 2024-09-21T15:49:49.331 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048412117999999997, 2024-09-21T15:49:49.331 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:49.331 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:49.331 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:49.331 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:49.331 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:49.331 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64035 2024-09-21T15:49:49.331 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.331 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:49.331 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.331 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:49.331 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:49:49.332 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:49.332 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.332 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.332 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:49.332 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:49:49.332 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:49.332 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.332 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.332 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:49.332 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:49:49.332 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:49.332 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.332 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.332 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:49.332 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167746+0000", 2024-09-21T15:49:49.333 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2031000000000001e-05 2024-09-21T15:49:49.333 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.333 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.333 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:49.333 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167753+0000", 2024-09-21T15:49:49.333 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0620000000000003e-06 2024-09-21T15:49:49.333 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.333 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.333 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:49.333 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167759+0000", 2024-09-21T15:49:49.333 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.596e-06 2024-09-21T15:49:49.333 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.333 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.333 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:49.334 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215977+0000", 2024-09-21T15:49:49.334 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048217318000000002 2024-09-21T15:49:49.334 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.334 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.334 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:49.334 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216076+0000", 2024-09-21T15:49:49.334 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9846000000000006e-05 2024-09-21T15:49:49.334 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.334 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.334 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:49.334 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216146+0000", 2024-09-21T15:49:49.334 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9578000000000004e-05 2024-09-21T15:49:49.334 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:49.335 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:49.335 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:49.335 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.335 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.335 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64036 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1099776~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:49.335 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167853+0000", 2024-09-21T15:49:49.335 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.120738294000001, 2024-09-21T15:49:49.335 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048450675999999998, 2024-09-21T15:49:49.335 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:49.335 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:49.335 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:49.335 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:49.335 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:49.335 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64036 2024-09-21T15:49:49.336 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.336 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:49.336 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.336 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:49.336 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167853+0000", 2024-09-21T15:49:49.336 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:49.336 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.336 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.336 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:49.336 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167853+0000", 2024-09-21T15:49:49.336 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:49.337 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.337 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.337 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:49.337 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167852+0000", 2024-09-21T15:49:49.337 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:49.337 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.337 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.337 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:49.337 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167855+0000", 2024-09-21T15:49:49.337 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5640000000000001e-06 2024-09-21T15:49:49.337 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.337 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.337 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:49.337 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167864+0000", 2024-09-21T15:49:49.338 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8140000000000007e-06 2024-09-21T15:49:49.338 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.338 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.338 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:49.338 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167871+0000", 2024-09-21T15:49:49.338 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5729999999999996e-06 2024-09-21T15:49:49.338 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.338 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.338 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:49.338 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216174+0000", 2024-09-21T15:49:49.338 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048303437999999997 2024-09-21T15:49:49.338 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.338 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.338 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:49.339 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216250+0000", 2024-09-21T15:49:49.339 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5661999999999995e-05 2024-09-21T15:49:49.339 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.339 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.339 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:49.339 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216303+0000", 2024-09-21T15:49:49.339 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3273000000000003e-05 2024-09-21T15:49:49.339 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:49.339 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:49.339 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:49.339 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.339 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.339 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64037 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1771520~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:49.339 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:49:49.340 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.120714751, 2024-09-21T15:49:49.340 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048588771000000003, 2024-09-21T15:49:49.340 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:49.340 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:49.340 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:49.340 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:49.340 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:49.340 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64037 2024-09-21T15:49:49.340 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.340 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:49.340 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.340 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:49.340 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:49:49.340 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:49.340 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.341 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.341 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:49.341 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:49:49.341 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:49.341 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.341 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.341 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:49.341 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:49:49.341 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:49.341 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.341 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.341 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:49.341 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167878+0000", 2024-09-21T15:49:49.341 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3e-06 2024-09-21T15:49:49.342 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.342 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.342 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:49.342 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167884+0000", 2024-09-21T15:49:49.342 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3269999999999997e-06 2024-09-21T15:49:49.342 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.342 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.342 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:49.342 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167889+0000", 2024-09-21T15:49:49.342 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9269999999999996e-06 2024-09-21T15:49:49.342 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.342 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.342 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:49.342 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216331+0000", 2024-09-21T15:49:49.342 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048442102000000001 2024-09-21T15:49:49.343 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.343 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.343 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:49.343 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216393+0000", 2024-09-21T15:49:49.343 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1132999999999999e-05 2024-09-21T15:49:49.343 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.343 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.343 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:49.343 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216465+0000", 2024-09-21T15:49:49.343 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2339e-05 2024-09-21T15:49:49.343 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:49.343 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:49.343 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:49.343 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.344 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.344 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64038 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1779712~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:49.344 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:49:49.344 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.12069692, 2024-09-21T15:49:49.344 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048749795999999998, 2024-09-21T15:49:49.344 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:49.344 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:49.344 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:49.344 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:49.344 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:49.344 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64038 2024-09-21T15:49:49.344 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.344 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:49.344 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.344 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:49.345 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:49:49.345 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:49.345 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.345 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.345 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:49.345 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:49:49.345 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:49.345 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.345 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.345 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:49.345 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:49:49.345 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:49.345 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.345 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.345 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:49.346 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167896+0000", 2024-09-21T15:49:49.346 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2419999999999999e-06 2024-09-21T15:49:49.346 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.346 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.346 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:49.346 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167902+0000", 2024-09-21T15:49:49.346 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0859999999999998e-06 2024-09-21T15:49:49.346 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.346 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.346 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:49.346 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167907+0000", 2024-09-21T15:49:49.346 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9679999999999997e-06 2024-09-21T15:49:49.346 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.346 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.347 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:49.347 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216506+0000", 2024-09-21T15:49:49.347 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048599374000000001 2024-09-21T15:49:49.347 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.347 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.347 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:49.347 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216579+0000", 2024-09-21T15:49:49.347 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2801999999999999e-05 2024-09-21T15:49:49.347 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.347 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.347 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:49.347 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216644+0000", 2024-09-21T15:49:49.347 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4715000000000006e-05 2024-09-21T15:49:49.347 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:49.348 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:49.348 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:49.348 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.348 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.348 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64039 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1107968~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:49.348 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:49:49.348 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.120679565, 2024-09-21T15:49:49.348 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048911370000000003, 2024-09-21T15:49:49.348 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:49.348 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:49.348 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:49.348 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:49.348 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:49.348 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64039 2024-09-21T15:49:49.349 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.349 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:49.349 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.349 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:49.349 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:49:49.349 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:49.349 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.349 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.349 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:49.349 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:49:49.349 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:49.349 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.349 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.349 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:49.349 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:49:49.350 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:49.350 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.350 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.350 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:49.350 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167913+0000", 2024-09-21T15:49:49.350 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0839999999999999e-06 2024-09-21T15:49:49.350 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.350 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.350 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:49.350 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167921+0000", 2024-09-21T15:49:49.350 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8310000000000001e-06 2024-09-21T15:49:49.350 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.350 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.350 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:49.351 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167926+0000", 2024-09-21T15:49:49.351 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0270000000000003e-06 2024-09-21T15:49:49.351 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.351 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.351 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:49.351 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216680+0000", 2024-09-21T15:49:49.351 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048753890000000001 2024-09-21T15:49:49.351 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.351 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.351 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:49.351 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216757+0000", 2024-09-21T15:49:49.351 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6848999999999996e-05 2024-09-21T15:49:49.351 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.351 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.351 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:49.352 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216823+0000", 2024-09-21T15:49:49.352 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6068000000000005e-05 2024-09-21T15:49:49.352 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:49.352 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:49.352 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:49.352 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.352 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.352 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64040 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1787904~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:49.352 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167931+0000", 2024-09-21T15:49:49.352 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.120660333, 2024-09-21T15:49:49.352 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049010561000000001, 2024-09-21T15:49:49.352 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:49.352 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:49.352 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:49.353 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:49.353 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:49.353 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64040 2024-09-21T15:49:49.353 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.353 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:49.353 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.353 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:49.353 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167931+0000", 2024-09-21T15:49:49.353 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:49.353 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.353 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.353 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:49.353 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167931+0000", 2024-09-21T15:49:49.353 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:49.353 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.354 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.354 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:49.354 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167930+0000", 2024-09-21T15:49:49.354 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:49.354 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.354 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.354 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:49.354 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167932+0000", 2024-09-21T15:49:49.354 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.198e-06 2024-09-21T15:49:49.354 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.354 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.354 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:49.354 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167938+0000", 2024-09-21T15:49:49.354 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.105e-06 2024-09-21T15:49:49.355 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.355 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.355 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:49.355 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167943+0000", 2024-09-21T15:49:49.355 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9429999999999999e-06 2024-09-21T15:49:49.355 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.355 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.355 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:49.355 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216848+0000", 2024-09-21T15:49:49.355 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048905200000000003 2024-09-21T15:49:49.355 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.355 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.355 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:49.355 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216903+0000", 2024-09-21T15:49:49.355 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4793999999999997e-05 2024-09-21T15:49:49.356 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.356 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.356 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:49.356 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216941+0000", 2024-09-21T15:49:49.356 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7790999999999998e-05 2024-09-21T15:49:49.356 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:49.356 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:49.356 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:49.356 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.356 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.356 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64041 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1796096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:49.356 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:49:49.356 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.120643361999999, 2024-09-21T15:49:49.356 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049181457999999997, 2024-09-21T15:49:49.356 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:49.357 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:49.357 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:49.357 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:49.357 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:49.357 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64041 2024-09-21T15:49:49.357 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.357 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:49.357 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.357 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:49.357 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:49:49.357 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:49.357 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.357 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.357 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:49.358 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:49:49.358 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:49.358 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.358 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.358 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:49.358 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:49:49.358 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:49.358 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.358 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.358 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:49.358 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167949+0000", 2024-09-21T15:49:49.358 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0200000000000001e-06 2024-09-21T15:49:49.358 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.358 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.358 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:49.359 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167955+0000", 2024-09-21T15:49:49.359 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5990000000000001e-06 2024-09-21T15:49:49.359 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.359 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.359 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:49.359 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167959+0000", 2024-09-21T15:49:49.359 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5589999999999999e-06 2024-09-21T15:49:49.359 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.359 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.359 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:49.359 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216972+0000", 2024-09-21T15:49:49.359 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049012574000000003 2024-09-21T15:49:49.359 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.359 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.360 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:49.360 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217057+0000", 2024-09-21T15:49:49.360 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4820000000000004e-05 2024-09-21T15:49:49.360 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.360 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.360 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:49.360 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217129+0000", 2024-09-21T15:49:49.360 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2186000000000003e-05 2024-09-21T15:49:49.360 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:49.360 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:49.360 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:49.360 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.360 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.360 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64042 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1804288~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:49.360 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167964+0000", 2024-09-21T15:49:49.361 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.120627070999999, 2024-09-21T15:49:49.361 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049333372, 2024-09-21T15:49:49.361 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:49.361 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:49.361 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:49.361 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:49.361 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:49.361 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64042 2024-09-21T15:49:49.361 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.361 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:49.361 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.361 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:49.361 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167964+0000", 2024-09-21T15:49:49.362 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:49.362 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.362 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.362 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:49.362 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167964+0000", 2024-09-21T15:49:49.362 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:49.362 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.362 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.362 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:49.362 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167963+0000", 2024-09-21T15:49:49.362 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:49.362 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.362 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.362 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:49.363 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167966+0000", 2024-09-21T15:49:49.363 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1119999999999999e-06 2024-09-21T15:49:49.363 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.363 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.363 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:49.363 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167975+0000", 2024-09-21T15:49:49.363 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1130000000000007e-06 2024-09-21T15:49:49.363 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.363 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.363 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:49.363 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167980+0000", 2024-09-21T15:49:49.363 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9599999999999999e-06 2024-09-21T15:49:49.363 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.363 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.364 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:49.364 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217158+0000", 2024-09-21T15:49:49.364 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049178569999999998 2024-09-21T15:49:49.364 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.364 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.364 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:49.364 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217232+0000", 2024-09-21T15:49:49.364 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3589000000000003e-05 2024-09-21T15:49:49.364 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.364 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.364 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:49.364 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217297+0000", 2024-09-21T15:49:49.364 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5339999999999994e-05 2024-09-21T15:49:49.364 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:49.365 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:49.365 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:49.365 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.365 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.365 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64043 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1812480~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:49.365 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.168085+0000", 2024-09-21T15:49:49.365 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.120506003999999, 2024-09-21T15:49:49.365 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049365157999999999, 2024-09-21T15:49:49.365 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:49.365 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:49.365 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:49.365 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:49.365 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:49.365 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64043 2024-09-21T15:49:49.365 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.366 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:49.366 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.366 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:49.366 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168085+0000", 2024-09-21T15:49:49.366 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:49.366 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.366 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.366 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:49.366 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168085+0000", 2024-09-21T15:49:49.366 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:49.366 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.366 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.366 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:49.367 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168084+0000", 2024-09-21T15:49:49.367 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:49.367 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.367 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.367 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:49.367 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168088+0000", 2024-09-21T15:49:49.367 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5420000000000001e-06 2024-09-21T15:49:49.367 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.367 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.367 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:49.367 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168097+0000", 2024-09-21T15:49:49.367 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5410000000000006e-06 2024-09-21T15:49:49.367 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.367 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.367 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:49.368 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168104+0000", 2024-09-21T15:49:49.368 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6719999999999998e-06 2024-09-21T15:49:49.368 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.368 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.368 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:49.368 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217323+0000", 2024-09-21T15:49:49.368 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049218857999999997 2024-09-21T15:49:49.368 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.368 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.368 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:49.368 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217397+0000", 2024-09-21T15:49:49.368 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3932999999999998e-05 2024-09-21T15:49:49.368 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.368 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.369 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:49.369 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217450+0000", 2024-09-21T15:49:49.369 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3168000000000003e-05 2024-09-21T15:49:49.369 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:49.369 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:49.369 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:49.369 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.369 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.369 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64044 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1820672~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:49.369 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.170036+0000", 2024-09-21T15:49:49.369 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.118554684999999, 2024-09-21T15:49:49.369 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047578069000000001, 2024-09-21T15:49:49.369 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:49.369 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:49.369 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:49.370 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:49.370 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:49.370 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64044 2024-09-21T15:49:49.370 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.370 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:49.370 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.370 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:49.370 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170036+0000", 2024-09-21T15:49:49.370 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:49.370 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.370 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.370 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:49.370 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170036+0000", 2024-09-21T15:49:49.370 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:49.371 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.371 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.371 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:49.371 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168109+0000", 2024-09-21T15:49:49.371 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9980721 2024-09-21T15:49:49.371 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.371 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.371 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:49.371 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170040+0000", 2024-09-21T15:49:49.371 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0019318079999999999 2024-09-21T15:49:49.371 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.371 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.371 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:49.371 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170051+0000", 2024-09-21T15:49:49.372 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0220999999999999e-05 2024-09-21T15:49:49.372 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.372 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.372 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:49.372 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170058+0000", 2024-09-21T15:49:49.372 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0110000000000003e-06 2024-09-21T15:49:49.372 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.372 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.372 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:49.372 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217478+0000", 2024-09-21T15:49:49.372 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047420334000000001 2024-09-21T15:49:49.372 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.372 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.372 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:49.372 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217547+0000", 2024-09-21T15:49:49.373 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9416000000000001e-05 2024-09-21T15:49:49.373 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.373 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.373 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:49.373 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217614+0000", 2024-09-21T15:49:49.373 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6909000000000006e-05 2024-09-21T15:49:49.373 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:49.373 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:49.373 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:49.373 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.373 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.373 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64045 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1828864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:49.373 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.170064+0000", 2024-09-21T15:49:49.373 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.118526645999999, 2024-09-21T15:49:49.373 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047709252000000001, 2024-09-21T15:49:49.374 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:49.374 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:49.374 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:49.374 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:49.374 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:49.374 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64045 2024-09-21T15:49:49.374 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.374 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:49.374 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.374 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:49.374 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170064+0000", 2024-09-21T15:49:49.374 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:49.374 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.374 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.375 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:49.375 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170064+0000", 2024-09-21T15:49:49.375 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:49.375 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.375 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.375 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:49.375 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170064+0000", 2024-09-21T15:49:49.375 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:49.375 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.375 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.375 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:49.375 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170066+0000", 2024-09-21T15:49:49.375 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8109999999999999e-06 2024-09-21T15:49:49.376 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.376 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.376 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:49.376 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170073+0000", 2024-09-21T15:49:49.376 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5570000000000001e-06 2024-09-21T15:49:49.376 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.376 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.376 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:49.376 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170078+0000", 2024-09-21T15:49:49.376 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9910000000000002e-06 2024-09-21T15:49:49.376 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.376 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.376 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:49.376 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217640+0000", 2024-09-21T15:49:49.377 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047561840000000001 2024-09-21T15:49:49.377 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.377 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.377 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:49.377 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217710+0000", 2024-09-21T15:49:49.377 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9967000000000003e-05 2024-09-21T15:49:49.377 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.377 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.377 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:49.377 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217773+0000", 2024-09-21T15:49:49.377 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3638000000000003e-05 2024-09-21T15:49:49.377 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:49.377 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:49.377 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:49.377 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.378 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.378 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64356 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 3076096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:49.378 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.183065+0000", 2024-09-21T15:49:49.378 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.105525515, 2024-09-21T15:49:49.378 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047484923999999998, 2024-09-21T15:49:49.378 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:49.378 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:49.378 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:49.378 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:49.378 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:49.378 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64356 2024-09-21T15:49:49.378 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.378 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:49.379 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.379 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:49.379 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.183065+0000", 2024-09-21T15:49:49.379 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:49.379 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.379 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.379 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:49.379 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.183065+0000", 2024-09-21T15:49:49.379 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:49.379 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.379 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.379 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:49.379 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.182922+0000", 2024-09-21T15:49:49.379 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999856 2024-09-21T15:49:49.379 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.380 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.380 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:49.380 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.183069+0000", 2024-09-21T15:49:49.380 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000147097 2024-09-21T15:49:49.380 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.380 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.380 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:49.380 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.183078+0000", 2024-09-21T15:49:49.380 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.9099999999999994e-06 2024-09-21T15:49:49.380 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.380 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.380 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:49.380 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.183085+0000", 2024-09-21T15:49:49.380 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0670000000000003e-06 2024-09-21T15:49:49.381 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.381 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.381 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:49.381 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.230377+0000", 2024-09-21T15:49:49.381 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047292698000000001 2024-09-21T15:49:49.381 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.381 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.381 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:49.381 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.230458+0000", 2024-09-21T15:49:49.381 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0980000000000001e-05 2024-09-21T15:49:49.381 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:49.381 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:49.381 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:49.381 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.230550+0000", 2024-09-21T15:49:49.381 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1954999999999996e-05 2024-09-21T15:49:49.382 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:49.382 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:49.382 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:49.382 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:49.382 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:49.382 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:49.383 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:49.382+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:49.383 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:49:49.484 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:49.483+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:49.486 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:49.487 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:49:49.487 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:49:49.487 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:49.495 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:49:49.584 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:49.584+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:49.638 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:49.638 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:49:49.639 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:49:49.639 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:49:49.639 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:49.647 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:49:49.692 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:49.691+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:49.750 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:49.750 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:49:49.750 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:49:49.750 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:49:49.750 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.750 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65283 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2836480~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:49.750 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.218024+0000", 2024-09-21T15:49:49.750 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.530069688999999, 2024-09-21T15:49:49.750 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043727861, 2024-09-21T15:49:49.751 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:49.751 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:49.751 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:49.751 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:49.751 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:49.751 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65283 2024-09-21T15:49:49.751 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.751 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:49.751 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.751 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:49.751 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218024+0000", 2024-09-21T15:49:49.751 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:49.751 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.752 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.752 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:49.752 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218024+0000", 2024-09-21T15:49:49.752 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:49.752 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.752 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.752 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:49.752 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218024+0000", 2024-09-21T15:49:49.752 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:49.752 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.752 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.752 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:49.752 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218026+0000", 2024-09-21T15:49:49.752 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.717e-06 2024-09-21T15:49:49.753 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.753 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.753 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:49.753 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218030+0000", 2024-09-21T15:49:49.753 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8630000000000001e-06 2024-09-21T15:49:49.753 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.753 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.753 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:49.753 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218033+0000", 2024-09-21T15:49:49.753 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9660000000000001e-06 2024-09-21T15:49:49.753 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.753 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.753 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:49.754 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.261601+0000", 2024-09-21T15:49:49.754 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043567862999999998 2024-09-21T15:49:49.754 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.754 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.754 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:49.754 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.261679+0000", 2024-09-21T15:49:49.754 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7807999999999998e-05 2024-09-21T15:49:49.754 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.754 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.754 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:49.754 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.261752+0000", 2024-09-21T15:49:49.755 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2977000000000003e-05 2024-09-21T15:49:49.755 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:49.755 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:49.755 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:49.755 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.755 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.755 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65284 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2844672~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:49.755 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.218037+0000", 2024-09-21T15:49:49.755 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.530056766, 2024-09-21T15:49:49.755 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043873034999999998, 2024-09-21T15:49:49.755 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:49.755 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:49.755 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:49.756 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:49.756 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:49.756 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65284 2024-09-21T15:49:49.756 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.756 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:49.756 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.756 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:49.756 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218037+0000", 2024-09-21T15:49:49.756 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:49.756 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.756 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.756 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:49.756 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218037+0000", 2024-09-21T15:49:49.756 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:49.756 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.757 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.757 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:49.757 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218037+0000", 2024-09-21T15:49:49.757 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:49:49.757 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.757 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.757 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:49.757 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218038+0000", 2024-09-21T15:49:49.757 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.533e-06 2024-09-21T15:49:49.757 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.757 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.757 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:49.757 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218041+0000", 2024-09-21T15:49:49.757 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9799999999999998e-06 2024-09-21T15:49:49.757 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.758 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.758 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:49.758 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218044+0000", 2024-09-21T15:49:49.758 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.458e-06 2024-09-21T15:49:49.758 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.758 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.758 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:49.758 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.261781+0000", 2024-09-21T15:49:49.758 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043736935999999997 2024-09-21T15:49:49.758 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.758 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.758 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:49.758 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.261846+0000", 2024-09-21T15:49:49.759 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5233000000000004e-05 2024-09-21T15:49:49.759 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.759 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.759 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:49.759 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.261910+0000", 2024-09-21T15:49:49.759 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4103000000000006e-05 2024-09-21T15:49:49.759 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:49.759 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:49.759 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:49.759 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.759 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.759 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65285 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2861056~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:49.759 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.218171+0000", 2024-09-21T15:49:49.759 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.529922619000001, 2024-09-21T15:49:49.759 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043909057000000001, 2024-09-21T15:49:49.760 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:49.760 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:49.760 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:49.760 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:49.760 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:49.760 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65285 2024-09-21T15:49:49.760 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.760 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:49.760 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.760 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:49.760 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218171+0000", 2024-09-21T15:49:49.760 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:49.760 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.760 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.760 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:49.760 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218171+0000", 2024-09-21T15:49:49.761 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:49.761 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.761 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.761 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:49.761 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218171+0000", 2024-09-21T15:49:49.761 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:49.761 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.761 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.761 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:49.761 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218173+0000", 2024-09-21T15:49:49.761 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.588e-06 2024-09-21T15:49:49.761 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.761 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.761 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:49.761 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218178+0000", 2024-09-21T15:49:49.762 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4719999999999997e-06 2024-09-21T15:49:49.762 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.762 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.762 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:49.762 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218181+0000", 2024-09-21T15:49:49.762 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0979999999999998e-06 2024-09-21T15:49:49.762 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.762 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.762 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:49.762 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.261943+0000", 2024-09-21T15:49:49.762 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043762652999999999 2024-09-21T15:49:49.762 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.762 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.763 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:49.763 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262018+0000", 2024-09-21T15:49:49.763 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4352999999999998e-05 2024-09-21T15:49:49.763 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.763 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.763 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:49.763 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262080+0000", 2024-09-21T15:49:49.763 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2477999999999994e-05 2024-09-21T15:49:49.763 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:49.763 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:49.763 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:49.763 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.763 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.764 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65286 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2869248~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:49.764 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.218284+0000", 2024-09-21T15:49:49.764 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.529810136, 2024-09-21T15:49:49.764 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043921241999999999, 2024-09-21T15:49:49.764 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:49.764 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:49.764 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:49.764 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:49.764 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:49.764 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65286 2024-09-21T15:49:49.764 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.764 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:49.764 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.764 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:49.765 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218284+0000", 2024-09-21T15:49:49.765 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:49.765 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.765 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.765 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:49.765 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218284+0000", 2024-09-21T15:49:49.765 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:49.765 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.765 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.765 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:49.765 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218283+0000", 2024-09-21T15:49:49.765 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:49.765 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.765 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.765 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:49.766 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218286+0000", 2024-09-21T15:49:49.766 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4159999999999998e-06 2024-09-21T15:49:49.766 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.766 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.766 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:49.766 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218290+0000", 2024-09-21T15:49:49.766 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3370000000000001e-06 2024-09-21T15:49:49.766 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.766 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.766 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:49.766 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218293+0000", 2024-09-21T15:49:49.766 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0960000000000001e-06 2024-09-21T15:49:49.766 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.767 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.767 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:49.767 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262105+0000", 2024-09-21T15:49:49.767 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043811914 2024-09-21T15:49:49.767 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.767 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.767 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:49.767 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262152+0000", 2024-09-21T15:49:49.767 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7341e-05 2024-09-21T15:49:49.767 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.767 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.767 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:49.767 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262205+0000", 2024-09-21T15:49:49.768 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2667000000000002e-05 2024-09-21T15:49:49.768 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:49.768 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:49.768 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:49.768 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.768 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.768 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65287 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2877440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:49.768 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.218326+0000", 2024-09-21T15:49:49.768 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.529767873000001, 2024-09-21T15:49:49.768 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.044001874000000003, 2024-09-21T15:49:49.768 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:49.768 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:49.768 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:49.769 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:49.769 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:49.769 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65287 2024-09-21T15:49:49.769 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.769 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:49.769 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.769 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:49.769 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218326+0000", 2024-09-21T15:49:49.769 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:49.769 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.769 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.769 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:49.769 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218326+0000", 2024-09-21T15:49:49.770 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:49.770 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.770 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.770 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:49.770 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218326+0000", 2024-09-21T15:49:49.770 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:49.770 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.770 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.770 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:49.770 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218328+0000", 2024-09-21T15:49:49.770 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.029e-06 2024-09-21T15:49:49.770 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.770 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.771 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:49.771 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218334+0000", 2024-09-21T15:49:49.771 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9519999999999999e-06 2024-09-21T15:49:49.771 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.771 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.771 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:49.771 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218338+0000", 2024-09-21T15:49:49.771 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7500000000000003e-06 2024-09-21T15:49:49.771 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.771 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.771 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:49.771 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262228+0000", 2024-09-21T15:49:49.771 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043890058000000003 2024-09-21T15:49:49.772 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.772 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.772 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:49.772 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262285+0000", 2024-09-21T15:49:49.772 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6779e-05 2024-09-21T15:49:49.772 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.772 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.772 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:49.772 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262328+0000", 2024-09-21T15:49:49.772 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2599999999999999e-05 2024-09-21T15:49:49.772 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:49.772 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:49.772 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:49.773 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.773 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.773 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65288 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2885632~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:49.773 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.218345+0000", 2024-09-21T15:49:49.773 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.529749141, 2024-09-21T15:49:49.773 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.044143245999999997, 2024-09-21T15:49:49.773 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:49.773 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:49.773 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:49.773 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:49.773 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:49.774 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65288 2024-09-21T15:49:49.774 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.774 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:49.774 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.774 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:49.774 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218345+0000", 2024-09-21T15:49:49.774 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:49.774 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.774 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.774 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:49.774 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218345+0000", 2024-09-21T15:49:49.774 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:49.775 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.775 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.775 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:49.775 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218344+0000", 2024-09-21T15:49:49.775 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:49.775 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.775 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.775 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:49.775 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218347+0000", 2024-09-21T15:49:49.775 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3920000000000001e-06 2024-09-21T15:49:49.775 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.775 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.775 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:49.775 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218353+0000", 2024-09-21T15:49:49.776 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9610000000000002e-06 2024-09-21T15:49:49.776 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.776 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.776 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:49.776 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218357+0000", 2024-09-21T15:49:49.776 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4700000000000004e-06 2024-09-21T15:49:49.776 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.776 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.776 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:49.776 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262405+0000", 2024-09-21T15:49:49.776 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.044047500000000003 2024-09-21T15:49:49.776 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.776 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.776 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:49.777 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262446+0000", 2024-09-21T15:49:49.777 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1151e-05 2024-09-21T15:49:49.777 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.777 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.777 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:49.777 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262488+0000", 2024-09-21T15:49:49.777 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2182999999999999e-05 2024-09-21T15:49:49.777 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:49.777 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:49.777 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:49.777 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.777 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.777 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65289 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2893824~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:49.777 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.218394+0000", 2024-09-21T15:49:49.777 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.529699938, 2024-09-21T15:49:49.777 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.044235021999999999, 2024-09-21T15:49:49.778 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:49.778 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:49.778 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:49.778 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:49.778 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:49.778 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65289 2024-09-21T15:49:49.778 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.778 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:49.778 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.778 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:49.778 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218394+0000", 2024-09-21T15:49:49.778 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:49.778 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.778 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.779 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:49.779 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218394+0000", 2024-09-21T15:49:49.779 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:49.779 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.779 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.779 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:49.779 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218393+0000", 2024-09-21T15:49:49.779 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:49.779 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.779 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.779 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:49.779 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218397+0000", 2024-09-21T15:49:49.779 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.27e-06 2024-09-21T15:49:49.779 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.779 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.780 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:49.780 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218404+0000", 2024-09-21T15:49:49.780 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6329999999999996e-06 2024-09-21T15:49:49.780 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.780 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.780 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:49.780 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218415+0000", 2024-09-21T15:49:49.780 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0373999999999999e-05 2024-09-21T15:49:49.780 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.780 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.780 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:49.780 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262511+0000", 2024-09-21T15:49:49.780 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.044096630999999997 2024-09-21T15:49:49.780 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.780 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.780 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:49.781 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262562+0000", 2024-09-21T15:49:49.781 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1266e-05 2024-09-21T15:49:49.781 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.781 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.781 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:49.781 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262629+0000", 2024-09-21T15:49:49.781 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6388000000000002e-05 2024-09-21T15:49:49.781 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:49.781 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:49.781 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:49.781 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.781 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.781 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65290 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2852864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:49.781 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.218451+0000", 2024-09-21T15:49:49.781 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.529642331, 2024-09-21T15:49:49.781 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.044280721000000002, 2024-09-21T15:49:49.782 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:49.782 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:49.782 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:49.782 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:49.782 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:49.782 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65290 2024-09-21T15:49:49.782 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.782 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:49.782 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.782 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:49.782 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218451+0000", 2024-09-21T15:49:49.782 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:49.782 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.782 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.783 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:49.783 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218451+0000", 2024-09-21T15:49:49.783 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:49.783 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.783 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.783 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:49.783 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218451+0000", 2024-09-21T15:49:49.783 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:49.783 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.783 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.783 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:49.783 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218454+0000", 2024-09-21T15:49:49.783 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7489999999999999e-06 2024-09-21T15:49:49.783 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.783 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.784 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:49.784 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218462+0000", 2024-09-21T15:49:49.784 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5329999999999994e-06 2024-09-21T15:49:49.784 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.784 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.784 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:49.784 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218465+0000", 2024-09-21T15:49:49.784 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1990000000000002e-06 2024-09-21T15:49:49.784 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.784 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.784 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:49.784 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262655+0000", 2024-09-21T15:49:49.784 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.044189472 2024-09-21T15:49:49.784 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.784 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.784 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:49.785 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262695+0000", 2024-09-21T15:49:49.785 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0476999999999999e-05 2024-09-21T15:49:49.785 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.785 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.785 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:49.785 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262732+0000", 2024-09-21T15:49:49.785 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6795000000000003e-05 2024-09-21T15:49:49.785 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:49.785 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:49.785 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:49.785 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.785 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.785 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65291 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2902016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:49.785 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.218644+0000", 2024-09-21T15:49:49.785 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.529449815, 2024-09-21T15:49:49.785 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.044258391000000001, 2024-09-21T15:49:49.786 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:49.786 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:49.786 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:49.786 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:49.786 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:49.786 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65291 2024-09-21T15:49:49.786 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.786 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:49.786 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.786 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:49.786 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218644+0000", 2024-09-21T15:49:49.786 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:49.786 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.786 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.787 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:49.787 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218644+0000", 2024-09-21T15:49:49.787 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:49.787 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.787 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.787 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:49.787 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218468+0000", 2024-09-21T15:49:49.787 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999824 2024-09-21T15:49:49.787 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.787 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.787 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:49.787 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218649+0000", 2024-09-21T15:49:49.787 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00018061999999999999 2024-09-21T15:49:49.787 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.787 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.788 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:49.788 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218657+0000", 2024-09-21T15:49:49.788 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6430000000000001e-06 2024-09-21T15:49:49.788 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.788 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.788 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:49.788 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218662+0000", 2024-09-21T15:49:49.788 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.814e-06 2024-09-21T15:49:49.788 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.788 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.788 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:49.788 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262766+0000", 2024-09-21T15:49:49.789 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.044103705999999999 2024-09-21T15:49:49.789 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.789 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.789 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:49.789 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262838+0000", 2024-09-21T15:49:49.789 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2312999999999997e-05 2024-09-21T15:49:49.789 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.789 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.789 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:49.789 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262902+0000", 2024-09-21T15:49:49.789 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4282999999999994e-05 2024-09-21T15:49:49.789 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:49.789 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:49.790 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:49.790 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.790 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.790 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65292 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2910208~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:49.790 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.218810+0000", 2024-09-21T15:49:49.790 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.529283603, 2024-09-21T15:49:49.790 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.044257131999999998, 2024-09-21T15:49:49.790 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:49.790 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:49.790 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:49.790 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:49.790 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:49.790 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65292 2024-09-21T15:49:49.791 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.791 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:49.791 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.791 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:49.791 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218810+0000", 2024-09-21T15:49:49.791 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:49.791 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.791 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.791 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:49.791 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218810+0000", 2024-09-21T15:49:49.791 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:49.791 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.791 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.791 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:49.792 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218666+0000", 2024-09-21T15:49:49.792 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999856 2024-09-21T15:49:49.792 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.792 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.792 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:49.792 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218814+0000", 2024-09-21T15:49:49.792 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014768299999999999 2024-09-21T15:49:49.792 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.792 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.792 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:49.792 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218823+0000", 2024-09-21T15:49:49.792 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2399999999999996e-06 2024-09-21T15:49:49.792 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.792 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.793 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:49.793 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218829+0000", 2024-09-21T15:49:49.793 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8279999999999999e-06 2024-09-21T15:49:49.793 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.793 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.793 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:49.793 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262933+0000", 2024-09-21T15:49:49.793 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.044104065999999997 2024-09-21T15:49:49.793 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.793 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.793 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:49.793 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263014+0000", 2024-09-21T15:49:49.793 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0717999999999995e-05 2024-09-21T15:49:49.793 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.793 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.794 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:49.794 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263067+0000", 2024-09-21T15:49:49.794 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3653999999999998e-05 2024-09-21T15:49:49.794 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:49.794 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:49.794 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:49.794 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.794 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.794 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65293 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2918400~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:49.794 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.218930+0000", 2024-09-21T15:49:49.794 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.529163922, 2024-09-21T15:49:49.794 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.044253536000000003, 2024-09-21T15:49:49.794 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:49.795 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:49.795 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:49.795 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:49.795 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:49.795 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65293 2024-09-21T15:49:49.795 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.795 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:49.795 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.795 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:49.795 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218930+0000", 2024-09-21T15:49:49.795 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:49.795 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.795 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.795 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:49.796 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218930+0000", 2024-09-21T15:49:49.796 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:49.796 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.796 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.796 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:49.796 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218834+0000", 2024-09-21T15:49:49.796 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999042 2024-09-21T15:49:49.796 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.796 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.796 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:49.796 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218933+0000", 2024-09-21T15:49:49.796 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9047000000000002e-05 2024-09-21T15:49:49.796 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.796 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.797 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:49.797 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218941+0000", 2024-09-21T15:49:49.797 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0849999999999997e-06 2024-09-21T15:49:49.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.797 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.797 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:49.797 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218945+0000", 2024-09-21T15:49:49.797 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9179999999999996e-06 2024-09-21T15:49:49.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.797 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.797 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:49.797 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263093+0000", 2024-09-21T15:49:49.797 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.044147906000000001 2024-09-21T15:49:49.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.798 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.798 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:49.798 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263144+0000", 2024-09-21T15:49:49.798 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0470999999999998e-05 2024-09-21T15:49:49.798 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.798 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.798 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:49.798 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263183+0000", 2024-09-21T15:49:49.798 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9839999999999998e-05 2024-09-21T15:49:49.798 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:49.798 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:49.798 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:49.799 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.799 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.799 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65294 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2926592~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:49.799 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.219137+0000", 2024-09-21T15:49:49.799 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.528956746, 2024-09-21T15:49:49.799 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.044168171999999999, 2024-09-21T15:49:49.799 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:49.799 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:49.799 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:49.799 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:49.799 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:49.799 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65294 2024-09-21T15:49:49.799 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.799 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:49.799 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.800 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:49.800 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.219137+0000", 2024-09-21T15:49:49.800 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:49.800 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.800 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.800 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:49.800 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.219137+0000", 2024-09-21T15:49:49.800 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:49.800 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.800 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.800 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:49.800 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218952+0000", 2024-09-21T15:49:49.800 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999815 2024-09-21T15:49:49.800 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.800 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.800 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:49.801 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.219140+0000", 2024-09-21T15:49:49.801 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00018848899999999999 2024-09-21T15:49:49.801 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.801 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.801 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:49.801 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.219147+0000", 2024-09-21T15:49:49.801 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0010000000000004e-06 2024-09-21T15:49:49.801 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.801 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.801 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:49.801 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.219153+0000", 2024-09-21T15:49:49.801 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4060000000000004e-06 2024-09-21T15:49:49.801 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.801 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.801 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:49.802 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263206+0000", 2024-09-21T15:49:49.802 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.044053467999999998 2024-09-21T15:49:49.802 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.802 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.802 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:49.802 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263248+0000", 2024-09-21T15:49:49.802 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1282000000000002e-05 2024-09-21T15:49:49.802 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.802 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.802 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:49.802 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263305+0000", 2024-09-21T15:49:49.802 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7645000000000001e-05 2024-09-21T15:49:49.802 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:49.802 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:49.802 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:49.803 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.803 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.803 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65295 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2934784~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:49.803 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.219274+0000", 2024-09-21T15:49:49.803 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.528819987, 2024-09-21T15:49:49.803 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.044207710999999997, 2024-09-21T15:49:49.803 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:49.803 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:49.803 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:49.803 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:49.803 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:49.803 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65295 2024-09-21T15:49:49.803 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.803 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:49.803 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.804 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:49.804 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.219274+0000", 2024-09-21T15:49:49.804 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:49.804 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.804 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.804 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:49.804 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.219274+0000", 2024-09-21T15:49:49.804 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:49.804 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.804 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.804 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:49.804 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.219158+0000", 2024-09-21T15:49:49.804 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9998837 2024-09-21T15:49:49.805 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.805 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.805 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:49.805 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.219276+0000", 2024-09-21T15:49:49.805 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000118656 2024-09-21T15:49:49.805 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.805 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.805 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:49.805 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.219281+0000", 2024-09-21T15:49:49.805 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7360000000000001e-06 2024-09-21T15:49:49.805 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.805 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.805 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:49.806 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.219285+0000", 2024-09-21T15:49:49.806 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6349999999999999e-06 2024-09-21T15:49:49.806 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.806 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.806 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:49.806 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263343+0000", 2024-09-21T15:49:49.806 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.044058264 2024-09-21T15:49:49.806 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.806 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.806 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:49.806 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263418+0000", 2024-09-21T15:49:49.806 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5001999999999998e-05 2024-09-21T15:49:49.806 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.806 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.806 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:49.807 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263482+0000", 2024-09-21T15:49:49.807 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3639999999999994e-05 2024-09-21T15:49:49.807 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:49.807 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:49.807 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:49.807 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.807 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.807 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65297 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2951168~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:49.807 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.220022+0000", 2024-09-21T15:49:49.807 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.528072094000001, 2024-09-21T15:49:49.807 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043682718000000002, 2024-09-21T15:49:49.807 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:49.807 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:49.807 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:49.807 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:49.807 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:49.808 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65297 2024-09-21T15:49:49.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.808 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:49.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.808 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:49.808 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220022+0000", 2024-09-21T15:49:49.808 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:49.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.808 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:49.808 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220022+0000", 2024-09-21T15:49:49.808 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:49.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.808 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:49.808 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220021+0000", 2024-09-21T15:49:49.809 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:49.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.809 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:49.809 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220024+0000", 2024-09-21T15:49:49.809 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3870000000000002e-06 2024-09-21T15:49:49.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.809 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:49.809 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220029+0000", 2024-09-21T15:49:49.809 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4439999999999999e-06 2024-09-21T15:49:49.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.809 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:49.809 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220033+0000", 2024-09-21T15:49:49.809 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7359999999999999e-06 2024-09-21T15:49:49.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.810 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.810 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:49.810 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263625+0000", 2024-09-21T15:49:49.810 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043592190000000003 2024-09-21T15:49:49.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.810 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.810 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:49.810 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263667+0000", 2024-09-21T15:49:49.810 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2048000000000001e-05 2024-09-21T15:49:49.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.810 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.810 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:49.810 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263704+0000", 2024-09-21T15:49:49.810 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7295000000000001e-05 2024-09-21T15:49:49.811 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:49.811 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:49.811 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:49.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.811 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65298 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2959360~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:49.811 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.220036+0000", 2024-09-21T15:49:49.811 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.528058066, 2024-09-21T15:49:49.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043821723999999999, 2024-09-21T15:49:49.811 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:49.811 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:49.811 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:49.812 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:49.812 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:49.812 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65298 2024-09-21T15:49:49.812 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.812 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:49.812 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.812 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:49.812 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220036+0000", 2024-09-21T15:49:49.812 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:49.812 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.812 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.812 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:49.812 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220036+0000", 2024-09-21T15:49:49.813 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:49.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.813 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:49.813 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220035+0000", 2024-09-21T15:49:49.813 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:49.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.813 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:49.813 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220037+0000", 2024-09-21T15:49:49.813 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.66e-06 2024-09-21T15:49:49.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.813 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:49.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220040+0000", 2024-09-21T15:49:49.814 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9699999999999999e-06 2024-09-21T15:49:49.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.814 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:49.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220042+0000", 2024-09-21T15:49:49.814 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4940000000000002e-06 2024-09-21T15:49:49.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.814 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:49.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263727+0000", 2024-09-21T15:49:49.814 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043684850999999997 2024-09-21T15:49:49.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.815 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.815 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:49.815 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263789+0000", 2024-09-21T15:49:49.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2147e-05 2024-09-21T15:49:49.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.815 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.815 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:49.815 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263857+0000", 2024-09-21T15:49:49.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7970000000000001e-05 2024-09-21T15:49:49.815 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:49.815 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:49.815 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:49.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.816 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.816 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65299 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2975744~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:49.816 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.220045+0000", 2024-09-21T15:49:49.816 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.528049266, 2024-09-21T15:49:49.816 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043920945000000003, 2024-09-21T15:49:49.816 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:49.816 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:49.816 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:49.816 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:49.816 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:49.816 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65299 2024-09-21T15:49:49.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.816 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:49.816 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.817 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:49.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220045+0000", 2024-09-21T15:49:49.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:49.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.817 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:49.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220045+0000", 2024-09-21T15:49:49.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:49.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.817 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:49.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220044+0000", 2024-09-21T15:49:49.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:49:49.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.818 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.818 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:49.818 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220046+0000", 2024-09-21T15:49:49.818 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.207e-06 2024-09-21T15:49:49.818 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.818 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.818 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:49.818 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220048+0000", 2024-09-21T15:49:49.818 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7939999999999998e-06 2024-09-21T15:49:49.818 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.818 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.818 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:49.818 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220051+0000", 2024-09-21T15:49:49.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4770000000000002e-06 2024-09-21T15:49:49.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.819 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:49.819 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263885+0000", 2024-09-21T15:49:49.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043833719 2024-09-21T15:49:49.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.819 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:49.819 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263927+0000", 2024-09-21T15:49:49.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2409e-05 2024-09-21T15:49:49.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.820 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:49.820 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263965+0000", 2024-09-21T15:49:49.820 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8550999999999997e-05 2024-09-21T15:49:49.820 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:49.820 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:49.820 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:49.820 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.820 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.820 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65773 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2961408~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:49.820 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.241336+0000", 2024-09-21T15:49:49.820 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.506757791, 2024-09-21T15:49:49.820 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043734599999999998, 2024-09-21T15:49:49.821 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:49.821 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:49.821 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:49.821 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:49.821 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:49.821 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65773 2024-09-21T15:49:49.821 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.821 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:49.821 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.821 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:49.821 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241336+0000", 2024-09-21T15:49:49.821 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:49.821 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.821 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.821 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:49.822 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241336+0000", 2024-09-21T15:49:49.822 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:49.822 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.822 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.822 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:49.822 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241336+0000", 2024-09-21T15:49:49.822 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:49.822 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.822 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.822 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:49.822 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241337+0000", 2024-09-21T15:49:49.822 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.505e-06 2024-09-21T15:49:49.822 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.822 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.823 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:49.823 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241340+0000", 2024-09-21T15:49:49.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8169999999999999e-06 2024-09-21T15:49:49.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.823 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.823 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:49.823 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241342+0000", 2024-09-21T15:49:49.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4820000000000001e-06 2024-09-21T15:49:49.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.823 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.823 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:49.823 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.284936+0000", 2024-09-21T15:49:49.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043593728999999998 2024-09-21T15:49:49.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.823 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.823 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:49.824 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.285002+0000", 2024-09-21T15:49:49.824 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5581999999999997e-05 2024-09-21T15:49:49.824 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.824 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:49.824 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.285071+0000", 2024-09-21T15:49:49.824 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8818000000000004e-05 2024-09-21T15:49:49.824 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:49.824 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:49.824 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:49.824 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.824 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65774 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2985984~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:49.824 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.241555+0000", 2024-09-21T15:49:49.824 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.506538977, 2024-09-21T15:49:49.824 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043704701999999998, 2024-09-21T15:49:49.825 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:49.825 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:49.825 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:49.825 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:49.825 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:49.825 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65774 2024-09-21T15:49:49.825 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.825 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:49.825 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.825 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:49.825 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241555+0000", 2024-09-21T15:49:49.825 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:49.825 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.825 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.825 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:49.826 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241555+0000", 2024-09-21T15:49:49.826 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:49.826 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.826 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.826 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:49.826 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241344+0000", 2024-09-21T15:49:49.826 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9997897 2024-09-21T15:49:49.826 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.826 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.826 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:49.826 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241560+0000", 2024-09-21T15:49:49.826 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00021524700000000001 2024-09-21T15:49:49.826 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.826 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.826 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:49.827 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241571+0000", 2024-09-21T15:49:49.827 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1627000000000001e-05 2024-09-21T15:49:49.827 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.827 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.827 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:49.827 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241578+0000", 2024-09-21T15:49:49.827 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6889999999999998e-06 2024-09-21T15:49:49.827 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.827 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.827 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:49.827 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.285108+0000", 2024-09-21T15:49:49.827 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043529644999999999 2024-09-21T15:49:49.827 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.827 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.828 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:49.828 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.285188+0000", 2024-09-21T15:49:49.828 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0774999999999998e-05 2024-09-21T15:49:49.828 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.828 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.828 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:49.828 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.285260+0000", 2024-09-21T15:49:49.828 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1173000000000004e-05 2024-09-21T15:49:49.828 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:49.828 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:49.828 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:49.828 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.828 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.828 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65775 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2977792~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:49.828 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.241672+0000", 2024-09-21T15:49:49.828 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.506421776, 2024-09-21T15:49:49.829 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043751534000000002, 2024-09-21T15:49:49.829 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:49.829 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:49.829 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:49.829 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:49.829 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:49.829 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65775 2024-09-21T15:49:49.829 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.829 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:49.829 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.829 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:49.829 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241672+0000", 2024-09-21T15:49:49.830 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:49.830 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.830 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.830 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:49.830 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241672+0000", 2024-09-21T15:49:49.830 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:49.830 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.830 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.830 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:49.830 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241584+0000", 2024-09-21T15:49:49.830 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999118 2024-09-21T15:49:49.830 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.830 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.831 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:49.831 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241676+0000", 2024-09-21T15:49:49.831 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2115999999999997e-05 2024-09-21T15:49:49.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.831 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.831 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:49.831 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241684+0000", 2024-09-21T15:49:49.831 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3149999999999992e-06 2024-09-21T15:49:49.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.831 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.831 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:49.831 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241691+0000", 2024-09-21T15:49:49.832 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2999999999999998e-06 2024-09-21T15:49:49.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.832 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.832 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:49.832 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.285289+0000", 2024-09-21T15:49:49.832 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043598494000000002 2024-09-21T15:49:49.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.832 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.832 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:49.832 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.285361+0000", 2024-09-21T15:49:49.832 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1731999999999997e-05 2024-09-21T15:49:49.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.832 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.833 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:49.833 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.285424+0000", 2024-09-21T15:49:49.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2577999999999996e-05 2024-09-21T15:49:49.833 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:49.833 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:49.833 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:49.833 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.833 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.833 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65776 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2994176~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:49.833 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.241867+0000", 2024-09-21T15:49:49.833 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.506227194999999, 2024-09-21T15:49:49.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043720638999999999, 2024-09-21T15:49:49.833 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:49.833 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:49.834 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:49.834 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:49.834 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:49.834 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65776 2024-09-21T15:49:49.834 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.834 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:49.834 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.834 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:49.834 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241867+0000", 2024-09-21T15:49:49.834 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:49.834 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.834 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.834 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:49.834 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241867+0000", 2024-09-21T15:49:49.835 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:49.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.835 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.835 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:49.835 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241696+0000", 2024-09-21T15:49:49.835 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9998293 2024-09-21T15:49:49.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.835 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.835 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:49.835 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241871+0000", 2024-09-21T15:49:49.835 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00017491300000000001 2024-09-21T15:49:49.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.835 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.835 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:49.836 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241889+0000", 2024-09-21T15:49:49.836 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8065000000000001e-05 2024-09-21T15:49:49.836 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.836 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.836 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:49.836 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241897+0000", 2024-09-21T15:49:49.836 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9310000000000008e-06 2024-09-21T15:49:49.836 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.836 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.836 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:49.836 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.285446+0000", 2024-09-21T15:49:49.836 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043548735999999998 2024-09-21T15:49:49.836 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.836 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.836 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:49.837 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.285523+0000", 2024-09-21T15:49:49.837 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7014000000000005e-05 2024-09-21T15:49:49.837 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:49.837 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:49.837 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:49.837 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.285587+0000", 2024-09-21T15:49:49.837 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4455000000000005e-05 2024-09-21T15:49:49.837 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:49.837 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:49.837 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:49.837 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:49.837 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:49.837 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:49.839 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:49:49.839 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:49.837+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:49.910 INFO:teuthology.orchestra.run.smithi089.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-21T15:49:49.939 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:49.938+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:49.967 INFO:teuthology.orchestra.run.smithi089.stdout:truncating image image_client.0-clone20 from 0xcecf873 (overlap 0x6f31fb) to 0x0 2024-09-21T15:49:49.995 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:49.995 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:49:49.995 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:49:49.995 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:50.005 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:49:50.015 INFO:teuthology.orchestra.run.smithi089.stdout:checking clone #19, image image_client.0-clone19 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent20 2024-09-21T15:49:50.040 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:50.039+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:50.121 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:50.121 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:49:50.121 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:49:50.121 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:49:50.121 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:50.131 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:49:50.141 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:50.140+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:50.242 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:50.240+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:50.255 INFO:teuthology.orchestra.run.smithi089.stdout:3434 punch from 0x10ac3e7 to 0x10b106e, (0x4c87 bytes) 2024-09-21T15:49:50.259 INFO:teuthology.orchestra.run.smithi089.stdout:3436 read 0xb50d848 thru 0xb51b2b8 (0xda71 bytes) 2024-09-21T15:49:50.259 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:50.259 INFO:teuthology.orchestra.run.smithi089.stdout:3438 writesame 0x10a3e26 thru 0x10adf98 data_size 0x17(0xa173 bytes) 2024-09-21T15:49:50.263 INFO:teuthology.orchestra.run.smithi089.stdout:3440 read 0xba60cd1 thru 0xba702b9 (0xf5e9 bytes) 2024-09-21T15:49:50.264 INFO:teuthology.orchestra.run.smithi089.stdout:3441 write 0xc2495b9 thru 0xc251b6f (0x85b7 bytes) 2024-09-21T15:49:50.267 INFO:teuthology.orchestra.run.smithi089.stdout:3442 read 0x3f87fa4 thru 0x3f8bd8c (0x3de9 bytes) 2024-09-21T15:49:50.281 INFO:teuthology.orchestra.run.smithi089.stdout:3443 trunc from 0xcecf873 to 0x1db3f4f 2024-09-21T15:49:50.282 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:50.282 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:49:50.282 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:49:50.282 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:49:50.282 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.282 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62833 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 192512~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:50.282 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:49:50.282 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.137444294, 2024-09-21T15:49:50.282 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050387007999999997, 2024-09-21T15:49:50.282 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:50.282 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:50.282 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:50.282 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:50.283 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:50.283 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62833 2024-09-21T15:49:50.283 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.283 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:50.283 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.283 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:50.283 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:49:50.283 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.283 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.283 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.283 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:50.283 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:49:50.283 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.283 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.283 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.284 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:50.284 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:49:50.284 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:49:50.284 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.284 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.284 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:50.284 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142150+0000", 2024-09-21T15:49:50.284 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1999999999999999e-06 2024-09-21T15:49:50.284 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.284 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.284 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:50.284 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142153+0000", 2024-09-21T15:49:50.284 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7410000000000001e-06 2024-09-21T15:49:50.284 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.284 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.285 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:50.285 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142156+0000", 2024-09-21T15:49:50.285 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5739999999999998e-06 2024-09-21T15:49:50.285 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.285 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.285 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:50.285 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.192358+0000", 2024-09-21T15:49:50.285 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050202452000000002 2024-09-21T15:49:50.285 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.285 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.285 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:50.285 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.192435+0000", 2024-09-21T15:49:50.285 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6738999999999998e-05 2024-09-21T15:49:50.285 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.285 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.286 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:50.286 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.192536+0000", 2024-09-21T15:49:50.286 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101508 2024-09-21T15:49:50.286 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.286 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:50.286 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.286 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.286 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.286 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62847 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 200704~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:50.286 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.144163+0000", 2024-09-21T15:49:50.286 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.135430314000001, 2024-09-21T15:49:50.286 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050248293999999999, 2024-09-21T15:49:50.286 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:50.286 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:50.287 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:50.287 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:50.287 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:50.287 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62847 2024-09-21T15:49:50.287 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.287 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:50.287 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.287 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:50.287 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144163+0000", 2024-09-21T15:49:50.287 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.287 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.287 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.287 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:50.287 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144163+0000", 2024-09-21T15:49:50.287 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.287 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.288 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.288 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:50.288 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144163+0000", 2024-09-21T15:49:50.288 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:50.288 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.288 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.288 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:50.288 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144165+0000", 2024-09-21T15:49:50.288 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4439999999999998e-06 2024-09-21T15:49:50.288 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.288 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.288 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:50.288 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144173+0000", 2024-09-21T15:49:50.288 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7100000000000007e-06 2024-09-21T15:49:50.288 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.288 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.289 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:50.289 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144177+0000", 2024-09-21T15:49:50.289 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3000000000000003e-06 2024-09-21T15:49:50.289 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.289 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.289 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:50.289 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.194237+0000", 2024-09-21T15:49:50.289 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050060064000000001 2024-09-21T15:49:50.289 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.289 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.289 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:50.289 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.194317+0000", 2024-09-21T15:49:50.289 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9683000000000003e-05 2024-09-21T15:49:50.289 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.289 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.290 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:50.290 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.194411+0000", 2024-09-21T15:49:50.290 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4734000000000005e-05 2024-09-21T15:49:50.290 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.290 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:50.290 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.290 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.290 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.290 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62849 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 208896~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:50.290 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.144181+0000", 2024-09-21T15:49:50.290 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.135412221999999, 2024-09-21T15:49:50.290 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050460390000000001, 2024-09-21T15:49:50.290 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:50.291 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:50.291 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:50.291 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:50.291 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:50.291 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62849 2024-09-21T15:49:50.291 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.291 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:50.291 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.291 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:50.291 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144181+0000", 2024-09-21T15:49:50.291 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.291 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.291 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.291 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:50.292 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144181+0000", 2024-09-21T15:49:50.292 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.292 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.292 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.292 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:50.292 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144181+0000", 2024-09-21T15:49:50.292 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:50.292 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.292 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.292 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:50.292 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144183+0000", 2024-09-21T15:49:50.292 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3860000000000001e-06 2024-09-21T15:49:50.292 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.293 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.293 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:50.293 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144189+0000", 2024-09-21T15:49:50.293 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0599999999999996e-06 2024-09-21T15:49:50.293 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.293 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.293 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:50.293 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144194+0000", 2024-09-21T15:49:50.293 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4379999999999997e-06 2024-09-21T15:49:50.293 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.293 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.293 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:50.293 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.194457+0000", 2024-09-21T15:49:50.293 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050263317000000002 2024-09-21T15:49:50.293 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.294 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.294 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:50.294 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.194548+0000", 2024-09-21T15:49:50.294 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0785000000000006e-05 2024-09-21T15:49:50.294 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.294 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.294 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:50.294 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.194642+0000", 2024-09-21T15:49:50.294 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3837000000000003e-05 2024-09-21T15:49:50.294 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.294 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:50.294 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.294 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.294 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.295 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62861 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 946176~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:50.295 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.146156+0000", 2024-09-21T15:49:50.295 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.133437398, 2024-09-21T15:49:50.295 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050260860999999997, 2024-09-21T15:49:50.295 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:50.295 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:50.295 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:50.295 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:50.295 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:50.295 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62861 2024-09-21T15:49:50.295 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.295 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:50.295 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.295 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:50.295 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146156+0000", 2024-09-21T15:49:50.296 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.296 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.296 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.296 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:50.296 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146156+0000", 2024-09-21T15:49:50.296 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.296 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.296 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.296 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:50.296 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146156+0000", 2024-09-21T15:49:50.296 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:50.296 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.296 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.297 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:50.297 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146159+0000", 2024-09-21T15:49:50.297 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.14e-06 2024-09-21T15:49:50.297 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.297 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.297 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:50.297 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146167+0000", 2024-09-21T15:49:50.297 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9500000000000001e-06 2024-09-21T15:49:50.297 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.297 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.297 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:50.297 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146172+0000", 2024-09-21T15:49:50.297 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2900000000000002e-06 2024-09-21T15:49:50.297 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.297 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.298 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:50.298 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196247+0000", 2024-09-21T15:49:50.298 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050075195000000003 2024-09-21T15:49:50.298 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.298 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.298 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:50.298 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196324+0000", 2024-09-21T15:49:50.298 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6621999999999999e-05 2024-09-21T15:49:50.298 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.298 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.298 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:50.298 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196417+0000", 2024-09-21T15:49:50.298 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3251000000000006e-05 2024-09-21T15:49:50.298 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.299 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:50.299 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.299 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.299 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.299 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62862 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 954368~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:50.299 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.146176+0000", 2024-09-21T15:49:50.299 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.133417159, 2024-09-21T15:49:50.299 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050442901999999998, 2024-09-21T15:49:50.299 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:50.299 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:50.299 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:50.299 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:50.299 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:50.299 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62862 2024-09-21T15:49:50.300 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.300 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:50.300 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.300 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:50.300 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146176+0000", 2024-09-21T15:49:50.300 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.300 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.300 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.300 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:50.300 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146176+0000", 2024-09-21T15:49:50.300 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.300 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.300 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.301 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:50.301 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146176+0000", 2024-09-21T15:49:50.301 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:50.301 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.301 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.301 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:50.301 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146178+0000", 2024-09-21T15:49:50.301 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4739999999999999e-06 2024-09-21T15:49:50.301 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.301 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.301 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:50.301 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146186+0000", 2024-09-21T15:49:50.301 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4989999999999997e-06 2024-09-21T15:49:50.301 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.302 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.302 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:50.302 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146191+0000", 2024-09-21T15:49:50.302 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7999999999999998e-06 2024-09-21T15:49:50.302 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.302 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.302 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:50.302 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196442+0000", 2024-09-21T15:49:50.302 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050251665000000001 2024-09-21T15:49:50.302 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.302 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.302 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:50.302 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196518+0000", 2024-09-21T15:49:50.303 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5677000000000001e-05 2024-09-21T15:49:50.303 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.303 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.303 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:50.303 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196619+0000", 2024-09-21T15:49:50.303 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101247 2024-09-21T15:49:50.303 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.303 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:50.303 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.303 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.303 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.303 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62863 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 937984~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:50.304 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.146195+0000", 2024-09-21T15:49:50.304 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.133398968, 2024-09-21T15:49:50.304 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050642305999999998, 2024-09-21T15:49:50.304 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:50.304 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:50.304 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:50.304 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:50.304 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:50.304 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62863 2024-09-21T15:49:50.304 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.304 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:50.304 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.304 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:50.305 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146195+0000", 2024-09-21T15:49:50.305 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.305 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.305 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.305 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:50.305 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146195+0000", 2024-09-21T15:49:50.305 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.305 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.305 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.305 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:50.305 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146194+0000", 2024-09-21T15:49:50.305 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:50.305 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.305 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.306 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:50.306 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146196+0000", 2024-09-21T15:49:50.306 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1900000000000002e-06 2024-09-21T15:49:50.306 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.306 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.306 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:50.306 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146202+0000", 2024-09-21T15:49:50.306 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.57e-06 2024-09-21T15:49:50.306 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.306 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.306 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:50.306 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146206+0000", 2024-09-21T15:49:50.306 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5079999999999999e-06 2024-09-21T15:49:50.306 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.306 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.307 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:50.307 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196655+0000", 2024-09-21T15:49:50.307 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050448973000000001 2024-09-21T15:49:50.307 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.307 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.307 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:50.307 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196730+0000", 2024-09-21T15:49:50.307 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4728999999999996e-05 2024-09-21T15:49:50.307 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.307 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.307 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:50.307 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196837+0000", 2024-09-21T15:49:50.308 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000106735 2024-09-21T15:49:50.308 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.308 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:50.308 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.308 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.308 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.308 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62871 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 307200~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:50.308 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148134+0000", 2024-09-21T15:49:50.308 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.131459759, 2024-09-21T15:49:50.308 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050310001, 2024-09-21T15:49:50.308 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:50.308 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:50.308 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:50.308 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:50.308 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:50.309 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62871 2024-09-21T15:49:50.309 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.309 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:50.309 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.309 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:50.309 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148134+0000", 2024-09-21T15:49:50.309 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.309 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.309 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.309 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:50.309 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148134+0000", 2024-09-21T15:49:50.309 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.309 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.309 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.310 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:50.310 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148133+0000", 2024-09-21T15:49:50.310 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:50.310 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.310 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.310 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:50.310 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148135+0000", 2024-09-21T15:49:50.310 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9829999999999999e-06 2024-09-21T15:49:50.310 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.310 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.310 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:50.310 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148141+0000", 2024-09-21T15:49:50.310 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5439999999999998e-06 2024-09-21T15:49:50.310 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.311 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.311 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:50.311 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148145+0000", 2024-09-21T15:49:50.311 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4229999999999998e-06 2024-09-21T15:49:50.311 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.311 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.311 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:50.311 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198294+0000", 2024-09-21T15:49:50.311 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050148558000000003 2024-09-21T15:49:50.311 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.311 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.311 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:50.311 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198372+0000", 2024-09-21T15:49:50.311 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8163000000000004e-05 2024-09-21T15:49:50.311 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.312 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.312 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:50.312 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198444+0000", 2024-09-21T15:49:50.312 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1620999999999997e-05 2024-09-21T15:49:50.312 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.312 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:50.312 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.312 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.312 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.312 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62872 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 290816~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:50.312 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148149+0000", 2024-09-21T15:49:50.312 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.131444291999999, 2024-09-21T15:49:50.312 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.05046991, 2024-09-21T15:49:50.312 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:50.312 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:50.313 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:50.313 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:50.313 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:50.313 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62872 2024-09-21T15:49:50.313 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.313 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:50.313 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.313 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:50.313 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148149+0000", 2024-09-21T15:49:50.313 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.313 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.313 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.313 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:50.313 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148149+0000", 2024-09-21T15:49:50.313 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.313 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.314 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.314 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:50.314 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148149+0000", 2024-09-21T15:49:50.314 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:50.314 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.314 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.314 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:50.314 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148157+0000", 2024-09-21T15:49:50.314 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4749999999999993e-06 2024-09-21T15:49:50.314 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.314 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.314 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:50.314 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148163+0000", 2024-09-21T15:49:50.314 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8250000000000001e-06 2024-09-21T15:49:50.315 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.315 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.315 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:50.315 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148168+0000", 2024-09-21T15:49:50.315 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4730000000000002e-06 2024-09-21T15:49:50.315 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.315 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.315 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:50.315 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198471+0000", 2024-09-21T15:49:50.315 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050302974 2024-09-21T15:49:50.315 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.315 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.315 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:50.315 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198555+0000", 2024-09-21T15:49:50.315 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4783999999999993e-05 2024-09-21T15:49:50.316 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.316 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.316 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:50.316 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198619+0000", 2024-09-21T15:49:50.316 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3751000000000007e-05 2024-09-21T15:49:50.316 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.316 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:50.316 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.316 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.316 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.316 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62873 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 315392~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:50.316 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148172+0000", 2024-09-21T15:49:50.316 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.131422001000001, 2024-09-21T15:49:50.316 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050707998999999997, 2024-09-21T15:49:50.316 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:50.316 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:50.317 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:50.317 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:50.317 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:50.317 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62873 2024-09-21T15:49:50.317 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.317 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:50.317 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.317 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:50.317 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148172+0000", 2024-09-21T15:49:50.317 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.317 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.317 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.317 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:50.317 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148172+0000", 2024-09-21T15:49:50.317 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.318 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.318 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.318 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:50.318 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148171+0000", 2024-09-21T15:49:50.318 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:50.318 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.318 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.318 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:50.318 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148173+0000", 2024-09-21T15:49:50.318 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2340000000000001e-06 2024-09-21T15:49:50.318 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.318 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.318 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:50.318 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148179+0000", 2024-09-21T15:49:50.318 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9340000000000003e-06 2024-09-21T15:49:50.319 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.319 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.319 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:50.319 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148186+0000", 2024-09-21T15:49:50.319 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2349999999999996e-06 2024-09-21T15:49:50.319 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.319 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.319 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:50.319 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198684+0000", 2024-09-21T15:49:50.319 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050498890999999997 2024-09-21T15:49:50.319 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.319 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.319 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:50.319 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198778+0000", 2024-09-21T15:49:50.319 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3134999999999995e-05 2024-09-21T15:49:50.319 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.320 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.320 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:50.320 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198880+0000", 2024-09-21T15:49:50.320 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101906 2024-09-21T15:49:50.320 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.320 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:50.320 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.320 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.320 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.320 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62874 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 323584~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:50.320 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148211+0000", 2024-09-21T15:49:50.320 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.131382241000001, 2024-09-21T15:49:50.320 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050887766000000001, 2024-09-21T15:49:50.320 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:50.320 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:50.320 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:50.321 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:50.321 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:50.321 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62874 2024-09-21T15:49:50.321 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.321 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:50.321 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.321 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:50.321 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148211+0000", 2024-09-21T15:49:50.321 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.321 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.321 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.321 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:50.322 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148211+0000", 2024-09-21T15:49:50.322 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.322 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.322 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.322 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:50.322 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148211+0000", 2024-09-21T15:49:50.322 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:50.322 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.322 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.322 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:50.322 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148213+0000", 2024-09-21T15:49:50.323 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6860000000000002e-06 2024-09-21T15:49:50.323 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.323 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.323 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:50.323 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148220+0000", 2024-09-21T15:49:50.323 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4740000000000002e-06 2024-09-21T15:49:50.323 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.323 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.323 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:50.323 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148226+0000", 2024-09-21T15:49:50.323 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0120000000000002e-06 2024-09-21T15:49:50.323 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.323 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.323 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:50.324 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198940+0000", 2024-09-21T15:49:50.324 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050714412 2024-09-21T15:49:50.324 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.324 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.324 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:50.324 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198995+0000", 2024-09-21T15:49:50.324 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4863e-05 2024-09-21T15:49:50.324 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.324 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.324 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:50.324 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.199099+0000", 2024-09-21T15:49:50.324 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010383299999999999 2024-09-21T15:49:50.324 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.324 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:50.324 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.324 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.325 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.325 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62877 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 995328~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:50.325 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148230+0000", 2024-09-21T15:49:50.325 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.131363381, 2024-09-21T15:49:50.325 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.051108949000000001, 2024-09-21T15:49:50.325 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:50.325 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:50.325 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:50.325 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:50.325 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:50.325 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62877 2024-09-21T15:49:50.325 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.325 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:50.325 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.325 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:50.325 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148230+0000", 2024-09-21T15:49:50.326 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.326 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.326 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.326 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:50.326 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148230+0000", 2024-09-21T15:49:50.326 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.326 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.326 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.326 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:50.326 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148230+0000", 2024-09-21T15:49:50.326 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:50.326 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.326 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.326 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:50.326 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148232+0000", 2024-09-21T15:49:50.327 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1220000000000002e-06 2024-09-21T15:49:50.327 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.327 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.327 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:50.327 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148238+0000", 2024-09-21T15:49:50.327 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.789e-06 2024-09-21T15:49:50.327 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.327 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.327 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:50.327 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148242+0000", 2024-09-21T15:49:50.327 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7369999999999997e-06 2024-09-21T15:49:50.327 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.327 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.327 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:50.327 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.199201+0000", 2024-09-21T15:49:50.328 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050959088 2024-09-21T15:49:50.328 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.328 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.328 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:50.328 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.199285+0000", 2024-09-21T15:49:50.328 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3147999999999997e-05 2024-09-21T15:49:50.328 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.328 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.328 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:50.328 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.199339+0000", 2024-09-21T15:49:50.328 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4484999999999998e-05 2024-09-21T15:49:50.328 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.328 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:50.328 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.328 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.328 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.329 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62886 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 1044480~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:50.329 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150138+0000", 2024-09-21T15:49:50.329 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.129455713, 2024-09-21T15:49:50.329 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050345527000000001, 2024-09-21T15:49:50.329 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:50.329 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:50.329 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:50.329 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:50.329 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:50.329 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62886 2024-09-21T15:49:50.329 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.329 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:50.329 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.329 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:50.329 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150138+0000", 2024-09-21T15:49:50.329 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.330 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.330 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.330 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:50.330 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150138+0000", 2024-09-21T15:49:50.330 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.330 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.330 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.330 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:50.330 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150137+0000", 2024-09-21T15:49:50.330 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:50.330 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.330 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.330 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:50.330 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150140+0000", 2024-09-21T15:49:50.331 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2460000000000002e-06 2024-09-21T15:49:50.331 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.331 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.331 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:50.331 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150145+0000", 2024-09-21T15:49:50.331 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6520000000000003e-06 2024-09-21T15:49:50.331 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.331 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.331 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:50.331 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150150+0000", 2024-09-21T15:49:50.331 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5310000000000003e-06 2024-09-21T15:49:50.331 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.331 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.331 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:50.331 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200356+0000", 2024-09-21T15:49:50.332 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050205759000000003 2024-09-21T15:49:50.332 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.332 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.332 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:50.332 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200415+0000", 2024-09-21T15:49:50.332 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9719999999999997e-05 2024-09-21T15:49:50.332 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.332 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.332 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:50.332 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200483+0000", 2024-09-21T15:49:50.332 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7959000000000004e-05 2024-09-21T15:49:50.332 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.332 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:50.332 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.332 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.332 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.333 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62888 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 331776~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:50.333 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150154+0000", 2024-09-21T15:49:50.333 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.129439787000001, 2024-09-21T15:49:50.333 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050483169000000001, 2024-09-21T15:49:50.333 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:50.333 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:50.333 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:50.333 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:50.333 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:50.333 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62888 2024-09-21T15:49:50.333 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.333 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:50.333 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.333 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:50.333 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150154+0000", 2024-09-21T15:49:50.333 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.334 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.334 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.334 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:50.334 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150154+0000", 2024-09-21T15:49:50.334 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.334 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.334 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.334 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:50.334 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150153+0000", 2024-09-21T15:49:50.334 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:50.334 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.334 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.334 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:50.334 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150155+0000", 2024-09-21T15:49:50.335 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9920000000000002e-06 2024-09-21T15:49:50.335 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.335 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.335 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:50.335 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150161+0000", 2024-09-21T15:49:50.335 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5439999999999998e-06 2024-09-21T15:49:50.335 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.335 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.335 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:50.335 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150165+0000", 2024-09-21T15:49:50.335 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3460000000000004e-06 2024-09-21T15:49:50.335 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.335 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.335 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:50.335 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200502+0000", 2024-09-21T15:49:50.335 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050337150999999997 2024-09-21T15:49:50.336 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.336 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.336 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:50.336 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200575+0000", 2024-09-21T15:49:50.336 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2250999999999997e-05 2024-09-21T15:49:50.336 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.336 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.336 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:50.336 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200637+0000", 2024-09-21T15:49:50.336 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2195999999999999e-05 2024-09-21T15:49:50.336 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.336 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:50.336 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.336 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.336 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.337 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62889 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 339968~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:50.337 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150169+0000", 2024-09-21T15:49:50.337 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.129424287999999, 2024-09-21T15:49:50.337 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050629678999999997, 2024-09-21T15:49:50.337 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:50.337 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:50.337 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:50.337 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:50.337 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:50.337 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62889 2024-09-21T15:49:50.337 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.337 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:50.337 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.337 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:50.337 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150169+0000", 2024-09-21T15:49:50.338 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.338 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.338 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.338 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:50.338 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150169+0000", 2024-09-21T15:49:50.338 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.338 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.338 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.338 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:50.338 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150169+0000", 2024-09-21T15:49:50.338 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:50.338 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.338 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.338 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:50.339 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150171+0000", 2024-09-21T15:49:50.339 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3910000000000001e-06 2024-09-21T15:49:50.339 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.339 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.339 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:50.339 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150177+0000", 2024-09-21T15:49:50.339 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4079999999999997e-06 2024-09-21T15:49:50.339 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.339 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.339 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:50.339 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150182+0000", 2024-09-21T15:49:50.339 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3890000000000004e-06 2024-09-21T15:49:50.339 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.339 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.339 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:50.339 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200664+0000", 2024-09-21T15:49:50.340 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050482212999999998 2024-09-21T15:49:50.340 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.340 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.340 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:50.340 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200738+0000", 2024-09-21T15:49:50.340 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4114000000000002e-05 2024-09-21T15:49:50.340 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.340 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.340 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:50.340 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200799+0000", 2024-09-21T15:49:50.340 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0503e-05 2024-09-21T15:49:50.340 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.340 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:50.340 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.340 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.341 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.341 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62891 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 348160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:50.341 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150186+0000", 2024-09-21T15:49:50.341 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.129407146, 2024-09-21T15:49:50.341 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050794552, 2024-09-21T15:49:50.341 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:50.341 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:50.341 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:50.341 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:50.341 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:50.341 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62891 2024-09-21T15:49:50.341 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.341 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:50.341 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.341 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:50.342 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150186+0000", 2024-09-21T15:49:50.342 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.342 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.342 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.342 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:50.342 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150186+0000", 2024-09-21T15:49:50.342 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.342 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.342 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.342 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:50.342 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150186+0000", 2024-09-21T15:49:50.342 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:50.342 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.342 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.342 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:50.343 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150193+0000", 2024-09-21T15:49:50.343 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6830000000000002e-06 2024-09-21T15:49:50.343 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.343 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.343 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:50.343 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150199+0000", 2024-09-21T15:49:50.343 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9859999999999999e-06 2024-09-21T15:49:50.343 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.343 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.343 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:50.343 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150203+0000", 2024-09-21T15:49:50.343 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5920000000000002e-06 2024-09-21T15:49:50.343 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.343 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.344 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:50.344 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200835+0000", 2024-09-21T15:49:50.344 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050631783 2024-09-21T15:49:50.344 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.344 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.344 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:50.344 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200894+0000", 2024-09-21T15:49:50.344 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8560999999999997e-05 2024-09-21T15:49:50.344 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.344 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.344 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:50.344 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200981+0000", 2024-09-21T15:49:50.344 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7305999999999994e-05 2024-09-21T15:49:50.344 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.344 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:50.345 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.345 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.345 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.345 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62892 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 356352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:50.345 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150208+0000", 2024-09-21T15:49:50.345 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.129385758, 2024-09-21T15:49:50.345 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050951769000000001, 2024-09-21T15:49:50.345 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:50.345 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:50.345 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:50.345 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:50.345 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:50.345 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62892 2024-09-21T15:49:50.345 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.345 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:50.345 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.346 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:50.346 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150208+0000", 2024-09-21T15:49:50.346 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.346 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.346 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.346 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:50.346 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150208+0000", 2024-09-21T15:49:50.346 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.346 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.346 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.346 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:50.346 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150207+0000", 2024-09-21T15:49:50.346 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:50.346 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.346 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.347 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:50.347 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150210+0000", 2024-09-21T15:49:50.347 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.238e-06 2024-09-21T15:49:50.347 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.347 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.347 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:50.347 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150215+0000", 2024-09-21T15:49:50.347 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6350000000000003e-06 2024-09-21T15:49:50.347 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.347 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.347 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:50.347 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150220+0000", 2024-09-21T15:49:50.347 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.544e-06 2024-09-21T15:49:50.347 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.347 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.347 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:50.348 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201014+0000", 2024-09-21T15:49:50.348 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050793932999999999 2024-09-21T15:49:50.348 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.348 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.348 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:50.348 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201092+0000", 2024-09-21T15:49:50.348 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7961999999999997e-05 2024-09-21T15:49:50.348 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.348 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.348 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:50.348 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201160+0000", 2024-09-21T15:49:50.348 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7836000000000006e-05 2024-09-21T15:49:50.348 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.349 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:50.349 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.349 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.349 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.349 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62894 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 364544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:50.349 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150820+0000", 2024-09-21T15:49:50.349 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.128773558000001, 2024-09-21T15:49:50.349 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050489196, 2024-09-21T15:49:50.349 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:50.349 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:50.349 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:50.349 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:50.349 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:50.350 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62894 2024-09-21T15:49:50.350 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.350 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:50.350 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.350 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:50.350 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150820+0000", 2024-09-21T15:49:50.350 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.350 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.350 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.350 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:50.350 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150820+0000", 2024-09-21T15:49:50.350 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.350 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.351 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.351 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:50.351 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150223+0000", 2024-09-21T15:49:50.351 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9994035 2024-09-21T15:49:50.351 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.351 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.351 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:50.351 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150825+0000", 2024-09-21T15:49:50.351 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00060137099999999996 2024-09-21T15:49:50.351 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.351 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.351 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:50.351 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150837+0000", 2024-09-21T15:49:50.351 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2503999999999999e-05 2024-09-21T15:49:50.352 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.352 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.352 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:50.352 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150844+0000", 2024-09-21T15:49:50.352 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6900000000000003e-06 2024-09-21T15:49:50.352 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.352 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.352 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:50.352 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201187+0000", 2024-09-21T15:49:50.352 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050342973999999999 2024-09-21T15:49:50.352 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.352 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.352 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:50.352 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201250+0000", 2024-09-21T15:49:50.353 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3274000000000004e-05 2024-09-21T15:49:50.353 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.353 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.353 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:50.353 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201309+0000", 2024-09-21T15:49:50.353 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9058000000000003e-05 2024-09-21T15:49:50.353 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.353 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:50.353 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.353 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.353 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.353 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62895 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 380928~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:50.353 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150851+0000", 2024-09-21T15:49:50.353 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.128742962, 2024-09-21T15:49:50.354 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050611097000000001, 2024-09-21T15:49:50.354 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:50.354 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:50.354 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:50.354 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:50.354 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:50.354 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62895 2024-09-21T15:49:50.354 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.354 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:50.354 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.354 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:50.354 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150851+0000", 2024-09-21T15:49:50.354 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.354 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.355 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.355 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:50.355 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150851+0000", 2024-09-21T15:49:50.355 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.355 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.355 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.355 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:50.355 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150850+0000", 2024-09-21T15:49:50.355 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:50.355 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.355 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.355 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:50.355 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150853+0000", 2024-09-21T15:49:50.355 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1039999999999998e-06 2024-09-21T15:49:50.355 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.356 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.356 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:50.356 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150860+0000", 2024-09-21T15:49:50.356 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3180000000000001e-06 2024-09-21T15:49:50.356 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.356 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.356 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:50.356 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150866+0000", 2024-09-21T15:49:50.356 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1599999999999997e-06 2024-09-21T15:49:50.356 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.356 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.356 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:50.356 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201332+0000", 2024-09-21T15:49:50.356 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050466100999999999 2024-09-21T15:49:50.356 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.356 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.357 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:50.357 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201392+0000", 2024-09-21T15:49:50.357 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0053999999999998e-05 2024-09-21T15:49:50.357 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.357 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.357 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:50.357 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201462+0000", 2024-09-21T15:49:50.357 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9861000000000001e-05 2024-09-21T15:49:50.357 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.357 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:50.357 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.357 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.357 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.357 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62896 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 372736~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:50.357 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150870+0000", 2024-09-21T15:49:50.357 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.128723182, 2024-09-21T15:49:50.358 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050753292999999998, 2024-09-21T15:49:50.358 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:50.358 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:50.358 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:50.358 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:50.358 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:50.358 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62896 2024-09-21T15:49:50.358 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.358 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:50.358 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.358 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:50.358 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150870+0000", 2024-09-21T15:49:50.358 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.358 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.358 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.359 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:50.359 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150870+0000", 2024-09-21T15:49:50.359 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.359 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.359 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.359 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:50.359 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150870+0000", 2024-09-21T15:49:50.359 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:50.359 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.359 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.359 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:50.359 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150872+0000", 2024-09-21T15:49:50.359 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.644e-06 2024-09-21T15:49:50.359 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.359 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.360 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:50.360 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150879+0000", 2024-09-21T15:49:50.360 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3509999999999998e-06 2024-09-21T15:49:50.360 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.360 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.360 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:50.360 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150884+0000", 2024-09-21T15:49:50.360 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9150000000000004e-06 2024-09-21T15:49:50.360 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.360 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.360 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:50.360 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201487+0000", 2024-09-21T15:49:50.360 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050603276000000003 2024-09-21T15:49:50.360 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.360 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.361 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:50.361 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201557+0000", 2024-09-21T15:49:50.361 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9847999999999999e-05 2024-09-21T15:49:50.361 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.361 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.361 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:50.361 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201624+0000", 2024-09-21T15:49:50.361 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6730999999999995e-05 2024-09-21T15:49:50.361 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.361 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:50.361 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.361 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.361 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.361 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62897 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 389120~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:50.361 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.151044+0000", 2024-09-21T15:49:50.361 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.128549142000001, 2024-09-21T15:49:50.362 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050720531999999999, 2024-09-21T15:49:50.362 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:50.362 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:50.362 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:50.362 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:50.362 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:50.362 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62897 2024-09-21T15:49:50.362 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.362 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:50.362 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.362 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:50.362 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151044+0000", 2024-09-21T15:49:50.362 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.362 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.362 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.363 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:50.363 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151044+0000", 2024-09-21T15:49:50.363 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.363 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.363 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.363 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:50.363 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150888+0000", 2024-09-21T15:49:50.363 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9998431 2024-09-21T15:49:50.363 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.363 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.363 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:50.363 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151048+0000", 2024-09-21T15:49:50.363 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00016077299999999999 2024-09-21T15:49:50.363 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.363 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.363 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:50.363 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151058+0000", 2024-09-21T15:49:50.364 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.465e-06 2024-09-21T15:49:50.364 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.364 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.364 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:50.364 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151064+0000", 2024-09-21T15:49:50.364 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3829999999999997e-06 2024-09-21T15:49:50.364 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.364 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.364 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:50.364 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201648+0000", 2024-09-21T15:49:50.364 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050583597000000001 2024-09-21T15:49:50.364 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.364 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.364 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:50.364 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201711+0000", 2024-09-21T15:49:50.365 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3616999999999998e-05 2024-09-21T15:49:50.365 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.365 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.365 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:50.365 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201765+0000", 2024-09-21T15:49:50.365 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3464000000000002e-05 2024-09-21T15:49:50.365 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.365 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:50.365 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.365 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.365 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:50.365 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:50.367 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:49:50.368 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:50.368 INFO:teuthology.orchestra.run.smithi089.stdout:3445 trunc from 0x1db3f4f to 0xed75b0 2024-09-21T15:49:50.368 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:50.368 INFO:teuthology.orchestra.run.smithi089.stdout:3448 writesame 0x52784bd thru 0x5281420 data_size 0x1c(0x8f64 bytes) 2024-09-21T15:49:50.368 INFO:teuthology.orchestra.run.smithi089.stdout:3449 trunc from 0x5281421 to 0xc2da73b 2024-09-21T15:49:50.368 INFO:teuthology.orchestra.run.smithi089.stdout:3451 punch from 0x477bcfe to 0x4785b65, (0x9e67 bytes) 2024-09-21T15:49:50.368 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:50.368 INFO:teuthology.orchestra.run.smithi089.stdout:3453 write 0xc06ed0a thru 0xc07b922 (0xcc19 bytes) 2024-09-21T15:49:50.369 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:50.365+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:50.370 INFO:teuthology.orchestra.run.smithi089.stdout:3454 read 0x14dd50a thru 0x14e8d87 (0xb87e bytes) 2024-09-21T15:49:50.370 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:50.380 INFO:teuthology.orchestra.run.smithi089.stdout:3456 write 0xd7c29ef thru 0xd7c84b5 (0x5ac7 bytes) 2024-09-21T15:49:50.384 INFO:teuthology.orchestra.run.smithi089.stdout:3457 punch from 0x4f6643b to 0x4f6c90b, (0x64d0 bytes) 2024-09-21T15:49:50.384 INFO:teuthology.orchestra.run.smithi089.stdout:3458 write 0x1d7c955 thru 0x1d8290a (0x5fb6 bytes) 2024-09-21T15:49:50.394 INFO:teuthology.orchestra.run.smithi089.stdout:3460 punch from 0xaa65997 to 0xaa66453, (0xabc bytes) 2024-09-21T15:49:50.394 INFO:teuthology.orchestra.run.smithi089.stdout:3462 write 0x5d6de23 thru 0x5d7606f (0x824d bytes) 2024-09-21T15:49:50.401 INFO:teuthology.orchestra.run.smithi089.stdout:3463 write 0x32996d thru 0x32d297 (0x392b bytes) 2024-09-21T15:49:50.407 INFO:teuthology.orchestra.run.smithi089.stdout:3464 read 0x97af37f thru 0x97b889d (0x951f bytes) 2024-09-21T15:49:50.412 INFO:teuthology.orchestra.run.smithi089.stdout:3465 trunc from 0xd7c84b6 to 0x9ff80db 2024-09-21T15:49:50.424 INFO:teuthology.orchestra.run.smithi089.stdout:3466 read 0x8693ae4 thru 0x86a109f (0xd5bc bytes) 2024-09-21T15:49:50.425 INFO:teuthology.orchestra.run.smithi089.stdout:3467 read 0x2d9b0f6 thru 0x2da5523 (0xa42e bytes) 2024-09-21T15:49:50.425 INFO:teuthology.orchestra.run.smithi089.stdout:3468 read 0x9f8226c thru 0x9f8d52d (0xb2c2 bytes) 2024-09-21T15:49:50.425 INFO:teuthology.orchestra.run.smithi089.stdout:3471 read 0x1499bf9 thru 0x14a4bf1 (0xaff9 bytes) 2024-09-21T15:49:50.433 INFO:teuthology.orchestra.run.smithi089.stdout:3472 write 0xc28f9a2 thru 0xc29ec6e (0xf2cd bytes) 2024-09-21T15:49:50.437 INFO:teuthology.orchestra.run.smithi089.stdout:3473 read 0x8af65c0 thru 0x8afbbe4 (0x5625 bytes) 2024-09-21T15:49:50.437 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:50.437 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:50.437 INFO:teuthology.orchestra.run.smithi089.stdout:3478 read 0x2378eb9 thru 0x23889ea (0xfb32 bytes) 2024-09-21T15:49:50.437 INFO:teuthology.orchestra.run.smithi089.stdout:3479 writesame 0x1ef5be9 thru 0x1efb14a data_size 0x3(0x5562 bytes) 2024-09-21T15:49:50.444 INFO:teuthology.orchestra.run.smithi089.stdout:3482 read 0x245d336 thru 0x2464afe (0x77c9 bytes) 2024-09-21T15:49:50.444 INFO:teuthology.orchestra.run.smithi089.stdout:3483 write 0x476cc90 thru 0x477a7ef (0xdb60 bytes) 2024-09-21T15:49:50.448 INFO:teuthology.orchestra.run.smithi089.stdout:3485 read 0x76545fb thru 0x7657930 (0x3336 bytes) 2024-09-21T15:49:50.448 INFO:teuthology.orchestra.run.smithi089.stdout:3486 punch from 0x3d7bd9c to 0x3d8b6d1, (0xf935 bytes) 2024-09-21T15:49:50.448 INFO:teuthology.orchestra.run.smithi089.stdout:3487 write 0xc08e51e thru 0xc0995c6 (0xb0a9 bytes) 2024-09-21T15:49:50.452 INFO:teuthology.orchestra.run.smithi089.stdout:3488 write 0xbf5d9e3 thru 0xbf62879 (0x4e97 bytes) 2024-09-21T15:49:50.462 INFO:teuthology.orchestra.run.smithi089.stdout:3489 punch from 0x10fcaca to 0x1104f81, (0x84b7 bytes) 2024-09-21T15:49:50.462 INFO:teuthology.orchestra.run.smithi089.stdout:3491 punch from 0x60d89ca to 0x60e3da2, (0xb3d8 bytes) 2024-09-21T15:49:50.467 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:50.466+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:50.468 INFO:teuthology.orchestra.run.smithi089.stdout:3492 write 0xd2f4e28 thru 0xd2f9bcb (0x4da4 bytes) 2024-09-21T15:49:50.471 INFO:teuthology.orchestra.run.smithi089.stdout:3493 read 0x8070f7e thru 0x8079e11 (0x8e94 bytes) 2024-09-21T15:49:50.473 INFO:teuthology.orchestra.run.smithi089.stdout:3495 trunc from 0xd2f9bcc to 0xe928e4c 2024-09-21T15:49:50.481 INFO:teuthology.orchestra.run.smithi089.stdout:3496 write 0x36192d5 thru 0x3627f1b (0xec47 bytes) 2024-09-21T15:49:50.481 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:50.481 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:49:50.481 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:49:50.481 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:50.487 INFO:teuthology.orchestra.run.smithi089.stdout:3497 write 0x3e0e1f4 thru 0x3e139ee (0x57fb bytes) 2024-09-21T15:49:50.490 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:49:50.491 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:50.491 INFO:teuthology.orchestra.run.smithi089.stdout:3499 read 0x2a93bc4 thru 0x2a9e033 (0xa470 bytes) 2024-09-21T15:49:50.491 INFO:teuthology.orchestra.run.smithi089.stdout:3502 read 0x3330b2d thru 0x3332665 (0x1b39 bytes) 2024-09-21T15:49:50.491 INFO:teuthology.orchestra.run.smithi089.stdout:3504 read 0x9143ba5 thru 0x9146f68 (0x33c4 bytes) 2024-09-21T15:49:50.491 INFO:teuthology.orchestra.run.smithi089.stdout:3505 read 0x448e92c thru 0x44953bc (0x6a91 bytes) 2024-09-21T15:49:50.491 INFO:teuthology.orchestra.run.smithi089.stdout:3507 read 0x32602cb thru 0x3270015 (0xfd4b bytes) 2024-09-21T15:49:50.499 INFO:teuthology.orchestra.run.smithi089.stdout:3508 trunc from 0xe928e4c to 0x653e827 2024-09-21T15:49:50.516 INFO:teuthology.orchestra.run.smithi089.stdout:3510 trunc from 0x653e827 to 0x245b041 2024-09-21T15:49:50.534 INFO:teuthology.orchestra.run.smithi089.stdout:3513 write 0xab1297 thru 0xac0710 (0xf47a bytes) 2024-09-21T15:49:50.538 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:50.539 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:50.539 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:50.539 INFO:teuthology.orchestra.run.smithi089.stdout:3517 punch from 0x1b1ad02 to 0x1b1b61f, (0x91d bytes) 2024-09-21T15:49:50.539 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:50.555 INFO:teuthology.orchestra.run.smithi089.stdout:3519 write 0xc086927 thru 0xc08fa98 (0x9172 bytes) 2024-09-21T15:49:50.558 INFO:teuthology.orchestra.run.smithi089.stdout:3520 read 0xb9d4de8 thru 0xb9e2909 (0xdb22 bytes) 2024-09-21T15:49:50.565 INFO:teuthology.orchestra.run.smithi089.stdout:3522 write 0xd40beb9 thru 0xd40e341 (0x2489 bytes) 2024-09-21T15:49:50.567 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:50.566+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:50.571 INFO:teuthology.orchestra.run.smithi089.stdout:3523 trunc from 0xd40e342 to 0xb9e72f7 2024-09-21T15:49:50.591 INFO:teuthology.orchestra.run.smithi089.stdout:3524 write 0xd425e30 thru 0xd43083f (0xaa10 bytes) 2024-09-21T15:49:50.595 INFO:teuthology.orchestra.run.smithi089.stdout:3525 read 0xc65918f thru 0xc65c6ca (0x353c bytes) 2024-09-21T15:49:50.595 INFO:teuthology.orchestra.run.smithi089.stdout:3531 punch from 0x9a22ef2 to 0x9a23d78, (0xe86 bytes) 2024-09-21T15:49:50.595 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:50.595 INFO:teuthology.orchestra.run.smithi089.stdout:3536 write 0x2f497d5 thru 0x2f5621e (0xca4a bytes) 2024-09-21T15:49:50.597 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:50.597 INFO:teuthology.orchestra.run.smithi089.stdout:3541 write 0x11a6a28 thru 0x11b210e (0xb6e7 bytes) 2024-09-21T15:49:50.600 INFO:teuthology.orchestra.run.smithi089.stdout:3542 read 0x8043c74 thru 0x804d906 (0x9c93 bytes) 2024-09-21T15:49:50.600 INFO:teuthology.orchestra.run.smithi089.stdout:3545 write 0x7e85870 thru 0x7e8d4a0 (0x7c31 bytes) 2024-09-21T15:49:50.605 INFO:teuthology.orchestra.run.smithi089.stdout:3546 read 0x60868f1 thru 0x60955e5 (0xecf5 bytes) 2024-09-21T15:49:50.605 INFO:teuthology.orchestra.run.smithi089.stdout:3547 write 0x65f5d87 thru 0x66016b8 (0xb932 bytes) 2024-09-21T15:49:50.612 INFO:teuthology.orchestra.run.smithi089.stdout:3549 read 0x7f52b29 thru 0x7f6119b (0xe673 bytes) 2024-09-21T15:49:50.612 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:50.612 INFO:teuthology.orchestra.run.smithi089.stdout:3551 writesame 0x8a5e37b thru 0x8a6a6ce data_size 0x36(0xc354 bytes) 2024-09-21T15:49:50.615 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:50.615 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:49:50.615 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:49:50.615 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:49:50.615 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:50.625 INFO:teuthology.orchestra.run.smithi089.stdout:3552 trunc from 0xd430840 to 0x991ac01 2024-09-21T15:49:50.628 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:49:50.636 INFO:teuthology.orchestra.run.smithi089.stdout:3554 punch from 0x1cc6f31 to 0x1ccd9dd, (0x6aac bytes) 2024-09-21T15:49:50.641 INFO:teuthology.orchestra.run.smithi089.stdout:3555 trunc from 0x991ac01 to 0x57ec626 2024-09-21T15:49:50.653 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:50.653 INFO:teuthology.orchestra.run.smithi089.stdout:3557 read 0x335f30d thru 0x336e5ec (0xf2e0 bytes) 2024-09-21T15:49:50.653 INFO:teuthology.orchestra.run.smithi089.stdout:3558 read 0x360ac1c thru 0x3618ce2 (0xe0c7 bytes) 2024-09-21T15:49:50.653 INFO:teuthology.orchestra.run.smithi089.stdout:3559 punch from 0x1f097ca to 0x1f111ca, (0x7a00 bytes) 2024-09-21T15:49:50.654 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:50.654 INFO:teuthology.orchestra.run.smithi089.stdout:3563 punch from 0x332d597 to 0x3338e50, (0xb8b9 bytes) 2024-09-21T15:49:50.654 INFO:teuthology.orchestra.run.smithi089.stdout:3564 write 0x4c2061e thru 0x4c2a6ed (0xa0d0 bytes) 2024-09-21T15:49:50.656 INFO:teuthology.orchestra.run.smithi089.stdout:3565 read 0x13dd70b thru 0x13df434 (0x1d2a bytes) 2024-09-21T15:49:50.656 INFO:teuthology.orchestra.run.smithi089.stdout:3566 read 0xce6f7b thru 0xce9315 (0x239b bytes) 2024-09-21T15:49:50.668 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:50.667+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:50.672 INFO:teuthology.orchestra.run.smithi089.stdout:3568 write 0xe248cd9 thru 0xe253878 (0xaba0 bytes) 2024-09-21T15:49:50.675 INFO:teuthology.orchestra.run.smithi089.stdout:3569 read 0x3e882d2 thru 0x3e898ac (0x15db bytes) 2024-09-21T15:49:50.675 INFO:teuthology.orchestra.run.smithi089.stdout:3570 writesame 0xcb8cd0 thru 0xcbd945 data_size 0x1(0x4c76 bytes) 2024-09-21T15:49:50.686 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:50.690 INFO:teuthology.orchestra.run.smithi089.stdout:3572 trunc from 0xe253879 to 0xadf7899 2024-09-21T15:49:50.712 INFO:teuthology.orchestra.run.smithi089.stdout:3574 write 0xdd4c7fa thru 0xdd4d9bc (0x11c3 bytes) 2024-09-21T15:49:50.715 INFO:teuthology.orchestra.run.smithi089.stdout:3576 write 0xab01a1e thru 0xab02b0b (0x10ee bytes) 2024-09-21T15:49:50.725 INFO:teuthology.orchestra.run.smithi089.stdout:3577 write 0xdebf5ad thru 0xdec382a (0x427e bytes) 2024-09-21T15:49:50.736 INFO:teuthology.orchestra.run.smithi089.stdout:3578 read 0x4fae7ba thru 0x4fb9a86 (0xb2cd bytes) 2024-09-21T15:49:50.737 INFO:teuthology.orchestra.run.smithi089.stdout:3579 write 0x8bed0ac thru 0x8bee5b5 (0x150a bytes) 2024-09-21T15:49:50.742 INFO:teuthology.orchestra.run.smithi089.stdout:3582 read 0x3c8f6f4 thru 0x3c9b8e8 (0xc1f5 bytes) 2024-09-21T15:49:50.743 INFO:teuthology.orchestra.run.smithi089.stdout:3583 write 0xc1373fb thru 0xc145b3e (0xe744 bytes) 2024-09-21T15:49:50.749 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:50.749 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:49:50.749 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:49:50.749 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:49:50.749 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.749 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:984 3.0 e27/22 3:025e53e7:::rbd_data.1133f3dfbd6b.0000000000000008:head v 27'21, mlcod=27'19)", 2024-09-21T15:49:50.749 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.619393+0000", 2024-09-21T15:49:50.749 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 38.128147865999999, 2024-09-21T15:49:50.749 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003401837, 2024-09-21T15:49:50.749 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:50.750 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:50.750 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:50.750 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.750 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:50.750 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619393+0000", 2024-09-21T15:49:50.750 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.750 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.750 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.750 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:50.750 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619393+0000", 2024-09-21T15:49:50.750 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.750 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.750 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.750 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:50.751 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619392+0000", 2024-09-21T15:49:50.751 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:50.751 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.751 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.751 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:50.751 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619439+0000", 2024-09-21T15:49:50.751 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7170999999999999e-05 2024-09-21T15:49:50.751 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.751 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.751 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:50.751 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619452+0000", 2024-09-21T15:49:50.751 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2986999999999999e-05 2024-09-21T15:49:50.751 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.752 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.752 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:50.752 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619459+0000", 2024-09-21T15:49:50.752 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6109999999999999e-06 2024-09-21T15:49:50.752 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.752 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.752 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:50.752 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619518+0000", 2024-09-21T15:49:50.752 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9117999999999999e-05 2024-09-21T15:49:50.752 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.752 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.752 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:50.752 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619532+0000", 2024-09-21T15:49:50.752 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4622e-05 2024-09-21T15:49:50.753 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.753 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.753 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:50.753 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.622760+0000", 2024-09-21T15:49:50.753 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032278839999999999 2024-09-21T15:49:50.753 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.753 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.753 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:50.753 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.622794+0000", 2024-09-21T15:49:50.753 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4144999999999999e-05 2024-09-21T15:49:50.753 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.753 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:50.753 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.753 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.754 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.754 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:4957 3.9 e29/22 3:95c5ed95:::rbd_data.11335fa2fe69.0000000000000018:head v 29'4, mlcod=26'2)", 2024-09-21T15:49:50.754 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:14.453951+0000", 2024-09-21T15:49:50.754 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 36.293589433999998, 2024-09-21T15:49:50.754 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0043804559999999996, 2024-09-21T15:49:50.754 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:50.754 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:50.754 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:50.754 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.754 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:50.754 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.453951+0000", 2024-09-21T15:49:50.754 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.754 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.754 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.755 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:50.755 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.453951+0000", 2024-09-21T15:49:50.755 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.755 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.755 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.755 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:50.755 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.453950+0000", 2024-09-21T15:49:50.755 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:50.755 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.755 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.755 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:50.755 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454495+0000", 2024-09-21T15:49:50.755 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00054437799999999998 2024-09-21T15:49:50.755 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.755 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.756 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:50.756 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454508+0000", 2024-09-21T15:49:50.756 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3475000000000001e-05 2024-09-21T15:49:50.756 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.756 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.756 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:50.756 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454515+0000", 2024-09-21T15:49:50.756 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8990000000000004e-06 2024-09-21T15:49:50.756 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.756 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.756 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:50.756 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454581+0000", 2024-09-21T15:49:50.756 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5987000000000004e-05 2024-09-21T15:49:50.756 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.757 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.757 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:50.757 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454609+0000", 2024-09-21T15:49:50.757 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7466000000000001e-05 2024-09-21T15:49:50.757 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.757 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.757 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:50.757 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.458303+0000", 2024-09-21T15:49:50.757 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003693979 2024-09-21T15:49:50.757 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.757 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.757 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:50.757 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.458331+0000", 2024-09-21T15:49:50.757 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8932999999999999e-05 2024-09-21T15:49:50.758 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.758 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:50.758 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.758 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.758 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.758 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13420 3.0 e31/22 3:038b8840:::rbd_data.1133dc5adfdf.0000000000000035:head v 31'49, mlcod=31'47)", 2024-09-21T15:49:50.758 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.209233+0000", 2024-09-21T15:49:50.758 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 33.538307818, 2024-09-21T15:49:50.758 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0042624960000000002, 2024-09-21T15:49:50.758 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:50.758 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:50.758 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:50.758 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.758 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:50.759 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209233+0000", 2024-09-21T15:49:50.759 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.759 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.759 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.759 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:50.759 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209233+0000", 2024-09-21T15:49:50.759 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.759 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.759 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.759 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:50.759 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209232+0000", 2024-09-21T15:49:50.759 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:50.759 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.759 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.759 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:50.759 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209251+0000", 2024-09-21T15:49:50.760 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9304e-05 2024-09-21T15:49:50.760 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.760 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.760 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:50.760 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209265+0000", 2024-09-21T15:49:50.760 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3532e-05 2024-09-21T15:49:50.760 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.760 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.760 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:50.760 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209272+0000", 2024-09-21T15:49:50.760 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.289e-06 2024-09-21T15:49:50.760 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.760 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.760 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:50.760 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209321+0000", 2024-09-21T15:49:50.760 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8773999999999998e-05 2024-09-21T15:49:50.761 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.761 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.761 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:50.761 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209341+0000", 2024-09-21T15:49:50.761 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0387999999999999e-05 2024-09-21T15:49:50.761 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.761 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.761 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:50.761 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.213464+0000", 2024-09-21T15:49:50.761 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0041232450000000002 2024-09-21T15:49:50.761 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.761 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.761 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:50.761 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.213495+0000", 2024-09-21T15:49:50.762 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0793999999999997e-05 2024-09-21T15:49:50.762 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.762 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:50.762 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.762 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.762 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.762 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13445 3.0 e31/22 3:038b8840:::rbd_data.1133dc5adfdf.0000000000000035:head v 31'50, mlcod=31'48)", 2024-09-21T15:49:50.762 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.232497+0000", 2024-09-21T15:49:50.762 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 33.515043806000001, 2024-09-21T15:49:50.762 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0040493559999999996, 2024-09-21T15:49:50.762 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:50.762 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:50.762 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:50.762 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.762 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:50.763 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232497+0000", 2024-09-21T15:49:50.763 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.763 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.763 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.763 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:50.763 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232497+0000", 2024-09-21T15:49:50.763 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.763 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.763 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.763 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:50.763 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232496+0000", 2024-09-21T15:49:50.763 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:50.763 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.763 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.764 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:50.764 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232503+0000", 2024-09-21T15:49:50.764 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7320000000000001e-06 2024-09-21T15:49:50.764 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.764 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.764 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:50.764 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232512+0000", 2024-09-21T15:49:50.764 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9259999999999995e-06 2024-09-21T15:49:50.764 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.764 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.764 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:50.764 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232520+0000", 2024-09-21T15:49:50.764 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3699999999999997e-06 2024-09-21T15:49:50.764 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.764 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.764 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:50.765 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232564+0000", 2024-09-21T15:49:50.765 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4128999999999997e-05 2024-09-21T15:49:50.765 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.765 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.765 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:50.765 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232585+0000", 2024-09-21T15:49:50.765 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0704999999999999e-05 2024-09-21T15:49:50.765 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.765 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.765 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:50.765 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.236507+0000", 2024-09-21T15:49:50.765 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0039218730000000002 2024-09-21T15:49:50.765 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.765 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.766 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:50.766 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.236546+0000", 2024-09-21T15:49:50.766 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9399e-05 2024-09-21T15:49:50.766 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.766 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:50.766 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.766 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.766 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.766 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13452 3.9 e31/22 3:93f5396a:::rbd_data.1133dc5adfdf.0000000000000078:head v 31'50, mlcod=31'47)", 2024-09-21T15:49:50.766 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.237263+0000", 2024-09-21T15:49:50.766 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 33.510277100000003, 2024-09-21T15:49:50.766 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033768180000000002, 2024-09-21T15:49:50.766 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:50.766 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:50.767 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:50.767 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.767 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:50.767 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237263+0000", 2024-09-21T15:49:50.767 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.767 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.767 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.767 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:50.767 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237263+0000", 2024-09-21T15:49:50.767 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.767 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.767 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.767 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:50.767 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237263+0000", 2024-09-21T15:49:50.767 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:50.768 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.768 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.768 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:50.768 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237269+0000", 2024-09-21T15:49:50.768 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.472e-06 2024-09-21T15:49:50.768 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.768 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.768 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:50.768 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237282+0000", 2024-09-21T15:49:50.768 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2788e-05 2024-09-21T15:49:50.768 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.768 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.768 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:50.768 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237288+0000", 2024-09-21T15:49:50.768 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4899999999999997e-06 2024-09-21T15:49:50.769 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.769 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.769 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:50.769 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237331+0000", 2024-09-21T15:49:50.769 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2425000000000001e-05 2024-09-21T15:49:50.769 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.769 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.769 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:50.769 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237349+0000", 2024-09-21T15:49:50.769 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8405000000000001e-05 2024-09-21T15:49:50.769 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.769 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.769 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:50.769 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.240610+0000", 2024-09-21T15:49:50.770 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032605379999999999 2024-09-21T15:49:50.770 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.770 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.770 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:50.770 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.240640+0000", 2024-09-21T15:49:50.770 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0448000000000001e-05 2024-09-21T15:49:50.770 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.770 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:50.770 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.770 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.770 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.770 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13832 3.0 e31/22 3:0792efd2:::rbd_data.1133dc5adfdf.0000000000000090:head v 31'58, mlcod=31'56)", 2024-09-21T15:49:50.770 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.527530+0000", 2024-09-21T15:49:50.770 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 33.220010279999997, 2024-09-21T15:49:50.771 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035256530000000001, 2024-09-21T15:49:50.771 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:50.771 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:50.771 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:50.771 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.771 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:50.771 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527530+0000", 2024-09-21T15:49:50.771 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.771 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.771 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.771 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:50.771 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527530+0000", 2024-09-21T15:49:50.771 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.771 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.771 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.772 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:50.772 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527529+0000", 2024-09-21T15:49:50.772 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:50.772 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.772 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.772 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:50.772 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527550+0000", 2024-09-21T15:49:50.772 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1387999999999999e-05 2024-09-21T15:49:50.772 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.772 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.772 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:50.772 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527568+0000", 2024-09-21T15:49:50.772 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8068e-05 2024-09-21T15:49:50.772 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.773 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.773 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:50.773 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527577+0000", 2024-09-21T15:49:50.773 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9919999999999996e-06 2024-09-21T15:49:50.773 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.773 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.773 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:50.773 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527626+0000", 2024-09-21T15:49:50.773 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8597000000000002e-05 2024-09-21T15:49:50.773 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.773 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.773 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:50.773 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527652+0000", 2024-09-21T15:49:50.773 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5993e-05 2024-09-21T15:49:50.774 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.774 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.774 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:50.774 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.531024+0000", 2024-09-21T15:49:50.774 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033721910000000001 2024-09-21T15:49:50.774 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.774 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.774 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:50.774 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.531056+0000", 2024-09-21T15:49:50.774 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1628000000000003e-05 2024-09-21T15:49:50.774 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.774 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:50.774 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.774 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.775 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.775 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13839 3.0 e31/22 3:03c4fb32:::rbd_data.1133dc5adfdf.0000000000000099:head v 31'59, mlcod=31'56)", 2024-09-21T15:49:50.775 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.528097+0000", 2024-09-21T15:49:50.775 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 33.219443681999998, 2024-09-21T15:49:50.775 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0057228840000000001, 2024-09-21T15:49:50.775 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:50.775 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:50.775 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:50.775 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.775 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:50.775 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528097+0000", 2024-09-21T15:49:50.775 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.775 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.775 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.775 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:50.776 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528097+0000", 2024-09-21T15:49:50.776 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.776 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.776 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.776 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:50.776 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528096+0000", 2024-09-21T15:49:50.776 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:50.776 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.776 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.776 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:50.776 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528105+0000", 2024-09-21T15:49:50.776 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7350000000000002e-06 2024-09-21T15:49:50.776 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.776 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.776 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:50.777 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528117+0000", 2024-09-21T15:49:50.777 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1909000000000001e-05 2024-09-21T15:49:50.777 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.777 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.777 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:50.777 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528124+0000", 2024-09-21T15:49:50.777 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8419999999999999e-06 2024-09-21T15:49:50.777 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.777 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.777 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:50.777 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528168+0000", 2024-09-21T15:49:50.777 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4301999999999997e-05 2024-09-21T15:49:50.777 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.777 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.777 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:50.777 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528191+0000", 2024-09-21T15:49:50.778 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2622000000000001e-05 2024-09-21T15:49:50.778 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.778 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.778 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:50.778 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.533780+0000", 2024-09-21T15:49:50.778 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0055897519999999999 2024-09-21T15:49:50.778 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.778 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.778 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:50.778 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.533820+0000", 2024-09-21T15:49:50.778 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9366999999999998e-05 2024-09-21T15:49:50.778 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.778 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:50.779 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.779 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.779 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.779 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13884 3.9 e31/22 3:93003736:::rbd_data.1133dc5adfdf.00000000000000a1:head v 31'54, mlcod=31'52)", 2024-09-21T15:49:50.779 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.562433+0000", 2024-09-21T15:49:50.779 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 33.185106933999997, 2024-09-21T15:49:50.779 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.005162313, 2024-09-21T15:49:50.779 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:50.779 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:50.779 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:50.779 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.779 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:50.779 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562433+0000", 2024-09-21T15:49:50.779 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.780 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.780 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.780 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:50.780 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562433+0000", 2024-09-21T15:49:50.780 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.780 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.780 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.780 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:50.780 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562432+0000", 2024-09-21T15:49:50.780 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:50.780 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.780 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.780 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:50.780 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562467+0000", 2024-09-21T15:49:50.780 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4885999999999998e-05 2024-09-21T15:49:50.781 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.781 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.781 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:50.781 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562481+0000", 2024-09-21T15:49:50.781 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3604000000000001e-05 2024-09-21T15:49:50.781 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.781 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.781 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:50.781 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562489+0000", 2024-09-21T15:49:50.781 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1829999999999994e-06 2024-09-21T15:49:50.781 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.781 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.781 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:50.782 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562536+0000", 2024-09-21T15:49:50.782 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6830999999999999e-05 2024-09-21T15:49:50.782 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.782 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.782 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:50.782 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562559+0000", 2024-09-21T15:49:50.782 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2765999999999999e-05 2024-09-21T15:49:50.782 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.782 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.782 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:50.782 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.567566+0000", 2024-09-21T15:49:50.782 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0050073219999999998 2024-09-21T15:49:50.782 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.783 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.783 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:50.783 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.567596+0000", 2024-09-21T15:49:50.783 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9802999999999999e-05 2024-09-21T15:49:50.783 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.783 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:50.783 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.783 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.783 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.783 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:22340 3.9 e34/22 3:910a5abe:::rbd_data.1133896adcfd.0000000000000023:head v 34'92, mlcod=34'90)", 2024-09-21T15:49:50.783 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:21.628025+0000", 2024-09-21T15:49:50.783 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 29.119515553999999, 2024-09-21T15:49:50.783 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046196010000000001, 2024-09-21T15:49:50.783 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:50.784 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:50.784 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:50.784 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.784 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:50.784 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628025+0000", 2024-09-21T15:49:50.784 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.784 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.784 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.784 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:50.784 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628025+0000", 2024-09-21T15:49:50.784 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.784 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.784 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.784 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:50.784 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628024+0000", 2024-09-21T15:49:50.785 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:50.785 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.785 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.785 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:50.785 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628034+0000", 2024-09-21T15:49:50.785 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3470000000000005e-06 2024-09-21T15:49:50.785 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.785 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.785 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:50.785 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628043+0000", 2024-09-21T15:49:50.785 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9450000000000005e-06 2024-09-21T15:49:50.785 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.785 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.785 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:50.786 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628048+0000", 2024-09-21T15:49:50.786 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8080000000000004e-06 2024-09-21T15:49:50.786 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.786 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.786 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:50.786 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628081+0000", 2024-09-21T15:49:50.786 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2738999999999999e-05 2024-09-21T15:49:50.786 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.786 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.786 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:50.786 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628094+0000", 2024-09-21T15:49:50.786 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3273e-05 2024-09-21T15:49:50.786 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.787 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.787 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:50.787 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.632616+0000", 2024-09-21T15:49:50.787 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045217269999999997 2024-09-21T15:49:50.787 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.787 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.787 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:50.787 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.632644+0000", 2024-09-21T15:49:50.787 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8469000000000001e-05 2024-09-21T15:49:50.787 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.787 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:50.787 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.787 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.787 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.788 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:22417 3.4 e34/22 3:25310a10:::rbd_data.1133896adcfd.0000000000000075:head v 34'48, mlcod=34'46)", 2024-09-21T15:49:50.788 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:21.704297+0000", 2024-09-21T15:49:50.788 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 29.043243455999999, 2024-09-21T15:49:50.788 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0038101799999999998, 2024-09-21T15:49:50.788 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:50.788 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:50.788 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:50.788 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.788 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:50.788 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704297+0000", 2024-09-21T15:49:50.788 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.788 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.788 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.788 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:50.788 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704297+0000", 2024-09-21T15:49:50.789 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.789 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.789 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.789 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:50.789 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704296+0000", 2024-09-21T15:49:50.789 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:50.789 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.789 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.789 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:50.789 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704308+0000", 2024-09-21T15:49:50.789 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2559e-05 2024-09-21T15:49:50.789 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.789 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.789 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:50.790 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704337+0000", 2024-09-21T15:49:50.790 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8960000000000001e-05 2024-09-21T15:49:50.790 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.790 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.790 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:50.790 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704345+0000", 2024-09-21T15:49:50.790 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3059999999999998e-06 2024-09-21T15:49:50.790 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.790 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.790 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:50.790 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704408+0000", 2024-09-21T15:49:50.790 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2156000000000006e-05 2024-09-21T15:49:50.790 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.790 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.791 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:50.791 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704430+0000", 2024-09-21T15:49:50.791 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1931000000000001e-05 2024-09-21T15:49:50.791 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.791 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.791 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:50.791 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.708073+0000", 2024-09-21T15:49:50.791 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003643028 2024-09-21T15:49:50.791 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.791 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.791 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:50.791 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.708107+0000", 2024-09-21T15:49:50.791 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4613999999999998e-05 2024-09-21T15:49:50.791 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.791 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:50.792 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.792 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.792 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.792 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:25278 3.4 e38/22 3:2251cc1c:::rbd_data.1133ebd6d3ba.0000000000000024:head v 38'51, mlcod=38'49)", 2024-09-21T15:49:50.792 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.317337+0000", 2024-09-21T15:49:50.792 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.430203696, 2024-09-21T15:49:50.792 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053904110000000003, 2024-09-21T15:49:50.792 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:50.792 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:50.792 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:50.792 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.792 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:50.792 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317337+0000", 2024-09-21T15:49:50.792 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.792 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.793 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.793 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:50.793 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317337+0000", 2024-09-21T15:49:50.793 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.793 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.793 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.793 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:50.793 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317335+0000", 2024-09-21T15:49:50.793 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:49:50.793 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.793 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.793 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:50.793 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317347+0000", 2024-09-21T15:49:50.793 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.19e-05 2024-09-21T15:49:50.794 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.794 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.794 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:50.794 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317369+0000", 2024-09-21T15:49:50.794 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2232e-05 2024-09-21T15:49:50.794 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.794 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.794 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:50.794 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317381+0000", 2024-09-21T15:49:50.794 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2231e-05 2024-09-21T15:49:50.794 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.794 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.794 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:50.795 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317425+0000", 2024-09-21T15:49:50.795 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3625000000000003e-05 2024-09-21T15:49:50.795 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.795 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.795 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:50.795 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317449+0000", 2024-09-21T15:49:50.795 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4301999999999999e-05 2024-09-21T15:49:50.795 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.795 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.795 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:50.795 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.322688+0000", 2024-09-21T15:49:50.795 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.005238609 2024-09-21T15:49:50.795 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.795 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.796 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:50.796 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.322727+0000", 2024-09-21T15:49:50.796 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9201e-05 2024-09-21T15:49:50.796 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.796 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:50.796 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.796 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.796 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.796 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:25286 3.0 e38/22 3:0248fb60:::rbd_data.1133ebd6d3ba.0000000000000066:head v 38'109, mlcod=34'107)", 2024-09-21T15:49:50.796 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.317638+0000", 2024-09-21T15:49:50.796 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.429902816999999, 2024-09-21T15:49:50.796 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0054074370000000002, 2024-09-21T15:49:50.796 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:50.797 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:50.797 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:50.797 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.797 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:50.797 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317638+0000", 2024-09-21T15:49:50.797 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.797 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.797 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:50.797 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317638+0000", 2024-09-21T15:49:50.797 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.797 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.797 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:50.797 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317637+0000", 2024-09-21T15:49:50.798 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:50.798 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.798 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.798 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:50.798 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317643+0000", 2024-09-21T15:49:50.798 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5609999999999998e-06 2024-09-21T15:49:50.798 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.798 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.798 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:50.798 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317653+0000", 2024-09-21T15:49:50.798 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0705e-05 2024-09-21T15:49:50.798 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.798 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.798 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:50.798 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317660+0000", 2024-09-21T15:49:50.799 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2099999999999998e-06 2024-09-21T15:49:50.799 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.799 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.799 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:50.799 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317704+0000", 2024-09-21T15:49:50.799 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4659000000000001e-05 2024-09-21T15:49:50.799 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.799 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.799 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:50.799 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317725+0000", 2024-09-21T15:49:50.799 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0537e-05 2024-09-21T15:49:50.799 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.799 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.799 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:50.800 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323010+0000", 2024-09-21T15:49:50.800 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052853420000000002 2024-09-21T15:49:50.800 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.800 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.800 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:50.800 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323045+0000", 2024-09-21T15:49:50.800 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4860000000000002e-05 2024-09-21T15:49:50.800 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.800 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:50.800 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.800 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.800 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.800 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:25280 3.4 e38/22 3:27fca38c:::rbd_data.1133ebd6d3ba.0000000000000026:head v 38'52, mlcod=38'49)", 2024-09-21T15:49:50.800 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.317684+0000", 2024-09-21T15:49:50.801 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.429856920999999, 2024-09-21T15:49:50.801 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053570809999999996, 2024-09-21T15:49:50.801 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:50.801 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:50.801 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:50.801 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.801 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:50.801 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317684+0000", 2024-09-21T15:49:50.801 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.801 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.801 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.801 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:50.801 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317684+0000", 2024-09-21T15:49:50.801 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.801 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.802 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.802 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:50.802 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317683+0000", 2024-09-21T15:49:50.802 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:50.802 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.802 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.802 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:50.802 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317688+0000", 2024-09-21T15:49:50.802 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5009999999999998e-06 2024-09-21T15:49:50.802 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.802 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.802 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:50.802 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317694+0000", 2024-09-21T15:49:50.802 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3579999999999999e-06 2024-09-21T15:49:50.803 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.803 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.803 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:50.803 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317698+0000", 2024-09-21T15:49:50.803 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4150000000000001e-06 2024-09-21T15:49:50.803 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.803 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.803 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:50.803 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317738+0000", 2024-09-21T15:49:50.803 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9575e-05 2024-09-21T15:49:50.803 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.803 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.803 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:50.803 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317762+0000", 2024-09-21T15:49:50.804 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4173999999999999e-05 2024-09-21T15:49:50.804 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.804 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.804 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:50.804 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323005+0000", 2024-09-21T15:49:50.804 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00524333 2024-09-21T15:49:50.804 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.804 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.804 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:50.804 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323041+0000", 2024-09-21T15:49:50.804 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5139999999999999e-05 2024-09-21T15:49:50.804 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.804 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:50.804 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.805 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.805 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.805 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:34837 3.0 e43/22 3:038527c8:::rbd_header.1133fed1a840:head v 43'130, mlcod=43'128)", 2024-09-21T15:49:50.805 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:49:50.805 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 19.763652223000001, 2024-09-21T15:49:50.805 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045776519999999998, 2024-09-21T15:49:50.805 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:50.805 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:50.805 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:50.805 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.805 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:50.805 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:49:50.805 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.805 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.805 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.806 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:50.806 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:49:50.806 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.806 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.806 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.806 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:50.806 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:49:50.806 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:50.806 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.806 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.806 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:50.806 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983892+0000", 2024-09-21T15:49:50.806 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8450000000000002e-06 2024-09-21T15:49:50.807 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.807 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.807 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:50.807 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983902+0000", 2024-09-21T15:49:50.807 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0299999999999999e-06 2024-09-21T15:49:50.807 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.807 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.807 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:50.807 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983906+0000", 2024-09-21T15:49:50.807 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2300000000000002e-06 2024-09-21T15:49:50.807 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.807 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.807 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:50.807 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983940+0000", 2024-09-21T15:49:50.808 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4563000000000001e-05 2024-09-21T15:49:50.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.808 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:50.808 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983960+0000", 2024-09-21T15:49:50.808 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9199999999999999e-05 2024-09-21T15:49:50.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.808 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:50.808 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.988430+0000", 2024-09-21T15:49:50.808 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0044709149999999998 2024-09-21T15:49:50.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.808 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:50.809 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.988466+0000", 2024-09-21T15:49:50.809 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5420000000000003e-05 2024-09-21T15:49:50.809 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.809 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:50.809 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.809 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "pg_update_log_missing(3.0 epoch 50/22 rep_tid 498 entries 50'171 (0'0) error 3:5df4a5a0:::rbd_data.11339e87a32a.000000000000008e:head by client.4403.0:80207 0.000000 -2 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 47'167)", 2024-09-21T15:49:50.809 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:49:50.809 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.8057793499999999, 2024-09-21T15:49:50.809 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0047253119999999997, 2024-09-21T15:49:50.809 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:50.809 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-09-21T15:49:50.809 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:50.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.810 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:50.810 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:49:50.810 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.810 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.810 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:50.810 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:49:50.810 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.810 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.810 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:50.810 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:49:50.810 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:50.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:50.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941768+0000", 2024-09-21T15:49:50.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7309999999999993e-06 2024-09-21T15:49:50.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:50.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941780+0000", 2024-09-21T15:49:50.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1399999999999999e-05 2024-09-21T15:49:50.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:50.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941789+0000", 2024-09-21T15:49:50.812 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5230000000000001e-06 2024-09-21T15:49:50.812 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.812 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.812 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:50.812 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941841+0000", 2024-09-21T15:49:50.812 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1863999999999997e-05 2024-09-21T15:49:50.812 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.812 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.812 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:50.812 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.946486+0000", 2024-09-21T15:49:50.812 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046452539999999997 2024-09-21T15:49:50.812 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.812 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:50.812 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.813 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:82086 3.0 e50/22 3:0114bc4d:::rbd_data.11334c14b8fd.0000000000000069:head v 50'178, mlcod=50'176)", 2024-09-21T15:49:50.813 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:41.155591+0000", 2024-09-21T15:49:50.813 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.5919490379999992, 2024-09-21T15:49:50.813 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003363497, 2024-09-21T15:49:50.813 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:50.813 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:50.813 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:50.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.813 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:50.813 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.155591+0000", 2024-09-21T15:49:50.813 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.814 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:50.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.155591+0000", 2024-09-21T15:49:50.814 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.814 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:50.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.155590+0000", 2024-09-21T15:49:50.814 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:50.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.814 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:50.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.155606+0000", 2024-09-21T15:49:50.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6075000000000001e-05 2024-09-21T15:49:50.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.815 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.815 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:50.815 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.155623+0000", 2024-09-21T15:49:50.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6838e-05 2024-09-21T15:49:50.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.815 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.815 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:50.815 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.155630+0000", 2024-09-21T15:49:50.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.627e-06 2024-09-21T15:49:50.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.815 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.815 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:50.816 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.155677+0000", 2024-09-21T15:49:50.816 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6669999999999999e-05 2024-09-21T15:49:50.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.816 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.816 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:50.816 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.155698+0000", 2024-09-21T15:49:50.816 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1330000000000001e-05 2024-09-21T15:49:50.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.816 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.816 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:50.816 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.158918+0000", 2024-09-21T15:49:50.816 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032198600000000002 2024-09-21T15:49:50.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.816 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.817 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:50.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.158955+0000", 2024-09-21T15:49:50.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6647000000000003e-05 2024-09-21T15:49:50.817 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.817 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:50.817 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.817 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:82470 3.4 e50/22 3:21ad7dfe:::rbd_data.11334c14b8fd.00000000000003a7:head v 50'190, mlcod=46'188)", 2024-09-21T15:49:50.817 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:41.291787+0000", 2024-09-21T15:49:50.817 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.4557534130000001, 2024-09-21T15:49:50.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033489969999999998, 2024-09-21T15:49:50.817 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:50.817 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:50.818 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:50.818 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.818 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:50.818 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.291787+0000", 2024-09-21T15:49:50.818 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.818 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.818 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.818 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:50.818 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.291787+0000", 2024-09-21T15:49:50.818 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.818 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.818 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.818 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:50.818 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.291786+0000", 2024-09-21T15:49:50.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:50.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.819 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:50.819 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.291795+0000", 2024-09-21T15:49:50.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7830000000000004e-06 2024-09-21T15:49:50.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.819 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:50.819 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.291814+0000", 2024-09-21T15:49:50.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9117999999999999e-05 2024-09-21T15:49:50.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.819 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:50.820 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.291822+0000", 2024-09-21T15:49:50.820 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0669999999999992e-06 2024-09-21T15:49:50.820 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.820 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.820 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:50.820 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.291866+0000", 2024-09-21T15:49:50.820 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4669000000000002e-05 2024-09-21T15:49:50.820 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.820 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.820 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:50.820 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.291888+0000", 2024-09-21T15:49:50.820 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1087999999999999e-05 2024-09-21T15:49:50.820 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.820 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.820 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:50.821 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.295104+0000", 2024-09-21T15:49:50.821 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032164910000000001 2024-09-21T15:49:50.821 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.821 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.821 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:50.821 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.295136+0000", 2024-09-21T15:49:50.821 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1931999999999999e-05 2024-09-21T15:49:50.821 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.821 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:50.821 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.821 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.821 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.821 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "pg_update_log_missing(3.0 epoch 53/22 rep_tid 585 entries 53'201 (0'0) error 3:6a1d4890:::rbd_data.1133e079cae4.0000000000000004:head by client.4403.0:101089 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 53'198)", 2024-09-21T15:49:50.821 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:44.832827+0000", 2024-09-21T15:49:50.821 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.9147135420000003, 2024-09-21T15:49:50.822 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053552929999999997, 2024-09-21T15:49:50.822 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:50.822 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-09-21T15:49:50.822 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:50.822 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.822 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:50.822 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832827+0000", 2024-09-21T15:49:50.822 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.822 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.822 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.822 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:50.822 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832827+0000", 2024-09-21T15:49:50.822 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.823 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.823 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:50.823 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832826+0000", 2024-09-21T15:49:50.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:50.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.823 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.823 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:50.823 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832835+0000", 2024-09-21T15:49:50.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4509999999999992e-06 2024-09-21T15:49:50.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.823 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.823 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:50.823 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832854+0000", 2024-09-21T15:49:50.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9352999999999999e-05 2024-09-21T15:49:50.824 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.824 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:50.824 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832866+0000", 2024-09-21T15:49:50.824 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1766000000000001e-05 2024-09-21T15:49:50.824 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.824 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:50.824 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.833091+0000", 2024-09-21T15:49:50.824 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00022490300000000001 2024-09-21T15:49:50.824 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.824 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:50.824 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.838182+0000", 2024-09-21T15:49:50.824 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0050915420000000001 2024-09-21T15:49:50.824 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.825 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:50.825 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.825 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.825 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.825 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:102822 3.4 e54/22 3:2729ba03:::rbd_object_map.113336ae1075:head v 54'208, mlcod=54'205)", 2024-09-21T15:49:50.825 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:46.249540+0000", 2024-09-21T15:49:50.825 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.4980003100000001, 2024-09-21T15:49:50.825 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052654859999999998, 2024-09-21T15:49:50.825 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:50.825 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:50.825 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:50.825 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.825 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:50.825 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249540+0000", 2024-09-21T15:49:50.826 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.826 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.826 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.826 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:50.826 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249540+0000", 2024-09-21T15:49:50.826 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.826 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.826 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.826 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:50.826 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249539+0000", 2024-09-21T15:49:50.826 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:50.826 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.826 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.826 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:50.826 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249546+0000", 2024-09-21T15:49:50.827 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7290000000000003e-06 2024-09-21T15:49:50.827 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.827 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.827 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:50.827 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249558+0000", 2024-09-21T15:49:50.827 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1844e-05 2024-09-21T15:49:50.827 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.827 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.827 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:50.827 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249565+0000", 2024-09-21T15:49:50.827 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8210000000000004e-06 2024-09-21T15:49:50.827 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.827 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.827 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:50.827 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249602+0000", 2024-09-21T15:49:50.828 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7057000000000001e-05 2024-09-21T15:49:50.828 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.828 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.828 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:50.828 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249622+0000", 2024-09-21T15:49:50.828 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9678999999999999e-05 2024-09-21T15:49:50.828 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.828 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.828 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:50.828 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254783+0000", 2024-09-21T15:49:50.828 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0051611449999999998 2024-09-21T15:49:50.828 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.828 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.828 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:50.828 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254806+0000", 2024-09-21T15:49:50.829 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2853e-05 2024-09-21T15:49:50.829 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.829 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:50.829 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.829 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.829 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.829 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:102823 3.4 e54/22 3:2729ba03:::rbd_object_map.113336ae1075:head v 54'209, mlcod=54'205)", 2024-09-21T15:49:50.829 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:46.250318+0000", 2024-09-21T15:49:50.829 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.4972227250000003, 2024-09-21T15:49:50.829 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045013539999999999, 2024-09-21T15:49:50.829 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:50.829 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:50.829 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:50.829 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.830 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:50.830 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250318+0000", 2024-09-21T15:49:50.830 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.830 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.830 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.830 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:50.830 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250318+0000", 2024-09-21T15:49:50.830 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:50.830 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.830 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.830 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:50.830 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250317+0000", 2024-09-21T15:49:50.830 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:50.830 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.831 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.831 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:50.831 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250322+0000", 2024-09-21T15:49:50.831 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.074e-06 2024-09-21T15:49:50.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.831 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.831 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:50.831 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250330+0000", 2024-09-21T15:49:50.831 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4730000000000003e-06 2024-09-21T15:49:50.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.831 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.831 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:50.831 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250335+0000", 2024-09-21T15:49:50.831 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8019999999999997e-06 2024-09-21T15:49:50.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.831 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.832 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:50.832 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250375+0000", 2024-09-21T15:49:50.832 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9459999999999998e-05 2024-09-21T15:49:50.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.832 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.832 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:50.832 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250388+0000", 2024-09-21T15:49:50.832 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3288999999999999e-05 2024-09-21T15:49:50.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.832 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.832 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:50.832 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254807+0000", 2024-09-21T15:49:50.832 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0044192220000000004 2024-09-21T15:49:50.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:50.832 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:50.833 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:50.833 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254819+0000", 2024-09-21T15:49:50.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1603999999999999e-05 2024-09-21T15:49:50.833 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.833 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:50.833 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.833 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:50.833 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:50.833 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:50.834 INFO:teuthology.orchestra.run.smithi089.stdout:3585 writesame 0x412dbf3 thru 0x412fa82 data_size 0x8(0x1e90 bytes) 2024-09-21T15:49:50.834 INFO:teuthology.orchestra.run.smithi089.stdout:3586 read 0xb4013fc thru 0xb404626 (0x322b bytes) 2024-09-21T15:49:50.835 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:49:50.836 INFO:teuthology.orchestra.run.smithi089.stdout:3588 trunc from 0xdec382b to 0x1758ed4 2024-09-21T15:49:50.836 INFO:teuthology.orchestra.run.smithi089.stdout:3590 trunc from 0x1758ed4 to 0x12f3005 2024-09-21T15:49:50.836 INFO:teuthology.orchestra.run.smithi089.stdout:3591 read 0xe48e1e thru 0xe4ffcc (0x71af bytes) 2024-09-21T15:49:50.836 INFO:teuthology.orchestra.run.smithi089.stdout:3592 write 0x15e59b1 thru 0x15f0a1f (0xb06f bytes) 2024-09-21T15:49:50.836 INFO:teuthology.orchestra.run.smithi089.stdout:3594 write 0x8975f45 thru 0x8980f32 (0xafee bytes) 2024-09-21T15:49:50.836 INFO:teuthology.orchestra.run.smithi089.stdout:3595 read 0x333ec8f thru 0x33419ab (0x2d1d bytes) 2024-09-21T15:49:50.836 INFO:teuthology.orchestra.run.smithi089.stdout:3596 read 0x4480bb7 thru 0x4482edd (0x2327 bytes) 2024-09-21T15:49:50.837 INFO:teuthology.orchestra.run.smithi089.stdout:3597 read 0x3e9b317 thru 0x3ea0306 (0x4ff0 bytes) 2024-09-21T15:49:50.837 INFO:teuthology.orchestra.run.smithi089.stdout:3598 writesame 0x46e4de3 thru 0x46eaa21 data_size 0x1(0x5c3f bytes) 2024-09-21T15:49:50.837 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:50.833+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:50.837 INFO:teuthology.orchestra.run.smithi089.stdout:3600 write 0x7dba1eb thru 0x7dc56d7 (0xb4ed bytes) 2024-09-21T15:49:50.844 INFO:teuthology.orchestra.run.smithi089.stdout:3601 read 0x42af601 thru 0x42bb903 (0xc303 bytes) 2024-09-21T15:49:50.844 INFO:teuthology.orchestra.run.smithi089.stdout:3602 punch from 0x517340a to 0x5181de0, (0xe9d6 bytes) 2024-09-21T15:49:50.844 INFO:teuthology.orchestra.run.smithi089.stdout:3604 write 0x2a98666 thru 0x2aa2b72 (0xa50d bytes) 2024-09-21T15:49:50.855 INFO:teuthology.orchestra.run.smithi089.stdout:3606 read 0x26a4c4b thru 0x26aab60 (0x5f16 bytes) 2024-09-21T15:49:50.855 INFO:teuthology.orchestra.run.smithi089.stdout:3607 read 0x14c7539 thru 0x14ce273 (0x6d3b bytes) 2024-09-21T15:49:50.863 INFO:teuthology.orchestra.run.smithi089.stdout:3608 trunc from 0x8980f33 to 0x1b6b45e 2024-09-21T15:49:50.878 INFO:teuthology.orchestra.run.smithi089.stdout:3609 read 0xc41662 thru 0xc49ef3 (0x8892 bytes) 2024-09-21T15:49:50.891 INFO:teuthology.orchestra.run.smithi089.stdout:3611 writesame 0x66af233 thru 0x66be4b0 data_size 0x1(0xf27e bytes) 2024-09-21T15:49:50.891 INFO:teuthology.orchestra.run.smithi089.stdout:3612 read 0x29adccd thru 0x29b8da9 (0xb0dd bytes) 2024-09-21T15:49:50.891 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:50.898 INFO:teuthology.orchestra.run.smithi089.stdout:3615 write 0x7888079 thru 0x789192a (0x98b2 bytes) 2024-09-21T15:49:50.901 INFO:teuthology.orchestra.run.smithi089.stdout:3616 punch from 0x6597564 to 0x65a0b33, (0x95cf bytes) 2024-09-21T15:49:50.901 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:50.901 INFO:teuthology.orchestra.run.smithi089.stdout:3618 write 0xa036a0 thru 0xa04da2 (0x1703 bytes) 2024-09-21T15:49:50.914 INFO:teuthology.orchestra.run.smithi089.stdout:3620 write 0xb4aaf67 thru 0xb4ac9c4 (0x1a5e bytes) 2024-09-21T15:49:50.917 INFO:teuthology.orchestra.run.smithi089.stdout:3621 writesame 0x805c186 thru 0x80603d1 data_size 0x2(0x424c bytes) 2024-09-21T15:49:50.920 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T15:49:50.932 INFO:teuthology.orchestra.run.smithi089.stdout:3622 write 0xcc00c76 thru 0xcc0ed05 (0xe090 bytes) 2024-09-21T15:49:50.935 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:50.934+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:50.935 INFO:teuthology.orchestra.run.smithi089.stdout:3623 writesame 0x843025c thru 0x8437c13 data_size 0x5f(0x79b8 bytes) 2024-09-21T15:49:50.939 INFO:teuthology.orchestra.run.smithi089.stdout:3624 read 0xbb915b thru 0xbbc7e5 (0x368b bytes) 2024-09-21T15:49:50.940 INFO:teuthology.orchestra.run.smithi089.stdout:3625 writesame 0xc17631b thru 0xc17d78a data_size 0xc(0x7470 bytes) 2024-09-21T15:49:50.947 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:50.947 INFO:teuthology.orchestra.run.smithi089.stdout:3628 read 0x942cdac thru 0x9435384 (0x85d9 bytes) 2024-09-21T15:49:50.947 INFO:teuthology.orchestra.run.smithi089.stdout:3629 read 0x4f03bc0 thru 0x4f07778 (0x3bb9 bytes) 2024-09-21T15:49:50.947 INFO:teuthology.orchestra.run.smithi089.stdout:3630 write 0x7f11485 thru 0x7f2083c (0xf3b8 bytes) 2024-09-21T15:49:50.953 INFO:teuthology.orchestra.run.smithi089.stdout:3631 punch from 0x78b72b1 to 0x78c184d, (0xa59c bytes) 2024-09-21T15:49:50.962 INFO:teuthology.orchestra.run.smithi089.stdout:3632 trunc from 0xcc0ed06 to 0x63bfb87 2024-09-21T15:49:50.974 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:50.974 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:49:50.974 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:49:50.974 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:50.980 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:50.980 INFO:teuthology.orchestra.run.smithi089.stdout:3635 punch from 0xa0921b to 0xa10e48, (0x7c2d bytes) 2024-09-21T15:49:50.980 INFO:teuthology.orchestra.run.smithi089.stdout:3636 write 0xec0b6a thru 0xec682b (0x5cc2 bytes) 2024-09-21T15:49:50.982 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:50.984 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:49:50.988 INFO:teuthology.orchestra.run.smithi089.stdout:3639 write 0x7652c9d thru 0x7660f90 (0xe2f4 bytes) 2024-09-21T15:49:50.995 INFO:teuthology.orchestra.run.smithi089.stdout:3640 punch from 0x43caebb to 0x43d57de, (0xa923 bytes) 2024-09-21T15:49:50.995 INFO:teuthology.orchestra.run.smithi089.stdout:3641 read 0x5627dec thru 0x5634f35 (0xd14a bytes) 2024-09-21T15:49:51.004 INFO:teuthology.orchestra.run.smithi089.stdout:3643 write 0xac52132 thru 0xac52ca9 (0xb78 bytes) 2024-09-21T15:49:51.007 INFO:teuthology.orchestra.run.smithi089.stdout:3645 write 0x23f07d2 thru 0x23ff4eb (0xed1a bytes) 2024-09-21T15:49:51.010 INFO:teuthology.orchestra.run.smithi089.stdout:3647 read 0x4f2eff4 thru 0x4f3df34 (0xef41 bytes) 2024-09-21T15:49:51.010 INFO:teuthology.orchestra.run.smithi089.stdout:3650 write 0x17b13f2 thru 0x17b58f0 (0x44ff bytes) 2024-09-21T15:49:51.013 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:51.014 INFO:teuthology.orchestra.run.smithi089.stdout:3653 write 0x371c9e9 thru 0x371ffd6 (0x35ee bytes) 2024-09-21T15:49:51.017 INFO:teuthology.orchestra.run.smithi089.stdout:3654 punch from 0x48a3c3 to 0x48dd09, (0x3946 bytes) 2024-09-21T15:49:51.017 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:51.017 INFO:teuthology.orchestra.run.smithi089.stdout:3656 read 0x875160f thru 0x87610ac (0xfa9e bytes) 2024-09-21T15:49:51.017 INFO:teuthology.orchestra.run.smithi089.stdout:3657 read 0x1674298 thru 0x1677775 (0x34de bytes) 2024-09-21T15:49:51.017 INFO:teuthology.orchestra.run.smithi089.stdout:3659 write 0x35affd0 thru 0x35b428a (0x42bb bytes) 2024-09-21T15:49:51.023 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:51.031 INFO:teuthology.orchestra.run.smithi089.stdout:3662 write 0xcafd658 thru 0xcb074bc (0x9e65 bytes) 2024-09-21T15:49:51.036 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:51.035+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:51.041 INFO:teuthology.orchestra.run.smithi089.stdout:3663 write 0xd517b33 thru 0xd51bdb9 (0x4287 bytes) 2024-09-21T15:49:51.044 INFO:teuthology.orchestra.run.smithi089.stdout:3665 write 0x8541114 thru 0x854e0f8 (0xcfe5 bytes) 2024-09-21T15:49:51.047 INFO:teuthology.orchestra.run.smithi089.stdout:3666 read 0xbb1bf8a thru 0xbb2ab30 (0xeba7 bytes) 2024-09-21T15:49:51.047 INFO:teuthology.orchestra.run.smithi089.stdout:3667 writesame 0x9fe1e0 thru 0xa068d5 data_size 0x19(0x86f6 bytes) 2024-09-21T15:49:51.051 INFO:teuthology.orchestra.run.smithi089.stdout:3669 write 0x60fc40 thru 0x614845 (0x4c06 bytes) 2024-09-21T15:49:51.055 INFO:teuthology.orchestra.run.smithi089.stdout:3673 punch from 0x365735a to 0x365cb6d, (0x5813 bytes) 2024-09-21T15:49:51.056 INFO:teuthology.orchestra.run.smithi089.stdout:3674 punch from 0xb4052d1 to 0xb4135b9, (0xe2e8 bytes) 2024-09-21T15:49:51.056 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:51.056 INFO:teuthology.orchestra.run.smithi089.stdout:3678 write 0x85af194 thru 0x85bba0c (0xc879 bytes) 2024-09-21T15:49:51.058 INFO:teuthology.orchestra.run.smithi089.stdout:3679 punch from 0x5ff2cdc to 0x5ff8442, (0x5766 bytes) 2024-09-21T15:49:51.058 INFO:teuthology.orchestra.run.smithi089.stdout:3680 write 0xb64d916 thru 0xb658246 (0xa931 bytes) 2024-09-21T15:49:51.061 INFO:teuthology.orchestra.run.smithi089.stdout:3681 write 0xd6a635 thru 0xd7565e (0xb02a bytes) 2024-09-21T15:49:51.062 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:51.062 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:51.062 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:51.067 INFO:teuthology.orchestra.run.smithi089.stdout:3685 trunc from 0xd51bdba to 0x9c823a4 2024-09-21T15:49:51.078 INFO:teuthology.orchestra.run.smithi089.stdout:3686 read 0x594001c thru 0x594a229 (0xa20e bytes) 2024-09-21T15:49:51.085 INFO:teuthology.orchestra.run.smithi089.stdout:3687 trunc from 0x9c823a4 to 0x390107e 2024-09-21T15:49:51.097 INFO:teuthology.orchestra.run.smithi089.stdout:3688 read 0x2051980 thru 0x205930d (0x798e bytes) 2024-09-21T15:49:51.097 INFO:teuthology.orchestra.run.smithi089.stdout:3689 read 0x1e360df thru 0x1e3940c (0x332e bytes) 2024-09-21T15:49:51.097 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:51.098 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:51.106 INFO:teuthology.orchestra.run.smithi089.stdout:3701 writesame 0x5e46a37 thru 0x5e485d5 data_size 0x1(0x1b9f bytes) 2024-09-21T15:49:51.106 INFO:teuthology.orchestra.run.smithi089.stdout:3702 write 0x486d411 thru 0x486dc35 (0x825 bytes) 2024-09-21T15:49:51.109 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:51.109 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:49:51.109 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:49:51.109 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:49:51.109 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:51.110 INFO:teuthology.orchestra.run.smithi089.stdout:3703 read 0x32b3227 thru 0x32b6a6f (0x3849 bytes) 2024-09-21T15:49:51.116 INFO:teuthology.orchestra.run.smithi089.stdout:3704 trunc from 0x5e485d6 to 0x379ef2 2024-09-21T15:49:51.123 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:49:51.147 INFO:teuthology.orchestra.run.smithi089.stdout:3706 write 0x3ebf8f7 thru 0x3ec93d2 (0x9adc bytes) 2024-09-21T15:49:51.147 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:51.146+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:51.171 INFO:teuthology.orchestra.run.smithi089.stdout:3708 write 0xe81bfef thru 0xe82388e (0x78a0 bytes) 2024-09-21T15:49:51.176 INFO:teuthology.orchestra.run.smithi089.stdout:3715 punch from 0x20914e9 to 0x2097939, (0x6450 bytes) 2024-09-21T15:49:51.177 INFO:teuthology.orchestra.run.smithi089.stdout:3716 punch from 0xd2f56d4 to 0xd300c53, (0xb57f bytes) 2024-09-21T15:49:51.177 INFO:teuthology.orchestra.run.smithi089.stdout:3718 write 0xa0cb3f5 thru 0xa0cd17a (0x1d86 bytes) 2024-09-21T15:49:51.181 INFO:teuthology.orchestra.run.smithi089.stdout:3719 write 0x61bc889 thru 0x61c8dd8 (0xc550 bytes) 2024-09-21T15:49:51.191 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:51.191 INFO:teuthology.orchestra.run.smithi089.stdout:3723 punch from 0x6a1be5e to 0x6a1c7ba, (0x95c bytes) 2024-09-21T15:49:51.191 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:51.191 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:51.191 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:51.191 INFO:teuthology.orchestra.run.smithi089.stdout:3731 writesame 0x76042df thru 0x760d9cf data_size 0x1(0x96f1 bytes) 2024-09-21T15:49:51.211 INFO:teuthology.orchestra.run.smithi089.stdout:3732 read 0x56edda0 thru 0x56fc36e (0xe5cf bytes) 2024-09-21T15:49:51.224 INFO:teuthology.orchestra.run.smithi089.stdout:3734 trunc from 0xe82388f to 0x3a2f13a 2024-09-21T15:49:51.237 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:51.236+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:51.244 INFO:teuthology.orchestra.run.smithi089.stdout:3735 trunc from 0x3a2f13a to 0x35e4844 2024-09-21T15:49:51.253 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:51.253 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:51.261 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:51.261 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:49:51.261 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:49:51.261 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:49:51.261 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.261 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:14051 3.8 3:13437810:::rbd_data.1133dc5adfdf.0000000000000039:head [write 716800~796] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-21T15:49:51.261 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:49:51.261 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 33.564119820999998, 2024-09-21T15:49:51.261 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.98064002699999997, 2024-09-21T15:49:51.261 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:51.262 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:51.262 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:51.262 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:51.262 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:51.262 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 14051 2024-09-21T15:49:51.262 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.262 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:51.262 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.262 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:51.262 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:49:51.262 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:51.262 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.262 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.263 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:51.263 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:49:51.263 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:51.263 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.263 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.263 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:51.263 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:49:51.263 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:51.263 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.263 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.263 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:51.263 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695131+0000", 2024-09-21T15:49:51.263 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4680000000000003e-06 2024-09-21T15:49:51.263 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.264 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.264 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:51.264 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695139+0000", 2024-09-21T15:49:51.264 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9400000000000002e-06 2024-09-21T15:49:51.264 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.264 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.264 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:51.264 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695143+0000", 2024-09-21T15:49:51.264 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9899999999999999e-06 2024-09-21T15:49:51.264 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.264 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.264 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:51.264 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695190+0000", 2024-09-21T15:49:51.264 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6458000000000001e-05 2024-09-21T15:49:51.264 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.265 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.265 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:49:51.265 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695201+0000", 2024-09-21T15:49:51.265 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0964999999999999e-05 2024-09-21T15:49:51.265 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.265 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.265 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:51.265 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.669490+0000", 2024-09-21T15:49:51.265 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.97428980499999995 2024-09-21T15:49:51.265 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.265 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.265 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-09-21T15:49:51.265 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.670033+0000", 2024-09-21T15:49:51.266 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00054252599999999999 2024-09-21T15:49:51.266 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.266 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.266 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:51.266 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.671095+0000", 2024-09-21T15:49:51.266 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001062183 2024-09-21T15:49:51.266 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.266 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.266 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:51.266 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.671186+0000", 2024-09-21T15:49:51.266 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1191000000000002e-05 2024-09-21T15:49:51.266 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.266 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.266 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-09-21T15:49:51.267 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.671346+0000", 2024-09-21T15:49:51.267 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015980399999999999 2024-09-21T15:49:51.267 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.267 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.267 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:51.267 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.672221+0000", 2024-09-21T15:49:51.267 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00087471999999999995 2024-09-21T15:49:51.267 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.267 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.267 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:51.267 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.675619+0000", 2024-09-21T15:49:51.267 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00339858 2024-09-21T15:49:51.268 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.268 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.268 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:51.268 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.675670+0000", 2024-09-21T15:49:51.268 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0664000000000001e-05 2024-09-21T15:49:51.268 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.268 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.268 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:51.268 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.675768+0000", 2024-09-21T15:49:51.268 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7415000000000002e-05 2024-09-21T15:49:51.268 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:51.268 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:51.268 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:51.268 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.269 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.269 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28537 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 409600~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:51.269 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.227783+0000", 2024-09-21T15:49:51.269 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.031464709000002, 2024-09-21T15:49:51.269 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.046702573999999997, 2024-09-21T15:49:51.269 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:51.269 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:51.269 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:51.269 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:51.269 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:51.269 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28537 2024-09-21T15:49:51.269 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.269 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:51.269 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.269 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:51.270 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227783+0000", 2024-09-21T15:49:51.270 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:51.270 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.270 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.270 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:51.270 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227783+0000", 2024-09-21T15:49:51.270 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:51.270 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.270 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.270 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:51.270 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227782+0000", 2024-09-21T15:49:51.270 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:51.270 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.270 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.271 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:51.271 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227785+0000", 2024-09-21T15:49:51.271 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.34e-06 2024-09-21T15:49:51.271 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.271 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.271 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:51.271 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227791+0000", 2024-09-21T15:49:51.271 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2020000000000001e-06 2024-09-21T15:49:51.271 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.271 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.271 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:51.271 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227796+0000", 2024-09-21T15:49:51.271 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8289999999999998e-06 2024-09-21T15:49:51.271 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.272 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.272 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:51.272 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.274387+0000", 2024-09-21T15:49:51.272 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.046591024000000002 2024-09-21T15:49:51.272 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.272 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.272 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:51.272 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.274427+0000", 2024-09-21T15:49:51.272 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0825000000000003e-05 2024-09-21T15:49:51.272 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.272 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.272 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:51.272 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.274485+0000", 2024-09-21T15:49:51.272 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7713000000000001e-05 2024-09-21T15:49:51.272 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:51.273 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:51.273 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:51.273 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.273 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.273 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28538 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 1212416~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:51.273 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.227800+0000", 2024-09-21T15:49:51.273 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.031447327999999, 2024-09-21T15:49:51.273 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.046874213999999997, 2024-09-21T15:49:51.273 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:51.273 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:51.273 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:51.273 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:51.273 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:51.273 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28538 2024-09-21T15:49:51.273 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.274 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:51.274 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.274 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:51.274 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227800+0000", 2024-09-21T15:49:51.274 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:51.274 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.274 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.274 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:51.274 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227800+0000", 2024-09-21T15:49:51.274 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:51.274 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.274 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.274 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:51.275 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227800+0000", 2024-09-21T15:49:51.275 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:51.275 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.275 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.275 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:51.275 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227809+0000", 2024-09-21T15:49:51.275 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4709999999999996e-06 2024-09-21T15:49:51.275 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.275 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.275 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:51.275 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227816+0000", 2024-09-21T15:49:51.275 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0809999999999996e-06 2024-09-21T15:49:51.275 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.275 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.276 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:51.276 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227821+0000", 2024-09-21T15:49:51.276 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9420000000000003e-06 2024-09-21T15:49:51.276 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.276 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.276 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:51.276 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.274515+0000", 2024-09-21T15:49:51.276 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.046694201999999997 2024-09-21T15:49:51.276 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.276 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.276 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:51.276 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.274585+0000", 2024-09-21T15:49:51.276 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9250000000000003e-05 2024-09-21T15:49:51.276 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.276 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.277 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:51.277 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.274674+0000", 2024-09-21T15:49:51.277 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9621000000000001e-05 2024-09-21T15:49:51.277 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:51.277 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:51.277 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:51.277 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.277 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.277 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28539 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 2015232~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:51.277 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.227826+0000", 2024-09-21T15:49:51.277 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.031421326, 2024-09-21T15:49:51.277 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.046969221999999998, 2024-09-21T15:49:51.277 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:51.277 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:51.277 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:51.278 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:51.278 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:51.278 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28539 2024-09-21T15:49:51.278 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.278 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:51.278 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.278 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:51.278 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227826+0000", 2024-09-21T15:49:51.278 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:51.278 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.278 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.278 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:51.278 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227826+0000", 2024-09-21T15:49:51.279 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:51.279 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.279 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.279 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:51.279 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227826+0000", 2024-09-21T15:49:51.279 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:51.279 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.279 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.279 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:51.279 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227828+0000", 2024-09-21T15:49:51.279 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4040000000000002e-06 2024-09-21T15:49:51.279 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.279 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.279 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:51.280 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227835+0000", 2024-09-21T15:49:51.280 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6100000000000002e-06 2024-09-21T15:49:51.280 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.280 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.280 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:51.280 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227839+0000", 2024-09-21T15:49:51.280 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7589999999999997e-06 2024-09-21T15:49:51.280 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.280 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.280 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:51.280 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.274701+0000", 2024-09-21T15:49:51.280 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.046861828000000001 2024-09-21T15:49:51.280 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.280 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.281 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:51.281 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.274742+0000", 2024-09-21T15:49:51.281 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1074e-05 2024-09-21T15:49:51.281 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.281 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.281 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:51.281 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.274795+0000", 2024-09-21T15:49:51.281 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2945999999999998e-05 2024-09-21T15:49:51.281 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:51.281 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:51.281 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:51.281 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.281 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.281 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28540 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 2760704~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:51.282 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.227844+0000", 2024-09-21T15:49:51.282 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.031403316999999, 2024-09-21T15:49:51.282 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047146018999999997, 2024-09-21T15:49:51.282 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:51.282 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:51.282 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:51.282 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:51.282 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:51.282 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28540 2024-09-21T15:49:51.282 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.282 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:51.282 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.282 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:51.282 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227844+0000", 2024-09-21T15:49:51.283 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:51.283 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.283 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.283 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:51.283 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227844+0000", 2024-09-21T15:49:51.283 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:51.283 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.283 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.283 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:51.283 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227844+0000", 2024-09-21T15:49:51.283 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:51.283 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.283 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.283 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:51.283 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227846+0000", 2024-09-21T15:49:51.284 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1959999999999998e-06 2024-09-21T15:49:51.284 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.284 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.284 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:51.284 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227852+0000", 2024-09-21T15:49:51.284 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1659999999999999e-06 2024-09-21T15:49:51.284 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.284 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.284 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:51.284 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227856+0000", 2024-09-21T15:49:51.284 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4789999999999998e-06 2024-09-21T15:49:51.284 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.284 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.284 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:51.285 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.274825+0000", 2024-09-21T15:49:51.285 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.046968325999999998 2024-09-21T15:49:51.285 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.285 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.285 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:51.285 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.274896+0000", 2024-09-21T15:49:51.285 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0859999999999996e-05 2024-09-21T15:49:51.285 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.285 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.285 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:51.285 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.274990+0000", 2024-09-21T15:49:51.285 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4345999999999994e-05 2024-09-21T15:49:51.285 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:51.285 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:51.285 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:51.286 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.286 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.286 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28541 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 3563520~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:51.286 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.227861+0000", 2024-09-21T15:49:51.286 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.031386328, 2024-09-21T15:49:51.286 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047277444000000002, 2024-09-21T15:49:51.286 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:51.286 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:51.286 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:51.286 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:51.286 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:51.286 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28541 2024-09-21T15:49:51.286 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.287 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:51.287 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.287 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:51.287 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227861+0000", 2024-09-21T15:49:51.287 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:51.287 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.287 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.287 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:51.287 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227861+0000", 2024-09-21T15:49:51.287 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:51.287 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.287 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.287 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:51.287 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227861+0000", 2024-09-21T15:49:51.287 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:51.288 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.288 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.288 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:51.288 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227863+0000", 2024-09-21T15:49:51.288 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0789999999999999e-06 2024-09-21T15:49:51.288 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.288 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.288 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:51.288 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227870+0000", 2024-09-21T15:49:51.288 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8070000000000002e-06 2024-09-21T15:49:51.288 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.288 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.288 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:51.288 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227874+0000", 2024-09-21T15:49:51.289 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9599999999999999e-06 2024-09-21T15:49:51.289 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.289 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.289 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:51.289 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275002+0000", 2024-09-21T15:49:51.289 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047127251000000002 2024-09-21T15:49:51.289 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.289 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.289 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:51.289 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275049+0000", 2024-09-21T15:49:51.289 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7592000000000001e-05 2024-09-21T15:49:51.289 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.289 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.289 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:51.290 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275138+0000", 2024-09-21T15:49:51.290 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9104000000000006e-05 2024-09-21T15:49:51.290 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:51.290 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:51.290 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:51.290 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.290 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.290 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28542 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 4366336~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:51.290 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.227879+0000", 2024-09-21T15:49:51.290 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.031368367999999, 2024-09-21T15:49:51.290 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047394002999999997, 2024-09-21T15:49:51.290 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:51.290 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:51.290 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:51.291 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:51.291 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:51.291 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28542 2024-09-21T15:49:51.291 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.291 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:51.291 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.291 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:51.291 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227879+0000", 2024-09-21T15:49:51.291 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:51.291 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.291 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.291 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:51.291 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227879+0000", 2024-09-21T15:49:51.291 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:51.292 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.292 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.292 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:51.292 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227878+0000", 2024-09-21T15:49:51.292 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:51.292 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.292 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.292 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:51.292 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227881+0000", 2024-09-21T15:49:51.292 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2340000000000001e-06 2024-09-21T15:49:51.292 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.292 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.292 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:51.292 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227887+0000", 2024-09-21T15:49:51.292 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.049e-06 2024-09-21T15:49:51.293 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.293 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.293 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:51.293 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227891+0000", 2024-09-21T15:49:51.293 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5480000000000003e-06 2024-09-21T15:49:51.293 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.293 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.293 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:51.293 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275168+0000", 2024-09-21T15:49:51.293 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047276986999999999 2024-09-21T15:49:51.293 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.293 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.293 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:51.293 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275219+0000", 2024-09-21T15:49:51.294 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1010000000000001e-05 2024-09-21T15:49:51.294 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.294 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.294 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:51.294 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275273+0000", 2024-09-21T15:49:51.294 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3649000000000001e-05 2024-09-21T15:49:51.294 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:51.294 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:51.294 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:51.294 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.294 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.294 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28544 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 5111808~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:51.294 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.227895+0000", 2024-09-21T15:49:51.294 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.03135211, 2024-09-21T15:49:51.295 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047502682999999997, 2024-09-21T15:49:51.295 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:51.295 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:51.295 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:51.295 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:51.295 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:51.295 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28544 2024-09-21T15:49:51.295 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.295 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:51.295 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.295 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:51.295 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227895+0000", 2024-09-21T15:49:51.295 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:51.295 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.296 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.296 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:51.296 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227895+0000", 2024-09-21T15:49:51.296 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:51.296 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.296 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.296 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:51.296 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227895+0000", 2024-09-21T15:49:51.296 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:51.296 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.296 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.296 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:51.296 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227897+0000", 2024-09-21T15:49:51.296 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.153e-06 2024-09-21T15:49:51.297 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.297 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.297 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:51.297 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227903+0000", 2024-09-21T15:49:51.297 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1940000000000003e-06 2024-09-21T15:49:51.297 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.297 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.297 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:51.297 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227908+0000", 2024-09-21T15:49:51.297 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6110000000000004e-06 2024-09-21T15:49:51.297 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.297 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.297 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:51.297 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275290+0000", 2024-09-21T15:49:51.297 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047381805999999999 2024-09-21T15:49:51.298 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.298 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.298 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:51.298 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275346+0000", 2024-09-21T15:49:51.298 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6313000000000001e-05 2024-09-21T15:49:51.298 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.298 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.298 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:51.298 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275398+0000", 2024-09-21T15:49:51.298 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2027000000000002e-05 2024-09-21T15:49:51.298 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:51.298 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:51.298 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:51.298 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.299 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.299 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28545 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 5169152~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:51.299 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.227912+0000", 2024-09-21T15:49:51.299 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.031335489, 2024-09-21T15:49:51.299 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047684807000000003, 2024-09-21T15:49:51.299 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:51.299 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:51.299 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:51.299 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:51.299 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:51.299 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28545 2024-09-21T15:49:51.299 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.299 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:51.300 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.300 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:51.300 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227912+0000", 2024-09-21T15:49:51.300 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:51.300 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.300 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.300 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:51.300 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227912+0000", 2024-09-21T15:49:51.300 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:51.300 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.300 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.300 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:51.300 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227911+0000", 2024-09-21T15:49:51.300 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:51.300 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.301 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.301 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:51.301 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227913+0000", 2024-09-21T15:49:51.301 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9970000000000001e-06 2024-09-21T15:49:51.301 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.301 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.301 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:51.301 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227920+0000", 2024-09-21T15:49:51.301 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0499999999999997e-06 2024-09-21T15:49:51.301 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.301 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.301 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:51.301 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227924+0000", 2024-09-21T15:49:51.301 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7219999999999999e-06 2024-09-21T15:49:51.301 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.302 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.302 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:51.302 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275424+0000", 2024-09-21T15:49:51.302 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047499574000000003 2024-09-21T15:49:51.302 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.302 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.302 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:51.302 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275495+0000", 2024-09-21T15:49:51.302 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1347999999999995e-05 2024-09-21T15:49:51.302 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.302 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.302 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:51.302 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275597+0000", 2024-09-21T15:49:51.303 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101446 2024-09-21T15:49:51.303 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:51.303 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:51.303 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:51.303 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.303 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.303 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28546 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 5914624~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:51.303 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.227929+0000", 2024-09-21T15:49:51.303 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.031318243000001, 2024-09-21T15:49:51.303 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047783297000000002, 2024-09-21T15:49:51.303 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:51.303 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:51.303 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:51.304 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:51.304 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:51.304 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28546 2024-09-21T15:49:51.304 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.304 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:51.304 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.304 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:51.304 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227929+0000", 2024-09-21T15:49:51.304 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:51.304 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.304 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.304 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:51.304 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227929+0000", 2024-09-21T15:49:51.304 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:51.304 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.305 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.305 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:51.305 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227929+0000", 2024-09-21T15:49:51.305 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:51.305 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.305 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.305 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:51.305 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227931+0000", 2024-09-21T15:49:51.305 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2529999999999999e-06 2024-09-21T15:49:51.305 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.305 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.305 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:51.305 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227937+0000", 2024-09-21T15:49:51.305 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9699999999999996e-06 2024-09-21T15:49:51.306 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.306 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.306 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:51.306 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227942+0000", 2024-09-21T15:49:51.306 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7650000000000001e-06 2024-09-21T15:49:51.306 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.306 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.306 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:51.306 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275621+0000", 2024-09-21T15:49:51.306 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047679694000000002 2024-09-21T15:49:51.306 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.306 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.306 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:51.306 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275661+0000", 2024-09-21T15:49:51.307 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9363999999999998e-05 2024-09-21T15:49:51.307 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.307 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.307 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:51.307 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275712+0000", 2024-09-21T15:49:51.307 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1598999999999998e-05 2024-09-21T15:49:51.307 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:51.307 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:51.307 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:51.307 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.307 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.307 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28547 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 6717440~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:51.307 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.228080+0000", 2024-09-21T15:49:51.308 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.031166966000001, 2024-09-21T15:49:51.308 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047744609, 2024-09-21T15:49:51.308 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:51.308 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:51.308 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:51.308 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:51.308 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:51.308 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28547 2024-09-21T15:49:51.308 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.308 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:51.308 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.308 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:51.308 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228080+0000", 2024-09-21T15:49:51.308 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:51.309 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.309 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.309 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:51.309 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228080+0000", 2024-09-21T15:49:51.309 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:51.309 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.309 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.309 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:51.309 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227946+0000", 2024-09-21T15:49:51.309 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999866 2024-09-21T15:49:51.309 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.309 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.309 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:51.310 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228084+0000", 2024-09-21T15:49:51.310 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000137517 2024-09-21T15:49:51.310 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.310 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.310 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:51.310 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228093+0000", 2024-09-21T15:49:51.310 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.645e-06 2024-09-21T15:49:51.310 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.310 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.310 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:51.310 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228101+0000", 2024-09-21T15:49:51.310 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9559999999999997e-06 2024-09-21T15:49:51.310 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.310 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.311 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:51.311 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275735+0000", 2024-09-21T15:49:51.311 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047633928999999998 2024-09-21T15:49:51.311 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.311 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.311 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:51.311 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275774+0000", 2024-09-21T15:49:51.311 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8699999999999999e-05 2024-09-21T15:49:51.311 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.311 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.311 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:51.311 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275825+0000", 2024-09-21T15:49:51.311 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0926e-05 2024-09-21T15:49:51.312 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:51.312 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:51.312 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:51.312 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.312 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.312 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28549 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 417792~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:51.312 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.228847+0000", 2024-09-21T15:49:51.312 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.030399860999999, 2024-09-21T15:49:51.312 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047086515000000002, 2024-09-21T15:49:51.312 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:51.312 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:51.312 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:51.312 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:51.312 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:51.313 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28549 2024-09-21T15:49:51.313 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.313 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:51.313 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.313 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:51.313 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228847+0000", 2024-09-21T15:49:51.313 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:51.313 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.313 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.313 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:51.313 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228847+0000", 2024-09-21T15:49:51.313 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:51.313 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.313 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.313 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:51.314 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228107+0000", 2024-09-21T15:49:51.314 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9992595 2024-09-21T15:49:51.314 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.314 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.314 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:51.314 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228852+0000", 2024-09-21T15:49:51.314 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00074505000000000003 2024-09-21T15:49:51.314 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.314 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.314 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:51.314 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228871+0000", 2024-09-21T15:49:51.314 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8635e-05 2024-09-21T15:49:51.314 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.314 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.315 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:51.315 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228880+0000", 2024-09-21T15:49:51.315 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9849999999999995e-06 2024-09-21T15:49:51.315 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.315 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.315 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:51.315 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275847+0000", 2024-09-21T15:49:51.315 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.046967239000000001 2024-09-21T15:49:51.315 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.315 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.315 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:51.315 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275885+0000", 2024-09-21T15:49:51.315 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8274e-05 2024-09-21T15:49:51.315 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.316 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.316 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:51.316 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275934+0000", 2024-09-21T15:49:51.316 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8764999999999998e-05 2024-09-21T15:49:51.316 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:51.316 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:51.316 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:51.316 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.316 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.316 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28551 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 1220608~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:51.316 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.228886+0000", 2024-09-21T15:49:51.316 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.030360815000002, 2024-09-21T15:49:51.316 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047166819999999998, 2024-09-21T15:49:51.316 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:51.317 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:51.317 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:51.317 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:51.317 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:51.317 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28551 2024-09-21T15:49:51.317 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.317 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:51.317 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.317 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:51.317 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228886+0000", 2024-09-21T15:49:51.317 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:51.317 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.317 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.317 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:51.318 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228886+0000", 2024-09-21T15:49:51.318 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:51.318 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.318 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.318 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:51.318 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228886+0000", 2024-09-21T15:49:51.318 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:51.318 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.318 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.318 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:51.319 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228889+0000", 2024-09-21T15:49:51.319 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8650000000000001e-06 2024-09-21T15:49:51.319 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.319 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.319 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:51.319 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228896+0000", 2024-09-21T15:49:51.319 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0790000000000003e-06 2024-09-21T15:49:51.319 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.319 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.319 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:51.319 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228903+0000", 2024-09-21T15:49:51.319 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9770000000000003e-06 2024-09-21T15:49:51.319 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.320 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.320 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:51.320 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275954+0000", 2024-09-21T15:49:51.320 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047050986000000003 2024-09-21T15:49:51.320 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.320 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.320 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:51.320 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276001+0000", 2024-09-21T15:49:51.320 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7527000000000001e-05 2024-09-21T15:49:51.320 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.320 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.320 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:51.320 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276053+0000", 2024-09-21T15:49:51.320 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1854999999999997e-05 2024-09-21T15:49:51.321 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:51.321 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:51.321 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:51.321 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.321 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.321 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28552 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 1966080~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:51.321 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.228908+0000", 2024-09-21T15:49:51.321 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.030339529999999, 2024-09-21T15:49:51.321 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047256193000000002, 2024-09-21T15:49:51.321 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:51.321 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:51.321 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:51.321 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:51.321 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:51.322 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28552 2024-09-21T15:49:51.322 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.322 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:51.322 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.322 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:51.322 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228908+0000", 2024-09-21T15:49:51.322 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:51.322 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.322 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.322 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:51.322 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228908+0000", 2024-09-21T15:49:51.322 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:51.322 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.322 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.323 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:51.323 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228907+0000", 2024-09-21T15:49:51.323 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:51.323 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.323 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.323 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:51.323 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228910+0000", 2024-09-21T15:49:51.323 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3819999999999998e-06 2024-09-21T15:49:51.323 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.323 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.323 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:51.323 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228916+0000", 2024-09-21T15:49:51.323 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6390000000000003e-06 2024-09-21T15:49:51.323 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.324 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.324 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:51.324 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228921+0000", 2024-09-21T15:49:51.324 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0350000000000001e-06 2024-09-21T15:49:51.324 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.324 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.324 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:51.324 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276077+0000", 2024-09-21T15:49:51.324 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047155326999999997 2024-09-21T15:49:51.324 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.324 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.324 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:51.324 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276114+0000", 2024-09-21T15:49:51.324 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6746999999999999e-05 2024-09-21T15:49:51.325 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.325 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.325 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:51.325 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276164+0000", 2024-09-21T15:49:51.325 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0438000000000001e-05 2024-09-21T15:49:51.325 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:51.325 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:51.325 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:51.325 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.325 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.325 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28553 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 2023424~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:51.325 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.228926+0000", 2024-09-21T15:49:51.325 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.030321323999999, 2024-09-21T15:49:51.325 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047351920999999998, 2024-09-21T15:49:51.326 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:51.326 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:51.326 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:51.326 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:51.326 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:51.326 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28553 2024-09-21T15:49:51.326 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.326 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:51.326 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.326 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:51.326 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228926+0000", 2024-09-21T15:49:51.326 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:51.326 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.326 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.327 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:51.327 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228926+0000", 2024-09-21T15:49:51.327 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:51.327 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.327 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.327 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:51.327 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228926+0000", 2024-09-21T15:49:51.327 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:51.327 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.327 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.327 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:51.327 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228932+0000", 2024-09-21T15:49:51.327 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6930000000000001e-06 2024-09-21T15:49:51.327 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.327 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.328 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:51.328 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228939+0000", 2024-09-21T15:49:51.328 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8199999999999999e-06 2024-09-21T15:49:51.328 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.328 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.328 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:51.328 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228953+0000", 2024-09-21T15:49:51.328 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.347e-05 2024-09-21T15:49:51.328 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.328 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.328 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:51.328 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276187+0000", 2024-09-21T15:49:51.328 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047234405 2024-09-21T15:49:51.328 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.329 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.329 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:51.329 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276228+0000", 2024-09-21T15:49:51.329 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1010000000000002e-05 2024-09-21T15:49:51.329 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.329 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.329 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:51.329 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276278+0000", 2024-09-21T15:49:51.329 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9874999999999999e-05 2024-09-21T15:49:51.329 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:51.329 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:51.329 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:51.329 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.329 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.330 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28554 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 2768896~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:51.330 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.228991+0000", 2024-09-21T15:49:51.330 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.030255846999999, 2024-09-21T15:49:51.330 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047394177000000003, 2024-09-21T15:49:51.330 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:51.330 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:51.330 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:51.330 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:51.330 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:51.330 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28554 2024-09-21T15:49:51.330 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.330 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:51.330 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.331 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:51.331 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228991+0000", 2024-09-21T15:49:51.331 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:51.331 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.331 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.331 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:51.331 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228991+0000", 2024-09-21T15:49:51.331 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:51.331 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.331 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.331 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:51.331 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228991+0000", 2024-09-21T15:49:51.331 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:51.331 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.331 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.332 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:51.332 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228994+0000", 2024-09-21T15:49:51.332 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.422e-06 2024-09-21T15:49:51.332 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.332 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.332 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:51.332 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229003+0000", 2024-09-21T15:49:51.332 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8699999999999998e-06 2024-09-21T15:49:51.332 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.332 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.332 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:51.332 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229009+0000", 2024-09-21T15:49:51.332 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3099999999999997e-06 2024-09-21T15:49:51.332 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.333 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.333 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:51.333 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276301+0000", 2024-09-21T15:49:51.333 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047291305999999998 2024-09-21T15:49:51.333 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.333 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.333 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:51.333 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276337+0000", 2024-09-21T15:49:51.333 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6378999999999998e-05 2024-09-21T15:49:51.333 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.333 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.333 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:51.333 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276386+0000", 2024-09-21T15:49:51.333 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8535999999999997e-05 2024-09-21T15:49:51.334 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:51.334 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:51.334 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:51.334 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.334 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.334 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28555 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 3571712~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:51.334 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.229015+0000", 2024-09-21T15:49:51.334 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.030232753, 2024-09-21T15:49:51.334 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047478713999999998, 2024-09-21T15:49:51.334 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:51.334 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:51.334 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:51.334 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:51.334 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:51.335 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28555 2024-09-21T15:49:51.335 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.335 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:51.335 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.335 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:51.335 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229015+0000", 2024-09-21T15:49:51.335 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:51.335 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.335 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.335 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:51.335 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229015+0000", 2024-09-21T15:49:51.335 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:51.335 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.335 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.336 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:51.336 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229014+0000", 2024-09-21T15:49:51.336 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:51.336 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.336 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.336 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:51.336 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229029+0000", 2024-09-21T15:49:51.336 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4503e-05 2024-09-21T15:49:51.336 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.336 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.336 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:51.336 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229036+0000", 2024-09-21T15:49:51.336 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5989999999999996e-06 2024-09-21T15:49:51.336 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.337 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.337 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:51.337 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229042+0000", 2024-09-21T15:49:51.337 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6699999999999999e-06 2024-09-21T15:49:51.337 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.337 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.337 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:51.337 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276407+0000", 2024-09-21T15:49:51.337 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047365646999999997 2024-09-21T15:49:51.337 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.337 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.337 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:51.337 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276445+0000", 2024-09-21T15:49:51.337 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7064000000000003e-05 2024-09-21T15:49:51.337 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.338 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.338 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:51.338 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276493+0000", 2024-09-21T15:49:51.338 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8704e-05 2024-09-21T15:49:51.338 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:51.338 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:51.338 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:51.338 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.338 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.338 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28556 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 4374528~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:51.338 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.229183+0000", 2024-09-21T15:49:51.338 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.030063962, 2024-09-21T15:49:51.338 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047417709000000002, 2024-09-21T15:49:51.339 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:51.339 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:51.339 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:51.339 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:51.339 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:51.339 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28556 2024-09-21T15:49:51.339 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.339 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:51.339 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.339 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:51.339 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229183+0000", 2024-09-21T15:49:51.339 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:51.339 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.339 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.340 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:51.340 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229183+0000", 2024-09-21T15:49:51.340 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:51.340 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.340 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.340 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:51.340 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229046+0000", 2024-09-21T15:49:51.340 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9998631 2024-09-21T15:49:51.340 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.340 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.340 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:51.340 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229186+0000", 2024-09-21T15:49:51.340 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014003900000000001 2024-09-21T15:49:51.340 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.340 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.341 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:51.341 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229195+0000", 2024-09-21T15:49:51.341 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.422e-06 2024-09-21T15:49:51.341 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.341 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.341 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:51.341 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229199+0000", 2024-09-21T15:49:51.341 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5379999999999996e-06 2024-09-21T15:49:51.341 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.341 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.341 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:51.341 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276516+0000", 2024-09-21T15:49:51.341 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047316432999999998 2024-09-21T15:49:51.341 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.342 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.342 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:51.342 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276552+0000", 2024-09-21T15:49:51.342 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6019000000000002e-05 2024-09-21T15:49:51.342 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.342 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.342 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:51.342 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276601+0000", 2024-09-21T15:49:51.342 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9172999999999998e-05 2024-09-21T15:49:51.342 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:51.342 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:51.342 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:51.342 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.342 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.343 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28558 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 5120000~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:51.343 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.229531+0000", 2024-09-21T15:49:51.343 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.029716079, 2024-09-21T15:49:51.343 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047179666000000002, 2024-09-21T15:49:51.343 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:51.343 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:51.343 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:51.343 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:51.343 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:51.343 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28558 2024-09-21T15:49:51.343 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.343 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:51.343 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.343 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:51.344 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229531+0000", 2024-09-21T15:49:51.344 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:51.344 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.344 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.344 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:51.344 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229531+0000", 2024-09-21T15:49:51.344 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:51.344 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.344 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.344 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:51.344 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229203+0000", 2024-09-21T15:49:51.344 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9996719 2024-09-21T15:49:51.344 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.344 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.345 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:51.345 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229536+0000", 2024-09-21T15:49:51.345 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000332885 2024-09-21T15:49:51.345 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.345 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.345 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:51.345 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229548+0000", 2024-09-21T15:49:51.345 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1861999999999999e-05 2024-09-21T15:49:51.345 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.345 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.345 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:51.345 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229555+0000", 2024-09-21T15:49:51.345 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6009999999999997e-06 2024-09-21T15:49:51.345 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.345 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.346 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:51.346 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276624+0000", 2024-09-21T15:49:51.346 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047068250999999998 2024-09-21T15:49:51.346 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.346 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.346 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:51.346 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276662+0000", 2024-09-21T15:49:51.346 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8705999999999998e-05 2024-09-21T15:49:51.346 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.346 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.346 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:51.346 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276711+0000", 2024-09-21T15:49:51.346 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8443999999999999e-05 2024-09-21T15:49:51.346 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:51.347 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:51.347 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:51.347 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.347 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.347 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28560 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 5922816~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:51.347 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.229562+0000", 2024-09-21T15:49:51.347 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 24.029684874000001, 2024-09-21T15:49:51.347 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047255377000000001, 2024-09-21T15:49:51.347 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:51.347 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:51.347 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:51.347 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:51.347 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:51.347 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28560 2024-09-21T15:49:51.348 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.348 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:51.348 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.348 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:51.348 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229562+0000", 2024-09-21T15:49:51.348 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:51.348 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.348 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.348 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:51.348 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229562+0000", 2024-09-21T15:49:51.348 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:51.348 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.348 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.348 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:51.348 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229562+0000", 2024-09-21T15:49:51.349 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:51.349 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.349 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.349 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:51.349 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229565+0000", 2024-09-21T15:49:51.349 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9450000000000002e-06 2024-09-21T15:49:51.349 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.349 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.349 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:51.349 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229572+0000", 2024-09-21T15:49:51.349 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4240000000000004e-06 2024-09-21T15:49:51.349 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.349 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.349 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:51.350 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229577+0000", 2024-09-21T15:49:51.350 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2000000000000002e-06 2024-09-21T15:49:51.350 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.350 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.350 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:51.350 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276733+0000", 2024-09-21T15:49:51.350 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047155202 2024-09-21T15:49:51.350 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.350 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.350 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:51.350 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276769+0000", 2024-09-21T15:49:51.350 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6013000000000002e-05 2024-09-21T15:49:51.350 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:51.350 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:51.351 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:51.351 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276818+0000", 2024-09-21T15:49:51.351 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9178000000000002e-05 2024-09-21T15:49:51.351 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:51.351 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:51.351 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:51.351 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:51.351 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:51.351 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:51.351 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:49:51.352 INFO:teuthology.orchestra.run.smithi089.stdout:3740 write 0x6487f85 thru 0x648ba82 (0x3afe bytes) 2024-09-21T15:49:51.352 INFO:teuthology.orchestra.run.smithi089.stdout:3741 clone 22 order 25 su 262144 sc 3 2024-09-21T15:49:51.353 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:49:51.354 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:51.351+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:51.453 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:51.452+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:51.455 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:51.455 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:51.455 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:49:51.455 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:51.463 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:49:51.511 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-09-21T15:49:51.511 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:49:51.527 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:49:51.528 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:49:51.543 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:49:51.543 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:49:51.554 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:51.553+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:51.558 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:49:51.558 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:49:51.574 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:49:51.575 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:49:51.590 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:49:51.590 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:49:51.605 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:49:51.606 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:49:51.606 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:51.606 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:51.615 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:49:51.621 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:49:51.622 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:49:51.656 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:51.655+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:51.756 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:51.755+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:51.757 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:51.757 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:51.766 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:49:51.857 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:51.856+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:51.908 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:51.908 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:51.917 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:49:51.923 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is unset 2024-09-21T15:49:51.958 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:51.957+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:51.984 INFO:teuthology.orchestra.run.smithi089.stdout:truncating image image_client.0-clone21 from 0x648ba83 (overlap 0x379ef2) to 0x0 2024-09-21T15:49:52.059 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:52.058+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:52.062 INFO:teuthology.orchestra.run.smithi089.stdout:checking clone #20, image image_client.0-clone20 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent21 2024-09-21T15:49:52.068 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:52.068 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:52.070 INFO:teuthology.orchestra.run.smithi089.stdout:3743 trunc from 0x648ba83 to 0x23de79f 2024-09-21T15:49:52.077 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:49:52.105 INFO:teuthology.orchestra.run.smithi089.stdout:3744 trunc from 0x23de79f to 0xdd8cac2 2024-09-21T15:49:52.111 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:52.116 INFO:teuthology.orchestra.run.smithi089.stdout:3746 write 0xdfd80ee thru 0xdfe47ce (0xc6e1 bytes) 2024-09-21T15:49:52.120 INFO:teuthology.orchestra.run.smithi089.stdout:3747 write 0xcff1e6d thru 0xcff7120 (0x52b4 bytes) 2024-09-21T15:49:52.125 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:52.125 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:52.125 INFO:teuthology.orchestra.run.smithi089.stdout:3750 writesame 0xa9f1929 thru 0xa9f80c7 data_size 0x1(0x679f bytes) 2024-09-21T15:49:52.140 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:52.141 INFO:teuthology.orchestra.run.smithi089.stdout:3752 read 0x45b0513 thru 0x45b1836 (0x1324 bytes) 2024-09-21T15:49:52.141 INFO:teuthology.orchestra.run.smithi089.stdout:3753 writesame 0xc20052f thru 0xc20a663 data_size 0x1(0xa135 bytes) 2024-09-21T15:49:52.161 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:52.160+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:52.163 INFO:teuthology.orchestra.run.smithi089.stdout:3754 read 0x189fc9c thru 0x18a0cb8 (0x101d bytes) 2024-09-21T15:49:52.168 INFO:teuthology.orchestra.run.smithi089.stdout:3756 trunc from 0xdfe47cf to 0xa24077d 2024-09-21T15:49:52.181 INFO:teuthology.orchestra.run.smithi089.stdout:3757 write 0x40feab1 thru 0x410b9ea (0xcf3a bytes) 2024-09-21T15:49:52.185 INFO:teuthology.orchestra.run.smithi089.stdout:3759 write 0x62d8d87 thru 0x62e7a63 (0xecdd bytes) 2024-09-21T15:49:52.188 INFO:teuthology.orchestra.run.smithi089.stdout:3761 write 0x1c61194 thru 0x1c638d0 (0x273d bytes) 2024-09-21T15:49:52.189 INFO:teuthology.orchestra.run.smithi089.stdout:3762 write 0x505d7ac thru 0x506b583 (0xddd8 bytes) 2024-09-21T15:49:52.191 INFO:teuthology.orchestra.run.smithi089.stdout:3763 write 0x108e949 thru 0x109c447 (0xdaff bytes) 2024-09-21T15:49:52.199 INFO:teuthology.orchestra.run.smithi089.stdout:3765 writesame 0xa2de5eb thru 0xa2e9735 data_size 0x29(0xb14b bytes) 2024-09-21T15:49:52.199 INFO:teuthology.orchestra.run.smithi089.stdout:3766 write 0x28e0dd2 thru 0x28eb818 (0xaa47 bytes) 2024-09-21T15:49:52.210 INFO:teuthology.orchestra.run.smithi089.stdout:3767 writesame 0xca01d2b thru 0xca0de2a data_size 0x40(0xc100 bytes) 2024-09-21T15:49:52.210 INFO:teuthology.orchestra.run.smithi089.stdout:3768 read 0x7102c64 thru 0x7111d8e (0xf12b bytes) 2024-09-21T15:49:52.210 INFO:teuthology.orchestra.run.smithi089.stdout:3769 read 0x58a0907 thru 0x58a778f (0x6e89 bytes) 2024-09-21T15:49:52.210 INFO:teuthology.orchestra.run.smithi089.stdout:3770 write 0x9fe25d9 thru 0x9fe4aaa (0x24d2 bytes) 2024-09-21T15:49:52.213 INFO:teuthology.orchestra.run.smithi089.stdout:3771 read 0x8cf7b52 thru 0x8cfdcdf (0x618e bytes) 2024-09-21T15:49:52.214 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:52.214 INFO:teuthology.orchestra.run.smithi089.stdout:3775 punch from 0x45af673 to 0x45b6078, (0x6a05 bytes) 2024-09-21T15:49:52.214 INFO:teuthology.orchestra.run.smithi089.stdout:3776 read 0x96d328f thru 0x96d4104 (0xe76 bytes) 2024-09-21T15:49:52.225 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:52.226 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:52.227 INFO:teuthology.orchestra.run.smithi089.stdout:3777 trunc from 0xca0de2b to 0x1368558 2024-09-21T15:49:52.235 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T15:49:52.261 INFO:teuthology.orchestra.run.smithi089.stdout:3779 trunc from 0x1368558 to 0xe3a0fbc 2024-09-21T15:49:52.272 INFO:teuthology.orchestra.run.smithi089.stdout:3780 write 0x65f2faf thru 0x6600bdd (0xdc2f bytes) 2024-09-21T15:49:52.276 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:52.275+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:52.281 INFO:teuthology.orchestra.run.smithi089.stdout:3782 read 0x8081fb thru 0x81403b (0xbe41 bytes) 2024-09-21T15:49:52.282 INFO:teuthology.orchestra.run.smithi089.stdout:3783 punch from 0xc17d823 to 0xc18a8c3, (0xd0a0 bytes) 2024-09-21T15:49:52.282 INFO:teuthology.orchestra.run.smithi089.stdout:3784 write 0x1aeffc0 thru 0x1af29e5 (0x2a26 bytes) 2024-09-21T15:49:52.288 INFO:teuthology.orchestra.run.smithi089.stdout:3785 trunc from 0xe3a0fbc to 0xafbbf15 2024-09-21T15:49:52.301 INFO:teuthology.orchestra.run.smithi089.stdout:3786 punch from 0x4b51720 to 0x4b553be, (0x3c9e bytes) 2024-09-21T15:49:52.301 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:52.301 INFO:teuthology.orchestra.run.smithi089.stdout:3791 writesame 0x129c5db thru 0x12a51f5 data_size 0x1f(0x8c1b bytes) 2024-09-21T15:49:52.313 INFO:teuthology.orchestra.run.smithi089.stdout:3792 write 0xdebcbf9 thru 0xdebe54f (0x1957 bytes) 2024-09-21T15:49:52.317 INFO:teuthology.orchestra.run.smithi089.stdout:3793 writesame 0x2ce39e7 thru 0x2cf30b2 data_size 0xc3(0xf6cc bytes) 2024-09-21T15:49:52.318 INFO:teuthology.orchestra.run.smithi089.stdout:3794 write 0xdaee984 thru 0xdafcde8 (0xe465 bytes) 2024-09-21T15:49:52.320 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:52.320 INFO:teuthology.orchestra.run.smithi089.stdout:3796 write 0x421bc8b thru 0x4220046 (0x43bc bytes) 2024-09-21T15:49:52.323 INFO:teuthology.orchestra.run.smithi089.stdout:3797 punch from 0xa239984 to 0xa2492c4, (0xf940 bytes) 2024-09-21T15:49:52.323 INFO:teuthology.orchestra.run.smithi089.stdout:3798 write 0x973fdf6 thru 0x9740542 (0x74d bytes) 2024-09-21T15:49:52.326 INFO:teuthology.orchestra.run.smithi089.stdout:3799 punch from 0x2570ddb to 0x257aac3, (0x9ce8 bytes) 2024-09-21T15:49:52.326 INFO:teuthology.orchestra.run.smithi089.stdout:3800 read 0xa5a1fe8 thru 0xa5a6377 (0x4390 bytes) 2024-09-21T15:49:52.326 INFO:teuthology.orchestra.run.smithi089.stdout:3801 read 0xc87dd75 thru 0xc88da9b (0xfd27 bytes) 2024-09-21T15:49:52.326 INFO:teuthology.orchestra.run.smithi089.stdout:3802 writesame 0xafb6144 thru 0xafbf954 data_size 0x1(0x9811 bytes) 2024-09-21T15:49:52.355 INFO:teuthology.orchestra.run.smithi089.stdout:3803 trunc from 0xdebe550 to 0x9dfb256 2024-09-21T15:49:52.361 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:52.360+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:52.369 INFO:teuthology.orchestra.run.smithi089.stdout:3804 punch from 0x23fcc69 to 0x23fcca6, (0x3d bytes) 2024-09-21T15:49:52.378 INFO:teuthology.orchestra.run.smithi089.stdout:3805 trunc from 0x9dfb256 to 0x28e4324 2024-09-21T15:49:52.390 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:52.391 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:52.400 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T15:49:52.405 INFO:teuthology.orchestra.run.smithi089.stdout:3806 trunc from 0x28e4324 to 0xa749ed7 2024-09-21T15:49:52.412 INFO:teuthology.orchestra.run.smithi089.stdout:3807 write 0x4cd6710 thru 0x4cd9469 (0x2d5a bytes) 2024-09-21T15:49:52.414 INFO:teuthology.orchestra.run.smithi089.stdout:3808 read 0x1fee481 thru 0x1ff962d (0xb1ad bytes) 2024-09-21T15:49:52.419 INFO:teuthology.orchestra.run.smithi089.stdout:3809 write 0xacb2318 thru 0xacb50ce (0x2db7 bytes) 2024-09-21T15:49:52.431 INFO:teuthology.orchestra.run.smithi089.stdout:3810 writesame 0xd9a241d thru 0xd9af425 data_size 0x1(0xd009 bytes) 2024-09-21T15:49:52.431 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:52.431 INFO:teuthology.orchestra.run.smithi089.stdout:3812 punch from 0x7ce8bc5 to 0x7cec95d, (0x3d98 bytes) 2024-09-21T15:49:52.431 INFO:teuthology.orchestra.run.smithi089.stdout:3813 read 0x90e3c2c thru 0x90e5c94 (0x2069 bytes) 2024-09-21T15:49:52.431 INFO:teuthology.orchestra.run.smithi089.stdout:3815 punch from 0xe16ee7 to 0xe198fe, (0x2a17 bytes) 2024-09-21T15:49:52.431 INFO:teuthology.orchestra.run.smithi089.stdout:3819 clone 23 order 20 su 524288 sc 13 2024-09-21T15:49:52.463 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:52.461+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:52.502 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:52.502 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:52.510 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T15:49:52.564 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:52.563+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:52.613 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:52.613 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:52.621 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T15:49:52.665 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:52.663+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:52.723 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:52.723 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:52.732 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T15:49:52.766 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:52.764+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:52.834 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:52.834 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:52.842 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T15:49:52.886 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:52.885+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:52.947 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:52.947 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:52.956 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:49:52.998 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:52.997+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:52.999 INFO:teuthology.orchestra.run.smithi089.stdout:truncating image image_client.0-clone22 from 0xd9af426 (overlap 0x1368558) to 0x3e1aab 2024-09-21T15:49:53.064 INFO:teuthology.orchestra.run.smithi089.stdout:checking clone #21, image image_client.0-clone21 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent22 2024-09-21T15:49:53.066 INFO:teuthology.orchestra.run.smithi089.stdout:3820 read 0x4a871b7 thru 0x4a8ec4b (0x7a95 bytes) 2024-09-21T15:49:53.067 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:53.067 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:53.067 INFO:teuthology.orchestra.run.smithi089.stdout:3822 read 0xd5f772d thru 0xd602977 (0xb24b bytes) 2024-09-21T15:49:53.068 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:53.067+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:53.079 INFO:teuthology.orchestra.run.smithi089.stdout:3823 trunc from 0xd9af426 to 0x43781c4 2024-09-21T15:49:53.080 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:49:53.109 INFO:teuthology.orchestra.run.smithi089.stdout:3826 read 0x849e17 thru 0x8546b2 (0xa89c bytes) 2024-09-21T15:49:53.109 INFO:teuthology.orchestra.run.smithi089.stdout:3827 punch from 0x13eb2f5 to 0x13f6835, (0xb540 bytes) 2024-09-21T15:49:53.109 INFO:teuthology.orchestra.run.smithi089.stdout:3829 punch from 0x19ff13b to 0x19ffc8b, (0xb50 bytes) 2024-09-21T15:49:53.109 INFO:teuthology.orchestra.run.smithi089.stdout:3831 punch from 0x198d2b5 to 0x1994c2a, (0x7975 bytes) 2024-09-21T15:49:53.111 INFO:teuthology.orchestra.run.smithi089.stdout:3832 trunc from 0x43781c4 to 0x20f67a6 2024-09-21T15:49:53.140 INFO:teuthology.orchestra.run.smithi089.stdout:3833 read 0xfcaab7 thru 0xfcad37 (0x281 bytes) 2024-09-21T15:49:53.140 INFO:teuthology.orchestra.run.smithi089.stdout:3834 read 0xaa6879 thru 0xaaca99 (0x6221 bytes) 2024-09-21T15:49:53.148 INFO:teuthology.orchestra.run.smithi089.stdout:3835 write 0x27d9589 thru 0x27e2fb4 (0x9a2c bytes) 2024-09-21T15:49:53.169 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:53.168+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:53.171 INFO:teuthology.orchestra.run.smithi089.stdout:3836 write 0xbb9c5cb thru 0xbba7b0f (0xb545 bytes) 2024-09-21T15:49:53.179 INFO:teuthology.orchestra.run.smithi089.stdout:3837 write 0x5c3f179 thru 0x5c4ed10 (0xfb98 bytes) 2024-09-21T15:49:53.187 INFO:teuthology.orchestra.run.smithi089.stdout:3838 read 0x76a346 thru 0x77247e (0x8139 bytes) 2024-09-21T15:49:53.187 INFO:teuthology.orchestra.run.smithi089.stdout:3839 writesame 0x7f862e1 thru 0x7f8fdf6 data_size 0x27(0x9b16 bytes) 2024-09-21T15:49:53.194 INFO:teuthology.orchestra.run.smithi089.stdout:3840 punch from 0x514ad70 to 0x5152463, (0x76f3 bytes) 2024-09-21T15:49:53.195 INFO:teuthology.orchestra.run.smithi089.stdout:3843 read 0x3cdb32d thru 0x3cdd88b (0x255f bytes) 2024-09-21T15:49:53.195 INFO:teuthology.orchestra.run.smithi089.stdout:3845 punch from 0xaf6520f to 0xaf66862, (0x1653 bytes) 2024-09-21T15:49:53.195 INFO:teuthology.orchestra.run.smithi089.stdout:3847 read 0xf81392 thru 0xf8a6a4 (0x9313 bytes) 2024-09-21T15:49:53.195 INFO:teuthology.orchestra.run.smithi089.stdout:3848 write 0x532b0c0 thru 0x532dc81 (0x2bc2 bytes) 2024-09-21T15:49:53.200 INFO:teuthology.orchestra.run.smithi089.stdout:3850 write 0x665643 thru 0x6673c5 (0x1d83 bytes) 2024-09-21T15:49:53.216 INFO:teuthology.orchestra.run.smithi089.stdout:3851 trunc from 0xbba7b10 to 0x2e91d70 2024-09-21T15:49:53.219 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:53.219 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:53.228 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:49:53.232 INFO:teuthology.orchestra.run.smithi089.stdout:3852 trunc from 0x2e91d70 to 0x282de23 2024-09-21T15:49:53.243 INFO:teuthology.orchestra.run.smithi089.stdout:3854 read 0x13d62bc thru 0x13e4429 (0xe16e bytes) 2024-09-21T15:49:53.243 INFO:teuthology.orchestra.run.smithi089.stdout:3855 read 0x8eefc3 thru 0x8fbd48 (0xcd86 bytes) 2024-09-21T15:49:53.261 INFO:teuthology.orchestra.run.smithi089.stdout:3859 write 0xb31321d thru 0xb31f15f (0xbf43 bytes) 2024-09-21T15:49:53.267 INFO:teuthology.orchestra.run.smithi089.stdout:3860 punch from 0x9b1559f to 0x9b19eaf, (0x4910 bytes) 2024-09-21T15:49:53.270 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:53.268+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:53.273 INFO:teuthology.orchestra.run.smithi089.stdout:3861 writesame 0xd48879a thru 0xd491443 data_size 0x1a(0x8caa bytes) 2024-09-21T15:49:53.274 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:53.289 INFO:teuthology.orchestra.run.smithi089.stdout:3865 trunc from 0xd491444 to 0x2bae0c 2024-09-21T15:49:53.336 INFO:teuthology.orchestra.run.smithi089.stdout:3866 trunc from 0x2bae0c to 0xa7dfa52 2024-09-21T15:49:53.342 INFO:teuthology.orchestra.run.smithi089.stdout:3867 write 0x89fc219 thru 0x89fce53 (0xc3b bytes) 2024-09-21T15:49:53.345 INFO:teuthology.orchestra.run.smithi089.stdout:3868 punch from 0x2c6c49c to 0x2c763e9, (0x9f4d bytes) 2024-09-21T15:49:53.345 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:53.345 INFO:teuthology.orchestra.run.smithi089.stdout:3872 punch from 0x21a767b to 0x21a8ae0, (0x1465 bytes) 2024-09-21T15:49:53.345 INFO:teuthology.orchestra.run.smithi089.stdout:3874 write 0x8f0460e thru 0x8f12c63 (0xe656 bytes) 2024-09-21T15:49:53.347 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:53.347 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:53.348 INFO:teuthology.orchestra.run.smithi089.stdout:3876 write 0x938bee1 thru 0x939b703 (0xf823 bytes) 2024-09-21T15:49:53.352 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:53.352 INFO:teuthology.orchestra.run.smithi089.stdout:3879 write 0x6e3afd3 thru 0x6e444bf (0x94ed bytes) 2024-09-21T15:49:53.355 INFO:teuthology.orchestra.run.smithi089.stdout:3880 write 0x4d3a6a8 thru 0x4d4201b (0x7974 bytes) 2024-09-21T15:49:53.356 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:49:53.358 INFO:teuthology.orchestra.run.smithi089.stdout:3881 write 0x63eb237 thru 0x63eec3e (0x3a08 bytes) 2024-09-21T15:49:53.361 INFO:teuthology.orchestra.run.smithi089.stdout:3882 read 0x5c7716 thru 0x5d6088 (0xe973 bytes) 2024-09-21T15:49:53.369 INFO:teuthology.orchestra.run.smithi089.stdout:3883 write 0xb5c383b thru 0xb5cd49f (0x9c65 bytes) 2024-09-21T15:49:53.370 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:53.369+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:53.375 INFO:teuthology.orchestra.run.smithi089.stdout:3884 read 0xa15918a thru 0xa15e876 (0x56ed bytes) 2024-09-21T15:49:53.384 INFO:teuthology.orchestra.run.smithi089.stdout:3885 trunc from 0xb5cd4a0 to 0x44d850d 2024-09-21T15:49:53.414 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:53.414 INFO:teuthology.orchestra.run.smithi089.stdout:3887 clone 24 order 21 su 262144 sc 3 2024-09-21T15:49:53.463 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:53.463 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:53.472 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:49:53.472 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:53.470+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:53.572 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:53.571+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:53.614 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:53.614 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:53.623 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:49:53.633 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:49:53.711 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:53.710+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:53.727 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:53.727 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:53.737 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:49:53.774 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:53.773+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:53.788 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-09-21T15:49:53.788 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:49:53.804 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:49:53.805 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:49:53.820 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:49:53.820 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:49:53.836 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:49:53.836 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:49:53.852 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:49:53.852 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:49:53.869 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:49:53.869 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:49:53.875 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:53.874+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:53.881 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:53.881 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:49:53.881 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:49:53.881 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:53.885 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:49:53.885 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:49:53.890 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:49:53.902 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:49:53.902 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:49:53.937 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T15:49:53.975 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:53.974+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:54.015 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:54.015 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:49:54.015 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:49:54.015 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:49:54.015 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:54.020 INFO:teuthology.orchestra.run.smithi089.stdout:truncating image image_client.0-clone23 from 0x44d850d (overlap 0x2bae0c) to 0xd1a9d 2024-09-21T15:49:54.024 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:49:54.076 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:54.075+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:54.128 INFO:teuthology.orchestra.run.smithi089.stdout:checking clone #22, image image_client.0-clone22 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent23 2024-09-21T15:49:54.134 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:54.134 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:49:54.134 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:49:54.134 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:49:54.134 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.135 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:984 3.0 e27/22 3:025e53e7:::rbd_data.1133f3dfbd6b.0000000000000008:head v 27'21, mlcod=27'19)", 2024-09-21T15:49:54.135 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.619393+0000", 2024-09-21T15:49:54.135 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 41.512998785999997, 2024-09-21T15:49:54.135 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003401837, 2024-09-21T15:49:54.135 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:54.135 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:54.135 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:54.135 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.135 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:54.135 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619393+0000", 2024-09-21T15:49:54.135 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.135 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.135 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.136 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:54.136 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619393+0000", 2024-09-21T15:49:54.136 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.136 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.136 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.136 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:54.136 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619392+0000", 2024-09-21T15:49:54.136 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:54.136 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.136 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.136 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:54.136 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619439+0000", 2024-09-21T15:49:54.136 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7170999999999999e-05 2024-09-21T15:49:54.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.137 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:54.137 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619452+0000", 2024-09-21T15:49:54.137 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2986999999999999e-05 2024-09-21T15:49:54.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.137 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:54.137 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619459+0000", 2024-09-21T15:49:54.137 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6109999999999999e-06 2024-09-21T15:49:54.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.138 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:54.138 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619518+0000", 2024-09-21T15:49:54.138 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9117999999999999e-05 2024-09-21T15:49:54.138 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.138 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.138 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:54.138 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619532+0000", 2024-09-21T15:49:54.138 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4622e-05 2024-09-21T15:49:54.138 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.138 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.138 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:54.138 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.622760+0000", 2024-09-21T15:49:54.138 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032278839999999999 2024-09-21T15:49:54.138 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.139 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:54.139 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.622794+0000", 2024-09-21T15:49:54.139 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4144999999999999e-05 2024-09-21T15:49:54.139 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.139 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:54.139 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.139 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.139 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:4957 3.9 e29/22 3:95c5ed95:::rbd_data.11335fa2fe69.0000000000000018:head v 29'4, mlcod=26'2)", 2024-09-21T15:49:54.139 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:14.453951+0000", 2024-09-21T15:49:54.139 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 39.678440354000003, 2024-09-21T15:49:54.139 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0043804559999999996, 2024-09-21T15:49:54.140 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:54.140 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:54.140 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:54.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.140 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:54.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.453951+0000", 2024-09-21T15:49:54.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.140 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:54.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.453951+0000", 2024-09-21T15:49:54.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.141 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:54.141 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.453950+0000", 2024-09-21T15:49:54.141 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:54.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.141 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:54.141 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454495+0000", 2024-09-21T15:49:54.141 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00054437799999999998 2024-09-21T15:49:54.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.141 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:54.141 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454508+0000", 2024-09-21T15:49:54.142 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3475000000000001e-05 2024-09-21T15:49:54.142 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.142 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.142 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:54.142 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454515+0000", 2024-09-21T15:49:54.142 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8990000000000004e-06 2024-09-21T15:49:54.142 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.142 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.142 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:54.142 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454581+0000", 2024-09-21T15:49:54.142 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5987000000000004e-05 2024-09-21T15:49:54.142 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.142 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.143 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:54.143 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454609+0000", 2024-09-21T15:49:54.143 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7466000000000001e-05 2024-09-21T15:49:54.143 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.143 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.143 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:54.143 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.458303+0000", 2024-09-21T15:49:54.143 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003693979 2024-09-21T15:49:54.143 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.143 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.143 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:54.143 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.458331+0000", 2024-09-21T15:49:54.143 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8932999999999999e-05 2024-09-21T15:49:54.143 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.144 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:54.144 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.144 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.144 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13420 3.0 e31/22 3:038b8840:::rbd_data.1133dc5adfdf.0000000000000035:head v 31'49, mlcod=31'47)", 2024-09-21T15:49:54.144 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.209233+0000", 2024-09-21T15:49:54.144 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 36.923158737999998, 2024-09-21T15:49:54.144 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0042624960000000002, 2024-09-21T15:49:54.144 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:54.144 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:54.144 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:54.144 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.144 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:54.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209233+0000", 2024-09-21T15:49:54.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.145 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:54.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209233+0000", 2024-09-21T15:49:54.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.145 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:54.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209232+0000", 2024-09-21T15:49:54.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:54.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.146 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.146 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:54.146 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209251+0000", 2024-09-21T15:49:54.146 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9304e-05 2024-09-21T15:49:54.146 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.146 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.146 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:54.146 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209265+0000", 2024-09-21T15:49:54.146 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3532e-05 2024-09-21T15:49:54.146 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.146 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.146 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:54.146 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209272+0000", 2024-09-21T15:49:54.146 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.289e-06 2024-09-21T15:49:54.147 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.147 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.147 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:54.147 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209321+0000", 2024-09-21T15:49:54.147 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8773999999999998e-05 2024-09-21T15:49:54.147 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.147 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.147 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:54.147 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209341+0000", 2024-09-21T15:49:54.147 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0387999999999999e-05 2024-09-21T15:49:54.147 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.147 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.147 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:54.147 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.213464+0000", 2024-09-21T15:49:54.148 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0041232450000000002 2024-09-21T15:49:54.148 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.148 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.148 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:54.148 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.213495+0000", 2024-09-21T15:49:54.148 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0793999999999997e-05 2024-09-21T15:49:54.148 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.148 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:54.148 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.148 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.148 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.148 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13445 3.0 e31/22 3:038b8840:::rbd_data.1133dc5adfdf.0000000000000035:head v 31'50, mlcod=31'48)", 2024-09-21T15:49:54.149 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.232497+0000", 2024-09-21T15:49:54.149 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 36.899894725999999, 2024-09-21T15:49:54.149 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0040493559999999996, 2024-09-21T15:49:54.149 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:54.149 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:54.149 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:54.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.149 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:54.149 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232497+0000", 2024-09-21T15:49:54.149 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.149 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:54.149 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232497+0000", 2024-09-21T15:49:54.150 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.150 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.150 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.150 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:54.150 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232496+0000", 2024-09-21T15:49:54.150 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:54.150 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.150 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.150 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:54.150 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232503+0000", 2024-09-21T15:49:54.150 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7320000000000001e-06 2024-09-21T15:49:54.150 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.150 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.151 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:54.151 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232512+0000", 2024-09-21T15:49:54.151 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9259999999999995e-06 2024-09-21T15:49:54.151 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.151 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.151 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:54.151 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232520+0000", 2024-09-21T15:49:54.151 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3699999999999997e-06 2024-09-21T15:49:54.151 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.151 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.151 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:54.151 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232564+0000", 2024-09-21T15:49:54.152 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4128999999999997e-05 2024-09-21T15:49:54.152 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.152 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.152 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:54.152 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232585+0000", 2024-09-21T15:49:54.152 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0704999999999999e-05 2024-09-21T15:49:54.152 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.152 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.152 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:54.152 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.236507+0000", 2024-09-21T15:49:54.152 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0039218730000000002 2024-09-21T15:49:54.152 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.152 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.153 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:54.153 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.236546+0000", 2024-09-21T15:49:54.153 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9399e-05 2024-09-21T15:49:54.153 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.153 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:54.153 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.153 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.153 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.153 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13452 3.9 e31/22 3:93f5396a:::rbd_data.1133dc5adfdf.0000000000000078:head v 31'50, mlcod=31'47)", 2024-09-21T15:49:54.153 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.237263+0000", 2024-09-21T15:49:54.153 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 36.895128020000001, 2024-09-21T15:49:54.153 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033768180000000002, 2024-09-21T15:49:54.153 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:54.154 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:54.154 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:54.154 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.154 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:54.154 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237263+0000", 2024-09-21T15:49:54.154 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.154 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.154 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.154 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:54.154 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237263+0000", 2024-09-21T15:49:54.155 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.155 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.155 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.155 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:54.155 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237263+0000", 2024-09-21T15:49:54.155 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:54.155 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.155 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.155 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:54.155 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237269+0000", 2024-09-21T15:49:54.155 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.472e-06 2024-09-21T15:49:54.155 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.156 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.156 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:54.156 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237282+0000", 2024-09-21T15:49:54.156 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2788e-05 2024-09-21T15:49:54.156 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.156 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.156 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:54.156 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237288+0000", 2024-09-21T15:49:54.156 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4899999999999997e-06 2024-09-21T15:49:54.156 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.156 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.156 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:54.156 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237331+0000", 2024-09-21T15:49:54.157 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2425000000000001e-05 2024-09-21T15:49:54.157 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.157 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.157 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:54.157 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237349+0000", 2024-09-21T15:49:54.157 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8405000000000001e-05 2024-09-21T15:49:54.157 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.157 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.157 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:54.157 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.240610+0000", 2024-09-21T15:49:54.157 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032605379999999999 2024-09-21T15:49:54.157 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.158 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.158 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:54.158 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.240640+0000", 2024-09-21T15:49:54.158 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0448000000000001e-05 2024-09-21T15:49:54.158 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.158 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:54.158 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.158 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.158 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.158 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13832 3.0 e31/22 3:0792efd2:::rbd_data.1133dc5adfdf.0000000000000090:head v 31'58, mlcod=31'56)", 2024-09-21T15:49:54.158 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.527530+0000", 2024-09-21T15:49:54.158 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 36.604861200000002, 2024-09-21T15:49:54.159 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035256530000000001, 2024-09-21T15:49:54.159 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:54.159 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:54.159 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:54.159 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.159 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:54.159 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527530+0000", 2024-09-21T15:49:54.159 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.159 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.159 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.159 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:54.160 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527530+0000", 2024-09-21T15:49:54.160 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.160 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.160 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.160 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:54.160 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527529+0000", 2024-09-21T15:49:54.160 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:54.160 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.160 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.160 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:54.160 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527550+0000", 2024-09-21T15:49:54.160 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1387999999999999e-05 2024-09-21T15:49:54.160 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.160 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.161 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:54.161 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527568+0000", 2024-09-21T15:49:54.161 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8068e-05 2024-09-21T15:49:54.161 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.161 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.161 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:54.161 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527577+0000", 2024-09-21T15:49:54.161 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9919999999999996e-06 2024-09-21T15:49:54.161 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.161 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.161 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:54.161 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527626+0000", 2024-09-21T15:49:54.161 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8597000000000002e-05 2024-09-21T15:49:54.162 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.162 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.162 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:54.162 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527652+0000", 2024-09-21T15:49:54.162 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5993e-05 2024-09-21T15:49:54.162 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.162 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.162 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:54.162 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.531024+0000", 2024-09-21T15:49:54.162 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033721910000000001 2024-09-21T15:49:54.162 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.162 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.163 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:54.163 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.531056+0000", 2024-09-21T15:49:54.163 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1628000000000003e-05 2024-09-21T15:49:54.163 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.163 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:54.163 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.163 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.163 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.163 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13839 3.0 e31/22 3:03c4fb32:::rbd_data.1133dc5adfdf.0000000000000099:head v 31'59, mlcod=31'56)", 2024-09-21T15:49:54.163 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.528097+0000", 2024-09-21T15:49:54.163 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 36.604294602000003, 2024-09-21T15:49:54.163 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0057228840000000001, 2024-09-21T15:49:54.163 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:54.164 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:54.164 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:54.164 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.164 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:54.164 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528097+0000", 2024-09-21T15:49:54.164 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.164 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.164 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.164 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:54.164 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528097+0000", 2024-09-21T15:49:54.164 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.164 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.165 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.165 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:54.165 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528096+0000", 2024-09-21T15:49:54.165 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:54.165 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.165 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.165 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:54.165 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528105+0000", 2024-09-21T15:49:54.165 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7350000000000002e-06 2024-09-21T15:49:54.165 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.166 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.166 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:54.166 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528117+0000", 2024-09-21T15:49:54.166 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1909000000000001e-05 2024-09-21T15:49:54.166 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.166 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.166 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:54.166 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528124+0000", 2024-09-21T15:49:54.166 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8419999999999999e-06 2024-09-21T15:49:54.166 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.166 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.166 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:54.166 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528168+0000", 2024-09-21T15:49:54.167 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4301999999999997e-05 2024-09-21T15:49:54.167 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.167 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.167 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:54.167 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528191+0000", 2024-09-21T15:49:54.167 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2622000000000001e-05 2024-09-21T15:49:54.167 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.167 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.167 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:54.167 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.533780+0000", 2024-09-21T15:49:54.167 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0055897519999999999 2024-09-21T15:49:54.167 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.167 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.168 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:54.168 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.533820+0000", 2024-09-21T15:49:54.168 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9366999999999998e-05 2024-09-21T15:49:54.168 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.168 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:54.168 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.168 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.168 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.168 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13884 3.9 e31/22 3:93003736:::rbd_data.1133dc5adfdf.00000000000000a1:head v 31'54, mlcod=31'52)", 2024-09-21T15:49:54.168 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.562433+0000", 2024-09-21T15:49:54.168 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 36.569957854000002, 2024-09-21T15:49:54.168 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.005162313, 2024-09-21T15:49:54.168 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:54.169 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:54.169 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:54.169 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.169 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:54.169 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562433+0000", 2024-09-21T15:49:54.169 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.169 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.169 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.169 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:54.169 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562433+0000", 2024-09-21T15:49:54.169 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.169 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.169 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.169 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:54.170 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562432+0000", 2024-09-21T15:49:54.170 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:54.170 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.170 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.170 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:54.170 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562467+0000", 2024-09-21T15:49:54.170 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4885999999999998e-05 2024-09-21T15:49:54.170 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.170 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.170 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:54.170 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562481+0000", 2024-09-21T15:49:54.170 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3604000000000001e-05 2024-09-21T15:49:54.170 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.170 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.171 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:54.171 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562489+0000", 2024-09-21T15:49:54.171 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1829999999999994e-06 2024-09-21T15:49:54.171 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.171 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.171 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:54.171 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562536+0000", 2024-09-21T15:49:54.171 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6830999999999999e-05 2024-09-21T15:49:54.171 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.171 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.171 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:54.171 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562559+0000", 2024-09-21T15:49:54.171 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2765999999999999e-05 2024-09-21T15:49:54.172 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.172 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.172 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:54.172 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.567566+0000", 2024-09-21T15:49:54.172 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0050073219999999998 2024-09-21T15:49:54.172 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.172 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.172 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:54.172 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.567596+0000", 2024-09-21T15:49:54.172 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9802999999999999e-05 2024-09-21T15:49:54.172 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.172 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:54.172 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.172 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.173 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.173 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:22340 3.9 e34/22 3:910a5abe:::rbd_data.1133896adcfd.0000000000000023:head v 34'92, mlcod=34'90)", 2024-09-21T15:49:54.173 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:21.628025+0000", 2024-09-21T15:49:54.173 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 32.504366474000001, 2024-09-21T15:49:54.173 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046196010000000001, 2024-09-21T15:49:54.173 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:54.173 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:54.173 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:54.173 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.173 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:54.173 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628025+0000", 2024-09-21T15:49:54.173 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.173 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.173 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.174 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:54.174 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628025+0000", 2024-09-21T15:49:54.174 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.174 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.174 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.174 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:54.174 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628024+0000", 2024-09-21T15:49:54.174 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:54.174 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.174 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.174 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:54.174 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628034+0000", 2024-09-21T15:49:54.174 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3470000000000005e-06 2024-09-21T15:49:54.175 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.175 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.175 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:54.175 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628043+0000", 2024-09-21T15:49:54.175 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9450000000000005e-06 2024-09-21T15:49:54.175 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.175 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.175 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:54.175 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628048+0000", 2024-09-21T15:49:54.175 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8080000000000004e-06 2024-09-21T15:49:54.175 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.175 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.175 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:54.176 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628081+0000", 2024-09-21T15:49:54.176 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2738999999999999e-05 2024-09-21T15:49:54.176 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.176 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.176 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:54.176 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628094+0000", 2024-09-21T15:49:54.176 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3273e-05 2024-09-21T15:49:54.176 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.176 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.176 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:54.176 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.632616+0000", 2024-09-21T15:49:54.176 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045217269999999997 2024-09-21T15:49:54.176 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.176 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.177 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:54.177 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.632644+0000", 2024-09-21T15:49:54.177 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8469000000000001e-05 2024-09-21T15:49:54.177 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.177 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:54.177 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.177 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.177 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.177 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:22417 3.4 e34/22 3:25310a10:::rbd_data.1133896adcfd.0000000000000075:head v 34'48, mlcod=34'46)", 2024-09-21T15:49:54.177 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:21.704297+0000", 2024-09-21T15:49:54.177 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 32.428094375999997, 2024-09-21T15:49:54.177 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0038101799999999998, 2024-09-21T15:49:54.177 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:54.177 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:54.178 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:54.178 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.178 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:54.178 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704297+0000", 2024-09-21T15:49:54.178 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.178 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.178 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.178 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:54.178 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704297+0000", 2024-09-21T15:49:54.178 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.178 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.178 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.178 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:54.179 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704296+0000", 2024-09-21T15:49:54.179 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:54.179 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.179 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.179 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:54.179 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704308+0000", 2024-09-21T15:49:54.179 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2559e-05 2024-09-21T15:49:54.179 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.179 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.179 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:54.179 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704337+0000", 2024-09-21T15:49:54.179 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8960000000000001e-05 2024-09-21T15:49:54.180 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.180 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.180 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:54.180 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704345+0000", 2024-09-21T15:49:54.180 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3059999999999998e-06 2024-09-21T15:49:54.180 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.180 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.180 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:54.180 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704408+0000", 2024-09-21T15:49:54.180 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2156000000000006e-05 2024-09-21T15:49:54.180 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.180 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.180 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:54.181 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704430+0000", 2024-09-21T15:49:54.181 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1931000000000001e-05 2024-09-21T15:49:54.181 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.181 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.181 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:54.181 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.708073+0000", 2024-09-21T15:49:54.181 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003643028 2024-09-21T15:49:54.181 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.181 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.181 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:54.181 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.708107+0000", 2024-09-21T15:49:54.181 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4613999999999998e-05 2024-09-21T15:49:54.181 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.182 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:54.182 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.182 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.182 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.182 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:25278 3.4 e38/22 3:2251cc1c:::rbd_data.1133ebd6d3ba.0000000000000024:head v 38'51, mlcod=38'49)", 2024-09-21T15:49:54.182 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.317337+0000", 2024-09-21T15:49:54.182 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.815054616000001, 2024-09-21T15:49:54.182 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053904110000000003, 2024-09-21T15:49:54.182 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:54.182 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:54.182 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:54.182 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.182 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:54.183 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317337+0000", 2024-09-21T15:49:54.183 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.183 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.183 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.183 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:54.183 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317337+0000", 2024-09-21T15:49:54.183 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.183 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.183 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.183 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:54.183 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317335+0000", 2024-09-21T15:49:54.183 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:49:54.184 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.184 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.184 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:54.184 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317347+0000", 2024-09-21T15:49:54.184 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.19e-05 2024-09-21T15:49:54.184 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.184 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.184 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:54.184 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317369+0000", 2024-09-21T15:49:54.184 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2232e-05 2024-09-21T15:49:54.184 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.184 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.185 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:54.185 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317381+0000", 2024-09-21T15:49:54.185 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2231e-05 2024-09-21T15:49:54.185 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.185 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.185 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:54.185 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317425+0000", 2024-09-21T15:49:54.185 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3625000000000003e-05 2024-09-21T15:49:54.185 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.185 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.185 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:54.185 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317449+0000", 2024-09-21T15:49:54.186 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4301999999999999e-05 2024-09-21T15:49:54.186 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.186 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.186 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:54.186 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.322688+0000", 2024-09-21T15:49:54.186 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.005238609 2024-09-21T15:49:54.186 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.186 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.186 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:54.186 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.322727+0000", 2024-09-21T15:49:54.186 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9201e-05 2024-09-21T15:49:54.186 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.186 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:54.187 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.187 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.187 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.187 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:25286 3.0 e38/22 3:0248fb60:::rbd_data.1133ebd6d3ba.0000000000000066:head v 38'109, mlcod=34'107)", 2024-09-21T15:49:54.187 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.317638+0000", 2024-09-21T15:49:54.187 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.814753737, 2024-09-21T15:49:54.187 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0054074370000000002, 2024-09-21T15:49:54.187 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:54.187 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:54.187 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:54.187 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.187 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:54.188 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317638+0000", 2024-09-21T15:49:54.188 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.188 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.188 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.188 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:54.188 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317638+0000", 2024-09-21T15:49:54.188 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.188 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.188 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.188 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:54.188 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317637+0000", 2024-09-21T15:49:54.188 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:54.189 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.189 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.189 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:54.189 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317643+0000", 2024-09-21T15:49:54.189 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5609999999999998e-06 2024-09-21T15:49:54.189 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.189 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.189 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:54.189 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317653+0000", 2024-09-21T15:49:54.189 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0705e-05 2024-09-21T15:49:54.189 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.189 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.189 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:54.189 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317660+0000", 2024-09-21T15:49:54.190 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2099999999999998e-06 2024-09-21T15:49:54.190 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.190 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.190 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:54.190 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317704+0000", 2024-09-21T15:49:54.190 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4659000000000001e-05 2024-09-21T15:49:54.190 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.190 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.190 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:54.190 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317725+0000", 2024-09-21T15:49:54.190 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0537e-05 2024-09-21T15:49:54.190 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.190 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.191 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:54.191 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323010+0000", 2024-09-21T15:49:54.191 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052853420000000002 2024-09-21T15:49:54.191 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.191 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.191 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:54.191 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323045+0000", 2024-09-21T15:49:54.191 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4860000000000002e-05 2024-09-21T15:49:54.191 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.191 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:54.191 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.191 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.191 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.191 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:25280 3.4 e38/22 3:27fca38c:::rbd_data.1133ebd6d3ba.0000000000000026:head v 38'52, mlcod=38'49)", 2024-09-21T15:49:54.192 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.317684+0000", 2024-09-21T15:49:54.192 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.814707841000001, 2024-09-21T15:49:54.192 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053570809999999996, 2024-09-21T15:49:54.192 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:54.192 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:54.192 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:54.192 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.192 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:54.192 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317684+0000", 2024-09-21T15:49:54.192 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.192 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.192 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.192 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:54.193 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317684+0000", 2024-09-21T15:49:54.193 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.193 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.193 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.193 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:54.193 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317683+0000", 2024-09-21T15:49:54.193 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:54.193 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.193 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.193 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:54.193 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317688+0000", 2024-09-21T15:49:54.193 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5009999999999998e-06 2024-09-21T15:49:54.193 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.193 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.194 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:54.194 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317694+0000", 2024-09-21T15:49:54.194 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3579999999999999e-06 2024-09-21T15:49:54.194 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.194 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.194 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:54.194 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317698+0000", 2024-09-21T15:49:54.194 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4150000000000001e-06 2024-09-21T15:49:54.194 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.194 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.194 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:54.194 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317738+0000", 2024-09-21T15:49:54.194 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9575e-05 2024-09-21T15:49:54.195 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.195 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.195 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:54.195 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317762+0000", 2024-09-21T15:49:54.195 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4173999999999999e-05 2024-09-21T15:49:54.195 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.195 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.195 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:54.195 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323005+0000", 2024-09-21T15:49:54.195 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00524333 2024-09-21T15:49:54.195 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.195 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.195 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:54.195 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323041+0000", 2024-09-21T15:49:54.196 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5139999999999999e-05 2024-09-21T15:49:54.196 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.196 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:54.196 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.196 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.196 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.196 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:34837 3.0 e43/22 3:038527c8:::rbd_header.1133fed1a840:head v 43'130, mlcod=43'128)", 2024-09-21T15:49:54.196 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:49:54.196 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.148503142999999, 2024-09-21T15:49:54.196 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045776519999999998, 2024-09-21T15:49:54.196 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:54.196 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:54.196 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:54.197 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.197 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:54.197 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:49:54.197 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.197 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.197 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.197 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:54.197 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:49:54.197 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.197 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.197 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.197 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:54.197 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:49:54.197 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:54.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:54.198 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983892+0000", 2024-09-21T15:49:54.198 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8450000000000002e-06 2024-09-21T15:49:54.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:54.198 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983902+0000", 2024-09-21T15:49:54.198 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0299999999999999e-06 2024-09-21T15:49:54.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:54.199 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983906+0000", 2024-09-21T15:49:54.199 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2300000000000002e-06 2024-09-21T15:49:54.199 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.199 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.199 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:54.199 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983940+0000", 2024-09-21T15:49:54.199 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4563000000000001e-05 2024-09-21T15:49:54.199 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.199 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.199 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:54.199 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983960+0000", 2024-09-21T15:49:54.199 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9199999999999999e-05 2024-09-21T15:49:54.199 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.200 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.200 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:54.200 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.988430+0000", 2024-09-21T15:49:54.200 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0044709149999999998 2024-09-21T15:49:54.200 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.200 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.200 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:54.200 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.988466+0000", 2024-09-21T15:49:54.200 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5420000000000003e-05 2024-09-21T15:49:54.200 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.200 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:54.200 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.200 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.200 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.201 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "pg_update_log_missing(3.0 epoch 50/22 rep_tid 498 entries 50'171 (0'0) error 3:5df4a5a0:::rbd_data.11339e87a32a.000000000000008e:head by client.4403.0:80207 0.000000 -2 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 47'167)", 2024-09-21T15:49:54.201 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:49:54.201 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.19063027, 2024-09-21T15:49:54.201 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0047253119999999997, 2024-09-21T15:49:54.201 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:54.201 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-09-21T15:49:54.201 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:54.201 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.201 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:54.201 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:49:54.201 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.201 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.201 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.201 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:54.202 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:49:54.202 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.202 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.202 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.202 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:54.202 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:49:54.202 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:54.202 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.202 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.202 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:54.202 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941768+0000", 2024-09-21T15:49:54.202 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7309999999999993e-06 2024-09-21T15:49:54.202 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.203 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.203 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:54.203 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941780+0000", 2024-09-21T15:49:54.203 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1399999999999999e-05 2024-09-21T15:49:54.203 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.203 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.203 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:54.203 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941789+0000", 2024-09-21T15:49:54.203 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5230000000000001e-06 2024-09-21T15:49:54.203 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.203 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.203 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:54.203 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941841+0000", 2024-09-21T15:49:54.204 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1863999999999997e-05 2024-09-21T15:49:54.204 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.204 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.204 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:54.204 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.946486+0000", 2024-09-21T15:49:54.204 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046452539999999997 2024-09-21T15:49:54.204 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.204 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:54.204 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.204 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.204 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.204 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:82086 3.0 e50/22 3:0114bc4d:::rbd_data.11334c14b8fd.0000000000000069:head v 50'178, mlcod=50'176)", 2024-09-21T15:49:54.205 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:41.155591+0000", 2024-09-21T15:49:54.205 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.976799958000001, 2024-09-21T15:49:54.205 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003363497, 2024-09-21T15:49:54.205 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:54.205 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:54.205 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:54.205 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.205 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:54.205 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.155591+0000", 2024-09-21T15:49:54.205 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.205 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.205 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.206 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:54.206 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.155591+0000", 2024-09-21T15:49:54.206 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.206 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.206 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.206 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:54.206 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.155590+0000", 2024-09-21T15:49:54.206 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:49:54.206 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.206 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.206 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:54.207 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.155606+0000", 2024-09-21T15:49:54.207 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6075000000000001e-05 2024-09-21T15:49:54.207 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.207 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.207 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:54.207 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.155623+0000", 2024-09-21T15:49:54.207 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6838e-05 2024-09-21T15:49:54.207 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.207 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.207 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:54.207 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.155630+0000", 2024-09-21T15:49:54.207 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.627e-06 2024-09-21T15:49:54.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.208 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.208 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:54.208 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.155677+0000", 2024-09-21T15:49:54.208 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6669999999999999e-05 2024-09-21T15:49:54.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.208 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.208 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:54.208 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.155698+0000", 2024-09-21T15:49:54.208 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1330000000000001e-05 2024-09-21T15:49:54.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.208 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.208 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:54.208 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.158918+0000", 2024-09-21T15:49:54.209 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032198600000000002 2024-09-21T15:49:54.209 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.209 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.209 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:54.209 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.158955+0000", 2024-09-21T15:49:54.209 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6647000000000003e-05 2024-09-21T15:49:54.209 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.209 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:54.209 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.209 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.209 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.209 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:82470 3.4 e50/22 3:21ad7dfe:::rbd_data.11334c14b8fd.00000000000003a7:head v 50'190, mlcod=46'188)", 2024-09-21T15:49:54.209 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:41.291787+0000", 2024-09-21T15:49:54.209 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.840604333, 2024-09-21T15:49:54.210 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033489969999999998, 2024-09-21T15:49:54.210 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:54.210 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:54.210 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:54.210 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.210 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:54.210 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.291787+0000", 2024-09-21T15:49:54.210 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.210 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.210 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.210 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:54.210 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.291787+0000", 2024-09-21T15:49:54.211 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.211 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.211 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.211 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:54.211 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.291786+0000", 2024-09-21T15:49:54.211 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:54.211 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.211 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.211 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:54.211 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.291795+0000", 2024-09-21T15:49:54.211 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7830000000000004e-06 2024-09-21T15:49:54.211 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.211 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.212 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:54.212 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.291814+0000", 2024-09-21T15:49:54.212 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9117999999999999e-05 2024-09-21T15:49:54.212 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.212 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.212 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:54.212 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.291822+0000", 2024-09-21T15:49:54.212 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0669999999999992e-06 2024-09-21T15:49:54.212 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.212 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.212 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:54.212 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.291866+0000", 2024-09-21T15:49:54.213 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4669000000000002e-05 2024-09-21T15:49:54.213 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.213 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.213 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:54.213 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.291888+0000", 2024-09-21T15:49:54.213 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1087999999999999e-05 2024-09-21T15:49:54.213 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.213 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.213 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:54.213 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.295104+0000", 2024-09-21T15:49:54.213 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032164910000000001 2024-09-21T15:49:54.213 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.213 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.214 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:54.214 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.295136+0000", 2024-09-21T15:49:54.214 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1931999999999999e-05 2024-09-21T15:49:54.214 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.214 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:54.214 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.214 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.214 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.214 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "pg_update_log_missing(3.0 epoch 53/22 rep_tid 585 entries 53'201 (0'0) error 3:6a1d4890:::rbd_data.1133e079cae4.0000000000000004:head by client.4403.0:101089 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 53'198)", 2024-09-21T15:49:54.214 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:44.832827+0000", 2024-09-21T15:49:54.214 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.2995644619999993, 2024-09-21T15:49:54.214 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053552929999999997, 2024-09-21T15:49:54.215 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:54.215 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-09-21T15:49:54.215 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:54.215 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.215 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:54.215 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832827+0000", 2024-09-21T15:49:54.215 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.215 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.215 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.215 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:54.215 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832827+0000", 2024-09-21T15:49:54.215 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.216 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.216 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.216 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:54.216 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832826+0000", 2024-09-21T15:49:54.216 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:49:54.216 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.216 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.216 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:54.216 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832835+0000", 2024-09-21T15:49:54.216 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4509999999999992e-06 2024-09-21T15:49:54.216 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.216 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.216 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:54.216 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832854+0000", 2024-09-21T15:49:54.217 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9352999999999999e-05 2024-09-21T15:49:54.217 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.217 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.217 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:54.217 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832866+0000", 2024-09-21T15:49:54.217 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1766000000000001e-05 2024-09-21T15:49:54.217 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.217 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.217 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:54.217 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.833091+0000", 2024-09-21T15:49:54.217 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00022490300000000001 2024-09-21T15:49:54.217 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.217 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.217 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:54.218 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.838182+0000", 2024-09-21T15:49:54.218 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0050915420000000001 2024-09-21T15:49:54.218 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.218 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:54.218 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.218 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.218 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.218 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:102822 3.4 e54/22 3:2729ba03:::rbd_object_map.113336ae1075:head v 54'208, mlcod=54'205)", 2024-09-21T15:49:54.218 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:46.249540+0000", 2024-09-21T15:49:54.218 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 7.88285123, 2024-09-21T15:49:54.218 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052654859999999998, 2024-09-21T15:49:54.218 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:54.218 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:54.219 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:54.219 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.219 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:54.219 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249540+0000", 2024-09-21T15:49:54.219 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.219 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.219 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.219 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:54.221 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249540+0000", 2024-09-21T15:49:54.221 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.221 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.221 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.221 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:54.221 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249539+0000", 2024-09-21T15:49:54.222 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:54.222 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.222 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.222 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:54.222 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249546+0000", 2024-09-21T15:49:54.222 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7290000000000003e-06 2024-09-21T15:49:54.222 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.222 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.222 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:54.222 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249558+0000", 2024-09-21T15:49:54.222 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1844e-05 2024-09-21T15:49:54.222 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.223 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.223 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:54.223 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249565+0000", 2024-09-21T15:49:54.223 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8210000000000004e-06 2024-09-21T15:49:54.223 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.223 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.223 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:54.223 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249602+0000", 2024-09-21T15:49:54.223 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7057000000000001e-05 2024-09-21T15:49:54.223 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.223 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.223 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:54.223 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249622+0000", 2024-09-21T15:49:54.224 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9678999999999999e-05 2024-09-21T15:49:54.224 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.224 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.224 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:54.224 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254783+0000", 2024-09-21T15:49:54.224 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0051611449999999998 2024-09-21T15:49:54.224 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.224 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.224 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:54.224 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254806+0000", 2024-09-21T15:49:54.224 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2853e-05 2024-09-21T15:49:54.224 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.224 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:54.225 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.225 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.225 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.225 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:102823 3.4 e54/22 3:2729ba03:::rbd_object_map.113336ae1075:head v 54'209, mlcod=54'205)", 2024-09-21T15:49:54.225 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:46.250318+0000", 2024-09-21T15:49:54.225 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 7.8820736450000002, 2024-09-21T15:49:54.225 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045013539999999999, 2024-09-21T15:49:54.225 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:54.225 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:54.225 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:54.225 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.225 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:54.226 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250318+0000", 2024-09-21T15:49:54.226 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.226 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.226 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.226 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:54.226 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250318+0000", 2024-09-21T15:49:54.226 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.226 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.226 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.226 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:54.226 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250317+0000", 2024-09-21T15:49:54.226 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:54.227 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.227 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.227 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:54.227 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250322+0000", 2024-09-21T15:49:54.227 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.074e-06 2024-09-21T15:49:54.227 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.227 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.227 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:54.227 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250330+0000", 2024-09-21T15:49:54.227 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4730000000000003e-06 2024-09-21T15:49:54.228 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.228 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.228 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:54.228 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250335+0000", 2024-09-21T15:49:54.228 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8019999999999997e-06 2024-09-21T15:49:54.228 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.228 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.228 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:54.228 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250375+0000", 2024-09-21T15:49:54.228 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9459999999999998e-05 2024-09-21T15:49:54.228 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.229 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.229 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:54.229 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250388+0000", 2024-09-21T15:49:54.229 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3288999999999999e-05 2024-09-21T15:49:54.229 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.229 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.229 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:54.229 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254807+0000", 2024-09-21T15:49:54.229 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0044192220000000004 2024-09-21T15:49:54.229 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.229 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.229 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:54.230 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254819+0000", 2024-09-21T15:49:54.230 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1603999999999999e-05 2024-09-21T15:49:54.230 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.230 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:54.230 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.230 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.230 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:54.230 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:54.231 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:49:54.233 INFO:teuthology.orchestra.run.smithi089.stdout:3888 writesame 0xb2f0dd8 thru 0xb2f5852 data_size 0x1(0x4a7b bytes) 2024-09-21T15:49:54.233 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:54.233 INFO:teuthology.orchestra.run.smithi089.stdout:3890 write 0xc88ed9a thru 0xc89b133 (0xc39a bytes) 2024-09-21T15:49:54.233 INFO:teuthology.orchestra.run.smithi089.stdout:3891 write 0x7105609 thru 0x7109967 (0x435f bytes) 2024-09-21T15:49:54.233 INFO:teuthology.orchestra.run.smithi089.stdout:3892 trunc from 0xc89b134 to 0x613840 2024-09-21T15:49:54.233 INFO:teuthology.orchestra.run.smithi089.stdout:3893 writesame 0x16e5f8d thru 0x16ed363 data_size 0xf(0x73d7 bytes) 2024-09-21T15:49:54.233 INFO:teuthology.orchestra.run.smithi089.stdout:3894 read 0x1613c06 thru 0x16190b5 (0x54b0 bytes) 2024-09-21T15:49:54.233 INFO:teuthology.orchestra.run.smithi089.stdout:3895 punch from 0x9f3366 to 0x9fe377, (0xb011 bytes) 2024-09-21T15:49:54.233 INFO:teuthology.orchestra.run.smithi089.stdout:3896 writesame 0x1391e74 thru 0x13920f3 data_size 0x5(0x280 bytes) 2024-09-21T15:49:54.234 INFO:teuthology.orchestra.run.smithi089.stdout:3897 trunc from 0x16ed364 to 0x2cd62a2 2024-09-21T15:49:54.234 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:54.230+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:54.248 INFO:teuthology.orchestra.run.smithi089.stdout:3898 write 0x3f8f008 thru 0x3f90646 (0x163f bytes) 2024-09-21T15:49:54.265 INFO:teuthology.orchestra.run.smithi089.stdout:3899 writesame 0x707b093 thru 0x707c52f data_size 0x3(0x149d bytes) 2024-09-21T15:49:54.265 INFO:teuthology.orchestra.run.smithi089.stdout:3900 read 0x432d170 thru 0x4337941 (0xa7d2 bytes) 2024-09-21T15:49:54.265 INFO:teuthology.orchestra.run.smithi089.stdout:3903 read 0x2da787d thru 0x2da9c65 (0x23e9 bytes) 2024-09-21T15:49:54.265 INFO:teuthology.orchestra.run.smithi089.stderr:2024-09-21T15:49:54.264+0000 7f79f2d9d640 0 --2- 172.21.15.89:0/359708235 >> [v2:172.21.15.89:3300/0,v1:172.21.15.89:6789/0] conn(0x7f79f4070970 0x7f79f413e4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-09-21T15:49:54.279 INFO:teuthology.orchestra.run.smithi089.stdout:3904 write 0xe289075 thru 0xe290cb4 (0x7c40 bytes) 2024-09-21T15:49:54.282 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:54.282 INFO:teuthology.orchestra.run.smithi089.stdout:3906 read 0x61faeda thru 0x6206e01 (0xbf28 bytes) 2024-09-21T15:49:54.282 INFO:teuthology.orchestra.run.smithi089.stdout:3908 punch from 0x663569e to 0x663e18d, (0x8aef bytes) 2024-09-21T15:49:54.283 INFO:teuthology.orchestra.run.smithi089.stdout:3909 trunc from 0xe290cb5 to 0xe564d2c 2024-09-21T15:49:54.287 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:54.287 INFO:teuthology.orchestra.run.smithi089.stdout:3911 trunc from 0xe564d2c to 0xe10d169 2024-09-21T15:49:54.294 INFO:teuthology.orchestra.run.smithi089.stdout:3912 write 0x366f27f thru 0x3677f02 (0x8c84 bytes) 2024-09-21T15:49:54.301 INFO:teuthology.orchestra.run.smithi089.stdout:3913 trunc from 0xe10d169 to 0xb6371b8 2024-09-21T15:49:54.311 INFO:teuthology.orchestra.run.smithi089.stdout:3917 read 0xabd4776 thru 0xabe25b9 (0xde44 bytes) 2024-09-21T15:49:54.316 INFO:teuthology.orchestra.run.smithi089.stdout:3918 write 0xbc6e0da thru 0xbc7cf05 (0xee2c bytes) 2024-09-21T15:49:54.319 INFO:teuthology.orchestra.run.smithi089.stdout:3921 write 0x3caa805 thru 0x3cb2bf5 (0x83f1 bytes) 2024-09-21T15:49:54.331 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:54.331+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:54.333 INFO:teuthology.orchestra.run.smithi089.stdout:3922 trunc from 0xbc7cf06 to 0x3909204 2024-09-21T15:49:54.349 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:54.349 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:49:54.349 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:49:54.349 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:54.361 INFO:teuthology.orchestra.run.smithi089.stdout:3923 trunc from 0x3909204 to 0xdff0517 2024-09-21T15:49:54.362 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:49:54.383 INFO:teuthology.orchestra.run.smithi089.stdout:3924 trunc from 0xdff0517 to 0x265ea0b 2024-09-21T15:49:54.398 INFO:teuthology.orchestra.run.smithi089.stdout:3927 trunc from 0x265ea0b to 0x1c16d3 2024-09-21T15:49:54.432 INFO:teuthology.orchestra.run.smithi089.stdout:3928 write 0x3905db4 thru 0x3913d82 (0xdfcf bytes) 2024-09-21T15:49:54.433 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:54.431+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:54.436 INFO:teuthology.orchestra.run.smithi089.stdout:3929 read 0x166a569 thru 0x166e2f9 (0x3d91 bytes) 2024-09-21T15:49:54.442 INFO:teuthology.orchestra.run.smithi089.stdout:3930 write 0x44dfdf4 thru 0x44e6d55 (0x6f62 bytes) 2024-09-21T15:49:54.450 INFO:teuthology.orchestra.run.smithi089.stdout:3931 trunc from 0x44e6d56 to 0x8087ddd 2024-09-21T15:49:54.456 INFO:teuthology.orchestra.run.smithi089.stdout:3933 trunc from 0x8087ddd to 0x997313b 2024-09-21T15:49:54.471 INFO:teuthology.orchestra.run.smithi089.stdout:3934 writesame 0xe0a8d8e thru 0xe0b5c9d data_size 0x8(0xcf10 bytes) 2024-09-21T15:49:54.471 INFO:teuthology.orchestra.run.smithi089.stdout:3935 writesame 0x6fca230 thru 0x6fcd075 data_size 0x1(0x2e46 bytes) 2024-09-21T15:49:54.479 INFO:teuthology.orchestra.run.smithi089.stdout:3936 read 0x336338e thru 0x3367a68 (0x46db bytes) 2024-09-21T15:49:54.479 INFO:teuthology.orchestra.run.smithi089.stdout:3937 write 0x3262db7 thru 0x3269ee8 (0x7132 bytes) 2024-09-21T15:49:54.482 INFO:teuthology.orchestra.run.smithi089.stdout:3938 punch from 0xa5b7dec to 0xa5c36fc, (0xb910 bytes) 2024-09-21T15:49:54.482 INFO:teuthology.orchestra.run.smithi089.stdout:3939 read 0x553a405 thru 0x553ad37 (0x933 bytes) 2024-09-21T15:49:54.482 INFO:teuthology.orchestra.run.smithi089.stdout:3941 clone 25 order 18 su 2048 sc 3 2024-09-21T15:49:54.496 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:54.496 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:49:54.496 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:49:54.496 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:49:54.496 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:54.505 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:49:54.546 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:54.545+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:54.634 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:54.633+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:54.648 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:49:54.649 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:49:54.649 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:49:54.649 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:49:54.649 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.649 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:14051 3.8 3:13437810:::rbd_data.1133dc5adfdf.0000000000000039:head [write 716800~796] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-21T15:49:54.649 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:49:54.649 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 36.951592239999997, 2024-09-21T15:49:54.649 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.98064002699999997, 2024-09-21T15:49:54.649 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:54.649 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:49:54.649 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:54.649 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:54.649 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:54.649 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 14051 2024-09-21T15:49:54.650 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.650 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:54.650 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.650 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:54.650 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:49:54.650 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.650 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.650 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.650 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:54.650 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:49:54.650 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.650 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.650 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.650 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:54.650 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:49:54.651 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:54.651 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.651 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.651 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:54.651 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695131+0000", 2024-09-21T15:49:54.651 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4680000000000003e-06 2024-09-21T15:49:54.651 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.651 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.651 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:54.651 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695139+0000", 2024-09-21T15:49:54.651 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9400000000000002e-06 2024-09-21T15:49:54.651 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.651 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.651 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:54.651 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695143+0000", 2024-09-21T15:49:54.652 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9899999999999999e-06 2024-09-21T15:49:54.652 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.652 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.652 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:54.652 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695190+0000", 2024-09-21T15:49:54.652 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6458000000000001e-05 2024-09-21T15:49:54.652 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.652 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.652 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:49:54.652 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695201+0000", 2024-09-21T15:49:54.652 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0964999999999999e-05 2024-09-21T15:49:54.652 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.652 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.652 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:54.652 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.669490+0000", 2024-09-21T15:49:54.652 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.97428980499999995 2024-09-21T15:49:54.653 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.653 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.653 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-09-21T15:49:54.653 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.670033+0000", 2024-09-21T15:49:54.653 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00054252599999999999 2024-09-21T15:49:54.653 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.653 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.653 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:54.653 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.671095+0000", 2024-09-21T15:49:54.653 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001062183 2024-09-21T15:49:54.653 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.653 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.653 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:54.653 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.671186+0000", 2024-09-21T15:49:54.653 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1191000000000002e-05 2024-09-21T15:49:54.653 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.654 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.654 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-09-21T15:49:54.654 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.671346+0000", 2024-09-21T15:49:54.654 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015980399999999999 2024-09-21T15:49:54.654 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.654 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.654 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:49:54.654 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.672221+0000", 2024-09-21T15:49:54.654 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00087471999999999995 2024-09-21T15:49:54.654 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.654 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.654 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:49:54.654 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.675619+0000", 2024-09-21T15:49:54.654 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00339858 2024-09-21T15:49:54.655 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.655 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.655 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:49:54.655 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.675670+0000", 2024-09-21T15:49:54.655 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0664000000000001e-05 2024-09-21T15:49:54.655 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.655 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.655 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:54.655 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.675768+0000", 2024-09-21T15:49:54.655 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7415000000000002e-05 2024-09-21T15:49:54.655 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.655 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:54.655 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.655 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.655 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.655 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28537 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 409600~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:54.656 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.227783+0000", 2024-09-21T15:49:54.656 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.418937128, 2024-09-21T15:49:54.656 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.046702573999999997, 2024-09-21T15:49:54.656 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:54.656 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:54.656 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:54.656 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:54.656 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:54.656 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28537 2024-09-21T15:49:54.656 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.656 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:54.656 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.656 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:54.656 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227783+0000", 2024-09-21T15:49:54.656 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.656 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.657 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.657 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:54.657 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227783+0000", 2024-09-21T15:49:54.657 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.657 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.657 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.657 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:54.657 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227782+0000", 2024-09-21T15:49:54.657 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:54.657 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.657 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.657 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:54.657 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227785+0000", 2024-09-21T15:49:54.657 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.34e-06 2024-09-21T15:49:54.657 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.658 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.658 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:54.658 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227791+0000", 2024-09-21T15:49:54.658 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2020000000000001e-06 2024-09-21T15:49:54.658 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.658 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.658 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:54.658 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227796+0000", 2024-09-21T15:49:54.658 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8289999999999998e-06 2024-09-21T15:49:54.658 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.658 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.658 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:54.658 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.274387+0000", 2024-09-21T15:49:54.658 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.046591024000000002 2024-09-21T15:49:54.658 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.658 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.659 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:54.659 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.274427+0000", 2024-09-21T15:49:54.659 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0825000000000003e-05 2024-09-21T15:49:54.659 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.659 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.659 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:54.659 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.274485+0000", 2024-09-21T15:49:54.659 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7713000000000001e-05 2024-09-21T15:49:54.659 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.659 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:54.659 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.659 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.659 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.659 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28538 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 1212416~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:54.659 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.227800+0000", 2024-09-21T15:49:54.660 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.418919747, 2024-09-21T15:49:54.660 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.046874213999999997, 2024-09-21T15:49:54.660 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:54.660 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:54.660 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:54.660 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:54.660 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:54.660 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28538 2024-09-21T15:49:54.660 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.660 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:54.660 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.660 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:54.660 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227800+0000", 2024-09-21T15:49:54.660 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.660 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.660 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.661 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:54.661 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227800+0000", 2024-09-21T15:49:54.661 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.661 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.661 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.661 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:54.661 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227800+0000", 2024-09-21T15:49:54.661 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:54.661 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.661 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.661 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:54.661 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227809+0000", 2024-09-21T15:49:54.661 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4709999999999996e-06 2024-09-21T15:49:54.661 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.661 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.661 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:54.662 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227816+0000", 2024-09-21T15:49:54.662 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0809999999999996e-06 2024-09-21T15:49:54.662 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.662 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.662 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:54.662 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227821+0000", 2024-09-21T15:49:54.662 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9420000000000003e-06 2024-09-21T15:49:54.662 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.662 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.662 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:54.662 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.274515+0000", 2024-09-21T15:49:54.662 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.046694201999999997 2024-09-21T15:49:54.662 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.662 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.662 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:54.663 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.274585+0000", 2024-09-21T15:49:54.663 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9250000000000003e-05 2024-09-21T15:49:54.663 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.663 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.663 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:54.663 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.274674+0000", 2024-09-21T15:49:54.663 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9621000000000001e-05 2024-09-21T15:49:54.663 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.663 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:54.663 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.663 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.663 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.663 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28539 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 2015232~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:54.663 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.227826+0000", 2024-09-21T15:49:54.664 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.418893744999998, 2024-09-21T15:49:54.664 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.046969221999999998, 2024-09-21T15:49:54.664 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:54.664 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:54.664 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:54.664 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:54.664 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:54.664 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28539 2024-09-21T15:49:54.664 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.664 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:54.664 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.664 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:54.664 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227826+0000", 2024-09-21T15:49:54.664 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.664 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.665 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.665 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:54.665 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227826+0000", 2024-09-21T15:49:54.665 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.665 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.665 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.665 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:54.665 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227826+0000", 2024-09-21T15:49:54.665 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:54.665 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.665 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.665 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:54.665 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227828+0000", 2024-09-21T15:49:54.665 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4040000000000002e-06 2024-09-21T15:49:54.665 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.666 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.666 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:54.666 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227835+0000", 2024-09-21T15:49:54.666 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6100000000000002e-06 2024-09-21T15:49:54.666 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.666 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.666 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:54.666 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227839+0000", 2024-09-21T15:49:54.666 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7589999999999997e-06 2024-09-21T15:49:54.666 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.666 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.666 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:54.666 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.274701+0000", 2024-09-21T15:49:54.666 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.046861828000000001 2024-09-21T15:49:54.667 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.667 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.667 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:54.667 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.274742+0000", 2024-09-21T15:49:54.667 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1074e-05 2024-09-21T15:49:54.667 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.667 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.667 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:54.667 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.274795+0000", 2024-09-21T15:49:54.667 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2945999999999998e-05 2024-09-21T15:49:54.667 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.667 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:54.667 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.667 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.667 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.668 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28540 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 2760704~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:54.668 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.227844+0000", 2024-09-21T15:49:54.668 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.418875736, 2024-09-21T15:49:54.668 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047146018999999997, 2024-09-21T15:49:54.668 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:54.668 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:54.668 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:54.668 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:54.668 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:54.668 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28540 2024-09-21T15:49:54.668 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.668 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:54.668 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.668 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:54.668 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227844+0000", 2024-09-21T15:49:54.669 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.669 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.669 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.669 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:54.669 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227844+0000", 2024-09-21T15:49:54.669 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.669 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.669 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.669 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:54.669 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227844+0000", 2024-09-21T15:49:54.669 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:54.669 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.669 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.669 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:54.669 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227846+0000", 2024-09-21T15:49:54.669 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1959999999999998e-06 2024-09-21T15:49:54.670 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.670 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.670 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:54.670 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227852+0000", 2024-09-21T15:49:54.670 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1659999999999999e-06 2024-09-21T15:49:54.670 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.670 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.670 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:54.670 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227856+0000", 2024-09-21T15:49:54.670 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4789999999999998e-06 2024-09-21T15:49:54.670 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.670 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.670 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:54.671 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.274825+0000", 2024-09-21T15:49:54.671 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.046968325999999998 2024-09-21T15:49:54.671 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.671 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.671 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:54.671 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.274896+0000", 2024-09-21T15:49:54.671 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0859999999999996e-05 2024-09-21T15:49:54.671 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.671 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.671 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:54.671 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.274990+0000", 2024-09-21T15:49:54.671 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4345999999999994e-05 2024-09-21T15:49:54.671 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.671 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:54.672 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.672 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.672 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.672 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28541 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 3563520~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:54.672 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.227861+0000", 2024-09-21T15:49:54.672 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.418858747000002, 2024-09-21T15:49:54.672 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047277444000000002, 2024-09-21T15:49:54.672 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:54.672 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:54.672 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:54.672 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:54.672 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:54.672 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28541 2024-09-21T15:49:54.672 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.672 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:54.673 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.673 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:54.673 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227861+0000", 2024-09-21T15:49:54.673 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.673 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.673 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.673 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:54.673 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227861+0000", 2024-09-21T15:49:54.673 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.673 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.673 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.673 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:54.673 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227861+0000", 2024-09-21T15:49:54.673 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:54.673 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.674 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.674 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:54.674 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227863+0000", 2024-09-21T15:49:54.674 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0789999999999999e-06 2024-09-21T15:49:54.674 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.674 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.674 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:54.674 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227870+0000", 2024-09-21T15:49:54.674 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8070000000000002e-06 2024-09-21T15:49:54.674 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.674 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.674 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:54.674 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227874+0000", 2024-09-21T15:49:54.675 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9599999999999999e-06 2024-09-21T15:49:54.675 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.675 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.675 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:54.675 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275002+0000", 2024-09-21T15:49:54.675 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047127251000000002 2024-09-21T15:49:54.675 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.675 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.675 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:54.675 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275049+0000", 2024-09-21T15:49:54.675 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7592000000000001e-05 2024-09-21T15:49:54.675 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.675 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.675 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:54.676 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275138+0000", 2024-09-21T15:49:54.676 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9104000000000006e-05 2024-09-21T15:49:54.676 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.676 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:54.676 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.676 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.676 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.676 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28542 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 4366336~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:54.676 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.227879+0000", 2024-09-21T15:49:54.676 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.418840787000001, 2024-09-21T15:49:54.676 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047394002999999997, 2024-09-21T15:49:54.676 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:54.676 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:54.676 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:54.677 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:54.677 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:54.677 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28542 2024-09-21T15:49:54.677 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.677 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:54.677 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.677 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:54.677 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227879+0000", 2024-09-21T15:49:54.677 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.677 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.677 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.677 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:54.677 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227879+0000", 2024-09-21T15:49:54.677 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.678 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.678 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.678 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:54.678 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227878+0000", 2024-09-21T15:49:54.678 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:54.678 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.678 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.678 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:54.678 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227881+0000", 2024-09-21T15:49:54.678 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2340000000000001e-06 2024-09-21T15:49:54.678 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.678 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.678 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:54.679 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227887+0000", 2024-09-21T15:49:54.679 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.049e-06 2024-09-21T15:49:54.679 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.679 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.679 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:54.679 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227891+0000", 2024-09-21T15:49:54.679 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5480000000000003e-06 2024-09-21T15:49:54.679 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.679 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.679 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:54.679 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275168+0000", 2024-09-21T15:49:54.679 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047276986999999999 2024-09-21T15:49:54.679 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.679 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.679 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:54.680 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275219+0000", 2024-09-21T15:49:54.680 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1010000000000001e-05 2024-09-21T15:49:54.680 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.680 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.680 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:54.680 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275273+0000", 2024-09-21T15:49:54.680 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3649000000000001e-05 2024-09-21T15:49:54.680 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.680 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:54.680 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.680 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.680 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.680 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28544 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 5111808~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:54.680 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.227895+0000", 2024-09-21T15:49:54.680 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.418824528999998, 2024-09-21T15:49:54.681 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047502682999999997, 2024-09-21T15:49:54.681 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:54.681 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:54.681 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:54.681 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:54.681 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:54.681 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28544 2024-09-21T15:49:54.681 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.681 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:54.681 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.681 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:54.681 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227895+0000", 2024-09-21T15:49:54.681 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.681 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.681 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.681 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:54.682 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227895+0000", 2024-09-21T15:49:54.682 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.682 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.682 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.682 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:54.682 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227895+0000", 2024-09-21T15:49:54.682 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:54.682 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.682 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.682 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:54.682 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227897+0000", 2024-09-21T15:49:54.682 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.153e-06 2024-09-21T15:49:54.682 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.682 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.682 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:54.682 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227903+0000", 2024-09-21T15:49:54.683 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1940000000000003e-06 2024-09-21T15:49:54.683 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.683 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.683 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:54.683 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227908+0000", 2024-09-21T15:49:54.683 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6110000000000004e-06 2024-09-21T15:49:54.683 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.683 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.683 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:54.683 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275290+0000", 2024-09-21T15:49:54.683 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047381805999999999 2024-09-21T15:49:54.683 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.683 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.683 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:54.683 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275346+0000", 2024-09-21T15:49:54.683 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6313000000000001e-05 2024-09-21T15:49:54.684 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.684 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.684 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:54.684 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275398+0000", 2024-09-21T15:49:54.684 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2027000000000002e-05 2024-09-21T15:49:54.684 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.684 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:54.684 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.684 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.684 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.684 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28545 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 5169152~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:54.684 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.227912+0000", 2024-09-21T15:49:54.684 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.418807908000002, 2024-09-21T15:49:54.684 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047684807000000003, 2024-09-21T15:49:54.684 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:54.684 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:54.685 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:54.685 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:54.685 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:54.685 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28545 2024-09-21T15:49:54.685 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.685 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:54.685 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.685 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:54.685 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227912+0000", 2024-09-21T15:49:54.685 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.685 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.685 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.685 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:54.685 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227912+0000", 2024-09-21T15:49:54.685 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.685 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.686 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.686 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:54.686 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227911+0000", 2024-09-21T15:49:54.686 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:54.686 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.686 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.686 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:54.686 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227913+0000", 2024-09-21T15:49:54.686 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9970000000000001e-06 2024-09-21T15:49:54.686 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.686 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.686 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:54.686 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227920+0000", 2024-09-21T15:49:54.686 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0499999999999997e-06 2024-09-21T15:49:54.686 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.686 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.686 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:54.687 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227924+0000", 2024-09-21T15:49:54.687 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7219999999999999e-06 2024-09-21T15:49:54.687 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.687 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.687 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:54.687 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275424+0000", 2024-09-21T15:49:54.687 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047499574000000003 2024-09-21T15:49:54.687 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.687 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.687 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:54.687 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275495+0000", 2024-09-21T15:49:54.687 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1347999999999995e-05 2024-09-21T15:49:54.687 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.687 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.687 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:54.688 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275597+0000", 2024-09-21T15:49:54.688 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101446 2024-09-21T15:49:54.688 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.688 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:54.688 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.688 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.688 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.688 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28546 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 5914624~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:54.688 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.227929+0000", 2024-09-21T15:49:54.688 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.418790661999999, 2024-09-21T15:49:54.688 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047783297000000002, 2024-09-21T15:49:54.688 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:54.688 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:54.688 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:54.688 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:54.688 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:54.689 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28546 2024-09-21T15:49:54.689 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.689 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:54.689 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.689 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:54.689 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227929+0000", 2024-09-21T15:49:54.689 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.689 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.689 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.689 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:54.689 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227929+0000", 2024-09-21T15:49:54.689 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.689 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.689 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.689 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:54.689 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227929+0000", 2024-09-21T15:49:54.689 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:54.690 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.690 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.690 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:54.690 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227931+0000", 2024-09-21T15:49:54.690 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2529999999999999e-06 2024-09-21T15:49:54.690 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.690 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.690 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:54.690 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227937+0000", 2024-09-21T15:49:54.690 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9699999999999996e-06 2024-09-21T15:49:54.690 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.690 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.690 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:54.690 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227942+0000", 2024-09-21T15:49:54.690 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7650000000000001e-06 2024-09-21T15:49:54.691 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.691 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.691 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:54.691 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275621+0000", 2024-09-21T15:49:54.691 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047679694000000002 2024-09-21T15:49:54.691 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.691 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.691 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:54.691 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275661+0000", 2024-09-21T15:49:54.691 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9363999999999998e-05 2024-09-21T15:49:54.691 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.691 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.691 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:54.691 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275712+0000", 2024-09-21T15:49:54.692 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1598999999999998e-05 2024-09-21T15:49:54.692 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.692 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:54.692 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.692 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.692 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.692 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28547 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 6717440~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:54.692 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.228080+0000", 2024-09-21T15:49:54.692 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.418639384999999, 2024-09-21T15:49:54.692 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047744609, 2024-09-21T15:49:54.692 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:54.692 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:54.692 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:54.692 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:54.692 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:54.693 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28547 2024-09-21T15:49:54.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.693 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:54.693 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.693 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:54.693 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228080+0000", 2024-09-21T15:49:54.693 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.693 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.693 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:54.693 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228080+0000", 2024-09-21T15:49:54.693 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.693 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.693 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:54.694 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.227946+0000", 2024-09-21T15:49:54.694 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999866 2024-09-21T15:49:54.694 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.694 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.694 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:54.694 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228084+0000", 2024-09-21T15:49:54.694 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000137517 2024-09-21T15:49:54.694 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.694 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.694 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:54.694 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228093+0000", 2024-09-21T15:49:54.694 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.645e-06 2024-09-21T15:49:54.694 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.694 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.695 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:54.695 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228101+0000", 2024-09-21T15:49:54.695 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9559999999999997e-06 2024-09-21T15:49:54.695 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.695 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.695 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:54.695 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275735+0000", 2024-09-21T15:49:54.695 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047633928999999998 2024-09-21T15:49:54.695 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.695 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.695 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:54.695 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275774+0000", 2024-09-21T15:49:54.695 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8699999999999999e-05 2024-09-21T15:49:54.695 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.695 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.695 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:54.695 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275825+0000", 2024-09-21T15:49:54.696 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0926e-05 2024-09-21T15:49:54.696 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.696 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:54.696 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.696 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.696 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.696 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28549 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 417792~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:54.696 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.228847+0000", 2024-09-21T15:49:54.696 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.417872280000001, 2024-09-21T15:49:54.696 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047086515000000002, 2024-09-21T15:49:54.696 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:54.696 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:54.696 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:54.696 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:54.696 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:54.696 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28549 2024-09-21T15:49:54.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.697 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:54.697 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.697 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:54.697 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228847+0000", 2024-09-21T15:49:54.697 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.697 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.697 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:54.697 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228847+0000", 2024-09-21T15:49:54.697 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.697 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.697 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:54.697 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228107+0000", 2024-09-21T15:49:54.697 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9992595 2024-09-21T15:49:54.698 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.698 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.698 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:54.698 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228852+0000", 2024-09-21T15:49:54.698 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00074505000000000003 2024-09-21T15:49:54.698 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.698 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.698 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:54.698 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228871+0000", 2024-09-21T15:49:54.698 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8635e-05 2024-09-21T15:49:54.698 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.698 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.698 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:54.698 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228880+0000", 2024-09-21T15:49:54.698 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9849999999999995e-06 2024-09-21T15:49:54.698 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.698 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.699 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:54.699 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275847+0000", 2024-09-21T15:49:54.699 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.046967239000000001 2024-09-21T15:49:54.699 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.699 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.699 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:54.699 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275885+0000", 2024-09-21T15:49:54.699 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8274e-05 2024-09-21T15:49:54.699 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.699 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.699 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:54.699 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275934+0000", 2024-09-21T15:49:54.699 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8764999999999998e-05 2024-09-21T15:49:54.699 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.699 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:54.699 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.700 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.700 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.700 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28551 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 1220608~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:54.700 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.228886+0000", 2024-09-21T15:49:54.700 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.417833234, 2024-09-21T15:49:54.700 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047166819999999998, 2024-09-21T15:49:54.700 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:54.700 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:54.700 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:54.700 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:54.700 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:54.700 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28551 2024-09-21T15:49:54.700 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.700 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:54.700 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.701 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:54.701 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228886+0000", 2024-09-21T15:49:54.701 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.701 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.701 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.701 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:54.701 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228886+0000", 2024-09-21T15:49:54.701 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.701 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.701 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.701 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:54.701 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228886+0000", 2024-09-21T15:49:54.701 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:54.701 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.701 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.702 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:54.702 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228889+0000", 2024-09-21T15:49:54.702 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8650000000000001e-06 2024-09-21T15:49:54.702 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.702 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.702 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:54.702 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228896+0000", 2024-09-21T15:49:54.702 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0790000000000003e-06 2024-09-21T15:49:54.702 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.702 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.702 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:54.702 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228903+0000", 2024-09-21T15:49:54.702 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9770000000000003e-06 2024-09-21T15:49:54.702 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.703 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.703 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:54.703 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.275954+0000", 2024-09-21T15:49:54.703 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047050986000000003 2024-09-21T15:49:54.703 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.703 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.703 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:54.703 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276001+0000", 2024-09-21T15:49:54.703 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7527000000000001e-05 2024-09-21T15:49:54.703 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.703 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.703 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:54.703 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276053+0000", 2024-09-21T15:49:54.703 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1854999999999997e-05 2024-09-21T15:49:54.703 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.704 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:54.704 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.704 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.704 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.704 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28552 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 1966080~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:54.704 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.228908+0000", 2024-09-21T15:49:54.704 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.417811949000001, 2024-09-21T15:49:54.704 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047256193000000002, 2024-09-21T15:49:54.704 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:54.704 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:54.704 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:54.704 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:54.704 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:54.704 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28552 2024-09-21T15:49:54.705 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.705 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:54.705 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.705 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:54.705 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228908+0000", 2024-09-21T15:49:54.705 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.705 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.705 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.705 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:54.705 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228908+0000", 2024-09-21T15:49:54.705 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.705 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.705 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.705 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:54.706 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228907+0000", 2024-09-21T15:49:54.706 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:54.706 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.706 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.706 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:54.706 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228910+0000", 2024-09-21T15:49:54.706 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3819999999999998e-06 2024-09-21T15:49:54.706 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.706 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.706 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:54.706 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228916+0000", 2024-09-21T15:49:54.706 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6390000000000003e-06 2024-09-21T15:49:54.706 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.706 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.706 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:54.707 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228921+0000", 2024-09-21T15:49:54.707 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0350000000000001e-06 2024-09-21T15:49:54.707 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.707 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.707 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:54.707 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276077+0000", 2024-09-21T15:49:54.707 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047155326999999997 2024-09-21T15:49:54.707 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.707 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.707 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:54.707 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276114+0000", 2024-09-21T15:49:54.707 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6746999999999999e-05 2024-09-21T15:49:54.707 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.707 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.707 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:54.708 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276164+0000", 2024-09-21T15:49:54.708 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0438000000000001e-05 2024-09-21T15:49:54.708 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.708 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:54.708 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.708 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.708 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.708 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28553 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 2023424~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:54.708 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.228926+0000", 2024-09-21T15:49:54.708 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.417793743000001, 2024-09-21T15:49:54.708 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047351920999999998, 2024-09-21T15:49:54.708 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:54.708 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:54.708 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:54.708 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:54.708 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:54.709 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28553 2024-09-21T15:49:54.709 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.709 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:54.709 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.709 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:54.709 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228926+0000", 2024-09-21T15:49:54.709 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.709 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.709 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.709 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:54.709 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228926+0000", 2024-09-21T15:49:54.709 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.709 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.709 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.709 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:54.709 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228926+0000", 2024-09-21T15:49:54.710 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:54.710 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.710 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.710 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:54.710 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228932+0000", 2024-09-21T15:49:54.710 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6930000000000001e-06 2024-09-21T15:49:54.710 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.710 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.710 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:54.710 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228939+0000", 2024-09-21T15:49:54.710 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8199999999999999e-06 2024-09-21T15:49:54.710 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.710 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.710 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:54.710 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228953+0000", 2024-09-21T15:49:54.710 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.347e-05 2024-09-21T15:49:54.710 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.711 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.711 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:54.711 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276187+0000", 2024-09-21T15:49:54.711 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047234405 2024-09-21T15:49:54.711 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.711 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.711 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:54.711 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276228+0000", 2024-09-21T15:49:54.711 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1010000000000002e-05 2024-09-21T15:49:54.711 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.711 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.711 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:54.711 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276278+0000", 2024-09-21T15:49:54.711 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9874999999999999e-05 2024-09-21T15:49:54.711 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.711 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:54.711 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.712 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.712 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.712 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28554 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 2768896~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:54.712 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.228991+0000", 2024-09-21T15:49:54.712 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.417728266000001, 2024-09-21T15:49:54.712 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047394177000000003, 2024-09-21T15:49:54.712 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:54.712 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:54.712 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:54.712 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:54.712 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:54.712 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28554 2024-09-21T15:49:54.712 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.712 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:54.712 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.712 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:54.713 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228991+0000", 2024-09-21T15:49:54.713 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.713 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.713 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.713 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:54.713 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228991+0000", 2024-09-21T15:49:54.713 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.713 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.713 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.713 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:54.713 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228991+0000", 2024-09-21T15:49:54.713 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:54.713 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.713 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.713 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:54.713 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.228994+0000", 2024-09-21T15:49:54.714 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.422e-06 2024-09-21T15:49:54.714 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.714 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.714 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:54.714 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229003+0000", 2024-09-21T15:49:54.714 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8699999999999998e-06 2024-09-21T15:49:54.714 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.714 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.714 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:54.714 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229009+0000", 2024-09-21T15:49:54.714 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3099999999999997e-06 2024-09-21T15:49:54.714 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.714 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.714 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:54.714 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276301+0000", 2024-09-21T15:49:54.714 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047291305999999998 2024-09-21T15:49:54.715 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.715 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.715 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:54.715 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276337+0000", 2024-09-21T15:49:54.715 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6378999999999998e-05 2024-09-21T15:49:54.715 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.715 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.715 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:54.715 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276386+0000", 2024-09-21T15:49:54.715 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8535999999999997e-05 2024-09-21T15:49:54.715 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.715 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:54.715 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.715 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.715 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.715 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28555 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 3571712~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:54.716 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.229015+0000", 2024-09-21T15:49:54.716 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.417705172000002, 2024-09-21T15:49:54.716 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047478713999999998, 2024-09-21T15:49:54.716 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:54.716 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:54.716 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:54.716 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:54.716 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:54.716 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28555 2024-09-21T15:49:54.716 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.716 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:54.716 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.716 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:54.716 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229015+0000", 2024-09-21T15:49:54.716 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.716 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.717 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.717 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:54.717 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229015+0000", 2024-09-21T15:49:54.717 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.717 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.717 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.717 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:54.717 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229014+0000", 2024-09-21T15:49:54.717 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:54.717 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.717 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.717 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:54.717 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229029+0000", 2024-09-21T15:49:54.717 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4503e-05 2024-09-21T15:49:54.717 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.718 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.718 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:54.718 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229036+0000", 2024-09-21T15:49:54.718 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5989999999999996e-06 2024-09-21T15:49:54.718 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.718 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.718 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:54.718 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229042+0000", 2024-09-21T15:49:54.718 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6699999999999999e-06 2024-09-21T15:49:54.718 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.718 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.718 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:54.718 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276407+0000", 2024-09-21T15:49:54.718 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047365646999999997 2024-09-21T15:49:54.718 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.718 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.719 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:54.719 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276445+0000", 2024-09-21T15:49:54.719 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7064000000000003e-05 2024-09-21T15:49:54.719 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.719 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.719 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:54.719 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276493+0000", 2024-09-21T15:49:54.719 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8704e-05 2024-09-21T15:49:54.719 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.719 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:54.719 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.719 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.719 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.719 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28556 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 4374528~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:54.719 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.229183+0000", 2024-09-21T15:49:54.719 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.417536381000001, 2024-09-21T15:49:54.720 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047417709000000002, 2024-09-21T15:49:54.720 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:54.720 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:54.720 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:54.720 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:54.720 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:54.720 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28556 2024-09-21T15:49:54.720 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.720 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:54.720 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.720 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:54.720 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229183+0000", 2024-09-21T15:49:54.720 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.720 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.720 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.720 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:54.720 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229183+0000", 2024-09-21T15:49:54.721 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.721 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.721 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.721 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:54.721 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229046+0000", 2024-09-21T15:49:54.721 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9998631 2024-09-21T15:49:54.721 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.721 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.721 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:54.721 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229186+0000", 2024-09-21T15:49:54.721 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014003900000000001 2024-09-21T15:49:54.721 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.721 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.721 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:54.721 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229195+0000", 2024-09-21T15:49:54.721 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.422e-06 2024-09-21T15:49:54.721 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.722 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.722 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:54.722 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229199+0000", 2024-09-21T15:49:54.722 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5379999999999996e-06 2024-09-21T15:49:54.722 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.722 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.722 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:54.722 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276516+0000", 2024-09-21T15:49:54.722 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047316432999999998 2024-09-21T15:49:54.722 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.722 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.722 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:54.722 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276552+0000", 2024-09-21T15:49:54.722 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6019000000000002e-05 2024-09-21T15:49:54.722 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.723 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.723 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:54.723 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276601+0000", 2024-09-21T15:49:54.723 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9172999999999998e-05 2024-09-21T15:49:54.723 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.723 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:54.723 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.723 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.723 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.723 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28558 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 5120000~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:54.723 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.229531+0000", 2024-09-21T15:49:54.723 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.417188498000002, 2024-09-21T15:49:54.723 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047179666000000002, 2024-09-21T15:49:54.723 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:54.723 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:54.723 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:54.724 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:54.724 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:54.724 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28558 2024-09-21T15:49:54.724 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.724 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:54.724 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.724 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:54.724 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229531+0000", 2024-09-21T15:49:54.724 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.724 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.724 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.724 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:54.724 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229531+0000", 2024-09-21T15:49:54.724 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.724 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.725 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.725 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:54.725 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229203+0000", 2024-09-21T15:49:54.725 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9996719 2024-09-21T15:49:54.725 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.725 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.725 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:54.725 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229536+0000", 2024-09-21T15:49:54.725 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000332885 2024-09-21T15:49:54.725 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.725 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.725 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:54.725 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229548+0000", 2024-09-21T15:49:54.725 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1861999999999999e-05 2024-09-21T15:49:54.725 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.726 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.726 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:54.726 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229555+0000", 2024-09-21T15:49:54.726 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6009999999999997e-06 2024-09-21T15:49:54.726 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.726 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.726 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:54.726 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276624+0000", 2024-09-21T15:49:54.726 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047068250999999998 2024-09-21T15:49:54.726 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.726 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.726 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:54.726 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276662+0000", 2024-09-21T15:49:54.726 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8705999999999998e-05 2024-09-21T15:49:54.726 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.726 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.726 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:54.727 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276711+0000", 2024-09-21T15:49:54.727 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8443999999999999e-05 2024-09-21T15:49:54.727 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.727 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:54.727 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.727 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.727 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.727 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:28560 3.8 3:15cf86d6:::rbd_data.11331466bd23.0000000000000000:9 [read 5922816~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e39)", 2024-09-21T15:49:54.727 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:27.229562+0000", 2024-09-21T15:49:54.727 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.417157292999999, 2024-09-21T15:49:54.727 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047255377000000001, 2024-09-21T15:49:54.727 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:49:54.727 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:49:54.727 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:49:54.727 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:49:54.727 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:54.727 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 28560 2024-09-21T15:49:54.728 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.728 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:49:54.728 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.728 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:49:54.728 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229562+0000", 2024-09-21T15:49:54.728 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.728 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.728 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.728 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:49:54.728 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229562+0000", 2024-09-21T15:49:54.728 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:49:54.728 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.728 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.728 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:49:54.728 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229562+0000", 2024-09-21T15:49:54.729 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:54.729 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.729 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.729 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:49:54.729 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229565+0000", 2024-09-21T15:49:54.729 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9450000000000002e-06 2024-09-21T15:49:54.729 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.729 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.729 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:49:54.729 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229572+0000", 2024-09-21T15:49:54.729 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4240000000000004e-06 2024-09-21T15:49:54.729 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.729 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.729 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:49:54.729 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.229577+0000", 2024-09-21T15:49:54.729 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2000000000000002e-06 2024-09-21T15:49:54.729 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.730 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.730 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:49:54.730 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276733+0000", 2024-09-21T15:49:54.730 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.047155202 2024-09-21T15:49:54.730 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.730 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.730 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:49:54.730 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276769+0000", 2024-09-21T15:49:54.730 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6013000000000002e-05 2024-09-21T15:49:54.730 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:49:54.730 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:49:54.730 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:49:54.730 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:27.276818+0000", 2024-09-21T15:49:54.730 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9178000000000002e-05 2024-09-21T15:49:54.730 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.730 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:54.731 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.731 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:49:54.731 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:49:54.731 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:49:54.732 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:49:54.775 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:54.774+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:54.834 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:54.835 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:54.835 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:49:54.835 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:54.837 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:54.835+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:54.843 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:49:54.925 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is set 2024-09-21T15:49:54.925 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T15:49:54.925 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T15:49:54.925 INFO:tasks.thrashosds.thrasher:Reweighting osd 2 to 0.30419318544284035 2024-09-21T15:49:54.926 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 0.30419318544284035 2024-09-21T15:49:54.940 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:54.939+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:54.991 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:54.991 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:54.991 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:49:54.991 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:49:54.991 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:55.000 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:49:55.018 INFO:teuthology.orchestra.run.smithi089.stdout:truncating image image_client.0-clone24 from 0xe0b5c9e (overlap 0x1c16d3) to 0x2cf15 2024-09-21T15:49:55.037 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:55.036+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:55.110 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:55.111 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:49:55.111 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:49:55.111 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:49:55.111 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.111 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26224 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 2326528~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:55.111 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.504487+0000", 2024-09-21T15:49:55.111 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 28.604501624000001, 2024-09-21T15:49:55.111 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032819441999999997, 2024-09-21T15:49:55.111 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:55.111 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:55.111 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:55.112 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:55.112 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:55.112 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26224 2024-09-21T15:49:55.112 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.112 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:55.112 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.112 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:55.112 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504487+0000", 2024-09-21T15:49:55.112 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.112 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.112 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.112 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:55.112 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504487+0000", 2024-09-21T15:49:55.112 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.113 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.113 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.113 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:55.113 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504487+0000", 2024-09-21T15:49:55.113 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:55.113 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.113 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.113 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:55.113 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504490+0000", 2024-09-21T15:49:55.113 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6049999999999998e-06 2024-09-21T15:49:55.113 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.113 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.113 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:55.113 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504493+0000", 2024-09-21T15:49:55.113 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1200000000000002e-06 2024-09-21T15:49:55.114 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.114 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.114 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:55.114 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504496+0000", 2024-09-21T15:49:55.114 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.977e-06 2024-09-21T15:49:55.114 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.114 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.114 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:55.114 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537221+0000", 2024-09-21T15:49:55.114 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032724529000000002 2024-09-21T15:49:55.114 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.114 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.114 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:55.114 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537259+0000", 2024-09-21T15:49:55.115 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7923999999999998e-05 2024-09-21T15:49:55.115 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.115 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.115 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:55.115 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537306+0000", 2024-09-21T15:49:55.115 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7512999999999997e-05 2024-09-21T15:49:55.115 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.115 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:55.115 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.115 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.115 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.115 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26225 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 2383872~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:55.115 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.504498+0000", 2024-09-21T15:49:55.116 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 28.604490054999999, 2024-09-21T15:49:55.116 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.03291703, 2024-09-21T15:49:55.116 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:55.116 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:55.116 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:55.116 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:55.116 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:55.116 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26225 2024-09-21T15:49:55.116 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.116 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:55.116 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.116 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:55.116 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504498+0000", 2024-09-21T15:49:55.116 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.116 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.117 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.117 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:55.117 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504498+0000", 2024-09-21T15:49:55.117 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.117 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.117 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.117 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:55.117 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504498+0000", 2024-09-21T15:49:55.117 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:55.117 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.117 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.117 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:55.117 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504499+0000", 2024-09-21T15:49:55.117 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0070000000000001e-06 2024-09-21T15:49:55.118 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.118 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.118 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:55.118 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504502+0000", 2024-09-21T15:49:55.118 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8119999999999999e-06 2024-09-21T15:49:55.118 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.118 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.118 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:55.118 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504504+0000", 2024-09-21T15:49:55.118 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4380000000000002e-06 2024-09-21T15:49:55.118 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.118 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.118 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:55.118 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537328+0000", 2024-09-21T15:49:55.119 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032823659999999998 2024-09-21T15:49:55.119 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.119 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.119 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:55.119 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537366+0000", 2024-09-21T15:49:55.119 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8083000000000001e-05 2024-09-21T15:49:55.119 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.119 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.119 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:55.119 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537415+0000", 2024-09-21T15:49:55.119 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9233000000000001e-05 2024-09-21T15:49:55.119 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.119 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:55.119 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.119 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.120 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.120 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26226 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 2441216~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:55.120 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.504506+0000", 2024-09-21T15:49:55.120 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 28.604481939999999, 2024-09-21T15:49:55.120 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033016726000000003, 2024-09-21T15:49:55.120 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:55.120 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:55.120 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:55.120 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:55.120 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:55.120 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26226 2024-09-21T15:49:55.120 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.120 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:55.120 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.121 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:55.121 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504506+0000", 2024-09-21T15:49:55.121 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.121 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.121 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.121 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:55.121 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504506+0000", 2024-09-21T15:49:55.121 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.121 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.121 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.121 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:55.121 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504506+0000", 2024-09-21T15:49:55.121 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:55.121 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.121 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.122 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:55.122 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504507+0000", 2024-09-21T15:49:55.122 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9699999999999994e-07 2024-09-21T15:49:55.122 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.122 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.122 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:55.122 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504510+0000", 2024-09-21T15:49:55.122 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.773e-06 2024-09-21T15:49:55.122 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.122 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.122 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:55.122 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504514+0000", 2024-09-21T15:49:55.122 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1150000000000004e-06 2024-09-21T15:49:55.122 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.123 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.123 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:55.123 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537437+0000", 2024-09-21T15:49:55.123 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032923249000000002 2024-09-21T15:49:55.123 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.123 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.123 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:55.123 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537476+0000", 2024-09-21T15:49:55.123 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8679e-05 2024-09-21T15:49:55.123 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.123 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.123 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:55.123 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537523+0000", 2024-09-21T15:49:55.123 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7119000000000001e-05 2024-09-21T15:49:55.124 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.124 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:55.124 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.124 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.124 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.124 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26227 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 2498560~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:55.124 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.504516+0000", 2024-09-21T15:49:55.124 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 28.604472339000001, 2024-09-21T15:49:55.124 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033117681000000003, 2024-09-21T15:49:55.124 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:55.124 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:55.124 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:55.124 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:55.124 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:55.125 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26227 2024-09-21T15:49:55.125 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.125 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:55.125 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.125 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:55.125 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504516+0000", 2024-09-21T15:49:55.125 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.125 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.125 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.125 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:55.125 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504516+0000", 2024-09-21T15:49:55.125 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.125 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.125 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.125 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:55.126 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504516+0000", 2024-09-21T15:49:55.126 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:55.126 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.126 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.126 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:55.126 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504517+0000", 2024-09-21T15:49:55.126 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0610000000000001e-06 2024-09-21T15:49:55.126 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.126 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.126 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:55.126 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504520+0000", 2024-09-21T15:49:55.126 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.83e-06 2024-09-21T15:49:55.126 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.126 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.127 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:55.127 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504522+0000", 2024-09-21T15:49:55.127 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.497e-06 2024-09-21T15:49:55.127 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.127 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.127 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:55.127 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537544+0000", 2024-09-21T15:49:55.127 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033021948000000002 2024-09-21T15:49:55.127 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.127 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.127 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:55.127 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537585+0000", 2024-09-21T15:49:55.127 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1094999999999999e-05 2024-09-21T15:49:55.128 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.128 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.128 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:55.128 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537634+0000", 2024-09-21T15:49:55.128 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8449000000000003e-05 2024-09-21T15:49:55.128 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.128 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:55.128 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.128 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.128 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.128 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26228 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 2555904~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:55.128 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.504524+0000", 2024-09-21T15:49:55.128 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 28.604464240999999, 2024-09-21T15:49:55.128 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.03321735, 2024-09-21T15:49:55.128 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:55.128 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:55.129 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:55.129 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:55.129 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:55.129 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26228 2024-09-21T15:49:55.129 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.129 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:55.129 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.129 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:55.129 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504524+0000", 2024-09-21T15:49:55.129 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.129 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.129 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.129 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:55.129 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504524+0000", 2024-09-21T15:49:55.130 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.130 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.130 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.130 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:55.130 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504524+0000", 2024-09-21T15:49:55.130 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:55.130 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.130 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.130 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:55.130 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504525+0000", 2024-09-21T15:49:55.130 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0419999999999999e-06 2024-09-21T15:49:55.130 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.130 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.130 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:55.130 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504528+0000", 2024-09-21T15:49:55.131 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7530000000000002e-06 2024-09-21T15:49:55.131 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.131 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.131 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:55.131 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504530+0000", 2024-09-21T15:49:55.131 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4530000000000001e-06 2024-09-21T15:49:55.131 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.131 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.131 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:55.131 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537656+0000", 2024-09-21T15:49:55.131 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033126023999999997 2024-09-21T15:49:55.131 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.131 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.131 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:55.131 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537694+0000", 2024-09-21T15:49:55.132 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7799000000000002e-05 2024-09-21T15:49:55.132 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.132 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.132 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:55.132 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537741+0000", 2024-09-21T15:49:55.132 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7484999999999997e-05 2024-09-21T15:49:55.132 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.132 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:55.132 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.132 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.132 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.132 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26229 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 2613248~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:55.132 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.504532+0000", 2024-09-21T15:49:55.132 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 28.604456362000001, 2024-09-21T15:49:55.133 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033315897999999997, 2024-09-21T15:49:55.133 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:55.133 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:55.133 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:55.133 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:55.133 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:55.133 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26229 2024-09-21T15:49:55.133 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.133 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:55.133 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.133 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:55.133 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504532+0000", 2024-09-21T15:49:55.133 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.133 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.133 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.134 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:55.134 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504532+0000", 2024-09-21T15:49:55.134 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.134 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.134 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.134 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:55.134 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504532+0000", 2024-09-21T15:49:55.134 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:55.134 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.134 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.134 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:55.134 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504533+0000", 2024-09-21T15:49:55.134 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0610000000000001e-06 2024-09-21T15:49:55.134 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.134 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.135 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:55.135 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504536+0000", 2024-09-21T15:49:55.135 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.309e-06 2024-09-21T15:49:55.135 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.135 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.135 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:55.135 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504539+0000", 2024-09-21T15:49:55.135 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.435e-06 2024-09-21T15:49:55.135 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.135 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.135 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:55.135 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537762+0000", 2024-09-21T15:49:55.135 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033223813999999997 2024-09-21T15:49:55.136 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.136 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.136 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:55.136 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537801+0000", 2024-09-21T15:49:55.136 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8136999999999998e-05 2024-09-21T15:49:55.136 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.136 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.136 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:55.136 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537848+0000", 2024-09-21T15:49:55.136 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7345999999999997e-05 2024-09-21T15:49:55.136 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.136 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:55.136 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.136 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.136 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.137 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26230 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 2670592~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:55.137 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.504606+0000", 2024-09-21T15:49:55.137 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 28.604382204, 2024-09-21T15:49:55.137 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033348801999999997, 2024-09-21T15:49:55.137 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:55.137 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:55.137 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:55.137 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:55.137 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:55.137 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26230 2024-09-21T15:49:55.137 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.137 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:55.137 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.137 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:55.137 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504606+0000", 2024-09-21T15:49:55.138 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.138 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.138 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.138 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:55.138 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504606+0000", 2024-09-21T15:49:55.138 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.138 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.138 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.138 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:55.138 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504541+0000", 2024-09-21T15:49:55.138 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999352 2024-09-21T15:49:55.138 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.138 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.138 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:55.139 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504608+0000", 2024-09-21T15:49:55.139 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6427000000000006e-05 2024-09-21T15:49:55.139 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.139 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.139 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:55.139 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504612+0000", 2024-09-21T15:49:55.139 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1799999999999998e-06 2024-09-21T15:49:55.139 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.139 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.139 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:55.139 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504615+0000", 2024-09-21T15:49:55.139 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0400000000000001e-06 2024-09-21T15:49:55.139 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.139 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.140 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:55.140 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537870+0000", 2024-09-21T15:49:55.140 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033254745000000002 2024-09-21T15:49:55.140 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.140 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.140 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:55.140 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537908+0000", 2024-09-21T15:49:55.140 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7882000000000001e-05 2024-09-21T15:49:55.140 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.140 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.140 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:55.140 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537955+0000", 2024-09-21T15:49:55.140 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7361999999999998e-05 2024-09-21T15:49:55.140 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.140 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:55.141 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.141 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.141 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.141 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26239 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 3186688~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:55.141 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.506097+0000", 2024-09-21T15:49:55.141 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 28.602891497999998, 2024-09-21T15:49:55.141 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032843331000000003, 2024-09-21T15:49:55.141 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:55.141 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:55.141 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:55.141 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:55.141 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:55.141 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26239 2024-09-21T15:49:55.141 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.141 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:55.141 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.141 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:55.142 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506097+0000", 2024-09-21T15:49:55.142 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.142 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.142 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.142 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:55.142 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506097+0000", 2024-09-21T15:49:55.142 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.142 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.142 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.142 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:55.142 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506097+0000", 2024-09-21T15:49:55.142 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:55.142 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.143 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.143 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:55.143 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506098+0000", 2024-09-21T15:49:55.143 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.079e-06 2024-09-21T15:49:55.143 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.143 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.143 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:55.143 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506101+0000", 2024-09-21T15:49:55.143 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8499999999999998e-06 2024-09-21T15:49:55.143 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.143 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.143 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:55.143 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506103+0000", 2024-09-21T15:49:55.143 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5770000000000001e-06 2024-09-21T15:49:55.143 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.143 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.144 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:55.144 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.538854+0000", 2024-09-21T15:49:55.144 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032750835999999998 2024-09-21T15:49:55.144 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.144 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.144 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:55.144 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.538892+0000", 2024-09-21T15:49:55.144 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7904000000000002e-05 2024-09-21T15:49:55.144 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.144 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.144 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:55.144 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.538940+0000", 2024-09-21T15:49:55.144 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8303999999999997e-05 2024-09-21T15:49:55.144 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.144 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:55.145 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.145 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.145 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.145 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26240 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 3244032~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:55.145 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.506106+0000", 2024-09-21T15:49:55.145 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 28.602882365999999, 2024-09-21T15:49:55.145 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032955696, 2024-09-21T15:49:55.145 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:55.145 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:55.145 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:55.145 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:55.145 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:55.145 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26240 2024-09-21T15:49:55.145 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.145 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:55.146 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.146 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:55.146 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506106+0000", 2024-09-21T15:49:55.146 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.146 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.146 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.146 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:55.146 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506106+0000", 2024-09-21T15:49:55.146 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.146 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.146 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.146 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:55.146 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506106+0000", 2024-09-21T15:49:55.147 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:55.147 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.147 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.147 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:55.147 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506107+0000", 2024-09-21T15:49:55.147 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1179999999999999e-06 2024-09-21T15:49:55.147 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.147 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.147 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:55.147 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506110+0000", 2024-09-21T15:49:55.147 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0259999999999999e-06 2024-09-21T15:49:55.147 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.147 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.147 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:55.147 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506112+0000", 2024-09-21T15:49:55.147 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5890000000000001e-06 2024-09-21T15:49:55.148 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.148 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.148 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:55.148 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.538962+0000", 2024-09-21T15:49:55.148 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032849861000000001 2024-09-21T15:49:55.148 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.148 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.148 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:55.148 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539003+0000", 2024-09-21T15:49:55.148 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0756000000000001e-05 2024-09-21T15:49:55.148 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.148 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.148 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:55.148 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539062+0000", 2024-09-21T15:49:55.148 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8597999999999997e-05 2024-09-21T15:49:55.149 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.149 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:55.149 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.149 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.149 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.149 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26241 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 3301376~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:55.149 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.506128+0000", 2024-09-21T15:49:55.149 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 28.602860199999999, 2024-09-21T15:49:55.149 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033043453, 2024-09-21T15:49:55.149 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:55.149 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:55.149 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:55.149 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:55.149 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:55.149 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26241 2024-09-21T15:49:55.150 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.150 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:55.150 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.150 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:55.150 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506128+0000", 2024-09-21T15:49:55.150 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.150 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.150 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.150 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:55.150 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506128+0000", 2024-09-21T15:49:55.150 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.150 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.150 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.150 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:55.151 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506128+0000", 2024-09-21T15:49:55.151 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:55.151 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.151 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.151 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:55.151 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506129+0000", 2024-09-21T15:49:55.151 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7290000000000001e-06 2024-09-21T15:49:55.151 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.151 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.151 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:55.151 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506132+0000", 2024-09-21T15:49:55.151 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9919999999999999e-06 2024-09-21T15:49:55.151 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.151 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.152 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:55.152 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506135+0000", 2024-09-21T15:49:55.152 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4090000000000001e-06 2024-09-21T15:49:55.152 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.152 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.152 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:55.152 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539084+0000", 2024-09-21T15:49:55.152 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032949231000000002 2024-09-21T15:49:55.152 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.152 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.152 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:55.152 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539124+0000", 2024-09-21T15:49:55.152 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9421000000000001e-05 2024-09-21T15:49:55.153 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.153 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.153 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:55.153 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539172+0000", 2024-09-21T15:49:55.153 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8068000000000001e-05 2024-09-21T15:49:55.153 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.153 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:55.153 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.153 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.153 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.153 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26242 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 3358720~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:55.153 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.506137+0000", 2024-09-21T15:49:55.153 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 28.602851266000002, 2024-09-21T15:49:55.153 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033142360000000003, 2024-09-21T15:49:55.154 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:55.154 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:55.154 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:55.154 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:55.154 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:55.154 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26242 2024-09-21T15:49:55.154 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.154 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:55.154 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.154 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:55.154 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506137+0000", 2024-09-21T15:49:55.154 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.154 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.154 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.155 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:55.155 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506137+0000", 2024-09-21T15:49:55.155 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.155 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.155 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.155 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:55.155 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506137+0000", 2024-09-21T15:49:55.155 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:55.155 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.155 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.155 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:55.155 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506138+0000", 2024-09-21T15:49:55.155 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2330000000000001e-06 2024-09-21T15:49:55.155 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.156 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.156 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:55.156 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506141+0000", 2024-09-21T15:49:55.156 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7999999999999999e-06 2024-09-21T15:49:55.156 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.156 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.156 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:55.156 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506143+0000", 2024-09-21T15:49:55.156 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5919999999999999e-06 2024-09-21T15:49:55.156 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.156 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.156 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:55.156 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539193+0000", 2024-09-21T15:49:55.156 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033049229999999999 2024-09-21T15:49:55.157 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.157 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.157 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:55.157 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539232+0000", 2024-09-21T15:49:55.157 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9743999999999998e-05 2024-09-21T15:49:55.157 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.157 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.157 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:55.157 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539279+0000", 2024-09-21T15:49:55.157 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7039999999999997e-05 2024-09-21T15:49:55.157 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.157 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:55.157 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.157 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.157 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.157 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26243 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 3416064~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:55.157 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.506145+0000", 2024-09-21T15:49:55.158 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 28.602842980999998, 2024-09-21T15:49:55.158 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033240933, 2024-09-21T15:49:55.158 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:55.158 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:55.158 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:55.158 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:55.158 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:55.158 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26243 2024-09-21T15:49:55.158 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.158 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:55.158 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.158 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:55.158 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506145+0000", 2024-09-21T15:49:55.159 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.159 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.159 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.159 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:55.159 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506145+0000", 2024-09-21T15:49:55.159 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.159 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.159 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.159 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:55.159 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506145+0000", 2024-09-21T15:49:55.159 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:55.159 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.159 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.159 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:55.159 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506146+0000", 2024-09-21T15:49:55.160 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0279999999999999e-06 2024-09-21T15:49:55.160 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.160 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.160 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:55.160 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506149+0000", 2024-09-21T15:49:55.160 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2449999999999998e-06 2024-09-21T15:49:55.160 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.160 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.160 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:55.160 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506152+0000", 2024-09-21T15:49:55.160 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5050000000000002e-06 2024-09-21T15:49:55.160 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.160 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.160 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:55.161 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539301+0000", 2024-09-21T15:49:55.161 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033149093999999997 2024-09-21T15:49:55.161 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.161 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.161 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:55.161 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539339+0000", 2024-09-21T15:49:55.161 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7685999999999998e-05 2024-09-21T15:49:55.161 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.161 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.161 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:55.161 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539386+0000", 2024-09-21T15:49:55.161 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7602000000000003e-05 2024-09-21T15:49:55.161 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.161 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:55.162 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.162 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.162 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.162 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26258 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 4276224~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:55.162 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.508167+0000", 2024-09-21T15:49:55.162 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 28.600821063000001, 2024-09-21T15:49:55.162 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032934857999999997, 2024-09-21T15:49:55.162 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:55.162 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:55.162 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:55.162 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:55.162 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:55.162 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26258 2024-09-21T15:49:55.162 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.162 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:55.163 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.163 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:55.163 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508167+0000", 2024-09-21T15:49:55.163 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.163 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.163 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.163 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:55.163 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508167+0000", 2024-09-21T15:49:55.163 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.163 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.163 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.163 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:55.163 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508167+0000", 2024-09-21T15:49:55.163 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:55.164 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.164 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.164 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:55.164 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508168+0000", 2024-09-21T15:49:55.164 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.088e-06 2024-09-21T15:49:55.164 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.164 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.164 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:55.164 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508171+0000", 2024-09-21T15:49:55.164 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7080000000000002e-06 2024-09-21T15:49:55.164 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.164 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.164 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:55.164 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508173+0000", 2024-09-21T15:49:55.165 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4030000000000001e-06 2024-09-21T15:49:55.165 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.165 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.165 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:55.165 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.540936+0000", 2024-09-21T15:49:55.165 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032762279999999998 2024-09-21T15:49:55.165 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.165 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.165 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:55.165 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.541007+0000", 2024-09-21T15:49:55.165 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1335999999999996e-05 2024-09-21T15:49:55.165 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.165 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.165 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:55.166 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.541102+0000", 2024-09-21T15:49:55.166 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5259000000000004e-05 2024-09-21T15:49:55.166 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.166 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:55.166 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.166 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.166 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.166 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26259 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 4333568~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:55.166 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.508175+0000", 2024-09-21T15:49:55.166 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 28.600813121000002, 2024-09-21T15:49:55.166 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033044845000000003, 2024-09-21T15:49:55.166 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:55.166 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:55.167 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:55.167 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:55.167 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:55.167 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26259 2024-09-21T15:49:55.167 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.167 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:55.167 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.167 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:55.167 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508175+0000", 2024-09-21T15:49:55.167 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.167 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.167 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.167 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:55.167 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508175+0000", 2024-09-21T15:49:55.168 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.168 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.168 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.168 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:55.168 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508175+0000", 2024-09-21T15:49:55.168 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:55.168 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.168 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.168 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:55.168 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508176+0000", 2024-09-21T15:49:55.168 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1039999999999999e-06 2024-09-21T15:49:55.168 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.168 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.168 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:55.169 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508179+0000", 2024-09-21T15:49:55.169 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.689e-06 2024-09-21T15:49:55.169 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.169 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.169 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:55.169 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508181+0000", 2024-09-21T15:49:55.169 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5610000000000001e-06 2024-09-21T15:49:55.169 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.169 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.169 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:55.169 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.541126+0000", 2024-09-21T15:49:55.169 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032944284999999997 2024-09-21T15:49:55.169 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.169 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.170 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:55.170 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.541168+0000", 2024-09-21T15:49:55.170 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2565000000000003e-05 2024-09-21T15:49:55.170 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.170 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.170 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:55.170 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.541220+0000", 2024-09-21T15:49:55.170 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1863000000000001e-05 2024-09-21T15:49:55.170 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.170 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:55.170 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.170 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.170 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.170 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26260 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 4390912~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:55.171 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.508183+0000", 2024-09-21T15:49:55.171 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 28.600805006000002, 2024-09-21T15:49:55.171 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033150304999999998, 2024-09-21T15:49:55.171 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:55.171 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:55.171 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:55.171 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:55.171 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:55.171 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26260 2024-09-21T15:49:55.171 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.171 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:55.171 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.171 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:55.172 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508183+0000", 2024-09-21T15:49:55.172 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.172 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.172 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.172 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:55.172 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508183+0000", 2024-09-21T15:49:55.172 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.172 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.172 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.172 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:55.172 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508183+0000", 2024-09-21T15:49:55.172 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:55.172 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.172 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.172 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:55.173 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508184+0000", 2024-09-21T15:49:55.173 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0580000000000001e-06 2024-09-21T15:49:55.173 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.173 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.173 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:55.173 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508187+0000", 2024-09-21T15:49:55.173 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7209999999999999e-06 2024-09-21T15:49:55.173 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.173 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.173 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:55.173 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508189+0000", 2024-09-21T15:49:55.173 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5239999999999999e-06 2024-09-21T15:49:55.173 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.173 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.173 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:55.174 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.541243+0000", 2024-09-21T15:49:55.174 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033053658 2024-09-21T15:49:55.174 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.174 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.174 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:55.174 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.541284+0000", 2024-09-21T15:49:55.174 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0834000000000002e-05 2024-09-21T15:49:55.174 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.174 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.174 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:55.174 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.541334+0000", 2024-09-21T15:49:55.174 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9724e-05 2024-09-21T15:49:55.174 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.174 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:55.174 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.175 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.175 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.175 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26275 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 5251072~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:55.175 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.510151+0000", 2024-09-21T15:49:55.175 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 28.598837227000001, 2024-09-21T15:49:55.175 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032836153999999999, 2024-09-21T15:49:55.175 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:55.175 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:55.175 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:55.175 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:55.175 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:55.175 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26275 2024-09-21T15:49:55.175 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.175 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:55.176 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.176 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:55.176 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510151+0000", 2024-09-21T15:49:55.176 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.176 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.176 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.176 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:55.176 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510151+0000", 2024-09-21T15:49:55.176 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.176 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.176 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.176 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:55.176 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510151+0000", 2024-09-21T15:49:55.176 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:55.176 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.177 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.177 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:55.177 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510152+0000", 2024-09-21T15:49:55.177 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1990000000000001e-06 2024-09-21T15:49:55.177 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.177 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.177 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:55.177 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510155+0000", 2024-09-21T15:49:55.177 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7089999999999998e-06 2024-09-21T15:49:55.177 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.177 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.177 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:55.177 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510157+0000", 2024-09-21T15:49:55.177 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5189999999999999e-06 2024-09-21T15:49:55.178 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.178 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.178 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:55.178 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.542902+0000", 2024-09-21T15:49:55.178 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032744599999999999 2024-09-21T15:49:55.178 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.178 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.178 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:55.178 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.542941+0000", 2024-09-21T15:49:55.178 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8761999999999999e-05 2024-09-21T15:49:55.178 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.178 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.178 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:55.178 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.542987+0000", 2024-09-21T15:49:55.179 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6665999999999997e-05 2024-09-21T15:49:55.179 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.179 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:55.179 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.179 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.179 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.179 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26276 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 5308416~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:55.179 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.510159+0000", 2024-09-21T15:49:55.179 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 28.598829262999999, 2024-09-21T15:49:55.179 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032943014999999999, 2024-09-21T15:49:55.179 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:55.179 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:55.179 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:55.180 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:55.180 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:55.180 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26276 2024-09-21T15:49:55.180 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.180 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:55.180 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.180 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:55.180 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510159+0000", 2024-09-21T15:49:55.180 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.180 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.180 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.180 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:55.180 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510159+0000", 2024-09-21T15:49:55.180 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.181 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.181 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.181 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:55.181 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510159+0000", 2024-09-21T15:49:55.181 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:55.181 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.181 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.181 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:55.181 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510160+0000", 2024-09-21T15:49:55.181 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.13e-06 2024-09-21T15:49:55.181 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.181 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.181 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:55.181 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510163+0000", 2024-09-21T15:49:55.181 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7489999999999999e-06 2024-09-21T15:49:55.181 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.182 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.182 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:55.182 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510165+0000", 2024-09-21T15:49:55.182 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4200000000000001e-06 2024-09-21T15:49:55.182 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.182 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.182 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:55.182 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.543013+0000", 2024-09-21T15:49:55.182 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032847736000000002 2024-09-21T15:49:55.182 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.182 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.182 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:55.182 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.543054+0000", 2024-09-21T15:49:55.182 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0725000000000001e-05 2024-09-21T15:49:55.183 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.183 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.183 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:55.183 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.543102+0000", 2024-09-21T15:49:55.183 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8471999999999999e-05 2024-09-21T15:49:55.183 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.183 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:55.183 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.183 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.183 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.183 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26277 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 5365760~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:55.183 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.510167+0000", 2024-09-21T15:49:55.183 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 28.598821132000001, 2024-09-21T15:49:55.183 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033113111000000001, 2024-09-21T15:49:55.184 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:55.184 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:55.184 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:55.184 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:55.184 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:55.184 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26277 2024-09-21T15:49:55.184 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.184 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:55.184 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.184 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:55.184 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510167+0000", 2024-09-21T15:49:55.184 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.184 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.184 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.185 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:55.185 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510167+0000", 2024-09-21T15:49:55.185 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.185 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.185 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.185 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:55.185 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510167+0000", 2024-09-21T15:49:55.185 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:55.185 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.185 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.185 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:55.185 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510168+0000", 2024-09-21T15:49:55.185 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.6899999999999996e-07 2024-09-21T15:49:55.185 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.185 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.185 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:55.186 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510171+0000", 2024-09-21T15:49:55.186 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7080000000000002e-06 2024-09-21T15:49:55.186 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.186 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.186 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:55.186 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510173+0000", 2024-09-21T15:49:55.186 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5050000000000002e-06 2024-09-21T15:49:55.186 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.186 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.186 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:55.186 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.543125+0000", 2024-09-21T15:49:55.186 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032951504 2024-09-21T15:49:55.186 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.186 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.186 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:55.187 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.543192+0000", 2024-09-21T15:49:55.187 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6938000000000002e-05 2024-09-21T15:49:55.187 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.187 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.187 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:55.187 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.543280+0000", 2024-09-21T15:49:55.187 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8702999999999994e-05 2024-09-21T15:49:55.187 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.187 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:55.187 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.187 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.187 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.187 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26292 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 6225920~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:55.187 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.512160+0000", 2024-09-21T15:49:55.188 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 28.596828318, 2024-09-21T15:49:55.188 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032838748000000001, 2024-09-21T15:49:55.188 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:55.188 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:55.188 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:55.188 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:55.188 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:55.188 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26292 2024-09-21T15:49:55.188 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.188 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:55.188 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.188 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:55.188 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512160+0000", 2024-09-21T15:49:55.188 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.188 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.188 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.189 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:55.189 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512160+0000", 2024-09-21T15:49:55.189 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.189 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.189 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.189 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:55.189 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512160+0000", 2024-09-21T15:49:55.189 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:55.189 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.189 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.189 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:55.189 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512161+0000", 2024-09-21T15:49:55.189 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.158e-06 2024-09-21T15:49:55.189 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.189 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.190 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:55.190 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512164+0000", 2024-09-21T15:49:55.190 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7429999999999998e-06 2024-09-21T15:49:55.190 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.190 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.190 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:55.190 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512166+0000", 2024-09-21T15:49:55.190 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5280000000000002e-06 2024-09-21T15:49:55.190 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.190 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.190 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:55.190 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.544912+0000", 2024-09-21T15:49:55.190 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032745962000000003 2024-09-21T15:49:55.190 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.190 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.191 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:55.191 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.544950+0000", 2024-09-21T15:49:55.191 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7657000000000002e-05 2024-09-21T15:49:55.191 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.191 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.191 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:55.191 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.544999+0000", 2024-09-21T15:49:55.191 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8912999999999997e-05 2024-09-21T15:49:55.191 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.191 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:55.191 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.191 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.191 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.191 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26293 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 6283264~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:55.191 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.512169+0000", 2024-09-21T15:49:55.191 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 28.596819169, 2024-09-21T15:49:55.192 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032937552000000002, 2024-09-21T15:49:55.192 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:55.192 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:55.192 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:55.192 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:55.192 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:55.192 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26293 2024-09-21T15:49:55.192 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.192 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:55.192 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.192 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:55.192 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512169+0000", 2024-09-21T15:49:55.192 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.192 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.192 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.192 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:55.192 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512169+0000", 2024-09-21T15:49:55.193 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.193 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.193 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.193 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:55.193 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512169+0000", 2024-09-21T15:49:55.193 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:55.193 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.193 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.193 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:55.193 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512170+0000", 2024-09-21T15:49:55.193 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1650000000000001e-06 2024-09-21T15:49:55.193 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.193 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.193 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:55.193 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512173+0000", 2024-09-21T15:49:55.193 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.773e-06 2024-09-21T15:49:55.194 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.194 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.194 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:55.194 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512175+0000", 2024-09-21T15:49:55.194 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4700000000000001e-06 2024-09-21T15:49:55.194 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.194 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.194 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:55.194 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.545018+0000", 2024-09-21T15:49:55.194 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032843043000000002 2024-09-21T15:49:55.194 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.194 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.194 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:55.194 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.545057+0000", 2024-09-21T15:49:55.194 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8862999999999997e-05 2024-09-21T15:49:55.195 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.195 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.195 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:55.195 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.545107+0000", 2024-09-21T15:49:55.195 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9455e-05 2024-09-21T15:49:55.195 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.195 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:55.195 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.195 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.195 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:55.195 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:55.197 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:49:55.198 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:55.195+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:55.198 INFO:teuthology.orchestra.run.smithi089.stdout:checking clone #23, image image_client.0-clone23 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent24 2024-09-21T15:49:55.198 INFO:teuthology.orchestra.run.smithi089.stdout:3943 punch from 0x80a2e73 to 0x80aca5b, (0x9be8 bytes) 2024-09-21T15:49:55.198 INFO:teuthology.orchestra.run.smithi089.stdout:3944 write 0x3867cbb thru 0x38771e2 (0xf528 bytes) 2024-09-21T15:49:55.198 INFO:teuthology.orchestra.run.smithi089.stdout:3945 punch from 0x4c50931 to 0x4c53de3, (0x34b2 bytes) 2024-09-21T15:49:55.198 INFO:teuthology.orchestra.run.smithi089.stdout:3948 write 0xcb9c482 thru 0xcba67dd (0xa35c bytes) 2024-09-21T15:49:55.198 INFO:teuthology.orchestra.run.smithi089.stdout:3950 writesame 0xcb16caa thru 0xcb18978 data_size 0x1(0x1ccf bytes) 2024-09-21T15:49:55.198 INFO:teuthology.orchestra.run.smithi089.stdout:3952 read 0x79db804 thru 0x79e96c9 (0xdec6 bytes) 2024-09-21T15:49:55.199 INFO:teuthology.orchestra.run.smithi089.stdout:3953 write 0x4223533 thru 0x422f25e (0xbd2c bytes) 2024-09-21T15:49:55.205 INFO:teuthology.orchestra.run.smithi089.stdout:3954 punch from 0xcb4fcc8 to 0xcb5ab73, (0xaeab bytes) 2024-09-21T15:49:55.205 INFO:teuthology.orchestra.run.smithi089.stdout:3955 write 0x31f439b thru 0x31fd8ce (0x9534 bytes) 2024-09-21T15:49:55.229 INFO:teuthology.orchestra.run.smithi089.stdout:3956 trunc from 0xe0b5c9e to 0xc381758 2024-09-21T15:49:55.248 INFO:teuthology.orchestra.run.smithi089.stdout:3957 read 0xb6b885f thru 0xb6bf293 (0x6a35 bytes) 2024-09-21T15:49:55.248 INFO:teuthology.orchestra.run.smithi089.stdout:3959 punch from 0x9a0a2e2 to 0x9a13a98, (0x97b6 bytes) 2024-09-21T15:49:55.258 INFO:teuthology.orchestra.run.smithi089.stdout:3960 trunc from 0xc381758 to 0x325be6a 2024-09-21T15:49:55.297 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:55.296+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:55.301 INFO:teuthology.orchestra.run.smithi089.stdout:3962 trunc from 0x325be6a to 0x3000de5 2024-09-21T15:49:55.335 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:55.344 INFO:teuthology.orchestra.run.smithi089.stdout:3964 write 0x483fda6 thru 0x484359d (0x37f8 bytes) 2024-09-21T15:49:55.352 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:55.352 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:55.352 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:49:55.352 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:55.358 INFO:teuthology.orchestra.run.smithi089.stdout:3965 writesame 0x544781c thru 0x5449888 data_size 0x1(0x206d bytes) 2024-09-21T15:49:55.358 INFO:teuthology.orchestra.run.smithi089.stdout:3966 writesame 0x4cdc571 thru 0x4ceb3c7 data_size 0x1(0xee57 bytes) 2024-09-21T15:49:55.362 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:49:55.375 INFO:teuthology.orchestra.run.smithi089.stdout:3969 read 0x53f261c thru 0x540141d (0xee02 bytes) 2024-09-21T15:49:55.376 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:55.376 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:55.376 INFO:teuthology.orchestra.run.smithi089.stdout:3974 read 0x2e5e374 thru 0x2e61a90 (0x371d bytes) 2024-09-21T15:49:55.376 INFO:teuthology.orchestra.run.smithi089.stdout:3975 punch from 0x4f78fa8 to 0x4f8233f, (0x9397 bytes) 2024-09-21T15:49:55.388 INFO:teuthology.orchestra.run.smithi089.stdout:3976 write 0xb0dee7c thru 0xb0e987b (0xaa00 bytes) 2024-09-21T15:49:55.394 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:55.403 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:55.402+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:55.404 INFO:teuthology.orchestra.run.smithi089.stdout:3979 write 0xdbb8051 thru 0xdbb99c6 (0x1976 bytes) 2024-09-21T15:49:55.412 INFO:teuthology.orchestra.run.smithi089.stdout:3980 read 0xc998f9e thru 0xc9a3542 (0xa5a5 bytes) 2024-09-21T15:49:55.412 INFO:teuthology.orchestra.run.smithi089.stdout:3981 write 0xb26ed40 thru 0xb274c21 (0x5ee2 bytes) 2024-09-21T15:49:55.422 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:55.422 INFO:teuthology.orchestra.run.smithi089.stdout:3983 punch from 0x6137f2e to 0x613dc08, (0x5cda bytes) 2024-09-21T15:49:55.436 INFO:teuthology.orchestra.run.smithi089.stdout:3984 trunc from 0xdbb99c7 to 0x162fd6f 2024-09-21T15:49:55.469 INFO:teuthology.orchestra.run.smithi089.stdout:3985 write 0x2b6de9c thru 0x2b71585 (0x36ea bytes) 2024-09-21T15:49:55.488 INFO:teuthology.orchestra.run.smithi089.stdout:3989 write 0x7424cd1 thru 0x7425169 (0x499 bytes) 2024-09-21T15:49:55.492 INFO:teuthology.orchestra.run.smithi089.stdout:3991 read 0x126487c thru 0x126af11 (0x6696 bytes) 2024-09-21T15:49:55.499 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:55.498+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:55.501 INFO:teuthology.orchestra.run.smithi089.stdout:3992 trunc from 0x742516a to 0xe0fdd0f 2024-09-21T15:49:55.507 INFO:teuthology.orchestra.run.smithi089.stdout:3993 write 0x6918f65 thru 0x6923c3e (0xacda bytes) 2024-09-21T15:49:55.516 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:55.516 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:55.516 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:49:55.517 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:49:55.517 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:55.517 INFO:teuthology.orchestra.run.smithi089.stdout:3994 read 0xbd4fcc thru 0xbd8049 (0x307e bytes) 2024-09-21T15:49:55.517 INFO:teuthology.orchestra.run.smithi089.stdout:3995 punch from 0x36376e0 to 0x364056e, (0x8e8e bytes) 2024-09-21T15:49:55.521 INFO:teuthology.orchestra.run.smithi089.stdout:3997 trunc from 0xe0fdd0f to 0xa68f6e8 2024-09-21T15:49:55.525 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:49:55.527 INFO:teuthology.orchestra.run.smithi089.stdout:3998 read 0x547f75f thru 0x5484a4f (0x52f1 bytes) 2024-09-21T15:49:55.527 INFO:teuthology.orchestra.run.smithi089.stdout:3999 read 0x575694a thru 0x5761bf3 (0xb2aa bytes) 2024-09-21T15:49:55.536 INFO:teuthology.orchestra.run.smithi089.stdout:4000 trunc from 0xa68f6e8 to 0x3194baa 2024-09-21T15:49:55.549 INFO:teuthology.orchestra.run.smithi089.stdout:4002 trunc from 0x3194baa to 0x3a12e28 2024-09-21T15:49:55.571 INFO:teuthology.orchestra.run.smithi089.stdout:4004 write 0x78676c2 thru 0x7872a10 (0xb34f bytes) 2024-09-21T15:49:55.586 INFO:teuthology.orchestra.run.smithi089.stdout:4005 write 0x8187d60 thru 0x8191ce1 (0x9f82 bytes) 2024-09-21T15:49:55.591 INFO:teuthology.orchestra.run.smithi089.stdout:4007 write 0xfbb75 thru 0xfbecd (0x359 bytes) 2024-09-21T15:49:55.601 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:55.599+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:55.617 INFO:teuthology.orchestra.run.smithi089.stdout:4008 write 0xc5f608b thru 0xc60236f (0xc2e5 bytes) 2024-09-21T15:49:55.634 INFO:teuthology.orchestra.run.smithi089.stdout:4009 write 0xe2997ce thru 0xe2a61ad (0xc9e0 bytes) 2024-09-21T15:49:55.641 INFO:teuthology.orchestra.run.smithi089.stdout:4010 read 0x949a8c9 thru 0x94a0e5f (0x6597 bytes) 2024-09-21T15:49:55.641 INFO:teuthology.orchestra.run.smithi089.stdout:4011 write 0x3e5991f thru 0x3e61629 (0x7d0b bytes) 2024-09-21T15:49:55.650 INFO:teuthology.orchestra.run.smithi089.stdout:4012 punch from 0xb3e3bb9 to 0xb3ef8b6, (0xbcfd bytes) 2024-09-21T15:49:55.658 INFO:teuthology.orchestra.run.smithi089.stdout:4013 trunc from 0xe2a61ae to 0x7978f3d 2024-09-21T15:49:55.675 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:55.675 INFO:teuthology.orchestra.run.smithi089.stdout:4015 punch from 0x5ab5634 to 0x5abe5df, (0x8fab bytes) 2024-09-21T15:49:55.675 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:55.681 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:55.681 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:49:55.681 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:49:55.681 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:49:55.681 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.681 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:65402 3.1d 3:bda7fcfc:::rbd_data.11336b2fdf4a.0000000000000000:c [read 4915200~0] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:55.681 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.279722+0000", 2024-09-21T15:49:55.682 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 18.398882679, 2024-09-21T15:49:55.682 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045558766000000001, 2024-09-21T15:49:55.682 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:55.682 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:55.682 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:55.682 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:55.682 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:55.683 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 65402 2024-09-21T15:49:55.683 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.683 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:55.683 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.683 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:55.683 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279722+0000", 2024-09-21T15:49:55.683 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.683 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.683 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.684 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:55.684 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279722+0000", 2024-09-21T15:49:55.684 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.684 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.684 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.684 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:55.684 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279722+0000", 2024-09-21T15:49:55.684 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:55.685 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.685 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.685 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:55.685 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279724+0000", 2024-09-21T15:49:55.685 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3929999999999998e-06 2024-09-21T15:49:55.685 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.685 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.685 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:55.685 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279731+0000", 2024-09-21T15:49:55.686 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3269999999999997e-06 2024-09-21T15:49:55.686 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.686 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.686 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:55.686 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279736+0000", 2024-09-21T15:49:55.686 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.113e-06 2024-09-21T15:49:55.686 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.686 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.687 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:55.687 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.325136+0000", 2024-09-21T15:49:55.687 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045399912000000001 2024-09-21T15:49:55.687 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.687 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.687 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:55.687 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.325213+0000", 2024-09-21T15:49:55.687 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7546999999999995e-05 2024-09-21T15:49:55.687 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.688 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.688 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:55.688 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.325281+0000", 2024-09-21T15:49:55.688 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7931999999999999e-05 2024-09-21T15:49:55.688 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.688 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:55.688 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.688 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.688 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.689 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:65403 3.1d 3:bda7fcfc:::rbd_data.11336b2fdf4a.0000000000000000:c [read 4923392~0] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:55.689 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.279740+0000", 2024-09-21T15:49:55.689 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 18.398864766999999, 2024-09-21T15:49:55.689 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045722756000000003, 2024-09-21T15:49:55.689 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:55.689 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:55.689 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:55.689 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:55.689 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:55.689 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 65403 2024-09-21T15:49:55.689 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.690 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:55.690 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.690 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:55.690 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279740+0000", 2024-09-21T15:49:55.690 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.690 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.690 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.690 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:55.690 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279740+0000", 2024-09-21T15:49:55.690 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.690 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.690 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.690 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:55.691 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279740+0000", 2024-09-21T15:49:55.691 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:55.691 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.691 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.691 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:55.691 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279742+0000", 2024-09-21T15:49:55.691 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1849999999999998e-06 2024-09-21T15:49:55.691 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.691 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.691 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:55.691 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279749+0000", 2024-09-21T15:49:55.691 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2450000000000001e-06 2024-09-21T15:49:55.692 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.692 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.692 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:55.692 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279754+0000", 2024-09-21T15:49:55.692 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8940000000000001e-06 2024-09-21T15:49:55.692 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.692 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.692 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:55.692 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.325316+0000", 2024-09-21T15:49:55.692 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045561938000000003 2024-09-21T15:49:55.692 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.692 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.692 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:55.692 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.325399+0000", 2024-09-21T15:49:55.692 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2647999999999999e-05 2024-09-21T15:49:55.692 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.693 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.693 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:55.693 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.325463+0000", 2024-09-21T15:49:55.693 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4247999999999999e-05 2024-09-21T15:49:55.693 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.693 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:55.693 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.693 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.693 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.693 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:65404 3.1d 3:bda7fcfc:::rbd_data.11336b2fdf4a.0000000000000000:c [read 4931584~0] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:55.693 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.279757+0000", 2024-09-21T15:49:55.693 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 18.398847936999999, 2024-09-21T15:49:55.693 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045883348999999997, 2024-09-21T15:49:55.693 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:55.693 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:55.693 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:55.694 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:55.694 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:55.694 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 65404 2024-09-21T15:49:55.694 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.694 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:55.694 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.694 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:55.694 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279757+0000", 2024-09-21T15:49:55.694 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.694 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.694 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.694 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:55.694 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279757+0000", 2024-09-21T15:49:55.695 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.695 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.695 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.695 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:55.695 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279757+0000", 2024-09-21T15:49:55.695 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:55.695 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.695 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.695 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:55.695 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279759+0000", 2024-09-21T15:49:55.695 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.64e-06 2024-09-21T15:49:55.695 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.695 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.695 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:55.695 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279764+0000", 2024-09-21T15:49:55.695 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0030000000000002e-06 2024-09-21T15:49:55.696 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.696 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.696 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:55.696 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279766+0000", 2024-09-21T15:49:55.696 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6620000000000001e-06 2024-09-21T15:49:55.696 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.696 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.696 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:55.696 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.325489+0000", 2024-09-21T15:49:55.696 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045722721000000001 2024-09-21T15:49:55.696 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.696 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.696 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:55.696 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.325574+0000", 2024-09-21T15:49:55.696 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5109000000000001e-05 2024-09-21T15:49:55.696 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.697 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.697 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:55.697 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.325641+0000", 2024-09-21T15:49:55.697 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6475000000000003e-05 2024-09-21T15:49:55.697 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.697 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:55.697 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.697 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.697 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.697 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:65405 3.1d 3:bda7fcfc:::rbd_data.11336b2fdf4a.0000000000000000:c [read 4939776~0] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:55.697 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.279768+0000", 2024-09-21T15:49:55.697 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 18.398836674000002, 2024-09-21T15:49:55.697 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046038737000000003, 2024-09-21T15:49:55.697 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:55.698 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:55.698 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:55.698 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:55.698 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:55.698 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 65405 2024-09-21T15:49:55.698 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.698 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:55.698 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.698 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:55.698 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279768+0000", 2024-09-21T15:49:55.698 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.698 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.698 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.698 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:55.699 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279768+0000", 2024-09-21T15:49:55.699 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.699 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.699 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.699 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:55.699 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279768+0000", 2024-09-21T15:49:55.699 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:55.699 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.699 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.699 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:55.699 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279769+0000", 2024-09-21T15:49:55.699 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.22e-06 2024-09-21T15:49:55.699 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.699 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.699 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:55.700 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279772+0000", 2024-09-21T15:49:55.700 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9349999999999999e-06 2024-09-21T15:49:55.700 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.700 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.700 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:55.700 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279775+0000", 2024-09-21T15:49:55.700 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4959999999999999e-06 2024-09-21T15:49:55.700 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.700 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.700 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:55.700 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.325668+0000", 2024-09-21T15:49:55.700 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045892779000000002 2024-09-21T15:49:55.700 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.700 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.700 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:55.701 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.325742+0000", 2024-09-21T15:49:55.701 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4092999999999997e-05 2024-09-21T15:49:55.701 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.701 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.701 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:55.701 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.325807+0000", 2024-09-21T15:49:55.701 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5419999999999994e-05 2024-09-21T15:49:55.701 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.701 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:55.701 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.701 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.701 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.701 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:65406 3.1d 3:bda7fcfc:::rbd_data.11336b2fdf4a.0000000000000000:c [read 4947968~0] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:55.701 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.279777+0000", 2024-09-21T15:49:55.701 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 18.398828115000001, 2024-09-21T15:49:55.702 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046198388999999999, 2024-09-21T15:49:55.702 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:55.702 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:55.702 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:55.702 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:55.702 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:55.702 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 65406 2024-09-21T15:49:55.702 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.702 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:55.702 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.702 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:55.702 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279777+0000", 2024-09-21T15:49:55.702 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.702 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.702 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.703 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:55.703 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279777+0000", 2024-09-21T15:49:55.703 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.703 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.703 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.703 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:55.703 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279777+0000", 2024-09-21T15:49:55.703 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:55.703 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.703 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.703 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:55.703 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279778+0000", 2024-09-21T15:49:55.703 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.238e-06 2024-09-21T15:49:55.703 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.703 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.703 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:55.704 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279781+0000", 2024-09-21T15:49:55.704 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8679999999999999e-06 2024-09-21T15:49:55.704 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.704 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.704 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:55.704 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279783+0000", 2024-09-21T15:49:55.704 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4729999999999999e-06 2024-09-21T15:49:55.704 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.704 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.704 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:55.704 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.325834+0000", 2024-09-21T15:49:55.704 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046050328000000001 2024-09-21T15:49:55.704 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.704 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.704 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:55.705 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.325909+0000", 2024-09-21T15:49:55.705 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5463999999999994e-05 2024-09-21T15:49:55.705 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.705 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.705 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:55.705 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.325975+0000", 2024-09-21T15:49:55.705 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6229999999999994e-05 2024-09-21T15:49:55.705 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.705 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:55.705 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.706 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.706 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.706 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:65407 3.1d 3:bda7fcfc:::rbd_data.11336b2fdf4a.0000000000000000:c [read 4956160~0] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:55.706 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.279786+0000", 2024-09-21T15:49:55.706 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 18.398819561, 2024-09-21T15:49:55.706 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046366821000000003, 2024-09-21T15:49:55.706 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:55.706 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:55.706 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:55.706 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:55.706 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:55.706 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 65407 2024-09-21T15:49:55.706 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.706 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:55.707 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.707 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:55.707 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279786+0000", 2024-09-21T15:49:55.707 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.707 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.707 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.707 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:55.707 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279786+0000", 2024-09-21T15:49:55.707 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.707 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.707 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.707 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:55.707 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279785+0000", 2024-09-21T15:49:55.707 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:55.708 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.708 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.708 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:55.708 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279786+0000", 2024-09-21T15:49:55.708 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1379999999999999e-06 2024-09-21T15:49:55.708 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.708 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.708 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:55.708 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279789+0000", 2024-09-21T15:49:55.708 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8629999999999999e-06 2024-09-21T15:49:55.708 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.708 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.708 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:55.708 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279792+0000", 2024-09-21T15:49:55.708 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.469e-06 2024-09-21T15:49:55.709 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.709 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.709 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:55.709 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.326002+0000", 2024-09-21T15:49:55.709 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046210481999999997 2024-09-21T15:49:55.709 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.709 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.709 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:55.709 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.326084+0000", 2024-09-21T15:49:55.709 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1570999999999995e-05 2024-09-21T15:49:55.709 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.709 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.709 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:55.709 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.326152+0000", 2024-09-21T15:49:55.709 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8523999999999996e-05 2024-09-21T15:49:55.710 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.710 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:55.710 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.710 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.710 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.710 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:65408 3.1d 3:bda7fcfc:::rbd_data.11336b2fdf4a.0000000000000000:c [read 4964352~0] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:55.710 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.279794+0000", 2024-09-21T15:49:55.710 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 18.398811259999999, 2024-09-21T15:49:55.710 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046551309999999999, 2024-09-21T15:49:55.710 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:55.710 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:55.710 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:55.710 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:55.710 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:55.711 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 65408 2024-09-21T15:49:55.711 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.711 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:55.711 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.711 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:55.711 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279794+0000", 2024-09-21T15:49:55.711 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.711 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.711 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.711 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:55.711 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279794+0000", 2024-09-21T15:49:55.711 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.711 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.711 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.712 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:55.712 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279794+0000", 2024-09-21T15:49:55.712 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:55.712 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.712 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.712 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:55.712 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279795+0000", 2024-09-21T15:49:55.712 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.105e-06 2024-09-21T15:49:55.712 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.712 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.712 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:55.712 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279797+0000", 2024-09-21T15:49:55.712 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7590000000000002e-06 2024-09-21T15:49:55.712 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.713 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.713 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:55.713 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279800+0000", 2024-09-21T15:49:55.713 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4030000000000001e-06 2024-09-21T15:49:55.713 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.713 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.713 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:55.713 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.326193+0000", 2024-09-21T15:49:55.713 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046392845000000002 2024-09-21T15:49:55.713 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.713 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.713 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:55.713 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.326272+0000", 2024-09-21T15:49:55.713 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8788999999999994e-05 2024-09-21T15:49:55.714 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.714 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.714 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:55.714 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.326345+0000", 2024-09-21T15:49:55.714 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3629999999999998e-05 2024-09-21T15:49:55.714 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.714 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:55.714 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.714 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.714 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.714 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:65409 3.1d 3:bda7fcfc:::rbd_data.11336b2fdf4a.0000000000000000:c [read 4972544~0] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:55.714 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.279802+0000", 2024-09-21T15:49:55.714 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 18.398802969999998, 2024-09-21T15:49:55.714 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046723775000000002, 2024-09-21T15:49:55.715 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:55.715 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:55.715 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:55.715 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:55.715 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:55.715 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 65409 2024-09-21T15:49:55.715 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.715 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:55.715 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.715 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:55.715 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279802+0000", 2024-09-21T15:49:55.715 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.715 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.716 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.716 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:55.716 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279802+0000", 2024-09-21T15:49:55.716 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.716 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.716 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.716 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:55.716 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279802+0000", 2024-09-21T15:49:55.716 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:55.716 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.716 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.716 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:55.716 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279803+0000", 2024-09-21T15:49:55.716 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.155e-06 2024-09-21T15:49:55.717 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.717 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.717 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:55.717 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279806+0000", 2024-09-21T15:49:55.717 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7949999999999999e-06 2024-09-21T15:49:55.717 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.717 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.717 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:55.717 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279808+0000", 2024-09-21T15:49:55.717 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.491e-06 2024-09-21T15:49:55.717 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.717 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.717 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:55.717 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.326375+0000", 2024-09-21T15:49:55.717 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046566824999999999 2024-09-21T15:49:55.717 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.718 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.718 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:55.718 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.326455+0000", 2024-09-21T15:49:55.718 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9680999999999998e-05 2024-09-21T15:49:55.718 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.718 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.718 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:55.718 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.326526+0000", 2024-09-21T15:49:55.718 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1057999999999996e-05 2024-09-21T15:49:55.718 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.718 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:55.718 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.718 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.718 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.719 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:70737 3.c 3:32ea75c5:::rbd_data.11336b2fdf4a.0000000000000001:c [read 1021952~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:55.719 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.495543+0000", 2024-09-21T15:49:55.719 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 18.183061781999999, 2024-09-21T15:49:55.719 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045418501, 2024-09-21T15:49:55.719 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:55.719 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:55.719 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:55.719 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:55.719 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:55.719 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 70737 2024-09-21T15:49:55.719 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.719 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:55.720 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.720 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:55.720 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495543+0000", 2024-09-21T15:49:55.720 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.720 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.720 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.720 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:55.720 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495543+0000", 2024-09-21T15:49:55.720 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.720 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.720 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.720 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:55.720 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495543+0000", 2024-09-21T15:49:55.720 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:55.720 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.721 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.721 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:55.721 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495545+0000", 2024-09-21T15:49:55.721 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5380000000000001e-06 2024-09-21T15:49:55.721 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.721 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.721 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:55.721 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495552+0000", 2024-09-21T15:49:55.721 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4740000000000002e-06 2024-09-21T15:49:55.721 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.721 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.721 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:55.721 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495557+0000", 2024-09-21T15:49:55.721 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.792e-06 2024-09-21T15:49:55.721 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.722 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.722 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:55.722 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.540761+0000", 2024-09-21T15:49:55.722 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045203967999999997 2024-09-21T15:49:55.722 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.722 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.722 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:55.722 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.540871+0000", 2024-09-21T15:49:55.722 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00011034799999999999 2024-09-21T15:49:55.722 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.722 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.722 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:55.722 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.540962+0000", 2024-09-21T15:49:55.723 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0931000000000001e-05 2024-09-21T15:49:55.723 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.723 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:55.723 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.723 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.723 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.723 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:70738 3.c 3:32ea75c5:::rbd_data.11336b2fdf4a.0000000000000001:c [read 1030144~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:55.723 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.495576+0000", 2024-09-21T15:49:55.723 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 18.183029414, 2024-09-21T15:49:55.723 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045596921999999998, 2024-09-21T15:49:55.723 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:55.723 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:55.723 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:55.723 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:55.723 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:55.724 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 70738 2024-09-21T15:49:55.724 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.724 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:55.724 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.724 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:55.724 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495576+0000", 2024-09-21T15:49:55.724 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.724 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.724 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.724 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:55.724 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495576+0000", 2024-09-21T15:49:55.724 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.724 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.724 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.724 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:55.724 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495575+0000", 2024-09-21T15:49:55.724 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:55.725 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.725 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.725 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:55.725 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495578+0000", 2024-09-21T15:49:55.725 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4779999999999998e-06 2024-09-21T15:49:55.725 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.725 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.725 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:55.725 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495584+0000", 2024-09-21T15:49:55.725 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1149999999999999e-06 2024-09-21T15:49:55.725 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.725 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.725 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:55.725 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495589+0000", 2024-09-21T15:49:55.725 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8300000000000003e-06 2024-09-21T15:49:55.726 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.726 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.726 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:55.726 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.540995+0000", 2024-09-21T15:49:55.726 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045406390999999997 2024-09-21T15:49:55.726 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.726 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.726 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:55.726 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.541083+0000", 2024-09-21T15:49:55.726 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7579999999999998e-05 2024-09-21T15:49:55.726 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.726 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.726 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:55.726 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.541173+0000", 2024-09-21T15:49:55.727 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0017000000000002e-05 2024-09-21T15:49:55.727 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.727 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:55.727 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.727 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.727 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.727 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:70739 3.c 3:32ea75c5:::rbd_data.11336b2fdf4a.0000000000000001:c [read 1038336~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:55.727 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.495608+0000", 2024-09-21T15:49:55.727 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 18.182997355000001, 2024-09-21T15:49:55.727 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045757673999999998, 2024-09-21T15:49:55.727 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:55.727 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:55.727 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:55.727 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:55.727 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:55.728 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 70739 2024-09-21T15:49:55.728 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.728 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:55.728 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.728 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:55.728 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495608+0000", 2024-09-21T15:49:55.728 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.728 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.728 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.728 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:55.728 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495608+0000", 2024-09-21T15:49:55.728 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.728 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.728 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.728 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:55.728 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495607+0000", 2024-09-21T15:49:55.729 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:55.729 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.729 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.729 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:55.729 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495610+0000", 2024-09-21T15:49:55.729 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5610000000000001e-06 2024-09-21T15:49:55.729 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.729 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.729 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:55.729 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495617+0000", 2024-09-21T15:49:55.729 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8290000000000002e-06 2024-09-21T15:49:55.730 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.730 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.730 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:55.730 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495622+0000", 2024-09-21T15:49:55.730 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0730000000000004e-06 2024-09-21T15:49:55.730 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.730 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.731 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:55.731 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.541203+0000", 2024-09-21T15:49:55.731 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045581165 2024-09-21T15:49:55.731 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.731 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.731 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:55.731 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.541279+0000", 2024-09-21T15:49:55.732 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6637000000000005e-05 2024-09-21T15:49:55.732 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.732 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.732 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:55.732 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.541365+0000", 2024-09-21T15:49:55.732 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5965999999999997e-05 2024-09-21T15:49:55.732 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.732 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:55.733 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.733 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.733 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.733 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:70742 3.c 3:32ea75c5:::rbd_data.11336b2fdf4a.0000000000000001:c [read 1046528~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:55.733 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.495639+0000", 2024-09-21T15:49:55.733 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 18.182966316000002, 2024-09-21T15:49:55.733 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045872673000000003, 2024-09-21T15:49:55.734 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:55.734 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:55.734 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:55.734 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:55.734 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:55.734 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 70742 2024-09-21T15:49:55.734 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.735 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:55.735 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.735 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:55.735 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495639+0000", 2024-09-21T15:49:55.735 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.735 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.735 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.735 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:55.735 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495639+0000", 2024-09-21T15:49:55.736 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.736 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.736 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.736 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:55.736 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495638+0000", 2024-09-21T15:49:55.736 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:55.736 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.736 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.737 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:55.737 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495641+0000", 2024-09-21T15:49:55.737 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5849999999999998e-06 2024-09-21T15:49:55.737 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.737 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.737 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:55.737 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495647+0000", 2024-09-21T15:49:55.737 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3069999999999999e-06 2024-09-21T15:49:55.737 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.738 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.738 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:55.738 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495652+0000", 2024-09-21T15:49:55.738 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2229999999999998e-06 2024-09-21T15:49:55.738 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.738 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.738 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:55.738 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.541400+0000", 2024-09-21T15:49:55.738 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045748024999999998 2024-09-21T15:49:55.739 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.739 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.739 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:55.739 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.541451+0000", 2024-09-21T15:49:55.739 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0093999999999998e-05 2024-09-21T15:49:55.739 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.739 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.739 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:55.739 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.541511+0000", 2024-09-21T15:49:55.740 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0928999999999999e-05 2024-09-21T15:49:55.740 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.740 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:55.740 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.740 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.740 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.740 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:70743 3.c 3:32ea75c5:::rbd_data.11336b2fdf4a.0000000000000001:c [read 333824~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:55.740 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.495672+0000", 2024-09-21T15:49:55.740 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 18.182933543000001, 2024-09-21T15:49:55.741 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045980785000000003, 2024-09-21T15:49:55.741 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:55.741 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:55.741 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:55.741 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:55.741 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:55.741 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 70743 2024-09-21T15:49:55.741 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.741 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:55.741 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.741 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:55.741 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495672+0000", 2024-09-21T15:49:55.741 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.741 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.741 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.742 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:55.742 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495672+0000", 2024-09-21T15:49:55.742 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.742 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.742 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.742 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:55.742 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495671+0000", 2024-09-21T15:49:55.742 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:55.742 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.742 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.742 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:55.742 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495674+0000", 2024-09-21T15:49:55.743 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5790000000000002e-06 2024-09-21T15:49:55.743 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.743 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.743 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:55.743 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495686+0000", 2024-09-21T15:49:55.743 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2276000000000001e-05 2024-09-21T15:49:55.743 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.743 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.743 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:55.743 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495692+0000", 2024-09-21T15:49:55.743 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8939999999999998e-06 2024-09-21T15:49:55.743 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.743 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.743 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:55.743 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.541538+0000", 2024-09-21T15:49:55.744 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045845761999999998 2024-09-21T15:49:55.744 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.744 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.744 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:55.744 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.541583+0000", 2024-09-21T15:49:55.744 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5343000000000003e-05 2024-09-21T15:49:55.744 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.744 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.744 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:55.744 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.541652+0000", 2024-09-21T15:49:55.744 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9468999999999995e-05 2024-09-21T15:49:55.744 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.744 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:55.744 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.744 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.744 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.745 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:70744 3.c 3:32ea75c5:::rbd_data.11336b2fdf4a.0000000000000001:c [read 342016~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:55.745 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.495705+0000", 2024-09-21T15:49:55.745 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 18.182899810999999, 2024-09-21T15:49:55.745 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046156794000000001, 2024-09-21T15:49:55.745 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:55.745 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:55.745 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:55.745 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:55.745 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:55.745 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 70744 2024-09-21T15:49:55.745 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.745 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:55.745 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.745 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:55.745 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495705+0000", 2024-09-21T15:49:55.746 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.746 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.746 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.746 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:55.746 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495705+0000", 2024-09-21T15:49:55.746 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.746 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.746 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.746 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:55.746 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495705+0000", 2024-09-21T15:49:55.746 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:55.747 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.747 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.747 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:55.747 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495707+0000", 2024-09-21T15:49:55.747 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4260000000000002e-06 2024-09-21T15:49:55.747 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.747 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.747 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:55.747 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495713+0000", 2024-09-21T15:49:55.747 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1029999999999998e-06 2024-09-21T15:49:55.747 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.747 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.747 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:55.747 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495720+0000", 2024-09-21T15:49:55.747 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.178e-06 2024-09-21T15:49:55.748 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.748 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.748 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:55.748 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.541690+0000", 2024-09-21T15:49:55.748 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045970069000000002 2024-09-21T15:49:55.748 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.748 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.748 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:55.748 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.541770+0000", 2024-09-21T15:49:55.748 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9976999999999997e-05 2024-09-21T15:49:55.748 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.748 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.748 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:55.748 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.541862+0000", 2024-09-21T15:49:55.748 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2532000000000001e-05 2024-09-21T15:49:55.749 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.749 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:55.749 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.749 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.749 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.749 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:70746 3.c 3:32ea75c5:::rbd_data.11336b2fdf4a.0000000000000001:c [read 350208~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:55.749 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.495741+0000", 2024-09-21T15:49:55.749 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 18.182864202000001, 2024-09-21T15:49:55.749 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046299229999999997, 2024-09-21T15:49:55.749 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:55.749 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:55.749 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:55.749 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:55.749 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:55.749 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 70746 2024-09-21T15:49:55.750 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.750 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:55.750 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.750 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:55.750 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495741+0000", 2024-09-21T15:49:55.750 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.750 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.750 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.750 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:55.750 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495741+0000", 2024-09-21T15:49:55.750 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.750 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.750 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.750 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:55.751 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495740+0000", 2024-09-21T15:49:55.751 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:55.751 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.751 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.751 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:55.751 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495743+0000", 2024-09-21T15:49:55.751 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5189999999999999e-06 2024-09-21T15:49:55.751 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.751 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.751 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:55.751 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495749+0000", 2024-09-21T15:49:55.751 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5470000000000002e-06 2024-09-21T15:49:55.751 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.751 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.751 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:55.751 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495755+0000", 2024-09-21T15:49:55.751 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2399999999999998e-06 2024-09-21T15:49:55.752 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.752 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.752 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:55.752 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.541877+0000", 2024-09-21T15:49:55.752 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046122375 2024-09-21T15:49:55.752 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.752 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.752 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:55.752 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.541945+0000", 2024-09-21T15:49:55.752 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7825999999999997e-05 2024-09-21T15:49:55.752 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.752 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.752 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:55.752 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.542040+0000", 2024-09-21T15:49:55.752 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5303999999999994e-05 2024-09-21T15:49:55.752 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.752 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:55.753 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.753 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.753 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.753 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:70748 3.c 3:32ea75c5:::rbd_data.11336b2fdf4a.0000000000000001:c [read 358400~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:55.753 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.495759+0000", 2024-09-21T15:49:55.753 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 18.182846047999998, 2024-09-21T15:49:55.753 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046462341999999997, 2024-09-21T15:49:55.753 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:55.753 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:55.753 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:55.753 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:55.753 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:55.753 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 70748 2024-09-21T15:49:55.753 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.753 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:55.753 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.754 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:55.754 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495759+0000", 2024-09-21T15:49:55.754 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.754 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.754 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.754 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:55.754 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495759+0000", 2024-09-21T15:49:55.754 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.754 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.754 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.754 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:55.754 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495759+0000", 2024-09-21T15:49:55.754 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:55.754 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.754 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.754 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:55.755 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495761+0000", 2024-09-21T15:49:55.755 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1900000000000002e-06 2024-09-21T15:49:55.755 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.755 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.755 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:55.755 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495767+0000", 2024-09-21T15:49:55.755 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0800000000000002e-06 2024-09-21T15:49:55.755 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.755 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.755 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:55.755 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495773+0000", 2024-09-21T15:49:55.755 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0000000000000002e-06 2024-09-21T15:49:55.755 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.755 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.755 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:55.756 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.542066+0000", 2024-09-21T15:49:55.756 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046293197000000001 2024-09-21T15:49:55.756 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.756 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.756 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:55.756 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.542139+0000", 2024-09-21T15:49:55.756 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2770999999999998e-05 2024-09-21T15:49:55.756 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.756 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.756 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:55.756 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.542221+0000", 2024-09-21T15:49:55.756 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2620000000000005e-05 2024-09-21T15:49:55.756 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.757 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:55.757 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.757 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.757 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.757 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:70750 3.c 3:32ea75c5:::rbd_data.11336b2fdf4a.0000000000000001:c [read 366592~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:55.757 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.495781+0000", 2024-09-21T15:49:55.757 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 18.182823704, 2024-09-21T15:49:55.757 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046624691000000003, 2024-09-21T15:49:55.757 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:55.757 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:55.757 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:55.757 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:55.757 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:55.757 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 70750 2024-09-21T15:49:55.758 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.758 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:55.758 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.758 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:55.758 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495781+0000", 2024-09-21T15:49:55.758 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.758 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.758 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.758 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:55.758 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495781+0000", 2024-09-21T15:49:55.758 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.758 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.759 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.759 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:55.759 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495781+0000", 2024-09-21T15:49:55.759 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:55.759 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.759 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.759 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:55.759 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495783+0000", 2024-09-21T15:49:55.759 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4729999999999999e-06 2024-09-21T15:49:55.759 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.759 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.759 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:55.759 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495790+0000", 2024-09-21T15:49:55.759 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3500000000000002e-06 2024-09-21T15:49:55.759 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.760 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.760 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:55.760 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495795+0000", 2024-09-21T15:49:55.760 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9679999999999997e-06 2024-09-21T15:49:55.760 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.760 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.760 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:55.760 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.542248+0000", 2024-09-21T15:49:55.760 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046453117000000002 2024-09-21T15:49:55.760 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.760 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.760 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:55.760 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.542312+0000", 2024-09-21T15:49:55.760 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4215999999999996e-05 2024-09-21T15:49:55.760 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.761 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.761 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:55.761 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.542406+0000", 2024-09-21T15:49:55.761 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3980000000000005e-05 2024-09-21T15:49:55.761 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.761 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:55.761 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.761 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.761 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.761 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:70751 3.c 3:32ea75c5:::rbd_data.11336b2fdf4a.0000000000000001:c [read 374784~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:55.761 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.495799+0000", 2024-09-21T15:49:55.761 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 18.182805909999999, 2024-09-21T15:49:55.761 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046807169000000003, 2024-09-21T15:49:55.762 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:55.762 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:55.762 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:55.762 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:55.762 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:55.762 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 70751 2024-09-21T15:49:55.762 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.762 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:55.762 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.762 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:55.762 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495799+0000", 2024-09-21T15:49:55.762 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.762 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.763 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.763 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:55.763 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495799+0000", 2024-09-21T15:49:55.763 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.763 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.763 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.763 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:55.763 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495799+0000", 2024-09-21T15:49:55.763 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:55.763 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.763 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.763 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:55.763 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495801+0000", 2024-09-21T15:49:55.763 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2560000000000001e-06 2024-09-21T15:49:55.763 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.764 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.764 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:55.764 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495807+0000", 2024-09-21T15:49:55.764 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8209999999999998e-06 2024-09-21T15:49:55.764 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.764 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.764 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:55.764 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495812+0000", 2024-09-21T15:49:55.764 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0289999999999996e-06 2024-09-21T15:49:55.764 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.764 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.764 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:55.764 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.542433+0000", 2024-09-21T15:49:55.764 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046621392999999997 2024-09-21T15:49:55.764 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.765 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.765 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:55.765 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.542510+0000", 2024-09-21T15:49:55.765 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7069000000000004e-05 2024-09-21T15:49:55.765 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.765 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.765 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:55.765 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.542606+0000", 2024-09-21T15:49:55.765 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.6036e-05 2024-09-21T15:49:55.765 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.765 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:55.765 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.765 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.765 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.766 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:70753 3.c 3:32ea75c5:::rbd_data.11336b2fdf4a.0000000000000001:c [read 382976~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:55.766 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.495816+0000", 2024-09-21T15:49:55.766 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 18.182788904999999, 2024-09-21T15:49:55.766 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046993252999999999, 2024-09-21T15:49:55.766 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:55.766 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:55.766 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:55.766 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:55.766 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:55.766 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 70753 2024-09-21T15:49:55.766 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.766 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:55.766 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.766 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:55.767 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495816+0000", 2024-09-21T15:49:55.767 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.767 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.767 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.767 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:55.767 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495816+0000", 2024-09-21T15:49:55.767 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.767 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.767 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.767 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:55.767 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495816+0000", 2024-09-21T15:49:55.767 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:55.767 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.767 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.767 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:55.768 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495818+0000", 2024-09-21T15:49:55.768 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1160000000000002e-06 2024-09-21T15:49:55.768 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.768 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.768 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:55.768 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495826+0000", 2024-09-21T15:49:55.768 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7800000000000001e-06 2024-09-21T15:49:55.768 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.768 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.768 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:55.768 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495831+0000", 2024-09-21T15:49:55.768 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9010000000000002e-06 2024-09-21T15:49:55.768 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.768 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.768 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:55.769 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.542665+0000", 2024-09-21T15:49:55.769 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046834037000000002 2024-09-21T15:49:55.769 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.769 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.769 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:55.769 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.542729+0000", 2024-09-21T15:49:55.769 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4199e-05 2024-09-21T15:49:55.769 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.769 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.769 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:55.769 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.542809+0000", 2024-09-21T15:49:55.769 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0592000000000004e-05 2024-09-21T15:49:55.769 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.769 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:55.770 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.770 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.770 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.770 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:70755 3.c 3:32ea75c5:::rbd_data.11336b2fdf4a.0000000000000001:c [read 391168~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:55.770 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.495835+0000", 2024-09-21T15:49:55.770 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 18.182770388000002, 2024-09-21T15:49:55.770 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047136708999999999, 2024-09-21T15:49:55.770 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:55.770 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:55.770 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:55.770 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:55.770 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:55.770 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 70755 2024-09-21T15:49:55.770 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.771 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:55.771 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.771 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:55.771 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495835+0000", 2024-09-21T15:49:55.771 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.771 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.771 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.771 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:55.771 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495835+0000", 2024-09-21T15:49:55.771 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:55.771 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.771 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.771 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:55.771 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495834+0000", 2024-09-21T15:49:55.771 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:55.772 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.772 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.772 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:55.772 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495836+0000", 2024-09-21T15:49:55.772 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1280000000000002e-06 2024-09-21T15:49:55.772 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.772 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.772 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:55.772 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495842+0000", 2024-09-21T15:49:55.772 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8819999999999998e-06 2024-09-21T15:49:55.772 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.772 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.772 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:55.773 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495848+0000", 2024-09-21T15:49:55.773 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1850000000000003e-06 2024-09-21T15:49:55.773 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.773 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.773 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:55.773 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.542836+0000", 2024-09-21T15:49:55.773 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046988156000000003 2024-09-21T15:49:55.773 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.773 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.773 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:55.773 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.542897+0000", 2024-09-21T15:49:55.773 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1505000000000002e-05 2024-09-21T15:49:55.773 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:55.773 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:55.774 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:55.774 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.542971+0000", 2024-09-21T15:49:55.774 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4221000000000006e-05 2024-09-21T15:49:55.774 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.774 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:55.774 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.775 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:55.775 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:55.775 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:55.778 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T15:49:55.780 INFO:teuthology.orchestra.run.smithi089.stdout:4017 write 0x8ea03f4 thru 0x8ea0758 (0x365 bytes) 2024-09-21T15:49:55.780 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:55.780 INFO:teuthology.orchestra.run.smithi089.stdout:4019 write 0x2c7e553 thru 0x2c84489 (0x5f37 bytes) 2024-09-21T15:49:55.780 INFO:teuthology.orchestra.run.smithi089.stdout:4024 punch from 0x43bf183 to 0x43bf619, (0x496 bytes) 2024-09-21T15:49:55.780 INFO:teuthology.orchestra.run.smithi089.stdout:4025 writesame 0x8ba3273 thru 0x8bae6cf data_size 0x1(0xb45d bytes) 2024-09-21T15:49:55.780 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:55.780 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:55.781 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:55.781 INFO:teuthology.orchestra.run.smithi089.stdout:4036 write 0x899e5c1 thru 0x89a253f (0x3f7f bytes) 2024-09-21T15:49:55.781 INFO:teuthology.orchestra.run.smithi089.stdout:4037 writesame 0x162ed2a thru 0x163a76d data_size 0x83(0xba44 bytes) 2024-09-21T15:49:55.781 INFO:teuthology.orchestra.run.smithi089.stdout:4038 write 0x11149eb thru 0x111dfd0 (0x95e6 bytes) 2024-09-21T15:49:55.781 INFO:teuthology.orchestra.run.smithi089.stdout:4040 write 0x67b4fbd thru 0x67b5b37 (0xb7b bytes) 2024-09-21T15:49:55.781 INFO:teuthology.orchestra.run.smithi089.stdout:4041 read 0x2176278 thru 0x21780ee (0x1e77 bytes) 2024-09-21T15:49:55.781 INFO:teuthology.orchestra.run.smithi089.stdout:4042 write 0x55345d2 thru 0x5540ea2 (0xc8d1 bytes) 2024-09-21T15:49:55.781 INFO:teuthology.orchestra.run.smithi089.stdout:4043 write 0x7ef54c8 thru 0x7ef8e94 (0x39cd bytes) 2024-09-21T15:49:55.782 INFO:teuthology.orchestra.run.smithi089.stdout:4044 writesame 0x5f0602 thru 0x5fa1a7 data_size 0x57(0x9ba6 bytes) 2024-09-21T15:49:55.782 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:55.775+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:55.788 INFO:teuthology.orchestra.run.smithi089.stdout:4045 writesame 0x94e334f thru 0x94ee898 data_size 0xaa(0xb54a bytes) 2024-09-21T15:49:55.788 INFO:teuthology.orchestra.run.smithi089.stdout:4046 punch from 0x5dc3c4e to 0x5dcf624, (0xb9d6 bytes) 2024-09-21T15:49:55.796 INFO:teuthology.orchestra.run.smithi089.stdout:4047 trunc from 0x94ee899 to 0x28d0507 2024-09-21T15:49:55.817 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:55.817 INFO:teuthology.orchestra.run.smithi089.stdout:4052 read 0x19be4e1 thru 0x19cb567 (0xd087 bytes) 2024-09-21T15:49:55.837 INFO:teuthology.orchestra.run.smithi089.stdout:4054 write 0xebff62f thru 0xec033bd (0x3d8f bytes) 2024-09-21T15:49:55.843 INFO:teuthology.orchestra.run.smithi089.stdout:4057 read 0xc7ff8c0 thru 0xc80adca (0xb50b bytes) 2024-09-21T15:49:55.844 INFO:teuthology.orchestra.run.smithi089.stdout:4058 read 0xc50619d thru 0xc50697c (0x7e0 bytes) 2024-09-21T15:49:55.844 INFO:teuthology.orchestra.run.smithi089.stdout:4060 clone 26 order 20 su 131072 sc 4 2024-09-21T15:49:55.877 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:55.876+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:55.889 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:55.889 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:55.889 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:49:55.889 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:55.898 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T15:49:55.913 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:49:55.945 INFO:teuthology.orchestra.run.smithi089.stderr:reweighted osd.2 to 0.304193 (4ddf) 2024-09-21T15:49:56.001 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:56.000+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:56.006 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:56.007 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:56.007 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:49:56.007 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:49:56.007 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:56.016 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T15:49:56.036 INFO:teuthology.orchestra.run.smithi089.stdout:flattening image image_client.0-clone25 2024-09-21T15:49:56.047 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-09-21T15:49:56.048 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:49:56.080 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:56.079+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:56.080 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:49:56.080 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:49:56.108 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:49:56.108 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:49:56.122 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:56.123 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:49:56.123 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:49:56.123 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:49:56.123 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.123 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63140 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 876544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:56.123 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.133649+0000", 2024-09-21T15:49:56.123 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 18.987447973999998, 2024-09-21T15:49:56.123 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053950201000000003, 2024-09-21T15:49:56.123 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:56.123 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:56.123 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:56.123 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:56.123 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:56.123 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63140 2024-09-21T15:49:56.124 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.124 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:56.124 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.124 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:56.124 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133649+0000", 2024-09-21T15:49:56.124 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:56.124 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.124 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.124 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:56.124 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133649+0000", 2024-09-21T15:49:56.124 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:56.124 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.124 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.124 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:56.125 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133648+0000", 2024-09-21T15:49:56.125 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:56.125 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.125 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.125 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:56.125 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133651+0000", 2024-09-21T15:49:56.125 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1549999999999999e-06 2024-09-21T15:49:56.125 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.125 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.125 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:56.125 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133660+0000", 2024-09-21T15:49:56.125 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3899999999999993e-06 2024-09-21T15:49:56.125 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.125 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.126 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:56.126 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133669+0000", 2024-09-21T15:49:56.126 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3219999999999999e-06 2024-09-21T15:49:56.126 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.126 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.126 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:56.126 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187409+0000", 2024-09-21T15:49:56.126 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053740257 2024-09-21T15:49:56.126 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.126 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.126 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:56.126 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187492+0000", 2024-09-21T15:49:56.126 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2409000000000003e-05 2024-09-21T15:49:56.126 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.126 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.127 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:56.127 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187599+0000", 2024-09-21T15:49:56.127 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000107334 2024-09-21T15:49:56.127 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:56.127 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:56.127 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:56.127 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.127 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.127 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63141 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 884736~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:56.127 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.133843+0000", 2024-09-21T15:49:56.127 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 18.987253854999999, 2024-09-21T15:49:56.127 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053954015000000001, 2024-09-21T15:49:56.127 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:56.127 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:56.128 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:56.128 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:56.128 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:56.128 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63141 2024-09-21T15:49:56.128 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.128 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:56.128 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.128 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:56.128 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133843+0000", 2024-09-21T15:49:56.128 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:56.128 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.128 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.128 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:56.128 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133843+0000", 2024-09-21T15:49:56.128 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:56.128 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.129 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.129 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:56.129 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133673+0000", 2024-09-21T15:49:56.129 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998302 2024-09-21T15:49:56.129 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.129 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.129 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:56.129 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133846+0000", 2024-09-21T15:49:56.129 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000172762 2024-09-21T15:49:56.129 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.129 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.129 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:56.129 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133855+0000", 2024-09-21T15:49:56.129 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2940000000000006e-06 2024-09-21T15:49:56.129 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.129 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.129 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:56.130 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133863+0000", 2024-09-21T15:49:56.130 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9740000000000002e-06 2024-09-21T15:49:56.130 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.130 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.130 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:56.130 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187630+0000", 2024-09-21T15:49:56.130 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053767211000000002 2024-09-21T15:49:56.130 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.130 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.130 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:56.130 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187707+0000", 2024-09-21T15:49:56.130 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7446000000000004e-05 2024-09-21T15:49:56.130 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.130 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.131 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:56.131 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187797+0000", 2024-09-21T15:49:56.131 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.9858000000000006e-05 2024-09-21T15:49:56.131 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:56.131 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:56.131 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:56.131 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.131 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.131 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63142 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 892928~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:56.131 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134046+0000", 2024-09-21T15:49:56.131 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 18.987050913000001, 2024-09-21T15:49:56.131 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054017849999999999, 2024-09-21T15:49:56.131 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:56.131 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:56.132 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:56.132 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:56.132 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:56.132 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63142 2024-09-21T15:49:56.132 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.132 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:56.132 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.132 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:56.132 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134046+0000", 2024-09-21T15:49:56.132 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:56.132 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.132 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.132 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:56.133 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134046+0000", 2024-09-21T15:49:56.133 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:56.133 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.133 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.133 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:56.133 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133867+0000", 2024-09-21T15:49:56.133 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998217 2024-09-21T15:49:56.133 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.133 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.133 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:56.133 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134049+0000", 2024-09-21T15:49:56.133 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000181695 2024-09-21T15:49:56.133 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.133 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.134 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:56.134 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134057+0000", 2024-09-21T15:49:56.134 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.303e-06 2024-09-21T15:49:56.134 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.134 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.134 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:56.134 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134063+0000", 2024-09-21T15:49:56.134 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9959999999999999e-06 2024-09-21T15:49:56.134 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.134 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.134 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:56.134 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187849+0000", 2024-09-21T15:49:56.134 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053785571999999997 2024-09-21T15:49:56.134 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.135 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.135 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:56.135 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187954+0000", 2024-09-21T15:49:56.135 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010524899999999999 2024-09-21T15:49:56.135 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.135 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.135 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:56.135 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188064+0000", 2024-09-21T15:49:56.135 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000109549 2024-09-21T15:49:56.135 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:56.135 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:56.135 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:56.135 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.135 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.135 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63143 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 901120~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:56.135 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134239+0000", 2024-09-21T15:49:56.136 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 18.98685832, 2024-09-21T15:49:56.136 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054026058000000002, 2024-09-21T15:49:56.136 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:56.136 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:56.136 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:56.136 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:56.136 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:56.136 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63143 2024-09-21T15:49:56.136 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.136 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:56.136 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.136 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:56.136 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134239+0000", 2024-09-21T15:49:56.136 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:56.136 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.137 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.137 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:56.137 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134239+0000", 2024-09-21T15:49:56.137 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:56.137 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.137 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.137 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:56.137 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134068+0000", 2024-09-21T15:49:56.137 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998298 2024-09-21T15:49:56.137 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.137 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.137 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:56.137 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134242+0000", 2024-09-21T15:49:56.137 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00017414199999999999 2024-09-21T15:49:56.137 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.138 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.138 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:56.138 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134264+0000", 2024-09-21T15:49:56.138 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1552e-05 2024-09-21T15:49:56.138 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.138 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.138 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:56.138 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134270+0000", 2024-09-21T15:49:56.138 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6939999999999998e-06 2024-09-21T15:49:56.138 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.138 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.138 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:56.138 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188108+0000", 2024-09-21T15:49:56.138 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.05383719 2024-09-21T15:49:56.139 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.139 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.139 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:56.139 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188186+0000", 2024-09-21T15:49:56.139 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8318000000000005e-05 2024-09-21T15:49:56.139 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.139 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.139 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:56.139 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188265+0000", 2024-09-21T15:49:56.139 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8597000000000007e-05 2024-09-21T15:49:56.139 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:56.139 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:56.139 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:56.139 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.139 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.140 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63144 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 909312~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:56.140 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134432+0000", 2024-09-21T15:49:56.140 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 18.986664764, 2024-09-21T15:49:56.140 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053968953, 2024-09-21T15:49:56.140 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:56.140 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:56.140 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:56.140 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:56.140 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:56.140 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63144 2024-09-21T15:49:56.140 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.140 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:56.140 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.141 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:56.141 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134432+0000", 2024-09-21T15:49:56.141 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:56.141 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.141 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.141 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:56.141 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134432+0000", 2024-09-21T15:49:56.141 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:56.141 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.141 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.141 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:56.141 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134276+0000", 2024-09-21T15:49:56.142 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998441 2024-09-21T15:49:56.142 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.142 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.142 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:56.142 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134436+0000", 2024-09-21T15:49:56.142 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000159591 2024-09-21T15:49:56.142 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.142 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.142 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:56.142 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134445+0000", 2024-09-21T15:49:56.142 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6780000000000006e-06 2024-09-21T15:49:56.142 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.142 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.142 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:56.143 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134452+0000", 2024-09-21T15:49:56.143 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6830000000000008e-06 2024-09-21T15:49:56.143 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.143 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.143 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:56.143 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188282+0000", 2024-09-21T15:49:56.143 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053829252000000001 2024-09-21T15:49:56.143 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.143 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.143 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:56.143 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188328+0000", 2024-09-21T15:49:56.143 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6734999999999999e-05 2024-09-21T15:49:56.143 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.143 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.143 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:56.144 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188401+0000", 2024-09-21T15:49:56.144 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2731000000000005e-05 2024-09-21T15:49:56.144 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:56.144 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:56.144 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:56.144 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.144 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.144 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63145 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 860160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:56.144 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134736+0000", 2024-09-21T15:49:56.144 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 18.986360979000001, 2024-09-21T15:49:56.144 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053865960999999997, 2024-09-21T15:49:56.144 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:56.144 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:56.145 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:56.145 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:56.145 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:56.145 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63145 2024-09-21T15:49:56.145 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.145 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:56.145 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.145 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:56.145 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134736+0000", 2024-09-21T15:49:56.145 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:56.145 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.145 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.146 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:56.146 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134736+0000", 2024-09-21T15:49:56.146 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:56.146 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.146 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.146 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:56.146 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134458+0000", 2024-09-21T15:49:56.146 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999722 2024-09-21T15:49:56.146 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.146 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.146 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:56.146 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134741+0000", 2024-09-21T15:49:56.146 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00028245599999999999 2024-09-21T15:49:56.146 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.147 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.147 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:56.147 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134753+0000", 2024-09-21T15:49:56.147 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2548000000000001e-05 2024-09-21T15:49:56.147 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.147 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.147 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:56.147 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134760+0000", 2024-09-21T15:49:56.147 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4189999999999996e-06 2024-09-21T15:49:56.147 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.147 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.147 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:56.148 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188426+0000", 2024-09-21T15:49:56.148 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053665894999999998 2024-09-21T15:49:56.148 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.148 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.148 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:56.148 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188502+0000", 2024-09-21T15:49:56.148 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5791000000000007e-05 2024-09-21T15:49:56.148 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.148 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.148 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:56.148 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188602+0000", 2024-09-21T15:49:56.148 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9667999999999995e-05 2024-09-21T15:49:56.149 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:56.149 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:56.149 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:56.149 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.149 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.149 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63148 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 933888~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:56.149 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.135289+0000", 2024-09-21T15:49:56.149 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 18.985808024000001, 2024-09-21T15:49:56.149 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053948488000000003, 2024-09-21T15:49:56.149 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:56.149 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:56.149 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:56.149 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:56.149 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:56.150 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63148 2024-09-21T15:49:56.150 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.150 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:56.150 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.150 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:56.150 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135289+0000", 2024-09-21T15:49:56.150 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:56.150 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.150 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.150 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:56.150 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135289+0000", 2024-09-21T15:49:56.151 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:56.151 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.151 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.151 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:56.151 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135288+0000", 2024-09-21T15:49:56.151 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:56.151 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.151 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.151 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:56.151 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135296+0000", 2024-09-21T15:49:56.151 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5399999999999998e-06 2024-09-21T15:49:56.151 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.151 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.151 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:56.151 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135303+0000", 2024-09-21T15:49:56.152 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0500000000000003e-06 2024-09-21T15:49:56.152 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.152 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.152 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:56.152 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135308+0000", 2024-09-21T15:49:56.152 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9749999999999998e-06 2024-09-21T15:49:56.152 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.152 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.152 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:56.152 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189066+0000", 2024-09-21T15:49:56.152 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053758501 2024-09-21T15:49:56.152 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.152 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.152 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:56.153 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189145+0000", 2024-09-21T15:49:56.153 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9017000000000006e-05 2024-09-21T15:49:56.153 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.153 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.153 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:56.153 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189237+0000", 2024-09-21T15:49:56.153 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1906999999999999e-05 2024-09-21T15:49:56.153 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:56.153 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:56.153 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:56.153 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.153 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.153 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63149 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 942080~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:56.153 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.135430+0000", 2024-09-21T15:49:56.153 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 18.985666694999999, 2024-09-21T15:49:56.154 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054005434999999997, 2024-09-21T15:49:56.154 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:56.154 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:56.154 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:56.154 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:56.154 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:56.154 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63149 2024-09-21T15:49:56.154 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.154 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:56.154 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.154 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:56.154 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135430+0000", 2024-09-21T15:49:56.154 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:56.154 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.155 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.155 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:56.155 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135430+0000", 2024-09-21T15:49:56.155 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:56.155 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.155 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.155 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:56.155 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135313+0000", 2024-09-21T15:49:56.155 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998827 2024-09-21T15:49:56.155 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.155 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.155 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:56.155 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135434+0000", 2024-09-21T15:49:56.155 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000120876 2024-09-21T15:49:56.155 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.156 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.156 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:56.156 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135443+0000", 2024-09-21T15:49:56.156 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3340000000000008e-06 2024-09-21T15:49:56.156 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.156 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.156 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:56.156 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135449+0000", 2024-09-21T15:49:56.156 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1580000000000002e-06 2024-09-21T15:49:56.156 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.156 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.156 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:56.156 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189267+0000", 2024-09-21T15:49:56.156 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053817775999999998 2024-09-21T15:49:56.157 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.157 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.157 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:56.157 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189336+0000", 2024-09-21T15:49:56.157 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9485000000000003e-05 2024-09-21T15:49:56.157 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.157 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.157 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:56.157 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189436+0000", 2024-09-21T15:49:56.157 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9277000000000005e-05 2024-09-21T15:49:56.157 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:56.157 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:56.157 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:56.158 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.158 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.158 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63150 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 950272~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:56.158 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.135618+0000", 2024-09-21T15:49:56.158 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 18.985478950000001, 2024-09-21T15:49:56.158 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054027393, 2024-09-21T15:49:56.158 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:56.158 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:56.158 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:56.158 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:56.158 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:56.158 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63150 2024-09-21T15:49:56.158 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.158 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:56.159 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.159 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:56.159 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135618+0000", 2024-09-21T15:49:56.159 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:56.159 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.159 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.159 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:56.159 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135618+0000", 2024-09-21T15:49:56.159 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:56.159 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.159 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.159 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:56.159 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135454+0000", 2024-09-21T15:49:56.159 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998364 2024-09-21T15:49:56.159 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.160 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.160 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:56.160 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135621+0000", 2024-09-21T15:49:56.160 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016714099999999999 2024-09-21T15:49:56.160 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.160 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.160 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:56.160 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135631+0000", 2024-09-21T15:49:56.160 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5340000000000005e-06 2024-09-21T15:49:56.160 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.160 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.160 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:56.160 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135638+0000", 2024-09-21T15:49:56.161 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9280000000000004e-06 2024-09-21T15:49:56.161 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.161 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.161 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:56.161 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189463+0000", 2024-09-21T15:49:56.161 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053825681 2024-09-21T15:49:56.161 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.161 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.161 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:56.161 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189546+0000", 2024-09-21T15:49:56.161 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2278000000000001e-05 2024-09-21T15:49:56.161 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.161 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.161 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:56.162 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189645+0000", 2024-09-21T15:49:56.162 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9543999999999994e-05 2024-09-21T15:49:56.162 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:56.162 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:56.162 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:56.162 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.162 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.162 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63152 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 966656~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:56.162 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.136029+0000", 2024-09-21T15:49:56.162 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 18.985068357999999, 2024-09-21T15:49:56.162 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054024823999999999, 2024-09-21T15:49:56.162 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:56.162 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:56.162 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:56.162 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:56.163 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:56.163 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63152 2024-09-21T15:49:56.163 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.163 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:56.163 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.163 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:56.163 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136029+0000", 2024-09-21T15:49:56.163 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:56.163 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.163 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.163 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:56.163 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136029+0000", 2024-09-21T15:49:56.163 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:56.163 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.163 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.164 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:56.164 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136028+0000", 2024-09-21T15:49:56.164 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:56.164 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.164 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.164 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:56.164 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136031+0000", 2024-09-21T15:49:56.164 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.033e-06 2024-09-21T15:49:56.164 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.164 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.164 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:56.164 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136039+0000", 2024-09-21T15:49:56.164 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7200000000000006e-06 2024-09-21T15:49:56.164 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.165 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.165 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:56.165 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136051+0000", 2024-09-21T15:49:56.165 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1987000000000001e-05 2024-09-21T15:49:56.165 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.165 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.165 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:56.165 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189876+0000", 2024-09-21T15:49:56.165 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053825264999999997 2024-09-21T15:49:56.165 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.165 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.165 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:56.165 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189955+0000", 2024-09-21T15:49:56.165 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9296999999999997e-05 2024-09-21T15:49:56.165 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.166 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.166 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:56.166 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190053+0000", 2024-09-21T15:49:56.166 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.8067999999999997e-05 2024-09-21T15:49:56.166 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:56.166 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:56.166 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:56.166 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.166 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.166 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63153 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 974848~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:56.166 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.136311+0000", 2024-09-21T15:49:56.166 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 18.984785845000001, 2024-09-21T15:49:56.166 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053933825999999997, 2024-09-21T15:49:56.166 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:56.167 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:56.167 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:56.167 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:56.167 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:56.167 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63153 2024-09-21T15:49:56.167 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.167 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:56.167 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.167 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:56.167 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136311+0000", 2024-09-21T15:49:56.167 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:56.167 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.167 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.167 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:56.167 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136311+0000", 2024-09-21T15:49:56.168 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:56.168 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.168 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.168 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:56.168 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136057+0000", 2024-09-21T15:49:56.168 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9997463 2024-09-21T15:49:56.168 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.168 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.168 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:56.168 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136316+0000", 2024-09-21T15:49:56.168 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00025872299999999998 2024-09-21T15:49:56.169 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.169 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.169 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:56.169 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136326+0000", 2024-09-21T15:49:56.169 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0552e-05 2024-09-21T15:49:56.169 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.169 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.169 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:56.169 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136334+0000", 2024-09-21T15:49:56.169 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1409999999999999e-06 2024-09-21T15:49:56.169 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.169 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.169 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:56.169 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190080+0000", 2024-09-21T15:49:56.170 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053745933000000003 2024-09-21T15:49:56.170 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.170 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.170 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:56.170 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190157+0000", 2024-09-21T15:49:56.170 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7412999999999999e-05 2024-09-21T15:49:56.170 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.170 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.170 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:56.170 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190245+0000", 2024-09-21T15:49:56.170 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7904000000000004e-05 2024-09-21T15:49:56.170 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:56.171 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:56.171 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:56.171 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.171 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.171 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63154 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 983040~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:56.171 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.136472+0000", 2024-09-21T15:49:56.171 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 18.984624493999998, 2024-09-21T15:49:56.171 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053968448000000002, 2024-09-21T15:49:56.171 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:56.171 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:56.171 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:56.171 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:56.171 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:56.171 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63154 2024-09-21T15:49:56.172 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.172 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:56.172 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.172 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:56.172 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136472+0000", 2024-09-21T15:49:56.172 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:56.172 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.172 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.172 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:56.172 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136472+0000", 2024-09-21T15:49:56.172 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:56.172 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.172 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.172 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:56.172 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136341+0000", 2024-09-21T15:49:56.173 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998684 2024-09-21T15:49:56.173 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.173 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.173 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:56.173 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136476+0000", 2024-09-21T15:49:56.173 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013528100000000001 2024-09-21T15:49:56.173 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.173 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.173 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:56.173 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136484+0000", 2024-09-21T15:49:56.173 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4760000000000001e-06 2024-09-21T15:49:56.173 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.173 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.173 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:56.173 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136490+0000", 2024-09-21T15:49:56.174 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8640000000000001e-06 2024-09-21T15:49:56.174 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.174 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.174 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:56.174 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190271+0000", 2024-09-21T15:49:56.174 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053781223000000003 2024-09-21T15:49:56.174 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.174 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.174 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:56.174 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190348+0000", 2024-09-21T15:49:56.174 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7058999999999995e-05 2024-09-21T15:49:56.174 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.175 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.175 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:56.175 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190441+0000", 2024-09-21T15:49:56.175 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2937999999999997e-05 2024-09-21T15:49:56.175 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:56.175 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:56.175 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:56.175 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.175 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.175 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63157 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1015808~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:56.175 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137112+0000", 2024-09-21T15:49:56.175 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 18.983985207, 2024-09-21T15:49:56.175 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053932345, 2024-09-21T15:49:56.175 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:56.175 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:56.176 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:56.176 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:56.176 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:56.176 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63157 2024-09-21T15:49:56.176 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.176 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:56.176 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.176 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:56.176 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137112+0000", 2024-09-21T15:49:56.176 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:56.176 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.176 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.176 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:56.176 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137112+0000", 2024-09-21T15:49:56.176 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:56.177 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.177 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.177 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:56.177 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137072+0000", 2024-09-21T15:49:56.177 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999604 2024-09-21T15:49:56.177 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.177 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.177 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:56.177 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137115+0000", 2024-09-21T15:49:56.177 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2379999999999997e-05 2024-09-21T15:49:56.177 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.177 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.177 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:56.178 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137121+0000", 2024-09-21T15:49:56.178 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6499999999999999e-06 2024-09-21T15:49:56.178 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.178 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.178 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:56.178 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137126+0000", 2024-09-21T15:49:56.178 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0769999999999998e-06 2024-09-21T15:49:56.178 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.178 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.178 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:56.178 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190868+0000", 2024-09-21T15:49:56.178 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053741555000000003 2024-09-21T15:49:56.178 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.179 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.179 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:56.179 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190945+0000", 2024-09-21T15:49:56.179 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7171999999999999e-05 2024-09-21T15:49:56.179 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.179 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.179 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:56.179 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191044+0000", 2024-09-21T15:49:56.179 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9031999999999996e-05 2024-09-21T15:49:56.179 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:56.179 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:56.179 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:56.180 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.180 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.180 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63158 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1007616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:56.180 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137293+0000", 2024-09-21T15:49:56.180 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 18.98380392, 2024-09-21T15:49:56.180 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054005642, 2024-09-21T15:49:56.180 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:56.180 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:56.180 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:56.180 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:56.180 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:56.180 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63158 2024-09-21T15:49:56.180 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.180 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:56.180 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.180 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:56.181 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137293+0000", 2024-09-21T15:49:56.181 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:56.181 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.181 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.181 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:56.181 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137293+0000", 2024-09-21T15:49:56.181 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:56.181 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.181 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.181 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:56.181 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137131+0000", 2024-09-21T15:49:56.181 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998379 2024-09-21T15:49:56.181 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.181 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.182 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:56.182 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137296+0000", 2024-09-21T15:49:56.182 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016568999999999999 2024-09-21T15:49:56.182 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.182 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.182 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:56.182 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137306+0000", 2024-09-21T15:49:56.182 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.6029999999999994e-06 2024-09-21T15:49:56.182 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.182 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.182 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:56.182 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137312+0000", 2024-09-21T15:49:56.182 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3210000000000001e-06 2024-09-21T15:49:56.182 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.183 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.183 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:56.183 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191087+0000", 2024-09-21T15:49:56.183 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053774686000000002 2024-09-21T15:49:56.183 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.183 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.183 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:56.183 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191175+0000", 2024-09-21T15:49:56.183 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8106e-05 2024-09-21T15:49:56.183 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.183 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.183 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:56.183 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191299+0000", 2024-09-21T15:49:56.183 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012343599999999999 2024-09-21T15:49:56.183 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:56.184 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:56.184 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:56.184 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.184 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.184 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63159 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1024000~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:56.184 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137568+0000", 2024-09-21T15:49:56.184 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 18.983528648, 2024-09-21T15:49:56.184 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053962999999999997, 2024-09-21T15:49:56.184 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:56.184 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:56.184 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:56.184 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:56.185 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:56.185 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63159 2024-09-21T15:49:56.185 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.185 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:56.185 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.185 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:56.185 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137568+0000", 2024-09-21T15:49:56.185 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:56.185 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.185 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.185 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:56.185 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137568+0000", 2024-09-21T15:49:56.185 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:56.185 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.186 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.186 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:56.186 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137317+0000", 2024-09-21T15:49:56.186 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9997492 2024-09-21T15:49:56.186 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.186 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.186 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:56.186 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137572+0000", 2024-09-21T15:49:56.186 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00025491499999999998 2024-09-21T15:49:56.186 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.186 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.186 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:56.186 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137582+0000", 2024-09-21T15:49:56.186 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4490000000000005e-06 2024-09-21T15:49:56.187 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.187 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.187 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:56.187 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137589+0000", 2024-09-21T15:49:56.187 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5580000000000003e-06 2024-09-21T15:49:56.187 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.187 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.187 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:56.187 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191344+0000", 2024-09-21T15:49:56.187 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053754272999999998 2024-09-21T15:49:56.187 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.187 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.187 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:56.188 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191429+0000", 2024-09-21T15:49:56.188 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.509e-05 2024-09-21T15:49:56.188 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.188 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.188 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:56.188 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191531+0000", 2024-09-21T15:49:56.188 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000102519 2024-09-21T15:49:56.188 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:56.188 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:56.188 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:56.188 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.188 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.188 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63160 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1032192~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:56.188 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137771+0000", 2024-09-21T15:49:56.188 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 18.983326114, 2024-09-21T15:49:56.189 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053957818999999997, 2024-09-21T15:49:56.189 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:56.189 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:56.189 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:56.189 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:56.189 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:56.189 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63160 2024-09-21T15:49:56.189 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.189 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:56.189 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.189 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:56.189 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137771+0000", 2024-09-21T15:49:56.189 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:56.189 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.190 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.190 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:56.190 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137771+0000", 2024-09-21T15:49:56.190 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:56.190 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.190 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.190 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:56.190 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137596+0000", 2024-09-21T15:49:56.190 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998255 2024-09-21T15:49:56.190 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.190 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.190 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:56.190 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137780+0000", 2024-09-21T15:49:56.190 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00018363900000000001 2024-09-21T15:49:56.191 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.191 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.191 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:56.191 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137791+0000", 2024-09-21T15:49:56.191 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.096e-05 2024-09-21T15:49:56.191 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.191 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.191 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:56.191 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137798+0000", 2024-09-21T15:49:56.191 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1169999999999998e-06 2024-09-21T15:49:56.192 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.192 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.192 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:56.192 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191560+0000", 2024-09-21T15:49:56.192 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053762163000000002 2024-09-21T15:49:56.192 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.192 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.192 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:56.192 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191634+0000", 2024-09-21T15:49:56.192 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4381999999999994e-05 2024-09-21T15:49:56.192 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.192 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.192 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:56.192 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191729+0000", 2024-09-21T15:49:56.193 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4098000000000006e-05 2024-09-21T15:49:56.193 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:56.193 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:56.193 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:56.193 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.193 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.193 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63161 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1040384~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:56.193 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.138018+0000", 2024-09-21T15:49:56.193 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 18.983079360000001, 2024-09-21T15:49:56.193 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053905657000000003, 2024-09-21T15:49:56.193 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:56.193 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:56.193 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:56.193 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:56.194 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:56.194 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63161 2024-09-21T15:49:56.194 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.194 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:56.194 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.194 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:56.194 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138018+0000", 2024-09-21T15:49:56.194 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:56.194 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.194 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.194 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:56.194 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138018+0000", 2024-09-21T15:49:56.195 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:56.195 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.195 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.195 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:56.195 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137804+0000", 2024-09-21T15:49:56.195 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9997859 2024-09-21T15:49:56.195 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.195 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.195 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:56.195 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138021+0000", 2024-09-21T15:49:56.195 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00021769899999999999 2024-09-21T15:49:56.195 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.195 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.195 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:56.196 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138043+0000", 2024-09-21T15:49:56.196 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1627000000000001e-05 2024-09-21T15:49:56.196 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.196 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.196 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:56.196 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138054+0000", 2024-09-21T15:49:56.196 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1321999999999999e-05 2024-09-21T15:49:56.196 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.196 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.196 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:56.196 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191755+0000", 2024-09-21T15:49:56.196 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053700354999999998 2024-09-21T15:49:56.196 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.196 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.196 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:56.197 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191830+0000", 2024-09-21T15:49:56.197 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5320000000000004e-05 2024-09-21T15:49:56.197 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.197 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.197 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:56.197 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191923+0000", 2024-09-21T15:49:56.197 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3282999999999994e-05 2024-09-21T15:49:56.197 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:56.197 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:56.197 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:56.197 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.197 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.197 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63176 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 108544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:56.197 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.140861+0000", 2024-09-21T15:49:56.198 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 18.980236041000001, 2024-09-21T15:49:56.198 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053935394999999997, 2024-09-21T15:49:56.198 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:56.198 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:56.198 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:56.198 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:56.198 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:56.198 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63176 2024-09-21T15:49:56.198 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.198 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:56.198 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.198 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:56.198 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140861+0000", 2024-09-21T15:49:56.198 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:56.199 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.199 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.199 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:56.199 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140861+0000", 2024-09-21T15:49:56.199 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:56.199 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.199 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.199 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:56.199 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140860+0000", 2024-09-21T15:49:56.199 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:56.199 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.199 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.199 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:56.199 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140863+0000", 2024-09-21T15:49:56.199 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8710000000000001e-06 2024-09-21T15:49:56.200 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.200 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.200 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:56.200 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140871+0000", 2024-09-21T15:49:56.200 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8089999999999993e-06 2024-09-21T15:49:56.200 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.200 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.200 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:56.200 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140877+0000", 2024-09-21T15:49:56.200 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5969999999999999e-06 2024-09-21T15:49:56.200 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.200 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.200 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:56.200 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.194608+0000", 2024-09-21T15:49:56.201 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053731749000000002 2024-09-21T15:49:56.201 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.201 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.201 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:56.201 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.194697+0000", 2024-09-21T15:49:56.201 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8504000000000005e-05 2024-09-21T15:49:56.201 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.201 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.201 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:56.201 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.194796+0000", 2024-09-21T15:49:56.201 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9414999999999996e-05 2024-09-21T15:49:56.201 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:56.201 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:56.201 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:56.202 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.202 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.202 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63182 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 157696~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:56.202 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:49:56.202 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 18.978963641, 2024-09-21T15:49:56.202 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053958825000000002, 2024-09-21T15:49:56.202 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:56.202 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:56.202 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:56.202 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:56.202 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:56.202 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63182 2024-09-21T15:49:56.202 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.202 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:56.203 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.203 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:56.203 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:49:56.203 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:56.203 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.203 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.203 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:56.203 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:49:56.203 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:56.203 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.203 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.203 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:56.203 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:49:56.203 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:56.203 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.204 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.204 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:56.204 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142136+0000", 2024-09-21T15:49:56.204 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8169999999999999e-06 2024-09-21T15:49:56.204 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.204 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.204 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:56.204 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:49:56.204 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3752e-05 2024-09-21T15:49:56.204 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.204 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.204 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:56.205 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142155+0000", 2024-09-21T15:49:56.205 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1859999999999997e-06 2024-09-21T15:49:56.205 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.205 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.205 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:56.205 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.195914+0000", 2024-09-21T15:49:56.205 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053758528999999999 2024-09-21T15:49:56.205 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.205 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.205 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:56.205 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.195995+0000", 2024-09-21T15:49:56.205 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0983999999999996e-05 2024-09-21T15:49:56.205 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.206 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.206 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:56.206 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.196092+0000", 2024-09-21T15:49:56.206 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7069000000000002e-05 2024-09-21T15:49:56.206 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:56.206 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:56.206 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:56.206 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.206 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.206 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63193 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 239616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:56.206 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:49:56.206 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 18.976876056000002, 2024-09-21T15:49:56.206 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053956577999999998, 2024-09-21T15:49:56.207 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:56.207 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:56.207 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:56.207 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:56.207 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:56.207 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63193 2024-09-21T15:49:56.207 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.207 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:56.207 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.207 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:56.207 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:49:56.207 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:56.207 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.207 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.207 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:56.208 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:49:56.208 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:56.208 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.208 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.208 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:56.208 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:49:56.208 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:56.208 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.208 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.208 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:56.208 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144223+0000", 2024-09-21T15:49:56.208 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0990000000000001e-06 2024-09-21T15:49:56.208 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.208 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.209 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:56.209 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144230+0000", 2024-09-21T15:49:56.209 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9070000000000001e-06 2024-09-21T15:49:56.209 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.209 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.209 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:56.209 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144234+0000", 2024-09-21T15:49:56.209 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9010000000000002e-06 2024-09-21T15:49:56.209 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.209 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.209 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:56.209 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.198004+0000", 2024-09-21T15:49:56.209 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053769539999999998 2024-09-21T15:49:56.209 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.210 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.210 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:56.210 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.198083+0000", 2024-09-21T15:49:56.210 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9531999999999997e-05 2024-09-21T15:49:56.210 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:56.210 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:56.210 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:56.210 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.198177+0000", 2024-09-21T15:49:56.210 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3906000000000005e-05 2024-09-21T15:49:56.210 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:56.210 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:56.210 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:56.210 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:56.210 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:56.211 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:56.211 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:49:56.212 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:49:56.213 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:49:56.213 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T15:49:56.213 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:49:56.214 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:49:56.214 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:49:56.214 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:56.210+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:56.227 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:49:56.227 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:49:56.245 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:49:56.245 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:49:56.312 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:56.311+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:56.317 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:56.318 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:56.326 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T15:49:56.414 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:56.412+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:56.469 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:56.469 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:56.478 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T15:49:56.519 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:56.518+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:56.615 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:56.614+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:56.620 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:56.620 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:56.629 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:49:56.716 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:56.715+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:56.733 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:56.733 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:56.742 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:49:56.826 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:56.825+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:56.883 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:56.884 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:56.892 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:49:56.933 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:56.932+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:56.941 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T15:49:57.019 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:57.018+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:57.039 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:57.039 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:57.049 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:49:57.119 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:57.119+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:57.154 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:57.154 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:57.163 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:49:57.220 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:57.219+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:57.266 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:57.266 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:57.275 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:49:57.321 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:57.320+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:57.417 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:57.417 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:57.421 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:57.420+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:57.426 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:49:57.522 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:57.521+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:57.567 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:57.567 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:57.575 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:49:57.624 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:57.623+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:57.678 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:57.678 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:57.687 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:49:57.724 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:57.723+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:57.789 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:57.789 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:57.797 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:49:57.841 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:57.840+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:57.900 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:57.900 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:57.909 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:49:57.944 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is set 2024-09-21T15:49:57.944 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:57.943+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:58.027 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:58.026+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:58.047 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:58.047 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:58.056 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:49:58.128 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:58.127+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:58.162 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:58.162 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:58.171 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:49:58.229 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:58.228+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:58.256 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:49:58.275 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:58.275 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:58.284 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:49:58.368 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:58.367+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:58.374 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-09-21T15:49:58.375 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:49:58.391 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:49:58.391 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:49:58.406 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:49:58.406 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:49:58.421 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:49:58.421 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:49:58.426 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:49:58.426 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:49:58.430 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:49:58.429+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:49:58.435 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:49:58.437 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:49:58.437 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:49:58.454 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:49:58.454 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:49:58.470 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:49:58.470 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:49:58.486 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:49:58.486 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:49:58.538 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:58.537+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:58.578 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:58.578 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:49:58.578 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:49:58.578 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:49:58.578 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.578 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26224 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 2326528~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:58.578 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.504487+0000", 2024-09-21T15:49:58.578 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.071622431000002, 2024-09-21T15:49:58.578 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032819441999999997, 2024-09-21T15:49:58.578 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:58.578 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:58.579 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:58.579 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:58.579 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:58.579 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26224 2024-09-21T15:49:58.579 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.579 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:58.579 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.579 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:58.579 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504487+0000", 2024-09-21T15:49:58.579 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:58.579 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.579 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.579 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:58.579 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504487+0000", 2024-09-21T15:49:58.580 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:58.580 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.580 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.580 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:58.580 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504487+0000", 2024-09-21T15:49:58.580 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:58.580 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.580 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.580 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:58.580 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504490+0000", 2024-09-21T15:49:58.580 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6049999999999998e-06 2024-09-21T15:49:58.580 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.580 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.580 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:58.580 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504493+0000", 2024-09-21T15:49:58.581 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1200000000000002e-06 2024-09-21T15:49:58.581 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.581 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.581 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:58.581 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504496+0000", 2024-09-21T15:49:58.581 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.977e-06 2024-09-21T15:49:58.581 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.581 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.581 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:58.581 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537221+0000", 2024-09-21T15:49:58.581 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032724529000000002 2024-09-21T15:49:58.581 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.581 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.581 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:58.581 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537259+0000", 2024-09-21T15:49:58.582 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7923999999999998e-05 2024-09-21T15:49:58.582 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.582 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.582 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:58.582 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537306+0000", 2024-09-21T15:49:58.582 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7512999999999997e-05 2024-09-21T15:49:58.582 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:58.582 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:58.582 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:58.582 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.582 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.582 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26225 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 2383872~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:58.582 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.504498+0000", 2024-09-21T15:49:58.582 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.071610862, 2024-09-21T15:49:58.583 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.03291703, 2024-09-21T15:49:58.583 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:58.583 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:58.583 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:58.583 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:58.583 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:58.583 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26225 2024-09-21T15:49:58.583 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.583 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:58.583 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.583 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:58.583 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504498+0000", 2024-09-21T15:49:58.583 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:58.583 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.583 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.584 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:58.584 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504498+0000", 2024-09-21T15:49:58.584 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:58.584 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.584 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.584 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:58.584 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504498+0000", 2024-09-21T15:49:58.584 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:58.584 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.584 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.584 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:58.584 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504499+0000", 2024-09-21T15:49:58.584 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0070000000000001e-06 2024-09-21T15:49:58.584 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.584 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.585 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:58.585 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504502+0000", 2024-09-21T15:49:58.585 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8119999999999999e-06 2024-09-21T15:49:58.585 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.585 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.585 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:58.585 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504504+0000", 2024-09-21T15:49:58.585 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4380000000000002e-06 2024-09-21T15:49:58.585 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.585 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.585 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:58.585 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537328+0000", 2024-09-21T15:49:58.585 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032823659999999998 2024-09-21T15:49:58.585 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.585 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.585 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:58.586 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537366+0000", 2024-09-21T15:49:58.586 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8083000000000001e-05 2024-09-21T15:49:58.586 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.586 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.586 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:58.586 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537415+0000", 2024-09-21T15:49:58.586 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9233000000000001e-05 2024-09-21T15:49:58.586 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:58.586 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:58.586 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:58.586 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.586 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.586 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26226 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 2441216~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:58.586 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.504506+0000", 2024-09-21T15:49:58.587 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.071602747, 2024-09-21T15:49:58.587 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033016726000000003, 2024-09-21T15:49:58.587 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:58.587 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:58.587 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:58.587 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:58.587 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:58.587 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26226 2024-09-21T15:49:58.587 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.587 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:58.587 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.587 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:58.587 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504506+0000", 2024-09-21T15:49:58.588 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:58.588 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.588 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.588 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:58.588 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504506+0000", 2024-09-21T15:49:58.588 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:58.588 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.588 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.588 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:58.588 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504506+0000", 2024-09-21T15:49:58.588 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:58.588 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.588 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.588 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:58.589 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504507+0000", 2024-09-21T15:49:58.589 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9699999999999994e-07 2024-09-21T15:49:58.589 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.589 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.589 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:58.589 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504510+0000", 2024-09-21T15:49:58.589 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.773e-06 2024-09-21T15:49:58.589 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.589 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.589 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:58.589 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504514+0000", 2024-09-21T15:49:58.589 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1150000000000004e-06 2024-09-21T15:49:58.589 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.589 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.590 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:58.590 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537437+0000", 2024-09-21T15:49:58.590 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032923249000000002 2024-09-21T15:49:58.590 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.590 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.590 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:58.590 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537476+0000", 2024-09-21T15:49:58.590 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8679e-05 2024-09-21T15:49:58.590 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.590 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.590 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:58.590 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537523+0000", 2024-09-21T15:49:58.590 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7119000000000001e-05 2024-09-21T15:49:58.590 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:58.591 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:58.591 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:58.591 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.591 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.591 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26227 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 2498560~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:58.591 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.504516+0000", 2024-09-21T15:49:58.591 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.071593145999998, 2024-09-21T15:49:58.591 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033117681000000003, 2024-09-21T15:49:58.591 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:58.591 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:58.591 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:58.591 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:58.591 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:58.592 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26227 2024-09-21T15:49:58.592 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.592 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:58.592 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.592 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:58.592 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504516+0000", 2024-09-21T15:49:58.592 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:58.592 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.592 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.592 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:58.592 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504516+0000", 2024-09-21T15:49:58.592 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:58.592 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.592 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.593 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:58.593 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504516+0000", 2024-09-21T15:49:58.593 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:58.593 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.593 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.593 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:58.593 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504517+0000", 2024-09-21T15:49:58.593 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0610000000000001e-06 2024-09-21T15:49:58.593 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.593 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.593 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:58.593 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504520+0000", 2024-09-21T15:49:58.593 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.83e-06 2024-09-21T15:49:58.593 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.593 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.594 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:58.594 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504522+0000", 2024-09-21T15:49:58.594 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.497e-06 2024-09-21T15:49:58.594 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.594 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.594 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:58.594 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537544+0000", 2024-09-21T15:49:58.594 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033021948000000002 2024-09-21T15:49:58.594 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.594 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.594 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:58.594 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537585+0000", 2024-09-21T15:49:58.594 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1094999999999999e-05 2024-09-21T15:49:58.594 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.595 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.595 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:58.595 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537634+0000", 2024-09-21T15:49:58.595 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8449000000000003e-05 2024-09-21T15:49:58.595 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:58.595 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:58.595 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:58.595 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.595 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.595 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26228 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 2555904~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:58.595 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.504524+0000", 2024-09-21T15:49:58.595 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.071585048000003, 2024-09-21T15:49:58.595 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.03321735, 2024-09-21T15:49:58.595 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:58.596 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:58.596 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:58.596 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:58.596 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:58.596 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26228 2024-09-21T15:49:58.596 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.596 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:58.596 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.596 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:58.596 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504524+0000", 2024-09-21T15:49:58.596 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:58.596 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.596 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.596 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:58.596 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504524+0000", 2024-09-21T15:49:58.596 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:58.597 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.597 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.597 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:58.597 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504524+0000", 2024-09-21T15:49:58.597 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:58.597 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.597 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.597 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:58.597 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504525+0000", 2024-09-21T15:49:58.597 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0419999999999999e-06 2024-09-21T15:49:58.597 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.597 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.597 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:58.597 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504528+0000", 2024-09-21T15:49:58.597 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7530000000000002e-06 2024-09-21T15:49:58.598 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.598 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.598 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:58.598 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504530+0000", 2024-09-21T15:49:58.598 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4530000000000001e-06 2024-09-21T15:49:58.598 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.598 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.598 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:58.598 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537656+0000", 2024-09-21T15:49:58.598 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033126023999999997 2024-09-21T15:49:58.598 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.598 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.598 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:58.598 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537694+0000", 2024-09-21T15:49:58.598 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7799000000000002e-05 2024-09-21T15:49:58.599 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.599 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.599 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:58.599 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537741+0000", 2024-09-21T15:49:58.599 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7484999999999997e-05 2024-09-21T15:49:58.599 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:58.599 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:58.599 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:58.599 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.599 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.599 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26229 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 2613248~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:58.599 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.504532+0000", 2024-09-21T15:49:58.599 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.071577169000001, 2024-09-21T15:49:58.599 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033315897999999997, 2024-09-21T15:49:58.600 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:58.600 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:58.600 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:58.600 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:58.600 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:58.600 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26229 2024-09-21T15:49:58.600 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.600 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:58.600 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.600 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:58.600 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504532+0000", 2024-09-21T15:49:58.600 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:58.600 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.600 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.600 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:58.601 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504532+0000", 2024-09-21T15:49:58.601 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:58.601 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.601 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.601 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:58.601 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504532+0000", 2024-09-21T15:49:58.601 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:58.601 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.601 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.601 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:58.601 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504533+0000", 2024-09-21T15:49:58.601 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0610000000000001e-06 2024-09-21T15:49:58.601 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.601 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.601 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:58.601 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504536+0000", 2024-09-21T15:49:58.602 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.309e-06 2024-09-21T15:49:58.602 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.602 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.602 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:58.602 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504539+0000", 2024-09-21T15:49:58.602 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.435e-06 2024-09-21T15:49:58.602 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.602 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.602 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:58.602 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537762+0000", 2024-09-21T15:49:58.602 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033223813999999997 2024-09-21T15:49:58.602 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.602 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.602 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:58.602 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537801+0000", 2024-09-21T15:49:58.603 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8136999999999998e-05 2024-09-21T15:49:58.603 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.603 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.603 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:58.603 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537848+0000", 2024-09-21T15:49:58.603 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7345999999999997e-05 2024-09-21T15:49:58.603 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:58.603 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:58.603 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:58.603 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.603 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.603 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26230 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 2670592~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:58.603 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.504606+0000", 2024-09-21T15:49:58.603 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.071503010999997, 2024-09-21T15:49:58.604 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033348801999999997, 2024-09-21T15:49:58.604 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:58.604 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:58.604 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:58.604 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:58.604 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:58.604 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26230 2024-09-21T15:49:58.604 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.604 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:58.604 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.604 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:58.604 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504606+0000", 2024-09-21T15:49:58.604 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:58.604 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.605 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.605 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:58.605 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504606+0000", 2024-09-21T15:49:58.605 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:58.605 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.605 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.605 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:58.605 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504541+0000", 2024-09-21T15:49:58.605 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999352 2024-09-21T15:49:58.605 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.605 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.605 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:58.605 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504608+0000", 2024-09-21T15:49:58.605 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6427000000000006e-05 2024-09-21T15:49:58.606 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.606 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.606 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:58.606 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504612+0000", 2024-09-21T15:49:58.606 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1799999999999998e-06 2024-09-21T15:49:58.606 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.606 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.606 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:58.606 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.504615+0000", 2024-09-21T15:49:58.606 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0400000000000001e-06 2024-09-21T15:49:58.606 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.606 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.606 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:58.606 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537870+0000", 2024-09-21T15:49:58.606 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033254745000000002 2024-09-21T15:49:58.607 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.607 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.607 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:58.607 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537908+0000", 2024-09-21T15:49:58.607 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7882000000000001e-05 2024-09-21T15:49:58.607 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.607 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.607 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:58.607 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.537955+0000", 2024-09-21T15:49:58.607 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7361999999999998e-05 2024-09-21T15:49:58.607 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:58.607 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:58.607 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:58.607 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.607 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.608 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26239 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 3186688~8192] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:58.608 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.506097+0000", 2024-09-21T15:49:58.608 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.070012304999999, 2024-09-21T15:49:58.608 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032843331000000003, 2024-09-21T15:49:58.608 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:58.608 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:58.608 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:58.608 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:58.608 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:58.608 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26239 2024-09-21T15:49:58.608 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.608 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:58.608 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.608 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:58.608 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506097+0000", 2024-09-21T15:49:58.609 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:58.609 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.609 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.609 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:58.609 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506097+0000", 2024-09-21T15:49:58.609 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:58.609 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.609 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.609 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:58.609 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506097+0000", 2024-09-21T15:49:58.609 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:58.609 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.609 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.609 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:58.609 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506098+0000", 2024-09-21T15:49:58.609 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.079e-06 2024-09-21T15:49:58.610 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.610 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.610 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:58.610 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506101+0000", 2024-09-21T15:49:58.610 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8499999999999998e-06 2024-09-21T15:49:58.610 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.610 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.610 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:58.610 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506103+0000", 2024-09-21T15:49:58.610 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5770000000000001e-06 2024-09-21T15:49:58.610 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.610 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.610 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:58.610 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.538854+0000", 2024-09-21T15:49:58.610 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032750835999999998 2024-09-21T15:49:58.611 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.611 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.611 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:58.611 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.538892+0000", 2024-09-21T15:49:58.611 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7904000000000002e-05 2024-09-21T15:49:58.611 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.611 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.611 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:58.611 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.538940+0000", 2024-09-21T15:49:58.611 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8303999999999997e-05 2024-09-21T15:49:58.611 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:58.611 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:58.611 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:58.611 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.612 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.612 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26240 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 3244032~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:58.612 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.506106+0000", 2024-09-21T15:49:58.612 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.070003173000003, 2024-09-21T15:49:58.612 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032955696, 2024-09-21T15:49:58.612 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:58.612 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:58.612 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:58.612 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:58.612 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:58.612 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26240 2024-09-21T15:49:58.612 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.612 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:58.612 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.612 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:58.613 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506106+0000", 2024-09-21T15:49:58.613 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:58.613 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.613 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.613 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:58.613 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506106+0000", 2024-09-21T15:49:58.613 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:58.613 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.613 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.613 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:58.613 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506106+0000", 2024-09-21T15:49:58.613 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:58.613 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.613 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.613 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:58.614 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506107+0000", 2024-09-21T15:49:58.614 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1179999999999999e-06 2024-09-21T15:49:58.614 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.614 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.614 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:58.614 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506110+0000", 2024-09-21T15:49:58.614 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0259999999999999e-06 2024-09-21T15:49:58.614 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.614 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.614 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:58.614 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506112+0000", 2024-09-21T15:49:58.614 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5890000000000001e-06 2024-09-21T15:49:58.614 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.614 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.615 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:58.615 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.538962+0000", 2024-09-21T15:49:58.615 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032849861000000001 2024-09-21T15:49:58.615 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.615 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.615 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:58.615 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539003+0000", 2024-09-21T15:49:58.615 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0756000000000001e-05 2024-09-21T15:49:58.615 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.615 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.615 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:58.615 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539062+0000", 2024-09-21T15:49:58.615 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8597999999999997e-05 2024-09-21T15:49:58.615 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:58.615 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:58.616 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:58.616 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.616 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.616 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26241 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 3301376~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:58.616 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.506128+0000", 2024-09-21T15:49:58.616 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.069981007000003, 2024-09-21T15:49:58.616 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033043453, 2024-09-21T15:49:58.616 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:58.616 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:58.616 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:58.616 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:58.616 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:58.616 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26241 2024-09-21T15:49:58.616 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.617 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:58.617 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.617 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:58.617 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506128+0000", 2024-09-21T15:49:58.617 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:58.617 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.617 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.617 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:58.617 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506128+0000", 2024-09-21T15:49:58.617 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:58.617 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.617 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.617 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:58.617 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506128+0000", 2024-09-21T15:49:58.617 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:58.618 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.618 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.618 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:58.618 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506129+0000", 2024-09-21T15:49:58.618 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7290000000000001e-06 2024-09-21T15:49:58.618 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.618 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.618 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:58.618 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506132+0000", 2024-09-21T15:49:58.618 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9919999999999999e-06 2024-09-21T15:49:58.618 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.618 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.618 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:58.618 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506135+0000", 2024-09-21T15:49:58.619 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4090000000000001e-06 2024-09-21T15:49:58.619 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.619 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.619 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:58.619 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539084+0000", 2024-09-21T15:49:58.619 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032949231000000002 2024-09-21T15:49:58.619 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.619 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.619 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:58.619 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539124+0000", 2024-09-21T15:49:58.619 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9421000000000001e-05 2024-09-21T15:49:58.619 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.619 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.619 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:58.620 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539172+0000", 2024-09-21T15:49:58.620 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8068000000000001e-05 2024-09-21T15:49:58.620 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:58.620 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:58.620 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:58.620 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.620 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.620 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26242 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 3358720~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:58.620 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.506137+0000", 2024-09-21T15:49:58.620 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.069972073000002, 2024-09-21T15:49:58.620 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033142360000000003, 2024-09-21T15:49:58.620 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:58.620 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:58.620 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:58.621 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:58.621 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:58.621 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26242 2024-09-21T15:49:58.621 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.621 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:58.621 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.621 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:58.621 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506137+0000", 2024-09-21T15:49:58.621 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:58.621 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.621 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.621 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:58.621 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506137+0000", 2024-09-21T15:49:58.621 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:58.621 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.622 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.622 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:58.622 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506137+0000", 2024-09-21T15:49:58.622 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:58.622 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.622 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.622 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:58.622 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506138+0000", 2024-09-21T15:49:58.622 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2330000000000001e-06 2024-09-21T15:49:58.622 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.622 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.622 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:58.622 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506141+0000", 2024-09-21T15:49:58.622 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7999999999999999e-06 2024-09-21T15:49:58.623 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.623 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.623 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:58.623 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506143+0000", 2024-09-21T15:49:58.623 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5919999999999999e-06 2024-09-21T15:49:58.623 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.623 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.623 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:58.623 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539193+0000", 2024-09-21T15:49:58.623 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033049229999999999 2024-09-21T15:49:58.623 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.623 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.623 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:58.624 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539232+0000", 2024-09-21T15:49:58.624 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9743999999999998e-05 2024-09-21T15:49:58.624 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.624 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.624 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:58.624 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539279+0000", 2024-09-21T15:49:58.624 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7039999999999997e-05 2024-09-21T15:49:58.624 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:58.624 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:58.624 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:58.624 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.624 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.624 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26243 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 3416064~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:58.624 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.506145+0000", 2024-09-21T15:49:58.624 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.069963788000003, 2024-09-21T15:49:58.625 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033240933, 2024-09-21T15:49:58.625 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:58.625 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:58.625 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:58.625 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:58.625 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:58.625 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26243 2024-09-21T15:49:58.625 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.625 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:58.625 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.625 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:58.625 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506145+0000", 2024-09-21T15:49:58.625 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:58.625 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.625 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.626 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:58.626 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506145+0000", 2024-09-21T15:49:58.626 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:58.626 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.626 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.626 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:58.626 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506145+0000", 2024-09-21T15:49:58.626 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:58.626 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.626 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.626 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:58.626 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506146+0000", 2024-09-21T15:49:58.626 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0279999999999999e-06 2024-09-21T15:49:58.626 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.627 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.627 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:58.627 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506149+0000", 2024-09-21T15:49:58.627 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2449999999999998e-06 2024-09-21T15:49:58.627 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.627 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.627 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:58.627 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.506152+0000", 2024-09-21T15:49:58.627 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5050000000000002e-06 2024-09-21T15:49:58.627 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.627 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.627 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:58.627 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539301+0000", 2024-09-21T15:49:58.627 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033149093999999997 2024-09-21T15:49:58.627 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.628 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.628 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:58.628 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539339+0000", 2024-09-21T15:49:58.628 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7685999999999998e-05 2024-09-21T15:49:58.628 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.628 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.628 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:58.628 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.539386+0000", 2024-09-21T15:49:58.628 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7602000000000003e-05 2024-09-21T15:49:58.628 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:58.628 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:58.628 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:58.628 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.628 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.628 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26258 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 4276224~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:58.629 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.508167+0000", 2024-09-21T15:49:58.629 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.067941869999999, 2024-09-21T15:49:58.629 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032934857999999997, 2024-09-21T15:49:58.629 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:58.629 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:58.629 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:58.629 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:58.629 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:58.629 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26258 2024-09-21T15:49:58.629 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.629 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:58.629 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.629 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:58.629 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508167+0000", 2024-09-21T15:49:58.630 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:58.630 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.630 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.630 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:58.630 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508167+0000", 2024-09-21T15:49:58.630 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:58.630 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.630 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.630 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:58.630 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508167+0000", 2024-09-21T15:49:58.630 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:58.630 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.630 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.630 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:58.630 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508168+0000", 2024-09-21T15:49:58.631 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.088e-06 2024-09-21T15:49:58.631 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.631 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.631 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:58.631 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508171+0000", 2024-09-21T15:49:58.631 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7080000000000002e-06 2024-09-21T15:49:58.631 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.631 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.631 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:58.631 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508173+0000", 2024-09-21T15:49:58.631 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4030000000000001e-06 2024-09-21T15:49:58.631 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.631 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.631 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:58.632 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.540936+0000", 2024-09-21T15:49:58.632 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032762279999999998 2024-09-21T15:49:58.632 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.632 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.632 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:58.632 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.541007+0000", 2024-09-21T15:49:58.632 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1335999999999996e-05 2024-09-21T15:49:58.632 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.632 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.632 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:58.632 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.541102+0000", 2024-09-21T15:49:58.632 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5259000000000004e-05 2024-09-21T15:49:58.632 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:58.633 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:58.633 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:58.633 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.633 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.633 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26259 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 4333568~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:58.633 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.508175+0000", 2024-09-21T15:49:58.633 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.067933928000002, 2024-09-21T15:49:58.633 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033044845000000003, 2024-09-21T15:49:58.633 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:58.633 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:58.633 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:58.633 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:58.633 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:58.633 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26259 2024-09-21T15:49:58.634 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.634 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:58.634 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.634 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:58.634 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508175+0000", 2024-09-21T15:49:58.634 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:58.634 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.634 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.634 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:58.634 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508175+0000", 2024-09-21T15:49:58.634 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:58.634 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.634 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.634 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:58.635 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508175+0000", 2024-09-21T15:49:58.635 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:58.635 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.635 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.635 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:58.635 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508176+0000", 2024-09-21T15:49:58.635 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1039999999999999e-06 2024-09-21T15:49:58.635 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.635 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.635 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:58.635 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508179+0000", 2024-09-21T15:49:58.635 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.689e-06 2024-09-21T15:49:58.635 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.635 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.636 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:58.636 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508181+0000", 2024-09-21T15:49:58.636 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5610000000000001e-06 2024-09-21T15:49:58.636 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.636 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.636 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:58.636 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.541126+0000", 2024-09-21T15:49:58.636 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032944284999999997 2024-09-21T15:49:58.636 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.636 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.636 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:58.636 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.541168+0000", 2024-09-21T15:49:58.636 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2565000000000003e-05 2024-09-21T15:49:58.636 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.637 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.637 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:58.637 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.541220+0000", 2024-09-21T15:49:58.637 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1863000000000001e-05 2024-09-21T15:49:58.637 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:58.637 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:58.637 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:58.637 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.637 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.637 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26260 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 4390912~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:58.637 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.508183+0000", 2024-09-21T15:49:58.637 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.067925813000002, 2024-09-21T15:49:58.637 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033150304999999998, 2024-09-21T15:49:58.637 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:58.637 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:58.638 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:58.638 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:58.638 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:58.638 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26260 2024-09-21T15:49:58.638 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.638 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:58.638 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.638 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:58.638 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508183+0000", 2024-09-21T15:49:58.638 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:58.638 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.638 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.638 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:58.639 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508183+0000", 2024-09-21T15:49:58.639 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:58.639 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.639 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.639 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:58.639 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508183+0000", 2024-09-21T15:49:58.639 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:58.639 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.639 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.639 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:58.639 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508184+0000", 2024-09-21T15:49:58.639 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0580000000000001e-06 2024-09-21T15:49:58.639 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.639 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.639 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:58.640 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508187+0000", 2024-09-21T15:49:58.640 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7209999999999999e-06 2024-09-21T15:49:58.640 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.640 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.640 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:58.640 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.508189+0000", 2024-09-21T15:49:58.640 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5239999999999999e-06 2024-09-21T15:49:58.640 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.640 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.640 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:58.640 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.541243+0000", 2024-09-21T15:49:58.640 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033053658 2024-09-21T15:49:58.640 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.640 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.640 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:58.641 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.541284+0000", 2024-09-21T15:49:58.641 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0834000000000002e-05 2024-09-21T15:49:58.641 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.641 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.641 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:58.641 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.541334+0000", 2024-09-21T15:49:58.641 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9724e-05 2024-09-21T15:49:58.641 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:58.641 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:58.641 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:58.641 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.641 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.641 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26275 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 5251072~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:58.641 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.510151+0000", 2024-09-21T15:49:58.641 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.065958033999998, 2024-09-21T15:49:58.642 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032836153999999999, 2024-09-21T15:49:58.642 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:58.642 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:58.642 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:58.642 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:58.642 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:58.642 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26275 2024-09-21T15:49:58.642 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.642 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:58.642 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.642 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:58.642 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510151+0000", 2024-09-21T15:49:58.642 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:58.642 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.643 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.643 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:58.643 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510151+0000", 2024-09-21T15:49:58.643 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:58.643 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.643 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.643 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:58.643 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510151+0000", 2024-09-21T15:49:58.643 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:58.643 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.643 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.643 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:58.643 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510152+0000", 2024-09-21T15:49:58.643 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1990000000000001e-06 2024-09-21T15:49:58.643 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.644 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.644 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:58.644 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510155+0000", 2024-09-21T15:49:58.644 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7089999999999998e-06 2024-09-21T15:49:58.644 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.644 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.644 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:58.644 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510157+0000", 2024-09-21T15:49:58.644 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5189999999999999e-06 2024-09-21T15:49:58.644 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.644 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.644 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:58.644 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.542902+0000", 2024-09-21T15:49:58.644 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032744599999999999 2024-09-21T15:49:58.644 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.645 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.645 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:58.645 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.542941+0000", 2024-09-21T15:49:58.645 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8761999999999999e-05 2024-09-21T15:49:58.645 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.645 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.645 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:58.645 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.542987+0000", 2024-09-21T15:49:58.645 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6665999999999997e-05 2024-09-21T15:49:58.645 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:58.645 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:58.645 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:58.645 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.645 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.646 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26276 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 5308416~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:58.646 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.510159+0000", 2024-09-21T15:49:58.646 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.06595007, 2024-09-21T15:49:58.646 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032943014999999999, 2024-09-21T15:49:58.646 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:58.646 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:58.646 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:58.646 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:58.646 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:58.646 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26276 2024-09-21T15:49:58.646 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.646 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:58.646 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.646 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:58.647 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510159+0000", 2024-09-21T15:49:58.647 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:58.647 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.647 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.647 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:58.647 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510159+0000", 2024-09-21T15:49:58.647 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:58.647 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.647 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.647 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:58.647 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510159+0000", 2024-09-21T15:49:58.647 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:58.647 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.647 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.647 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:58.648 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510160+0000", 2024-09-21T15:49:58.648 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.13e-06 2024-09-21T15:49:58.648 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.648 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.648 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:58.648 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510163+0000", 2024-09-21T15:49:58.648 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7489999999999999e-06 2024-09-21T15:49:58.648 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.648 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.648 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:58.648 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510165+0000", 2024-09-21T15:49:58.648 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4200000000000001e-06 2024-09-21T15:49:58.648 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.648 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.648 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:58.648 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.543013+0000", 2024-09-21T15:49:58.649 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032847736000000002 2024-09-21T15:49:58.649 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.649 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.649 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:58.649 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.543054+0000", 2024-09-21T15:49:58.649 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0725000000000001e-05 2024-09-21T15:49:58.649 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.649 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.649 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:58.649 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.543102+0000", 2024-09-21T15:49:58.649 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8471999999999999e-05 2024-09-21T15:49:58.649 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:58.649 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:58.650 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:58.650 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.650 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.650 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26277 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 5365760~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:58.650 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.510167+0000", 2024-09-21T15:49:58.650 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.065941938999998, 2024-09-21T15:49:58.650 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.033113111000000001, 2024-09-21T15:49:58.650 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:58.650 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:58.650 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:58.650 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:58.650 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:58.650 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26277 2024-09-21T15:49:58.650 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.651 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:58.651 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.651 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:58.651 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510167+0000", 2024-09-21T15:49:58.651 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:58.651 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.651 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.651 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:58.651 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510167+0000", 2024-09-21T15:49:58.651 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:58.651 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.651 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.651 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:58.651 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510167+0000", 2024-09-21T15:49:58.652 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:58.652 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.652 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.652 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:58.652 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510168+0000", 2024-09-21T15:49:58.652 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.6899999999999996e-07 2024-09-21T15:49:58.652 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.652 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.652 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:58.652 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510171+0000", 2024-09-21T15:49:58.652 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7080000000000002e-06 2024-09-21T15:49:58.652 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.652 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.652 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:58.652 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.510173+0000", 2024-09-21T15:49:58.653 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5050000000000002e-06 2024-09-21T15:49:58.653 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.653 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.653 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:58.653 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.543125+0000", 2024-09-21T15:49:58.653 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032951504 2024-09-21T15:49:58.653 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.653 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.653 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:58.653 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.543192+0000", 2024-09-21T15:49:58.653 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6938000000000002e-05 2024-09-21T15:49:58.653 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.653 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.653 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:58.654 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.543280+0000", 2024-09-21T15:49:58.654 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8702999999999994e-05 2024-09-21T15:49:58.654 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:58.654 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:58.654 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:58.654 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.654 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.654 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26292 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 6225920~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:58.654 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.512160+0000", 2024-09-21T15:49:58.654 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.063949125000001, 2024-09-21T15:49:58.654 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032838748000000001, 2024-09-21T15:49:58.654 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:58.654 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:58.654 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:58.654 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:58.655 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:58.655 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26292 2024-09-21T15:49:58.655 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.655 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:58.655 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.655 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:58.655 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512160+0000", 2024-09-21T15:49:58.655 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:58.655 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.655 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.655 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:58.655 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512160+0000", 2024-09-21T15:49:58.655 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:58.655 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.656 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.656 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:58.656 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512160+0000", 2024-09-21T15:49:58.656 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:58.656 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.656 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.656 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:58.656 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512161+0000", 2024-09-21T15:49:58.656 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.158e-06 2024-09-21T15:49:58.656 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.656 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.656 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:58.656 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512164+0000", 2024-09-21T15:49:58.656 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7429999999999998e-06 2024-09-21T15:49:58.656 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.657 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.657 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:58.657 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512166+0000", 2024-09-21T15:49:58.657 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5280000000000002e-06 2024-09-21T15:49:58.657 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.657 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.657 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:58.657 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.544912+0000", 2024-09-21T15:49:58.657 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032745962000000003 2024-09-21T15:49:58.657 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.657 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.657 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:58.657 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.544950+0000", 2024-09-21T15:49:58.657 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7657000000000002e-05 2024-09-21T15:49:58.657 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.658 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.658 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:58.658 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.544999+0000", 2024-09-21T15:49:58.658 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8912999999999997e-05 2024-09-21T15:49:58.658 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:58.658 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:58.658 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:58.658 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.658 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.658 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:26293 3.1a 3:5d1d4fcd:::rbd_data.11331466bd23.0000000000000001:9 [read 6283264~8192] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e38)", 2024-09-21T15:49:58.658 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:26.512169+0000", 2024-09-21T15:49:58.658 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.063939976, 2024-09-21T15:49:58.658 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032937552000000002, 2024-09-21T15:49:58.658 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:58.659 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:58.659 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:58.659 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:58.659 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:58.659 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 26293 2024-09-21T15:49:58.659 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.659 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:58.659 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.659 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:58.659 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512169+0000", 2024-09-21T15:49:58.659 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:58.659 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.659 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.660 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:58.660 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512169+0000", 2024-09-21T15:49:58.660 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:58.660 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.660 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.660 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:58.660 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512169+0000", 2024-09-21T15:49:58.660 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:58.660 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.660 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.660 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:58.660 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512170+0000", 2024-09-21T15:49:58.660 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1650000000000001e-06 2024-09-21T15:49:58.660 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.660 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.661 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:58.661 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512173+0000", 2024-09-21T15:49:58.661 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.773e-06 2024-09-21T15:49:58.661 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.661 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.661 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:58.661 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.512175+0000", 2024-09-21T15:49:58.661 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4700000000000001e-06 2024-09-21T15:49:58.661 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.661 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.661 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:58.661 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.545018+0000", 2024-09-21T15:49:58.661 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.032843043000000002 2024-09-21T15:49:58.661 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.662 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.662 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:58.662 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.545057+0000", 2024-09-21T15:49:58.662 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8862999999999997e-05 2024-09-21T15:49:58.662 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:58.662 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:58.662 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:58.662 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:26.545107+0000", 2024-09-21T15:49:58.662 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9455e-05 2024-09-21T15:49:58.662 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:58.662 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:58.662 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:58.662 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:58.662 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:58.663 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:58.664 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:49:58.664 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:58.662+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:58.765 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:49:58.763+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:49:58.810 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:58.811 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:58.811 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:49:58.811 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:58.816 INFO:teuthology.orchestra.run.smithi089.stdout:checking clone #24, image image_client.0-clone24 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent25 2024-09-21T15:49:58.820 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:49:58.825 INFO:teuthology.orchestra.run.smithi089.stdout:4061 trunc from 0xec033be to 0xaa10a09 2024-09-21T15:49:58.848 INFO:teuthology.orchestra.run.smithi089.stdout:4062 write 0x676eee5 thru 0x677a103 (0xb21f bytes) 2024-09-21T15:49:58.857 INFO:teuthology.orchestra.run.smithi089.stdout:4065 read 0x9fe7f8e thru 0x9ff57e0 (0xd853 bytes) 2024-09-21T15:49:58.857 INFO:teuthology.orchestra.run.smithi089.stdout:4067 read 0x1b70b2e thru 0x1b746f6 (0x3bc9 bytes) 2024-09-21T15:49:58.857 INFO:teuthology.orchestra.run.smithi089.stdout:4069 write 0x12f6bc4 thru 0x1305126 (0xe563 bytes) 2024-09-21T15:49:58.864 INFO:teuthology.orchestra.run.smithi089.stdout:4071 write 0x5a491fa thru 0x5a562a6 (0xd0ad bytes) 2024-09-21T15:49:58.867 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:58.867 INFO:teuthology.orchestra.run.smithi089.stdout:4073 read 0x253b8cb thru 0x254aa7e (0xf1b4 bytes) 2024-09-21T15:49:58.867 INFO:teuthology.orchestra.run.smithi089.stdout:4076 read 0xa143d4b thru 0xa148eae (0x5164 bytes) 2024-09-21T15:49:58.868 INFO:teuthology.orchestra.run.smithi089.stdout:4077 write 0x1c213bc thru 0x1c30e8e (0xfad3 bytes) 2024-09-21T15:49:58.879 INFO:teuthology.orchestra.run.smithi089.stdout:4078 write 0xd22ded5 thru 0xd238dd9 (0xaf05 bytes) 2024-09-21T15:49:58.884 INFO:teuthology.orchestra.run.smithi089.stdout:4079 read 0x892c488 thru 0x8938d0d (0xc886 bytes) 2024-09-21T15:49:58.884 INFO:teuthology.orchestra.run.smithi089.stdout:4080 read 0x6a11f5c thru 0x6a14e2b (0x2ed0 bytes) 2024-09-21T15:49:58.884 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:58.884 INFO:teuthology.orchestra.run.smithi089.stdout:4083 writesame 0x157a9f1 thru 0x157c7fe data_size 0x1(0x1e0e bytes) 2024-09-21T15:49:58.891 INFO:teuthology.orchestra.run.smithi089.stdout:4084 write 0xb50bd90 thru 0xb51b446 (0xf6b7 bytes) 2024-09-21T15:49:58.893 INFO:teuthology.orchestra.run.smithi089.stdout:4085 writesame 0x3a0022d thru 0x3a0bfcb data_size 0xb(0xbd9f bytes) 2024-09-21T15:49:58.904 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:58.903+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:58.908 INFO:teuthology.orchestra.run.smithi089.stdout:4086 trunc from 0xd238dda to 0x6233da8 2024-09-21T15:49:58.926 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:49:58.926 INFO:teuthology.orchestra.run.smithi089.stdout:4088 read 0x104a60f thru 0x104cba8 (0x259a bytes) 2024-09-21T15:49:58.926 INFO:teuthology.orchestra.run.smithi089.stdout:4091 read 0x57768e8 thru 0x577a2ac (0x39c5 bytes) 2024-09-21T15:49:58.926 INFO:teuthology.orchestra.run.smithi089.stdout:4092 punch from 0x4dfa391 to 0x4dfcd08, (0x2977 bytes) 2024-09-21T15:49:58.926 INFO:teuthology.orchestra.run.smithi089.stdout:4094 flatten 2024-09-21T15:49:58.961 INFO:teuthology.orchestra.run.smithi089.stdout:4095 write 0xccf923b thru 0xccf9c1e (0x9e4 bytes) 2024-09-21T15:49:58.967 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:58.965+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:58.971 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:58.971 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:58.971 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:49:58.971 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:49:58.971 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:58.974 INFO:teuthology.orchestra.run.smithi089.stdout:4097 trunc from 0xccf9c1f to 0x4a2328d 2024-09-21T15:49:58.980 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:49:59.068 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:59.066+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:59.123 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:59.123 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:49:59.123 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:49:59.124 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:49:59.124 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.124 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:65402 3.1d 3:bda7fcfc:::rbd_data.11336b2fdf4a.0000000000000000:c [read 4915200~0] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:59.124 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.279722+0000", 2024-09-21T15:49:59.124 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 21.842221316, 2024-09-21T15:49:59.124 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045558766000000001, 2024-09-21T15:49:59.124 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:59.124 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:59.124 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:59.124 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:59.124 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:59.124 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 65402 2024-09-21T15:49:59.124 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.124 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:59.125 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.125 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:59.125 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279722+0000", 2024-09-21T15:49:59.125 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.125 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.125 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.125 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:59.125 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279722+0000", 2024-09-21T15:49:59.125 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.125 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.125 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.125 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:59.125 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279722+0000", 2024-09-21T15:49:59.125 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:59.126 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.126 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.126 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:59.126 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279724+0000", 2024-09-21T15:49:59.126 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3929999999999998e-06 2024-09-21T15:49:59.126 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.126 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.126 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:59.126 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279731+0000", 2024-09-21T15:49:59.126 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3269999999999997e-06 2024-09-21T15:49:59.126 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.126 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.126 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:59.127 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279736+0000", 2024-09-21T15:49:59.127 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.113e-06 2024-09-21T15:49:59.127 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.127 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.127 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:59.127 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.325136+0000", 2024-09-21T15:49:59.127 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045399912000000001 2024-09-21T15:49:59.127 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.127 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.127 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:59.127 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.325213+0000", 2024-09-21T15:49:59.127 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7546999999999995e-05 2024-09-21T15:49:59.127 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.127 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.128 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:59.128 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.325281+0000", 2024-09-21T15:49:59.128 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7931999999999999e-05 2024-09-21T15:49:59.128 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.128 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:59.128 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.128 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.128 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.128 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:65403 3.1d 3:bda7fcfc:::rbd_data.11336b2fdf4a.0000000000000000:c [read 4923392~0] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:59.128 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.279740+0000", 2024-09-21T15:49:59.128 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 21.842203403999999, 2024-09-21T15:49:59.128 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045722756000000003, 2024-09-21T15:49:59.128 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:59.128 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:59.128 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:59.129 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:59.129 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:59.129 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 65403 2024-09-21T15:49:59.129 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.129 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:59.129 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.129 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:59.129 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279740+0000", 2024-09-21T15:49:59.129 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.129 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.129 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.129 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:59.129 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279740+0000", 2024-09-21T15:49:59.130 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.130 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.130 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.130 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:59.130 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279740+0000", 2024-09-21T15:49:59.130 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:59.130 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.130 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.130 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:59.130 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279742+0000", 2024-09-21T15:49:59.130 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1849999999999998e-06 2024-09-21T15:49:59.130 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.130 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.130 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:59.131 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279749+0000", 2024-09-21T15:49:59.131 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2450000000000001e-06 2024-09-21T15:49:59.131 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.131 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.131 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:59.131 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279754+0000", 2024-09-21T15:49:59.131 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8940000000000001e-06 2024-09-21T15:49:59.131 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.131 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.131 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:59.131 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.325316+0000", 2024-09-21T15:49:59.131 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045561938000000003 2024-09-21T15:49:59.131 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.131 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.132 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:59.132 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.325399+0000", 2024-09-21T15:49:59.132 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2647999999999999e-05 2024-09-21T15:49:59.132 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.132 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.132 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:59.132 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.325463+0000", 2024-09-21T15:49:59.132 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4247999999999999e-05 2024-09-21T15:49:59.132 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.132 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:59.132 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.132 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.132 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.132 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:65404 3.1d 3:bda7fcfc:::rbd_data.11336b2fdf4a.0000000000000000:c [read 4931584~0] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:59.132 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.279757+0000", 2024-09-21T15:49:59.133 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 21.842186573999999, 2024-09-21T15:49:59.133 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045883348999999997, 2024-09-21T15:49:59.133 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:59.133 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:59.133 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:59.133 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:59.133 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:59.133 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 65404 2024-09-21T15:49:59.133 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.133 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:59.133 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.133 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:59.133 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279757+0000", 2024-09-21T15:49:59.133 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.133 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.133 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.134 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:59.134 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279757+0000", 2024-09-21T15:49:59.134 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.134 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.134 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.134 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:59.134 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279757+0000", 2024-09-21T15:49:59.134 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:59.134 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.134 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.134 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:59.134 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279759+0000", 2024-09-21T15:49:59.134 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.64e-06 2024-09-21T15:49:59.134 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.134 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.135 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:59.135 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279764+0000", 2024-09-21T15:49:59.135 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0030000000000002e-06 2024-09-21T15:49:59.135 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.135 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.135 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:59.137 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279766+0000", 2024-09-21T15:49:59.137 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6620000000000001e-06 2024-09-21T15:49:59.137 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.138 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.138 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:59.138 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.325489+0000", 2024-09-21T15:49:59.138 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045722721000000001 2024-09-21T15:49:59.138 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.138 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.138 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:59.138 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.325574+0000", 2024-09-21T15:49:59.138 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5109000000000001e-05 2024-09-21T15:49:59.138 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.138 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.138 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:59.138 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.325641+0000", 2024-09-21T15:49:59.138 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6475000000000003e-05 2024-09-21T15:49:59.139 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.139 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:59.139 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.139 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.139 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.139 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:65405 3.1d 3:bda7fcfc:::rbd_data.11336b2fdf4a.0000000000000000:c [read 4939776~0] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:59.139 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.279768+0000", 2024-09-21T15:49:59.139 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 21.842175310999998, 2024-09-21T15:49:59.139 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046038737000000003, 2024-09-21T15:49:59.139 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:59.139 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:59.139 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:59.139 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:59.139 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:59.140 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 65405 2024-09-21T15:49:59.140 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.140 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:59.140 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.140 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:59.140 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279768+0000", 2024-09-21T15:49:59.140 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.140 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.140 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.140 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:59.140 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279768+0000", 2024-09-21T15:49:59.140 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.140 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.140 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.141 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:59.141 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279768+0000", 2024-09-21T15:49:59.141 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:59.141 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.141 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.141 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:59.141 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279769+0000", 2024-09-21T15:49:59.141 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.22e-06 2024-09-21T15:49:59.141 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.141 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.141 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:59.141 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279772+0000", 2024-09-21T15:49:59.141 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9349999999999999e-06 2024-09-21T15:49:59.141 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.141 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.141 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:59.142 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279775+0000", 2024-09-21T15:49:59.142 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4959999999999999e-06 2024-09-21T15:49:59.142 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.142 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.142 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:59.142 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.325668+0000", 2024-09-21T15:49:59.142 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045892779000000002 2024-09-21T15:49:59.142 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.142 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.142 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:59.142 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.325742+0000", 2024-09-21T15:49:59.142 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4092999999999997e-05 2024-09-21T15:49:59.142 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.143 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.143 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:59.143 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.325807+0000", 2024-09-21T15:49:59.143 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5419999999999994e-05 2024-09-21T15:49:59.143 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.143 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:59.143 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.143 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.143 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.143 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:65406 3.1d 3:bda7fcfc:::rbd_data.11336b2fdf4a.0000000000000000:c [read 4947968~0] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:59.143 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.279777+0000", 2024-09-21T15:49:59.143 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 21.842166752000001, 2024-09-21T15:49:59.143 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046198388999999999, 2024-09-21T15:49:59.143 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:59.143 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:59.144 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:59.144 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:59.144 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:59.144 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 65406 2024-09-21T15:49:59.144 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.144 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:59.144 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.144 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:59.144 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279777+0000", 2024-09-21T15:49:59.144 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.144 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.144 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.144 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:59.144 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279777+0000", 2024-09-21T15:49:59.145 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.145 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.145 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.145 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:59.145 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279777+0000", 2024-09-21T15:49:59.145 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:59.145 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.145 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.145 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:59.145 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279778+0000", 2024-09-21T15:49:59.145 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.238e-06 2024-09-21T15:49:59.145 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.145 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.145 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:59.145 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279781+0000", 2024-09-21T15:49:59.145 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8679999999999999e-06 2024-09-21T15:49:59.146 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.146 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.146 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:59.146 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279783+0000", 2024-09-21T15:49:59.146 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4729999999999999e-06 2024-09-21T15:49:59.146 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.146 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.146 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:59.146 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.325834+0000", 2024-09-21T15:49:59.146 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046050328000000001 2024-09-21T15:49:59.146 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.146 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.146 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:59.147 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.325909+0000", 2024-09-21T15:49:59.147 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5463999999999994e-05 2024-09-21T15:49:59.147 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.147 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.147 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:59.147 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.325975+0000", 2024-09-21T15:49:59.147 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6229999999999994e-05 2024-09-21T15:49:59.147 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.147 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:59.147 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.147 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.147 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.147 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:65407 3.1d 3:bda7fcfc:::rbd_data.11336b2fdf4a.0000000000000000:c [read 4956160~0] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:59.148 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.279786+0000", 2024-09-21T15:49:59.148 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 21.842158198, 2024-09-21T15:49:59.148 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046366821000000003, 2024-09-21T15:49:59.148 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:59.148 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:59.148 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:59.148 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:59.148 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:59.148 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 65407 2024-09-21T15:49:59.148 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.148 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:59.148 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.148 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:59.148 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279786+0000", 2024-09-21T15:49:59.148 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.149 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.149 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.149 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:59.149 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279786+0000", 2024-09-21T15:49:59.149 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.149 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.149 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.149 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:59.149 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279785+0000", 2024-09-21T15:49:59.149 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:59.149 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.149 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.149 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:59.149 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279786+0000", 2024-09-21T15:49:59.149 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1379999999999999e-06 2024-09-21T15:49:59.150 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.150 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.150 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:59.150 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279789+0000", 2024-09-21T15:49:59.150 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8629999999999999e-06 2024-09-21T15:49:59.150 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.150 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.150 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:59.150 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279792+0000", 2024-09-21T15:49:59.150 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.469e-06 2024-09-21T15:49:59.150 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.150 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.150 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:59.151 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.326002+0000", 2024-09-21T15:49:59.151 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046210481999999997 2024-09-21T15:49:59.151 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.151 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.151 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:59.151 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.326084+0000", 2024-09-21T15:49:59.151 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1570999999999995e-05 2024-09-21T15:49:59.151 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.151 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.151 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:59.151 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.326152+0000", 2024-09-21T15:49:59.151 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8523999999999996e-05 2024-09-21T15:49:59.151 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.151 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:59.151 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.152 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.152 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.152 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:65408 3.1d 3:bda7fcfc:::rbd_data.11336b2fdf4a.0000000000000000:c [read 4964352~0] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:59.152 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.279794+0000", 2024-09-21T15:49:59.152 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 21.842149896999999, 2024-09-21T15:49:59.152 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046551309999999999, 2024-09-21T15:49:59.152 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:59.152 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:59.152 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:59.152 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:59.152 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:59.152 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 65408 2024-09-21T15:49:59.152 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.152 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:59.152 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.153 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:59.153 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279794+0000", 2024-09-21T15:49:59.153 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.153 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.153 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.153 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:59.153 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279794+0000", 2024-09-21T15:49:59.153 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.153 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.153 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.153 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:59.153 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279794+0000", 2024-09-21T15:49:59.153 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:59.153 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.154 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.154 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:59.154 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279795+0000", 2024-09-21T15:49:59.154 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.105e-06 2024-09-21T15:49:59.154 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.154 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.154 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:59.154 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279797+0000", 2024-09-21T15:49:59.154 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7590000000000002e-06 2024-09-21T15:49:59.154 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.154 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.154 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:59.154 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279800+0000", 2024-09-21T15:49:59.155 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4030000000000001e-06 2024-09-21T15:49:59.155 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.155 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.155 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:59.155 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.326193+0000", 2024-09-21T15:49:59.155 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046392845000000002 2024-09-21T15:49:59.155 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.155 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.155 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:59.155 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.326272+0000", 2024-09-21T15:49:59.155 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8788999999999994e-05 2024-09-21T15:49:59.155 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.155 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.155 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:59.155 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.326345+0000", 2024-09-21T15:49:59.155 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3629999999999998e-05 2024-09-21T15:49:59.156 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.156 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:59.156 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.156 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.156 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.156 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:65409 3.1d 3:bda7fcfc:::rbd_data.11336b2fdf4a.0000000000000000:c [read 4972544~0] snapc 0=[] RETRY=4 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:59.156 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.279802+0000", 2024-09-21T15:49:59.156 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 21.842141606999999, 2024-09-21T15:49:59.156 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046723775000000002, 2024-09-21T15:49:59.156 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:59.156 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:59.156 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:59.156 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:59.156 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:59.157 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 65409 2024-09-21T15:49:59.157 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.157 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:59.157 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.157 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:59.157 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279802+0000", 2024-09-21T15:49:59.157 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.157 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.157 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.157 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:59.157 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279802+0000", 2024-09-21T15:49:59.157 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.157 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.157 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.157 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:59.158 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279802+0000", 2024-09-21T15:49:59.158 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:49:59.158 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.158 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.158 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:59.158 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279803+0000", 2024-09-21T15:49:59.158 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.155e-06 2024-09-21T15:49:59.158 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.158 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.158 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:59.158 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279806+0000", 2024-09-21T15:49:59.158 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7949999999999999e-06 2024-09-21T15:49:59.158 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.158 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.159 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:59.159 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.279808+0000", 2024-09-21T15:49:59.159 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.491e-06 2024-09-21T15:49:59.159 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.159 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.159 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:59.159 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.326375+0000", 2024-09-21T15:49:59.159 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046566824999999999 2024-09-21T15:49:59.159 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.159 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.159 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:59.159 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.326455+0000", 2024-09-21T15:49:59.159 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9680999999999998e-05 2024-09-21T15:49:59.159 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.160 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.160 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:59.160 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.326526+0000", 2024-09-21T15:49:59.160 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1057999999999996e-05 2024-09-21T15:49:59.160 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.160 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:59.160 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.160 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.160 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.160 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:70737 3.c 3:32ea75c5:::rbd_data.11336b2fdf4a.0000000000000001:c [read 1021952~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:59.160 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.495543+0000", 2024-09-21T15:49:59.160 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 21.626400418999999, 2024-09-21T15:49:59.160 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045418501, 2024-09-21T15:49:59.160 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:59.160 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:59.161 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:59.161 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:59.161 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:59.161 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 70737 2024-09-21T15:49:59.161 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.161 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:59.161 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.161 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:59.161 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495543+0000", 2024-09-21T15:49:59.161 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.161 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.161 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.161 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:59.161 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495543+0000", 2024-09-21T15:49:59.161 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.162 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.162 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.162 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:59.162 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495543+0000", 2024-09-21T15:49:59.162 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:59.162 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.162 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.162 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:59.162 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495545+0000", 2024-09-21T15:49:59.162 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5380000000000001e-06 2024-09-21T15:49:59.162 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.162 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.162 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:59.162 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495552+0000", 2024-09-21T15:49:59.163 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4740000000000002e-06 2024-09-21T15:49:59.163 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.163 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.163 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:59.163 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495557+0000", 2024-09-21T15:49:59.163 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.792e-06 2024-09-21T15:49:59.163 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.163 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.163 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:59.163 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.540761+0000", 2024-09-21T15:49:59.163 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045203967999999997 2024-09-21T15:49:59.163 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.163 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.163 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:59.164 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.540871+0000", 2024-09-21T15:49:59.164 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00011034799999999999 2024-09-21T15:49:59.164 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.164 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.164 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:59.164 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.540962+0000", 2024-09-21T15:49:59.164 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0931000000000001e-05 2024-09-21T15:49:59.164 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.164 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:59.164 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.164 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.164 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.164 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:70738 3.c 3:32ea75c5:::rbd_data.11336b2fdf4a.0000000000000001:c [read 1030144~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:59.164 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.495576+0000", 2024-09-21T15:49:59.164 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 21.626368051, 2024-09-21T15:49:59.165 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045596921999999998, 2024-09-21T15:49:59.165 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:59.165 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:59.165 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:59.165 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:59.165 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:59.165 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 70738 2024-09-21T15:49:59.165 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.165 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:59.165 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.165 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:59.165 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495576+0000", 2024-09-21T15:49:59.165 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.165 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.165 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.166 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:59.166 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495576+0000", 2024-09-21T15:49:59.166 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.166 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.166 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.166 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:59.166 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495575+0000", 2024-09-21T15:49:59.166 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:59.166 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.166 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.166 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:59.166 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495578+0000", 2024-09-21T15:49:59.166 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4779999999999998e-06 2024-09-21T15:49:59.166 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.167 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.167 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:59.167 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495584+0000", 2024-09-21T15:49:59.167 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1149999999999999e-06 2024-09-21T15:49:59.167 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.167 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.167 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:59.167 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495589+0000", 2024-09-21T15:49:59.167 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8300000000000003e-06 2024-09-21T15:49:59.167 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.167 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.167 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:59.167 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.540995+0000", 2024-09-21T15:49:59.167 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045406390999999997 2024-09-21T15:49:59.167 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.168 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.168 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:59.168 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.541083+0000", 2024-09-21T15:49:59.168 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7579999999999998e-05 2024-09-21T15:49:59.168 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.168 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.168 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:59.168 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.541173+0000", 2024-09-21T15:49:59.168 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0017000000000002e-05 2024-09-21T15:49:59.168 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.168 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:59.168 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.168 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.168 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.169 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:70739 3.c 3:32ea75c5:::rbd_data.11336b2fdf4a.0000000000000001:c [read 1038336~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:59.169 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.495608+0000", 2024-09-21T15:49:59.169 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 21.626335992000001, 2024-09-21T15:49:59.169 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045757673999999998, 2024-09-21T15:49:59.169 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:59.169 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:59.169 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:59.169 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:59.169 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:59.169 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 70739 2024-09-21T15:49:59.169 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.169 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:59.169 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.169 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:59.170 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495608+0000", 2024-09-21T15:49:59.170 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.170 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.170 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.170 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:59.170 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495608+0000", 2024-09-21T15:49:59.170 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.170 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.170 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.170 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:59.170 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495607+0000", 2024-09-21T15:49:59.170 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:59.170 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.170 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.170 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:59.171 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495610+0000", 2024-09-21T15:49:59.171 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5610000000000001e-06 2024-09-21T15:49:59.171 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.171 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.171 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:59.171 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495617+0000", 2024-09-21T15:49:59.171 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8290000000000002e-06 2024-09-21T15:49:59.171 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.171 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.171 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:59.171 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495622+0000", 2024-09-21T15:49:59.171 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0730000000000004e-06 2024-09-21T15:49:59.171 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.171 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.172 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:59.172 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.541203+0000", 2024-09-21T15:49:59.172 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045581165 2024-09-21T15:49:59.172 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.172 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.172 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:59.172 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.541279+0000", 2024-09-21T15:49:59.172 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6637000000000005e-05 2024-09-21T15:49:59.172 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.172 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.172 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:59.172 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.541365+0000", 2024-09-21T15:49:59.172 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5965999999999997e-05 2024-09-21T15:49:59.172 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.172 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:59.173 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.173 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.173 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.173 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:70742 3.c 3:32ea75c5:::rbd_data.11336b2fdf4a.0000000000000001:c [read 1046528~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:59.173 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.495639+0000", 2024-09-21T15:49:59.173 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 21.626304953000002, 2024-09-21T15:49:59.173 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045872673000000003, 2024-09-21T15:49:59.173 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:59.173 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:59.173 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:59.173 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:59.173 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:59.173 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 70742 2024-09-21T15:49:59.173 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.173 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:59.174 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.174 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:59.174 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495639+0000", 2024-09-21T15:49:59.174 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.174 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.174 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.174 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:59.174 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495639+0000", 2024-09-21T15:49:59.174 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.174 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.174 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.174 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:59.174 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495638+0000", 2024-09-21T15:49:59.174 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:59.174 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.175 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.175 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:59.175 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495641+0000", 2024-09-21T15:49:59.175 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5849999999999998e-06 2024-09-21T15:49:59.175 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.175 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.175 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:59.175 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495647+0000", 2024-09-21T15:49:59.175 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3069999999999999e-06 2024-09-21T15:49:59.175 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.175 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.175 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:59.175 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495652+0000", 2024-09-21T15:49:59.175 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2229999999999998e-06 2024-09-21T15:49:59.175 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.176 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.176 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:59.176 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.541400+0000", 2024-09-21T15:49:59.176 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045748024999999998 2024-09-21T15:49:59.176 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.176 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.176 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:59.176 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.541451+0000", 2024-09-21T15:49:59.176 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0093999999999998e-05 2024-09-21T15:49:59.176 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.176 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.176 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:59.176 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.541511+0000", 2024-09-21T15:49:59.176 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0928999999999999e-05 2024-09-21T15:49:59.177 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.177 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:59.177 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.177 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.177 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.177 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:70743 3.c 3:32ea75c5:::rbd_data.11336b2fdf4a.0000000000000001:c [read 333824~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:59.177 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.495672+0000", 2024-09-21T15:49:59.177 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 21.626272180000001, 2024-09-21T15:49:59.177 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045980785000000003, 2024-09-21T15:49:59.177 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:59.177 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:59.177 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:59.178 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:59.178 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:59.178 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 70743 2024-09-21T15:49:59.178 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.178 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:59.178 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.178 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:59.178 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495672+0000", 2024-09-21T15:49:59.178 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.178 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.178 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.178 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:59.178 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495672+0000", 2024-09-21T15:49:59.178 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.179 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.179 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.179 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:59.179 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495671+0000", 2024-09-21T15:49:59.179 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:59.179 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.179 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.179 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:59.179 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495674+0000", 2024-09-21T15:49:59.179 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5790000000000002e-06 2024-09-21T15:49:59.179 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.179 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.179 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:59.179 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495686+0000", 2024-09-21T15:49:59.180 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2276000000000001e-05 2024-09-21T15:49:59.180 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.180 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.180 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:59.180 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495692+0000", 2024-09-21T15:49:59.180 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8939999999999998e-06 2024-09-21T15:49:59.180 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.180 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.180 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:59.180 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.541538+0000", 2024-09-21T15:49:59.180 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045845761999999998 2024-09-21T15:49:59.180 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.180 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.180 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:59.181 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.541583+0000", 2024-09-21T15:49:59.181 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5343000000000003e-05 2024-09-21T15:49:59.181 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.181 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.181 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:59.181 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.541652+0000", 2024-09-21T15:49:59.181 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9468999999999995e-05 2024-09-21T15:49:59.181 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.181 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:59.181 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.181 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.181 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.181 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:70744 3.c 3:32ea75c5:::rbd_data.11336b2fdf4a.0000000000000001:c [read 342016~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:59.181 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.495705+0000", 2024-09-21T15:49:59.181 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 21.626238447999999, 2024-09-21T15:49:59.182 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046156794000000001, 2024-09-21T15:49:59.182 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:59.182 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:59.182 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:59.182 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:59.182 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:59.182 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 70744 2024-09-21T15:49:59.182 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.182 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:59.182 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.182 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:59.182 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495705+0000", 2024-09-21T15:49:59.182 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.183 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.183 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.183 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:59.183 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495705+0000", 2024-09-21T15:49:59.183 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.183 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.183 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.183 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:59.183 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495705+0000", 2024-09-21T15:49:59.183 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:59.183 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.183 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.183 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:59.183 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495707+0000", 2024-09-21T15:49:59.184 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4260000000000002e-06 2024-09-21T15:49:59.184 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.184 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.184 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:59.184 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495713+0000", 2024-09-21T15:49:59.184 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1029999999999998e-06 2024-09-21T15:49:59.184 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.184 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.184 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:59.184 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495720+0000", 2024-09-21T15:49:59.184 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.178e-06 2024-09-21T15:49:59.184 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.184 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.184 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:59.184 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.541690+0000", 2024-09-21T15:49:59.185 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045970069000000002 2024-09-21T15:49:59.185 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.185 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.185 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:59.185 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.541770+0000", 2024-09-21T15:49:59.185 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9976999999999997e-05 2024-09-21T15:49:59.185 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.185 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.185 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:59.185 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.541862+0000", 2024-09-21T15:49:59.185 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2532000000000001e-05 2024-09-21T15:49:59.185 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.185 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:59.186 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.186 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.186 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.186 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:70746 3.c 3:32ea75c5:::rbd_data.11336b2fdf4a.0000000000000001:c [read 350208~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:59.186 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.495741+0000", 2024-09-21T15:49:59.186 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 21.626202839000001, 2024-09-21T15:49:59.186 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046299229999999997, 2024-09-21T15:49:59.186 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:59.186 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:59.186 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:59.186 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:59.186 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:59.186 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 70746 2024-09-21T15:49:59.186 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.186 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:59.187 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.187 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:59.187 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495741+0000", 2024-09-21T15:49:59.187 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.187 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.187 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.187 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:59.187 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495741+0000", 2024-09-21T15:49:59.187 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.187 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.187 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.187 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:59.187 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495740+0000", 2024-09-21T15:49:59.187 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:59.188 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.188 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.188 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:59.188 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495743+0000", 2024-09-21T15:49:59.188 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5189999999999999e-06 2024-09-21T15:49:59.188 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.188 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.188 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:59.188 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495749+0000", 2024-09-21T15:49:59.188 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5470000000000002e-06 2024-09-21T15:49:59.188 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.188 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.188 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:59.188 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495755+0000", 2024-09-21T15:49:59.188 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2399999999999998e-06 2024-09-21T15:49:59.189 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.189 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.189 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:59.189 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.541877+0000", 2024-09-21T15:49:59.189 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046122375 2024-09-21T15:49:59.189 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.189 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.189 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:59.189 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.541945+0000", 2024-09-21T15:49:59.189 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7825999999999997e-05 2024-09-21T15:49:59.189 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.189 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.189 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:59.189 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.542040+0000", 2024-09-21T15:49:59.189 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5303999999999994e-05 2024-09-21T15:49:59.190 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.190 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:59.190 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.190 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.190 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.190 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:70748 3.c 3:32ea75c5:::rbd_data.11336b2fdf4a.0000000000000001:c [read 358400~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:59.190 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.495759+0000", 2024-09-21T15:49:59.190 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 21.626184684999998, 2024-09-21T15:49:59.190 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046462341999999997, 2024-09-21T15:49:59.190 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:59.190 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:59.190 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:59.190 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:59.190 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:59.191 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 70748 2024-09-21T15:49:59.191 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.191 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:59.191 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.191 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:59.191 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495759+0000", 2024-09-21T15:49:59.191 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.191 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.191 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.191 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:59.191 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495759+0000", 2024-09-21T15:49:59.191 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.191 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.191 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.192 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:59.192 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495759+0000", 2024-09-21T15:49:59.192 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:59.192 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.192 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.192 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:59.192 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495761+0000", 2024-09-21T15:49:59.192 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1900000000000002e-06 2024-09-21T15:49:59.192 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.192 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.192 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:59.192 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495767+0000", 2024-09-21T15:49:59.192 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0800000000000002e-06 2024-09-21T15:49:59.192 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.192 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.193 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:59.193 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495773+0000", 2024-09-21T15:49:59.193 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0000000000000002e-06 2024-09-21T15:49:59.193 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.193 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.193 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:59.193 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.542066+0000", 2024-09-21T15:49:59.193 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046293197000000001 2024-09-21T15:49:59.193 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.193 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.193 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:59.193 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.542139+0000", 2024-09-21T15:49:59.193 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2770999999999998e-05 2024-09-21T15:49:59.193 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.194 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.194 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:59.194 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.542221+0000", 2024-09-21T15:49:59.194 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2620000000000005e-05 2024-09-21T15:49:59.194 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.194 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:59.194 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.194 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.194 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.194 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:70750 3.c 3:32ea75c5:::rbd_data.11336b2fdf4a.0000000000000001:c [read 366592~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:59.194 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.495781+0000", 2024-09-21T15:49:59.194 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 21.626162341000001, 2024-09-21T15:49:59.194 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046624691000000003, 2024-09-21T15:49:59.194 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:59.195 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:59.195 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:59.195 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:59.195 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:59.195 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 70750 2024-09-21T15:49:59.195 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.195 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:59.195 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.195 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:59.195 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495781+0000", 2024-09-21T15:49:59.195 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.195 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.195 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.195 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:59.195 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495781+0000", 2024-09-21T15:49:59.196 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.196 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.196 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.196 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:59.196 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495781+0000", 2024-09-21T15:49:59.196 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:59.196 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.196 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.196 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:59.196 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495783+0000", 2024-09-21T15:49:59.196 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4729999999999999e-06 2024-09-21T15:49:59.196 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.196 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.196 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:59.196 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495790+0000", 2024-09-21T15:49:59.197 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3500000000000002e-06 2024-09-21T15:49:59.197 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.197 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.197 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:59.197 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495795+0000", 2024-09-21T15:49:59.197 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9679999999999997e-06 2024-09-21T15:49:59.197 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.197 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.197 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:59.197 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.542248+0000", 2024-09-21T15:49:59.197 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046453117000000002 2024-09-21T15:49:59.197 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.197 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.197 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:59.198 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.542312+0000", 2024-09-21T15:49:59.198 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4215999999999996e-05 2024-09-21T15:49:59.198 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.198 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.198 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:59.198 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.542406+0000", 2024-09-21T15:49:59.198 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3980000000000005e-05 2024-09-21T15:49:59.198 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.198 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:59.198 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.198 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.198 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.198 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:70751 3.c 3:32ea75c5:::rbd_data.11336b2fdf4a.0000000000000001:c [read 374784~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:59.198 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.495799+0000", 2024-09-21T15:49:59.198 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 21.626144546999999, 2024-09-21T15:49:59.199 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046807169000000003, 2024-09-21T15:49:59.199 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:59.199 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:59.199 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:59.199 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:59.199 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:59.199 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 70751 2024-09-21T15:49:59.199 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.199 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:59.199 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.199 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:59.199 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495799+0000", 2024-09-21T15:49:59.199 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.199 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.200 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.200 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:59.200 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495799+0000", 2024-09-21T15:49:59.200 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.200 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.200 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.200 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:59.200 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495799+0000", 2024-09-21T15:49:59.200 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:59.200 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.200 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.200 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:59.200 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495801+0000", 2024-09-21T15:49:59.200 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2560000000000001e-06 2024-09-21T15:49:59.200 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.201 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.201 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:59.201 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495807+0000", 2024-09-21T15:49:59.201 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8209999999999998e-06 2024-09-21T15:49:59.201 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.201 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.201 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:59.201 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495812+0000", 2024-09-21T15:49:59.201 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0289999999999996e-06 2024-09-21T15:49:59.201 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.201 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.201 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:59.201 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.542433+0000", 2024-09-21T15:49:59.201 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046621392999999997 2024-09-21T15:49:59.201 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.202 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.202 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:59.202 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.542510+0000", 2024-09-21T15:49:59.202 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7069000000000004e-05 2024-09-21T15:49:59.202 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.202 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.202 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:59.202 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.542606+0000", 2024-09-21T15:49:59.202 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.6036e-05 2024-09-21T15:49:59.202 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.202 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:59.202 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.202 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.202 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.203 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:70753 3.c 3:32ea75c5:::rbd_data.11336b2fdf4a.0000000000000001:c [read 382976~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:59.203 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.495816+0000", 2024-09-21T15:49:59.203 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 21.626127541999999, 2024-09-21T15:49:59.203 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046993252999999999, 2024-09-21T15:49:59.203 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:59.203 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:59.203 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:59.203 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:59.203 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:59.203 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 70753 2024-09-21T15:49:59.203 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.203 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:59.203 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.203 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:59.203 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495816+0000", 2024-09-21T15:49:59.204 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.204 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.204 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.204 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:59.204 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495816+0000", 2024-09-21T15:49:59.204 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.204 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.204 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.204 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:59.204 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495816+0000", 2024-09-21T15:49:59.204 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:59.204 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.204 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.204 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:59.204 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495818+0000", 2024-09-21T15:49:59.205 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1160000000000002e-06 2024-09-21T15:49:59.205 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.205 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.205 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:59.205 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495826+0000", 2024-09-21T15:49:59.205 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7800000000000001e-06 2024-09-21T15:49:59.205 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.205 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.205 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:59.205 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495831+0000", 2024-09-21T15:49:59.205 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9010000000000002e-06 2024-09-21T15:49:59.205 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.205 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.205 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:59.205 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.542665+0000", 2024-09-21T15:49:59.205 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046834037000000002 2024-09-21T15:49:59.206 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.206 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.206 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:59.206 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.542729+0000", 2024-09-21T15:49:59.206 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4199e-05 2024-09-21T15:49:59.206 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.206 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.206 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:59.206 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.542809+0000", 2024-09-21T15:49:59.206 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0592000000000004e-05 2024-09-21T15:49:59.206 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.206 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:59.206 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.207 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.207 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.207 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:70755 3.c 3:32ea75c5:::rbd_data.11336b2fdf4a.0000000000000001:c [read 391168~2048] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:59.207 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.495835+0000", 2024-09-21T15:49:59.207 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 21.626109025000002, 2024-09-21T15:49:59.207 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047136708999999999, 2024-09-21T15:49:59.207 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:59.207 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:59.207 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:59.207 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:59.207 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:59.207 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 70755 2024-09-21T15:49:59.207 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.207 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:59.207 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.208 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:59.208 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495835+0000", 2024-09-21T15:49:59.208 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.208 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.208 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.208 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:59.208 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495835+0000", 2024-09-21T15:49:59.208 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.208 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.208 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.208 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:59.208 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495834+0000", 2024-09-21T15:49:59.208 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:59.208 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.208 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.209 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:59.209 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495836+0000", 2024-09-21T15:49:59.209 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1280000000000002e-06 2024-09-21T15:49:59.209 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.209 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.209 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:59.209 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495842+0000", 2024-09-21T15:49:59.209 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8819999999999998e-06 2024-09-21T15:49:59.209 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.209 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.209 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:59.209 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.495848+0000", 2024-09-21T15:49:59.209 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1850000000000003e-06 2024-09-21T15:49:59.209 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.209 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.210 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:59.210 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.542836+0000", 2024-09-21T15:49:59.210 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046988156000000003 2024-09-21T15:49:59.210 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.210 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.210 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:59.210 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.542897+0000", 2024-09-21T15:49:59.210 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1505000000000002e-05 2024-09-21T15:49:59.210 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.210 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.210 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:59.210 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.542971+0000", 2024-09-21T15:49:59.210 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4221000000000006e-05 2024-09-21T15:49:59.210 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.211 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:59.211 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.211 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.211 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:59.211 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:59.212 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T15:49:59.212 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:49:59.211+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:49:59.313 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:49:59.312+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:49:59.314 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:59.315 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:59.315 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:49:59.315 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:59.323 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T15:49:59.414 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:49:59.412+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:49:59.465 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:59.465 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:59.465 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:49:59.465 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:49:59.465 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:59.473 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T15:49:59.515 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:59.514+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:59.616 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:59.616 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:49:59.617 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:49:59.617 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:49:59.617 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.617 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63140 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 876544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:59.617 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.133649+0000", 2024-09-21T15:49:59.617 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.481444789000001, 2024-09-21T15:49:59.617 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053950201000000003, 2024-09-21T15:49:59.617 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:59.617 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:59.617 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:59.617 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:59.617 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:59.617 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63140 2024-09-21T15:49:59.617 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.618 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:59.618 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.618 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:59.618 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133649+0000", 2024-09-21T15:49:59.618 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.618 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.618 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.618 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:59.618 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133649+0000", 2024-09-21T15:49:59.618 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.618 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.618 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.618 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:59.618 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133648+0000", 2024-09-21T15:49:59.619 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:59.619 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.619 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.619 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:59.619 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133651+0000", 2024-09-21T15:49:59.619 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1549999999999999e-06 2024-09-21T15:49:59.619 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.619 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.619 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:59.619 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133660+0000", 2024-09-21T15:49:59.619 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3899999999999993e-06 2024-09-21T15:49:59.619 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.619 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.619 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:59.619 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133669+0000", 2024-09-21T15:49:59.620 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3219999999999999e-06 2024-09-21T15:49:59.620 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.620 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.620 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:59.620 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187409+0000", 2024-09-21T15:49:59.620 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053740257 2024-09-21T15:49:59.620 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.620 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.620 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:59.620 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187492+0000", 2024-09-21T15:49:59.620 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2409000000000003e-05 2024-09-21T15:49:59.620 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.620 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.620 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:59.620 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187599+0000", 2024-09-21T15:49:59.621 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000107334 2024-09-21T15:49:59.621 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.621 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:59.621 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.621 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.621 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.621 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63141 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 884736~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:59.621 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.133843+0000", 2024-09-21T15:49:59.621 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.481250670000001, 2024-09-21T15:49:59.621 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053954015000000001, 2024-09-21T15:49:59.621 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:59.621 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:59.621 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:59.621 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:59.621 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:59.622 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63141 2024-09-21T15:49:59.622 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.622 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:59.622 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.622 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:59.622 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133843+0000", 2024-09-21T15:49:59.622 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.622 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.622 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.622 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:59.622 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133843+0000", 2024-09-21T15:49:59.622 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.622 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.622 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.622 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:59.622 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133673+0000", 2024-09-21T15:49:59.623 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998302 2024-09-21T15:49:59.623 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.623 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.623 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:59.623 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133846+0000", 2024-09-21T15:49:59.623 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000172762 2024-09-21T15:49:59.623 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.623 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.623 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:59.623 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133855+0000", 2024-09-21T15:49:59.623 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2940000000000006e-06 2024-09-21T15:49:59.623 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.623 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.623 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:59.623 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133863+0000", 2024-09-21T15:49:59.624 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9740000000000002e-06 2024-09-21T15:49:59.624 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.624 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.624 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:59.624 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187630+0000", 2024-09-21T15:49:59.624 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053767211000000002 2024-09-21T15:49:59.624 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.624 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.624 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:59.624 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187707+0000", 2024-09-21T15:49:59.624 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7446000000000004e-05 2024-09-21T15:49:59.624 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.624 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.624 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:59.624 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187797+0000", 2024-09-21T15:49:59.625 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.9858000000000006e-05 2024-09-21T15:49:59.625 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.625 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:59.625 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.625 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.625 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.625 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63142 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 892928~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:59.625 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134046+0000", 2024-09-21T15:49:59.625 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.481047728, 2024-09-21T15:49:59.625 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054017849999999999, 2024-09-21T15:49:59.625 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:59.625 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:59.625 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:59.625 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:59.625 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:59.626 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63142 2024-09-21T15:49:59.626 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.626 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:59.626 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.626 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:59.626 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134046+0000", 2024-09-21T15:49:59.626 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.626 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.626 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.626 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:59.626 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134046+0000", 2024-09-21T15:49:59.626 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.626 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.626 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.627 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:59.627 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133867+0000", 2024-09-21T15:49:59.627 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998217 2024-09-21T15:49:59.627 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.627 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.627 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:59.627 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134049+0000", 2024-09-21T15:49:59.627 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000181695 2024-09-21T15:49:59.627 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.627 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.627 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:59.627 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134057+0000", 2024-09-21T15:49:59.627 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.303e-06 2024-09-21T15:49:59.627 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.627 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.627 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:59.628 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134063+0000", 2024-09-21T15:49:59.628 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9959999999999999e-06 2024-09-21T15:49:59.628 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.628 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.628 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:59.628 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187849+0000", 2024-09-21T15:49:59.628 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053785571999999997 2024-09-21T15:49:59.628 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.628 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.628 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:59.628 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187954+0000", 2024-09-21T15:49:59.628 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010524899999999999 2024-09-21T15:49:59.628 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.628 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.628 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:59.629 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188064+0000", 2024-09-21T15:49:59.629 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000109549 2024-09-21T15:49:59.629 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.629 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:59.629 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.629 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.629 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.629 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63143 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 901120~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:59.629 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134239+0000", 2024-09-21T15:49:59.629 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.480855134999999, 2024-09-21T15:49:59.629 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054026058000000002, 2024-09-21T15:49:59.629 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:59.629 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:59.629 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:59.629 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:59.630 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:59.630 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63143 2024-09-21T15:49:59.630 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.630 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:59.630 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.630 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:59.630 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134239+0000", 2024-09-21T15:49:59.630 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.630 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.630 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.630 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:59.630 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134239+0000", 2024-09-21T15:49:59.630 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.630 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.630 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.631 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:59.631 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134068+0000", 2024-09-21T15:49:59.631 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998298 2024-09-21T15:49:59.631 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.631 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.631 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:59.631 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134242+0000", 2024-09-21T15:49:59.631 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00017414199999999999 2024-09-21T15:49:59.631 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.631 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.631 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:59.631 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134264+0000", 2024-09-21T15:49:59.631 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1552e-05 2024-09-21T15:49:59.631 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.631 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.632 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:59.632 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134270+0000", 2024-09-21T15:49:59.632 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6939999999999998e-06 2024-09-21T15:49:59.632 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.632 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.632 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:59.632 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188108+0000", 2024-09-21T15:49:59.632 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.05383719 2024-09-21T15:49:59.632 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.632 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.632 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:59.632 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188186+0000", 2024-09-21T15:49:59.632 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8318000000000005e-05 2024-09-21T15:49:59.632 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.632 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.632 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:59.633 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188265+0000", 2024-09-21T15:49:59.633 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8597000000000007e-05 2024-09-21T15:49:59.633 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.633 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:59.633 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.633 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.633 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.633 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63144 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 909312~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:59.633 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134432+0000", 2024-09-21T15:49:59.633 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.480661579, 2024-09-21T15:49:59.633 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053968953, 2024-09-21T15:49:59.633 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:59.633 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:59.633 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:59.633 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:59.634 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:59.634 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63144 2024-09-21T15:49:59.634 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.634 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:59.634 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.634 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:59.634 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134432+0000", 2024-09-21T15:49:59.634 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.634 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.634 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.634 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:59.634 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134432+0000", 2024-09-21T15:49:59.634 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.634 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.635 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.635 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:59.635 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134276+0000", 2024-09-21T15:49:59.635 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998441 2024-09-21T15:49:59.635 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.635 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.635 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:59.635 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134436+0000", 2024-09-21T15:49:59.635 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000159591 2024-09-21T15:49:59.635 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.635 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.635 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:59.635 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134445+0000", 2024-09-21T15:49:59.635 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6780000000000006e-06 2024-09-21T15:49:59.635 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.636 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.636 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:59.636 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134452+0000", 2024-09-21T15:49:59.636 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6830000000000008e-06 2024-09-21T15:49:59.636 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.636 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.636 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:59.636 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188282+0000", 2024-09-21T15:49:59.636 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053829252000000001 2024-09-21T15:49:59.636 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.636 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.636 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:59.636 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188328+0000", 2024-09-21T15:49:59.636 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6734999999999999e-05 2024-09-21T15:49:59.636 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.636 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.637 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:59.637 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188401+0000", 2024-09-21T15:49:59.637 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2731000000000005e-05 2024-09-21T15:49:59.637 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.637 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:59.637 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.637 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.637 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.637 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63145 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 860160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:59.637 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134736+0000", 2024-09-21T15:49:59.637 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.480357794, 2024-09-21T15:49:59.637 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053865960999999997, 2024-09-21T15:49:59.637 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:59.637 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:59.637 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:59.638 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:59.638 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:59.638 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63145 2024-09-21T15:49:59.638 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.638 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:59.638 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.638 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:59.638 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134736+0000", 2024-09-21T15:49:59.638 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.638 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.638 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.638 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:59.638 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134736+0000", 2024-09-21T15:49:59.638 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.638 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.639 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.639 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:59.639 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134458+0000", 2024-09-21T15:49:59.639 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999722 2024-09-21T15:49:59.639 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.639 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.639 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:59.639 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134741+0000", 2024-09-21T15:49:59.639 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00028245599999999999 2024-09-21T15:49:59.639 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.639 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.639 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:59.639 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134753+0000", 2024-09-21T15:49:59.639 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2548000000000001e-05 2024-09-21T15:49:59.639 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.640 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.640 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:59.640 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134760+0000", 2024-09-21T15:49:59.640 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4189999999999996e-06 2024-09-21T15:49:59.640 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.640 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.640 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:59.640 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188426+0000", 2024-09-21T15:49:59.640 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053665894999999998 2024-09-21T15:49:59.640 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.640 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.640 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:59.640 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188502+0000", 2024-09-21T15:49:59.640 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5791000000000007e-05 2024-09-21T15:49:59.640 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.640 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.641 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:59.641 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188602+0000", 2024-09-21T15:49:59.641 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9667999999999995e-05 2024-09-21T15:49:59.641 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.641 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:59.641 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.641 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.641 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.641 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63148 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 933888~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:59.641 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.135289+0000", 2024-09-21T15:49:59.641 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.479804839, 2024-09-21T15:49:59.641 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053948488000000003, 2024-09-21T15:49:59.641 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:59.641 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:59.641 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:59.642 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:59.642 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:59.642 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63148 2024-09-21T15:49:59.642 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.642 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:59.642 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.642 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:59.642 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135289+0000", 2024-09-21T15:49:59.642 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.642 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.642 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.642 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:59.642 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135289+0000", 2024-09-21T15:49:59.642 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.643 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.643 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.643 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:59.643 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135288+0000", 2024-09-21T15:49:59.643 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:59.643 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.643 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.643 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:59.643 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135296+0000", 2024-09-21T15:49:59.643 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5399999999999998e-06 2024-09-21T15:49:59.643 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.643 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.643 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:59.643 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135303+0000", 2024-09-21T15:49:59.643 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0500000000000003e-06 2024-09-21T15:49:59.644 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.644 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.644 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:59.644 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135308+0000", 2024-09-21T15:49:59.644 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9749999999999998e-06 2024-09-21T15:49:59.644 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.644 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.644 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:59.644 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189066+0000", 2024-09-21T15:49:59.644 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053758501 2024-09-21T15:49:59.644 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.644 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.644 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:59.644 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189145+0000", 2024-09-21T15:49:59.644 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9017000000000006e-05 2024-09-21T15:49:59.645 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.645 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.645 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:59.645 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189237+0000", 2024-09-21T15:49:59.645 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1906999999999999e-05 2024-09-21T15:49:59.645 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.645 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:59.645 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.645 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.645 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.645 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63149 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 942080~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:59.645 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.135430+0000", 2024-09-21T15:49:59.645 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.479663510000002, 2024-09-21T15:49:59.645 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054005434999999997, 2024-09-21T15:49:59.645 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:59.646 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:59.646 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:59.646 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:59.646 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:59.646 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63149 2024-09-21T15:49:59.646 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.646 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:59.646 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.646 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:59.646 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135430+0000", 2024-09-21T15:49:59.646 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.646 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.646 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.646 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:59.647 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135430+0000", 2024-09-21T15:49:59.647 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.647 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.647 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.647 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:59.647 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135313+0000", 2024-09-21T15:49:59.647 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998827 2024-09-21T15:49:59.647 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.647 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.647 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:59.647 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135434+0000", 2024-09-21T15:49:59.647 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000120876 2024-09-21T15:49:59.647 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.647 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.647 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:59.648 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135443+0000", 2024-09-21T15:49:59.648 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3340000000000008e-06 2024-09-21T15:49:59.648 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.648 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.648 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:59.648 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135449+0000", 2024-09-21T15:49:59.648 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1580000000000002e-06 2024-09-21T15:49:59.648 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.648 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.648 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:59.648 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189267+0000", 2024-09-21T15:49:59.648 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053817775999999998 2024-09-21T15:49:59.648 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.648 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.648 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:59.649 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189336+0000", 2024-09-21T15:49:59.649 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9485000000000003e-05 2024-09-21T15:49:59.649 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.649 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.649 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:59.649 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189436+0000", 2024-09-21T15:49:59.649 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9277000000000005e-05 2024-09-21T15:49:59.649 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.649 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:59.649 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.649 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.649 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.649 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63150 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 950272~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:59.649 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.135618+0000", 2024-09-21T15:49:59.649 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.479475765, 2024-09-21T15:49:59.650 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054027393, 2024-09-21T15:49:59.650 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:59.650 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:59.650 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:59.650 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:59.650 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:59.650 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63150 2024-09-21T15:49:59.650 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.650 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:59.650 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.650 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:59.650 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135618+0000", 2024-09-21T15:49:59.650 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.650 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.650 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.651 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:59.651 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135618+0000", 2024-09-21T15:49:59.651 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.651 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.651 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.651 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:59.651 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135454+0000", 2024-09-21T15:49:59.651 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998364 2024-09-21T15:49:59.651 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.651 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.651 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:59.651 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135621+0000", 2024-09-21T15:49:59.651 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016714099999999999 2024-09-21T15:49:59.651 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.651 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.651 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:59.652 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135631+0000", 2024-09-21T15:49:59.652 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5340000000000005e-06 2024-09-21T15:49:59.652 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.652 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.652 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:59.652 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135638+0000", 2024-09-21T15:49:59.652 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9280000000000004e-06 2024-09-21T15:49:59.652 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.652 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.652 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:59.652 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189463+0000", 2024-09-21T15:49:59.652 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053825681 2024-09-21T15:49:59.652 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.652 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.653 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:59.653 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189546+0000", 2024-09-21T15:49:59.653 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2278000000000001e-05 2024-09-21T15:49:59.653 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.653 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.653 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:59.653 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189645+0000", 2024-09-21T15:49:59.653 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9543999999999994e-05 2024-09-21T15:49:59.653 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.653 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:59.653 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.653 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.653 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.653 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63152 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 966656~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:59.653 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.136029+0000", 2024-09-21T15:49:59.653 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.479065172999999, 2024-09-21T15:49:59.654 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054024823999999999, 2024-09-21T15:49:59.654 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:59.654 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:59.654 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:59.654 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:59.654 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:59.654 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63152 2024-09-21T15:49:59.654 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.654 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:59.654 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.654 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:59.654 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136029+0000", 2024-09-21T15:49:59.654 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.654 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.654 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.655 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:59.655 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136029+0000", 2024-09-21T15:49:59.655 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.655 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.655 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.655 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:59.655 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136028+0000", 2024-09-21T15:49:59.655 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:59.655 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.655 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.655 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:59.655 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136031+0000", 2024-09-21T15:49:59.655 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.033e-06 2024-09-21T15:49:59.655 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.655 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.656 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:59.656 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136039+0000", 2024-09-21T15:49:59.656 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7200000000000006e-06 2024-09-21T15:49:59.656 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.656 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.656 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:59.656 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136051+0000", 2024-09-21T15:49:59.656 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1987000000000001e-05 2024-09-21T15:49:59.656 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.656 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.656 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:59.656 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189876+0000", 2024-09-21T15:49:59.656 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053825264999999997 2024-09-21T15:49:59.656 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.656 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.657 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:59.657 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189955+0000", 2024-09-21T15:49:59.657 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9296999999999997e-05 2024-09-21T15:49:59.657 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.657 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.657 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:59.657 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190053+0000", 2024-09-21T15:49:59.657 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.8067999999999997e-05 2024-09-21T15:49:59.657 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.657 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:59.657 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.657 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.657 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.657 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63153 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 974848~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:59.657 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.136311+0000", 2024-09-21T15:49:59.658 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.47878266, 2024-09-21T15:49:59.658 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053933825999999997, 2024-09-21T15:49:59.658 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:59.658 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:59.658 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:59.658 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:59.658 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:59.658 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63153 2024-09-21T15:49:59.658 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.658 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:59.658 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.658 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:59.658 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136311+0000", 2024-09-21T15:49:59.658 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.658 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.659 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.659 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:59.659 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136311+0000", 2024-09-21T15:49:59.659 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.659 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.659 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.659 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:59.659 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136057+0000", 2024-09-21T15:49:59.659 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9997463 2024-09-21T15:49:59.659 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.659 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.659 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:59.659 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136316+0000", 2024-09-21T15:49:59.659 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00025872299999999998 2024-09-21T15:49:59.660 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.660 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.660 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:59.660 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136326+0000", 2024-09-21T15:49:59.660 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0552e-05 2024-09-21T15:49:59.660 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.660 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.660 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:59.660 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136334+0000", 2024-09-21T15:49:59.660 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1409999999999999e-06 2024-09-21T15:49:59.660 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.660 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.660 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:59.660 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190080+0000", 2024-09-21T15:49:59.661 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053745933000000003 2024-09-21T15:49:59.661 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.661 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.661 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:59.661 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190157+0000", 2024-09-21T15:49:59.661 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7412999999999999e-05 2024-09-21T15:49:59.661 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.661 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.661 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:59.661 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190245+0000", 2024-09-21T15:49:59.661 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7904000000000004e-05 2024-09-21T15:49:59.661 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.661 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:59.661 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.661 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.662 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.662 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63154 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 983040~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:59.662 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.136472+0000", 2024-09-21T15:49:59.662 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.478621309000001, 2024-09-21T15:49:59.662 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053968448000000002, 2024-09-21T15:49:59.662 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:59.662 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:59.662 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:59.662 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:59.662 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:59.662 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63154 2024-09-21T15:49:59.662 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.662 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:59.663 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.663 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:59.663 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136472+0000", 2024-09-21T15:49:59.663 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.663 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.663 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.663 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:59.663 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136472+0000", 2024-09-21T15:49:59.663 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.663 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.663 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.663 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:59.663 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136341+0000", 2024-09-21T15:49:59.663 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998684 2024-09-21T15:49:59.664 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.664 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.664 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:59.664 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136476+0000", 2024-09-21T15:49:59.664 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013528100000000001 2024-09-21T15:49:59.664 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.664 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.664 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:59.664 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136484+0000", 2024-09-21T15:49:59.664 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4760000000000001e-06 2024-09-21T15:49:59.664 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.664 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.664 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:59.664 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136490+0000", 2024-09-21T15:49:59.665 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8640000000000001e-06 2024-09-21T15:49:59.665 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.665 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.665 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:59.665 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190271+0000", 2024-09-21T15:49:59.665 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053781223000000003 2024-09-21T15:49:59.665 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.665 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.665 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:59.665 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190348+0000", 2024-09-21T15:49:59.665 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7058999999999995e-05 2024-09-21T15:49:59.665 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.665 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.665 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:59.665 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190441+0000", 2024-09-21T15:49:59.666 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2937999999999997e-05 2024-09-21T15:49:59.666 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.666 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:59.666 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.666 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.666 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.666 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63157 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1015808~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:59.666 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137112+0000", 2024-09-21T15:49:59.666 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.477982021999999, 2024-09-21T15:49:59.666 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053932345, 2024-09-21T15:49:59.666 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:59.666 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:59.666 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:59.666 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:59.667 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:59.667 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63157 2024-09-21T15:49:59.667 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.667 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:59.667 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.667 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:59.667 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137112+0000", 2024-09-21T15:49:59.667 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.667 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.667 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.667 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:59.667 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137112+0000", 2024-09-21T15:49:59.667 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.667 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.667 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.668 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:59.668 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137072+0000", 2024-09-21T15:49:59.668 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999604 2024-09-21T15:49:59.668 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.668 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.668 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:59.668 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137115+0000", 2024-09-21T15:49:59.668 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2379999999999997e-05 2024-09-21T15:49:59.668 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.668 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.668 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:59.668 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137121+0000", 2024-09-21T15:49:59.668 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6499999999999999e-06 2024-09-21T15:49:59.668 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.668 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.668 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:59.669 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137126+0000", 2024-09-21T15:49:59.669 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0769999999999998e-06 2024-09-21T15:49:59.669 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.669 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.669 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:59.669 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190868+0000", 2024-09-21T15:49:59.669 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053741555000000003 2024-09-21T15:49:59.669 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.669 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.669 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:59.669 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190945+0000", 2024-09-21T15:49:59.669 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7171999999999999e-05 2024-09-21T15:49:59.669 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.669 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.669 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:59.670 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191044+0000", 2024-09-21T15:49:59.670 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9031999999999996e-05 2024-09-21T15:49:59.670 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.670 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:59.670 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.670 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.670 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.670 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63158 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1007616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:59.670 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137293+0000", 2024-09-21T15:49:59.670 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.477800734999999, 2024-09-21T15:49:59.670 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054005642, 2024-09-21T15:49:59.670 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:59.670 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:59.670 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:59.671 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:59.671 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:59.671 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63158 2024-09-21T15:49:59.671 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.671 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:59.671 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.671 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:59.671 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137293+0000", 2024-09-21T15:49:59.671 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.671 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.671 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.671 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:59.671 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137293+0000", 2024-09-21T15:49:59.671 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.671 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.671 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.672 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:59.672 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137131+0000", 2024-09-21T15:49:59.672 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998379 2024-09-21T15:49:59.672 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.672 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.672 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:59.672 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137296+0000", 2024-09-21T15:49:59.672 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016568999999999999 2024-09-21T15:49:59.672 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.672 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.672 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:59.672 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137306+0000", 2024-09-21T15:49:59.672 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.6029999999999994e-06 2024-09-21T15:49:59.672 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.673 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.673 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:59.673 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137312+0000", 2024-09-21T15:49:59.673 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3210000000000001e-06 2024-09-21T15:49:59.673 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.673 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.673 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:59.673 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191087+0000", 2024-09-21T15:49:59.673 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053774686000000002 2024-09-21T15:49:59.673 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.673 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.673 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:59.673 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191175+0000", 2024-09-21T15:49:59.673 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8106e-05 2024-09-21T15:49:59.673 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.674 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.674 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:59.674 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191299+0000", 2024-09-21T15:49:59.674 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012343599999999999 2024-09-21T15:49:59.674 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.674 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:59.674 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.674 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.674 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.674 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63159 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1024000~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:59.674 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137568+0000", 2024-09-21T15:49:59.674 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.477525462999999, 2024-09-21T15:49:59.674 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053962999999999997, 2024-09-21T15:49:59.674 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:59.674 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:59.675 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:59.675 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:59.675 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:59.675 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63159 2024-09-21T15:49:59.675 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.675 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:59.675 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.675 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:59.675 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137568+0000", 2024-09-21T15:49:59.675 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.675 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.675 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.675 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:59.675 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137568+0000", 2024-09-21T15:49:59.675 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.676 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.676 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.676 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:59.676 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137317+0000", 2024-09-21T15:49:59.676 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9997492 2024-09-21T15:49:59.676 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.676 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.676 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:59.676 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137572+0000", 2024-09-21T15:49:59.676 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00025491499999999998 2024-09-21T15:49:59.676 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.676 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.676 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:59.676 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137582+0000", 2024-09-21T15:49:59.676 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4490000000000005e-06 2024-09-21T15:49:59.677 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.677 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.677 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:59.677 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137589+0000", 2024-09-21T15:49:59.677 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5580000000000003e-06 2024-09-21T15:49:59.677 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.677 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.677 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:59.677 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191344+0000", 2024-09-21T15:49:59.677 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053754272999999998 2024-09-21T15:49:59.677 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.677 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.677 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:59.677 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191429+0000", 2024-09-21T15:49:59.678 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.509e-05 2024-09-21T15:49:59.678 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.678 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.678 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:59.678 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191531+0000", 2024-09-21T15:49:59.678 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000102519 2024-09-21T15:49:59.678 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.678 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:59.678 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.678 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.678 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.678 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63160 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1032192~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:59.678 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137771+0000", 2024-09-21T15:49:59.678 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.477322929, 2024-09-21T15:49:59.678 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053957818999999997, 2024-09-21T15:49:59.679 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:59.679 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:59.679 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:59.679 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:59.679 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:59.679 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63160 2024-09-21T15:49:59.679 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.679 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:59.679 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.679 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:59.679 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137771+0000", 2024-09-21T15:49:59.679 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.679 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.679 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.680 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:59.680 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137771+0000", 2024-09-21T15:49:59.680 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.680 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.680 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.680 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:59.680 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137596+0000", 2024-09-21T15:49:59.680 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998255 2024-09-21T15:49:59.680 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.680 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.680 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:59.680 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137780+0000", 2024-09-21T15:49:59.680 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00018363900000000001 2024-09-21T15:49:59.680 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.680 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.681 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:59.681 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137791+0000", 2024-09-21T15:49:59.681 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.096e-05 2024-09-21T15:49:59.681 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.681 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.681 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:59.681 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137798+0000", 2024-09-21T15:49:59.681 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1169999999999998e-06 2024-09-21T15:49:59.681 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.681 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.681 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:59.681 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191560+0000", 2024-09-21T15:49:59.681 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053762163000000002 2024-09-21T15:49:59.681 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.681 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.681 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:59.682 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191634+0000", 2024-09-21T15:49:59.682 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4381999999999994e-05 2024-09-21T15:49:59.682 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.682 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.682 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:59.682 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191729+0000", 2024-09-21T15:49:59.682 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4098000000000006e-05 2024-09-21T15:49:59.682 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.682 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:59.682 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.682 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.682 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.682 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63161 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1040384~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:59.682 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.138018+0000", 2024-09-21T15:49:59.683 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.477076175000001, 2024-09-21T15:49:59.683 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053905657000000003, 2024-09-21T15:49:59.683 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:59.683 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:59.683 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:59.683 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:59.683 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:59.683 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63161 2024-09-21T15:49:59.683 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.683 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:59.683 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.683 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:59.683 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138018+0000", 2024-09-21T15:49:59.683 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.684 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.684 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.684 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:59.684 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138018+0000", 2024-09-21T15:49:59.684 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.684 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.684 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.684 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:59.684 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137804+0000", 2024-09-21T15:49:59.684 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9997859 2024-09-21T15:49:59.684 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.684 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.684 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:59.684 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138021+0000", 2024-09-21T15:49:59.684 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00021769899999999999 2024-09-21T15:49:59.685 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.685 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.685 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:59.685 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138043+0000", 2024-09-21T15:49:59.685 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1627000000000001e-05 2024-09-21T15:49:59.685 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.685 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.685 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:59.685 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138054+0000", 2024-09-21T15:49:59.685 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1321999999999999e-05 2024-09-21T15:49:59.685 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.685 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.685 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:59.685 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191755+0000", 2024-09-21T15:49:59.685 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053700354999999998 2024-09-21T15:49:59.685 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.686 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.686 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:59.686 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191830+0000", 2024-09-21T15:49:59.686 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5320000000000004e-05 2024-09-21T15:49:59.686 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.686 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.686 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:59.686 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191923+0000", 2024-09-21T15:49:59.686 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3282999999999994e-05 2024-09-21T15:49:59.686 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.686 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:59.686 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.686 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.686 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.687 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63176 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 108544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:59.687 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.140861+0000", 2024-09-21T15:49:59.687 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.474232856, 2024-09-21T15:49:59.687 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053935394999999997, 2024-09-21T15:49:59.687 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:59.687 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:59.687 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:59.687 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:59.687 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:59.687 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63176 2024-09-21T15:49:59.687 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.687 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:59.687 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.687 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:59.687 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140861+0000", 2024-09-21T15:49:59.688 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.688 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.688 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.688 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:59.688 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140861+0000", 2024-09-21T15:49:59.688 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.688 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.688 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.688 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:59.688 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140860+0000", 2024-09-21T15:49:59.688 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:59.688 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.688 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.688 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:59.688 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140863+0000", 2024-09-21T15:49:59.689 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8710000000000001e-06 2024-09-21T15:49:59.689 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.689 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.689 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:59.689 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140871+0000", 2024-09-21T15:49:59.689 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8089999999999993e-06 2024-09-21T15:49:59.689 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.689 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.689 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:59.689 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140877+0000", 2024-09-21T15:49:59.689 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5969999999999999e-06 2024-09-21T15:49:59.689 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.689 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.689 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:59.690 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.194608+0000", 2024-09-21T15:49:59.690 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053731749000000002 2024-09-21T15:49:59.690 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.690 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.690 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:59.690 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.194697+0000", 2024-09-21T15:49:59.690 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8504000000000005e-05 2024-09-21T15:49:59.690 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.690 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.690 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:59.690 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.194796+0000", 2024-09-21T15:49:59.690 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9414999999999996e-05 2024-09-21T15:49:59.690 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.690 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:59.690 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.691 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.691 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.691 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63182 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 157696~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:59.691 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:49:59.691 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.472960455999999, 2024-09-21T15:49:59.691 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053958825000000002, 2024-09-21T15:49:59.691 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:59.691 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:59.691 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:59.691 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:59.691 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:59.691 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63182 2024-09-21T15:49:59.691 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.691 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:59.692 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.692 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:59.692 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:49:59.692 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.692 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.692 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.692 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:59.692 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:49:59.692 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.692 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.692 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.692 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:59.692 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:49:59.692 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:59.692 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.693 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.693 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:59.693 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142136+0000", 2024-09-21T15:49:59.693 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8169999999999999e-06 2024-09-21T15:49:59.693 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.693 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.693 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:59.693 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:49:59.693 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3752e-05 2024-09-21T15:49:59.693 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.693 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.693 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:59.693 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142155+0000", 2024-09-21T15:49:59.693 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1859999999999997e-06 2024-09-21T15:49:59.693 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.694 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.694 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:59.694 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.195914+0000", 2024-09-21T15:49:59.694 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053758528999999999 2024-09-21T15:49:59.694 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.694 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.694 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:59.694 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.195995+0000", 2024-09-21T15:49:59.694 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0983999999999996e-05 2024-09-21T15:49:59.694 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.694 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.694 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:59.694 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.196092+0000", 2024-09-21T15:49:59.694 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7069000000000002e-05 2024-09-21T15:49:59.694 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.695 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:59.695 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.695 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.695 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.695 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63193 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 239616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:49:59.695 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:49:59.695 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.470872871000001, 2024-09-21T15:49:59.695 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053956577999999998, 2024-09-21T15:49:59.695 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:49:59.695 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:49:59.695 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:49:59.695 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:49:59.695 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:49:59.695 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63193 2024-09-21T15:49:59.696 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.696 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:49:59.696 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.696 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:49:59.696 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:49:59.696 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.696 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.696 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.696 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:49:59.696 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:49:59.696 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:49:59.696 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.696 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.696 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:49:59.696 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:49:59.697 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:49:59.697 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.697 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.697 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:49:59.697 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144223+0000", 2024-09-21T15:49:59.697 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0990000000000001e-06 2024-09-21T15:49:59.697 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.697 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.697 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:49:59.697 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144230+0000", 2024-09-21T15:49:59.697 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9070000000000001e-06 2024-09-21T15:49:59.697 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.697 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.697 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:49:59.697 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144234+0000", 2024-09-21T15:49:59.697 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9010000000000002e-06 2024-09-21T15:49:59.698 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.698 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.698 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:49:59.698 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.198004+0000", 2024-09-21T15:49:59.698 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053769539999999998 2024-09-21T15:49:59.698 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.698 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.698 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:49:59.698 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.198083+0000", 2024-09-21T15:49:59.698 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9531999999999997e-05 2024-09-21T15:49:59.698 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:49:59.698 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:49:59.698 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:49:59.698 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.198177+0000", 2024-09-21T15:49:59.699 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3906000000000005e-05 2024-09-21T15:49:59.699 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.699 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:59.699 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.699 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:49:59.699 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:49:59.699 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:59.699 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:59.615+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:59.700 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T15:49:59.717 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:49:59.715+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:49:59.803 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:59.803 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:59.803 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:49:59.804 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:59.812 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T15:49:59.818 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:49:59.817+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:49:59.914 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:49:59.914 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:49:59.914 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:49:59.914 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:49:59.914 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:49:59.918 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:49:59.917+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:49:59.922 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T15:49:59.960 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T15:49:59.980 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:00.000 INFO:teuthology.orchestra.run.smithi089.stdout:4099 write 0xc099a52 thru 0xc09b4a1 (0x1a50 bytes) 2024-09-21T15:50:00.003 INFO:teuthology.orchestra.run.smithi089.stdout:4100 read 0x143029d thru 0x143ccc5 (0xca29 bytes) 2024-09-21T15:50:00.003 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:00.004 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:00.005 INFO:teuthology.orchestra.run.smithi089.stdout:4104 trunc from 0xc09b4a2 to 0xd161900 2024-09-21T15:50:00.011 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:00.012 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:00.012 INFO:teuthology.orchestra.run.smithi089.stdout:4108 read 0x9be822a thru 0x9bf616e (0xdf45 bytes) 2024-09-21T15:50:00.012 INFO:teuthology.orchestra.run.smithi089.stdout:4110 read 0x38a49b1 thru 0x38ac049 (0x7699 bytes) 2024-09-21T15:50:00.012 INFO:teuthology.orchestra.run.smithi089.stdout:4113 read 0x6eaba6d thru 0x6eb069a (0x4c2e bytes) 2024-09-21T15:50:00.012 INFO:teuthology.orchestra.run.smithi089.stdout:4114 punch from 0xb3bf8d7 to 0xb3cdf3a, (0xe663 bytes) 2024-09-21T15:50:00.018 INFO:teuthology.orchestra.run.smithi089.stdout:4115 trunc from 0xd161900 to 0x8949c7f 2024-09-21T15:50:00.018 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:00.017+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:00.028 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:00.028 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:00.028 INFO:teuthology.orchestra.run.smithi089.stdout:4119 read 0x732ef4e thru 0x733ccdd (0xdd90 bytes) 2024-09-21T15:50:00.028 INFO:teuthology.orchestra.run.smithi089.stdout:4121 read 0x3bb0ea0 thru 0x3bba913 (0x9a74 bytes) 2024-09-21T15:50:00.028 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:00.029 INFO:teuthology.orchestra.run.smithi089.stdout:4125 write 0x84df62a thru 0x84ee893 (0xf26a bytes) 2024-09-21T15:50:00.032 INFO:teuthology.orchestra.run.smithi089.stdout:4126 read 0x474c505 thru 0x475725f (0xad5b bytes) 2024-09-21T15:50:00.040 INFO:teuthology.orchestra.run.smithi089.stdout:4128 write 0xa53cb92 thru 0xa53eb37 (0x1fa6 bytes) 2024-09-21T15:50:00.053 INFO:teuthology.orchestra.run.smithi089.stdout:4129 trunc from 0xa53eb38 to 0x38ad4fd 2024-09-21T15:50:00.066 INFO:teuthology.orchestra.run.smithi089.stdout:4131 writesame 0xdce0c2 thru 0xddb4f3 data_size 0x9d(0xd432 bytes) 2024-09-21T15:50:00.069 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:00.070 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:50:00.070 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:50:00.070 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:50:00.070 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.070 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4209.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:50:00.070 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:50:00.070 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 68.877028254999999, 2024-09-21T15:50:00.070 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.19452909900000001, 2024-09-21T15:50:00.070 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:00.070 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:00.070 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:00.070 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4209", 2024-09-21T15:50:00.070 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/4045143050", 2024-09-21T15:50:00.071 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1 2024-09-21T15:50:00.071 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.071 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:00.071 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.071 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:00.071 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:50:00.071 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:00.071 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.071 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.071 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:00.071 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:50:00.071 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:00.071 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.071 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.072 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:00.072 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191161+0000", 2024-09-21T15:50:00.072 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-21T15:50:00.072 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.072 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.072 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:00.072 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191169+0000", 2024-09-21T15:50:00.072 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0090000000000007e-06 2024-09-21T15:50:00.072 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.072 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.072 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:00.072 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191180+0000", 2024-09-21T15:50:00.072 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0728e-05 2024-09-21T15:50:00.073 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.073 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.073 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:00.073 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191187+0000", 2024-09-21T15:50:00.073 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9009999999999994e-06 2024-09-21T15:50:00.073 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.073 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.073 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:00.073 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191238+0000", 2024-09-21T15:50:00.073 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0287000000000002e-05 2024-09-21T15:50:00.073 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.073 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.073 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-21T15:50:00.073 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191245+0000", 2024-09-21T15:50:00.073 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0709999999999997e-06 2024-09-21T15:50:00.074 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.074 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.074 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:00.074 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380309+0000", 2024-09-21T15:50:00.074 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.189064277 2024-09-21T15:50:00.074 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.074 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.074 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:00.074 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380454+0000", 2024-09-21T15:50:00.074 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014490799999999999 2024-09-21T15:50:00.074 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.074 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.074 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0", 2024-09-21T15:50:00.074 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380610+0000", 2024-09-21T15:50:00.074 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015650199999999999 2024-09-21T15:50:00.075 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.075 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.075 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:50:00.075 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.381160+0000", 2024-09-21T15:50:00.075 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00054903799999999996 2024-09-21T15:50:00.075 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.075 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.075 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:00.075 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385591+0000", 2024-09-21T15:50:00.075 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.004431272 2024-09-21T15:50:00.075 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.075 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.075 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:50:00.075 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385651+0000", 2024-09-21T15:50:00.076 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0455999999999999e-05 2024-09-21T15:50:00.076 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.076 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.076 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:00.076 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385691+0000", 2024-09-21T15:50:00.076 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0234999999999997e-05 2024-09-21T15:50:00.076 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:00.076 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:00.076 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:00.076 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.076 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.076 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64028 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1763328~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:00.076 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167504+0000", 2024-09-21T15:50:00.076 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.900686699000001, 2024-09-21T15:50:00.077 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047494153999999997, 2024-09-21T15:50:00.077 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:00.077 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:00.077 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:00.077 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:00.077 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:00.077 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64028 2024-09-21T15:50:00.077 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.077 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:00.077 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.077 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:00.077 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167504+0000", 2024-09-21T15:50:00.077 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:00.077 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.077 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.077 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:00.078 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167504+0000", 2024-09-21T15:50:00.078 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:00.078 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.078 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.078 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:00.078 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167504+0000", 2024-09-21T15:50:00.078 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:00.078 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.078 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.078 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:00.078 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167506+0000", 2024-09-21T15:50:00.078 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0600000000000002e-06 2024-09-21T15:50:00.078 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.078 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.078 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:00.079 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167512+0000", 2024-09-21T15:50:00.079 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.083e-06 2024-09-21T15:50:00.079 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.079 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.079 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:00.079 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167517+0000", 2024-09-21T15:50:00.079 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7650000000000001e-06 2024-09-21T15:50:00.079 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.079 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.079 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:00.079 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.214835+0000", 2024-09-21T15:50:00.079 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047318067999999998 2024-09-21T15:50:00.079 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.079 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.080 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:00.080 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.214916+0000", 2024-09-21T15:50:00.080 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1815000000000002e-05 2024-09-21T15:50:00.080 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.080 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.080 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:00.080 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.214998+0000", 2024-09-21T15:50:00.080 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1642000000000002e-05 2024-09-21T15:50:00.080 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:00.080 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:00.080 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:00.080 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.080 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.080 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64029 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1050624~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:00.080 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167521+0000", 2024-09-21T15:50:00.080 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.900669787999998, 2024-09-21T15:50:00.081 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047649915000000001, 2024-09-21T15:50:00.081 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:00.081 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:00.081 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:00.081 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:00.081 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:00.081 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64029 2024-09-21T15:50:00.081 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.081 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:00.081 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.081 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:00.081 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167521+0000", 2024-09-21T15:50:00.081 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:00.081 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.081 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.081 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:00.082 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167521+0000", 2024-09-21T15:50:00.082 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:00.082 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.082 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.082 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:00.082 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167521+0000", 2024-09-21T15:50:00.082 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:00.082 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.082 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.082 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:00.082 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167523+0000", 2024-09-21T15:50:00.082 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9810000000000002e-06 2024-09-21T15:50:00.082 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.082 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.083 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:00.083 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167529+0000", 2024-09-21T15:50:00.083 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0900000000000001e-06 2024-09-21T15:50:00.083 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.083 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.083 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:00.083 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167533+0000", 2024-09-21T15:50:00.083 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5730000000000001e-06 2024-09-21T15:50:00.083 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.083 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.083 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:00.083 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215031+0000", 2024-09-21T15:50:00.083 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047497893999999999 2024-09-21T15:50:00.083 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.084 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.084 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:00.084 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215099+0000", 2024-09-21T15:50:00.084 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8416000000000004e-05 2024-09-21T15:50:00.084 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.084 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.084 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:00.084 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215171+0000", 2024-09-21T15:50:00.084 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1279000000000007e-05 2024-09-21T15:50:00.084 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:00.084 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:00.084 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:00.084 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.084 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.084 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64030 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1058816~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:00.085 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167537+0000", 2024-09-21T15:50:00.085 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.900653279, 2024-09-21T15:50:00.085 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047783816999999999, 2024-09-21T15:50:00.085 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:00.085 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:00.085 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:00.085 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:00.085 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:00.085 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64030 2024-09-21T15:50:00.085 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.085 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:00.085 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.085 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:00.085 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167537+0000", 2024-09-21T15:50:00.086 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:00.086 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.086 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.086 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:00.086 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167537+0000", 2024-09-21T15:50:00.086 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:00.086 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.086 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.086 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:00.086 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167537+0000", 2024-09-21T15:50:00.086 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:00.086 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.086 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.087 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:00.087 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167539+0000", 2024-09-21T15:50:00.087 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9700000000000002e-06 2024-09-21T15:50:00.087 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.087 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.087 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:00.087 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167545+0000", 2024-09-21T15:50:00.087 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8629999999999996e-06 2024-09-21T15:50:00.087 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.087 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.087 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:00.087 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167549+0000", 2024-09-21T15:50:00.087 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.386e-06 2024-09-21T15:50:00.087 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.088 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.088 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:00.088 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215197+0000", 2024-09-21T15:50:00.088 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047647241 2024-09-21T15:50:00.088 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.088 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.088 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:00.088 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215263+0000", 2024-09-21T15:50:00.088 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6295000000000001e-05 2024-09-21T15:50:00.088 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.088 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.088 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:00.088 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215321+0000", 2024-09-21T15:50:00.089 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8362000000000001e-05 2024-09-21T15:50:00.089 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:00.089 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:00.089 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:00.089 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.089 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.089 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64031 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1738752~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:00.089 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167554+0000", 2024-09-21T15:50:00.089 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.900636775999999, 2024-09-21T15:50:00.089 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047917855000000002, 2024-09-21T15:50:00.089 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:00.089 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:00.089 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:00.089 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:00.090 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:00.090 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64031 2024-09-21T15:50:00.090 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.090 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:00.090 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.090 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:00.090 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167554+0000", 2024-09-21T15:50:00.090 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:00.090 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.090 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.090 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:00.090 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167554+0000", 2024-09-21T15:50:00.090 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:00.090 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.091 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.091 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:00.091 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167554+0000", 2024-09-21T15:50:00.091 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:00.091 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.091 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.091 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:00.091 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167556+0000", 2024-09-21T15:50:00.091 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3870000000000002e-06 2024-09-21T15:50:00.091 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.091 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.091 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:00.091 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167562+0000", 2024-09-21T15:50:00.091 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9100000000000002e-06 2024-09-21T15:50:00.091 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.092 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.092 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:00.092 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167566+0000", 2024-09-21T15:50:00.092 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6029999999999998e-06 2024-09-21T15:50:00.092 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.092 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.092 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:00.092 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215346+0000", 2024-09-21T15:50:00.092 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.04777998 2024-09-21T15:50:00.092 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.092 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.092 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:00.092 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215412+0000", 2024-09-21T15:50:00.092 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5598000000000005e-05 2024-09-21T15:50:00.092 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.093 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.093 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:00.093 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215472+0000", 2024-09-21T15:50:00.093 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9728000000000001e-05 2024-09-21T15:50:00.093 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:00.093 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:00.093 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:00.093 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.093 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.093 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64032 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1067008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:00.093 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167673+0000", 2024-09-21T15:50:00.093 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.900518063, 2024-09-21T15:50:00.093 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047947667999999999, 2024-09-21T15:50:00.094 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:00.094 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:00.094 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:00.094 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:00.094 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:00.094 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64032 2024-09-21T15:50:00.094 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.094 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:00.094 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.094 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:00.094 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167673+0000", 2024-09-21T15:50:00.094 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:00.094 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.094 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.095 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:00.095 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167673+0000", 2024-09-21T15:50:00.095 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:00.095 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.095 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.095 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:00.095 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167672+0000", 2024-09-21T15:50:00.095 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:00.095 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.095 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.095 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:00.095 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167675+0000", 2024-09-21T15:50:00.095 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3780000000000001e-06 2024-09-21T15:50:00.095 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.096 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.096 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:00.096 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167685+0000", 2024-09-21T15:50:00.096 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3039999999999994e-06 2024-09-21T15:50:00.096 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.096 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.096 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:00.096 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167692+0000", 2024-09-21T15:50:00.096 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2559999999999996e-06 2024-09-21T15:50:00.096 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.096 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.096 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:00.096 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215495+0000", 2024-09-21T15:50:00.096 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047803267000000003 2024-09-21T15:50:00.097 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.097 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.097 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:00.097 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215560+0000", 2024-09-21T15:50:00.097 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4703999999999996e-05 2024-09-21T15:50:00.097 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.097 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.097 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:00.097 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215620+0000", 2024-09-21T15:50:00.097 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0328000000000002e-05 2024-09-21T15:50:00.097 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:00.097 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:00.097 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:00.097 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.097 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.098 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64033 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1075200~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:00.098 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167697+0000", 2024-09-21T15:50:00.098 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.900493490999999, 2024-09-21T15:50:00.098 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048082863000000003, 2024-09-21T15:50:00.098 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:00.098 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:00.098 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:00.098 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:00.098 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:00.098 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64033 2024-09-21T15:50:00.098 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.098 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:00.098 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.099 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:00.099 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167697+0000", 2024-09-21T15:50:00.099 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:00.099 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.099 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.099 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:00.099 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167697+0000", 2024-09-21T15:50:00.099 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:00.099 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.099 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.099 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:00.099 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167697+0000", 2024-09-21T15:50:00.099 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:00.099 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.100 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.100 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:00.100 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167699+0000", 2024-09-21T15:50:00.100 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5299999999999999e-06 2024-09-21T15:50:00.100 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.100 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.100 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:00.100 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167706+0000", 2024-09-21T15:50:00.100 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7490000000000001e-06 2024-09-21T15:50:00.100 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.100 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.100 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:00.100 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167712+0000", 2024-09-21T15:50:00.100 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1940000000000003e-06 2024-09-21T15:50:00.101 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.101 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.101 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:00.101 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215643+0000", 2024-09-21T15:50:00.101 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047930647999999999 2024-09-21T15:50:00.101 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.101 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.101 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:00.101 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215713+0000", 2024-09-21T15:50:00.101 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0488999999999995e-05 2024-09-21T15:50:00.101 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.101 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.101 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:00.101 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215780+0000", 2024-09-21T15:50:00.102 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6656999999999996e-05 2024-09-21T15:50:00.102 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:00.102 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:00.102 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:00.102 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.102 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.102 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64034 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1083392~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:00.102 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167717+0000", 2024-09-21T15:50:00.102 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.900473925, 2024-09-21T15:50:00.102 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048233316999999998, 2024-09-21T15:50:00.102 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:00.102 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:00.102 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:00.102 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:00.103 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:00.103 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64034 2024-09-21T15:50:00.103 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.103 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:00.103 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.103 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:00.103 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167717+0000", 2024-09-21T15:50:00.103 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:00.103 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.103 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.103 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:00.103 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167717+0000", 2024-09-21T15:50:00.103 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:00.103 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.104 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.104 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:00.104 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167716+0000", 2024-09-21T15:50:00.104 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:00.104 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.104 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.104 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:00.104 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167718+0000", 2024-09-21T15:50:00.104 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.165e-06 2024-09-21T15:50:00.104 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.104 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.104 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:00.104 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167725+0000", 2024-09-21T15:50:00.104 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3380000000000001e-06 2024-09-21T15:50:00.104 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.105 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.105 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:00.105 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167730+0000", 2024-09-21T15:50:00.105 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8010000000000003e-06 2024-09-21T15:50:00.105 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.105 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.105 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:00.105 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215807+0000", 2024-09-21T15:50:00.105 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048077823999999998 2024-09-21T15:50:00.105 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.105 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.105 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:00.105 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215884+0000", 2024-09-21T15:50:00.105 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6283000000000001e-05 2024-09-21T15:50:00.105 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.105 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.106 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:00.106 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215950+0000", 2024-09-21T15:50:00.106 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6290000000000004e-05 2024-09-21T15:50:00.106 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:00.106 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:00.106 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:00.106 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.106 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.106 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64035 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1091584~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:00.106 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:50:00.106 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.900456712, 2024-09-21T15:50:00.106 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048412117999999997, 2024-09-21T15:50:00.106 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:00.106 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:00.107 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:00.107 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:00.107 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:00.107 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64035 2024-09-21T15:50:00.107 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.107 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:00.107 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.107 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:00.107 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:50:00.107 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:00.107 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.107 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.107 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:00.107 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:50:00.108 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:00.108 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.108 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.108 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:00.108 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:50:00.108 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:00.108 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.108 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.108 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:00.108 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167746+0000", 2024-09-21T15:50:00.108 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2031000000000001e-05 2024-09-21T15:50:00.108 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.108 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.109 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:00.109 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167753+0000", 2024-09-21T15:50:00.109 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0620000000000003e-06 2024-09-21T15:50:00.109 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.109 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.109 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:00.109 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167759+0000", 2024-09-21T15:50:00.109 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.596e-06 2024-09-21T15:50:00.109 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.109 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.109 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:00.109 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215977+0000", 2024-09-21T15:50:00.109 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048217318000000002 2024-09-21T15:50:00.109 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.109 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.110 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:00.110 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216076+0000", 2024-09-21T15:50:00.110 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9846000000000006e-05 2024-09-21T15:50:00.110 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.110 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.110 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:00.110 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216146+0000", 2024-09-21T15:50:00.110 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9578000000000004e-05 2024-09-21T15:50:00.110 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:00.110 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:00.110 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:00.110 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.110 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.111 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64036 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1099776~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:00.111 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167853+0000", 2024-09-21T15:50:00.111 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.900338098999999, 2024-09-21T15:50:00.111 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048450675999999998, 2024-09-21T15:50:00.111 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:00.111 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:00.111 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:00.111 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:00.111 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:00.111 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64036 2024-09-21T15:50:00.111 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.111 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:00.111 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.111 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:00.112 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167853+0000", 2024-09-21T15:50:00.112 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:00.112 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.112 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.112 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:00.112 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167853+0000", 2024-09-21T15:50:00.112 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:00.112 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.112 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.112 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:00.112 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167852+0000", 2024-09-21T15:50:00.112 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:00.112 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.112 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.113 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:00.113 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167855+0000", 2024-09-21T15:50:00.113 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5640000000000001e-06 2024-09-21T15:50:00.113 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.113 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.113 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:00.113 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167864+0000", 2024-09-21T15:50:00.113 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8140000000000007e-06 2024-09-21T15:50:00.113 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.113 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.113 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:00.113 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167871+0000", 2024-09-21T15:50:00.113 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5729999999999996e-06 2024-09-21T15:50:00.113 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.114 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.114 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:00.114 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216174+0000", 2024-09-21T15:50:00.114 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048303437999999997 2024-09-21T15:50:00.114 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.114 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.114 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:00.114 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216250+0000", 2024-09-21T15:50:00.114 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5661999999999995e-05 2024-09-21T15:50:00.114 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.114 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.114 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:00.114 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216303+0000", 2024-09-21T15:50:00.114 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3273000000000003e-05 2024-09-21T15:50:00.115 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:00.115 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:00.115 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:00.115 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.115 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.115 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64037 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1771520~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:00.115 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:50:00.115 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.900314556000001, 2024-09-21T15:50:00.115 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048588771000000003, 2024-09-21T15:50:00.115 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:00.115 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:00.115 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:00.115 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:00.115 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:00.116 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64037 2024-09-21T15:50:00.116 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.116 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:00.116 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.116 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:00.116 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:50:00.116 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:00.116 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.116 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.116 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:00.116 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:50:00.116 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:00.116 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.116 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.117 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:00.117 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:50:00.117 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:00.117 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.117 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.117 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:00.117 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167878+0000", 2024-09-21T15:50:00.117 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3e-06 2024-09-21T15:50:00.117 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.117 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.117 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:00.117 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167884+0000", 2024-09-21T15:50:00.117 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3269999999999997e-06 2024-09-21T15:50:00.117 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.118 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.118 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:00.118 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167889+0000", 2024-09-21T15:50:00.118 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9269999999999996e-06 2024-09-21T15:50:00.118 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.118 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.118 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:00.118 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216331+0000", 2024-09-21T15:50:00.118 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048442102000000001 2024-09-21T15:50:00.118 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.118 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.118 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:00.118 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216393+0000", 2024-09-21T15:50:00.118 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1132999999999999e-05 2024-09-21T15:50:00.119 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.119 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.119 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:00.119 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216465+0000", 2024-09-21T15:50:00.119 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2339e-05 2024-09-21T15:50:00.119 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:00.119 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:00.119 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:00.119 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.119 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.119 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64038 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1779712~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:00.119 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:50:00.119 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.900296725, 2024-09-21T15:50:00.120 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048749795999999998, 2024-09-21T15:50:00.120 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:00.120 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:00.120 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:00.120 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:00.120 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:00.120 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64038 2024-09-21T15:50:00.120 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.120 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:00.120 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.120 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:00.120 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:50:00.120 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:00.120 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.121 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.121 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:00.121 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:50:00.121 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:00.121 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.121 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.121 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:00.121 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:50:00.121 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:00.121 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.121 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.121 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:00.121 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167896+0000", 2024-09-21T15:50:00.121 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2419999999999999e-06 2024-09-21T15:50:00.122 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.122 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.122 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:00.122 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167902+0000", 2024-09-21T15:50:00.122 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0859999999999998e-06 2024-09-21T15:50:00.122 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.122 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.122 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:00.122 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167907+0000", 2024-09-21T15:50:00.122 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9679999999999997e-06 2024-09-21T15:50:00.122 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.122 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.122 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:00.122 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216506+0000", 2024-09-21T15:50:00.123 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048599374000000001 2024-09-21T15:50:00.123 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.123 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.123 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:00.123 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216579+0000", 2024-09-21T15:50:00.123 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2801999999999999e-05 2024-09-21T15:50:00.123 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.123 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.123 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:00.123 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216644+0000", 2024-09-21T15:50:00.123 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4715000000000006e-05 2024-09-21T15:50:00.123 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:00.123 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:00.123 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:00.124 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.124 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.124 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64039 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1107968~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:00.124 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:50:00.124 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.90027937, 2024-09-21T15:50:00.124 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048911370000000003, 2024-09-21T15:50:00.124 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:00.124 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:00.124 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:00.124 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:00.124 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:00.124 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64039 2024-09-21T15:50:00.124 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.125 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:00.125 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.125 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:00.125 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:50:00.125 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:00.125 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.125 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.125 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:00.125 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:50:00.125 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:00.125 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.125 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.125 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:00.125 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:50:00.126 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:00.126 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.126 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.126 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:00.126 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167913+0000", 2024-09-21T15:50:00.126 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0839999999999999e-06 2024-09-21T15:50:00.126 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.126 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.126 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:00.126 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167921+0000", 2024-09-21T15:50:00.126 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8310000000000001e-06 2024-09-21T15:50:00.126 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.126 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.126 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:00.127 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167926+0000", 2024-09-21T15:50:00.127 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0270000000000003e-06 2024-09-21T15:50:00.127 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.127 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.127 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:00.127 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216680+0000", 2024-09-21T15:50:00.127 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048753890000000001 2024-09-21T15:50:00.127 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.127 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.127 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:00.127 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216757+0000", 2024-09-21T15:50:00.127 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6848999999999996e-05 2024-09-21T15:50:00.127 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.128 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.128 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:00.128 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216823+0000", 2024-09-21T15:50:00.128 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6068000000000005e-05 2024-09-21T15:50:00.128 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:00.128 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:00.128 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:00.128 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.128 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.128 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64040 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1787904~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:00.128 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167931+0000", 2024-09-21T15:50:00.128 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.900260138, 2024-09-21T15:50:00.128 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049010561000000001, 2024-09-21T15:50:00.128 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:00.129 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:00.129 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:00.129 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:00.129 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:00.129 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64040 2024-09-21T15:50:00.129 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.129 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:00.129 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.129 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:00.129 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167931+0000", 2024-09-21T15:50:00.129 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:00.129 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.129 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.129 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:00.130 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167931+0000", 2024-09-21T15:50:00.130 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:00.130 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.130 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.130 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:00.130 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167930+0000", 2024-09-21T15:50:00.130 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:00.130 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.130 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.130 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:00.130 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167932+0000", 2024-09-21T15:50:00.131 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.198e-06 2024-09-21T15:50:00.131 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.131 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.131 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:00.131 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167938+0000", 2024-09-21T15:50:00.131 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.105e-06 2024-09-21T15:50:00.131 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.131 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.131 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:00.131 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167943+0000", 2024-09-21T15:50:00.131 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9429999999999999e-06 2024-09-21T15:50:00.131 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.131 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.131 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:00.132 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216848+0000", 2024-09-21T15:50:00.132 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048905200000000003 2024-09-21T15:50:00.132 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.132 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.132 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:00.132 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216903+0000", 2024-09-21T15:50:00.132 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4793999999999997e-05 2024-09-21T15:50:00.132 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.132 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.132 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:00.132 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216941+0000", 2024-09-21T15:50:00.132 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7790999999999998e-05 2024-09-21T15:50:00.132 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:00.132 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:00.133 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:00.133 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.133 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.133 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64041 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1796096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:00.133 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:50:00.133 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.900243166999999, 2024-09-21T15:50:00.133 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049181457999999997, 2024-09-21T15:50:00.133 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:00.133 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:00.133 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:00.133 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:00.133 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:00.133 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64041 2024-09-21T15:50:00.133 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.134 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:00.134 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.134 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:00.134 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:50:00.134 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:00.134 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.134 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.134 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:00.134 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:50:00.134 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:00.134 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.134 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.134 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:00.135 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:50:00.135 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:00.135 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.135 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.135 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:00.135 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167949+0000", 2024-09-21T15:50:00.135 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0200000000000001e-06 2024-09-21T15:50:00.135 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.135 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.135 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:00.135 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167955+0000", 2024-09-21T15:50:00.135 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5990000000000001e-06 2024-09-21T15:50:00.135 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.135 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.136 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:00.136 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167959+0000", 2024-09-21T15:50:00.136 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5589999999999999e-06 2024-09-21T15:50:00.136 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.136 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.136 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:00.136 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216972+0000", 2024-09-21T15:50:00.136 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049012574000000003 2024-09-21T15:50:00.136 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.136 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.136 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:00.136 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217057+0000", 2024-09-21T15:50:00.136 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4820000000000004e-05 2024-09-21T15:50:00.136 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.137 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.137 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:00.137 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217129+0000", 2024-09-21T15:50:00.137 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2186000000000003e-05 2024-09-21T15:50:00.137 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:00.137 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:00.137 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:00.137 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.137 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.137 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64042 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1804288~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:00.137 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167964+0000", 2024-09-21T15:50:00.137 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.900226876000001, 2024-09-21T15:50:00.137 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049333372, 2024-09-21T15:50:00.137 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:00.138 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:00.138 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:00.138 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:00.138 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:00.138 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64042 2024-09-21T15:50:00.138 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.138 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:00.138 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.138 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:00.138 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167964+0000", 2024-09-21T15:50:00.138 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:00.138 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.138 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.138 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:00.139 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167964+0000", 2024-09-21T15:50:00.139 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:00.139 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.139 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.139 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:00.139 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167963+0000", 2024-09-21T15:50:00.139 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:00.139 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.139 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.139 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:00.139 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167966+0000", 2024-09-21T15:50:00.139 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1119999999999999e-06 2024-09-21T15:50:00.139 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.139 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.140 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:00.140 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167975+0000", 2024-09-21T15:50:00.140 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1130000000000007e-06 2024-09-21T15:50:00.140 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.140 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.140 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:00.140 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167980+0000", 2024-09-21T15:50:00.140 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9599999999999999e-06 2024-09-21T15:50:00.140 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.140 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.140 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:00.140 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217158+0000", 2024-09-21T15:50:00.140 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049178569999999998 2024-09-21T15:50:00.140 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.140 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.141 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:00.141 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217232+0000", 2024-09-21T15:50:00.141 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3589000000000003e-05 2024-09-21T15:50:00.141 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.141 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.141 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:00.141 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217297+0000", 2024-09-21T15:50:00.141 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5339999999999994e-05 2024-09-21T15:50:00.141 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:00.141 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:00.141 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:00.141 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.141 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.141 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64043 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1812480~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:00.141 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.168085+0000", 2024-09-21T15:50:00.142 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.900105808999999, 2024-09-21T15:50:00.142 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049365157999999999, 2024-09-21T15:50:00.142 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:00.142 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:00.142 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:00.142 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:00.142 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:00.142 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64043 2024-09-21T15:50:00.142 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.142 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:00.142 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.142 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:00.142 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168085+0000", 2024-09-21T15:50:00.143 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:00.143 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.143 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.143 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:00.143 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168085+0000", 2024-09-21T15:50:00.143 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:00.143 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.143 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.143 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:00.143 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168084+0000", 2024-09-21T15:50:00.143 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:00.143 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.143 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.143 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:00.144 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168088+0000", 2024-09-21T15:50:00.144 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5420000000000001e-06 2024-09-21T15:50:00.144 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.144 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.144 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:00.144 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168097+0000", 2024-09-21T15:50:00.144 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5410000000000006e-06 2024-09-21T15:50:00.144 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.144 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.144 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:00.144 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168104+0000", 2024-09-21T15:50:00.144 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6719999999999998e-06 2024-09-21T15:50:00.144 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.144 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.145 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:00.145 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217323+0000", 2024-09-21T15:50:00.145 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049218857999999997 2024-09-21T15:50:00.145 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.145 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.145 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:00.145 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217397+0000", 2024-09-21T15:50:00.145 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3932999999999998e-05 2024-09-21T15:50:00.145 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.145 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.145 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:00.145 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217450+0000", 2024-09-21T15:50:00.145 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3168000000000003e-05 2024-09-21T15:50:00.145 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:00.146 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:00.146 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:00.146 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.146 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.146 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64044 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1820672~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:00.146 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.170036+0000", 2024-09-21T15:50:00.146 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.89815449, 2024-09-21T15:50:00.146 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047578069000000001, 2024-09-21T15:50:00.146 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:00.146 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:00.146 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:00.146 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:00.146 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:00.146 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64044 2024-09-21T15:50:00.146 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.147 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:00.147 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.147 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:00.147 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170036+0000", 2024-09-21T15:50:00.147 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:00.147 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.147 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.147 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:00.147 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170036+0000", 2024-09-21T15:50:00.147 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:00.147 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.147 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.147 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:00.147 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168109+0000", 2024-09-21T15:50:00.148 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9980721 2024-09-21T15:50:00.148 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.148 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.148 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:00.148 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170040+0000", 2024-09-21T15:50:00.148 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0019318079999999999 2024-09-21T15:50:00.148 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.148 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.148 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:00.148 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170051+0000", 2024-09-21T15:50:00.148 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0220999999999999e-05 2024-09-21T15:50:00.148 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.148 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.148 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:00.149 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170058+0000", 2024-09-21T15:50:00.149 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0110000000000003e-06 2024-09-21T15:50:00.149 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.149 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.149 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:00.149 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217478+0000", 2024-09-21T15:50:00.149 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047420334000000001 2024-09-21T15:50:00.149 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.149 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.149 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:00.149 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217547+0000", 2024-09-21T15:50:00.149 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9416000000000001e-05 2024-09-21T15:50:00.149 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.149 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.149 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:00.149 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217614+0000", 2024-09-21T15:50:00.150 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6909000000000006e-05 2024-09-21T15:50:00.150 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:00.150 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:00.150 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:00.150 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.150 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.150 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64045 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1828864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:00.150 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.170064+0000", 2024-09-21T15:50:00.150 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.898126451, 2024-09-21T15:50:00.150 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047709252000000001, 2024-09-21T15:50:00.150 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:00.150 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:00.150 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:00.150 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:00.151 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:00.151 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64045 2024-09-21T15:50:00.151 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.151 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:00.151 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.151 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:00.151 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170064+0000", 2024-09-21T15:50:00.151 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:00.151 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.151 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.151 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:00.151 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170064+0000", 2024-09-21T15:50:00.151 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:00.151 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.152 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.152 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:00.152 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170064+0000", 2024-09-21T15:50:00.152 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:00.152 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.152 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.152 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:00.152 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170066+0000", 2024-09-21T15:50:00.152 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8109999999999999e-06 2024-09-21T15:50:00.152 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.152 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.152 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:00.152 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170073+0000", 2024-09-21T15:50:00.152 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5570000000000001e-06 2024-09-21T15:50:00.152 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.152 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.153 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:00.153 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170078+0000", 2024-09-21T15:50:00.153 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9910000000000002e-06 2024-09-21T15:50:00.153 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.153 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.153 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:00.153 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217640+0000", 2024-09-21T15:50:00.153 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047561840000000001 2024-09-21T15:50:00.153 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.153 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.153 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:00.153 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217710+0000", 2024-09-21T15:50:00.153 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9967000000000003e-05 2024-09-21T15:50:00.153 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.153 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.153 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:00.154 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217773+0000", 2024-09-21T15:50:00.154 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3638000000000003e-05 2024-09-21T15:50:00.154 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:00.154 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:00.154 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:00.154 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.154 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.154 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64356 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 3076096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:00.154 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.183065+0000", 2024-09-21T15:50:00.154 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 22.88512532, 2024-09-21T15:50:00.154 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047484923999999998, 2024-09-21T15:50:00.154 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:00.154 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:00.155 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:00.155 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:00.155 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:00.155 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64356 2024-09-21T15:50:00.155 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.155 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:00.155 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.155 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:00.155 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.183065+0000", 2024-09-21T15:50:00.155 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:00.155 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.155 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.155 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:00.155 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.183065+0000", 2024-09-21T15:50:00.155 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:00.156 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.156 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.156 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:00.156 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.182922+0000", 2024-09-21T15:50:00.156 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999856 2024-09-21T15:50:00.156 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.156 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.156 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:00.156 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.183069+0000", 2024-09-21T15:50:00.156 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000147097 2024-09-21T15:50:00.156 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.156 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.156 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:00.156 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.183078+0000", 2024-09-21T15:50:00.156 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.9099999999999994e-06 2024-09-21T15:50:00.157 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.157 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.157 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:00.157 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.183085+0000", 2024-09-21T15:50:00.157 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0670000000000003e-06 2024-09-21T15:50:00.157 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.157 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.157 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:00.157 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.230377+0000", 2024-09-21T15:50:00.157 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047292698000000001 2024-09-21T15:50:00.157 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.157 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.157 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:00.157 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.230458+0000", 2024-09-21T15:50:00.157 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0980000000000001e-05 2024-09-21T15:50:00.158 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:00.158 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:00.158 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:00.158 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.230550+0000", 2024-09-21T15:50:00.158 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1954999999999996e-05 2024-09-21T15:50:00.158 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:00.158 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:00.158 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:00.158 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:00.158 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:00.158 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:00.160 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:00.158+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:00.160 INFO:teuthology.orchestra.run.smithi089.stdout:4132 write 0x3d39607 thru 0x3d4664a (0xd044 bytes) 2024-09-21T15:50:00.160 INFO:teuthology.orchestra.run.smithi089.stdout:4133 write 0x3bfd91a thru 0x3c080d0 (0xa7b7 bytes) 2024-09-21T15:50:00.160 INFO:teuthology.orchestra.run.smithi089.stdout:4135 read 0x15734 thru 0x222c4 (0xcb91 bytes) 2024-09-21T15:50:00.160 INFO:teuthology.orchestra.run.smithi089.stdout:4136 punch from 0x1a9bac3 to 0x1aa47b1, (0x8cee bytes) 2024-09-21T15:50:00.160 INFO:teuthology.orchestra.run.smithi089.stdout:4137 trunc from 0x3d4664b to 0x632fb89 2024-09-21T15:50:00.161 INFO:teuthology.orchestra.run.smithi089.stdout:4138 write 0xe2f0afc thru 0xe2f842f (0x7934 bytes) 2024-09-21T15:50:00.161 INFO:teuthology.orchestra.run.smithi089.stdout:4139 clone 27 order 19 su 2048 sc 10 2024-09-21T15:50:00.161 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:50:00.260 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:00.260+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:00.266 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:00.266 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:00.266 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:50:00.267 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:00.275 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:50:00.361 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:00.361+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:00.377 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:00.377 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:00.377 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:50:00.377 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:50:00.377 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:00.386 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:50:00.463 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:00.463+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:00.489 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:00.489 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:50:00.489 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:50:00.489 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:50:00.489 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.490 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65283 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2836480~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:00.490 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.218024+0000", 2024-09-21T15:50:00.490 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.269337457999999, 2024-09-21T15:50:00.490 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043727861, 2024-09-21T15:50:00.490 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:00.490 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:00.490 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:00.490 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:00.490 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:00.490 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65283 2024-09-21T15:50:00.490 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.490 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:00.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.491 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:00.491 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218024+0000", 2024-09-21T15:50:00.491 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:00.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.491 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:00.491 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218024+0000", 2024-09-21T15:50:00.491 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:00.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.491 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:00.491 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218024+0000", 2024-09-21T15:50:00.491 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:00.492 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.492 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.492 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:00.492 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218026+0000", 2024-09-21T15:50:00.492 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.717e-06 2024-09-21T15:50:00.492 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.492 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.492 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:00.492 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218030+0000", 2024-09-21T15:50:00.492 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8630000000000001e-06 2024-09-21T15:50:00.492 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.492 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.492 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:00.492 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218033+0000", 2024-09-21T15:50:00.493 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9660000000000001e-06 2024-09-21T15:50:00.493 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.493 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.493 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:00.493 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.261601+0000", 2024-09-21T15:50:00.493 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043567862999999998 2024-09-21T15:50:00.493 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.493 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.493 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:00.493 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.261679+0000", 2024-09-21T15:50:00.493 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7807999999999998e-05 2024-09-21T15:50:00.493 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.493 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.493 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:00.494 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.261752+0000", 2024-09-21T15:50:00.494 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2977000000000003e-05 2024-09-21T15:50:00.494 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:00.494 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:00.494 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:00.494 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.494 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.494 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65284 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2844672~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:00.494 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.218037+0000", 2024-09-21T15:50:00.494 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.269324534999999, 2024-09-21T15:50:00.494 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043873034999999998, 2024-09-21T15:50:00.494 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:00.495 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:00.495 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:00.495 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:00.495 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:00.495 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65284 2024-09-21T15:50:00.495 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.495 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:00.495 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.495 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:00.495 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218037+0000", 2024-09-21T15:50:00.495 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:00.495 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.495 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.496 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:00.496 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218037+0000", 2024-09-21T15:50:00.496 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:00.496 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.496 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.496 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:00.496 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218037+0000", 2024-09-21T15:50:00.496 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:00.496 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.496 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.496 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:00.496 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218038+0000", 2024-09-21T15:50:00.496 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.533e-06 2024-09-21T15:50:00.496 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.496 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.497 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:00.497 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218041+0000", 2024-09-21T15:50:00.497 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9799999999999998e-06 2024-09-21T15:50:00.497 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.497 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.497 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:00.497 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218044+0000", 2024-09-21T15:50:00.497 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.458e-06 2024-09-21T15:50:00.497 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.497 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.497 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:00.497 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.261781+0000", 2024-09-21T15:50:00.497 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043736935999999997 2024-09-21T15:50:00.497 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.497 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.497 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:00.498 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.261846+0000", 2024-09-21T15:50:00.498 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5233000000000004e-05 2024-09-21T15:50:00.498 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.498 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.498 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:00.498 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.261910+0000", 2024-09-21T15:50:00.498 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4103000000000006e-05 2024-09-21T15:50:00.498 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:00.498 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:00.498 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:00.498 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.498 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.498 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65285 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2861056~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:00.498 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.218171+0000", 2024-09-21T15:50:00.498 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.269190387999998, 2024-09-21T15:50:00.499 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043909057000000001, 2024-09-21T15:50:00.499 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:00.499 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:00.499 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:00.499 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:00.499 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:00.499 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65285 2024-09-21T15:50:00.499 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.499 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:00.499 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.499 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:00.499 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218171+0000", 2024-09-21T15:50:00.499 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:00.499 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.499 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.500 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:00.500 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218171+0000", 2024-09-21T15:50:00.500 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:00.500 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.500 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.500 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:00.500 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218171+0000", 2024-09-21T15:50:00.500 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:00.500 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.500 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.500 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:00.500 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218173+0000", 2024-09-21T15:50:00.500 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.588e-06 2024-09-21T15:50:00.500 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.501 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.501 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:00.501 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218178+0000", 2024-09-21T15:50:00.501 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4719999999999997e-06 2024-09-21T15:50:00.501 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.501 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.501 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:00.501 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218181+0000", 2024-09-21T15:50:00.501 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0979999999999998e-06 2024-09-21T15:50:00.501 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.501 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.501 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:00.501 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.261943+0000", 2024-09-21T15:50:00.502 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043762652999999999 2024-09-21T15:50:00.502 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.502 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.502 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:00.502 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262018+0000", 2024-09-21T15:50:00.502 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4352999999999998e-05 2024-09-21T15:50:00.502 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.502 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.502 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:00.502 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262080+0000", 2024-09-21T15:50:00.502 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2477999999999994e-05 2024-09-21T15:50:00.502 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:00.502 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:00.502 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:00.502 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.503 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.503 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65286 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2869248~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:00.503 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.218284+0000", 2024-09-21T15:50:00.503 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.269077905, 2024-09-21T15:50:00.503 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043921241999999999, 2024-09-21T15:50:00.503 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:00.503 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:00.503 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:00.503 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:00.503 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:00.503 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65286 2024-09-21T15:50:00.503 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.503 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:00.503 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.504 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:00.504 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218284+0000", 2024-09-21T15:50:00.504 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:00.504 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.504 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.504 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:00.504 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218284+0000", 2024-09-21T15:50:00.504 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:00.504 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.504 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.504 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:00.504 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218283+0000", 2024-09-21T15:50:00.504 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:00.504 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.505 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.505 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:00.505 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218286+0000", 2024-09-21T15:50:00.505 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4159999999999998e-06 2024-09-21T15:50:00.505 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.505 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.505 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:00.505 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218290+0000", 2024-09-21T15:50:00.505 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3370000000000001e-06 2024-09-21T15:50:00.505 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.505 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.505 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:00.505 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218293+0000", 2024-09-21T15:50:00.505 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0960000000000001e-06 2024-09-21T15:50:00.505 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.506 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.506 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:00.506 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262105+0000", 2024-09-21T15:50:00.506 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043811914 2024-09-21T15:50:00.506 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.506 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.506 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:00.506 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262152+0000", 2024-09-21T15:50:00.506 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7341e-05 2024-09-21T15:50:00.506 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.506 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.506 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:00.506 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262205+0000", 2024-09-21T15:50:00.506 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2667000000000002e-05 2024-09-21T15:50:00.507 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:00.507 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:00.507 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:00.507 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.507 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.507 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65287 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2877440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:00.507 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.218326+0000", 2024-09-21T15:50:00.507 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.269035641999999, 2024-09-21T15:50:00.507 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.044001874000000003, 2024-09-21T15:50:00.507 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:00.507 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:00.507 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:00.507 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:00.508 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:00.508 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65287 2024-09-21T15:50:00.508 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.508 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:00.508 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.508 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:00.508 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218326+0000", 2024-09-21T15:50:00.508 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:00.508 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.508 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.508 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:00.508 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218326+0000", 2024-09-21T15:50:00.508 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:00.508 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.508 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.509 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:00.509 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218326+0000", 2024-09-21T15:50:00.509 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:00.509 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.509 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.509 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:00.509 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218328+0000", 2024-09-21T15:50:00.509 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.029e-06 2024-09-21T15:50:00.509 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.509 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.509 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:00.509 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218334+0000", 2024-09-21T15:50:00.509 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9519999999999999e-06 2024-09-21T15:50:00.509 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.509 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.509 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:00.510 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218338+0000", 2024-09-21T15:50:00.510 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7500000000000003e-06 2024-09-21T15:50:00.510 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.510 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.510 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:00.510 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262228+0000", 2024-09-21T15:50:00.510 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043890058000000003 2024-09-21T15:50:00.510 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.510 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.510 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:00.510 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262285+0000", 2024-09-21T15:50:00.510 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6779e-05 2024-09-21T15:50:00.510 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.510 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.510 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:00.511 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262328+0000", 2024-09-21T15:50:00.511 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2599999999999999e-05 2024-09-21T15:50:00.511 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:00.511 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:00.511 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:00.511 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.511 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.511 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65288 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2885632~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:00.511 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.218345+0000", 2024-09-21T15:50:00.511 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.269016910000001, 2024-09-21T15:50:00.511 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.044143245999999997, 2024-09-21T15:50:00.511 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:00.511 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:00.511 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:00.511 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:00.511 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:00.512 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65288 2024-09-21T15:50:00.512 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.512 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:00.512 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.512 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:00.512 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218345+0000", 2024-09-21T15:50:00.512 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:00.512 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.512 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.512 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:00.512 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218345+0000", 2024-09-21T15:50:00.512 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:00.512 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.512 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.512 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:00.512 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218344+0000", 2024-09-21T15:50:00.512 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:00.513 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.513 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.513 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:00.513 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218347+0000", 2024-09-21T15:50:00.513 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3920000000000001e-06 2024-09-21T15:50:00.513 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.513 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.513 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:00.513 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218353+0000", 2024-09-21T15:50:00.513 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9610000000000002e-06 2024-09-21T15:50:00.513 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.513 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.513 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:00.513 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218357+0000", 2024-09-21T15:50:00.513 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4700000000000004e-06 2024-09-21T15:50:00.514 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.514 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.514 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:00.514 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262405+0000", 2024-09-21T15:50:00.514 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.044047500000000003 2024-09-21T15:50:00.514 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.514 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.514 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:00.514 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262446+0000", 2024-09-21T15:50:00.514 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1151e-05 2024-09-21T15:50:00.514 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.514 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.514 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:00.514 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262488+0000", 2024-09-21T15:50:00.514 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2182999999999999e-05 2024-09-21T15:50:00.515 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:00.515 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:00.515 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:00.515 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.515 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.515 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65289 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2893824~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:00.515 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.218394+0000", 2024-09-21T15:50:00.515 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.268967707000002, 2024-09-21T15:50:00.515 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.044235021999999999, 2024-09-21T15:50:00.515 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:00.515 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:00.515 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:00.515 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:00.515 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:00.516 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65289 2024-09-21T15:50:00.516 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.516 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:00.516 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.516 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:00.516 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218394+0000", 2024-09-21T15:50:00.516 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:00.516 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.516 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.516 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:00.516 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218394+0000", 2024-09-21T15:50:00.516 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:00.516 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.516 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.516 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:00.517 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218393+0000", 2024-09-21T15:50:00.517 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:00.517 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.517 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.517 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:00.517 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218397+0000", 2024-09-21T15:50:00.517 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.27e-06 2024-09-21T15:50:00.517 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.517 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.517 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:00.517 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218404+0000", 2024-09-21T15:50:00.517 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6329999999999996e-06 2024-09-21T15:50:00.517 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.517 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.518 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:00.518 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218415+0000", 2024-09-21T15:50:00.518 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0373999999999999e-05 2024-09-21T15:50:00.518 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.518 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.518 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:00.518 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262511+0000", 2024-09-21T15:50:00.518 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.044096630999999997 2024-09-21T15:50:00.518 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.518 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.518 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:00.518 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262562+0000", 2024-09-21T15:50:00.518 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1266e-05 2024-09-21T15:50:00.518 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.519 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.519 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:00.519 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262629+0000", 2024-09-21T15:50:00.519 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6388000000000002e-05 2024-09-21T15:50:00.519 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:00.519 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:00.519 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:00.519 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.519 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.519 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65290 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2852864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:00.519 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.218451+0000", 2024-09-21T15:50:00.519 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.268910099999999, 2024-09-21T15:50:00.519 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.044280721000000002, 2024-09-21T15:50:00.519 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:00.520 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:00.520 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:00.520 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:00.520 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:00.520 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65290 2024-09-21T15:50:00.520 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.520 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:00.520 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.520 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:00.520 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218451+0000", 2024-09-21T15:50:00.520 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:00.520 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.520 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.520 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:00.520 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218451+0000", 2024-09-21T15:50:00.521 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:00.521 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.521 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.521 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:00.521 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218451+0000", 2024-09-21T15:50:00.521 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:00.521 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.521 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.521 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:00.521 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218454+0000", 2024-09-21T15:50:00.521 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7489999999999999e-06 2024-09-21T15:50:00.521 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.521 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.521 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:00.521 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218462+0000", 2024-09-21T15:50:00.522 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5329999999999994e-06 2024-09-21T15:50:00.522 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.522 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.522 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:00.522 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218465+0000", 2024-09-21T15:50:00.522 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1990000000000002e-06 2024-09-21T15:50:00.522 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.522 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.522 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:00.522 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262655+0000", 2024-09-21T15:50:00.522 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.044189472 2024-09-21T15:50:00.522 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.522 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.522 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:00.523 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262695+0000", 2024-09-21T15:50:00.523 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0476999999999999e-05 2024-09-21T15:50:00.523 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.523 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.523 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:00.523 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262732+0000", 2024-09-21T15:50:00.523 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6795000000000003e-05 2024-09-21T15:50:00.523 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:00.523 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:00.523 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:00.523 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.523 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.523 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65291 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2902016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:00.523 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.218644+0000", 2024-09-21T15:50:00.523 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.268717584000001, 2024-09-21T15:50:00.524 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.044258391000000001, 2024-09-21T15:50:00.524 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:00.524 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:00.524 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:00.524 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:00.524 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:00.524 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65291 2024-09-21T15:50:00.524 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.524 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:00.524 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.524 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:00.524 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218644+0000", 2024-09-21T15:50:00.524 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:00.524 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.524 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.525 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:00.525 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218644+0000", 2024-09-21T15:50:00.525 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:00.525 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.525 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.525 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:00.525 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218468+0000", 2024-09-21T15:50:00.525 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999824 2024-09-21T15:50:00.525 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.525 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.525 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:00.525 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218649+0000", 2024-09-21T15:50:00.525 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00018061999999999999 2024-09-21T15:50:00.525 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.525 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.526 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:00.526 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218657+0000", 2024-09-21T15:50:00.526 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.6430000000000001e-06 2024-09-21T15:50:00.526 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.526 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.526 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:00.526 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218662+0000", 2024-09-21T15:50:00.526 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.814e-06 2024-09-21T15:50:00.526 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.526 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.526 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:00.526 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262766+0000", 2024-09-21T15:50:00.526 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.044103705999999999 2024-09-21T15:50:00.526 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.526 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.527 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:00.527 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262838+0000", 2024-09-21T15:50:00.527 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2312999999999997e-05 2024-09-21T15:50:00.527 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.527 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.527 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:00.527 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262902+0000", 2024-09-21T15:50:00.527 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4282999999999994e-05 2024-09-21T15:50:00.527 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:00.527 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:00.527 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:00.527 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.527 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.527 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65292 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2910208~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:00.528 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.218810+0000", 2024-09-21T15:50:00.528 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.268551372000001, 2024-09-21T15:50:00.528 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.044257131999999998, 2024-09-21T15:50:00.528 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:00.528 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:00.528 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:00.528 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:00.528 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:00.528 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65292 2024-09-21T15:50:00.528 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.528 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:00.528 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.528 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:00.528 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218810+0000", 2024-09-21T15:50:00.528 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:00.529 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.529 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.529 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:00.529 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218810+0000", 2024-09-21T15:50:00.529 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:00.529 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.529 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.529 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:00.529 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218666+0000", 2024-09-21T15:50:00.529 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999856 2024-09-21T15:50:00.529 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.529 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.529 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:00.529 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218814+0000", 2024-09-21T15:50:00.529 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00014768299999999999 2024-09-21T15:50:00.530 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.530 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.530 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:00.530 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218823+0000", 2024-09-21T15:50:00.530 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2399999999999996e-06 2024-09-21T15:50:00.530 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.530 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.530 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:00.530 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218829+0000", 2024-09-21T15:50:00.530 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8279999999999999e-06 2024-09-21T15:50:00.530 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.530 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.530 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:00.530 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.262933+0000", 2024-09-21T15:50:00.531 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.044104065999999997 2024-09-21T15:50:00.531 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.531 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.531 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:00.531 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263014+0000", 2024-09-21T15:50:00.531 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0717999999999995e-05 2024-09-21T15:50:00.531 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.531 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.531 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:00.531 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263067+0000", 2024-09-21T15:50:00.531 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3653999999999998e-05 2024-09-21T15:50:00.531 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:00.531 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:00.531 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:00.531 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.532 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.532 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65293 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2918400~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:00.532 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.218930+0000", 2024-09-21T15:50:00.532 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.268431691, 2024-09-21T15:50:00.532 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.044253536000000003, 2024-09-21T15:50:00.532 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:00.532 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:00.532 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:00.532 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:00.532 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:00.532 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65293 2024-09-21T15:50:00.532 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.532 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:00.532 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.532 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:00.533 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218930+0000", 2024-09-21T15:50:00.533 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:00.533 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.533 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.533 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:00.533 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218930+0000", 2024-09-21T15:50:00.533 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:00.533 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.533 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.533 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:00.533 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218834+0000", 2024-09-21T15:50:00.533 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999042 2024-09-21T15:50:00.533 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.533 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.533 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:00.534 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218933+0000", 2024-09-21T15:50:00.534 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9047000000000002e-05 2024-09-21T15:50:00.534 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.534 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.534 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:00.534 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218941+0000", 2024-09-21T15:50:00.534 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0849999999999997e-06 2024-09-21T15:50:00.534 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.534 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.534 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:00.534 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218945+0000", 2024-09-21T15:50:00.534 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9179999999999996e-06 2024-09-21T15:50:00.534 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.534 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.535 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:00.535 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263093+0000", 2024-09-21T15:50:00.535 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.044147906000000001 2024-09-21T15:50:00.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.535 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.535 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:00.535 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263144+0000", 2024-09-21T15:50:00.535 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0470999999999998e-05 2024-09-21T15:50:00.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.535 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.535 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:00.535 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263183+0000", 2024-09-21T15:50:00.535 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9839999999999998e-05 2024-09-21T15:50:00.535 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:00.535 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:00.536 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:00.536 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.536 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.536 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65294 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2926592~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:00.536 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.219137+0000", 2024-09-21T15:50:00.536 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.268224515, 2024-09-21T15:50:00.536 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.044168171999999999, 2024-09-21T15:50:00.536 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:00.536 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:00.536 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:00.536 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:00.536 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:00.536 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65294 2024-09-21T15:50:00.536 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.536 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:00.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.537 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:00.537 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.219137+0000", 2024-09-21T15:50:00.537 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:00.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.537 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:00.537 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.219137+0000", 2024-09-21T15:50:00.537 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:00.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.537 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:00.537 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.218952+0000", 2024-09-21T15:50:00.537 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999815 2024-09-21T15:50:00.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.538 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.538 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:00.538 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.219140+0000", 2024-09-21T15:50:00.538 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00018848899999999999 2024-09-21T15:50:00.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.538 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.538 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:00.538 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.219147+0000", 2024-09-21T15:50:00.538 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0010000000000004e-06 2024-09-21T15:50:00.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.538 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.538 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:00.538 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.219153+0000", 2024-09-21T15:50:00.538 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4060000000000004e-06 2024-09-21T15:50:00.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.539 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:00.539 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263206+0000", 2024-09-21T15:50:00.539 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.044053467999999998 2024-09-21T15:50:00.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.539 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:00.539 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263248+0000", 2024-09-21T15:50:00.539 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1282000000000002e-05 2024-09-21T15:50:00.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.539 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:00.539 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263305+0000", 2024-09-21T15:50:00.539 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7645000000000001e-05 2024-09-21T15:50:00.540 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:00.540 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:00.540 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:00.540 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.540 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.540 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65295 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2934784~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:00.540 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.219274+0000", 2024-09-21T15:50:00.540 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.268087756, 2024-09-21T15:50:00.540 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.044207710999999997, 2024-09-21T15:50:00.540 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:00.540 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:00.540 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:00.540 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:00.540 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:00.540 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65295 2024-09-21T15:50:00.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.541 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:00.541 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.541 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:00.541 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.219274+0000", 2024-09-21T15:50:00.541 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:00.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.541 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.541 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:00.541 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.219274+0000", 2024-09-21T15:50:00.541 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:00.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.541 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.541 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:00.541 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.219158+0000", 2024-09-21T15:50:00.542 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9998837 2024-09-21T15:50:00.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.542 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:00.542 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.219276+0000", 2024-09-21T15:50:00.542 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000118656 2024-09-21T15:50:00.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.542 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:00.542 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.219281+0000", 2024-09-21T15:50:00.542 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7360000000000001e-06 2024-09-21T15:50:00.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.542 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:00.543 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.219285+0000", 2024-09-21T15:50:00.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6349999999999999e-06 2024-09-21T15:50:00.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.543 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.543 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:00.543 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263343+0000", 2024-09-21T15:50:00.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.044058264 2024-09-21T15:50:00.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.543 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.543 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:00.543 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263418+0000", 2024-09-21T15:50:00.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5001999999999998e-05 2024-09-21T15:50:00.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.543 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.544 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:00.544 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263482+0000", 2024-09-21T15:50:00.544 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3639999999999994e-05 2024-09-21T15:50:00.544 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:00.544 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:00.544 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:00.544 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.544 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.544 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65297 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2951168~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:00.544 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.220022+0000", 2024-09-21T15:50:00.544 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.267339863, 2024-09-21T15:50:00.544 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043682718000000002, 2024-09-21T15:50:00.544 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:00.544 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:00.544 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:00.545 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:00.545 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:00.545 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65297 2024-09-21T15:50:00.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.545 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:00.545 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.545 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:00.545 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220022+0000", 2024-09-21T15:50:00.545 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:00.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.545 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.545 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:00.545 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220022+0000", 2024-09-21T15:50:00.545 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:00.546 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.546 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:00.546 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220021+0000", 2024-09-21T15:50:00.546 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:00.546 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.546 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:00.546 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220024+0000", 2024-09-21T15:50:00.546 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3870000000000002e-06 2024-09-21T15:50:00.546 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.546 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:00.546 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220029+0000", 2024-09-21T15:50:00.546 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4439999999999999e-06 2024-09-21T15:50:00.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.547 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:00.547 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220033+0000", 2024-09-21T15:50:00.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7359999999999999e-06 2024-09-21T15:50:00.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.547 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:00.547 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263625+0000", 2024-09-21T15:50:00.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043592190000000003 2024-09-21T15:50:00.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.547 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:00.547 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263667+0000", 2024-09-21T15:50:00.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2048000000000001e-05 2024-09-21T15:50:00.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.548 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.548 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:00.548 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263704+0000", 2024-09-21T15:50:00.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7295000000000001e-05 2024-09-21T15:50:00.548 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:00.548 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:00.548 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:00.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.548 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.548 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65298 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2959360~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:00.548 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.220036+0000", 2024-09-21T15:50:00.548 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.267325835000001, 2024-09-21T15:50:00.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043821723999999999, 2024-09-21T15:50:00.549 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:00.549 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:00.549 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:00.549 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:00.549 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:00.549 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65298 2024-09-21T15:50:00.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.549 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:00.549 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.549 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:00.549 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220036+0000", 2024-09-21T15:50:00.549 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:00.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.549 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.549 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:00.550 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220036+0000", 2024-09-21T15:50:00.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:00.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.550 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.550 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:00.550 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220035+0000", 2024-09-21T15:50:00.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:00.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.550 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.550 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:00.550 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220037+0000", 2024-09-21T15:50:00.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.66e-06 2024-09-21T15:50:00.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.550 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:00.551 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220040+0000", 2024-09-21T15:50:00.551 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9699999999999999e-06 2024-09-21T15:50:00.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:00.551 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220042+0000", 2024-09-21T15:50:00.551 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4940000000000002e-06 2024-09-21T15:50:00.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:00.551 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263727+0000", 2024-09-21T15:50:00.551 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043684850999999997 2024-09-21T15:50:00.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.552 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:00.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263789+0000", 2024-09-21T15:50:00.552 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2147e-05 2024-09-21T15:50:00.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.552 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:00.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263857+0000", 2024-09-21T15:50:00.552 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7970000000000001e-05 2024-09-21T15:50:00.552 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:00.552 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:00.552 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:00.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.552 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65299 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2975744~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:00.553 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.220045+0000", 2024-09-21T15:50:00.553 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.267317035000001, 2024-09-21T15:50:00.553 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043920945000000003, 2024-09-21T15:50:00.553 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:00.553 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:00.553 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:00.553 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:00.553 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:00.553 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65299 2024-09-21T15:50:00.553 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.553 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:00.553 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.553 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:00.553 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220045+0000", 2024-09-21T15:50:00.553 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:00.554 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.554 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.554 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:00.554 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220045+0000", 2024-09-21T15:50:00.554 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:00.554 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.554 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.554 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:00.554 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220044+0000", 2024-09-21T15:50:00.554 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:00.554 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.554 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.554 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:00.554 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220046+0000", 2024-09-21T15:50:00.555 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.207e-06 2024-09-21T15:50:00.555 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.555 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.555 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:00.555 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220048+0000", 2024-09-21T15:50:00.555 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7939999999999998e-06 2024-09-21T15:50:00.555 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.555 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.555 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:00.555 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.220051+0000", 2024-09-21T15:50:00.555 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4770000000000002e-06 2024-09-21T15:50:00.555 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.555 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.555 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:00.556 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263885+0000", 2024-09-21T15:50:00.556 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043833719 2024-09-21T15:50:00.556 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.556 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.556 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:00.556 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263927+0000", 2024-09-21T15:50:00.556 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2409e-05 2024-09-21T15:50:00.556 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.556 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.556 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:00.556 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.263965+0000", 2024-09-21T15:50:00.556 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8550999999999997e-05 2024-09-21T15:50:00.556 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:00.556 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:00.556 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:00.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.557 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.557 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65773 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2961408~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:00.557 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.241336+0000", 2024-09-21T15:50:00.557 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.24602556, 2024-09-21T15:50:00.557 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043734599999999998, 2024-09-21T15:50:00.557 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:00.557 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:00.557 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:00.557 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:00.557 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:00.557 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65773 2024-09-21T15:50:00.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.557 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:00.557 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.558 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:00.558 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241336+0000", 2024-09-21T15:50:00.558 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:00.558 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.558 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.558 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:00.558 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241336+0000", 2024-09-21T15:50:00.558 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:00.558 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.558 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.558 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:00.558 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241336+0000", 2024-09-21T15:50:00.558 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:00.558 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.559 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.559 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:00.559 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241337+0000", 2024-09-21T15:50:00.559 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.505e-06 2024-09-21T15:50:00.559 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.559 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.559 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:00.559 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241340+0000", 2024-09-21T15:50:00.559 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8169999999999999e-06 2024-09-21T15:50:00.559 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.559 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.559 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:00.559 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241342+0000", 2024-09-21T15:50:00.559 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4820000000000001e-06 2024-09-21T15:50:00.559 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.560 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:00.560 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.284936+0000", 2024-09-21T15:50:00.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043593728999999998 2024-09-21T15:50:00.560 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.560 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:00.560 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.285002+0000", 2024-09-21T15:50:00.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5581999999999997e-05 2024-09-21T15:50:00.560 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.560 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:00.560 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.285071+0000", 2024-09-21T15:50:00.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8818000000000004e-05 2024-09-21T15:50:00.560 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:00.561 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:00.561 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:00.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.561 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.561 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65774 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2985984~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:00.561 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.241555+0000", 2024-09-21T15:50:00.561 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.245806746, 2024-09-21T15:50:00.561 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043704701999999998, 2024-09-21T15:50:00.561 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:00.561 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:00.561 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:00.561 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:00.561 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:00.561 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65774 2024-09-21T15:50:00.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.562 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:00.562 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.562 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:00.562 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241555+0000", 2024-09-21T15:50:00.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:00.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.562 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.562 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:00.562 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241555+0000", 2024-09-21T15:50:00.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:00.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.562 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.562 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:00.562 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241344+0000", 2024-09-21T15:50:00.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9997897 2024-09-21T15:50:00.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.563 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:00.563 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241560+0000", 2024-09-21T15:50:00.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00021524700000000001 2024-09-21T15:50:00.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.563 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:00.563 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241571+0000", 2024-09-21T15:50:00.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1627000000000001e-05 2024-09-21T15:50:00.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.563 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:00.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241578+0000", 2024-09-21T15:50:00.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6889999999999998e-06 2024-09-21T15:50:00.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:00.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.285108+0000", 2024-09-21T15:50:00.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043529644999999999 2024-09-21T15:50:00.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:00.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.285188+0000", 2024-09-21T15:50:00.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0774999999999998e-05 2024-09-21T15:50:00.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:00.565 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.285260+0000", 2024-09-21T15:50:00.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1173000000000004e-05 2024-09-21T15:50:00.565 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:00.565 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:00.565 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:00.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.565 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65775 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2977792~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:00.565 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.241672+0000", 2024-09-21T15:50:00.565 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.245689545000001, 2024-09-21T15:50:00.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043751534000000002, 2024-09-21T15:50:00.565 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:00.565 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:00.565 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:00.565 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:00.566 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:00.566 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65775 2024-09-21T15:50:00.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.566 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:00.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.566 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:00.566 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241672+0000", 2024-09-21T15:50:00.566 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:00.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.566 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:00.566 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241672+0000", 2024-09-21T15:50:00.566 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:00.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.567 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.567 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:00.567 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241584+0000", 2024-09-21T15:50:00.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999118 2024-09-21T15:50:00.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.567 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.567 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:00.567 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241676+0000", 2024-09-21T15:50:00.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2115999999999997e-05 2024-09-21T15:50:00.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.567 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.567 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:00.567 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241684+0000", 2024-09-21T15:50:00.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3149999999999992e-06 2024-09-21T15:50:00.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.568 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:00.568 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241691+0000", 2024-09-21T15:50:00.568 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2999999999999998e-06 2024-09-21T15:50:00.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.568 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:00.568 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.285289+0000", 2024-09-21T15:50:00.568 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043598494000000002 2024-09-21T15:50:00.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.568 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:00.568 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.285361+0000", 2024-09-21T15:50:00.568 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1731999999999997e-05 2024-09-21T15:50:00.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.569 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:00.569 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.285424+0000", 2024-09-21T15:50:00.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2577999999999996e-05 2024-09-21T15:50:00.569 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:00.569 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:00.570 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:00.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.570 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:65776 3.17 3:ed66a0cb:::rbd_data.11336b2fdf4a.0000000000000002:c [read 2994176~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:00.570 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.241867+0000", 2024-09-21T15:50:00.570 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.245494963999999, 2024-09-21T15:50:00.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043720638999999999, 2024-09-21T15:50:00.570 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:00.570 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:00.570 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:00.570 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:00.570 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:00.570 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 65776 2024-09-21T15:50:00.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.571 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:00.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:00.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241867+0000", 2024-09-21T15:50:00.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:00.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:00.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241867+0000", 2024-09-21T15:50:00.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:00.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:00.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241696+0000", 2024-09-21T15:50:00.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9998293 2024-09-21T15:50:00.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.572 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:00.572 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241871+0000", 2024-09-21T15:50:00.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00017491300000000001 2024-09-21T15:50:00.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.572 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:00.572 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241889+0000", 2024-09-21T15:50:00.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8065000000000001e-05 2024-09-21T15:50:00.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.572 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:00.572 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.241897+0000", 2024-09-21T15:50:00.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9310000000000008e-06 2024-09-21T15:50:00.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:00.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.285446+0000", 2024-09-21T15:50:00.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.043548735999999998 2024-09-21T15:50:00.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:00.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.285523+0000", 2024-09-21T15:50:00.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7014000000000005e-05 2024-09-21T15:50:00.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:00.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:00.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:00.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.285587+0000", 2024-09-21T15:50:00.574 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4455000000000005e-05 2024-09-21T15:50:00.574 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:00.574 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:00.574 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:00.574 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:00.574 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:00.574 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:00.574 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:50:00.576 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:50:00.576 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:00.574+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:00.677 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:50:00.676+0000 7f3d4f460640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 33314) UID: 0 2024-09-21T15:50:00.722 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:00.722 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:00.722 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:50:00.722 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:00.731 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:50:00.737 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-09-21T15:50:00.737 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:50:00.755 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:50:00.756 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:50:00.771 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:50:00.772 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:50:00.778 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:00.777+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:00.788 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:50:00.788 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:50:00.804 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:50:00.804 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:50:00.820 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:50:00.821 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:50:00.837 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:50:00.837 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:50:00.842 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:00.843 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:00.852 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:50:00.853 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:50:00.854 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:50:00.879 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:00.877+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:00.955 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:00.955 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:00.960 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T15:50:00.960 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T15:50:00.960 INFO:tasks.thrashosds.thrasher:Killing osd 0, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T15:50:00.961 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-21T15:50:00.961 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T15:50:00.965 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:50:00.966 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is unset 2024-09-21T15:50:00.979 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:00.979+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:01.002 INFO:tasks.ceph.osd.0:Stopped 2024-09-21T15:50:01.002 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.0 2024-09-21T15:50:01.002 INFO:tasks.thrashosds.thrasher:remote for osd 0 is ubuntu@smithi089.front.sepia.ceph.com 2024-09-21T15:50:01.002 DEBUG:teuthology.orchestra.run.smithi089:> type ceph-objectstore-tool 2024-09-21T15:50:01.056 INFO:teuthology.orchestra.run.smithi089.stdout:ceph-objectstore-tool is /usr/bin/ceph-objectstore-tool 2024-09-21T15:50:01.057 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --op list-pgs 2024-09-21T15:50:01.070 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:01.071 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:01.080 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:50:01.080 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:01.079+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:01.181 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:01.180+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:01.229 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:01.230 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:01.238 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:50:01.282 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:01.281+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:01.341 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:01.341 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:01.350 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:50:01.394 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:01.393+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:01.467 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:01.467 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:01.476 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:50:01.517 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:01.517+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:01.584 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:01.583+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:01.620 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:01.620 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:01.629 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:50:01.713 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:01.712+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:01.771 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:01.771 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:01.779 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:50:01.820 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:01.819+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:01.886 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:01.885+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:01.921 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:01.922 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:01.930 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:50:01.987 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:01.986+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:02.033 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:02.034 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:02.042 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:50:02.087 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:02.086+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:02.144 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:02.144 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:02.153 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T15:50:02.194 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:02.193+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:02.290 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:02.289+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:02.295 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:02.295 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:02.304 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T15:50:02.391 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:02.390+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:02.408 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:02.408 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:02.417 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T15:50:02.492 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:02.491+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:02.520 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:02.520 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:02.528 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T15:50:02.572 INFO:teuthology.orchestra.run.smithi089.stdout:3.6 2024-09-21T15:50:02.572 INFO:teuthology.orchestra.run.smithi089.stdout:3.3 2024-09-21T15:50:02.572 INFO:teuthology.orchestra.run.smithi089.stdout:3.1f 2024-09-21T15:50:02.572 INFO:teuthology.orchestra.run.smithi089.stdout:3.1b 2024-09-21T15:50:02.572 INFO:teuthology.orchestra.run.smithi089.stdout:3.1 2024-09-21T15:50:02.572 INFO:teuthology.orchestra.run.smithi089.stdout:3.18 2024-09-21T15:50:02.572 INFO:teuthology.orchestra.run.smithi089.stdout:3.17 2024-09-21T15:50:02.572 INFO:teuthology.orchestra.run.smithi089.stdout:3.12 2024-09-21T15:50:02.573 INFO:teuthology.orchestra.run.smithi089.stdout:3.10 2024-09-21T15:50:02.573 INFO:teuthology.orchestra.run.smithi089.stdout:2.5 2024-09-21T15:50:02.573 INFO:teuthology.orchestra.run.smithi089.stdout:3.b 2024-09-21T15:50:02.573 INFO:teuthology.orchestra.run.smithi089.stdout:2.1 2024-09-21T15:50:02.573 INFO:teuthology.orchestra.run.smithi089.stdout:2.4 2024-09-21T15:50:02.573 INFO:teuthology.orchestra.run.smithi089.stdout:1.0 2024-09-21T15:50:02.593 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:02.592+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:02.630 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:02.630 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:02.639 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T15:50:02.694 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:02.693+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:02.781 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:02.782 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:02.790 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T15:50:02.834 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:02.834+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:02.865 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:50:02.892 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:02.892 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:02.900 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:50:02.910 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:02.909+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:02.980 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T15:50:02.988 INFO:teuthology.orchestra.run.smithi089.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T15:50:02.996 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:02.995+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:03.020 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:50:03.021 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:50:03.054 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:03.055 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:03.055 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:50:03.055 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:03.055 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:50:03.055 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:50:03.070 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:50:03.077 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:50:03.078 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:50:03.095 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:50:03.095 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:50:03.097 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:03.096+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:03.100 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --op export --pgid 2.4 --file /var/log/ceph/exp.2.4.0 2024-09-21T15:50:03.114 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:50:03.114 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:50:03.131 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:50:03.132 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:50:03.149 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:50:03.149 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:50:03.153 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T15:50:03.153 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T15:50:03.191 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:03.191 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:03.191 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:50:03.191 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:50:03.192 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:03.197 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:03.196+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:03.204 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:50:03.214 INFO:teuthology.orchestra.run.smithi089.stdout:checking clone #25, image image_client.0-clone25 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent26 2024-09-21T15:50:03.298 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:03.297+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:03.339 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:03.339 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:50:03.339 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:50:03.339 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:50:03.339 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.339 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62833 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 192512~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:03.339 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:50:03.339 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 26.194544638, 2024-09-21T15:50:03.339 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050387007999999997, 2024-09-21T15:50:03.339 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:03.339 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:03.340 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:03.340 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:03.340 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:03.340 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62833 2024-09-21T15:50:03.340 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.340 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:03.340 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.340 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:03.340 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:50:03.340 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.340 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.340 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.340 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:03.340 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:50:03.340 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.341 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.341 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.341 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:03.341 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:50:03.341 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:03.341 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.341 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.341 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:03.341 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142150+0000", 2024-09-21T15:50:03.341 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1999999999999999e-06 2024-09-21T15:50:03.341 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.341 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.341 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:03.341 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142153+0000", 2024-09-21T15:50:03.342 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7410000000000001e-06 2024-09-21T15:50:03.342 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.342 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.342 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:03.342 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142156+0000", 2024-09-21T15:50:03.342 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5739999999999998e-06 2024-09-21T15:50:03.342 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.342 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.342 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:03.342 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.192358+0000", 2024-09-21T15:50:03.342 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050202452000000002 2024-09-21T15:50:03.342 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.342 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.342 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:03.343 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.192435+0000", 2024-09-21T15:50:03.343 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6738999999999998e-05 2024-09-21T15:50:03.343 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.343 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.343 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:03.343 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.192536+0000", 2024-09-21T15:50:03.343 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101508 2024-09-21T15:50:03.343 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.343 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:03.343 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.343 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.343 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.343 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62849 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 208896~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:03.343 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.144181+0000", 2024-09-21T15:50:03.343 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 26.192512566000001, 2024-09-21T15:50:03.344 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050460390000000001, 2024-09-21T15:50:03.344 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:03.344 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:03.344 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:03.344 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:03.344 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:03.344 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62849 2024-09-21T15:50:03.344 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.344 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:03.344 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.344 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:03.344 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144181+0000", 2024-09-21T15:50:03.344 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.345 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.345 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.345 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:03.345 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144181+0000", 2024-09-21T15:50:03.345 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.345 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.345 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.345 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:03.345 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144181+0000", 2024-09-21T15:50:03.345 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:03.345 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.345 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.345 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:03.345 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144183+0000", 2024-09-21T15:50:03.345 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3860000000000001e-06 2024-09-21T15:50:03.346 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.346 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.346 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:03.346 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144189+0000", 2024-09-21T15:50:03.346 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0599999999999996e-06 2024-09-21T15:50:03.346 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.346 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.346 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:03.346 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144194+0000", 2024-09-21T15:50:03.346 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4379999999999997e-06 2024-09-21T15:50:03.346 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.346 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.346 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:03.347 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.194457+0000", 2024-09-21T15:50:03.347 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050263317000000002 2024-09-21T15:50:03.347 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.347 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.347 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:03.347 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.194548+0000", 2024-09-21T15:50:03.347 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0785000000000006e-05 2024-09-21T15:50:03.347 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.347 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.347 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:03.347 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.194642+0000", 2024-09-21T15:50:03.347 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3837000000000003e-05 2024-09-21T15:50:03.347 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.347 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:03.347 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.348 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.348 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.348 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62862 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 954368~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:03.348 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.146176+0000", 2024-09-21T15:50:03.348 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 26.190517502999999, 2024-09-21T15:50:03.348 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050442901999999998, 2024-09-21T15:50:03.348 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:03.348 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:03.348 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:03.348 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:03.348 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:03.348 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62862 2024-09-21T15:50:03.348 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.348 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:03.349 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.349 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:03.349 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146176+0000", 2024-09-21T15:50:03.349 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.349 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.349 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.349 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:03.349 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146176+0000", 2024-09-21T15:50:03.349 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.349 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.349 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.349 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:03.349 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146176+0000", 2024-09-21T15:50:03.349 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:03.350 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.350 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.350 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:03.350 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146178+0000", 2024-09-21T15:50:03.350 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4739999999999999e-06 2024-09-21T15:50:03.350 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.350 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.350 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:03.350 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146186+0000", 2024-09-21T15:50:03.350 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4989999999999997e-06 2024-09-21T15:50:03.350 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.350 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.350 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:03.350 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146191+0000", 2024-09-21T15:50:03.350 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7999999999999998e-06 2024-09-21T15:50:03.351 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.351 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.351 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:03.351 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196442+0000", 2024-09-21T15:50:03.351 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050251665000000001 2024-09-21T15:50:03.351 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.351 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.351 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:03.351 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196518+0000", 2024-09-21T15:50:03.351 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5677000000000001e-05 2024-09-21T15:50:03.351 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.351 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.351 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:03.351 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196619+0000", 2024-09-21T15:50:03.352 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101247 2024-09-21T15:50:03.352 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.352 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:03.352 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.352 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.352 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.352 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62863 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 937984~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:03.352 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.146195+0000", 2024-09-21T15:50:03.352 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 26.190499312, 2024-09-21T15:50:03.352 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050642305999999998, 2024-09-21T15:50:03.352 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:03.352 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:03.352 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:03.352 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:03.352 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:03.352 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62863 2024-09-21T15:50:03.352 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.353 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:03.353 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.353 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:03.353 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146195+0000", 2024-09-21T15:50:03.353 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.353 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.353 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.353 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:03.353 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146195+0000", 2024-09-21T15:50:03.353 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.353 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.353 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.353 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:03.353 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146194+0000", 2024-09-21T15:50:03.353 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:03.353 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.354 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.354 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:03.354 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146196+0000", 2024-09-21T15:50:03.354 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1900000000000002e-06 2024-09-21T15:50:03.354 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.354 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.354 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:03.354 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146202+0000", 2024-09-21T15:50:03.354 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.57e-06 2024-09-21T15:50:03.354 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.354 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.354 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:03.354 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146206+0000", 2024-09-21T15:50:03.354 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5079999999999999e-06 2024-09-21T15:50:03.355 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.355 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.355 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:03.355 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196655+0000", 2024-09-21T15:50:03.355 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050448973000000001 2024-09-21T15:50:03.355 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.355 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.355 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:03.355 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196730+0000", 2024-09-21T15:50:03.355 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4728999999999996e-05 2024-09-21T15:50:03.355 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.355 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.355 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:03.355 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196837+0000", 2024-09-21T15:50:03.355 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000106735 2024-09-21T15:50:03.355 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.356 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:03.356 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.356 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.356 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.356 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62871 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 307200~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:03.356 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148134+0000", 2024-09-21T15:50:03.356 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 26.188560103, 2024-09-21T15:50:03.356 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050310001, 2024-09-21T15:50:03.356 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:03.356 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:03.356 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:03.356 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:03.356 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:03.356 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62871 2024-09-21T15:50:03.356 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.356 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:03.357 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.357 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:03.357 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148134+0000", 2024-09-21T15:50:03.357 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.357 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.357 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.357 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:03.357 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148134+0000", 2024-09-21T15:50:03.357 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.357 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.357 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.357 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:03.357 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148133+0000", 2024-09-21T15:50:03.357 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:03.357 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.357 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.357 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:03.358 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148135+0000", 2024-09-21T15:50:03.358 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9829999999999999e-06 2024-09-21T15:50:03.358 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.358 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.358 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:03.358 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148141+0000", 2024-09-21T15:50:03.358 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5439999999999998e-06 2024-09-21T15:50:03.358 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.358 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.358 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:03.358 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148145+0000", 2024-09-21T15:50:03.358 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4229999999999998e-06 2024-09-21T15:50:03.358 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.358 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.359 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:03.359 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198294+0000", 2024-09-21T15:50:03.359 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050148558000000003 2024-09-21T15:50:03.359 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.359 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.359 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:03.359 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198372+0000", 2024-09-21T15:50:03.359 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8163000000000004e-05 2024-09-21T15:50:03.359 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.359 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.359 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:03.359 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198444+0000", 2024-09-21T15:50:03.359 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1620999999999997e-05 2024-09-21T15:50:03.359 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.359 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:03.360 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.360 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.360 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.360 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62872 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 290816~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:03.360 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148149+0000", 2024-09-21T15:50:03.360 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 26.188544636, 2024-09-21T15:50:03.360 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.05046991, 2024-09-21T15:50:03.360 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:03.360 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:03.360 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:03.360 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:03.360 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:03.360 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62872 2024-09-21T15:50:03.360 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.361 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:03.361 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.361 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:03.361 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148149+0000", 2024-09-21T15:50:03.361 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.361 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.361 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.361 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:03.361 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148149+0000", 2024-09-21T15:50:03.361 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.361 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.361 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.361 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:03.361 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148149+0000", 2024-09-21T15:50:03.361 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:03.361 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.362 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.362 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:03.362 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148157+0000", 2024-09-21T15:50:03.362 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4749999999999993e-06 2024-09-21T15:50:03.362 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.362 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.362 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:03.362 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148163+0000", 2024-09-21T15:50:03.362 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8250000000000001e-06 2024-09-21T15:50:03.362 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.362 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.362 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:03.363 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148168+0000", 2024-09-21T15:50:03.363 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4730000000000002e-06 2024-09-21T15:50:03.363 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.363 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.363 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:03.363 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198471+0000", 2024-09-21T15:50:03.363 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050302974 2024-09-21T15:50:03.363 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.363 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.363 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:03.363 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198555+0000", 2024-09-21T15:50:03.363 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4783999999999993e-05 2024-09-21T15:50:03.363 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.363 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.363 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:03.363 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198619+0000", 2024-09-21T15:50:03.364 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3751000000000007e-05 2024-09-21T15:50:03.364 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.364 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:03.364 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.364 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.364 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.364 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62873 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 315392~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:03.364 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148172+0000", 2024-09-21T15:50:03.364 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 26.188522344999999, 2024-09-21T15:50:03.364 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050707998999999997, 2024-09-21T15:50:03.364 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:03.364 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:03.364 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:03.364 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:03.364 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:03.364 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62873 2024-09-21T15:50:03.364 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.365 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:03.365 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.365 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:03.365 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148172+0000", 2024-09-21T15:50:03.365 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.365 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.365 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.365 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:03.365 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148172+0000", 2024-09-21T15:50:03.365 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.365 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.365 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.365 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:03.365 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148171+0000", 2024-09-21T15:50:03.365 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:03.366 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.366 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.366 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:03.366 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148173+0000", 2024-09-21T15:50:03.366 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2340000000000001e-06 2024-09-21T15:50:03.366 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.366 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.366 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:03.366 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148179+0000", 2024-09-21T15:50:03.366 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9340000000000003e-06 2024-09-21T15:50:03.366 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.366 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.366 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:03.366 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148186+0000", 2024-09-21T15:50:03.366 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2349999999999996e-06 2024-09-21T15:50:03.366 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.367 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.367 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:03.367 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198684+0000", 2024-09-21T15:50:03.367 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050498890999999997 2024-09-21T15:50:03.367 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.367 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.367 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:03.367 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198778+0000", 2024-09-21T15:50:03.367 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3134999999999995e-05 2024-09-21T15:50:03.367 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.367 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.367 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:03.367 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198880+0000", 2024-09-21T15:50:03.367 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101906 2024-09-21T15:50:03.367 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.367 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:03.368 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.368 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.368 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.368 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62874 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 323584~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:03.368 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148211+0000", 2024-09-21T15:50:03.368 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 26.188482584999999, 2024-09-21T15:50:03.368 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050887766000000001, 2024-09-21T15:50:03.368 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:03.368 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:03.368 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:03.368 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:03.368 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:03.368 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62874 2024-09-21T15:50:03.368 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.368 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:03.368 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.369 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:03.369 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148211+0000", 2024-09-21T15:50:03.369 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.369 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.369 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.369 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:03.369 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148211+0000", 2024-09-21T15:50:03.369 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.369 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.369 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.369 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:03.369 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148211+0000", 2024-09-21T15:50:03.369 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:03.369 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.369 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.370 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:03.370 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148213+0000", 2024-09-21T15:50:03.370 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6860000000000002e-06 2024-09-21T15:50:03.370 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.370 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.370 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:03.370 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148220+0000", 2024-09-21T15:50:03.370 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4740000000000002e-06 2024-09-21T15:50:03.370 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.370 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.370 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:03.370 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148226+0000", 2024-09-21T15:50:03.370 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0120000000000002e-06 2024-09-21T15:50:03.370 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.370 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.370 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:03.371 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198940+0000", 2024-09-21T15:50:03.371 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050714412 2024-09-21T15:50:03.371 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.371 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.371 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:03.371 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198995+0000", 2024-09-21T15:50:03.371 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4863e-05 2024-09-21T15:50:03.371 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.371 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.371 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:03.371 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.199099+0000", 2024-09-21T15:50:03.371 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010383299999999999 2024-09-21T15:50:03.371 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.371 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:03.371 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.371 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.372 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.372 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62877 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 995328~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:03.372 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148230+0000", 2024-09-21T15:50:03.372 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 26.188463724999998, 2024-09-21T15:50:03.372 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.051108949000000001, 2024-09-21T15:50:03.372 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:03.372 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:03.372 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:03.372 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:03.372 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:03.372 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62877 2024-09-21T15:50:03.372 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.372 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:03.372 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.372 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:03.373 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148230+0000", 2024-09-21T15:50:03.373 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.373 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.373 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.373 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:03.373 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148230+0000", 2024-09-21T15:50:03.373 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.373 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.373 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.373 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:03.373 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148230+0000", 2024-09-21T15:50:03.373 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:03.373 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.373 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.374 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:03.374 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148232+0000", 2024-09-21T15:50:03.374 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1220000000000002e-06 2024-09-21T15:50:03.374 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.374 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.374 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:03.374 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148238+0000", 2024-09-21T15:50:03.374 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.789e-06 2024-09-21T15:50:03.374 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.374 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.374 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:03.374 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148242+0000", 2024-09-21T15:50:03.374 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7369999999999997e-06 2024-09-21T15:50:03.374 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.374 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.375 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:03.375 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.199201+0000", 2024-09-21T15:50:03.375 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050959088 2024-09-21T15:50:03.375 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.375 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.375 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:03.375 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.199285+0000", 2024-09-21T15:50:03.375 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3147999999999997e-05 2024-09-21T15:50:03.375 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.375 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.375 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:03.375 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.199339+0000", 2024-09-21T15:50:03.375 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4484999999999998e-05 2024-09-21T15:50:03.375 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.375 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:03.376 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.376 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.376 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.376 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62886 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 1044480~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:03.376 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150138+0000", 2024-09-21T15:50:03.376 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 26.186556057000001, 2024-09-21T15:50:03.376 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050345527000000001, 2024-09-21T15:50:03.376 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:03.376 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:03.376 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:03.376 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:03.376 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:03.376 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62886 2024-09-21T15:50:03.376 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.376 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:03.376 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.377 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:03.377 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150138+0000", 2024-09-21T15:50:03.377 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.377 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.377 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.377 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:03.377 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150138+0000", 2024-09-21T15:50:03.377 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.377 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.377 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.377 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:03.377 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150137+0000", 2024-09-21T15:50:03.377 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:03.377 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.377 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.377 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:03.377 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150140+0000", 2024-09-21T15:50:03.378 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2460000000000002e-06 2024-09-21T15:50:03.378 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.378 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.378 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:03.378 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150145+0000", 2024-09-21T15:50:03.378 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6520000000000003e-06 2024-09-21T15:50:03.378 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.378 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.378 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:03.378 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150150+0000", 2024-09-21T15:50:03.378 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5310000000000003e-06 2024-09-21T15:50:03.378 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.378 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.378 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:03.378 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200356+0000", 2024-09-21T15:50:03.378 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050205759000000003 2024-09-21T15:50:03.379 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.379 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.379 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:03.379 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200415+0000", 2024-09-21T15:50:03.379 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9719999999999997e-05 2024-09-21T15:50:03.379 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.379 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.379 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:03.379 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200483+0000", 2024-09-21T15:50:03.379 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7959000000000004e-05 2024-09-21T15:50:03.379 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.379 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:03.379 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.379 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.379 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.379 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62888 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 331776~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:03.380 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150154+0000", 2024-09-21T15:50:03.380 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 26.186540131000001, 2024-09-21T15:50:03.380 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050483169000000001, 2024-09-21T15:50:03.380 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:03.380 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:03.380 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:03.380 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:03.380 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:03.380 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62888 2024-09-21T15:50:03.380 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.380 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:03.380 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.380 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:03.380 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150154+0000", 2024-09-21T15:50:03.381 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.381 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.381 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.381 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:03.381 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150154+0000", 2024-09-21T15:50:03.381 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.381 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.381 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.381 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:03.381 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150153+0000", 2024-09-21T15:50:03.381 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:03.381 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.382 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.382 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:03.382 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150155+0000", 2024-09-21T15:50:03.382 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9920000000000002e-06 2024-09-21T15:50:03.382 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.382 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.382 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:03.382 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150161+0000", 2024-09-21T15:50:03.382 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5439999999999998e-06 2024-09-21T15:50:03.382 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.382 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.382 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:03.382 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150165+0000", 2024-09-21T15:50:03.382 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3460000000000004e-06 2024-09-21T15:50:03.383 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.383 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.383 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:03.383 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200502+0000", 2024-09-21T15:50:03.383 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050337150999999997 2024-09-21T15:50:03.383 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.383 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.383 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:03.383 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200575+0000", 2024-09-21T15:50:03.383 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2250999999999997e-05 2024-09-21T15:50:03.383 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.383 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.383 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:03.383 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200637+0000", 2024-09-21T15:50:03.383 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2195999999999999e-05 2024-09-21T15:50:03.383 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.383 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:03.384 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.384 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.384 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.384 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62889 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 339968~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:03.384 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150169+0000", 2024-09-21T15:50:03.384 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 26.186524632000001, 2024-09-21T15:50:03.384 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050629678999999997, 2024-09-21T15:50:03.384 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:03.384 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:03.384 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:03.384 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:03.384 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:03.384 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62889 2024-09-21T15:50:03.384 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.384 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:03.385 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.385 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:03.385 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150169+0000", 2024-09-21T15:50:03.385 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.385 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.385 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.385 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:03.385 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150169+0000", 2024-09-21T15:50:03.385 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.385 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.385 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.385 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:03.385 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150169+0000", 2024-09-21T15:50:03.385 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:03.385 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.385 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.386 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:03.386 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150171+0000", 2024-09-21T15:50:03.386 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3910000000000001e-06 2024-09-21T15:50:03.386 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.386 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.386 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:03.386 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150177+0000", 2024-09-21T15:50:03.386 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4079999999999997e-06 2024-09-21T15:50:03.386 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.386 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.386 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:03.386 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150182+0000", 2024-09-21T15:50:03.386 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3890000000000004e-06 2024-09-21T15:50:03.386 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.387 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.387 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:03.387 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200664+0000", 2024-09-21T15:50:03.387 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050482212999999998 2024-09-21T15:50:03.387 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.387 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.387 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:03.387 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200738+0000", 2024-09-21T15:50:03.387 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4114000000000002e-05 2024-09-21T15:50:03.387 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.387 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.387 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:03.387 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200799+0000", 2024-09-21T15:50:03.387 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0503e-05 2024-09-21T15:50:03.387 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.388 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:03.388 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.388 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.388 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.388 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62891 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 348160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:03.388 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150186+0000", 2024-09-21T15:50:03.388 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 26.18650749, 2024-09-21T15:50:03.388 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050794552, 2024-09-21T15:50:03.388 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:03.388 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:03.388 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:03.388 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:03.388 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:03.388 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62891 2024-09-21T15:50:03.388 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.388 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:03.389 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.389 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:03.389 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150186+0000", 2024-09-21T15:50:03.389 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.389 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.389 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.389 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:03.389 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150186+0000", 2024-09-21T15:50:03.389 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.389 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.389 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.389 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:03.389 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150186+0000", 2024-09-21T15:50:03.389 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:03.389 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.390 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.390 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:03.390 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150193+0000", 2024-09-21T15:50:03.390 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6830000000000002e-06 2024-09-21T15:50:03.390 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.390 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.390 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:03.390 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150199+0000", 2024-09-21T15:50:03.390 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9859999999999999e-06 2024-09-21T15:50:03.390 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.390 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.390 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:03.390 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150203+0000", 2024-09-21T15:50:03.390 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5920000000000002e-06 2024-09-21T15:50:03.390 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.391 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.391 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:03.391 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200835+0000", 2024-09-21T15:50:03.391 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050631783 2024-09-21T15:50:03.391 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.391 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.391 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:03.391 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200894+0000", 2024-09-21T15:50:03.391 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8560999999999997e-05 2024-09-21T15:50:03.391 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.391 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.391 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:03.391 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200981+0000", 2024-09-21T15:50:03.391 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7305999999999994e-05 2024-09-21T15:50:03.391 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.391 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:03.392 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.392 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.392 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.392 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62892 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 356352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:03.392 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150208+0000", 2024-09-21T15:50:03.392 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 26.186486102, 2024-09-21T15:50:03.392 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050951769000000001, 2024-09-21T15:50:03.392 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:03.392 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:03.392 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:03.392 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:03.392 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:03.392 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62892 2024-09-21T15:50:03.392 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.392 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:03.392 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.393 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:03.393 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150208+0000", 2024-09-21T15:50:03.393 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.393 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.393 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.393 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:03.393 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150208+0000", 2024-09-21T15:50:03.393 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.393 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.393 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.393 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:03.393 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150207+0000", 2024-09-21T15:50:03.393 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:03.393 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.393 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.393 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:03.394 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150210+0000", 2024-09-21T15:50:03.394 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.238e-06 2024-09-21T15:50:03.394 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.394 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.394 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:03.394 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150215+0000", 2024-09-21T15:50:03.394 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6350000000000003e-06 2024-09-21T15:50:03.394 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.394 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.394 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:03.394 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150220+0000", 2024-09-21T15:50:03.394 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.544e-06 2024-09-21T15:50:03.394 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.394 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.394 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:03.394 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201014+0000", 2024-09-21T15:50:03.395 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050793932999999999 2024-09-21T15:50:03.395 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.395 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.395 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:03.395 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201092+0000", 2024-09-21T15:50:03.395 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7961999999999997e-05 2024-09-21T15:50:03.395 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.395 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.395 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:03.395 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201160+0000", 2024-09-21T15:50:03.395 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7836000000000006e-05 2024-09-21T15:50:03.395 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.395 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:03.395 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.395 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.396 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.396 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62894 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 364544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:03.396 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150820+0000", 2024-09-21T15:50:03.396 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 26.185873902000001, 2024-09-21T15:50:03.396 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050489196, 2024-09-21T15:50:03.396 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:03.396 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:03.396 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:03.396 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:03.396 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:03.396 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62894 2024-09-21T15:50:03.396 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.396 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:03.396 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.396 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:03.396 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150820+0000", 2024-09-21T15:50:03.396 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.397 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.397 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.397 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:03.397 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150820+0000", 2024-09-21T15:50:03.397 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.397 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.397 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.397 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:03.397 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150223+0000", 2024-09-21T15:50:03.397 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9994035 2024-09-21T15:50:03.397 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.397 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.397 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:03.397 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150825+0000", 2024-09-21T15:50:03.397 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00060137099999999996 2024-09-21T15:50:03.398 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.398 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.398 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:03.398 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150837+0000", 2024-09-21T15:50:03.398 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2503999999999999e-05 2024-09-21T15:50:03.398 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.398 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.398 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:03.398 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150844+0000", 2024-09-21T15:50:03.398 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6900000000000003e-06 2024-09-21T15:50:03.398 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.398 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.398 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:03.398 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201187+0000", 2024-09-21T15:50:03.399 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050342973999999999 2024-09-21T15:50:03.399 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.399 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.399 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:03.399 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201250+0000", 2024-09-21T15:50:03.399 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3274000000000004e-05 2024-09-21T15:50:03.399 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.399 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.399 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:03.399 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201309+0000", 2024-09-21T15:50:03.399 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9058000000000003e-05 2024-09-21T15:50:03.399 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.399 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:03.399 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.400 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.400 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.400 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62895 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 380928~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:03.400 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150851+0000", 2024-09-21T15:50:03.400 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 26.185843305999999, 2024-09-21T15:50:03.400 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050611097000000001, 2024-09-21T15:50:03.400 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:03.400 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:03.400 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:03.400 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:03.400 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:03.400 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62895 2024-09-21T15:50:03.400 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.400 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:03.401 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.401 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:03.401 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150851+0000", 2024-09-21T15:50:03.401 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.401 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.401 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.401 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:03.401 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150851+0000", 2024-09-21T15:50:03.401 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.401 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.401 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.401 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:03.401 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150850+0000", 2024-09-21T15:50:03.401 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:03.401 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.401 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.402 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:03.402 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150853+0000", 2024-09-21T15:50:03.402 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1039999999999998e-06 2024-09-21T15:50:03.402 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.402 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.402 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:03.402 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150860+0000", 2024-09-21T15:50:03.402 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3180000000000001e-06 2024-09-21T15:50:03.402 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.402 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.402 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:03.402 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150866+0000", 2024-09-21T15:50:03.402 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1599999999999997e-06 2024-09-21T15:50:03.402 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.402 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.402 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:03.403 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201332+0000", 2024-09-21T15:50:03.403 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050466100999999999 2024-09-21T15:50:03.403 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.403 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.403 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:03.403 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201392+0000", 2024-09-21T15:50:03.403 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0053999999999998e-05 2024-09-21T15:50:03.403 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.403 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.403 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:03.403 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201462+0000", 2024-09-21T15:50:03.403 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9861000000000001e-05 2024-09-21T15:50:03.403 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.403 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:03.403 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.403 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.404 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.404 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62896 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 372736~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:03.404 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150870+0000", 2024-09-21T15:50:03.404 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 26.185823526, 2024-09-21T15:50:03.404 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050753292999999998, 2024-09-21T15:50:03.404 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:03.404 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:03.404 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:03.404 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:03.404 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:03.404 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62896 2024-09-21T15:50:03.404 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.404 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:03.404 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.404 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:03.404 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150870+0000", 2024-09-21T15:50:03.405 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.405 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.405 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.405 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:03.405 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150870+0000", 2024-09-21T15:50:03.405 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.405 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.405 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.405 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:03.405 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150870+0000", 2024-09-21T15:50:03.405 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:03.405 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.405 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.405 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:03.405 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150872+0000", 2024-09-21T15:50:03.406 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.644e-06 2024-09-21T15:50:03.406 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.406 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.406 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:03.406 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150879+0000", 2024-09-21T15:50:03.406 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3509999999999998e-06 2024-09-21T15:50:03.406 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.406 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.406 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:03.406 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150884+0000", 2024-09-21T15:50:03.406 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9150000000000004e-06 2024-09-21T15:50:03.406 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.406 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.406 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:03.406 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201487+0000", 2024-09-21T15:50:03.406 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050603276000000003 2024-09-21T15:50:03.407 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.407 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.407 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:03.407 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201557+0000", 2024-09-21T15:50:03.407 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9847999999999999e-05 2024-09-21T15:50:03.407 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.407 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.407 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:03.407 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201624+0000", 2024-09-21T15:50:03.407 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6730999999999995e-05 2024-09-21T15:50:03.407 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.407 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:03.407 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.407 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.407 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.407 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62897 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 389120~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:03.408 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.151044+0000", 2024-09-21T15:50:03.408 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 26.185649485999999, 2024-09-21T15:50:03.408 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050720531999999999, 2024-09-21T15:50:03.408 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:03.408 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:03.408 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:03.408 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:03.408 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:03.408 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62897 2024-09-21T15:50:03.408 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.408 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:03.408 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.408 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:03.408 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151044+0000", 2024-09-21T15:50:03.408 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.408 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.409 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.409 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:03.409 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151044+0000", 2024-09-21T15:50:03.409 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.409 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.409 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.409 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:03.409 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150888+0000", 2024-09-21T15:50:03.409 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9998431 2024-09-21T15:50:03.409 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.409 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.409 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:03.409 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151048+0000", 2024-09-21T15:50:03.409 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00016077299999999999 2024-09-21T15:50:03.409 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.409 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.409 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:03.410 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151058+0000", 2024-09-21T15:50:03.410 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.465e-06 2024-09-21T15:50:03.410 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.410 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.410 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:03.410 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151064+0000", 2024-09-21T15:50:03.410 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3829999999999997e-06 2024-09-21T15:50:03.410 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.410 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.410 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:03.410 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201648+0000", 2024-09-21T15:50:03.410 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050583597000000001 2024-09-21T15:50:03.410 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.410 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.410 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:03.411 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201711+0000", 2024-09-21T15:50:03.411 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3616999999999998e-05 2024-09-21T15:50:03.411 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.411 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.411 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:03.411 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201765+0000", 2024-09-21T15:50:03.411 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3464000000000002e-05 2024-09-21T15:50:03.411 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.411 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:03.411 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.411 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.411 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.411 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:118351 3.4 3:25980f86:::rbd_data.1133545d3fa6.000000000000005a:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-09-21T15:50:03.411 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:58.975795+0000", 2024-09-21T15:50:03.411 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.360898852, 2024-09-21T15:50:03.411 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.99016351599999997, 2024-09-21T15:50:03.412 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:03.412 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:03.412 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:03.412 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:03.412 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:03.412 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 118351 2024-09-21T15:50:03.412 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.412 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:03.412 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.412 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:03.412 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975795+0000", 2024-09-21T15:50:03.412 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.412 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.412 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.412 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:03.412 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975795+0000", 2024-09-21T15:50:03.412 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.413 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.413 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.413 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:03.413 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975793+0000", 2024-09-21T15:50:03.413 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:50:03.413 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.413 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.413 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:03.413 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975800+0000", 2024-09-21T15:50:03.413 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8220000000000001e-06 2024-09-21T15:50:03.413 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.413 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.413 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:03.413 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975814+0000", 2024-09-21T15:50:03.413 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3893000000000001e-05 2024-09-21T15:50:03.413 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.414 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.414 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:03.414 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975821+0000", 2024-09-21T15:50:03.414 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1319999999999996e-06 2024-09-21T15:50:03.414 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.414 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.414 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:03.414 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975878+0000", 2024-09-21T15:50:03.414 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7238999999999999e-05 2024-09-21T15:50:03.414 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.414 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.414 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:50:03.414 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975889+0000", 2024-09-21T15:50:03.414 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0509000000000001e-05 2024-09-21T15:50:03.414 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.414 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.415 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:03.415 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.964573+0000", 2024-09-21T15:50:03.415 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.98868396300000005 2024-09-21T15:50:03.415 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.415 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.415 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:03.415 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.964672+0000", 2024-09-21T15:50:03.415 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9601999999999999e-05 2024-09-21T15:50:03.415 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.415 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.415 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-09-21T15:50:03.415 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.964758+0000", 2024-09-21T15:50:03.415 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5537000000000005e-05 2024-09-21T15:50:03.415 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.415 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.416 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:03.416 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965847+0000", 2024-09-21T15:50:03.416 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001089404 2024-09-21T15:50:03.416 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.416 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.416 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:03.416 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965890+0000", 2024-09-21T15:50:03.416 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2521999999999997e-05 2024-09-21T15:50:03.416 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.416 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.416 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:03.416 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965934+0000", 2024-09-21T15:50:03.416 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4489e-05 2024-09-21T15:50:03.416 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.416 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.416 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:03.417 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965959+0000", 2024-09-21T15:50:03.417 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4301999999999999e-05 2024-09-21T15:50:03.417 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.417 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:03.417 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.417 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.417 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.417 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:118353 3.4 3:23841e18:::rbd_data.1133545d3fa6.0000000000000061:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-09-21T15:50:03.417 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:58.975836+0000", 2024-09-21T15:50:03.417 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.3608573530000001, 2024-09-21T15:50:03.417 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.99077746700000002, 2024-09-21T15:50:03.417 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:03.417 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:03.417 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:03.417 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:03.417 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:03.417 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 118353 2024-09-21T15:50:03.418 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.418 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:03.418 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.418 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:03.418 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975836+0000", 2024-09-21T15:50:03.418 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.418 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.418 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.418 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:03.418 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975836+0000", 2024-09-21T15:50:03.418 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.418 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.418 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.418 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:03.418 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975836+0000", 2024-09-21T15:50:03.419 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:03.419 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.419 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.419 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:03.419 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975839+0000", 2024-09-21T15:50:03.419 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6129999999999998e-06 2024-09-21T15:50:03.419 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.419 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.419 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:03.419 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975844+0000", 2024-09-21T15:50:03.419 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5130000000000004e-06 2024-09-21T15:50:03.419 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.419 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.419 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:03.419 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975850+0000", 2024-09-21T15:50:03.419 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3179999999999997e-06 2024-09-21T15:50:03.420 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.420 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.420 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:03.420 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975915+0000", 2024-09-21T15:50:03.420 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5168999999999999e-05 2024-09-21T15:50:03.420 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.420 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.420 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:50:03.420 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975922+0000", 2024-09-21T15:50:03.420 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2749999999999998e-06 2024-09-21T15:50:03.420 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.420 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.420 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:03.420 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965431+0000", 2024-09-21T15:50:03.420 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.98950864699999996 2024-09-21T15:50:03.420 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.421 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.421 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:03.421 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965535+0000", 2024-09-21T15:50:03.421 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000103949 2024-09-21T15:50:03.421 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.421 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.421 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-09-21T15:50:03.421 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965607+0000", 2024-09-21T15:50:03.421 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2478000000000006e-05 2024-09-21T15:50:03.421 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.421 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.421 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:03.421 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.966205+0000", 2024-09-21T15:50:03.421 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00059775200000000001 2024-09-21T15:50:03.421 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.422 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.422 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:03.422 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.966562+0000", 2024-09-21T15:50:03.422 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00035732199999999998 2024-09-21T15:50:03.422 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.422 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.422 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:03.422 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.966593+0000", 2024-09-21T15:50:03.422 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0642999999999998e-05 2024-09-21T15:50:03.422 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.422 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.422 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:03.422 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.966614+0000", 2024-09-21T15:50:03.422 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1149999999999999e-05 2024-09-21T15:50:03.422 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.422 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:03.422 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.423 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.423 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:03.423 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:03.424 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:50:03.424 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:03.423+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:03.525 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:03.525+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:03.526 INFO:teuthology.orchestra.run.smithi089.stdout:4140 writesame 0xc7b922b thru 0xc7bdce0 data_size 0x2(0x4ab6 bytes) 2024-09-21T15:50:03.540 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:03.540 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:03.540 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:50:03.540 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:03.548 INFO:teuthology.orchestra.run.smithi089.stdout:4141 write 0x417013f thru 0x41768be (0x6780 bytes) 2024-09-21T15:50:03.551 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:50:03.566 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:03.566 INFO:teuthology.orchestra.run.smithi089.stdout:4143 punch from 0x25b1f6 to 0x2623db, (0x71e5 bytes) 2024-09-21T15:50:03.566 INFO:teuthology.orchestra.run.smithi089.stdout:4144 write 0x176157c thru 0x1767d16 (0x679b bytes) 2024-09-21T15:50:03.626 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:03.625+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:03.659 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:03.660 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:03.660 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:50:03.660 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:50:03.660 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:03.669 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:50:03.673 INFO:teuthology.orchestra.run.smithi089.stdout:4145 writesame 0x5c7b232 thru 0x5c812b8 data_size 0x1(0x6087 bytes) 2024-09-21T15:50:03.690 INFO:teuthology.orchestra.run.smithi089.stdout:4146 write 0x49de20e thru 0x49edef8 (0xfceb bytes) 2024-09-21T15:50:03.703 INFO:teuthology.orchestra.run.smithi089.stdout:4151 write 0xd72573c thru 0xd7313f4 (0xbcb9 bytes) 2024-09-21T15:50:03.716 INFO:teuthology.orchestra.run.smithi089.stdout:4152 writesame 0x28bbfac thru 0x28be835 data_size 0x1(0x288a bytes) 2024-09-21T15:50:03.726 INFO:teuthology.orchestra.run.smithi089.stdout:4153 writesame 0xac9e0c1 thru 0xaca6649 data_size 0x35(0x8589 bytes) 2024-09-21T15:50:03.727 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:03.726+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:03.741 INFO:teuthology.orchestra.run.smithi089.stdout:4154 read 0xd94bd5b thru 0xd9584a0 (0xc746 bytes) 2024-09-21T15:50:03.742 INFO:teuthology.orchestra.run.smithi089.stdout:4156 writesame 0xcd853e thru 0xce730e data_size 0x1(0xedd1 bytes) 2024-09-21T15:50:03.795 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:03.796 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:50:03.796 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:50:03.796 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:50:03.796 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.796 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:984 3.0 e27/22 3:025e53e7:::rbd_data.1133f3dfbd6b.0000000000000008:head v 27'21, mlcod=27'19)", 2024-09-21T15:50:03.796 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.619393+0000", 2024-09-21T15:50:03.796 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 51.174387682999999, 2024-09-21T15:50:03.796 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003401837, 2024-09-21T15:50:03.796 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:03.796 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:03.796 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:03.796 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.797 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:03.797 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619393+0000", 2024-09-21T15:50:03.797 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.797 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.797 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:03.797 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619393+0000", 2024-09-21T15:50:03.797 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.797 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.797 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:03.797 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619392+0000", 2024-09-21T15:50:03.797 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:03.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.798 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.798 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:03.798 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619439+0000", 2024-09-21T15:50:03.798 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7170999999999999e-05 2024-09-21T15:50:03.798 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.798 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.798 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:03.798 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619452+0000", 2024-09-21T15:50:03.798 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2986999999999999e-05 2024-09-21T15:50:03.798 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.798 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.798 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:03.798 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619459+0000", 2024-09-21T15:50:03.798 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6109999999999999e-06 2024-09-21T15:50:03.799 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.799 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.799 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:03.799 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619518+0000", 2024-09-21T15:50:03.799 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9117999999999999e-05 2024-09-21T15:50:03.799 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.799 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.799 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:03.799 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619532+0000", 2024-09-21T15:50:03.799 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4622e-05 2024-09-21T15:50:03.799 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.799 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.799 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:03.800 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.622760+0000", 2024-09-21T15:50:03.800 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032278839999999999 2024-09-21T15:50:03.800 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.800 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.800 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:03.800 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.622794+0000", 2024-09-21T15:50:03.800 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4144999999999999e-05 2024-09-21T15:50:03.800 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.800 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:03.800 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.800 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.800 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.800 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:4957 3.9 e29/22 3:95c5ed95:::rbd_data.11335fa2fe69.0000000000000018:head v 29'4, mlcod=26'2)", 2024-09-21T15:50:03.801 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:14.453951+0000", 2024-09-21T15:50:03.801 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 49.339829250999998, 2024-09-21T15:50:03.801 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0043804559999999996, 2024-09-21T15:50:03.801 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:03.801 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:03.801 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:03.801 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.801 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:03.801 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.453951+0000", 2024-09-21T15:50:03.801 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.801 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.801 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.801 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:03.802 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.453951+0000", 2024-09-21T15:50:03.802 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.802 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.802 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.802 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:03.802 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.453950+0000", 2024-09-21T15:50:03.802 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:03.802 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.802 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.802 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:03.802 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454495+0000", 2024-09-21T15:50:03.802 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00054437799999999998 2024-09-21T15:50:03.802 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.802 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.803 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:03.803 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454508+0000", 2024-09-21T15:50:03.803 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3475000000000001e-05 2024-09-21T15:50:03.803 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.803 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.803 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:03.803 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454515+0000", 2024-09-21T15:50:03.803 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8990000000000004e-06 2024-09-21T15:50:03.803 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.803 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.803 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:03.803 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454581+0000", 2024-09-21T15:50:03.803 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5987000000000004e-05 2024-09-21T15:50:03.803 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.803 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.804 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:03.804 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454609+0000", 2024-09-21T15:50:03.804 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7466000000000001e-05 2024-09-21T15:50:03.804 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.804 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.804 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:03.804 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.458303+0000", 2024-09-21T15:50:03.804 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003693979 2024-09-21T15:50:03.804 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.804 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.804 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:03.804 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.458331+0000", 2024-09-21T15:50:03.804 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8932999999999999e-05 2024-09-21T15:50:03.805 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.805 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:03.805 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.805 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.805 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.805 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13420 3.0 e31/22 3:038b8840:::rbd_data.1133dc5adfdf.0000000000000035:head v 31'49, mlcod=31'47)", 2024-09-21T15:50:03.805 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.209233+0000", 2024-09-21T15:50:03.805 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 46.584547635, 2024-09-21T15:50:03.805 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0042624960000000002, 2024-09-21T15:50:03.805 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:03.805 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:03.805 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:03.805 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.805 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:03.806 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209233+0000", 2024-09-21T15:50:03.806 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.806 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.806 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.806 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:03.806 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209233+0000", 2024-09-21T15:50:03.806 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.806 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.806 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.806 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:03.806 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209232+0000", 2024-09-21T15:50:03.806 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:03.806 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.806 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.806 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:03.807 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209251+0000", 2024-09-21T15:50:03.807 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9304e-05 2024-09-21T15:50:03.807 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.807 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.807 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:03.807 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209265+0000", 2024-09-21T15:50:03.807 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3532e-05 2024-09-21T15:50:03.807 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.807 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.807 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:03.807 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209272+0000", 2024-09-21T15:50:03.807 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.289e-06 2024-09-21T15:50:03.807 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.807 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.808 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:03.808 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209321+0000", 2024-09-21T15:50:03.808 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8773999999999998e-05 2024-09-21T15:50:03.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.808 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:03.808 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209341+0000", 2024-09-21T15:50:03.808 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0387999999999999e-05 2024-09-21T15:50:03.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.808 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:03.808 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.213464+0000", 2024-09-21T15:50:03.808 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0041232450000000002 2024-09-21T15:50:03.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.808 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:03.809 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.213495+0000", 2024-09-21T15:50:03.809 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0793999999999997e-05 2024-09-21T15:50:03.809 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.809 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:03.809 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.809 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13445 3.0 e31/22 3:038b8840:::rbd_data.1133dc5adfdf.0000000000000035:head v 31'50, mlcod=31'48)", 2024-09-21T15:50:03.809 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.232497+0000", 2024-09-21T15:50:03.809 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 46.561283623000001, 2024-09-21T15:50:03.809 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0040493559999999996, 2024-09-21T15:50:03.809 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:03.809 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:03.810 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:03.810 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.810 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:03.810 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232497+0000", 2024-09-21T15:50:03.810 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.810 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.810 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:03.810 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232497+0000", 2024-09-21T15:50:03.810 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.810 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.810 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:03.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232496+0000", 2024-09-21T15:50:03.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:03.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:03.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232503+0000", 2024-09-21T15:50:03.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7320000000000001e-06 2024-09-21T15:50:03.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:03.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232512+0000", 2024-09-21T15:50:03.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9259999999999995e-06 2024-09-21T15:50:03.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:03.812 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232520+0000", 2024-09-21T15:50:03.812 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3699999999999997e-06 2024-09-21T15:50:03.812 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.812 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.812 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:03.812 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232564+0000", 2024-09-21T15:50:03.812 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4128999999999997e-05 2024-09-21T15:50:03.812 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.812 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.812 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:03.812 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232585+0000", 2024-09-21T15:50:03.812 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0704999999999999e-05 2024-09-21T15:50:03.812 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.812 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.813 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:03.813 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.236507+0000", 2024-09-21T15:50:03.813 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0039218730000000002 2024-09-21T15:50:03.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.813 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:03.813 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.236546+0000", 2024-09-21T15:50:03.813 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9399e-05 2024-09-21T15:50:03.813 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.813 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:03.813 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.813 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13452 3.9 e31/22 3:93f5396a:::rbd_data.1133dc5adfdf.0000000000000078:head v 31'50, mlcod=31'47)", 2024-09-21T15:50:03.813 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.237263+0000", 2024-09-21T15:50:03.814 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 46.556516917000003, 2024-09-21T15:50:03.814 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033768180000000002, 2024-09-21T15:50:03.814 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:03.814 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:03.814 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:03.814 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:03.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237263+0000", 2024-09-21T15:50:03.814 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.814 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:03.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237263+0000", 2024-09-21T15:50:03.814 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.815 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.815 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:03.815 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237263+0000", 2024-09-21T15:50:03.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:03.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.815 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.815 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:03.815 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237269+0000", 2024-09-21T15:50:03.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.472e-06 2024-09-21T15:50:03.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.815 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.815 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:03.815 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237282+0000", 2024-09-21T15:50:03.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2788e-05 2024-09-21T15:50:03.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.816 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.816 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:03.816 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237288+0000", 2024-09-21T15:50:03.816 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4899999999999997e-06 2024-09-21T15:50:03.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.816 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.816 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:03.816 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237331+0000", 2024-09-21T15:50:03.816 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2425000000000001e-05 2024-09-21T15:50:03.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.816 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.816 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:03.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237349+0000", 2024-09-21T15:50:03.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8405000000000001e-05 2024-09-21T15:50:03.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.817 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:03.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.240610+0000", 2024-09-21T15:50:03.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032605379999999999 2024-09-21T15:50:03.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.817 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:03.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.240640+0000", 2024-09-21T15:50:03.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0448000000000001e-05 2024-09-21T15:50:03.817 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.817 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:03.817 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.818 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.818 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.818 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13832 3.0 e31/22 3:0792efd2:::rbd_data.1133dc5adfdf.0000000000000090:head v 31'58, mlcod=31'56)", 2024-09-21T15:50:03.818 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.527530+0000", 2024-09-21T15:50:03.818 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 46.266250096999997, 2024-09-21T15:50:03.818 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035256530000000001, 2024-09-21T15:50:03.818 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:03.818 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:03.818 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:03.818 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.818 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:03.818 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527530+0000", 2024-09-21T15:50:03.818 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.818 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.819 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:03.819 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527530+0000", 2024-09-21T15:50:03.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.819 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:03.819 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527529+0000", 2024-09-21T15:50:03.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:50:03.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.819 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:03.819 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527550+0000", 2024-09-21T15:50:03.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1387999999999999e-05 2024-09-21T15:50:03.820 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.820 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.820 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:03.820 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527568+0000", 2024-09-21T15:50:03.820 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8068e-05 2024-09-21T15:50:03.820 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.820 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.820 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:03.820 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527577+0000", 2024-09-21T15:50:03.820 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9919999999999996e-06 2024-09-21T15:50:03.820 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.820 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.820 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:03.820 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527626+0000", 2024-09-21T15:50:03.820 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8597000000000002e-05 2024-09-21T15:50:03.821 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.821 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.821 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:03.821 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527652+0000", 2024-09-21T15:50:03.821 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5993e-05 2024-09-21T15:50:03.821 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.821 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.821 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:03.821 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.531024+0000", 2024-09-21T15:50:03.821 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033721910000000001 2024-09-21T15:50:03.821 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.821 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.821 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:03.821 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.531056+0000", 2024-09-21T15:50:03.821 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1628000000000003e-05 2024-09-21T15:50:03.822 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.822 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:03.822 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.822 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.822 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.822 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13839 3.0 e31/22 3:03c4fb32:::rbd_data.1133dc5adfdf.0000000000000099:head v 31'59, mlcod=31'56)", 2024-09-21T15:50:03.822 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.528097+0000", 2024-09-21T15:50:03.822 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 46.265683498999998, 2024-09-21T15:50:03.822 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0057228840000000001, 2024-09-21T15:50:03.822 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:03.822 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:03.822 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:03.822 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.822 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:03.823 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528097+0000", 2024-09-21T15:50:03.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.823 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.823 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:03.823 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528097+0000", 2024-09-21T15:50:03.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.823 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.823 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:03.823 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528096+0000", 2024-09-21T15:50:03.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:03.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.823 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.823 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:03.824 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528105+0000", 2024-09-21T15:50:03.824 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7350000000000002e-06 2024-09-21T15:50:03.824 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.824 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:03.824 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528117+0000", 2024-09-21T15:50:03.824 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1909000000000001e-05 2024-09-21T15:50:03.824 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.824 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:03.824 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528124+0000", 2024-09-21T15:50:03.824 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8419999999999999e-06 2024-09-21T15:50:03.824 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.825 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:03.825 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528168+0000", 2024-09-21T15:50:03.825 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4301999999999997e-05 2024-09-21T15:50:03.825 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.825 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.825 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:03.825 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528191+0000", 2024-09-21T15:50:03.825 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2622000000000001e-05 2024-09-21T15:50:03.825 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.825 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.825 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:03.825 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.533780+0000", 2024-09-21T15:50:03.825 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0055897519999999999 2024-09-21T15:50:03.825 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.825 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.826 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:03.826 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.533820+0000", 2024-09-21T15:50:03.826 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9366999999999998e-05 2024-09-21T15:50:03.826 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.826 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:03.826 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.826 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.826 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.826 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13884 3.9 e31/22 3:93003736:::rbd_data.1133dc5adfdf.00000000000000a1:head v 31'54, mlcod=31'52)", 2024-09-21T15:50:03.826 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.562433+0000", 2024-09-21T15:50:03.826 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 46.231346750999997, 2024-09-21T15:50:03.826 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.005162313, 2024-09-21T15:50:03.826 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:03.826 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:03.826 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:03.827 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.827 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:03.827 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562433+0000", 2024-09-21T15:50:03.827 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.827 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.827 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.827 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:03.827 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562433+0000", 2024-09-21T15:50:03.827 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.827 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.827 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.827 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:03.827 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562432+0000", 2024-09-21T15:50:03.827 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:03.828 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.828 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.828 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:03.828 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562467+0000", 2024-09-21T15:50:03.828 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4885999999999998e-05 2024-09-21T15:50:03.828 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.828 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.828 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:03.828 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562481+0000", 2024-09-21T15:50:03.828 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3604000000000001e-05 2024-09-21T15:50:03.828 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.828 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.828 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:03.828 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562489+0000", 2024-09-21T15:50:03.828 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1829999999999994e-06 2024-09-21T15:50:03.829 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.829 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.829 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:03.829 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562536+0000", 2024-09-21T15:50:03.829 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6830999999999999e-05 2024-09-21T15:50:03.829 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.829 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.829 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:03.829 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562559+0000", 2024-09-21T15:50:03.829 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2765999999999999e-05 2024-09-21T15:50:03.829 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.829 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.829 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:03.829 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.567566+0000", 2024-09-21T15:50:03.829 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0050073219999999998 2024-09-21T15:50:03.830 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.830 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.830 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:03.830 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.567596+0000", 2024-09-21T15:50:03.830 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9802999999999999e-05 2024-09-21T15:50:03.830 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.830 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:03.830 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.830 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.830 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.830 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:22340 3.9 e34/22 3:910a5abe:::rbd_data.1133896adcfd.0000000000000023:head v 34'92, mlcod=34'90)", 2024-09-21T15:50:03.830 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:21.628025+0000", 2024-09-21T15:50:03.830 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.165755371000003, 2024-09-21T15:50:03.831 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046196010000000001, 2024-09-21T15:50:03.831 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:03.831 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:03.831 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:03.831 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.831 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:03.831 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628025+0000", 2024-09-21T15:50:03.831 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.831 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.831 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:03.831 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628025+0000", 2024-09-21T15:50:03.831 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.832 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.832 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:03.832 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628024+0000", 2024-09-21T15:50:03.832 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:03.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.832 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.832 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:03.832 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628034+0000", 2024-09-21T15:50:03.832 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3470000000000005e-06 2024-09-21T15:50:03.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.832 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.832 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:03.832 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628043+0000", 2024-09-21T15:50:03.832 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9450000000000005e-06 2024-09-21T15:50:03.833 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.833 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.833 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:03.833 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628048+0000", 2024-09-21T15:50:03.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8080000000000004e-06 2024-09-21T15:50:03.833 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.833 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.833 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:03.833 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628081+0000", 2024-09-21T15:50:03.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2738999999999999e-05 2024-09-21T15:50:03.833 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.833 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.833 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:03.833 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628094+0000", 2024-09-21T15:50:03.834 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3273e-05 2024-09-21T15:50:03.834 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.834 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.834 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:03.834 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.632616+0000", 2024-09-21T15:50:03.834 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045217269999999997 2024-09-21T15:50:03.834 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.834 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.834 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:03.834 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.632644+0000", 2024-09-21T15:50:03.834 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8469000000000001e-05 2024-09-21T15:50:03.834 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.835 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:03.835 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.835 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.835 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:22417 3.4 e34/22 3:25310a10:::rbd_data.1133896adcfd.0000000000000075:head v 34'48, mlcod=34'46)", 2024-09-21T15:50:03.835 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:21.704297+0000", 2024-09-21T15:50:03.835 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.089483272999999, 2024-09-21T15:50:03.835 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0038101799999999998, 2024-09-21T15:50:03.835 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:03.835 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:03.835 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:03.835 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.835 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:03.835 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704297+0000", 2024-09-21T15:50:03.836 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.836 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.836 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.836 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:03.836 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704297+0000", 2024-09-21T15:50:03.836 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.836 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.836 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.836 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:03.836 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704296+0000", 2024-09-21T15:50:03.836 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:50:03.836 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.836 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.836 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:03.837 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704308+0000", 2024-09-21T15:50:03.837 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2559e-05 2024-09-21T15:50:03.837 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.837 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.837 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:03.837 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704337+0000", 2024-09-21T15:50:03.837 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8960000000000001e-05 2024-09-21T15:50:03.837 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.837 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.837 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:03.837 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704345+0000", 2024-09-21T15:50:03.837 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3059999999999998e-06 2024-09-21T15:50:03.837 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.838 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.838 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:03.838 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704408+0000", 2024-09-21T15:50:03.838 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2156000000000006e-05 2024-09-21T15:50:03.838 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.838 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.838 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:03.838 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704430+0000", 2024-09-21T15:50:03.838 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1931000000000001e-05 2024-09-21T15:50:03.838 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.838 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.838 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:03.838 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.708073+0000", 2024-09-21T15:50:03.838 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003643028 2024-09-21T15:50:03.839 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.839 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.839 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:03.839 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.708107+0000", 2024-09-21T15:50:03.839 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4613999999999998e-05 2024-09-21T15:50:03.839 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.839 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:03.839 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.839 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.839 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.839 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:25278 3.4 e38/22 3:2251cc1c:::rbd_data.1133ebd6d3ba.0000000000000024:head v 38'51, mlcod=38'49)", 2024-09-21T15:50:03.839 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.317337+0000", 2024-09-21T15:50:03.839 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 37.476443513, 2024-09-21T15:50:03.839 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053904110000000003, 2024-09-21T15:50:03.840 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:03.840 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:03.840 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:03.840 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.840 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:03.840 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317337+0000", 2024-09-21T15:50:03.840 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.840 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.840 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.840 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:03.840 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317337+0000", 2024-09-21T15:50:03.840 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.840 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.841 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.841 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:03.841 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317335+0000", 2024-09-21T15:50:03.841 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:50:03.841 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.841 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.841 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:03.841 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317347+0000", 2024-09-21T15:50:03.841 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.19e-05 2024-09-21T15:50:03.841 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.841 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.841 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:03.841 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317369+0000", 2024-09-21T15:50:03.841 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2232e-05 2024-09-21T15:50:03.842 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.842 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.842 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:03.842 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317381+0000", 2024-09-21T15:50:03.842 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2231e-05 2024-09-21T15:50:03.842 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.842 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.842 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:03.842 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317425+0000", 2024-09-21T15:50:03.842 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3625000000000003e-05 2024-09-21T15:50:03.842 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.842 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.842 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:03.843 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317449+0000", 2024-09-21T15:50:03.843 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4301999999999999e-05 2024-09-21T15:50:03.843 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.843 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.843 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:03.843 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.322688+0000", 2024-09-21T15:50:03.843 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.005238609 2024-09-21T15:50:03.843 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.843 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.843 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:03.843 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.322727+0000", 2024-09-21T15:50:03.843 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9201e-05 2024-09-21T15:50:03.843 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.843 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:03.843 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.844 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.844 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.844 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:25286 3.0 e38/22 3:0248fb60:::rbd_data.1133ebd6d3ba.0000000000000066:head v 38'109, mlcod=34'107)", 2024-09-21T15:50:03.844 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.317638+0000", 2024-09-21T15:50:03.844 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 37.476142633999999, 2024-09-21T15:50:03.844 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0054074370000000002, 2024-09-21T15:50:03.844 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:03.844 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:03.844 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:03.844 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.844 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:03.844 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317638+0000", 2024-09-21T15:50:03.844 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.844 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.845 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.845 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:03.845 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317638+0000", 2024-09-21T15:50:03.845 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.845 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.845 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.845 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:03.845 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317637+0000", 2024-09-21T15:50:03.845 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:03.845 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.845 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.845 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:03.845 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317643+0000", 2024-09-21T15:50:03.845 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5609999999999998e-06 2024-09-21T15:50:03.846 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.846 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.846 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:03.846 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317653+0000", 2024-09-21T15:50:03.846 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0705e-05 2024-09-21T15:50:03.846 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.846 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.846 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:03.846 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317660+0000", 2024-09-21T15:50:03.846 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2099999999999998e-06 2024-09-21T15:50:03.846 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.846 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.846 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:03.847 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317704+0000", 2024-09-21T15:50:03.847 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4659000000000001e-05 2024-09-21T15:50:03.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.847 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.847 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:03.847 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317725+0000", 2024-09-21T15:50:03.847 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0537e-05 2024-09-21T15:50:03.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.847 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.847 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:03.847 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323010+0000", 2024-09-21T15:50:03.847 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052853420000000002 2024-09-21T15:50:03.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.847 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.848 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:03.848 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323045+0000", 2024-09-21T15:50:03.848 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4860000000000002e-05 2024-09-21T15:50:03.848 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.848 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:03.848 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.848 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.848 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.848 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:25280 3.4 e38/22 3:27fca38c:::rbd_data.1133ebd6d3ba.0000000000000026:head v 38'52, mlcod=38'49)", 2024-09-21T15:50:03.848 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.317684+0000", 2024-09-21T15:50:03.848 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 37.476096738000003, 2024-09-21T15:50:03.848 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053570809999999996, 2024-09-21T15:50:03.848 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:03.848 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:03.849 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:03.849 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.849 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:03.849 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317684+0000", 2024-09-21T15:50:03.849 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.849 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.849 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.849 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:03.849 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317684+0000", 2024-09-21T15:50:03.849 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.849 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.849 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.849 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:03.849 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317683+0000", 2024-09-21T15:50:03.849 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:03.849 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.850 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.850 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:03.850 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317688+0000", 2024-09-21T15:50:03.850 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5009999999999998e-06 2024-09-21T15:50:03.850 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.850 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.850 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:03.850 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317694+0000", 2024-09-21T15:50:03.850 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3579999999999999e-06 2024-09-21T15:50:03.850 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.850 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.850 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:03.850 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317698+0000", 2024-09-21T15:50:03.850 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4150000000000001e-06 2024-09-21T15:50:03.850 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.851 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.851 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:03.851 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317738+0000", 2024-09-21T15:50:03.851 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9575e-05 2024-09-21T15:50:03.851 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.851 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.851 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:03.851 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317762+0000", 2024-09-21T15:50:03.851 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4173999999999999e-05 2024-09-21T15:50:03.851 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.851 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.851 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:03.851 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323005+0000", 2024-09-21T15:50:03.852 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00524333 2024-09-21T15:50:03.852 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.852 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.852 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:03.852 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323041+0000", 2024-09-21T15:50:03.852 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5139999999999999e-05 2024-09-21T15:50:03.852 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.852 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:03.852 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.852 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.852 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.852 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:34837 3.0 e43/22 3:038527c8:::rbd_header.1133fed1a840:head v 43'130, mlcod=43'128)", 2024-09-21T15:50:03.852 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:50:03.852 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 32.809892040000001, 2024-09-21T15:50:03.853 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045776519999999998, 2024-09-21T15:50:03.853 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:03.853 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:03.853 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:03.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.853 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:03.853 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:50:03.853 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.853 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.853 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:03.853 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:50:03.853 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.853 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.854 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:03.854 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:50:03.854 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:03.854 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.854 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.854 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:03.854 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983892+0000", 2024-09-21T15:50:03.854 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8450000000000002e-06 2024-09-21T15:50:03.854 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.854 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.854 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:03.854 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983902+0000", 2024-09-21T15:50:03.854 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0299999999999999e-06 2024-09-21T15:50:03.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.855 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:03.855 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983906+0000", 2024-09-21T15:50:03.855 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2300000000000002e-06 2024-09-21T15:50:03.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.855 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:03.855 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983940+0000", 2024-09-21T15:50:03.855 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4563000000000001e-05 2024-09-21T15:50:03.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.855 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:03.855 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983960+0000", 2024-09-21T15:50:03.856 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9199999999999999e-05 2024-09-21T15:50:03.856 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.856 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.856 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:03.856 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.988430+0000", 2024-09-21T15:50:03.856 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0044709149999999998 2024-09-21T15:50:03.856 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.856 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.856 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:03.856 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.988466+0000", 2024-09-21T15:50:03.856 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5420000000000003e-05 2024-09-21T15:50:03.856 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.856 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:03.856 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.856 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.857 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.857 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "pg_update_log_missing(3.0 epoch 50/22 rep_tid 498 entries 50'171 (0'0) error 3:5df4a5a0:::rbd_data.11339e87a32a.000000000000008e:head by client.4403.0:80207 0.000000 -2 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 47'167)", 2024-09-21T15:50:03.857 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:50:03.857 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 22.852019167000002, 2024-09-21T15:50:03.857 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0047253119999999997, 2024-09-21T15:50:03.857 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:03.857 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-09-21T15:50:03.857 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:03.857 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.857 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:03.857 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:50:03.857 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.857 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.857 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.857 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:03.858 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:50:03.858 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.858 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.858 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.858 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:03.858 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:50:03.858 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:03.858 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.858 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.858 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:03.858 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941768+0000", 2024-09-21T15:50:03.858 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7309999999999993e-06 2024-09-21T15:50:03.858 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.858 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.859 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:03.859 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941780+0000", 2024-09-21T15:50:03.859 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1399999999999999e-05 2024-09-21T15:50:03.859 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.859 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.859 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:03.859 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941789+0000", 2024-09-21T15:50:03.859 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5230000000000001e-06 2024-09-21T15:50:03.859 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.859 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.859 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:03.859 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941841+0000", 2024-09-21T15:50:03.860 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1863999999999997e-05 2024-09-21T15:50:03.860 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.860 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.860 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:03.860 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.946486+0000", 2024-09-21T15:50:03.860 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046452539999999997 2024-09-21T15:50:03.860 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.860 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:03.860 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.860 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.860 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.860 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:82086 3.0 e50/22 3:0114bc4d:::rbd_data.11334c14b8fd.0000000000000069:head v 50'178, mlcod=50'176)", 2024-09-21T15:50:03.860 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:41.155591+0000", 2024-09-21T15:50:03.860 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 22.638188854999999, 2024-09-21T15:50:03.861 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003363497, 2024-09-21T15:50:03.861 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:03.861 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:03.861 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:03.861 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.861 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:03.861 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.155591+0000", 2024-09-21T15:50:03.861 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.861 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.861 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.861 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:03.861 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.155591+0000", 2024-09-21T15:50:03.861 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.861 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.862 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.862 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:03.862 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.155590+0000", 2024-09-21T15:50:03.862 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:50:03.862 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.862 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.862 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:03.862 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.155606+0000", 2024-09-21T15:50:03.862 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6075000000000001e-05 2024-09-21T15:50:03.862 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.862 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.862 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:03.862 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.155623+0000", 2024-09-21T15:50:03.862 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6838e-05 2024-09-21T15:50:03.863 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.863 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.863 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:03.863 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.155630+0000", 2024-09-21T15:50:03.863 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.627e-06 2024-09-21T15:50:03.863 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.863 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.863 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:03.863 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.155677+0000", 2024-09-21T15:50:03.863 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6669999999999999e-05 2024-09-21T15:50:03.863 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.863 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.863 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:03.863 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.155698+0000", 2024-09-21T15:50:03.864 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1330000000000001e-05 2024-09-21T15:50:03.864 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.864 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.864 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:03.864 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.158918+0000", 2024-09-21T15:50:03.864 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032198600000000002 2024-09-21T15:50:03.864 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.864 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.864 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:03.864 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.158955+0000", 2024-09-21T15:50:03.864 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6647000000000003e-05 2024-09-21T15:50:03.864 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.864 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:03.864 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.865 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.865 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.865 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:82470 3.4 e50/22 3:21ad7dfe:::rbd_data.11334c14b8fd.00000000000003a7:head v 50'190, mlcod=46'188)", 2024-09-21T15:50:03.865 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:41.291787+0000", 2024-09-21T15:50:03.865 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 22.50199323, 2024-09-21T15:50:03.865 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033489969999999998, 2024-09-21T15:50:03.865 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:03.865 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:03.865 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:03.865 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.865 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:03.865 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.291787+0000", 2024-09-21T15:50:03.865 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.865 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.866 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.866 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:03.866 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.291787+0000", 2024-09-21T15:50:03.866 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.866 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.866 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.866 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:03.866 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.291786+0000", 2024-09-21T15:50:03.866 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:03.866 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.866 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.866 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:03.866 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.291795+0000", 2024-09-21T15:50:03.866 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7830000000000004e-06 2024-09-21T15:50:03.866 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.867 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.867 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:03.867 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.291814+0000", 2024-09-21T15:50:03.867 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9117999999999999e-05 2024-09-21T15:50:03.867 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.867 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.867 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:03.867 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.291822+0000", 2024-09-21T15:50:03.867 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0669999999999992e-06 2024-09-21T15:50:03.867 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.867 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.867 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:03.867 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.291866+0000", 2024-09-21T15:50:03.868 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4669000000000002e-05 2024-09-21T15:50:03.868 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.868 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.868 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:03.868 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.291888+0000", 2024-09-21T15:50:03.868 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1087999999999999e-05 2024-09-21T15:50:03.868 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.868 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.868 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:03.868 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.295104+0000", 2024-09-21T15:50:03.868 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032164910000000001 2024-09-21T15:50:03.868 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.868 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.868 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:03.869 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.295136+0000", 2024-09-21T15:50:03.869 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1931999999999999e-05 2024-09-21T15:50:03.869 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.869 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:03.869 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.869 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.869 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.869 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "pg_update_log_missing(3.0 epoch 53/22 rep_tid 585 entries 53'201 (0'0) error 3:6a1d4890:::rbd_data.1133e079cae4.0000000000000004:head by client.4403.0:101089 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 53'198)", 2024-09-21T15:50:03.869 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:44.832827+0000", 2024-09-21T15:50:03.869 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.960953359000001, 2024-09-21T15:50:03.869 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053552929999999997, 2024-09-21T15:50:03.869 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:03.869 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-09-21T15:50:03.869 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:03.869 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.870 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:03.870 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832827+0000", 2024-09-21T15:50:03.870 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.870 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.870 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.870 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:03.870 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832827+0000", 2024-09-21T15:50:03.870 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.870 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.870 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.870 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:03.870 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832826+0000", 2024-09-21T15:50:03.870 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:03.870 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.871 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.871 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:03.871 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832835+0000", 2024-09-21T15:50:03.871 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4509999999999992e-06 2024-09-21T15:50:03.871 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.871 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.871 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:03.871 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832854+0000", 2024-09-21T15:50:03.871 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9352999999999999e-05 2024-09-21T15:50:03.871 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.871 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.871 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:03.871 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832866+0000", 2024-09-21T15:50:03.871 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1766000000000001e-05 2024-09-21T15:50:03.872 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.872 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.872 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:03.872 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.833091+0000", 2024-09-21T15:50:03.872 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00022490300000000001 2024-09-21T15:50:03.872 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.872 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.872 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:03.872 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.838182+0000", 2024-09-21T15:50:03.872 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0050915420000000001 2024-09-21T15:50:03.872 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.872 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:03.872 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.872 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.873 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.873 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:102822 3.4 e54/22 3:2729ba03:::rbd_object_map.113336ae1075:head v 54'208, mlcod=54'205)", 2024-09-21T15:50:03.873 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:46.249540+0000", 2024-09-21T15:50:03.873 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.544240126999998, 2024-09-21T15:50:03.873 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052654859999999998, 2024-09-21T15:50:03.873 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:03.873 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:03.873 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:03.873 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.873 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:03.873 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249540+0000", 2024-09-21T15:50:03.873 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.873 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.873 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.874 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:03.874 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249540+0000", 2024-09-21T15:50:03.874 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.874 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.874 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.874 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:03.874 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249539+0000", 2024-09-21T15:50:03.874 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:03.874 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.874 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.874 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:03.874 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249546+0000", 2024-09-21T15:50:03.874 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7290000000000003e-06 2024-09-21T15:50:03.874 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.875 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.875 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:03.875 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249558+0000", 2024-09-21T15:50:03.875 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1844e-05 2024-09-21T15:50:03.875 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.875 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.875 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:03.875 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249565+0000", 2024-09-21T15:50:03.875 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8210000000000004e-06 2024-09-21T15:50:03.875 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.875 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.875 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:03.875 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249602+0000", 2024-09-21T15:50:03.875 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7057000000000001e-05 2024-09-21T15:50:03.876 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.876 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.876 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:03.876 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249622+0000", 2024-09-21T15:50:03.876 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9678999999999999e-05 2024-09-21T15:50:03.876 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.876 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.876 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:03.876 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254783+0000", 2024-09-21T15:50:03.876 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0051611449999999998 2024-09-21T15:50:03.876 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.876 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.876 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:03.876 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254806+0000", 2024-09-21T15:50:03.876 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2853e-05 2024-09-21T15:50:03.876 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.877 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:03.877 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.877 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.877 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.877 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:102823 3.4 e54/22 3:2729ba03:::rbd_object_map.113336ae1075:head v 54'209, mlcod=54'205)", 2024-09-21T15:50:03.877 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:46.250318+0000", 2024-09-21T15:50:03.877 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.543462542, 2024-09-21T15:50:03.877 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045013539999999999, 2024-09-21T15:50:03.877 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:03.877 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:03.877 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:03.877 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.877 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:03.877 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250318+0000", 2024-09-21T15:50:03.877 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.878 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.878 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.878 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:03.878 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250318+0000", 2024-09-21T15:50:03.878 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:03.878 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.878 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.878 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:03.878 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250317+0000", 2024-09-21T15:50:03.878 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:03.878 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.878 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.878 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:03.878 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250322+0000", 2024-09-21T15:50:03.879 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.074e-06 2024-09-21T15:50:03.879 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.879 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.879 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:03.879 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250330+0000", 2024-09-21T15:50:03.879 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4730000000000003e-06 2024-09-21T15:50:03.879 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.879 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.879 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:03.879 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250335+0000", 2024-09-21T15:50:03.879 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8019999999999997e-06 2024-09-21T15:50:03.879 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.879 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.880 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:03.880 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250375+0000", 2024-09-21T15:50:03.880 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9459999999999998e-05 2024-09-21T15:50:03.880 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.880 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.880 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:03.880 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250388+0000", 2024-09-21T15:50:03.880 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3288999999999999e-05 2024-09-21T15:50:03.880 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.880 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.880 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:03.880 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254807+0000", 2024-09-21T15:50:03.880 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0044192220000000004 2024-09-21T15:50:03.880 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:03.880 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:03.880 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:03.880 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254819+0000", 2024-09-21T15:50:03.881 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1603999999999999e-05 2024-09-21T15:50:03.881 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.881 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:03.881 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.881 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:03.881 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:03.881 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:03.882 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:50:03.883 INFO:teuthology.orchestra.run.smithi089.stdout:4157 write 0x80f950f thru 0x80f9e47 (0x939 bytes) 2024-09-21T15:50:03.884 INFO:teuthology.orchestra.run.smithi089.stdout:4158 writesame 0x83401d3 thru 0x834ed07 data_size 0x1(0xeb35 bytes) 2024-09-21T15:50:03.884 INFO:teuthology.orchestra.run.smithi089.stdout:4159 read 0x33a9d90 thru 0x33b30a7 (0x9318 bytes) 2024-09-21T15:50:03.884 INFO:teuthology.orchestra.run.smithi089.stdout:4160 punch from 0xd2d0a9d to 0xd2d5183, (0x46e6 bytes) 2024-09-21T15:50:03.884 INFO:teuthology.orchestra.run.smithi089.stdout:4161 read 0xc42da06 thru 0xc434c60 (0x725b bytes) 2024-09-21T15:50:03.884 INFO:teuthology.orchestra.run.smithi089.stdout:4162 writesame 0x927800 thru 0x92e71e data_size 0x1(0x6f1f bytes) 2024-09-21T15:50:03.884 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:03.881+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:03.909 INFO:teuthology.orchestra.run.smithi089.stdout:4163 writesame 0x5152f66 thru 0x515579a data_size 0x2f(0x2835 bytes) 2024-09-21T15:50:03.917 INFO:teuthology.orchestra.run.smithi089.stdout:4164 punch from 0xe2bdca to 0xe36a5e, (0xac94 bytes) 2024-09-21T15:50:03.917 INFO:teuthology.orchestra.run.smithi089.stdout:4165 punch from 0xc87656a to 0xc87b971, (0x5407 bytes) 2024-09-21T15:50:03.917 INFO:teuthology.orchestra.run.smithi089.stdout:4167 writesame 0xd5babad thru 0xd5c4228 data_size 0x2(0x967c bytes) 2024-09-21T15:50:03.932 INFO:teuthology.orchestra.run.smithi089.stdout:4168 read 0xbe3a949 thru 0xbe4037c (0x5a34 bytes) 2024-09-21T15:50:03.932 INFO:teuthology.orchestra.run.smithi089.stdout:4169 writesame 0xcd916be thru 0xcd9fd21 data_size 0x1e(0xe664 bytes) 2024-09-21T15:50:03.945 INFO:teuthology.orchestra.run.smithi089.stdout:4171 write 0x9246192 thru 0x924de71 (0x7ce0 bytes) 2024-09-21T15:50:03.960 INFO:teuthology.orchestra.run.smithi089.stdout:4173 punch from 0x43f4731 to 0x44019ba, (0xd289 bytes) 2024-09-21T15:50:03.960 INFO:teuthology.orchestra.run.smithi089.stdout:4174 write 0x3fd61d9 thru 0x3fdf26a (0x9092 bytes) 2024-09-21T15:50:03.983 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:03.982+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:03.998 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is unset 2024-09-21T15:50:04.059 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:04.194 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:50:04.195 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.195 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:123069 3.5 3.9ec96085 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-21T15:50:04.195 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:04.020216+0000", 2024-09-21T15:50:04.195 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.036779805999999998, 2024-09-21T15:50:04.195 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.036798422999999997, 2024-09-21T15:50:04.195 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:04.195 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-09-21T15:50:04.195 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:04.195 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:04.196 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:04.196 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 123069 2024-09-21T15:50:04.196 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.196 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:04.196 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.196 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:04.196 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020216+0000", 2024-09-21T15:50:04.196 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.196 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.196 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.196 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:04.196 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020216+0000", 2024-09-21T15:50:04.196 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.197 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.197 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.197 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:04.197 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020215+0000", 2024-09-21T15:50:04.197 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:04.197 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.197 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.197 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:04.197 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020218+0000", 2024-09-21T15:50:04.197 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6429999999999999e-06 2024-09-21T15:50:04.197 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.197 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:04.198 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020224+0000", 2024-09-21T15:50:04.198 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1720000000000004e-06 2024-09-21T15:50:04.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:04.198 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020229+0000", 2024-09-21T15:50:04.198 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.955e-06 2024-09-21T15:50:04.198 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.198 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:04.198 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.199 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:123070 3.5 3.9ec96085 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-21T15:50:04.199 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:04.020257+0000", 2024-09-21T15:50:04.199 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.036738986000000001, 2024-09-21T15:50:04.199 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.036794097999999997, 2024-09-21T15:50:04.199 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:04.199 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-09-21T15:50:04.199 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:04.199 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:04.199 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:04.199 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 123070 2024-09-21T15:50:04.199 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.199 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:04.199 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.199 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:04.199 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020257+0000", 2024-09-21T15:50:04.199 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.200 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.200 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.200 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:04.200 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020257+0000", 2024-09-21T15:50:04.200 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.200 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.200 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.200 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:04.200 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020256+0000", 2024-09-21T15:50:04.200 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:04.200 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.200 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.200 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:04.201 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020261+0000", 2024-09-21T15:50:04.201 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7110000000000003e-06 2024-09-21T15:50:04.201 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.201 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.201 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:04.201 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020276+0000", 2024-09-21T15:50:04.201 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5211999999999999e-05 2024-09-21T15:50:04.201 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.201 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.201 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:04.201 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020284+0000", 2024-09-21T15:50:04.201 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7379999999999994e-06 2024-09-21T15:50:04.201 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.201 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:04.202 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.202 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.202 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.202 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:123071 3.5 3.9ec96085 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-21T15:50:04.202 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:04.020291+0000", 2024-09-21T15:50:04.202 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.036704903999999997, 2024-09-21T15:50:04.202 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.036804641999999999, 2024-09-21T15:50:04.202 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:04.202 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-09-21T15:50:04.202 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:04.202 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:04.202 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:04.202 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 123071 2024-09-21T15:50:04.203 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.203 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:04.203 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.203 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:04.203 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020291+0000", 2024-09-21T15:50:04.203 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.203 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.203 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.203 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:04.203 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020291+0000", 2024-09-21T15:50:04.203 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.203 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.203 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.203 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:04.204 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020290+0000", 2024-09-21T15:50:04.204 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:04.204 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.204 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.204 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:04.204 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020293+0000", 2024-09-21T15:50:04.204 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9620000000000002e-06 2024-09-21T15:50:04.204 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.204 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.204 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:04.204 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020300+0000", 2024-09-21T15:50:04.204 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5980000000000002e-06 2024-09-21T15:50:04.204 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.204 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.205 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:04.205 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020305+0000", 2024-09-21T15:50:04.205 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0300000000000001e-06 2024-09-21T15:50:04.205 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.205 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:04.205 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.205 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.205 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.205 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:123072 3.5 3.9ec96085 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-21T15:50:04.205 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:04.020309+0000", 2024-09-21T15:50:04.205 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.036686694999999998, 2024-09-21T15:50:04.205 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.036820331999999997, 2024-09-21T15:50:04.205 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:04.205 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-09-21T15:50:04.206 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:04.206 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:04.206 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:04.206 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 123072 2024-09-21T15:50:04.206 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.206 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:04.206 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.206 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:04.206 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020309+0000", 2024-09-21T15:50:04.206 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.206 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.206 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.206 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:04.206 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020309+0000", 2024-09-21T15:50:04.207 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.207 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.207 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.207 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:04.207 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020309+0000", 2024-09-21T15:50:04.207 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:04.207 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.207 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.207 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:04.207 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020311+0000", 2024-09-21T15:50:04.207 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3259999999999998e-06 2024-09-21T15:50:04.207 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.207 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.207 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:04.208 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020317+0000", 2024-09-21T15:50:04.208 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7570000000000001e-06 2024-09-21T15:50:04.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.208 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.208 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:04.208 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020321+0000", 2024-09-21T15:50:04.208 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5739999999999997e-06 2024-09-21T15:50:04.208 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.208 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:04.208 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.208 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.208 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:123073 3.5 3.9ec96085 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-21T15:50:04.209 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:04.020325+0000", 2024-09-21T15:50:04.209 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.036670295999999998, 2024-09-21T15:50:04.209 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.036831225000000002, 2024-09-21T15:50:04.209 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:04.209 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-09-21T15:50:04.209 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:04.209 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:04.209 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:04.209 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 123073 2024-09-21T15:50:04.209 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.209 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:04.209 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.209 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:04.209 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020325+0000", 2024-09-21T15:50:04.210 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.210 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.210 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.210 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:04.210 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020325+0000", 2024-09-21T15:50:04.210 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.210 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.210 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.210 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:04.210 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020325+0000", 2024-09-21T15:50:04.210 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:04.210 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.210 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.211 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:04.211 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020333+0000", 2024-09-21T15:50:04.211 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9710000000000004e-06 2024-09-21T15:50:04.211 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.211 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.211 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:04.211 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020339+0000", 2024-09-21T15:50:04.211 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2299999999999996e-06 2024-09-21T15:50:04.211 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.211 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.211 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:04.211 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020344+0000", 2024-09-21T15:50:04.211 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9810000000000003e-06 2024-09-21T15:50:04.211 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.211 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:04.212 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.212 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.212 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.212 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:123074 3.5 3.9ec96085 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-21T15:50:04.212 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:04.020349+0000", 2024-09-21T15:50:04.212 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.036646293000000003, 2024-09-21T15:50:04.212 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.036841207000000001, 2024-09-21T15:50:04.212 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:04.212 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-09-21T15:50:04.212 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:04.212 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:04.212 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:04.212 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 123074 2024-09-21T15:50:04.212 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.213 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:04.213 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.213 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:04.213 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020349+0000", 2024-09-21T15:50:04.213 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.213 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.213 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.213 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:04.213 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020349+0000", 2024-09-21T15:50:04.213 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.213 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.213 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.213 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:04.213 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020349+0000", 2024-09-21T15:50:04.213 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:04.214 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.214 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.214 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:04.214 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020351+0000", 2024-09-21T15:50:04.214 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1890000000000001e-06 2024-09-21T15:50:04.214 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.214 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.214 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:04.214 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020357+0000", 2024-09-21T15:50:04.214 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0639999999999999e-06 2024-09-21T15:50:04.214 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.214 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.214 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:04.214 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020363+0000", 2024-09-21T15:50:04.215 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4210000000000003e-06 2024-09-21T15:50:04.215 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.215 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:04.215 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.215 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.215 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.215 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:123075 3.5 3.9ec96085 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-21T15:50:04.215 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:04.020367+0000", 2024-09-21T15:50:04.215 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.036628616000000003, 2024-09-21T15:50:04.215 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.036851683000000003, 2024-09-21T15:50:04.215 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:04.215 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-09-21T15:50:04.215 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:04.216 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:04.216 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:04.216 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 123075 2024-09-21T15:50:04.216 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.216 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:04.216 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.216 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:04.216 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020367+0000", 2024-09-21T15:50:04.216 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.216 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.216 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.216 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:04.216 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020367+0000", 2024-09-21T15:50:04.216 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.217 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.217 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.217 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:04.217 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020367+0000", 2024-09-21T15:50:04.217 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:04.217 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.217 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.217 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:04.217 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020369+0000", 2024-09-21T15:50:04.217 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3709999999999998e-06 2024-09-21T15:50:04.217 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.217 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.217 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:04.217 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020388+0000", 2024-09-21T15:50:04.218 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9261000000000001e-05 2024-09-21T15:50:04.218 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.218 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.218 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:04.218 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020403+0000", 2024-09-21T15:50:04.218 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4297000000000001e-05 2024-09-21T15:50:04.218 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.218 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:04.218 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.218 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.218 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.218 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:123076 3.5 3.9ec96085 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-21T15:50:04.218 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:04.020412+0000", 2024-09-21T15:50:04.218 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.036584136000000003, 2024-09-21T15:50:04.219 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.036835622999999998, 2024-09-21T15:50:04.219 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:04.219 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-09-21T15:50:04.219 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:04.219 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:04.219 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:04.219 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 123076 2024-09-21T15:50:04.219 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.219 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:04.219 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.219 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:04.219 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020412+0000", 2024-09-21T15:50:04.219 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.219 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.220 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.220 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:04.220 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020412+0000", 2024-09-21T15:50:04.220 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.220 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.220 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.220 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:04.220 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020411+0000", 2024-09-21T15:50:04.220 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:04.220 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.220 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.220 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:04.220 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020415+0000", 2024-09-21T15:50:04.220 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6779999999999999e-06 2024-09-21T15:50:04.221 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.221 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.221 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:04.221 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020426+0000", 2024-09-21T15:50:04.221 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0366999999999999e-05 2024-09-21T15:50:04.221 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.221 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.221 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:04.221 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020432+0000", 2024-09-21T15:50:04.221 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4609999999999996e-06 2024-09-21T15:50:04.221 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.221 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:04.221 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.221 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.222 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.222 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:123077 3.5 3.9ec96085 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-21T15:50:04.222 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:04.020438+0000", 2024-09-21T15:50:04.222 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.036557944000000002, 2024-09-21T15:50:04.222 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.03685066, 2024-09-21T15:50:04.222 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:04.222 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-09-21T15:50:04.222 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:04.222 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:04.222 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:04.222 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 123077 2024-09-21T15:50:04.222 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.222 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:04.222 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.223 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:04.223 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020438+0000", 2024-09-21T15:50:04.223 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.223 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.223 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.223 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:04.223 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020438+0000", 2024-09-21T15:50:04.223 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.223 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.223 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.223 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:04.223 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020437+0000", 2024-09-21T15:50:04.223 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:04.223 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.224 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.224 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:04.224 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020440+0000", 2024-09-21T15:50:04.224 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.694e-06 2024-09-21T15:50:04.224 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.224 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.224 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:04.224 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020449+0000", 2024-09-21T15:50:04.224 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5399999999999996e-06 2024-09-21T15:50:04.224 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.224 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.224 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:04.224 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020454+0000", 2024-09-21T15:50:04.224 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4199999999999998e-06 2024-09-21T15:50:04.225 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.225 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:04.225 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.225 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.225 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.225 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:123078 3.5 3.9ec96085 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-21T15:50:04.225 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:04.020458+0000", 2024-09-21T15:50:04.225 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.036537434000000001, 2024-09-21T15:50:04.225 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.036865809999999999, 2024-09-21T15:50:04.225 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:04.225 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-09-21T15:50:04.225 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:04.225 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:04.225 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:04.225 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 123078 2024-09-21T15:50:04.226 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.226 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:04.226 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.226 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:04.226 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020458+0000", 2024-09-21T15:50:04.226 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.226 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.226 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.226 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:04.226 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020458+0000", 2024-09-21T15:50:04.226 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.226 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.226 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.226 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:04.226 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020458+0000", 2024-09-21T15:50:04.227 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:04.227 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.227 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.227 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:04.227 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020460+0000", 2024-09-21T15:50:04.227 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.26e-06 2024-09-21T15:50:04.227 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.227 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.227 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:04.227 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020467+0000", 2024-09-21T15:50:04.227 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0759999999999997e-06 2024-09-21T15:50:04.227 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.227 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.227 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:04.228 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020472+0000", 2024-09-21T15:50:04.228 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7910000000000004e-06 2024-09-21T15:50:04.228 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.228 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:04.228 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.228 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.228 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.228 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:123080 3.5 3.9ec96085 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-21T15:50:04.228 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:04.021994+0000", 2024-09-21T15:50:04.228 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.035001745000000001, 2024-09-21T15:50:04.228 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.035357583999999997, 2024-09-21T15:50:04.228 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:04.228 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-09-21T15:50:04.228 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:04.228 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:04.229 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:04.229 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 123080 2024-09-21T15:50:04.229 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.229 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:04.229 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.229 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:04.229 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.021994+0000", 2024-09-21T15:50:04.229 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.229 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.229 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.229 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:04.229 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.021994+0000", 2024-09-21T15:50:04.229 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.229 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.229 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.230 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:04.230 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020483+0000", 2024-09-21T15:50:04.230 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9984889 2024-09-21T15:50:04.230 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.230 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.230 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:04.230 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.021999+0000", 2024-09-21T15:50:04.230 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001515558 2024-09-21T15:50:04.230 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.230 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.230 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:04.230 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.022017+0000", 2024-09-21T15:50:04.230 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8034e-05 2024-09-21T15:50:04.230 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.231 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.231 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:04.231 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.022025+0000", 2024-09-21T15:50:04.231 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7900000000000005e-06 2024-09-21T15:50:04.231 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.231 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:04.231 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.231 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.231 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.231 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:123081 3.5 3.9ec96085 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-21T15:50:04.231 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:04.022032+0000", 2024-09-21T15:50:04.231 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.034963230999999997, 2024-09-21T15:50:04.231 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.035346324999999998, 2024-09-21T15:50:04.231 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:04.232 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-09-21T15:50:04.232 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:04.232 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:04.232 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:04.232 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 123081 2024-09-21T15:50:04.232 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.232 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:04.232 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.232 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:04.232 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.022032+0000", 2024-09-21T15:50:04.232 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.232 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.232 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.232 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:04.233 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.022032+0000", 2024-09-21T15:50:04.233 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.233 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.233 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.233 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:04.233 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.022032+0000", 2024-09-21T15:50:04.233 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:04.233 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.233 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.233 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:04.233 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.022035+0000", 2024-09-21T15:50:04.233 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5610000000000001e-06 2024-09-21T15:50:04.233 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.233 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.233 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:04.234 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.022043+0000", 2024-09-21T15:50:04.234 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9670000000000001e-06 2024-09-21T15:50:04.234 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.234 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.234 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:04.234 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.022048+0000", 2024-09-21T15:50:04.234 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3750000000000002e-06 2024-09-21T15:50:04.234 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.234 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:04.234 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.234 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.234 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.234 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:123082 3.5 3.9ec96085 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-21T15:50:04.235 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:04.022052+0000", 2024-09-21T15:50:04.235 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.034943330000000002, 2024-09-21T15:50:04.235 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.035353170000000003, 2024-09-21T15:50:04.235 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:04.235 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-09-21T15:50:04.235 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:04.235 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:04.235 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:04.235 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 123082 2024-09-21T15:50:04.235 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.235 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:04.235 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.235 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:04.235 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.022052+0000", 2024-09-21T15:50:04.235 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.236 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.236 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.236 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:04.236 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.022052+0000", 2024-09-21T15:50:04.236 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.236 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.236 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.236 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:04.236 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.022052+0000", 2024-09-21T15:50:04.236 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:04.236 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.236 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.237 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:04.237 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.022054+0000", 2024-09-21T15:50:04.237 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1720000000000001e-06 2024-09-21T15:50:04.237 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.237 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.237 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:04.237 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.022063+0000", 2024-09-21T15:50:04.237 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1220000000000001e-06 2024-09-21T15:50:04.237 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.237 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.237 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:04.237 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.022068+0000", 2024-09-21T15:50:04.237 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6999999999999999e-06 2024-09-21T15:50:04.237 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.238 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:04.238 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.238 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.238 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.238 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:123083 3.5 3.9ec96085 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-21T15:50:04.238 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:04.022072+0000", 2024-09-21T15:50:04.238 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.034923754000000001, 2024-09-21T15:50:04.238 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.035371329999999999, 2024-09-21T15:50:04.238 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:04.238 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-09-21T15:50:04.238 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:04.238 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:04.238 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:04.238 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 123083 2024-09-21T15:50:04.239 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.239 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:04.239 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.239 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:04.239 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.022072+0000", 2024-09-21T15:50:04.239 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.239 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.239 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.239 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:04.239 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.022072+0000", 2024-09-21T15:50:04.239 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.239 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.239 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.239 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:04.240 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.022072+0000", 2024-09-21T15:50:04.240 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:04.240 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.240 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.240 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:04.240 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.022074+0000", 2024-09-21T15:50:04.240 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.153e-06 2024-09-21T15:50:04.240 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.240 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.240 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:04.240 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.022079+0000", 2024-09-21T15:50:04.240 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5999999999999997e-06 2024-09-21T15:50:04.240 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.240 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.241 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:04.241 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.022084+0000", 2024-09-21T15:50:04.241 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5000000000000001e-06 2024-09-21T15:50:04.241 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.241 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:04.241 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.241 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.241 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.241 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:123084 3.5 3.9ec96085 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-21T15:50:04.241 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:04.022088+0000", 2024-09-21T15:50:04.241 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.034907834999999998, 2024-09-21T15:50:04.241 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.03538906, 2024-09-21T15:50:04.241 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:04.241 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-09-21T15:50:04.242 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:04.242 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:04.242 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:04.242 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 123084 2024-09-21T15:50:04.242 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.242 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:04.242 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.242 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:04.242 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.022088+0000", 2024-09-21T15:50:04.242 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.242 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.242 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.242 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:04.242 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.022088+0000", 2024-09-21T15:50:04.242 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.243 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.243 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.243 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:04.243 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.022088+0000", 2024-09-21T15:50:04.243 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:04.243 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.243 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.243 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:04.243 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.022089+0000", 2024-09-21T15:50:04.243 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9400000000000001e-06 2024-09-21T15:50:04.243 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.243 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.243 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:04.243 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.022095+0000", 2024-09-21T15:50:04.244 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.75e-06 2024-09-21T15:50:04.244 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.244 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.244 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:04.244 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.022100+0000", 2024-09-21T15:50:04.244 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7489999999999998e-06 2024-09-21T15:50:04.244 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.244 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:04.244 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.244 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.244 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.244 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:123085 3.5 3.9ec96085 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-21T15:50:04.244 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:04.022130+0000", 2024-09-21T15:50:04.244 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.034865592000000001, 2024-09-21T15:50:04.245 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.035373807, 2024-09-21T15:50:04.245 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:04.245 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-09-21T15:50:04.245 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:04.245 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:04.245 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:04.245 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 123085 2024-09-21T15:50:04.245 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.245 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:04.245 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.245 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:04.245 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.022130+0000", 2024-09-21T15:50:04.245 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.245 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.246 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.246 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:04.246 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.022130+0000", 2024-09-21T15:50:04.246 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.246 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.246 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.246 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:04.246 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.022130+0000", 2024-09-21T15:50:04.246 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:04.246 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.246 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.246 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:04.246 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.022132+0000", 2024-09-21T15:50:04.247 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.599e-06 2024-09-21T15:50:04.247 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.247 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.247 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:04.247 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.022139+0000", 2024-09-21T15:50:04.247 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4960000000000001e-06 2024-09-21T15:50:04.247 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.247 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.247 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:04.247 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.022144+0000", 2024-09-21T15:50:04.247 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8810000000000004e-06 2024-09-21T15:50:04.247 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.247 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:04.247 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.247 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.248 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.248 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:123198 3.8 3:16adab62:::rbd_data.11333a557424.000000000000007e:head [call rbd.sparse_copyup in=71977b,set-alloc-hint object_size 524288 write_size 524288,write 401408~4096 in=4096b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e71)", 2024-09-21T15:50:04.248 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:04.053689+0000", 2024-09-21T15:50:04.248 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.0033067439999999999, 2024-09-21T15:50:04.248 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0038540219999999999, 2024-09-21T15:50:04.249 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:04.249 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "waiting for sub ops", 2024-09-21T15:50:04.249 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:04.249 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:04.250 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:04.250 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 123198 2024-09-21T15:50:04.250 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.250 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:04.250 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.250 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:04.250 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.053689+0000", 2024-09-21T15:50:04.250 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.250 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.250 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.250 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:04.250 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.053689+0000", 2024-09-21T15:50:04.250 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.250 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.251 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.251 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:04.251 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.053687+0000", 2024-09-21T15:50:04.251 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:50:04.251 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.251 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.251 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:04.251 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.053727+0000", 2024-09-21T15:50:04.251 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0201999999999999e-05 2024-09-21T15:50:04.251 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.251 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.251 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:04.251 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.053736+0000", 2024-09-21T15:50:04.251 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.363e-06 2024-09-21T15:50:04.252 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.252 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.252 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:04.252 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.053749+0000", 2024-09-21T15:50:04.252 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2089999999999999e-05 2024-09-21T15:50:04.252 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.252 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.252 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:04.252 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.053799+0000", 2024-09-21T15:50:04.252 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0148000000000002e-05 2024-09-21T15:50:04.252 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.252 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.252 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:04.252 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.053964+0000", 2024-09-21T15:50:04.252 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00016493500000000001 2024-09-21T15:50:04.253 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.253 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.254 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-09-21T15:50:04.254 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.054249+0000", 2024-09-21T15:50:04.254 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000284846 2024-09-21T15:50:04.255 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.255 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.255 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:04.255 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.055841+0000", 2024-09-21T15:50:04.255 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0015928450000000001 2024-09-21T15:50:04.255 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.255 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:04.255 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.255 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.255 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.255 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:123057 3.5 3:a1069379:::rbd_data.1133545d3fa6.000000000000003c:1e [read 477184~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-21T15:50:04.255 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:04.018017+0000", 2024-09-21T15:50:04.255 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.038978741999999997, 2024-09-21T15:50:04.256 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.039570398999999999, 2024-09-21T15:50:04.256 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:04.256 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:04.256 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:04.256 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:04.256 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:04.256 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 123057 2024-09-21T15:50:04.256 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.256 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:04.256 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.256 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:04.256 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018017+0000", 2024-09-21T15:50:04.256 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.256 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.256 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.256 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:04.257 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018017+0000", 2024-09-21T15:50:04.257 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.257 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.257 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.257 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:04.257 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018017+0000", 2024-09-21T15:50:04.257 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:04.257 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.257 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.257 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:04.257 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018019+0000", 2024-09-21T15:50:04.257 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3439999999999999e-06 2024-09-21T15:50:04.257 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.257 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.257 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:04.258 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018026+0000", 2024-09-21T15:50:04.258 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4279999999999999e-06 2024-09-21T15:50:04.258 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.258 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.258 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:04.258 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018031+0000", 2024-09-21T15:50:04.258 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8300000000000003e-06 2024-09-21T15:50:04.258 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.258 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.258 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:04.258 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.057456+0000", 2024-09-21T15:50:04.258 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.039424976 2024-09-21T15:50:04.258 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.258 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.259 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:04.259 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.057534+0000", 2024-09-21T15:50:04.259 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7857999999999999e-05 2024-09-21T15:50:04.259 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.259 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:04.259 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.259 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.259 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.259 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:123058 3.5 3.9ec96085 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-21T15:50:04.259 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:04.018036+0000", 2024-09-21T15:50:04.259 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.038960123999999999, 2024-09-21T15:50:04.259 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.039598950000000001, 2024-09-21T15:50:04.259 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:04.259 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-09-21T15:50:04.260 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:04.260 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:04.260 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:04.260 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 123058 2024-09-21T15:50:04.260 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.260 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:04.260 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.260 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:04.260 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018036+0000", 2024-09-21T15:50:04.260 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.260 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.260 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.260 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:04.260 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018036+0000", 2024-09-21T15:50:04.261 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.261 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.261 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.261 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:04.261 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018035+0000", 2024-09-21T15:50:04.261 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:04.261 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.261 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.261 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:04.261 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018037+0000", 2024-09-21T15:50:04.261 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1959999999999998e-06 2024-09-21T15:50:04.261 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.261 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.261 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:04.261 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018048+0000", 2024-09-21T15:50:04.261 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0504e-05 2024-09-21T15:50:04.262 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.262 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.262 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:04.262 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018055+0000", 2024-09-21T15:50:04.262 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.759e-06 2024-09-21T15:50:04.262 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.262 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.262 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:04.262 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.057625+0000", 2024-09-21T15:50:04.262 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.039570596999999999 2024-09-21T15:50:04.262 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.262 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:04.262 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.262 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.262 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.263 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:123059 3.5 3.9ec96085 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-21T15:50:04.263 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:04.018060+0000", 2024-09-21T15:50:04.263 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.038936050999999999, 2024-09-21T15:50:04.263 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.039610547000000003, 2024-09-21T15:50:04.263 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:04.263 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-09-21T15:50:04.263 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:04.263 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:04.263 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:04.263 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 123059 2024-09-21T15:50:04.263 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.263 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:04.263 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.263 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:04.263 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018060+0000", 2024-09-21T15:50:04.264 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.264 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.264 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.264 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:04.264 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018060+0000", 2024-09-21T15:50:04.264 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.264 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.264 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.264 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:04.264 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018059+0000", 2024-09-21T15:50:04.264 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:04.264 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.264 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.264 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:04.265 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018062+0000", 2024-09-21T15:50:04.265 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.661e-06 2024-09-21T15:50:04.265 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.265 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.265 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:04.265 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018073+0000", 2024-09-21T15:50:04.265 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0696000000000001e-05 2024-09-21T15:50:04.265 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.265 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.265 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:04.265 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018077+0000", 2024-09-21T15:50:04.265 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6759999999999998e-06 2024-09-21T15:50:04.265 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.265 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:04.266 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.266 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.266 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.266 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:123060 3.5 3.9ec96085 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-21T15:50:04.266 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:04.018095+0000", 2024-09-21T15:50:04.266 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.038900675000000003, 2024-09-21T15:50:04.266 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.039607069000000002, 2024-09-21T15:50:04.266 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:04.266 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-09-21T15:50:04.266 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:04.266 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:04.266 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:04.266 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 123060 2024-09-21T15:50:04.266 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.267 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:04.267 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.267 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:04.267 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018095+0000", 2024-09-21T15:50:04.267 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.267 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.267 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.267 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:04.267 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018095+0000", 2024-09-21T15:50:04.267 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.267 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.267 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.267 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:04.267 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018095+0000", 2024-09-21T15:50:04.267 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:04.268 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.268 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.268 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:04.268 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018097+0000", 2024-09-21T15:50:04.268 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.503e-06 2024-09-21T15:50:04.268 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.268 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.268 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:04.268 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018104+0000", 2024-09-21T15:50:04.268 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1749999999999999e-06 2024-09-21T15:50:04.268 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.268 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.268 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:04.268 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018109+0000", 2024-09-21T15:50:04.269 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1449999999999999e-06 2024-09-21T15:50:04.269 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.269 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:04.269 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.269 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.269 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.269 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:123061 3.5 3.9ec96085 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-21T15:50:04.269 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:04.018114+0000", 2024-09-21T15:50:04.269 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.038881582999999997, 2024-09-21T15:50:04.269 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.039615600000000001, 2024-09-21T15:50:04.269 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:04.269 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-09-21T15:50:04.269 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:04.269 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:04.270 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:04.270 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 123061 2024-09-21T15:50:04.270 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.270 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:04.270 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.270 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:04.270 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018114+0000", 2024-09-21T15:50:04.270 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.270 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.270 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.270 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:04.270 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018114+0000", 2024-09-21T15:50:04.270 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.270 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.271 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.271 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:04.271 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018114+0000", 2024-09-21T15:50:04.271 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:04.271 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.271 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.271 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:04.271 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018121+0000", 2024-09-21T15:50:04.271 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8580000000000002e-06 2024-09-21T15:50:04.271 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.271 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.271 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:04.271 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018127+0000", 2024-09-21T15:50:04.271 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6819999999999997e-06 2024-09-21T15:50:04.271 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.272 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.272 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:04.272 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018132+0000", 2024-09-21T15:50:04.272 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0660000000000003e-06 2024-09-21T15:50:04.272 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.272 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:04.272 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.272 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.272 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.272 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:123062 3.5 3.9ec96085 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-21T15:50:04.272 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:04.018137+0000", 2024-09-21T15:50:04.272 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.038858856999999997, 2024-09-21T15:50:04.272 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.039620052000000003, 2024-09-21T15:50:04.272 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:04.272 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-09-21T15:50:04.273 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:04.273 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:04.273 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:04.273 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 123062 2024-09-21T15:50:04.273 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.273 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:04.273 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.273 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:04.273 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018137+0000", 2024-09-21T15:50:04.273 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.273 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.273 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.273 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:04.273 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018137+0000", 2024-09-21T15:50:04.273 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.274 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.274 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.274 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:04.274 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018136+0000", 2024-09-21T15:50:04.274 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:04.274 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.274 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.274 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:04.274 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018139+0000", 2024-09-21T15:50:04.274 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2409999999999998e-06 2024-09-21T15:50:04.274 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.274 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.274 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:04.275 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018145+0000", 2024-09-21T15:50:04.275 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3069999999999999e-06 2024-09-21T15:50:04.275 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.275 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.275 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:04.275 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018150+0000", 2024-09-21T15:50:04.275 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7559999999999999e-06 2024-09-21T15:50:04.275 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.275 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:04.275 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.275 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.275 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.275 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:123063 3.5 3.9ec96085 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-21T15:50:04.275 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:04.018154+0000", 2024-09-21T15:50:04.275 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.038841605000000001, 2024-09-21T15:50:04.275 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.039646046999999997, 2024-09-21T15:50:04.276 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:04.276 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-09-21T15:50:04.276 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:04.276 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:04.276 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:04.276 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 123063 2024-09-21T15:50:04.276 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.276 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:04.276 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.276 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:04.276 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018154+0000", 2024-09-21T15:50:04.276 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.276 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.276 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.276 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:04.276 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018154+0000", 2024-09-21T15:50:04.276 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.277 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.277 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.277 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:04.277 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018154+0000", 2024-09-21T15:50:04.277 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:04.277 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.277 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.277 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:04.277 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018156+0000", 2024-09-21T15:50:04.277 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0899999999999999e-06 2024-09-21T15:50:04.277 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.277 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.277 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:04.277 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018166+0000", 2024-09-21T15:50:04.277 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7569999999999999e-06 2024-09-21T15:50:04.277 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.278 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.278 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:04.278 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018171+0000", 2024-09-21T15:50:04.278 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0579999999999997e-06 2024-09-21T15:50:04.278 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.278 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:04.278 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.278 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.278 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.278 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:123064 3.5 3.9ec96085 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-21T15:50:04.278 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:04.018175+0000", 2024-09-21T15:50:04.278 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.038820425999999998, 2024-09-21T15:50:04.278 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.039652623999999997, 2024-09-21T15:50:04.278 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:04.278 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-09-21T15:50:04.279 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:04.279 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:04.279 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:04.279 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 123064 2024-09-21T15:50:04.279 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.279 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:04.279 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.279 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:04.279 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018175+0000", 2024-09-21T15:50:04.279 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.279 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.279 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.279 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:04.279 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018175+0000", 2024-09-21T15:50:04.280 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.280 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.280 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.280 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:04.280 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018175+0000", 2024-09-21T15:50:04.280 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:04.280 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.280 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.280 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:04.280 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018177+0000", 2024-09-21T15:50:04.280 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9149999999999999e-06 2024-09-21T15:50:04.280 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.280 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.280 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:04.280 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018183+0000", 2024-09-21T15:50:04.281 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.879e-06 2024-09-21T15:50:04.281 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.281 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.281 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:04.281 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018187+0000", 2024-09-21T15:50:04.281 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.69e-06 2024-09-21T15:50:04.281 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.281 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:04.281 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.281 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.281 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.281 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:123065 3.5 3.9ec96085 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-21T15:50:04.281 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:04.018192+0000", 2024-09-21T15:50:04.281 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.038803811000000001, 2024-09-21T15:50:04.282 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.039667354000000002, 2024-09-21T15:50:04.282 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:04.282 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-09-21T15:50:04.282 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:04.282 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:04.282 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:04.282 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 123065 2024-09-21T15:50:04.282 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.282 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:04.282 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.282 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:04.282 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018192+0000", 2024-09-21T15:50:04.282 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.282 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.283 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.283 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:04.283 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018192+0000", 2024-09-21T15:50:04.283 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.283 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.283 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.283 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:04.283 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018192+0000", 2024-09-21T15:50:04.283 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:04.283 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.283 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.283 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:04.283 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018194+0000", 2024-09-21T15:50:04.284 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.159e-06 2024-09-21T15:50:04.284 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.284 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.284 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:04.284 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018200+0000", 2024-09-21T15:50:04.284 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1750000000000002e-06 2024-09-21T15:50:04.284 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.284 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.284 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:04.284 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018205+0000", 2024-09-21T15:50:04.284 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7269999999999998e-06 2024-09-21T15:50:04.284 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.284 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:04.284 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.284 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.285 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.285 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:123066 3.5 3.9ec96085 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-21T15:50:04.285 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:04.020146+0000", 2024-09-21T15:50:04.285 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.036849932000000002, 2024-09-21T15:50:04.285 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.037740783999999999, 2024-09-21T15:50:04.285 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:04.285 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-09-21T15:50:04.285 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:04.285 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:04.285 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:04.285 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 123066 2024-09-21T15:50:04.285 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.285 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:04.285 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.285 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:04.285 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020146+0000", 2024-09-21T15:50:04.286 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.286 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.286 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.286 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:04.286 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020146+0000", 2024-09-21T15:50:04.286 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.286 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.286 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.286 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:04.286 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.018209+0000", 2024-09-21T15:50:04.286 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9980636 2024-09-21T15:50:04.286 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.286 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.286 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:04.287 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020151+0000", 2024-09-21T15:50:04.287 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001941356 2024-09-21T15:50:04.287 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.287 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.287 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:04.287 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020163+0000", 2024-09-21T15:50:04.287 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2687999999999999e-05 2024-09-21T15:50:04.287 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.287 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.287 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:04.287 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020171+0000", 2024-09-21T15:50:04.287 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3710000000000002e-06 2024-09-21T15:50:04.287 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.287 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:04.287 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.288 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.288 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.288 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:123067 3.5 3.9ec96085 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-21T15:50:04.288 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:04.020178+0000", 2024-09-21T15:50:04.288 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.036818091999999997, 2024-09-21T15:50:04.288 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.037736005000000003, 2024-09-21T15:50:04.288 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:04.288 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-09-21T15:50:04.288 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:04.288 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:04.288 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:04.288 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 123067 2024-09-21T15:50:04.288 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.288 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:04.288 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.288 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:04.289 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020178+0000", 2024-09-21T15:50:04.289 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.289 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.289 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.289 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:04.289 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020178+0000", 2024-09-21T15:50:04.289 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.289 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.289 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.289 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:04.289 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020177+0000", 2024-09-21T15:50:04.289 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:04.289 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.289 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.289 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:04.290 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020180+0000", 2024-09-21T15:50:04.290 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9079999999999999e-06 2024-09-21T15:50:04.290 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.290 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.290 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:04.290 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020187+0000", 2024-09-21T15:50:04.290 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8290000000000002e-06 2024-09-21T15:50:04.290 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.290 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.290 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:04.290 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020192+0000", 2024-09-21T15:50:04.290 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7310000000000001e-06 2024-09-21T15:50:04.290 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.290 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:04.290 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.291 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.291 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.291 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:123068 3.5 3.9ec96085 (undecoded) ondisk+read+known_if_redirected+supports_pool_eio e71)", 2024-09-21T15:50:04.291 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:04.020196+0000", 2024-09-21T15:50:04.291 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.036799642, 2024-09-21T15:50:04.291 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.037757433, 2024-09-21T15:50:04.291 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:04.291 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "queued for pg", 2024-09-21T15:50:04.291 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:04.291 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:04.291 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:04.291 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 123068 2024-09-21T15:50:04.291 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.291 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:04.291 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.291 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:04.292 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020196+0000", 2024-09-21T15:50:04.292 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.292 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.292 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.292 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:04.292 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020196+0000", 2024-09-21T15:50:04.292 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.292 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.292 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.292 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:04.292 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020196+0000", 2024-09-21T15:50:04.292 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:04.292 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.292 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.292 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:04.293 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020198+0000", 2024-09-21T15:50:04.293 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3470000000000001e-06 2024-09-21T15:50:04.293 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.293 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.293 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:04.293 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020204+0000", 2024-09-21T15:50:04.293 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2670000000000003e-06 2024-09-21T15:50:04.293 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.293 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.293 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:04.293 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:04.020211+0000", 2024-09-21T15:50:04.293 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8839999999999997e-06 2024-09-21T15:50:04.293 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.293 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:04.293 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.293 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.294 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-21T15:50:04.294 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 29 2024-09-21T15:50:04.294 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:04.294 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:50:04.295 INFO:teuthology.orchestra.run.smithi089.stdout:4175 write 0x825b0e7 thru 0x8267593 (0xc4ad bytes) 2024-09-21T15:50:04.296 INFO:teuthology.orchestra.run.smithi089.stdout:4176 write 0xaa336c1 thru 0xaa3a31f (0x6c5f bytes) 2024-09-21T15:50:04.296 INFO:teuthology.orchestra.run.smithi089.stdout:4177 writesame 0x1652afb thru 0x1662a94 data_size 0x1(0xff9a bytes) 2024-09-21T15:50:04.296 INFO:teuthology.orchestra.run.smithi089.stdout:4178 read 0x2cd32f3 thru 0x2cdf60d (0xc31b bytes) 2024-09-21T15:50:04.296 INFO:teuthology.orchestra.run.smithi089.stdout:4179 writesame 0x6d39b58 thru 0x6d4400b data_size 0x7f(0xa4b4 bytes) 2024-09-21T15:50:04.296 INFO:teuthology.orchestra.run.smithi089.stdout:4180 write 0x63df70e thru 0x63ee89a (0xf18d bytes) 2024-09-21T15:50:04.296 INFO:teuthology.orchestra.run.smithi089.stdout:4181 punch from 0xa7c51ac to 0xa7ccae2, (0x7936 bytes) 2024-09-21T15:50:04.296 INFO:teuthology.orchestra.run.smithi089.stdout:4182 read 0x8f42393 thru 0x8f476f5 (0x5363 bytes) 2024-09-21T15:50:04.296 INFO:teuthology.orchestra.run.smithi089.stdout:4183 trunc from 0xe2f8430 to 0x221954a 2024-09-21T15:50:04.296 INFO:teuthology.orchestra.run.smithi089.stderr:Exporting 2.4 info 2.4( empty local-lis/les=17/18 n=0 ec=17/17 lis/c=17/17 les/c/f=18/18/0 sis=17) 2024-09-21T15:50:04.297 INFO:teuthology.orchestra.run.smithi089.stderr:Export successful 2024-09-21T15:50:04.297 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:04.253+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:04.297 INFO:teuthology.orchestra.run.smithi089.stdout:4184 read 0x13eaa8f thru 0x13f49f8 (0x9f6a bytes) 2024-09-21T15:50:04.297 INFO:teuthology.orchestra.run.smithi089.stdout:4185 punch from 0x12d9333 to 0x12dfa64, (0x6731 bytes) 2024-09-21T15:50:04.298 INFO:teuthology.orchestra.run.smithi089.stdout:4188 punch from 0x3405c2 to 0x3501f5, (0xfc33 bytes) 2024-09-21T15:50:04.298 INFO:teuthology.orchestra.run.smithi089.stdout:4189 read 0x1c6a7c2 thru 0x1c73cf9 (0x9538 bytes) 2024-09-21T15:50:04.298 INFO:teuthology.orchestra.run.smithi089.stdout:4190 write 0x9ecb327 thru 0x9ed0c1e (0x58f8 bytes) 2024-09-21T15:50:04.298 INFO:teuthology.orchestra.run.smithi089.stdout:4192 trunc from 0x9ed0c1f to 0xe43990d 2024-09-21T15:50:04.303 INFO:teuthology.orchestra.run.smithi089.stdout:4194 read 0x5141eb8 thru 0x5147bb9 (0x5d02 bytes) 2024-09-21T15:50:04.303 INFO:teuthology.orchestra.run.smithi089.stdout:4195 write 0x7e984b4 thru 0x7e9fe06 (0x7953 bytes) 2024-09-21T15:50:04.327 INFO:teuthology.orchestra.run.smithi089.stdout:4196 writesame 0xedd8013 thru 0xede34d3 data_size 0x1(0xb4c1 bytes) 2024-09-21T15:50:04.327 INFO:teuthology.orchestra.run.smithi089.stdout:4197 read 0x294314f thru 0x2949e0f (0x6cc1 bytes) 2024-09-21T15:50:04.327 INFO:teuthology.orchestra.run.smithi089.stdout:4198 read 0x14a978c thru 0x14b2dca (0x963f bytes) 2024-09-21T15:50:04.340 INFO:teuthology.orchestra.run.smithi089.stdout:4199 trunc from 0xede34d4 to 0x48d2372 2024-09-21T15:50:04.355 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:04.354+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:04.359 INFO:teuthology.orchestra.run.smithi089.stdout:4200 punch from 0x9fc563 to 0xa01173, (0x4c10 bytes) 2024-09-21T15:50:04.360 INFO:teuthology.orchestra.run.smithi089.stdout:4201 read 0x48701e8 thru 0x48781d1 (0x7fea bytes) 2024-09-21T15:50:04.360 INFO:teuthology.orchestra.run.smithi089.stdout:4203 read 0x1bd352a thru 0x1bd9a9d (0x6574 bytes) 2024-09-21T15:50:04.360 INFO:teuthology.orchestra.run.smithi089.stdout:4207 punch from 0x383251 to 0x3852e8, (0x2097 bytes) 2024-09-21T15:50:04.362 INFO:teuthology.orchestra.run.smithi089.stdout:4208 trunc from 0x48d2372 to 0x2b558fe 2024-09-21T15:50:04.367 INFO:teuthology.orchestra.run.smithi089.stdout:4209 read 0x1357cf3 thru 0x1363123 (0xb431 bytes) 2024-09-21T15:50:04.368 INFO:teuthology.orchestra.run.smithi089.stdout:4210 clone 28 order 24 su 8388608 sc 12 2024-09-21T15:50:04.406 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:04.406 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:04.406 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:50:04.406 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:50:04.406 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:04.414 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:50:04.456 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:04.454+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:04.557 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:04.556+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:04.559 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:04.559 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:50:04.559 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:50:04.559 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:50:04.559 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.559 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:14051 3.8 3:13437810:::rbd_data.1133dc5adfdf.0000000000000039:head [write 716800~796] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-21T15:50:04.559 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:50:04.559 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 46.862392559, 2024-09-21T15:50:04.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.98064002699999997, 2024-09-21T15:50:04.560 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:04.560 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:04.560 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:04.560 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:04.560 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:04.560 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 14051 2024-09-21T15:50:04.560 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.560 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:04.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.560 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:04.560 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:50:04.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.560 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.561 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.561 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:04.561 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:50:04.561 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.561 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.561 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:04.561 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:50:04.561 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:04.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.561 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.561 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:04.561 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695131+0000", 2024-09-21T15:50:04.561 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4680000000000003e-06 2024-09-21T15:50:04.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.562 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.562 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:04.562 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695139+0000", 2024-09-21T15:50:04.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9400000000000002e-06 2024-09-21T15:50:04.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.562 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.562 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:04.562 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695143+0000", 2024-09-21T15:50:04.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9899999999999999e-06 2024-09-21T15:50:04.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.562 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.562 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:04.562 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695190+0000", 2024-09-21T15:50:04.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6458000000000001e-05 2024-09-21T15:50:04.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.563 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:50:04.563 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695201+0000", 2024-09-21T15:50:04.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0964999999999999e-05 2024-09-21T15:50:04.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.563 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:04.563 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.669490+0000", 2024-09-21T15:50:04.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.97428980499999995 2024-09-21T15:50:04.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.563 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-09-21T15:50:04.563 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.670033+0000", 2024-09-21T15:50:04.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00054252599999999999 2024-09-21T15:50:04.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:04.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.671095+0000", 2024-09-21T15:50:04.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001062183 2024-09-21T15:50:04.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:04.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.671186+0000", 2024-09-21T15:50:04.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1191000000000002e-05 2024-09-21T15:50:04.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-09-21T15:50:04.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.671346+0000", 2024-09-21T15:50:04.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015980399999999999 2024-09-21T15:50:04.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.565 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:04.565 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.672221+0000", 2024-09-21T15:50:04.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00087471999999999995 2024-09-21T15:50:04.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.565 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:04.565 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.675619+0000", 2024-09-21T15:50:04.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00339858 2024-09-21T15:50:04.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.566 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:04.566 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.675670+0000", 2024-09-21T15:50:04.566 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0664000000000001e-05 2024-09-21T15:50:04.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.566 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:04.566 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.675768+0000", 2024-09-21T15:50:04.566 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7415000000000002e-05 2024-09-21T15:50:04.566 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.566 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:04.566 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.566 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116235 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 761856~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:04.566 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048806+0000", 2024-09-21T15:50:04.567 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.5087136749999992, 2024-09-21T15:50:04.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056818556999999999, 2024-09-21T15:50:04.567 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:04.567 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:04.567 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:04.567 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:04.567 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:04.567 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116235 2024-09-21T15:50:04.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.567 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:04.567 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.567 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:04.567 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048806+0000", 2024-09-21T15:50:04.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.568 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:04.568 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048806+0000", 2024-09-21T15:50:04.568 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.568 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:04.568 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048806+0000", 2024-09-21T15:50:04.568 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:04.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.568 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:04.568 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048807+0000", 2024-09-21T15:50:04.568 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.001e-06 2024-09-21T15:50:04.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.569 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:04.569 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048810+0000", 2024-09-21T15:50:04.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.711e-06 2024-09-21T15:50:04.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.569 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:04.569 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048812+0000", 2024-09-21T15:50:04.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2290000000000002e-06 2024-09-21T15:50:04.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.569 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:04.569 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105495+0000", 2024-09-21T15:50:04.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056682679999999999 2024-09-21T15:50:04.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:04.570 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105565+0000", 2024-09-21T15:50:04.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9868000000000003e-05 2024-09-21T15:50:04.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:04.570 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105625+0000", 2024-09-21T15:50:04.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0268e-05 2024-09-21T15:50:04.570 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.570 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:04.570 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.570 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116236 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 768000~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:04.571 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048814+0000", 2024-09-21T15:50:04.571 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.5087060119999993, 2024-09-21T15:50:04.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056959587999999998, 2024-09-21T15:50:04.571 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:04.571 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:04.571 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:04.571 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:04.571 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:04.571 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116236 2024-09-21T15:50:04.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.571 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:04.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:04.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048814+0000", 2024-09-21T15:50:04.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.572 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:04.572 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048814+0000", 2024-09-21T15:50:04.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.572 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:04.572 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048814+0000", 2024-09-21T15:50:04.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:04.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.572 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:04.572 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048815+0000", 2024-09-21T15:50:04.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7999999999999993e-07 2024-09-21T15:50:04.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:04.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048817+0000", 2024-09-21T15:50:04.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6620000000000001e-06 2024-09-21T15:50:04.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:04.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048820+0000", 2024-09-21T15:50:04.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2800000000000002e-06 2024-09-21T15:50:04.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:04.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105647+0000", 2024-09-21T15:50:04.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056827253000000001 2024-09-21T15:50:04.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.574 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:04.574 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105716+0000", 2024-09-21T15:50:04.574 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8761000000000001e-05 2024-09-21T15:50:04.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.574 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:04.574 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105774+0000", 2024-09-21T15:50:04.574 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7855000000000001e-05 2024-09-21T15:50:04.574 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.574 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:04.574 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.575 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116237 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 774144~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:04.575 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048822+0000", 2024-09-21T15:50:04.575 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.5086983269999994, 2024-09-21T15:50:04.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057114028999999997, 2024-09-21T15:50:04.575 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:04.575 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:04.575 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:04.575 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:04.575 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:04.575 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116237 2024-09-21T15:50:04.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.575 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:04.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:04.575 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048822+0000", 2024-09-21T15:50:04.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.576 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:04.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048822+0000", 2024-09-21T15:50:04.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.576 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:04.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048822+0000", 2024-09-21T15:50:04.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:04.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.576 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:04.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048823+0000", 2024-09-21T15:50:04.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9699999999999994e-07 2024-09-21T15:50:04.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:04.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048825+0000", 2024-09-21T15:50:04.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.65e-06 2024-09-21T15:50:04.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:04.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048828+0000", 2024-09-21T15:50:04.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8890000000000002e-06 2024-09-21T15:50:04.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:04.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105802+0000", 2024-09-21T15:50:04.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056974415 2024-09-21T15:50:04.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.578 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:04.578 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105871+0000", 2024-09-21T15:50:04.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8727999999999997e-05 2024-09-21T15:50:04.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.578 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:04.578 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105936+0000", 2024-09-21T15:50:04.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4554999999999994e-05 2024-09-21T15:50:04.578 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.578 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:04.578 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.579 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116238 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 780288~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:04.579 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048830+0000", 2024-09-21T15:50:04.579 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.5086900679999999, 2024-09-21T15:50:04.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057296574000000003, 2024-09-21T15:50:04.579 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:04.579 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:04.579 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:04.579 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:04.579 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:04.579 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116238 2024-09-21T15:50:04.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.579 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:04.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:04.580 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048830+0000", 2024-09-21T15:50:04.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:04.580 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048830+0000", 2024-09-21T15:50:04.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:04.580 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048830+0000", 2024-09-21T15:50:04.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:04.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:04.581 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048831+0000", 2024-09-21T15:50:04.581 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0160000000000001e-06 2024-09-21T15:50:04.581 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.581 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:04.581 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048834+0000", 2024-09-21T15:50:04.581 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.813e-06 2024-09-21T15:50:04.581 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.581 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:04.581 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048836+0000", 2024-09-21T15:50:04.581 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6790000000000001e-06 2024-09-21T15:50:04.581 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.581 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:04.582 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105961+0000", 2024-09-21T15:50:04.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057124705999999997 2024-09-21T15:50:04.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.582 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.582 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:04.582 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106042+0000", 2024-09-21T15:50:04.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0820000000000002e-05 2024-09-21T15:50:04.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.582 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.582 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:04.582 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106127+0000", 2024-09-21T15:50:04.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4746000000000005e-05 2024-09-21T15:50:04.582 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.582 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:04.583 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.583 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116324 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 524288~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:04.583 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048838+0000", 2024-09-21T15:50:04.583 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.5086817589999999, 2024-09-21T15:50:04.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057477185, 2024-09-21T15:50:04.583 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:04.583 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:04.583 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:04.583 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:04.583 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:04.583 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116324 2024-09-21T15:50:04.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.583 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:04.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:04.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048838+0000", 2024-09-21T15:50:04.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:04.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048838+0000", 2024-09-21T15:50:04.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:04.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048838+0000", 2024-09-21T15:50:04.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:04.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.585 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:04.585 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048842+0000", 2024-09-21T15:50:04.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4829999999999999e-06 2024-09-21T15:50:04.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.585 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:04.585 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048846+0000", 2024-09-21T15:50:04.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0470000000000004e-06 2024-09-21T15:50:04.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.585 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:04.585 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048849+0000", 2024-09-21T15:50:04.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3689999999999998e-06 2024-09-21T15:50:04.586 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.586 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:04.586 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106157+0000", 2024-09-21T15:50:04.586 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057307942000000001 2024-09-21T15:50:04.586 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.586 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:04.586 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106237+0000", 2024-09-21T15:50:04.586 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0320000000000003e-05 2024-09-21T15:50:04.586 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.586 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:04.586 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106315+0000", 2024-09-21T15:50:04.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8227999999999997e-05 2024-09-21T15:50:04.587 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.587 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:04.587 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.587 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.587 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116325 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 530432~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:04.587 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048851+0000", 2024-09-21T15:50:04.587 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.5086685870000007, 2024-09-21T15:50:04.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057581036000000002, 2024-09-21T15:50:04.587 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:04.587 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:04.587 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:04.587 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:04.588 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:04.588 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116325 2024-09-21T15:50:04.588 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.588 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:04.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.588 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:04.588 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048851+0000", 2024-09-21T15:50:04.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.588 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.588 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:04.588 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048851+0000", 2024-09-21T15:50:04.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.588 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.589 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:04.589 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048851+0000", 2024-09-21T15:50:04.589 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:04.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.589 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.589 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:04.589 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048852+0000", 2024-09-21T15:50:04.589 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.077e-06 2024-09-21T15:50:04.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.589 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.589 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:04.589 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048855+0000", 2024-09-21T15:50:04.589 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9859999999999999e-06 2024-09-21T15:50:04.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.589 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.589 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:04.590 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048858+0000", 2024-09-21T15:50:04.590 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3810000000000002e-06 2024-09-21T15:50:04.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.590 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:04.590 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106335+0000", 2024-09-21T15:50:04.590 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057477473000000001 2024-09-21T15:50:04.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.590 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:04.590 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106393+0000", 2024-09-21T15:50:04.590 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8050000000000002e-05 2024-09-21T15:50:04.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.590 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:04.591 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106432+0000", 2024-09-21T15:50:04.591 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9274999999999999e-05 2024-09-21T15:50:04.591 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.591 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:04.591 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.591 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.591 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116326 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 536576~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:04.591 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048860+0000", 2024-09-21T15:50:04.591 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.5086605360000007, 2024-09-21T15:50:04.591 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057774197999999999, 2024-09-21T15:50:04.591 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:04.591 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:04.591 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:04.591 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:04.591 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:04.592 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116326 2024-09-21T15:50:04.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.592 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:04.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.592 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:04.592 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048860+0000", 2024-09-21T15:50:04.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.592 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:04.592 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048860+0000", 2024-09-21T15:50:04.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.592 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:04.592 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048859+0000", 2024-09-21T15:50:04.593 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:04.593 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.593 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.593 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:04.593 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048860+0000", 2024-09-21T15:50:04.593 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.02e-06 2024-09-21T15:50:04.593 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.593 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.593 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:04.593 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048863+0000", 2024-09-21T15:50:04.593 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1039999999999998e-06 2024-09-21T15:50:04.593 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.593 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.593 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:04.593 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048867+0000", 2024-09-21T15:50:04.593 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.563e-06 2024-09-21T15:50:04.594 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.594 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.594 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:04.594 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106465+0000", 2024-09-21T15:50:04.594 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057597789000000003 2024-09-21T15:50:04.594 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.594 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.594 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:04.594 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106552+0000", 2024-09-21T15:50:04.594 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7602000000000006e-05 2024-09-21T15:50:04.594 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.594 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.594 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:04.594 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106634+0000", 2024-09-21T15:50:04.595 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1321000000000003e-05 2024-09-21T15:50:04.595 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.595 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:04.595 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.595 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.595 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116327 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 542720~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:04.595 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048870+0000", 2024-09-21T15:50:04.595 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.508649664, 2024-09-21T15:50:04.595 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057883148000000002, 2024-09-21T15:50:04.595 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:04.595 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:04.595 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:04.595 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:04.596 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:04.596 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116327 2024-09-21T15:50:04.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.596 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:04.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:04.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048870+0000", 2024-09-21T15:50:04.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:04.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048870+0000", 2024-09-21T15:50:04.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:04.597 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048870+0000", 2024-09-21T15:50:04.597 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:04.597 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.597 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.597 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:04.597 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048872+0000", 2024-09-21T15:50:04.597 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.824e-06 2024-09-21T15:50:04.597 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.597 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.597 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:04.597 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048879+0000", 2024-09-21T15:50:04.597 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6749999999999996e-06 2024-09-21T15:50:04.597 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.597 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.597 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:04.597 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048883+0000", 2024-09-21T15:50:04.597 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2200000000000003e-06 2024-09-21T15:50:04.598 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.598 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.598 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:04.598 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106663+0000", 2024-09-21T15:50:04.598 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057780049 2024-09-21T15:50:04.598 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.598 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.598 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:04.598 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106711+0000", 2024-09-21T15:50:04.598 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8433999999999997e-05 2024-09-21T15:50:04.598 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.598 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.598 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:04.598 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106754+0000", 2024-09-21T15:50:04.598 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2236e-05 2024-09-21T15:50:04.599 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.599 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:04.599 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.599 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.599 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.599 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116328 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 548864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:04.599 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048887+0000", 2024-09-21T15:50:04.599 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.5086331440000009, 2024-09-21T15:50:04.599 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057981984, 2024-09-21T15:50:04.599 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:04.599 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:04.599 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:04.599 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:04.599 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:04.599 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116328 2024-09-21T15:50:04.599 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.600 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:04.600 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.600 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:04.600 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048887+0000", 2024-09-21T15:50:04.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.600 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.600 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.600 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:04.600 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048887+0000", 2024-09-21T15:50:04.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.600 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.600 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.600 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:04.600 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048887+0000", 2024-09-21T15:50:04.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:04.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.601 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.601 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:04.601 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048891+0000", 2024-09-21T15:50:04.601 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.843e-06 2024-09-21T15:50:04.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.601 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.601 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:04.601 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048897+0000", 2024-09-21T15:50:04.601 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5419999999999996e-06 2024-09-21T15:50:04.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.601 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.601 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:04.601 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048901+0000", 2024-09-21T15:50:04.601 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4689999999999999e-06 2024-09-21T15:50:04.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.602 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.602 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:04.602 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106780+0000", 2024-09-21T15:50:04.602 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057878259000000001 2024-09-21T15:50:04.602 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.602 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.602 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:04.602 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106827+0000", 2024-09-21T15:50:04.602 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7098000000000002e-05 2024-09-21T15:50:04.602 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.602 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.602 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:04.602 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106869+0000", 2024-09-21T15:50:04.602 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2129999999999998e-05 2024-09-21T15:50:04.602 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.602 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:04.603 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.603 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116329 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 555008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:04.603 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048905+0000", 2024-09-21T15:50:04.603 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.508615034, 2024-09-21T15:50:04.603 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058115554999999999, 2024-09-21T15:50:04.603 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:04.603 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:04.603 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:04.603 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:04.603 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:04.603 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116329 2024-09-21T15:50:04.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.604 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:04.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.604 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:04.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048905+0000", 2024-09-21T15:50:04.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.604 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:04.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048905+0000", 2024-09-21T15:50:04.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.604 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:04.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048905+0000", 2024-09-21T15:50:04.605 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:04.605 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.605 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.605 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:04.605 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048907+0000", 2024-09-21T15:50:04.605 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.086e-06 2024-09-21T15:50:04.605 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.605 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.605 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:04.605 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048912+0000", 2024-09-21T15:50:04.605 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2830000000000001e-06 2024-09-21T15:50:04.605 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.605 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.605 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:04.605 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048916+0000", 2024-09-21T15:50:04.606 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.98e-06 2024-09-21T15:50:04.606 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.606 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.606 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:04.606 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106907+0000", 2024-09-21T15:50:04.606 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057990719000000003 2024-09-21T15:50:04.606 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.606 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.606 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:04.606 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106969+0000", 2024-09-21T15:50:04.606 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1789000000000001e-05 2024-09-21T15:50:04.606 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.606 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.606 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:04.606 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107021+0000", 2024-09-21T15:50:04.607 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2030999999999997e-05 2024-09-21T15:50:04.607 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.607 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:04.607 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.607 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.607 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.607 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116330 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 561152~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:04.607 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048922+0000", 2024-09-21T15:50:04.607 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.5085978759999996, 2024-09-21T15:50:04.607 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058242343000000002, 2024-09-21T15:50:04.607 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:04.607 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:04.607 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:04.607 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:04.607 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:04.608 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116330 2024-09-21T15:50:04.608 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.608 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:04.608 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.608 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:04.608 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048922+0000", 2024-09-21T15:50:04.608 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.608 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.608 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.608 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:04.609 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048922+0000", 2024-09-21T15:50:04.609 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.609 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.609 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.609 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:04.609 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048922+0000", 2024-09-21T15:50:04.609 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:04.609 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.609 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.610 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:04.610 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048924+0000", 2024-09-21T15:50:04.610 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1660000000000001e-06 2024-09-21T15:50:04.610 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.610 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.610 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:04.610 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048930+0000", 2024-09-21T15:50:04.610 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4529999999999999e-06 2024-09-21T15:50:04.610 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.610 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.610 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:04.610 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048934+0000", 2024-09-21T15:50:04.610 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0269999999999997e-06 2024-09-21T15:50:04.610 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.610 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.611 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:04.611 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107047+0000", 2024-09-21T15:50:04.611 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058112913000000002 2024-09-21T15:50:04.611 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.611 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.611 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:04.611 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107095+0000", 2024-09-21T15:50:04.611 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7778000000000002e-05 2024-09-21T15:50:04.611 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.611 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.611 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:04.611 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107165+0000", 2024-09-21T15:50:04.611 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9370999999999997e-05 2024-09-21T15:50:04.611 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.611 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:04.612 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.612 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.612 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.612 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116331 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 567296~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:04.612 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.049440+0000", 2024-09-21T15:50:04.612 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.5080797819999994, 2024-09-21T15:50:04.612 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057878485, 2024-09-21T15:50:04.612 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:04.612 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:04.612 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:04.612 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:04.612 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:04.612 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116331 2024-09-21T15:50:04.612 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.612 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:04.613 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.613 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:04.613 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049440+0000", 2024-09-21T15:50:04.613 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.613 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.613 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.613 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:04.613 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049440+0000", 2024-09-21T15:50:04.613 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.613 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.613 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.613 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:04.613 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048939+0000", 2024-09-21T15:50:04.613 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9994984 2024-09-21T15:50:04.614 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.614 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.614 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:04.614 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049444+0000", 2024-09-21T15:50:04.614 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00050557200000000003 2024-09-21T15:50:04.614 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.614 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.614 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:04.614 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049460+0000", 2024-09-21T15:50:04.614 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6120000000000002e-05 2024-09-21T15:50:04.614 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.614 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.614 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:04.614 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049468+0000", 2024-09-21T15:50:04.614 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4209999999999998e-06 2024-09-21T15:50:04.615 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.615 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.615 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:04.615 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107189+0000", 2024-09-21T15:50:04.615 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057721019999999998 2024-09-21T15:50:04.615 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.615 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.615 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:04.615 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107254+0000", 2024-09-21T15:50:04.615 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5265000000000006e-05 2024-09-21T15:50:04.615 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.615 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.615 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:04.615 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107319+0000", 2024-09-21T15:50:04.615 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4784999999999997e-05 2024-09-21T15:50:04.615 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.616 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:04.616 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.616 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.616 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.616 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116332 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 573440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:04.616 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.049474+0000", 2024-09-21T15:50:04.616 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.5080465210000007, 2024-09-21T15:50:04.616 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058013809, 2024-09-21T15:50:04.616 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:04.616 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:04.616 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:04.616 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:04.616 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:04.616 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116332 2024-09-21T15:50:04.616 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.617 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:04.617 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.617 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:04.617 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049474+0000", 2024-09-21T15:50:04.617 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.617 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.617 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.617 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:04.617 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049474+0000", 2024-09-21T15:50:04.617 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.617 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.617 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.617 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:04.617 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049473+0000", 2024-09-21T15:50:04.618 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:04.618 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.618 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.618 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:04.618 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049476+0000", 2024-09-21T15:50:04.618 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6249999999999999e-06 2024-09-21T15:50:04.618 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.618 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.618 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:04.618 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049482+0000", 2024-09-21T15:50:04.618 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0959999999999997e-06 2024-09-21T15:50:04.618 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.618 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.618 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:04.619 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049487+0000", 2024-09-21T15:50:04.619 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5949999999999998e-06 2024-09-21T15:50:04.619 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.619 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.619 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:04.619 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107344+0000", 2024-09-21T15:50:04.619 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057856594999999997 2024-09-21T15:50:04.619 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.619 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.619 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:04.619 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107422+0000", 2024-09-21T15:50:04.619 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7582000000000004e-05 2024-09-21T15:50:04.619 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.619 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.620 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:04.620 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107487+0000", 2024-09-21T15:50:04.620 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5723999999999996e-05 2024-09-21T15:50:04.620 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.620 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:04.620 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.620 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.620 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.620 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116333 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 579584~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:04.620 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.049993+0000", 2024-09-21T15:50:04.620 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.5075267379999993, 2024-09-21T15:50:04.620 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057663537000000001, 2024-09-21T15:50:04.620 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:04.620 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:04.620 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:04.620 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:04.621 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:04.621 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116333 2024-09-21T15:50:04.621 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.621 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:04.621 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.621 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:04.621 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049993+0000", 2024-09-21T15:50:04.621 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.621 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.621 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.621 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:04.621 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049993+0000", 2024-09-21T15:50:04.621 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.621 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.621 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.621 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:04.622 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049491+0000", 2024-09-21T15:50:04.622 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9994979 2024-09-21T15:50:04.622 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.622 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.622 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:04.622 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049998+0000", 2024-09-21T15:50:04.622 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00050654200000000002 2024-09-21T15:50:04.622 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.622 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.622 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:04.622 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050010+0000", 2024-09-21T15:50:04.622 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1790999999999999e-05 2024-09-21T15:50:04.622 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.622 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.622 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:04.622 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050016+0000", 2024-09-21T15:50:04.623 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2179999999999996e-06 2024-09-21T15:50:04.623 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.623 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.623 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:04.623 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107513+0000", 2024-09-21T15:50:04.623 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057496927000000003 2024-09-21T15:50:04.623 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.623 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.623 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:04.623 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107590+0000", 2024-09-21T15:50:04.623 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7547999999999997e-05 2024-09-21T15:50:04.623 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.623 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.623 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:04.623 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107657+0000", 2024-09-21T15:50:04.624 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6630000000000004e-05 2024-09-21T15:50:04.625 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.625 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:04.625 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.625 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.625 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.625 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116334 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 585728~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:04.625 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.050091+0000", 2024-09-21T15:50:04.625 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.5074291100000003, 2024-09-21T15:50:04.626 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057734355000000001, 2024-09-21T15:50:04.626 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:04.626 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:04.626 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:04.626 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:04.626 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:04.626 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116334 2024-09-21T15:50:04.626 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.626 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:04.626 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.626 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:04.626 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050091+0000", 2024-09-21T15:50:04.626 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.626 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.627 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.627 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:04.627 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050091+0000", 2024-09-21T15:50:04.627 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.627 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.627 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.627 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:04.627 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050021+0000", 2024-09-21T15:50:04.627 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999304 2024-09-21T15:50:04.627 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.627 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.627 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:04.627 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050095+0000", 2024-09-21T15:50:04.627 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4158000000000004e-05 2024-09-21T15:50:04.627 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.628 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.628 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:04.628 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050103+0000", 2024-09-21T15:50:04.628 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0770000000000002e-06 2024-09-21T15:50:04.628 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.628 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.628 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:04.628 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050108+0000", 2024-09-21T15:50:04.628 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6330000000000001e-06 2024-09-21T15:50:04.628 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.628 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.628 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:04.628 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107681+0000", 2024-09-21T15:50:04.628 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057572996000000001 2024-09-21T15:50:04.628 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.629 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.629 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:04.629 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107759+0000", 2024-09-21T15:50:04.629 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7368999999999997e-05 2024-09-21T15:50:04.629 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.629 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.629 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:04.629 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107825+0000", 2024-09-21T15:50:04.629 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6783000000000001e-05 2024-09-21T15:50:04.629 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.629 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:04.629 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.629 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.629 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.629 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116335 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 591872~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:04.630 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.050542+0000", 2024-09-21T15:50:04.630 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.5069785249999992, 2024-09-21T15:50:04.630 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057466455999999999, 2024-09-21T15:50:04.630 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:04.630 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:04.630 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:04.630 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:04.630 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:04.630 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116335 2024-09-21T15:50:04.630 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.630 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:04.630 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.630 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:04.630 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050542+0000", 2024-09-21T15:50:04.631 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.631 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.631 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.631 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:04.631 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050542+0000", 2024-09-21T15:50:04.631 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.631 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.631 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.631 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:04.631 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050113+0000", 2024-09-21T15:50:04.631 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9995713 2024-09-21T15:50:04.631 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.631 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.631 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:04.631 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050546+0000", 2024-09-21T15:50:04.631 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00043328699999999997 2024-09-21T15:50:04.632 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.632 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.632 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:04.632 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050557+0000", 2024-09-21T15:50:04.632 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1011e-05 2024-09-21T15:50:04.632 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.632 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.632 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:04.632 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050564+0000", 2024-09-21T15:50:04.632 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1130000000000003e-06 2024-09-21T15:50:04.632 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.632 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.632 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:04.632 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107852+0000", 2024-09-21T15:50:04.632 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057287350000000001 2024-09-21T15:50:04.633 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.633 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.633 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:04.633 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107928+0000", 2024-09-21T15:50:04.633 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6668000000000005e-05 2024-09-21T15:50:04.633 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.633 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.633 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:04.633 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108008+0000", 2024-09-21T15:50:04.633 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9783999999999994e-05 2024-09-21T15:50:04.633 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.633 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:04.633 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.633 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.634 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.634 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116336 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 598016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:04.634 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.050834+0000", 2024-09-21T15:50:04.634 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.5066862590000003, 2024-09-21T15:50:04.634 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057353572999999998, 2024-09-21T15:50:04.634 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:04.634 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:04.634 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:04.634 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:04.634 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:04.634 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116336 2024-09-21T15:50:04.634 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.634 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:04.634 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.635 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:04.635 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050834+0000", 2024-09-21T15:50:04.635 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.635 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.635 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.635 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:04.635 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050834+0000", 2024-09-21T15:50:04.635 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.635 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.635 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.635 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:04.635 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050571+0000", 2024-09-21T15:50:04.635 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9997373 2024-09-21T15:50:04.635 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.635 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.636 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:04.636 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050842+0000", 2024-09-21T15:50:04.636 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00027029099999999998 2024-09-21T15:50:04.636 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.636 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.636 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:04.636 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050853+0000", 2024-09-21T15:50:04.636 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1532000000000001e-05 2024-09-21T15:50:04.636 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.636 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.636 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:04.636 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050863+0000", 2024-09-21T15:50:04.636 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4239999999999999e-06 2024-09-21T15:50:04.636 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.637 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.637 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:04.637 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108045+0000", 2024-09-21T15:50:04.637 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057182415 2024-09-21T15:50:04.637 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.637 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.637 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:04.637 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108123+0000", 2024-09-21T15:50:04.637 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8275000000000005e-05 2024-09-21T15:50:04.637 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.637 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.637 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:04.638 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108187+0000", 2024-09-21T15:50:04.638 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4152000000000005e-05 2024-09-21T15:50:04.638 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.638 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:04.638 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.638 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.638 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.638 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116337 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 604160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:04.638 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.051192+0000", 2024-09-21T15:50:04.638 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.5063276309999996, 2024-09-21T15:50:04.638 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057175981000000001, 2024-09-21T15:50:04.638 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:04.638 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:04.638 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:04.639 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:04.639 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:04.639 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116337 2024-09-21T15:50:04.639 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.639 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:04.639 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.639 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:04.639 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051192+0000", 2024-09-21T15:50:04.639 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.639 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.639 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.639 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:04.639 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051192+0000", 2024-09-21T15:50:04.639 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.640 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.640 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.640 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:04.640 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050870+0000", 2024-09-21T15:50:04.640 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9996777 2024-09-21T15:50:04.640 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.640 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.640 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:04.640 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051197+0000", 2024-09-21T15:50:04.640 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00032656299999999998 2024-09-21T15:50:04.640 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.640 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.640 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:04.640 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051209+0000", 2024-09-21T15:50:04.640 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2636000000000001e-05 2024-09-21T15:50:04.641 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.641 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.641 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:04.641 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051217+0000", 2024-09-21T15:50:04.641 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5989999999999996e-06 2024-09-21T15:50:04.641 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.641 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.641 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:04.641 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108224+0000", 2024-09-21T15:50:04.641 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057007622000000001 2024-09-21T15:50:04.641 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.641 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.641 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:04.641 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108299+0000", 2024-09-21T15:50:04.641 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5099999999999996e-05 2024-09-21T15:50:04.641 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.642 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.642 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:04.642 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108368+0000", 2024-09-21T15:50:04.642 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8918000000000007e-05 2024-09-21T15:50:04.642 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.642 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:04.642 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.642 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.642 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.642 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116338 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 610304~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:04.642 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.051583+0000", 2024-09-21T15:50:04.642 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.5059371779999999, 2024-09-21T15:50:04.642 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056967716000000002, 2024-09-21T15:50:04.642 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:04.643 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:04.643 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:04.643 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:04.643 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:04.643 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116338 2024-09-21T15:50:04.643 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.643 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:04.643 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.643 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:04.643 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051583+0000", 2024-09-21T15:50:04.643 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.643 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.643 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.643 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:04.643 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051583+0000", 2024-09-21T15:50:04.644 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:04.644 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.644 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.644 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:04.644 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051223+0000", 2024-09-21T15:50:04.644 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.99964 2024-09-21T15:50:04.644 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.644 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.644 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:04.644 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051587+0000", 2024-09-21T15:50:04.644 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00036352999999999998 2024-09-21T15:50:04.644 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.644 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.644 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:04.645 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051597+0000", 2024-09-21T15:50:04.645 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.893e-06 2024-09-21T15:50:04.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.645 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.645 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:04.645 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051603+0000", 2024-09-21T15:50:04.645 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0519999999999998e-06 2024-09-21T15:50:04.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.645 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.645 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:04.645 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108394+0000", 2024-09-21T15:50:04.645 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056791531999999999 2024-09-21T15:50:04.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.645 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.646 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:04.646 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108480+0000", 2024-09-21T15:50:04.646 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.577e-05 2024-09-21T15:50:04.646 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:04.646 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:04.646 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:04.646 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108551+0000", 2024-09-21T15:50:04.646 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0722000000000005e-05 2024-09-21T15:50:04.646 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.646 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:04.646 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.646 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:04.646 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:04.646 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:04.648 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:50:04.689 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:04.688+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:04.724 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --force --op remove --pgid 2.4 2024-09-21T15:50:04.754 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:04.754 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:04.754 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:50:04.754 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:04.762 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:50:04.765 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:04.764+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:04.859 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:04.858+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:04.863 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:04.863 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:04.863 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:50:04.863 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:50:04.863 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:04.871 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:50:04.960 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:04.959+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:05.020 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:05.020 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:50:05.020 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:50:05.020 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:50:05.020 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.020 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120589 3.1d 3:bf04a118:::rbd_data.1133545d3fa6.0000000000000017:1e [read 653312~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:05.020 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.618510+0000", 2024-09-21T15:50:05.020 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 1.3994900809999999, 2024-09-21T15:50:05.020 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.035506942999999999, 2024-09-21T15:50:05.021 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:05.021 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:05.021 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:05.021 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:05.021 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:05.021 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120589 2024-09-21T15:50:05.021 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.021 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:05.021 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.021 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:05.021 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618510+0000", 2024-09-21T15:50:05.022 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:05.022 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.022 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.022 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:05.022 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618510+0000", 2024-09-21T15:50:05.022 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:05.022 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.022 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.022 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:05.022 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618509+0000", 2024-09-21T15:50:05.023 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:50:05.023 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.023 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.023 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:05.023 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618511+0000", 2024-09-21T15:50:05.023 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6309999999999999e-06 2024-09-21T15:50:05.023 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.023 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.023 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:05.023 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618515+0000", 2024-09-21T15:50:05.023 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9450000000000003e-06 2024-09-21T15:50:05.023 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.023 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.024 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:05.024 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618518+0000", 2024-09-21T15:50:05.024 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7140000000000002e-06 2024-09-21T15:50:05.024 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.024 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.024 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:05.024 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.653856+0000", 2024-09-21T15:50:05.024 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.035338304000000001 2024-09-21T15:50:05.024 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.024 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.024 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:05.024 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.653934+0000", 2024-09-21T15:50:05.025 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7900999999999998e-05 2024-09-21T15:50:05.025 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.025 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.025 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:05.025 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.654016+0000", 2024-09-21T15:50:05.025 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2684999999999999e-05 2024-09-21T15:50:05.025 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:05.025 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:05.025 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:05.025 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.025 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.025 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120590 3.1d 3:bf04a118:::rbd_data.1133545d3fa6.0000000000000017:1e [read 669696~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:05.026 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.618520+0000", 2024-09-21T15:50:05.026 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 1.3994798259999999, 2024-09-21T15:50:05.026 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.035676202999999997, 2024-09-21T15:50:05.026 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:05.026 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:05.026 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:05.026 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:05.026 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:05.026 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120590 2024-09-21T15:50:05.026 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.026 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:05.027 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.027 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:05.027 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618520+0000", 2024-09-21T15:50:05.027 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:05.027 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.027 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.027 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:05.027 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618520+0000", 2024-09-21T15:50:05.027 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:05.027 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.027 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.027 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:05.027 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618520+0000", 2024-09-21T15:50:05.028 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:50:05.028 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.028 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.028 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:05.028 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618521+0000", 2024-09-21T15:50:05.028 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1540000000000001e-06 2024-09-21T15:50:05.028 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.028 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.028 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:05.028 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618524+0000", 2024-09-21T15:50:05.028 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9840000000000001e-06 2024-09-21T15:50:05.029 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.029 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.029 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:05.029 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618526+0000", 2024-09-21T15:50:05.029 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4889999999999998e-06 2024-09-21T15:50:05.029 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.029 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.029 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:05.029 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.654057+0000", 2024-09-21T15:50:05.029 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.035530335000000003 2024-09-21T15:50:05.029 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.029 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.029 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:05.030 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.654130+0000", 2024-09-21T15:50:05.030 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3757999999999994e-05 2024-09-21T15:50:05.030 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.030 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.030 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:05.030 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.654196+0000", 2024-09-21T15:50:05.030 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5686999999999996e-05 2024-09-21T15:50:05.030 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:05.030 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:05.030 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:05.030 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.030 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.031 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120591 3.1d 3:bf04a118:::rbd_data.1133545d3fa6.0000000000000017:1e [read 690176~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:05.031 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.618529+0000", 2024-09-21T15:50:05.031 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 1.399470561, 2024-09-21T15:50:05.031 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.035861056000000002, 2024-09-21T15:50:05.031 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:05.031 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:05.031 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:05.031 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:05.031 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:05.031 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120591 2024-09-21T15:50:05.031 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.032 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:05.032 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.032 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:05.032 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618529+0000", 2024-09-21T15:50:05.032 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:05.032 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.032 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.032 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:05.032 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618529+0000", 2024-09-21T15:50:05.032 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:05.032 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.032 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.032 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:05.033 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618529+0000", 2024-09-21T15:50:05.033 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:05.033 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.033 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.033 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:05.033 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618530+0000", 2024-09-21T15:50:05.033 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3540000000000001e-06 2024-09-21T15:50:05.033 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.033 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.033 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:05.033 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618533+0000", 2024-09-21T15:50:05.033 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8339999999999999e-06 2024-09-21T15:50:05.034 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.034 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.034 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:05.034 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618535+0000", 2024-09-21T15:50:05.034 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.396e-06 2024-09-21T15:50:05.034 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.034 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.034 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:05.034 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.654238+0000", 2024-09-21T15:50:05.034 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.035702456 2024-09-21T15:50:05.034 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.035 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.035 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:05.035 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.654315+0000", 2024-09-21T15:50:05.035 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7754000000000001e-05 2024-09-21T15:50:05.035 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.035 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.035 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:05.035 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.654390+0000", 2024-09-21T15:50:05.035 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4702000000000004e-05 2024-09-21T15:50:05.035 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:05.035 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:05.036 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:05.036 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.036 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.036 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120592 3.1d 3:bf04a118:::rbd_data.1133545d3fa6.0000000000000017:1e [read 710656~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:05.036 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.618537+0000", 2024-09-21T15:50:05.036 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 1.399462217, 2024-09-21T15:50:05.036 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.036018739000000001, 2024-09-21T15:50:05.036 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:05.036 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:05.036 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:05.036 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:05.036 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:05.037 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120592 2024-09-21T15:50:05.037 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.037 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:05.037 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.037 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:05.037 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618537+0000", 2024-09-21T15:50:05.037 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:05.037 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.037 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.037 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:05.037 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618537+0000", 2024-09-21T15:50:05.038 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:05.038 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.038 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.038 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:05.038 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618537+0000", 2024-09-21T15:50:05.038 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:50:05.038 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.038 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.038 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:05.038 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618538+0000", 2024-09-21T15:50:05.038 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1599999999999999e-06 2024-09-21T15:50:05.039 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.039 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.039 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:05.039 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618541+0000", 2024-09-21T15:50:05.039 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8590000000000001e-06 2024-09-21T15:50:05.039 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.039 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.039 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:05.039 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618543+0000", 2024-09-21T15:50:05.039 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2199999999999999e-06 2024-09-21T15:50:05.040 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.040 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.040 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:05.040 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.654423+0000", 2024-09-21T15:50:05.040 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.035879200999999999 2024-09-21T15:50:05.040 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.040 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.040 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:05.040 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.654490+0000", 2024-09-21T15:50:05.040 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7142000000000002e-05 2024-09-21T15:50:05.040 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.040 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.041 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:05.041 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.654556+0000", 2024-09-21T15:50:05.041 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6378999999999996e-05 2024-09-21T15:50:05.041 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:05.041 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:05.041 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:05.041 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.041 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.041 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120593 3.1d 3:bf04a118:::rbd_data.1133545d3fa6.0000000000000017:1e [read 731136~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:05.041 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.618552+0000", 2024-09-21T15:50:05.041 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 1.3994477320000001, 2024-09-21T15:50:05.041 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.036193327999999997, 2024-09-21T15:50:05.042 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:05.042 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:05.042 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:05.042 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:05.042 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:05.042 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120593 2024-09-21T15:50:05.042 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.042 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:05.042 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.042 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:05.042 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618552+0000", 2024-09-21T15:50:05.042 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:05.043 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.043 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.043 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:05.043 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618552+0000", 2024-09-21T15:50:05.043 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:05.043 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.043 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.043 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:05.043 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618552+0000", 2024-09-21T15:50:05.043 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:05.043 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.043 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.044 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:05.044 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618553+0000", 2024-09-21T15:50:05.044 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5260000000000001e-06 2024-09-21T15:50:05.044 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.044 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.044 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:05.044 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618558+0000", 2024-09-21T15:50:05.044 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4000000000000002e-06 2024-09-21T15:50:05.044 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.044 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.044 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:05.045 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618564+0000", 2024-09-21T15:50:05.045 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9699999999999996e-06 2024-09-21T15:50:05.045 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.045 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.045 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:05.045 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.654591+0000", 2024-09-21T15:50:05.045 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.036027058000000001 2024-09-21T15:50:05.045 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.045 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.045 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:05.045 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.654674+0000", 2024-09-21T15:50:05.046 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3901999999999997e-05 2024-09-21T15:50:05.046 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.046 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.046 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:05.046 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.654745+0000", 2024-09-21T15:50:05.046 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0718999999999998e-05 2024-09-21T15:50:05.046 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:05.046 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:05.046 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:05.046 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.046 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.046 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120594 3.1d 3:bf04a118:::rbd_data.1133545d3fa6.0000000000000017:1e [read 751616~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:05.047 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.618566+0000", 2024-09-21T15:50:05.047 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 1.399433927, 2024-09-21T15:50:05.047 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.036328510000000001, 2024-09-21T15:50:05.047 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:05.047 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:05.047 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:05.047 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:05.047 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:05.047 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120594 2024-09-21T15:50:05.047 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.047 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:05.048 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.048 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:05.048 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618566+0000", 2024-09-21T15:50:05.048 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:05.048 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.048 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.048 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:05.048 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618566+0000", 2024-09-21T15:50:05.048 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:05.048 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.048 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.048 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:05.048 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618565+0000", 2024-09-21T15:50:05.049 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:50:05.049 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.049 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.049 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:05.049 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618567+0000", 2024-09-21T15:50:05.049 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.248e-06 2024-09-21T15:50:05.049 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.049 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.049 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:05.049 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618570+0000", 2024-09-21T15:50:05.049 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3560000000000001e-06 2024-09-21T15:50:05.050 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.050 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.050 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:05.050 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618573+0000", 2024-09-21T15:50:05.050 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5150000000000001e-06 2024-09-21T15:50:05.050 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.050 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.050 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:05.050 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.654786+0000", 2024-09-21T15:50:05.050 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.036213093000000002 2024-09-21T15:50:05.050 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.050 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.051 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:05.051 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.654843+0000", 2024-09-21T15:50:05.051 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7825999999999998e-05 2024-09-21T15:50:05.051 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.051 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.051 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:05.051 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.654894+0000", 2024-09-21T15:50:05.051 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0692999999999997e-05 2024-09-21T15:50:05.051 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:05.051 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:05.051 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:05.051 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.052 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.052 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120595 3.1d 3:bf04a118:::rbd_data.1133545d3fa6.0000000000000017:1e [read 772096~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:05.052 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.618592+0000", 2024-09-21T15:50:05.052 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 1.399407439, 2024-09-21T15:50:05.052 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.036430806000000003, 2024-09-21T15:50:05.052 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:05.052 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:05.052 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:05.052 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:05.052 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:05.052 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120595 2024-09-21T15:50:05.053 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.053 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:05.053 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.053 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:05.053 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618592+0000", 2024-09-21T15:50:05.053 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:05.053 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.053 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.053 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:05.053 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618592+0000", 2024-09-21T15:50:05.053 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:05.054 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.054 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.054 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:05.054 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618592+0000", 2024-09-21T15:50:05.054 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:05.054 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.054 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.054 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:05.054 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618593+0000", 2024-09-21T15:50:05.054 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.6750000000000001e-06 2024-09-21T15:50:05.054 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.054 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.055 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:05.055 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618597+0000", 2024-09-21T15:50:05.055 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1690000000000001e-06 2024-09-21T15:50:05.055 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.055 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.055 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:05.055 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618599+0000", 2024-09-21T15:50:05.055 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4439999999999998e-06 2024-09-21T15:50:05.055 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.055 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.055 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:05.056 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.654921+0000", 2024-09-21T15:50:05.056 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.036322037000000001 2024-09-21T15:50:05.056 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.056 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.056 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:05.056 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.654976+0000", 2024-09-21T15:50:05.056 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4897e-05 2024-09-21T15:50:05.056 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.056 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.056 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:05.057 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.655023+0000", 2024-09-21T15:50:05.057 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6941999999999999e-05 2024-09-21T15:50:05.057 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:05.057 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:05.057 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:05.057 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.057 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.057 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120596 3.1d 3:bf04a118:::rbd_data.1133545d3fa6.0000000000000017:1e [read 788480~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:05.057 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.618609+0000", 2024-09-21T15:50:05.057 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 1.399390095, 2024-09-21T15:50:05.057 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.036510599999999997, 2024-09-21T15:50:05.057 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:05.057 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:05.058 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:05.058 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:05.058 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:05.058 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120596 2024-09-21T15:50:05.058 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.058 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:05.058 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.058 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:05.058 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618609+0000", 2024-09-21T15:50:05.058 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:05.058 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.058 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.059 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:05.059 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618609+0000", 2024-09-21T15:50:05.059 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:05.059 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.059 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.059 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:05.059 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618609+0000", 2024-09-21T15:50:05.059 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:05.059 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.059 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.059 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:05.059 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618611+0000", 2024-09-21T15:50:05.059 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7400000000000001e-06 2024-09-21T15:50:05.060 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.060 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.060 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:05.060 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618615+0000", 2024-09-21T15:50:05.060 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9859999999999996e-06 2024-09-21T15:50:05.060 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.060 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.060 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:05.060 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618617+0000", 2024-09-21T15:50:05.060 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4490000000000002e-06 2024-09-21T15:50:05.060 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.060 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.061 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:05.061 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.655046+0000", 2024-09-21T15:50:05.061 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.036428501000000002 2024-09-21T15:50:05.061 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.061 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.061 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:05.061 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.655083+0000", 2024-09-21T15:50:05.061 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7688999999999998e-05 2024-09-21T15:50:05.061 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.061 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.061 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:05.061 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.655120+0000", 2024-09-21T15:50:05.061 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6690000000000003e-05 2024-09-21T15:50:05.062 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:05.062 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:05.062 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:05.062 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.062 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.062 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120597 3.1d 3:bf04a118:::rbd_data.1133545d3fa6.0000000000000017:1e [read 808960~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:05.062 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.618619+0000", 2024-09-21T15:50:05.062 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 1.399380211, 2024-09-21T15:50:05.062 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.036608329000000002, 2024-09-21T15:50:05.062 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:05.062 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:05.063 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:05.063 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:05.063 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:05.063 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120597 2024-09-21T15:50:05.063 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.063 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:05.063 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.063 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:05.063 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618619+0000", 2024-09-21T15:50:05.063 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:05.063 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.063 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.064 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:05.064 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618619+0000", 2024-09-21T15:50:05.064 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:05.064 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.064 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.064 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:05.064 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618619+0000", 2024-09-21T15:50:05.064 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:50:05.064 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.064 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.064 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:05.064 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618625+0000", 2024-09-21T15:50:05.064 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5570000000000003e-06 2024-09-21T15:50:05.065 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.065 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.065 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:05.065 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618628+0000", 2024-09-21T15:50:05.065 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.269e-06 2024-09-21T15:50:05.065 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.065 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.065 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:05.065 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618630+0000", 2024-09-21T15:50:05.065 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.401e-06 2024-09-21T15:50:05.065 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.065 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.065 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:05.066 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.655141+0000", 2024-09-21T15:50:05.066 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.036511096999999999 2024-09-21T15:50:05.066 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.066 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.066 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:05.066 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.655184+0000", 2024-09-21T15:50:05.066 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2042999999999997e-05 2024-09-21T15:50:05.066 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.066 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.066 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:05.066 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.655228+0000", 2024-09-21T15:50:05.066 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4181999999999998e-05 2024-09-21T15:50:05.067 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:05.067 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:05.067 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:05.067 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.067 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.067 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120598 3.1d 3:bf04a118:::rbd_data.1133545d3fa6.0000000000000017:1e [read 829440~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:05.067 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.618633+0000", 2024-09-21T15:50:05.067 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 1.399366975, 2024-09-21T15:50:05.067 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.036695428000000002, 2024-09-21T15:50:05.067 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:05.067 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:05.067 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:05.068 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:05.068 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:05.068 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120598 2024-09-21T15:50:05.069 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.069 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:05.069 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.069 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:05.070 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618633+0000", 2024-09-21T15:50:05.070 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:05.070 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.070 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.070 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:05.070 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618633+0000", 2024-09-21T15:50:05.070 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:05.070 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.070 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.070 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:05.070 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618632+0000", 2024-09-21T15:50:05.071 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:05.071 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.071 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.071 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:05.071 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618634+0000", 2024-09-21T15:50:05.071 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1990000000000001e-06 2024-09-21T15:50:05.071 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.071 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.071 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:05.071 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618636+0000", 2024-09-21T15:50:05.071 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8650000000000001e-06 2024-09-21T15:50:05.072 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.072 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.072 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:05.072 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618639+0000", 2024-09-21T15:50:05.072 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4399999999999999e-06 2024-09-21T15:50:05.072 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.072 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.072 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:05.072 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.655250+0000", 2024-09-21T15:50:05.072 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.036610868999999997 2024-09-21T15:50:05.072 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.072 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.073 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:05.073 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.655293+0000", 2024-09-21T15:50:05.073 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3748000000000002e-05 2024-09-21T15:50:05.073 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.073 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.073 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:05.073 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.655328+0000", 2024-09-21T15:50:05.073 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4551000000000002e-05 2024-09-21T15:50:05.073 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:05.073 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:05.073 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:05.073 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.074 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.074 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120599 3.1d 3:bf04a118:::rbd_data.1133545d3fa6.0000000000000017:1e [read 849920~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:05.074 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.618641+0000", 2024-09-21T15:50:05.074 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 1.3993584349999999, 2024-09-21T15:50:05.074 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.036790430999999998, 2024-09-21T15:50:05.074 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:05.074 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:05.074 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:05.074 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:05.074 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:05.074 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120599 2024-09-21T15:50:05.074 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.075 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:05.075 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.075 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:05.075 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618641+0000", 2024-09-21T15:50:05.075 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:05.075 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.075 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.075 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:05.075 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618641+0000", 2024-09-21T15:50:05.075 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:05.075 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.075 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.076 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:05.076 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618641+0000", 2024-09-21T15:50:05.076 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:05.076 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.076 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.076 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:05.076 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618642+0000", 2024-09-21T15:50:05.076 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.308e-06 2024-09-21T15:50:05.076 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.076 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.076 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:05.076 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618645+0000", 2024-09-21T15:50:05.076 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7719999999999999e-06 2024-09-21T15:50:05.077 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.077 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.077 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:05.077 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618648+0000", 2024-09-21T15:50:05.077 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2880000000000001e-06 2024-09-21T15:50:05.077 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.077 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.077 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:05.077 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.655348+0000", 2024-09-21T15:50:05.077 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.036699831000000002 2024-09-21T15:50:05.077 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.077 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.077 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:05.078 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.655386+0000", 2024-09-21T15:50:05.078 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8457000000000001e-05 2024-09-21T15:50:05.078 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.078 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.078 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:05.078 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.655432+0000", 2024-09-21T15:50:05.078 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5209e-05 2024-09-21T15:50:05.078 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:05.078 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:05.078 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:05.078 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.078 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.078 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120600 3.1d 3:bf04a118:::rbd_data.1133545d3fa6.0000000000000017:1e [read 870400~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:05.079 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.618660+0000", 2024-09-21T15:50:05.079 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 1.3993398889999999, 2024-09-21T15:50:05.079 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.036866313999999997, 2024-09-21T15:50:05.079 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:05.079 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:05.079 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:05.079 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:05.079 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:05.079 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120600 2024-09-21T15:50:05.079 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.079 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:05.079 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.080 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:05.080 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618660+0000", 2024-09-21T15:50:05.080 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:05.080 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.080 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.080 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:05.080 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618660+0000", 2024-09-21T15:50:05.080 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:05.080 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.080 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.080 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:05.081 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618659+0000", 2024-09-21T15:50:05.081 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:05.081 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.081 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.081 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:05.081 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618661+0000", 2024-09-21T15:50:05.081 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.818e-06 2024-09-21T15:50:05.081 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.081 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.081 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:05.081 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618664+0000", 2024-09-21T15:50:05.081 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1219999999999999e-06 2024-09-21T15:50:05.082 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.082 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.082 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:05.082 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618667+0000", 2024-09-21T15:50:05.082 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.402e-06 2024-09-21T15:50:05.082 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.082 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.082 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:05.082 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.655453+0000", 2024-09-21T15:50:05.082 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.03678609 2024-09-21T15:50:05.082 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.082 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.083 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:05.083 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.655489+0000", 2024-09-21T15:50:05.083 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6675999999999999e-05 2024-09-21T15:50:05.083 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.083 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.083 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:05.083 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.655526+0000", 2024-09-21T15:50:05.083 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6659000000000003e-05 2024-09-21T15:50:05.083 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:05.083 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:05.083 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:05.083 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.084 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.084 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120601 3.1d 3:bf04a118:::rbd_data.1133545d3fa6.0000000000000017:1e [read 890880~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:05.084 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.618670+0000", 2024-09-21T15:50:05.084 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 1.399329974, 2024-09-21T15:50:05.084 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.036947299000000003, 2024-09-21T15:50:05.084 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:05.084 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:05.084 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:05.084 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:05.084 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:05.084 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120601 2024-09-21T15:50:05.085 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.085 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:05.085 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.085 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:05.085 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618670+0000", 2024-09-21T15:50:05.085 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:05.085 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.085 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.085 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:05.085 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618670+0000", 2024-09-21T15:50:05.085 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:05.085 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.086 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.086 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:05.086 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618669+0000", 2024-09-21T15:50:05.086 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:50:05.086 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.086 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.086 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:05.086 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618671+0000", 2024-09-21T15:50:05.086 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.587e-06 2024-09-21T15:50:05.086 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.086 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.086 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:05.087 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618674+0000", 2024-09-21T15:50:05.087 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.9100000000000001e-06 2024-09-21T15:50:05.087 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.087 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.087 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:05.087 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618676+0000", 2024-09-21T15:50:05.087 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4200000000000001e-06 2024-09-21T15:50:05.087 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.087 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.087 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:05.087 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.655547+0000", 2024-09-21T15:50:05.088 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.03687095 2024-09-21T15:50:05.088 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.088 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.088 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:05.088 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.655583+0000", 2024-09-21T15:50:05.088 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5646999999999999e-05 2024-09-21T15:50:05.088 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.088 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.088 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:05.088 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.655617+0000", 2024-09-21T15:50:05.088 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4013000000000001e-05 2024-09-21T15:50:05.088 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:05.089 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:05.089 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:05.089 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.089 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.089 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120602 3.1d 3:bf04a118:::rbd_data.1133545d3fa6.0000000000000017:1e [read 911360~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:05.089 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.618679+0000", 2024-09-21T15:50:05.089 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 1.3993209, 2024-09-21T15:50:05.089 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.037074337999999998, 2024-09-21T15:50:05.089 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:05.089 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:05.089 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:05.090 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:05.090 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:05.090 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120602 2024-09-21T15:50:05.090 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.090 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:05.090 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.090 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:05.090 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618679+0000", 2024-09-21T15:50:05.090 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:05.090 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.090 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.090 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:05.091 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618679+0000", 2024-09-21T15:50:05.091 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:05.091 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.091 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.091 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:05.091 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618678+0000", 2024-09-21T15:50:05.091 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:05.091 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.091 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.091 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:05.091 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618680+0000", 2024-09-21T15:50:05.091 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3230000000000001e-06 2024-09-21T15:50:05.092 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.092 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.092 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:05.092 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618682+0000", 2024-09-21T15:50:05.092 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8169999999999999e-06 2024-09-21T15:50:05.092 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.092 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.092 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:05.092 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618689+0000", 2024-09-21T15:50:05.092 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1419999999999998e-06 2024-09-21T15:50:05.092 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.092 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.093 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:05.093 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.655637+0000", 2024-09-21T15:50:05.093 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.036948478999999999 2024-09-21T15:50:05.093 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.093 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.093 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:05.093 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.655701+0000", 2024-09-21T15:50:05.093 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3717e-05 2024-09-21T15:50:05.093 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.093 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.093 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:05.093 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.655753+0000", 2024-09-21T15:50:05.094 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2286e-05 2024-09-21T15:50:05.094 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:05.094 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:05.094 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:05.094 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.094 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.094 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120603 3.1d 3:bf04a118:::rbd_data.1133545d3fa6.0000000000000017:1e [read 927744~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:05.094 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.618701+0000", 2024-09-21T15:50:05.094 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 1.39929836, 2024-09-21T15:50:05.094 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.037165192999999999, 2024-09-21T15:50:05.094 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:05.094 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:05.094 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:05.095 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:05.095 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:05.095 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120603 2024-09-21T15:50:05.095 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.095 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:05.095 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.095 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:05.095 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618701+0000", 2024-09-21T15:50:05.095 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:05.095 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.095 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.095 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:05.095 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618701+0000", 2024-09-21T15:50:05.096 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:05.096 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.096 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.096 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:05.096 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618701+0000", 2024-09-21T15:50:05.096 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:05.096 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.096 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.096 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:05.096 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618703+0000", 2024-09-21T15:50:05.096 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.812e-06 2024-09-21T15:50:05.096 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.096 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.096 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:05.097 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618706+0000", 2024-09-21T15:50:05.097 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1640000000000001e-06 2024-09-21T15:50:05.097 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.097 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.097 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:05.097 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618708+0000", 2024-09-21T15:50:05.097 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3379999999999999e-06 2024-09-21T15:50:05.097 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.097 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.097 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:05.097 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.655775+0000", 2024-09-21T15:50:05.097 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.037067127999999998 2024-09-21T15:50:05.098 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.098 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.098 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:05.098 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.655826+0000", 2024-09-21T15:50:05.098 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1187999999999998e-05 2024-09-21T15:50:05.098 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.098 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.098 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:05.098 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.655866+0000", 2024-09-21T15:50:05.098 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9978999999999998e-05 2024-09-21T15:50:05.098 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:05.098 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:05.098 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:05.099 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.099 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.099 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120604 3.1d 3:bf04a118:::rbd_data.1133545d3fa6.0000000000000017:1e [read 948224~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:05.099 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.618727+0000", 2024-09-21T15:50:05.099 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 1.3992727469999999, 2024-09-21T15:50:05.099 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.037301102000000003, 2024-09-21T15:50:05.099 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:05.099 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:05.099 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:05.099 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:05.099 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:05.099 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120604 2024-09-21T15:50:05.100 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.100 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:05.100 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.100 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:05.100 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618727+0000", 2024-09-21T15:50:05.100 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:05.100 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.100 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.100 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:05.100 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618727+0000", 2024-09-21T15:50:05.101 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:05.101 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.101 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.101 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:05.101 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618727+0000", 2024-09-21T15:50:05.101 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:05.101 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.101 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.101 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:05.101 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618728+0000", 2024-09-21T15:50:05.101 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.612e-06 2024-09-21T15:50:05.101 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.101 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.102 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:05.102 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618731+0000", 2024-09-21T15:50:05.102 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.027e-06 2024-09-21T15:50:05.102 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.102 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.102 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:05.102 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618734+0000", 2024-09-21T15:50:05.102 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4169999999999999e-06 2024-09-21T15:50:05.102 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.102 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.102 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:05.102 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.655887+0000", 2024-09-21T15:50:05.103 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.037153409999999998 2024-09-21T15:50:05.103 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.103 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.103 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:05.103 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.655934+0000", 2024-09-21T15:50:05.103 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6755000000000002e-05 2024-09-21T15:50:05.103 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.103 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.103 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:05.103 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.656028+0000", 2024-09-21T15:50:05.103 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4221000000000005e-05 2024-09-21T15:50:05.103 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:05.103 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:05.104 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:05.104 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.104 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.104 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120605 3.1d 3:bf04a118:::rbd_data.1133545d3fa6.0000000000000017:1e [read 968704~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:05.104 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.618755+0000", 2024-09-21T15:50:05.104 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 1.399244911, 2024-09-21T15:50:05.104 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.037399375999999998, 2024-09-21T15:50:05.104 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:05.104 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:05.104 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:05.104 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:05.104 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:05.105 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120605 2024-09-21T15:50:05.105 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.105 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:05.105 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.105 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:05.105 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618755+0000", 2024-09-21T15:50:05.105 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:05.105 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.105 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.105 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:05.105 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618755+0000", 2024-09-21T15:50:05.106 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:05.106 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.106 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.106 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:05.106 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618754+0000", 2024-09-21T15:50:05.106 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:05.106 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.106 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.106 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:05.106 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618756+0000", 2024-09-21T15:50:05.106 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.734e-06 2024-09-21T15:50:05.106 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.107 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.107 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:05.107 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618759+0000", 2024-09-21T15:50:05.107 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.365e-06 2024-09-21T15:50:05.107 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.107 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.107 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:05.107 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618762+0000", 2024-09-21T15:50:05.107 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8159999999999998e-06 2024-09-21T15:50:05.107 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.107 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.107 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:05.107 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.656062+0000", 2024-09-21T15:50:05.108 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.037299448999999998 2024-09-21T15:50:05.108 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.108 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.108 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:05.108 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.656111+0000", 2024-09-21T15:50:05.108 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9568999999999999e-05 2024-09-21T15:50:05.108 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.108 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.108 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:05.108 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.656154+0000", 2024-09-21T15:50:05.108 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2892999999999997e-05 2024-09-21T15:50:05.108 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:05.109 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:05.109 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:05.109 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.109 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.109 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120606 3.1d 3:bf04a118:::rbd_data.1133545d3fa6.0000000000000017:1e [read 989184~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:05.109 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.618767+0000", 2024-09-21T15:50:05.109 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 1.3992328679999999, 2024-09-21T15:50:05.109 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.037514430000000001, 2024-09-21T15:50:05.109 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:05.109 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:05.109 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:05.109 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:05.110 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:05.110 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120606 2024-09-21T15:50:05.110 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.110 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:05.110 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.110 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:05.110 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618767+0000", 2024-09-21T15:50:05.110 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:05.110 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.110 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.110 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:05.111 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618767+0000", 2024-09-21T15:50:05.111 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:05.111 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.111 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.111 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:05.111 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618766+0000", 2024-09-21T15:50:05.111 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:05.111 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.111 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.111 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:05.111 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618768+0000", 2024-09-21T15:50:05.111 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7409999999999999e-06 2024-09-21T15:50:05.112 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.112 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.112 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:05.112 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618774+0000", 2024-09-21T15:50:05.112 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4009999999999996e-06 2024-09-21T15:50:05.112 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.112 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.112 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:05.112 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618778+0000", 2024-09-21T15:50:05.112 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3340000000000003e-06 2024-09-21T15:50:05.112 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.112 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.113 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:05.113 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.656183+0000", 2024-09-21T15:50:05.113 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.037405074000000003 2024-09-21T15:50:05.113 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.113 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.113 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:05.113 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.656241+0000", 2024-09-21T15:50:05.113 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7723999999999998e-05 2024-09-21T15:50:05.113 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.113 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.113 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:05.113 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.656281+0000", 2024-09-21T15:50:05.113 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0436999999999999e-05 2024-09-21T15:50:05.114 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:05.114 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:05.114 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:05.114 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.114 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.114 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120607 3.1d 3:bf04a118:::rbd_data.1133545d3fa6.0000000000000017:1e [read 1009664~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:05.114 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.618793+0000", 2024-09-21T15:50:05.114 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 1.399206838, 2024-09-21T15:50:05.114 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.037678489000000003, 2024-09-21T15:50:05.114 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:05.114 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:05.115 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:05.115 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:05.115 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:05.115 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120607 2024-09-21T15:50:05.115 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.115 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:05.115 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.115 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:05.115 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618793+0000", 2024-09-21T15:50:05.115 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:05.115 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.115 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.115 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:05.116 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618793+0000", 2024-09-21T15:50:05.116 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:05.116 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.116 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.116 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:05.116 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618792+0000", 2024-09-21T15:50:05.116 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:05.116 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.116 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.116 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:05.116 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618794+0000", 2024-09-21T15:50:05.116 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1600000000000001e-06 2024-09-21T15:50:05.116 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.116 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.117 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:05.117 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618798+0000", 2024-09-21T15:50:05.117 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5729999999999999e-06 2024-09-21T15:50:05.117 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.117 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.117 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:05.117 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618801+0000", 2024-09-21T15:50:05.117 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6220000000000001e-06 2024-09-21T15:50:05.117 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.117 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.117 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:05.117 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.656314+0000", 2024-09-21T15:50:05.117 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.037513611000000002 2024-09-21T15:50:05.118 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.118 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.118 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:05.118 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.656383+0000", 2024-09-21T15:50:05.118 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8917000000000004e-05 2024-09-21T15:50:05.118 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.118 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.118 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:05.118 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.656471+0000", 2024-09-21T15:50:05.118 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8085999999999997e-05 2024-09-21T15:50:05.118 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:05.118 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:05.119 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:05.119 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.119 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.119 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120608 3.1d 3:bf04a118:::rbd_data.1133545d3fa6.0000000000000017:1e [read 1030144~0] snapc 0=[] RETRY=2 ondisk+retry+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:05.119 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.618803+0000", 2024-09-21T15:50:05.119 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 1.3991968530000001, 2024-09-21T15:50:05.119 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.037854451999999997, 2024-09-21T15:50:05.119 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:05.119 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:05.119 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:05.119 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:05.119 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:05.119 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120608 2024-09-21T15:50:05.120 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.120 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:05.120 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.120 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:05.120 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618803+0000", 2024-09-21T15:50:05.120 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:05.120 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.120 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.120 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:05.120 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618803+0000", 2024-09-21T15:50:05.120 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:05.120 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.121 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.121 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:05.121 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618803+0000", 2024-09-21T15:50:05.121 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:50:05.121 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.121 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.121 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:05.121 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618804+0000", 2024-09-21T15:50:05.121 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1909999999999999e-06 2024-09-21T15:50:05.121 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.121 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.121 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:05.122 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618807+0000", 2024-09-21T15:50:05.122 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8100000000000002e-06 2024-09-21T15:50:05.122 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.122 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.122 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:05.122 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.618809+0000", 2024-09-21T15:50:05.122 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.503e-06 2024-09-21T15:50:05.122 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.122 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.122 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:05.122 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.656502+0000", 2024-09-21T15:50:05.122 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.037692812999999999 2024-09-21T15:50:05.123 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.123 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.123 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:05.123 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.656580+0000", 2024-09-21T15:50:05.123 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7755000000000003e-05 2024-09-21T15:50:05.123 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:05.123 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:05.123 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:05.123 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.656657+0000", 2024-09-21T15:50:05.123 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7583999999999994e-05 2024-09-21T15:50:05.123 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:05.123 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:05.124 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:05.124 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:05.124 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:05.124 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:05.125 INFO:teuthology.orchestra.run.smithi089.stdout:truncating image image_client.0-clone27 from 0x2b558fe (overlap 0x221954a) to 0x110caa5 2024-09-21T15:50:05.126 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:50:05.126 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:05.124+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:05.154 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:50:05.226 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:05.224+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:05.259 INFO:teuthology.orchestra.run.smithi089.stdout:checking clone #26, image image_client.0-clone26 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent27 2024-09-21T15:50:05.280 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:05.280 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:05.280 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:50:05.280 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:05.281 INFO:teuthology.orchestra.run.smithi089.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T15:50:05.289 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:50:05.297 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:50:05.297 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:50:05.318 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:50:05.318 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:50:05.327 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:05.326+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:05.349 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:50:05.349 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:50:05.367 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:50:05.367 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:50:05.383 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:50:05.384 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:50:05.400 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:50:05.400 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:50:05.416 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:50:05.417 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:50:05.421 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T15:50:05.421 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T15:50:05.428 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:05.427+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:05.434 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:05.434 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:05.443 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:50:05.515 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:05.515 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:05.528 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:05.527+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:05.531 INFO:teuthology.orchestra.run.smithi089.stdout:4213 write 0x761f339 thru 0x7629c29 (0xa8f1 bytes) 2024-09-21T15:50:05.558 INFO:teuthology.orchestra.run.smithi089.stdout:4214 trunc from 0x7629c2a to 0xa812cab 2024-09-21T15:50:05.565 INFO:teuthology.orchestra.run.smithi089.stdout:4215 trunc from 0xa812cab to 0xab90707 2024-09-21T15:50:05.574 INFO:teuthology.orchestra.run.smithi089.stdout:4217 trunc from 0xab90707 to 0x69291d1 2024-09-21T15:50:05.596 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:05.596 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:05.600 INFO:teuthology.orchestra.run.smithi089.stdout:4218 write 0xaa6a1df thru 0xaa74efa (0xad1c bytes) 2024-09-21T15:50:05.603 INFO:teuthology.orchestra.run.smithi089.stdout:4219 read 0x9994ad6 thru 0x99a3211 (0xe73c bytes) 2024-09-21T15:50:05.603 INFO:teuthology.orchestra.run.smithi089.stdout:4220 writesame 0x8438165 thru 0x843934a data_size 0x1(0x11e6 bytes) 2024-09-21T15:50:05.605 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T15:50:05.612 INFO:teuthology.orchestra.run.smithi089.stdout:4221 writesame 0xb5c50ee thru 0xb5cab0c data_size 0x1(0x5a1f bytes) 2024-09-21T15:50:05.620 INFO:teuthology.orchestra.run.smithi089.stdout:4224 write 0xd84da50 thru 0xd85bbad (0xe15e bytes) 2024-09-21T15:50:05.624 INFO:teuthology.orchestra.run.smithi089.stdout:4225 read 0x44cddba thru 0x44d0ee4 (0x312b bytes) 2024-09-21T15:50:05.624 INFO:teuthology.orchestra.run.smithi089.stdout:4228 write 0x9c272b9 thru 0x9c32aad (0xb7f5 bytes) 2024-09-21T15:50:05.627 INFO:teuthology.orchestra.run.smithi089.stdout:4229 writesame 0x553fd5e thru 0x55425f0 data_size 0x1(0x2893 bytes) 2024-09-21T15:50:05.629 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:05.628+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:05.633 INFO:teuthology.orchestra.run.smithi089.stdout:4230 write 0xce5ab0 thru 0xce9fbf (0x4510 bytes) 2024-09-21T15:50:05.635 INFO:teuthology.orchestra.run.smithi089.stdout:4231 read 0x81a6843 thru 0x81af368 (0x8b26 bytes) 2024-09-21T15:50:05.635 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:05.648 INFO:teuthology.orchestra.run.smithi089.stdout:4233 trunc from 0xd85bbae to 0x27b06d5 2024-09-21T15:50:05.731 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:05.729+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:05.741 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:05.741 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:05.750 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T15:50:05.811 INFO:teuthology.orchestra.run.smithi089.stdout:4234 write 0x5e9e372 thru 0x5eac4dc (0xe16b bytes) 2024-09-21T15:50:05.819 INFO:teuthology.orchestra.run.smithi089.stdout:4235 trunc from 0x5eac4dd to 0x1705e7e 2024-09-21T15:50:05.834 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:05.833+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:05.841 INFO:teuthology.orchestra.run.smithi089.stdout:4236 read 0x12bed56 thru 0x12cdaed (0xed98 bytes) 2024-09-21T15:50:05.841 INFO:teuthology.orchestra.run.smithi089.stdout:4237 read 0x16295d thru 0x1673f5 (0x4a99 bytes) 2024-09-21T15:50:05.853 INFO:teuthology.orchestra.run.smithi089.stdout:4240 writesame 0x5408fb1 thru 0x54134e9 data_size 0x17(0xa539 bytes) 2024-09-21T15:50:05.855 INFO:teuthology.orchestra.run.smithi089.stdout:4241 trunc from 0x54134ea to 0x665b87a 2024-09-21T15:50:05.870 INFO:teuthology.orchestra.run.smithi089.stdout: marking collection for removal 2024-09-21T15:50:05.870 INFO:teuthology.orchestra.run.smithi089.stdout:setting '_remove' omap key 2024-09-21T15:50:05.870 INFO:teuthology.orchestra.run.smithi089.stdout:finish_remove_pgs 2.4_head removing 2.4 2024-09-21T15:50:05.871 INFO:teuthology.orchestra.run.smithi089.stdout:Remove successful 2024-09-21T15:50:05.873 INFO:teuthology.orchestra.run.smithi089.stdout:4243 write 0xcbe11a3 thru 0xcbeb816 (0xa674 bytes) 2024-09-21T15:50:05.876 INFO:teuthology.orchestra.run.smithi089.stdout:4244 punch from 0x30a390 to 0x313106, (0x8d76 bytes) 2024-09-21T15:50:05.880 INFO:teuthology.orchestra.run.smithi089.stdout:4245 trunc from 0xcbeb817 to 0x981ffff 2024-09-21T15:50:05.894 INFO:teuthology.orchestra.run.smithi089.stdout:4246 trunc from 0x981ffff to 0x660908b 2024-09-21T15:50:05.900 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:05.900 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:05.901 INFO:teuthology.orchestra.run.smithi089.stdout:4247 read 0x6467b7c thru 0x646cc5f (0x50e4 bytes) 2024-09-21T15:50:05.901 INFO:teuthology.orchestra.run.smithi089.stdout:4248 read 0x600a4fa thru 0x6014abc (0xa5c3 bytes) 2024-09-21T15:50:05.901 INFO:teuthology.orchestra.run.smithi089.stdout:4249 write 0x226a596 thru 0x2275f67 (0xb9d2 bytes) 2024-09-21T15:50:05.906 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:05.906 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:05.906 INFO:teuthology.orchestra.run.smithi089.stdout:4252 write 0x23c17ad thru 0x23cfb37 (0xe38b bytes) 2024-09-21T15:50:05.907 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:05.910 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T15:50:05.919 INFO:teuthology.orchestra.run.smithi089.stdout:4254 write 0xba8409b thru 0xba87fc5 (0x3f2b bytes) 2024-09-21T15:50:05.922 INFO:teuthology.orchestra.run.smithi089.stdout:4255 write 0x9c36026 thru 0x9c449e2 (0xe9bd bytes) 2024-09-21T15:50:05.925 INFO:teuthology.orchestra.run.smithi089.stdout:4256 write 0x961692e thru 0x9626917 (0xffea bytes) 2024-09-21T15:50:05.928 INFO:teuthology.orchestra.run.smithi089.stdout:4257 read 0x3442ddb thru 0x344cfe1 (0xa207 bytes) 2024-09-21T15:50:05.932 INFO:teuthology.orchestra.run.smithi089.stdout:4258 trunc from 0xba87fc6 to 0xe4ff0f5 2024-09-21T15:50:05.932 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:05.931+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:05.947 INFO:teuthology.orchestra.run.smithi089.stdout:4261 trunc from 0xe4ff0f5 to 0x4d5255c 2024-09-21T15:50:05.963 INFO:teuthology.orchestra.run.smithi089.stdout:4262 trunc from 0x4d5255c to 0x855ead9 2024-09-21T15:50:05.968 INFO:teuthology.orchestra.run.smithi089.stdout:4264 read 0x4ce1afd thru 0x4ce1fa5 (0x4a9 bytes) 2024-09-21T15:50:05.975 INFO:teuthology.orchestra.run.smithi089.stdout:4265 trunc from 0x855ead9 to 0xe31efaf 2024-09-21T15:50:05.986 INFO:teuthology.orchestra.run.smithi089.stdout:4266 writesame 0xeb8114f thru 0xeb81aff data_size 0x1(0x9b1 bytes) 2024-09-21T15:50:05.991 INFO:teuthology.orchestra.run.smithi089.stdout:4267 trunc from 0xeb81b00 to 0xa18a23c 2024-09-21T15:50:05.997 INFO:teuthology.orchestra.run.smithi089.stdout:4268 read 0x88c3bc7 thru 0x88c5a91 (0x1ecb bytes) 2024-09-21T15:50:05.997 INFO:teuthology.orchestra.run.smithi089.stdout:4271 write 0x34d2aaa thru 0x34e1318 (0xe86f bytes) 2024-09-21T15:50:05.998 INFO:teuthology.orchestra.run.smithi089.stdout:4272 punch from 0x7a4f228 to 0x7a59dbd, (0xab95 bytes) 2024-09-21T15:50:05.999 INFO:teuthology.orchestra.run.smithi089.stdout:4273 write 0x881737d thru 0x881b39a (0x401e bytes) 2024-09-21T15:50:06.007 INFO:teuthology.orchestra.run.smithi089.stdout:4275 trunc from 0xa18a23c to 0x60c2473 2024-09-21T15:50:06.017 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T15:50:06.017 INFO:teuthology.orchestra.run.smithi089.stdout:4278 write 0x161c873 thru 0x1628722 (0xbeb0 bytes) 2024-09-21T15:50:06.019 INFO:teuthology.orchestra.run.smithi089.stdout:4279 writesame 0x541d1c9 thru 0x5426e2a data_size 0x2(0x9c62 bytes) 2024-09-21T15:50:06.027 INFO:teuthology.orchestra.run.smithi089.stdout:4283 punch from 0xc8f9bb to 0xc91f61, (0x25a6 bytes) 2024-09-21T15:50:06.028 INFO:teuthology.orchestra.run.smithi089.stdout:4284 clone 29 order 25 su 16777216 sc 4 2024-09-21T15:50:06.033 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:06.032+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:06.059 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:06.059 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:06.068 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T15:50:06.133 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:06.132+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:06.211 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:06.211 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:06.219 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T15:50:06.234 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:06.233+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:06.320 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:06.320 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:06.329 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T15:50:06.335 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:06.334+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:06.380 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.2.4.0 2024-09-21T15:50:06.429 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:06.430 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:06.438 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:50:06.465 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:06.464+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:06.537 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:06.535+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:06.566 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:06.566 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:06.575 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:50:06.638 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:06.636+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:06.717 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:06.717 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:06.726 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:50:06.769 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:06.768+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:06.835 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:06.835 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:06.839 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:06.838+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:06.846 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:50:06.940 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:06.939+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:06.990 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:06.990 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:06.999 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:50:07.004 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is set 2024-09-21T15:50:07.041 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:07.040+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:07.081 INFO:teuthology.orchestra.run.smithi089.stdout:truncating image image_client.0-clone28 from 0x60c2473 (overlap 0x1705e7e) to 0xb82f3f 2024-09-21T15:50:07.113 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:07.114 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:07.123 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:50:07.163 INFO:teuthology.orchestra.run.smithi089.stdout:checking clone #27, image image_client.0-clone27 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent28 2024-09-21T15:50:07.164 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:07.162+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:07.243 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:07.242+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:07.258 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:07.258 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:07.268 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:50:07.275 INFO:teuthology.orchestra.run.smithi089.stdout:4285 read 0x43b9dd5 thru 0x43bb699 (0x18c5 bytes) 2024-09-21T15:50:07.276 INFO:teuthology.orchestra.run.smithi089.stdout:4287 read 0x16c18f0 thru 0x16c35d4 (0x1ce5 bytes) 2024-09-21T15:50:07.277 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:07.277 INFO:teuthology.orchestra.run.smithi089.stdout:4290 write 0x2ffc02a thru 0x3000e78 (0x4e4f bytes) 2024-09-21T15:50:07.287 INFO:teuthology.orchestra.run.smithi089.stdout:4292 read 0x5c2587 thru 0x5cde05 (0xb87f bytes) 2024-09-21T15:50:07.293 INFO:teuthology.orchestra.run.smithi089.stdout:4294 write 0x6d10a0d thru 0x6d12b35 (0x2129 bytes) 2024-09-21T15:50:07.295 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:07.295 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:07.295 INFO:teuthology.orchestra.run.smithi089.stdout:4299 read 0x58e4452 thru 0x58e8959 (0x4508 bytes) 2024-09-21T15:50:07.295 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:07.300 INFO:teuthology.orchestra.run.smithi089.stdout:4302 writesame 0x7d8f1a0 thru 0x7d931db data_size 0x2(0x403c bytes) 2024-09-21T15:50:07.309 INFO:teuthology.orchestra.run.smithi089.stdout:4303 trunc from 0x7d931dc to 0x1b81c96 2024-09-21T15:50:07.344 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:07.343+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:07.352 INFO:teuthology.orchestra.run.smithi089.stdout:4307 trunc from 0x1b81c96 to 0x38a11c0 2024-09-21T15:50:07.358 INFO:teuthology.orchestra.run.smithi089.stdout:4308 write 0x633f02 thru 0x636146 (0x2245 bytes) 2024-09-21T15:50:07.369 INFO:teuthology.orchestra.run.smithi089.stdout:4309 trunc from 0x38a11c0 to 0xbffbcec 2024-09-21T15:50:07.374 INFO:teuthology.orchestra.run.smithi089.stdout:4311 write 0x94e1060 thru 0x94e6ca7 (0x5c48 bytes) 2024-09-21T15:50:07.377 INFO:teuthology.orchestra.run.smithi089.stdout:4312 flatten 2024-09-21T15:50:07.382 INFO:teuthology.orchestra.run.smithi089.stdout:4313 write 0xb7592d5 thru 0xb766943 (0xd66f bytes) 2024-09-21T15:50:07.386 INFO:teuthology.orchestra.run.smithi089.stdout:4314 punch from 0x522b74f to 0x522c77b, (0x102c bytes) 2024-09-21T15:50:07.390 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:07.390 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:07.399 INFO:teuthology.orchestra.run.smithi089.stdout:4315 trunc from 0xbffbcec to 0x1263728 2024-09-21T15:50:07.403 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:50:07.409 INFO:teuthology.orchestra.run.smithi089.stdout:4316 read 0x105254 thru 0x114b3e (0xf8eb bytes) 2024-09-21T15:50:07.418 INFO:teuthology.orchestra.run.smithi089.stdout:4317 write 0x4395267 thru 0x43952bd (0x57 bytes) 2024-09-21T15:50:07.422 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:50:07.427 INFO:teuthology.orchestra.run.smithi089.stdout:4318 trunc from 0x43952be to 0xaf29675 2024-09-21T15:50:07.441 INFO:teuthology.orchestra.run.smithi089.stdout:4319 write 0xcb8cb69 thru 0xcb916d6 (0x4b6e bytes) 2024-09-21T15:50:07.445 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:07.444+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:07.445 INFO:teuthology.orchestra.run.smithi089.stdout:4322 trunc from 0xcb916d7 to 0xcb760fd 2024-09-21T15:50:07.452 INFO:teuthology.orchestra.run.smithi089.stdout:4324 punch from 0x2e8f125 to 0x2e9a2db, (0xb1b6 bytes) 2024-09-21T15:50:07.452 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:07.452 INFO:teuthology.orchestra.run.smithi089.stdout:4328 read 0x678f70f thru 0x6794430 (0x4d22 bytes) 2024-09-21T15:50:07.453 INFO:teuthology.orchestra.run.smithi089.stdout:4332 writesame 0x72075d7 thru 0x720ad96 data_size 0x8(0x37c0 bytes) 2024-09-21T15:50:07.456 INFO:teuthology.orchestra.run.smithi089.stdout:4334 write 0xa9cb053 thru 0xa9d7c9d (0xcc4b bytes) 2024-09-21T15:50:07.461 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:07.461 INFO:teuthology.orchestra.run.smithi089.stdout:4336 read 0xc733c92 thru 0xc741c0c (0xdf7b bytes) 2024-09-21T15:50:07.462 INFO:teuthology.orchestra.run.smithi089.stdout:4337 punch from 0x320a153 to 0x3215edf, (0xbd8c bytes) 2024-09-21T15:50:07.462 INFO:teuthology.orchestra.run.smithi089.stdout:4338 read 0x1e97c0e thru 0x1e9aff4 (0x33e7 bytes) 2024-09-21T15:50:07.462 INFO:teuthology.orchestra.run.smithi089.stdout:4341 writesame 0x4ab19a thru 0x4ac24b data_size 0x1(0x10b2 bytes) 2024-09-21T15:50:07.465 INFO:teuthology.orchestra.run.smithi089.stdout:4343 punch from 0x1b3ed19 to 0x1b4a60e, (0xb8f5 bytes) 2024-09-21T15:50:07.466 INFO:teuthology.orchestra.run.smithi089.stdout:4345 read 0x467e7a0 thru 0x4680902 (0x2163 bytes) 2024-09-21T15:50:07.466 INFO:teuthology.orchestra.run.smithi089.stdout:4348 write 0x9f9eff7 thru 0x9fa6f69 (0x7f73 bytes) 2024-09-21T15:50:07.470 INFO:teuthology.orchestra.run.smithi089.stdout:4349 writesame 0x4d08006 thru 0x4d0bfe0 data_size 0x1(0x3fdb bytes) 2024-09-21T15:50:07.481 INFO:teuthology.orchestra.run.smithi089.stdout:4350 write 0x486577e thru 0x487347a (0xdcfd bytes) 2024-09-21T15:50:07.484 INFO:teuthology.orchestra.run.smithi089.stdout:4351 trunc from 0xcb760fd to 0xb7e8109 2024-09-21T15:50:07.490 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:07.491 INFO:teuthology.orchestra.run.smithi089.stdout:4353 write 0xb455168 thru 0xb45af3f (0x5dd8 bytes) 2024-09-21T15:50:07.493 INFO:teuthology.orchestra.run.smithi089.stdout:4355 write 0x50fd23a thru 0x5101b2e (0x48f5 bytes) 2024-09-21T15:50:07.495 INFO:teuthology.orchestra.run.smithi089.stdout:4357 writesame 0x4a1aeb thru 0x4af0dc data_size 0x5(0xd5f2 bytes) 2024-09-21T15:50:07.502 INFO:teuthology.orchestra.run.smithi089.stdout:4359 read 0xb78bef9 thru 0xb79b94f (0xfa57 bytes) 2024-09-21T15:50:07.503 INFO:teuthology.orchestra.run.smithi089.stdout:4360 trunc from 0xb7e8109 to 0xaee35c4 2024-09-21T15:50:07.514 INFO:teuthology.orchestra.run.smithi089.stdout:4361 read 0x42fd10c thru 0x430778a (0xa67f bytes) 2024-09-21T15:50:07.514 INFO:teuthology.orchestra.run.smithi089.stdout:4362 read 0x298a625 thru 0x2990954 (0x6330 bytes) 2024-09-21T15:50:07.514 INFO:teuthology.orchestra.run.smithi089.stdout:4363 read 0x2833bb2 thru 0x2834d66 (0x11b5 bytes) 2024-09-21T15:50:07.514 INFO:teuthology.orchestra.run.smithi089.stdout:4364 read 0x6e16ed2 thru 0x6e1bcda (0x4e09 bytes) 2024-09-21T15:50:07.515 INFO:teuthology.orchestra.run.smithi089.stdout:4365 read 0x2c43ddc thru 0x2c4e21f (0xa444 bytes) 2024-09-21T15:50:07.515 INFO:teuthology.orchestra.run.smithi089.stdout:4368 punch from 0x33ac2bd to 0x33b8f0a, (0xcc4d bytes) 2024-09-21T15:50:07.515 INFO:teuthology.orchestra.run.smithi089.stdout:4370 trunc from 0xaee35c4 to 0xaad3f86 2024-09-21T15:50:07.520 INFO:teuthology.orchestra.run.smithi089.stdout:4371 punch from 0xc4d617 to 0xc4f9ec, (0x23d5 bytes) 2024-09-21T15:50:07.520 INFO:teuthology.orchestra.run.smithi089.stdout:4375 read 0x18e5192 thru 0x18f236c (0xd1db bytes) 2024-09-21T15:50:07.520 INFO:teuthology.orchestra.run.smithi089.stdout:4376 punch from 0x36e3780 to 0x36eeeba, (0xb73a bytes) 2024-09-21T15:50:07.523 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:07.523 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:07.531 INFO:teuthology.orchestra.run.smithi089.stdout:get_pg_num_history pg_num_history pg_num_history(e67 pg_nums {1={14=1},2={17=8},3={22=32}} deleted_pools ) 2024-09-21T15:50:07.532 INFO:teuthology.orchestra.run.smithi089.stdout:Importing pgid 2.4 2024-09-21T15:50:07.532 INFO:teuthology.orchestra.run.smithi089.stdout:4377 write 0xd2c108b thru 0xd2c4032 (0x2fa8 bytes) 2024-09-21T15:50:07.532 INFO:teuthology.orchestra.run.smithi089.stdout:write_pg epoch 62 info 2.4( empty local-lis/les=17/18 n=0 ec=17/17 lis/c=17/17 les/c/f=18/18/0 sis=17) 2024-09-21T15:50:07.533 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:50:07.534 INFO:teuthology.orchestra.run.smithi089.stdout:4379 write 0x849973 thru 0x858a86 (0xf114 bytes) 2024-09-21T15:50:07.536 INFO:teuthology.orchestra.run.smithi089.stdout:4380 read 0x91e6fa6 thru 0x91edab1 (0x6b0c bytes) 2024-09-21T15:50:07.536 INFO:teuthology.orchestra.run.smithi089.stdout:4381 write 0x3443334 thru 0x344eddb (0xbaa8 bytes) 2024-09-21T15:50:07.538 INFO:teuthology.orchestra.run.smithi089.stdout:Import successful 2024-09-21T15:50:07.538 INFO:teuthology.orchestra.run.smithi089.stdout:4382 read 0x9a321f3 thru 0x9a404f1 (0xe2ff bytes) 2024-09-21T15:50:07.539 INFO:teuthology.orchestra.run.smithi089.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T15:50:07.543 INFO:teuthology.orchestra.run.smithi089.stdout:4383 trunc from 0xd2c4033 to 0xa3d5ae6 2024-09-21T15:50:07.546 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:07.545+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:07.552 INFO:teuthology.orchestra.run.smithi089.stdout:4385 punch from 0x4723081 to 0x47311e5, (0xe164 bytes) 2024-09-21T15:50:07.557 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:50:07.557 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:50:07.562 INFO:teuthology.orchestra.run.smithi089.stdout:4386 writesame 0xe943377 thru 0xe945510 data_size 0x2e(0x219a bytes) 2024-09-21T15:50:07.563 INFO:teuthology.orchestra.run.smithi089.stdout:4387 write 0xce4a25e thru 0xce584ab (0xe24e bytes) 2024-09-21T15:50:07.567 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:07.568 INFO:teuthology.orchestra.run.smithi089.stdout:4389 read 0xb77c0ea thru 0xb77ddcd (0x1ce4 bytes) 2024-09-21T15:50:07.568 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:07.568 INFO:teuthology.orchestra.run.smithi089.stdout:4393 punch from 0x76023e1 to 0x760978e, (0x73ad bytes) 2024-09-21T15:50:07.568 INFO:teuthology.orchestra.run.smithi089.stdout:4394 punch from 0x84e0328 to 0x84e4d86, (0x4a5e bytes) 2024-09-21T15:50:07.568 INFO:teuthology.orchestra.run.smithi089.stdout:4395 write 0x6553327 thru 0x6559044 (0x5d1e bytes) 2024-09-21T15:50:07.570 INFO:teuthology.orchestra.run.smithi089.stdout:4396 read 0x2320492 thru 0x23298e8 (0x9457 bytes) 2024-09-21T15:50:07.571 INFO:teuthology.orchestra.run.smithi089.stdout:4397 read 0xcb479e5 thru 0xcb4cae6 (0x5102 bytes) 2024-09-21T15:50:07.571 INFO:teuthology.orchestra.run.smithi089.stdout:4398 punch from 0xc9bf90f to 0xc9c69d5, (0x70c6 bytes) 2024-09-21T15:50:07.571 INFO:teuthology.orchestra.run.smithi089.stdout:4400 punch from 0x2fd77ae to 0x2fd9cc6, (0x2518 bytes) 2024-09-21T15:50:07.575 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:50:07.575 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:50:07.583 INFO:teuthology.orchestra.run.smithi089.stdout:4402 trunc from 0xe945511 to 0x49da0db 2024-09-21T15:50:07.592 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:50:07.592 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:50:07.598 INFO:teuthology.orchestra.run.smithi089.stdout:4403 trunc from 0x49da0db to 0x3a53bf3 2024-09-21T15:50:07.610 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:50:07.610 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:50:07.620 INFO:teuthology.orchestra.run.smithi089.stdout:4405 write 0xc9b32e2 thru 0xc9bdc52 (0xa971 bytes) 2024-09-21T15:50:07.623 INFO:teuthology.orchestra.run.smithi089.stdout:4407 read 0x9dd3afc thru 0x9de2586 (0xea8b bytes) 2024-09-21T15:50:07.627 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:50:07.627 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:50:07.628 INFO:teuthology.orchestra.run.smithi089.stdout:4408 write 0xd2cf667 thru 0xd2d7d22 (0x86bc bytes) 2024-09-21T15:50:07.632 INFO:teuthology.orchestra.run.smithi089.stdout:4409 write 0xa05ddfe thru 0xa05f0ed (0x12f0 bytes) 2024-09-21T15:50:07.634 INFO:teuthology.orchestra.run.smithi089.stdout:4410 clone 30 order 25 su 16777216 sc 6 2024-09-21T15:50:07.644 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:50:07.644 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:50:07.647 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:07.646+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:07.656 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:07.656 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:50:07.656 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:50:07.656 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:50:07.656 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.656 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:14051 3.8 3:13437810:::rbd_data.1133dc5adfdf.0000000000000039:head [write 716800~796] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-21T15:50:07.656 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:50:07.656 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 49.958255747999999, 2024-09-21T15:50:07.656 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.98064002699999997, 2024-09-21T15:50:07.656 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:07.656 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:07.656 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:07.656 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:07.657 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:07.657 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 14051 2024-09-21T15:50:07.657 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.657 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:07.657 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.657 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:07.657 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:50:07.657 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:07.657 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.657 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.657 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:07.657 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:50:07.657 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:07.657 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.657 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.657 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:07.658 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:50:07.658 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:07.658 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.658 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.658 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:07.658 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695131+0000", 2024-09-21T15:50:07.658 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4680000000000003e-06 2024-09-21T15:50:07.658 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.658 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.658 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:07.658 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695139+0000", 2024-09-21T15:50:07.658 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9400000000000002e-06 2024-09-21T15:50:07.658 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.658 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.658 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:07.659 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695143+0000", 2024-09-21T15:50:07.659 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9899999999999999e-06 2024-09-21T15:50:07.659 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.659 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.659 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:07.659 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695190+0000", 2024-09-21T15:50:07.659 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6458000000000001e-05 2024-09-21T15:50:07.659 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.659 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.659 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:50:07.659 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695201+0000", 2024-09-21T15:50:07.659 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0964999999999999e-05 2024-09-21T15:50:07.659 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.659 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.659 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:07.660 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.669490+0000", 2024-09-21T15:50:07.660 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.97428980499999995 2024-09-21T15:50:07.660 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.660 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.660 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-09-21T15:50:07.660 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.670033+0000", 2024-09-21T15:50:07.660 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00054252599999999999 2024-09-21T15:50:07.660 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.660 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.660 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:07.660 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.671095+0000", 2024-09-21T15:50:07.660 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001062183 2024-09-21T15:50:07.660 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.660 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.660 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:07.660 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.671186+0000", 2024-09-21T15:50:07.660 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1191000000000002e-05 2024-09-21T15:50:07.661 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.661 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.661 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-09-21T15:50:07.661 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.671346+0000", 2024-09-21T15:50:07.661 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015980399999999999 2024-09-21T15:50:07.661 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.661 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.661 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:07.661 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.672221+0000", 2024-09-21T15:50:07.661 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00087471999999999995 2024-09-21T15:50:07.661 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.661 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.661 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:07.661 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.675619+0000", 2024-09-21T15:50:07.662 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00339858 2024-09-21T15:50:07.662 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.662 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.662 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:07.662 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.675670+0000", 2024-09-21T15:50:07.662 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0664000000000001e-05 2024-09-21T15:50:07.662 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.662 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.662 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:07.662 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.675768+0000", 2024-09-21T15:50:07.662 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7415000000000002e-05 2024-09-21T15:50:07.662 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:07.662 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:07.662 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:07.662 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.663 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.663 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116235 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 761856~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:07.663 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048806+0000", 2024-09-21T15:50:07.663 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 11.604576864, 2024-09-21T15:50:07.663 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056818556999999999, 2024-09-21T15:50:07.663 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:07.663 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:07.663 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:07.663 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:07.663 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:07.663 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116235 2024-09-21T15:50:07.663 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.663 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:07.663 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.663 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:07.663 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048806+0000", 2024-09-21T15:50:07.664 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:07.664 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.664 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.664 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:07.664 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048806+0000", 2024-09-21T15:50:07.664 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:07.664 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.664 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.664 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:07.664 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048806+0000", 2024-09-21T15:50:07.664 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:07.664 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.664 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.664 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:07.665 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048807+0000", 2024-09-21T15:50:07.665 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.001e-06 2024-09-21T15:50:07.665 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.665 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.665 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:07.665 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048810+0000", 2024-09-21T15:50:07.665 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.711e-06 2024-09-21T15:50:07.665 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.665 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.665 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:07.665 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048812+0000", 2024-09-21T15:50:07.665 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2290000000000002e-06 2024-09-21T15:50:07.665 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.665 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.665 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:07.666 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105495+0000", 2024-09-21T15:50:07.666 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056682679999999999 2024-09-21T15:50:07.666 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.666 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.666 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:07.666 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105565+0000", 2024-09-21T15:50:07.666 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9868000000000003e-05 2024-09-21T15:50:07.666 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.666 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.666 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:07.666 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105625+0000", 2024-09-21T15:50:07.666 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0268e-05 2024-09-21T15:50:07.666 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:07.666 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:07.667 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:07.667 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.667 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.667 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116236 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 768000~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:07.667 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048814+0000", 2024-09-21T15:50:07.667 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 11.604569201, 2024-09-21T15:50:07.667 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056959587999999998, 2024-09-21T15:50:07.667 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:07.667 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:07.667 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:07.667 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:07.667 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:07.667 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116236 2024-09-21T15:50:07.667 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.668 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:07.668 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.668 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:07.668 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048814+0000", 2024-09-21T15:50:07.668 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:07.668 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.668 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.668 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:07.668 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048814+0000", 2024-09-21T15:50:07.668 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:07.668 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.668 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.668 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:07.668 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048814+0000", 2024-09-21T15:50:07.668 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:07.668 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.669 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.669 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:07.669 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048815+0000", 2024-09-21T15:50:07.669 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7999999999999993e-07 2024-09-21T15:50:07.669 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.669 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.669 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:07.669 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048817+0000", 2024-09-21T15:50:07.669 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6620000000000001e-06 2024-09-21T15:50:07.669 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.669 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.669 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:07.669 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048820+0000", 2024-09-21T15:50:07.669 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2800000000000002e-06 2024-09-21T15:50:07.669 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.669 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.670 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:07.670 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105647+0000", 2024-09-21T15:50:07.670 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056827253000000001 2024-09-21T15:50:07.670 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.670 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.670 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:07.670 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105716+0000", 2024-09-21T15:50:07.670 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8761000000000001e-05 2024-09-21T15:50:07.670 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.670 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.670 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:07.670 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105774+0000", 2024-09-21T15:50:07.670 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7855000000000001e-05 2024-09-21T15:50:07.670 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:07.670 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:07.671 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:07.671 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.671 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.671 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116237 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 774144~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:07.671 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048822+0000", 2024-09-21T15:50:07.671 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 11.604561516, 2024-09-21T15:50:07.671 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057114028999999997, 2024-09-21T15:50:07.671 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:07.671 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:07.671 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:07.671 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:07.671 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:07.671 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116237 2024-09-21T15:50:07.671 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.671 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:07.671 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.672 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:07.672 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048822+0000", 2024-09-21T15:50:07.672 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:07.672 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.672 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.672 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:07.672 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048822+0000", 2024-09-21T15:50:07.672 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:07.672 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.672 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.672 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:07.672 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048822+0000", 2024-09-21T15:50:07.672 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:07.672 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.672 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.673 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:07.673 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048823+0000", 2024-09-21T15:50:07.673 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9699999999999994e-07 2024-09-21T15:50:07.673 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.673 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.673 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:07.673 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048825+0000", 2024-09-21T15:50:07.673 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.65e-06 2024-09-21T15:50:07.673 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.673 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.673 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:07.673 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048828+0000", 2024-09-21T15:50:07.673 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8890000000000002e-06 2024-09-21T15:50:07.673 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.674 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.674 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:07.674 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105802+0000", 2024-09-21T15:50:07.674 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056974415 2024-09-21T15:50:07.674 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.674 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.674 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:07.674 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105871+0000", 2024-09-21T15:50:07.674 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8727999999999997e-05 2024-09-21T15:50:07.674 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.674 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.674 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:07.674 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105936+0000", 2024-09-21T15:50:07.674 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4554999999999994e-05 2024-09-21T15:50:07.674 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:07.674 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:07.675 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:07.675 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.675 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.675 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116238 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 780288~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:07.675 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048830+0000", 2024-09-21T15:50:07.675 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 11.604553256999999, 2024-09-21T15:50:07.675 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057296574000000003, 2024-09-21T15:50:07.675 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:07.675 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:07.675 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:07.675 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:07.675 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:07.675 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116238 2024-09-21T15:50:07.676 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.676 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:07.676 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.676 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:07.676 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048830+0000", 2024-09-21T15:50:07.676 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:07.676 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.676 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.676 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:07.676 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048830+0000", 2024-09-21T15:50:07.676 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:07.676 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.676 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.676 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:07.677 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048830+0000", 2024-09-21T15:50:07.677 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:07.677 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.677 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.677 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:07.677 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048831+0000", 2024-09-21T15:50:07.677 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0160000000000001e-06 2024-09-21T15:50:07.677 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.677 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.677 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:07.677 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048834+0000", 2024-09-21T15:50:07.677 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.813e-06 2024-09-21T15:50:07.677 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.677 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.677 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:07.677 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048836+0000", 2024-09-21T15:50:07.677 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6790000000000001e-06 2024-09-21T15:50:07.678 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.678 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.678 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:07.678 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105961+0000", 2024-09-21T15:50:07.678 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057124705999999997 2024-09-21T15:50:07.678 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.678 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.678 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:07.678 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106042+0000", 2024-09-21T15:50:07.678 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0820000000000002e-05 2024-09-21T15:50:07.678 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.678 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.678 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:07.678 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106127+0000", 2024-09-21T15:50:07.679 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4746000000000005e-05 2024-09-21T15:50:07.679 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:07.679 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:07.679 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:07.679 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.679 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.679 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116324 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 524288~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:07.679 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048838+0000", 2024-09-21T15:50:07.679 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 11.604544948000001, 2024-09-21T15:50:07.679 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057477185, 2024-09-21T15:50:07.679 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:07.679 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:07.679 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:07.679 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:07.679 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:07.680 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116324 2024-09-21T15:50:07.680 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.680 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:07.680 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.680 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:07.680 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048838+0000", 2024-09-21T15:50:07.680 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:07.680 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.680 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.680 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:07.680 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048838+0000", 2024-09-21T15:50:07.680 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:07.680 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.680 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.680 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:07.680 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048838+0000", 2024-09-21T15:50:07.681 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:07.681 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.681 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.681 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:07.681 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048842+0000", 2024-09-21T15:50:07.681 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4829999999999999e-06 2024-09-21T15:50:07.681 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.681 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.681 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:07.681 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048846+0000", 2024-09-21T15:50:07.681 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0470000000000004e-06 2024-09-21T15:50:07.681 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.681 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.681 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:07.682 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048849+0000", 2024-09-21T15:50:07.682 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3689999999999998e-06 2024-09-21T15:50:07.682 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.682 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.682 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:07.682 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106157+0000", 2024-09-21T15:50:07.682 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057307942000000001 2024-09-21T15:50:07.682 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.682 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.682 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:07.682 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106237+0000", 2024-09-21T15:50:07.682 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0320000000000003e-05 2024-09-21T15:50:07.682 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.682 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.683 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:07.683 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106315+0000", 2024-09-21T15:50:07.683 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8227999999999997e-05 2024-09-21T15:50:07.683 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:07.683 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:07.683 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:07.683 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.683 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.683 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116325 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 530432~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:07.683 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048851+0000", 2024-09-21T15:50:07.683 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 11.604531776, 2024-09-21T15:50:07.683 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057581036000000002, 2024-09-21T15:50:07.683 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:07.683 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:07.684 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:07.684 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:07.684 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:07.684 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116325 2024-09-21T15:50:07.684 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.684 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:07.684 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.684 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:07.684 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048851+0000", 2024-09-21T15:50:07.684 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:07.684 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.684 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.684 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:07.685 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048851+0000", 2024-09-21T15:50:07.685 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:07.685 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.685 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.685 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:07.685 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048851+0000", 2024-09-21T15:50:07.685 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:07.685 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.685 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.685 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:07.685 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048852+0000", 2024-09-21T15:50:07.685 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.077e-06 2024-09-21T15:50:07.685 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.685 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.685 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:07.686 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048855+0000", 2024-09-21T15:50:07.686 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9859999999999999e-06 2024-09-21T15:50:07.686 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.686 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.686 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:07.686 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048858+0000", 2024-09-21T15:50:07.686 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3810000000000002e-06 2024-09-21T15:50:07.686 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.686 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.686 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:07.686 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106335+0000", 2024-09-21T15:50:07.686 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057477473000000001 2024-09-21T15:50:07.686 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.687 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.687 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:07.687 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106393+0000", 2024-09-21T15:50:07.687 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8050000000000002e-05 2024-09-21T15:50:07.687 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.687 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.687 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:07.687 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106432+0000", 2024-09-21T15:50:07.687 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9274999999999999e-05 2024-09-21T15:50:07.687 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:07.687 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:07.687 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:07.687 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.687 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.687 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116326 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 536576~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:07.687 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048860+0000", 2024-09-21T15:50:07.688 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 11.604523725, 2024-09-21T15:50:07.688 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057774197999999999, 2024-09-21T15:50:07.688 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:07.688 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:07.688 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:07.688 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:07.688 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:07.688 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116326 2024-09-21T15:50:07.688 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.688 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:07.688 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.688 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:07.688 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048860+0000", 2024-09-21T15:50:07.688 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:07.689 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.689 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.689 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:07.689 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048860+0000", 2024-09-21T15:50:07.689 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:07.689 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.689 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.689 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:07.689 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048859+0000", 2024-09-21T15:50:07.689 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:07.689 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.689 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.689 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:07.689 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048860+0000", 2024-09-21T15:50:07.689 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.02e-06 2024-09-21T15:50:07.690 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.690 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.690 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:07.690 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048863+0000", 2024-09-21T15:50:07.690 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1039999999999998e-06 2024-09-21T15:50:07.690 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.690 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.690 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:07.690 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048867+0000", 2024-09-21T15:50:07.690 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.563e-06 2024-09-21T15:50:07.690 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.690 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.690 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:07.690 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106465+0000", 2024-09-21T15:50:07.691 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057597789000000003 2024-09-21T15:50:07.691 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.691 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.691 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:07.691 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106552+0000", 2024-09-21T15:50:07.691 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7602000000000006e-05 2024-09-21T15:50:07.691 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.691 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.691 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:07.691 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106634+0000", 2024-09-21T15:50:07.691 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1321000000000003e-05 2024-09-21T15:50:07.691 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:07.691 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:07.691 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:07.691 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.692 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.692 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116327 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 542720~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:07.692 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048870+0000", 2024-09-21T15:50:07.692 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 11.604512852999999, 2024-09-21T15:50:07.692 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057883148000000002, 2024-09-21T15:50:07.692 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:07.692 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:07.692 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:07.692 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:07.692 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:07.692 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116327 2024-09-21T15:50:07.692 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.692 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:07.692 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.692 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:07.692 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048870+0000", 2024-09-21T15:50:07.693 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:07.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.693 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.693 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:07.693 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048870+0000", 2024-09-21T15:50:07.693 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:07.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.693 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.693 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:07.693 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048870+0000", 2024-09-21T15:50:07.693 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:07.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.693 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.693 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:07.693 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048872+0000", 2024-09-21T15:50:07.694 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.824e-06 2024-09-21T15:50:07.694 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.694 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.694 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:07.694 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048879+0000", 2024-09-21T15:50:07.694 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6749999999999996e-06 2024-09-21T15:50:07.694 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.694 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.694 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:07.694 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048883+0000", 2024-09-21T15:50:07.694 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2200000000000003e-06 2024-09-21T15:50:07.694 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.694 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.694 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:07.695 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106663+0000", 2024-09-21T15:50:07.695 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057780049 2024-09-21T15:50:07.695 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.695 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.695 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:07.695 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106711+0000", 2024-09-21T15:50:07.695 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8433999999999997e-05 2024-09-21T15:50:07.695 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.695 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.695 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:07.695 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106754+0000", 2024-09-21T15:50:07.695 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2236e-05 2024-09-21T15:50:07.695 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:07.695 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:07.695 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:07.696 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.696 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.696 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116328 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 548864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:07.696 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048887+0000", 2024-09-21T15:50:07.696 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 11.604496333, 2024-09-21T15:50:07.696 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057981984, 2024-09-21T15:50:07.696 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:07.696 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:07.696 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:07.696 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:07.696 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:07.696 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116328 2024-09-21T15:50:07.696 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.696 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:07.696 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.697 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:07.697 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048887+0000", 2024-09-21T15:50:07.697 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:07.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.697 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.697 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:07.697 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048887+0000", 2024-09-21T15:50:07.697 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:07.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.697 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.697 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:07.697 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048887+0000", 2024-09-21T15:50:07.697 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:07.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.697 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.698 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:07.698 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048891+0000", 2024-09-21T15:50:07.698 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.843e-06 2024-09-21T15:50:07.698 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.698 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.698 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:07.698 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048897+0000", 2024-09-21T15:50:07.698 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5419999999999996e-06 2024-09-21T15:50:07.698 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.698 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.698 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:07.698 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048901+0000", 2024-09-21T15:50:07.698 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4689999999999999e-06 2024-09-21T15:50:07.698 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.699 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.699 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:07.699 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106780+0000", 2024-09-21T15:50:07.699 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057878259000000001 2024-09-21T15:50:07.699 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.699 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.699 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:07.699 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106827+0000", 2024-09-21T15:50:07.699 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7098000000000002e-05 2024-09-21T15:50:07.699 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.699 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.699 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:07.699 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106869+0000", 2024-09-21T15:50:07.699 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2129999999999998e-05 2024-09-21T15:50:07.699 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:07.699 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:07.700 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:07.700 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.700 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.700 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116329 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 555008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:07.700 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048905+0000", 2024-09-21T15:50:07.700 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 11.604478222999999, 2024-09-21T15:50:07.700 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058115554999999999, 2024-09-21T15:50:07.700 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:07.700 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:07.700 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:07.700 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:07.700 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:07.700 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116329 2024-09-21T15:50:07.700 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.701 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:07.701 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.701 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:07.701 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048905+0000", 2024-09-21T15:50:07.701 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:07.701 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.701 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.701 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:07.701 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048905+0000", 2024-09-21T15:50:07.701 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:07.701 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.701 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.701 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:07.701 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048905+0000", 2024-09-21T15:50:07.701 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:07.702 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.702 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.702 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:07.702 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048907+0000", 2024-09-21T15:50:07.702 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.086e-06 2024-09-21T15:50:07.702 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.702 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.702 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:07.702 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048912+0000", 2024-09-21T15:50:07.702 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2830000000000001e-06 2024-09-21T15:50:07.702 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.702 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.702 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:07.702 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048916+0000", 2024-09-21T15:50:07.702 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.98e-06 2024-09-21T15:50:07.703 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.703 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.703 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:07.703 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106907+0000", 2024-09-21T15:50:07.703 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057990719000000003 2024-09-21T15:50:07.703 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.703 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.703 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:07.703 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106969+0000", 2024-09-21T15:50:07.703 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1789000000000001e-05 2024-09-21T15:50:07.703 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.703 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.703 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:07.703 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107021+0000", 2024-09-21T15:50:07.703 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2030999999999997e-05 2024-09-21T15:50:07.704 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:07.704 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:07.704 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:07.704 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.704 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.704 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116330 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 561152~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:07.704 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048922+0000", 2024-09-21T15:50:07.704 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 11.604461065000001, 2024-09-21T15:50:07.704 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058242343000000002, 2024-09-21T15:50:07.704 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:07.704 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:07.704 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:07.704 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:07.704 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:07.704 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116330 2024-09-21T15:50:07.705 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.705 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:07.705 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.705 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:07.705 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048922+0000", 2024-09-21T15:50:07.705 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:07.705 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.705 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.705 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:07.705 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048922+0000", 2024-09-21T15:50:07.705 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:07.705 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.705 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.705 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:07.705 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048922+0000", 2024-09-21T15:50:07.706 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:07.706 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.706 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.706 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:07.706 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048924+0000", 2024-09-21T15:50:07.706 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1660000000000001e-06 2024-09-21T15:50:07.706 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.706 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.706 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:07.706 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048930+0000", 2024-09-21T15:50:07.706 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4529999999999999e-06 2024-09-21T15:50:07.706 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.706 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.706 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:07.706 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048934+0000", 2024-09-21T15:50:07.707 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0269999999999997e-06 2024-09-21T15:50:07.707 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.707 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.707 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:07.707 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107047+0000", 2024-09-21T15:50:07.707 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058112913000000002 2024-09-21T15:50:07.707 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.707 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.707 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:07.707 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107095+0000", 2024-09-21T15:50:07.707 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7778000000000002e-05 2024-09-21T15:50:07.707 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.707 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.707 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:07.707 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107165+0000", 2024-09-21T15:50:07.707 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9370999999999997e-05 2024-09-21T15:50:07.708 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:07.708 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:07.708 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:07.708 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.708 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.708 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116331 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 567296~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:07.708 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.049440+0000", 2024-09-21T15:50:07.708 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 11.603942971, 2024-09-21T15:50:07.708 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057878485, 2024-09-21T15:50:07.708 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:07.708 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:07.708 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:07.708 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:07.708 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:07.708 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116331 2024-09-21T15:50:07.708 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.709 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:07.709 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.709 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:07.709 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049440+0000", 2024-09-21T15:50:07.709 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:07.709 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.709 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.709 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:07.709 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049440+0000", 2024-09-21T15:50:07.709 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:07.709 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.709 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.709 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:07.709 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048939+0000", 2024-09-21T15:50:07.709 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9994984 2024-09-21T15:50:07.709 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.710 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.710 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:07.710 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049444+0000", 2024-09-21T15:50:07.710 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00050557200000000003 2024-09-21T15:50:07.710 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.710 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.710 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:07.710 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049460+0000", 2024-09-21T15:50:07.710 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6120000000000002e-05 2024-09-21T15:50:07.710 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.710 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.710 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:07.710 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049468+0000", 2024-09-21T15:50:07.710 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4209999999999998e-06 2024-09-21T15:50:07.710 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.710 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.711 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:07.711 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107189+0000", 2024-09-21T15:50:07.711 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057721019999999998 2024-09-21T15:50:07.711 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.711 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.711 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:07.711 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107254+0000", 2024-09-21T15:50:07.711 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5265000000000006e-05 2024-09-21T15:50:07.711 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.711 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.711 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:07.711 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107319+0000", 2024-09-21T15:50:07.711 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4784999999999997e-05 2024-09-21T15:50:07.711 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:07.711 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:07.711 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:07.712 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.712 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.712 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116332 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 573440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:07.712 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.049474+0000", 2024-09-21T15:50:07.712 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 11.60390971, 2024-09-21T15:50:07.712 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058013809, 2024-09-21T15:50:07.712 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:07.712 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:07.712 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:07.712 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:07.712 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:07.712 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116332 2024-09-21T15:50:07.712 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.712 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:07.712 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.712 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:07.713 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049474+0000", 2024-09-21T15:50:07.713 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:07.713 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.713 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.713 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:07.713 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049474+0000", 2024-09-21T15:50:07.713 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:07.713 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.713 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.713 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:07.713 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049473+0000", 2024-09-21T15:50:07.713 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:07.713 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.713 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.713 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:07.714 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049476+0000", 2024-09-21T15:50:07.714 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6249999999999999e-06 2024-09-21T15:50:07.714 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.714 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.714 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:07.714 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049482+0000", 2024-09-21T15:50:07.714 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0959999999999997e-06 2024-09-21T15:50:07.714 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.714 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.714 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:07.714 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049487+0000", 2024-09-21T15:50:07.714 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5949999999999998e-06 2024-09-21T15:50:07.714 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.714 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.715 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:07.715 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107344+0000", 2024-09-21T15:50:07.715 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057856594999999997 2024-09-21T15:50:07.715 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.715 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.715 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:07.715 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107422+0000", 2024-09-21T15:50:07.715 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7582000000000004e-05 2024-09-21T15:50:07.715 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.715 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.715 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:07.715 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107487+0000", 2024-09-21T15:50:07.715 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5723999999999996e-05 2024-09-21T15:50:07.715 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:07.715 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:07.715 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:07.716 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.716 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.716 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116333 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 579584~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:07.716 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.049993+0000", 2024-09-21T15:50:07.716 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 11.603389927, 2024-09-21T15:50:07.716 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057663537000000001, 2024-09-21T15:50:07.716 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:07.716 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:07.716 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:07.716 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:07.716 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:07.716 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116333 2024-09-21T15:50:07.716 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.716 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:07.716 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.717 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:07.717 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049993+0000", 2024-09-21T15:50:07.717 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:07.717 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.717 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.717 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:07.717 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049993+0000", 2024-09-21T15:50:07.717 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:07.717 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.717 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.717 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:07.717 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049491+0000", 2024-09-21T15:50:07.717 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9994979 2024-09-21T15:50:07.717 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.717 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.717 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:07.718 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049998+0000", 2024-09-21T15:50:07.718 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00050654200000000002 2024-09-21T15:50:07.718 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.718 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.718 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:07.718 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050010+0000", 2024-09-21T15:50:07.718 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1790999999999999e-05 2024-09-21T15:50:07.718 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.718 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.718 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:07.718 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050016+0000", 2024-09-21T15:50:07.718 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2179999999999996e-06 2024-09-21T15:50:07.718 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.718 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.718 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:07.719 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107513+0000", 2024-09-21T15:50:07.719 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057496927000000003 2024-09-21T15:50:07.719 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.719 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.719 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:07.719 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107590+0000", 2024-09-21T15:50:07.719 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7547999999999997e-05 2024-09-21T15:50:07.719 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.719 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.719 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:07.719 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107657+0000", 2024-09-21T15:50:07.719 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6630000000000004e-05 2024-09-21T15:50:07.719 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:07.719 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:07.719 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:07.720 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.720 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.720 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116334 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 585728~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:07.720 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.050091+0000", 2024-09-21T15:50:07.720 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 11.603292299, 2024-09-21T15:50:07.720 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057734355000000001, 2024-09-21T15:50:07.720 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:07.720 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:07.720 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:07.720 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:07.720 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:07.720 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116334 2024-09-21T15:50:07.720 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.720 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:07.721 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.721 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:07.721 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050091+0000", 2024-09-21T15:50:07.721 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:07.721 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.721 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.721 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:07.721 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050091+0000", 2024-09-21T15:50:07.721 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:07.721 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.721 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.721 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:07.721 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050021+0000", 2024-09-21T15:50:07.721 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999304 2024-09-21T15:50:07.721 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.722 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.722 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:07.722 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050095+0000", 2024-09-21T15:50:07.722 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4158000000000004e-05 2024-09-21T15:50:07.722 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.722 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.722 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:07.722 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050103+0000", 2024-09-21T15:50:07.722 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0770000000000002e-06 2024-09-21T15:50:07.722 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.722 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.722 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:07.722 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050108+0000", 2024-09-21T15:50:07.722 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6330000000000001e-06 2024-09-21T15:50:07.722 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.723 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.723 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:07.723 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107681+0000", 2024-09-21T15:50:07.723 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057572996000000001 2024-09-21T15:50:07.723 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.723 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.723 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:07.723 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107759+0000", 2024-09-21T15:50:07.723 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7368999999999997e-05 2024-09-21T15:50:07.723 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.723 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.723 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:07.723 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107825+0000", 2024-09-21T15:50:07.723 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6783000000000001e-05 2024-09-21T15:50:07.724 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:07.724 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:07.724 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:07.724 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.724 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.724 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116335 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 591872~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:07.724 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.050542+0000", 2024-09-21T15:50:07.724 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 11.602841714, 2024-09-21T15:50:07.724 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057466455999999999, 2024-09-21T15:50:07.724 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:07.724 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:07.724 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:07.724 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:07.724 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:07.725 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116335 2024-09-21T15:50:07.725 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.725 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:07.725 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.725 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:07.725 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050542+0000", 2024-09-21T15:50:07.725 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:07.725 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.725 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.725 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:07.725 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050542+0000", 2024-09-21T15:50:07.725 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:07.725 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.725 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.725 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:07.726 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050113+0000", 2024-09-21T15:50:07.726 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9995713 2024-09-21T15:50:07.726 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.726 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.726 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:07.726 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050546+0000", 2024-09-21T15:50:07.726 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00043328699999999997 2024-09-21T15:50:07.726 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.726 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.726 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:07.726 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050557+0000", 2024-09-21T15:50:07.726 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1011e-05 2024-09-21T15:50:07.726 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.726 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.727 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:07.727 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050564+0000", 2024-09-21T15:50:07.727 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1130000000000003e-06 2024-09-21T15:50:07.727 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.727 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.727 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:07.727 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107852+0000", 2024-09-21T15:50:07.727 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057287350000000001 2024-09-21T15:50:07.727 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.727 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.727 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:07.727 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107928+0000", 2024-09-21T15:50:07.727 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6668000000000005e-05 2024-09-21T15:50:07.727 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.728 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.728 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:07.728 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108008+0000", 2024-09-21T15:50:07.728 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9783999999999994e-05 2024-09-21T15:50:07.728 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:07.728 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:07.728 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:07.728 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.728 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.728 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116336 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 598016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:07.728 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.050834+0000", 2024-09-21T15:50:07.728 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 11.602549448, 2024-09-21T15:50:07.728 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057353572999999998, 2024-09-21T15:50:07.728 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:07.728 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:07.728 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:07.729 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:07.729 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:07.729 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116336 2024-09-21T15:50:07.729 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.729 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:07.729 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.729 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:07.729 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050834+0000", 2024-09-21T15:50:07.729 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:07.729 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.729 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.729 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:07.729 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050834+0000", 2024-09-21T15:50:07.729 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:07.729 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.730 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.730 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:07.730 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050571+0000", 2024-09-21T15:50:07.730 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9997373 2024-09-21T15:50:07.730 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.730 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.730 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:07.730 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050842+0000", 2024-09-21T15:50:07.730 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00027029099999999998 2024-09-21T15:50:07.730 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.730 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.730 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:07.730 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050853+0000", 2024-09-21T15:50:07.730 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1532000000000001e-05 2024-09-21T15:50:07.730 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.731 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.731 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:07.731 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050863+0000", 2024-09-21T15:50:07.731 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4239999999999999e-06 2024-09-21T15:50:07.731 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.731 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.731 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:07.731 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108045+0000", 2024-09-21T15:50:07.731 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057182415 2024-09-21T15:50:07.731 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.731 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.731 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:07.731 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108123+0000", 2024-09-21T15:50:07.731 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8275000000000005e-05 2024-09-21T15:50:07.731 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.732 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.732 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:07.732 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108187+0000", 2024-09-21T15:50:07.732 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4152000000000005e-05 2024-09-21T15:50:07.732 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:07.732 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:07.732 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:07.732 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.732 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.732 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116337 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 604160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:07.732 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.051192+0000", 2024-09-21T15:50:07.732 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 11.602190820000001, 2024-09-21T15:50:07.732 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057175981000000001, 2024-09-21T15:50:07.732 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:07.732 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:07.732 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:07.733 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:07.733 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:07.733 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116337 2024-09-21T15:50:07.733 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.733 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:07.733 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.733 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:07.733 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051192+0000", 2024-09-21T15:50:07.733 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:07.733 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.733 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.733 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:07.733 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051192+0000", 2024-09-21T15:50:07.733 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:07.734 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.734 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.734 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:07.734 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050870+0000", 2024-09-21T15:50:07.734 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9996777 2024-09-21T15:50:07.734 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.734 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.734 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:07.734 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051197+0000", 2024-09-21T15:50:07.734 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00032656299999999998 2024-09-21T15:50:07.734 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.734 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.734 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:07.734 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051209+0000", 2024-09-21T15:50:07.735 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2636000000000001e-05 2024-09-21T15:50:07.735 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.735 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.735 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:07.735 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051217+0000", 2024-09-21T15:50:07.735 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5989999999999996e-06 2024-09-21T15:50:07.735 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.735 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.735 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:07.735 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108224+0000", 2024-09-21T15:50:07.735 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057007622000000001 2024-09-21T15:50:07.735 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.735 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.735 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:07.735 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108299+0000", 2024-09-21T15:50:07.736 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5099999999999996e-05 2024-09-21T15:50:07.736 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.736 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.736 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:07.736 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108368+0000", 2024-09-21T15:50:07.736 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8918000000000007e-05 2024-09-21T15:50:07.736 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:07.736 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:07.736 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:07.736 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.736 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.736 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116338 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 610304~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:07.736 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.051583+0000", 2024-09-21T15:50:07.736 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 11.601800366999999, 2024-09-21T15:50:07.736 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056967716000000002, 2024-09-21T15:50:07.737 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:07.737 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:07.737 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:07.737 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:07.737 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:07.737 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116338 2024-09-21T15:50:07.737 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.737 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:07.737 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.737 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:07.737 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051583+0000", 2024-09-21T15:50:07.737 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:07.737 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.738 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.738 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:07.738 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051583+0000", 2024-09-21T15:50:07.738 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:07.738 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.738 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.738 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:07.738 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051223+0000", 2024-09-21T15:50:07.738 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.99964 2024-09-21T15:50:07.738 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.738 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.738 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:07.739 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051587+0000", 2024-09-21T15:50:07.739 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00036352999999999998 2024-09-21T15:50:07.739 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.739 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.739 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:07.739 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051597+0000", 2024-09-21T15:50:07.739 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.893e-06 2024-09-21T15:50:07.739 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.739 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.739 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:07.739 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051603+0000", 2024-09-21T15:50:07.739 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0519999999999998e-06 2024-09-21T15:50:07.739 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.740 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.740 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:07.740 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108394+0000", 2024-09-21T15:50:07.740 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056791531999999999 2024-09-21T15:50:07.740 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.740 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.740 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:07.740 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108480+0000", 2024-09-21T15:50:07.740 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.577e-05 2024-09-21T15:50:07.740 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:07.740 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:07.740 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:07.740 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108551+0000", 2024-09-21T15:50:07.741 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0722000000000005e-05 2024-09-21T15:50:07.741 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:07.741 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:07.741 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:07.741 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:07.741 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:07.741 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:07.741 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:50:07.742 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:50:07.743 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:50:07.743 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T15:50:07.744 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T15:50:07.787 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:07.786+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:07.847 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:07.847 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:07.847 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:50:07.847 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:07.848 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:07.847+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:07.855 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:50:07.949 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:07.948+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:07.996 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:07.996 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:07.996 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:50:07.997 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:50:07.997 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:08.006 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:50:08.049 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -f /var/log/ceph/exp.2.4.0 2024-09-21T15:50:08.050 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:08.049+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:08.151 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:08.151 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:50:08.151 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:50:08.151 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:50:08.151 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.151 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:125291 3.1b 3:d87db7c4:::rbd_data.1133545d3fa6.0000000000000013:1e [read 884736~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:50:08.151 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:05.145761+0000", 2024-09-21T15:50:08.151 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 3.003151055, 2024-09-21T15:50:08.151 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045764303999999999, 2024-09-21T15:50:08.151 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:08.152 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:08.152 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:08.152 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:08.152 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:08.152 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 125291 2024-09-21T15:50:08.152 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.152 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:08.152 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.152 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:08.152 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.145761+0000", 2024-09-21T15:50:08.152 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.152 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.152 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.152 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:08.152 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.145761+0000", 2024-09-21T15:50:08.153 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.153 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.153 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.153 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:08.153 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.145761+0000", 2024-09-21T15:50:08.153 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:08.153 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.153 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.153 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:08.153 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.145762+0000", 2024-09-21T15:50:08.153 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1799999999999999e-06 2024-09-21T15:50:08.153 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.153 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.153 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:08.154 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.145765+0000", 2024-09-21T15:50:08.154 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7630000000000001e-06 2024-09-21T15:50:08.154 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.154 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.154 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:08.154 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.145767+0000", 2024-09-21T15:50:08.154 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4559999999999999e-06 2024-09-21T15:50:08.154 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.154 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.154 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:08.154 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.191362+0000", 2024-09-21T15:50:08.154 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045594329000000003 2024-09-21T15:50:08.154 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.154 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.154 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:08.155 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.191429+0000", 2024-09-21T15:50:08.155 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7185000000000001e-05 2024-09-21T15:50:08.155 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.155 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.155 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:08.155 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.191525+0000", 2024-09-21T15:50:08.155 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.6631000000000004e-05 2024-09-21T15:50:08.155 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.155 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:08.155 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.155 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.155 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.155 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:125292 3.1b 3:d87db7c4:::rbd_data.1133545d3fa6.0000000000000013:1e [read 905216~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:50:08.155 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:05.145790+0000", 2024-09-21T15:50:08.155 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 3.0031219060000001, 2024-09-21T15:50:08.156 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045873948999999997, 2024-09-21T15:50:08.156 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:08.156 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:08.156 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:08.156 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:08.156 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:08.156 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 125292 2024-09-21T15:50:08.156 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.156 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:08.156 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.156 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:08.156 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.145790+0000", 2024-09-21T15:50:08.156 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.156 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.156 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.157 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:08.157 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.145790+0000", 2024-09-21T15:50:08.157 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.157 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.157 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.157 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:08.157 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.145790+0000", 2024-09-21T15:50:08.157 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:08.157 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.157 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.157 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:08.157 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.145792+0000", 2024-09-21T15:50:08.157 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7549999999999999e-06 2024-09-21T15:50:08.157 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.157 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.158 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:08.158 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.145795+0000", 2024-09-21T15:50:08.158 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1319999999999998e-06 2024-09-21T15:50:08.158 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.158 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.158 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:08.158 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.145797+0000", 2024-09-21T15:50:08.158 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5739999999999998e-06 2024-09-21T15:50:08.158 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.158 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.158 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:08.158 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.191555+0000", 2024-09-21T15:50:08.158 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045757412999999997 2024-09-21T15:50:08.158 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.158 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.159 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:08.159 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.191603+0000", 2024-09-21T15:50:08.159 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8368000000000002e-05 2024-09-21T15:50:08.159 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.159 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.159 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:08.159 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.191664+0000", 2024-09-21T15:50:08.159 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1104000000000003e-05 2024-09-21T15:50:08.159 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.159 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:08.159 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.159 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.159 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.159 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:125293 3.1b 3:d87db7c4:::rbd_data.1133545d3fa6.0000000000000013:1e [read 921600~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:50:08.159 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:05.145800+0000", 2024-09-21T15:50:08.160 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 3.0031126829999999, 2024-09-21T15:50:08.160 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045990704, 2024-09-21T15:50:08.160 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:08.160 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:08.160 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:08.160 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:08.160 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:08.160 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 125293 2024-09-21T15:50:08.160 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.160 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:08.160 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.160 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:08.160 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.145800+0000", 2024-09-21T15:50:08.160 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.160 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.161 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.161 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:08.161 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.145800+0000", 2024-09-21T15:50:08.161 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.161 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.161 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.161 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:08.161 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.145799+0000", 2024-09-21T15:50:08.161 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:08.161 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.161 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.161 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:08.161 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.145800+0000", 2024-09-21T15:50:08.161 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2300000000000001e-06 2024-09-21T15:50:08.161 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.162 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.162 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:08.162 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.145803+0000", 2024-09-21T15:50:08.162 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.712e-06 2024-09-21T15:50:08.162 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.162 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.162 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:08.162 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.145806+0000", 2024-09-21T15:50:08.162 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4360000000000001e-06 2024-09-21T15:50:08.162 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.162 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.162 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:08.162 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.191694+0000", 2024-09-21T15:50:08.162 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045888311000000001 2024-09-21T15:50:08.162 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.163 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.163 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:08.163 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.191736+0000", 2024-09-21T15:50:08.163 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1959000000000003e-05 2024-09-21T15:50:08.163 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.163 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.163 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:08.163 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.191790+0000", 2024-09-21T15:50:08.163 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4348999999999998e-05 2024-09-21T15:50:08.163 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.163 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:08.163 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.163 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.163 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.164 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:125294 3.1b 3:d87db7c4:::rbd_data.1133545d3fa6.0000000000000013:1e [read 942080~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:50:08.164 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:05.145808+0000", 2024-09-21T15:50:08.164 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 3.0031046429999999, 2024-09-21T15:50:08.164 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046102812, 2024-09-21T15:50:08.164 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:08.164 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:08.164 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:08.164 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:08.164 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:08.164 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 125294 2024-09-21T15:50:08.164 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.164 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:08.164 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.164 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:08.164 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.145808+0000", 2024-09-21T15:50:08.165 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.165 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.165 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.165 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:08.165 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.145808+0000", 2024-09-21T15:50:08.165 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.165 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.165 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.165 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:08.165 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.145807+0000", 2024-09-21T15:50:08.165 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:50:08.165 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.165 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.165 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:08.165 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.145809+0000", 2024-09-21T15:50:08.166 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1909999999999999e-06 2024-09-21T15:50:08.166 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.166 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.166 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:08.166 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.145811+0000", 2024-09-21T15:50:08.166 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7580000000000001e-06 2024-09-21T15:50:08.166 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.166 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.166 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:08.166 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.145814+0000", 2024-09-21T15:50:08.166 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4360000000000001e-06 2024-09-21T15:50:08.166 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.166 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.166 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:08.167 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.191803+0000", 2024-09-21T15:50:08.167 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045989533999999999 2024-09-21T15:50:08.167 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.167 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.167 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:08.167 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.191839+0000", 2024-09-21T15:50:08.167 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5441000000000002e-05 2024-09-21T15:50:08.167 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.167 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.167 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:08.167 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.191910+0000", 2024-09-21T15:50:08.167 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1675999999999996e-05 2024-09-21T15:50:08.167 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.168 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:08.168 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.168 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.168 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.168 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:125295 3.1b 3:d87db7c4:::rbd_data.1133545d3fa6.0000000000000013:1e [read 962560~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:50:08.168 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:05.146015+0000", 2024-09-21T15:50:08.168 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 3.002897344, 2024-09-21T15:50:08.168 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046083249999999999, 2024-09-21T15:50:08.168 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:08.168 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:08.168 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:08.168 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:08.168 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:08.168 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 125295 2024-09-21T15:50:08.168 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.169 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:08.169 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.169 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:08.169 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.146015+0000", 2024-09-21T15:50:08.169 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.169 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.169 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.169 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:08.169 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.146015+0000", 2024-09-21T15:50:08.169 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.169 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.169 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.169 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:08.169 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.145815+0000", 2024-09-21T15:50:08.169 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998007 2024-09-21T15:50:08.170 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.170 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.170 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:08.170 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.146016+0000", 2024-09-21T15:50:08.170 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00020100000000000001 2024-09-21T15:50:08.170 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.170 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.170 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:08.170 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.146021+0000", 2024-09-21T15:50:08.170 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1230000000000002e-06 2024-09-21T15:50:08.170 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.170 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.170 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:08.170 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.146024+0000", 2024-09-21T15:50:08.171 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8949999999999998e-06 2024-09-21T15:50:08.171 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.171 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.171 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:08.171 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.191931+0000", 2024-09-21T15:50:08.171 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045907798 2024-09-21T15:50:08.171 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.171 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.171 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:08.171 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.191998+0000", 2024-09-21T15:50:08.171 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6774999999999997e-05 2024-09-21T15:50:08.171 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.171 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.171 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:08.172 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.192098+0000", 2024-09-21T15:50:08.172 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000100058 2024-09-21T15:50:08.172 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.172 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:08.172 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.172 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.172 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.172 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:125382 3.1b 3:d87db7c4:::rbd_data.1133545d3fa6.0000000000000013:1e [read 350208~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:50:08.172 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:05.150211+0000", 2024-09-21T15:50:08.172 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 2.9987016990000002, 2024-09-21T15:50:08.172 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045716650999999997, 2024-09-21T15:50:08.172 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:08.172 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:08.173 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:08.173 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:08.173 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:08.173 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 125382 2024-09-21T15:50:08.173 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.173 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:08.173 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.173 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:08.173 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.150211+0000", 2024-09-21T15:50:08.173 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.173 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.173 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.173 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:08.173 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.150211+0000", 2024-09-21T15:50:08.173 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.173 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.174 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.174 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:08.174 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.150210+0000", 2024-09-21T15:50:08.174 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:08.174 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.174 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.174 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:08.174 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.150212+0000", 2024-09-21T15:50:08.174 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0899999999999999e-06 2024-09-21T15:50:08.174 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.174 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.174 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:08.174 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.150219+0000", 2024-09-21T15:50:08.174 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.939e-06 2024-09-21T15:50:08.175 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.175 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.175 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:08.175 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.150224+0000", 2024-09-21T15:50:08.175 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6399999999999996e-06 2024-09-21T15:50:08.175 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.175 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.175 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:08.175 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.195813+0000", 2024-09-21T15:50:08.175 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045589400000000002 2024-09-21T15:50:08.175 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.175 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.175 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:08.175 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.195857+0000", 2024-09-21T15:50:08.175 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3998999999999997e-05 2024-09-21T15:50:08.176 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.176 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.176 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:08.176 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.195927+0000", 2024-09-21T15:50:08.176 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9888999999999994e-05 2024-09-21T15:50:08.176 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.176 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:08.176 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.176 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.176 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.176 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:125383 3.1b 3:d87db7c4:::rbd_data.1133545d3fa6.0000000000000013:1e [read 370688~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:50:08.176 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:05.150229+0000", 2024-09-21T15:50:08.176 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 2.9986835530000002, 2024-09-21T15:50:08.176 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045962198000000003, 2024-09-21T15:50:08.176 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:08.177 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:08.177 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:08.177 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:08.177 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:08.177 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 125383 2024-09-21T15:50:08.177 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.177 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:08.177 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.177 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:08.177 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.150229+0000", 2024-09-21T15:50:08.177 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.177 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.177 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.177 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:08.177 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.150229+0000", 2024-09-21T15:50:08.178 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.178 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.178 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.178 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:08.178 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.150228+0000", 2024-09-21T15:50:08.178 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:08.178 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.178 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.178 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:08.178 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.150231+0000", 2024-09-21T15:50:08.178 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.357e-06 2024-09-21T15:50:08.178 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.178 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.178 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:08.179 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.150236+0000", 2024-09-21T15:50:08.179 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6810000000000003e-06 2024-09-21T15:50:08.179 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.179 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.179 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:08.179 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.150241+0000", 2024-09-21T15:50:08.179 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7600000000000002e-06 2024-09-21T15:50:08.179 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.179 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.179 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:08.179 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.195960+0000", 2024-09-21T15:50:08.179 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045719013000000003 2024-09-21T15:50:08.179 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.179 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.179 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:08.180 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.196051+0000", 2024-09-21T15:50:08.180 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0822999999999994e-05 2024-09-21T15:50:08.180 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.180 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.180 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:08.180 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.196191+0000", 2024-09-21T15:50:08.180 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013989699999999999 2024-09-21T15:50:08.180 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.180 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:08.180 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.180 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.180 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.180 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:125384 3.1b 3:d87db7c4:::rbd_data.1133545d3fa6.0000000000000013:1e [read 391168~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:50:08.180 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:05.150245+0000", 2024-09-21T15:50:08.181 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 2.998666804, 2024-09-21T15:50:08.181 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046069553999999999, 2024-09-21T15:50:08.181 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:08.181 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:08.181 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:08.181 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:08.181 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:08.181 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 125384 2024-09-21T15:50:08.181 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.181 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:08.181 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.181 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:08.181 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.150245+0000", 2024-09-21T15:50:08.181 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.181 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.181 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.182 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:08.182 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.150245+0000", 2024-09-21T15:50:08.182 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.182 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.182 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.182 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:08.182 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.150245+0000", 2024-09-21T15:50:08.182 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:08.182 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.182 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.182 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:08.182 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.150247+0000", 2024-09-21T15:50:08.182 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3659999999999999e-06 2024-09-21T15:50:08.182 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.183 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.183 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:08.183 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.150253+0000", 2024-09-21T15:50:08.183 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0689999999999998e-06 2024-09-21T15:50:08.183 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.183 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.183 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:08.183 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.150258+0000", 2024-09-21T15:50:08.183 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6859999999999997e-06 2024-09-21T15:50:08.183 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.183 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.183 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:08.183 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.196205+0000", 2024-09-21T15:50:08.183 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045946873999999999 2024-09-21T15:50:08.184 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.184 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.184 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:08.184 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.196246+0000", 2024-09-21T15:50:08.184 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0664999999999998e-05 2024-09-21T15:50:08.184 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.184 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.184 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:08.184 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.196315+0000", 2024-09-21T15:50:08.184 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9395999999999998e-05 2024-09-21T15:50:08.184 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.184 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:08.184 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.184 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.184 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.184 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:125391 3.1b 3:d87db7c4:::rbd_data.1133545d3fa6.0000000000000013:1e [read 526336~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:50:08.185 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:05.151721+0000", 2024-09-21T15:50:08.185 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 2.9971916250000001, 2024-09-21T15:50:08.185 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045674394, 2024-09-21T15:50:08.185 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:08.185 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:08.185 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:08.185 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:08.185 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:08.185 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 125391 2024-09-21T15:50:08.185 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.185 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:08.185 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.185 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:08.185 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.151721+0000", 2024-09-21T15:50:08.185 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.186 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.186 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.186 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:08.186 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.151721+0000", 2024-09-21T15:50:08.186 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.186 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.186 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.186 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:08.186 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.151720+0000", 2024-09-21T15:50:08.186 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:08.186 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.186 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.186 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:08.186 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.151722+0000", 2024-09-21T15:50:08.186 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.255e-06 2024-09-21T15:50:08.186 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.187 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.187 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:08.187 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.151728+0000", 2024-09-21T15:50:08.187 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9070000000000004e-06 2024-09-21T15:50:08.187 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.187 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.187 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:08.187 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.151733+0000", 2024-09-21T15:50:08.187 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5730000000000001e-06 2024-09-21T15:50:08.187 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.187 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.187 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:08.187 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.197290+0000", 2024-09-21T15:50:08.187 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045556685 2024-09-21T15:50:08.187 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.188 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.188 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:08.188 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.197330+0000", 2024-09-21T15:50:08.188 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0352999999999998e-05 2024-09-21T15:50:08.188 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.188 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.188 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:08.188 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.197395+0000", 2024-09-21T15:50:08.188 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5001999999999999e-05 2024-09-21T15:50:08.188 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.188 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:08.188 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.188 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.188 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.189 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:125392 3.1b 3:d87db7c4:::rbd_data.1133545d3fa6.0000000000000013:1e [read 546816~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:50:08.189 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:05.151739+0000", 2024-09-21T15:50:08.189 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 2.9971732869999999, 2024-09-21T15:50:08.189 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045785246000000002, 2024-09-21T15:50:08.189 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:08.189 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:08.189 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:08.189 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:08.189 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:08.189 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 125392 2024-09-21T15:50:08.189 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.189 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:08.189 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.189 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:08.189 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.151739+0000", 2024-09-21T15:50:08.190 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.190 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.190 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.190 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:08.190 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.151739+0000", 2024-09-21T15:50:08.190 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.190 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.190 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.190 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:08.190 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.151739+0000", 2024-09-21T15:50:08.190 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:08.190 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.190 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.190 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:08.191 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.151741+0000", 2024-09-21T15:50:08.191 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.046e-06 2024-09-21T15:50:08.191 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.191 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.191 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:08.191 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.151746+0000", 2024-09-21T15:50:08.191 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6289999999999998e-06 2024-09-21T15:50:08.191 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.191 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.191 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:08.191 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.151751+0000", 2024-09-21T15:50:08.191 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5859999999999998e-06 2024-09-21T15:50:08.191 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.191 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.191 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:08.192 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.197416+0000", 2024-09-21T15:50:08.192 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045665310000000001 2024-09-21T15:50:08.192 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.192 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.192 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:08.192 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.197458+0000", 2024-09-21T15:50:08.192 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1625000000000002e-05 2024-09-21T15:50:08.192 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.192 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.192 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:08.192 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.197524+0000", 2024-09-21T15:50:08.192 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6425000000000002e-05 2024-09-21T15:50:08.192 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.192 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:08.192 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.193 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.193 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.193 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:125393 3.1b 3:d87db7c4:::rbd_data.1133545d3fa6.0000000000000013:1e [read 567296~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:50:08.193 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:05.151755+0000", 2024-09-21T15:50:08.193 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 2.9971576120000001, 2024-09-21T15:50:08.193 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045898429999999997, 2024-09-21T15:50:08.193 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:08.193 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:08.193 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:08.193 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:08.193 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:08.193 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 125393 2024-09-21T15:50:08.193 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.193 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:08.194 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.194 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:08.194 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.151755+0000", 2024-09-21T15:50:08.194 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.194 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.194 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.194 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:08.194 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.151755+0000", 2024-09-21T15:50:08.194 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.194 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.194 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.194 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:08.194 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.151754+0000", 2024-09-21T15:50:08.194 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:08.194 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.195 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.195 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:08.195 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.151756+0000", 2024-09-21T15:50:08.195 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0049999999999999e-06 2024-09-21T15:50:08.195 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.195 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.195 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:08.195 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.151762+0000", 2024-09-21T15:50:08.195 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6049999999999997e-06 2024-09-21T15:50:08.195 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.195 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.195 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:08.195 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.151767+0000", 2024-09-21T15:50:08.195 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5580000000000002e-06 2024-09-21T15:50:08.195 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.195 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.196 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:08.196 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.197545+0000", 2024-09-21T15:50:08.196 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045778555999999998 2024-09-21T15:50:08.196 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.196 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.196 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:08.196 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.197587+0000", 2024-09-21T15:50:08.196 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1863999999999998e-05 2024-09-21T15:50:08.196 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.196 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.196 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:08.196 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.197653+0000", 2024-09-21T15:50:08.196 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6131999999999996e-05 2024-09-21T15:50:08.196 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.196 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:08.197 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.197 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.197 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.197 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:125394 3.1b 3:d87db7c4:::rbd_data.1133545d3fa6.0000000000000013:1e [read 587776~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:50:08.197 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:05.151776+0000", 2024-09-21T15:50:08.197 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 2.9971362089999998, 2024-09-21T15:50:08.197 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046003321999999999, 2024-09-21T15:50:08.197 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:08.197 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:08.197 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:08.197 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:08.197 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:08.197 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 125394 2024-09-21T15:50:08.197 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.197 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:08.198 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.198 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:08.198 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.151776+0000", 2024-09-21T15:50:08.198 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.198 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.198 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.198 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:08.198 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.151776+0000", 2024-09-21T15:50:08.198 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.198 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.198 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.198 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:08.198 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.151775+0000", 2024-09-21T15:50:08.198 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:08.198 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.199 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.199 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:08.199 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.151778+0000", 2024-09-21T15:50:08.199 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5919999999999999e-06 2024-09-21T15:50:08.199 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.199 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.199 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:08.199 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.151784+0000", 2024-09-21T15:50:08.199 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.054e-06 2024-09-21T15:50:08.199 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.199 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.199 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:08.199 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.151789+0000", 2024-09-21T15:50:08.199 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7489999999999998e-06 2024-09-21T15:50:08.199 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.200 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.200 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:08.200 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.197674+0000", 2024-09-21T15:50:08.200 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045885232999999997 2024-09-21T15:50:08.200 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.200 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.200 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:08.200 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.197714+0000", 2024-09-21T15:50:08.200 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0352999999999998e-05 2024-09-21T15:50:08.200 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.200 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.200 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:08.200 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.197779+0000", 2024-09-21T15:50:08.200 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4875999999999993e-05 2024-09-21T15:50:08.200 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.200 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:08.201 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.201 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.201 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.201 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:125395 3.1b 3:d87db7c4:::rbd_data.1133545d3fa6.0000000000000013:1e [read 608256~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:50:08.201 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:05.152015+0000", 2024-09-21T15:50:08.201 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 2.9968974619999997, 2024-09-21T15:50:08.201 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045892923000000002, 2024-09-21T15:50:08.201 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:08.201 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:08.201 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:08.201 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:08.201 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:08.201 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 125395 2024-09-21T15:50:08.201 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.201 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:08.201 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.202 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:08.202 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.152015+0000", 2024-09-21T15:50:08.202 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.202 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.202 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.202 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:08.202 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.152015+0000", 2024-09-21T15:50:08.202 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.202 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.202 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.202 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:08.202 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.151793+0000", 2024-09-21T15:50:08.202 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9997778 2024-09-21T15:50:08.202 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.202 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.203 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:08.203 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.152018+0000", 2024-09-21T15:50:08.203 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00022526100000000001 2024-09-21T15:50:08.203 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.203 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.203 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:08.203 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.152026+0000", 2024-09-21T15:50:08.203 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1869999999999997e-06 2024-09-21T15:50:08.203 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.203 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.203 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:08.203 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.152032+0000", 2024-09-21T15:50:08.203 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3559999999999998e-06 2024-09-21T15:50:08.203 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.203 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.204 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:08.204 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.197802+0000", 2024-09-21T15:50:08.204 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045769481000000001 2024-09-21T15:50:08.204 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.204 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.204 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:08.204 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.197842+0000", 2024-09-21T15:50:08.204 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9836000000000003e-05 2024-09-21T15:50:08.204 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.204 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.204 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:08.204 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.197908+0000", 2024-09-21T15:50:08.204 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5938999999999993e-05 2024-09-21T15:50:08.204 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.204 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:08.205 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.205 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.205 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.205 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:125396 3.1b 3:d87db7c4:::rbd_data.1133545d3fa6.0000000000000013:1e [read 628736~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:50:08.205 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:05.152038+0000", 2024-09-21T15:50:08.205 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 2.9968746949999998, 2024-09-21T15:50:08.205 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046005533000000001, 2024-09-21T15:50:08.205 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:08.205 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:08.205 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:08.205 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:08.205 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:08.205 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 125396 2024-09-21T15:50:08.205 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.205 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:08.206 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.206 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:08.206 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.152038+0000", 2024-09-21T15:50:08.206 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.206 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.206 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.206 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:08.206 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.152038+0000", 2024-09-21T15:50:08.206 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.206 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.206 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.206 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:08.206 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.152037+0000", 2024-09-21T15:50:08.206 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:08.206 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.207 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.207 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:08.207 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.152040+0000", 2024-09-21T15:50:08.207 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5689999999999999e-06 2024-09-21T15:50:08.207 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.207 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.207 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:08.207 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.152046+0000", 2024-09-21T15:50:08.207 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0449999999999997e-06 2024-09-21T15:50:08.207 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.207 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.207 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:08.207 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.152051+0000", 2024-09-21T15:50:08.207 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7529999999999998e-06 2024-09-21T15:50:08.207 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.208 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.208 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:08.208 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.197930+0000", 2024-09-21T15:50:08.208 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045878752000000002 2024-09-21T15:50:08.208 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.208 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.208 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:08.208 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.197970+0000", 2024-09-21T15:50:08.208 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9444999999999999e-05 2024-09-21T15:50:08.208 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.208 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.208 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:08.208 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.198043+0000", 2024-09-21T15:50:08.208 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3386999999999994e-05 2024-09-21T15:50:08.208 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.209 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:08.209 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.209 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.209 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.209 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:125405 3.1b 3:d87db7c4:::rbd_data.1133545d3fa6.0000000000000013:1e [read 825344~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:50:08.209 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:05.154106+0000", 2024-09-21T15:50:08.209 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 2.9948059169999999, 2024-09-21T15:50:08.209 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045656167999999997, 2024-09-21T15:50:08.209 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:08.209 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:08.209 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:08.209 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:08.209 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:08.209 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 125405 2024-09-21T15:50:08.209 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.209 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:08.210 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.210 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:08.210 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154106+0000", 2024-09-21T15:50:08.210 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.210 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.210 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.210 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:08.210 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154106+0000", 2024-09-21T15:50:08.210 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.210 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.210 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.210 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:08.210 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154106+0000", 2024-09-21T15:50:08.210 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:50:08.210 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.211 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.211 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:08.211 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154107+0000", 2024-09-21T15:50:08.211 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2720000000000001e-06 2024-09-21T15:50:08.211 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.211 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.211 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:08.211 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154110+0000", 2024-09-21T15:50:08.211 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7829999999999999e-06 2024-09-21T15:50:08.211 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.211 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.211 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:08.211 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154113+0000", 2024-09-21T15:50:08.211 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6690000000000002e-06 2024-09-21T15:50:08.211 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.212 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.212 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:08.212 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.199540+0000", 2024-09-21T15:50:08.212 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045427109 2024-09-21T15:50:08.212 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.212 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.212 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:08.212 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.199631+0000", 2024-09-21T15:50:08.212 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1279000000000005e-05 2024-09-21T15:50:08.212 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.212 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.212 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:08.212 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.199762+0000", 2024-09-21T15:50:08.212 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013128400000000001 2024-09-21T15:50:08.212 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.213 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:08.213 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.213 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.213 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.213 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:125406 3.1b 3:d87db7c4:::rbd_data.1133545d3fa6.0000000000000013:1e [read 845824~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:50:08.213 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:05.154129+0000", 2024-09-21T15:50:08.213 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 2.9947835729999999, 2024-09-21T15:50:08.213 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045864239000000001, 2024-09-21T15:50:08.213 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:08.213 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:08.213 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:08.213 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:08.213 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:08.213 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 125406 2024-09-21T15:50:08.213 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.213 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:08.214 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.214 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:08.214 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154129+0000", 2024-09-21T15:50:08.214 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.214 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.214 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.214 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:08.214 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154129+0000", 2024-09-21T15:50:08.214 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.214 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.214 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.214 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:08.214 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154128+0000", 2024-09-21T15:50:08.214 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:08.214 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.215 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.215 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:08.215 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154130+0000", 2024-09-21T15:50:08.215 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9e-06 2024-09-21T15:50:08.215 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.215 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.215 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:08.215 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154134+0000", 2024-09-21T15:50:08.215 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5810000000000001e-06 2024-09-21T15:50:08.215 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.215 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.215 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:08.215 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154136+0000", 2024-09-21T15:50:08.215 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5459999999999998e-06 2024-09-21T15:50:08.215 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.216 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.216 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:08.216 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.199793+0000", 2024-09-21T15:50:08.216 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045656799999999997 2024-09-21T15:50:08.216 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.216 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.216 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:08.216 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.199876+0000", 2024-09-21T15:50:08.216 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2730000000000002e-05 2024-09-21T15:50:08.216 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.216 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.216 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:08.216 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.199993+0000", 2024-09-21T15:50:08.216 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000117124 2024-09-21T15:50:08.216 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.217 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:08.217 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.217 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.217 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.217 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:125407 3.1b 3:d87db7c4:::rbd_data.1133545d3fa6.0000000000000013:1e [read 866304~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:50:08.217 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:05.154139+0000", 2024-09-21T15:50:08.217 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 2.994773736, 2024-09-21T15:50:08.217 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046086304000000002, 2024-09-21T15:50:08.217 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:08.217 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:08.217 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:08.217 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:08.217 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:08.217 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 125407 2024-09-21T15:50:08.217 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.218 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:08.218 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.218 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:08.218 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154139+0000", 2024-09-21T15:50:08.218 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.218 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.218 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.218 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:08.218 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154139+0000", 2024-09-21T15:50:08.218 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.218 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.218 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.218 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:08.218 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154138+0000", 2024-09-21T15:50:08.218 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:08.219 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.219 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.219 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:08.219 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154139+0000", 2024-09-21T15:50:08.219 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3120000000000001e-06 2024-09-21T15:50:08.219 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.219 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.219 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:08.219 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154142+0000", 2024-09-21T15:50:08.219 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.796e-06 2024-09-21T15:50:08.219 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.219 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.219 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:08.219 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154145+0000", 2024-09-21T15:50:08.219 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3800000000000001e-06 2024-09-21T15:50:08.220 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.220 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.220 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:08.220 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.200025+0000", 2024-09-21T15:50:08.220 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045880524999999998 2024-09-21T15:50:08.220 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.220 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.220 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:08.220 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.200116+0000", 2024-09-21T15:50:08.220 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0963000000000003e-05 2024-09-21T15:50:08.220 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.220 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.220 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:08.220 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.200225+0000", 2024-09-21T15:50:08.220 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000108646 2024-09-21T15:50:08.221 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.221 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:08.221 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.221 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.221 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.221 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:125408 3.1b 3:d87db7c4:::rbd_data.1133545d3fa6.0000000000000013:1e [read 886784~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:50:08.221 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:05.154148+0000", 2024-09-21T15:50:08.221 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 2.994764677, 2024-09-21T15:50:08.221 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046333535000000002, 2024-09-21T15:50:08.221 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:08.221 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:08.221 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:08.221 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:08.221 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:08.221 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 125408 2024-09-21T15:50:08.222 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.222 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:08.222 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.222 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:08.222 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154148+0000", 2024-09-21T15:50:08.222 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.222 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.222 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.222 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:08.222 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154148+0000", 2024-09-21T15:50:08.222 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.222 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.222 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.222 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:08.222 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154147+0000", 2024-09-21T15:50:08.222 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:50:08.223 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.223 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.223 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:08.223 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154149+0000", 2024-09-21T15:50:08.223 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.19e-06 2024-09-21T15:50:08.223 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.223 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.223 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:08.223 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154151+0000", 2024-09-21T15:50:08.223 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8059999999999999e-06 2024-09-21T15:50:08.223 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.223 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.223 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:08.223 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154154+0000", 2024-09-21T15:50:08.223 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.909e-06 2024-09-21T15:50:08.224 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.224 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.224 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:08.224 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.200270+0000", 2024-09-21T15:50:08.224 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046115505000000001 2024-09-21T15:50:08.224 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.224 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.224 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:08.224 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.200361+0000", 2024-09-21T15:50:08.224 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1457000000000002e-05 2024-09-21T15:50:08.224 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.224 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.224 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:08.224 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.200481+0000", 2024-09-21T15:50:08.224 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000119902 2024-09-21T15:50:08.225 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.225 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:08.225 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.225 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.225 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.225 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:125409 3.1b 3:d87db7c4:::rbd_data.1133545d3fa6.0000000000000013:1e [read 907264~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:50:08.225 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:05.154156+0000", 2024-09-21T15:50:08.225 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 2.9947559670000001, 2024-09-21T15:50:08.225 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046547416000000001, 2024-09-21T15:50:08.225 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:08.225 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:08.225 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:08.225 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:08.225 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:08.225 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 125409 2024-09-21T15:50:08.225 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.226 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:08.226 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.226 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:08.226 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154156+0000", 2024-09-21T15:50:08.226 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.226 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.226 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.226 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:08.226 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154156+0000", 2024-09-21T15:50:08.226 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.226 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.226 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.226 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:08.226 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154156+0000", 2024-09-21T15:50:08.226 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:50:08.227 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.227 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.227 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:08.227 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154157+0000", 2024-09-21T15:50:08.227 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1680000000000001e-06 2024-09-21T15:50:08.227 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.227 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.227 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:08.227 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154160+0000", 2024-09-21T15:50:08.227 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.802e-06 2024-09-21T15:50:08.227 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.227 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.227 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:08.227 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154163+0000", 2024-09-21T15:50:08.227 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.509e-06 2024-09-21T15:50:08.227 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.228 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.228 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:08.228 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.200520+0000", 2024-09-21T15:50:08.228 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046357281 2024-09-21T15:50:08.228 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.228 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.228 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:08.228 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.200597+0000", 2024-09-21T15:50:08.228 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7205000000000004e-05 2024-09-21T15:50:08.228 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.228 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.228 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:08.228 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.200704+0000", 2024-09-21T15:50:08.228 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010668 2024-09-21T15:50:08.228 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.229 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:08.229 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.229 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.229 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.229 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:125416 3.1b 3:d87db7c4:::rbd_data.1133545d3fa6.0000000000000013:1e [read 1046528~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:50:08.229 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:05.156074+0000", 2024-09-21T15:50:08.229 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 2.9928385899999999, 2024-09-21T15:50:08.229 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045646541999999998, 2024-09-21T15:50:08.229 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:08.229 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:08.229 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:08.229 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:08.229 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:08.229 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 125416 2024-09-21T15:50:08.230 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.230 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:08.230 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.230 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:08.230 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.156074+0000", 2024-09-21T15:50:08.230 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.230 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.230 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.230 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:08.230 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.156074+0000", 2024-09-21T15:50:08.230 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.230 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.230 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.230 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:08.230 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.156073+0000", 2024-09-21T15:50:08.231 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:50:08.231 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.231 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.231 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:08.231 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.156075+0000", 2024-09-21T15:50:08.231 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2699999999999999e-06 2024-09-21T15:50:08.231 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.231 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.231 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:08.231 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.156078+0000", 2024-09-21T15:50:08.231 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8389999999999998e-06 2024-09-21T15:50:08.231 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.231 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.231 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:08.231 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.156081+0000", 2024-09-21T15:50:08.232 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4640000000000002e-06 2024-09-21T15:50:08.232 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.232 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.232 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:08.232 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.201622+0000", 2024-09-21T15:50:08.232 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045541352 2024-09-21T15:50:08.232 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.232 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.232 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:08.232 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.201664+0000", 2024-09-21T15:50:08.232 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1848000000000003e-05 2024-09-21T15:50:08.232 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.232 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.232 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:08.233 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.201720+0000", 2024-09-21T15:50:08.233 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5983999999999998e-05 2024-09-21T15:50:08.233 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.233 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:08.233 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.233 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.233 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:08.233 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:08.233 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:08.149+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:08.234 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:50:08.235 INFO:teuthology.orchestra.run.smithi089.stdout:checking clone #28, image image_client.0-clone28 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent29 2024-09-21T15:50:08.252 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:08.251+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:08.255 INFO:teuthology.orchestra.run.smithi089.stdout:4411 write 0xe10ead7 thru 0xe11710f (0x8639 bytes) 2024-09-21T15:50:08.257 INFO:teuthology.orchestra.run.smithi089.stdout:4412 read 0x311d9c8 thru 0x311dbf4 (0x22d bytes) 2024-09-21T15:50:08.258 INFO:teuthology.orchestra.run.smithi089.stdout:4413 read 0x995ec81 thru 0x996ca9d (0xde1d bytes) 2024-09-21T15:50:08.267 INFO:teuthology.orchestra.run.smithi089.stdout:4414 trunc from 0xe117110 to 0x620e6b3 2024-09-21T15:50:08.328 INFO:teuthology.orchestra.run.smithi089.stdout:4416 trunc from 0x620e6b3 to 0xd6c4ad3 2024-09-21T15:50:08.334 INFO:teuthology.orchestra.run.smithi089.stdout:4419 read 0xc1d28be thru 0xc1d5c56 (0x3399 bytes) 2024-09-21T15:50:08.334 INFO:teuthology.orchestra.run.smithi089.stdout:4420 write 0x17279b4 thru 0x172ed7e (0x73cb bytes) 2024-09-21T15:50:08.335 INFO:teuthology.orchestra.run.smithi089.stdout:4421 read 0x560a137 thru 0x560e960 (0x482a bytes) 2024-09-21T15:50:08.339 INFO:teuthology.orchestra.run.smithi089.stdout:4422 trunc from 0xd6c4ad3 to 0xa9905ac 2024-09-21T15:50:08.346 INFO:teuthology.orchestra.run.smithi089.stdout:4423 read 0x5a4aff0 thru 0x5a5251a (0x752b bytes) 2024-09-21T15:50:08.346 INFO:teuthology.orchestra.run.smithi089.stdout:4425 write 0x94edba6 thru 0x94fb568 (0xd9c3 bytes) 2024-09-21T15:50:08.348 INFO:teuthology.orchestra.run.smithi089.stdout:4426 read 0x32e42ab thru 0x32f2417 (0xe16d bytes) 2024-09-21T15:50:08.348 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:08.348 INFO:teuthology.orchestra.run.smithi089.stdout:4430 read 0x9f67f96 thru 0x9f6dd6a (0x5dd5 bytes) 2024-09-21T15:50:08.349 INFO:teuthology.orchestra.run.smithi089.stdout:4431 punch from 0x5c9a063 to 0x5ca2a4a, (0x89e7 bytes) 2024-09-21T15:50:08.349 INFO:teuthology.orchestra.run.smithi089.stdout:4432 write 0x4b201f2 thru 0x4b2bc78 (0xba87 bytes) 2024-09-21T15:50:08.350 INFO:teuthology.orchestra.run.smithi089.stdout:4433 read 0x9649df8 thru 0x9651714 (0x791d bytes) 2024-09-21T15:50:08.351 INFO:teuthology.orchestra.run.smithi089.stdout:4434 write 0x1d72394 thru 0x1d778e1 (0x554e bytes) 2024-09-21T15:50:08.352 INFO:teuthology.orchestra.run.smithi089.stdout:4435 read 0x3ed49f3 thru 0x3ee351f (0xeb2d bytes) 2024-09-21T15:50:08.353 INFO:teuthology.orchestra.run.smithi089.stdout:4436 write 0x142040b thru 0x142e2f2 (0xdee8 bytes) 2024-09-21T15:50:08.353 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:08.352+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:08.355 INFO:teuthology.orchestra.run.smithi089.stdout:4439 writesame 0x7cc48a8 thru 0x7cd2666 data_size 0x1(0xddbf bytes) 2024-09-21T15:50:08.375 INFO:teuthology.orchestra.run.smithi089.stdout:4440 read 0x422b284 thru 0x423412b (0x8ea8 bytes) 2024-09-21T15:50:08.375 INFO:teuthology.orchestra.run.smithi089.stdout:4441 write 0x25db0a8 thru 0x25e8064 (0xcfbd bytes) 2024-09-21T15:50:08.377 INFO:teuthology.orchestra.run.smithi089.stdout:4444 read 0x1f40b2a thru 0x1f4d247 (0xc71e bytes) 2024-09-21T15:50:08.377 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:08.377 INFO:teuthology.orchestra.run.smithi089.stdout:4447 read 0x31d463c thru 0x31e3d84 (0xf749 bytes) 2024-09-21T15:50:08.378 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:08.378 INFO:teuthology.orchestra.run.smithi089.stdout:4449 write 0x756d737 thru 0x75795c8 (0xbe92 bytes) 2024-09-21T15:50:08.385 INFO:teuthology.orchestra.run.smithi089.stdout:4450 write 0xad73ee0 thru 0xad7d416 (0x9537 bytes) 2024-09-21T15:50:08.386 INFO:teuthology.orchestra.run.smithi089.stdout:4451 read 0x3ba293d thru 0x3ba872b (0x5def bytes) 2024-09-21T15:50:08.387 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:08.387 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:08.387 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:50:08.387 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:08.396 INFO:teuthology.orchestra.run.smithi089.stdout:4453 write 0xd98fd47 thru 0xd99def4 (0xe1ae bytes) 2024-09-21T15:50:08.397 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:50:08.402 INFO:teuthology.orchestra.run.smithi089.stdout:4456 read 0xa15a950 thru 0xa165c06 (0xb2b7 bytes) 2024-09-21T15:50:08.402 INFO:teuthology.orchestra.run.smithi089.stdout:4457 read 0xc992f7f thru 0xc99fa3c (0xcabe bytes) 2024-09-21T15:50:08.412 INFO:teuthology.orchestra.run.smithi089.stdout:4461 trunc from 0xd99def5 to 0x4ea72b8 2024-09-21T15:50:08.439 INFO:teuthology.orchestra.run.smithi089.stdout:4463 write 0x659566b thru 0x6598353 (0x2ce9 bytes) 2024-09-21T15:50:08.441 INFO:teuthology.orchestra.run.smithi089.stdout:4464 punch from 0x435b31a to 0x4366aed, (0xb7d3 bytes) 2024-09-21T15:50:08.441 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:08.441 INFO:teuthology.orchestra.run.smithi089.stdout:4467 read 0x10887ab thru 0x10969ce (0xe224 bytes) 2024-09-21T15:50:08.442 INFO:teuthology.orchestra.run.smithi089.stdout:4468 punch from 0x347e03e to 0x34879b0, (0x9972 bytes) 2024-09-21T15:50:08.442 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:08.442 INFO:teuthology.orchestra.run.smithi089.stdout:4470 read 0x5769aa3 thru 0x5776288 (0xc7e6 bytes) 2024-09-21T15:50:08.443 INFO:teuthology.orchestra.run.smithi089.stdout:4471 trunc from 0x6598354 to 0x7859a0f 2024-09-21T15:50:08.453 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:08.452+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:08.455 INFO:teuthology.orchestra.run.smithi089.stdout:4472 trunc from 0x7859a0f to 0xcab220a 2024-09-21T15:50:08.461 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:08.461 INFO:teuthology.orchestra.run.smithi089.stdout:4475 read 0x49d2544 thru 0x49dd624 (0xb0e1 bytes) 2024-09-21T15:50:08.462 INFO:teuthology.orchestra.run.smithi089.stdout:4476 write 0x2864c10 thru 0x286ffe1 (0xb3d2 bytes) 2024-09-21T15:50:08.464 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:08.464 INFO:teuthology.orchestra.run.smithi089.stdout:4480 writesame 0x7b4b086 thru 0x7b53353 data_size 0x3(0x82ce bytes) 2024-09-21T15:50:08.469 INFO:teuthology.orchestra.run.smithi089.stdout:4481 punch from 0xb458bc0 to 0xb45ad62, (0x21a2 bytes) 2024-09-21T15:50:08.469 INFO:teuthology.orchestra.run.smithi089.stdout:4482 read 0x8b106eb thru 0x8b1d300 (0xcc16 bytes) 2024-09-21T15:50:08.469 INFO:teuthology.orchestra.run.smithi089.stdout:4483 write 0x6b15bd0 thru 0x6b16063 (0x494 bytes) 2024-09-21T15:50:08.471 INFO:teuthology.orchestra.run.smithi089.stdout:4484 punch from 0xc11eb94 to 0xc12337d, (0x47e9 bytes) 2024-09-21T15:50:08.471 INFO:teuthology.orchestra.run.smithi089.stdout:4485 read 0x3f80bfb thru 0x3f8db73 (0xcf79 bytes) 2024-09-21T15:50:08.471 INFO:teuthology.orchestra.run.smithi089.stdout:4487 punch from 0x1ccb1d0 to 0x1cd52a2, (0xa0d2 bytes) 2024-09-21T15:50:08.471 INFO:teuthology.orchestra.run.smithi089.stdout:4488 writesame 0x9f8c971 thru 0x9f9c657 data_size 0x7(0xfce7 bytes) 2024-09-21T15:50:08.483 INFO:teuthology.orchestra.run.smithi089.stdout:4489 write 0xdc6d091 thru 0xdc6d6b5 (0x625 bytes) 2024-09-21T15:50:08.488 INFO:teuthology.orchestra.run.smithi089.stdout:4492 punch from 0xc616159 to 0xc625629, (0xf4d0 bytes) 2024-09-21T15:50:08.488 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:08.488 INFO:teuthology.orchestra.run.smithi089.stdout:4494 read 0x84fa892 thru 0x850671d (0xbe8c bytes) 2024-09-21T15:50:08.488 INFO:teuthology.orchestra.run.smithi089.stdout:4495 writesame 0xc2e251c thru 0xc2eb4bb data_size 0x30(0x8fa0 bytes) 2024-09-21T15:50:08.491 INFO:teuthology.orchestra.run.smithi089.stdout:4496 write 0xa187119 thru 0xa1918dd (0xa7c5 bytes) 2024-09-21T15:50:08.504 INFO:teuthology.orchestra.run.smithi089.stdout:4498 trunc from 0xdc6d6b6 to 0x6149050 2024-09-21T15:50:08.505 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:08.505 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:08.506 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:50:08.506 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:50:08.506 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:08.515 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:50:08.525 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:08.525 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:08.525 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:08.525 INFO:teuthology.orchestra.run.smithi089.stdout:4503 read 0x25d7cb6 thru 0x25e15a3 (0x98ee bytes) 2024-09-21T15:50:08.526 INFO:teuthology.orchestra.run.smithi089.stdout:4504 read 0x3940c54 thru 0x3942c00 (0x1fad bytes) 2024-09-21T15:50:08.526 INFO:teuthology.orchestra.run.smithi089.stdout:4505 punch from 0x5840b1c to 0x584d172, (0xc656 bytes) 2024-09-21T15:50:08.526 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:08.526 INFO:teuthology.orchestra.run.smithi089.stdout:4508 punch from 0x5a53457 to 0x5a53549, (0xf2 bytes) 2024-09-21T15:50:08.527 INFO:teuthology.orchestra.run.smithi089.stdout:4509 write 0x503daac thru 0x5049bea (0xc13f bytes) 2024-09-21T15:50:08.542 INFO:teuthology.orchestra.run.smithi089.stdout:4510 writesame 0xd93c1dc thru 0xd942577 data_size 0x7d(0x639c bytes) 2024-09-21T15:50:08.554 INFO:teuthology.orchestra.run.smithi089.stdout:4511 trunc from 0xd942578 to 0x3d59881 2024-09-21T15:50:08.555 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:08.553+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:08.582 INFO:teuthology.orchestra.run.smithi089.stdout:4512 write 0x9a9b48c thru 0x9aa9818 (0xe38d bytes) 2024-09-21T15:50:08.584 INFO:teuthology.orchestra.run.smithi089.stdout:4514 read 0x7b3bfc3 thru 0x7b3c433 (0x471 bytes) 2024-09-21T15:50:08.593 INFO:teuthology.orchestra.run.smithi089.stdout:4516 write 0xc656273 thru 0xc65bacc (0x585a bytes) 2024-09-21T15:50:08.605 INFO:teuthology.orchestra.run.smithi089.stdout:4517 write 0xe693a87 thru 0xe699faf (0x6529 bytes) 2024-09-21T15:50:08.608 INFO:teuthology.orchestra.run.smithi089.stdout:4520 write 0x725fa3e thru 0x72691fc (0x97bf bytes) 2024-09-21T15:50:08.609 INFO:teuthology.orchestra.run.smithi089.stdout:4521 read 0xb03f204 thru 0xb04c317 (0xd114 bytes) 2024-09-21T15:50:08.609 INFO:teuthology.orchestra.run.smithi089.stdout:4522 write 0x8948a88 thru 0x89498e3 (0xe5c bytes) 2024-09-21T15:50:08.611 INFO:teuthology.orchestra.run.smithi089.stdout:4523 write 0x82ff061 thru 0x830a3ef (0xb38f bytes) 2024-09-21T15:50:08.613 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:08.613 INFO:teuthology.orchestra.run.smithi089.stdout:4525 write 0x95df6b2 thru 0x95eac0a (0xb559 bytes) 2024-09-21T15:50:08.614 INFO:teuthology.orchestra.run.smithi089.stdout:4526 writesame 0x40a8157 thru 0x40b1318 data_size 0x1(0x91c2 bytes) 2024-09-21T15:50:08.632 INFO:teuthology.orchestra.run.smithi089.stdout:4527 read 0x2499b5a thru 0x24a47f6 (0xac9d bytes) 2024-09-21T15:50:08.633 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:08.633 INFO:teuthology.orchestra.run.smithi089.stdout:4529 write 0xbc54a0 thru 0xbcce9d (0x79fe bytes) 2024-09-21T15:50:08.635 INFO:teuthology.orchestra.run.smithi089.stdout:4531 write 0x4d3d442 thru 0x4d42a25 (0x55e4 bytes) 2024-09-21T15:50:08.638 INFO:teuthology.orchestra.run.smithi089.stdout:4532 read 0xa9f11ad thru 0xa9f2f7d (0x1dd1 bytes) 2024-09-21T15:50:08.638 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:08.638 INFO:teuthology.orchestra.run.smithi089.stdout:4534 read 0xd81bff9 thru 0xd81e60a (0x2612 bytes) 2024-09-21T15:50:08.638 INFO:teuthology.orchestra.run.smithi089.stdout:4537 read 0xd29ebab thru 0xd2a0543 (0x1999 bytes) 2024-09-21T15:50:08.638 INFO:teuthology.orchestra.run.smithi089.stdout:4538 writesame 0xa53ec6a thru 0xa54d466 data_size 0x1(0xe7fd bytes) 2024-09-21T15:50:08.655 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:08.654+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:08.662 INFO:teuthology.orchestra.run.smithi089.stdout:4539 write 0xab247cf thru 0xab32117 (0xd949 bytes) 2024-09-21T15:50:08.666 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:08.666 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:50:08.666 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:50:08.666 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:50:08.666 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.666 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120111 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 321536~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:08.666 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572216+0000", 2024-09-21T15:50:08.667 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.0922593809999999, 2024-09-21T15:50:08.667 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054657399000000002, 2024-09-21T15:50:08.667 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:08.667 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:08.667 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:08.667 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:08.667 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:08.667 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120111 2024-09-21T15:50:08.667 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.667 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:08.667 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.667 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:08.667 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572216+0000", 2024-09-21T15:50:08.667 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.668 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.668 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.668 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:08.668 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572216+0000", 2024-09-21T15:50:08.668 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.668 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.668 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.668 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:08.668 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572216+0000", 2024-09-21T15:50:08.668 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:08.668 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.668 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.668 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:08.669 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572218+0000", 2024-09-21T15:50:08.669 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1840000000000002e-06 2024-09-21T15:50:08.669 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.669 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.669 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:08.669 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572224+0000", 2024-09-21T15:50:08.669 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7880000000000003e-06 2024-09-21T15:50:08.669 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.669 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.669 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:08.669 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572229+0000", 2024-09-21T15:50:08.669 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8239999999999999e-06 2024-09-21T15:50:08.670 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.670 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.670 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:08.670 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626680+0000", 2024-09-21T15:50:08.670 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054451431000000002 2024-09-21T15:50:08.670 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.670 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.670 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:08.670 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626755+0000", 2024-09-21T15:50:08.670 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4641000000000006e-05 2024-09-21T15:50:08.670 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.670 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.670 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:08.670 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626874+0000", 2024-09-21T15:50:08.671 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00011886 2024-09-21T15:50:08.671 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.671 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:08.671 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.671 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.671 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.671 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120112 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 342016~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:08.672 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572233+0000", 2024-09-21T15:50:08.672 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.092243013, 2024-09-21T15:50:08.672 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054823997999999999, 2024-09-21T15:50:08.672 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:08.672 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:08.672 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:08.672 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:08.673 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:08.673 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120112 2024-09-21T15:50:08.673 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.673 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:08.673 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.673 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:08.673 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572233+0000", 2024-09-21T15:50:08.674 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.674 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.674 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.674 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:08.674 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572233+0000", 2024-09-21T15:50:08.674 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.675 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.675 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.675 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:08.675 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572232+0000", 2024-09-21T15:50:08.675 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:08.675 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.675 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.676 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:08.676 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572234+0000", 2024-09-21T15:50:08.676 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0660000000000002e-06 2024-09-21T15:50:08.676 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.676 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.676 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:08.676 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572242+0000", 2024-09-21T15:50:08.677 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0430000000000002e-06 2024-09-21T15:50:08.677 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.677 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.677 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:08.677 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572246+0000", 2024-09-21T15:50:08.677 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9080000000000003e-06 2024-09-21T15:50:08.677 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.678 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.678 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:08.678 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626916+0000", 2024-09-21T15:50:08.678 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054669482999999998 2024-09-21T15:50:08.678 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.678 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.679 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:08.679 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626977+0000", 2024-09-21T15:50:08.679 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0597000000000003e-05 2024-09-21T15:50:08.679 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.679 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.679 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:08.680 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627057+0000", 2024-09-21T15:50:08.680 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0258000000000003e-05 2024-09-21T15:50:08.680 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.680 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:08.680 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.680 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.680 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.681 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120113 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 362496~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:08.681 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572253+0000", 2024-09-21T15:50:08.681 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.0922230280000003, 2024-09-21T15:50:08.681 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.05503044, 2024-09-21T15:50:08.681 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:08.681 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:08.682 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:08.682 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:08.682 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:08.682 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120113 2024-09-21T15:50:08.682 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.682 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:08.682 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.683 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:08.683 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572253+0000", 2024-09-21T15:50:08.683 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.683 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.683 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.683 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:08.684 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572253+0000", 2024-09-21T15:50:08.684 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.684 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.684 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.684 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:08.684 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572252+0000", 2024-09-21T15:50:08.684 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:08.685 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.685 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.685 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:08.685 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572258+0000", 2024-09-21T15:50:08.685 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6169999999999998e-06 2024-09-21T15:50:08.685 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.686 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.686 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:08.686 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572266+0000", 2024-09-21T15:50:08.686 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9410000000000007e-06 2024-09-21T15:50:08.686 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.686 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.687 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:08.687 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572272+0000", 2024-09-21T15:50:08.687 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.241e-06 2024-09-21T15:50:08.687 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.687 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.687 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:08.687 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627087+0000", 2024-09-21T15:50:08.688 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054814850999999998 2024-09-21T15:50:08.688 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.688 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.688 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:08.688 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627162+0000", 2024-09-21T15:50:08.688 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5012999999999995e-05 2024-09-21T15:50:08.689 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.689 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.689 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:08.689 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627283+0000", 2024-09-21T15:50:08.689 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012113 2024-09-21T15:50:08.689 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.689 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:08.690 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.690 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.690 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.690 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120114 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 382976~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:08.690 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572277+0000", 2024-09-21T15:50:08.690 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.0921990099999999, 2024-09-21T15:50:08.691 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055184395999999997, 2024-09-21T15:50:08.691 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:08.691 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:08.691 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:08.691 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:08.691 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:08.692 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120114 2024-09-21T15:50:08.692 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.692 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:08.692 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.692 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:08.692 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572277+0000", 2024-09-21T15:50:08.692 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.693 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.693 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.693 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:08.693 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572277+0000", 2024-09-21T15:50:08.693 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.693 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.694 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.694 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:08.694 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572276+0000", 2024-09-21T15:50:08.694 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:08.694 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.694 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.695 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:08.695 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572279+0000", 2024-09-21T15:50:08.695 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.373e-06 2024-09-21T15:50:08.695 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.695 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.695 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:08.696 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572285+0000", 2024-09-21T15:50:08.696 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9359999999999996e-06 2024-09-21T15:50:08.696 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.696 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.696 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:08.696 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572289+0000", 2024-09-21T15:50:08.696 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.634e-06 2024-09-21T15:50:08.697 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.697 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.697 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:08.697 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627311+0000", 2024-09-21T15:50:08.697 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055021211 2024-09-21T15:50:08.697 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.697 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.698 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:08.698 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627360+0000", 2024-09-21T15:50:08.698 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9203999999999998e-05 2024-09-21T15:50:08.698 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.698 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.698 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:08.699 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627461+0000", 2024-09-21T15:50:08.699 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000101469 2024-09-21T15:50:08.699 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.699 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:08.699 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.699 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.699 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.700 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120115 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 399360~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:08.700 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572294+0000", 2024-09-21T15:50:08.700 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.0921817029999996, 2024-09-21T15:50:08.700 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055400092999999997, 2024-09-21T15:50:08.700 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:08.700 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:08.701 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:08.701 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:08.701 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:08.701 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120115 2024-09-21T15:50:08.701 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.701 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:08.702 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.702 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:08.702 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572294+0000", 2024-09-21T15:50:08.702 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.702 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.702 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.703 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:08.703 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572294+0000", 2024-09-21T15:50:08.703 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.703 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.703 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.703 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:08.704 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572294+0000", 2024-09-21T15:50:08.704 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:08.704 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.704 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.704 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:08.704 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572296+0000", 2024-09-21T15:50:08.704 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1100000000000001e-06 2024-09-21T15:50:08.705 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.705 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.705 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:08.705 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572303+0000", 2024-09-21T15:50:08.705 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8580000000000002e-06 2024-09-21T15:50:08.705 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.706 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.706 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:08.706 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572314+0000", 2024-09-21T15:50:08.706 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.101e-05 2024-09-21T15:50:08.706 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.706 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.707 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:08.707 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627496+0000", 2024-09-21T15:50:08.707 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055182522999999997 2024-09-21T15:50:08.707 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.707 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.707 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:08.708 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627568+0000", 2024-09-21T15:50:08.708 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1569999999999994e-05 2024-09-21T15:50:08.708 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.708 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.708 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:08.708 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627694+0000", 2024-09-21T15:50:08.708 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000126372 2024-09-21T15:50:08.709 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.709 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:08.709 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.709 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.709 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.709 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120116 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 419840~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:08.710 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572318+0000", 2024-09-21T15:50:08.710 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.092157512, 2024-09-21T15:50:08.710 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055539416000000001, 2024-09-21T15:50:08.710 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:08.710 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:08.710 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:08.711 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:08.711 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:08.711 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120116 2024-09-21T15:50:08.711 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.711 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:08.711 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.712 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:08.712 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572318+0000", 2024-09-21T15:50:08.712 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.712 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.712 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.712 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:08.712 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572318+0000", 2024-09-21T15:50:08.713 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.713 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.713 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.713 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:08.713 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572318+0000", 2024-09-21T15:50:08.713 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:08.714 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.714 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.714 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:08.714 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572322+0000", 2024-09-21T15:50:08.714 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4839999999999998e-06 2024-09-21T15:50:08.714 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.715 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.715 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:08.715 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572329+0000", 2024-09-21T15:50:08.715 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0959999999999997e-06 2024-09-21T15:50:08.715 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.715 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.716 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:08.716 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572333+0000", 2024-09-21T15:50:08.716 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9359999999999998e-06 2024-09-21T15:50:08.716 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.716 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.716 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:08.717 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627726+0000", 2024-09-21T15:50:08.717 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055392829999999997 2024-09-21T15:50:08.717 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.717 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.717 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:08.717 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627779+0000", 2024-09-21T15:50:08.718 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2185999999999998e-05 2024-09-21T15:50:08.718 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.718 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.718 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:08.718 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627858+0000", 2024-09-21T15:50:08.718 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9190999999999994e-05 2024-09-21T15:50:08.718 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.719 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:08.719 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.719 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.719 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.719 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120117 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 440320~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:08.719 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572338+0000", 2024-09-21T15:50:08.719 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.0921381989999999, 2024-09-21T15:50:08.719 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055680888999999997, 2024-09-21T15:50:08.719 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:08.720 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:08.720 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:08.720 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:08.720 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:08.720 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120117 2024-09-21T15:50:08.720 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.720 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:08.720 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.720 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:08.721 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572338+0000", 2024-09-21T15:50:08.721 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.721 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.721 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.721 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:08.721 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572338+0000", 2024-09-21T15:50:08.721 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.721 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.721 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.722 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:08.722 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572337+0000", 2024-09-21T15:50:08.722 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:08.722 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.722 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.722 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:08.722 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572339+0000", 2024-09-21T15:50:08.722 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9379999999999999e-06 2024-09-21T15:50:08.723 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.723 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.723 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:08.723 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572349+0000", 2024-09-21T15:50:08.723 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7780000000000002e-06 2024-09-21T15:50:08.723 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.723 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.723 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:08.723 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572354+0000", 2024-09-21T15:50:08.723 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8740000000000003e-06 2024-09-21T15:50:08.724 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.724 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.724 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:08.724 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627885+0000", 2024-09-21T15:50:08.724 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055531448999999997 2024-09-21T15:50:08.724 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.724 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.724 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:08.724 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627935+0000", 2024-09-21T15:50:08.724 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9348999999999998e-05 2024-09-21T15:50:08.724 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.724 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.724 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:08.725 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628018+0000", 2024-09-21T15:50:08.725 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3851999999999996e-05 2024-09-21T15:50:08.725 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.725 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:08.725 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.725 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.725 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.725 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120118 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 460800~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:08.725 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572359+0000", 2024-09-21T15:50:08.725 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.092116957, 2024-09-21T15:50:08.725 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055951714, 2024-09-21T15:50:08.725 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:08.725 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:08.725 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:08.726 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:08.726 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:08.726 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120118 2024-09-21T15:50:08.726 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.726 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:08.726 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.726 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:08.726 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572359+0000", 2024-09-21T15:50:08.726 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.726 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.726 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.726 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:08.726 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572359+0000", 2024-09-21T15:50:08.727 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.727 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.727 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.727 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:08.727 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572358+0000", 2024-09-21T15:50:08.727 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:08.727 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.727 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.727 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:08.727 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572361+0000", 2024-09-21T15:50:08.727 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.26e-06 2024-09-21T15:50:08.727 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.727 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.727 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:08.727 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572367+0000", 2024-09-21T15:50:08.727 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3640000000000004e-06 2024-09-21T15:50:08.728 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.728 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.728 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:08.728 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572376+0000", 2024-09-21T15:50:08.728 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2310000000000002e-06 2024-09-21T15:50:08.728 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.728 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.728 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:08.728 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628096+0000", 2024-09-21T15:50:08.728 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055719369999999997 2024-09-21T15:50:08.728 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.728 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.728 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:08.728 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628178+0000", 2024-09-21T15:50:08.728 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2667e-05 2024-09-21T15:50:08.729 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.729 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.729 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:08.729 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628311+0000", 2024-09-21T15:50:08.729 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000132175 2024-09-21T15:50:08.729 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.729 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:08.729 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.729 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.729 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.729 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120119 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 481280~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:08.729 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572381+0000", 2024-09-21T15:50:08.729 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.0920945130000002, 2024-09-21T15:50:08.729 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056088791999999998, 2024-09-21T15:50:08.729 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:08.730 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:08.730 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:08.730 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:08.730 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:08.730 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120119 2024-09-21T15:50:08.730 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.730 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:08.730 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.730 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:08.730 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572381+0000", 2024-09-21T15:50:08.730 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.730 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.730 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.730 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:08.730 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572381+0000", 2024-09-21T15:50:08.731 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.731 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.731 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.731 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:08.731 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572381+0000", 2024-09-21T15:50:08.731 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:08.731 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.731 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.731 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:08.731 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572383+0000", 2024-09-21T15:50:08.731 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0619999999999999e-06 2024-09-21T15:50:08.731 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.731 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.731 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:08.731 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572391+0000", 2024-09-21T15:50:08.731 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8769999999999993e-06 2024-09-21T15:50:08.732 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.732 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.732 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:08.732 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572399+0000", 2024-09-21T15:50:08.732 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6240000000000008e-06 2024-09-21T15:50:08.732 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.732 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.732 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:08.732 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628344+0000", 2024-09-21T15:50:08.732 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055944828000000002 2024-09-21T15:50:08.732 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.732 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.732 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:08.732 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628393+0000", 2024-09-21T15:50:08.733 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8653000000000003e-05 2024-09-21T15:50:08.733 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.733 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.733 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:08.733 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628470+0000", 2024-09-21T15:50:08.733 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7121999999999998e-05 2024-09-21T15:50:08.733 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.733 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:08.733 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.733 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.733 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.733 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120120 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 501760~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:08.733 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572418+0000", 2024-09-21T15:50:08.733 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.0920580150000001, 2024-09-21T15:50:08.734 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056209138999999998, 2024-09-21T15:50:08.734 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:08.734 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:08.734 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:08.734 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:08.734 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:08.734 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120120 2024-09-21T15:50:08.734 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.734 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:08.734 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.734 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:08.734 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572418+0000", 2024-09-21T15:50:08.734 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.735 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.735 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.735 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:08.735 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572418+0000", 2024-09-21T15:50:08.735 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.735 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.735 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.735 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:08.735 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572417+0000", 2024-09-21T15:50:08.735 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:08.735 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.735 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.735 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:08.735 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572420+0000", 2024-09-21T15:50:08.736 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5270000000000001e-06 2024-09-21T15:50:08.736 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.736 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.736 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:08.736 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572426+0000", 2024-09-21T15:50:08.736 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3380000000000001e-06 2024-09-21T15:50:08.736 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.736 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.736 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:08.736 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572431+0000", 2024-09-21T15:50:08.736 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.865e-06 2024-09-21T15:50:08.736 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.736 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.736 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:08.737 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628500+0000", 2024-09-21T15:50:08.737 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056068917000000003 2024-09-21T15:50:08.737 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.737 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.737 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:08.737 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628542+0000", 2024-09-21T15:50:08.737 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1761999999999998e-05 2024-09-21T15:50:08.737 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.737 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.737 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:08.737 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628627+0000", 2024-09-21T15:50:08.737 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5247999999999994e-05 2024-09-21T15:50:08.737 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.737 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:08.738 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.738 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.738 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.738 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120121 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 522240~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:08.738 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572630+0000", 2024-09-21T15:50:08.738 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.0918456289999998, 2024-09-21T15:50:08.738 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056207209000000001, 2024-09-21T15:50:08.738 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:08.738 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:08.738 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:08.738 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:08.738 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:08.738 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120121 2024-09-21T15:50:08.738 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.738 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:08.739 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.739 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:08.739 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572630+0000", 2024-09-21T15:50:08.739 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.739 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.739 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.739 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:08.739 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572630+0000", 2024-09-21T15:50:08.739 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.739 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.739 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.739 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:08.739 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572435+0000", 2024-09-21T15:50:08.739 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998045 2024-09-21T15:50:08.739 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.740 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.740 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:08.740 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572633+0000", 2024-09-21T15:50:08.740 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000198786 2024-09-21T15:50:08.740 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.740 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.740 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:08.740 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572641+0000", 2024-09-21T15:50:08.740 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9449999999999993e-06 2024-09-21T15:50:08.740 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.740 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.740 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:08.740 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572648+0000", 2024-09-21T15:50:08.740 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1929999999999998e-06 2024-09-21T15:50:08.740 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.741 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.741 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:08.741 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628669+0000", 2024-09-21T15:50:08.741 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056021690999999998 2024-09-21T15:50:08.741 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.741 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.741 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:08.741 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628751+0000", 2024-09-21T15:50:08.741 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1710000000000002e-05 2024-09-21T15:50:08.741 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.741 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.741 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:08.741 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628837+0000", 2024-09-21T15:50:08.741 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6383000000000003e-05 2024-09-21T15:50:08.742 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.742 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:08.742 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.742 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.742 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.742 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120122 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 538624~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:08.742 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.573471+0000", 2024-09-21T15:50:08.742 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.0910043839999997, 2024-09-21T15:50:08.742 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055563935000000002, 2024-09-21T15:50:08.742 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:08.742 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:08.742 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:08.742 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:08.743 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:08.743 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120122 2024-09-21T15:50:08.743 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.743 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:08.743 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.743 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:08.743 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573471+0000", 2024-09-21T15:50:08.743 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.743 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.743 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.743 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:08.743 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573471+0000", 2024-09-21T15:50:08.743 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.743 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.743 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.744 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:08.744 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572653+0000", 2024-09-21T15:50:08.744 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9991813 2024-09-21T15:50:08.744 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.744 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.744 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:08.744 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573475+0000", 2024-09-21T15:50:08.744 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00082196500000000002 2024-09-21T15:50:08.744 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.744 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.744 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:08.744 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573483+0000", 2024-09-21T15:50:08.744 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5939999999999994e-06 2024-09-21T15:50:08.744 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.744 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.745 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:08.745 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573489+0000", 2024-09-21T15:50:08.745 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1689999999999997e-06 2024-09-21T15:50:08.745 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.745 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.745 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:08.745 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628882+0000", 2024-09-21T15:50:08.745 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055392230000000001 2024-09-21T15:50:08.745 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.745 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.745 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:08.745 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628934+0000", 2024-09-21T15:50:08.745 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1870000000000003e-05 2024-09-21T15:50:08.745 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.745 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.745 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:08.746 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629035+0000", 2024-09-21T15:50:08.746 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000101753 2024-09-21T15:50:08.746 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.746 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:08.746 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.746 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.746 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.746 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120123 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 559104~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:08.746 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.573495+0000", 2024-09-21T15:50:08.746 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.0909806849999999, 2024-09-21T15:50:08.746 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055704765000000003, 2024-09-21T15:50:08.746 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:08.746 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:08.746 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:08.747 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:08.747 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:08.747 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120123 2024-09-21T15:50:08.747 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.747 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:08.747 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.747 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:08.747 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573495+0000", 2024-09-21T15:50:08.747 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.747 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.747 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.747 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:08.747 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573495+0000", 2024-09-21T15:50:08.747 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.747 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.748 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.748 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:08.748 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573495+0000", 2024-09-21T15:50:08.748 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:08.748 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.748 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.748 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:08.748 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573497+0000", 2024-09-21T15:50:08.748 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3420000000000002e-06 2024-09-21T15:50:08.748 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.748 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.748 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:08.748 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573503+0000", 2024-09-21T15:50:08.748 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7030000000000003e-06 2024-09-21T15:50:08.748 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.749 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.749 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:08.749 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573511+0000", 2024-09-21T15:50:08.749 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0790000000000001e-06 2024-09-21T15:50:08.749 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.749 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.749 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:08.749 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629067+0000", 2024-09-21T15:50:08.749 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055556573999999997 2024-09-21T15:50:08.749 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.749 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.749 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:08.749 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629118+0000", 2024-09-21T15:50:08.749 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1053000000000001e-05 2024-09-21T15:50:08.750 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.750 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.750 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:08.750 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629200+0000", 2024-09-21T15:50:08.750 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1452999999999994e-05 2024-09-21T15:50:08.750 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.750 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:08.750 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.750 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.750 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.750 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120124 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 579584~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:08.750 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.574014+0000", 2024-09-21T15:50:08.750 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.0904616459999996, 2024-09-21T15:50:08.750 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055440284999999999, 2024-09-21T15:50:08.750 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:08.751 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:08.751 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:08.751 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:08.751 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:08.751 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120124 2024-09-21T15:50:08.751 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.751 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:08.751 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.751 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:08.751 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574014+0000", 2024-09-21T15:50:08.751 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.751 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.751 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.751 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:08.751 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574014+0000", 2024-09-21T15:50:08.752 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.752 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.752 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.752 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:08.752 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573515+0000", 2024-09-21T15:50:08.752 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9995008 2024-09-21T15:50:08.752 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.752 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.752 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:08.752 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574017+0000", 2024-09-21T15:50:08.752 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00050241100000000003 2024-09-21T15:50:08.752 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.752 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.752 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:08.752 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574027+0000", 2024-09-21T15:50:08.753 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4700000000000008e-06 2024-09-21T15:50:08.753 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.753 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.753 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:08.753 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574033+0000", 2024-09-21T15:50:08.753 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1489999999999999e-06 2024-09-21T15:50:08.753 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.753 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.753 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:08.753 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629238+0000", 2024-09-21T15:50:08.753 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055204880999999997 2024-09-21T15:50:08.753 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.753 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.753 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:08.753 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629317+0000", 2024-09-21T15:50:08.754 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8942000000000004e-05 2024-09-21T15:50:08.754 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.754 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.754 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:08.754 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629454+0000", 2024-09-21T15:50:08.754 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013767099999999999 2024-09-21T15:50:08.754 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.754 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:08.754 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.754 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.754 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.754 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120125 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 600064~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:08.754 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.574482+0000", 2024-09-21T15:50:08.754 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.0899933470000001, 2024-09-21T15:50:08.755 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055243608, 2024-09-21T15:50:08.755 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:08.755 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:08.755 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:08.755 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:08.755 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:08.755 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120125 2024-09-21T15:50:08.755 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.755 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:08.755 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.755 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:08.755 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574482+0000", 2024-09-21T15:50:08.755 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.755 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.755 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.756 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:08.756 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574482+0000", 2024-09-21T15:50:08.756 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.756 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.756 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.756 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:08.756 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574038+0000", 2024-09-21T15:50:08.756 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9995556 2024-09-21T15:50:08.756 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.756 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.756 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:08.756 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574497+0000", 2024-09-21T15:50:08.756 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00045854400000000002 2024-09-21T15:50:08.756 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.756 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.757 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:08.757 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574509+0000", 2024-09-21T15:50:08.757 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2486e-05 2024-09-21T15:50:08.757 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.757 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.757 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:08.757 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574516+0000", 2024-09-21T15:50:08.758 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2269999999999996e-06 2024-09-21T15:50:08.758 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.758 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.758 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:08.758 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629497+0000", 2024-09-21T15:50:08.758 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054980785999999997 2024-09-21T15:50:08.758 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.758 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.758 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:08.758 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629580+0000", 2024-09-21T15:50:08.758 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2528000000000007e-05 2024-09-21T15:50:08.758 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.758 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.759 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:08.759 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629726+0000", 2024-09-21T15:50:08.759 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014630399999999999 2024-09-21T15:50:08.759 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.759 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:08.759 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.759 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.759 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.759 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120126 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 620544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:08.759 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.575315+0000", 2024-09-21T15:50:08.759 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.0891603170000002, 2024-09-21T15:50:08.759 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054709559999999997, 2024-09-21T15:50:08.759 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:08.760 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:08.760 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:08.760 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:08.760 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:08.760 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120126 2024-09-21T15:50:08.760 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.760 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:08.760 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.760 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:08.760 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575315+0000", 2024-09-21T15:50:08.760 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.760 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.760 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.760 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:08.761 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575315+0000", 2024-09-21T15:50:08.761 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.761 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.761 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.761 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:08.761 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574522+0000", 2024-09-21T15:50:08.761 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9992065 2024-09-21T15:50:08.761 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.761 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.761 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:08.761 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575319+0000", 2024-09-21T15:50:08.761 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00079626800000000004 2024-09-21T15:50:08.762 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.762 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.762 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:08.762 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575329+0000", 2024-09-21T15:50:08.762 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.028e-05 2024-09-21T15:50:08.762 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.762 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.762 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:08.762 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575335+0000", 2024-09-21T15:50:08.762 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9290000000000003e-06 2024-09-21T15:50:08.762 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.762 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.762 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:08.762 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629806+0000", 2024-09-21T15:50:08.763 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054471558000000003 2024-09-21T15:50:08.763 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.763 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.763 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:08.763 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629885+0000", 2024-09-21T15:50:08.763 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8563e-05 2024-09-21T15:50:08.763 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.763 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.763 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:08.763 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630025+0000", 2024-09-21T15:50:08.763 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014019 2024-09-21T15:50:08.763 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.763 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:08.764 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.764 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.764 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.764 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120127 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 641024~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:08.764 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.575340+0000", 2024-09-21T15:50:08.764 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.0891356099999996, 2024-09-21T15:50:08.764 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054895589000000002, 2024-09-21T15:50:08.764 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:08.764 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:08.764 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:08.764 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:08.764 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:08.764 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120127 2024-09-21T15:50:08.764 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.765 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:08.765 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.765 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:08.765 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575340+0000", 2024-09-21T15:50:08.765 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.765 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.765 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.765 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:08.765 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575340+0000", 2024-09-21T15:50:08.765 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.765 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.765 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.765 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:08.765 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575340+0000", 2024-09-21T15:50:08.765 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:08.766 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.766 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.766 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:08.766 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575342+0000", 2024-09-21T15:50:08.766 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5600000000000001e-06 2024-09-21T15:50:08.766 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.766 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.766 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:08.766 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575351+0000", 2024-09-21T15:50:08.766 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2349999999999999e-06 2024-09-21T15:50:08.766 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.766 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.766 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:08.767 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575356+0000", 2024-09-21T15:50:08.767 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4600000000000002e-06 2024-09-21T15:50:08.767 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.767 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.767 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:08.767 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630053+0000", 2024-09-21T15:50:08.767 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054697190999999999 2024-09-21T15:50:08.767 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.767 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.767 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:08.767 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630128+0000", 2024-09-21T15:50:08.767 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4777000000000006e-05 2024-09-21T15:50:08.768 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.768 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.768 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:08.768 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630236+0000", 2024-09-21T15:50:08.768 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000107837 2024-09-21T15:50:08.768 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.768 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:08.768 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.768 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.768 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.768 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120128 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 657408~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:08.768 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.575461+0000", 2024-09-21T15:50:08.768 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.0890150309999997, 2024-09-21T15:50:08.768 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054945121999999999, 2024-09-21T15:50:08.769 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:08.769 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:08.769 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:08.769 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:08.769 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:08.769 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120128 2024-09-21T15:50:08.769 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.769 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:08.769 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.769 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:08.769 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575461+0000", 2024-09-21T15:50:08.769 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.769 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.769 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.769 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:08.770 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575461+0000", 2024-09-21T15:50:08.770 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.770 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.770 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.770 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:08.770 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575360+0000", 2024-09-21T15:50:08.770 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998994 2024-09-21T15:50:08.770 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.770 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.770 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:08.770 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575464+0000", 2024-09-21T15:50:08.770 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000103397 2024-09-21T15:50:08.770 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.771 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.771 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:08.771 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575472+0000", 2024-09-21T15:50:08.771 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3189999999999995e-06 2024-09-21T15:50:08.771 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.771 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.771 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:08.771 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575478+0000", 2024-09-21T15:50:08.771 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2779999999999999e-06 2024-09-21T15:50:08.771 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.771 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.771 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:08.771 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630268+0000", 2024-09-21T15:50:08.772 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054789545000000002 2024-09-21T15:50:08.772 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.772 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.772 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:08.772 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630315+0000", 2024-09-21T15:50:08.772 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7244999999999999e-05 2024-09-21T15:50:08.772 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.772 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.772 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:08.772 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630406+0000", 2024-09-21T15:50:08.772 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0811999999999997e-05 2024-09-21T15:50:08.772 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.772 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:08.773 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.773 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.773 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.773 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120129 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 677888~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:08.773 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.576016+0000", 2024-09-21T15:50:08.773 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.0884594869999997, 2024-09-21T15:50:08.773 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054602823000000002, 2024-09-21T15:50:08.773 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:08.773 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:08.773 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:08.773 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:08.773 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:08.773 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120129 2024-09-21T15:50:08.774 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.774 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:08.774 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.774 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:08.774 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576016+0000", 2024-09-21T15:50:08.774 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.774 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.774 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.774 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:08.774 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576016+0000", 2024-09-21T15:50:08.774 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.774 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.774 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.774 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:08.775 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575483+0000", 2024-09-21T15:50:08.775 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9994669 2024-09-21T15:50:08.775 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.775 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.775 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:08.775 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576020+0000", 2024-09-21T15:50:08.775 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00053657900000000003 2024-09-21T15:50:08.775 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.775 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.775 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:08.775 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576030+0000", 2024-09-21T15:50:08.775 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0326000000000001e-05 2024-09-21T15:50:08.775 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.776 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.776 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:08.776 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576037+0000", 2024-09-21T15:50:08.776 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6490000000000002e-06 2024-09-21T15:50:08.776 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.776 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.776 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:08.776 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630446+0000", 2024-09-21T15:50:08.776 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054409698999999999 2024-09-21T15:50:08.776 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.776 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.776 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:08.776 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630524+0000", 2024-09-21T15:50:08.777 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7422999999999994e-05 2024-09-21T15:50:08.777 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.777 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.777 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:08.777 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630619+0000", 2024-09-21T15:50:08.777 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5332000000000001e-05 2024-09-21T15:50:08.777 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.777 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:08.777 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.777 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.777 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.777 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120130 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 698368~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:08.777 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.576043+0000", 2024-09-21T15:50:08.777 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 5.088433116, 2024-09-21T15:50:08.778 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054781798, 2024-09-21T15:50:08.778 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:08.778 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:08.778 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:08.778 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:08.778 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:08.778 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120130 2024-09-21T15:50:08.778 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.778 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:08.778 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.778 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:08.778 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576043+0000", 2024-09-21T15:50:08.778 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.778 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.779 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.779 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:08.779 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576043+0000", 2024-09-21T15:50:08.779 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:08.779 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.779 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.779 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:08.779 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576042+0000", 2024-09-21T15:50:08.779 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:08.779 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.779 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.779 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:08.779 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576044+0000", 2024-09-21T15:50:08.779 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0999999999999998e-06 2024-09-21T15:50:08.780 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.780 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.780 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:08.780 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576051+0000", 2024-09-21T15:50:08.780 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4779999999999996e-06 2024-09-21T15:50:08.780 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.780 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.780 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:08.780 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576058+0000", 2024-09-21T15:50:08.780 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2180000000000002e-06 2024-09-21T15:50:08.780 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.780 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.780 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:08.780 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630661+0000", 2024-09-21T15:50:08.780 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054603240999999997 2024-09-21T15:50:08.780 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.781 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.781 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:08.781 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630747+0000", 2024-09-21T15:50:08.781 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6046999999999998e-05 2024-09-21T15:50:08.781 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:08.781 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:08.781 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:08.781 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630824+0000", 2024-09-21T15:50:08.781 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7119999999999993e-05 2024-09-21T15:50:08.781 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.781 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:08.781 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.781 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:08.781 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:08.781 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:08.783 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T15:50:08.786 INFO:teuthology.orchestra.run.smithi089.stdout:4540 trunc from 0xe699fb0 to 0x8729296 2024-09-21T15:50:08.786 INFO:teuthology.orchestra.run.smithi089.stdout:4541 writesame 0x6400c54 thru 0x64067ff data_size 0x2(0x5bac bytes) 2024-09-21T15:50:08.786 INFO:teuthology.orchestra.run.smithi089.stdout:4542 read 0x62b5e21 thru 0x62bb055 (0x5235 bytes) 2024-09-21T15:50:08.786 INFO:teuthology.orchestra.run.smithi089.stdout:4544 read 0x262cf15 thru 0x262f263 (0x234f bytes) 2024-09-21T15:50:08.786 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:08.786 INFO:teuthology.orchestra.run.smithi089.stdout:4547 read 0x2903cd7 thru 0x290d3bb (0x96e5 bytes) 2024-09-21T15:50:08.787 INFO:teuthology.orchestra.run.smithi089.stdout:4548 read 0xedc7a8 thru 0xee2bed (0x6446 bytes) 2024-09-21T15:50:08.787 INFO:teuthology.orchestra.run.smithi089.stdout:4549 read 0x2704a75 thru 0x2711596 (0xcb22 bytes) 2024-09-21T15:50:08.787 INFO:teuthology.orchestra.run.smithi089.stdout:4550 trunc from 0x8729296 to 0xd9055c6 2024-09-21T15:50:08.787 INFO:teuthology.orchestra.run.smithi089.stdout:4551 punch from 0x772e849 to 0x773ca9f, (0xe256 bytes) 2024-09-21T15:50:08.787 INFO:teuthology.orchestra.run.smithi089.stdout:4552 trunc from 0xd9055c6 to 0xdf27de7 2024-09-21T15:50:08.787 INFO:teuthology.orchestra.run.smithi089.stdout:4553 trunc from 0xdf27de7 to 0xa046efe 2024-09-21T15:50:08.787 INFO:teuthology.orchestra.run.smithi089.stdout:4554 read 0x94e221b thru 0x94ed8b4 (0xb69a bytes) 2024-09-21T15:50:08.787 INFO:teuthology.orchestra.run.smithi089.stdout:4555 write 0xed456f8 thru 0xed4ea6b (0x9374 bytes) 2024-09-21T15:50:08.787 INFO:teuthology.orchestra.run.smithi089.stdout:4556 read 0xaba0b57 thru 0xaba86b9 (0x7b63 bytes) 2024-09-21T15:50:08.787 INFO:teuthology.orchestra.run.smithi089.stdout:4557 read 0x8592ddb thru 0x859cce6 (0x9f0c bytes) 2024-09-21T15:50:08.787 INFO:teuthology.orchestra.run.smithi089.stdout:4558 read 0x540f2c3 thru 0x541415c (0x4e9a bytes) 2024-09-21T15:50:08.787 INFO:teuthology.orchestra.run.smithi089.stdout:4559 punch from 0xcf34f55 to 0xcf39753, (0x47fe bytes) 2024-09-21T15:50:08.788 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:08.788 INFO:teuthology.orchestra.run.smithi089.stdout:4561 read 0x1e872a3 thru 0x1e8d90c (0x666a bytes) 2024-09-21T15:50:08.788 INFO:teuthology.orchestra.run.smithi089.stdout:4562 write 0xd33a92e thru 0xd340473 (0x5b46 bytes) 2024-09-21T15:50:08.788 INFO:teuthology.orchestra.run.smithi089.stdout:4563 read 0x52ba97e thru 0x52c01f9 (0x587c bytes) 2024-09-21T15:50:08.788 INFO:teuthology.orchestra.run.smithi089.stdout:4564 write 0x675d217 thru 0x6765532 (0x831c bytes) 2024-09-21T15:50:08.788 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:08.788 INFO:teuthology.orchestra.run.smithi089.stdout:4566 write 0x61aa14c thru 0x61ac354 (0x2209 bytes) 2024-09-21T15:50:08.789 INFO:teuthology.orchestra.run.smithi089.stdout:4567 writesame 0x9ac7c8d thru 0x9ad2416 data_size 0x5(0xa78a bytes) 2024-09-21T15:50:08.789 INFO:teuthology.orchestra.run.smithi089.stdout:4568 punch from 0xbfebbf1 to 0xbff5232, (0x9641 bytes) 2024-09-21T15:50:08.789 INFO:teuthology.orchestra.run.smithi089.stdout:4570 trunc from 0xed4ea6c to 0xb20f899 2024-09-21T15:50:08.789 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:08.789 INFO:teuthology.orchestra.run.smithi089.stdout:4574 read 0x737dded thru 0x7382be0 (0x4df4 bytes) 2024-09-21T15:50:08.789 INFO:teuthology.orchestra.run.smithi089.stdout:4575 write 0x5468e5c thru 0x5472e9c (0xa041 bytes) 2024-09-21T15:50:08.789 INFO:teuthology.orchestra.run.smithi089.stdout:4577 writesame 0x694cf7b thru 0x6953c6c data_size 0x5(0x6cf2 bytes) 2024-09-21T15:50:08.790 INFO:teuthology.orchestra.run.smithi089.stdout:4580 read 0x4be4c8b thru 0x4bed72f (0x8aa5 bytes) 2024-09-21T15:50:08.790 INFO:teuthology.orchestra.run.smithi089.stdout:4581 read 0x5b00c7b thru 0x5b0dea3 (0xd229 bytes) 2024-09-21T15:50:08.790 INFO:teuthology.orchestra.run.smithi089.stdout:4582 write 0x3037f05 thru 0x303f5b7 (0x76b3 bytes) 2024-09-21T15:50:08.790 INFO:teuthology.orchestra.run.smithi089.stdout:4583 trunc from 0xb20f899 to 0xa73c538 2024-09-21T15:50:08.790 INFO:teuthology.orchestra.run.smithi089.stdout:4584 read 0x54849e1 thru 0x5485c10 (0x1230 bytes) 2024-09-21T15:50:08.790 INFO:teuthology.orchestra.run.smithi089.stdout:4585 punch from 0x467b18c to 0x467e753, (0x35c7 bytes) 2024-09-21T15:50:08.790 INFO:teuthology.orchestra.run.smithi089.stdout:4586 read 0x83fcb78 thru 0x840aa35 (0xdebe bytes) 2024-09-21T15:50:08.791 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:08.781+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:08.792 INFO:teuthology.orchestra.run.smithi089.stdout:4587 flatten 2024-09-21T15:50:08.792 INFO:teuthology.orchestra.run.smithi089.stdout:4588 read 0x2ad805d thru 0x2ae74d7 (0xf47b bytes) 2024-09-21T15:50:08.792 INFO:teuthology.orchestra.run.smithi089.stdout:4589 punch from 0x9b2382d to 0x9b2f780, (0xbf53 bytes) 2024-09-21T15:50:08.802 INFO:teuthology.orchestra.run.smithi089.stdout:4590 writesame 0xe0c3d01 thru 0xe0d3156 data_size 0xa(0xf456 bytes) 2024-09-21T15:50:08.813 INFO:teuthology.orchestra.run.smithi089.stdout:4591 trunc from 0xe0d3157 to 0x51906f1 2024-09-21T15:50:08.821 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:08.821 INFO:teuthology.orchestra.run.smithi089.stdout:4593 punch from 0x2df21b3 to 0x2dffe17, (0xdc64 bytes) 2024-09-21T15:50:08.829 INFO:teuthology.orchestra.run.smithi089.stdout:4594 write 0x80e6889 thru 0x80e82c2 (0x1a3a bytes) 2024-09-21T15:50:08.840 INFO:teuthology.orchestra.run.smithi089.stdout:4595 trunc from 0x80e82c3 to 0xbe3576 2024-09-21T15:50:08.864 INFO:teuthology.orchestra.run.smithi089.stdout:4596 trunc from 0xbe3576 to 0x31d9ce7 2024-09-21T15:50:08.869 INFO:teuthology.orchestra.run.smithi089.stdout:4598 read 0xc955e6 thru 0xca19d4 (0xc3ef bytes) 2024-09-21T15:50:08.869 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:08.869 INFO:teuthology.orchestra.run.smithi089.stdout:4600 read 0x14a00e1 thru 0x14a6b93 (0x6ab3 bytes) 2024-09-21T15:50:08.879 INFO:teuthology.orchestra.run.smithi089.stdout:4603 writesame 0x6d7cf3c thru 0x6d83e4c data_size 0x1(0x6f11 bytes) 2024-09-21T15:50:08.881 INFO:teuthology.orchestra.run.smithi089.stdout:4604 write 0x5029b4b thru 0x502e509 (0x49bf bytes) 2024-09-21T15:50:08.884 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:08.883+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:08.885 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:08.885 INFO:teuthology.orchestra.run.smithi089.stdout:4607 read 0x41b2d9f thru 0x41c03a5 (0xd607 bytes) 2024-09-21T15:50:08.885 INFO:teuthology.orchestra.run.smithi089.stdout:4608 read 0x15b42c7 thru 0x15bcd27 (0x8a61 bytes) 2024-09-21T15:50:08.893 INFO:teuthology.orchestra.run.smithi089.stdout:4610 trunc from 0x6d83e4d to 0xdd0ccf4 2024-09-21T15:50:08.899 INFO:teuthology.orchestra.run.smithi089.stdout:4613 write 0x492ebbf thru 0x4932929 (0x3d6b bytes) 2024-09-21T15:50:08.902 INFO:teuthology.orchestra.run.smithi089.stdout:4614 read 0x1399fed thru 0x139d080 (0x3094 bytes) 2024-09-21T15:50:08.902 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:08.902 INFO:teuthology.orchestra.run.smithi089.stdout:4617 read 0x8ab4581 thru 0x8abe32b (0x9dab bytes) 2024-09-21T15:50:08.903 INFO:teuthology.orchestra.run.smithi089.stdout:4618 write 0xd9da0f1 thru 0xd9e27e5 (0x86f5 bytes) 2024-09-21T15:50:08.905 INFO:teuthology.orchestra.run.smithi089.stdout:4619 write 0xbb316d0 thru 0xbb32dfc (0x172d bytes) 2024-09-21T15:50:08.906 INFO:teuthology.orchestra.run.smithi089.stdout:4620 writesame 0x8e30d88 thru 0x8e3d1ad data_size 0x3(0xc426 bytes) 2024-09-21T15:50:08.926 INFO:teuthology.orchestra.run.smithi089.stdout:4621 trunc from 0xdd0ccf4 to 0x5eae702 2024-09-21T15:50:08.933 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:08.933 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:08.933 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:50:08.933 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:08.940 INFO:teuthology.orchestra.run.smithi089.stdout:4622 read 0xe67c1c thru 0xe6f491 (0x7876 bytes) 2024-09-21T15:50:08.942 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T15:50:08.952 INFO:teuthology.orchestra.run.smithi089.stdout:4623 write 0xbab7eca thru 0xbac22e3 (0xa41a bytes) 2024-09-21T15:50:08.960 INFO:teuthology.orchestra.run.smithi089.stdout:4627 trunc from 0xbac22e4 to 0x6c9fa0f 2024-09-21T15:50:08.969 INFO:teuthology.orchestra.run.smithi089.stdout:4628 read 0x4370726 thru 0x4380099 (0xf974 bytes) 2024-09-21T15:50:08.970 INFO:teuthology.orchestra.run.smithi089.stdout:4629 punch from 0x4d10b38 to 0x4d179d5, (0x6e9d bytes) 2024-09-21T15:50:08.970 INFO:teuthology.orchestra.run.smithi089.stdout:4630 write 0x543bc57 thru 0x544b089 (0xf433 bytes) 2024-09-21T15:50:08.974 INFO:teuthology.orchestra.run.smithi089.stdout:4631 read 0x1cb871b thru 0x1cc868b (0xff71 bytes) 2024-09-21T15:50:08.974 INFO:teuthology.orchestra.run.smithi089.stdout:4633 clone 31 order 25 su 524288 sc 15 2024-09-21T15:50:08.985 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:08.983+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:09.018 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T15:50:09.085 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:09.084+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:09.090 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:09.090 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:09.090 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:50:09.090 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:50:09.090 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:09.099 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T15:50:09.166 INFO:teuthology.orchestra.run.smithi089.stdout:checking clone #29, image image_client.0-clone29 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent30 2024-09-21T15:50:09.187 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:09.185+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:09.202 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:09.203 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:50:09.203 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:50:09.203 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:50:09.203 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.203 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63140 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 876544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:09.203 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.133649+0000", 2024-09-21T15:50:09.203 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.067220845999998, 2024-09-21T15:50:09.203 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053950201000000003, 2024-09-21T15:50:09.203 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:09.203 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:09.204 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:09.204 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:09.204 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:09.204 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63140 2024-09-21T15:50:09.204 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.204 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:09.204 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.204 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:09.204 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133649+0000", 2024-09-21T15:50:09.204 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.204 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.205 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.205 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:09.205 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133649+0000", 2024-09-21T15:50:09.205 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.205 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.205 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.205 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:09.205 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133648+0000", 2024-09-21T15:50:09.205 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:09.205 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.205 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.206 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:09.206 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133651+0000", 2024-09-21T15:50:09.206 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1549999999999999e-06 2024-09-21T15:50:09.206 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.206 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.206 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:09.206 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133660+0000", 2024-09-21T15:50:09.206 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3899999999999993e-06 2024-09-21T15:50:09.206 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.206 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.206 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:09.207 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133669+0000", 2024-09-21T15:50:09.207 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3219999999999999e-06 2024-09-21T15:50:09.207 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.207 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.207 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:09.207 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187409+0000", 2024-09-21T15:50:09.207 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053740257 2024-09-21T15:50:09.207 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.207 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.207 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:09.208 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187492+0000", 2024-09-21T15:50:09.208 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2409000000000003e-05 2024-09-21T15:50:09.208 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.208 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.208 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:09.208 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187599+0000", 2024-09-21T15:50:09.208 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000107334 2024-09-21T15:50:09.208 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.208 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:09.208 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.208 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.208 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.209 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63141 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 884736~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:09.209 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.133843+0000", 2024-09-21T15:50:09.209 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.067026726999998, 2024-09-21T15:50:09.209 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053954015000000001, 2024-09-21T15:50:09.209 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:09.209 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:09.209 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:09.209 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:09.209 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:09.209 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63141 2024-09-21T15:50:09.209 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.210 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:09.210 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.210 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:09.210 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133843+0000", 2024-09-21T15:50:09.210 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.210 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.210 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.210 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:09.210 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133843+0000", 2024-09-21T15:50:09.210 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.210 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.211 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.211 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:09.211 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133673+0000", 2024-09-21T15:50:09.211 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998302 2024-09-21T15:50:09.211 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.211 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.211 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:09.211 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133846+0000", 2024-09-21T15:50:09.211 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000172762 2024-09-21T15:50:09.211 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.211 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.211 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:09.212 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133855+0000", 2024-09-21T15:50:09.212 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2940000000000006e-06 2024-09-21T15:50:09.212 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.212 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.212 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:09.212 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133863+0000", 2024-09-21T15:50:09.212 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9740000000000002e-06 2024-09-21T15:50:09.212 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.212 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.212 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:09.213 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187630+0000", 2024-09-21T15:50:09.213 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053767211000000002 2024-09-21T15:50:09.213 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.213 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.213 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:09.213 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187707+0000", 2024-09-21T15:50:09.213 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7446000000000004e-05 2024-09-21T15:50:09.213 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.213 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.213 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:09.214 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187797+0000", 2024-09-21T15:50:09.214 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.9858000000000006e-05 2024-09-21T15:50:09.214 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.214 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:09.214 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.214 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.214 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.214 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63142 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 892928~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:09.214 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134046+0000", 2024-09-21T15:50:09.214 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.066823784999997, 2024-09-21T15:50:09.215 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054017849999999999, 2024-09-21T15:50:09.215 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:09.215 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:09.215 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:09.215 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:09.215 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:09.215 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63142 2024-09-21T15:50:09.215 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.215 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:09.216 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.216 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:09.216 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134046+0000", 2024-09-21T15:50:09.216 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.216 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.216 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.216 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:09.216 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134046+0000", 2024-09-21T15:50:09.216 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.217 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.217 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.217 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:09.217 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133867+0000", 2024-09-21T15:50:09.217 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998217 2024-09-21T15:50:09.217 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.217 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.217 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:09.217 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134049+0000", 2024-09-21T15:50:09.217 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000181695 2024-09-21T15:50:09.217 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.217 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.218 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:09.218 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134057+0000", 2024-09-21T15:50:09.218 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.303e-06 2024-09-21T15:50:09.218 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.218 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.218 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:09.218 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134063+0000", 2024-09-21T15:50:09.218 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9959999999999999e-06 2024-09-21T15:50:09.218 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.218 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.218 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:09.219 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187849+0000", 2024-09-21T15:50:09.219 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053785571999999997 2024-09-21T15:50:09.219 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.219 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.219 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:09.219 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187954+0000", 2024-09-21T15:50:09.219 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010524899999999999 2024-09-21T15:50:09.219 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.219 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.219 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:09.219 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188064+0000", 2024-09-21T15:50:09.220 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000109549 2024-09-21T15:50:09.220 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.220 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:09.220 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.220 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.220 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.220 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63143 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 901120~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:09.220 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134239+0000", 2024-09-21T15:50:09.220 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.066631192000003, 2024-09-21T15:50:09.220 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054026058000000002, 2024-09-21T15:50:09.220 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:09.220 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:09.221 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:09.221 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:09.221 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:09.221 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63143 2024-09-21T15:50:09.221 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.221 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:09.221 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.221 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:09.221 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134239+0000", 2024-09-21T15:50:09.221 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.221 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.222 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.222 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:09.222 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134239+0000", 2024-09-21T15:50:09.222 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.222 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.222 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.222 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:09.222 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134068+0000", 2024-09-21T15:50:09.222 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998298 2024-09-21T15:50:09.222 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.223 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.223 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:09.223 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134242+0000", 2024-09-21T15:50:09.223 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00017414199999999999 2024-09-21T15:50:09.223 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.223 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.223 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:09.223 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134264+0000", 2024-09-21T15:50:09.223 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1552e-05 2024-09-21T15:50:09.223 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.224 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.224 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:09.224 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134270+0000", 2024-09-21T15:50:09.224 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6939999999999998e-06 2024-09-21T15:50:09.224 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.224 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.224 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:09.224 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188108+0000", 2024-09-21T15:50:09.224 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.05383719 2024-09-21T15:50:09.224 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.224 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.224 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:09.225 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188186+0000", 2024-09-21T15:50:09.225 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8318000000000005e-05 2024-09-21T15:50:09.225 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.225 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.225 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:09.225 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188265+0000", 2024-09-21T15:50:09.225 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8597000000000007e-05 2024-09-21T15:50:09.225 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.225 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:09.225 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.225 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.225 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.226 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63144 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 909312~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:09.226 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134432+0000", 2024-09-21T15:50:09.226 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.066437636000003, 2024-09-21T15:50:09.226 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053968953, 2024-09-21T15:50:09.226 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:09.226 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:09.226 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:09.226 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:09.226 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:09.226 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63144 2024-09-21T15:50:09.227 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.227 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:09.227 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.227 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:09.227 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134432+0000", 2024-09-21T15:50:09.227 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.227 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.227 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.227 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:09.227 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134432+0000", 2024-09-21T15:50:09.228 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.228 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.228 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.228 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:09.228 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134276+0000", 2024-09-21T15:50:09.228 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998441 2024-09-21T15:50:09.228 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.228 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.228 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:09.228 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134436+0000", 2024-09-21T15:50:09.228 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000159591 2024-09-21T15:50:09.228 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.228 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.229 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:09.229 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134445+0000", 2024-09-21T15:50:09.229 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6780000000000006e-06 2024-09-21T15:50:09.229 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.229 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.229 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:09.229 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134452+0000", 2024-09-21T15:50:09.229 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6830000000000008e-06 2024-09-21T15:50:09.229 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.230 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.230 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:09.230 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188282+0000", 2024-09-21T15:50:09.230 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053829252000000001 2024-09-21T15:50:09.230 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.230 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.230 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:09.230 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188328+0000", 2024-09-21T15:50:09.230 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6734999999999999e-05 2024-09-21T15:50:09.230 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.230 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.231 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:09.231 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188401+0000", 2024-09-21T15:50:09.231 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2731000000000005e-05 2024-09-21T15:50:09.231 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.231 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:09.231 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.231 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.231 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.231 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63145 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 860160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:09.231 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134736+0000", 2024-09-21T15:50:09.231 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.066133850999996, 2024-09-21T15:50:09.232 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053865960999999997, 2024-09-21T15:50:09.232 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:09.232 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:09.232 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:09.232 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:09.232 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:09.232 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63145 2024-09-21T15:50:09.232 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.232 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:09.232 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.232 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:09.232 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134736+0000", 2024-09-21T15:50:09.233 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.233 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.233 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.233 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:09.233 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134736+0000", 2024-09-21T15:50:09.233 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.233 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.233 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.233 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:09.234 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134458+0000", 2024-09-21T15:50:09.234 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999722 2024-09-21T15:50:09.234 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.234 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.234 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:09.234 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134741+0000", 2024-09-21T15:50:09.234 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00028245599999999999 2024-09-21T15:50:09.234 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.234 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.234 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:09.235 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134753+0000", 2024-09-21T15:50:09.235 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2548000000000001e-05 2024-09-21T15:50:09.235 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.235 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.235 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:09.235 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134760+0000", 2024-09-21T15:50:09.235 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4189999999999996e-06 2024-09-21T15:50:09.235 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.235 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.235 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:09.236 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188426+0000", 2024-09-21T15:50:09.236 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053665894999999998 2024-09-21T15:50:09.236 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.236 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.236 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:09.236 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188502+0000", 2024-09-21T15:50:09.236 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5791000000000007e-05 2024-09-21T15:50:09.236 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.236 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.236 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:09.236 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188602+0000", 2024-09-21T15:50:09.236 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9667999999999995e-05 2024-09-21T15:50:09.237 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.237 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:09.237 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.237 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.237 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.237 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63148 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 933888~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:09.237 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.135289+0000", 2024-09-21T15:50:09.237 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.065580896, 2024-09-21T15:50:09.237 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053948488000000003, 2024-09-21T15:50:09.237 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:09.237 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:09.237 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:09.238 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:09.238 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:09.238 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63148 2024-09-21T15:50:09.238 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.238 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:09.238 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.238 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:09.238 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135289+0000", 2024-09-21T15:50:09.238 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.238 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.238 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.239 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:09.239 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135289+0000", 2024-09-21T15:50:09.239 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.239 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.239 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.239 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:09.239 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135288+0000", 2024-09-21T15:50:09.239 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:09.239 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.239 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.240 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:09.240 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135296+0000", 2024-09-21T15:50:09.240 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5399999999999998e-06 2024-09-21T15:50:09.240 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.240 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.240 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:09.240 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135303+0000", 2024-09-21T15:50:09.240 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0500000000000003e-06 2024-09-21T15:50:09.240 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.240 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.240 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:09.241 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135308+0000", 2024-09-21T15:50:09.241 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9749999999999998e-06 2024-09-21T15:50:09.241 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.241 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.241 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:09.241 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189066+0000", 2024-09-21T15:50:09.241 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053758501 2024-09-21T15:50:09.241 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.241 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.241 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:09.241 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189145+0000", 2024-09-21T15:50:09.242 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9017000000000006e-05 2024-09-21T15:50:09.242 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.242 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.242 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:09.242 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189237+0000", 2024-09-21T15:50:09.242 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1906999999999999e-05 2024-09-21T15:50:09.242 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.243 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:09.243 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.243 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.243 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.243 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63149 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 942080~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:09.243 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.135430+0000", 2024-09-21T15:50:09.243 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.065439566999999, 2024-09-21T15:50:09.243 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054005434999999997, 2024-09-21T15:50:09.243 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:09.243 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:09.244 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:09.244 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:09.244 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:09.244 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63149 2024-09-21T15:50:09.244 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.244 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:09.244 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.244 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:09.245 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135430+0000", 2024-09-21T15:50:09.245 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.245 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.245 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.245 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:09.245 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135430+0000", 2024-09-21T15:50:09.245 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.245 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.245 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.245 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:09.245 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135313+0000", 2024-09-21T15:50:09.245 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998827 2024-09-21T15:50:09.245 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.246 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.246 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:09.246 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135434+0000", 2024-09-21T15:50:09.246 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000120876 2024-09-21T15:50:09.246 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.246 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.246 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:09.246 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135443+0000", 2024-09-21T15:50:09.246 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3340000000000008e-06 2024-09-21T15:50:09.246 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.247 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.247 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:09.247 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135449+0000", 2024-09-21T15:50:09.247 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1580000000000002e-06 2024-09-21T15:50:09.247 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.247 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.247 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:09.247 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189267+0000", 2024-09-21T15:50:09.247 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053817775999999998 2024-09-21T15:50:09.247 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.247 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.248 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:09.248 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189336+0000", 2024-09-21T15:50:09.248 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9485000000000003e-05 2024-09-21T15:50:09.248 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.248 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.248 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:09.248 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189436+0000", 2024-09-21T15:50:09.248 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9277000000000005e-05 2024-09-21T15:50:09.248 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.248 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:09.248 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.248 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.249 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.249 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63150 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 950272~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:09.249 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.135618+0000", 2024-09-21T15:50:09.249 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.065251822, 2024-09-21T15:50:09.249 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054027393, 2024-09-21T15:50:09.249 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:09.249 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:09.249 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:09.249 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:09.249 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:09.249 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63150 2024-09-21T15:50:09.250 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.250 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:09.250 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.250 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:09.250 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135618+0000", 2024-09-21T15:50:09.250 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.250 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.250 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.250 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:09.250 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135618+0000", 2024-09-21T15:50:09.250 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.251 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.251 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.251 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:09.251 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135454+0000", 2024-09-21T15:50:09.251 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998364 2024-09-21T15:50:09.251 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.251 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.251 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:09.251 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135621+0000", 2024-09-21T15:50:09.251 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016714099999999999 2024-09-21T15:50:09.251 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.251 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.251 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:09.252 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135631+0000", 2024-09-21T15:50:09.252 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5340000000000005e-06 2024-09-21T15:50:09.252 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.252 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.252 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:09.252 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135638+0000", 2024-09-21T15:50:09.252 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9280000000000004e-06 2024-09-21T15:50:09.252 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.252 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.252 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:09.252 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189463+0000", 2024-09-21T15:50:09.253 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053825681 2024-09-21T15:50:09.253 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.253 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.253 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:09.253 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189546+0000", 2024-09-21T15:50:09.253 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2278000000000001e-05 2024-09-21T15:50:09.253 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.253 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.253 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:09.254 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189645+0000", 2024-09-21T15:50:09.254 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9543999999999994e-05 2024-09-21T15:50:09.254 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.254 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:09.254 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.254 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.254 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.254 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63152 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 966656~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:09.254 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.136029+0000", 2024-09-21T15:50:09.254 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.064841229999999, 2024-09-21T15:50:09.254 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054024823999999999, 2024-09-21T15:50:09.255 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:09.255 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:09.255 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:09.255 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:09.255 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:09.255 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63152 2024-09-21T15:50:09.255 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.255 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:09.255 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.255 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:09.255 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136029+0000", 2024-09-21T15:50:09.256 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.256 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.256 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.256 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:09.256 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136029+0000", 2024-09-21T15:50:09.256 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.256 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.256 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.256 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:09.256 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136028+0000", 2024-09-21T15:50:09.256 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:09.256 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.257 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.257 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:09.257 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136031+0000", 2024-09-21T15:50:09.257 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.033e-06 2024-09-21T15:50:09.257 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.257 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.257 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:09.257 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136039+0000", 2024-09-21T15:50:09.257 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7200000000000006e-06 2024-09-21T15:50:09.257 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.257 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.258 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:09.258 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136051+0000", 2024-09-21T15:50:09.258 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1987000000000001e-05 2024-09-21T15:50:09.258 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.258 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.258 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:09.258 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189876+0000", 2024-09-21T15:50:09.258 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053825264999999997 2024-09-21T15:50:09.258 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.258 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.258 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:09.258 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189955+0000", 2024-09-21T15:50:09.259 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9296999999999997e-05 2024-09-21T15:50:09.259 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.259 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.259 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:09.259 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190053+0000", 2024-09-21T15:50:09.259 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.8067999999999997e-05 2024-09-21T15:50:09.259 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.259 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:09.259 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.259 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.259 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.259 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63153 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 974848~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:09.259 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.136311+0000", 2024-09-21T15:50:09.260 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.064558716999997, 2024-09-21T15:50:09.260 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053933825999999997, 2024-09-21T15:50:09.260 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:09.260 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:09.260 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:09.260 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:09.260 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:09.260 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63153 2024-09-21T15:50:09.260 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.260 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:09.260 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.261 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:09.261 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136311+0000", 2024-09-21T15:50:09.261 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.261 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.261 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.261 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:09.261 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136311+0000", 2024-09-21T15:50:09.261 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.261 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.261 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.261 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:09.261 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136057+0000", 2024-09-21T15:50:09.261 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9997463 2024-09-21T15:50:09.262 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.262 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.262 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:09.262 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136316+0000", 2024-09-21T15:50:09.262 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00025872299999999998 2024-09-21T15:50:09.262 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.262 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.262 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:09.262 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136326+0000", 2024-09-21T15:50:09.262 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0552e-05 2024-09-21T15:50:09.262 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.262 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.263 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:09.263 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136334+0000", 2024-09-21T15:50:09.263 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1409999999999999e-06 2024-09-21T15:50:09.263 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.263 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.263 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:09.263 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190080+0000", 2024-09-21T15:50:09.263 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053745933000000003 2024-09-21T15:50:09.263 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.263 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.263 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:09.263 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190157+0000", 2024-09-21T15:50:09.264 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7412999999999999e-05 2024-09-21T15:50:09.264 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.264 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.264 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:09.264 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190245+0000", 2024-09-21T15:50:09.264 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7904000000000004e-05 2024-09-21T15:50:09.264 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.264 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:09.264 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.264 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.264 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.264 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63154 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 983040~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:09.265 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.136472+0000", 2024-09-21T15:50:09.265 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.064397366000001, 2024-09-21T15:50:09.265 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053968448000000002, 2024-09-21T15:50:09.265 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:09.265 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:09.265 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:09.265 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:09.265 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:09.265 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63154 2024-09-21T15:50:09.265 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.265 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:09.265 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.266 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:09.266 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136472+0000", 2024-09-21T15:50:09.266 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.266 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.266 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.266 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:09.266 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136472+0000", 2024-09-21T15:50:09.266 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.266 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.266 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.266 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:09.267 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136341+0000", 2024-09-21T15:50:09.267 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998684 2024-09-21T15:50:09.267 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.267 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.267 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:09.267 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136476+0000", 2024-09-21T15:50:09.267 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013528100000000001 2024-09-21T15:50:09.267 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.267 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.267 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:09.267 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136484+0000", 2024-09-21T15:50:09.267 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4760000000000001e-06 2024-09-21T15:50:09.267 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.268 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.268 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:09.268 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136490+0000", 2024-09-21T15:50:09.268 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8640000000000001e-06 2024-09-21T15:50:09.268 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.268 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.268 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:09.268 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190271+0000", 2024-09-21T15:50:09.268 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053781223000000003 2024-09-21T15:50:09.268 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.268 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.268 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:09.269 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190348+0000", 2024-09-21T15:50:09.269 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7058999999999995e-05 2024-09-21T15:50:09.269 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.269 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.269 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:09.269 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190441+0000", 2024-09-21T15:50:09.269 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2937999999999997e-05 2024-09-21T15:50:09.269 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.269 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:09.269 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.269 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.269 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.270 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63157 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1015808~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:09.270 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137112+0000", 2024-09-21T15:50:09.270 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.063758079000003, 2024-09-21T15:50:09.270 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053932345, 2024-09-21T15:50:09.270 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:09.270 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:09.270 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:09.270 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:09.270 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:09.270 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63157 2024-09-21T15:50:09.270 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.270 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:09.271 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.271 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:09.271 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137112+0000", 2024-09-21T15:50:09.271 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.271 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.271 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.271 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:09.271 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137112+0000", 2024-09-21T15:50:09.271 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.271 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.271 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.271 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:09.272 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137072+0000", 2024-09-21T15:50:09.272 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999604 2024-09-21T15:50:09.272 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.272 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.272 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:09.272 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137115+0000", 2024-09-21T15:50:09.272 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2379999999999997e-05 2024-09-21T15:50:09.272 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.272 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.272 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:09.272 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137121+0000", 2024-09-21T15:50:09.272 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6499999999999999e-06 2024-09-21T15:50:09.272 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.273 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.273 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:09.273 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137126+0000", 2024-09-21T15:50:09.273 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0769999999999998e-06 2024-09-21T15:50:09.273 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.273 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.273 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:09.273 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190868+0000", 2024-09-21T15:50:09.273 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053741555000000003 2024-09-21T15:50:09.273 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.273 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.273 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:09.274 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190945+0000", 2024-09-21T15:50:09.274 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7171999999999999e-05 2024-09-21T15:50:09.274 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.274 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.274 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:09.274 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191044+0000", 2024-09-21T15:50:09.274 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9031999999999996e-05 2024-09-21T15:50:09.274 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.274 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:09.274 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.274 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.274 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.275 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63158 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1007616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:09.275 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137293+0000", 2024-09-21T15:50:09.275 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.063576791999999, 2024-09-21T15:50:09.275 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054005642, 2024-09-21T15:50:09.275 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:09.275 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:09.275 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:09.275 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:09.275 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:09.275 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63158 2024-09-21T15:50:09.275 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.276 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:09.276 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.276 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:09.276 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137293+0000", 2024-09-21T15:50:09.276 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.276 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.276 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.276 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:09.276 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137293+0000", 2024-09-21T15:50:09.276 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.276 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.276 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.277 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:09.277 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137131+0000", 2024-09-21T15:50:09.277 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998379 2024-09-21T15:50:09.277 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.277 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.277 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:09.277 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137296+0000", 2024-09-21T15:50:09.277 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016568999999999999 2024-09-21T15:50:09.277 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.278 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.278 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:09.278 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137306+0000", 2024-09-21T15:50:09.278 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.6029999999999994e-06 2024-09-21T15:50:09.278 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.278 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.278 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:09.278 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137312+0000", 2024-09-21T15:50:09.278 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3210000000000001e-06 2024-09-21T15:50:09.278 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.279 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.279 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:09.279 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191087+0000", 2024-09-21T15:50:09.279 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053774686000000002 2024-09-21T15:50:09.279 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.279 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.279 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:09.279 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191175+0000", 2024-09-21T15:50:09.279 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8106e-05 2024-09-21T15:50:09.279 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.279 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.279 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:09.279 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191299+0000", 2024-09-21T15:50:09.280 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012343599999999999 2024-09-21T15:50:09.280 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.280 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:09.280 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.280 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.280 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.280 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63159 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1024000~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:09.280 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137568+0000", 2024-09-21T15:50:09.280 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.063301520000003, 2024-09-21T15:50:09.280 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053962999999999997, 2024-09-21T15:50:09.280 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:09.281 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:09.281 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:09.281 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:09.281 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:09.281 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63159 2024-09-21T15:50:09.281 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.281 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:09.281 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.281 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:09.281 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137568+0000", 2024-09-21T15:50:09.281 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.282 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.282 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.282 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:09.282 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137568+0000", 2024-09-21T15:50:09.282 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.282 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.282 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.282 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:09.282 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137317+0000", 2024-09-21T15:50:09.282 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9997492 2024-09-21T15:50:09.282 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.283 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.283 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:09.283 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137572+0000", 2024-09-21T15:50:09.283 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00025491499999999998 2024-09-21T15:50:09.283 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.283 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.283 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:09.283 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137582+0000", 2024-09-21T15:50:09.283 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4490000000000005e-06 2024-09-21T15:50:09.283 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.283 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.284 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:09.284 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137589+0000", 2024-09-21T15:50:09.284 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5580000000000003e-06 2024-09-21T15:50:09.284 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.284 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.284 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:09.284 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191344+0000", 2024-09-21T15:50:09.284 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053754272999999998 2024-09-21T15:50:09.284 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.284 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.284 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:09.284 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191429+0000", 2024-09-21T15:50:09.284 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.509e-05 2024-09-21T15:50:09.285 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.285 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.285 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:09.285 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191531+0000", 2024-09-21T15:50:09.285 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000102519 2024-09-21T15:50:09.285 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.285 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:09.285 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.285 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.285 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.285 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63160 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1032192~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:09.286 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137771+0000", 2024-09-21T15:50:09.286 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.063098986, 2024-09-21T15:50:09.286 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053957818999999997, 2024-09-21T15:50:09.286 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:09.286 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:09.286 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:09.286 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:09.286 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:09.286 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63160 2024-09-21T15:50:09.286 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.286 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:09.286 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.287 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:09.287 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137771+0000", 2024-09-21T15:50:09.287 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.287 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.287 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.287 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:09.287 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137771+0000", 2024-09-21T15:50:09.287 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.287 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.287 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.287 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:09.287 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137596+0000", 2024-09-21T15:50:09.288 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998255 2024-09-21T15:50:09.288 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.288 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.288 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:09.288 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137780+0000", 2024-09-21T15:50:09.288 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00018363900000000001 2024-09-21T15:50:09.288 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.288 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.288 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:09.288 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137791+0000", 2024-09-21T15:50:09.288 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.096e-05 2024-09-21T15:50:09.288 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.288 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.289 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:09.289 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137798+0000", 2024-09-21T15:50:09.289 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1169999999999998e-06 2024-09-21T15:50:09.289 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.289 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.289 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:09.289 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191560+0000", 2024-09-21T15:50:09.289 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053762163000000002 2024-09-21T15:50:09.289 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.289 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.289 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:09.289 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191634+0000", 2024-09-21T15:50:09.290 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4381999999999994e-05 2024-09-21T15:50:09.290 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.290 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.290 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:09.290 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191729+0000", 2024-09-21T15:50:09.290 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4098000000000006e-05 2024-09-21T15:50:09.290 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.290 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:09.290 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.290 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.290 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.290 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63161 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1040384~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:09.291 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.138018+0000", 2024-09-21T15:50:09.291 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.062852231999997, 2024-09-21T15:50:09.291 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053905657000000003, 2024-09-21T15:50:09.291 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:09.291 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:09.291 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:09.291 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:09.291 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:09.291 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63161 2024-09-21T15:50:09.291 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.291 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:09.292 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.292 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:09.292 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138018+0000", 2024-09-21T15:50:09.292 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.292 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.292 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.292 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:09.292 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138018+0000", 2024-09-21T15:50:09.292 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.292 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.293 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.293 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:09.293 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137804+0000", 2024-09-21T15:50:09.293 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9997859 2024-09-21T15:50:09.293 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.293 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.293 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:09.293 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138021+0000", 2024-09-21T15:50:09.293 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00021769899999999999 2024-09-21T15:50:09.293 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.293 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.293 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:09.293 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138043+0000", 2024-09-21T15:50:09.294 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1627000000000001e-05 2024-09-21T15:50:09.294 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.294 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.294 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:09.294 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138054+0000", 2024-09-21T15:50:09.294 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1321999999999999e-05 2024-09-21T15:50:09.294 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.294 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.294 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:09.294 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191755+0000", 2024-09-21T15:50:09.295 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053700354999999998 2024-09-21T15:50:09.295 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.295 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.295 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:09.295 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191830+0000", 2024-09-21T15:50:09.295 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5320000000000004e-05 2024-09-21T15:50:09.295 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.295 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.295 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:09.295 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191923+0000", 2024-09-21T15:50:09.295 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3282999999999994e-05 2024-09-21T15:50:09.295 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.296 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:09.296 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.296 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.296 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.296 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63176 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 108544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:09.296 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.140861+0000", 2024-09-21T15:50:09.296 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.060008912999997, 2024-09-21T15:50:09.296 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053935394999999997, 2024-09-21T15:50:09.296 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:09.296 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:09.296 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:09.296 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:09.297 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:09.297 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63176 2024-09-21T15:50:09.297 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.297 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:09.297 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.297 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:09.297 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140861+0000", 2024-09-21T15:50:09.297 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.297 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.297 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.297 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:09.298 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140861+0000", 2024-09-21T15:50:09.298 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.298 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.298 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.298 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:09.298 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140860+0000", 2024-09-21T15:50:09.298 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:09.298 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.298 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.298 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:09.298 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140863+0000", 2024-09-21T15:50:09.298 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8710000000000001e-06 2024-09-21T15:50:09.299 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.299 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.299 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:09.299 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140871+0000", 2024-09-21T15:50:09.299 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8089999999999993e-06 2024-09-21T15:50:09.299 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.299 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.299 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:09.299 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140877+0000", 2024-09-21T15:50:09.299 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5969999999999999e-06 2024-09-21T15:50:09.299 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.300 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.300 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:09.300 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.194608+0000", 2024-09-21T15:50:09.300 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053731749000000002 2024-09-21T15:50:09.300 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.300 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.300 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:09.300 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.194697+0000", 2024-09-21T15:50:09.300 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8504000000000005e-05 2024-09-21T15:50:09.300 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.301 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.301 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:09.301 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.194796+0000", 2024-09-21T15:50:09.301 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9414999999999996e-05 2024-09-21T15:50:09.301 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.301 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:09.301 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.301 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.302 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.302 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63182 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 157696~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:09.302 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:50:09.302 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.058736512999999, 2024-09-21T15:50:09.302 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053958825000000002, 2024-09-21T15:50:09.302 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:09.302 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:09.302 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:09.302 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:09.302 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:09.303 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63182 2024-09-21T15:50:09.303 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.303 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:09.303 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.303 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:09.303 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:50:09.303 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.303 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.303 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.304 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:09.304 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:50:09.304 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.304 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.304 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.304 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:09.304 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:50:09.304 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:09.304 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.304 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.304 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:09.305 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142136+0000", 2024-09-21T15:50:09.305 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8169999999999999e-06 2024-09-21T15:50:09.305 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.305 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.305 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:09.305 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:50:09.305 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3752e-05 2024-09-21T15:50:09.305 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.305 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.305 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:09.305 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142155+0000", 2024-09-21T15:50:09.306 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1859999999999997e-06 2024-09-21T15:50:09.306 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.306 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.306 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:09.306 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.195914+0000", 2024-09-21T15:50:09.306 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053758528999999999 2024-09-21T15:50:09.306 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.306 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.306 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:09.306 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.195995+0000", 2024-09-21T15:50:09.307 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0983999999999996e-05 2024-09-21T15:50:09.307 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.307 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.307 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:09.307 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.196092+0000", 2024-09-21T15:50:09.307 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7069000000000002e-05 2024-09-21T15:50:09.307 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.307 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:09.307 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.307 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.308 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.308 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63193 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 239616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:09.308 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:50:09.308 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.056648928000001, 2024-09-21T15:50:09.308 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053956577999999998, 2024-09-21T15:50:09.308 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:09.308 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:09.308 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:09.308 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:09.308 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:09.309 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63193 2024-09-21T15:50:09.309 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.309 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:09.309 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.309 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:09.309 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:50:09.309 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.309 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.309 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.309 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:09.310 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:50:09.310 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.310 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.310 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.310 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:09.310 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:50:09.310 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:09.310 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.310 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.310 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:09.310 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144223+0000", 2024-09-21T15:50:09.310 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0990000000000001e-06 2024-09-21T15:50:09.311 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.311 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.311 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:09.311 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144230+0000", 2024-09-21T15:50:09.311 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9070000000000001e-06 2024-09-21T15:50:09.311 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.311 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.311 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:09.311 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144234+0000", 2024-09-21T15:50:09.311 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9010000000000002e-06 2024-09-21T15:50:09.311 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.311 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.311 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:09.312 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.198004+0000", 2024-09-21T15:50:09.312 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053769539999999998 2024-09-21T15:50:09.312 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.312 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.312 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:09.312 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.198083+0000", 2024-09-21T15:50:09.312 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9531999999999997e-05 2024-09-21T15:50:09.312 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.312 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.312 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:09.312 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.198177+0000", 2024-09-21T15:50:09.312 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3906000000000005e-05 2024-09-21T15:50:09.313 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.313 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:09.313 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.313 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.313 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:09.313 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:09.314 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T15:50:09.315 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:09.313+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:09.395 INFO:teuthology.orchestra.run.smithi089.stdout:4635 read 0x5eadea2 thru 0x5ebd755 (0xf8b4 bytes) 2024-09-21T15:50:09.396 INFO:teuthology.orchestra.run.smithi089.stdout:4636 punch from 0x3020739 to 0x30289e7, (0x82ae bytes) 2024-09-21T15:50:09.405 INFO:teuthology.orchestra.run.smithi089.stdout:4639 writesame 0x437bec8 thru 0x43841bb data_size 0x3a(0x82f4 bytes) 2024-09-21T15:50:09.415 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:09.414+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:09.421 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:09.421 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:09.421 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:50:09.421 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:09.422 INFO:teuthology.orchestra.run.smithi089.stdout:4640 trunc from 0x6c9fa0f to 0x124ecc 2024-09-21T15:50:09.433 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T15:50:09.459 INFO:teuthology.orchestra.run.smithi089.stdout:4641 write 0xd1429ed thru 0xd14d7dd (0xadf1 bytes) 2024-09-21T15:50:09.462 INFO:teuthology.orchestra.run.smithi089.stdout:4642 punch from 0x1ff848a to 0x200370b, (0xb281 bytes) 2024-09-21T15:50:09.462 INFO:teuthology.orchestra.run.smithi089.stdout:4643 write 0x5810f20 thru 0x5820db0 (0xfe91 bytes) 2024-09-21T15:50:09.465 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:09.465 INFO:teuthology.orchestra.run.smithi089.stdout:4646 write 0x64df9cf thru 0x64e58ef (0x5f21 bytes) 2024-09-21T15:50:09.468 INFO:teuthology.orchestra.run.smithi089.stdout:4647 write 0x2ec5bed thru 0x2ec7730 (0x1b44 bytes) 2024-09-21T15:50:09.472 INFO:teuthology.orchestra.run.smithi089.stdout:4650 read 0x951f659 thru 0x95297b4 (0xa15c bytes) 2024-09-21T15:50:09.472 INFO:teuthology.orchestra.run.smithi089.stdout:4651 read 0x55fc44c thru 0x560abc0 (0xe775 bytes) 2024-09-21T15:50:09.472 INFO:teuthology.orchestra.run.smithi089.stdout:4652 write 0x8e2b453 thru 0x8e2c998 (0x1546 bytes) 2024-09-21T15:50:09.475 INFO:teuthology.orchestra.run.smithi089.stdout:4654 read 0x275503b thru 0x275bf11 (0x6ed7 bytes) 2024-09-21T15:50:09.475 INFO:teuthology.orchestra.run.smithi089.stdout:4655 write 0xc454a93 thru 0xc45aff3 (0x6561 bytes) 2024-09-21T15:50:09.480 INFO:teuthology.orchestra.run.smithi089.stdout:4656 trunc from 0xd14d7de to 0xa6fe925 2024-09-21T15:50:09.497 INFO:teuthology.orchestra.run.smithi089.stdout:4657 writesame 0xaf19521 thru 0xaf228f1 data_size 0x1(0x93d1 bytes) 2024-09-21T15:50:09.505 INFO:teuthology.orchestra.run.smithi089.stdout:4658 write 0xdcce9c0 thru 0xdcd8972 (0x9fb3 bytes) 2024-09-21T15:50:09.507 INFO:teuthology.orchestra.run.smithi089.stdout:4659 write 0x4a261b6 thru 0x4a318ac (0xb6f7 bytes) 2024-09-21T15:50:09.509 INFO:teuthology.orchestra.run.smithi089.stdout:4660 write 0xb5af3f1 thru 0xb5afae2 (0x6f2 bytes) 2024-09-21T15:50:09.510 INFO:teuthology.orchestra.run.smithi089.stdout:4661 read 0x2a3e1a7 thru 0x2a4946b (0xb2c5 bytes) 2024-09-21T15:50:09.510 INFO:teuthology.orchestra.run.smithi089.stdout:4662 read 0x64b8b42 thru 0x64bf172 (0x6631 bytes) 2024-09-21T15:50:09.510 INFO:teuthology.orchestra.run.smithi089.stdout:4663 read 0x4132d28 thru 0x4141b31 (0xee0a bytes) 2024-09-21T15:50:09.511 INFO:teuthology.orchestra.run.smithi089.stdout:4664 read 0x4680176 thru 0x468d6bd (0xd548 bytes) 2024-09-21T15:50:09.511 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:09.511 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:09.511 INFO:teuthology.orchestra.run.smithi089.stdout:4668 read 0x13daa48 thru 0x13e4cd0 (0xa289 bytes) 2024-09-21T15:50:09.516 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:09.515+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:09.526 INFO:teuthology.orchestra.run.smithi089.stdout:4669 trunc from 0xdcd8973 to 0x8b9de4 2024-09-21T15:50:09.543 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:09.543 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:09.544 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:50:09.544 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:50:09.544 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:09.548 INFO:teuthology.orchestra.run.smithi089.stdout:4670 write 0x404c2b0 thru 0x40599aa (0xd6fb bytes) 2024-09-21T15:50:09.553 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T15:50:09.568 INFO:teuthology.orchestra.run.smithi089.stdout:4671 write 0xc597b88 thru 0xc597fe0 (0x459 bytes) 2024-09-21T15:50:09.574 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:09.574 INFO:teuthology.orchestra.run.smithi089.stdout:4676 write 0xfacc8d thru 0xfb570c (0x8a80 bytes) 2024-09-21T15:50:09.576 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:09.576 INFO:teuthology.orchestra.run.smithi089.stdout:4678 read 0x9b9bd27 thru 0x9bab9d7 (0xfcb1 bytes) 2024-09-21T15:50:09.577 INFO:teuthology.orchestra.run.smithi089.stdout:4679 write 0x54bc5fe thru 0x54c4c8a (0x868d bytes) 2024-09-21T15:50:09.585 INFO:teuthology.orchestra.run.smithi089.stdout:4680 write 0x10bf97 thru 0x11b056 (0xf0c0 bytes) 2024-09-21T15:50:09.589 INFO:teuthology.orchestra.run.smithi089.stdout:4681 write 0x3bed4f1 thru 0x3bf7a1f (0xa52f bytes) 2024-09-21T15:50:09.591 INFO:teuthology.orchestra.run.smithi089.stdout:4684 write 0x70fb223 thru 0x710088d (0x566b bytes) 2024-09-21T15:50:09.596 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:09.596 INFO:teuthology.orchestra.run.smithi089.stdout:4688 read 0xab9498e thru 0xab9e86c (0x9edf bytes) 2024-09-21T15:50:09.596 INFO:teuthology.orchestra.run.smithi089.stdout:4689 write 0x9b49820 thru 0x9b58314 (0xeaf5 bytes) 2024-09-21T15:50:09.599 INFO:teuthology.orchestra.run.smithi089.stdout:4691 writesame 0x426d126 thru 0x4274fbc data_size 0x1(0x7e97 bytes) 2024-09-21T15:50:09.619 INFO:teuthology.orchestra.run.smithi089.stdout:4692 write 0x9d676eb thru 0x9d6f5c4 (0x7eda bytes) 2024-09-21T15:50:09.621 INFO:teuthology.orchestra.run.smithi089.stdout:4693 write 0x3d82ac6 thru 0x3d8a150 (0x768b bytes) 2024-09-21T15:50:09.625 INFO:teuthology.orchestra.run.smithi089.stdout:4695 write 0x9420ecb thru 0x942d8de (0xca14 bytes) 2024-09-21T15:50:09.627 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:09.627 INFO:teuthology.orchestra.run.smithi089.stdout:4698 write 0x2326723 thru 0x232f8ba (0x9198 bytes) 2024-09-21T15:50:09.628 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:09.628 INFO:teuthology.orchestra.run.smithi089.stdout:4700 writesame 0x685076c thru 0x68593ad data_size 0xd(0x8c42 bytes) 2024-09-21T15:50:09.630 INFO:teuthology.orchestra.run.smithi089.stdout:4701 clone 32 order 20 su 131072 sc 4 2024-09-21T15:50:09.637 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:09.636+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:09.698 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:09.698 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:50:09.698 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:50:09.698 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:50:09.698 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.698 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4209.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:50:09.698 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:50:09.698 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 78.504811130999997, 2024-09-21T15:50:09.698 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.19452909900000001, 2024-09-21T15:50:09.698 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:09.698 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:09.699 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:09.699 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4209", 2024-09-21T15:50:09.699 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/4045143050", 2024-09-21T15:50:09.699 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1 2024-09-21T15:50:09.699 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.699 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:09.699 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.699 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:09.699 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:50:09.699 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.699 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.699 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.699 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:09.699 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:50:09.700 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.700 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.700 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.700 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:09.700 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191161+0000", 2024-09-21T15:50:09.700 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-21T15:50:09.700 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.700 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.700 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:09.700 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191169+0000", 2024-09-21T15:50:09.700 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0090000000000007e-06 2024-09-21T15:50:09.700 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.700 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.700 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:09.701 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191180+0000", 2024-09-21T15:50:09.701 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0728e-05 2024-09-21T15:50:09.701 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.701 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.701 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:09.701 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191187+0000", 2024-09-21T15:50:09.701 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9009999999999994e-06 2024-09-21T15:50:09.701 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.701 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.701 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:09.701 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191238+0000", 2024-09-21T15:50:09.701 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0287000000000002e-05 2024-09-21T15:50:09.701 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.701 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.702 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-21T15:50:09.702 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191245+0000", 2024-09-21T15:50:09.702 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0709999999999997e-06 2024-09-21T15:50:09.702 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.702 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.702 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:09.702 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380309+0000", 2024-09-21T15:50:09.702 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.189064277 2024-09-21T15:50:09.702 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.702 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.702 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:09.702 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380454+0000", 2024-09-21T15:50:09.702 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014490799999999999 2024-09-21T15:50:09.703 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.703 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.703 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0", 2024-09-21T15:50:09.703 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380610+0000", 2024-09-21T15:50:09.703 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015650199999999999 2024-09-21T15:50:09.703 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.703 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.703 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:50:09.703 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.381160+0000", 2024-09-21T15:50:09.703 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00054903799999999996 2024-09-21T15:50:09.703 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.703 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.703 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:09.703 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385591+0000", 2024-09-21T15:50:09.703 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.004431272 2024-09-21T15:50:09.704 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.704 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.704 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:50:09.704 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385651+0000", 2024-09-21T15:50:09.704 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0455999999999999e-05 2024-09-21T15:50:09.704 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.704 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.704 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:09.704 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385691+0000", 2024-09-21T15:50:09.704 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0234999999999997e-05 2024-09-21T15:50:09.704 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.704 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:09.704 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.704 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.704 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.705 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64028 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1763328~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:09.705 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167504+0000", 2024-09-21T15:50:09.705 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.528469575000003, 2024-09-21T15:50:09.705 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047494153999999997, 2024-09-21T15:50:09.705 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:09.705 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:09.705 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:09.705 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:09.705 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:09.705 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64028 2024-09-21T15:50:09.705 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.705 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:09.705 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.705 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:09.706 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167504+0000", 2024-09-21T15:50:09.706 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.706 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.706 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.706 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:09.706 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167504+0000", 2024-09-21T15:50:09.706 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.706 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.706 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.706 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:09.706 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167504+0000", 2024-09-21T15:50:09.706 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:09.706 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.706 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.706 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:09.707 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167506+0000", 2024-09-21T15:50:09.707 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0600000000000002e-06 2024-09-21T15:50:09.707 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.707 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.707 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:09.707 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167512+0000", 2024-09-21T15:50:09.707 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.083e-06 2024-09-21T15:50:09.707 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.707 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.707 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:09.707 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167517+0000", 2024-09-21T15:50:09.707 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7650000000000001e-06 2024-09-21T15:50:09.707 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.707 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.707 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:09.707 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.214835+0000", 2024-09-21T15:50:09.708 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047318067999999998 2024-09-21T15:50:09.708 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.708 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.708 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:09.708 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.214916+0000", 2024-09-21T15:50:09.708 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1815000000000002e-05 2024-09-21T15:50:09.708 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.708 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.708 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:09.708 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.214998+0000", 2024-09-21T15:50:09.708 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1642000000000002e-05 2024-09-21T15:50:09.708 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.708 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:09.708 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.709 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.709 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.709 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64029 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1050624~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:09.709 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167521+0000", 2024-09-21T15:50:09.709 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.528452664, 2024-09-21T15:50:09.709 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047649915000000001, 2024-09-21T15:50:09.709 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:09.709 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:09.709 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:09.709 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:09.709 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:09.709 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64029 2024-09-21T15:50:09.709 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.709 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:09.709 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.709 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:09.709 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167521+0000", 2024-09-21T15:50:09.710 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.710 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.710 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.710 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:09.710 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167521+0000", 2024-09-21T15:50:09.710 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.710 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.710 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.710 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:09.710 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167521+0000", 2024-09-21T15:50:09.710 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:09.710 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.710 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.710 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:09.711 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167523+0000", 2024-09-21T15:50:09.711 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9810000000000002e-06 2024-09-21T15:50:09.711 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.711 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.711 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:09.711 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167529+0000", 2024-09-21T15:50:09.711 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0900000000000001e-06 2024-09-21T15:50:09.711 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.711 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.711 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:09.711 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167533+0000", 2024-09-21T15:50:09.711 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5730000000000001e-06 2024-09-21T15:50:09.711 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.711 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.711 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:09.712 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215031+0000", 2024-09-21T15:50:09.712 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047497893999999999 2024-09-21T15:50:09.712 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.712 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.712 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:09.712 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215099+0000", 2024-09-21T15:50:09.712 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8416000000000004e-05 2024-09-21T15:50:09.712 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.712 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.712 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:09.712 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215171+0000", 2024-09-21T15:50:09.712 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1279000000000007e-05 2024-09-21T15:50:09.712 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.712 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:09.712 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.713 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.713 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.713 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64030 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1058816~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:09.713 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167537+0000", 2024-09-21T15:50:09.713 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.528436155000001, 2024-09-21T15:50:09.713 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047783816999999999, 2024-09-21T15:50:09.713 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:09.713 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:09.713 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:09.713 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:09.713 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:09.713 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64030 2024-09-21T15:50:09.713 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.713 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:09.713 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.713 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:09.714 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167537+0000", 2024-09-21T15:50:09.714 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.714 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.714 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.714 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:09.714 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167537+0000", 2024-09-21T15:50:09.714 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.714 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.714 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.715 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:09.715 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167537+0000", 2024-09-21T15:50:09.715 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:09.715 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.715 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.715 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:09.715 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167539+0000", 2024-09-21T15:50:09.715 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9700000000000002e-06 2024-09-21T15:50:09.715 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.715 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.715 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:09.715 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167545+0000", 2024-09-21T15:50:09.715 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8629999999999996e-06 2024-09-21T15:50:09.716 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.716 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.716 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:09.716 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167549+0000", 2024-09-21T15:50:09.716 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.386e-06 2024-09-21T15:50:09.716 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.716 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.716 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:09.716 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215197+0000", 2024-09-21T15:50:09.716 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047647241 2024-09-21T15:50:09.716 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.716 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.716 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:09.716 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215263+0000", 2024-09-21T15:50:09.717 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6295000000000001e-05 2024-09-21T15:50:09.717 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.717 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.717 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:09.717 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215321+0000", 2024-09-21T15:50:09.717 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8362000000000001e-05 2024-09-21T15:50:09.717 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.717 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:09.717 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.717 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.717 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.717 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64031 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1738752~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:09.717 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167554+0000", 2024-09-21T15:50:09.717 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.528419651999997, 2024-09-21T15:50:09.717 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047917855000000002, 2024-09-21T15:50:09.718 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:09.718 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:09.718 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:09.718 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:09.718 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:09.718 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64031 2024-09-21T15:50:09.718 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.718 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:09.718 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.718 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:09.718 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167554+0000", 2024-09-21T15:50:09.718 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.718 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.719 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.719 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:09.719 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167554+0000", 2024-09-21T15:50:09.719 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.719 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.719 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.719 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:09.719 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167554+0000", 2024-09-21T15:50:09.719 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:09.719 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.719 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.719 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:09.719 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167556+0000", 2024-09-21T15:50:09.719 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3870000000000002e-06 2024-09-21T15:50:09.719 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.720 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.720 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:09.720 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167562+0000", 2024-09-21T15:50:09.720 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9100000000000002e-06 2024-09-21T15:50:09.720 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.720 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.720 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:09.720 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167566+0000", 2024-09-21T15:50:09.720 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6029999999999998e-06 2024-09-21T15:50:09.720 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.720 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.720 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:09.720 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215346+0000", 2024-09-21T15:50:09.720 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.04777998 2024-09-21T15:50:09.720 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.721 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.721 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:09.721 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215412+0000", 2024-09-21T15:50:09.721 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5598000000000005e-05 2024-09-21T15:50:09.721 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.721 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.721 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:09.721 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215472+0000", 2024-09-21T15:50:09.721 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9728000000000001e-05 2024-09-21T15:50:09.721 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.721 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:09.721 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.721 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.721 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.721 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64032 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1067008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:09.722 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167673+0000", 2024-09-21T15:50:09.722 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.528300938999998, 2024-09-21T15:50:09.722 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047947667999999999, 2024-09-21T15:50:09.722 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:09.722 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:09.722 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:09.722 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:09.722 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:09.722 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64032 2024-09-21T15:50:09.722 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.722 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:09.722 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.722 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:09.722 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167673+0000", 2024-09-21T15:50:09.723 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.723 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.723 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.723 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:09.723 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167673+0000", 2024-09-21T15:50:09.723 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.723 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.723 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.723 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:09.723 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167672+0000", 2024-09-21T15:50:09.723 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:09.723 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.723 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.723 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:09.723 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167675+0000", 2024-09-21T15:50:09.724 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3780000000000001e-06 2024-09-21T15:50:09.724 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.724 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.724 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:09.724 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167685+0000", 2024-09-21T15:50:09.724 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3039999999999994e-06 2024-09-21T15:50:09.724 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.724 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.724 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:09.724 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167692+0000", 2024-09-21T15:50:09.724 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2559999999999996e-06 2024-09-21T15:50:09.724 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.724 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.724 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:09.724 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215495+0000", 2024-09-21T15:50:09.724 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047803267000000003 2024-09-21T15:50:09.725 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.725 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.725 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:09.725 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215560+0000", 2024-09-21T15:50:09.725 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4703999999999996e-05 2024-09-21T15:50:09.725 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.725 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.725 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:09.725 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215620+0000", 2024-09-21T15:50:09.725 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0328000000000002e-05 2024-09-21T15:50:09.725 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.725 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:09.725 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.725 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.725 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.725 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64033 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1075200~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:09.726 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167697+0000", 2024-09-21T15:50:09.726 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.528276366999997, 2024-09-21T15:50:09.726 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048082863000000003, 2024-09-21T15:50:09.726 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:09.726 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:09.726 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:09.726 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:09.726 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:09.726 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64033 2024-09-21T15:50:09.726 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.726 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:09.726 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.726 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:09.726 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167697+0000", 2024-09-21T15:50:09.727 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.727 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.727 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.727 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:09.727 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167697+0000", 2024-09-21T15:50:09.727 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.727 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.727 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.727 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:09.727 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167697+0000", 2024-09-21T15:50:09.727 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:09.727 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.727 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.727 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:09.727 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167699+0000", 2024-09-21T15:50:09.727 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5299999999999999e-06 2024-09-21T15:50:09.728 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.728 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.728 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:09.728 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167706+0000", 2024-09-21T15:50:09.728 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7490000000000001e-06 2024-09-21T15:50:09.728 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.728 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.728 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:09.728 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167712+0000", 2024-09-21T15:50:09.728 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1940000000000003e-06 2024-09-21T15:50:09.728 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.728 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.728 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:09.728 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215643+0000", 2024-09-21T15:50:09.728 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047930647999999999 2024-09-21T15:50:09.728 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.729 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.729 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:09.729 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215713+0000", 2024-09-21T15:50:09.729 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0488999999999995e-05 2024-09-21T15:50:09.729 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.729 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.729 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:09.729 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215780+0000", 2024-09-21T15:50:09.729 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6656999999999996e-05 2024-09-21T15:50:09.729 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.729 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:09.729 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.729 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.729 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.729 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64034 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1083392~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:09.729 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167717+0000", 2024-09-21T15:50:09.730 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.528256800999998, 2024-09-21T15:50:09.730 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048233316999999998, 2024-09-21T15:50:09.730 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:09.730 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:09.730 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:09.730 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:09.730 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:09.730 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64034 2024-09-21T15:50:09.730 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.730 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:09.730 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.730 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:09.730 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167717+0000", 2024-09-21T15:50:09.730 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.731 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.731 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.731 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:09.731 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167717+0000", 2024-09-21T15:50:09.731 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.731 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.731 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.731 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:09.731 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167716+0000", 2024-09-21T15:50:09.731 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:09.731 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.731 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.731 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:09.731 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167718+0000", 2024-09-21T15:50:09.732 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.165e-06 2024-09-21T15:50:09.732 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.732 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.732 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:09.732 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167725+0000", 2024-09-21T15:50:09.732 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3380000000000001e-06 2024-09-21T15:50:09.732 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.732 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.732 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:09.732 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167730+0000", 2024-09-21T15:50:09.732 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8010000000000003e-06 2024-09-21T15:50:09.732 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.732 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.732 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:09.733 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215807+0000", 2024-09-21T15:50:09.733 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048077823999999998 2024-09-21T15:50:09.733 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.733 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.733 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:09.733 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215884+0000", 2024-09-21T15:50:09.733 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6283000000000001e-05 2024-09-21T15:50:09.733 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.733 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.733 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:09.733 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215950+0000", 2024-09-21T15:50:09.733 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6290000000000004e-05 2024-09-21T15:50:09.733 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.733 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:09.733 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.734 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.734 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.734 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64035 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1091584~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:09.734 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:50:09.734 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.528239587999998, 2024-09-21T15:50:09.734 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048412117999999997, 2024-09-21T15:50:09.734 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:09.734 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:09.734 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:09.734 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:09.734 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:09.734 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64035 2024-09-21T15:50:09.734 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.734 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:09.735 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.735 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:09.735 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:50:09.735 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.735 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.735 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.735 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:09.735 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:50:09.735 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.735 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.735 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.735 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:09.735 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:50:09.735 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:09.735 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.736 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.736 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:09.736 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167746+0000", 2024-09-21T15:50:09.736 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2031000000000001e-05 2024-09-21T15:50:09.736 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.736 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.736 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:09.736 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167753+0000", 2024-09-21T15:50:09.736 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0620000000000003e-06 2024-09-21T15:50:09.736 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.736 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.736 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:09.736 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167759+0000", 2024-09-21T15:50:09.737 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.596e-06 2024-09-21T15:50:09.737 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.737 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.737 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:09.737 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215977+0000", 2024-09-21T15:50:09.737 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048217318000000002 2024-09-21T15:50:09.737 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.737 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.737 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:09.737 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216076+0000", 2024-09-21T15:50:09.737 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9846000000000006e-05 2024-09-21T15:50:09.737 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.737 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.737 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:09.738 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216146+0000", 2024-09-21T15:50:09.738 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9578000000000004e-05 2024-09-21T15:50:09.738 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.738 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:09.738 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.738 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.738 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.738 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64036 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1099776~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:09.738 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167853+0000", 2024-09-21T15:50:09.738 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.528120975, 2024-09-21T15:50:09.738 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048450675999999998, 2024-09-21T15:50:09.738 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:09.738 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:09.738 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:09.739 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:09.739 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:09.739 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64036 2024-09-21T15:50:09.739 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.739 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:09.739 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.739 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:09.739 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167853+0000", 2024-09-21T15:50:09.739 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.739 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.739 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.739 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:09.739 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167853+0000", 2024-09-21T15:50:09.739 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.739 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.740 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.740 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:09.740 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167852+0000", 2024-09-21T15:50:09.740 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:09.740 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.740 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.740 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:09.740 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167855+0000", 2024-09-21T15:50:09.740 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5640000000000001e-06 2024-09-21T15:50:09.740 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.740 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.740 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:09.740 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167864+0000", 2024-09-21T15:50:09.740 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8140000000000007e-06 2024-09-21T15:50:09.740 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.740 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.741 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:09.741 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167871+0000", 2024-09-21T15:50:09.741 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5729999999999996e-06 2024-09-21T15:50:09.741 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.741 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.741 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:09.741 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216174+0000", 2024-09-21T15:50:09.741 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048303437999999997 2024-09-21T15:50:09.741 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.741 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.741 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:09.741 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216250+0000", 2024-09-21T15:50:09.741 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5661999999999995e-05 2024-09-21T15:50:09.741 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.741 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.742 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:09.742 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216303+0000", 2024-09-21T15:50:09.742 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3273000000000003e-05 2024-09-21T15:50:09.742 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.742 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:09.742 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.742 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.742 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.742 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64037 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1771520~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:09.742 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:50:09.742 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.528097432000003, 2024-09-21T15:50:09.742 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048588771000000003, 2024-09-21T15:50:09.742 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:09.742 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:09.742 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:09.743 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:09.743 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:09.743 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64037 2024-09-21T15:50:09.743 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.743 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:09.743 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.743 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:09.743 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:50:09.743 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.743 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.743 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.743 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:09.743 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:50:09.743 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.743 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.744 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.744 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:09.744 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:50:09.744 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:09.744 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.744 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.744 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:09.744 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167878+0000", 2024-09-21T15:50:09.744 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3e-06 2024-09-21T15:50:09.744 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.744 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.744 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:09.744 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167884+0000", 2024-09-21T15:50:09.744 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3269999999999997e-06 2024-09-21T15:50:09.744 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.745 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.745 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:09.745 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167889+0000", 2024-09-21T15:50:09.745 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9269999999999996e-06 2024-09-21T15:50:09.745 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.745 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.745 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:09.745 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216331+0000", 2024-09-21T15:50:09.745 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048442102000000001 2024-09-21T15:50:09.745 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.745 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.745 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:09.745 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216393+0000", 2024-09-21T15:50:09.745 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1132999999999999e-05 2024-09-21T15:50:09.745 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.745 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.746 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:09.746 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216465+0000", 2024-09-21T15:50:09.746 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2339e-05 2024-09-21T15:50:09.746 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.746 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:09.746 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.746 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.746 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.746 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64038 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1779712~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:09.746 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:50:09.746 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.528079601000002, 2024-09-21T15:50:09.746 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048749795999999998, 2024-09-21T15:50:09.746 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:09.746 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:09.747 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:09.747 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:09.747 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:09.747 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64038 2024-09-21T15:50:09.747 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.747 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:09.747 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.747 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:09.747 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:50:09.747 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.747 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.747 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.747 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:09.747 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:50:09.747 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.748 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.748 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.748 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:09.748 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:50:09.748 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:09.748 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.748 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.748 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:09.748 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167896+0000", 2024-09-21T15:50:09.748 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2419999999999999e-06 2024-09-21T15:50:09.748 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.748 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.748 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:09.748 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167902+0000", 2024-09-21T15:50:09.749 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0859999999999998e-06 2024-09-21T15:50:09.749 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.749 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.749 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:09.749 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167907+0000", 2024-09-21T15:50:09.749 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9679999999999997e-06 2024-09-21T15:50:09.749 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.749 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.749 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:09.749 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216506+0000", 2024-09-21T15:50:09.749 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048599374000000001 2024-09-21T15:50:09.749 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.749 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.749 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:09.749 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216579+0000", 2024-09-21T15:50:09.750 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2801999999999999e-05 2024-09-21T15:50:09.750 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.750 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.750 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:09.750 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216644+0000", 2024-09-21T15:50:09.750 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4715000000000006e-05 2024-09-21T15:50:09.750 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.750 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:09.750 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.750 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.750 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.750 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64039 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1107968~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:09.750 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:50:09.750 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.528062245999998, 2024-09-21T15:50:09.750 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048911370000000003, 2024-09-21T15:50:09.751 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:09.751 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:09.751 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:09.751 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:09.751 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:09.751 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64039 2024-09-21T15:50:09.751 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.751 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:09.751 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.751 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:09.751 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:50:09.751 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.751 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.751 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.752 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:09.752 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:50:09.752 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.752 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.752 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.752 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:09.752 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:50:09.752 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:09.752 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.752 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.752 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:09.752 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167913+0000", 2024-09-21T15:50:09.752 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0839999999999999e-06 2024-09-21T15:50:09.752 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.753 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.753 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:09.753 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167921+0000", 2024-09-21T15:50:09.753 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8310000000000001e-06 2024-09-21T15:50:09.753 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.753 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.753 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:09.753 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167926+0000", 2024-09-21T15:50:09.753 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0270000000000003e-06 2024-09-21T15:50:09.753 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.753 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.753 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:09.753 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216680+0000", 2024-09-21T15:50:09.753 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048753890000000001 2024-09-21T15:50:09.753 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.754 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.754 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:09.754 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216757+0000", 2024-09-21T15:50:09.754 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6848999999999996e-05 2024-09-21T15:50:09.754 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.754 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.754 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:09.754 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216823+0000", 2024-09-21T15:50:09.754 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6068000000000005e-05 2024-09-21T15:50:09.754 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.754 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:09.754 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.754 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.754 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.754 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64040 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1787904~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:09.755 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167931+0000", 2024-09-21T15:50:09.755 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.528043013999998, 2024-09-21T15:50:09.755 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049010561000000001, 2024-09-21T15:50:09.755 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:09.755 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:09.755 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:09.755 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:09.755 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:09.755 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64040 2024-09-21T15:50:09.755 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.755 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:09.755 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.755 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:09.755 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167931+0000", 2024-09-21T15:50:09.755 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.756 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.756 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.756 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:09.756 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167931+0000", 2024-09-21T15:50:09.756 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.756 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.756 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.756 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:09.756 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167930+0000", 2024-09-21T15:50:09.756 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:09.756 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.756 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.756 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:09.756 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167932+0000", 2024-09-21T15:50:09.756 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.198e-06 2024-09-21T15:50:09.756 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.757 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.757 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:09.757 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167938+0000", 2024-09-21T15:50:09.757 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.105e-06 2024-09-21T15:50:09.757 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.757 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.757 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:09.757 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167943+0000", 2024-09-21T15:50:09.757 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9429999999999999e-06 2024-09-21T15:50:09.757 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.757 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.757 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:09.757 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216848+0000", 2024-09-21T15:50:09.757 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048905200000000003 2024-09-21T15:50:09.757 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.757 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.757 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:09.758 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216903+0000", 2024-09-21T15:50:09.758 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4793999999999997e-05 2024-09-21T15:50:09.758 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.758 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.758 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:09.758 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216941+0000", 2024-09-21T15:50:09.758 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7790999999999998e-05 2024-09-21T15:50:09.758 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.758 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:09.758 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.758 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.758 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.758 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64041 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1796096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:09.758 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:50:09.759 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.528026042999997, 2024-09-21T15:50:09.759 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049181457999999997, 2024-09-21T15:50:09.759 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:09.759 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:09.759 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:09.759 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:09.759 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:09.759 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64041 2024-09-21T15:50:09.759 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.759 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:09.759 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.759 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:09.759 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:50:09.759 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.759 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.760 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.760 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:09.760 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:50:09.760 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.760 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.760 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.760 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:09.760 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:50:09.760 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:09.760 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.760 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.760 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:09.760 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167949+0000", 2024-09-21T15:50:09.760 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0200000000000001e-06 2024-09-21T15:50:09.760 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.760 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.760 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:09.761 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167955+0000", 2024-09-21T15:50:09.761 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5990000000000001e-06 2024-09-21T15:50:09.761 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.761 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.761 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:09.761 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167959+0000", 2024-09-21T15:50:09.761 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5589999999999999e-06 2024-09-21T15:50:09.761 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.761 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.761 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:09.761 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216972+0000", 2024-09-21T15:50:09.761 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049012574000000003 2024-09-21T15:50:09.761 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.761 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.761 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:09.761 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217057+0000", 2024-09-21T15:50:09.761 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4820000000000004e-05 2024-09-21T15:50:09.762 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.762 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.762 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:09.762 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217129+0000", 2024-09-21T15:50:09.762 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2186000000000003e-05 2024-09-21T15:50:09.762 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.762 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:09.762 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.762 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.762 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.762 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64042 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1804288~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:09.762 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167964+0000", 2024-09-21T15:50:09.762 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.528009752000003, 2024-09-21T15:50:09.762 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049333372, 2024-09-21T15:50:09.763 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:09.763 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:09.763 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:09.763 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:09.763 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:09.763 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64042 2024-09-21T15:50:09.763 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.763 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:09.763 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.763 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:09.763 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167964+0000", 2024-09-21T15:50:09.763 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.763 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.764 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.764 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:09.764 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167964+0000", 2024-09-21T15:50:09.764 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.764 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.764 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.764 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:09.764 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167963+0000", 2024-09-21T15:50:09.764 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:09.764 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.764 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.764 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:09.764 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167966+0000", 2024-09-21T15:50:09.764 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1119999999999999e-06 2024-09-21T15:50:09.764 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.765 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.765 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:09.765 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167975+0000", 2024-09-21T15:50:09.765 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1130000000000007e-06 2024-09-21T15:50:09.765 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.765 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.765 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:09.765 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167980+0000", 2024-09-21T15:50:09.765 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9599999999999999e-06 2024-09-21T15:50:09.765 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.765 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.765 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:09.765 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217158+0000", 2024-09-21T15:50:09.766 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049178569999999998 2024-09-21T15:50:09.766 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.766 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.766 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:09.766 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217232+0000", 2024-09-21T15:50:09.766 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3589000000000003e-05 2024-09-21T15:50:09.766 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.766 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.766 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:09.766 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217297+0000", 2024-09-21T15:50:09.766 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5339999999999994e-05 2024-09-21T15:50:09.766 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.766 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:09.766 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.767 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.767 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.767 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64043 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1812480~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:09.767 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.168085+0000", 2024-09-21T15:50:09.767 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.527888685000001, 2024-09-21T15:50:09.767 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049365157999999999, 2024-09-21T15:50:09.767 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:09.767 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:09.767 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:09.767 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:09.767 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:09.767 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64043 2024-09-21T15:50:09.767 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.767 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:09.768 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.768 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:09.768 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168085+0000", 2024-09-21T15:50:09.768 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.768 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.768 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.768 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:09.768 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168085+0000", 2024-09-21T15:50:09.768 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.768 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.768 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.768 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:09.768 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168084+0000", 2024-09-21T15:50:09.768 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:09.769 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.769 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.769 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:09.769 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168088+0000", 2024-09-21T15:50:09.769 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5420000000000001e-06 2024-09-21T15:50:09.769 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.769 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.769 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:09.769 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168097+0000", 2024-09-21T15:50:09.769 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5410000000000006e-06 2024-09-21T15:50:09.769 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.769 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.769 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:09.769 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168104+0000", 2024-09-21T15:50:09.769 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6719999999999998e-06 2024-09-21T15:50:09.770 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.770 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.770 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:09.770 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217323+0000", 2024-09-21T15:50:09.770 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049218857999999997 2024-09-21T15:50:09.770 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.770 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.770 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:09.770 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217397+0000", 2024-09-21T15:50:09.770 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3932999999999998e-05 2024-09-21T15:50:09.770 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.770 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.770 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:09.770 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217450+0000", 2024-09-21T15:50:09.770 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3168000000000003e-05 2024-09-21T15:50:09.771 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.771 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:09.771 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.771 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.771 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.771 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64044 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1820672~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:09.771 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.170036+0000", 2024-09-21T15:50:09.771 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.525937366000001, 2024-09-21T15:50:09.771 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047578069000000001, 2024-09-21T15:50:09.771 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:09.771 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:09.771 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:09.771 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:09.772 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:09.772 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64044 2024-09-21T15:50:09.772 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.772 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:09.772 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.772 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:09.772 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170036+0000", 2024-09-21T15:50:09.772 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.772 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.772 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.772 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:09.772 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170036+0000", 2024-09-21T15:50:09.772 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.772 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.773 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.773 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:09.773 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168109+0000", 2024-09-21T15:50:09.773 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9980721 2024-09-21T15:50:09.773 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.773 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.773 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:09.773 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170040+0000", 2024-09-21T15:50:09.773 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0019318079999999999 2024-09-21T15:50:09.773 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.773 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.773 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:09.773 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170051+0000", 2024-09-21T15:50:09.773 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0220999999999999e-05 2024-09-21T15:50:09.773 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.773 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.774 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:09.774 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170058+0000", 2024-09-21T15:50:09.774 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0110000000000003e-06 2024-09-21T15:50:09.774 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.774 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.774 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:09.774 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217478+0000", 2024-09-21T15:50:09.774 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047420334000000001 2024-09-21T15:50:09.774 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.774 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.774 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:09.774 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217547+0000", 2024-09-21T15:50:09.774 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9416000000000001e-05 2024-09-21T15:50:09.774 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.774 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.775 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:09.775 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217614+0000", 2024-09-21T15:50:09.775 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6909000000000006e-05 2024-09-21T15:50:09.775 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.775 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:09.775 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.775 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.775 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.775 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64045 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1828864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:09.775 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.170064+0000", 2024-09-21T15:50:09.775 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.525909327000001, 2024-09-21T15:50:09.775 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047709252000000001, 2024-09-21T15:50:09.775 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:09.775 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:09.775 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:09.775 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:09.776 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:09.776 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64045 2024-09-21T15:50:09.777 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.777 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:09.777 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.777 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:09.777 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170064+0000", 2024-09-21T15:50:09.777 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.777 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.778 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.778 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:09.778 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170064+0000", 2024-09-21T15:50:09.778 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.778 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.778 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.778 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:09.778 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170064+0000", 2024-09-21T15:50:09.778 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:09.778 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.778 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.778 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:09.778 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170066+0000", 2024-09-21T15:50:09.778 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8109999999999999e-06 2024-09-21T15:50:09.778 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.779 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.779 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:09.779 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170073+0000", 2024-09-21T15:50:09.779 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5570000000000001e-06 2024-09-21T15:50:09.779 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.779 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.779 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:09.779 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170078+0000", 2024-09-21T15:50:09.779 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9910000000000002e-06 2024-09-21T15:50:09.779 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.779 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.779 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:09.779 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217640+0000", 2024-09-21T15:50:09.779 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047561840000000001 2024-09-21T15:50:09.779 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.780 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.780 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:09.780 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217710+0000", 2024-09-21T15:50:09.780 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9967000000000003e-05 2024-09-21T15:50:09.780 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.780 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.780 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:09.780 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217773+0000", 2024-09-21T15:50:09.780 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3638000000000003e-05 2024-09-21T15:50:09.780 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.780 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:09.780 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.780 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.780 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.781 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64356 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 3076096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:09.781 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.183065+0000", 2024-09-21T15:50:09.781 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.512908195999998, 2024-09-21T15:50:09.781 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047484923999999998, 2024-09-21T15:50:09.781 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:09.781 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:09.781 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:09.781 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:09.781 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:09.781 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64356 2024-09-21T15:50:09.781 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.781 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:09.781 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.781 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:09.781 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.183065+0000", 2024-09-21T15:50:09.782 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.782 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.782 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.782 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:09.782 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.183065+0000", 2024-09-21T15:50:09.782 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:09.782 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.782 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.782 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:09.782 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.182922+0000", 2024-09-21T15:50:09.782 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999856 2024-09-21T15:50:09.782 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.782 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.782 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:09.782 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.183069+0000", 2024-09-21T15:50:09.783 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000147097 2024-09-21T15:50:09.783 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.783 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.783 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:09.783 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.183078+0000", 2024-09-21T15:50:09.783 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.9099999999999994e-06 2024-09-21T15:50:09.783 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.783 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.783 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:09.783 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.183085+0000", 2024-09-21T15:50:09.783 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0670000000000003e-06 2024-09-21T15:50:09.783 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.783 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.783 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:09.784 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.230377+0000", 2024-09-21T15:50:09.784 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047292698000000001 2024-09-21T15:50:09.784 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.784 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.784 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:09.784 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.230458+0000", 2024-09-21T15:50:09.784 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0980000000000001e-05 2024-09-21T15:50:09.784 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:09.784 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:09.784 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:09.784 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.230550+0000", 2024-09-21T15:50:09.784 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1954999999999996e-05 2024-09-21T15:50:09.784 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.784 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:09.784 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.784 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:09.785 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:09.785 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:09.785 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:50:09.786 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:50:09.787 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:09.785+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:09.886 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:09.885+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:09.893 INFO:teuthology.orchestra.run.smithi089.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T15:50:09.895 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:09.895 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:09.896 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:50:09.896 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:09.905 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:50:09.911 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:50:09.911 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:50:09.928 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:50:09.928 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:50:09.944 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:50:09.944 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:50:09.961 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:50:09.961 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:50:09.977 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:50:09.978 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:50:09.988 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:09.987+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:09.994 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:50:09.994 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:50:10.009 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:10.009 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:10.010 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:50:10.010 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:50:10.013 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T15:50:10.014 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T15:50:10.019 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:50:10.027 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is set 2024-09-21T15:50:10.081 INFO:teuthology.orchestra.run.smithi089.stdout:truncating image image_client.0-clone31 from 0xc597fe1 (overlap 0x124ecc) to 0x1d4ae 2024-09-21T15:50:10.088 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:10.087+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:10.135 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:10.135 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:10.142 INFO:teuthology.orchestra.run.smithi089.stdout:checking clone #30, image image_client.0-clone30 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent31 2024-09-21T15:50:10.145 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:50:10.190 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:10.188+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:10.272 INFO:teuthology.orchestra.run.smithi089.stdout:4702 read 0x8be9ace thru 0x8bf3ad0 (0xa003 bytes) 2024-09-21T15:50:10.273 INFO:teuthology.orchestra.run.smithi089.stdout:4704 writesame 0x1692ea7 thru 0x169956e data_size 0x5c(0x66c8 bytes) 2024-09-21T15:50:10.280 INFO:teuthology.orchestra.run.smithi089.stdout:4705 writesame 0x54b031 thru 0x556345 data_size 0x35(0xb315 bytes) 2024-09-21T15:50:10.291 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:10.289+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:10.294 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:10.294 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:10.297 INFO:teuthology.orchestra.run.smithi089.stdout:4706 trunc from 0xc597fe1 to 0x226dac2 2024-09-21T15:50:10.305 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:50:10.308 INFO:teuthology.orchestra.run.smithi089.stdout:4708 punch from 0xcb76a0 to 0xcc630e, (0xec6e bytes) 2024-09-21T15:50:10.329 INFO:teuthology.orchestra.run.smithi089.stdout:4711 writesame 0xecfa14a thru 0xed00cc6 data_size 0x1(0x6b7d bytes) 2024-09-21T15:50:10.329 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:10.329 INFO:teuthology.orchestra.run.smithi089.stdout:4715 write 0x6b97d1f thru 0x6ba47cc (0xcaae bytes) 2024-09-21T15:50:10.333 INFO:teuthology.orchestra.run.smithi089.stdout:4716 punch from 0xd766740 to 0xd76f3d4, (0x8c94 bytes) 2024-09-21T15:50:10.333 INFO:teuthology.orchestra.run.smithi089.stdout:4718 read 0x951a3c7 thru 0x951d5e6 (0x3220 bytes) 2024-09-21T15:50:10.344 INFO:teuthology.orchestra.run.smithi089.stdout:4721 trunc from 0xed00cc7 to 0x5276c55 2024-09-21T15:50:10.354 INFO:teuthology.orchestra.run.smithi089.stdout:4723 write 0x394201a thru 0x394c13a (0xa121 bytes) 2024-09-21T15:50:10.375 INFO:teuthology.orchestra.run.smithi089.stdout:4724 writesame 0xe8613b7 thru 0xe86b5c3 data_size 0x1(0xa20d bytes) 2024-09-21T15:50:10.375 INFO:teuthology.orchestra.run.smithi089.stdout:4725 write 0xd5b32b5 thru 0xd5c2845 (0xf591 bytes) 2024-09-21T15:50:10.378 INFO:teuthology.orchestra.run.smithi089.stdout:4727 punch from 0xba9c896 to 0xbaa5bc4, (0x932e bytes) 2024-09-21T15:50:10.378 INFO:teuthology.orchestra.run.smithi089.stdout:4728 read 0x1facaf1 thru 0x1fb30b6 (0x65c6 bytes) 2024-09-21T15:50:10.379 INFO:teuthology.orchestra.run.smithi089.stdout:4730 write 0x1549cd2 thru 0x1555ce3 (0xc012 bytes) 2024-09-21T15:50:10.382 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:10.382 INFO:teuthology.orchestra.run.smithi089.stdout:4735 writesame 0xb6ac3a thru 0xb7880e data_size 0x75(0xdbd5 bytes) 2024-09-21T15:50:10.386 INFO:teuthology.orchestra.run.smithi089.stdout:4737 read 0xcbd4692 thru 0xcbd8c5c (0x45cb bytes) 2024-09-21T15:50:10.386 INFO:teuthology.orchestra.run.smithi089.stdout:4738 read 0x2b174b2 thru 0x2b214d1 (0xa020 bytes) 2024-09-21T15:50:10.386 INFO:teuthology.orchestra.run.smithi089.stdout:4739 read 0xde35a6e thru 0xde3f728 (0x9cbb bytes) 2024-09-21T15:50:10.386 INFO:teuthology.orchestra.run.smithi089.stdout:4741 writesame 0x62a905e thru 0x62b0f39 data_size 0x2e(0x7edc bytes) 2024-09-21T15:50:10.389 INFO:teuthology.orchestra.run.smithi089.stdout:4743 read 0x43c2389 thru 0x43c938e (0x7006 bytes) 2024-09-21T15:50:10.389 INFO:teuthology.orchestra.run.smithi089.stdout:4745 read 0x2a8c944 thru 0x2a99ff0 (0xd6ad bytes) 2024-09-21T15:50:10.391 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:10.390+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:10.399 INFO:teuthology.orchestra.run.smithi089.stdout:4747 trunc from 0xe86b5c4 to 0x64ce989 2024-09-21T15:50:10.408 INFO:teuthology.orchestra.run.smithi089.stdout:4750 read 0xbc7bce thru 0xbcbe2f (0x4262 bytes) 2024-09-21T15:50:10.415 INFO:teuthology.orchestra.run.smithi089.stdout:4751 trunc from 0x64ce989 to 0xa630df 2024-09-21T15:50:10.426 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:10.426 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:10.436 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:50:10.441 INFO:teuthology.orchestra.run.smithi089.stdout:4752 punch from 0x9d28d6 to 0x9d38ff, (0x1029 bytes) 2024-09-21T15:50:10.441 INFO:teuthology.orchestra.run.smithi089.stdout:4753 read 0x3e8768 thru 0x3f6be2 (0xe47b bytes) 2024-09-21T15:50:10.448 INFO:teuthology.orchestra.run.smithi089.stdout:4755 writesame 0x1b2c641 thru 0x1b36bf8 data_size 0x4(0xa5b8 bytes) 2024-09-21T15:50:10.460 INFO:teuthology.orchestra.run.smithi089.stdout:4756 write 0x6da2920 thru 0x6db0c28 (0xe309 bytes) 2024-09-21T15:50:10.463 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:10.463 INFO:teuthology.orchestra.run.smithi089.stdout:4758 punch from 0xc5a957 to 0xc656d1, (0xad7a bytes) 2024-09-21T15:50:10.463 INFO:teuthology.orchestra.run.smithi089.stdout:4760 read 0x3522ea3 thru 0x352efb6 (0xc114 bytes) 2024-09-21T15:50:10.463 INFO:teuthology.orchestra.run.smithi089.stdout:4761 writesame 0x2001f24 thru 0x200cd31 data_size 0x1(0xae0e bytes) 2024-09-21T15:50:10.487 INFO:teuthology.orchestra.run.smithi089.stdout:4762 writesame 0x43c1983 thru 0x43cb3b9 data_size 0x25(0x9a37 bytes) 2024-09-21T15:50:10.491 INFO:teuthology.orchestra.run.smithi089.stdout:4763 punch from 0x23ac959 to 0x23af9f7, (0x309e bytes) 2024-09-21T15:50:10.491 INFO:teuthology.orchestra.run.smithi089.stdout:4764 punch from 0x45df83c to 0x45eec81, (0xf445 bytes) 2024-09-21T15:50:10.491 INFO:teuthology.orchestra.run.smithi089.stdout:4765 read 0x25e221 thru 0x264718 (0x64f8 bytes) 2024-09-21T15:50:10.492 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:10.492 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:10.492 INFO:teuthology.orchestra.run.smithi089.stdout:4768 write 0x3b4c1cd thru 0x3b5bfc8 (0xfdfc bytes) 2024-09-21T15:50:10.492 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:10.491+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:10.494 INFO:teuthology.orchestra.run.smithi089.stdout:4769 read 0x46fd655 thru 0x4703064 (0x5a10 bytes) 2024-09-21T15:50:10.494 INFO:teuthology.orchestra.run.smithi089.stdout:4771 read 0x692c103 thru 0x6937f40 (0xbe3e bytes) 2024-09-21T15:50:10.495 INFO:teuthology.orchestra.run.smithi089.stdout:4773 trunc from 0x6db0c29 to 0x7c0d4b8 2024-09-21T15:50:10.508 INFO:teuthology.orchestra.run.smithi089.stdout:4775 writesame 0x9edcad5 thru 0x9ee64b7 data_size 0x1(0x99e3 bytes) 2024-09-21T15:50:10.508 INFO:teuthology.orchestra.run.smithi089.stdout:4776 read 0x9761eeb thru 0x976a682 (0x8798 bytes) 2024-09-21T15:50:10.515 INFO:teuthology.orchestra.run.smithi089.stdout:4777 write 0xb6e6308 thru 0xb6f2959 (0xc652 bytes) 2024-09-21T15:50:10.521 INFO:teuthology.orchestra.run.smithi089.stdout:4779 writesame 0x140dcef thru 0x1416064 data_size 0x6(0x8376 bytes) 2024-09-21T15:50:10.526 INFO:teuthology.orchestra.run.smithi089.stdout:4780 write 0xaeb1640 thru 0xaeb6ba0 (0x5561 bytes) 2024-09-21T15:50:10.528 INFO:teuthology.orchestra.run.smithi089.stdout:4781 write 0x15c7e42 thru 0x15ceea1 (0x7060 bytes) 2024-09-21T15:50:10.531 INFO:teuthology.orchestra.run.smithi089.stdout:4783 write 0x260ff2 thru 0x261023 (0x32 bytes) 2024-09-21T15:50:10.538 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:10.538 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:10.538 INFO:teuthology.orchestra.run.smithi089.stdout:4786 read 0x3521f74 thru 0x3529602 (0x768f bytes) 2024-09-21T15:50:10.538 INFO:teuthology.orchestra.run.smithi089.stdout:4787 writesame 0x9a2aea1 thru 0x9a38cf8 data_size 0x14(0xde58 bytes) 2024-09-21T15:50:10.541 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:10.553 INFO:teuthology.orchestra.run.smithi089.stdout:4792 write 0xec980d2 thru 0xec99af7 (0x1a26 bytes) 2024-09-21T15:50:10.556 INFO:teuthology.orchestra.run.smithi089.stdout:4793 write 0xddb1f70 thru 0xddbd566 (0xb5f7 bytes) 2024-09-21T15:50:10.560 INFO:teuthology.orchestra.run.smithi089.stdout:4795 read 0xd79b9fa thru 0xd7a7ab2 (0xc0b9 bytes) 2024-09-21T15:50:10.560 INFO:teuthology.orchestra.run.smithi089.stdout:4796 write 0x7b3ea03 thru 0x7b4af47 (0xc545 bytes) 2024-09-21T15:50:10.560 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:10.560 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:10.572 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:50:10.580 INFO:teuthology.orchestra.run.smithi089.stdout:4797 trunc from 0xec99af8 to 0x3c80c9d 2024-09-21T15:50:10.598 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:10.598 INFO:teuthology.orchestra.run.smithi089.stdout:4803 writesame 0x37deaa0 thru 0x37e81a9 data_size 0x1(0x970a bytes) 2024-09-21T15:50:10.598 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:10.597+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:10.617 INFO:teuthology.orchestra.run.smithi089.stdout:4805 read 0x2f252d5 thru 0x2f2e3d7 (0x9103 bytes) 2024-09-21T15:50:10.629 INFO:teuthology.orchestra.run.smithi089.stdout:4806 writesame 0x9a08587 thru 0x9a17607 data_size 0x3(0xf081 bytes) 2024-09-21T15:50:10.629 INFO:teuthology.orchestra.run.smithi089.stdout:4807 read 0x9373006 thru 0x9378092 (0x508d bytes) 2024-09-21T15:50:10.629 INFO:teuthology.orchestra.run.smithi089.stdout:4808 write 0x57001ca thru 0x570c432 (0xc269 bytes) 2024-09-21T15:50:10.642 INFO:teuthology.orchestra.run.smithi089.stdout:4813 trunc from 0x9a17608 to 0xf8460b 2024-09-21T15:50:10.668 INFO:teuthology.orchestra.run.smithi089.stdout:4815 write 0x7dea112 thru 0x7df98c4 (0xf7b3 bytes) 2024-09-21T15:50:10.674 INFO:teuthology.orchestra.run.smithi089.stdout:4816 read 0x5364e thru 0x5a57f (0x6f32 bytes) 2024-09-21T15:50:10.674 INFO:teuthology.orchestra.run.smithi089.stdout:4817 writesame 0x2e9d823 thru 0x2eab4e7 data_size 0x1(0xdcc5 bytes) 2024-09-21T15:50:10.692 INFO:teuthology.orchestra.run.smithi089.stdout:4818 punch from 0x7152ffd to 0x71573d1, (0x43d4 bytes) 2024-09-21T15:50:10.693 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:10.692+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:10.694 INFO:teuthology.orchestra.run.smithi089.stdout:4819 trunc from 0x7df98c5 to 0x8c4c43b 2024-09-21T15:50:10.694 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:10.694 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:10.698 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:10.698 INFO:teuthology.orchestra.run.smithi089.stdout:4822 read 0x19dea37 thru 0x19deed0 (0x49a bytes) 2024-09-21T15:50:10.698 INFO:teuthology.orchestra.run.smithi089.stdout:4824 trunc from 0x8c4c43b to 0x8f2ff89 2024-09-21T15:50:10.703 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:50:10.704 INFO:teuthology.orchestra.run.smithi089.stdout:4825 read 0xcc165e thru 0xcce95c (0xd2ff bytes) 2024-09-21T15:50:10.704 INFO:teuthology.orchestra.run.smithi089.stdout:4826 read 0x25348de thru 0x253e8cd (0x9ff0 bytes) 2024-09-21T15:50:10.704 INFO:teuthology.orchestra.run.smithi089.stdout:4829 punch from 0x3c58f08 to 0x3c6083b, (0x7933 bytes) 2024-09-21T15:50:10.704 INFO:teuthology.orchestra.run.smithi089.stdout:4830 write 0x1c437a0 thru 0x1c4fd65 (0xc5c6 bytes) 2024-09-21T15:50:10.706 INFO:teuthology.orchestra.run.smithi089.stdout:4831 writesame 0x2eea512 thru 0x2eef116 data_size 0xd(0x4c05 bytes) 2024-09-21T15:50:10.717 INFO:teuthology.orchestra.run.smithi089.stdout:4832 writesame 0xaf27a8b thru 0xaf31f46 data_size 0x1a(0xa4bc bytes) 2024-09-21T15:50:10.717 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:10.718 INFO:teuthology.orchestra.run.smithi089.stdout:4834 read 0x2b6815d thru 0x2b73094 (0xaf38 bytes) 2024-09-21T15:50:10.718 INFO:teuthology.orchestra.run.smithi089.stdout:4835 write 0x8d903fc thru 0x8d93c77 (0x387c bytes) 2024-09-21T15:50:10.721 INFO:teuthology.orchestra.run.smithi089.stdout:4836 punch from 0x93705ef to 0x9370dad, (0x7be bytes) 2024-09-21T15:50:10.721 INFO:teuthology.orchestra.run.smithi089.stdout:4837 punch from 0x65dfa34 to 0x65e353b, (0x3b07 bytes) 2024-09-21T15:50:10.721 INFO:teuthology.orchestra.run.smithi089.stdout:4838 read 0xa5f15f thru 0xa6906b (0x9f0d bytes) 2024-09-21T15:50:10.721 INFO:teuthology.orchestra.run.smithi089.stdout:4839 write 0x8c1eb thru 0x9a9a9 (0xe7bf bytes) 2024-09-21T15:50:10.733 INFO:teuthology.orchestra.run.smithi089.stdout:4841 trunc from 0xaf31f47 to 0x3f7e832 2024-09-21T15:50:10.756 INFO:teuthology.orchestra.run.smithi089.stdout:4842 write 0xb6c79c0 thru 0xb6d09ae (0x8fef bytes) 2024-09-21T15:50:10.768 INFO:teuthology.orchestra.run.smithi089.stdout:4843 writesame 0xe91c865 thru 0xe923648 data_size 0x1a(0x6de4 bytes) 2024-09-21T15:50:10.768 INFO:teuthology.orchestra.run.smithi089.stdout:4844 write 0xd3d825a thru 0xd3df042 (0x6de9 bytes) 2024-09-21T15:50:10.771 INFO:teuthology.orchestra.run.smithi089.stdout:4846 write 0x9f1b693 thru 0x9f221a6 (0x6b14 bytes) 2024-09-21T15:50:10.778 INFO:teuthology.orchestra.run.smithi089.stdout:4848 read 0xa68e8ab thru 0xa6983e1 (0x9b37 bytes) 2024-09-21T15:50:10.778 INFO:teuthology.orchestra.run.smithi089.stdout:4849 punch from 0x9a8df2f to 0x9a9842c, (0xa4fd bytes) 2024-09-21T15:50:10.778 INFO:teuthology.orchestra.run.smithi089.stdout:4850 punch from 0xa18c696 to 0xa195987, (0x92f1 bytes) 2024-09-21T15:50:10.778 INFO:teuthology.orchestra.run.smithi089.stdout:4852 read 0xbe19d4a thru 0xbe2434f (0xa606 bytes) 2024-09-21T15:50:10.779 INFO:teuthology.orchestra.run.smithi089.stdout:4853 write 0xdf9537a thru 0xdf9ea62 (0x96e9 bytes) 2024-09-21T15:50:10.784 INFO:teuthology.orchestra.run.smithi089.stdout:4856 write 0x90728c5 thru 0x907fef9 (0xd635 bytes) 2024-09-21T15:50:10.790 INFO:teuthology.orchestra.run.smithi089.stdout:4857 write 0xa9b5a05 thru 0xa9c31b0 (0xd7ac bytes) 2024-09-21T15:50:10.794 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:10.793+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:10.806 INFO:teuthology.orchestra.run.smithi089.stdout:4858 write 0xe9d950a thru 0xe9dee51 (0x5948 bytes) 2024-09-21T15:50:10.811 INFO:teuthology.orchestra.run.smithi089.stdout:4860 read 0x7f4ea7f thru 0x7f5c441 (0xd9c3 bytes) 2024-09-21T15:50:10.814 INFO:teuthology.orchestra.run.smithi089.stdout:4861 trunc from 0xe9dee52 to 0xc3051e0 2024-09-21T15:50:10.821 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:10.821 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:10.829 INFO:teuthology.orchestra.run.smithi089.stdout:4862 write 0xb6e914d thru 0xb6f312d (0x9fe1 bytes) 2024-09-21T15:50:10.830 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:50:10.835 INFO:teuthology.orchestra.run.smithi089.stdout:4863 trunc from 0xc3051e0 to 0x9ebf68f 2024-09-21T15:50:10.848 INFO:teuthology.orchestra.run.smithi089.stdout:4866 read 0x3215169 thru 0x3222c68 (0xdb00 bytes) 2024-09-21T15:50:10.848 INFO:teuthology.orchestra.run.smithi089.stdout:4867 punch from 0x13a3f5a to 0x13a4805, (0x8ab bytes) 2024-09-21T15:50:10.848 INFO:teuthology.orchestra.run.smithi089.stdout:4868 clone 33 order 24 su 2097152 sc 13 2024-09-21T15:50:10.894 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:10.893+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:10.937 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:10.937 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:10.946 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:50:10.995 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:10.995+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:11.053 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:11.053 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:11.061 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:50:11.093 INFO:teuthology.orchestra.run.smithi089.stdout:flattening image image_client.0-clone32 2024-09-21T15:50:11.096 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:11.095+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:11.166 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:11.166 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:11.175 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:50:11.197 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:11.196+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:11.223 INFO:teuthology.orchestra.run.smithi089.stdout:checking clone #31, image image_client.0-clone31 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent32 2024-09-21T15:50:11.225 INFO:teuthology.orchestra.run.smithi089.stdout:4870 write 0x9e483c8 thru 0x9e4dd97 (0x59d0 bytes) 2024-09-21T15:50:11.238 INFO:teuthology.orchestra.run.smithi089.stdout:4875 trunc from 0x9ebf68f to 0x5cf9074 2024-09-21T15:50:11.264 INFO:teuthology.orchestra.run.smithi089.stdout:4876 punch from 0x5a4b09e to 0x5a571f3, (0xc155 bytes) 2024-09-21T15:50:11.264 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:11.264 INFO:teuthology.orchestra.run.smithi089.stdout:4879 write 0x2ffe4b5 thru 0x30054a7 (0x6ff3 bytes) 2024-09-21T15:50:11.271 INFO:teuthology.orchestra.run.smithi089.stdout:4881 trunc from 0x5cf9074 to 0x13b4985 2024-09-21T15:50:11.286 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:11.286 INFO:teuthology.orchestra.run.smithi089.stdout:4883 punch from 0x106d59f to 0x1076aca, (0x952b bytes) 2024-09-21T15:50:11.286 INFO:teuthology.orchestra.run.smithi089.stdout:truncating to largest ever: 0xeb5d8a2 2024-09-21T15:50:11.289 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:11.289 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:11.297 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:11.296+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:11.298 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:50:11.302 INFO:teuthology.orchestra.run.smithi089.stdout:4884 trunc from 0x13b4985 to 0xeb5d8a2 2024-09-21T15:50:11.307 INFO:teuthology.orchestra.run.smithi089.stdout:4885 writesame 0xdfe9253 thru 0xdff0fbe data_size 0x2e(0x7d6c bytes) 2024-09-21T15:50:11.310 INFO:teuthology.orchestra.run.smithi089.stdout:4886 write 0x22351ca thru 0x2241a9c (0xc8d3 bytes) 2024-09-21T15:50:11.312 INFO:teuthology.orchestra.run.smithi089.stdout:4887 read 0xb838f74 thru 0xb847d30 (0xedbd bytes) 2024-09-21T15:50:11.312 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:11.312 INFO:teuthology.orchestra.run.smithi089.stdout:4890 punch from 0xca40e00 to 0xca420ee, (0x12ee bytes) 2024-09-21T15:50:11.312 INFO:teuthology.orchestra.run.smithi089.stdout:4891 punch from 0x4d55188 to 0x4d6168e, (0xc506 bytes) 2024-09-21T15:50:11.312 INFO:teuthology.orchestra.run.smithi089.stdout:4892 writesame 0x3aaca3a thru 0x3ab4d8c data_size 0x1(0x8353 bytes) 2024-09-21T15:50:11.323 INFO:teuthology.orchestra.run.smithi089.stdout:4893 write 0x4a59718 thru 0x4a603e2 (0x6ccb bytes) 2024-09-21T15:50:11.328 INFO:teuthology.orchestra.run.smithi089.stdout:4896 writesame 0xa49684 thru 0xa50a63 data_size 0x67(0x73e0 bytes) 2024-09-21T15:50:11.329 INFO:teuthology.orchestra.run.smithi089.stdout:4897 read 0xe38bf75 thru 0xe38e92b (0x29b7 bytes) 2024-09-21T15:50:11.329 INFO:teuthology.orchestra.run.smithi089.stdout:4898 write 0x9ca4493 thru 0x9cae8d6 (0xa444 bytes) 2024-09-21T15:50:11.337 INFO:teuthology.orchestra.run.smithi089.stdout:4900 write 0xedc298e thru 0xedc681d (0x3e90 bytes) 2024-09-21T15:50:11.341 INFO:teuthology.orchestra.run.smithi089.stdout:4901 write 0xc1787c9 thru 0xc188714 (0xff4c bytes) 2024-09-21T15:50:11.348 INFO:teuthology.orchestra.run.smithi089.stdout:4902 trunc from 0xedc681e to 0x985f8f8 2024-09-21T15:50:11.362 INFO:teuthology.orchestra.run.smithi089.stdout:4903 read 0xb9b2c5 thru 0xba5b6e (0xa8aa bytes) 2024-09-21T15:50:11.363 INFO:teuthology.orchestra.run.smithi089.stdout:4904 write 0x4587d5f thru 0x4590462 (0x8704 bytes) 2024-09-21T15:50:11.374 INFO:teuthology.orchestra.run.smithi089.stdout:4905 write 0xdb04447 thru 0xdb0a6e8 (0x62a2 bytes) 2024-09-21T15:50:11.378 INFO:teuthology.orchestra.run.smithi089.stdout:4906 write 0xb8b7496 thru 0xb8bc6b0 (0x521b bytes) 2024-09-21T15:50:11.380 INFO:teuthology.orchestra.run.smithi089.stdout:4909 read 0xb81bc80 thru 0xb820e17 (0x5198 bytes) 2024-09-21T15:50:11.380 INFO:teuthology.orchestra.run.smithi089.stdout:4910 clone 34 order 23 su 32768 sc 2 2024-09-21T15:50:11.398 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:11.397+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:11.443 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:11.443 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:11.452 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:50:11.499 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:11.498+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:11.554 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:11.554 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:11.563 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:50:11.604 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:11.603+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:11.701 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:11.700+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:11.705 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:11.705 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:11.713 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T15:50:11.802 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:11.801+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:11.856 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:11.856 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:11.865 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T15:50:11.903 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:11.901+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:11.966 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:11.966 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:11.975 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T15:50:12.004 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:12.003+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:12.015 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:50:12.041 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T15:50:12.080 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:12.080 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:12.089 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T15:50:12.094 INFO:teuthology.orchestra.run.smithi089.stdout:flattening image image_client.0-clone33 2024-09-21T15:50:12.131 INFO:teuthology.orchestra.run.smithi089.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T15:50:12.132 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:12.131+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:12.147 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:50:12.148 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:50:12.166 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:50:12.166 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:50:12.183 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:50:12.184 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:50:12.191 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:12.191 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:12.200 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T15:50:12.201 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:50:12.201 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:50:12.217 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:50:12.217 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:50:12.234 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:50:12.234 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:50:12.241 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:12.240+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:12.244 INFO:teuthology.orchestra.run.smithi089.stdout:checking clone #32, image image_client.0-clone32 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent33 2024-09-21T15:50:12.250 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:50:12.251 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:50:12.254 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T15:50:12.255 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T15:50:12.306 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:12.305+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:12.345 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:12.345 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:12.345 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:50:12.345 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:50:12.345 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:12.354 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T15:50:12.407 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:12.405+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:12.422 INFO:teuthology.orchestra.run.smithi089.stdout:4911 write 0x282b8bb thru 0x28381b3 (0xc8f9 bytes) 2024-09-21T15:50:12.490 INFO:teuthology.orchestra.run.smithi089.stdout:4912 writesame 0x45595a1 thru 0x455c98b data_size 0x1(0x33eb bytes) 2024-09-21T15:50:12.507 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:12.506+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:12.520 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:12.521 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:50:12.521 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:50:12.521 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:50:12.521 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.521 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4209.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:50:12.521 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:50:12.521 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 81.327785902000002, 2024-09-21T15:50:12.521 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.19452909900000001, 2024-09-21T15:50:12.521 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:12.521 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:12.521 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:12.521 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4209", 2024-09-21T15:50:12.521 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/4045143050", 2024-09-21T15:50:12.522 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1 2024-09-21T15:50:12.522 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.522 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:12.522 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.522 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:12.522 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:50:12.522 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:12.522 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.522 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.522 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:12.522 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:50:12.522 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:12.522 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.522 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.523 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:12.523 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191161+0000", 2024-09-21T15:50:12.523 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-21T15:50:12.523 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.523 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.523 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:12.523 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191169+0000", 2024-09-21T15:50:12.523 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0090000000000007e-06 2024-09-21T15:50:12.523 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.523 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.523 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:12.523 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191180+0000", 2024-09-21T15:50:12.523 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0728e-05 2024-09-21T15:50:12.523 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.523 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.524 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:12.524 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191187+0000", 2024-09-21T15:50:12.524 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9009999999999994e-06 2024-09-21T15:50:12.524 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.524 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.524 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:12.524 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191238+0000", 2024-09-21T15:50:12.524 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0287000000000002e-05 2024-09-21T15:50:12.524 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.524 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.524 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-21T15:50:12.524 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191245+0000", 2024-09-21T15:50:12.524 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0709999999999997e-06 2024-09-21T15:50:12.524 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.524 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.525 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:12.525 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380309+0000", 2024-09-21T15:50:12.525 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.189064277 2024-09-21T15:50:12.525 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.525 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.525 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:12.525 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380454+0000", 2024-09-21T15:50:12.525 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014490799999999999 2024-09-21T15:50:12.525 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.525 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.525 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0", 2024-09-21T15:50:12.525 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380610+0000", 2024-09-21T15:50:12.525 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015650199999999999 2024-09-21T15:50:12.525 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.526 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.526 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:50:12.526 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.381160+0000", 2024-09-21T15:50:12.526 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00054903799999999996 2024-09-21T15:50:12.526 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.526 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.526 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:12.526 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385591+0000", 2024-09-21T15:50:12.526 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.004431272 2024-09-21T15:50:12.526 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.526 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.526 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:50:12.526 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385651+0000", 2024-09-21T15:50:12.526 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0455999999999999e-05 2024-09-21T15:50:12.527 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.527 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.527 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:12.527 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385691+0000", 2024-09-21T15:50:12.527 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0234999999999997e-05 2024-09-21T15:50:12.527 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:12.527 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:12.527 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:12.527 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.527 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.527 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64028 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1763328~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:12.527 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167504+0000", 2024-09-21T15:50:12.527 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 35.351444346000001, 2024-09-21T15:50:12.527 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047494153999999997, 2024-09-21T15:50:12.527 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:12.528 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:12.528 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:12.528 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:12.528 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:12.528 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64028 2024-09-21T15:50:12.528 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.528 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:12.528 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.528 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:12.528 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167504+0000", 2024-09-21T15:50:12.528 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:12.528 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.528 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.528 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:12.528 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167504+0000", 2024-09-21T15:50:12.529 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:12.529 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.529 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.529 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:12.529 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167504+0000", 2024-09-21T15:50:12.529 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:12.529 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.529 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.529 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:12.529 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167506+0000", 2024-09-21T15:50:12.529 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0600000000000002e-06 2024-09-21T15:50:12.529 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.529 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.529 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:12.529 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167512+0000", 2024-09-21T15:50:12.529 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.083e-06 2024-09-21T15:50:12.530 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.530 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.530 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:12.530 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167517+0000", 2024-09-21T15:50:12.530 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7650000000000001e-06 2024-09-21T15:50:12.530 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.530 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.530 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:12.530 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.214835+0000", 2024-09-21T15:50:12.530 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047318067999999998 2024-09-21T15:50:12.530 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.530 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.530 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:12.530 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.214916+0000", 2024-09-21T15:50:12.530 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1815000000000002e-05 2024-09-21T15:50:12.531 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.531 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.531 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:12.531 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.214998+0000", 2024-09-21T15:50:12.531 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1642000000000002e-05 2024-09-21T15:50:12.531 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:12.531 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:12.531 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:12.531 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.531 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.531 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64029 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1050624~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:12.531 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167521+0000", 2024-09-21T15:50:12.531 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 35.351427434999998, 2024-09-21T15:50:12.531 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047649915000000001, 2024-09-21T15:50:12.532 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:12.532 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:12.532 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:12.532 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:12.532 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:12.532 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64029 2024-09-21T15:50:12.532 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.532 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:12.532 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.532 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:12.532 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167521+0000", 2024-09-21T15:50:12.532 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:12.532 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.532 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.532 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:12.533 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167521+0000", 2024-09-21T15:50:12.533 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:12.533 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.533 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.533 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:12.533 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167521+0000", 2024-09-21T15:50:12.533 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:12.533 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.533 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.533 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:12.533 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167523+0000", 2024-09-21T15:50:12.533 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9810000000000002e-06 2024-09-21T15:50:12.533 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.533 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.533 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:12.534 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167529+0000", 2024-09-21T15:50:12.534 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0900000000000001e-06 2024-09-21T15:50:12.534 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.534 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.534 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:12.534 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167533+0000", 2024-09-21T15:50:12.534 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5730000000000001e-06 2024-09-21T15:50:12.534 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.534 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.534 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:12.534 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215031+0000", 2024-09-21T15:50:12.534 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047497893999999999 2024-09-21T15:50:12.534 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.534 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.535 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:12.535 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215099+0000", 2024-09-21T15:50:12.535 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8416000000000004e-05 2024-09-21T15:50:12.535 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.535 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.535 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:12.535 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215171+0000", 2024-09-21T15:50:12.535 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1279000000000007e-05 2024-09-21T15:50:12.535 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:12.535 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:12.535 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:12.535 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.535 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.535 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64030 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1058816~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:12.536 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167537+0000", 2024-09-21T15:50:12.536 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 35.351410926, 2024-09-21T15:50:12.536 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047783816999999999, 2024-09-21T15:50:12.536 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:12.536 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:12.536 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:12.536 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:12.536 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:12.536 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64030 2024-09-21T15:50:12.536 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.536 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:12.536 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.536 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:12.536 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167537+0000", 2024-09-21T15:50:12.536 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:12.537 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.537 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.537 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:12.537 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167537+0000", 2024-09-21T15:50:12.537 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:12.537 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.537 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.537 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:12.537 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167537+0000", 2024-09-21T15:50:12.537 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:12.537 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.537 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.537 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:12.537 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167539+0000", 2024-09-21T15:50:12.537 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9700000000000002e-06 2024-09-21T15:50:12.538 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.538 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.538 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:12.538 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167545+0000", 2024-09-21T15:50:12.538 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8629999999999996e-06 2024-09-21T15:50:12.538 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.538 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.538 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:12.538 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167549+0000", 2024-09-21T15:50:12.538 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.386e-06 2024-09-21T15:50:12.538 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.538 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.538 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:12.538 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215197+0000", 2024-09-21T15:50:12.539 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047647241 2024-09-21T15:50:12.539 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.539 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.539 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:12.539 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215263+0000", 2024-09-21T15:50:12.539 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6295000000000001e-05 2024-09-21T15:50:12.539 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.539 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.539 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:12.539 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215321+0000", 2024-09-21T15:50:12.539 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8362000000000001e-05 2024-09-21T15:50:12.539 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:12.539 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:12.539 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:12.539 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.540 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.540 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64031 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1738752~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:12.540 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167554+0000", 2024-09-21T15:50:12.540 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 35.351394423000002, 2024-09-21T15:50:12.540 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047917855000000002, 2024-09-21T15:50:12.540 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:12.540 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:12.540 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:12.540 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:12.540 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:12.540 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64031 2024-09-21T15:50:12.540 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.540 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:12.540 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.541 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:12.541 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167554+0000", 2024-09-21T15:50:12.541 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:12.541 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.541 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.541 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:12.541 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167554+0000", 2024-09-21T15:50:12.541 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:12.541 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.541 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.541 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:12.541 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167554+0000", 2024-09-21T15:50:12.541 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:12.541 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.541 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.541 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:12.542 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167556+0000", 2024-09-21T15:50:12.542 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3870000000000002e-06 2024-09-21T15:50:12.542 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.542 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.542 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:12.542 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167562+0000", 2024-09-21T15:50:12.542 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9100000000000002e-06 2024-09-21T15:50:12.542 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.542 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.542 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:12.542 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167566+0000", 2024-09-21T15:50:12.542 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6029999999999998e-06 2024-09-21T15:50:12.542 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.542 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.543 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:12.543 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215346+0000", 2024-09-21T15:50:12.543 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.04777998 2024-09-21T15:50:12.543 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.543 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.543 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:12.543 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215412+0000", 2024-09-21T15:50:12.543 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5598000000000005e-05 2024-09-21T15:50:12.543 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.543 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.543 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:12.543 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215472+0000", 2024-09-21T15:50:12.543 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9728000000000001e-05 2024-09-21T15:50:12.543 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:12.543 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:12.544 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:12.544 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.544 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.544 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64032 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1067008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:12.544 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167673+0000", 2024-09-21T15:50:12.544 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 35.351275710000003, 2024-09-21T15:50:12.544 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047947667999999999, 2024-09-21T15:50:12.544 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:12.544 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:12.544 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:12.544 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:12.544 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:12.544 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64032 2024-09-21T15:50:12.544 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.544 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:12.545 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.545 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:12.545 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167673+0000", 2024-09-21T15:50:12.545 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:12.545 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.545 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.545 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:12.545 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167673+0000", 2024-09-21T15:50:12.545 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:12.545 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.545 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.545 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:12.545 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167672+0000", 2024-09-21T15:50:12.545 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:12.546 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.546 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.546 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:12.546 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167675+0000", 2024-09-21T15:50:12.546 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3780000000000001e-06 2024-09-21T15:50:12.546 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.546 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.546 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:12.546 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167685+0000", 2024-09-21T15:50:12.546 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3039999999999994e-06 2024-09-21T15:50:12.546 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.546 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.546 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:12.546 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167692+0000", 2024-09-21T15:50:12.546 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2559999999999996e-06 2024-09-21T15:50:12.547 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.547 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.547 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:12.547 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215495+0000", 2024-09-21T15:50:12.547 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047803267000000003 2024-09-21T15:50:12.547 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.547 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.547 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:12.547 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215560+0000", 2024-09-21T15:50:12.547 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4703999999999996e-05 2024-09-21T15:50:12.547 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.547 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.547 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:12.547 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215620+0000", 2024-09-21T15:50:12.548 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0328000000000002e-05 2024-09-21T15:50:12.548 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:12.548 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:12.548 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:12.548 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.548 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.548 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64033 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1075200~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:12.548 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167697+0000", 2024-09-21T15:50:12.548 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 35.351251138000002, 2024-09-21T15:50:12.548 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048082863000000003, 2024-09-21T15:50:12.548 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:12.548 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:12.548 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:12.548 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:12.548 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:12.549 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64033 2024-09-21T15:50:12.549 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.549 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:12.549 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.549 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:12.549 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167697+0000", 2024-09-21T15:50:12.549 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:12.549 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.549 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.549 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:12.549 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167697+0000", 2024-09-21T15:50:12.549 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:12.549 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.549 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.549 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:12.550 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167697+0000", 2024-09-21T15:50:12.550 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:12.550 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.550 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.550 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:12.550 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167699+0000", 2024-09-21T15:50:12.550 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5299999999999999e-06 2024-09-21T15:50:12.550 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.550 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.550 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:12.550 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167706+0000", 2024-09-21T15:50:12.550 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7490000000000001e-06 2024-09-21T15:50:12.550 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.550 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.551 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:12.551 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167712+0000", 2024-09-21T15:50:12.551 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1940000000000003e-06 2024-09-21T15:50:12.551 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.551 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.551 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:12.551 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215643+0000", 2024-09-21T15:50:12.551 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047930647999999999 2024-09-21T15:50:12.551 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.551 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.551 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:12.551 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215713+0000", 2024-09-21T15:50:12.551 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0488999999999995e-05 2024-09-21T15:50:12.551 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.551 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.552 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:12.552 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215780+0000", 2024-09-21T15:50:12.552 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6656999999999996e-05 2024-09-21T15:50:12.552 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:12.552 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:12.552 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:12.552 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.552 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.552 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64034 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1083392~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:12.552 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167717+0000", 2024-09-21T15:50:12.552 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 35.351231572000003, 2024-09-21T15:50:12.552 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048233316999999998, 2024-09-21T15:50:12.552 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:12.552 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:12.553 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:12.553 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:12.553 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:12.553 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64034 2024-09-21T15:50:12.553 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.553 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:12.553 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.553 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:12.553 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167717+0000", 2024-09-21T15:50:12.553 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:12.553 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.553 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.553 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:12.553 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167717+0000", 2024-09-21T15:50:12.553 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:12.554 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.554 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.554 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:12.554 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167716+0000", 2024-09-21T15:50:12.554 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:12.554 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.554 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.554 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:12.554 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167718+0000", 2024-09-21T15:50:12.554 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.165e-06 2024-09-21T15:50:12.554 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.554 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.554 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:12.555 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167725+0000", 2024-09-21T15:50:12.555 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3380000000000001e-06 2024-09-21T15:50:12.555 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.555 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.555 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:12.555 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167730+0000", 2024-09-21T15:50:12.555 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8010000000000003e-06 2024-09-21T15:50:12.555 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.555 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.555 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:12.555 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215807+0000", 2024-09-21T15:50:12.555 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048077823999999998 2024-09-21T15:50:12.555 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.555 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.555 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:12.555 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215884+0000", 2024-09-21T15:50:12.556 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6283000000000001e-05 2024-09-21T15:50:12.556 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.556 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.556 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:12.556 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215950+0000", 2024-09-21T15:50:12.556 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6290000000000004e-05 2024-09-21T15:50:12.556 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:12.556 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:12.556 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:12.556 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.556 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.556 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64035 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1091584~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:12.556 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:50:12.556 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 35.351214358999997, 2024-09-21T15:50:12.556 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048412117999999997, 2024-09-21T15:50:12.556 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:12.557 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:12.557 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:12.557 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:12.557 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:12.557 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64035 2024-09-21T15:50:12.557 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.557 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:12.557 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.557 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:12.557 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:50:12.557 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:12.557 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.557 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.557 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:12.557 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:50:12.558 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:12.558 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.558 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.558 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:12.558 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:50:12.558 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:12.558 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.558 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.558 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:12.558 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167746+0000", 2024-09-21T15:50:12.558 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2031000000000001e-05 2024-09-21T15:50:12.558 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.558 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.558 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:12.558 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167753+0000", 2024-09-21T15:50:12.559 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0620000000000003e-06 2024-09-21T15:50:12.559 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.559 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.559 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:12.559 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167759+0000", 2024-09-21T15:50:12.559 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.596e-06 2024-09-21T15:50:12.559 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.559 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.559 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:12.559 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215977+0000", 2024-09-21T15:50:12.559 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048217318000000002 2024-09-21T15:50:12.559 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.559 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.559 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:12.560 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216076+0000", 2024-09-21T15:50:12.560 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9846000000000006e-05 2024-09-21T15:50:12.560 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.560 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.560 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:12.560 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216146+0000", 2024-09-21T15:50:12.560 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9578000000000004e-05 2024-09-21T15:50:12.560 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:12.560 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:12.560 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:12.560 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.560 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.560 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64036 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1099776~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:12.560 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167853+0000", 2024-09-21T15:50:12.560 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 35.351095745999999, 2024-09-21T15:50:12.561 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048450675999999998, 2024-09-21T15:50:12.561 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:12.561 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:12.561 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:12.561 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:12.561 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:12.561 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64036 2024-09-21T15:50:12.561 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.561 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:12.561 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.561 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:12.561 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167853+0000", 2024-09-21T15:50:12.561 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:12.561 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.561 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.561 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:12.562 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167853+0000", 2024-09-21T15:50:12.562 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:12.562 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.562 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.562 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:12.562 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167852+0000", 2024-09-21T15:50:12.562 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:12.562 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.562 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.562 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:12.562 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167855+0000", 2024-09-21T15:50:12.562 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5640000000000001e-06 2024-09-21T15:50:12.562 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.562 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.562 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:12.562 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167864+0000", 2024-09-21T15:50:12.563 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8140000000000007e-06 2024-09-21T15:50:12.563 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.563 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.563 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:12.563 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167871+0000", 2024-09-21T15:50:12.563 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5729999999999996e-06 2024-09-21T15:50:12.563 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.563 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.563 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:12.563 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216174+0000", 2024-09-21T15:50:12.563 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048303437999999997 2024-09-21T15:50:12.563 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.563 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.563 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:12.563 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216250+0000", 2024-09-21T15:50:12.563 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5661999999999995e-05 2024-09-21T15:50:12.564 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.564 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.564 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:12.564 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216303+0000", 2024-09-21T15:50:12.564 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3273000000000003e-05 2024-09-21T15:50:12.564 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:12.564 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:12.564 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:12.564 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.564 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.564 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64037 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1771520~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:12.564 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:50:12.564 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 35.351072203000001, 2024-09-21T15:50:12.564 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048588771000000003, 2024-09-21T15:50:12.564 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:12.565 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:12.565 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:12.565 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:12.565 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:12.565 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64037 2024-09-21T15:50:12.565 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.565 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:12.565 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.565 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:12.565 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:50:12.565 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:12.565 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.565 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.565 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:12.565 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:50:12.566 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:12.566 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.566 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.566 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:12.566 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:50:12.566 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:12.566 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.566 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.566 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:12.566 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167878+0000", 2024-09-21T15:50:12.566 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3e-06 2024-09-21T15:50:12.566 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.566 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.566 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:12.566 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167884+0000", 2024-09-21T15:50:12.567 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3269999999999997e-06 2024-09-21T15:50:12.567 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.567 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.567 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:12.567 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167889+0000", 2024-09-21T15:50:12.567 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9269999999999996e-06 2024-09-21T15:50:12.567 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.567 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.567 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:12.567 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216331+0000", 2024-09-21T15:50:12.567 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048442102000000001 2024-09-21T15:50:12.567 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.567 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.567 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:12.567 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216393+0000", 2024-09-21T15:50:12.568 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1132999999999999e-05 2024-09-21T15:50:12.568 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.568 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.568 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:12.568 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216465+0000", 2024-09-21T15:50:12.568 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2339e-05 2024-09-21T15:50:12.568 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:12.568 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:12.568 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:12.568 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.568 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.568 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64038 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1779712~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:12.568 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:50:12.568 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 35.351054372, 2024-09-21T15:50:12.568 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048749795999999998, 2024-09-21T15:50:12.569 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:12.569 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:12.569 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:12.569 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:12.569 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:12.569 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64038 2024-09-21T15:50:12.569 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.569 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:12.569 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.569 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:12.569 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:50:12.569 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:12.569 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.569 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.569 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:12.570 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:50:12.570 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:12.570 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.570 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.570 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:12.570 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:50:12.570 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:12.570 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.570 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.570 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:12.570 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167896+0000", 2024-09-21T15:50:12.570 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2419999999999999e-06 2024-09-21T15:50:12.570 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.570 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.571 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:12.571 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167902+0000", 2024-09-21T15:50:12.571 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0859999999999998e-06 2024-09-21T15:50:12.571 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.571 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.571 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:12.571 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167907+0000", 2024-09-21T15:50:12.571 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9679999999999997e-06 2024-09-21T15:50:12.571 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.571 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.571 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:12.571 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216506+0000", 2024-09-21T15:50:12.571 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048599374000000001 2024-09-21T15:50:12.571 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.571 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.572 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:12.572 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216579+0000", 2024-09-21T15:50:12.572 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2801999999999999e-05 2024-09-21T15:50:12.572 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.572 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.572 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:12.572 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216644+0000", 2024-09-21T15:50:12.572 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4715000000000006e-05 2024-09-21T15:50:12.572 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:12.572 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:12.572 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:12.572 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.572 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.572 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64039 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1107968~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:12.572 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:50:12.573 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 35.351037017000003, 2024-09-21T15:50:12.573 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048911370000000003, 2024-09-21T15:50:12.573 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:12.573 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:12.573 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:12.573 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:12.573 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:12.573 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64039 2024-09-21T15:50:12.573 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.573 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:12.573 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.573 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:12.573 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:50:12.573 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:12.574 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.574 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.574 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:12.574 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:50:12.574 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:12.574 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.574 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.574 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:12.574 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:50:12.574 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:12.574 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.574 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.574 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:12.574 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167913+0000", 2024-09-21T15:50:12.575 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0839999999999999e-06 2024-09-21T15:50:12.575 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.575 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.575 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:12.575 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167921+0000", 2024-09-21T15:50:12.575 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8310000000000001e-06 2024-09-21T15:50:12.575 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.575 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.575 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:12.575 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167926+0000", 2024-09-21T15:50:12.575 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0270000000000003e-06 2024-09-21T15:50:12.575 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.575 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.575 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:12.575 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216680+0000", 2024-09-21T15:50:12.576 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048753890000000001 2024-09-21T15:50:12.576 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.576 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.576 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:12.576 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216757+0000", 2024-09-21T15:50:12.576 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6848999999999996e-05 2024-09-21T15:50:12.576 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.576 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.576 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:12.576 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216823+0000", 2024-09-21T15:50:12.576 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6068000000000005e-05 2024-09-21T15:50:12.576 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:12.576 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:12.576 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:12.576 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.577 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.577 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64040 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1787904~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:12.577 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167931+0000", 2024-09-21T15:50:12.577 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 35.351017785000003, 2024-09-21T15:50:12.577 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049010561000000001, 2024-09-21T15:50:12.577 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:12.577 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:12.577 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:12.577 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:12.577 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:12.577 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64040 2024-09-21T15:50:12.577 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.577 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:12.577 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.578 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:12.578 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167931+0000", 2024-09-21T15:50:12.578 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:12.578 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.578 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.578 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:12.578 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167931+0000", 2024-09-21T15:50:12.578 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:12.578 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.578 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.578 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:12.578 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167930+0000", 2024-09-21T15:50:12.578 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:12.578 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.578 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.579 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:12.579 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167932+0000", 2024-09-21T15:50:12.579 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.198e-06 2024-09-21T15:50:12.579 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.579 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.579 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:12.579 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167938+0000", 2024-09-21T15:50:12.579 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.105e-06 2024-09-21T15:50:12.579 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.579 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.579 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:12.579 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167943+0000", 2024-09-21T15:50:12.579 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9429999999999999e-06 2024-09-21T15:50:12.579 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.580 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.580 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:12.580 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216848+0000", 2024-09-21T15:50:12.580 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048905200000000003 2024-09-21T15:50:12.580 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.580 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.580 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:12.580 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216903+0000", 2024-09-21T15:50:12.580 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4793999999999997e-05 2024-09-21T15:50:12.580 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.580 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.580 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:12.580 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216941+0000", 2024-09-21T15:50:12.580 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7790999999999998e-05 2024-09-21T15:50:12.580 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:12.581 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:12.581 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:12.581 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.581 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.581 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64041 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1796096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:12.581 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:50:12.581 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 35.351000814000002, 2024-09-21T15:50:12.581 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049181457999999997, 2024-09-21T15:50:12.581 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:12.581 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:12.581 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:12.581 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:12.581 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:12.581 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64041 2024-09-21T15:50:12.581 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.582 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:12.582 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.582 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:12.582 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:50:12.582 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:12.582 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.582 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.582 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:12.582 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:50:12.582 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:12.582 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.582 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.582 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:12.582 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:50:12.583 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:12.583 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.583 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.583 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:12.583 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167949+0000", 2024-09-21T15:50:12.583 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0200000000000001e-06 2024-09-21T15:50:12.583 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.583 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.583 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:12.583 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167955+0000", 2024-09-21T15:50:12.583 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5990000000000001e-06 2024-09-21T15:50:12.583 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.583 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.583 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:12.584 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167959+0000", 2024-09-21T15:50:12.584 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5589999999999999e-06 2024-09-21T15:50:12.584 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.584 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.584 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:12.584 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216972+0000", 2024-09-21T15:50:12.584 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049012574000000003 2024-09-21T15:50:12.584 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.584 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.584 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:12.584 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217057+0000", 2024-09-21T15:50:12.584 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4820000000000004e-05 2024-09-21T15:50:12.584 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.584 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.584 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:12.585 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217129+0000", 2024-09-21T15:50:12.585 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2186000000000003e-05 2024-09-21T15:50:12.585 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:12.585 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:12.585 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:12.585 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.585 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.585 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64042 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1804288~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:12.585 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167964+0000", 2024-09-21T15:50:12.585 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 35.350984523000001, 2024-09-21T15:50:12.585 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049333372, 2024-09-21T15:50:12.585 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:12.585 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:12.585 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:12.585 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:12.586 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:12.586 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64042 2024-09-21T15:50:12.586 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.586 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:12.586 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.586 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:12.586 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167964+0000", 2024-09-21T15:50:12.586 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:12.586 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.586 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.586 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:12.586 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167964+0000", 2024-09-21T15:50:12.586 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:12.586 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.587 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.587 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:12.587 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167963+0000", 2024-09-21T15:50:12.587 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:12.587 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.587 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.587 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:12.587 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167966+0000", 2024-09-21T15:50:12.587 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1119999999999999e-06 2024-09-21T15:50:12.587 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.587 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.587 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:12.587 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167975+0000", 2024-09-21T15:50:12.587 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1130000000000007e-06 2024-09-21T15:50:12.588 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.588 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.588 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:12.588 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167980+0000", 2024-09-21T15:50:12.588 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9599999999999999e-06 2024-09-21T15:50:12.588 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.588 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.588 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:12.588 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217158+0000", 2024-09-21T15:50:12.588 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049178569999999998 2024-09-21T15:50:12.588 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.588 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.588 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:12.588 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217232+0000", 2024-09-21T15:50:12.588 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3589000000000003e-05 2024-09-21T15:50:12.589 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.589 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.589 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:12.589 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217297+0000", 2024-09-21T15:50:12.589 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5339999999999994e-05 2024-09-21T15:50:12.589 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:12.589 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:12.589 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:12.589 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.589 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.589 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64043 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1812480~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:12.589 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.168085+0000", 2024-09-21T15:50:12.589 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 35.350863455999999, 2024-09-21T15:50:12.589 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049365157999999999, 2024-09-21T15:50:12.590 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:12.590 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:12.590 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:12.590 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:12.590 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:12.590 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64043 2024-09-21T15:50:12.590 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.590 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:12.590 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.590 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:12.590 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168085+0000", 2024-09-21T15:50:12.590 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:12.590 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.590 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.590 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:12.591 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168085+0000", 2024-09-21T15:50:12.591 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:12.591 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.591 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.591 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:12.591 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168084+0000", 2024-09-21T15:50:12.591 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:12.591 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.591 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.591 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:12.591 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168088+0000", 2024-09-21T15:50:12.591 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5420000000000001e-06 2024-09-21T15:50:12.591 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.591 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.592 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:12.592 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168097+0000", 2024-09-21T15:50:12.592 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5410000000000006e-06 2024-09-21T15:50:12.592 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.592 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.592 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:12.592 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168104+0000", 2024-09-21T15:50:12.592 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6719999999999998e-06 2024-09-21T15:50:12.592 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.592 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.592 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:12.592 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217323+0000", 2024-09-21T15:50:12.592 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049218857999999997 2024-09-21T15:50:12.592 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.592 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.593 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:12.593 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217397+0000", 2024-09-21T15:50:12.593 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3932999999999998e-05 2024-09-21T15:50:12.593 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.593 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.593 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:12.593 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217450+0000", 2024-09-21T15:50:12.593 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3168000000000003e-05 2024-09-21T15:50:12.593 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:12.593 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:12.593 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:12.593 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.593 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.593 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64044 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1820672~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:12.593 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.170036+0000", 2024-09-21T15:50:12.594 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 35.348912136999999, 2024-09-21T15:50:12.594 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047578069000000001, 2024-09-21T15:50:12.594 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:12.594 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:12.594 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:12.594 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:12.594 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:12.594 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64044 2024-09-21T15:50:12.594 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.594 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:12.594 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.594 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:12.594 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170036+0000", 2024-09-21T15:50:12.594 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:12.595 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.595 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.595 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:12.595 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170036+0000", 2024-09-21T15:50:12.595 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:12.595 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.595 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.595 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:12.595 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168109+0000", 2024-09-21T15:50:12.595 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9980721 2024-09-21T15:50:12.595 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.595 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.595 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:12.595 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170040+0000", 2024-09-21T15:50:12.595 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0019318079999999999 2024-09-21T15:50:12.596 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.596 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.596 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:12.596 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170051+0000", 2024-09-21T15:50:12.596 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0220999999999999e-05 2024-09-21T15:50:12.596 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.596 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.596 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:12.596 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170058+0000", 2024-09-21T15:50:12.596 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0110000000000003e-06 2024-09-21T15:50:12.596 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.596 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.596 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:12.596 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217478+0000", 2024-09-21T15:50:12.596 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047420334000000001 2024-09-21T15:50:12.597 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.597 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.597 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:12.597 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217547+0000", 2024-09-21T15:50:12.597 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9416000000000001e-05 2024-09-21T15:50:12.597 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.597 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.597 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:12.597 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217614+0000", 2024-09-21T15:50:12.597 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6909000000000006e-05 2024-09-21T15:50:12.597 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:12.597 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:12.597 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:12.597 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.597 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.598 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64045 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1828864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:12.598 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.170064+0000", 2024-09-21T15:50:12.598 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 35.348884097999999, 2024-09-21T15:50:12.598 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047709252000000001, 2024-09-21T15:50:12.598 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:12.598 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:12.598 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:12.598 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:12.598 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:12.598 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64045 2024-09-21T15:50:12.598 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.598 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:12.598 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.598 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:12.598 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170064+0000", 2024-09-21T15:50:12.599 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:12.599 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.599 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.599 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:12.599 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170064+0000", 2024-09-21T15:50:12.599 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:12.599 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.599 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.599 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:12.599 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170064+0000", 2024-09-21T15:50:12.599 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:12.599 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.599 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.599 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:12.600 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170066+0000", 2024-09-21T15:50:12.600 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8109999999999999e-06 2024-09-21T15:50:12.600 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.600 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.600 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:12.600 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170073+0000", 2024-09-21T15:50:12.600 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5570000000000001e-06 2024-09-21T15:50:12.600 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.600 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.600 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:12.600 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170078+0000", 2024-09-21T15:50:12.600 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9910000000000002e-06 2024-09-21T15:50:12.600 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.600 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.600 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:12.601 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217640+0000", 2024-09-21T15:50:12.601 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047561840000000001 2024-09-21T15:50:12.601 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.601 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.601 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:12.601 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217710+0000", 2024-09-21T15:50:12.601 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9967000000000003e-05 2024-09-21T15:50:12.601 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.601 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.601 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:12.601 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217773+0000", 2024-09-21T15:50:12.601 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3638000000000003e-05 2024-09-21T15:50:12.601 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:12.601 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:12.601 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:12.602 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.602 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.602 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64356 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 3076096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:12.602 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.183065+0000", 2024-09-21T15:50:12.602 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 35.335882967000003, 2024-09-21T15:50:12.602 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047484923999999998, 2024-09-21T15:50:12.602 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:12.602 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:12.602 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:12.602 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:12.602 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:12.602 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64356 2024-09-21T15:50:12.602 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.602 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:12.603 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.603 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:12.603 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.183065+0000", 2024-09-21T15:50:12.603 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:12.603 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.603 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.603 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:12.603 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.183065+0000", 2024-09-21T15:50:12.603 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:12.603 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.603 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.603 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:12.603 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.182922+0000", 2024-09-21T15:50:12.603 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999856 2024-09-21T15:50:12.603 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.603 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.604 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:12.604 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.183069+0000", 2024-09-21T15:50:12.604 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000147097 2024-09-21T15:50:12.604 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.604 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.604 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:12.604 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.183078+0000", 2024-09-21T15:50:12.604 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.9099999999999994e-06 2024-09-21T15:50:12.604 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.604 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.604 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:12.604 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.183085+0000", 2024-09-21T15:50:12.604 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0670000000000003e-06 2024-09-21T15:50:12.604 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.604 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.604 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:12.604 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.230377+0000", 2024-09-21T15:50:12.605 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047292698000000001 2024-09-21T15:50:12.605 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.605 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.605 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:12.605 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.230458+0000", 2024-09-21T15:50:12.605 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0980000000000001e-05 2024-09-21T15:50:12.605 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:12.605 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:12.605 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:12.605 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.230550+0000", 2024-09-21T15:50:12.605 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1954999999999996e-05 2024-09-21T15:50:12.605 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:12.605 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:12.605 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:12.605 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:12.605 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:12.606 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:12.607 INFO:teuthology.orchestra.run.smithi089.stdout:4913 trunc from 0xdb0a6e9 to 0xa40cec6 2024-09-21T15:50:12.607 INFO:teuthology.orchestra.run.smithi089.stdout:4914 write 0x232ed4c thru 0x23349ee (0x5ca3 bytes) 2024-09-21T15:50:12.607 INFO:teuthology.orchestra.run.smithi089.stdout:4916 read 0x5708e58 thru 0x571746d (0xe616 bytes) 2024-09-21T15:50:12.607 INFO:teuthology.orchestra.run.smithi089.stdout:4918 write 0x914f023 thru 0x915367f (0x465d bytes) 2024-09-21T15:50:12.608 INFO:teuthology.orchestra.run.smithi089.stdout:4919 read 0x928e898 thru 0x929b4be (0xcc27 bytes) 2024-09-21T15:50:12.608 INFO:teuthology.orchestra.run.smithi089.stdout:4922 write 0xe348a1e thru 0xe34d61f (0x4c02 bytes) 2024-09-21T15:50:12.608 INFO:teuthology.orchestra.run.smithi089.stdout:4924 trunc from 0xe34d620 to 0x68319f9 2024-09-21T15:50:12.608 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:50:12.609 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:12.608+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:12.641 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:12.647 INFO:teuthology.orchestra.run.smithi089.stdout:4927 trunc from 0x68319f9 to 0x7d43b0 2024-09-21T15:50:12.700 INFO:teuthology.orchestra.run.smithi089.stdout:4928 punch from 0x566b49 to 0x56b0e3, (0x459a bytes) 2024-09-21T15:50:12.700 INFO:teuthology.orchestra.run.smithi089.stdout:4929 flatten 2024-09-21T15:50:12.705 INFO:teuthology.orchestra.run.smithi089.stdout:4930 read 0x54a063 thru 0x557275 (0xd213 bytes) 2024-09-21T15:50:12.706 INFO:teuthology.orchestra.run.smithi089.stdout:4931 write 0x383e0f thru 0x38dbca (0x9dbc bytes) 2024-09-21T15:50:12.709 INFO:teuthology.orchestra.run.smithi089.stdout:4932 read 0x16ec91 thru 0x17034c (0x16bc bytes) 2024-09-21T15:50:12.709 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:12.708+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:12.709 INFO:teuthology.orchestra.run.smithi089.stdout:truncating to largest ever: 0xee29f44 2024-09-21T15:50:12.726 INFO:teuthology.orchestra.run.smithi089.stdout:4935 trunc from 0x7d43b0 to 0xee29f44 2024-09-21T15:50:12.731 INFO:teuthology.orchestra.run.smithi089.stdout:4936 write 0x8ff8e13 thru 0x9003ec7 (0xb0b5 bytes) 2024-09-21T15:50:12.734 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:12.734 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:12.734 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:50:12.734 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:12.736 INFO:teuthology.orchestra.run.smithi089.stdout:4937 writesame 0xa2c7b88 thru 0xa2d27a3 data_size 0xa(0xac1c bytes) 2024-09-21T15:50:12.746 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:50:12.753 INFO:teuthology.orchestra.run.smithi089.stdout:4938 trunc from 0xee29f44 to 0x4788454 2024-09-21T15:50:12.766 INFO:teuthology.orchestra.run.smithi089.stdout:4940 writesame 0x1a1d21b thru 0x1a21ecd data_size 0x37(0x4cb3 bytes) 2024-09-21T15:50:12.770 INFO:teuthology.orchestra.run.smithi089.stdout:4941 read 0x196815b thru 0x1974a8d (0xc933 bytes) 2024-09-21T15:50:12.770 INFO:teuthology.orchestra.run.smithi089.stdout:4942 read 0x45c0375 thru 0x45ce8c6 (0xe552 bytes) 2024-09-21T15:50:12.770 INFO:teuthology.orchestra.run.smithi089.stdout:4943 writesame 0x3316d16 thru 0x33214ea data_size 0xd(0xa7d5 bytes) 2024-09-21T15:50:12.774 INFO:teuthology.orchestra.run.smithi089.stdout:4945 read 0xb66728 thru 0xb71aff (0xb3d8 bytes) 2024-09-21T15:50:12.774 INFO:teuthology.orchestra.run.smithi089.stdout:4947 read 0x13e8d95 thru 0x13f3a88 (0xacf4 bytes) 2024-09-21T15:50:12.775 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:12.775 INFO:teuthology.orchestra.run.smithi089.stdout:4951 punch from 0x1200dae to 0x120d232, (0xc484 bytes) 2024-09-21T15:50:12.775 INFO:teuthology.orchestra.run.smithi089.stdout:4952 read 0x12daafa thru 0x12dc17f (0x1686 bytes) 2024-09-21T15:50:12.775 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:12.791 INFO:teuthology.orchestra.run.smithi089.stdout:4954 write 0xd2f04cc thru 0xd2fd8cd (0xd402 bytes) 2024-09-21T15:50:12.794 INFO:teuthology.orchestra.run.smithi089.stdout:4955 writesame 0x122bbd3 thru 0x12368f9 data_size 0x1(0xad27 bytes) 2024-09-21T15:50:12.807 INFO:teuthology.orchestra.run.smithi089.stdout:4956 punch from 0xafdb7d1 to 0xafe7999, (0xc1c8 bytes) 2024-09-21T15:50:12.807 INFO:teuthology.orchestra.run.smithi089.stdout:4957 read 0x7ef68a5 thru 0x7efb4a8 (0x4c04 bytes) 2024-09-21T15:50:12.807 INFO:teuthology.orchestra.run.smithi089.stdout:4960 writesame 0xfa7af9 thru 0xfb2ae2 data_size 0x59(0xafea bytes) 2024-09-21T15:50:12.808 INFO:teuthology.orchestra.run.smithi089.stdout:4961 read 0x5ea7ce3 thru 0x5eadc9e (0x5fbc bytes) 2024-09-21T15:50:12.808 INFO:teuthology.orchestra.run.smithi089.stdout:4962 read 0x7bad292 thru 0x7bb9888 (0xc5f7 bytes) 2024-09-21T15:50:12.809 INFO:teuthology.orchestra.run.smithi089.stdout:4964 write 0xc8bcabf thru 0xc8bf400 (0x2942 bytes) 2024-09-21T15:50:12.810 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:12.809+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:12.811 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:12.811 INFO:teuthology.orchestra.run.smithi089.stdout:4967 write 0x5047e0d thru 0x505280e (0xaa02 bytes) 2024-09-21T15:50:12.817 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:12.817 INFO:teuthology.orchestra.run.smithi089.stdout:4971 read 0x36fb97f thru 0x370b170 (0xf7f2 bytes) 2024-09-21T15:50:12.824 INFO:teuthology.orchestra.run.smithi089.stdout:4972 writesame 0xe33c921 thru 0xe33df89 data_size 0x1(0x1669 bytes) 2024-09-21T15:50:12.824 INFO:teuthology.orchestra.run.smithi089.stdout:4973 read 0x47ac92f thru 0x47bb761 (0xee33 bytes) 2024-09-21T15:50:12.824 INFO:teuthology.orchestra.run.smithi089.stdout:4974 write 0x21b348a thru 0x21bd44c (0x9fc3 bytes) 2024-09-21T15:50:12.828 INFO:teuthology.orchestra.run.smithi089.stdout:4975 punch from 0xb6529c4 to 0xb6590fb, (0x6737 bytes) 2024-09-21T15:50:12.828 INFO:teuthology.orchestra.run.smithi089.stdout:4976 read 0x9a9674b thru 0x9a9e820 (0x80d6 bytes) 2024-09-21T15:50:12.828 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:12.828 INFO:teuthology.orchestra.run.smithi089.stdout:4979 writesame 0x3f89bcb thru 0x3f8d74c data_size 0x3(0x3b82 bytes) 2024-09-21T15:50:12.831 INFO:teuthology.orchestra.run.smithi089.stdout:4980 write 0x35fca0d thru 0x35fe27d (0x1871 bytes) 2024-09-21T15:50:12.832 INFO:teuthology.orchestra.run.smithi089.stdout:4981 punch from 0x27c183d to 0x27cf51c, (0xdcdf bytes) 2024-09-21T15:50:12.832 INFO:teuthology.orchestra.run.smithi089.stdout:4982 punch from 0xa1eca6e to 0xa1f5233, (0x87c5 bytes) 2024-09-21T15:50:12.832 INFO:teuthology.orchestra.run.smithi089.stdout:4983 write 0x6f1b95a thru 0x6f29c13 (0xe2ba bytes) 2024-09-21T15:50:12.836 INFO:teuthology.orchestra.run.smithi089.stdout:4984 writesame 0x89b93dd thru 0x89c1453 data_size 0x1(0x8077 bytes) 2024-09-21T15:50:12.851 INFO:teuthology.orchestra.run.smithi089.stdout:4985 read 0x59e91a0 thru 0x59f2ad2 (0x9933 bytes) 2024-09-21T15:50:12.852 INFO:teuthology.orchestra.run.smithi089.stdout:4987 write 0xcc6760e thru 0xcc68de5 (0x17d8 bytes) 2024-09-21T15:50:12.854 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:12.854 INFO:teuthology.orchestra.run.smithi089.stdout:4989 read 0x6c078cc thru 0x6c1652f (0xec64 bytes) 2024-09-21T15:50:12.855 INFO:teuthology.orchestra.run.smithi089.stdout:4990 write 0xbb1eb0e thru 0xbb22cb0 (0x41a3 bytes) 2024-09-21T15:50:12.861 INFO:teuthology.orchestra.run.smithi089.stdout:4991 punch from 0xab7904f to 0xab7a74c, (0x16fd bytes) 2024-09-21T15:50:12.861 INFO:teuthology.orchestra.run.smithi089.stdout:4992 read 0xd92188f thru 0xd9303f0 (0xeb62 bytes) 2024-09-21T15:50:12.862 INFO:teuthology.orchestra.run.smithi089.stdout:4993 read 0x87774bb thru 0x877d91c (0x6462 bytes) 2024-09-21T15:50:12.862 INFO:teuthology.orchestra.run.smithi089.stdout:4994 read 0x840ba4b thru 0x841a316 (0xe8cc bytes) 2024-09-21T15:50:12.862 INFO:teuthology.orchestra.run.smithi089.stdout:4995 read 0x7018216 thru 0x701f6da (0x74c5 bytes) 2024-09-21T15:50:12.862 INFO:teuthology.orchestra.run.smithi089.stdout:4997 write 0x6a14490 thru 0x6a17127 (0x2c98 bytes) 2024-09-21T15:50:12.863 INFO:teuthology.orchestra.run.smithi089.stdout:4998 writesame 0x94a170 thru 0x94cbb3 data_size 0xa(0x2a44 bytes) 2024-09-21T15:50:12.865 INFO:teuthology.orchestra.run.smithi089.stdout:5000 punch from 0x77159b8 to 0x7717e37, (0x247f bytes) 2024-09-21T15:50:12.865 INFO:teuthology.orchestra.run.smithi089.stdout:5002 read 0x13e1221 thru 0x13ed021 (0xbe01 bytes) 2024-09-21T15:50:12.865 INFO:teuthology.orchestra.run.smithi089.stdout:5004 writesame 0xb92c1fc thru 0xb93baa1 data_size 0x6(0xf8a6 bytes) 2024-09-21T15:50:12.868 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:12.868 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:12.868 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:50:12.868 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:50:12.868 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:12.878 INFO:teuthology.orchestra.run.smithi089.stdout:5005 trunc from 0xe33df8a to 0x7aa9e27 2024-09-21T15:50:12.881 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:50:12.894 INFO:teuthology.orchestra.run.smithi089.stdout:5006 trunc from 0x7aa9e27 to 0x8775d61 2024-09-21T15:50:12.899 INFO:teuthology.orchestra.run.smithi089.stdout:5007 punch from 0x3c21a25 to 0x3c2514d, (0x3728 bytes) 2024-09-21T15:50:12.899 INFO:teuthology.orchestra.run.smithi089.stdout:5008 read 0x3d1a91d thru 0x3d1bfad (0x1691 bytes) 2024-09-21T15:50:12.900 INFO:teuthology.orchestra.run.smithi089.stdout:5009 punch from 0xc04d59 to 0xc054fe, (0x7a5 bytes) 2024-09-21T15:50:12.900 INFO:teuthology.orchestra.run.smithi089.stdout:5010 read 0x3f4478 thru 0x3f6037 (0x1bc0 bytes) 2024-09-21T15:50:12.908 INFO:teuthology.orchestra.run.smithi089.stdout:5013 write 0xa9775ac thru 0xa986111 (0xeb66 bytes) 2024-09-21T15:50:12.911 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:12.910+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:12.912 INFO:teuthology.orchestra.run.smithi089.stdout:5014 clone 35 order 23 su 1048576 sc 4 2024-09-21T15:50:12.998 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:12.999 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:50:12.999 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:50:12.999 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:50:12.999 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:12.999 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62833 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 192512~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:12.999 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:50:12.999 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 35.854536791999998, 2024-09-21T15:50:12.999 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050387007999999997, 2024-09-21T15:50:12.999 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:12.999 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:12.999 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:12.999 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:12.999 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:12.999 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62833 2024-09-21T15:50:13.000 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.000 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:13.000 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.000 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:13.000 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:50:13.000 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.000 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.000 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.000 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:13.000 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:50:13.000 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.000 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.000 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.000 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:13.000 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:50:13.000 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:13.001 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.001 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.001 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:13.001 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142150+0000", 2024-09-21T15:50:13.001 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1999999999999999e-06 2024-09-21T15:50:13.001 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.001 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.001 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:13.001 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142153+0000", 2024-09-21T15:50:13.001 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7410000000000001e-06 2024-09-21T15:50:13.001 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.001 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.001 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:13.001 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142156+0000", 2024-09-21T15:50:13.001 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5739999999999998e-06 2024-09-21T15:50:13.002 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.002 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.002 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:13.002 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.192358+0000", 2024-09-21T15:50:13.002 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050202452000000002 2024-09-21T15:50:13.002 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.002 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.002 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:13.002 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.192435+0000", 2024-09-21T15:50:13.002 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6738999999999998e-05 2024-09-21T15:50:13.002 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.002 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.002 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:13.002 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.192536+0000", 2024-09-21T15:50:13.002 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101508 2024-09-21T15:50:13.003 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.003 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:13.003 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.003 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.003 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.003 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62849 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 208896~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:13.003 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.144181+0000", 2024-09-21T15:50:13.003 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 35.852504719999999, 2024-09-21T15:50:13.003 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050460390000000001, 2024-09-21T15:50:13.003 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:13.003 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:13.003 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:13.003 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:13.003 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:13.003 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62849 2024-09-21T15:50:13.003 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.004 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:13.004 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.004 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:13.004 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144181+0000", 2024-09-21T15:50:13.004 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.004 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.004 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.004 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:13.004 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144181+0000", 2024-09-21T15:50:13.004 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.004 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.004 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.004 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:13.004 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144181+0000", 2024-09-21T15:50:13.005 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:13.005 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.005 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.005 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:13.005 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144183+0000", 2024-09-21T15:50:13.005 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3860000000000001e-06 2024-09-21T15:50:13.005 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.005 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.005 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:13.005 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144189+0000", 2024-09-21T15:50:13.005 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0599999999999996e-06 2024-09-21T15:50:13.005 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.005 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.005 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:13.005 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144194+0000", 2024-09-21T15:50:13.005 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4379999999999997e-06 2024-09-21T15:50:13.005 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.006 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.006 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:13.006 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.194457+0000", 2024-09-21T15:50:13.006 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050263317000000002 2024-09-21T15:50:13.006 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.006 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.006 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:13.006 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.194548+0000", 2024-09-21T15:50:13.006 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0785000000000006e-05 2024-09-21T15:50:13.006 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.006 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.006 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:13.006 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.194642+0000", 2024-09-21T15:50:13.006 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3837000000000003e-05 2024-09-21T15:50:13.007 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.007 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:13.007 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.007 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.007 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.007 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62862 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 954368~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:13.007 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.146176+0000", 2024-09-21T15:50:13.007 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 35.850509657000003, 2024-09-21T15:50:13.007 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050442901999999998, 2024-09-21T15:50:13.007 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:13.007 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:13.007 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:13.007 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:13.007 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:13.007 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62862 2024-09-21T15:50:13.007 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.008 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:13.008 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.008 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:13.008 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146176+0000", 2024-09-21T15:50:13.008 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.008 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.008 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.008 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:13.008 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146176+0000", 2024-09-21T15:50:13.008 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.008 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.008 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.008 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:13.008 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146176+0000", 2024-09-21T15:50:13.008 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:13.008 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.009 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.009 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:13.009 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146178+0000", 2024-09-21T15:50:13.009 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4739999999999999e-06 2024-09-21T15:50:13.009 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.009 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.009 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:13.009 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146186+0000", 2024-09-21T15:50:13.009 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4989999999999997e-06 2024-09-21T15:50:13.009 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.009 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.009 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:13.009 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146191+0000", 2024-09-21T15:50:13.009 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7999999999999998e-06 2024-09-21T15:50:13.009 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.010 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.010 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:13.010 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196442+0000", 2024-09-21T15:50:13.010 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050251665000000001 2024-09-21T15:50:13.010 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.010 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.010 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:13.010 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196518+0000", 2024-09-21T15:50:13.010 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5677000000000001e-05 2024-09-21T15:50:13.010 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.010 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.010 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:13.010 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196619+0000", 2024-09-21T15:50:13.010 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101247 2024-09-21T15:50:13.010 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.010 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:13.011 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.011 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.011 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.011 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62863 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 937984~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:13.011 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.146195+0000", 2024-09-21T15:50:13.011 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 35.850491466000001, 2024-09-21T15:50:13.011 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050642305999999998, 2024-09-21T15:50:13.011 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:13.011 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:13.011 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:13.011 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:13.011 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:13.011 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62863 2024-09-21T15:50:13.011 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.011 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:13.012 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.012 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:13.012 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146195+0000", 2024-09-21T15:50:13.012 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.012 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.012 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.012 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:13.012 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146195+0000", 2024-09-21T15:50:13.012 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.012 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.012 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.012 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:13.012 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146194+0000", 2024-09-21T15:50:13.012 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:13.012 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.013 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.013 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:13.013 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146196+0000", 2024-09-21T15:50:13.013 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1900000000000002e-06 2024-09-21T15:50:13.013 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.013 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.013 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:13.013 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146202+0000", 2024-09-21T15:50:13.013 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.57e-06 2024-09-21T15:50:13.013 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.013 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.013 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:13.013 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146206+0000", 2024-09-21T15:50:13.013 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5079999999999999e-06 2024-09-21T15:50:13.013 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.013 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.014 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:13.014 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196655+0000", 2024-09-21T15:50:13.014 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050448973000000001 2024-09-21T15:50:13.014 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.014 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.014 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:13.014 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196730+0000", 2024-09-21T15:50:13.014 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4728999999999996e-05 2024-09-21T15:50:13.014 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.014 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.014 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:13.014 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196837+0000", 2024-09-21T15:50:13.014 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000106735 2024-09-21T15:50:13.014 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.014 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:13.015 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.015 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.015 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.015 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62871 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 307200~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:13.015 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148134+0000", 2024-09-21T15:50:13.015 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 35.848552257000001, 2024-09-21T15:50:13.015 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050310001, 2024-09-21T15:50:13.015 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:13.015 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:13.015 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:13.015 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:13.015 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:13.015 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62871 2024-09-21T15:50:13.015 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.015 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:13.016 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.016 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:13.016 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148134+0000", 2024-09-21T15:50:13.016 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.016 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.016 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.016 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:13.016 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148134+0000", 2024-09-21T15:50:13.016 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.016 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.016 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.016 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:13.016 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148133+0000", 2024-09-21T15:50:13.016 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:13.016 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.016 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.017 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:13.017 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148135+0000", 2024-09-21T15:50:13.017 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9829999999999999e-06 2024-09-21T15:50:13.017 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.017 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.017 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:13.017 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148141+0000", 2024-09-21T15:50:13.017 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5439999999999998e-06 2024-09-21T15:50:13.017 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.017 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.017 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:13.017 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148145+0000", 2024-09-21T15:50:13.017 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4229999999999998e-06 2024-09-21T15:50:13.017 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.017 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.017 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:13.018 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198294+0000", 2024-09-21T15:50:13.018 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050148558000000003 2024-09-21T15:50:13.018 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.018 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.018 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:13.018 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198372+0000", 2024-09-21T15:50:13.018 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8163000000000004e-05 2024-09-21T15:50:13.018 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.018 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.018 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:13.018 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198444+0000", 2024-09-21T15:50:13.018 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1620999999999997e-05 2024-09-21T15:50:13.018 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.018 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:13.018 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.019 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.019 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.019 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62872 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 290816~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:13.019 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148149+0000", 2024-09-21T15:50:13.019 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 35.848536789999997, 2024-09-21T15:50:13.019 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.05046991, 2024-09-21T15:50:13.019 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:13.019 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:13.019 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:13.019 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:13.019 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:13.019 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62872 2024-09-21T15:50:13.019 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.019 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:13.019 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.019 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:13.020 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148149+0000", 2024-09-21T15:50:13.020 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.020 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.020 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.020 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:13.020 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148149+0000", 2024-09-21T15:50:13.020 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.020 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.020 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.020 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:13.020 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148149+0000", 2024-09-21T15:50:13.020 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:13.020 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.020 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.020 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:13.021 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148157+0000", 2024-09-21T15:50:13.021 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4749999999999993e-06 2024-09-21T15:50:13.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.021 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.021 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:13.021 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148163+0000", 2024-09-21T15:50:13.021 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8250000000000001e-06 2024-09-21T15:50:13.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.021 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.021 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:13.021 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148168+0000", 2024-09-21T15:50:13.021 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4730000000000002e-06 2024-09-21T15:50:13.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.021 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.021 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:13.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198471+0000", 2024-09-21T15:50:13.022 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050302974 2024-09-21T15:50:13.022 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.022 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.022 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:13.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198555+0000", 2024-09-21T15:50:13.022 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4783999999999993e-05 2024-09-21T15:50:13.022 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.022 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.022 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:13.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198619+0000", 2024-09-21T15:50:13.022 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3751000000000007e-05 2024-09-21T15:50:13.022 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.022 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:13.022 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.023 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.023 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62873 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 315392~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:13.023 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148172+0000", 2024-09-21T15:50:13.023 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 35.848514498999997, 2024-09-21T15:50:13.023 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050707998999999997, 2024-09-21T15:50:13.023 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:13.023 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:13.023 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:13.023 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:13.023 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:13.023 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62873 2024-09-21T15:50:13.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.023 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:13.023 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.024 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:13.024 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148172+0000", 2024-09-21T15:50:13.024 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.024 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.024 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.024 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:13.024 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148172+0000", 2024-09-21T15:50:13.024 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.024 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.024 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.024 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:13.024 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148171+0000", 2024-09-21T15:50:13.024 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:13.024 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.024 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.025 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:13.025 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148173+0000", 2024-09-21T15:50:13.025 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2340000000000001e-06 2024-09-21T15:50:13.025 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.025 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.025 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:13.025 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148179+0000", 2024-09-21T15:50:13.025 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9340000000000003e-06 2024-09-21T15:50:13.025 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.025 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.025 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:13.025 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148186+0000", 2024-09-21T15:50:13.025 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2349999999999996e-06 2024-09-21T15:50:13.025 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.025 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.025 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:13.026 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198684+0000", 2024-09-21T15:50:13.026 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050498890999999997 2024-09-21T15:50:13.026 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.026 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.026 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:13.026 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198778+0000", 2024-09-21T15:50:13.026 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3134999999999995e-05 2024-09-21T15:50:13.026 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.026 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.026 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:13.026 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198880+0000", 2024-09-21T15:50:13.026 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101906 2024-09-21T15:50:13.026 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.026 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:13.026 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.027 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.027 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.027 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62874 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 323584~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:13.027 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148211+0000", 2024-09-21T15:50:13.027 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 35.848474738999997, 2024-09-21T15:50:13.027 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050887766000000001, 2024-09-21T15:50:13.027 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:13.027 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:13.027 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:13.027 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:13.027 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:13.027 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62874 2024-09-21T15:50:13.027 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.027 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:13.028 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.028 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:13.028 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148211+0000", 2024-09-21T15:50:13.028 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.028 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.028 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:13.028 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148211+0000", 2024-09-21T15:50:13.028 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.028 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.028 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:13.028 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148211+0000", 2024-09-21T15:50:13.028 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:13.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.029 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.029 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:13.029 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148213+0000", 2024-09-21T15:50:13.029 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6860000000000002e-06 2024-09-21T15:50:13.029 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.029 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.029 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:13.029 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148220+0000", 2024-09-21T15:50:13.029 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4740000000000002e-06 2024-09-21T15:50:13.029 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.029 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.029 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:13.029 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148226+0000", 2024-09-21T15:50:13.029 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0120000000000002e-06 2024-09-21T15:50:13.029 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.029 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.030 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:13.030 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198940+0000", 2024-09-21T15:50:13.030 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050714412 2024-09-21T15:50:13.030 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.030 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.030 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:13.030 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198995+0000", 2024-09-21T15:50:13.030 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4863e-05 2024-09-21T15:50:13.030 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.030 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.030 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:13.030 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.199099+0000", 2024-09-21T15:50:13.030 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010383299999999999 2024-09-21T15:50:13.030 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.030 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:13.031 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.031 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.031 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.031 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62877 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 995328~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:13.031 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148230+0000", 2024-09-21T15:50:13.031 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 35.848455878999999, 2024-09-21T15:50:13.031 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.051108949000000001, 2024-09-21T15:50:13.031 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:13.031 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:13.031 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:13.031 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:13.031 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:13.031 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62877 2024-09-21T15:50:13.031 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.031 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:13.031 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.031 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:13.032 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148230+0000", 2024-09-21T15:50:13.032 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.032 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.032 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.032 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:13.032 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148230+0000", 2024-09-21T15:50:13.032 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.032 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.032 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.032 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:13.032 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148230+0000", 2024-09-21T15:50:13.032 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:13.032 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.032 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.032 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:13.032 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148232+0000", 2024-09-21T15:50:13.033 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1220000000000002e-06 2024-09-21T15:50:13.033 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.033 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.033 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:13.033 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148238+0000", 2024-09-21T15:50:13.033 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.789e-06 2024-09-21T15:50:13.033 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.033 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.033 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:13.033 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148242+0000", 2024-09-21T15:50:13.033 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7369999999999997e-06 2024-09-21T15:50:13.033 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.033 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.033 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:13.033 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.199201+0000", 2024-09-21T15:50:13.034 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050959088 2024-09-21T15:50:13.034 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.034 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:13.034 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.199285+0000", 2024-09-21T15:50:13.034 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3147999999999997e-05 2024-09-21T15:50:13.034 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.034 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:13.034 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.199339+0000", 2024-09-21T15:50:13.034 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4484999999999998e-05 2024-09-21T15:50:13.034 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.034 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:13.034 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.034 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.035 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62886 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 1044480~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:13.035 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150138+0000", 2024-09-21T15:50:13.035 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 35.846548210999998, 2024-09-21T15:50:13.035 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050345527000000001, 2024-09-21T15:50:13.035 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:13.035 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:13.035 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:13.035 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:13.035 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:13.035 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62886 2024-09-21T15:50:13.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.035 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:13.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.036 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:13.036 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150138+0000", 2024-09-21T15:50:13.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.036 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:13.036 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150138+0000", 2024-09-21T15:50:13.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.036 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:13.036 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150137+0000", 2024-09-21T15:50:13.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:13.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.037 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:13.037 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150140+0000", 2024-09-21T15:50:13.037 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2460000000000002e-06 2024-09-21T15:50:13.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.037 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:13.037 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150145+0000", 2024-09-21T15:50:13.037 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6520000000000003e-06 2024-09-21T15:50:13.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.037 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:13.037 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150150+0000", 2024-09-21T15:50:13.037 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5310000000000003e-06 2024-09-21T15:50:13.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.038 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:13.038 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200356+0000", 2024-09-21T15:50:13.038 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050205759000000003 2024-09-21T15:50:13.038 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.038 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.038 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:13.038 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200415+0000", 2024-09-21T15:50:13.038 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9719999999999997e-05 2024-09-21T15:50:13.038 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.038 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.038 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:13.038 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200483+0000", 2024-09-21T15:50:13.038 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7959000000000004e-05 2024-09-21T15:50:13.038 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.038 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:13.039 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.039 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.039 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.039 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62888 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 331776~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:13.039 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150154+0000", 2024-09-21T15:50:13.039 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 35.846532285000002, 2024-09-21T15:50:13.039 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050483169000000001, 2024-09-21T15:50:13.039 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:13.039 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:13.039 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:13.039 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:13.039 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:13.039 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62888 2024-09-21T15:50:13.039 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.040 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:13.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.040 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:13.040 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150154+0000", 2024-09-21T15:50:13.040 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.040 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.040 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:13.040 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150154+0000", 2024-09-21T15:50:13.040 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.040 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.040 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:13.040 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150153+0000", 2024-09-21T15:50:13.040 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:13.040 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.041 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.041 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:13.041 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150155+0000", 2024-09-21T15:50:13.041 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9920000000000002e-06 2024-09-21T15:50:13.041 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.041 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.041 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:13.041 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150161+0000", 2024-09-21T15:50:13.041 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5439999999999998e-06 2024-09-21T15:50:13.041 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.041 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.041 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:13.041 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150165+0000", 2024-09-21T15:50:13.041 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3460000000000004e-06 2024-09-21T15:50:13.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.042 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.042 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:13.042 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200502+0000", 2024-09-21T15:50:13.042 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050337150999999997 2024-09-21T15:50:13.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.042 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.042 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:13.042 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200575+0000", 2024-09-21T15:50:13.042 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2250999999999997e-05 2024-09-21T15:50:13.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.042 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.042 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:13.042 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200637+0000", 2024-09-21T15:50:13.043 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2195999999999999e-05 2024-09-21T15:50:13.043 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.043 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:13.043 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.043 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.043 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.043 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62889 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 339968~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:13.043 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150169+0000", 2024-09-21T15:50:13.043 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 35.846516786000002, 2024-09-21T15:50:13.043 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050629678999999997, 2024-09-21T15:50:13.043 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:13.043 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:13.043 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:13.043 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:13.044 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:13.044 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62889 2024-09-21T15:50:13.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.044 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:13.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.044 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:13.044 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150169+0000", 2024-09-21T15:50:13.044 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.044 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:13.044 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150169+0000", 2024-09-21T15:50:13.044 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.044 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:13.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150169+0000", 2024-09-21T15:50:13.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:13.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:13.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150171+0000", 2024-09-21T15:50:13.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3910000000000001e-06 2024-09-21T15:50:13.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:13.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150177+0000", 2024-09-21T15:50:13.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4079999999999997e-06 2024-09-21T15:50:13.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:13.046 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150182+0000", 2024-09-21T15:50:13.046 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3890000000000004e-06 2024-09-21T15:50:13.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.046 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:13.046 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200664+0000", 2024-09-21T15:50:13.046 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050482212999999998 2024-09-21T15:50:13.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.046 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:13.046 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200738+0000", 2024-09-21T15:50:13.046 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4114000000000002e-05 2024-09-21T15:50:13.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.046 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:13.047 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200799+0000", 2024-09-21T15:50:13.047 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0503e-05 2024-09-21T15:50:13.047 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.047 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:13.047 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.047 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.047 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.047 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62891 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 348160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:13.047 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150186+0000", 2024-09-21T15:50:13.047 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 35.846499643999998, 2024-09-21T15:50:13.047 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050794552, 2024-09-21T15:50:13.047 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:13.047 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:13.047 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:13.048 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:13.048 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:13.048 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62891 2024-09-21T15:50:13.048 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.048 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:13.048 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.048 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:13.048 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150186+0000", 2024-09-21T15:50:13.048 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.048 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.048 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.048 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:13.048 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150186+0000", 2024-09-21T15:50:13.048 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.048 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.049 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.049 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:13.049 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150186+0000", 2024-09-21T15:50:13.049 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:13.049 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.049 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.049 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:13.049 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150193+0000", 2024-09-21T15:50:13.049 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6830000000000002e-06 2024-09-21T15:50:13.049 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.049 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.049 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:13.049 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150199+0000", 2024-09-21T15:50:13.049 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9859999999999999e-06 2024-09-21T15:50:13.049 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.049 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.050 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:13.050 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150203+0000", 2024-09-21T15:50:13.050 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5920000000000002e-06 2024-09-21T15:50:13.050 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.050 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.050 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:13.050 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200835+0000", 2024-09-21T15:50:13.050 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050631783 2024-09-21T15:50:13.050 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.050 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.050 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:13.050 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200894+0000", 2024-09-21T15:50:13.050 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8560999999999997e-05 2024-09-21T15:50:13.050 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.051 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.051 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:13.051 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200981+0000", 2024-09-21T15:50:13.051 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7305999999999994e-05 2024-09-21T15:50:13.051 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.051 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:13.051 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.051 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.051 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62892 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 356352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:13.051 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150208+0000", 2024-09-21T15:50:13.051 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 35.846478255999997, 2024-09-21T15:50:13.051 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050951769000000001, 2024-09-21T15:50:13.051 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:13.052 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:13.052 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:13.052 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:13.052 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:13.052 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62892 2024-09-21T15:50:13.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.052 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:13.052 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.052 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:13.052 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150208+0000", 2024-09-21T15:50:13.052 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.052 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.052 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:13.052 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150208+0000", 2024-09-21T15:50:13.053 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.053 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.053 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.053 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:13.053 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150207+0000", 2024-09-21T15:50:13.053 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:13.053 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.053 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.053 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:13.053 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150210+0000", 2024-09-21T15:50:13.053 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.238e-06 2024-09-21T15:50:13.053 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.053 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.053 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:13.054 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150215+0000", 2024-09-21T15:50:13.054 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6350000000000003e-06 2024-09-21T15:50:13.054 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.054 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.054 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:13.054 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150220+0000", 2024-09-21T15:50:13.054 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.544e-06 2024-09-21T15:50:13.054 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.054 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.054 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:13.054 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201014+0000", 2024-09-21T15:50:13.054 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050793932999999999 2024-09-21T15:50:13.054 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.054 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.055 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:13.055 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201092+0000", 2024-09-21T15:50:13.055 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7961999999999997e-05 2024-09-21T15:50:13.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.055 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.055 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:13.055 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201160+0000", 2024-09-21T15:50:13.055 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7836000000000006e-05 2024-09-21T15:50:13.055 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.055 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:13.055 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.055 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.055 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62894 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 364544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:13.056 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150820+0000", 2024-09-21T15:50:13.056 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 35.845866055999998, 2024-09-21T15:50:13.056 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050489196, 2024-09-21T15:50:13.056 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:13.056 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:13.056 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:13.056 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:13.056 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:13.056 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62894 2024-09-21T15:50:13.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.056 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:13.056 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.056 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:13.056 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150820+0000", 2024-09-21T15:50:13.056 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.057 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:13.057 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150820+0000", 2024-09-21T15:50:13.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.057 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:13.057 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150223+0000", 2024-09-21T15:50:13.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9994035 2024-09-21T15:50:13.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.057 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:13.057 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150825+0000", 2024-09-21T15:50:13.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00060137099999999996 2024-09-21T15:50:13.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.058 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:13.058 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150837+0000", 2024-09-21T15:50:13.058 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2503999999999999e-05 2024-09-21T15:50:13.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.058 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:13.058 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150844+0000", 2024-09-21T15:50:13.058 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6900000000000003e-06 2024-09-21T15:50:13.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.058 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:13.058 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201187+0000", 2024-09-21T15:50:13.058 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050342973999999999 2024-09-21T15:50:13.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.059 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.059 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:13.059 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201250+0000", 2024-09-21T15:50:13.059 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3274000000000004e-05 2024-09-21T15:50:13.059 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.059 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.059 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:13.059 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201309+0000", 2024-09-21T15:50:13.059 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9058000000000003e-05 2024-09-21T15:50:13.059 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.059 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:13.059 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.059 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.059 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.060 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62895 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 380928~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:13.060 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150851+0000", 2024-09-21T15:50:13.060 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 35.845835460000004, 2024-09-21T15:50:13.060 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050611097000000001, 2024-09-21T15:50:13.060 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:13.060 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:13.060 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:13.060 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:13.060 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:13.060 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62895 2024-09-21T15:50:13.060 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.060 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:13.060 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.060 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:13.060 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150851+0000", 2024-09-21T15:50:13.060 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.061 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:13.061 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150851+0000", 2024-09-21T15:50:13.061 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.061 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:13.061 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150850+0000", 2024-09-21T15:50:13.061 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:13.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.061 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:13.061 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150853+0000", 2024-09-21T15:50:13.061 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1039999999999998e-06 2024-09-21T15:50:13.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.062 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:13.062 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150860+0000", 2024-09-21T15:50:13.062 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3180000000000001e-06 2024-09-21T15:50:13.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.062 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:13.062 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150866+0000", 2024-09-21T15:50:13.062 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1599999999999997e-06 2024-09-21T15:50:13.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.062 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:13.062 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201332+0000", 2024-09-21T15:50:13.063 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050466100999999999 2024-09-21T15:50:13.063 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.063 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.063 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:13.063 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201392+0000", 2024-09-21T15:50:13.063 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0053999999999998e-05 2024-09-21T15:50:13.063 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.063 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.063 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:13.063 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201462+0000", 2024-09-21T15:50:13.063 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9861000000000001e-05 2024-09-21T15:50:13.063 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.063 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:13.064 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.064 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.064 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.064 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62896 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 372736~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:13.064 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150870+0000", 2024-09-21T15:50:13.064 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 35.845815680000001, 2024-09-21T15:50:13.064 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050753292999999998, 2024-09-21T15:50:13.064 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:13.064 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:13.064 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:13.064 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:13.064 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:13.064 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62896 2024-09-21T15:50:13.064 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.064 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:13.064 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.065 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:13.065 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150870+0000", 2024-09-21T15:50:13.065 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.065 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:13.065 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150870+0000", 2024-09-21T15:50:13.065 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.065 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:13.065 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150870+0000", 2024-09-21T15:50:13.065 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:13.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.065 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:13.066 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150872+0000", 2024-09-21T15:50:13.066 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.644e-06 2024-09-21T15:50:13.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.066 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.066 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:13.066 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150879+0000", 2024-09-21T15:50:13.066 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3509999999999998e-06 2024-09-21T15:50:13.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.066 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.066 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:13.066 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150884+0000", 2024-09-21T15:50:13.066 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9150000000000004e-06 2024-09-21T15:50:13.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.066 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.067 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:13.067 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201487+0000", 2024-09-21T15:50:13.067 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050603276000000003 2024-09-21T15:50:13.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.067 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:13.067 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201557+0000", 2024-09-21T15:50:13.067 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9847999999999999e-05 2024-09-21T15:50:13.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.067 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:13.067 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201624+0000", 2024-09-21T15:50:13.067 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6730999999999995e-05 2024-09-21T15:50:13.067 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.068 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:13.068 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.068 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.068 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.068 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62897 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 389120~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:13.068 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.151044+0000", 2024-09-21T15:50:13.068 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 35.845641639999997, 2024-09-21T15:50:13.068 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050720531999999999, 2024-09-21T15:50:13.068 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:13.068 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:13.068 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:13.068 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:13.068 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:13.068 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62897 2024-09-21T15:50:13.068 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.069 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:13.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.069 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:13.069 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151044+0000", 2024-09-21T15:50:13.069 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.069 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:13.069 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151044+0000", 2024-09-21T15:50:13.069 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.069 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:13.069 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150888+0000", 2024-09-21T15:50:13.069 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9998431 2024-09-21T15:50:13.070 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.070 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:13.070 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151048+0000", 2024-09-21T15:50:13.070 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00016077299999999999 2024-09-21T15:50:13.070 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.070 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:13.070 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151058+0000", 2024-09-21T15:50:13.070 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.465e-06 2024-09-21T15:50:13.070 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.070 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:13.071 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151064+0000", 2024-09-21T15:50:13.071 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3829999999999997e-06 2024-09-21T15:50:13.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.071 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:13.071 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201648+0000", 2024-09-21T15:50:13.071 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050583597000000001 2024-09-21T15:50:13.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.071 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:13.071 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201711+0000", 2024-09-21T15:50:13.071 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3616999999999998e-05 2024-09-21T15:50:13.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.072 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:13.072 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201765+0000", 2024-09-21T15:50:13.072 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3464000000000002e-05 2024-09-21T15:50:13.072 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.072 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:13.072 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.072 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.072 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.072 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:118351 3.4 3:25980f86:::rbd_data.1133545d3fa6.000000000000005a:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-09-21T15:50:13.072 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:58.975795+0000", 2024-09-21T15:50:13.072 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.020891005999999, 2024-09-21T15:50:13.072 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.99016351599999997, 2024-09-21T15:50:13.072 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:13.072 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:13.072 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:13.073 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:13.073 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:13.073 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 118351 2024-09-21T15:50:13.073 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.073 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:13.073 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.073 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:13.073 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975795+0000", 2024-09-21T15:50:13.073 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.073 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.073 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.073 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:13.073 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975795+0000", 2024-09-21T15:50:13.073 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.073 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.073 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.074 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:13.074 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975793+0000", 2024-09-21T15:50:13.074 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:50:13.074 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.074 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.074 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:13.074 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975800+0000", 2024-09-21T15:50:13.074 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8220000000000001e-06 2024-09-21T15:50:13.074 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.074 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.074 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:13.074 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975814+0000", 2024-09-21T15:50:13.074 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3893000000000001e-05 2024-09-21T15:50:13.074 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:13.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975821+0000", 2024-09-21T15:50:13.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1319999999999996e-06 2024-09-21T15:50:13.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:13.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975878+0000", 2024-09-21T15:50:13.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7238999999999999e-05 2024-09-21T15:50:13.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:50:13.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975889+0000", 2024-09-21T15:50:13.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0509000000000001e-05 2024-09-21T15:50:13.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.076 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:13.076 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.964573+0000", 2024-09-21T15:50:13.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.98868396300000005 2024-09-21T15:50:13.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.076 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:13.076 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.964672+0000", 2024-09-21T15:50:13.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9601999999999999e-05 2024-09-21T15:50:13.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.076 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-09-21T15:50:13.076 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.964758+0000", 2024-09-21T15:50:13.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5537000000000005e-05 2024-09-21T15:50:13.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.077 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.077 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:13.077 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965847+0000", 2024-09-21T15:50:13.077 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001089404 2024-09-21T15:50:13.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.077 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.077 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:13.077 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965890+0000", 2024-09-21T15:50:13.077 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2521999999999997e-05 2024-09-21T15:50:13.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.077 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.077 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:13.077 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965934+0000", 2024-09-21T15:50:13.077 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4489e-05 2024-09-21T15:50:13.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.078 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:13.078 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965959+0000", 2024-09-21T15:50:13.078 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4301999999999999e-05 2024-09-21T15:50:13.078 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.078 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:13.078 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.078 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.078 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:118353 3.4 3:23841e18:::rbd_data.1133545d3fa6.0000000000000061:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-09-21T15:50:13.078 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:58.975836+0000", 2024-09-21T15:50:13.078 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.020849506999999, 2024-09-21T15:50:13.078 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.99077746700000002, 2024-09-21T15:50:13.078 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:13.079 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:13.079 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:13.079 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:13.079 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:13.079 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 118353 2024-09-21T15:50:13.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.079 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:13.079 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.079 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:13.079 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975836+0000", 2024-09-21T15:50:13.079 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.079 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.079 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:13.079 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975836+0000", 2024-09-21T15:50:13.079 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.080 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:13.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975836+0000", 2024-09-21T15:50:13.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:13.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.080 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:13.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975839+0000", 2024-09-21T15:50:13.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6129999999999998e-06 2024-09-21T15:50:13.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.080 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:13.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975844+0000", 2024-09-21T15:50:13.081 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5130000000000004e-06 2024-09-21T15:50:13.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.081 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:13.081 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975850+0000", 2024-09-21T15:50:13.081 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3179999999999997e-06 2024-09-21T15:50:13.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.081 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:13.081 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975915+0000", 2024-09-21T15:50:13.081 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5168999999999999e-05 2024-09-21T15:50:13.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.081 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:50:13.081 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975922+0000", 2024-09-21T15:50:13.082 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2749999999999998e-06 2024-09-21T15:50:13.082 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.082 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.082 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:13.082 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965431+0000", 2024-09-21T15:50:13.082 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.98950864699999996 2024-09-21T15:50:13.082 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.082 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.082 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:13.082 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965535+0000", 2024-09-21T15:50:13.082 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000103949 2024-09-21T15:50:13.082 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.082 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.082 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-09-21T15:50:13.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965607+0000", 2024-09-21T15:50:13.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2478000000000006e-05 2024-09-21T15:50:13.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.083 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:13.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.966205+0000", 2024-09-21T15:50:13.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00059775200000000001 2024-09-21T15:50:13.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.083 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:13.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.966562+0000", 2024-09-21T15:50:13.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00035732199999999998 2024-09-21T15:50:13.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.084 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:13.084 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.966593+0000", 2024-09-21T15:50:13.084 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0642999999999998e-05 2024-09-21T15:50:13.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.084 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:13.084 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.966614+0000", 2024-09-21T15:50:13.084 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1149999999999999e-05 2024-09-21T15:50:13.084 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.084 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:13.084 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.084 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.084 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:13.084 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:13.085 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [0] live_osds: [1, 2, 3, 4, 5, 6, 7] 2024-09-21T15:50:13.085 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T15:50:13.085 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-09-21T15:50:13.085 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T15:50:13.085 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T15:50:13.086 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok config set filestore_inject_stall 3 2024-09-21T15:50:13.087 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:13.084+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:13.087 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:50:13.088 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is unset 2024-09-21T15:50:13.186 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:13.185+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:13.208 INFO:teuthology.orchestra.run.smithi089.stdout:checking clone #33, image image_client.0-clone33 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent34 2024-09-21T15:50:13.237 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:13.237 INFO:teuthology.orchestra.run.smithi089.stdout: "success": "" 2024-09-21T15:50:13.237 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:13.238 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:13.238 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:13.238 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:50:13.238 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:13.252 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:50:13.287 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:13.286+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:13.387 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:13.386+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:13.397 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:13.398 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:13.398 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:50:13.398 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:50:13.398 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:13.407 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:50:13.458 INFO:teuthology.orchestra.run.smithi089.stdout:5015 write 0xb59e0ee thru 0xb5a383f (0x5752 bytes) 2024-09-21T15:50:13.471 INFO:teuthology.orchestra.run.smithi089.stdout:5016 trunc from 0xb5a3840 to 0x3861009 2024-09-21T15:50:13.489 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:13.488+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:13.496 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:13.505 INFO:teuthology.orchestra.run.smithi089.stdout:5020 trunc from 0x3861009 to 0xac37e0e 2024-09-21T15:50:13.510 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:13.510 INFO:teuthology.orchestra.run.smithi089.stdout:5023 punch from 0x4b8b22f to 0x4b8c570, (0x1341 bytes) 2024-09-21T15:50:13.520 INFO:teuthology.orchestra.run.smithi089.stdout:5024 trunc from 0xac37e0e to 0x254aea6 2024-09-21T15:50:13.531 INFO:teuthology.orchestra.run.smithi089.stdout:5025 writesame 0x17a19c7 thru 0x17a6ba7 data_size 0x11(0x51e1 bytes) 2024-09-21T15:50:13.546 INFO:teuthology.orchestra.run.smithi089.stdout:5026 read 0x22efd0 thru 0x231127 (0x2158 bytes) 2024-09-21T15:50:13.547 INFO:teuthology.orchestra.run.smithi089.stdout:5027 read 0x16c231d thru 0x16c8e05 (0x6ae9 bytes) 2024-09-21T15:50:13.564 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:13.564 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:50:13.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:50:13.564 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:50:13.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.564 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:984 3.0 e27/22 3:025e53e7:::rbd_data.1133f3dfbd6b.0000000000000008:head v 27'21, mlcod=27'19)", 2024-09-21T15:50:13.564 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.619393+0000", 2024-09-21T15:50:13.564 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 60.942493218999999, 2024-09-21T15:50:13.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003401837, 2024-09-21T15:50:13.564 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:13.564 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:13.565 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:13.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.565 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:13.565 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619393+0000", 2024-09-21T15:50:13.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.565 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:13.565 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619393+0000", 2024-09-21T15:50:13.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.565 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:13.565 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619392+0000", 2024-09-21T15:50:13.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:13.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.566 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:13.566 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619439+0000", 2024-09-21T15:50:13.566 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7170999999999999e-05 2024-09-21T15:50:13.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.566 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:13.566 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619452+0000", 2024-09-21T15:50:13.566 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2986999999999999e-05 2024-09-21T15:50:13.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.566 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:13.566 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619459+0000", 2024-09-21T15:50:13.566 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6109999999999999e-06 2024-09-21T15:50:13.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.567 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.567 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:13.567 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619518+0000", 2024-09-21T15:50:13.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9117999999999999e-05 2024-09-21T15:50:13.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.567 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.567 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:13.567 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619532+0000", 2024-09-21T15:50:13.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4622e-05 2024-09-21T15:50:13.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.567 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.567 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:13.567 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.622760+0000", 2024-09-21T15:50:13.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032278839999999999 2024-09-21T15:50:13.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.568 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:13.568 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.622794+0000", 2024-09-21T15:50:13.568 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4144999999999999e-05 2024-09-21T15:50:13.568 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.568 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:13.568 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.568 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:4957 3.9 e29/22 3:95c5ed95:::rbd_data.11335fa2fe69.0000000000000018:head v 29'4, mlcod=26'2)", 2024-09-21T15:50:13.568 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:14.453951+0000", 2024-09-21T15:50:13.568 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 59.107934786999998, 2024-09-21T15:50:13.568 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0043804559999999996, 2024-09-21T15:50:13.568 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:13.568 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:13.568 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:13.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.569 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:13.569 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.453951+0000", 2024-09-21T15:50:13.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.569 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:13.569 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.453951+0000", 2024-09-21T15:50:13.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.569 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:13.569 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.453950+0000", 2024-09-21T15:50:13.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:13.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:13.570 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454495+0000", 2024-09-21T15:50:13.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00054437799999999998 2024-09-21T15:50:13.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:13.570 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454508+0000", 2024-09-21T15:50:13.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3475000000000001e-05 2024-09-21T15:50:13.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:13.570 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454515+0000", 2024-09-21T15:50:13.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8990000000000004e-06 2024-09-21T15:50:13.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:13.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454581+0000", 2024-09-21T15:50:13.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5987000000000004e-05 2024-09-21T15:50:13.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:13.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454609+0000", 2024-09-21T15:50:13.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7466000000000001e-05 2024-09-21T15:50:13.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:13.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.458303+0000", 2024-09-21T15:50:13.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003693979 2024-09-21T15:50:13.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:13.572 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.458331+0000", 2024-09-21T15:50:13.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8932999999999999e-05 2024-09-21T15:50:13.572 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.572 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:13.572 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.572 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13420 3.0 e31/22 3:038b8840:::rbd_data.1133dc5adfdf.0000000000000035:head v 31'49, mlcod=31'47)", 2024-09-21T15:50:13.572 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.209233+0000", 2024-09-21T15:50:13.572 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.352653171, 2024-09-21T15:50:13.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0042624960000000002, 2024-09-21T15:50:13.572 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:13.572 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:13.572 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:13.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:13.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209233+0000", 2024-09-21T15:50:13.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:13.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209233+0000", 2024-09-21T15:50:13.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:13.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209232+0000", 2024-09-21T15:50:13.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:13.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.574 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:13.574 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209251+0000", 2024-09-21T15:50:13.574 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9304e-05 2024-09-21T15:50:13.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.574 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:13.574 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209265+0000", 2024-09-21T15:50:13.574 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3532e-05 2024-09-21T15:50:13.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.574 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:13.574 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209272+0000", 2024-09-21T15:50:13.574 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.289e-06 2024-09-21T15:50:13.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:13.575 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209321+0000", 2024-09-21T15:50:13.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8773999999999998e-05 2024-09-21T15:50:13.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:13.575 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209341+0000", 2024-09-21T15:50:13.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0387999999999999e-05 2024-09-21T15:50:13.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:13.575 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.213464+0000", 2024-09-21T15:50:13.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0041232450000000002 2024-09-21T15:50:13.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:13.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.213495+0000", 2024-09-21T15:50:13.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0793999999999997e-05 2024-09-21T15:50:13.576 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.576 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:13.576 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.576 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13445 3.0 e31/22 3:038b8840:::rbd_data.1133dc5adfdf.0000000000000035:head v 31'50, mlcod=31'48)", 2024-09-21T15:50:13.576 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.232497+0000", 2024-09-21T15:50:13.576 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.329389159000002, 2024-09-21T15:50:13.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0040493559999999996, 2024-09-21T15:50:13.576 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:13.576 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:13.576 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:13.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.576 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:13.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232497+0000", 2024-09-21T15:50:13.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:13.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232497+0000", 2024-09-21T15:50:13.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:13.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232496+0000", 2024-09-21T15:50:13.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:13.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:13.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232503+0000", 2024-09-21T15:50:13.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7320000000000001e-06 2024-09-21T15:50:13.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.578 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:13.578 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232512+0000", 2024-09-21T15:50:13.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9259999999999995e-06 2024-09-21T15:50:13.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.578 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:13.578 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232520+0000", 2024-09-21T15:50:13.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3699999999999997e-06 2024-09-21T15:50:13.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.578 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:13.578 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232564+0000", 2024-09-21T15:50:13.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4128999999999997e-05 2024-09-21T15:50:13.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:13.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232585+0000", 2024-09-21T15:50:13.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0704999999999999e-05 2024-09-21T15:50:13.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:13.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.236507+0000", 2024-09-21T15:50:13.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0039218730000000002 2024-09-21T15:50:13.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:13.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.236546+0000", 2024-09-21T15:50:13.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9399e-05 2024-09-21T15:50:13.579 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.579 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:13.580 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.580 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13452 3.9 e31/22 3:93f5396a:::rbd_data.1133dc5adfdf.0000000000000078:head v 31'50, mlcod=31'47)", 2024-09-21T15:50:13.580 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.237263+0000", 2024-09-21T15:50:13.580 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.324622453000003, 2024-09-21T15:50:13.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033768180000000002, 2024-09-21T15:50:13.580 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:13.580 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:13.580 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:13.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:13.580 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237263+0000", 2024-09-21T15:50:13.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:13.581 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237263+0000", 2024-09-21T15:50:13.581 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.581 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.581 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:13.581 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237263+0000", 2024-09-21T15:50:13.581 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:13.581 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.581 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:13.581 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237269+0000", 2024-09-21T15:50:13.581 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.472e-06 2024-09-21T15:50:13.581 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.581 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:13.581 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237282+0000", 2024-09-21T15:50:13.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2788e-05 2024-09-21T15:50:13.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.582 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.582 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:13.582 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237288+0000", 2024-09-21T15:50:13.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4899999999999997e-06 2024-09-21T15:50:13.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.582 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.582 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:13.582 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237331+0000", 2024-09-21T15:50:13.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2425000000000001e-05 2024-09-21T15:50:13.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.582 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.582 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:13.582 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237349+0000", 2024-09-21T15:50:13.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8405000000000001e-05 2024-09-21T15:50:13.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:13.583 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.240610+0000", 2024-09-21T15:50:13.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032605379999999999 2024-09-21T15:50:13.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:13.583 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.240640+0000", 2024-09-21T15:50:13.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0448000000000001e-05 2024-09-21T15:50:13.583 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.583 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:13.583 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.583 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13832 3.0 e31/22 3:0792efd2:::rbd_data.1133dc5adfdf.0000000000000090:head v 31'58, mlcod=31'56)", 2024-09-21T15:50:13.584 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.527530+0000", 2024-09-21T15:50:13.584 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.034355632999997, 2024-09-21T15:50:13.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035256530000000001, 2024-09-21T15:50:13.584 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:13.584 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:13.584 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:13.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:13.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527530+0000", 2024-09-21T15:50:13.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:13.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527530+0000", 2024-09-21T15:50:13.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.585 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:13.585 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527529+0000", 2024-09-21T15:50:13.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:50:13.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.585 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:13.585 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527550+0000", 2024-09-21T15:50:13.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1387999999999999e-05 2024-09-21T15:50:13.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.585 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:13.585 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527568+0000", 2024-09-21T15:50:13.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8068e-05 2024-09-21T15:50:13.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.586 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:13.586 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527577+0000", 2024-09-21T15:50:13.586 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9919999999999996e-06 2024-09-21T15:50:13.586 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.586 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:13.586 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527626+0000", 2024-09-21T15:50:13.586 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8597000000000002e-05 2024-09-21T15:50:13.586 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.586 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:13.586 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527652+0000", 2024-09-21T15:50:13.586 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5993e-05 2024-09-21T15:50:13.586 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.586 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:13.586 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.531024+0000", 2024-09-21T15:50:13.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033721910000000001 2024-09-21T15:50:13.587 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.587 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:13.587 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.531056+0000", 2024-09-21T15:50:13.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1628000000000003e-05 2024-09-21T15:50:13.587 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.587 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:13.587 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.587 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.587 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13839 3.0 e31/22 3:03c4fb32:::rbd_data.1133dc5adfdf.0000000000000099:head v 31'59, mlcod=31'56)", 2024-09-21T15:50:13.587 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.528097+0000", 2024-09-21T15:50:13.587 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.033789034999998, 2024-09-21T15:50:13.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0057228840000000001, 2024-09-21T15:50:13.588 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:13.588 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:13.588 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:13.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.588 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:13.588 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528097+0000", 2024-09-21T15:50:13.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.588 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.588 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:13.588 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528097+0000", 2024-09-21T15:50:13.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.588 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.588 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:13.589 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528096+0000", 2024-09-21T15:50:13.589 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:13.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.589 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.589 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:13.589 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528105+0000", 2024-09-21T15:50:13.589 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7350000000000002e-06 2024-09-21T15:50:13.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.589 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.589 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:13.589 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528117+0000", 2024-09-21T15:50:13.589 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1909000000000001e-05 2024-09-21T15:50:13.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.589 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.589 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:13.590 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528124+0000", 2024-09-21T15:50:13.590 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8419999999999999e-06 2024-09-21T15:50:13.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.590 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:13.590 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528168+0000", 2024-09-21T15:50:13.590 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4301999999999997e-05 2024-09-21T15:50:13.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.590 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:13.590 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528191+0000", 2024-09-21T15:50:13.590 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2622000000000001e-05 2024-09-21T15:50:13.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.590 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:13.591 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.533780+0000", 2024-09-21T15:50:13.591 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0055897519999999999 2024-09-21T15:50:13.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.591 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.591 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:13.591 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.533820+0000", 2024-09-21T15:50:13.591 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9366999999999998e-05 2024-09-21T15:50:13.591 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.591 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:13.591 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.591 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.591 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13884 3.9 e31/22 3:93003736:::rbd_data.1133dc5adfdf.00000000000000a1:head v 31'54, mlcod=31'52)", 2024-09-21T15:50:13.591 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.562433+0000", 2024-09-21T15:50:13.591 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 55.999452286999997, 2024-09-21T15:50:13.591 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.005162313, 2024-09-21T15:50:13.592 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:13.592 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:13.592 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:13.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.592 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:13.592 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562433+0000", 2024-09-21T15:50:13.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.592 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:13.592 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562433+0000", 2024-09-21T15:50:13.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.592 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:13.592 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562432+0000", 2024-09-21T15:50:13.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:13.593 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.593 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.593 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:13.593 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562467+0000", 2024-09-21T15:50:13.593 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4885999999999998e-05 2024-09-21T15:50:13.593 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.593 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.593 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:13.593 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562481+0000", 2024-09-21T15:50:13.593 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3604000000000001e-05 2024-09-21T15:50:13.593 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.593 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.593 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:13.593 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562489+0000", 2024-09-21T15:50:13.593 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1829999999999994e-06 2024-09-21T15:50:13.593 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.593 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.594 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:13.594 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562536+0000", 2024-09-21T15:50:13.594 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6830999999999999e-05 2024-09-21T15:50:13.594 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.594 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.594 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:13.594 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562559+0000", 2024-09-21T15:50:13.594 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2765999999999999e-05 2024-09-21T15:50:13.594 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.594 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.594 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:13.594 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.567566+0000", 2024-09-21T15:50:13.594 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0050073219999999998 2024-09-21T15:50:13.594 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.594 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.595 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:13.595 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.567596+0000", 2024-09-21T15:50:13.595 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9802999999999999e-05 2024-09-21T15:50:13.595 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.595 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:13.595 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.595 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.595 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:22340 3.9 e34/22 3:910a5abe:::rbd_data.1133896adcfd.0000000000000023:head v 34'92, mlcod=34'90)", 2024-09-21T15:50:13.595 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:21.628025+0000", 2024-09-21T15:50:13.595 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 51.933860907000003, 2024-09-21T15:50:13.595 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046196010000000001, 2024-09-21T15:50:13.595 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:13.595 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:13.595 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:13.595 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:13.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628025+0000", 2024-09-21T15:50:13.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:13.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628025+0000", 2024-09-21T15:50:13.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:13.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628024+0000", 2024-09-21T15:50:13.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:13.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:13.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628034+0000", 2024-09-21T15:50:13.597 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3470000000000005e-06 2024-09-21T15:50:13.597 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.597 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.597 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:13.597 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628043+0000", 2024-09-21T15:50:13.597 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9450000000000005e-06 2024-09-21T15:50:13.597 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.597 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.597 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:13.597 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628048+0000", 2024-09-21T15:50:13.597 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8080000000000004e-06 2024-09-21T15:50:13.597 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.597 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.597 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:13.597 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628081+0000", 2024-09-21T15:50:13.597 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2738999999999999e-05 2024-09-21T15:50:13.597 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.598 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.598 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:13.598 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628094+0000", 2024-09-21T15:50:13.598 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3273e-05 2024-09-21T15:50:13.598 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.598 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.598 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:13.598 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.632616+0000", 2024-09-21T15:50:13.598 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045217269999999997 2024-09-21T15:50:13.598 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.598 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.598 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:13.598 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.632644+0000", 2024-09-21T15:50:13.598 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8469000000000001e-05 2024-09-21T15:50:13.598 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.598 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:13.599 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.599 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.599 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.599 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:22417 3.4 e34/22 3:25310a10:::rbd_data.1133896adcfd.0000000000000075:head v 34'48, mlcod=34'46)", 2024-09-21T15:50:13.599 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:21.704297+0000", 2024-09-21T15:50:13.599 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 51.857588808999999, 2024-09-21T15:50:13.599 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0038101799999999998, 2024-09-21T15:50:13.599 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:13.599 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:13.599 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:13.599 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.599 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:13.599 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704297+0000", 2024-09-21T15:50:13.599 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.599 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.599 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.599 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:13.600 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704297+0000", 2024-09-21T15:50:13.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.600 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.600 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.600 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:13.600 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704296+0000", 2024-09-21T15:50:13.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:50:13.600 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.600 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.600 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:13.600 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704308+0000", 2024-09-21T15:50:13.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2559e-05 2024-09-21T15:50:13.600 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.600 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.600 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:13.600 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704337+0000", 2024-09-21T15:50:13.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8960000000000001e-05 2024-09-21T15:50:13.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.601 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.601 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:13.601 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704345+0000", 2024-09-21T15:50:13.601 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3059999999999998e-06 2024-09-21T15:50:13.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.601 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.601 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:13.601 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704408+0000", 2024-09-21T15:50:13.601 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2156000000000006e-05 2024-09-21T15:50:13.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.601 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.601 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:13.601 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704430+0000", 2024-09-21T15:50:13.601 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1931000000000001e-05 2024-09-21T15:50:13.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.602 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.602 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:13.602 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.708073+0000", 2024-09-21T15:50:13.602 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003643028 2024-09-21T15:50:13.602 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.602 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.602 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:13.602 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.708107+0000", 2024-09-21T15:50:13.602 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4613999999999998e-05 2024-09-21T15:50:13.602 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.602 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:13.602 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.602 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.603 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:25278 3.4 e38/22 3:2251cc1c:::rbd_data.1133ebd6d3ba.0000000000000024:head v 38'51, mlcod=38'49)", 2024-09-21T15:50:13.603 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.317337+0000", 2024-09-21T15:50:13.603 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 47.244549049, 2024-09-21T15:50:13.603 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053904110000000003, 2024-09-21T15:50:13.603 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:13.603 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:13.603 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:13.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.603 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:13.603 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317337+0000", 2024-09-21T15:50:13.603 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.603 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:13.603 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317337+0000", 2024-09-21T15:50:13.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.604 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:13.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317335+0000", 2024-09-21T15:50:13.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:50:13.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.604 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:13.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317347+0000", 2024-09-21T15:50:13.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.19e-05 2024-09-21T15:50:13.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.604 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:13.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317369+0000", 2024-09-21T15:50:13.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2232e-05 2024-09-21T15:50:13.605 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.605 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.605 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:13.605 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317381+0000", 2024-09-21T15:50:13.605 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2231e-05 2024-09-21T15:50:13.605 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.605 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.605 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:13.605 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317425+0000", 2024-09-21T15:50:13.605 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3625000000000003e-05 2024-09-21T15:50:13.605 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.605 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.605 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:13.605 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317449+0000", 2024-09-21T15:50:13.605 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4301999999999999e-05 2024-09-21T15:50:13.606 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.606 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.606 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:13.606 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.322688+0000", 2024-09-21T15:50:13.606 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.005238609 2024-09-21T15:50:13.606 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.606 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.606 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:13.606 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.322727+0000", 2024-09-21T15:50:13.606 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9201e-05 2024-09-21T15:50:13.606 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.606 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:13.606 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.606 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.606 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.606 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:25286 3.0 e38/22 3:0248fb60:::rbd_data.1133ebd6d3ba.0000000000000066:head v 38'109, mlcod=34'107)", 2024-09-21T15:50:13.606 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.317638+0000", 2024-09-21T15:50:13.607 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 47.244248169999999, 2024-09-21T15:50:13.607 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0054074370000000002, 2024-09-21T15:50:13.607 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:13.607 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:13.607 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:13.607 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.607 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:13.607 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317638+0000", 2024-09-21T15:50:13.607 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.607 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.607 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.607 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:13.607 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317638+0000", 2024-09-21T15:50:13.607 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.607 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.608 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.608 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:13.608 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317637+0000", 2024-09-21T15:50:13.608 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:13.608 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.608 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.608 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:13.608 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317643+0000", 2024-09-21T15:50:13.608 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5609999999999998e-06 2024-09-21T15:50:13.608 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.608 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.608 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:13.608 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317653+0000", 2024-09-21T15:50:13.608 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0705e-05 2024-09-21T15:50:13.608 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.608 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.608 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:13.609 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317660+0000", 2024-09-21T15:50:13.609 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2099999999999998e-06 2024-09-21T15:50:13.609 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.609 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.609 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:13.609 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317704+0000", 2024-09-21T15:50:13.609 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4659000000000001e-05 2024-09-21T15:50:13.609 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.609 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.609 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:13.609 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317725+0000", 2024-09-21T15:50:13.609 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0537e-05 2024-09-21T15:50:13.609 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.609 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.609 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:13.609 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323010+0000", 2024-09-21T15:50:13.609 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052853420000000002 2024-09-21T15:50:13.610 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.610 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.610 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:13.610 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323045+0000", 2024-09-21T15:50:13.610 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4860000000000002e-05 2024-09-21T15:50:13.610 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.610 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:13.610 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.610 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.610 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.610 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:25280 3.4 e38/22 3:27fca38c:::rbd_data.1133ebd6d3ba.0000000000000026:head v 38'52, mlcod=38'49)", 2024-09-21T15:50:13.610 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.317684+0000", 2024-09-21T15:50:13.610 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 47.244202274000003, 2024-09-21T15:50:13.610 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053570809999999996, 2024-09-21T15:50:13.610 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:13.610 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:13.611 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:13.611 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.611 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:13.611 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317684+0000", 2024-09-21T15:50:13.611 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.611 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.611 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.611 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:13.611 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317684+0000", 2024-09-21T15:50:13.611 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.611 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.611 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.611 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:13.611 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317683+0000", 2024-09-21T15:50:13.611 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:13.611 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.612 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.612 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:13.612 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317688+0000", 2024-09-21T15:50:13.612 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5009999999999998e-06 2024-09-21T15:50:13.612 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.612 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.612 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:13.612 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317694+0000", 2024-09-21T15:50:13.612 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3579999999999999e-06 2024-09-21T15:50:13.612 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.612 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.612 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:13.612 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317698+0000", 2024-09-21T15:50:13.612 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4150000000000001e-06 2024-09-21T15:50:13.613 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.613 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.613 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:13.613 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317738+0000", 2024-09-21T15:50:13.613 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9575e-05 2024-09-21T15:50:13.613 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.613 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.613 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:13.613 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317762+0000", 2024-09-21T15:50:13.613 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4173999999999999e-05 2024-09-21T15:50:13.613 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.613 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.613 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:13.613 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323005+0000", 2024-09-21T15:50:13.613 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00524333 2024-09-21T15:50:13.613 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.613 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.614 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:13.614 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323041+0000", 2024-09-21T15:50:13.614 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5139999999999999e-05 2024-09-21T15:50:13.614 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.614 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:13.614 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.614 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.614 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.614 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:34837 3.0 e43/22 3:038527c8:::rbd_header.1133fed1a840:head v 43'130, mlcod=43'128)", 2024-09-21T15:50:13.614 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:50:13.614 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.577997576000001, 2024-09-21T15:50:13.614 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045776519999999998, 2024-09-21T15:50:13.614 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:13.614 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:13.614 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:13.615 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.615 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:13.615 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:50:13.615 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.615 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.615 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.615 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:13.615 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:50:13.615 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.615 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.615 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.615 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:13.615 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:50:13.615 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:13.615 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.615 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.616 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:13.616 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983892+0000", 2024-09-21T15:50:13.616 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8450000000000002e-06 2024-09-21T15:50:13.616 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.616 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.616 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:13.616 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983902+0000", 2024-09-21T15:50:13.616 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0299999999999999e-06 2024-09-21T15:50:13.616 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.616 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.616 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:13.616 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983906+0000", 2024-09-21T15:50:13.616 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2300000000000002e-06 2024-09-21T15:50:13.616 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.616 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.616 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:13.616 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983940+0000", 2024-09-21T15:50:13.617 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4563000000000001e-05 2024-09-21T15:50:13.617 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.617 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.617 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:13.617 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983960+0000", 2024-09-21T15:50:13.617 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9199999999999999e-05 2024-09-21T15:50:13.617 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.617 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.617 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:13.617 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.988430+0000", 2024-09-21T15:50:13.617 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0044709149999999998 2024-09-21T15:50:13.617 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.617 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.617 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:13.617 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.988466+0000", 2024-09-21T15:50:13.617 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5420000000000003e-05 2024-09-21T15:50:13.617 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.618 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:13.618 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.618 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.618 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.618 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "pg_update_log_missing(3.0 epoch 50/22 rep_tid 498 entries 50'171 (0'0) error 3:5df4a5a0:::rbd_data.11339e87a32a.000000000000008e:head by client.4403.0:80207 0.000000 -2 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 47'167)", 2024-09-21T15:50:13.618 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:50:13.618 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 32.620124703000002, 2024-09-21T15:50:13.618 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0047253119999999997, 2024-09-21T15:50:13.618 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:13.618 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-09-21T15:50:13.618 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:13.618 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.618 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:13.618 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:50:13.618 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.618 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.619 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.619 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:13.619 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:50:13.619 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.619 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.619 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.619 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:13.619 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:50:13.619 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:13.619 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.619 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.619 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:13.619 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941768+0000", 2024-09-21T15:50:13.619 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7309999999999993e-06 2024-09-21T15:50:13.619 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.620 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.620 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:13.620 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941780+0000", 2024-09-21T15:50:13.620 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1399999999999999e-05 2024-09-21T15:50:13.620 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.620 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.620 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:13.620 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941789+0000", 2024-09-21T15:50:13.620 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5230000000000001e-06 2024-09-21T15:50:13.620 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.620 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.620 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:13.620 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941841+0000", 2024-09-21T15:50:13.620 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1863999999999997e-05 2024-09-21T15:50:13.620 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.620 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.621 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:13.621 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.946486+0000", 2024-09-21T15:50:13.621 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046452539999999997 2024-09-21T15:50:13.621 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.621 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:13.621 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.621 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.621 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.621 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:82086 3.0 e50/22 3:0114bc4d:::rbd_data.11334c14b8fd.0000000000000069:head v 50'178, mlcod=50'176)", 2024-09-21T15:50:13.621 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:41.155591+0000", 2024-09-21T15:50:13.621 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 32.406294391000003, 2024-09-21T15:50:13.621 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003363497, 2024-09-21T15:50:13.621 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:13.621 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:13.621 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:13.621 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.621 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:13.622 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.155591+0000", 2024-09-21T15:50:13.622 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.622 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.622 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.622 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:13.622 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.155591+0000", 2024-09-21T15:50:13.622 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.622 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.622 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.622 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:13.622 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.155590+0000", 2024-09-21T15:50:13.622 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:50:13.622 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.622 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.622 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:13.622 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.155606+0000", 2024-09-21T15:50:13.622 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6075000000000001e-05 2024-09-21T15:50:13.623 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.623 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.623 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:13.623 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.155623+0000", 2024-09-21T15:50:13.623 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6838e-05 2024-09-21T15:50:13.623 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.623 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.623 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:13.623 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.155630+0000", 2024-09-21T15:50:13.623 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.627e-06 2024-09-21T15:50:13.623 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.623 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.623 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:13.623 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.155677+0000", 2024-09-21T15:50:13.623 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6669999999999999e-05 2024-09-21T15:50:13.623 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.623 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.624 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:13.624 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.155698+0000", 2024-09-21T15:50:13.624 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1330000000000001e-05 2024-09-21T15:50:13.624 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.624 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.624 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:13.624 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.158918+0000", 2024-09-21T15:50:13.624 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032198600000000002 2024-09-21T15:50:13.624 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.624 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.624 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:13.624 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.158955+0000", 2024-09-21T15:50:13.624 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6647000000000003e-05 2024-09-21T15:50:13.624 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.624 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:13.625 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.625 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.625 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.625 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:82470 3.4 e50/22 3:21ad7dfe:::rbd_data.11334c14b8fd.00000000000003a7:head v 50'190, mlcod=46'188)", 2024-09-21T15:50:13.625 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:41.291787+0000", 2024-09-21T15:50:13.625 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 32.270098765999997, 2024-09-21T15:50:13.625 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033489969999999998, 2024-09-21T15:50:13.625 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:13.625 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:13.625 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:13.625 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.625 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:13.625 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.291787+0000", 2024-09-21T15:50:13.625 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.625 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.626 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.626 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:13.626 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.291787+0000", 2024-09-21T15:50:13.626 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.626 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.626 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.626 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:13.626 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.291786+0000", 2024-09-21T15:50:13.626 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:13.626 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.626 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.626 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:13.626 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.291795+0000", 2024-09-21T15:50:13.626 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7830000000000004e-06 2024-09-21T15:50:13.626 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.626 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.627 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:13.627 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.291814+0000", 2024-09-21T15:50:13.627 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9117999999999999e-05 2024-09-21T15:50:13.627 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.627 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.627 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:13.627 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.291822+0000", 2024-09-21T15:50:13.627 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0669999999999992e-06 2024-09-21T15:50:13.627 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.627 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.627 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:13.627 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.291866+0000", 2024-09-21T15:50:13.627 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4669000000000002e-05 2024-09-21T15:50:13.627 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.627 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.627 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:13.627 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.291888+0000", 2024-09-21T15:50:13.628 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1087999999999999e-05 2024-09-21T15:50:13.628 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.628 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.628 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:13.628 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.295104+0000", 2024-09-21T15:50:13.628 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032164910000000001 2024-09-21T15:50:13.628 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.628 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.628 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:13.628 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.295136+0000", 2024-09-21T15:50:13.628 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1931999999999999e-05 2024-09-21T15:50:13.628 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.628 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:13.628 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.628 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.628 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.629 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "pg_update_log_missing(3.0 epoch 53/22 rep_tid 585 entries 53'201 (0'0) error 3:6a1d4890:::rbd_data.1133e079cae4.0000000000000004:head by client.4403.0:101089 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 53'198)", 2024-09-21T15:50:13.629 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:44.832827+0000", 2024-09-21T15:50:13.629 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.729058895000001, 2024-09-21T15:50:13.629 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053552929999999997, 2024-09-21T15:50:13.629 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:13.629 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-09-21T15:50:13.629 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:13.629 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.629 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:13.629 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832827+0000", 2024-09-21T15:50:13.629 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.629 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.629 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.629 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:13.629 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832827+0000", 2024-09-21T15:50:13.629 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.629 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.630 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.630 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:13.630 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832826+0000", 2024-09-21T15:50:13.630 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:13.630 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.630 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.630 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:13.630 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832835+0000", 2024-09-21T15:50:13.630 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4509999999999992e-06 2024-09-21T15:50:13.630 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.630 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.630 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:13.630 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832854+0000", 2024-09-21T15:50:13.630 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9352999999999999e-05 2024-09-21T15:50:13.631 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.631 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.631 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:13.631 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832866+0000", 2024-09-21T15:50:13.631 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1766000000000001e-05 2024-09-21T15:50:13.631 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.631 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.631 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:13.631 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.833091+0000", 2024-09-21T15:50:13.631 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00022490300000000001 2024-09-21T15:50:13.631 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.631 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.631 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:13.631 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.838182+0000", 2024-09-21T15:50:13.631 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0050915420000000001 2024-09-21T15:50:13.631 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.631 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:13.632 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.632 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.632 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.632 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:102822 3.4 e54/22 3:2729ba03:::rbd_object_map.113336ae1075:head v 54'208, mlcod=54'205)", 2024-09-21T15:50:13.632 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:46.249540+0000", 2024-09-21T15:50:13.632 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.312345662999999, 2024-09-21T15:50:13.632 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052654859999999998, 2024-09-21T15:50:13.632 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:13.632 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:13.632 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:13.632 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.632 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:13.632 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249540+0000", 2024-09-21T15:50:13.632 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.632 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.633 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.633 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:13.633 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249540+0000", 2024-09-21T15:50:13.633 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.633 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.633 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.633 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:13.633 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249539+0000", 2024-09-21T15:50:13.633 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:13.633 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.633 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.633 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:13.633 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249546+0000", 2024-09-21T15:50:13.633 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7290000000000003e-06 2024-09-21T15:50:13.634 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.634 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.634 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:13.634 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249558+0000", 2024-09-21T15:50:13.634 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1844e-05 2024-09-21T15:50:13.634 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.634 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.634 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:13.634 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249565+0000", 2024-09-21T15:50:13.634 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8210000000000004e-06 2024-09-21T15:50:13.634 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.634 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.634 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:13.634 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249602+0000", 2024-09-21T15:50:13.635 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7057000000000001e-05 2024-09-21T15:50:13.635 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.635 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.635 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:13.635 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249622+0000", 2024-09-21T15:50:13.635 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9678999999999999e-05 2024-09-21T15:50:13.635 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.635 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.635 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:13.635 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254783+0000", 2024-09-21T15:50:13.635 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0051611449999999998 2024-09-21T15:50:13.635 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.635 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.635 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:13.635 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254806+0000", 2024-09-21T15:50:13.636 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2853e-05 2024-09-21T15:50:13.636 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.636 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:13.636 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.636 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.636 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.636 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:102823 3.4 e54/22 3:2729ba03:::rbd_object_map.113336ae1075:head v 54'209, mlcod=54'205)", 2024-09-21T15:50:13.636 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:46.250318+0000", 2024-09-21T15:50:13.636 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.311568078000001, 2024-09-21T15:50:13.636 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045013539999999999, 2024-09-21T15:50:13.636 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:13.636 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:13.636 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:13.636 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.636 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:13.636 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250318+0000", 2024-09-21T15:50:13.637 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.637 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.637 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.637 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:13.637 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250318+0000", 2024-09-21T15:50:13.637 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:13.637 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.637 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.637 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:13.637 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250317+0000", 2024-09-21T15:50:13.637 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:13.637 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.637 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.637 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:13.637 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250322+0000", 2024-09-21T15:50:13.638 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.074e-06 2024-09-21T15:50:13.638 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.638 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.638 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:13.638 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250330+0000", 2024-09-21T15:50:13.638 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4730000000000003e-06 2024-09-21T15:50:13.638 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.638 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.638 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:13.638 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250335+0000", 2024-09-21T15:50:13.638 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8019999999999997e-06 2024-09-21T15:50:13.638 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.638 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.638 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:13.638 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250375+0000", 2024-09-21T15:50:13.638 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9459999999999998e-05 2024-09-21T15:50:13.639 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.639 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.639 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:13.639 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250388+0000", 2024-09-21T15:50:13.639 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3288999999999999e-05 2024-09-21T15:50:13.639 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.639 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.639 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:13.639 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254807+0000", 2024-09-21T15:50:13.639 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0044192220000000004 2024-09-21T15:50:13.639 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:13.639 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:13.639 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:13.639 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254819+0000", 2024-09-21T15:50:13.639 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1603999999999999e-05 2024-09-21T15:50:13.639 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.639 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:13.640 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.640 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:13.640 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:13.640 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:13.640 INFO:teuthology.orchestra.run.smithi089.stdout:5028 write 0xe8ca571 thru 0xe8cd140 (0x2bd0 bytes) 2024-09-21T15:50:13.641 INFO:teuthology.orchestra.run.smithi089.stdout:5030 write 0xc04e113 thru 0xc055a56 (0x7944 bytes) 2024-09-21T15:50:13.641 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:50:13.643 INFO:teuthology.orchestra.run.smithi089.stdout:5031 trunc from 0xe8cd141 to 0xd2d338c 2024-09-21T15:50:13.643 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:13.643 INFO:teuthology.orchestra.run.smithi089.stdout:5034 writesame 0x71db3bb thru 0x71e2d36 data_size 0xa(0x797c bytes) 2024-09-21T15:50:13.643 INFO:teuthology.orchestra.run.smithi089.stdout:5035 write 0x42d3296 thru 0x42d9ed4 (0x6c3f bytes) 2024-09-21T15:50:13.643 INFO:teuthology.orchestra.run.smithi089.stdout:5036 writesame 0x91c1454 thru 0x91cfa75 data_size 0x1b(0xe622 bytes) 2024-09-21T15:50:13.643 INFO:teuthology.orchestra.run.smithi089.stdout:5037 read 0xd288617 thru 0xd28f953 (0x733d bytes) 2024-09-21T15:50:13.643 INFO:teuthology.orchestra.run.smithi089.stdout:5038 write 0x1c83db thru 0x1cc336 (0x3f5c bytes) 2024-09-21T15:50:13.644 INFO:teuthology.orchestra.run.smithi089.stdout:5039 write 0x9742144 thru 0x974f7c2 (0xd67f bytes) 2024-09-21T15:50:13.644 INFO:teuthology.orchestra.run.smithi089.stdout:5042 write 0x899b0f6 thru 0x89aa6a0 (0xf5ab bytes) 2024-09-21T15:50:13.644 INFO:teuthology.orchestra.run.smithi089.stdout:5043 read 0xcef2638 thru 0xcef8990 (0x6359 bytes) 2024-09-21T15:50:13.644 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:13.644 INFO:teuthology.orchestra.run.smithi089.stdout:5045 trunc from 0xd2d338c to 0xa1f500c 2024-09-21T15:50:13.644 INFO:teuthology.orchestra.run.smithi089.stdout:5046 read 0x39ef314 thru 0x39fa2b8 (0xafa5 bytes) 2024-09-21T15:50:13.644 INFO:teuthology.orchestra.run.smithi089.stdout:5049 writesame 0x9c26df thru 0x9cfdf7 data_size 0x5(0xd719 bytes) 2024-09-21T15:50:13.644 INFO:teuthology.orchestra.run.smithi089.stdout:5050 read 0x6185971 thru 0x6188192 (0x2822 bytes) 2024-09-21T15:50:13.644 INFO:teuthology.orchestra.run.smithi089.stdout:5051 trunc from 0xa1f500c to 0xa8f40a1 2024-09-21T15:50:13.644 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:13.639+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:13.645 INFO:teuthology.orchestra.run.smithi089.stdout:5052 read 0x325b763 thru 0x32682ae (0xcb4c bytes) 2024-09-21T15:50:13.645 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:13.645 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:13.645 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:13.650 INFO:teuthology.orchestra.run.smithi089.stdout:5059 writesame 0xc338d8e thru 0xc3434e0 data_size 0xd(0xa753 bytes) 2024-09-21T15:50:13.650 INFO:teuthology.orchestra.run.smithi089.stdout:5060 read 0xb6cff28 thru 0xb6d7c19 (0x7cf2 bytes) 2024-09-21T15:50:13.650 INFO:teuthology.orchestra.run.smithi089.stdout:5063 read 0x5910fe7 thru 0x59138e0 (0x28fa bytes) 2024-09-21T15:50:13.650 INFO:teuthology.orchestra.run.smithi089.stdout:5065 write 0x84ad094 thru 0x84b66e3 (0x9650 bytes) 2024-09-21T15:50:13.660 INFO:teuthology.orchestra.run.smithi089.stdout:5066 trunc from 0xc3434e1 to 0x82885d1 2024-09-21T15:50:13.671 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:13.671 INFO:teuthology.orchestra.run.smithi089.stdout:5069 read 0x6dd1ed7 thru 0x6dda580 (0x86aa bytes) 2024-09-21T15:50:13.672 INFO:teuthology.orchestra.run.smithi089.stdout:5071 writesame 0x20b7afa thru 0x20c679e data_size 0x1(0xeca5 bytes) 2024-09-21T15:50:13.704 INFO:teuthology.orchestra.run.smithi089.stdout:5072 write 0x365de30 thru 0x3661f5e (0x412f bytes) 2024-09-21T15:50:13.716 INFO:teuthology.orchestra.run.smithi089.stdout:5073 write 0xcb9af7c thru 0xcbaa39b (0xf420 bytes) 2024-09-21T15:50:13.722 INFO:teuthology.orchestra.run.smithi089.stdout:5074 write 0x34ae13e thru 0x34af124 (0xfe7 bytes) 2024-09-21T15:50:13.723 INFO:teuthology.orchestra.run.smithi089.stdout:5075 write 0x2420544 thru 0x242399b (0x3458 bytes) 2024-09-21T15:50:13.724 INFO:teuthology.orchestra.run.smithi089.stdout:5076 read 0x7f93587 thru 0x7f93dd9 (0x853 bytes) 2024-09-21T15:50:13.728 INFO:teuthology.orchestra.run.smithi089.stdout:5077 trunc from 0xcbaa39c to 0x99c086c 2024-09-21T15:50:13.741 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:13.740+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:13.742 INFO:teuthology.orchestra.run.smithi089.stdout:5078 read 0xbe590b thru 0xbefb11 (0xa207 bytes) 2024-09-21T15:50:13.748 INFO:teuthology.orchestra.run.smithi089.stdout:5079 trunc from 0x99c086c to 0x42660a2 2024-09-21T15:50:13.761 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:13.761 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:13.761 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:50:13.761 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:13.765 INFO:teuthology.orchestra.run.smithi089.stdout:5081 punch from 0x628b32 to 0x635cf9, (0xd1c7 bytes) 2024-09-21T15:50:13.765 INFO:teuthology.orchestra.run.smithi089.stdout:5082 punch from 0x564b50 to 0x5668ae, (0x1d5e bytes) 2024-09-21T15:50:13.772 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:50:13.776 INFO:teuthology.orchestra.run.smithi089.stdout:5083 write 0x9aec080 thru 0x9af7fe2 (0xbf63 bytes) 2024-09-21T15:50:13.783 INFO:teuthology.orchestra.run.smithi089.stdout:5085 trunc from 0x9af7fe3 to 0xa9ee1d1 2024-09-21T15:50:13.788 INFO:teuthology.orchestra.run.smithi089.stdout:5087 read 0x690755c thru 0x690fa31 (0x84d6 bytes) 2024-09-21T15:50:13.794 INFO:teuthology.orchestra.run.smithi089.stdout:5088 trunc from 0xa9ee1d1 to 0x52e43b6 2024-09-21T15:50:13.806 INFO:teuthology.orchestra.run.smithi089.stdout:5090 read 0x5197124 thru 0x519e4ab (0x7388 bytes) 2024-09-21T15:50:13.807 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:13.807 INFO:teuthology.orchestra.run.smithi089.stdout:5092 read 0x908f45 thru 0x90b39d (0x2459 bytes) 2024-09-21T15:50:13.816 INFO:teuthology.orchestra.run.smithi089.stdout:5093 trunc from 0x52e43b6 to 0xcf32456 2024-09-21T15:50:13.827 INFO:teuthology.orchestra.run.smithi089.stdout:5094 trunc from 0xcf32456 to 0x7771aa4 2024-09-21T15:50:13.833 INFO:teuthology.orchestra.run.smithi089.stdout:5095 read 0x5bdb1f4 thru 0x5bdbfd4 (0xde1 bytes) 2024-09-21T15:50:13.842 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:13.841+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:13.843 INFO:teuthology.orchestra.run.smithi089.stdout:5096 writesame 0xb944074 thru 0xb9447e5 data_size 0x1(0x772 bytes) 2024-09-21T15:50:13.843 INFO:teuthology.orchestra.run.smithi089.stdout:5097 write 0x1a0620 thru 0x1ac71d (0xc0fe bytes) 2024-09-21T15:50:13.845 INFO:teuthology.orchestra.run.smithi089.stdout:5098 punch from 0x6930a34 to 0x693c375, (0xb941 bytes) 2024-09-21T15:50:13.845 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:13.853 INFO:teuthology.orchestra.run.smithi089.stdout:5100 write 0xeb3fbef thru 0xeb41702 (0x1b14 bytes) 2024-09-21T15:50:13.857 INFO:teuthology.orchestra.run.smithi089.stdout:5106 writesame 0x1822442 thru 0x1822903 data_size 0x3(0x4c2 bytes) 2024-09-21T15:50:13.870 INFO:teuthology.orchestra.run.smithi089.stdout:5107 read 0x936b406 thru 0x93761cc (0xadc7 bytes) 2024-09-21T15:50:13.870 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:13.882 INFO:teuthology.orchestra.run.smithi089.stdout:5109 trunc from 0xeb41703 to 0x471d14c 2024-09-21T15:50:13.891 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:13.891 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:13.891 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:50:13.891 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:50:13.891 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:13.896 INFO:teuthology.orchestra.run.smithi089.stdout:5110 read 0xc8e9c1 thru 0xc9615a (0x779a bytes) 2024-09-21T15:50:13.901 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:50:13.905 INFO:teuthology.orchestra.run.smithi089.stdout:5111 write 0x77f195a thru 0x77f9b28 (0x81cf bytes) 2024-09-21T15:50:13.910 INFO:teuthology.orchestra.run.smithi089.stdout:5112 punch from 0x5cef293 to 0x5cfa330, (0xb09d bytes) 2024-09-21T15:50:13.918 INFO:teuthology.orchestra.run.smithi089.stdout:5113 write 0x9af54da thru 0x9b01ede (0xca05 bytes) 2024-09-21T15:50:13.924 INFO:teuthology.orchestra.run.smithi089.stdout:5114 punch from 0x2dc55fa to 0x2dd5359, (0xfd5f bytes) 2024-09-21T15:50:13.924 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:13.932 INFO:teuthology.orchestra.run.smithi089.stdout:5116 trunc from 0x9b01edf to 0x2a8bf8e 2024-09-21T15:50:13.942 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:13.941+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:13.945 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:13.950 INFO:teuthology.orchestra.run.smithi089.stdout:5118 trunc from 0x2a8bf8e to 0x728002b 2024-09-21T15:50:13.955 INFO:teuthology.orchestra.run.smithi089.stdout:5119 read 0x22ad0c3 thru 0x22b0246 (0x3184 bytes) 2024-09-21T15:50:13.956 INFO:teuthology.orchestra.run.smithi089.stdout:5120 write 0x6274278 thru 0x627bb48 (0x78d1 bytes) 2024-09-21T15:50:13.961 INFO:teuthology.orchestra.run.smithi089.stdout:5121 read 0x491079e thru 0x4918f00 (0x8763 bytes) 2024-09-21T15:50:13.961 INFO:teuthology.orchestra.run.smithi089.stdout:5123 write 0xe4172f thru 0xe4764b (0x5f1d bytes) 2024-09-21T15:50:13.975 INFO:teuthology.orchestra.run.smithi089.stdout:5124 write 0x1932a90 thru 0x193b81d (0x8d8e bytes) 2024-09-21T15:50:13.984 INFO:teuthology.orchestra.run.smithi089.stdout:5125 trunc from 0x728002b to 0xd3ef8a7 2024-09-21T15:50:13.989 INFO:teuthology.orchestra.run.smithi089.stdout:5126 write 0xc4905a5 thru 0xc49e9af (0xe40b bytes) 2024-09-21T15:50:13.994 INFO:teuthology.orchestra.run.smithi089.stdout:5128 trunc from 0xd3ef8a7 to 0xb23c173 2024-09-21T15:50:14.004 INFO:teuthology.orchestra.run.smithi089.stdout:5129 trunc from 0xb23c173 to 0xa7eed6d 2024-09-21T15:50:14.010 INFO:teuthology.orchestra.run.smithi089.stdout:5130 punch from 0x62351dc to 0x62444f7, (0xf31b bytes) 2024-09-21T15:50:14.010 INFO:teuthology.orchestra.run.smithi089.stdout:5131 read 0x59dc24b thru 0x59ddd2e (0x1ae4 bytes) 2024-09-21T15:50:14.010 INFO:teuthology.orchestra.run.smithi089.stdout:5132 write 0x2485b68 thru 0x248840b (0x28a4 bytes) 2024-09-21T15:50:14.011 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:14.011 INFO:teuthology.orchestra.run.smithi089.stdout:5134 read 0xf44267 thru 0xf47f8e (0x3d28 bytes) 2024-09-21T15:50:14.012 INFO:teuthology.orchestra.run.smithi089.stdout:5135 read 0x630c0c8 thru 0x630c384 (0x2bd bytes) 2024-09-21T15:50:14.012 INFO:teuthology.orchestra.run.smithi089.stdout:5138 read 0x4921b0f thru 0x49286a5 (0x6b97 bytes) 2024-09-21T15:50:14.012 INFO:teuthology.orchestra.run.smithi089.stdout:5140 write 0x1e7b7c3 thru 0x1e8079b (0x4fd9 bytes) 2024-09-21T15:50:14.013 INFO:teuthology.orchestra.run.smithi089.stdout:5141 write 0x66aa4e3 thru 0x66af962 (0x5480 bytes) 2024-09-21T15:50:14.016 INFO:teuthology.orchestra.run.smithi089.stdout:5142 writesame 0x5f866ca thru 0x5f8ed2a data_size 0x1(0x8661 bytes) 2024-09-21T15:50:14.019 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:14.019 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:50:14.019 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:50:14.019 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:50:14.019 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.019 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:14051 3.8 3:13437810:::rbd_data.1133dc5adfdf.0000000000000039:head [write 716800~796] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-21T15:50:14.019 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:50:14.019 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.322156872999997, 2024-09-21T15:50:14.019 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.98064002699999997, 2024-09-21T15:50:14.019 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:14.020 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:14.020 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:14.020 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:14.020 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:14.020 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 14051 2024-09-21T15:50:14.020 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.020 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:14.020 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.020 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:14.020 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:50:14.020 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:14.020 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.020 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.020 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:14.020 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:50:14.020 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:14.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.021 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.021 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:14.021 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:50:14.021 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:14.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.021 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.021 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:14.021 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695131+0000", 2024-09-21T15:50:14.021 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4680000000000003e-06 2024-09-21T15:50:14.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.021 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.021 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:14.021 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695139+0000", 2024-09-21T15:50:14.021 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9400000000000002e-06 2024-09-21T15:50:14.022 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.022 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.022 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:14.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695143+0000", 2024-09-21T15:50:14.022 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9899999999999999e-06 2024-09-21T15:50:14.022 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.022 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.022 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:14.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695190+0000", 2024-09-21T15:50:14.022 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6458000000000001e-05 2024-09-21T15:50:14.022 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.022 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.022 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:50:14.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695201+0000", 2024-09-21T15:50:14.023 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0964999999999999e-05 2024-09-21T15:50:14.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.023 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.023 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:14.023 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.669490+0000", 2024-09-21T15:50:14.023 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.97428980499999995 2024-09-21T15:50:14.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.023 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.023 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-09-21T15:50:14.023 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.670033+0000", 2024-09-21T15:50:14.023 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00054252599999999999 2024-09-21T15:50:14.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.023 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.023 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:14.024 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.671095+0000", 2024-09-21T15:50:14.024 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001062183 2024-09-21T15:50:14.024 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.024 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.024 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:14.024 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.671186+0000", 2024-09-21T15:50:14.024 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1191000000000002e-05 2024-09-21T15:50:14.024 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.024 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.024 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-09-21T15:50:14.024 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.671346+0000", 2024-09-21T15:50:14.024 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015980399999999999 2024-09-21T15:50:14.024 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.024 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.025 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:14.025 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.672221+0000", 2024-09-21T15:50:14.025 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00087471999999999995 2024-09-21T15:50:14.025 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.025 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.025 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:14.025 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.675619+0000", 2024-09-21T15:50:14.025 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00339858 2024-09-21T15:50:14.025 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.025 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.025 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:14.025 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.675670+0000", 2024-09-21T15:50:14.025 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0664000000000001e-05 2024-09-21T15:50:14.025 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.026 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.026 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:14.026 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.675768+0000", 2024-09-21T15:50:14.026 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7415000000000002e-05 2024-09-21T15:50:14.026 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:14.026 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:14.026 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:14.026 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.026 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.026 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116235 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 761856~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:14.026 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048806+0000", 2024-09-21T15:50:14.026 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.968477989, 2024-09-21T15:50:14.026 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056818556999999999, 2024-09-21T15:50:14.026 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:14.026 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:14.027 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:14.027 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:14.027 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:14.027 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116235 2024-09-21T15:50:14.027 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.027 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:14.027 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.027 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:14.027 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048806+0000", 2024-09-21T15:50:14.027 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:14.027 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.027 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.027 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:14.027 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048806+0000", 2024-09-21T15:50:14.027 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:14.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.028 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.028 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:14.028 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048806+0000", 2024-09-21T15:50:14.028 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:14.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.028 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.028 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:14.028 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048807+0000", 2024-09-21T15:50:14.028 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.001e-06 2024-09-21T15:50:14.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.028 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.028 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:14.028 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048810+0000", 2024-09-21T15:50:14.028 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.711e-06 2024-09-21T15:50:14.029 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.029 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.029 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:14.029 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048812+0000", 2024-09-21T15:50:14.029 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2290000000000002e-06 2024-09-21T15:50:14.029 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.029 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.029 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:14.029 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105495+0000", 2024-09-21T15:50:14.029 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056682679999999999 2024-09-21T15:50:14.029 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.029 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.029 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:14.029 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105565+0000", 2024-09-21T15:50:14.029 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9868000000000003e-05 2024-09-21T15:50:14.030 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.030 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.030 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:14.030 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105625+0000", 2024-09-21T15:50:14.030 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0268e-05 2024-09-21T15:50:14.030 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:14.030 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:14.030 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:14.030 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.030 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.030 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116236 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 768000~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:14.030 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048814+0000", 2024-09-21T15:50:14.030 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.968470325999998, 2024-09-21T15:50:14.030 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056959587999999998, 2024-09-21T15:50:14.030 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:14.031 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:14.031 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:14.031 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:14.031 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:14.031 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116236 2024-09-21T15:50:14.031 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.031 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:14.031 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.031 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:14.031 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048814+0000", 2024-09-21T15:50:14.031 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:14.031 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.031 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.031 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:14.031 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048814+0000", 2024-09-21T15:50:14.032 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:14.032 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.032 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.032 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:14.032 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048814+0000", 2024-09-21T15:50:14.032 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:14.032 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.032 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.032 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:14.032 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048815+0000", 2024-09-21T15:50:14.032 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7999999999999993e-07 2024-09-21T15:50:14.032 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.032 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.032 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:14.032 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048817+0000", 2024-09-21T15:50:14.032 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6620000000000001e-06 2024-09-21T15:50:14.033 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.033 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.033 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:14.033 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048820+0000", 2024-09-21T15:50:14.033 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2800000000000002e-06 2024-09-21T15:50:14.033 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.033 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.033 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:14.033 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105647+0000", 2024-09-21T15:50:14.033 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056827253000000001 2024-09-21T15:50:14.033 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.033 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.033 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:14.033 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105716+0000", 2024-09-21T15:50:14.033 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8761000000000001e-05 2024-09-21T15:50:14.033 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.034 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:14.034 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105774+0000", 2024-09-21T15:50:14.034 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7855000000000001e-05 2024-09-21T15:50:14.034 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:14.034 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:14.034 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:14.034 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.034 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116237 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 774144~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:14.034 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048822+0000", 2024-09-21T15:50:14.034 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.968462640999999, 2024-09-21T15:50:14.034 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057114028999999997, 2024-09-21T15:50:14.035 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:14.035 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:14.035 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:14.035 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:14.035 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:14.035 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116237 2024-09-21T15:50:14.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.035 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:14.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.035 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:14.035 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048822+0000", 2024-09-21T15:50:14.035 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:14.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.036 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:14.036 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048822+0000", 2024-09-21T15:50:14.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:14.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.036 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:14.036 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048822+0000", 2024-09-21T15:50:14.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:14.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.036 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:14.036 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048823+0000", 2024-09-21T15:50:14.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9699999999999994e-07 2024-09-21T15:50:14.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.036 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:14.037 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048825+0000", 2024-09-21T15:50:14.037 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.65e-06 2024-09-21T15:50:14.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.037 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:14.037 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048828+0000", 2024-09-21T15:50:14.037 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8890000000000002e-06 2024-09-21T15:50:14.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.037 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:14.037 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105802+0000", 2024-09-21T15:50:14.037 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056974415 2024-09-21T15:50:14.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.038 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:14.038 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105871+0000", 2024-09-21T15:50:14.038 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8727999999999997e-05 2024-09-21T15:50:14.038 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.038 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.038 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:14.038 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105936+0000", 2024-09-21T15:50:14.038 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4554999999999994e-05 2024-09-21T15:50:14.038 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:14.038 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:14.038 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:14.038 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.038 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.038 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116238 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 780288~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:14.038 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048830+0000", 2024-09-21T15:50:14.039 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.968454382000001, 2024-09-21T15:50:14.039 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057296574000000003, 2024-09-21T15:50:14.039 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:14.039 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:14.039 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:14.039 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:14.039 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:14.039 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116238 2024-09-21T15:50:14.039 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.039 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:14.039 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.039 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:14.039 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048830+0000", 2024-09-21T15:50:14.039 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:14.040 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.040 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:14.040 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048830+0000", 2024-09-21T15:50:14.040 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:14.040 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.040 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:14.040 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048830+0000", 2024-09-21T15:50:14.040 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:14.040 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.040 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:14.040 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048831+0000", 2024-09-21T15:50:14.040 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0160000000000001e-06 2024-09-21T15:50:14.041 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.041 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.041 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:14.041 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048834+0000", 2024-09-21T15:50:14.041 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.813e-06 2024-09-21T15:50:14.041 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.041 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.041 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:14.041 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048836+0000", 2024-09-21T15:50:14.041 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6790000000000001e-06 2024-09-21T15:50:14.041 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.041 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.041 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:14.041 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105961+0000", 2024-09-21T15:50:14.042 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057124705999999997 2024-09-21T15:50:14.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.042 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.042 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:14.042 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106042+0000", 2024-09-21T15:50:14.042 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0820000000000002e-05 2024-09-21T15:50:14.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.042 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.042 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:14.042 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106127+0000", 2024-09-21T15:50:14.042 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4746000000000005e-05 2024-09-21T15:50:14.042 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:14.042 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:14.042 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:14.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.043 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.043 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116324 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 524288~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:14.043 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048838+0000", 2024-09-21T15:50:14.043 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.968446072999999, 2024-09-21T15:50:14.043 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057477185, 2024-09-21T15:50:14.043 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:14.043 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:14.043 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:14.043 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:14.043 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:14.043 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116324 2024-09-21T15:50:14.043 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.043 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:14.043 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.043 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:14.044 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048838+0000", 2024-09-21T15:50:14.044 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:14.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.044 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:14.044 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048838+0000", 2024-09-21T15:50:14.044 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:14.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.044 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:14.044 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048838+0000", 2024-09-21T15:50:14.044 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:14.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.044 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:14.044 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048842+0000", 2024-09-21T15:50:14.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4829999999999999e-06 2024-09-21T15:50:14.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:14.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048846+0000", 2024-09-21T15:50:14.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0470000000000004e-06 2024-09-21T15:50:14.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:14.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048849+0000", 2024-09-21T15:50:14.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3689999999999998e-06 2024-09-21T15:50:14.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:14.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106157+0000", 2024-09-21T15:50:14.046 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057307942000000001 2024-09-21T15:50:14.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.046 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:14.046 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106237+0000", 2024-09-21T15:50:14.046 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0320000000000003e-05 2024-09-21T15:50:14.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.046 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:14.046 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106315+0000", 2024-09-21T15:50:14.046 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8227999999999997e-05 2024-09-21T15:50:14.046 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:14.046 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:14.046 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:14.047 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.047 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.047 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116325 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 530432~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:14.047 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048851+0000", 2024-09-21T15:50:14.047 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.968432901, 2024-09-21T15:50:14.047 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057581036000000002, 2024-09-21T15:50:14.047 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:14.047 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:14.047 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:14.047 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:14.047 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:14.047 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116325 2024-09-21T15:50:14.047 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.047 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:14.047 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.047 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:14.048 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048851+0000", 2024-09-21T15:50:14.048 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:14.048 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.048 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.048 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:14.048 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048851+0000", 2024-09-21T15:50:14.048 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:14.048 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.048 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.048 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:14.048 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048851+0000", 2024-09-21T15:50:14.048 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:14.048 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.049 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.049 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:14.049 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048852+0000", 2024-09-21T15:50:14.049 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.077e-06 2024-09-21T15:50:14.049 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.049 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.049 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:14.049 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048855+0000", 2024-09-21T15:50:14.049 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9859999999999999e-06 2024-09-21T15:50:14.049 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.049 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.049 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:14.049 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048858+0000", 2024-09-21T15:50:14.049 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3810000000000002e-06 2024-09-21T15:50:14.050 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.050 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.050 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:14.050 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106335+0000", 2024-09-21T15:50:14.050 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057477473000000001 2024-09-21T15:50:14.050 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.050 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.050 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:14.050 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106393+0000", 2024-09-21T15:50:14.050 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8050000000000002e-05 2024-09-21T15:50:14.050 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.050 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.050 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:14.050 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106432+0000", 2024-09-21T15:50:14.051 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9274999999999999e-05 2024-09-21T15:50:14.051 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:14.051 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:14.051 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:14.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.051 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.051 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116326 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 536576~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:14.051 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048860+0000", 2024-09-21T15:50:14.051 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.968424850000002, 2024-09-21T15:50:14.051 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057774197999999999, 2024-09-21T15:50:14.051 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:14.051 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:14.051 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:14.051 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:14.051 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:14.052 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116326 2024-09-21T15:50:14.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.052 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:14.052 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.052 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:14.052 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048860+0000", 2024-09-21T15:50:14.052 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:14.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.052 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.052 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:14.052 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048860+0000", 2024-09-21T15:50:14.052 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:14.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.052 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.052 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:14.052 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048859+0000", 2024-09-21T15:50:14.053 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:14.053 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.053 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.053 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:14.053 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048860+0000", 2024-09-21T15:50:14.053 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.02e-06 2024-09-21T15:50:14.053 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.053 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.053 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:14.053 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048863+0000", 2024-09-21T15:50:14.053 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1039999999999998e-06 2024-09-21T15:50:14.053 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.053 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.053 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:14.053 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048867+0000", 2024-09-21T15:50:14.053 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.563e-06 2024-09-21T15:50:14.054 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.054 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.054 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:14.054 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106465+0000", 2024-09-21T15:50:14.054 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057597789000000003 2024-09-21T15:50:14.054 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.054 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.054 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:14.054 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106552+0000", 2024-09-21T15:50:14.054 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7602000000000006e-05 2024-09-21T15:50:14.054 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.054 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.054 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:14.054 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106634+0000", 2024-09-21T15:50:14.055 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1321000000000003e-05 2024-09-21T15:50:14.055 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:14.055 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:14.055 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:14.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.055 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.055 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116327 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 542720~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:14.055 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048870+0000", 2024-09-21T15:50:14.055 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.968413978000001, 2024-09-21T15:50:14.055 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057883148000000002, 2024-09-21T15:50:14.055 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:14.055 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:14.055 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:14.055 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:14.055 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:14.056 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116327 2024-09-21T15:50:14.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.056 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:14.056 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.056 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:14.056 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048870+0000", 2024-09-21T15:50:14.056 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:14.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.056 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.056 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:14.056 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048870+0000", 2024-09-21T15:50:14.056 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:14.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.056 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.057 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:14.057 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048870+0000", 2024-09-21T15:50:14.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:14.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.057 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:14.057 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048872+0000", 2024-09-21T15:50:14.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.824e-06 2024-09-21T15:50:14.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.057 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:14.057 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048879+0000", 2024-09-21T15:50:14.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6749999999999996e-06 2024-09-21T15:50:14.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.057 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:14.058 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048883+0000", 2024-09-21T15:50:14.058 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2200000000000003e-06 2024-09-21T15:50:14.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.058 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:14.058 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106663+0000", 2024-09-21T15:50:14.058 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057780049 2024-09-21T15:50:14.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.058 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:14.058 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106711+0000", 2024-09-21T15:50:14.058 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8433999999999997e-05 2024-09-21T15:50:14.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.059 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:14.059 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106754+0000", 2024-09-21T15:50:14.059 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2236e-05 2024-09-21T15:50:14.059 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:14.059 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:14.059 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:14.059 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.059 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.059 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116328 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 548864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:14.059 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048887+0000", 2024-09-21T15:50:14.059 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.968397457999998, 2024-09-21T15:50:14.059 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057981984, 2024-09-21T15:50:14.059 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:14.059 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:14.060 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:14.060 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:14.060 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:14.060 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116328 2024-09-21T15:50:14.060 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.060 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:14.060 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.060 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:14.060 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048887+0000", 2024-09-21T15:50:14.060 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:14.060 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.060 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.060 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:14.060 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048887+0000", 2024-09-21T15:50:14.060 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:14.060 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.061 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:14.061 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048887+0000", 2024-09-21T15:50:14.061 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:14.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.061 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:14.061 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048891+0000", 2024-09-21T15:50:14.061 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.843e-06 2024-09-21T15:50:14.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.061 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:14.061 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048897+0000", 2024-09-21T15:50:14.061 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5419999999999996e-06 2024-09-21T15:50:14.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.062 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:14.062 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048901+0000", 2024-09-21T15:50:14.062 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4689999999999999e-06 2024-09-21T15:50:14.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.062 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:14.062 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106780+0000", 2024-09-21T15:50:14.062 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057878259000000001 2024-09-21T15:50:14.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.062 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:14.062 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106827+0000", 2024-09-21T15:50:14.063 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7098000000000002e-05 2024-09-21T15:50:14.063 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.063 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.063 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:14.063 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106869+0000", 2024-09-21T15:50:14.063 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2129999999999998e-05 2024-09-21T15:50:14.063 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:14.063 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:14.063 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:14.063 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.063 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.063 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116329 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 555008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:14.063 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048905+0000", 2024-09-21T15:50:14.063 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.968379347999999, 2024-09-21T15:50:14.063 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058115554999999999, 2024-09-21T15:50:14.064 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:14.064 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:14.064 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:14.065 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:14.065 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:14.065 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116329 2024-09-21T15:50:14.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.065 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:14.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.066 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:14.066 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048905+0000", 2024-09-21T15:50:14.066 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:14.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.066 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.066 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:14.066 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048905+0000", 2024-09-21T15:50:14.066 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:14.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.066 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.066 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:14.066 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048905+0000", 2024-09-21T15:50:14.066 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:14.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.067 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:14.067 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048907+0000", 2024-09-21T15:50:14.067 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.086e-06 2024-09-21T15:50:14.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.067 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:14.067 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048912+0000", 2024-09-21T15:50:14.067 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2830000000000001e-06 2024-09-21T15:50:14.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.067 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:14.067 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048916+0000", 2024-09-21T15:50:14.067 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.98e-06 2024-09-21T15:50:14.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.068 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.068 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:14.068 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106907+0000", 2024-09-21T15:50:14.068 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057990719000000003 2024-09-21T15:50:14.068 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.068 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.068 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:14.068 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106969+0000", 2024-09-21T15:50:14.068 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1789000000000001e-05 2024-09-21T15:50:14.068 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.068 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.068 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:14.068 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107021+0000", 2024-09-21T15:50:14.068 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2030999999999997e-05 2024-09-21T15:50:14.068 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:14.068 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:14.069 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:14.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.069 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116330 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 561152~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:14.069 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048922+0000", 2024-09-21T15:50:14.069 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.968362190000001, 2024-09-21T15:50:14.069 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058242343000000002, 2024-09-21T15:50:14.069 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:14.069 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:14.069 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:14.069 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:14.069 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:14.069 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116330 2024-09-21T15:50:14.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.070 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:14.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.070 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:14.070 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048922+0000", 2024-09-21T15:50:14.070 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:14.070 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.070 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:14.070 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048922+0000", 2024-09-21T15:50:14.070 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:14.070 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.070 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:14.070 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048922+0000", 2024-09-21T15:50:14.071 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:14.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.071 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:14.071 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048924+0000", 2024-09-21T15:50:14.071 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1660000000000001e-06 2024-09-21T15:50:14.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.071 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:14.071 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048930+0000", 2024-09-21T15:50:14.071 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4529999999999999e-06 2024-09-21T15:50:14.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.071 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:14.071 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048934+0000", 2024-09-21T15:50:14.072 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0269999999999997e-06 2024-09-21T15:50:14.072 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.072 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.072 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:14.072 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107047+0000", 2024-09-21T15:50:14.072 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058112913000000002 2024-09-21T15:50:14.072 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.072 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.072 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:14.072 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107095+0000", 2024-09-21T15:50:14.072 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7778000000000002e-05 2024-09-21T15:50:14.072 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.072 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.073 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:14.073 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107165+0000", 2024-09-21T15:50:14.073 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9370999999999997e-05 2024-09-21T15:50:14.073 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:14.073 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:14.073 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:14.073 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.073 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.073 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116331 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 567296~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:14.073 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.049440+0000", 2024-09-21T15:50:14.073 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.967844096, 2024-09-21T15:50:14.073 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057878485, 2024-09-21T15:50:14.073 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:14.073 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:14.073 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:14.074 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:14.074 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:14.074 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116331 2024-09-21T15:50:14.074 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.074 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:14.074 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.074 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:14.074 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049440+0000", 2024-09-21T15:50:14.074 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:14.074 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.074 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.074 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:14.074 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049440+0000", 2024-09-21T15:50:14.074 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:14.074 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:14.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048939+0000", 2024-09-21T15:50:14.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9994984 2024-09-21T15:50:14.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:14.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049444+0000", 2024-09-21T15:50:14.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00050557200000000003 2024-09-21T15:50:14.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:14.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049460+0000", 2024-09-21T15:50:14.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6120000000000002e-05 2024-09-21T15:50:14.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.076 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:14.076 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049468+0000", 2024-09-21T15:50:14.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4209999999999998e-06 2024-09-21T15:50:14.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.076 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:14.076 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107189+0000", 2024-09-21T15:50:14.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057721019999999998 2024-09-21T15:50:14.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.076 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:14.076 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107254+0000", 2024-09-21T15:50:14.077 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5265000000000006e-05 2024-09-21T15:50:14.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.077 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.077 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:14.077 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107319+0000", 2024-09-21T15:50:14.077 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4784999999999997e-05 2024-09-21T15:50:14.077 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:14.077 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:14.077 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:14.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.077 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.077 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116332 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 573440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:14.077 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.049474+0000", 2024-09-21T15:50:14.077 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.967810835000002, 2024-09-21T15:50:14.077 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058013809, 2024-09-21T15:50:14.078 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:14.078 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:14.078 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:14.078 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:14.078 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:14.078 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116332 2024-09-21T15:50:14.078 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.078 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:14.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.078 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:14.078 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049474+0000", 2024-09-21T15:50:14.078 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:14.078 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.078 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:14.079 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049474+0000", 2024-09-21T15:50:14.079 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:14.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.079 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.079 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:14.079 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049473+0000", 2024-09-21T15:50:14.079 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:14.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.079 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.079 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:14.079 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049476+0000", 2024-09-21T15:50:14.079 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6249999999999999e-06 2024-09-21T15:50:14.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.079 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.079 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:14.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049482+0000", 2024-09-21T15:50:14.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0959999999999997e-06 2024-09-21T15:50:14.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.080 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:14.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049487+0000", 2024-09-21T15:50:14.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5949999999999998e-06 2024-09-21T15:50:14.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.080 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:14.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107344+0000", 2024-09-21T15:50:14.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057856594999999997 2024-09-21T15:50:14.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.080 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:14.081 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107422+0000", 2024-09-21T15:50:14.081 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7582000000000004e-05 2024-09-21T15:50:14.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.081 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:14.081 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107487+0000", 2024-09-21T15:50:14.081 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5723999999999996e-05 2024-09-21T15:50:14.081 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:14.081 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:14.081 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:14.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.081 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116333 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 579584~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:14.081 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.049993+0000", 2024-09-21T15:50:14.081 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.967291052, 2024-09-21T15:50:14.082 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057663537000000001, 2024-09-21T15:50:14.082 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:14.082 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:14.082 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:14.082 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:14.082 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:14.082 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116333 2024-09-21T15:50:14.082 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.082 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:14.082 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.082 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:14.082 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049993+0000", 2024-09-21T15:50:14.082 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:14.082 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.083 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:14.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049993+0000", 2024-09-21T15:50:14.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:14.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.083 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:14.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049491+0000", 2024-09-21T15:50:14.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9994979 2024-09-21T15:50:14.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.083 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:14.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049998+0000", 2024-09-21T15:50:14.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00050654200000000002 2024-09-21T15:50:14.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.084 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:14.084 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050010+0000", 2024-09-21T15:50:14.084 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1790999999999999e-05 2024-09-21T15:50:14.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.084 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:14.084 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050016+0000", 2024-09-21T15:50:14.084 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2179999999999996e-06 2024-09-21T15:50:14.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.084 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:14.084 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107513+0000", 2024-09-21T15:50:14.085 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057496927000000003 2024-09-21T15:50:14.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.085 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.085 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:14.085 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107590+0000", 2024-09-21T15:50:14.085 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7547999999999997e-05 2024-09-21T15:50:14.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.085 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.085 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:14.085 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107657+0000", 2024-09-21T15:50:14.085 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6630000000000004e-05 2024-09-21T15:50:14.085 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:14.085 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:14.085 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:14.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.085 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.086 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116334 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 585728~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:14.086 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.050091+0000", 2024-09-21T15:50:14.086 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.967193424000001, 2024-09-21T15:50:14.086 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057734355000000001, 2024-09-21T15:50:14.086 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:14.086 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:14.086 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:14.086 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:14.086 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:14.086 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116334 2024-09-21T15:50:14.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.086 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:14.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.086 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:14.087 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050091+0000", 2024-09-21T15:50:14.087 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:14.087 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.087 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.087 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:14.087 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050091+0000", 2024-09-21T15:50:14.087 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:14.087 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.087 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.087 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:14.087 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050021+0000", 2024-09-21T15:50:14.087 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999304 2024-09-21T15:50:14.087 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.088 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:14.088 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050095+0000", 2024-09-21T15:50:14.088 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4158000000000004e-05 2024-09-21T15:50:14.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.088 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:14.088 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050103+0000", 2024-09-21T15:50:14.088 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0770000000000002e-06 2024-09-21T15:50:14.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.088 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:14.088 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050108+0000", 2024-09-21T15:50:14.089 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6330000000000001e-06 2024-09-21T15:50:14.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.089 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.089 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:14.089 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107681+0000", 2024-09-21T15:50:14.089 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057572996000000001 2024-09-21T15:50:14.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.089 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.089 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:14.089 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107759+0000", 2024-09-21T15:50:14.089 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7368999999999997e-05 2024-09-21T15:50:14.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.090 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.090 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:14.090 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107825+0000", 2024-09-21T15:50:14.090 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6783000000000001e-05 2024-09-21T15:50:14.090 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:14.090 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:14.090 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:14.090 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.090 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.090 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116335 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 591872~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:14.090 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.050542+0000", 2024-09-21T15:50:14.090 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.966742838999998, 2024-09-21T15:50:14.091 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057466455999999999, 2024-09-21T15:50:14.091 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:14.091 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:14.091 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:14.091 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:14.091 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:14.091 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116335 2024-09-21T15:50:14.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.091 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:14.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.091 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:14.091 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050542+0000", 2024-09-21T15:50:14.091 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:14.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.092 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:14.092 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050542+0000", 2024-09-21T15:50:14.092 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:14.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.092 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:14.092 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050113+0000", 2024-09-21T15:50:14.092 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9995713 2024-09-21T15:50:14.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.092 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:14.092 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050546+0000", 2024-09-21T15:50:14.092 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00043328699999999997 2024-09-21T15:50:14.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.093 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.093 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:14.093 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050557+0000", 2024-09-21T15:50:14.093 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1011e-05 2024-09-21T15:50:14.093 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.093 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.093 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:14.093 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050564+0000", 2024-09-21T15:50:14.093 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1130000000000003e-06 2024-09-21T15:50:14.093 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.093 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.093 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:14.093 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107852+0000", 2024-09-21T15:50:14.093 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057287350000000001 2024-09-21T15:50:14.093 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.094 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.094 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:14.094 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107928+0000", 2024-09-21T15:50:14.094 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6668000000000005e-05 2024-09-21T15:50:14.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.094 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.094 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:14.094 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108008+0000", 2024-09-21T15:50:14.094 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9783999999999994e-05 2024-09-21T15:50:14.094 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:14.094 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:14.094 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:14.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.094 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.095 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116336 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 598016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:14.095 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.050834+0000", 2024-09-21T15:50:14.095 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.966450572999999, 2024-09-21T15:50:14.095 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057353572999999998, 2024-09-21T15:50:14.095 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:14.095 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:14.095 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:14.095 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:14.095 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:14.095 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116336 2024-09-21T15:50:14.095 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.095 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:14.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.095 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:14.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050834+0000", 2024-09-21T15:50:14.096 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:14.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:14.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050834+0000", 2024-09-21T15:50:14.096 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:14.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:14.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050571+0000", 2024-09-21T15:50:14.096 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9997373 2024-09-21T15:50:14.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:14.097 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050842+0000", 2024-09-21T15:50:14.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00027029099999999998 2024-09-21T15:50:14.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.097 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:14.097 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050853+0000", 2024-09-21T15:50:14.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1532000000000001e-05 2024-09-21T15:50:14.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.097 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:14.097 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050863+0000", 2024-09-21T15:50:14.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4239999999999999e-06 2024-09-21T15:50:14.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.097 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:14.098 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108045+0000", 2024-09-21T15:50:14.098 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057182415 2024-09-21T15:50:14.098 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:14.098 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108123+0000", 2024-09-21T15:50:14.098 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8275000000000005e-05 2024-09-21T15:50:14.098 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:14.098 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108187+0000", 2024-09-21T15:50:14.098 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4152000000000005e-05 2024-09-21T15:50:14.098 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:14.098 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:14.099 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:14.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.099 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116337 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 604160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:14.099 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.051192+0000", 2024-09-21T15:50:14.099 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.966091944999999, 2024-09-21T15:50:14.099 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057175981000000001, 2024-09-21T15:50:14.099 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:14.099 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:14.099 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:14.099 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:14.099 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:14.099 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116337 2024-09-21T15:50:14.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.099 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:14.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.100 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:14.100 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051192+0000", 2024-09-21T15:50:14.100 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:14.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.100 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:14.100 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051192+0000", 2024-09-21T15:50:14.100 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:14.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.100 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:14.100 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050870+0000", 2024-09-21T15:50:14.100 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9996777 2024-09-21T15:50:14.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.101 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:14.101 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051197+0000", 2024-09-21T15:50:14.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00032656299999999998 2024-09-21T15:50:14.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.101 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:14.101 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051209+0000", 2024-09-21T15:50:14.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2636000000000001e-05 2024-09-21T15:50:14.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.101 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:14.101 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051217+0000", 2024-09-21T15:50:14.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5989999999999996e-06 2024-09-21T15:50:14.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.102 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:14.102 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108224+0000", 2024-09-21T15:50:14.102 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057007622000000001 2024-09-21T15:50:14.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.102 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:14.102 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108299+0000", 2024-09-21T15:50:14.102 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5099999999999996e-05 2024-09-21T15:50:14.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.102 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:14.102 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108368+0000", 2024-09-21T15:50:14.102 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8918000000000007e-05 2024-09-21T15:50:14.102 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:14.102 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:14.103 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:14.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.103 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116338 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 610304~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:14.103 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.051583+0000", 2024-09-21T15:50:14.103 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.965701492000001, 2024-09-21T15:50:14.103 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056967716000000002, 2024-09-21T15:50:14.103 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:14.103 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:14.103 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:14.103 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:14.103 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:14.103 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116338 2024-09-21T15:50:14.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.104 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:14.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.104 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:14.104 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051583+0000", 2024-09-21T15:50:14.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:14.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.104 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:14.104 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051583+0000", 2024-09-21T15:50:14.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:14.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.104 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:14.104 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051223+0000", 2024-09-21T15:50:14.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.99964 2024-09-21T15:50:14.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.105 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:14.105 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051587+0000", 2024-09-21T15:50:14.105 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00036352999999999998 2024-09-21T15:50:14.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.105 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:14.105 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051597+0000", 2024-09-21T15:50:14.105 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.893e-06 2024-09-21T15:50:14.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.105 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:14.105 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051603+0000", 2024-09-21T15:50:14.106 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0519999999999998e-06 2024-09-21T15:50:14.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.106 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:14.106 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108394+0000", 2024-09-21T15:50:14.106 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056791531999999999 2024-09-21T15:50:14.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.106 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:14.106 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108480+0000", 2024-09-21T15:50:14.106 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.577e-05 2024-09-21T15:50:14.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:14.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:14.106 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:14.107 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108551+0000", 2024-09-21T15:50:14.107 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0722000000000005e-05 2024-09-21T15:50:14.107 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:14.107 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:14.107 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:14.107 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:14.107 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:14.107 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:14.109 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:50:14.110 INFO:teuthology.orchestra.run.smithi089.stdout:5143 write 0x66d6873 thru 0x66e57dc (0xef6a bytes) 2024-09-21T15:50:14.110 INFO:teuthology.orchestra.run.smithi089.stdout:5144 read 0x2bb86e thru 0x2bfabf (0x4252 bytes) 2024-09-21T15:50:14.110 INFO:teuthology.orchestra.run.smithi089.stdout:5145 read 0x79303b5 thru 0x793c76f (0xc3bb bytes) 2024-09-21T15:50:14.111 INFO:teuthology.orchestra.run.smithi089.stdout:5146 writesame 0x58429f thru 0x5850fa data_size 0x2(0xe5c bytes) 2024-09-21T15:50:14.111 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:14.111 INFO:teuthology.orchestra.run.smithi089.stdout:5148 trunc from 0xa7eed6d to 0x58e72c2 2024-09-21T15:50:14.111 INFO:teuthology.orchestra.run.smithi089.stdout:5149 trunc from 0x58e72c2 to 0x76e250c 2024-09-21T15:50:14.111 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:14.111 INFO:teuthology.orchestra.run.smithi089.stdout:5151 read 0x1c05c40 thru 0x1c094ad (0x386e bytes) 2024-09-21T15:50:14.111 INFO:teuthology.orchestra.run.smithi089.stdout:5153 writesame 0xa31190d thru 0xa31ffc4 data_size 0x8a(0xe6b8 bytes) 2024-09-21T15:50:14.111 INFO:teuthology.orchestra.run.smithi089.stdout:5154 read 0x6544ef9 thru 0x65518d1 (0xc9d9 bytes) 2024-09-21T15:50:14.111 INFO:teuthology.orchestra.run.smithi089.stdout:5155 read 0x7f2adf8 thru 0x7f2d9e2 (0x2beb bytes) 2024-09-21T15:50:14.111 INFO:teuthology.orchestra.run.smithi089.stdout:5156 read 0x9b5ab8d thru 0x9b63625 (0x8a99 bytes) 2024-09-21T15:50:14.111 INFO:teuthology.orchestra.run.smithi089.stdout:5157 trunc from 0xa31ffc5 to 0x17f1d44 2024-09-21T15:50:14.111 INFO:teuthology.orchestra.run.smithi089.stdout:5158 writesame 0x5abe3b2 thru 0x5ac96ec data_size 0x1(0xb33b bytes) 2024-09-21T15:50:14.111 INFO:teuthology.orchestra.run.smithi089.stdout:5159 read 0x2f89556 thru 0x2f8ea1d (0x54c8 bytes) 2024-09-21T15:50:14.112 INFO:teuthology.orchestra.run.smithi089.stdout:5160 write 0x39f8891 thru 0x39f9489 (0xbf9 bytes) 2024-09-21T15:50:14.112 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:14.107+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:14.122 INFO:teuthology.orchestra.run.smithi089.stdout:5161 writesame 0xa1991cb thru 0xa1a8478 data_size 0x1(0xf2ae bytes) 2024-09-21T15:50:14.122 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:14.122 INFO:teuthology.orchestra.run.smithi089.stdout:5163 trunc from 0xa1a8479 to 0x9b07766 2024-09-21T15:50:14.129 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:14.130 INFO:teuthology.orchestra.run.smithi089.stdout:5165 punch from 0x9082d8c to 0x9083e96, (0x110a bytes) 2024-09-21T15:50:14.130 INFO:teuthology.orchestra.run.smithi089.stdout:5166 punch from 0x59b024f to 0x59bc580, (0xc331 bytes) 2024-09-21T15:50:14.134 INFO:teuthology.orchestra.run.smithi089.stdout:5167 write 0xa139056 thru 0xa13d507 (0x44b2 bytes) 2024-09-21T15:50:14.139 INFO:teuthology.orchestra.run.smithi089.stdout:5168 read 0x57cc4fb thru 0x57d3475 (0x6f7b bytes) 2024-09-21T15:50:14.139 INFO:teuthology.orchestra.run.smithi089.stdout:5169 write 0x61e29ca thru 0x61e9701 (0x6d38 bytes) 2024-09-21T15:50:14.141 INFO:teuthology.orchestra.run.smithi089.stdout:5170 read 0x49e2d3 thru 0x4a6ec3 (0x8bf1 bytes) 2024-09-21T15:50:14.152 INFO:teuthology.orchestra.run.smithi089.stdout:5171 write 0xeb07455 thru 0xeb16865 (0xf411 bytes) 2024-09-21T15:50:14.156 INFO:teuthology.orchestra.run.smithi089.stdout:5172 read 0x7f3db02 thru 0x7f47f0a (0xa409 bytes) 2024-09-21T15:50:14.156 INFO:teuthology.orchestra.run.smithi089.stdout:5173 write 0x74093a4 thru 0x7414268 (0xaec5 bytes) 2024-09-21T15:50:14.159 INFO:teuthology.orchestra.run.smithi089.stdout:5174 read 0xd6e8e6c thru 0xd6ec207 (0x339c bytes) 2024-09-21T15:50:14.159 INFO:teuthology.orchestra.run.smithi089.stdout:5175 punch from 0x3dc18d9 to 0x3dc6eeb, (0x5612 bytes) 2024-09-21T15:50:14.159 INFO:teuthology.orchestra.run.smithi089.stdout:5176 write 0x8150f9f thru 0x8160b4b (0xfbad bytes) 2024-09-21T15:50:14.162 INFO:teuthology.orchestra.run.smithi089.stdout:5177 read 0xbd1a9c thru 0xbd3ff5 (0x255a bytes) 2024-09-21T15:50:14.163 INFO:teuthology.orchestra.run.smithi089.stdout:5179 read 0x299ec22 thru 0x29a535c (0x673b bytes) 2024-09-21T15:50:14.163 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:14.163 INFO:teuthology.orchestra.run.smithi089.stdout:5181 punch from 0x2f8f294 to 0x2f93252, (0x3fbe bytes) 2024-09-21T15:50:14.163 INFO:teuthology.orchestra.run.smithi089.stdout:5184 read 0xd5061fe thru 0xd508758 (0x255b bytes) 2024-09-21T15:50:14.165 INFO:teuthology.orchestra.run.smithi089.stdout:5185 trunc from 0xeb16866 to 0xd2bc32b 2024-09-21T15:50:14.173 INFO:teuthology.orchestra.run.smithi089.stdout:5186 punch from 0x25396cd to 0x253d74c, (0x407f bytes) 2024-09-21T15:50:14.174 INFO:teuthology.orchestra.run.smithi089.stdout:5187 write 0x2c97939 thru 0x2c97b21 (0x1e9 bytes) 2024-09-21T15:50:14.176 INFO:teuthology.orchestra.run.smithi089.stdout:5188 write 0xb1b2541 thru 0xb1bb6fd (0x91bd bytes) 2024-09-21T15:50:14.180 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:14.180 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:14.180 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:14.180 INFO:teuthology.orchestra.run.smithi089.stdout:5192 read 0x1f6d327 thru 0x1f6f816 (0x24f0 bytes) 2024-09-21T15:50:14.180 INFO:teuthology.orchestra.run.smithi089.stdout:5193 writesame 0x7ba9b87 thru 0x7bb01cb data_size 0x3(0x6645 bytes) 2024-09-21T15:50:14.186 INFO:teuthology.orchestra.run.smithi089.stdout:5195 read 0x1521fff thru 0x15287a6 (0x67a8 bytes) 2024-09-21T15:50:14.187 INFO:teuthology.orchestra.run.smithi089.stdout:5196 writesame 0xa59dfb0 thru 0xa5a5569 data_size 0x3(0x75ba bytes) 2024-09-21T15:50:14.191 INFO:teuthology.orchestra.run.smithi089.stdout:5197 write 0x2d6fa15 thru 0x2d74ce3 (0x52cf bytes) 2024-09-21T15:50:14.193 INFO:teuthology.orchestra.run.smithi089.stdout:5198 read 0x13e932f thru 0x13ed308 (0x3fda bytes) 2024-09-21T15:50:14.193 INFO:teuthology.orchestra.run.smithi089.stdout:5199 punch from 0x49889c3 to 0x498be92, (0x34cf bytes) 2024-09-21T15:50:14.193 INFO:teuthology.orchestra.run.smithi089.stdout:5202 read 0x88cbf32 thru 0x88cdaa1 (0x1b70 bytes) 2024-09-21T15:50:14.193 INFO:teuthology.orchestra.run.smithi089.stdout:5203 read 0x4e0d317 thru 0x4e176be (0xa3a8 bytes) 2024-09-21T15:50:14.193 INFO:teuthology.orchestra.run.smithi089.stdout:5204 read 0x121a81a thru 0x122908c (0xe873 bytes) 2024-09-21T15:50:14.193 INFO:teuthology.orchestra.run.smithi089.stdout:5205 write 0x5db3d87 thru 0x5db5cb8 (0x1f32 bytes) 2024-09-21T15:50:14.197 INFO:teuthology.orchestra.run.smithi089.stdout:5206 write 0x6fd9c04 thru 0x6fdcc58 (0x3055 bytes) 2024-09-21T15:50:14.199 INFO:teuthology.orchestra.run.smithi089.stdout:5207 write 0x222e41b thru 0x2239276 (0xae5c bytes) 2024-09-21T15:50:14.203 INFO:teuthology.orchestra.run.smithi089.stdout:5209 write 0xa57ad59 thru 0xa582722 (0x79ca bytes) 2024-09-21T15:50:14.204 INFO:teuthology.orchestra.run.smithi089.stdout:5210 write 0x4acce5f thru 0x4ad8e78 (0xc01a bytes) 2024-09-21T15:50:14.207 INFO:teuthology.orchestra.run.smithi089.stdout:5211 read 0x663693d thru 0x6637574 (0xc38 bytes) 2024-09-21T15:50:14.208 INFO:teuthology.orchestra.run.smithi089.stdout:5213 read 0x25025da thru 0x250f6cf (0xd0f6 bytes) 2024-09-21T15:50:14.209 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:14.207+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:14.215 INFO:teuthology.orchestra.run.smithi089.stdout:5214 trunc from 0xd2bc32b to 0x6d72f8a 2024-09-21T15:50:14.219 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:14.219 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:50:14.219 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:14.219 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:135933 3.3 3:c340e5d3:::rbd_object_map.1133487c00b0:head [call lock.assert_locked in=44b,call rbd.object_map_update in=39b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e79)", 2024-09-21T15:50:14.219 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:14.218328+0000", 2024-09-21T15:50:14.219 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.000410565, 2024-09-21T15:50:14.219 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00042821800000000001, 2024-09-21T15:50:14.219 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:14.219 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:14.219 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:14.219 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:14.220 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:14.220 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 135933 2024-09-21T15:50:14.220 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:14.220 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:14.220 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:14.220 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:14.220 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:14.218328+0000", 2024-09-21T15:50:14.220 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:14.220 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:14.220 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:14.220 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:14.220 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:14.218328+0000", 2024-09-21T15:50:14.220 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:14.220 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:14.220 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:14.220 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:14.221 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:14.218327+0000", 2024-09-21T15:50:14.221 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-21T15:50:14.221 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:14.221 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:14.221 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:14.221 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:14.218332+0000", 2024-09-21T15:50:14.221 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3090000000000003e-06 2024-09-21T15:50:14.221 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:14.221 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:14.221 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:14.221 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:14.218343+0000", 2024-09-21T15:50:14.221 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0905e-05 2024-09-21T15:50:14.221 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:14.221 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:14.221 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:14.221 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:14.218348+0000", 2024-09-21T15:50:14.222 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1499999999999998e-06 2024-09-21T15:50:14.222 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:14.222 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:14.222 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:14.222 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:14.218375+0000", 2024-09-21T15:50:14.222 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7279000000000001e-05 2024-09-21T15:50:14.222 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:14.222 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:14.222 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:14.222 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:14.218423+0000", 2024-09-21T15:50:14.222 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7978e-05 2024-09-21T15:50:14.222 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:14.222 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:14.222 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:14.222 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:14.223 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-21T15:50:14.223 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 1 2024-09-21T15:50:14.223 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:14.225 INFO:teuthology.orchestra.run.smithi089.stdout:5216 read 0x1a86319 thru 0x1a8a33c (0x4024 bytes) 2024-09-21T15:50:14.226 INFO:teuthology.orchestra.run.smithi089.stdout:5217 read 0x4377c8 thru 0x43d074 (0x58ad bytes) 2024-09-21T15:50:14.226 INFO:teuthology.orchestra.run.smithi089.stdout:5218 write 0x3469b34 thru 0x346c306 (0x27d3 bytes) 2024-09-21T15:50:14.229 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:50:14.240 INFO:teuthology.orchestra.run.smithi089.stdout:5219 write 0xd0540ae thru 0xd059df6 (0x5d49 bytes) 2024-09-21T15:50:14.251 INFO:teuthology.orchestra.run.smithi089.stdout:5222 trunc from 0xd059df7 to 0x6b31fc3 2024-09-21T15:50:14.255 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:50:14.272 INFO:teuthology.orchestra.run.smithi089.stdout:5225 trunc from 0x6b31fc3 to 0xa3775e8 2024-09-21T15:50:14.277 INFO:teuthology.orchestra.run.smithi089.stdout:5226 punch from 0x25d3516 to 0x25e2a12, (0xf4fc bytes) 2024-09-21T15:50:14.277 INFO:teuthology.orchestra.run.smithi089.stdout:5227 trunc from 0xa3775e8 to 0xa30bddd 2024-09-21T15:50:14.281 INFO:teuthology.orchestra.run.smithi089.stdout:5231 write 0x5c3e069 thru 0x5c41e07 (0x3d9f bytes) 2024-09-21T15:50:14.284 INFO:teuthology.orchestra.run.smithi089.stdout:5232 read 0x5214fb2 thru 0x5221c01 (0xcc50 bytes) 2024-09-21T15:50:14.292 INFO:teuthology.orchestra.run.smithi089.stdout:5233 writesame 0xc81bf37 thru 0xc81e3d0 data_size 0x5(0x249a bytes) 2024-09-21T15:50:14.292 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:14.299 INFO:teuthology.orchestra.run.smithi089.stdout:5236 writesame 0xe5131b4 thru 0xe5202c5 data_size 0x7(0xd112 bytes) 2024-09-21T15:50:14.299 INFO:teuthology.orchestra.run.smithi089.stdout:5238 read 0x4e95e5e thru 0x4ea44c6 (0xe669 bytes) 2024-09-21T15:50:14.299 INFO:teuthology.orchestra.run.smithi089.stdout:5239 write 0xc1e8bab thru 0xc1f7895 (0xeceb bytes) 2024-09-21T15:50:14.302 INFO:teuthology.orchestra.run.smithi089.stdout:5240 write 0x45841b2 thru 0x45915db (0xd42a bytes) 2024-09-21T15:50:14.304 INFO:teuthology.orchestra.run.smithi089.stdout:5241 write 0xe015ab9 thru 0xe016ea5 (0x13ed bytes) 2024-09-21T15:50:14.307 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:14.307 INFO:teuthology.orchestra.run.smithi089.stdout:5243 write 0x4c4111a thru 0x4c4d93a (0xc821 bytes) 2024-09-21T15:50:14.309 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:14.308+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:14.312 INFO:teuthology.orchestra.run.smithi089.stdout:5245 trunc from 0xe5202c6 to 0xe2db0c2 2024-09-21T15:50:14.318 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:14.318 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:14.318 INFO:teuthology.orchestra.run.smithi089.stdout:5249 read 0xaf2e735 thru 0xaf35885 (0x7151 bytes) 2024-09-21T15:50:14.318 INFO:teuthology.orchestra.run.smithi089.stdout:5251 punch from 0x5db616b to 0x5dbab0a, (0x499f bytes) 2024-09-21T15:50:14.318 INFO:teuthology.orchestra.run.smithi089.stdout:5252 read 0x6aaa62d thru 0x6ab473f (0xa113 bytes) 2024-09-21T15:50:14.318 INFO:teuthology.orchestra.run.smithi089.stdout:5253 read 0x80c523a thru 0x80c706d (0x1e34 bytes) 2024-09-21T15:50:14.318 INFO:teuthology.orchestra.run.smithi089.stdout:5254 writesame 0x7828ea2 thru 0x78335ed data_size 0x6(0xa74c bytes) 2024-09-21T15:50:14.321 INFO:teuthology.orchestra.run.smithi089.stdout:5257 read 0x29f7c0e thru 0x29fc3fa (0x47ed bytes) 2024-09-21T15:50:14.322 INFO:teuthology.orchestra.run.smithi089.stdout:5258 punch from 0x1e40ae6 to 0x1e425b7, (0x1ad1 bytes) 2024-09-21T15:50:14.322 INFO:teuthology.orchestra.run.smithi089.stdout:5259 write 0x84fff44 thru 0x8501907 (0x19c4 bytes) 2024-09-21T15:50:14.327 INFO:teuthology.orchestra.run.smithi089.stdout:5260 read 0xc2dcf3a thru 0xc2ea48d (0xd554 bytes) 2024-09-21T15:50:14.334 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:14.334 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:14.334 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:50:14.334 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:50:14.334 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:14.335 INFO:teuthology.orchestra.run.smithi089.stdout:5261 trunc from 0xe2db0c2 to 0x737a23c 2024-09-21T15:50:14.344 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:50:14.351 INFO:teuthology.orchestra.run.smithi089.stdout:5262 write 0x62e881f thru 0x62f1108 (0x88ea bytes) 2024-09-21T15:50:14.355 INFO:teuthology.orchestra.run.smithi089.stdout:5266 read 0x1935ae thru 0x19827e (0x4cd1 bytes) 2024-09-21T15:50:14.357 INFO:teuthology.orchestra.run.smithi089.stdout:5267 trunc from 0x737a23c to 0x5ee6afc 2024-09-21T15:50:14.368 INFO:teuthology.orchestra.run.smithi089.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T15:50:14.377 INFO:teuthology.orchestra.run.smithi089.stdout:5268 punch from 0x4f708a5 to 0x4f70f93, (0x6ee bytes) 2024-09-21T15:50:14.377 INFO:teuthology.orchestra.run.smithi089.stdout:5269 read 0x4b9a74f thru 0x4ba93e8 (0xec9a bytes) 2024-09-21T15:50:14.383 INFO:teuthology.orchestra.run.smithi089.stdout:5270 write 0x789d50d thru 0x789e154 (0xc48 bytes) 2024-09-21T15:50:14.386 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:50:14.386 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:14.386 INFO:teuthology.orchestra.run.smithi089.stdout:5274 punch from 0x5fe19a3 to 0x5fe1b81, (0x1de bytes) 2024-09-21T15:50:14.387 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:50:14.387 INFO:teuthology.orchestra.run.smithi089.stdout:5275 read 0x14ea28 thru 0x159c69 (0xb242 bytes) 2024-09-21T15:50:14.387 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:14.387 INFO:teuthology.orchestra.run.smithi089.stdout:5277 write 0x60473bf thru 0x6051eb6 (0xaaf8 bytes) 2024-09-21T15:50:14.388 INFO:teuthology.orchestra.run.smithi089.stdout:5278 trunc from 0x789e155 to 0x774309b 2024-09-21T15:50:14.395 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:14.395 INFO:teuthology.orchestra.run.smithi089.stdout:5280 write 0x6131198 thru 0x613cb51 (0xb9ba bytes) 2024-09-21T15:50:14.398 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:14.398 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:14.404 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:50:14.404 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:50:14.407 INFO:teuthology.orchestra.run.smithi089.stdout:5284 write 0xaee3ef7 thru 0xaee6e5b (0x2f65 bytes) 2024-09-21T15:50:14.410 INFO:teuthology.orchestra.run.smithi089.stdout:5285 write 0x73b0c8c thru 0x73bf069 (0xe3de bytes) 2024-09-21T15:50:14.410 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:14.409+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:14.412 INFO:teuthology.orchestra.run.smithi089.stdout:5287 write 0x653c614 thru 0x6542496 (0x5e83 bytes) 2024-09-21T15:50:14.414 INFO:teuthology.orchestra.run.smithi089.stdout:5288 trunc from 0xaee6e5c to 0xa5212c8 2024-09-21T15:50:14.420 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:50:14.421 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:50:14.432 INFO:teuthology.orchestra.run.smithi089.stdout:5289 write 0xed6caef thru 0xed6d05d (0x56f bytes) 2024-09-21T15:50:14.434 INFO:teuthology.orchestra.run.smithi089.stdout:5290 write 0xeaefd4d thru 0xeaf4257 (0x450b bytes) 2024-09-21T15:50:14.437 INFO:teuthology.orchestra.run.smithi089.stdout:5293 writesame 0x2591f6c thru 0x259e505 data_size 0x3(0xc59a bytes) 2024-09-21T15:50:14.438 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:50:14.438 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:50:14.444 INFO:teuthology.orchestra.run.smithi089.stdout:5294 write 0xb9addbc thru 0xb9bd26c (0xf4b1 bytes) 2024-09-21T15:50:14.447 INFO:teuthology.orchestra.run.smithi089.stdout:5295 read 0x4cc7faf thru 0x4cd432e (0xc380 bytes) 2024-09-21T15:50:14.447 INFO:teuthology.orchestra.run.smithi089.stdout:5298 punch from 0xcc7444f to 0xcc79152, (0x4d03 bytes) 2024-09-21T15:50:14.448 INFO:teuthology.orchestra.run.smithi089.stdout:5300 writesame 0x244b424 thru 0x2453f2b data_size 0xc(0x8b08 bytes) 2024-09-21T15:50:14.449 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:14.450 INFO:teuthology.orchestra.run.smithi089.stdout:5302 read 0x7163aa3 thru 0x716a8d4 (0x6e32 bytes) 2024-09-21T15:50:14.450 INFO:teuthology.orchestra.run.smithi089.stdout:5304 writesame 0xa3e103 thru 0xa4a3d1 data_size 0x1(0xc2cf bytes) 2024-09-21T15:50:14.453 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:14.453 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:14.455 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:50:14.455 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:50:14.463 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:50:14.471 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:50:14.471 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:50:14.473 INFO:teuthology.orchestra.run.smithi089.stdout:5306 read 0x60f5e70 thru 0x60fa4b6 (0x4647 bytes) 2024-09-21T15:50:14.474 INFO:teuthology.orchestra.run.smithi089.stdout:5308 read 0x1ff335d thru 0x20025b8 (0xf25c bytes) 2024-09-21T15:50:14.474 INFO:teuthology.orchestra.run.smithi089.stdout:5310 read 0x1946747 thru 0x195184a (0xb104 bytes) 2024-09-21T15:50:14.475 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:14.475 INFO:teuthology.orchestra.run.smithi089.stdout:5312 writesame 0xbb189a5 thru 0xbb1d1a9 data_size 0x1(0x4805 bytes) 2024-09-21T15:50:14.487 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:14.487 INFO:teuthology.orchestra.run.smithi089.stdout:5314 writesame 0x6481781 thru 0x648360a data_size 0x3(0x1e8a bytes) 2024-09-21T15:50:14.487 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:50:14.488 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:50:14.489 INFO:teuthology.orchestra.run.smithi089.stdout:5315 read 0xc93d7c4 thru 0xc94407f (0x68bc bytes) 2024-09-21T15:50:14.489 INFO:teuthology.orchestra.run.smithi089.stdout:5316 write 0xbb98e10 thru 0xbba5a1f (0xcc10 bytes) 2024-09-21T15:50:14.491 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T15:50:14.491 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T15:50:14.494 INFO:teuthology.orchestra.run.smithi089.stdout:5317 read 0x5a99188 thru 0x5a9d479 (0x42f2 bytes) 2024-09-21T15:50:14.495 INFO:teuthology.orchestra.run.smithi089.stdout:5319 read 0x5954fce thru 0x595604e (0x1081 bytes) 2024-09-21T15:50:14.495 INFO:teuthology.orchestra.run.smithi089.stdout:5320 writesame 0x7d40c0b thru 0x7d46749 data_size 0x2f(0x5b3f bytes) 2024-09-21T15:50:14.497 INFO:teuthology.orchestra.run.smithi089.stdout:5324 write 0x860f0bd thru 0x861bc68 (0xcbac bytes) 2024-09-21T15:50:14.500 INFO:teuthology.orchestra.run.smithi089.stdout:5325 write 0x3ba21d7 thru 0x3bac7bd (0xa5e7 bytes) 2024-09-21T15:50:14.504 INFO:teuthology.orchestra.run.smithi089.stdout:5326 read 0x3b354df thru 0x3b35f9d (0xabf bytes) 2024-09-21T15:50:14.504 INFO:teuthology.orchestra.run.smithi089.stdout:5327 read 0xc0916cc thru 0xc094649 (0x2f7e bytes) 2024-09-21T15:50:14.504 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:14.504 INFO:teuthology.orchestra.run.smithi089.stdout:5329 writesame 0xcb048e3 thru 0xcb0f8aa data_size 0x64(0xafc8 bytes) 2024-09-21T15:50:14.507 INFO:teuthology.orchestra.run.smithi089.stdout:5331 punch from 0x337f8dd to 0x338d0b9, (0xd7dc bytes) 2024-09-21T15:50:14.507 INFO:teuthology.orchestra.run.smithi089.stdout:5332 read 0x31fc645 thru 0x3204ca5 (0x8661 bytes) 2024-09-21T15:50:14.507 INFO:teuthology.orchestra.run.smithi089.stdout:5334 write 0xe77f0e2 thru 0xe7875bc (0x84db bytes) 2024-09-21T15:50:14.510 INFO:teuthology.orchestra.run.smithi089.stdout:5336 write 0xcd7ab1 thru 0xcdfcd3 (0x8223 bytes) 2024-09-21T15:50:14.512 INFO:teuthology.orchestra.run.smithi089.stdout:5338 punch from 0x1593b7b to 0x159e915, (0xad9a bytes) 2024-09-21T15:50:14.512 INFO:teuthology.orchestra.run.smithi089.stdout:5340 punch from 0xa67293 to 0xa77126, (0xfe93 bytes) 2024-09-21T15:50:14.512 INFO:teuthology.orchestra.run.smithi089.stdout:5341 write 0x6035a17 thru 0x6039a7e (0x4068 bytes) 2024-09-21T15:50:14.513 INFO:teuthology.orchestra.run.smithi089.stdout:5342 write 0xc56f4df thru 0xc570179 (0xc9b bytes) 2024-09-21T15:50:14.516 INFO:teuthology.orchestra.run.smithi089.stdout:5343 write 0xc6a40a5 thru 0xc6b0cd8 (0xcc34 bytes) 2024-09-21T15:50:14.519 INFO:teuthology.orchestra.run.smithi089.stdout:5345 writesame 0xee44d6 thru 0xeefd1a data_size 0x17(0xb845 bytes) 2024-09-21T15:50:14.521 INFO:teuthology.orchestra.run.smithi089.stdout:5346 writesame 0xd1b9f49 thru 0xd1be854 data_size 0x55(0x490c bytes) 2024-09-21T15:50:14.523 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:14.523 INFO:teuthology.orchestra.run.smithi089.stdout:5348 write 0xd671e5f thru 0xd67b13d (0x92df bytes) 2024-09-21T15:50:14.524 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:14.524 INFO:teuthology.orchestra.run.smithi089.stdout:5352 read 0xc2f9243 thru 0xc2fdbe6 (0x49a4 bytes) 2024-09-21T15:50:14.524 INFO:teuthology.orchestra.run.smithi089.stdout:5353 read 0x954d750 thru 0x95548ef (0x71a0 bytes) 2024-09-21T15:50:14.525 INFO:teuthology.orchestra.run.smithi089.stdout:5354 write 0x558a93a thru 0x5599d34 (0xf3fb bytes) 2024-09-21T15:50:14.526 INFO:teuthology.orchestra.run.smithi089.stdout:5355 writesame 0xa9e4aad thru 0xa9f4212 data_size 0x1(0xf766 bytes) 2024-09-21T15:50:14.549 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:14.548+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:14.556 INFO:teuthology.orchestra.run.smithi089.stdout:5358 read 0x33ab9a6 thru 0x33b339f (0x79fa bytes) 2024-09-21T15:50:14.556 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:14.556 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:14.556 INFO:teuthology.orchestra.run.smithi089.stdout:5362 punch from 0xe3c3bcc to 0xe3ccbea, (0x901e bytes) 2024-09-21T15:50:14.556 INFO:teuthology.orchestra.run.smithi089.stdout:5364 writesame 0x65b985c thru 0x65bdf6a data_size 0x1(0x470f bytes) 2024-09-21T15:50:14.556 INFO:teuthology.orchestra.run.smithi089.stdout:5366 read 0xc8efc6e thru 0xc8fd2af (0xd642 bytes) 2024-09-21T15:50:14.559 INFO:teuthology.orchestra.run.smithi089.stdout:5367 trunc from 0xed6d05e to 0xc545343 2024-09-21T15:50:14.570 INFO:teuthology.orchestra.run.smithi089.stdout:5369 write 0xab7030a thru 0xab7b2f4 (0xafeb bytes) 2024-09-21T15:50:14.572 INFO:teuthology.orchestra.run.smithi089.stdout:5370 read 0x5af097a thru 0x5af8f47 (0x85ce bytes) 2024-09-21T15:50:14.585 INFO:teuthology.orchestra.run.smithi089.stdout:5371 trunc from 0xc545343 to 0x1cf204a 2024-09-21T15:50:14.601 INFO:teuthology.orchestra.run.smithi089.stdout:5372 read 0xed6303 thru 0xed80f3 (0x1df1 bytes) 2024-09-21T15:50:14.602 INFO:teuthology.orchestra.run.smithi089.stdout:5373 read 0xc6bc3 thru 0xcacb2 (0x40f0 bytes) 2024-09-21T15:50:14.606 INFO:teuthology.orchestra.run.smithi089.stdout:5374 trunc from 0x1cf204a to 0x47a995f 2024-09-21T15:50:14.610 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:14.609+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:14.614 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:14.614 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:14.622 INFO:teuthology.orchestra.run.smithi089.stdout:5375 write 0x7a56519 thru 0x7a62ce1 (0xc7c9 bytes) 2024-09-21T15:50:14.623 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:50:14.626 INFO:teuthology.orchestra.run.smithi089.stdout:5376 punch from 0x3c1876a to 0x3c1d29a, (0x4b30 bytes) 2024-09-21T15:50:14.627 INFO:teuthology.orchestra.run.smithi089.stdout:5377 writesame 0x1c9732f thru 0x1ca3490 data_size 0x6f(0xc162 bytes) 2024-09-21T15:50:14.632 INFO:teuthology.orchestra.run.smithi089.stdout:5378 punch from 0x37ee2e0 to 0x37f31e2, (0x4f02 bytes) 2024-09-21T15:50:14.632 INFO:teuthology.orchestra.run.smithi089.stdout:5379 punch from 0x23fa951 to 0x2402c0a, (0x82b9 bytes) 2024-09-21T15:50:14.632 INFO:teuthology.orchestra.run.smithi089.stdout:5380 punch from 0x4c12ef7 to 0x4c22e18, (0xff21 bytes) 2024-09-21T15:50:14.632 INFO:teuthology.orchestra.run.smithi089.stdout:5382 read 0x73decc1 thru 0x73ec779 (0xdab9 bytes) 2024-09-21T15:50:14.633 INFO:teuthology.orchestra.run.smithi089.stdout:5384 read 0x4ead121 thru 0x4eb8170 (0xb050 bytes) 2024-09-21T15:50:14.639 INFO:teuthology.orchestra.run.smithi089.stdout:5385 write 0x7b90ad9 thru 0x7b93570 (0x2a98 bytes) 2024-09-21T15:50:14.646 INFO:teuthology.orchestra.run.smithi089.stdout:5386 read 0x344736b thru 0x344b9c7 (0x465d bytes) 2024-09-21T15:50:14.646 INFO:teuthology.orchestra.run.smithi089.stdout:5387 read 0x3b2bed3 thru 0x3b32965 (0x6a93 bytes) 2024-09-21T15:50:14.646 INFO:teuthology.orchestra.run.smithi089.stdout:5388 punch from 0x87ca6b to 0x882e9c, (0x6431 bytes) 2024-09-21T15:50:14.646 INFO:teuthology.orchestra.run.smithi089.stdout:5389 write 0x2db1c9 thru 0x2e6d87 (0xbbbf bytes) 2024-09-21T15:50:14.648 INFO:teuthology.orchestra.run.smithi089.stdout:5390 read 0x5bc39e7 thru 0x5bd24f5 (0xeb0f bytes) 2024-09-21T15:50:14.648 INFO:teuthology.orchestra.run.smithi089.stdout:5391 punch from 0xe81614 to 0xe8f977, (0xe363 bytes) 2024-09-21T15:50:14.649 INFO:teuthology.orchestra.run.smithi089.stdout:5393 trunc from 0x7b93571 to 0x893e2c7 2024-09-21T15:50:14.655 INFO:teuthology.orchestra.run.smithi089.stdout:5394 trunc from 0x893e2c7 to 0x71f1c83 2024-09-21T15:50:14.662 INFO:teuthology.orchestra.run.smithi089.stdout:5395 writesame 0x9f82e5 thru 0x9fd4ea data_size 0x1(0x5206 bytes) 2024-09-21T15:50:14.687 INFO:teuthology.orchestra.run.smithi089.stdout:5396 write 0xec29288 thru 0xec2fc8e (0x6a07 bytes) 2024-09-21T15:50:14.708 INFO:teuthology.orchestra.run.smithi089.stdout:5398 trunc from 0xec2fc8f to 0x17b8ffa 2024-09-21T15:50:14.712 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:14.710+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:14.735 INFO:teuthology.orchestra.run.smithi089.stdout:5399 write 0x7c19e56 thru 0x7c1ad59 (0xf04 bytes) 2024-09-21T15:50:14.746 INFO:teuthology.orchestra.run.smithi089.stdout:5400 writesame 0x9ea9d61 thru 0x9eb2ac4 data_size 0x2(0x8d64 bytes) 2024-09-21T15:50:14.746 INFO:teuthology.orchestra.run.smithi089.stdout:5401 clone 36 order 18 su 8192 sc 11 2024-09-21T15:50:14.768 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:14.768 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:14.776 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:50:14.812 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:14.812+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:14.878 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:14.878 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:14.887 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T15:50:14.931 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:14.930+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:14.989 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:14.989 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:14.998 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T15:50:15.014 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:15.013+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:15.089 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T15:50:15.116 INFO:teuthology.orchestra.run.smithi089.stdout:truncating image image_client.0-clone35 from 0x9eb2ac5 (overlap 0x17b8ffa) to 0x12fa661 2024-09-21T15:50:15.117 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:15.116+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:15.150 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:15.150 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:15.159 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T15:50:15.180 INFO:teuthology.orchestra.run.smithi089.stdout:checking clone #34, image image_client.0-clone34 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent35 2024-09-21T15:50:15.216 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:15.214+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:15.262 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:15.262 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:15.271 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T15:50:15.317 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:15.315+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:15.373 INFO:teuthology.orchestra.run.smithi089.stdout:5402 read 0x8dab426 thru 0x8dac481 (0x105c bytes) 2024-09-21T15:50:15.374 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:15.374 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:15.383 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T15:50:15.389 INFO:teuthology.orchestra.run.smithi089.stdout:5403 write 0xd433448 thru 0xd437c8c (0x4845 bytes) 2024-09-21T15:50:15.394 INFO:teuthology.orchestra.run.smithi089.stdout:5404 write 0x496210 thru 0x499ad8 (0x38c9 bytes) 2024-09-21T15:50:15.409 INFO:teuthology.orchestra.run.smithi089.stdout:5405 write 0x688eba3 thru 0x689a182 (0xb5e0 bytes) 2024-09-21T15:50:15.421 INFO:teuthology.orchestra.run.smithi089.stdout:5406 trunc from 0xd437c8d to 0xb5cafec 2024-09-21T15:50:15.425 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:15.424+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:15.437 INFO:teuthology.orchestra.run.smithi089.stdout:5407 read 0x851998a thru 0x8526817 (0xce8e bytes) 2024-09-21T15:50:15.437 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:15.437 INFO:teuthology.orchestra.run.smithi089.stdout:5410 punch from 0x3da03cc to 0x3da9a97, (0x96cb bytes) 2024-09-21T15:50:15.437 INFO:teuthology.orchestra.run.smithi089.stdout:5411 clone 37 order 21 su 524288 sc 15 2024-09-21T15:50:15.518 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:15.517+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:15.527 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:15.527 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:15.535 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T15:50:15.619 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:15.618+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:15.678 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:15.678 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:15.687 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:50:15.730 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:15.729+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:15.791 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:15.791 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:15.800 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:50:15.820 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:15.819+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:15.921 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:15.920+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:15.942 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:15.943 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:15.951 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:50:16.022 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:16.021+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:16.053 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:16.054 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:16.057 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is unset 2024-09-21T15:50:16.065 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:50:16.123 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:16.122+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:16.140 INFO:teuthology.orchestra.run.smithi089.stdout:truncating image image_client.0-clone36 from 0xb5cafec (overlap 0x9eb2ac5) to 0x1fbd55a 2024-09-21T15:50:16.224 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:16.223+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:16.232 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:16.232 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:16.242 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:50:16.325 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:16.324+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:16.395 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:16.395 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:16.405 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:50:16.449 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:16.448+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:16.492 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:50:16.527 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:16.528 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:16.528 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:16.527+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:16.539 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:50:16.614 INFO:teuthology.orchestra.run.smithi089.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T15:50:16.627 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:16.626+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:16.633 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:50:16.633 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:50:16.655 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:50:16.655 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:50:16.673 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:50:16.673 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:50:16.696 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:50:16.696 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:50:16.709 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:16.709 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:16.710 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:50:16.710 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:16.723 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:50:16.725 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:50:16.726 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:50:16.728 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:16.726+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:16.742 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:50:16.742 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:50:16.760 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:50:16.760 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:50:16.763 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T15:50:16.763 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T15:50:16.779 INFO:teuthology.orchestra.run.smithi089.stdout:checking clone #35, image image_client.0-clone35 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent36 2024-09-21T15:50:16.811 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:16.823 INFO:teuthology.orchestra.run.smithi089.stdout:5414 write 0xe49ea9f thru 0xe4a9db2 (0xb314 bytes) 2024-09-21T15:50:16.828 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:16.827+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:16.829 INFO:teuthology.orchestra.run.smithi089.stdout:5415 write 0x9705dc3 thru 0x9710444 (0xa682 bytes) 2024-09-21T15:50:16.833 INFO:teuthology.orchestra.run.smithi089.stdout:5416 writesame 0x5d8722e thru 0x5d8f2a2 data_size 0x1(0x8075 bytes) 2024-09-21T15:50:16.838 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:16.838 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:16.838 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:50:16.838 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:50:16.838 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:16.848 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:50:16.869 INFO:teuthology.orchestra.run.smithi089.stdout:5417 write 0xb56aa6b thru 0xb579dec (0xf382 bytes) 2024-09-21T15:50:16.872 INFO:teuthology.orchestra.run.smithi089.stdout:5419 writesame 0x4776fd2 thru 0x47816b1 data_size 0x8(0xa6e0 bytes) 2024-09-21T15:50:16.882 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:16.882 INFO:teuthology.orchestra.run.smithi089.stdout:5421 read 0xb683fab thru 0xb68c7bc (0x8812 bytes) 2024-09-21T15:50:16.882 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:16.882 INFO:teuthology.orchestra.run.smithi089.stdout:5423 writesame 0x221283b thru 0x221bd30 data_size 0x17(0x94f6 bytes) 2024-09-21T15:50:16.886 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:16.886 INFO:teuthology.orchestra.run.smithi089.stdout:5425 read 0xcff6b96 thru 0xcff81a1 (0x160c bytes) 2024-09-21T15:50:16.886 INFO:teuthology.orchestra.run.smithi089.stdout:5427 write 0xce66068 thru 0xce6cf34 (0x6ecd bytes) 2024-09-21T15:50:16.892 INFO:teuthology.orchestra.run.smithi089.stdout:5428 write 0xc473f9f thru 0xc4757fb (0x185d bytes) 2024-09-21T15:50:16.893 INFO:teuthology.orchestra.run.smithi089.stdout:5429 punch from 0xa77ef82 to 0xa78db72, (0xebf0 bytes) 2024-09-21T15:50:16.893 INFO:teuthology.orchestra.run.smithi089.stdout:5430 write 0xb05818f thru 0xb05cc61 (0x4ad3 bytes) 2024-09-21T15:50:16.906 INFO:teuthology.orchestra.run.smithi089.stdout:5431 trunc from 0xe4a9db3 to 0x78ae9b9 2024-09-21T15:50:16.929 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:16.928+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:16.943 INFO:teuthology.orchestra.run.smithi089.stdout:5432 read 0x3c99bcd thru 0x3c9bd6b (0x219f bytes) 2024-09-21T15:50:16.943 INFO:teuthology.orchestra.run.smithi089.stdout:5433 read 0x55acac2 thru 0x55b72c7 (0xa806 bytes) 2024-09-21T15:50:16.944 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:16.944 INFO:teuthology.orchestra.run.smithi089.stdout:5435 read 0x68e5f7c thru 0x68f48b3 (0xe938 bytes) 2024-09-21T15:50:16.952 INFO:teuthology.orchestra.run.smithi089.stdout:5436 trunc from 0x78ae9b9 to 0xdeeca0f 2024-09-21T15:50:16.959 INFO:teuthology.orchestra.run.smithi089.stdout:5438 trunc from 0xdeeca0f to 0xc20147c 2024-09-21T15:50:16.967 INFO:teuthology.orchestra.run.smithi089.stdout:5439 read 0x6873403 thru 0x6877534 (0x4132 bytes) 2024-09-21T15:50:16.968 INFO:teuthology.orchestra.run.smithi089.stdout:5440 write 0xa178322 thru 0xa179a64 (0x1743 bytes) 2024-09-21T15:50:16.971 INFO:teuthology.orchestra.run.smithi089.stdout:5441 writesame 0x8a3ce08 thru 0x8a457ba data_size 0x1(0x89b3 bytes) 2024-09-21T15:50:16.983 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:16.983 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:50:16.983 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:50:16.983 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:50:16.984 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:16.984 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:14051 3.8 3:13437810:::rbd_data.1133dc5adfdf.0000000000000039:head [write 716800~796] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-21T15:50:16.984 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:50:16.984 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 59.285707791999997, 2024-09-21T15:50:16.984 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.98064002699999997, 2024-09-21T15:50:16.984 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:16.984 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:16.984 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:16.984 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:16.984 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:16.984 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 14051 2024-09-21T15:50:16.984 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:16.985 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:16.985 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:16.985 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:16.985 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:50:16.985 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:16.985 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:16.985 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:16.985 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:16.985 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:50:16.985 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:16.985 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:16.985 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:16.985 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:16.985 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:50:16.986 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:16.986 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:16.986 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:16.986 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:16.986 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695131+0000", 2024-09-21T15:50:16.986 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4680000000000003e-06 2024-09-21T15:50:16.986 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:16.986 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:16.986 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:16.986 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695139+0000", 2024-09-21T15:50:16.986 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9400000000000002e-06 2024-09-21T15:50:16.986 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:16.986 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:16.987 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:16.987 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695143+0000", 2024-09-21T15:50:16.987 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9899999999999999e-06 2024-09-21T15:50:16.987 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:16.987 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:16.987 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:16.987 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695190+0000", 2024-09-21T15:50:16.987 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6458000000000001e-05 2024-09-21T15:50:16.987 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:16.987 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:16.987 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:50:16.987 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695201+0000", 2024-09-21T15:50:16.987 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0964999999999999e-05 2024-09-21T15:50:16.987 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:16.987 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:16.988 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:16.988 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.669490+0000", 2024-09-21T15:50:16.988 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.97428980499999995 2024-09-21T15:50:16.988 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:16.988 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:16.988 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-09-21T15:50:16.988 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.670033+0000", 2024-09-21T15:50:16.988 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00054252599999999999 2024-09-21T15:50:16.988 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:16.988 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:16.988 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:16.988 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.671095+0000", 2024-09-21T15:50:16.988 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001062183 2024-09-21T15:50:16.988 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:16.988 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:16.989 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:16.989 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.671186+0000", 2024-09-21T15:50:16.989 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1191000000000002e-05 2024-09-21T15:50:16.989 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:16.989 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:16.989 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-09-21T15:50:16.989 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.671346+0000", 2024-09-21T15:50:16.989 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015980399999999999 2024-09-21T15:50:16.989 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:16.989 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:16.989 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:16.989 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.672221+0000", 2024-09-21T15:50:16.989 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00087471999999999995 2024-09-21T15:50:16.989 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:16.989 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:16.990 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:16.990 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.675619+0000", 2024-09-21T15:50:16.990 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00339858 2024-09-21T15:50:16.990 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:16.990 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:16.990 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:16.990 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.675670+0000", 2024-09-21T15:50:16.990 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0664000000000001e-05 2024-09-21T15:50:16.990 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:16.990 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:16.990 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:16.990 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.675768+0000", 2024-09-21T15:50:16.990 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7415000000000002e-05 2024-09-21T15:50:16.990 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:16.991 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:16.991 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:16.991 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:16.991 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:16.991 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116235 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 761856~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:16.991 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048806+0000", 2024-09-21T15:50:16.991 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 20.932028907999999, 2024-09-21T15:50:16.991 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056818556999999999, 2024-09-21T15:50:16.991 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:16.991 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:16.991 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:16.991 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:16.991 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:16.991 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116235 2024-09-21T15:50:16.992 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:16.992 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:16.992 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:16.992 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:16.992 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048806+0000", 2024-09-21T15:50:16.992 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:16.992 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:16.992 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:16.992 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:16.992 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048806+0000", 2024-09-21T15:50:16.992 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:16.992 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:16.992 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:16.992 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:16.992 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048806+0000", 2024-09-21T15:50:16.993 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:16.993 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:16.993 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:16.993 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:16.993 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048807+0000", 2024-09-21T15:50:16.993 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.001e-06 2024-09-21T15:50:16.993 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:16.993 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:16.993 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:16.993 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048810+0000", 2024-09-21T15:50:16.993 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.711e-06 2024-09-21T15:50:16.993 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:16.993 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:16.993 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:16.993 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048812+0000", 2024-09-21T15:50:16.994 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2290000000000002e-06 2024-09-21T15:50:16.994 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:16.994 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:16.994 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:16.994 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105495+0000", 2024-09-21T15:50:16.994 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056682679999999999 2024-09-21T15:50:16.994 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:16.994 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:16.994 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:16.994 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105565+0000", 2024-09-21T15:50:16.995 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9868000000000003e-05 2024-09-21T15:50:16.995 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:16.995 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:16.995 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:16.995 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105625+0000", 2024-09-21T15:50:16.995 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0268e-05 2024-09-21T15:50:16.995 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:16.995 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:16.995 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:16.995 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:16.995 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:16.995 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116236 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 768000~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:16.995 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048814+0000", 2024-09-21T15:50:16.996 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 20.932021245000001, 2024-09-21T15:50:16.996 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056959587999999998, 2024-09-21T15:50:16.996 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:16.996 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:16.996 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:16.996 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:16.996 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:16.996 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116236 2024-09-21T15:50:16.996 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:16.996 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:16.996 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:16.996 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:16.996 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048814+0000", 2024-09-21T15:50:16.996 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:16.996 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:16.997 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:16.997 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:16.997 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048814+0000", 2024-09-21T15:50:16.997 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:16.997 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:16.997 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:16.997 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:16.997 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048814+0000", 2024-09-21T15:50:16.997 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:16.997 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:16.997 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:16.997 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:16.997 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048815+0000", 2024-09-21T15:50:16.997 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7999999999999993e-07 2024-09-21T15:50:16.998 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:16.998 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:16.998 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:16.998 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048817+0000", 2024-09-21T15:50:16.998 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6620000000000001e-06 2024-09-21T15:50:16.998 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:16.998 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:16.998 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:16.998 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048820+0000", 2024-09-21T15:50:16.998 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2800000000000002e-06 2024-09-21T15:50:16.998 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:16.998 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:16.998 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:16.998 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105647+0000", 2024-09-21T15:50:16.999 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056827253000000001 2024-09-21T15:50:16.999 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:16.999 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:16.999 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:16.999 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105716+0000", 2024-09-21T15:50:16.999 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8761000000000001e-05 2024-09-21T15:50:16.999 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:16.999 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:16.999 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:16.999 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105774+0000", 2024-09-21T15:50:16.999 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7855000000000001e-05 2024-09-21T15:50:16.999 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:16.999 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:16.999 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:16.999 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.000 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.000 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116237 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 774144~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:17.000 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048822+0000", 2024-09-21T15:50:17.000 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 20.932013560000001, 2024-09-21T15:50:17.000 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057114028999999997, 2024-09-21T15:50:17.000 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:17.000 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:17.000 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:17.000 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:17.000 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:17.000 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116237 2024-09-21T15:50:17.000 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.000 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:17.001 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.001 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:17.001 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048822+0000", 2024-09-21T15:50:17.001 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:17.001 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.001 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.001 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:17.001 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048822+0000", 2024-09-21T15:50:17.001 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:17.001 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.001 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.001 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:17.002 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048822+0000", 2024-09-21T15:50:17.002 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:17.002 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.002 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.002 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:17.002 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048823+0000", 2024-09-21T15:50:17.002 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9699999999999994e-07 2024-09-21T15:50:17.002 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.002 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.002 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:17.002 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048825+0000", 2024-09-21T15:50:17.002 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.65e-06 2024-09-21T15:50:17.003 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.003 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.003 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.003 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048828+0000", 2024-09-21T15:50:17.003 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8890000000000002e-06 2024-09-21T15:50:17.003 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.003 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.003 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:17.003 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105802+0000", 2024-09-21T15:50:17.003 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056974415 2024-09-21T15:50:17.003 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.003 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.003 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:17.003 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105871+0000", 2024-09-21T15:50:17.004 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8727999999999997e-05 2024-09-21T15:50:17.004 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.004 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.004 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:17.004 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105936+0000", 2024-09-21T15:50:17.004 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4554999999999994e-05 2024-09-21T15:50:17.004 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:17.004 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:17.004 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:17.004 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.004 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.004 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116238 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 780288~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:17.004 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048830+0000", 2024-09-21T15:50:17.004 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 20.932005301, 2024-09-21T15:50:17.004 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057296574000000003, 2024-09-21T15:50:17.005 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:17.005 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:17.005 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:17.005 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:17.005 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:17.005 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116238 2024-09-21T15:50:17.005 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.005 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:17.005 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.005 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:17.005 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048830+0000", 2024-09-21T15:50:17.005 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:17.005 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.005 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.005 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:17.006 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048830+0000", 2024-09-21T15:50:17.006 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:17.006 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.006 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.006 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:17.006 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048830+0000", 2024-09-21T15:50:17.006 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:17.006 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.006 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.006 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:17.006 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048831+0000", 2024-09-21T15:50:17.006 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0160000000000001e-06 2024-09-21T15:50:17.006 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.006 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.007 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:17.007 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048834+0000", 2024-09-21T15:50:17.007 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.813e-06 2024-09-21T15:50:17.007 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.007 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.007 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.007 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048836+0000", 2024-09-21T15:50:17.007 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6790000000000001e-06 2024-09-21T15:50:17.007 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.007 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.007 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:17.007 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105961+0000", 2024-09-21T15:50:17.007 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057124705999999997 2024-09-21T15:50:17.007 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.007 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.008 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:17.008 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106042+0000", 2024-09-21T15:50:17.008 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0820000000000002e-05 2024-09-21T15:50:17.008 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.008 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.008 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:17.008 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106127+0000", 2024-09-21T15:50:17.008 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4746000000000005e-05 2024-09-21T15:50:17.008 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:17.008 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:17.008 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:17.008 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.008 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.008 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116324 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 524288~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:17.008 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048838+0000", 2024-09-21T15:50:17.009 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 20.931996991999998, 2024-09-21T15:50:17.009 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057477185, 2024-09-21T15:50:17.009 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:17.009 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:17.009 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:17.009 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:17.009 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:17.009 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116324 2024-09-21T15:50:17.009 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.009 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:17.009 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.009 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:17.009 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048838+0000", 2024-09-21T15:50:17.009 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:17.009 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.010 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.010 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:17.010 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048838+0000", 2024-09-21T15:50:17.010 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:17.010 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.010 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.010 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:17.010 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048838+0000", 2024-09-21T15:50:17.010 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:17.010 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.010 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.010 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:17.010 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048842+0000", 2024-09-21T15:50:17.010 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4829999999999999e-06 2024-09-21T15:50:17.011 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.011 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.011 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:17.011 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048846+0000", 2024-09-21T15:50:17.011 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0470000000000004e-06 2024-09-21T15:50:17.011 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.011 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.011 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.011 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048849+0000", 2024-09-21T15:50:17.011 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3689999999999998e-06 2024-09-21T15:50:17.011 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.011 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.011 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:17.011 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106157+0000", 2024-09-21T15:50:17.012 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057307942000000001 2024-09-21T15:50:17.012 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.012 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.012 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:17.012 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106237+0000", 2024-09-21T15:50:17.012 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0320000000000003e-05 2024-09-21T15:50:17.012 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.012 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.012 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:17.012 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106315+0000", 2024-09-21T15:50:17.012 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8227999999999997e-05 2024-09-21T15:50:17.012 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:17.012 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:17.012 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:17.013 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.013 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.013 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116325 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 530432~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:17.013 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048851+0000", 2024-09-21T15:50:17.013 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 20.931983819999999, 2024-09-21T15:50:17.013 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057581036000000002, 2024-09-21T15:50:17.013 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:17.013 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:17.013 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:17.013 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:17.013 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:17.013 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116325 2024-09-21T15:50:17.013 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.013 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:17.013 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.014 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:17.014 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048851+0000", 2024-09-21T15:50:17.014 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:17.014 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.014 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.014 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:17.014 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048851+0000", 2024-09-21T15:50:17.014 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:17.014 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.014 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.014 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:17.014 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048851+0000", 2024-09-21T15:50:17.014 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:17.014 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.015 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.015 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:17.015 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048852+0000", 2024-09-21T15:50:17.015 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.077e-06 2024-09-21T15:50:17.015 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.015 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.015 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:17.015 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048855+0000", 2024-09-21T15:50:17.015 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9859999999999999e-06 2024-09-21T15:50:17.015 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.015 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.015 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.015 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048858+0000", 2024-09-21T15:50:17.015 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3810000000000002e-06 2024-09-21T15:50:17.015 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.016 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.016 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:17.016 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106335+0000", 2024-09-21T15:50:17.016 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057477473000000001 2024-09-21T15:50:17.016 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.016 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.016 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:17.016 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106393+0000", 2024-09-21T15:50:17.016 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8050000000000002e-05 2024-09-21T15:50:17.016 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.016 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.016 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:17.016 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106432+0000", 2024-09-21T15:50:17.016 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9274999999999999e-05 2024-09-21T15:50:17.016 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:17.016 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:17.017 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:17.017 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.017 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.017 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116326 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 536576~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:17.017 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048860+0000", 2024-09-21T15:50:17.017 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 20.931975769000001, 2024-09-21T15:50:17.017 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057774197999999999, 2024-09-21T15:50:17.017 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:17.017 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:17.017 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:17.017 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:17.017 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:17.017 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116326 2024-09-21T15:50:17.017 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.017 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:17.017 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.018 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:17.018 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048860+0000", 2024-09-21T15:50:17.018 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:17.018 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.018 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.018 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:17.018 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048860+0000", 2024-09-21T15:50:17.018 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:17.018 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.018 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.018 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:17.018 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048859+0000", 2024-09-21T15:50:17.018 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:17.018 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.018 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.019 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:17.019 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048860+0000", 2024-09-21T15:50:17.019 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.02e-06 2024-09-21T15:50:17.019 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.019 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.019 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:17.019 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048863+0000", 2024-09-21T15:50:17.019 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1039999999999998e-06 2024-09-21T15:50:17.019 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.019 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.019 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.019 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048867+0000", 2024-09-21T15:50:17.019 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.563e-06 2024-09-21T15:50:17.019 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.019 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.020 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:17.020 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106465+0000", 2024-09-21T15:50:17.020 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057597789000000003 2024-09-21T15:50:17.020 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.020 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.020 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:17.020 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106552+0000", 2024-09-21T15:50:17.020 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7602000000000006e-05 2024-09-21T15:50:17.020 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.020 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.020 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:17.020 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106634+0000", 2024-09-21T15:50:17.020 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1321000000000003e-05 2024-09-21T15:50:17.020 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:17.020 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:17.021 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:17.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.021 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.021 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116327 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 542720~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:17.021 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048870+0000", 2024-09-21T15:50:17.021 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 20.931964897, 2024-09-21T15:50:17.021 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057883148000000002, 2024-09-21T15:50:17.021 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:17.021 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:17.021 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:17.021 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:17.021 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:17.021 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116327 2024-09-21T15:50:17.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.021 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:17.022 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.022 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:17.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048870+0000", 2024-09-21T15:50:17.022 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:17.022 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.022 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.022 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:17.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048870+0000", 2024-09-21T15:50:17.022 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:17.022 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.022 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.022 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:17.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048870+0000", 2024-09-21T15:50:17.022 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:17.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.023 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.023 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:17.023 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048872+0000", 2024-09-21T15:50:17.023 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.824e-06 2024-09-21T15:50:17.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.023 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.023 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:17.023 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048879+0000", 2024-09-21T15:50:17.023 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6749999999999996e-06 2024-09-21T15:50:17.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.023 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.023 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.023 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048883+0000", 2024-09-21T15:50:17.023 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2200000000000003e-06 2024-09-21T15:50:17.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.024 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.024 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:17.024 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106663+0000", 2024-09-21T15:50:17.024 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057780049 2024-09-21T15:50:17.024 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.024 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.024 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:17.024 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106711+0000", 2024-09-21T15:50:17.024 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8433999999999997e-05 2024-09-21T15:50:17.024 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.024 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.024 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:17.024 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106754+0000", 2024-09-21T15:50:17.025 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2236e-05 2024-09-21T15:50:17.025 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:17.025 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:17.025 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:17.025 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.025 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.025 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116328 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 548864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:17.025 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048887+0000", 2024-09-21T15:50:17.025 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 20.931948377000001, 2024-09-21T15:50:17.025 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057981984, 2024-09-21T15:50:17.025 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:17.025 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:17.025 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:17.025 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:17.025 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:17.026 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116328 2024-09-21T15:50:17.026 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.026 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:17.026 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.026 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:17.026 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048887+0000", 2024-09-21T15:50:17.026 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:17.026 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.026 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.026 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:17.026 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048887+0000", 2024-09-21T15:50:17.026 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:17.026 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.026 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.026 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:17.027 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048887+0000", 2024-09-21T15:50:17.027 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:17.027 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.027 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.027 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:17.027 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048891+0000", 2024-09-21T15:50:17.027 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.843e-06 2024-09-21T15:50:17.027 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.027 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.027 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:17.027 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048897+0000", 2024-09-21T15:50:17.027 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5419999999999996e-06 2024-09-21T15:50:17.027 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.027 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.028 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.028 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048901+0000", 2024-09-21T15:50:17.028 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4689999999999999e-06 2024-09-21T15:50:17.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.028 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.028 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:17.028 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106780+0000", 2024-09-21T15:50:17.028 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057878259000000001 2024-09-21T15:50:17.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.028 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.028 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:17.028 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106827+0000", 2024-09-21T15:50:17.028 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7098000000000002e-05 2024-09-21T15:50:17.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.028 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.028 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:17.029 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106869+0000", 2024-09-21T15:50:17.029 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2129999999999998e-05 2024-09-21T15:50:17.029 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:17.029 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:17.029 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:17.029 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.029 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.029 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116329 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 555008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:17.029 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048905+0000", 2024-09-21T15:50:17.029 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 20.931930266999998, 2024-09-21T15:50:17.029 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058115554999999999, 2024-09-21T15:50:17.029 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:17.029 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:17.029 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:17.029 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:17.029 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:17.030 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116329 2024-09-21T15:50:17.030 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.030 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:17.030 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.030 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:17.030 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048905+0000", 2024-09-21T15:50:17.030 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:17.030 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.030 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.030 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:17.030 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048905+0000", 2024-09-21T15:50:17.030 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:17.030 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.030 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.030 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:17.031 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048905+0000", 2024-09-21T15:50:17.031 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:17.031 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.031 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.031 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:17.031 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048907+0000", 2024-09-21T15:50:17.031 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.086e-06 2024-09-21T15:50:17.031 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.031 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.031 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:17.031 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048912+0000", 2024-09-21T15:50:17.031 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2830000000000001e-06 2024-09-21T15:50:17.031 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.031 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.032 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.032 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048916+0000", 2024-09-21T15:50:17.032 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.98e-06 2024-09-21T15:50:17.032 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.032 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.032 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:17.032 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106907+0000", 2024-09-21T15:50:17.032 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057990719000000003 2024-09-21T15:50:17.032 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.032 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.032 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:17.032 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106969+0000", 2024-09-21T15:50:17.032 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1789000000000001e-05 2024-09-21T15:50:17.032 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.032 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.032 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:17.032 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107021+0000", 2024-09-21T15:50:17.033 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2030999999999997e-05 2024-09-21T15:50:17.033 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:17.033 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:17.033 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:17.033 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.033 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.033 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116330 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 561152~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:17.033 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048922+0000", 2024-09-21T15:50:17.033 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 20.931913109, 2024-09-21T15:50:17.033 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058242343000000002, 2024-09-21T15:50:17.033 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:17.033 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:17.033 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:17.033 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:17.033 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:17.033 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116330 2024-09-21T15:50:17.034 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.034 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:17.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.034 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:17.034 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048922+0000", 2024-09-21T15:50:17.034 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:17.034 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.034 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:17.034 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048922+0000", 2024-09-21T15:50:17.034 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:17.034 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.034 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:17.035 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048922+0000", 2024-09-21T15:50:17.035 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:17.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.035 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:17.035 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048924+0000", 2024-09-21T15:50:17.035 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1660000000000001e-06 2024-09-21T15:50:17.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.035 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:17.035 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048930+0000", 2024-09-21T15:50:17.035 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4529999999999999e-06 2024-09-21T15:50:17.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.035 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.036 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048934+0000", 2024-09-21T15:50:17.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0269999999999997e-06 2024-09-21T15:50:17.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.036 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:17.036 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107047+0000", 2024-09-21T15:50:17.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058112913000000002 2024-09-21T15:50:17.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.036 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:17.036 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107095+0000", 2024-09-21T15:50:17.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7778000000000002e-05 2024-09-21T15:50:17.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.036 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:17.037 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107165+0000", 2024-09-21T15:50:17.037 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9370999999999997e-05 2024-09-21T15:50:17.037 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:17.037 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:17.037 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:17.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.037 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116331 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 567296~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:17.037 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.049440+0000", 2024-09-21T15:50:17.037 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 20.931395015, 2024-09-21T15:50:17.037 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057878485, 2024-09-21T15:50:17.037 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:17.037 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:17.038 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:17.038 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:17.038 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:17.038 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116331 2024-09-21T15:50:17.038 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.038 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:17.038 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.038 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:17.038 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049440+0000", 2024-09-21T15:50:17.038 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:17.038 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.038 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.038 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:17.039 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049440+0000", 2024-09-21T15:50:17.039 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:17.039 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.039 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.039 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:17.039 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048939+0000", 2024-09-21T15:50:17.039 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9994984 2024-09-21T15:50:17.039 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.039 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.039 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:17.039 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049444+0000", 2024-09-21T15:50:17.039 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00050557200000000003 2024-09-21T15:50:17.039 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.039 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.040 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:17.040 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049460+0000", 2024-09-21T15:50:17.040 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6120000000000002e-05 2024-09-21T15:50:17.040 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.040 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.040 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049468+0000", 2024-09-21T15:50:17.040 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4209999999999998e-06 2024-09-21T15:50:17.040 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.040 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:17.040 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107189+0000", 2024-09-21T15:50:17.040 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057721019999999998 2024-09-21T15:50:17.041 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.041 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.041 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:17.041 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107254+0000", 2024-09-21T15:50:17.041 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5265000000000006e-05 2024-09-21T15:50:17.041 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.041 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.041 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:17.041 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107319+0000", 2024-09-21T15:50:17.041 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4784999999999997e-05 2024-09-21T15:50:17.041 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:17.041 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:17.041 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:17.041 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.041 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.042 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116332 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 573440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:17.042 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.049474+0000", 2024-09-21T15:50:17.042 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 20.931361754000001, 2024-09-21T15:50:17.042 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058013809, 2024-09-21T15:50:17.042 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:17.042 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:17.042 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:17.042 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:17.042 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:17.042 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116332 2024-09-21T15:50:17.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.042 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:17.042 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.042 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:17.043 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049474+0000", 2024-09-21T15:50:17.043 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:17.043 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.043 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.043 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:17.043 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049474+0000", 2024-09-21T15:50:17.043 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:17.043 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.043 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.043 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:17.043 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049473+0000", 2024-09-21T15:50:17.043 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:17.043 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.043 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.044 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:17.044 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049476+0000", 2024-09-21T15:50:17.044 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6249999999999999e-06 2024-09-21T15:50:17.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.044 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:17.044 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049482+0000", 2024-09-21T15:50:17.044 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0959999999999997e-06 2024-09-21T15:50:17.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.044 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.044 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049487+0000", 2024-09-21T15:50:17.044 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5949999999999998e-06 2024-09-21T15:50:17.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:17.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107344+0000", 2024-09-21T15:50:17.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057856594999999997 2024-09-21T15:50:17.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:17.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107422+0000", 2024-09-21T15:50:17.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7582000000000004e-05 2024-09-21T15:50:17.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:17.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107487+0000", 2024-09-21T15:50:17.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5723999999999996e-05 2024-09-21T15:50:17.045 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:17.045 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:17.046 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:17.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.046 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116333 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 579584~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:17.046 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.049993+0000", 2024-09-21T15:50:17.046 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 20.930841971, 2024-09-21T15:50:17.046 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057663537000000001, 2024-09-21T15:50:17.046 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:17.046 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:17.046 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:17.046 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:17.046 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:17.046 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116333 2024-09-21T15:50:17.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.047 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:17.047 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.047 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:17.047 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049993+0000", 2024-09-21T15:50:17.047 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:17.047 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.047 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.047 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:17.047 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049993+0000", 2024-09-21T15:50:17.047 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:17.047 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.047 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.047 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:17.047 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049491+0000", 2024-09-21T15:50:17.047 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9994979 2024-09-21T15:50:17.048 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.048 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.048 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:17.048 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049998+0000", 2024-09-21T15:50:17.048 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00050654200000000002 2024-09-21T15:50:17.048 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.048 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.048 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:17.048 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050010+0000", 2024-09-21T15:50:17.048 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1790999999999999e-05 2024-09-21T15:50:17.048 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.048 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.048 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.048 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050016+0000", 2024-09-21T15:50:17.048 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2179999999999996e-06 2024-09-21T15:50:17.049 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.049 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.049 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:17.049 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107513+0000", 2024-09-21T15:50:17.049 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057496927000000003 2024-09-21T15:50:17.049 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.049 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.049 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:17.049 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107590+0000", 2024-09-21T15:50:17.049 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7547999999999997e-05 2024-09-21T15:50:17.049 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.049 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.049 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:17.049 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107657+0000", 2024-09-21T15:50:17.049 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6630000000000004e-05 2024-09-21T15:50:17.050 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:17.050 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:17.050 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:17.050 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.050 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.050 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116334 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 585728~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:17.050 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.050091+0000", 2024-09-21T15:50:17.050 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 20.930744343000001, 2024-09-21T15:50:17.050 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057734355000000001, 2024-09-21T15:50:17.050 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:17.050 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:17.050 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:17.050 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:17.050 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:17.050 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116334 2024-09-21T15:50:17.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.051 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:17.051 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.051 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:17.051 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050091+0000", 2024-09-21T15:50:17.051 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:17.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.051 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.051 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:17.051 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050091+0000", 2024-09-21T15:50:17.051 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:17.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.051 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.051 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:17.051 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050021+0000", 2024-09-21T15:50:17.052 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999304 2024-09-21T15:50:17.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.052 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.052 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:17.052 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050095+0000", 2024-09-21T15:50:17.052 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4158000000000004e-05 2024-09-21T15:50:17.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.052 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.052 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:17.052 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050103+0000", 2024-09-21T15:50:17.052 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0770000000000002e-06 2024-09-21T15:50:17.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.052 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.052 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.053 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050108+0000", 2024-09-21T15:50:17.053 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6330000000000001e-06 2024-09-21T15:50:17.053 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.053 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.053 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:17.053 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107681+0000", 2024-09-21T15:50:17.053 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057572996000000001 2024-09-21T15:50:17.053 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.053 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.053 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:17.053 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107759+0000", 2024-09-21T15:50:17.053 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7368999999999997e-05 2024-09-21T15:50:17.053 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.053 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.053 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:17.054 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107825+0000", 2024-09-21T15:50:17.054 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6783000000000001e-05 2024-09-21T15:50:17.054 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:17.054 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:17.054 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:17.054 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.054 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.054 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116335 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 591872~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:17.054 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.050542+0000", 2024-09-21T15:50:17.054 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 20.930293758000001, 2024-09-21T15:50:17.054 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057466455999999999, 2024-09-21T15:50:17.054 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:17.055 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:17.055 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:17.055 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:17.055 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:17.055 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116335 2024-09-21T15:50:17.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.055 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:17.055 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.055 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:17.055 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050542+0000", 2024-09-21T15:50:17.055 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:17.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.055 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.055 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:17.055 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050542+0000", 2024-09-21T15:50:17.055 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:17.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.056 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.056 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:17.056 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050113+0000", 2024-09-21T15:50:17.056 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9995713 2024-09-21T15:50:17.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.056 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.056 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:17.056 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050546+0000", 2024-09-21T15:50:17.056 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00043328699999999997 2024-09-21T15:50:17.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.056 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.056 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:17.056 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050557+0000", 2024-09-21T15:50:17.056 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1011e-05 2024-09-21T15:50:17.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.057 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.057 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050564+0000", 2024-09-21T15:50:17.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1130000000000003e-06 2024-09-21T15:50:17.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.057 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:17.057 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107852+0000", 2024-09-21T15:50:17.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057287350000000001 2024-09-21T15:50:17.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.057 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:17.057 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107928+0000", 2024-09-21T15:50:17.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6668000000000005e-05 2024-09-21T15:50:17.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.058 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:17.058 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108008+0000", 2024-09-21T15:50:17.058 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9783999999999994e-05 2024-09-21T15:50:17.058 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:17.058 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:17.058 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:17.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.058 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116336 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 598016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:17.058 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.050834+0000", 2024-09-21T15:50:17.058 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 20.930001491999999, 2024-09-21T15:50:17.058 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057353572999999998, 2024-09-21T15:50:17.058 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:17.058 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:17.058 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:17.059 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:17.059 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:17.059 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116336 2024-09-21T15:50:17.059 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.059 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:17.059 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.059 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:17.059 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050834+0000", 2024-09-21T15:50:17.059 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:17.059 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.059 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.059 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:17.059 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050834+0000", 2024-09-21T15:50:17.059 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:17.059 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.059 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.060 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:17.060 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050571+0000", 2024-09-21T15:50:17.060 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9997373 2024-09-21T15:50:17.060 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.060 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.060 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:17.060 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050842+0000", 2024-09-21T15:50:17.060 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00027029099999999998 2024-09-21T15:50:17.060 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.060 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.060 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:17.060 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050853+0000", 2024-09-21T15:50:17.060 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1532000000000001e-05 2024-09-21T15:50:17.060 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.061 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.061 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050863+0000", 2024-09-21T15:50:17.061 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4239999999999999e-06 2024-09-21T15:50:17.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.061 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:17.061 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108045+0000", 2024-09-21T15:50:17.061 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057182415 2024-09-21T15:50:17.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.061 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:17.061 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108123+0000", 2024-09-21T15:50:17.061 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8275000000000005e-05 2024-09-21T15:50:17.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.062 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:17.062 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108187+0000", 2024-09-21T15:50:17.062 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4152000000000005e-05 2024-09-21T15:50:17.062 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:17.062 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:17.062 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:17.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.062 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116337 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 604160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:17.062 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.051192+0000", 2024-09-21T15:50:17.062 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 20.929642864000002, 2024-09-21T15:50:17.062 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057175981000000001, 2024-09-21T15:50:17.062 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:17.062 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:17.062 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:17.062 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:17.063 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:17.063 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116337 2024-09-21T15:50:17.063 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.063 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:17.063 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.063 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:17.063 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051192+0000", 2024-09-21T15:50:17.063 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:17.063 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.063 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.063 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:17.063 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051192+0000", 2024-09-21T15:50:17.063 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:17.063 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.063 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.064 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:17.064 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050870+0000", 2024-09-21T15:50:17.064 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9996777 2024-09-21T15:50:17.064 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.064 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.064 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:17.064 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051197+0000", 2024-09-21T15:50:17.064 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00032656299999999998 2024-09-21T15:50:17.064 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.064 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.064 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:17.064 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051209+0000", 2024-09-21T15:50:17.064 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2636000000000001e-05 2024-09-21T15:50:17.064 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.064 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.065 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.065 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051217+0000", 2024-09-21T15:50:17.065 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5989999999999996e-06 2024-09-21T15:50:17.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.065 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:17.065 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108224+0000", 2024-09-21T15:50:17.065 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057007622000000001 2024-09-21T15:50:17.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.065 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:17.065 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108299+0000", 2024-09-21T15:50:17.065 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5099999999999996e-05 2024-09-21T15:50:17.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.065 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:17.066 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108368+0000", 2024-09-21T15:50:17.066 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8918000000000007e-05 2024-09-21T15:50:17.066 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:17.066 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:17.066 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:17.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.066 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.066 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116338 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 610304~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:17.066 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.051583+0000", 2024-09-21T15:50:17.066 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 20.929252411, 2024-09-21T15:50:17.066 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056967716000000002, 2024-09-21T15:50:17.066 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:17.066 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:17.066 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:17.066 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:17.067 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:17.067 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116338 2024-09-21T15:50:17.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.067 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:17.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.067 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:17.067 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051583+0000", 2024-09-21T15:50:17.067 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:17.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.067 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:17.067 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051583+0000", 2024-09-21T15:50:17.067 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:17.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.067 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:17.068 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051223+0000", 2024-09-21T15:50:17.068 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.99964 2024-09-21T15:50:17.068 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.068 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.068 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:17.068 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051587+0000", 2024-09-21T15:50:17.068 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00036352999999999998 2024-09-21T15:50:17.068 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.068 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.068 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:17.068 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051597+0000", 2024-09-21T15:50:17.068 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.893e-06 2024-09-21T15:50:17.068 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.068 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.068 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.068 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051603+0000", 2024-09-21T15:50:17.069 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0519999999999998e-06 2024-09-21T15:50:17.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.069 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:17.069 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108394+0000", 2024-09-21T15:50:17.069 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056791531999999999 2024-09-21T15:50:17.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.069 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:17.069 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108480+0000", 2024-09-21T15:50:17.069 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.577e-05 2024-09-21T15:50:17.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:17.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:17.069 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:17.069 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108551+0000", 2024-09-21T15:50:17.069 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0722000000000005e-05 2024-09-21T15:50:17.070 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:17.070 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:17.070 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:17.070 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:17.070 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:17.070 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:17.071 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:50:17.072 INFO:teuthology.orchestra.run.smithi089.stdout:5442 trunc from 0xc20147c to 0xbc37a3 2024-09-21T15:50:17.072 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:17.070+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:17.171 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:17.170+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:17.229 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:17.229 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:50:17.229 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.229 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4403.0:143290 3.e e81/22 3:72d90f68:::rbd_data.11338b2e77e3.0000000000000004:head v 81'396, mlcod=81'394)", 2024-09-21T15:50:17.230 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:17.222714+0000", 2024-09-21T15:50:17.230 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.0061883140000000003, 2024-09-21T15:50:17.230 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.006205276, 2024-09-21T15:50:17.230 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:17.230 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:17.230 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:17.230 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.230 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:17.230 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:17.222714+0000", 2024-09-21T15:50:17.230 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.230 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.230 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.231 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:17.231 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:17.222714+0000", 2024-09-21T15:50:17.231 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.231 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.231 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.231 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:17.231 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:17.222712+0000", 2024-09-21T15:50:17.231 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-21T15:50:17.231 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.231 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.231 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:17.231 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:17.223374+0000", 2024-09-21T15:50:17.231 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00066212400000000005 2024-09-21T15:50:17.231 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.232 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.232 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:17.232 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:17.223395+0000", 2024-09-21T15:50:17.232 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0372000000000001e-05 2024-09-21T15:50:17.232 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.232 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.232 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.232 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:17.223404+0000", 2024-09-21T15:50:17.232 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.9449999999999999e-06 2024-09-21T15:50:17.232 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.232 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.232 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:17.232 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:17.223477+0000", 2024-09-21T15:50:17.233 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3540000000000004e-05 2024-09-21T15:50:17.233 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.233 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.233 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:17.233 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:17.223501+0000", 2024-09-21T15:50:17.233 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3563000000000001e-05 2024-09-21T15:50:17.233 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.233 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:17.233 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.233 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.233 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.233 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4403.0:143292 3.e e81/22 3:779de3ac:::rbd_data.11338b2e77e3.0000000000000006:head v 81'397, mlcod=81'394)", 2024-09-21T15:50:17.233 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:17.225374+0000", 2024-09-21T15:50:17.233 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.0035283810000000001, 2024-09-21T15:50:17.234 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.003581908, 2024-09-21T15:50:17.234 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:17.234 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:17.234 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:17.234 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.234 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:17.234 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:17.225374+0000", 2024-09-21T15:50:17.234 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.234 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.234 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.234 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:17.234 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:17.225374+0000", 2024-09-21T15:50:17.234 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.234 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.235 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.235 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:17.235 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:17.225373+0000", 2024-09-21T15:50:17.235 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-21T15:50:17.235 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.235 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.235 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:17.235 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:17.225380+0000", 2024-09-21T15:50:17.235 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4619999999999999e-06 2024-09-21T15:50:17.235 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.235 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.235 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:17.235 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:17.225395+0000", 2024-09-21T15:50:17.236 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4994000000000001e-05 2024-09-21T15:50:17.236 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.236 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.236 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.236 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:17.225403+0000", 2024-09-21T15:50:17.236 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4220000000000003e-06 2024-09-21T15:50:17.236 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.236 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.236 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:17.236 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:17.225453+0000", 2024-09-21T15:50:17.236 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0163000000000001e-05 2024-09-21T15:50:17.236 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.236 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.236 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:17.237 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:17.225473+0000", 2024-09-21T15:50:17.237 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9986000000000001e-05 2024-09-21T15:50:17.237 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.237 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:17.237 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.237 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.237 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-21T15:50:17.237 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 2 2024-09-21T15:50:17.237 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:17.240 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:50:17.281 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:17.280+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:17.373 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:17.372+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:17.378 INFO:teuthology.orchestra.run.smithi089.stdout:5443 write 0x68ea88 thru 0x69d902 (0xee7b bytes) 2024-09-21T15:50:17.396 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:17.396 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:17.396 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:50:17.396 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:50:17.397 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:17.397 INFO:teuthology.orchestra.run.smithi089.stdout:5446 write 0x9712efc thru 0x9721680 (0xe785 bytes) 2024-09-21T15:50:17.401 INFO:teuthology.orchestra.run.smithi089.stdout:5447 writesame 0xb0a5d7 thru 0xb0e4c6 data_size 0x4c(0x3ef0 bytes) 2024-09-21T15:50:17.403 INFO:teuthology.orchestra.run.smithi089.stdout:5449 write 0x4cfb357 thru 0x4d04b06 (0x97b0 bytes) 2024-09-21T15:50:17.406 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:50:17.418 INFO:teuthology.orchestra.run.smithi089.stdout:5450 write 0xb867580 thru 0xb8722e2 (0xad63 bytes) 2024-09-21T15:50:17.421 INFO:teuthology.orchestra.run.smithi089.stdout:5451 writesame 0x3424e4a thru 0x3432a4e data_size 0x19(0xdc05 bytes) 2024-09-21T15:50:17.425 INFO:teuthology.orchestra.run.smithi089.stdout:5452 punch from 0xb5dafb3 to 0xb5e1c67, (0x6cb4 bytes) 2024-09-21T15:50:17.425 INFO:teuthology.orchestra.run.smithi089.stdout:5453 read 0x2de6e9a thru 0x2df4a1a (0xdb81 bytes) 2024-09-21T15:50:17.437 INFO:teuthology.orchestra.run.smithi089.stdout:5454 trunc from 0xb8722e3 to 0xbc97bf 2024-09-21T15:50:17.452 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:17.469 INFO:teuthology.orchestra.run.smithi089.stdout:5456 write 0x9c58643 thru 0x9c615f2 (0x8fb0 bytes) 2024-09-21T15:50:17.473 INFO:teuthology.orchestra.run.smithi089.stdout:5457 write 0x3f217b8 thru 0x3f27182 (0x59cb bytes) 2024-09-21T15:50:17.473 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:17.472+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:17.476 INFO:teuthology.orchestra.run.smithi089.stdout:5458 write 0x98306ab thru 0x98405e7 (0xff3d bytes) 2024-09-21T15:50:17.488 INFO:teuthology.orchestra.run.smithi089.stdout:5459 write 0xc07c9db thru 0xc088e31 (0xc457 bytes) 2024-09-21T15:50:17.495 INFO:teuthology.orchestra.run.smithi089.stdout:5460 punch from 0x81a7f8 to 0x81d5ff, (0x2e07 bytes) 2024-09-21T15:50:17.495 INFO:teuthology.orchestra.run.smithi089.stdout:5462 read 0xa9ac6f thru 0xaa3979 (0x8d0b bytes) 2024-09-21T15:50:17.496 INFO:teuthology.orchestra.run.smithi089.stdout:5463 writesame 0xbb86df5 thru 0xbb9510d data_size 0x1(0xe319 bytes) 2024-09-21T15:50:17.519 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:17.519 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:50:17.520 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:50:17.520 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:50:17.520 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.520 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:125291 3.1b 3:d87db7c4:::rbd_data.1133545d3fa6.0000000000000013:1e [read 884736~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:50:17.520 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:05.145761+0000", 2024-09-21T15:50:17.520 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.371986406, 2024-09-21T15:50:17.520 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045764303999999999, 2024-09-21T15:50:17.520 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:17.520 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:17.520 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:17.520 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:17.520 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:17.520 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 125291 2024-09-21T15:50:17.520 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.521 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:17.521 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.521 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:17.521 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.145761+0000", 2024-09-21T15:50:17.521 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.521 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.521 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.521 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:17.521 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.145761+0000", 2024-09-21T15:50:17.521 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.521 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.521 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.521 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:17.521 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.145761+0000", 2024-09-21T15:50:17.522 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:17.522 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.522 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.522 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:17.522 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.145762+0000", 2024-09-21T15:50:17.522 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1799999999999999e-06 2024-09-21T15:50:17.522 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.522 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.522 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:17.522 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.145765+0000", 2024-09-21T15:50:17.522 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7630000000000001e-06 2024-09-21T15:50:17.522 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.522 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.523 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.523 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.145767+0000", 2024-09-21T15:50:17.523 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4559999999999999e-06 2024-09-21T15:50:17.523 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.523 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.523 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:17.523 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.191362+0000", 2024-09-21T15:50:17.523 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045594329000000003 2024-09-21T15:50:17.523 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.523 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.523 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:17.523 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.191429+0000", 2024-09-21T15:50:17.523 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7185000000000001e-05 2024-09-21T15:50:17.524 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.524 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.524 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:17.524 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.191525+0000", 2024-09-21T15:50:17.524 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.6631000000000004e-05 2024-09-21T15:50:17.524 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.524 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:17.524 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.524 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.524 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.524 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:125292 3.1b 3:d87db7c4:::rbd_data.1133545d3fa6.0000000000000013:1e [read 905216~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:50:17.524 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:05.145790+0000", 2024-09-21T15:50:17.524 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.371957257, 2024-09-21T15:50:17.524 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045873948999999997, 2024-09-21T15:50:17.524 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:17.525 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:17.525 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:17.525 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:17.525 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:17.525 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 125292 2024-09-21T15:50:17.525 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.525 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:17.525 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.525 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:17.525 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.145790+0000", 2024-09-21T15:50:17.525 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.525 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.525 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.525 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:17.526 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.145790+0000", 2024-09-21T15:50:17.526 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.526 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.526 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.526 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:17.526 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.145790+0000", 2024-09-21T15:50:17.526 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:17.526 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.526 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.526 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:17.526 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.145792+0000", 2024-09-21T15:50:17.526 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7549999999999999e-06 2024-09-21T15:50:17.526 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.527 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.527 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:17.527 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.145795+0000", 2024-09-21T15:50:17.527 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1319999999999998e-06 2024-09-21T15:50:17.527 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.527 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.527 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.527 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.145797+0000", 2024-09-21T15:50:17.527 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5739999999999998e-06 2024-09-21T15:50:17.527 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.527 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.527 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:17.527 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.191555+0000", 2024-09-21T15:50:17.527 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045757412999999997 2024-09-21T15:50:17.527 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.528 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.528 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:17.528 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.191603+0000", 2024-09-21T15:50:17.528 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8368000000000002e-05 2024-09-21T15:50:17.528 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.528 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.528 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:17.528 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.191664+0000", 2024-09-21T15:50:17.528 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1104000000000003e-05 2024-09-21T15:50:17.528 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.528 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:17.528 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.528 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.528 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.528 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:125293 3.1b 3:d87db7c4:::rbd_data.1133545d3fa6.0000000000000013:1e [read 921600~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:50:17.529 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:05.145800+0000", 2024-09-21T15:50:17.529 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.371948034000001, 2024-09-21T15:50:17.529 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045990704, 2024-09-21T15:50:17.529 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:17.529 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:17.529 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:17.529 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:17.529 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:17.529 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 125293 2024-09-21T15:50:17.529 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.529 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:17.529 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.529 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:17.529 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.145800+0000", 2024-09-21T15:50:17.529 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.530 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.530 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.530 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:17.530 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.145800+0000", 2024-09-21T15:50:17.530 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.530 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.530 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.530 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:17.530 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.145799+0000", 2024-09-21T15:50:17.530 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:17.530 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.530 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.530 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:17.531 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.145800+0000", 2024-09-21T15:50:17.531 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2300000000000001e-06 2024-09-21T15:50:17.531 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.531 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.531 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:17.531 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.145803+0000", 2024-09-21T15:50:17.531 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.712e-06 2024-09-21T15:50:17.531 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.531 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.531 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.531 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.145806+0000", 2024-09-21T15:50:17.531 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4360000000000001e-06 2024-09-21T15:50:17.531 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.531 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.531 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:17.532 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.191694+0000", 2024-09-21T15:50:17.532 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045888311000000001 2024-09-21T15:50:17.532 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.532 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.532 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:17.532 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.191736+0000", 2024-09-21T15:50:17.532 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1959000000000003e-05 2024-09-21T15:50:17.532 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.532 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.532 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:17.532 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.191790+0000", 2024-09-21T15:50:17.532 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4348999999999998e-05 2024-09-21T15:50:17.532 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.532 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:17.532 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.532 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.533 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.533 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:125294 3.1b 3:d87db7c4:::rbd_data.1133545d3fa6.0000000000000013:1e [read 942080~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:50:17.533 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:05.145808+0000", 2024-09-21T15:50:17.533 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.371939994, 2024-09-21T15:50:17.533 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046102812, 2024-09-21T15:50:17.533 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:17.533 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:17.533 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:17.533 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:17.533 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:17.533 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 125294 2024-09-21T15:50:17.533 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.533 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:17.533 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.533 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:17.534 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.145808+0000", 2024-09-21T15:50:17.534 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.534 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.534 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.534 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:17.534 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.145808+0000", 2024-09-21T15:50:17.534 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.534 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.534 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.534 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:17.534 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.145807+0000", 2024-09-21T15:50:17.534 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:50:17.534 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.534 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.535 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:17.535 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.145809+0000", 2024-09-21T15:50:17.535 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1909999999999999e-06 2024-09-21T15:50:17.535 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.535 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.535 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:17.535 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.145811+0000", 2024-09-21T15:50:17.535 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7580000000000001e-06 2024-09-21T15:50:17.535 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.535 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.535 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.535 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.145814+0000", 2024-09-21T15:50:17.535 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4360000000000001e-06 2024-09-21T15:50:17.535 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.535 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.536 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:17.536 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.191803+0000", 2024-09-21T15:50:17.536 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045989533999999999 2024-09-21T15:50:17.536 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.536 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.536 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:17.536 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.191839+0000", 2024-09-21T15:50:17.536 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5441000000000002e-05 2024-09-21T15:50:17.536 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.536 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.536 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:17.536 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.191910+0000", 2024-09-21T15:50:17.536 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1675999999999996e-05 2024-09-21T15:50:17.536 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.537 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:17.537 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.537 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.537 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.537 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:125295 3.1b 3:d87db7c4:::rbd_data.1133545d3fa6.0000000000000013:1e [read 962560~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:50:17.537 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:05.146015+0000", 2024-09-21T15:50:17.537 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.371732695, 2024-09-21T15:50:17.537 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046083249999999999, 2024-09-21T15:50:17.537 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:17.537 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:17.537 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:17.537 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:17.537 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:17.538 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 125295 2024-09-21T15:50:17.538 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.538 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:17.538 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.538 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:17.538 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.146015+0000", 2024-09-21T15:50:17.538 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.538 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.538 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.538 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:17.538 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.146015+0000", 2024-09-21T15:50:17.538 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.538 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.538 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.539 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:17.539 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.145815+0000", 2024-09-21T15:50:17.539 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998007 2024-09-21T15:50:17.539 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.539 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.539 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:17.539 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.146016+0000", 2024-09-21T15:50:17.539 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00020100000000000001 2024-09-21T15:50:17.539 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.539 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.539 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:17.539 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.146021+0000", 2024-09-21T15:50:17.539 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1230000000000002e-06 2024-09-21T15:50:17.539 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.539 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.540 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.540 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.146024+0000", 2024-09-21T15:50:17.540 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8949999999999998e-06 2024-09-21T15:50:17.540 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.540 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.540 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:17.540 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.191931+0000", 2024-09-21T15:50:17.540 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045907798 2024-09-21T15:50:17.540 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.540 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.540 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:17.540 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.191998+0000", 2024-09-21T15:50:17.540 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6774999999999997e-05 2024-09-21T15:50:17.540 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.541 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.541 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:17.541 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.192098+0000", 2024-09-21T15:50:17.541 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000100058 2024-09-21T15:50:17.541 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.541 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:17.541 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.541 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.541 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.541 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:125382 3.1b 3:d87db7c4:::rbd_data.1133545d3fa6.0000000000000013:1e [read 350208~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:50:17.541 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:05.150211+0000", 2024-09-21T15:50:17.541 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.367537049999999, 2024-09-21T15:50:17.541 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045716650999999997, 2024-09-21T15:50:17.542 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:17.542 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:17.542 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:17.542 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:17.542 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:17.542 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 125382 2024-09-21T15:50:17.542 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.542 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:17.542 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.542 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:17.542 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.150211+0000", 2024-09-21T15:50:17.542 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.542 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.542 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.543 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:17.543 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.150211+0000", 2024-09-21T15:50:17.543 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.543 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.543 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.543 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:17.543 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.150210+0000", 2024-09-21T15:50:17.543 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:17.543 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.543 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.543 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:17.543 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.150212+0000", 2024-09-21T15:50:17.543 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0899999999999999e-06 2024-09-21T15:50:17.543 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.544 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.544 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:17.544 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.150219+0000", 2024-09-21T15:50:17.544 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.939e-06 2024-09-21T15:50:17.544 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.544 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.544 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.544 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.150224+0000", 2024-09-21T15:50:17.544 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6399999999999996e-06 2024-09-21T15:50:17.544 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.544 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.544 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:17.544 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.195813+0000", 2024-09-21T15:50:17.545 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045589400000000002 2024-09-21T15:50:17.545 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.545 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.545 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:17.545 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.195857+0000", 2024-09-21T15:50:17.545 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3998999999999997e-05 2024-09-21T15:50:17.545 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.545 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.545 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:17.545 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.195927+0000", 2024-09-21T15:50:17.545 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9888999999999994e-05 2024-09-21T15:50:17.545 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.545 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:17.545 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.546 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.546 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.546 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:125383 3.1b 3:d87db7c4:::rbd_data.1133545d3fa6.0000000000000013:1e [read 370688~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:50:17.546 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:05.150229+0000", 2024-09-21T15:50:17.546 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.367518904000001, 2024-09-21T15:50:17.546 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045962198000000003, 2024-09-21T15:50:17.546 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:17.546 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:17.546 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:17.546 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:17.546 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:17.546 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 125383 2024-09-21T15:50:17.546 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.546 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:17.546 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.546 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:17.547 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.150229+0000", 2024-09-21T15:50:17.547 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.547 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.547 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.547 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:17.547 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.150229+0000", 2024-09-21T15:50:17.547 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.547 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.547 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.547 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:17.547 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.150228+0000", 2024-09-21T15:50:17.547 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:17.547 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.547 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.547 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:17.548 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.150231+0000", 2024-09-21T15:50:17.548 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.357e-06 2024-09-21T15:50:17.548 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.548 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.548 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:17.548 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.150236+0000", 2024-09-21T15:50:17.548 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6810000000000003e-06 2024-09-21T15:50:17.548 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.548 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.548 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.548 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.150241+0000", 2024-09-21T15:50:17.548 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7600000000000002e-06 2024-09-21T15:50:17.548 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.548 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.548 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:17.548 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.195960+0000", 2024-09-21T15:50:17.549 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045719013000000003 2024-09-21T15:50:17.549 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.549 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.549 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:17.549 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.196051+0000", 2024-09-21T15:50:17.549 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0822999999999994e-05 2024-09-21T15:50:17.549 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.549 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.549 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:17.549 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.196191+0000", 2024-09-21T15:50:17.549 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013989699999999999 2024-09-21T15:50:17.549 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.549 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:17.549 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.549 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.549 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.550 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:125384 3.1b 3:d87db7c4:::rbd_data.1133545d3fa6.0000000000000013:1e [read 391168~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:50:17.550 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:05.150245+0000", 2024-09-21T15:50:17.550 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.367502155, 2024-09-21T15:50:17.550 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046069553999999999, 2024-09-21T15:50:17.550 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:17.550 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:17.550 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:17.550 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:17.550 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:17.550 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 125384 2024-09-21T15:50:17.550 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.550 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:17.550 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.551 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:17.551 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.150245+0000", 2024-09-21T15:50:17.551 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.551 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.551 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.551 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:17.551 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.150245+0000", 2024-09-21T15:50:17.551 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.551 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.551 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.551 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:17.551 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.150245+0000", 2024-09-21T15:50:17.551 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:17.551 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.551 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.551 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:17.552 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.150247+0000", 2024-09-21T15:50:17.552 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3659999999999999e-06 2024-09-21T15:50:17.552 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.552 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.552 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:17.552 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.150253+0000", 2024-09-21T15:50:17.552 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0689999999999998e-06 2024-09-21T15:50:17.552 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.552 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.552 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.552 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.150258+0000", 2024-09-21T15:50:17.552 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6859999999999997e-06 2024-09-21T15:50:17.552 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.552 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.552 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:17.553 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.196205+0000", 2024-09-21T15:50:17.553 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045946873999999999 2024-09-21T15:50:17.553 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.553 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.553 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:17.553 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.196246+0000", 2024-09-21T15:50:17.553 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0664999999999998e-05 2024-09-21T15:50:17.553 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.553 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.553 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:17.553 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.196315+0000", 2024-09-21T15:50:17.553 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9395999999999998e-05 2024-09-21T15:50:17.553 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.553 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:17.554 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.554 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.554 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.554 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:125391 3.1b 3:d87db7c4:::rbd_data.1133545d3fa6.0000000000000013:1e [read 526336~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:50:17.554 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:05.151721+0000", 2024-09-21T15:50:17.554 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.366026976000001, 2024-09-21T15:50:17.554 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045674394, 2024-09-21T15:50:17.554 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:17.554 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:17.554 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:17.554 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:17.554 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:17.554 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 125391 2024-09-21T15:50:17.554 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.554 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:17.555 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.555 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:17.555 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.151721+0000", 2024-09-21T15:50:17.555 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.555 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.555 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.555 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:17.555 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.151721+0000", 2024-09-21T15:50:17.555 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.555 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.555 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.555 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:17.555 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.151720+0000", 2024-09-21T15:50:17.555 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:17.555 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.556 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.556 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:17.556 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.151722+0000", 2024-09-21T15:50:17.556 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.255e-06 2024-09-21T15:50:17.556 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.556 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.556 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:17.556 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.151728+0000", 2024-09-21T15:50:17.556 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9070000000000004e-06 2024-09-21T15:50:17.556 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.556 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.556 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.556 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.151733+0000", 2024-09-21T15:50:17.556 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5730000000000001e-06 2024-09-21T15:50:17.557 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.557 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.557 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:17.557 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.197290+0000", 2024-09-21T15:50:17.557 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045556685 2024-09-21T15:50:17.557 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.557 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.557 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:17.557 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.197330+0000", 2024-09-21T15:50:17.557 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0352999999999998e-05 2024-09-21T15:50:17.557 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.557 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.557 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:17.557 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.197395+0000", 2024-09-21T15:50:17.557 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5001999999999999e-05 2024-09-21T15:50:17.558 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.558 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:17.558 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.558 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.558 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.558 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:125392 3.1b 3:d87db7c4:::rbd_data.1133545d3fa6.0000000000000013:1e [read 546816~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:50:17.558 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:05.151739+0000", 2024-09-21T15:50:17.558 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.366008638, 2024-09-21T15:50:17.558 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045785246000000002, 2024-09-21T15:50:17.558 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:17.558 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:17.558 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:17.558 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:17.559 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:17.559 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 125392 2024-09-21T15:50:17.559 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.559 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:17.559 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.559 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:17.559 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.151739+0000", 2024-09-21T15:50:17.559 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.559 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.559 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.559 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:17.559 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.151739+0000", 2024-09-21T15:50:17.560 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.560 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.560 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.560 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:17.560 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.151739+0000", 2024-09-21T15:50:17.560 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:17.560 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.560 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.560 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:17.560 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.151741+0000", 2024-09-21T15:50:17.560 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.046e-06 2024-09-21T15:50:17.560 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.560 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.561 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:17.561 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.151746+0000", 2024-09-21T15:50:17.561 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6289999999999998e-06 2024-09-21T15:50:17.561 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.561 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.561 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.561 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.151751+0000", 2024-09-21T15:50:17.561 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5859999999999998e-06 2024-09-21T15:50:17.561 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.561 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.561 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:17.561 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.197416+0000", 2024-09-21T15:50:17.561 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045665310000000001 2024-09-21T15:50:17.561 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.561 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.562 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:17.562 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.197458+0000", 2024-09-21T15:50:17.562 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1625000000000002e-05 2024-09-21T15:50:17.562 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.562 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.562 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:17.562 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.197524+0000", 2024-09-21T15:50:17.562 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6425000000000002e-05 2024-09-21T15:50:17.562 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.562 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:17.562 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.562 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.562 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.562 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:125393 3.1b 3:d87db7c4:::rbd_data.1133545d3fa6.0000000000000013:1e [read 567296~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:50:17.562 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:05.151755+0000", 2024-09-21T15:50:17.563 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.365992963, 2024-09-21T15:50:17.563 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045898429999999997, 2024-09-21T15:50:17.563 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:17.563 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:17.563 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:17.563 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:17.563 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:17.563 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 125393 2024-09-21T15:50:17.563 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.563 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:17.563 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.563 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:17.563 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.151755+0000", 2024-09-21T15:50:17.563 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.564 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.564 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.564 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:17.564 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.151755+0000", 2024-09-21T15:50:17.564 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.564 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.564 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.564 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:17.564 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.151754+0000", 2024-09-21T15:50:17.564 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:17.564 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.564 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.564 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:17.564 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.151756+0000", 2024-09-21T15:50:17.565 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0049999999999999e-06 2024-09-21T15:50:17.565 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.565 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.565 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:17.565 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.151762+0000", 2024-09-21T15:50:17.565 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6049999999999997e-06 2024-09-21T15:50:17.565 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.565 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.565 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.565 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.151767+0000", 2024-09-21T15:50:17.565 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5580000000000002e-06 2024-09-21T15:50:17.565 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.565 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.565 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:17.566 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.197545+0000", 2024-09-21T15:50:17.566 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045778555999999998 2024-09-21T15:50:17.566 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.566 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.566 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:17.566 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.197587+0000", 2024-09-21T15:50:17.566 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1863999999999998e-05 2024-09-21T15:50:17.566 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.566 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.566 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:17.566 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.197653+0000", 2024-09-21T15:50:17.566 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6131999999999996e-05 2024-09-21T15:50:17.566 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.566 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:17.567 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.567 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.567 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.567 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:125394 3.1b 3:d87db7c4:::rbd_data.1133545d3fa6.0000000000000013:1e [read 587776~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:50:17.567 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:05.151776+0000", 2024-09-21T15:50:17.567 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.36597156, 2024-09-21T15:50:17.567 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046003321999999999, 2024-09-21T15:50:17.567 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:17.567 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:17.567 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:17.567 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:17.567 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:17.567 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 125394 2024-09-21T15:50:17.567 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.568 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:17.568 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.568 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:17.568 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.151776+0000", 2024-09-21T15:50:17.568 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.568 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.568 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.568 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:17.568 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.151776+0000", 2024-09-21T15:50:17.568 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.568 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.568 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.568 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:17.569 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.151775+0000", 2024-09-21T15:50:17.569 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:17.569 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.569 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.569 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:17.569 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.151778+0000", 2024-09-21T15:50:17.569 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5919999999999999e-06 2024-09-21T15:50:17.569 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.569 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.569 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:17.569 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.151784+0000", 2024-09-21T15:50:17.569 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.054e-06 2024-09-21T15:50:17.569 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.569 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.570 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.570 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.151789+0000", 2024-09-21T15:50:17.570 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7489999999999998e-06 2024-09-21T15:50:17.570 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.570 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.570 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:17.570 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.197674+0000", 2024-09-21T15:50:17.570 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045885232999999997 2024-09-21T15:50:17.570 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.570 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.570 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:17.570 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.197714+0000", 2024-09-21T15:50:17.570 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0352999999999998e-05 2024-09-21T15:50:17.571 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.571 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.571 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:17.571 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.197779+0000", 2024-09-21T15:50:17.571 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4875999999999993e-05 2024-09-21T15:50:17.571 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.571 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:17.571 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.571 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.571 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.571 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:125395 3.1b 3:d87db7c4:::rbd_data.1133545d3fa6.0000000000000013:1e [read 608256~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:50:17.571 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:05.152015+0000", 2024-09-21T15:50:17.571 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.365732812999999, 2024-09-21T15:50:17.571 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045892923000000002, 2024-09-21T15:50:17.571 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:17.572 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:17.572 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:17.572 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:17.572 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:17.572 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 125395 2024-09-21T15:50:17.572 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.572 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:17.572 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.572 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:17.572 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.152015+0000", 2024-09-21T15:50:17.572 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.572 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.572 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.572 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:17.572 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.152015+0000", 2024-09-21T15:50:17.573 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.573 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.573 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.573 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:17.573 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.151793+0000", 2024-09-21T15:50:17.573 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9997778 2024-09-21T15:50:17.573 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.573 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.573 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:17.573 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.152018+0000", 2024-09-21T15:50:17.573 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00022526100000000001 2024-09-21T15:50:17.573 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.573 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.573 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:17.574 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.152026+0000", 2024-09-21T15:50:17.574 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1869999999999997e-06 2024-09-21T15:50:17.574 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.574 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.574 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.574 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.152032+0000", 2024-09-21T15:50:17.574 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3559999999999998e-06 2024-09-21T15:50:17.574 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.574 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.574 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:17.574 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.197802+0000", 2024-09-21T15:50:17.574 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045769481000000001 2024-09-21T15:50:17.574 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.575 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.575 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:17.575 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.197842+0000", 2024-09-21T15:50:17.575 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9836000000000003e-05 2024-09-21T15:50:17.575 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.575 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.575 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:17.575 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.197908+0000", 2024-09-21T15:50:17.575 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5938999999999993e-05 2024-09-21T15:50:17.575 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.575 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:17.575 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.575 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.576 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.576 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:125396 3.1b 3:d87db7c4:::rbd_data.1133545d3fa6.0000000000000013:1e [read 628736~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:50:17.576 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:05.152038+0000", 2024-09-21T15:50:17.576 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.365710046, 2024-09-21T15:50:17.576 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046005533000000001, 2024-09-21T15:50:17.576 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:17.576 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:17.576 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:17.576 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:17.576 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:17.576 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 125396 2024-09-21T15:50:17.576 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.576 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:17.576 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.577 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:17.577 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.152038+0000", 2024-09-21T15:50:17.577 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.577 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.577 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.577 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:17.577 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.152038+0000", 2024-09-21T15:50:17.577 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.577 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.577 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.577 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:17.577 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.152037+0000", 2024-09-21T15:50:17.577 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:17.578 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.578 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.578 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:17.578 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.152040+0000", 2024-09-21T15:50:17.578 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5689999999999999e-06 2024-09-21T15:50:17.578 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.578 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.578 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:17.578 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.152046+0000", 2024-09-21T15:50:17.578 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0449999999999997e-06 2024-09-21T15:50:17.578 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.578 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.578 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.578 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.152051+0000", 2024-09-21T15:50:17.579 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7529999999999998e-06 2024-09-21T15:50:17.579 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.579 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.579 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:17.579 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.197930+0000", 2024-09-21T15:50:17.579 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045878752000000002 2024-09-21T15:50:17.579 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.579 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.579 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:17.579 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.197970+0000", 2024-09-21T15:50:17.579 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9444999999999999e-05 2024-09-21T15:50:17.579 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.579 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.579 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:17.580 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.198043+0000", 2024-09-21T15:50:17.580 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3386999999999994e-05 2024-09-21T15:50:17.580 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.580 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:17.580 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.580 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.580 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.580 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:125405 3.1b 3:d87db7c4:::rbd_data.1133545d3fa6.0000000000000013:1e [read 825344~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:50:17.580 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:05.154106+0000", 2024-09-21T15:50:17.580 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.363641268, 2024-09-21T15:50:17.580 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045656167999999997, 2024-09-21T15:50:17.580 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:17.580 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:17.580 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:17.581 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:17.581 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:17.581 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 125405 2024-09-21T15:50:17.581 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.581 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:17.581 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.581 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:17.581 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154106+0000", 2024-09-21T15:50:17.581 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.581 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.581 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.581 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:17.581 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154106+0000", 2024-09-21T15:50:17.581 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.582 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.582 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.582 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:17.582 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154106+0000", 2024-09-21T15:50:17.582 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:50:17.582 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.582 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.582 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:17.582 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154107+0000", 2024-09-21T15:50:17.582 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2720000000000001e-06 2024-09-21T15:50:17.582 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.582 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.582 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:17.582 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154110+0000", 2024-09-21T15:50:17.582 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.7829999999999999e-06 2024-09-21T15:50:17.583 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.583 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.583 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.583 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154113+0000", 2024-09-21T15:50:17.583 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.6690000000000002e-06 2024-09-21T15:50:17.583 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.583 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.583 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:17.583 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.199540+0000", 2024-09-21T15:50:17.583 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045427109 2024-09-21T15:50:17.583 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.583 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.583 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:17.584 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.199631+0000", 2024-09-21T15:50:17.584 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1279000000000005e-05 2024-09-21T15:50:17.584 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.584 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.584 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:17.584 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.199762+0000", 2024-09-21T15:50:17.584 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013128400000000001 2024-09-21T15:50:17.584 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.584 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:17.584 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.584 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.584 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.584 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:125406 3.1b 3:d87db7c4:::rbd_data.1133545d3fa6.0000000000000013:1e [read 845824~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:50:17.584 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:05.154129+0000", 2024-09-21T15:50:17.584 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.363618924000001, 2024-09-21T15:50:17.585 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045864239000000001, 2024-09-21T15:50:17.585 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:17.585 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:17.585 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:17.585 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:17.585 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:17.585 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 125406 2024-09-21T15:50:17.585 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.585 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:17.585 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.585 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:17.585 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154129+0000", 2024-09-21T15:50:17.585 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.585 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.586 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.586 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:17.586 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154129+0000", 2024-09-21T15:50:17.586 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.586 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.586 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.586 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:17.586 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154128+0000", 2024-09-21T15:50:17.586 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:17.586 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.586 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.586 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:17.586 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154130+0000", 2024-09-21T15:50:17.587 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9e-06 2024-09-21T15:50:17.587 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.587 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.587 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:17.587 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154134+0000", 2024-09-21T15:50:17.587 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5810000000000001e-06 2024-09-21T15:50:17.587 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.587 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.587 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.587 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154136+0000", 2024-09-21T15:50:17.587 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5459999999999998e-06 2024-09-21T15:50:17.587 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.587 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.587 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:17.587 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.199793+0000", 2024-09-21T15:50:17.588 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045656799999999997 2024-09-21T15:50:17.588 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.588 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.588 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:17.588 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.199876+0000", 2024-09-21T15:50:17.588 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2730000000000002e-05 2024-09-21T15:50:17.588 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.588 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.588 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:17.588 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.199993+0000", 2024-09-21T15:50:17.588 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000117124 2024-09-21T15:50:17.588 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.588 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:17.589 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.589 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.589 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.589 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:125407 3.1b 3:d87db7c4:::rbd_data.1133545d3fa6.0000000000000013:1e [read 866304~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:50:17.589 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:05.154139+0000", 2024-09-21T15:50:17.589 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.363609087, 2024-09-21T15:50:17.589 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046086304000000002, 2024-09-21T15:50:17.589 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:17.589 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:17.589 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:17.589 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:17.589 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:17.589 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 125407 2024-09-21T15:50:17.589 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.589 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:17.590 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.590 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:17.590 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154139+0000", 2024-09-21T15:50:17.590 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.590 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.590 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.590 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:17.590 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154139+0000", 2024-09-21T15:50:17.590 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.590 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.590 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.590 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:17.590 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154138+0000", 2024-09-21T15:50:17.591 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:17.591 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.591 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.591 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:17.591 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154139+0000", 2024-09-21T15:50:17.591 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3120000000000001e-06 2024-09-21T15:50:17.591 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.591 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.591 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:17.591 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154142+0000", 2024-09-21T15:50:17.591 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.796e-06 2024-09-21T15:50:17.591 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.591 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.591 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.591 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154145+0000", 2024-09-21T15:50:17.592 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3800000000000001e-06 2024-09-21T15:50:17.592 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.592 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.592 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:17.592 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.200025+0000", 2024-09-21T15:50:17.592 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045880524999999998 2024-09-21T15:50:17.592 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.592 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.592 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:17.592 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.200116+0000", 2024-09-21T15:50:17.592 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0963000000000003e-05 2024-09-21T15:50:17.592 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.592 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.592 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:17.592 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.200225+0000", 2024-09-21T15:50:17.593 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000108646 2024-09-21T15:50:17.593 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.593 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:17.593 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.593 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.593 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.593 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:125408 3.1b 3:d87db7c4:::rbd_data.1133545d3fa6.0000000000000013:1e [read 886784~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:50:17.593 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:05.154148+0000", 2024-09-21T15:50:17.593 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.363600028, 2024-09-21T15:50:17.593 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046333535000000002, 2024-09-21T15:50:17.593 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:17.593 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:17.593 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:17.593 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:17.593 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:17.594 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 125408 2024-09-21T15:50:17.594 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.594 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:17.594 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.594 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:17.594 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154148+0000", 2024-09-21T15:50:17.594 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.594 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.594 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.594 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:17.594 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154148+0000", 2024-09-21T15:50:17.595 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.595 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.595 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.595 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:17.595 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154147+0000", 2024-09-21T15:50:17.595 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:50:17.595 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.595 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.595 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:17.595 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154149+0000", 2024-09-21T15:50:17.595 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.19e-06 2024-09-21T15:50:17.595 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.595 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.596 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:17.596 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154151+0000", 2024-09-21T15:50:17.596 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8059999999999999e-06 2024-09-21T15:50:17.596 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.596 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.596 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.596 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154154+0000", 2024-09-21T15:50:17.596 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.909e-06 2024-09-21T15:50:17.596 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.596 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.596 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:17.596 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.200270+0000", 2024-09-21T15:50:17.596 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046115505000000001 2024-09-21T15:50:17.596 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.596 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.597 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:17.597 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.200361+0000", 2024-09-21T15:50:17.597 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1457000000000002e-05 2024-09-21T15:50:17.597 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.597 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.597 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:17.597 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.200481+0000", 2024-09-21T15:50:17.597 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000119902 2024-09-21T15:50:17.597 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.597 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:17.597 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.597 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.597 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.597 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:125409 3.1b 3:d87db7c4:::rbd_data.1133545d3fa6.0000000000000013:1e [read 907264~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:50:17.597 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:05.154156+0000", 2024-09-21T15:50:17.598 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.363591317999999, 2024-09-21T15:50:17.598 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046547416000000001, 2024-09-21T15:50:17.598 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:17.598 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:17.598 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:17.598 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:17.598 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:17.598 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 125409 2024-09-21T15:50:17.598 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.598 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:17.598 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.598 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:17.598 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154156+0000", 2024-09-21T15:50:17.598 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.599 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.599 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.599 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:17.599 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154156+0000", 2024-09-21T15:50:17.599 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.599 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.599 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.599 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:17.599 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154156+0000", 2024-09-21T15:50:17.599 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:50:17.599 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.599 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.599 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:17.600 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154157+0000", 2024-09-21T15:50:17.600 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1680000000000001e-06 2024-09-21T15:50:17.600 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.600 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.600 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:17.600 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154160+0000", 2024-09-21T15:50:17.600 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.802e-06 2024-09-21T15:50:17.600 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.600 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.600 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.600 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.154163+0000", 2024-09-21T15:50:17.600 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.509e-06 2024-09-21T15:50:17.600 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.600 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.601 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:17.601 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.200520+0000", 2024-09-21T15:50:17.601 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.046357281 2024-09-21T15:50:17.601 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.601 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.601 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:17.601 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.200597+0000", 2024-09-21T15:50:17.601 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7205000000000004e-05 2024-09-21T15:50:17.601 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.601 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.601 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:17.601 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.200704+0000", 2024-09-21T15:50:17.601 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010668 2024-09-21T15:50:17.601 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.602 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:17.602 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.602 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.602 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.602 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:125416 3.1b 3:d87db7c4:::rbd_data.1133545d3fa6.0000000000000013:1e [read 1046528~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e72)", 2024-09-21T15:50:17.602 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:05.156074+0000", 2024-09-21T15:50:17.602 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 12.361673940999999, 2024-09-21T15:50:17.602 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045646541999999998, 2024-09-21T15:50:17.602 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:17.602 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:17.602 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:17.602 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:17.603 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:17.603 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 125416 2024-09-21T15:50:17.603 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.603 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:17.603 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.603 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:17.603 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.156074+0000", 2024-09-21T15:50:17.603 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.603 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.603 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.603 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:17.603 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.156074+0000", 2024-09-21T15:50:17.603 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.603 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.603 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.604 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:17.604 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.156073+0000", 2024-09-21T15:50:17.604 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967296 2024-09-21T15:50:17.604 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.604 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.604 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:17.604 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.156075+0000", 2024-09-21T15:50:17.604 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2699999999999999e-06 2024-09-21T15:50:17.604 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.604 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.604 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:17.604 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.156078+0000", 2024-09-21T15:50:17.604 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8389999999999998e-06 2024-09-21T15:50:17.604 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.605 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.605 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.605 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.156081+0000", 2024-09-21T15:50:17.605 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4640000000000002e-06 2024-09-21T15:50:17.605 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.605 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.605 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:17.605 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.201622+0000", 2024-09-21T15:50:17.605 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.045541352 2024-09-21T15:50:17.605 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.605 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.605 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:17.605 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.201664+0000", 2024-09-21T15:50:17.605 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1848000000000003e-05 2024-09-21T15:50:17.605 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.606 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.606 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:17.606 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:05.201720+0000", 2024-09-21T15:50:17.606 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5983999999999998e-05 2024-09-21T15:50:17.606 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.606 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:17.606 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.606 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.606 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:17.606 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:17.608 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:50:17.609 INFO:teuthology.orchestra.run.smithi089.stdout:5464 writesame 0x4ba5d08 thru 0x4baca62 data_size 0xb(0x6d5b bytes) 2024-09-21T15:50:17.609 INFO:teuthology.orchestra.run.smithi089.stdout:5465 writesame 0xbbfa44 thru 0xbc2079 data_size 0x43(0x2636 bytes) 2024-09-21T15:50:17.609 INFO:teuthology.orchestra.run.smithi089.stdout:5466 read 0xa31268 thru 0xa35c7a (0x4a13 bytes) 2024-09-21T15:50:17.609 INFO:teuthology.orchestra.run.smithi089.stdout:5467 read 0x7c91128 thru 0x7c97ee5 (0x6dbe bytes) 2024-09-21T15:50:17.610 INFO:teuthology.orchestra.run.smithi089.stdout:5468 trunc from 0xc088e32 to 0x7ac2404 2024-09-21T15:50:17.610 INFO:teuthology.orchestra.run.smithi089.stdout:5469 writesame 0x497da91 thru 0x498710b data_size 0x1(0x967b bytes) 2024-09-21T15:50:17.610 INFO:teuthology.orchestra.run.smithi089.stdout:5471 punch from 0x8bfdd2 to 0x8cb9dc, (0xbc0a bytes) 2024-09-21T15:50:17.610 INFO:teuthology.orchestra.run.smithi089.stdout:5473 read 0x73e4754 thru 0x73e9971 (0x521e bytes) 2024-09-21T15:50:17.610 INFO:teuthology.orchestra.run.smithi089.stdout:5474 write 0x8aced10 thru 0x8ad62a0 (0x7591 bytes) 2024-09-21T15:50:17.610 INFO:teuthology.orchestra.run.smithi089.stdout:5475 write 0x4f3f227 thru 0x4f41de7 (0x2bc1 bytes) 2024-09-21T15:50:17.610 INFO:teuthology.orchestra.run.smithi089.stdout:5477 read 0x6b9cb39 thru 0x6ba3a70 (0x6f38 bytes) 2024-09-21T15:50:17.610 INFO:teuthology.orchestra.run.smithi089.stdout:5478 write 0x281da8a thru 0x28252c0 (0x7837 bytes) 2024-09-21T15:50:17.610 INFO:teuthology.orchestra.run.smithi089.stdout:5479 trunc from 0x8ad62a1 to 0xdd650c2 2024-09-21T15:50:17.610 INFO:teuthology.orchestra.run.smithi089.stdout:5480 write 0x84eebb8 thru 0x84fe10d (0xf556 bytes) 2024-09-21T15:50:17.610 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:17.606+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:17.611 INFO:teuthology.orchestra.run.smithi089.stdout:5481 read 0x79cbad0 thru 0x79ccb8d (0x10be bytes) 2024-09-21T15:50:17.611 INFO:teuthology.orchestra.run.smithi089.stdout:5482 read 0x203e314 thru 0x204c07a (0xdd67 bytes) 2024-09-21T15:50:17.611 INFO:teuthology.orchestra.run.smithi089.stdout:5484 read 0x6078021 thru 0x6084c4e (0xcc2e bytes) 2024-09-21T15:50:17.611 INFO:teuthology.orchestra.run.smithi089.stdout:5485 read 0x61a446f thru 0x61a6873 (0x2405 bytes) 2024-09-21T15:50:17.612 INFO:teuthology.orchestra.run.smithi089.stdout:5487 punch from 0xc1d15fe to 0xc1e0c41, (0xf643 bytes) 2024-09-21T15:50:17.612 INFO:teuthology.orchestra.run.smithi089.stdout:5488 read 0xcd3986a thru 0xcd3ea53 (0x51ea bytes) 2024-09-21T15:50:17.620 INFO:teuthology.orchestra.run.smithi089.stdout:5489 trunc from 0xdd650c2 to 0x55d312f 2024-09-21T15:50:17.636 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:17.652 INFO:teuthology.orchestra.run.smithi089.stdout:5492 write 0xb5c94f2 thru 0xb5cdd91 (0x48a0 bytes) 2024-09-21T15:50:17.656 INFO:teuthology.orchestra.run.smithi089.stdout:5494 read 0x4eac0f8 thru 0x4eb1609 (0x5512 bytes) 2024-09-21T15:50:17.656 INFO:teuthology.orchestra.run.smithi089.stdout:5495 read 0x32f9a10 thru 0x32fecce (0x52bf bytes) 2024-09-21T15:50:17.656 INFO:teuthology.orchestra.run.smithi089.stdout:5496 punch from 0x367b00f to 0x367eb2c, (0x3b1d bytes) 2024-09-21T15:50:17.656 INFO:teuthology.orchestra.run.smithi089.stdout:5497 clone 38 order 23 su 65536 sc 11 2024-09-21T15:50:17.708 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:17.707+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:17.715 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:17.716 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:17.716 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:50:17.716 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:17.724 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:50:17.809 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:17.809+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:17.825 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:17.826 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:17.826 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:50:17.826 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:50:17.826 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:17.834 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:50:17.910 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:17.909+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:17.938 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:17.938 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:50:17.938 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:50:17.938 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:50:17.938 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.938 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120111 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 321536~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:17.938 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572216+0000", 2024-09-21T15:50:17.938 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 14.363982044, 2024-09-21T15:50:17.938 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054657399000000002, 2024-09-21T15:50:17.939 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:17.939 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:17.939 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:17.939 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:17.939 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:17.939 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120111 2024-09-21T15:50:17.939 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.939 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:17.939 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.939 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:17.939 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572216+0000", 2024-09-21T15:50:17.939 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.939 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.939 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.939 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:17.940 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572216+0000", 2024-09-21T15:50:17.940 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.940 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.940 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.940 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:17.940 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572216+0000", 2024-09-21T15:50:17.940 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:17.940 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.940 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.940 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:17.940 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572218+0000", 2024-09-21T15:50:17.940 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1840000000000002e-06 2024-09-21T15:50:17.940 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.940 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.941 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:17.941 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572224+0000", 2024-09-21T15:50:17.941 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7880000000000003e-06 2024-09-21T15:50:17.941 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.941 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.941 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.941 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572229+0000", 2024-09-21T15:50:17.941 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8239999999999999e-06 2024-09-21T15:50:17.941 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.941 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.941 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:17.941 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626680+0000", 2024-09-21T15:50:17.941 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054451431000000002 2024-09-21T15:50:17.941 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.941 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.941 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:17.942 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626755+0000", 2024-09-21T15:50:17.942 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4641000000000006e-05 2024-09-21T15:50:17.942 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.942 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.942 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:17.942 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626874+0000", 2024-09-21T15:50:17.942 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00011886 2024-09-21T15:50:17.942 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.942 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:17.942 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.942 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.942 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.942 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120112 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 342016~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:17.942 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572233+0000", 2024-09-21T15:50:17.943 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 14.363965675999999, 2024-09-21T15:50:17.943 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054823997999999999, 2024-09-21T15:50:17.943 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:17.943 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:17.943 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:17.943 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:17.943 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:17.943 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120112 2024-09-21T15:50:17.943 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.943 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:17.943 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.943 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:17.943 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572233+0000", 2024-09-21T15:50:17.943 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.944 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.944 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.944 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:17.944 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572233+0000", 2024-09-21T15:50:17.944 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.944 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.944 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.944 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:17.944 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572232+0000", 2024-09-21T15:50:17.944 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:17.944 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.944 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.944 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:17.944 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572234+0000", 2024-09-21T15:50:17.944 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0660000000000002e-06 2024-09-21T15:50:17.945 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.945 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.945 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:17.945 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572242+0000", 2024-09-21T15:50:17.945 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0430000000000002e-06 2024-09-21T15:50:17.945 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.945 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.945 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.945 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572246+0000", 2024-09-21T15:50:17.945 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9080000000000003e-06 2024-09-21T15:50:17.945 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.945 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.945 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:17.946 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626916+0000", 2024-09-21T15:50:17.946 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054669482999999998 2024-09-21T15:50:17.946 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.946 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.946 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:17.946 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626977+0000", 2024-09-21T15:50:17.946 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0597000000000003e-05 2024-09-21T15:50:17.946 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.946 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.946 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:17.946 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627057+0000", 2024-09-21T15:50:17.946 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0258000000000003e-05 2024-09-21T15:50:17.946 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.946 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:17.946 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.947 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.947 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.947 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120113 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 362496~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:17.947 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572253+0000", 2024-09-21T15:50:17.947 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 14.363945691, 2024-09-21T15:50:17.947 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.05503044, 2024-09-21T15:50:17.947 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:17.947 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:17.947 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:17.947 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:17.947 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:17.947 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120113 2024-09-21T15:50:17.947 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.947 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:17.948 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.948 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:17.948 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572253+0000", 2024-09-21T15:50:17.948 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.948 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.948 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.948 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:17.948 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572253+0000", 2024-09-21T15:50:17.948 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.948 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.948 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.948 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:17.948 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572252+0000", 2024-09-21T15:50:17.948 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:17.948 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.949 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.949 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:17.949 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572258+0000", 2024-09-21T15:50:17.949 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6169999999999998e-06 2024-09-21T15:50:17.949 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.949 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.949 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:17.949 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572266+0000", 2024-09-21T15:50:17.949 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9410000000000007e-06 2024-09-21T15:50:17.949 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.949 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.949 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.949 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572272+0000", 2024-09-21T15:50:17.949 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.241e-06 2024-09-21T15:50:17.949 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.949 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.950 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:17.950 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627087+0000", 2024-09-21T15:50:17.950 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054814850999999998 2024-09-21T15:50:17.950 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.950 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.950 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:17.950 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627162+0000", 2024-09-21T15:50:17.950 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5012999999999995e-05 2024-09-21T15:50:17.950 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.950 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.950 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:17.950 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627283+0000", 2024-09-21T15:50:17.950 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012113 2024-09-21T15:50:17.950 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.950 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:17.950 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.951 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.951 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.951 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120114 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 382976~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:17.951 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572277+0000", 2024-09-21T15:50:17.951 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 14.363921673, 2024-09-21T15:50:17.951 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055184395999999997, 2024-09-21T15:50:17.951 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:17.951 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:17.951 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:17.951 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:17.951 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:17.951 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120114 2024-09-21T15:50:17.951 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.951 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:17.951 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.952 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:17.952 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572277+0000", 2024-09-21T15:50:17.952 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.952 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.952 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.952 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:17.952 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572277+0000", 2024-09-21T15:50:17.952 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.952 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.952 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.952 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:17.952 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572276+0000", 2024-09-21T15:50:17.952 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:17.952 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.952 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.953 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:17.953 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572279+0000", 2024-09-21T15:50:17.953 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.373e-06 2024-09-21T15:50:17.953 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.953 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.953 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:17.953 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572285+0000", 2024-09-21T15:50:17.953 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9359999999999996e-06 2024-09-21T15:50:17.953 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.953 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.953 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.953 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572289+0000", 2024-09-21T15:50:17.953 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.634e-06 2024-09-21T15:50:17.953 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.953 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.953 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:17.953 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627311+0000", 2024-09-21T15:50:17.954 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055021211 2024-09-21T15:50:17.954 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.954 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.954 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:17.954 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627360+0000", 2024-09-21T15:50:17.954 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9203999999999998e-05 2024-09-21T15:50:17.954 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.954 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.954 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:17.954 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627461+0000", 2024-09-21T15:50:17.954 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000101469 2024-09-21T15:50:17.954 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.954 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:17.954 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.955 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.955 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.955 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120115 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 399360~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:17.955 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572294+0000", 2024-09-21T15:50:17.955 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 14.363904366, 2024-09-21T15:50:17.955 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055400092999999997, 2024-09-21T15:50:17.955 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:17.955 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:17.955 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:17.955 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:17.955 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:17.955 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120115 2024-09-21T15:50:17.955 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.955 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:17.956 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.956 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:17.956 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572294+0000", 2024-09-21T15:50:17.956 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.956 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.956 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.956 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:17.956 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572294+0000", 2024-09-21T15:50:17.956 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.956 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.956 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.956 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:17.956 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572294+0000", 2024-09-21T15:50:17.956 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:17.957 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.957 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.957 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:17.957 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572296+0000", 2024-09-21T15:50:17.957 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1100000000000001e-06 2024-09-21T15:50:17.957 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.957 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.957 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:17.957 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572303+0000", 2024-09-21T15:50:17.957 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8580000000000002e-06 2024-09-21T15:50:17.957 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.957 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.957 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.957 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572314+0000", 2024-09-21T15:50:17.958 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.101e-05 2024-09-21T15:50:17.958 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.958 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.958 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:17.958 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627496+0000", 2024-09-21T15:50:17.958 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055182522999999997 2024-09-21T15:50:17.958 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.958 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.958 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:17.958 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627568+0000", 2024-09-21T15:50:17.958 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1569999999999994e-05 2024-09-21T15:50:17.958 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.958 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.958 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:17.958 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627694+0000", 2024-09-21T15:50:17.959 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000126372 2024-09-21T15:50:17.959 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.959 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:17.959 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.959 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.959 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.959 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120116 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 419840~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:17.959 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572318+0000", 2024-09-21T15:50:17.959 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 14.363880175, 2024-09-21T15:50:17.959 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055539416000000001, 2024-09-21T15:50:17.959 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:17.959 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:17.959 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:17.959 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:17.960 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:17.960 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120116 2024-09-21T15:50:17.960 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.960 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:17.960 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.960 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:17.960 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572318+0000", 2024-09-21T15:50:17.960 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.960 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.960 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.960 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:17.960 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572318+0000", 2024-09-21T15:50:17.960 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.960 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.961 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.961 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:17.961 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572318+0000", 2024-09-21T15:50:17.961 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:17.961 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.961 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.961 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:17.961 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572322+0000", 2024-09-21T15:50:17.961 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4839999999999998e-06 2024-09-21T15:50:17.961 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.961 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.961 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:17.961 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572329+0000", 2024-09-21T15:50:17.961 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0959999999999997e-06 2024-09-21T15:50:17.961 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.962 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.962 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.962 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572333+0000", 2024-09-21T15:50:17.962 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9359999999999998e-06 2024-09-21T15:50:17.962 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.962 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.962 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:17.962 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627726+0000", 2024-09-21T15:50:17.962 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055392829999999997 2024-09-21T15:50:17.962 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.962 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.962 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:17.962 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627779+0000", 2024-09-21T15:50:17.962 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2185999999999998e-05 2024-09-21T15:50:17.963 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.963 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.963 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:17.963 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627858+0000", 2024-09-21T15:50:17.963 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9190999999999994e-05 2024-09-21T15:50:17.963 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.963 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:17.963 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.963 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.963 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.963 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120117 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 440320~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:17.963 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572338+0000", 2024-09-21T15:50:17.963 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 14.363860861999999, 2024-09-21T15:50:17.963 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055680888999999997, 2024-09-21T15:50:17.964 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:17.964 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:17.964 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:17.964 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:17.964 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:17.964 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120117 2024-09-21T15:50:17.964 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.964 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:17.964 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.964 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:17.964 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572338+0000", 2024-09-21T15:50:17.964 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.964 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.964 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.965 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:17.965 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572338+0000", 2024-09-21T15:50:17.965 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.965 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.965 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.965 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:17.965 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572337+0000", 2024-09-21T15:50:17.965 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:17.965 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.965 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.965 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:17.965 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572339+0000", 2024-09-21T15:50:17.965 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9379999999999999e-06 2024-09-21T15:50:17.965 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.966 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.966 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:17.966 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572349+0000", 2024-09-21T15:50:17.966 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7780000000000002e-06 2024-09-21T15:50:17.966 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.966 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.966 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.966 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572354+0000", 2024-09-21T15:50:17.966 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8740000000000003e-06 2024-09-21T15:50:17.966 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.966 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.966 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:17.966 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627885+0000", 2024-09-21T15:50:17.966 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055531448999999997 2024-09-21T15:50:17.966 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.967 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.967 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:17.967 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627935+0000", 2024-09-21T15:50:17.967 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9348999999999998e-05 2024-09-21T15:50:17.967 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.967 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.967 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:17.967 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628018+0000", 2024-09-21T15:50:17.967 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3851999999999996e-05 2024-09-21T15:50:17.967 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.967 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:17.967 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.967 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.967 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.967 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120118 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 460800~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:17.968 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572359+0000", 2024-09-21T15:50:17.968 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 14.36383962, 2024-09-21T15:50:17.968 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055951714, 2024-09-21T15:50:17.968 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:17.968 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:17.968 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:17.968 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:17.968 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:17.968 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120118 2024-09-21T15:50:17.968 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.968 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:17.968 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.968 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:17.968 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572359+0000", 2024-09-21T15:50:17.968 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.969 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.969 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.969 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:17.969 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572359+0000", 2024-09-21T15:50:17.969 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.969 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.969 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.969 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:17.969 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572358+0000", 2024-09-21T15:50:17.969 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:17.969 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.969 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.969 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:17.969 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572361+0000", 2024-09-21T15:50:17.969 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.26e-06 2024-09-21T15:50:17.970 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.970 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.970 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:17.970 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572367+0000", 2024-09-21T15:50:17.970 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3640000000000004e-06 2024-09-21T15:50:17.970 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.970 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.970 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.970 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572376+0000", 2024-09-21T15:50:17.970 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2310000000000002e-06 2024-09-21T15:50:17.970 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.970 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.970 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:17.970 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628096+0000", 2024-09-21T15:50:17.971 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055719369999999997 2024-09-21T15:50:17.971 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.971 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.971 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:17.971 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628178+0000", 2024-09-21T15:50:17.971 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2667e-05 2024-09-21T15:50:17.971 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.971 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.971 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:17.971 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628311+0000", 2024-09-21T15:50:17.971 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000132175 2024-09-21T15:50:17.971 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.971 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:17.971 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.971 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.971 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.972 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120119 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 481280~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:17.972 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572381+0000", 2024-09-21T15:50:17.972 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 14.363817176, 2024-09-21T15:50:17.972 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056088791999999998, 2024-09-21T15:50:17.972 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:17.972 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:17.972 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:17.972 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:17.972 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:17.972 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120119 2024-09-21T15:50:17.972 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.972 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:17.972 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.972 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:17.973 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572381+0000", 2024-09-21T15:50:17.973 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.973 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.973 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.973 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:17.973 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572381+0000", 2024-09-21T15:50:17.973 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.973 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.973 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.973 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:17.973 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572381+0000", 2024-09-21T15:50:17.973 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:17.973 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.973 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.973 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:17.973 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572383+0000", 2024-09-21T15:50:17.974 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0619999999999999e-06 2024-09-21T15:50:17.974 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.974 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.974 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:17.974 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572391+0000", 2024-09-21T15:50:17.974 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8769999999999993e-06 2024-09-21T15:50:17.974 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.974 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.974 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.974 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572399+0000", 2024-09-21T15:50:17.974 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6240000000000008e-06 2024-09-21T15:50:17.974 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.974 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.974 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:17.974 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628344+0000", 2024-09-21T15:50:17.975 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055944828000000002 2024-09-21T15:50:17.975 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.975 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.975 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:17.975 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628393+0000", 2024-09-21T15:50:17.975 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8653000000000003e-05 2024-09-21T15:50:17.975 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.975 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.975 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:17.975 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628470+0000", 2024-09-21T15:50:17.975 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7121999999999998e-05 2024-09-21T15:50:17.975 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.975 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:17.975 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.975 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.976 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.976 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120120 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 501760~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:17.976 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572418+0000", 2024-09-21T15:50:17.976 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 14.363780677999999, 2024-09-21T15:50:17.976 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056209138999999998, 2024-09-21T15:50:17.976 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:17.976 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:17.976 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:17.976 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:17.976 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:17.976 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120120 2024-09-21T15:50:17.976 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.976 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:17.976 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.977 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:17.977 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572418+0000", 2024-09-21T15:50:17.977 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.977 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.977 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.977 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:17.977 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572418+0000", 2024-09-21T15:50:17.977 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.977 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.977 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.977 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:17.977 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572417+0000", 2024-09-21T15:50:17.977 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:17.977 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.977 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.978 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:17.978 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572420+0000", 2024-09-21T15:50:17.978 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5270000000000001e-06 2024-09-21T15:50:17.978 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.978 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.978 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:17.978 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572426+0000", 2024-09-21T15:50:17.978 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3380000000000001e-06 2024-09-21T15:50:17.978 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.978 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.978 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.978 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572431+0000", 2024-09-21T15:50:17.978 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.865e-06 2024-09-21T15:50:17.979 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.979 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.979 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:17.979 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628500+0000", 2024-09-21T15:50:17.979 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056068917000000003 2024-09-21T15:50:17.979 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.979 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.979 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:17.979 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628542+0000", 2024-09-21T15:50:17.979 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1761999999999998e-05 2024-09-21T15:50:17.979 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.979 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.979 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:17.979 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628627+0000", 2024-09-21T15:50:17.979 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5247999999999994e-05 2024-09-21T15:50:17.980 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.980 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:17.980 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.980 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.980 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.980 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120121 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 522240~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:17.980 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572630+0000", 2024-09-21T15:50:17.980 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 14.363568292, 2024-09-21T15:50:17.980 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056207209000000001, 2024-09-21T15:50:17.980 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:17.980 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:17.980 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:17.980 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:17.980 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:17.980 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120121 2024-09-21T15:50:17.980 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.981 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:17.981 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.981 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:17.981 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572630+0000", 2024-09-21T15:50:17.981 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.981 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.981 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.981 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:17.981 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572630+0000", 2024-09-21T15:50:17.981 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.981 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.981 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.981 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:17.981 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572435+0000", 2024-09-21T15:50:17.981 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998045 2024-09-21T15:50:17.981 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.982 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.982 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:17.982 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572633+0000", 2024-09-21T15:50:17.982 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000198786 2024-09-21T15:50:17.982 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.982 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.982 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:17.982 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572641+0000", 2024-09-21T15:50:17.982 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9449999999999993e-06 2024-09-21T15:50:17.982 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.982 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.982 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.982 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572648+0000", 2024-09-21T15:50:17.982 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1929999999999998e-06 2024-09-21T15:50:17.982 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.983 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.983 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:17.983 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628669+0000", 2024-09-21T15:50:17.983 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056021690999999998 2024-09-21T15:50:17.983 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.983 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.983 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:17.983 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628751+0000", 2024-09-21T15:50:17.983 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1710000000000002e-05 2024-09-21T15:50:17.983 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.983 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.983 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:17.983 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628837+0000", 2024-09-21T15:50:17.983 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6383000000000003e-05 2024-09-21T15:50:17.983 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.984 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:17.984 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.984 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.984 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.984 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120122 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 538624~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:17.984 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.573471+0000", 2024-09-21T15:50:17.984 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 14.362727047, 2024-09-21T15:50:17.984 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055563935000000002, 2024-09-21T15:50:17.984 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:17.984 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:17.984 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:17.984 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:17.984 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:17.984 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120122 2024-09-21T15:50:17.984 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.984 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:17.985 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.985 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:17.985 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573471+0000", 2024-09-21T15:50:17.985 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.985 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.985 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.985 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:17.985 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573471+0000", 2024-09-21T15:50:17.985 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.985 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.985 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.985 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:17.985 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572653+0000", 2024-09-21T15:50:17.985 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9991813 2024-09-21T15:50:17.985 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.985 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.986 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:17.986 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573475+0000", 2024-09-21T15:50:17.986 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00082196500000000002 2024-09-21T15:50:17.986 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.986 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.986 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:17.986 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573483+0000", 2024-09-21T15:50:17.986 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5939999999999994e-06 2024-09-21T15:50:17.986 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.986 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.986 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.986 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573489+0000", 2024-09-21T15:50:17.986 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1689999999999997e-06 2024-09-21T15:50:17.986 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.986 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.987 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:17.987 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628882+0000", 2024-09-21T15:50:17.987 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055392230000000001 2024-09-21T15:50:17.987 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.987 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.987 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:17.987 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628934+0000", 2024-09-21T15:50:17.987 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1870000000000003e-05 2024-09-21T15:50:17.987 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.987 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.987 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:17.987 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629035+0000", 2024-09-21T15:50:17.987 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000101753 2024-09-21T15:50:17.987 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.987 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:17.987 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.987 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.988 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.988 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120123 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 559104~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:17.988 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.573495+0000", 2024-09-21T15:50:17.988 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 14.362703348, 2024-09-21T15:50:17.988 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055704765000000003, 2024-09-21T15:50:17.988 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:17.988 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:17.988 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:17.988 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:17.988 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:17.988 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120123 2024-09-21T15:50:17.988 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.988 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:17.988 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.988 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:17.988 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573495+0000", 2024-09-21T15:50:17.989 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.989 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.989 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.989 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:17.989 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573495+0000", 2024-09-21T15:50:17.989 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.989 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.989 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.989 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:17.989 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573495+0000", 2024-09-21T15:50:17.989 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:17.989 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.989 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.989 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:17.989 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573497+0000", 2024-09-21T15:50:17.989 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3420000000000002e-06 2024-09-21T15:50:17.989 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.990 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.990 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:17.990 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573503+0000", 2024-09-21T15:50:17.990 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7030000000000003e-06 2024-09-21T15:50:17.990 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.990 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.990 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.990 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573511+0000", 2024-09-21T15:50:17.990 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0790000000000001e-06 2024-09-21T15:50:17.990 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.990 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.990 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:17.990 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629067+0000", 2024-09-21T15:50:17.990 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055556573999999997 2024-09-21T15:50:17.990 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.991 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.991 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:17.991 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629118+0000", 2024-09-21T15:50:17.991 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1053000000000001e-05 2024-09-21T15:50:17.991 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.991 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.991 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:17.991 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629200+0000", 2024-09-21T15:50:17.991 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1452999999999994e-05 2024-09-21T15:50:17.991 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.991 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:17.991 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.991 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.991 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.991 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120124 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 579584~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:17.991 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.574014+0000", 2024-09-21T15:50:17.992 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 14.362184309, 2024-09-21T15:50:17.992 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055440284999999999, 2024-09-21T15:50:17.992 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:17.992 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:17.992 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:17.992 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:17.992 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:17.992 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120124 2024-09-21T15:50:17.992 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.992 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:17.992 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.992 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:17.992 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574014+0000", 2024-09-21T15:50:17.992 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.992 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.992 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.992 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:17.993 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574014+0000", 2024-09-21T15:50:17.993 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.993 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.993 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.993 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:17.993 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573515+0000", 2024-09-21T15:50:17.993 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9995008 2024-09-21T15:50:17.993 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.993 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.993 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:17.993 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574017+0000", 2024-09-21T15:50:17.993 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00050241100000000003 2024-09-21T15:50:17.993 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.993 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.993 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:17.993 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574027+0000", 2024-09-21T15:50:17.994 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4700000000000008e-06 2024-09-21T15:50:17.994 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.994 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.994 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.994 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574033+0000", 2024-09-21T15:50:17.994 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1489999999999999e-06 2024-09-21T15:50:17.994 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.994 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.994 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:17.994 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629238+0000", 2024-09-21T15:50:17.994 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055204880999999997 2024-09-21T15:50:17.994 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.994 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.994 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:17.995 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629317+0000", 2024-09-21T15:50:17.995 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8942000000000004e-05 2024-09-21T15:50:17.995 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.995 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.995 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:17.995 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629454+0000", 2024-09-21T15:50:17.995 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013767099999999999 2024-09-21T15:50:17.995 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.995 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:17.995 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.995 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.995 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.996 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120125 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 600064~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:17.996 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.574482+0000", 2024-09-21T15:50:17.996 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 14.36171601, 2024-09-21T15:50:17.996 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055243608, 2024-09-21T15:50:17.996 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:17.996 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:17.996 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:17.996 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:17.996 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:17.996 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120125 2024-09-21T15:50:17.996 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.996 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:17.996 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.996 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:17.997 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574482+0000", 2024-09-21T15:50:17.997 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.997 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.997 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.997 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:17.997 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574482+0000", 2024-09-21T15:50:17.997 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:17.997 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.997 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.997 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:17.997 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574038+0000", 2024-09-21T15:50:17.997 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9995556 2024-09-21T15:50:17.997 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.997 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.997 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:17.997 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574497+0000", 2024-09-21T15:50:17.998 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00045854400000000002 2024-09-21T15:50:17.998 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.998 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.998 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:17.998 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574509+0000", 2024-09-21T15:50:17.998 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2486e-05 2024-09-21T15:50:17.998 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.998 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.998 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:17.998 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574516+0000", 2024-09-21T15:50:17.998 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2269999999999996e-06 2024-09-21T15:50:17.998 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.998 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.998 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:17.998 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629497+0000", 2024-09-21T15:50:17.999 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054980785999999997 2024-09-21T15:50:17.999 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.999 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.999 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:17.999 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629580+0000", 2024-09-21T15:50:17.999 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2528000000000007e-05 2024-09-21T15:50:17.999 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.999 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:17.999 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:17.999 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629726+0000", 2024-09-21T15:50:17.999 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014630399999999999 2024-09-21T15:50:17.999 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.999 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:17.999 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:17.999 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:17.999 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.000 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120126 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 620544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:18.000 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.575315+0000", 2024-09-21T15:50:18.000 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 14.36088298, 2024-09-21T15:50:18.000 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054709559999999997, 2024-09-21T15:50:18.000 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:18.000 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:18.000 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:18.000 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:18.000 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:18.000 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120126 2024-09-21T15:50:18.000 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.000 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:18.000 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.000 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:18.000 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575315+0000", 2024-09-21T15:50:18.000 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:18.001 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.001 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.001 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:18.001 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575315+0000", 2024-09-21T15:50:18.001 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:18.001 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.001 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.001 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:18.001 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574522+0000", 2024-09-21T15:50:18.001 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9992065 2024-09-21T15:50:18.001 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.001 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.001 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:18.001 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575319+0000", 2024-09-21T15:50:18.001 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00079626800000000004 2024-09-21T15:50:18.001 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.001 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.002 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:18.002 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575329+0000", 2024-09-21T15:50:18.002 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.028e-05 2024-09-21T15:50:18.002 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.002 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.002 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:18.002 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575335+0000", 2024-09-21T15:50:18.002 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9290000000000003e-06 2024-09-21T15:50:18.002 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.002 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.002 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:18.002 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629806+0000", 2024-09-21T15:50:18.002 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054471558000000003 2024-09-21T15:50:18.002 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.002 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.003 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:18.003 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629885+0000", 2024-09-21T15:50:18.003 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8563e-05 2024-09-21T15:50:18.003 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.003 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.003 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:18.003 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630025+0000", 2024-09-21T15:50:18.003 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014019 2024-09-21T15:50:18.003 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:18.003 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:18.003 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:18.003 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.003 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.003 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120127 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 641024~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:18.004 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.575340+0000", 2024-09-21T15:50:18.004 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 14.360858273, 2024-09-21T15:50:18.004 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054895589000000002, 2024-09-21T15:50:18.004 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:18.004 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:18.004 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:18.004 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:18.004 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:18.004 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120127 2024-09-21T15:50:18.004 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.004 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:18.004 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.004 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:18.004 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575340+0000", 2024-09-21T15:50:18.005 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:18.005 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.005 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.005 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:18.005 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575340+0000", 2024-09-21T15:50:18.005 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:18.005 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.005 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.005 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:18.005 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575340+0000", 2024-09-21T15:50:18.005 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:18.005 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.005 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.005 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:18.006 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575342+0000", 2024-09-21T15:50:18.006 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5600000000000001e-06 2024-09-21T15:50:18.006 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.006 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.006 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:18.006 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575351+0000", 2024-09-21T15:50:18.006 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2349999999999999e-06 2024-09-21T15:50:18.006 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.006 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.006 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:18.006 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575356+0000", 2024-09-21T15:50:18.006 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4600000000000002e-06 2024-09-21T15:50:18.006 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.006 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.006 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:18.007 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630053+0000", 2024-09-21T15:50:18.007 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054697190999999999 2024-09-21T15:50:18.007 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.007 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.007 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:18.007 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630128+0000", 2024-09-21T15:50:18.007 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4777000000000006e-05 2024-09-21T15:50:18.007 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.007 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.007 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:18.007 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630236+0000", 2024-09-21T15:50:18.007 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000107837 2024-09-21T15:50:18.007 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:18.008 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:18.008 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:18.008 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.008 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.008 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120128 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 657408~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:18.008 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.575461+0000", 2024-09-21T15:50:18.008 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 14.360737694000001, 2024-09-21T15:50:18.008 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054945121999999999, 2024-09-21T15:50:18.008 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:18.008 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:18.008 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:18.008 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:18.008 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:18.008 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120128 2024-09-21T15:50:18.008 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.009 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:18.009 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.009 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:18.009 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575461+0000", 2024-09-21T15:50:18.009 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:18.009 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.009 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.009 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:18.009 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575461+0000", 2024-09-21T15:50:18.009 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:18.009 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.009 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.009 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:18.009 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575360+0000", 2024-09-21T15:50:18.010 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998994 2024-09-21T15:50:18.010 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.010 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.010 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:18.010 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575464+0000", 2024-09-21T15:50:18.010 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000103397 2024-09-21T15:50:18.010 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.010 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.010 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:18.010 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575472+0000", 2024-09-21T15:50:18.010 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3189999999999995e-06 2024-09-21T15:50:18.010 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.010 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.010 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:18.011 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575478+0000", 2024-09-21T15:50:18.011 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2779999999999999e-06 2024-09-21T15:50:18.011 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.011 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.011 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:18.011 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630268+0000", 2024-09-21T15:50:18.011 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054789545000000002 2024-09-21T15:50:18.011 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.011 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.011 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:18.011 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630315+0000", 2024-09-21T15:50:18.011 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7244999999999999e-05 2024-09-21T15:50:18.011 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.011 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.012 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:18.012 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630406+0000", 2024-09-21T15:50:18.012 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0811999999999997e-05 2024-09-21T15:50:18.012 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:18.012 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:18.012 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:18.012 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.012 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.012 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120129 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 677888~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:18.012 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.576016+0000", 2024-09-21T15:50:18.012 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 14.36018215, 2024-09-21T15:50:18.012 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054602823000000002, 2024-09-21T15:50:18.012 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:18.012 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:18.012 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:18.013 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:18.013 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:18.013 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120129 2024-09-21T15:50:18.013 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.013 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:18.013 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.013 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:18.013 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576016+0000", 2024-09-21T15:50:18.013 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:18.013 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.013 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.013 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:18.013 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576016+0000", 2024-09-21T15:50:18.013 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:18.013 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.014 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.014 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:18.014 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575483+0000", 2024-09-21T15:50:18.014 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9994669 2024-09-21T15:50:18.014 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.014 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.014 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:18.014 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576020+0000", 2024-09-21T15:50:18.014 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00053657900000000003 2024-09-21T15:50:18.014 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.014 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.014 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:18.014 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576030+0000", 2024-09-21T15:50:18.014 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0326000000000001e-05 2024-09-21T15:50:18.014 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.014 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.015 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:18.015 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576037+0000", 2024-09-21T15:50:18.015 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6490000000000002e-06 2024-09-21T15:50:18.015 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.015 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.015 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:18.015 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630446+0000", 2024-09-21T15:50:18.015 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054409698999999999 2024-09-21T15:50:18.015 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.015 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.015 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:18.015 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630524+0000", 2024-09-21T15:50:18.015 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7422999999999994e-05 2024-09-21T15:50:18.015 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.015 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.016 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:18.016 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630619+0000", 2024-09-21T15:50:18.016 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5332000000000001e-05 2024-09-21T15:50:18.016 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:18.016 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:18.016 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:18.016 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.016 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.016 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120130 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 698368~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:18.016 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.576043+0000", 2024-09-21T15:50:18.016 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 14.360155778999999, 2024-09-21T15:50:18.016 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054781798, 2024-09-21T15:50:18.016 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:18.016 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:18.016 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:18.017 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:18.017 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:18.017 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120130 2024-09-21T15:50:18.017 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.017 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:18.017 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.017 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:18.017 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576043+0000", 2024-09-21T15:50:18.017 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:18.017 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.017 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.017 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:18.017 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576043+0000", 2024-09-21T15:50:18.017 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:18.017 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.017 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.018 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:18.018 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576042+0000", 2024-09-21T15:50:18.018 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:18.018 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.018 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.018 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:18.018 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576044+0000", 2024-09-21T15:50:18.018 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0999999999999998e-06 2024-09-21T15:50:18.018 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.018 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.018 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:18.018 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576051+0000", 2024-09-21T15:50:18.018 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4779999999999996e-06 2024-09-21T15:50:18.018 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.018 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.018 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:18.019 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576058+0000", 2024-09-21T15:50:18.019 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2180000000000002e-06 2024-09-21T15:50:18.019 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.019 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.019 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:18.019 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630661+0000", 2024-09-21T15:50:18.019 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054603240999999997 2024-09-21T15:50:18.019 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.019 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.019 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:18.019 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630747+0000", 2024-09-21T15:50:18.019 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6046999999999998e-05 2024-09-21T15:50:18.019 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.019 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.019 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:18.019 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630824+0000", 2024-09-21T15:50:18.019 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7119999999999993e-05 2024-09-21T15:50:18.020 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:18.020 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:18.020 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:18.020 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:18.020 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:18.020 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:18.021 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T15:50:18.022 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:18.020+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:18.073 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T15:50:18.121 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:18.120+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:18.146 INFO:teuthology.orchestra.run.smithi089.stdout:truncating image image_client.0-clone37 from 0xb5cdd92 (overlap 0xbc37a3) to 0x0 2024-09-21T15:50:18.175 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:18.175 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:18.176 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:50:18.176 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:18.184 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T15:50:18.228 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:18.227+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:18.233 INFO:teuthology.orchestra.run.smithi089.stdout:checking clone #36, image image_client.0-clone36 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent37 2024-09-21T15:50:18.252 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [0] live_osds: [1, 2, 3, 4, 5, 6, 7] 2024-09-21T15:50:18.252 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T15:50:18.252 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-21T15:50:18.252 INFO:tasks.thrashosds.thrasher:Setting osd 3 primary_affinity to 0.605768 2024-09-21T15:50:18.252 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 0.6057684044845939 2024-09-21T15:50:18.317 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:18.317 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:18.317 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:50:18.317 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:50:18.317 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:18.322 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:18.321+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:18.329 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T15:50:18.423 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:18.422+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:18.504 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:18.504 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:50:18.504 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:50:18.504 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:50:18.504 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.504 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63140 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 876544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:18.504 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.133649+0000", 2024-09-21T15:50:18.504 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 41.368668814999999, 2024-09-21T15:50:18.504 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053950201000000003, 2024-09-21T15:50:18.504 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:18.504 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:18.505 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:18.505 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:18.505 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:18.505 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63140 2024-09-21T15:50:18.505 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.505 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:18.505 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.505 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:18.505 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133649+0000", 2024-09-21T15:50:18.505 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:18.505 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.505 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.505 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:18.505 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133649+0000", 2024-09-21T15:50:18.506 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:18.506 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.506 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.506 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:18.506 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133648+0000", 2024-09-21T15:50:18.506 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:18.506 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.506 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.506 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:18.506 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133651+0000", 2024-09-21T15:50:18.506 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1549999999999999e-06 2024-09-21T15:50:18.506 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.506 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.506 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:18.507 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133660+0000", 2024-09-21T15:50:18.507 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3899999999999993e-06 2024-09-21T15:50:18.507 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.507 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.507 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:18.507 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133669+0000", 2024-09-21T15:50:18.507 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3219999999999999e-06 2024-09-21T15:50:18.507 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.507 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.507 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:18.507 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187409+0000", 2024-09-21T15:50:18.507 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053740257 2024-09-21T15:50:18.507 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.507 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.508 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:18.508 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187492+0000", 2024-09-21T15:50:18.508 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2409000000000003e-05 2024-09-21T15:50:18.508 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.508 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.508 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:18.508 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187599+0000", 2024-09-21T15:50:18.508 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000107334 2024-09-21T15:50:18.508 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:18.508 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:18.508 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:18.508 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.508 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.508 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63141 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 884736~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:18.509 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.133843+0000", 2024-09-21T15:50:18.509 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 41.368474696, 2024-09-21T15:50:18.509 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053954015000000001, 2024-09-21T15:50:18.509 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:18.509 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:18.509 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:18.509 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:18.509 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:18.509 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63141 2024-09-21T15:50:18.509 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.509 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:18.509 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.509 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:18.509 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133843+0000", 2024-09-21T15:50:18.509 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:18.510 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.510 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.510 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:18.510 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133843+0000", 2024-09-21T15:50:18.510 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:18.510 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.510 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.510 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:18.510 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133673+0000", 2024-09-21T15:50:18.510 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998302 2024-09-21T15:50:18.510 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.510 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.510 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:18.510 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133846+0000", 2024-09-21T15:50:18.511 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000172762 2024-09-21T15:50:18.511 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.511 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.511 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:18.511 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133855+0000", 2024-09-21T15:50:18.511 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2940000000000006e-06 2024-09-21T15:50:18.511 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.511 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.511 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:18.511 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133863+0000", 2024-09-21T15:50:18.511 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9740000000000002e-06 2024-09-21T15:50:18.511 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.511 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.511 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:18.511 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187630+0000", 2024-09-21T15:50:18.512 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053767211000000002 2024-09-21T15:50:18.512 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.512 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.512 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:18.512 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187707+0000", 2024-09-21T15:50:18.512 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7446000000000004e-05 2024-09-21T15:50:18.512 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.512 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.512 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:18.512 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187797+0000", 2024-09-21T15:50:18.512 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.9858000000000006e-05 2024-09-21T15:50:18.512 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:18.512 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:18.512 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:18.512 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.512 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.513 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63142 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 892928~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:18.513 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134046+0000", 2024-09-21T15:50:18.513 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 41.368271753999998, 2024-09-21T15:50:18.513 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054017849999999999, 2024-09-21T15:50:18.513 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:18.513 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:18.513 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:18.513 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:18.513 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:18.513 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63142 2024-09-21T15:50:18.513 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.513 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:18.513 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.513 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:18.513 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134046+0000", 2024-09-21T15:50:18.514 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:18.514 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.514 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.514 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:18.514 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134046+0000", 2024-09-21T15:50:18.514 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:18.514 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.514 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.514 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:18.514 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133867+0000", 2024-09-21T15:50:18.514 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998217 2024-09-21T15:50:18.514 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.514 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.514 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:18.514 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134049+0000", 2024-09-21T15:50:18.514 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000181695 2024-09-21T15:50:18.515 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.515 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.515 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:18.515 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134057+0000", 2024-09-21T15:50:18.515 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.303e-06 2024-09-21T15:50:18.515 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.515 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.515 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:18.515 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134063+0000", 2024-09-21T15:50:18.515 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9959999999999999e-06 2024-09-21T15:50:18.515 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.515 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.515 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:18.515 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187849+0000", 2024-09-21T15:50:18.515 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053785571999999997 2024-09-21T15:50:18.516 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.516 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.516 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:18.516 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187954+0000", 2024-09-21T15:50:18.516 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010524899999999999 2024-09-21T15:50:18.516 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.516 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.516 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:18.516 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188064+0000", 2024-09-21T15:50:18.516 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000109549 2024-09-21T15:50:18.516 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:18.516 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:18.516 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:18.516 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.516 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.517 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63143 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 901120~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:18.517 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134239+0000", 2024-09-21T15:50:18.517 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 41.368079160999997, 2024-09-21T15:50:18.517 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054026058000000002, 2024-09-21T15:50:18.517 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:18.517 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:18.517 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:18.517 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:18.517 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:18.517 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63143 2024-09-21T15:50:18.517 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.517 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:18.517 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.517 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:18.517 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134239+0000", 2024-09-21T15:50:18.518 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:18.518 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.518 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.518 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:18.518 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134239+0000", 2024-09-21T15:50:18.518 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:18.518 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.518 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.518 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:18.518 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134068+0000", 2024-09-21T15:50:18.518 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998298 2024-09-21T15:50:18.518 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.518 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.518 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:18.518 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134242+0000", 2024-09-21T15:50:18.518 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00017414199999999999 2024-09-21T15:50:18.519 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.519 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.519 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:18.519 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134264+0000", 2024-09-21T15:50:18.519 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1552e-05 2024-09-21T15:50:18.519 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.519 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.519 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:18.519 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134270+0000", 2024-09-21T15:50:18.519 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6939999999999998e-06 2024-09-21T15:50:18.519 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.519 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.519 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:18.519 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188108+0000", 2024-09-21T15:50:18.520 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.05383719 2024-09-21T15:50:18.520 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.520 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.520 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:18.520 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188186+0000", 2024-09-21T15:50:18.520 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8318000000000005e-05 2024-09-21T15:50:18.520 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.520 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.520 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:18.520 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188265+0000", 2024-09-21T15:50:18.520 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8597000000000007e-05 2024-09-21T15:50:18.520 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:18.520 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:18.520 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:18.520 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.521 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.521 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63144 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 909312~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:18.521 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134432+0000", 2024-09-21T15:50:18.521 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 41.367885604999998, 2024-09-21T15:50:18.521 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053968953, 2024-09-21T15:50:18.521 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:18.521 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:18.521 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:18.521 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:18.521 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:18.521 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63144 2024-09-21T15:50:18.521 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.521 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:18.521 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.521 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:18.522 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134432+0000", 2024-09-21T15:50:18.522 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:18.522 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.522 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.522 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:18.522 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134432+0000", 2024-09-21T15:50:18.522 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:18.522 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.522 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.522 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:18.522 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134276+0000", 2024-09-21T15:50:18.522 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998441 2024-09-21T15:50:18.522 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.522 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.522 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:18.523 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134436+0000", 2024-09-21T15:50:18.523 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000159591 2024-09-21T15:50:18.523 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.523 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.523 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:18.523 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134445+0000", 2024-09-21T15:50:18.523 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6780000000000006e-06 2024-09-21T15:50:18.523 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.523 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.523 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:18.523 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134452+0000", 2024-09-21T15:50:18.523 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6830000000000008e-06 2024-09-21T15:50:18.523 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.523 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.523 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:18.524 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188282+0000", 2024-09-21T15:50:18.524 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053829252000000001 2024-09-21T15:50:18.524 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.524 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.524 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:18.524 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188328+0000", 2024-09-21T15:50:18.524 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6734999999999999e-05 2024-09-21T15:50:18.524 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.524 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.524 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:18.524 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188401+0000", 2024-09-21T15:50:18.524 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2731000000000005e-05 2024-09-21T15:50:18.524 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:18.524 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:18.525 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:18.525 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.525 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.525 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63145 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 860160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:18.525 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134736+0000", 2024-09-21T15:50:18.525 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 41.367581819999998, 2024-09-21T15:50:18.525 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053865960999999997, 2024-09-21T15:50:18.525 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:18.525 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:18.525 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:18.525 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:18.525 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:18.525 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63145 2024-09-21T15:50:18.525 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.525 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:18.526 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.526 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:18.526 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134736+0000", 2024-09-21T15:50:18.526 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:18.526 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.526 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.526 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:18.526 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134736+0000", 2024-09-21T15:50:18.526 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:18.526 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.526 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.526 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:18.526 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134458+0000", 2024-09-21T15:50:18.526 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999722 2024-09-21T15:50:18.527 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.527 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.527 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:18.527 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134741+0000", 2024-09-21T15:50:18.527 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00028245599999999999 2024-09-21T15:50:18.527 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.527 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.527 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:18.527 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134753+0000", 2024-09-21T15:50:18.527 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2548000000000001e-05 2024-09-21T15:50:18.527 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.527 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.527 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:18.527 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134760+0000", 2024-09-21T15:50:18.527 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4189999999999996e-06 2024-09-21T15:50:18.528 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.528 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.528 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:18.528 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188426+0000", 2024-09-21T15:50:18.528 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053665894999999998 2024-09-21T15:50:18.528 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.528 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.528 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:18.528 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188502+0000", 2024-09-21T15:50:18.528 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5791000000000007e-05 2024-09-21T15:50:18.528 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.528 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.528 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:18.528 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188602+0000", 2024-09-21T15:50:18.528 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9667999999999995e-05 2024-09-21T15:50:18.529 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:18.529 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:18.529 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:18.529 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.529 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.529 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63148 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 933888~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:18.529 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.135289+0000", 2024-09-21T15:50:18.529 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 41.367028865000002, 2024-09-21T15:50:18.529 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053948488000000003, 2024-09-21T15:50:18.529 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:18.529 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:18.529 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:18.529 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:18.529 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:18.529 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63148 2024-09-21T15:50:18.530 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.530 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:18.530 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.530 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:18.530 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135289+0000", 2024-09-21T15:50:18.530 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:18.530 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.530 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.530 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:18.530 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135289+0000", 2024-09-21T15:50:18.530 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:18.530 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.530 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.530 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:18.530 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135288+0000", 2024-09-21T15:50:18.531 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:18.531 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.531 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.531 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:18.531 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135296+0000", 2024-09-21T15:50:18.531 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5399999999999998e-06 2024-09-21T15:50:18.531 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.531 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.531 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:18.531 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135303+0000", 2024-09-21T15:50:18.531 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0500000000000003e-06 2024-09-21T15:50:18.531 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.531 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.531 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:18.531 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135308+0000", 2024-09-21T15:50:18.532 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9749999999999998e-06 2024-09-21T15:50:18.532 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.532 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.532 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:18.532 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189066+0000", 2024-09-21T15:50:18.532 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053758501 2024-09-21T15:50:18.532 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.532 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.532 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:18.532 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189145+0000", 2024-09-21T15:50:18.532 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9017000000000006e-05 2024-09-21T15:50:18.532 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.532 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.532 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:18.532 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189237+0000", 2024-09-21T15:50:18.533 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1906999999999999e-05 2024-09-21T15:50:18.533 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:18.533 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:18.533 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:18.533 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.533 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.533 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63149 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 942080~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:18.533 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.135430+0000", 2024-09-21T15:50:18.533 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 41.366887536, 2024-09-21T15:50:18.533 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054005434999999997, 2024-09-21T15:50:18.533 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:18.533 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:18.533 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:18.533 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:18.534 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:18.534 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63149 2024-09-21T15:50:18.534 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.534 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:18.534 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.534 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:18.534 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135430+0000", 2024-09-21T15:50:18.534 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:18.534 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.534 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.534 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:18.534 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135430+0000", 2024-09-21T15:50:18.534 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:18.534 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.535 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.535 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:18.535 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135313+0000", 2024-09-21T15:50:18.535 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998827 2024-09-21T15:50:18.535 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.535 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.535 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:18.535 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135434+0000", 2024-09-21T15:50:18.535 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000120876 2024-09-21T15:50:18.535 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.535 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.535 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:18.535 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135443+0000", 2024-09-21T15:50:18.535 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3340000000000008e-06 2024-09-21T15:50:18.535 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.535 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.536 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:18.536 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135449+0000", 2024-09-21T15:50:18.536 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1580000000000002e-06 2024-09-21T15:50:18.536 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.536 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.536 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:18.536 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189267+0000", 2024-09-21T15:50:18.536 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053817775999999998 2024-09-21T15:50:18.536 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.536 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.536 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:18.536 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189336+0000", 2024-09-21T15:50:18.536 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9485000000000003e-05 2024-09-21T15:50:18.536 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.536 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.536 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:18.537 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189436+0000", 2024-09-21T15:50:18.537 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9277000000000005e-05 2024-09-21T15:50:18.537 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:18.537 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:18.537 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:18.537 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.537 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.537 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63150 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 950272~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:18.537 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.135618+0000", 2024-09-21T15:50:18.537 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 41.366699791000002, 2024-09-21T15:50:18.537 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054027393, 2024-09-21T15:50:18.537 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:18.537 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:18.537 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:18.537 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:18.538 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:18.538 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63150 2024-09-21T15:50:18.538 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.538 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:18.538 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.538 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:18.538 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135618+0000", 2024-09-21T15:50:18.538 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:18.538 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.538 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.538 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:18.538 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135618+0000", 2024-09-21T15:50:18.538 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:18.538 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.538 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.539 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:18.539 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135454+0000", 2024-09-21T15:50:18.539 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998364 2024-09-21T15:50:18.539 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.539 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.539 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:18.539 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135621+0000", 2024-09-21T15:50:18.539 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016714099999999999 2024-09-21T15:50:18.539 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.539 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.539 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:18.539 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135631+0000", 2024-09-21T15:50:18.539 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5340000000000005e-06 2024-09-21T15:50:18.539 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.539 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.539 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:18.540 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135638+0000", 2024-09-21T15:50:18.540 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9280000000000004e-06 2024-09-21T15:50:18.540 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.540 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.540 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:18.540 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189463+0000", 2024-09-21T15:50:18.540 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053825681 2024-09-21T15:50:18.540 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.540 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.540 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:18.540 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189546+0000", 2024-09-21T15:50:18.540 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2278000000000001e-05 2024-09-21T15:50:18.540 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.540 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.540 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:18.541 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189645+0000", 2024-09-21T15:50:18.541 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9543999999999994e-05 2024-09-21T15:50:18.541 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:18.541 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:18.541 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:18.541 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.541 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.541 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63152 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 966656~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:18.541 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.136029+0000", 2024-09-21T15:50:18.541 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 41.366289199000001, 2024-09-21T15:50:18.541 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054024823999999999, 2024-09-21T15:50:18.541 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:18.541 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:18.541 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:18.541 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:18.542 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:18.542 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63152 2024-09-21T15:50:18.542 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.542 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:18.542 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.542 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:18.542 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136029+0000", 2024-09-21T15:50:18.542 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:18.542 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.542 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.542 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:18.542 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136029+0000", 2024-09-21T15:50:18.542 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:18.542 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.542 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.542 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:18.543 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136028+0000", 2024-09-21T15:50:18.543 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:18.543 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.543 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.543 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:18.543 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136031+0000", 2024-09-21T15:50:18.543 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.033e-06 2024-09-21T15:50:18.543 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.543 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.543 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:18.543 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136039+0000", 2024-09-21T15:50:18.543 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7200000000000006e-06 2024-09-21T15:50:18.543 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.544 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.544 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:18.544 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136051+0000", 2024-09-21T15:50:18.544 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1987000000000001e-05 2024-09-21T15:50:18.544 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.544 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.544 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:18.544 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189876+0000", 2024-09-21T15:50:18.544 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053825264999999997 2024-09-21T15:50:18.544 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.544 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.544 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:18.544 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189955+0000", 2024-09-21T15:50:18.544 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9296999999999997e-05 2024-09-21T15:50:18.544 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.545 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.545 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:18.545 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190053+0000", 2024-09-21T15:50:18.545 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.8067999999999997e-05 2024-09-21T15:50:18.545 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:18.545 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:18.545 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:18.545 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.545 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.545 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63153 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 974848~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:18.545 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.136311+0000", 2024-09-21T15:50:18.545 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 41.366006685999999, 2024-09-21T15:50:18.545 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053933825999999997, 2024-09-21T15:50:18.545 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:18.545 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:18.546 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:18.546 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:18.546 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:18.546 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63153 2024-09-21T15:50:18.546 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.546 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:18.546 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.546 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:18.546 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136311+0000", 2024-09-21T15:50:18.546 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:18.546 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.546 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.546 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:18.546 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136311+0000", 2024-09-21T15:50:18.546 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:18.547 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.547 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.547 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:18.547 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136057+0000", 2024-09-21T15:50:18.547 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9997463 2024-09-21T15:50:18.547 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.547 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.547 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:18.547 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136316+0000", 2024-09-21T15:50:18.547 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00025872299999999998 2024-09-21T15:50:18.547 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.547 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.547 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:18.547 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136326+0000", 2024-09-21T15:50:18.547 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0552e-05 2024-09-21T15:50:18.547 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.547 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.548 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:18.548 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136334+0000", 2024-09-21T15:50:18.548 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1409999999999999e-06 2024-09-21T15:50:18.548 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.548 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.548 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:18.548 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190080+0000", 2024-09-21T15:50:18.548 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053745933000000003 2024-09-21T15:50:18.548 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.548 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.548 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:18.548 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190157+0000", 2024-09-21T15:50:18.548 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7412999999999999e-05 2024-09-21T15:50:18.548 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.548 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.549 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:18.549 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190245+0000", 2024-09-21T15:50:18.549 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7904000000000004e-05 2024-09-21T15:50:18.549 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:18.549 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:18.549 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:18.549 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.549 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.549 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63154 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 983040~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:18.549 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.136472+0000", 2024-09-21T15:50:18.549 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 41.365845335000003, 2024-09-21T15:50:18.549 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053968448000000002, 2024-09-21T15:50:18.549 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:18.549 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:18.550 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:18.550 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:18.550 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:18.550 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63154 2024-09-21T15:50:18.550 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.550 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:18.550 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.550 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:18.550 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136472+0000", 2024-09-21T15:50:18.550 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:18.550 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.550 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.550 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:18.550 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136472+0000", 2024-09-21T15:50:18.551 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:18.551 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.551 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.551 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:18.551 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136341+0000", 2024-09-21T15:50:18.551 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998684 2024-09-21T15:50:18.551 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.551 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.551 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:18.551 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136476+0000", 2024-09-21T15:50:18.551 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013528100000000001 2024-09-21T15:50:18.551 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.551 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.551 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:18.551 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136484+0000", 2024-09-21T15:50:18.552 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4760000000000001e-06 2024-09-21T15:50:18.552 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.552 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.552 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:18.552 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136490+0000", 2024-09-21T15:50:18.552 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8640000000000001e-06 2024-09-21T15:50:18.552 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.552 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.552 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:18.552 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190271+0000", 2024-09-21T15:50:18.552 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053781223000000003 2024-09-21T15:50:18.552 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.552 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.552 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:18.552 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190348+0000", 2024-09-21T15:50:18.553 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7058999999999995e-05 2024-09-21T15:50:18.553 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.553 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.553 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:18.553 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190441+0000", 2024-09-21T15:50:18.553 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2937999999999997e-05 2024-09-21T15:50:18.553 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:18.553 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:18.553 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:18.553 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.553 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.553 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63157 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1015808~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:18.553 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137112+0000", 2024-09-21T15:50:18.553 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 41.365206047999997, 2024-09-21T15:50:18.553 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053932345, 2024-09-21T15:50:18.553 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:18.553 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:18.554 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:18.554 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:18.554 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:18.554 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63157 2024-09-21T15:50:18.554 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.554 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:18.554 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.554 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:18.554 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137112+0000", 2024-09-21T15:50:18.554 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:18.554 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.554 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.554 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:18.554 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137112+0000", 2024-09-21T15:50:18.555 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:18.555 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.555 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.555 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:18.555 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137072+0000", 2024-09-21T15:50:18.555 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999604 2024-09-21T15:50:18.555 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.555 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.555 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:18.555 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137115+0000", 2024-09-21T15:50:18.555 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2379999999999997e-05 2024-09-21T15:50:18.555 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.555 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.555 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:18.555 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137121+0000", 2024-09-21T15:50:18.556 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6499999999999999e-06 2024-09-21T15:50:18.556 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.556 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.556 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:18.556 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137126+0000", 2024-09-21T15:50:18.556 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0769999999999998e-06 2024-09-21T15:50:18.556 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.556 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.556 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:18.556 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190868+0000", 2024-09-21T15:50:18.556 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053741555000000003 2024-09-21T15:50:18.556 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.556 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.556 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:18.556 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190945+0000", 2024-09-21T15:50:18.557 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7171999999999999e-05 2024-09-21T15:50:18.557 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.557 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.557 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:18.557 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191044+0000", 2024-09-21T15:50:18.557 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9031999999999996e-05 2024-09-21T15:50:18.557 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:18.557 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:18.557 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:18.557 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.557 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.557 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63158 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1007616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:18.557 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137293+0000", 2024-09-21T15:50:18.557 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 41.365024761000001, 2024-09-21T15:50:18.558 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054005642, 2024-09-21T15:50:18.558 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:18.558 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:18.558 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:18.558 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:18.558 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:18.558 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63158 2024-09-21T15:50:18.558 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.558 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:18.558 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.558 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:18.558 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137293+0000", 2024-09-21T15:50:18.558 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:18.558 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.558 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.558 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:18.559 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137293+0000", 2024-09-21T15:50:18.559 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:18.559 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.559 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.559 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:18.559 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137131+0000", 2024-09-21T15:50:18.559 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998379 2024-09-21T15:50:18.559 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.559 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.559 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:18.559 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137296+0000", 2024-09-21T15:50:18.559 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016568999999999999 2024-09-21T15:50:18.559 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.559 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.559 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:18.559 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137306+0000", 2024-09-21T15:50:18.559 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.6029999999999994e-06 2024-09-21T15:50:18.560 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.560 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.560 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:18.560 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137312+0000", 2024-09-21T15:50:18.560 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3210000000000001e-06 2024-09-21T15:50:18.560 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.560 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.560 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:18.560 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191087+0000", 2024-09-21T15:50:18.560 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053774686000000002 2024-09-21T15:50:18.560 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.560 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.560 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:18.560 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191175+0000", 2024-09-21T15:50:18.560 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8106e-05 2024-09-21T15:50:18.560 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.561 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.561 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:18.561 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191299+0000", 2024-09-21T15:50:18.561 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012343599999999999 2024-09-21T15:50:18.561 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:18.561 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:18.561 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:18.561 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.561 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.561 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63159 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1024000~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:18.561 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137568+0000", 2024-09-21T15:50:18.561 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 41.364749488999998, 2024-09-21T15:50:18.561 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053962999999999997, 2024-09-21T15:50:18.561 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:18.561 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:18.561 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:18.561 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:18.562 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:18.562 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63159 2024-09-21T15:50:18.562 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.562 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:18.562 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.562 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:18.562 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137568+0000", 2024-09-21T15:50:18.562 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:18.562 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.562 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.562 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:18.562 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137568+0000", 2024-09-21T15:50:18.562 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:18.562 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.562 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.562 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:18.563 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137317+0000", 2024-09-21T15:50:18.563 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9997492 2024-09-21T15:50:18.563 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.563 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.563 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:18.563 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137572+0000", 2024-09-21T15:50:18.563 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00025491499999999998 2024-09-21T15:50:18.563 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.563 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.563 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:18.563 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137582+0000", 2024-09-21T15:50:18.563 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4490000000000005e-06 2024-09-21T15:50:18.563 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.563 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.563 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:18.563 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137589+0000", 2024-09-21T15:50:18.563 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5580000000000003e-06 2024-09-21T15:50:18.564 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.564 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.564 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:18.564 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191344+0000", 2024-09-21T15:50:18.564 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053754272999999998 2024-09-21T15:50:18.564 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.564 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.564 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:18.564 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191429+0000", 2024-09-21T15:50:18.564 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.509e-05 2024-09-21T15:50:18.564 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.564 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.564 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:18.564 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191531+0000", 2024-09-21T15:50:18.564 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000102519 2024-09-21T15:50:18.565 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:18.565 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:18.565 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:18.565 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.565 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.565 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63160 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1032192~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:18.565 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137771+0000", 2024-09-21T15:50:18.565 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 41.364546955000002, 2024-09-21T15:50:18.565 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053957818999999997, 2024-09-21T15:50:18.565 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:18.565 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:18.565 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:18.565 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:18.565 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:18.565 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63160 2024-09-21T15:50:18.565 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.566 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:18.566 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.566 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:18.566 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137771+0000", 2024-09-21T15:50:18.566 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:18.566 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.566 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.566 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:18.566 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137771+0000", 2024-09-21T15:50:18.566 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:18.566 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.566 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.566 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:18.566 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137596+0000", 2024-09-21T15:50:18.566 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998255 2024-09-21T15:50:18.566 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.567 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.567 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:18.567 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137780+0000", 2024-09-21T15:50:18.567 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00018363900000000001 2024-09-21T15:50:18.567 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.567 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.567 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:18.567 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137791+0000", 2024-09-21T15:50:18.567 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.096e-05 2024-09-21T15:50:18.567 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.567 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.567 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:18.567 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137798+0000", 2024-09-21T15:50:18.567 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1169999999999998e-06 2024-09-21T15:50:18.567 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.567 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.567 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:18.568 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191560+0000", 2024-09-21T15:50:18.568 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053762163000000002 2024-09-21T15:50:18.568 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.568 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.568 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:18.568 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191634+0000", 2024-09-21T15:50:18.568 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4381999999999994e-05 2024-09-21T15:50:18.568 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.568 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.568 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:18.568 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191729+0000", 2024-09-21T15:50:18.568 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4098000000000006e-05 2024-09-21T15:50:18.568 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:18.568 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:18.568 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:18.568 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.569 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.569 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63161 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1040384~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:18.569 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.138018+0000", 2024-09-21T15:50:18.569 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 41.364300200999999, 2024-09-21T15:50:18.569 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053905657000000003, 2024-09-21T15:50:18.569 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:18.569 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:18.569 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:18.569 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:18.569 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:18.569 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63161 2024-09-21T15:50:18.569 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.569 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:18.569 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.569 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:18.569 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138018+0000", 2024-09-21T15:50:18.569 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:18.570 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.570 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.570 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:18.570 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138018+0000", 2024-09-21T15:50:18.570 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:18.570 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.570 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.570 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:18.570 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137804+0000", 2024-09-21T15:50:18.570 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9997859 2024-09-21T15:50:18.570 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.570 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.570 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:18.570 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138021+0000", 2024-09-21T15:50:18.570 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00021769899999999999 2024-09-21T15:50:18.571 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.571 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.571 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:18.571 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138043+0000", 2024-09-21T15:50:18.571 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1627000000000001e-05 2024-09-21T15:50:18.571 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.571 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.571 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:18.571 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138054+0000", 2024-09-21T15:50:18.571 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1321999999999999e-05 2024-09-21T15:50:18.571 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.571 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.571 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:18.571 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191755+0000", 2024-09-21T15:50:18.571 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053700354999999998 2024-09-21T15:50:18.571 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.572 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.572 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:18.572 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191830+0000", 2024-09-21T15:50:18.572 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5320000000000004e-05 2024-09-21T15:50:18.572 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.572 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.572 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:18.572 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191923+0000", 2024-09-21T15:50:18.572 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3282999999999994e-05 2024-09-21T15:50:18.572 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:18.572 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:18.572 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:18.572 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.572 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.572 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63176 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 108544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:18.573 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.140861+0000", 2024-09-21T15:50:18.573 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 41.361456881999999, 2024-09-21T15:50:18.573 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053935394999999997, 2024-09-21T15:50:18.573 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:18.573 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:18.573 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:18.573 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:18.573 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:18.573 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63176 2024-09-21T15:50:18.573 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.573 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:18.573 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.573 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:18.573 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140861+0000", 2024-09-21T15:50:18.573 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:18.573 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.574 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.574 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:18.574 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140861+0000", 2024-09-21T15:50:18.574 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:18.574 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.574 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.574 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:18.574 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140860+0000", 2024-09-21T15:50:18.574 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:18.574 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.574 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.574 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:18.574 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140863+0000", 2024-09-21T15:50:18.574 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8710000000000001e-06 2024-09-21T15:50:18.574 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.575 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.575 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:18.575 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140871+0000", 2024-09-21T15:50:18.575 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8089999999999993e-06 2024-09-21T15:50:18.575 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.575 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.575 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:18.575 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140877+0000", 2024-09-21T15:50:18.575 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5969999999999999e-06 2024-09-21T15:50:18.575 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.575 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.575 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:18.575 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.194608+0000", 2024-09-21T15:50:18.575 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053731749000000002 2024-09-21T15:50:18.575 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.576 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.576 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:18.576 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.194697+0000", 2024-09-21T15:50:18.576 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8504000000000005e-05 2024-09-21T15:50:18.576 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.576 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.576 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:18.576 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.194796+0000", 2024-09-21T15:50:18.576 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9414999999999996e-05 2024-09-21T15:50:18.576 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:18.576 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:18.576 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:18.576 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.576 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.576 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63182 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 157696~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:18.576 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:50:18.576 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 41.360184482000001, 2024-09-21T15:50:18.577 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053958825000000002, 2024-09-21T15:50:18.577 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:18.577 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:18.577 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:18.577 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:18.577 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:18.577 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63182 2024-09-21T15:50:18.577 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.577 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:18.577 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.577 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:18.577 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:50:18.577 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:18.577 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.577 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.578 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:18.578 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:50:18.578 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:18.578 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.578 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.578 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:18.578 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:50:18.578 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:18.578 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.578 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.578 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:18.578 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142136+0000", 2024-09-21T15:50:18.578 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8169999999999999e-06 2024-09-21T15:50:18.578 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.578 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.579 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:18.579 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:50:18.579 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3752e-05 2024-09-21T15:50:18.579 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.579 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.579 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:18.579 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142155+0000", 2024-09-21T15:50:18.579 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1859999999999997e-06 2024-09-21T15:50:18.579 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.579 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.579 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:18.579 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.195914+0000", 2024-09-21T15:50:18.579 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053758528999999999 2024-09-21T15:50:18.579 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.579 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.579 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:18.580 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.195995+0000", 2024-09-21T15:50:18.580 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0983999999999996e-05 2024-09-21T15:50:18.580 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.580 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.580 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:18.580 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.196092+0000", 2024-09-21T15:50:18.580 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7069000000000002e-05 2024-09-21T15:50:18.580 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:18.580 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:18.580 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:18.580 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.580 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.580 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63193 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 239616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:18.580 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:50:18.580 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 41.358096897000003, 2024-09-21T15:50:18.581 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053956577999999998, 2024-09-21T15:50:18.581 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:18.581 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:18.581 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:18.581 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:18.581 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:18.581 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63193 2024-09-21T15:50:18.581 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.581 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:18.581 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.581 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:18.581 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:50:18.581 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:18.581 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.582 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.582 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:18.582 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:50:18.582 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:18.582 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.582 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.582 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:18.582 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:50:18.582 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:18.582 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.582 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.582 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:18.582 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144223+0000", 2024-09-21T15:50:18.583 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0990000000000001e-06 2024-09-21T15:50:18.583 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.583 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.583 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:18.583 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144230+0000", 2024-09-21T15:50:18.583 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9070000000000001e-06 2024-09-21T15:50:18.583 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.583 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.583 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:18.583 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144234+0000", 2024-09-21T15:50:18.583 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9010000000000002e-06 2024-09-21T15:50:18.583 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.583 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.583 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:18.583 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.198004+0000", 2024-09-21T15:50:18.584 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053769539999999998 2024-09-21T15:50:18.584 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.584 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.584 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:18.584 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.198083+0000", 2024-09-21T15:50:18.584 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9531999999999997e-05 2024-09-21T15:50:18.584 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:18.584 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:18.584 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:18.584 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.198177+0000", 2024-09-21T15:50:18.584 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3906000000000005e-05 2024-09-21T15:50:18.584 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:18.584 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:18.584 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:18.584 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:18.584 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:18.585 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:18.586 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T15:50:18.586 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:18.584+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:18.644 INFO:teuthology.orchestra.run.smithi089.stdout:5498 clone 39 order 22 su 524288 sc 9 2024-09-21T15:50:18.687 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:18.686+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:18.729 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:18.729 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:18.730 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:50:18.730 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:18.738 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T15:50:18.765 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:50:18.787 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:18.786+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:18.839 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:18.839 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:18.840 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:50:18.840 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:50:18.840 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:18.848 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T15:50:18.867 INFO:teuthology.orchestra.run.smithi089.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T15:50:18.883 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:50:18.883 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:50:18.888 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:18.887+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:18.899 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:50:18.900 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:50:18.915 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:50:18.915 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:50:18.932 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:50:18.932 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:50:18.948 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:50:18.948 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:50:18.965 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:50:18.965 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:50:18.981 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:50:18.982 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:50:18.985 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T15:50:18.985 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T15:50:18.989 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:18.988+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:18.994 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:18.995 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:19.003 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:50:19.080 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is set 2024-09-21T15:50:19.081 INFO:teuthology.orchestra.run.smithi089.stderr:set osd.3 primary-affinity to 0.605768 (8396992) 2024-09-21T15:50:19.090 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:19.089+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:19.135 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:19.135 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:19.144 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:50:19.228 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:19.227+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:19.286 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:19.286 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:19.291 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:19.290+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:19.295 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:50:19.392 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:19.391+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:19.438 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:19.438 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:19.446 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:50:19.493 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:19.492+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:19.549 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:19.549 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:19.557 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:50:19.594 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:19.593+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:19.695 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:19.694+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:19.699 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:19.699 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:19.708 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:50:19.796 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:19.795+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:19.851 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:19.851 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:19.859 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:50:19.897 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:19.896+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:19.962 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:19.962 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:19.971 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:50:19.998 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:19.997+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:20.073 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:20.073 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:20.083 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:50:20.099 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:20.097+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:20.119 INFO:teuthology.orchestra.run.smithi089.stdout:truncating image image_client.0-clone38 from 0xb5cdd92 (overlap 0xb5cdd92) to 0x122e2f5 2024-09-21T15:50:20.200 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:20.199+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:20.206 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:20.206 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:20.218 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:50:20.275 INFO:teuthology.orchestra.run.smithi089.stdout:checking clone #37, image image_client.0-clone37 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent38 2024-09-21T15:50:20.276 INFO:teuthology.orchestra.run.smithi089.stdout:5499 punch from 0x1d23e53 to 0x1d27ddd, (0x3f8a bytes) 2024-09-21T15:50:20.279 INFO:teuthology.orchestra.run.smithi089.stdout:5500 read 0x15caebc thru 0x15d8c27 (0xdd6c bytes) 2024-09-21T15:50:20.280 INFO:teuthology.orchestra.run.smithi089.stdout:5501 write 0x9d9519e thru 0x9d98d9a (0x3bfd bytes) 2024-09-21T15:50:20.285 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:20.285 INFO:teuthology.orchestra.run.smithi089.stdout:5503 read 0x1160159 thru 0x116ddef (0xdc97 bytes) 2024-09-21T15:50:20.285 INFO:teuthology.orchestra.run.smithi089.stdout:5505 read 0x5ac4bef thru 0x5ad1b06 (0xcf18 bytes) 2024-09-21T15:50:20.286 INFO:teuthology.orchestra.run.smithi089.stdout:5506 punch from 0x86d25ec to 0x86e14e3, (0xeef7 bytes) 2024-09-21T15:50:20.286 INFO:teuthology.orchestra.run.smithi089.stdout:5507 trunc from 0xb5cdd92 to 0xae4b1f5 2024-09-21T15:50:20.301 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:20.300+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:20.304 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:20.304 INFO:teuthology.orchestra.run.smithi089.stdout:5509 write 0x8427b91 thru 0x84371e1 (0xf651 bytes) 2024-09-21T15:50:20.309 INFO:teuthology.orchestra.run.smithi089.stdout:5510 write 0x9280bab thru 0x928a945 (0x9d9b bytes) 2024-09-21T15:50:20.311 INFO:teuthology.orchestra.run.smithi089.stdout:5511 read 0x5eca923 thru 0x5ecff0e (0x55ec bytes) 2024-09-21T15:50:20.315 INFO:teuthology.orchestra.run.smithi089.stdout:5512 trunc from 0xae4b1f5 to 0xea8d879 2024-09-21T15:50:20.330 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:20.330 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:20.335 INFO:teuthology.orchestra.run.smithi089.stdout:5514 trunc from 0xea8d879 to 0x2c43fd9 2024-09-21T15:50:20.339 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:50:20.396 INFO:teuthology.orchestra.run.smithi089.stdout:5515 clone 40 order 19 su 4096 sc 8 2024-09-21T15:50:20.402 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:20.401+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:20.482 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:20.482 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:20.490 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:50:20.534 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:20.533+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:20.591 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:20.591 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:20.600 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:50:20.641 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:20.640+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:20.704 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:20.703+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:20.742 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:20.743 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:20.751 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:50:20.805 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:20.803+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:20.852 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:20.853 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:20.861 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:50:20.945 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:20.944+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:20.986 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:50:21.005 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:21.005 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:21.014 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T15:50:21.030 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:21.029+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:21.097 INFO:teuthology.orchestra.run.smithi089.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T15:50:21.098 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T15:50:21.107 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:21.105+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:21.114 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:50:21.114 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:50:21.132 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:50:21.133 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:50:21.163 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:21.163 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:21.164 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:50:21.165 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:50:21.173 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T15:50:21.180 INFO:teuthology.orchestra.run.smithi089.stdout:truncating image image_client.0-clone39 from 0x2c43fd9 (overlap 0x2c43fd9) to 0x1efc64b 2024-09-21T15:50:21.181 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:50:21.181 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:50:21.203 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:50:21.204 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:50:21.214 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:21.213+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:21.226 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:50:21.226 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:50:21.253 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:50:21.253 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:50:21.256 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T15:50:21.257 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T15:50:21.309 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:21.308+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:21.332 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:21.332 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:21.333 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:50:21.333 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:50:21.333 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:21.342 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T15:50:21.369 INFO:teuthology.orchestra.run.smithi089.stdout:checking clone #38, image image_client.0-clone38 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent39 2024-09-21T15:50:21.398 INFO:teuthology.orchestra.run.smithi089.stdout:5516 read 0x1b7df31 thru 0x1b7fa12 (0x1ae2 bytes) 2024-09-21T15:50:21.405 INFO:teuthology.orchestra.run.smithi089.stdout:5518 trunc from 0x2c43fd9 to 0x771b800 2024-09-21T15:50:21.410 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:21.409+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:21.415 INFO:teuthology.orchestra.run.smithi089.stdout:5519 writesame 0xb7deb5 thru 0xb8dd5c data_size 0x8(0xfea8 bytes) 2024-09-21T15:50:21.472 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:21.472 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:50:21.472 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:50:21.472 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:50:21.472 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.473 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63140 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 876544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:21.473 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.133649+0000", 2024-09-21T15:50:21.473 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 44.336204461000001, 2024-09-21T15:50:21.473 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053950201000000003, 2024-09-21T15:50:21.473 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:21.473 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:21.473 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:21.473 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:21.473 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:21.473 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63140 2024-09-21T15:50:21.473 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.473 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:21.473 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.474 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:21.474 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133649+0000", 2024-09-21T15:50:21.474 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:21.474 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.474 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.474 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:21.474 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133649+0000", 2024-09-21T15:50:21.474 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:21.474 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.474 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.474 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:21.474 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133648+0000", 2024-09-21T15:50:21.474 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:21.475 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.475 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.475 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:21.475 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133651+0000", 2024-09-21T15:50:21.475 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1549999999999999e-06 2024-09-21T15:50:21.475 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.475 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.475 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:21.475 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133660+0000", 2024-09-21T15:50:21.475 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3899999999999993e-06 2024-09-21T15:50:21.475 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.475 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.475 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:21.475 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133669+0000", 2024-09-21T15:50:21.475 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3219999999999999e-06 2024-09-21T15:50:21.475 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.476 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.476 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:21.476 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187409+0000", 2024-09-21T15:50:21.476 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053740257 2024-09-21T15:50:21.476 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.476 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.476 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:21.476 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187492+0000", 2024-09-21T15:50:21.476 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2409000000000003e-05 2024-09-21T15:50:21.476 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.476 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.476 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:21.476 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187599+0000", 2024-09-21T15:50:21.476 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000107334 2024-09-21T15:50:21.476 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:21.477 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:21.477 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:21.477 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.477 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.477 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63141 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 884736~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:21.477 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.133843+0000", 2024-09-21T15:50:21.477 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 44.336010342000002, 2024-09-21T15:50:21.477 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053954015000000001, 2024-09-21T15:50:21.477 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:21.477 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:21.477 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:21.477 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:21.477 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:21.477 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63141 2024-09-21T15:50:21.477 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.478 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:21.478 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.478 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:21.478 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133843+0000", 2024-09-21T15:50:21.478 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:21.478 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.478 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.478 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:21.478 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133843+0000", 2024-09-21T15:50:21.478 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:21.478 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.478 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.478 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:21.478 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133673+0000", 2024-09-21T15:50:21.479 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998302 2024-09-21T15:50:21.479 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.479 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.479 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:21.479 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133846+0000", 2024-09-21T15:50:21.479 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000172762 2024-09-21T15:50:21.479 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.479 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.479 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:21.479 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133855+0000", 2024-09-21T15:50:21.479 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2940000000000006e-06 2024-09-21T15:50:21.479 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.479 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.479 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:21.480 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133863+0000", 2024-09-21T15:50:21.480 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9740000000000002e-06 2024-09-21T15:50:21.480 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.480 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.480 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:21.480 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187630+0000", 2024-09-21T15:50:21.480 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053767211000000002 2024-09-21T15:50:21.480 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.480 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.480 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:21.480 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187707+0000", 2024-09-21T15:50:21.480 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7446000000000004e-05 2024-09-21T15:50:21.480 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.480 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.480 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:21.481 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187797+0000", 2024-09-21T15:50:21.481 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.9858000000000006e-05 2024-09-21T15:50:21.481 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:21.481 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:21.481 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:21.481 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.481 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.481 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63142 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 892928~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:21.481 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134046+0000", 2024-09-21T15:50:21.481 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 44.3358074, 2024-09-21T15:50:21.481 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054017849999999999, 2024-09-21T15:50:21.481 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:21.481 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:21.481 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:21.481 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:21.482 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:21.482 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63142 2024-09-21T15:50:21.482 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.482 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:21.482 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.482 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:21.482 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134046+0000", 2024-09-21T15:50:21.482 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:21.482 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.482 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.482 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:21.482 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134046+0000", 2024-09-21T15:50:21.482 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:21.482 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.482 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.483 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:21.483 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133867+0000", 2024-09-21T15:50:21.483 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998217 2024-09-21T15:50:21.483 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.483 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.483 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:21.483 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134049+0000", 2024-09-21T15:50:21.483 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000181695 2024-09-21T15:50:21.483 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.483 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.483 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:21.483 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134057+0000", 2024-09-21T15:50:21.483 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.303e-06 2024-09-21T15:50:21.483 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.483 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.484 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:21.484 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134063+0000", 2024-09-21T15:50:21.484 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9959999999999999e-06 2024-09-21T15:50:21.484 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.484 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.484 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:21.484 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187849+0000", 2024-09-21T15:50:21.484 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053785571999999997 2024-09-21T15:50:21.484 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.484 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.484 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:21.484 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187954+0000", 2024-09-21T15:50:21.484 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010524899999999999 2024-09-21T15:50:21.484 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.484 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.484 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:21.484 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188064+0000", 2024-09-21T15:50:21.485 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000109549 2024-09-21T15:50:21.485 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:21.485 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:21.485 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:21.485 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.485 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.485 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63143 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 901120~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:21.485 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134239+0000", 2024-09-21T15:50:21.485 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 44.335614806999999, 2024-09-21T15:50:21.485 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054026058000000002, 2024-09-21T15:50:21.485 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:21.485 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:21.485 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:21.486 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:21.486 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:21.486 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63143 2024-09-21T15:50:21.486 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.486 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:21.486 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.486 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:21.486 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134239+0000", 2024-09-21T15:50:21.486 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:21.486 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.486 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.486 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:21.486 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134239+0000", 2024-09-21T15:50:21.486 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:21.486 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.487 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.487 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:21.487 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134068+0000", 2024-09-21T15:50:21.487 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998298 2024-09-21T15:50:21.487 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.487 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.487 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:21.487 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134242+0000", 2024-09-21T15:50:21.487 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00017414199999999999 2024-09-21T15:50:21.487 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.487 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.487 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:21.487 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134264+0000", 2024-09-21T15:50:21.488 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1552e-05 2024-09-21T15:50:21.488 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.488 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.488 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:21.488 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134270+0000", 2024-09-21T15:50:21.488 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6939999999999998e-06 2024-09-21T15:50:21.488 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.488 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.488 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:21.488 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188108+0000", 2024-09-21T15:50:21.488 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.05383719 2024-09-21T15:50:21.488 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.488 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.488 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:21.488 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188186+0000", 2024-09-21T15:50:21.489 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8318000000000005e-05 2024-09-21T15:50:21.489 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.489 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.489 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:21.489 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188265+0000", 2024-09-21T15:50:21.489 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8597000000000007e-05 2024-09-21T15:50:21.489 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:21.489 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:21.489 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:21.489 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.489 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.489 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63144 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 909312~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:21.489 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134432+0000", 2024-09-21T15:50:21.489 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 44.335421251, 2024-09-21T15:50:21.490 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053968953, 2024-09-21T15:50:21.490 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:21.490 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:21.490 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:21.490 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:21.490 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:21.490 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63144 2024-09-21T15:50:21.490 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.490 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:21.490 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.490 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:21.490 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134432+0000", 2024-09-21T15:50:21.490 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:21.491 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.491 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.491 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:21.491 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134432+0000", 2024-09-21T15:50:21.491 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:21.491 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.491 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.491 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:21.491 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134276+0000", 2024-09-21T15:50:21.491 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998441 2024-09-21T15:50:21.491 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.491 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.491 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:21.491 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134436+0000", 2024-09-21T15:50:21.492 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000159591 2024-09-21T15:50:21.492 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.492 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.492 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:21.492 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134445+0000", 2024-09-21T15:50:21.492 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6780000000000006e-06 2024-09-21T15:50:21.492 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.492 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.492 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:21.492 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134452+0000", 2024-09-21T15:50:21.492 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6830000000000008e-06 2024-09-21T15:50:21.492 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.492 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.492 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:21.493 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188282+0000", 2024-09-21T15:50:21.493 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053829252000000001 2024-09-21T15:50:21.493 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.493 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.493 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:21.493 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188328+0000", 2024-09-21T15:50:21.493 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6734999999999999e-05 2024-09-21T15:50:21.493 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.493 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.493 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:21.493 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188401+0000", 2024-09-21T15:50:21.493 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2731000000000005e-05 2024-09-21T15:50:21.493 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:21.493 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:21.493 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:21.494 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.494 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.494 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63145 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 860160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:21.494 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134736+0000", 2024-09-21T15:50:21.494 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 44.335117466, 2024-09-21T15:50:21.494 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053865960999999997, 2024-09-21T15:50:21.494 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:21.494 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:21.494 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:21.494 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:21.494 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:21.494 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63145 2024-09-21T15:50:21.494 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.494 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:21.495 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.495 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:21.495 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134736+0000", 2024-09-21T15:50:21.495 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:21.495 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.495 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.495 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:21.495 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134736+0000", 2024-09-21T15:50:21.495 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:21.495 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.495 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.495 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:21.495 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134458+0000", 2024-09-21T15:50:21.495 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999722 2024-09-21T15:50:21.496 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.496 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.496 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:21.496 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134741+0000", 2024-09-21T15:50:21.496 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00028245599999999999 2024-09-21T15:50:21.496 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.496 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.496 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:21.496 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134753+0000", 2024-09-21T15:50:21.496 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2548000000000001e-05 2024-09-21T15:50:21.496 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.496 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.496 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:21.496 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134760+0000", 2024-09-21T15:50:21.497 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4189999999999996e-06 2024-09-21T15:50:21.497 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.497 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.497 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:21.497 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188426+0000", 2024-09-21T15:50:21.497 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053665894999999998 2024-09-21T15:50:21.497 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.497 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.497 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:21.497 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188502+0000", 2024-09-21T15:50:21.497 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5791000000000007e-05 2024-09-21T15:50:21.497 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.497 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.497 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:21.497 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188602+0000", 2024-09-21T15:50:21.498 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9667999999999995e-05 2024-09-21T15:50:21.498 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:21.498 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:21.498 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:21.498 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.498 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.498 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63148 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 933888~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:21.498 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.135289+0000", 2024-09-21T15:50:21.498 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 44.334564511000004, 2024-09-21T15:50:21.498 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053948488000000003, 2024-09-21T15:50:21.498 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:21.498 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:21.498 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:21.499 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:21.499 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:21.499 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63148 2024-09-21T15:50:21.499 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.499 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:21.499 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.499 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:21.499 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135289+0000", 2024-09-21T15:50:21.499 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:21.499 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.499 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.499 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:21.499 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135289+0000", 2024-09-21T15:50:21.499 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:21.499 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.500 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.500 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:21.500 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135288+0000", 2024-09-21T15:50:21.500 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:21.500 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.500 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.500 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:21.500 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135296+0000", 2024-09-21T15:50:21.500 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5399999999999998e-06 2024-09-21T15:50:21.500 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.500 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.500 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:21.500 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135303+0000", 2024-09-21T15:50:21.500 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0500000000000003e-06 2024-09-21T15:50:21.500 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.501 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.501 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:21.501 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135308+0000", 2024-09-21T15:50:21.501 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9749999999999998e-06 2024-09-21T15:50:21.501 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.501 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.501 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:21.501 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189066+0000", 2024-09-21T15:50:21.501 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053758501 2024-09-21T15:50:21.501 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.501 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.501 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:21.501 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189145+0000", 2024-09-21T15:50:21.501 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9017000000000006e-05 2024-09-21T15:50:21.501 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.502 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.502 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:21.502 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189237+0000", 2024-09-21T15:50:21.502 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1906999999999999e-05 2024-09-21T15:50:21.502 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:21.502 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:21.502 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:21.502 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.502 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.502 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63149 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 942080~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:21.502 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.135430+0000", 2024-09-21T15:50:21.502 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 44.334423182000002, 2024-09-21T15:50:21.502 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054005434999999997, 2024-09-21T15:50:21.503 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:21.503 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:21.503 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:21.503 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:21.503 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:21.503 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63149 2024-09-21T15:50:21.503 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.503 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:21.503 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.503 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:21.503 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135430+0000", 2024-09-21T15:50:21.503 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:21.503 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.503 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.503 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:21.504 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135430+0000", 2024-09-21T15:50:21.504 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:21.504 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.504 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.504 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:21.504 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135313+0000", 2024-09-21T15:50:21.504 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998827 2024-09-21T15:50:21.504 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.504 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.504 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:21.504 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135434+0000", 2024-09-21T15:50:21.504 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000120876 2024-09-21T15:50:21.504 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.505 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.505 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:21.505 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135443+0000", 2024-09-21T15:50:21.505 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3340000000000008e-06 2024-09-21T15:50:21.505 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.505 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.505 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:21.505 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135449+0000", 2024-09-21T15:50:21.505 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1580000000000002e-06 2024-09-21T15:50:21.505 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.505 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.505 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:21.505 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189267+0000", 2024-09-21T15:50:21.505 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053817775999999998 2024-09-21T15:50:21.506 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.506 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.506 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:21.506 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189336+0000", 2024-09-21T15:50:21.506 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9485000000000003e-05 2024-09-21T15:50:21.506 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.506 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.506 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:21.506 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189436+0000", 2024-09-21T15:50:21.506 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9277000000000005e-05 2024-09-21T15:50:21.506 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:21.506 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:21.506 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:21.506 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.506 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.507 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63150 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 950272~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:21.507 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.135618+0000", 2024-09-21T15:50:21.507 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 44.334235436999997, 2024-09-21T15:50:21.507 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054027393, 2024-09-21T15:50:21.507 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:21.507 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:21.507 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:21.507 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:21.507 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:21.507 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63150 2024-09-21T15:50:21.507 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.507 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:21.507 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.507 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:21.508 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135618+0000", 2024-09-21T15:50:21.508 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:21.508 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.508 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.508 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:21.508 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135618+0000", 2024-09-21T15:50:21.508 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:21.508 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.508 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.508 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:21.508 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135454+0000", 2024-09-21T15:50:21.508 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998364 2024-09-21T15:50:21.508 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.508 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.508 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:21.508 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135621+0000", 2024-09-21T15:50:21.509 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016714099999999999 2024-09-21T15:50:21.509 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.509 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.509 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:21.509 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135631+0000", 2024-09-21T15:50:21.509 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5340000000000005e-06 2024-09-21T15:50:21.509 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.509 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.509 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:21.509 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135638+0000", 2024-09-21T15:50:21.509 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9280000000000004e-06 2024-09-21T15:50:21.509 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.509 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.509 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:21.510 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189463+0000", 2024-09-21T15:50:21.510 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053825681 2024-09-21T15:50:21.510 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.510 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.510 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:21.510 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189546+0000", 2024-09-21T15:50:21.510 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2278000000000001e-05 2024-09-21T15:50:21.510 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.510 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.510 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:21.510 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189645+0000", 2024-09-21T15:50:21.510 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9543999999999994e-05 2024-09-21T15:50:21.510 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:21.510 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:21.511 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:21.511 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.511 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.511 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63152 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 966656~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:21.511 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.136029+0000", 2024-09-21T15:50:21.511 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 44.333824845000002, 2024-09-21T15:50:21.511 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054024823999999999, 2024-09-21T15:50:21.511 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:21.511 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:21.511 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:21.511 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:21.511 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:21.511 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63152 2024-09-21T15:50:21.511 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.511 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:21.511 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.512 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:21.512 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136029+0000", 2024-09-21T15:50:21.512 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:21.512 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.512 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.512 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:21.512 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136029+0000", 2024-09-21T15:50:21.512 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:21.512 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.512 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.512 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:21.512 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136028+0000", 2024-09-21T15:50:21.512 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:21.512 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.512 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.512 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:21.513 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136031+0000", 2024-09-21T15:50:21.513 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.033e-06 2024-09-21T15:50:21.513 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.513 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.513 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:21.513 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136039+0000", 2024-09-21T15:50:21.513 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7200000000000006e-06 2024-09-21T15:50:21.513 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.513 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.513 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:21.513 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136051+0000", 2024-09-21T15:50:21.513 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1987000000000001e-05 2024-09-21T15:50:21.513 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.513 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.513 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:21.513 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189876+0000", 2024-09-21T15:50:21.514 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053825264999999997 2024-09-21T15:50:21.514 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.514 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.514 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:21.514 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189955+0000", 2024-09-21T15:50:21.514 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9296999999999997e-05 2024-09-21T15:50:21.514 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.514 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.514 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:21.514 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190053+0000", 2024-09-21T15:50:21.514 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.8067999999999997e-05 2024-09-21T15:50:21.514 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:21.514 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:21.514 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:21.515 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.515 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.515 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63153 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 974848~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:21.515 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.136311+0000", 2024-09-21T15:50:21.515 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 44.333542332, 2024-09-21T15:50:21.515 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053933825999999997, 2024-09-21T15:50:21.515 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:21.515 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:21.515 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:21.515 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:21.515 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:21.515 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63153 2024-09-21T15:50:21.515 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.515 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:21.515 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.515 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:21.516 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136311+0000", 2024-09-21T15:50:21.516 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:21.516 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.516 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.516 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:21.516 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136311+0000", 2024-09-21T15:50:21.516 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:21.516 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.516 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.516 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:21.516 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136057+0000", 2024-09-21T15:50:21.516 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9997463 2024-09-21T15:50:21.516 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.516 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.516 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:21.517 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136316+0000", 2024-09-21T15:50:21.517 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00025872299999999998 2024-09-21T15:50:21.517 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.517 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.517 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:21.517 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136326+0000", 2024-09-21T15:50:21.517 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0552e-05 2024-09-21T15:50:21.517 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.517 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.517 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:21.517 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136334+0000", 2024-09-21T15:50:21.517 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1409999999999999e-06 2024-09-21T15:50:21.517 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.517 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.517 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:21.518 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190080+0000", 2024-09-21T15:50:21.518 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053745933000000003 2024-09-21T15:50:21.518 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.518 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.518 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:21.518 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190157+0000", 2024-09-21T15:50:21.518 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7412999999999999e-05 2024-09-21T15:50:21.518 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.518 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.518 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:21.518 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190245+0000", 2024-09-21T15:50:21.518 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7904000000000004e-05 2024-09-21T15:50:21.518 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:21.518 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:21.519 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:21.519 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.519 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.519 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63154 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 983040~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:21.519 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.136472+0000", 2024-09-21T15:50:21.519 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 44.333380980999998, 2024-09-21T15:50:21.519 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053968448000000002, 2024-09-21T15:50:21.519 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:21.519 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:21.519 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:21.519 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:21.519 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:21.519 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63154 2024-09-21T15:50:21.520 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.520 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:21.520 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.520 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:21.520 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136472+0000", 2024-09-21T15:50:21.520 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:21.520 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.520 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.520 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:21.520 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136472+0000", 2024-09-21T15:50:21.520 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:21.520 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.520 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.520 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:21.520 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136341+0000", 2024-09-21T15:50:21.521 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998684 2024-09-21T15:50:21.521 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.521 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.521 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:21.521 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136476+0000", 2024-09-21T15:50:21.521 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013528100000000001 2024-09-21T15:50:21.521 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.521 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.521 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:21.521 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136484+0000", 2024-09-21T15:50:21.521 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4760000000000001e-06 2024-09-21T15:50:21.521 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.521 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.521 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:21.522 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136490+0000", 2024-09-21T15:50:21.522 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8640000000000001e-06 2024-09-21T15:50:21.522 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.522 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.522 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:21.522 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190271+0000", 2024-09-21T15:50:21.522 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053781223000000003 2024-09-21T15:50:21.522 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.522 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.522 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:21.522 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190348+0000", 2024-09-21T15:50:21.522 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7058999999999995e-05 2024-09-21T15:50:21.522 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.522 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.522 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:21.522 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190441+0000", 2024-09-21T15:50:21.523 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2937999999999997e-05 2024-09-21T15:50:21.523 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:21.523 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:21.523 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:21.523 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.523 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.523 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63157 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1015808~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:21.523 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137112+0000", 2024-09-21T15:50:21.523 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 44.332741693999999, 2024-09-21T15:50:21.523 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053932345, 2024-09-21T15:50:21.523 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:21.523 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:21.523 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:21.523 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:21.523 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:21.524 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63157 2024-09-21T15:50:21.524 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.524 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:21.524 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.524 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:21.524 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137112+0000", 2024-09-21T15:50:21.524 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:21.524 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.524 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.524 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:21.524 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137112+0000", 2024-09-21T15:50:21.524 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:21.524 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.524 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.524 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:21.524 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137072+0000", 2024-09-21T15:50:21.525 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999604 2024-09-21T15:50:21.525 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.525 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.525 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:21.525 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137115+0000", 2024-09-21T15:50:21.525 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2379999999999997e-05 2024-09-21T15:50:21.525 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.525 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.525 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:21.525 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137121+0000", 2024-09-21T15:50:21.525 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6499999999999999e-06 2024-09-21T15:50:21.525 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.525 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.526 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:21.526 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137126+0000", 2024-09-21T15:50:21.526 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0769999999999998e-06 2024-09-21T15:50:21.526 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.526 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.526 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:21.526 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190868+0000", 2024-09-21T15:50:21.526 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053741555000000003 2024-09-21T15:50:21.526 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.526 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.526 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:21.526 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190945+0000", 2024-09-21T15:50:21.526 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7171999999999999e-05 2024-09-21T15:50:21.527 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.527 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.527 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:21.527 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191044+0000", 2024-09-21T15:50:21.527 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9031999999999996e-05 2024-09-21T15:50:21.527 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:21.527 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:21.527 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:21.527 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.527 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.527 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63158 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1007616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:21.527 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137293+0000", 2024-09-21T15:50:21.527 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 44.332560407000003, 2024-09-21T15:50:21.528 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054005642, 2024-09-21T15:50:21.528 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:21.528 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:21.528 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:21.528 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:21.528 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:21.528 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63158 2024-09-21T15:50:21.528 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.528 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:21.528 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.528 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:21.528 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137293+0000", 2024-09-21T15:50:21.528 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:21.528 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.528 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.528 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:21.529 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137293+0000", 2024-09-21T15:50:21.529 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:21.529 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.529 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.529 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:21.529 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137131+0000", 2024-09-21T15:50:21.529 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998379 2024-09-21T15:50:21.529 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.529 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.529 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:21.529 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137296+0000", 2024-09-21T15:50:21.529 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016568999999999999 2024-09-21T15:50:21.529 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.529 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.530 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:21.530 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137306+0000", 2024-09-21T15:50:21.530 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.6029999999999994e-06 2024-09-21T15:50:21.530 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.530 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.530 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:21.530 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137312+0000", 2024-09-21T15:50:21.530 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3210000000000001e-06 2024-09-21T15:50:21.530 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.530 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.530 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:21.530 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191087+0000", 2024-09-21T15:50:21.530 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053774686000000002 2024-09-21T15:50:21.530 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.530 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.531 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:21.531 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191175+0000", 2024-09-21T15:50:21.531 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8106e-05 2024-09-21T15:50:21.531 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.531 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.531 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:21.531 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191299+0000", 2024-09-21T15:50:21.531 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012343599999999999 2024-09-21T15:50:21.531 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:21.531 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:21.531 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:21.531 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.531 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.532 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63159 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1024000~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:21.532 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137568+0000", 2024-09-21T15:50:21.532 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 44.332285134999999, 2024-09-21T15:50:21.532 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053962999999999997, 2024-09-21T15:50:21.532 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:21.532 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:21.532 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:21.532 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:21.532 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:21.532 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63159 2024-09-21T15:50:21.532 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.532 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:21.532 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.532 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:21.532 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137568+0000", 2024-09-21T15:50:21.533 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:21.533 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.533 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.533 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:21.533 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137568+0000", 2024-09-21T15:50:21.533 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:21.533 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.533 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.533 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:21.533 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137317+0000", 2024-09-21T15:50:21.533 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9997492 2024-09-21T15:50:21.533 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.533 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.533 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:21.533 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137572+0000", 2024-09-21T15:50:21.534 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00025491499999999998 2024-09-21T15:50:21.534 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.534 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.534 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:21.534 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137582+0000", 2024-09-21T15:50:21.534 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4490000000000005e-06 2024-09-21T15:50:21.534 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.534 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.534 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:21.534 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137589+0000", 2024-09-21T15:50:21.534 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5580000000000003e-06 2024-09-21T15:50:21.534 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.534 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.534 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:21.535 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191344+0000", 2024-09-21T15:50:21.535 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053754272999999998 2024-09-21T15:50:21.535 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.535 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.535 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:21.535 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191429+0000", 2024-09-21T15:50:21.535 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.509e-05 2024-09-21T15:50:21.535 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.535 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.535 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:21.535 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191531+0000", 2024-09-21T15:50:21.535 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000102519 2024-09-21T15:50:21.535 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:21.535 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:21.535 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:21.535 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.536 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.536 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63160 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1032192~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:21.536 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137771+0000", 2024-09-21T15:50:21.536 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 44.332082601000003, 2024-09-21T15:50:21.536 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053957818999999997, 2024-09-21T15:50:21.536 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:21.536 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:21.536 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:21.536 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:21.536 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:21.536 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63160 2024-09-21T15:50:21.536 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.536 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:21.537 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.537 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:21.537 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137771+0000", 2024-09-21T15:50:21.537 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:21.537 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.537 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.537 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:21.537 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137771+0000", 2024-09-21T15:50:21.537 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:21.537 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.537 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.537 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:21.537 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137596+0000", 2024-09-21T15:50:21.537 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998255 2024-09-21T15:50:21.537 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.538 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.538 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:21.538 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137780+0000", 2024-09-21T15:50:21.538 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00018363900000000001 2024-09-21T15:50:21.538 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.538 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.538 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:21.538 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137791+0000", 2024-09-21T15:50:21.538 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.096e-05 2024-09-21T15:50:21.538 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.538 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.538 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:21.538 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137798+0000", 2024-09-21T15:50:21.538 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1169999999999998e-06 2024-09-21T15:50:21.539 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.539 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.539 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:21.539 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191560+0000", 2024-09-21T15:50:21.539 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053762163000000002 2024-09-21T15:50:21.539 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.539 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.539 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:21.539 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191634+0000", 2024-09-21T15:50:21.539 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4381999999999994e-05 2024-09-21T15:50:21.539 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.539 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.539 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:21.540 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191729+0000", 2024-09-21T15:50:21.540 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4098000000000006e-05 2024-09-21T15:50:21.540 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:21.540 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:21.540 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:21.540 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.540 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.540 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63161 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1040384~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:21.540 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.138018+0000", 2024-09-21T15:50:21.540 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 44.331835847000001, 2024-09-21T15:50:21.540 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053905657000000003, 2024-09-21T15:50:21.540 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:21.540 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:21.540 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:21.540 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:21.540 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:21.540 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63161 2024-09-21T15:50:21.541 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.541 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:21.541 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.541 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:21.541 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138018+0000", 2024-09-21T15:50:21.541 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:21.541 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.541 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.541 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:21.541 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138018+0000", 2024-09-21T15:50:21.541 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:21.541 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.541 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.541 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:21.541 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137804+0000", 2024-09-21T15:50:21.541 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9997859 2024-09-21T15:50:21.542 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.542 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.542 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:21.542 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138021+0000", 2024-09-21T15:50:21.542 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00021769899999999999 2024-09-21T15:50:21.542 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.542 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.542 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:21.542 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138043+0000", 2024-09-21T15:50:21.542 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1627000000000001e-05 2024-09-21T15:50:21.542 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.542 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.542 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:21.542 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138054+0000", 2024-09-21T15:50:21.542 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1321999999999999e-05 2024-09-21T15:50:21.542 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.543 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.543 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:21.543 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191755+0000", 2024-09-21T15:50:21.543 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053700354999999998 2024-09-21T15:50:21.543 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.543 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.543 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:21.543 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191830+0000", 2024-09-21T15:50:21.543 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5320000000000004e-05 2024-09-21T15:50:21.543 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.543 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.543 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:21.543 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191923+0000", 2024-09-21T15:50:21.543 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3282999999999994e-05 2024-09-21T15:50:21.543 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:21.544 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:21.544 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:21.544 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.544 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.544 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63176 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 108544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:21.544 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.140861+0000", 2024-09-21T15:50:21.544 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 44.328992528000001, 2024-09-21T15:50:21.544 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053935394999999997, 2024-09-21T15:50:21.544 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:21.544 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:21.544 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:21.544 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:21.544 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:21.544 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63176 2024-09-21T15:50:21.544 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.544 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:21.545 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.545 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:21.545 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140861+0000", 2024-09-21T15:50:21.545 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:21.545 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.545 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.545 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:21.545 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140861+0000", 2024-09-21T15:50:21.545 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:21.545 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.545 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.545 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:21.545 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140860+0000", 2024-09-21T15:50:21.545 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:21.545 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.546 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.546 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:21.546 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140863+0000", 2024-09-21T15:50:21.546 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8710000000000001e-06 2024-09-21T15:50:21.546 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.546 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.546 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:21.546 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140871+0000", 2024-09-21T15:50:21.546 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8089999999999993e-06 2024-09-21T15:50:21.546 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.546 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.546 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:21.546 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140877+0000", 2024-09-21T15:50:21.546 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5969999999999999e-06 2024-09-21T15:50:21.546 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.547 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.547 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:21.547 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.194608+0000", 2024-09-21T15:50:21.547 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053731749000000002 2024-09-21T15:50:21.547 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.547 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.547 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:21.547 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.194697+0000", 2024-09-21T15:50:21.547 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8504000000000005e-05 2024-09-21T15:50:21.547 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.547 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.547 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:21.547 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.194796+0000", 2024-09-21T15:50:21.548 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9414999999999996e-05 2024-09-21T15:50:21.548 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:21.548 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:21.548 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:21.548 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.548 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.548 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63182 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 157696~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:21.548 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:50:21.548 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 44.327720128000003, 2024-09-21T15:50:21.548 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053958825000000002, 2024-09-21T15:50:21.548 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:21.548 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:21.548 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:21.548 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:21.548 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:21.548 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63182 2024-09-21T15:50:21.549 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.549 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:21.549 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.549 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:21.549 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:50:21.549 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:21.549 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.549 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.549 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:21.549 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:50:21.549 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:21.549 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.549 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.549 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:21.549 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:50:21.549 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:21.550 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.550 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.550 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:21.550 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142136+0000", 2024-09-21T15:50:21.550 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8169999999999999e-06 2024-09-21T15:50:21.550 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.550 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.550 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:21.550 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:50:21.550 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3752e-05 2024-09-21T15:50:21.550 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.550 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.550 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:21.550 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142155+0000", 2024-09-21T15:50:21.550 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1859999999999997e-06 2024-09-21T15:50:21.551 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.551 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.551 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:21.551 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.195914+0000", 2024-09-21T15:50:21.551 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053758528999999999 2024-09-21T15:50:21.551 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.551 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.551 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:21.551 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.195995+0000", 2024-09-21T15:50:21.551 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0983999999999996e-05 2024-09-21T15:50:21.551 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.551 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.551 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:21.551 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.196092+0000", 2024-09-21T15:50:21.551 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7069000000000002e-05 2024-09-21T15:50:21.552 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:21.552 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:21.552 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:21.552 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.552 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.552 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63193 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 239616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:21.552 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:50:21.552 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 44.325632542999998, 2024-09-21T15:50:21.552 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053956577999999998, 2024-09-21T15:50:21.552 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:21.552 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:21.552 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:21.552 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:21.552 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:21.552 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63193 2024-09-21T15:50:21.552 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.553 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:21.553 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.553 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:21.553 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:50:21.553 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:21.553 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.553 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.553 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:21.553 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:50:21.553 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:21.553 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.553 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.553 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:21.553 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:50:21.553 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:21.553 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.554 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.554 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:21.554 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144223+0000", 2024-09-21T15:50:21.554 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0990000000000001e-06 2024-09-21T15:50:21.554 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.554 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.554 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:21.554 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144230+0000", 2024-09-21T15:50:21.554 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9070000000000001e-06 2024-09-21T15:50:21.554 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.554 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.554 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:21.554 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144234+0000", 2024-09-21T15:50:21.554 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9010000000000002e-06 2024-09-21T15:50:21.555 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.555 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.555 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:21.555 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.198004+0000", 2024-09-21T15:50:21.555 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053769539999999998 2024-09-21T15:50:21.555 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.555 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.555 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:21.555 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.198083+0000", 2024-09-21T15:50:21.555 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9531999999999997e-05 2024-09-21T15:50:21.555 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:21.555 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:21.555 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:21.555 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.198177+0000", 2024-09-21T15:50:21.555 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3906000000000005e-05 2024-09-21T15:50:21.555 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:21.556 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:21.556 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:21.556 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:21.556 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:21.556 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:21.557 INFO:teuthology.orchestra.run.smithi089.stdout:5522 punch from 0x451eafe to 0x4528c85, (0xa187 bytes) 2024-09-21T15:50:21.557 INFO:teuthology.orchestra.run.smithi089.stdout:5523 write 0xc2da531 thru 0xc2e1021 (0x6af1 bytes) 2024-09-21T15:50:21.557 INFO:teuthology.orchestra.run.smithi089.stdout:5524 read 0x7a9b1e2 thru 0x7aa935b (0xe17a bytes) 2024-09-21T15:50:21.557 INFO:teuthology.orchestra.run.smithi089.stdout:5526 punch from 0x5d795a8 to 0x5d8910f, (0xfb67 bytes) 2024-09-21T15:50:21.558 INFO:teuthology.orchestra.run.smithi089.stdout:5527 trunc from 0xc2e1022 to 0xf2712b 2024-09-21T15:50:21.558 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T15:50:21.558 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:21.556+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:21.625 INFO:teuthology.orchestra.run.smithi089.stdout:5529 trunc from 0xf2712b to 0x37fd05a 2024-09-21T15:50:21.646 INFO:teuthology.orchestra.run.smithi089.stdout:5530 write 0xa1744df thru 0xa17d475 (0x8f97 bytes) 2024-09-21T15:50:21.658 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:21.657+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:21.660 INFO:teuthology.orchestra.run.smithi089.stdout:5532 write 0x29a8eae thru 0x29ae880 (0x59d3 bytes) 2024-09-21T15:50:21.683 INFO:teuthology.orchestra.run.smithi089.stdout:5533 writesame 0xb9b73a5 thru 0xb9c2d1d data_size 0x39(0xb979 bytes) 2024-09-21T15:50:21.683 INFO:teuthology.orchestra.run.smithi089.stdout:5534 read 0xb84426f thru 0xb84a8b5 (0x6647 bytes) 2024-09-21T15:50:21.683 INFO:teuthology.orchestra.run.smithi089.stdout:5535 read 0x90c46a2 thru 0x90c66f5 (0x2054 bytes) 2024-09-21T15:50:21.683 INFO:teuthology.orchestra.run.smithi089.stdout:5536 read 0x6041202 thru 0x604c8fb (0xb6fa bytes) 2024-09-21T15:50:21.683 INFO:teuthology.orchestra.run.smithi089.stdout:5537 read 0x62391c thru 0x62d107 (0x97ec bytes) 2024-09-21T15:50:21.684 INFO:teuthology.orchestra.run.smithi089.stdout:5538 punch from 0x28ff0af to 0x29039d0, (0x4921 bytes) 2024-09-21T15:50:21.685 INFO:teuthology.orchestra.run.smithi089.stdout:5542 trunc from 0xb9c2d1e to 0xa55856e 2024-09-21T15:50:21.693 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:21.693 INFO:teuthology.orchestra.run.smithi089.stdout:5544 punch from 0x904fb4 to 0x9095a9, (0x45f5 bytes) 2024-09-21T15:50:21.701 INFO:teuthology.orchestra.run.smithi089.stdout:5547 writesame 0xc441c1e thru 0xc44320d data_size 0x1b(0x15f0 bytes) 2024-09-21T15:50:21.701 INFO:teuthology.orchestra.run.smithi089.stdout:5548 read 0x82bfcb6 thru 0x82cf561 (0xf8ac bytes) 2024-09-21T15:50:21.701 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:21.703 INFO:teuthology.orchestra.run.smithi089.stdout:5550 trunc from 0xc44320e to 0xd9f0050 2024-09-21T15:50:21.708 INFO:teuthology.orchestra.run.smithi089.stdout:5551 read 0x8c15d05 thru 0x8c1a32e (0x462a bytes) 2024-09-21T15:50:21.708 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:21.708 INFO:teuthology.orchestra.run.smithi089.stdout:5554 punch from 0x45507f0 to 0x4556139, (0x5949 bytes) 2024-09-21T15:50:21.708 INFO:teuthology.orchestra.run.smithi089.stdout:5555 read 0xccfeffa thru 0xcd031a4 (0x41ab bytes) 2024-09-21T15:50:21.709 INFO:teuthology.orchestra.run.smithi089.stdout:5556 read 0x96772b2 thru 0x9677a79 (0x7c8 bytes) 2024-09-21T15:50:21.710 INFO:teuthology.orchestra.run.smithi089.stdout:5558 trunc from 0xd9f0050 to 0xc13ef30 2024-09-21T15:50:21.714 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:21.714 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:21.714 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:50:21.714 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:21.723 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T15:50:21.724 INFO:teuthology.orchestra.run.smithi089.stdout:5559 trunc from 0xc13ef30 to 0x416f300 2024-09-21T15:50:21.748 INFO:teuthology.orchestra.run.smithi089.stdout:5560 trunc from 0x416f300 to 0xd33517b 2024-09-21T15:50:21.754 INFO:teuthology.orchestra.run.smithi089.stdout:5562 read 0x2ffca1e thru 0x3004a01 (0x7fe4 bytes) 2024-09-21T15:50:21.754 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:21.754 INFO:teuthology.orchestra.run.smithi089.stdout:5566 read 0x9d2eda thru 0x9d9a5c (0x6b83 bytes) 2024-09-21T15:50:21.755 INFO:teuthology.orchestra.run.smithi089.stdout:5567 punch from 0x4d01a10 to 0x4d0e582, (0xcb72 bytes) 2024-09-21T15:50:21.755 INFO:teuthology.orchestra.run.smithi089.stdout:5569 punch from 0x732fa6d to 0x733c16e, (0xc701 bytes) 2024-09-21T15:50:21.755 INFO:teuthology.orchestra.run.smithi089.stdout:5571 writesame 0x41d6231 thru 0x41e19ff data_size 0x5(0xb7cf bytes) 2024-09-21T15:50:21.766 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:21.765+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:21.771 INFO:teuthology.orchestra.run.smithi089.stdout:5572 read 0xb21c6e4 thru 0xb21e48b (0x1da8 bytes) 2024-09-21T15:50:21.771 INFO:teuthology.orchestra.run.smithi089.stdout:5573 writesame 0x8de0343 thru 0x8de9c1a data_size 0x8(0x98d8 bytes) 2024-09-21T15:50:21.785 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:21.785 INFO:teuthology.orchestra.run.smithi089.stdout:5578 writesame 0x6b170be thru 0x6b1cae3 data_size 0xb(0x5a26 bytes) 2024-09-21T15:50:21.798 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:21.798 INFO:teuthology.orchestra.run.smithi089.stdout:5581 read 0xcd8202 thru 0xce2934 (0xa733 bytes) 2024-09-21T15:50:21.799 INFO:teuthology.orchestra.run.smithi089.stdout:5582 read 0x5f21cbf thru 0x5f2f18e (0xd4d0 bytes) 2024-09-21T15:50:21.799 INFO:teuthology.orchestra.run.smithi089.stdout:5588 read 0xadc978b thru 0xadcdf92 (0x4808 bytes) 2024-09-21T15:50:21.799 INFO:teuthology.orchestra.run.smithi089.stdout:5589 writesame 0x3506308 thru 0x350900f data_size 0xb(0x2d08 bytes) 2024-09-21T15:50:21.806 INFO:teuthology.orchestra.run.smithi089.stdout:5590 write 0x1bfc1b4 thru 0x1c00971 (0x47be bytes) 2024-09-21T15:50:21.817 INFO:teuthology.orchestra.run.smithi089.stdout:5591 read 0x9b21102 thru 0x9b25059 (0x3f58 bytes) 2024-09-21T15:50:21.826 INFO:teuthology.orchestra.run.smithi089.stdout:5592 trunc from 0xd33517b to 0x5a8e022 2024-09-21T15:50:21.841 INFO:teuthology.orchestra.run.smithi089.stdout:5593 read 0x3067fa2 thru 0x3073190 (0xb1ef bytes) 2024-09-21T15:50:21.841 INFO:teuthology.orchestra.run.smithi089.stdout:5594 read 0x17d7693 thru 0x17da351 (0x2cbf bytes) 2024-09-21T15:50:21.841 INFO:teuthology.orchestra.run.smithi089.stdout:5595 punch from 0x43770a7 to 0x437d106, (0x605f bytes) 2024-09-21T15:50:21.841 INFO:teuthology.orchestra.run.smithi089.stdout:5597 trunc from 0x5a8e022 to 0x5410488 2024-09-21T15:50:21.842 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:21.842 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:21.842 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:50:21.842 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:50:21.842 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:21.847 INFO:teuthology.orchestra.run.smithi089.stdout:5598 read 0x1dc6e00 thru 0x1dd3f89 (0xd18a bytes) 2024-09-21T15:50:21.847 INFO:teuthology.orchestra.run.smithi089.stdout:5599 punch from 0x172ecec to 0x17371f2, (0x8506 bytes) 2024-09-21T15:50:21.848 INFO:teuthology.orchestra.run.smithi089.stdout:5600 trunc from 0x5410488 to 0x5abf6e8 2024-09-21T15:50:21.851 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T15:50:21.869 INFO:teuthology.orchestra.run.smithi089.stdout:5601 write 0xc33a02b thru 0xc342d17 (0x8ced bytes) 2024-09-21T15:50:21.882 INFO:teuthology.orchestra.run.smithi089.stdout:5602 write 0x346e86b thru 0x346f1fa (0x990 bytes) 2024-09-21T15:50:21.884 INFO:teuthology.orchestra.run.smithi089.stdout:5605 write 0xaf357d7 thru 0xaf3b8d0 (0x60fa bytes) 2024-09-21T15:50:21.892 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:21.891+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:21.902 INFO:teuthology.orchestra.run.smithi089.stdout:5606 trunc from 0xc342d18 to 0x71f9853 2024-09-21T15:50:21.935 INFO:teuthology.orchestra.run.smithi089.stdout:5607 write 0xe0c723d thru 0xe0d0292 (0x9056 bytes) 2024-09-21T15:50:21.952 INFO:teuthology.orchestra.run.smithi089.stdout:5609 trunc from 0xe0d0293 to 0xd88fd1f 2024-09-21T15:50:21.960 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:21.958+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:21.970 INFO:teuthology.orchestra.run.smithi089.stdout:5611 writesame 0xd19176a thru 0xd19ee2b data_size 0x33(0xd6c2 bytes) 2024-09-21T15:50:21.991 INFO:teuthology.orchestra.run.smithi089.stdout:5612 read 0x5ab3fe3 thru 0x5ab9b9f (0x5bbd bytes) 2024-09-21T15:50:21.991 INFO:teuthology.orchestra.run.smithi089.stdout:5613 write 0x4a871e thru 0x4b4ce7 (0xc5ca bytes) 2024-09-21T15:50:22.025 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:22.025 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:50:22.025 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:50:22.025 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:50:22.025 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.025 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4209.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:50:22.025 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:50:22.025 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 90.830594262999995, 2024-09-21T15:50:22.025 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.19452909900000001, 2024-09-21T15:50:22.025 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:22.026 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:22.026 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:22.026 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4209", 2024-09-21T15:50:22.026 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/4045143050", 2024-09-21T15:50:22.026 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1 2024-09-21T15:50:22.026 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.026 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:22.026 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.026 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:22.026 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:50:22.026 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:22.026 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.026 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.026 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:22.027 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:50:22.027 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:22.027 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.027 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.027 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:22.027 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191161+0000", 2024-09-21T15:50:22.027 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-21T15:50:22.027 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.027 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.027 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:22.027 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191169+0000", 2024-09-21T15:50:22.027 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0090000000000007e-06 2024-09-21T15:50:22.027 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.027 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.028 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:22.028 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191180+0000", 2024-09-21T15:50:22.028 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0728e-05 2024-09-21T15:50:22.028 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.028 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.028 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:22.028 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191187+0000", 2024-09-21T15:50:22.028 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9009999999999994e-06 2024-09-21T15:50:22.028 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.028 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.028 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:22.028 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191238+0000", 2024-09-21T15:50:22.028 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0287000000000002e-05 2024-09-21T15:50:22.028 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.029 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.029 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-21T15:50:22.029 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191245+0000", 2024-09-21T15:50:22.029 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0709999999999997e-06 2024-09-21T15:50:22.029 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.029 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.029 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:22.029 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380309+0000", 2024-09-21T15:50:22.029 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.189064277 2024-09-21T15:50:22.029 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.029 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.029 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:22.029 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380454+0000", 2024-09-21T15:50:22.029 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014490799999999999 2024-09-21T15:50:22.029 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.030 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.030 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0", 2024-09-21T15:50:22.030 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380610+0000", 2024-09-21T15:50:22.030 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015650199999999999 2024-09-21T15:50:22.030 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.030 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.030 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:50:22.030 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.381160+0000", 2024-09-21T15:50:22.030 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00054903799999999996 2024-09-21T15:50:22.030 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.030 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.030 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:22.031 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385591+0000", 2024-09-21T15:50:22.031 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.004431272 2024-09-21T15:50:22.031 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.031 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.031 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:50:22.031 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385651+0000", 2024-09-21T15:50:22.031 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0455999999999999e-05 2024-09-21T15:50:22.031 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.031 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.031 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:22.031 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385691+0000", 2024-09-21T15:50:22.031 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0234999999999997e-05 2024-09-21T15:50:22.031 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:22.031 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:22.032 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:22.032 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.032 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.032 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64028 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1763328~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:22.032 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167504+0000", 2024-09-21T15:50:22.032 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 44.854252707000001, 2024-09-21T15:50:22.032 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047494153999999997, 2024-09-21T15:50:22.032 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:22.032 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:22.032 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:22.032 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:22.032 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:22.032 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64028 2024-09-21T15:50:22.032 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.033 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:22.033 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.033 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:22.033 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167504+0000", 2024-09-21T15:50:22.033 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:22.033 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.033 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.033 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:22.033 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167504+0000", 2024-09-21T15:50:22.033 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:22.033 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.033 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.033 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:22.033 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167504+0000", 2024-09-21T15:50:22.033 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:22.033 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.034 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.034 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:22.034 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167506+0000", 2024-09-21T15:50:22.034 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0600000000000002e-06 2024-09-21T15:50:22.034 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.034 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.034 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:22.034 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167512+0000", 2024-09-21T15:50:22.034 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.083e-06 2024-09-21T15:50:22.034 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.034 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.034 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:22.034 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167517+0000", 2024-09-21T15:50:22.034 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7650000000000001e-06 2024-09-21T15:50:22.035 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.035 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.035 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:22.035 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.214835+0000", 2024-09-21T15:50:22.035 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047318067999999998 2024-09-21T15:50:22.035 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.035 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.035 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:22.035 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.214916+0000", 2024-09-21T15:50:22.035 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1815000000000002e-05 2024-09-21T15:50:22.035 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.035 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.035 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:22.035 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.214998+0000", 2024-09-21T15:50:22.035 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1642000000000002e-05 2024-09-21T15:50:22.036 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:22.036 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:22.036 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:22.036 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.036 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.036 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64029 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1050624~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:22.036 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167521+0000", 2024-09-21T15:50:22.036 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 44.854235795999998, 2024-09-21T15:50:22.036 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047649915000000001, 2024-09-21T15:50:22.036 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:22.036 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:22.036 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:22.036 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:22.036 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:22.037 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64029 2024-09-21T15:50:22.037 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.037 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:22.037 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.037 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:22.037 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167521+0000", 2024-09-21T15:50:22.037 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:22.037 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.037 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.037 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:22.037 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167521+0000", 2024-09-21T15:50:22.037 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:22.037 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.037 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.037 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:22.037 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167521+0000", 2024-09-21T15:50:22.037 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:22.038 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.038 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.038 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:22.038 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167523+0000", 2024-09-21T15:50:22.038 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9810000000000002e-06 2024-09-21T15:50:22.038 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.038 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.038 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:22.038 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167529+0000", 2024-09-21T15:50:22.038 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0900000000000001e-06 2024-09-21T15:50:22.038 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.038 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.038 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:22.038 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167533+0000", 2024-09-21T15:50:22.039 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5730000000000001e-06 2024-09-21T15:50:22.039 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.039 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.039 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:22.039 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215031+0000", 2024-09-21T15:50:22.039 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047497893999999999 2024-09-21T15:50:22.039 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.039 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.039 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:22.039 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215099+0000", 2024-09-21T15:50:22.039 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8416000000000004e-05 2024-09-21T15:50:22.039 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.039 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.039 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:22.040 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215171+0000", 2024-09-21T15:50:22.040 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1279000000000007e-05 2024-09-21T15:50:22.040 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:22.040 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:22.040 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:22.040 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.040 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.040 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64030 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1058816~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:22.040 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167537+0000", 2024-09-21T15:50:22.040 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 44.854219286999999, 2024-09-21T15:50:22.040 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047783816999999999, 2024-09-21T15:50:22.040 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:22.040 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:22.040 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:22.040 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:22.040 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:22.041 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64030 2024-09-21T15:50:22.041 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.041 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:22.041 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.041 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:22.041 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167537+0000", 2024-09-21T15:50:22.041 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:22.041 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.041 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.041 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:22.041 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167537+0000", 2024-09-21T15:50:22.041 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:22.041 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.041 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.041 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:22.041 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167537+0000", 2024-09-21T15:50:22.042 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:22.042 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.042 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.042 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:22.042 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167539+0000", 2024-09-21T15:50:22.042 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9700000000000002e-06 2024-09-21T15:50:22.042 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.042 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.042 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:22.042 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167545+0000", 2024-09-21T15:50:22.042 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8629999999999996e-06 2024-09-21T15:50:22.042 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.042 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.042 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:22.043 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167549+0000", 2024-09-21T15:50:22.043 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.386e-06 2024-09-21T15:50:22.043 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.043 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.043 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:22.043 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215197+0000", 2024-09-21T15:50:22.043 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047647241 2024-09-21T15:50:22.043 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.043 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.043 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:22.043 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215263+0000", 2024-09-21T15:50:22.043 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6295000000000001e-05 2024-09-21T15:50:22.043 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.043 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.043 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:22.043 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215321+0000", 2024-09-21T15:50:22.044 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8362000000000001e-05 2024-09-21T15:50:22.044 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:22.044 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:22.044 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:22.044 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.044 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.044 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64031 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1738752~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:22.044 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167554+0000", 2024-09-21T15:50:22.044 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 44.854202784000002, 2024-09-21T15:50:22.044 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047917855000000002, 2024-09-21T15:50:22.044 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:22.044 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:22.044 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:22.044 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:22.044 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:22.044 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64031 2024-09-21T15:50:22.045 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.045 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:22.045 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.045 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:22.045 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167554+0000", 2024-09-21T15:50:22.045 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:22.045 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.045 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.045 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:22.045 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167554+0000", 2024-09-21T15:50:22.045 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:22.045 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.045 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.045 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:22.045 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167554+0000", 2024-09-21T15:50:22.045 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:22.046 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.046 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.046 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:22.046 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167556+0000", 2024-09-21T15:50:22.046 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3870000000000002e-06 2024-09-21T15:50:22.046 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.046 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.046 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:22.046 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167562+0000", 2024-09-21T15:50:22.046 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9100000000000002e-06 2024-09-21T15:50:22.046 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.046 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.046 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:22.046 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167566+0000", 2024-09-21T15:50:22.047 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6029999999999998e-06 2024-09-21T15:50:22.047 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.047 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.047 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:22.047 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215346+0000", 2024-09-21T15:50:22.047 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.04777998 2024-09-21T15:50:22.047 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.047 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.047 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:22.047 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215412+0000", 2024-09-21T15:50:22.047 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5598000000000005e-05 2024-09-21T15:50:22.047 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.047 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.047 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:22.047 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215472+0000", 2024-09-21T15:50:22.048 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9728000000000001e-05 2024-09-21T15:50:22.048 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:22.048 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:22.048 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:22.048 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.048 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.048 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64032 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1067008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:22.048 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167673+0000", 2024-09-21T15:50:22.048 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 44.854084071000003, 2024-09-21T15:50:22.048 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047947667999999999, 2024-09-21T15:50:22.048 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:22.048 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:22.048 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:22.048 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:22.049 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:22.049 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64032 2024-09-21T15:50:22.049 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.049 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:22.049 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.049 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:22.049 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167673+0000", 2024-09-21T15:50:22.049 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:22.049 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.049 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.049 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:22.049 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167673+0000", 2024-09-21T15:50:22.049 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:22.049 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.049 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.049 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:22.050 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167672+0000", 2024-09-21T15:50:22.050 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:22.050 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.050 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.050 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:22.050 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167675+0000", 2024-09-21T15:50:22.050 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3780000000000001e-06 2024-09-21T15:50:22.050 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.050 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.050 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:22.050 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167685+0000", 2024-09-21T15:50:22.050 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3039999999999994e-06 2024-09-21T15:50:22.050 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.050 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.050 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:22.050 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167692+0000", 2024-09-21T15:50:22.051 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2559999999999996e-06 2024-09-21T15:50:22.051 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.051 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.051 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:22.051 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215495+0000", 2024-09-21T15:50:22.051 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047803267000000003 2024-09-21T15:50:22.051 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.051 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.051 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:22.051 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215560+0000", 2024-09-21T15:50:22.051 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4703999999999996e-05 2024-09-21T15:50:22.051 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.051 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.051 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:22.052 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215620+0000", 2024-09-21T15:50:22.052 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0328000000000002e-05 2024-09-21T15:50:22.052 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:22.052 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:22.052 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:22.052 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.052 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.052 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64033 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1075200~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:22.052 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167697+0000", 2024-09-21T15:50:22.052 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 44.854059499000002, 2024-09-21T15:50:22.052 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048082863000000003, 2024-09-21T15:50:22.052 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:22.052 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:22.052 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:22.052 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:22.053 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:22.053 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64033 2024-09-21T15:50:22.053 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.053 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:22.053 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.053 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:22.053 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167697+0000", 2024-09-21T15:50:22.053 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:22.053 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.053 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.053 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:22.053 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167697+0000", 2024-09-21T15:50:22.053 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:22.053 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.053 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.054 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:22.054 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167697+0000", 2024-09-21T15:50:22.054 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:22.054 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.054 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.054 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:22.054 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167699+0000", 2024-09-21T15:50:22.054 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5299999999999999e-06 2024-09-21T15:50:22.054 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.054 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.054 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:22.054 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167706+0000", 2024-09-21T15:50:22.054 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7490000000000001e-06 2024-09-21T15:50:22.054 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.054 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.055 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:22.055 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167712+0000", 2024-09-21T15:50:22.055 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1940000000000003e-06 2024-09-21T15:50:22.055 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.055 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.055 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:22.055 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215643+0000", 2024-09-21T15:50:22.055 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047930647999999999 2024-09-21T15:50:22.055 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.055 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.055 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:22.055 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215713+0000", 2024-09-21T15:50:22.055 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0488999999999995e-05 2024-09-21T15:50:22.055 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.055 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.055 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:22.056 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215780+0000", 2024-09-21T15:50:22.056 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6656999999999996e-05 2024-09-21T15:50:22.056 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:22.056 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:22.056 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:22.056 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.056 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.056 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64034 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1083392~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:22.056 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167717+0000", 2024-09-21T15:50:22.056 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 44.854039933000003, 2024-09-21T15:50:22.056 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048233316999999998, 2024-09-21T15:50:22.056 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:22.056 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:22.056 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:22.056 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:22.056 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:22.057 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64034 2024-09-21T15:50:22.057 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.057 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:22.057 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.057 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:22.057 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167717+0000", 2024-09-21T15:50:22.057 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:22.057 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.057 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.057 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:22.057 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167717+0000", 2024-09-21T15:50:22.057 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:22.057 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.057 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.057 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:22.058 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167716+0000", 2024-09-21T15:50:22.058 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:22.058 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.058 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.058 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:22.058 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167718+0000", 2024-09-21T15:50:22.058 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.165e-06 2024-09-21T15:50:22.058 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.058 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.058 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:22.058 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167725+0000", 2024-09-21T15:50:22.058 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3380000000000001e-06 2024-09-21T15:50:22.058 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.058 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.059 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:22.059 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167730+0000", 2024-09-21T15:50:22.059 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8010000000000003e-06 2024-09-21T15:50:22.059 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.059 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.059 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:22.059 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215807+0000", 2024-09-21T15:50:22.059 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048077823999999998 2024-09-21T15:50:22.059 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.059 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.059 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:22.059 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215884+0000", 2024-09-21T15:50:22.059 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6283000000000001e-05 2024-09-21T15:50:22.059 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.060 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.060 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:22.060 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215950+0000", 2024-09-21T15:50:22.060 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6290000000000004e-05 2024-09-21T15:50:22.060 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:22.060 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:22.060 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:22.060 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.060 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.060 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64035 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1091584~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:22.060 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:50:22.060 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 44.854022720000003, 2024-09-21T15:50:22.060 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048412117999999997, 2024-09-21T15:50:22.060 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:22.061 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:22.061 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:22.061 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:22.061 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:22.061 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64035 2024-09-21T15:50:22.061 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.061 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:22.061 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.061 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:22.061 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:50:22.061 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:22.061 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.061 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.061 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:22.061 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:50:22.062 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:22.062 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.062 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.062 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:22.062 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:50:22.062 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:22.062 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.062 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.062 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:22.062 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167746+0000", 2024-09-21T15:50:22.062 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2031000000000001e-05 2024-09-21T15:50:22.062 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.062 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.062 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:22.063 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167753+0000", 2024-09-21T15:50:22.063 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0620000000000003e-06 2024-09-21T15:50:22.063 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.063 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.063 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:22.063 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167759+0000", 2024-09-21T15:50:22.063 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.596e-06 2024-09-21T15:50:22.063 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.063 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.063 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:22.063 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215977+0000", 2024-09-21T15:50:22.063 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048217318000000002 2024-09-21T15:50:22.063 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.063 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.064 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:22.064 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216076+0000", 2024-09-21T15:50:22.064 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9846000000000006e-05 2024-09-21T15:50:22.064 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.064 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.064 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:22.064 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216146+0000", 2024-09-21T15:50:22.064 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9578000000000004e-05 2024-09-21T15:50:22.064 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:22.064 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:22.064 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:22.064 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.064 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.064 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64036 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1099776~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:22.064 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167853+0000", 2024-09-21T15:50:22.065 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 44.853904106999998, 2024-09-21T15:50:22.065 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048450675999999998, 2024-09-21T15:50:22.065 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:22.065 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:22.065 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:22.065 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:22.065 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:22.065 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64036 2024-09-21T15:50:22.065 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.065 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:22.065 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.065 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:22.065 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167853+0000", 2024-09-21T15:50:22.065 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:22.066 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.066 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.066 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:22.066 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167853+0000", 2024-09-21T15:50:22.066 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:22.066 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.066 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.066 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:22.066 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167852+0000", 2024-09-21T15:50:22.066 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:22.066 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.066 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.066 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:22.066 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167855+0000", 2024-09-21T15:50:22.067 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5640000000000001e-06 2024-09-21T15:50:22.067 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.067 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.067 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:22.067 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167864+0000", 2024-09-21T15:50:22.067 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8140000000000007e-06 2024-09-21T15:50:22.067 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.067 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.067 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:22.067 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167871+0000", 2024-09-21T15:50:22.067 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5729999999999996e-06 2024-09-21T15:50:22.067 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.067 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.068 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:22.068 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216174+0000", 2024-09-21T15:50:22.068 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048303437999999997 2024-09-21T15:50:22.068 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.068 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.068 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:22.068 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216250+0000", 2024-09-21T15:50:22.068 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5661999999999995e-05 2024-09-21T15:50:22.068 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.068 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.068 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:22.068 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216303+0000", 2024-09-21T15:50:22.068 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3273000000000003e-05 2024-09-21T15:50:22.068 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:22.069 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:22.069 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:22.069 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.069 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.069 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64037 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1771520~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:22.069 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:50:22.069 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 44.853880564000001, 2024-09-21T15:50:22.069 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048588771000000003, 2024-09-21T15:50:22.069 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:22.069 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:22.069 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:22.069 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:22.069 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:22.069 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64037 2024-09-21T15:50:22.070 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.070 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:22.070 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.070 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:22.070 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:50:22.070 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:22.070 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.070 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.070 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:22.070 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:50:22.070 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:22.070 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.070 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.070 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:22.070 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:50:22.070 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:22.071 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.071 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.071 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:22.071 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167878+0000", 2024-09-21T15:50:22.071 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3e-06 2024-09-21T15:50:22.071 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.071 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.071 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:22.071 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167884+0000", 2024-09-21T15:50:22.071 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3269999999999997e-06 2024-09-21T15:50:22.071 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.071 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.071 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:22.071 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167889+0000", 2024-09-21T15:50:22.071 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9269999999999996e-06 2024-09-21T15:50:22.072 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.072 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.072 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:22.072 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216331+0000", 2024-09-21T15:50:22.072 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048442102000000001 2024-09-21T15:50:22.072 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.072 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.072 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:22.072 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216393+0000", 2024-09-21T15:50:22.072 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1132999999999999e-05 2024-09-21T15:50:22.072 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.072 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.072 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:22.072 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216465+0000", 2024-09-21T15:50:22.072 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2339e-05 2024-09-21T15:50:22.072 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:22.073 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:22.073 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:22.073 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.073 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.073 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64038 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1779712~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:22.073 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:50:22.073 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 44.853862733, 2024-09-21T15:50:22.073 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048749795999999998, 2024-09-21T15:50:22.073 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:22.073 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:22.073 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:22.073 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:22.073 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:22.073 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64038 2024-09-21T15:50:22.073 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.073 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:22.074 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.074 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:22.074 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:50:22.074 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:22.074 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.074 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.074 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:22.074 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:50:22.074 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:22.074 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.074 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.074 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:22.074 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:50:22.074 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:22.074 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.075 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.075 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:22.075 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167896+0000", 2024-09-21T15:50:22.075 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2419999999999999e-06 2024-09-21T15:50:22.075 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.075 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.075 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:22.075 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167902+0000", 2024-09-21T15:50:22.075 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0859999999999998e-06 2024-09-21T15:50:22.075 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.075 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.075 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:22.075 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167907+0000", 2024-09-21T15:50:22.075 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9679999999999997e-06 2024-09-21T15:50:22.075 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.075 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.076 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:22.076 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216506+0000", 2024-09-21T15:50:22.076 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048599374000000001 2024-09-21T15:50:22.076 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.076 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.076 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:22.076 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216579+0000", 2024-09-21T15:50:22.076 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2801999999999999e-05 2024-09-21T15:50:22.076 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.076 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.076 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:22.076 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216644+0000", 2024-09-21T15:50:22.076 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4715000000000006e-05 2024-09-21T15:50:22.076 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:22.076 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:22.076 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:22.077 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.077 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.077 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64039 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1107968~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:22.077 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:50:22.077 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 44.853845378000003, 2024-09-21T15:50:22.077 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048911370000000003, 2024-09-21T15:50:22.077 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:22.077 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:22.077 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:22.077 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:22.077 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:22.077 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64039 2024-09-21T15:50:22.077 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.077 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:22.077 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.077 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:22.078 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:50:22.078 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:22.078 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.078 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.078 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:22.078 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:50:22.078 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:22.078 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.078 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.078 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:22.078 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:50:22.078 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:22.078 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.078 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.079 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:22.079 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167913+0000", 2024-09-21T15:50:22.079 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0839999999999999e-06 2024-09-21T15:50:22.079 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.079 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.079 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:22.079 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167921+0000", 2024-09-21T15:50:22.079 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8310000000000001e-06 2024-09-21T15:50:22.079 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.079 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.079 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:22.079 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167926+0000", 2024-09-21T15:50:22.079 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0270000000000003e-06 2024-09-21T15:50:22.079 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.079 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.080 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:22.080 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216680+0000", 2024-09-21T15:50:22.080 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048753890000000001 2024-09-21T15:50:22.080 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.080 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.080 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:22.080 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216757+0000", 2024-09-21T15:50:22.080 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6848999999999996e-05 2024-09-21T15:50:22.080 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.080 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.080 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:22.080 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216823+0000", 2024-09-21T15:50:22.080 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6068000000000005e-05 2024-09-21T15:50:22.080 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:22.080 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:22.080 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:22.081 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.081 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.081 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64040 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1787904~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:22.081 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167931+0000", 2024-09-21T15:50:22.081 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 44.853826146000003, 2024-09-21T15:50:22.081 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049010561000000001, 2024-09-21T15:50:22.081 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:22.081 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:22.081 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:22.081 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:22.081 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:22.081 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64040 2024-09-21T15:50:22.081 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.081 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:22.081 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.081 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:22.082 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167931+0000", 2024-09-21T15:50:22.082 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:22.082 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.082 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.082 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:22.082 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167931+0000", 2024-09-21T15:50:22.082 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:22.082 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.082 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.082 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:22.082 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167930+0000", 2024-09-21T15:50:22.082 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:22.082 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.082 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.083 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:22.083 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167932+0000", 2024-09-21T15:50:22.083 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.198e-06 2024-09-21T15:50:22.083 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.083 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.083 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:22.083 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167938+0000", 2024-09-21T15:50:22.083 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.105e-06 2024-09-21T15:50:22.083 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.083 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.083 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:22.083 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167943+0000", 2024-09-21T15:50:22.083 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9429999999999999e-06 2024-09-21T15:50:22.083 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.083 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.084 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:22.084 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216848+0000", 2024-09-21T15:50:22.084 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048905200000000003 2024-09-21T15:50:22.084 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.084 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.084 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:22.084 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216903+0000", 2024-09-21T15:50:22.084 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4793999999999997e-05 2024-09-21T15:50:22.084 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.084 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.084 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:22.084 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216941+0000", 2024-09-21T15:50:22.084 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7790999999999998e-05 2024-09-21T15:50:22.084 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:22.085 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:22.085 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:22.085 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.085 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.085 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64041 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1796096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:22.085 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:50:22.085 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 44.853809175000002, 2024-09-21T15:50:22.085 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049181457999999997, 2024-09-21T15:50:22.085 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:22.085 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:22.085 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:22.085 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:22.085 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:22.085 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64041 2024-09-21T15:50:22.085 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.085 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:22.085 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.086 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:22.086 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:50:22.086 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:22.086 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.086 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.086 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:22.086 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:50:22.086 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:22.086 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.086 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.086 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:22.086 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:50:22.086 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:22.086 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.086 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.087 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:22.087 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167949+0000", 2024-09-21T15:50:22.087 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0200000000000001e-06 2024-09-21T15:50:22.087 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.087 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.087 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:22.087 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167955+0000", 2024-09-21T15:50:22.087 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5990000000000001e-06 2024-09-21T15:50:22.087 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.087 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.087 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:22.087 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167959+0000", 2024-09-21T15:50:22.087 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5589999999999999e-06 2024-09-21T15:50:22.087 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.087 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.088 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:22.088 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216972+0000", 2024-09-21T15:50:22.088 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049012574000000003 2024-09-21T15:50:22.088 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.088 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.088 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:22.088 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217057+0000", 2024-09-21T15:50:22.088 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4820000000000004e-05 2024-09-21T15:50:22.088 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.088 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.088 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:22.088 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217129+0000", 2024-09-21T15:50:22.088 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2186000000000003e-05 2024-09-21T15:50:22.088 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:22.088 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:22.088 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:22.089 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.089 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.089 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64042 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1804288~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:22.089 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167964+0000", 2024-09-21T15:50:22.089 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 44.853792884000001, 2024-09-21T15:50:22.089 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049333372, 2024-09-21T15:50:22.089 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:22.089 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:22.089 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:22.089 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:22.089 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:22.089 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64042 2024-09-21T15:50:22.089 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.089 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:22.089 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.090 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:22.090 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167964+0000", 2024-09-21T15:50:22.090 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:22.090 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.090 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.090 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:22.090 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167964+0000", 2024-09-21T15:50:22.090 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:22.090 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.090 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.090 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:22.090 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167963+0000", 2024-09-21T15:50:22.090 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:22.090 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.091 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.091 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:22.091 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167966+0000", 2024-09-21T15:50:22.091 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1119999999999999e-06 2024-09-21T15:50:22.091 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.091 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.091 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:22.091 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167975+0000", 2024-09-21T15:50:22.091 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1130000000000007e-06 2024-09-21T15:50:22.091 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.091 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.091 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:22.091 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167980+0000", 2024-09-21T15:50:22.091 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9599999999999999e-06 2024-09-21T15:50:22.091 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.092 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.092 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:22.092 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217158+0000", 2024-09-21T15:50:22.092 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049178569999999998 2024-09-21T15:50:22.092 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.092 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.092 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:22.092 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217232+0000", 2024-09-21T15:50:22.092 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3589000000000003e-05 2024-09-21T15:50:22.092 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.092 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.092 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:22.092 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217297+0000", 2024-09-21T15:50:22.092 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5339999999999994e-05 2024-09-21T15:50:22.092 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:22.093 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:22.093 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:22.093 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.093 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.093 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64043 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1812480~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:22.093 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.168085+0000", 2024-09-21T15:50:22.093 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 44.853671816999999, 2024-09-21T15:50:22.093 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049365157999999999, 2024-09-21T15:50:22.093 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:22.093 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:22.093 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:22.093 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:22.093 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:22.093 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64043 2024-09-21T15:50:22.093 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.093 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:22.094 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.094 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:22.094 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168085+0000", 2024-09-21T15:50:22.094 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:22.094 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.094 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.094 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:22.094 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168085+0000", 2024-09-21T15:50:22.094 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:22.094 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.094 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.094 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:22.094 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168084+0000", 2024-09-21T15:50:22.094 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:22.094 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.095 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.095 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:22.095 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168088+0000", 2024-09-21T15:50:22.095 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5420000000000001e-06 2024-09-21T15:50:22.095 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.095 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.095 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:22.095 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168097+0000", 2024-09-21T15:50:22.095 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5410000000000006e-06 2024-09-21T15:50:22.095 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.095 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.095 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:22.095 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168104+0000", 2024-09-21T15:50:22.095 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6719999999999998e-06 2024-09-21T15:50:22.095 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.095 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.095 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:22.096 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217323+0000", 2024-09-21T15:50:22.096 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049218857999999997 2024-09-21T15:50:22.096 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.096 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.096 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:22.096 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217397+0000", 2024-09-21T15:50:22.096 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3932999999999998e-05 2024-09-21T15:50:22.096 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.096 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.096 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:22.096 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217450+0000", 2024-09-21T15:50:22.096 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3168000000000003e-05 2024-09-21T15:50:22.097 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:22.097 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:22.097 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:22.097 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.097 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.097 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64044 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1820672~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:22.097 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.170036+0000", 2024-09-21T15:50:22.097 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 44.851720497999999, 2024-09-21T15:50:22.097 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047578069000000001, 2024-09-21T15:50:22.097 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:22.097 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:22.097 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:22.097 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:22.097 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:22.098 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64044 2024-09-21T15:50:22.098 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.098 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:22.098 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.098 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:22.098 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170036+0000", 2024-09-21T15:50:22.098 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:22.098 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.098 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.098 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:22.098 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170036+0000", 2024-09-21T15:50:22.098 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:22.098 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.098 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.099 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:22.099 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168109+0000", 2024-09-21T15:50:22.099 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9980721 2024-09-21T15:50:22.099 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.099 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.099 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:22.099 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170040+0000", 2024-09-21T15:50:22.099 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0019318079999999999 2024-09-21T15:50:22.099 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.099 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.099 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:22.099 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170051+0000", 2024-09-21T15:50:22.099 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0220999999999999e-05 2024-09-21T15:50:22.099 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.099 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.099 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:22.100 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170058+0000", 2024-09-21T15:50:22.100 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0110000000000003e-06 2024-09-21T15:50:22.100 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.100 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.100 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:22.100 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217478+0000", 2024-09-21T15:50:22.100 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047420334000000001 2024-09-21T15:50:22.100 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.100 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.100 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:22.100 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217547+0000", 2024-09-21T15:50:22.100 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9416000000000001e-05 2024-09-21T15:50:22.100 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.100 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.101 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:22.101 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217614+0000", 2024-09-21T15:50:22.101 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6909000000000006e-05 2024-09-21T15:50:22.101 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:22.101 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:22.101 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:22.101 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.101 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.101 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64045 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1828864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:22.101 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.170064+0000", 2024-09-21T15:50:22.101 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 44.851692458999999, 2024-09-21T15:50:22.101 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047709252000000001, 2024-09-21T15:50:22.101 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:22.101 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:22.101 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:22.102 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:22.102 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:22.102 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64045 2024-09-21T15:50:22.102 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.102 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:22.102 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.102 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:22.102 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170064+0000", 2024-09-21T15:50:22.102 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:22.102 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.102 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.102 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:22.102 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170064+0000", 2024-09-21T15:50:22.103 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:22.103 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.103 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.103 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:22.103 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170064+0000", 2024-09-21T15:50:22.103 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:22.103 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.103 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.103 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:22.103 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170066+0000", 2024-09-21T15:50:22.103 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8109999999999999e-06 2024-09-21T15:50:22.103 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.103 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.103 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:22.103 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170073+0000", 2024-09-21T15:50:22.104 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5570000000000001e-06 2024-09-21T15:50:22.104 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.104 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.104 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:22.104 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.170078+0000", 2024-09-21T15:50:22.104 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9910000000000002e-06 2024-09-21T15:50:22.104 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.104 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.104 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:22.104 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217640+0000", 2024-09-21T15:50:22.104 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047561840000000001 2024-09-21T15:50:22.104 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.104 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.104 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:22.104 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217710+0000", 2024-09-21T15:50:22.104 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9967000000000003e-05 2024-09-21T15:50:22.105 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.105 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.105 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:22.105 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217773+0000", 2024-09-21T15:50:22.105 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3638000000000003e-05 2024-09-21T15:50:22.105 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:22.105 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:22.105 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:22.105 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.105 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.105 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64356 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 3076096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:22.105 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.183065+0000", 2024-09-21T15:50:22.106 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 44.838691328000003, 2024-09-21T15:50:22.106 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047484923999999998, 2024-09-21T15:50:22.106 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:22.106 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:22.106 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:22.106 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:22.106 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:22.106 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64356 2024-09-21T15:50:22.106 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.106 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:22.106 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.106 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:22.106 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.183065+0000", 2024-09-21T15:50:22.106 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:22.107 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.107 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.107 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:22.107 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.183065+0000", 2024-09-21T15:50:22.107 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:22.107 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.107 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.107 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:22.107 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.182922+0000", 2024-09-21T15:50:22.107 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999856 2024-09-21T15:50:22.107 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.107 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.107 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:22.107 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.183069+0000", 2024-09-21T15:50:22.107 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000147097 2024-09-21T15:50:22.107 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.108 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.108 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:22.108 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.183078+0000", 2024-09-21T15:50:22.108 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.9099999999999994e-06 2024-09-21T15:50:22.108 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.108 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.108 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:22.108 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.183085+0000", 2024-09-21T15:50:22.108 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0670000000000003e-06 2024-09-21T15:50:22.108 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.108 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.108 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:22.108 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.230377+0000", 2024-09-21T15:50:22.108 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.047292698000000001 2024-09-21T15:50:22.108 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.109 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.109 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:22.109 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.230458+0000", 2024-09-21T15:50:22.109 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0980000000000001e-05 2024-09-21T15:50:22.109 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:22.109 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:22.109 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:22.109 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.230550+0000", 2024-09-21T15:50:22.109 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1954999999999996e-05 2024-09-21T15:50:22.109 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:22.109 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:22.109 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:22.109 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:22.109 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:22.109 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:22.111 INFO:teuthology.orchestra.run.smithi089.stdout:5614 write 0x72212e thru 0x72bd2b (0x9bfe bytes) 2024-09-21T15:50:22.111 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:22.111 INFO:teuthology.orchestra.run.smithi089.stdout:5616 read 0xc2c6b78 thru 0xc2ceaee (0x7f77 bytes) 2024-09-21T15:50:22.111 INFO:teuthology.orchestra.run.smithi089.stdout:5617 writesame 0x95851e6 thru 0x9593baf data_size 0xd2(0xe9ca bytes) 2024-09-21T15:50:22.112 INFO:teuthology.orchestra.run.smithi089.stdout:5618 read 0x9ee8b8c thru 0x9eee6d5 (0x5b4a bytes) 2024-09-21T15:50:22.112 INFO:teuthology.orchestra.run.smithi089.stdout:5619 write 0xa82b382 thru 0xa838e37 (0xdab6 bytes) 2024-09-21T15:50:22.112 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is set 2024-09-21T15:50:22.112 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:22.109+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:22.112 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:50:22.113 INFO:teuthology.orchestra.run.smithi089.stdout:5620 write 0x121a320 thru 0x121fcde (0x59bf bytes) 2024-09-21T15:50:22.119 INFO:teuthology.orchestra.run.smithi089.stdout:5622 punch from 0xadb15eb to 0xadbead4, (0xd4e9 bytes) 2024-09-21T15:50:22.126 INFO:teuthology.orchestra.run.smithi089.stdout:5623 trunc from 0xd88fd1f to 0x7152f7b 2024-09-21T15:50:22.148 INFO:teuthology.orchestra.run.smithi089.stdout:5625 trunc from 0x7152f7b to 0xddbdf7 2024-09-21T15:50:22.189 INFO:teuthology.orchestra.run.smithi089.stdout:5626 write 0x633b53a thru 0x6347374 (0xbe3b bytes) 2024-09-21T15:50:22.203 INFO:teuthology.orchestra.run.smithi089.stdout:5627 read 0x5be115f thru 0x5beed05 (0xdba7 bytes) 2024-09-21T15:50:22.203 INFO:teuthology.orchestra.run.smithi089.stdout:5629 writesame 0xbd7057 thru 0xbdce65 data_size 0xb(0x5e0f bytes) 2024-09-21T15:50:22.208 INFO:teuthology.orchestra.run.smithi089.stdout:5630 punch from 0x3f218f6 to 0x3f300a6, (0xe7b0 bytes) 2024-09-21T15:50:22.208 INFO:teuthology.orchestra.run.smithi089.stdout:5632 write 0x5db55e4 thru 0x5db76d0 (0x20ed bytes) 2024-09-21T15:50:22.212 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:22.211+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:22.212 INFO:teuthology.orchestra.run.smithi089.stdout:5633 read 0x4bb6718 thru 0x4bbda6e (0x7357 bytes) 2024-09-21T15:50:22.223 INFO:teuthology.orchestra.run.smithi089.stdout:5634 write 0xb8b092c thru 0xb8b6772 (0x5e47 bytes) 2024-09-21T15:50:22.226 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:22.226 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:22.226 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:50:22.226 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:22.234 INFO:teuthology.orchestra.run.smithi089.stdout:5635 read 0x1522901 thru 0x1527510 (0x4c10 bytes) 2024-09-21T15:50:22.234 INFO:teuthology.orchestra.run.smithi089.stdout:5636 read 0x74e72f4 thru 0x74eb3d2 (0x40df bytes) 2024-09-21T15:50:22.234 INFO:teuthology.orchestra.run.smithi089.stdout:5637 read 0x71a6193 thru 0x71ae4cc (0x833a bytes) 2024-09-21T15:50:22.234 INFO:teuthology.orchestra.run.smithi089.stdout:5638 read 0x7fb891 thru 0x7fe6bc (0x2e2c bytes) 2024-09-21T15:50:22.236 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:50:22.236 INFO:teuthology.orchestra.run.smithi089.stdout:5639 writesame 0x5a9e5dd thru 0x5aa8ab4 data_size 0x32(0xa4d8 bytes) 2024-09-21T15:50:22.248 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:22.248 INFO:teuthology.orchestra.run.smithi089.stdout:5641 read 0x5fec7b7 thru 0x5ff5994 (0x91de bytes) 2024-09-21T15:50:22.254 INFO:teuthology.orchestra.run.smithi089.stdout:5643 write 0xc1372b4 thru 0xc14005c (0x8da9 bytes) 2024-09-21T15:50:22.266 INFO:teuthology.orchestra.run.smithi089.stdout:5645 read 0x9bd4ca2 thru 0x9be2269 (0xd5c8 bytes) 2024-09-21T15:50:22.266 INFO:teuthology.orchestra.run.smithi089.stdout:5646 punch from 0xb77b93d to 0xb785038, (0x96fb bytes) 2024-09-21T15:50:22.266 INFO:teuthology.orchestra.run.smithi089.stdout:5647 punch from 0xa546a73 to 0xa547270, (0x7fd bytes) 2024-09-21T15:50:22.266 INFO:teuthology.orchestra.run.smithi089.stdout:5648 read 0x46643a1 thru 0x466e5fc (0xa25c bytes) 2024-09-21T15:50:22.266 INFO:teuthology.orchestra.run.smithi089.stdout:5649 read 0x5023bc4 thru 0x5024c57 (0x1094 bytes) 2024-09-21T15:50:22.266 INFO:teuthology.orchestra.run.smithi089.stdout:5650 read 0x4f6915a thru 0x4f6dbbc (0x4a63 bytes) 2024-09-21T15:50:22.273 INFO:teuthology.orchestra.run.smithi089.stdout:5651 write 0xe56fe72 thru 0xe571e8f (0x201e bytes) 2024-09-21T15:50:22.281 INFO:teuthology.orchestra.run.smithi089.stdout:5652 write 0x5af4454 thru 0x5b000c4 (0xbc71 bytes) 2024-09-21T15:50:22.291 INFO:teuthology.orchestra.run.smithi089.stdout:5654 read 0xacada2b thru 0xacbb0f6 (0xd6cc bytes) 2024-09-21T15:50:22.291 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:22.291 INFO:teuthology.orchestra.run.smithi089.stdout:5658 write 0x66129d3 thru 0x6615c91 (0x32bf bytes) 2024-09-21T15:50:22.298 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:22.298 INFO:teuthology.orchestra.run.smithi089.stdout:5660 read 0xa341f7f thru 0xa34f6f2 (0xd774 bytes) 2024-09-21T15:50:22.298 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:22.298 INFO:teuthology.orchestra.run.smithi089.stdout:5662 read 0x6836b0b thru 0x683ee7c (0x8372 bytes) 2024-09-21T15:50:22.298 INFO:teuthology.orchestra.run.smithi089.stdout:5663 write 0xb1b3af1 thru 0xb1b64ea (0x29fa bytes) 2024-09-21T15:50:22.305 INFO:teuthology.orchestra.run.smithi089.stdout:5665 trunc from 0xe571e90 to 0xdb24ab8 2024-09-21T15:50:22.313 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:22.312+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:22.315 INFO:teuthology.orchestra.run.smithi089.stdout:5666 punch from 0xb31593f to 0xb323baa, (0xe26b bytes) 2024-09-21T15:50:22.315 INFO:teuthology.orchestra.run.smithi089.stdout:5667 writesame 0x3dda807 thru 0x3de49d4 data_size 0x2(0xa1ce bytes) 2024-09-21T15:50:22.331 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:22.331 INFO:teuthology.orchestra.run.smithi089.stdout:5669 punch from 0x5796007 to 0x579eebd, (0x8eb6 bytes) 2024-09-21T15:50:22.335 INFO:teuthology.orchestra.run.smithi089.stdout:5670 trunc from 0xdb24ab8 to 0xa917128 2024-09-21T15:50:22.347 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:22.347 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:22.347 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:50:22.347 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:50:22.347 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:22.356 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:50:22.359 INFO:teuthology.orchestra.run.smithi089.stdout:5671 read 0x50b11cd thru 0x50b9277 (0x80ab bytes) 2024-09-21T15:50:22.359 INFO:teuthology.orchestra.run.smithi089.stdout:5674 punch from 0x85d3dff to 0x85d69f0, (0x2bf1 bytes) 2024-09-21T15:50:22.359 INFO:teuthology.orchestra.run.smithi089.stdout:5675 write 0x37db096 thru 0x37e3b9d (0x8b08 bytes) 2024-09-21T15:50:22.371 INFO:teuthology.orchestra.run.smithi089.stdout:5678 punch from 0x261ee88 to 0x2622350, (0x34c8 bytes) 2024-09-21T15:50:22.371 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:22.377 INFO:teuthology.orchestra.run.smithi089.stdout:5680 write 0xbe3c515 thru 0xbe47c9d (0xb789 bytes) 2024-09-21T15:50:22.391 INFO:teuthology.orchestra.run.smithi089.stdout:5681 read 0x43d7a72 thru 0x43df118 (0x76a7 bytes) 2024-09-21T15:50:22.397 INFO:teuthology.orchestra.run.smithi089.stdout:5684 trunc from 0xbe47c9e to 0x68937a5 2024-09-21T15:50:22.413 INFO:teuthology.orchestra.run.smithi089.stdout:5686 read 0x3233a79 thru 0x323c8c5 (0x8e4d bytes) 2024-09-21T15:50:22.413 INFO:teuthology.orchestra.run.smithi089.stdout:5687 read 0x49f0f6c thru 0x49fee36 (0xdecb bytes) 2024-09-21T15:50:22.413 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:22.414 INFO:teuthology.orchestra.run.smithi089.stdout:5689 write 0x51440e2 thru 0x514c3b1 (0x82d0 bytes) 2024-09-21T15:50:22.414 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:22.412+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:22.431 INFO:teuthology.orchestra.run.smithi089.stdout:5690 flatten 2024-09-21T15:50:22.477 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:22.478 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:50:22.478 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:50:22.478 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:50:22.478 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.478 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62833 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 192512~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:22.478 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:50:22.478 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 45.333816349000003, 2024-09-21T15:50:22.478 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050387007999999997, 2024-09-21T15:50:22.478 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:22.478 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:22.478 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:22.478 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:22.479 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:22.479 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62833 2024-09-21T15:50:22.479 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.479 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:22.479 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.479 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:22.479 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:50:22.479 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.479 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.479 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.479 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:22.479 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:50:22.479 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.479 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.479 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.479 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:22.480 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:50:22.480 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:22.480 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.480 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.480 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:22.480 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142150+0000", 2024-09-21T15:50:22.480 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1999999999999999e-06 2024-09-21T15:50:22.480 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.480 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.480 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:22.480 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142153+0000", 2024-09-21T15:50:22.480 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7410000000000001e-06 2024-09-21T15:50:22.480 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.480 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.481 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:22.481 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142156+0000", 2024-09-21T15:50:22.481 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5739999999999998e-06 2024-09-21T15:50:22.481 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.481 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.481 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:22.481 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.192358+0000", 2024-09-21T15:50:22.481 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050202452000000002 2024-09-21T15:50:22.481 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.481 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.481 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:22.481 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.192435+0000", 2024-09-21T15:50:22.481 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6738999999999998e-05 2024-09-21T15:50:22.481 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.481 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.482 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:22.482 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.192536+0000", 2024-09-21T15:50:22.482 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101508 2024-09-21T15:50:22.482 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.482 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:22.482 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.482 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.482 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.482 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62849 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 208896~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:22.482 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.144181+0000", 2024-09-21T15:50:22.482 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 45.331784276999997, 2024-09-21T15:50:22.482 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050460390000000001, 2024-09-21T15:50:22.482 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:22.482 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:22.482 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:22.483 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:22.483 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:22.483 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62849 2024-09-21T15:50:22.483 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.483 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:22.483 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.483 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:22.483 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144181+0000", 2024-09-21T15:50:22.483 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.483 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.483 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.483 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:22.483 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144181+0000", 2024-09-21T15:50:22.483 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.483 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.484 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.484 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:22.484 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144181+0000", 2024-09-21T15:50:22.484 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:22.484 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.484 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.484 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:22.484 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144183+0000", 2024-09-21T15:50:22.484 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3860000000000001e-06 2024-09-21T15:50:22.484 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.484 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.484 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:22.484 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144189+0000", 2024-09-21T15:50:22.484 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0599999999999996e-06 2024-09-21T15:50:22.484 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.485 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.485 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:22.485 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144194+0000", 2024-09-21T15:50:22.485 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4379999999999997e-06 2024-09-21T15:50:22.485 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.485 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.485 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:22.485 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.194457+0000", 2024-09-21T15:50:22.485 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050263317000000002 2024-09-21T15:50:22.485 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.485 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.485 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:22.485 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.194548+0000", 2024-09-21T15:50:22.485 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0785000000000006e-05 2024-09-21T15:50:22.485 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.486 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.486 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:22.486 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.194642+0000", 2024-09-21T15:50:22.486 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3837000000000003e-05 2024-09-21T15:50:22.486 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.486 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:22.486 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.486 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.486 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.486 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62862 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 954368~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:22.486 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.146176+0000", 2024-09-21T15:50:22.486 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 45.329789214000002, 2024-09-21T15:50:22.486 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050442901999999998, 2024-09-21T15:50:22.486 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:22.486 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:22.487 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:22.487 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:22.487 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:22.487 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62862 2024-09-21T15:50:22.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.487 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:22.487 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.487 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:22.487 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146176+0000", 2024-09-21T15:50:22.487 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.487 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.487 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:22.487 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146176+0000", 2024-09-21T15:50:22.487 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.488 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.488 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:22.488 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146176+0000", 2024-09-21T15:50:22.488 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:22.488 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.488 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.488 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:22.488 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146178+0000", 2024-09-21T15:50:22.488 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4739999999999999e-06 2024-09-21T15:50:22.488 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.488 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.488 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:22.488 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146186+0000", 2024-09-21T15:50:22.488 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4989999999999997e-06 2024-09-21T15:50:22.488 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.489 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.489 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:22.489 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146191+0000", 2024-09-21T15:50:22.489 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7999999999999998e-06 2024-09-21T15:50:22.489 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.489 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.489 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:22.489 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196442+0000", 2024-09-21T15:50:22.489 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050251665000000001 2024-09-21T15:50:22.489 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.489 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.489 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:22.489 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196518+0000", 2024-09-21T15:50:22.489 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5677000000000001e-05 2024-09-21T15:50:22.489 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.490 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.490 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:22.490 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196619+0000", 2024-09-21T15:50:22.490 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101247 2024-09-21T15:50:22.490 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.490 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:22.490 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.490 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.490 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.490 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62863 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 937984~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:22.490 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.146195+0000", 2024-09-21T15:50:22.490 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 45.329771022999999, 2024-09-21T15:50:22.490 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050642305999999998, 2024-09-21T15:50:22.490 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:22.491 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:22.491 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:22.491 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:22.491 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:22.491 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62863 2024-09-21T15:50:22.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.491 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:22.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.491 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:22.491 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146195+0000", 2024-09-21T15:50:22.491 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.491 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:22.491 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146195+0000", 2024-09-21T15:50:22.491 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.492 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.492 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.492 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:22.492 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146194+0000", 2024-09-21T15:50:22.492 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:22.492 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.492 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.492 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:22.492 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146196+0000", 2024-09-21T15:50:22.492 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1900000000000002e-06 2024-09-21T15:50:22.492 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.492 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.492 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:22.492 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146202+0000", 2024-09-21T15:50:22.492 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.57e-06 2024-09-21T15:50:22.492 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.493 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.493 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:22.493 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146206+0000", 2024-09-21T15:50:22.493 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5079999999999999e-06 2024-09-21T15:50:22.493 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.493 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.493 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:22.493 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196655+0000", 2024-09-21T15:50:22.493 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050448973000000001 2024-09-21T15:50:22.493 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.493 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.493 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:22.493 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196730+0000", 2024-09-21T15:50:22.493 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4728999999999996e-05 2024-09-21T15:50:22.493 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.494 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.494 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:22.494 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196837+0000", 2024-09-21T15:50:22.494 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000106735 2024-09-21T15:50:22.494 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.494 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:22.494 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.494 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.494 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.494 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62871 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 307200~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:22.494 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148134+0000", 2024-09-21T15:50:22.494 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 45.327831814, 2024-09-21T15:50:22.494 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050310001, 2024-09-21T15:50:22.494 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:22.495 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:22.495 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:22.495 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:22.495 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:22.495 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62871 2024-09-21T15:50:22.495 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.495 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:22.495 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.495 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:22.495 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148134+0000", 2024-09-21T15:50:22.495 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.495 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.495 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.495 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:22.495 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148134+0000", 2024-09-21T15:50:22.495 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.496 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.496 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.496 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:22.496 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148133+0000", 2024-09-21T15:50:22.496 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:22.496 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.496 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.496 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:22.496 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148135+0000", 2024-09-21T15:50:22.496 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9829999999999999e-06 2024-09-21T15:50:22.496 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.496 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.496 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:22.496 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148141+0000", 2024-09-21T15:50:22.496 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5439999999999998e-06 2024-09-21T15:50:22.497 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.497 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.497 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:22.497 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148145+0000", 2024-09-21T15:50:22.497 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4229999999999998e-06 2024-09-21T15:50:22.497 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.497 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.497 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:22.497 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198294+0000", 2024-09-21T15:50:22.497 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050148558000000003 2024-09-21T15:50:22.497 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.497 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.497 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:22.497 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198372+0000", 2024-09-21T15:50:22.497 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8163000000000004e-05 2024-09-21T15:50:22.498 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.498 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.498 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:22.498 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198444+0000", 2024-09-21T15:50:22.498 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1620999999999997e-05 2024-09-21T15:50:22.498 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.498 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:22.498 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.498 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.498 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.498 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62872 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 290816~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:22.498 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148149+0000", 2024-09-21T15:50:22.498 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 45.327816347000002, 2024-09-21T15:50:22.498 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.05046991, 2024-09-21T15:50:22.499 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:22.499 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:22.499 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:22.499 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:22.499 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:22.499 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62872 2024-09-21T15:50:22.499 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.499 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:22.499 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.499 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:22.499 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148149+0000", 2024-09-21T15:50:22.499 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.499 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.499 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.500 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:22.500 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148149+0000", 2024-09-21T15:50:22.500 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.500 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.500 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.500 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:22.500 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148149+0000", 2024-09-21T15:50:22.500 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:22.500 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.500 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.500 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:22.500 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148157+0000", 2024-09-21T15:50:22.500 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4749999999999993e-06 2024-09-21T15:50:22.500 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.500 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.501 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:22.501 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148163+0000", 2024-09-21T15:50:22.501 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8250000000000001e-06 2024-09-21T15:50:22.501 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.501 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.501 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:22.501 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148168+0000", 2024-09-21T15:50:22.501 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4730000000000002e-06 2024-09-21T15:50:22.501 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.501 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.501 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:22.501 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198471+0000", 2024-09-21T15:50:22.501 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050302974 2024-09-21T15:50:22.501 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.501 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.502 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:22.502 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198555+0000", 2024-09-21T15:50:22.502 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4783999999999993e-05 2024-09-21T15:50:22.502 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.502 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.502 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:22.502 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198619+0000", 2024-09-21T15:50:22.502 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3751000000000007e-05 2024-09-21T15:50:22.502 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.502 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:22.502 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.502 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.502 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.502 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62873 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 315392~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:22.503 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148172+0000", 2024-09-21T15:50:22.503 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 45.327794056000002, 2024-09-21T15:50:22.503 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050707998999999997, 2024-09-21T15:50:22.503 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:22.503 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:22.503 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:22.503 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:22.503 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:22.503 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62873 2024-09-21T15:50:22.503 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.503 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:22.503 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.503 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:22.503 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148172+0000", 2024-09-21T15:50:22.503 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.504 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.504 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.504 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:22.504 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148172+0000", 2024-09-21T15:50:22.504 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.504 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.504 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.504 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:22.504 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148171+0000", 2024-09-21T15:50:22.504 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:22.504 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.504 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.504 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:22.504 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148173+0000", 2024-09-21T15:50:22.505 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2340000000000001e-06 2024-09-21T15:50:22.505 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.505 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.505 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:22.505 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148179+0000", 2024-09-21T15:50:22.505 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9340000000000003e-06 2024-09-21T15:50:22.505 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.505 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.505 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:22.505 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148186+0000", 2024-09-21T15:50:22.505 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2349999999999996e-06 2024-09-21T15:50:22.505 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.505 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.505 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:22.505 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198684+0000", 2024-09-21T15:50:22.505 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050498890999999997 2024-09-21T15:50:22.506 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.506 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.506 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:22.506 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198778+0000", 2024-09-21T15:50:22.506 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3134999999999995e-05 2024-09-21T15:50:22.506 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.506 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.506 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:22.506 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198880+0000", 2024-09-21T15:50:22.506 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101906 2024-09-21T15:50:22.506 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.506 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:22.506 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.506 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.507 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.507 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62874 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 323584~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:22.507 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148211+0000", 2024-09-21T15:50:22.507 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 45.327754296000002, 2024-09-21T15:50:22.507 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050887766000000001, 2024-09-21T15:50:22.507 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:22.507 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:22.507 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:22.507 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:22.507 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:22.507 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62874 2024-09-21T15:50:22.507 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.507 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:22.507 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.507 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:22.508 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148211+0000", 2024-09-21T15:50:22.508 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.508 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.508 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.508 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:22.508 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148211+0000", 2024-09-21T15:50:22.508 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.508 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.508 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.508 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:22.508 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148211+0000", 2024-09-21T15:50:22.508 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:22.508 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.508 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.508 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:22.509 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148213+0000", 2024-09-21T15:50:22.509 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6860000000000002e-06 2024-09-21T15:50:22.509 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.509 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.509 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:22.509 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148220+0000", 2024-09-21T15:50:22.509 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4740000000000002e-06 2024-09-21T15:50:22.509 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.509 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.509 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:22.509 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148226+0000", 2024-09-21T15:50:22.509 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0120000000000002e-06 2024-09-21T15:50:22.509 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.509 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.509 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:22.509 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198940+0000", 2024-09-21T15:50:22.510 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050714412 2024-09-21T15:50:22.510 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.510 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.510 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:22.510 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198995+0000", 2024-09-21T15:50:22.510 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4863e-05 2024-09-21T15:50:22.510 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.510 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.510 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:22.510 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.199099+0000", 2024-09-21T15:50:22.510 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010383299999999999 2024-09-21T15:50:22.510 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.510 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:22.510 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.510 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.511 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.511 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62877 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 995328~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:22.511 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148230+0000", 2024-09-21T15:50:22.511 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 45.327735435999998, 2024-09-21T15:50:22.511 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.051108949000000001, 2024-09-21T15:50:22.511 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:22.511 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:22.511 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:22.511 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:22.511 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:22.511 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62877 2024-09-21T15:50:22.511 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.511 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:22.511 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.512 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:22.512 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148230+0000", 2024-09-21T15:50:22.512 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.512 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.512 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.512 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:22.512 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148230+0000", 2024-09-21T15:50:22.512 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.512 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.512 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.512 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:22.512 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148230+0000", 2024-09-21T15:50:22.512 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:22.512 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.512 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.513 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:22.513 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148232+0000", 2024-09-21T15:50:22.513 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1220000000000002e-06 2024-09-21T15:50:22.513 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.513 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.513 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:22.513 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148238+0000", 2024-09-21T15:50:22.513 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.789e-06 2024-09-21T15:50:22.513 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.513 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.513 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:22.513 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148242+0000", 2024-09-21T15:50:22.513 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7369999999999997e-06 2024-09-21T15:50:22.513 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.513 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.513 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:22.514 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.199201+0000", 2024-09-21T15:50:22.514 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050959088 2024-09-21T15:50:22.514 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.514 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.514 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:22.514 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.199285+0000", 2024-09-21T15:50:22.514 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3147999999999997e-05 2024-09-21T15:50:22.514 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.514 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.514 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:22.514 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.199339+0000", 2024-09-21T15:50:22.514 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4484999999999998e-05 2024-09-21T15:50:22.515 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.515 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:22.515 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.515 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.515 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.515 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62886 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 1044480~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:22.515 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150138+0000", 2024-09-21T15:50:22.515 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 45.325827768000003, 2024-09-21T15:50:22.515 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050345527000000001, 2024-09-21T15:50:22.515 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:22.515 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:22.515 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:22.515 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:22.515 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:22.516 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62886 2024-09-21T15:50:22.516 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.516 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:22.516 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.516 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:22.516 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150138+0000", 2024-09-21T15:50:22.516 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.516 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.516 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.516 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:22.516 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150138+0000", 2024-09-21T15:50:22.516 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.516 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.516 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.516 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:22.516 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150137+0000", 2024-09-21T15:50:22.517 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:22.517 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.517 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.517 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:22.517 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150140+0000", 2024-09-21T15:50:22.517 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2460000000000002e-06 2024-09-21T15:50:22.517 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.517 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.517 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:22.517 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150145+0000", 2024-09-21T15:50:22.517 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6520000000000003e-06 2024-09-21T15:50:22.517 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.517 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.517 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:22.517 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150150+0000", 2024-09-21T15:50:22.518 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5310000000000003e-06 2024-09-21T15:50:22.518 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.518 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.518 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:22.518 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200356+0000", 2024-09-21T15:50:22.518 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050205759000000003 2024-09-21T15:50:22.518 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.518 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.518 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:22.518 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200415+0000", 2024-09-21T15:50:22.518 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9719999999999997e-05 2024-09-21T15:50:22.518 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.518 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.518 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:22.518 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200483+0000", 2024-09-21T15:50:22.519 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7959000000000004e-05 2024-09-21T15:50:22.519 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.519 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:22.519 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.519 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.519 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.519 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62888 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 331776~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:22.519 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150154+0000", 2024-09-21T15:50:22.519 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 45.325811842, 2024-09-21T15:50:22.519 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050483169000000001, 2024-09-21T15:50:22.519 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:22.519 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:22.519 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:22.519 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:22.520 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:22.520 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62888 2024-09-21T15:50:22.520 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.520 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:22.520 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.520 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:22.520 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150154+0000", 2024-09-21T15:50:22.520 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.520 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.520 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.520 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:22.520 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150154+0000", 2024-09-21T15:50:22.520 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.520 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.521 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.521 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:22.521 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150153+0000", 2024-09-21T15:50:22.521 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:22.521 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.521 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.521 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:22.521 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150155+0000", 2024-09-21T15:50:22.521 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9920000000000002e-06 2024-09-21T15:50:22.521 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.521 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.521 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:22.521 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150161+0000", 2024-09-21T15:50:22.521 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5439999999999998e-06 2024-09-21T15:50:22.522 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.522 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.522 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:22.522 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150165+0000", 2024-09-21T15:50:22.522 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3460000000000004e-06 2024-09-21T15:50:22.522 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.522 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.522 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:22.522 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200502+0000", 2024-09-21T15:50:22.522 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050337150999999997 2024-09-21T15:50:22.522 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.522 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.522 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:22.522 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200575+0000", 2024-09-21T15:50:22.522 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2250999999999997e-05 2024-09-21T15:50:22.523 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.523 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.523 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:22.523 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200637+0000", 2024-09-21T15:50:22.523 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2195999999999999e-05 2024-09-21T15:50:22.523 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.523 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:22.523 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.523 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.523 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.523 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62889 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 339968~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:22.523 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150169+0000", 2024-09-21T15:50:22.523 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 45.325796343, 2024-09-21T15:50:22.523 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050629678999999997, 2024-09-21T15:50:22.523 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:22.524 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:22.524 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:22.524 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:22.524 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:22.524 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62889 2024-09-21T15:50:22.524 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.524 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:22.524 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.524 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:22.524 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150169+0000", 2024-09-21T15:50:22.524 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.524 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.524 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.524 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:22.525 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150169+0000", 2024-09-21T15:50:22.525 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.525 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.525 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.525 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:22.525 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150169+0000", 2024-09-21T15:50:22.525 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:22.525 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.525 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.525 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:22.525 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150171+0000", 2024-09-21T15:50:22.525 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3910000000000001e-06 2024-09-21T15:50:22.525 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.525 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.525 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:22.526 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150177+0000", 2024-09-21T15:50:22.526 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4079999999999997e-06 2024-09-21T15:50:22.526 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.526 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.526 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:22.526 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150182+0000", 2024-09-21T15:50:22.526 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3890000000000004e-06 2024-09-21T15:50:22.526 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.526 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.526 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:22.526 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200664+0000", 2024-09-21T15:50:22.526 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050482212999999998 2024-09-21T15:50:22.526 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.526 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.527 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:22.527 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200738+0000", 2024-09-21T15:50:22.527 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4114000000000002e-05 2024-09-21T15:50:22.527 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.527 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.527 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:22.527 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200799+0000", 2024-09-21T15:50:22.527 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0503e-05 2024-09-21T15:50:22.527 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.527 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:22.527 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.527 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.527 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.527 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62891 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 348160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:22.527 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150186+0000", 2024-09-21T15:50:22.528 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 45.325779201000003, 2024-09-21T15:50:22.528 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050794552, 2024-09-21T15:50:22.528 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:22.528 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:22.528 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:22.528 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:22.528 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:22.528 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62891 2024-09-21T15:50:22.528 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.528 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:22.528 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.528 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:22.528 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150186+0000", 2024-09-21T15:50:22.528 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.528 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.529 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.529 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:22.529 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150186+0000", 2024-09-21T15:50:22.529 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.529 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.529 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.529 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:22.529 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150186+0000", 2024-09-21T15:50:22.529 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:22.529 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.529 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.529 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:22.529 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150193+0000", 2024-09-21T15:50:22.529 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6830000000000002e-06 2024-09-21T15:50:22.530 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.530 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.530 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:22.530 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150199+0000", 2024-09-21T15:50:22.530 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9859999999999999e-06 2024-09-21T15:50:22.530 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.530 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.530 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:22.530 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150203+0000", 2024-09-21T15:50:22.530 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5920000000000002e-06 2024-09-21T15:50:22.530 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.530 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.530 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:22.530 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200835+0000", 2024-09-21T15:50:22.531 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050631783 2024-09-21T15:50:22.531 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.531 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.531 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:22.531 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200894+0000", 2024-09-21T15:50:22.531 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8560999999999997e-05 2024-09-21T15:50:22.531 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.531 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.531 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:22.531 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200981+0000", 2024-09-21T15:50:22.531 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7305999999999994e-05 2024-09-21T15:50:22.531 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.531 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:22.531 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.532 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.532 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.532 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62892 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 356352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:22.532 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150208+0000", 2024-09-21T15:50:22.532 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 45.325757813000003, 2024-09-21T15:50:22.532 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050951769000000001, 2024-09-21T15:50:22.532 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:22.532 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:22.532 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:22.532 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:22.532 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:22.532 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62892 2024-09-21T15:50:22.532 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.532 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:22.532 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.533 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:22.533 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150208+0000", 2024-09-21T15:50:22.533 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.533 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.533 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.533 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:22.533 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150208+0000", 2024-09-21T15:50:22.533 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.533 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.533 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.533 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:22.533 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150207+0000", 2024-09-21T15:50:22.533 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:22.533 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.533 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.534 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:22.534 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150210+0000", 2024-09-21T15:50:22.534 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.238e-06 2024-09-21T15:50:22.534 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.534 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.534 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:22.534 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150215+0000", 2024-09-21T15:50:22.534 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6350000000000003e-06 2024-09-21T15:50:22.534 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.534 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.534 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:22.534 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150220+0000", 2024-09-21T15:50:22.534 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.544e-06 2024-09-21T15:50:22.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.535 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.535 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:22.535 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201014+0000", 2024-09-21T15:50:22.535 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050793932999999999 2024-09-21T15:50:22.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.535 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.535 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:22.535 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201092+0000", 2024-09-21T15:50:22.535 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7961999999999997e-05 2024-09-21T15:50:22.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.535 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.535 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:22.535 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201160+0000", 2024-09-21T15:50:22.535 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7836000000000006e-05 2024-09-21T15:50:22.536 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.536 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:22.536 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.536 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.536 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.536 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62894 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 364544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:22.536 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150820+0000", 2024-09-21T15:50:22.536 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 45.325145612999997, 2024-09-21T15:50:22.536 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050489196, 2024-09-21T15:50:22.536 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:22.536 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:22.536 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:22.536 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:22.536 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:22.537 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62894 2024-09-21T15:50:22.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.537 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:22.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.537 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:22.537 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150820+0000", 2024-09-21T15:50:22.537 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.537 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:22.537 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150820+0000", 2024-09-21T15:50:22.537 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.537 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:22.538 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150223+0000", 2024-09-21T15:50:22.538 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9994035 2024-09-21T15:50:22.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.538 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.538 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:22.538 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150825+0000", 2024-09-21T15:50:22.538 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00060137099999999996 2024-09-21T15:50:22.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.538 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.538 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:22.538 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150837+0000", 2024-09-21T15:50:22.538 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2503999999999999e-05 2024-09-21T15:50:22.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.539 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:22.539 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150844+0000", 2024-09-21T15:50:22.539 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6900000000000003e-06 2024-09-21T15:50:22.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.539 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:22.539 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201187+0000", 2024-09-21T15:50:22.539 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050342973999999999 2024-09-21T15:50:22.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.539 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:22.539 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201250+0000", 2024-09-21T15:50:22.539 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3274000000000004e-05 2024-09-21T15:50:22.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.540 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:22.540 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201309+0000", 2024-09-21T15:50:22.540 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9058000000000003e-05 2024-09-21T15:50:22.540 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.540 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:22.540 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.540 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.540 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.540 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62895 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 380928~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:22.540 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150851+0000", 2024-09-21T15:50:22.540 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 45.325115017000002, 2024-09-21T15:50:22.540 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050611097000000001, 2024-09-21T15:50:22.540 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:22.540 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:22.540 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:22.540 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:22.541 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:22.541 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62895 2024-09-21T15:50:22.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.541 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:22.541 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.541 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:22.541 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150851+0000", 2024-09-21T15:50:22.541 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.541 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.541 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:22.541 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150851+0000", 2024-09-21T15:50:22.541 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.542 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:22.542 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150850+0000", 2024-09-21T15:50:22.542 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:22.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.542 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:22.542 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150853+0000", 2024-09-21T15:50:22.542 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1039999999999998e-06 2024-09-21T15:50:22.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.542 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:22.542 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150860+0000", 2024-09-21T15:50:22.542 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3180000000000001e-06 2024-09-21T15:50:22.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.543 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.543 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:22.543 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150866+0000", 2024-09-21T15:50:22.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1599999999999997e-06 2024-09-21T15:50:22.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.543 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.543 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:22.543 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201332+0000", 2024-09-21T15:50:22.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050466100999999999 2024-09-21T15:50:22.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.543 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.543 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:22.543 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201392+0000", 2024-09-21T15:50:22.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0053999999999998e-05 2024-09-21T15:50:22.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.544 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.544 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:22.544 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201462+0000", 2024-09-21T15:50:22.544 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9861000000000001e-05 2024-09-21T15:50:22.544 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.544 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:22.544 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.544 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.544 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.544 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62896 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 372736~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:22.544 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150870+0000", 2024-09-21T15:50:22.544 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 45.325095236999999, 2024-09-21T15:50:22.544 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050753292999999998, 2024-09-21T15:50:22.544 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:22.545 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:22.545 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:22.545 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:22.545 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:22.545 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62896 2024-09-21T15:50:22.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.545 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:22.545 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.545 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:22.545 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150870+0000", 2024-09-21T15:50:22.545 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.545 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.545 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:22.545 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150870+0000", 2024-09-21T15:50:22.546 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.546 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.546 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:22.546 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150870+0000", 2024-09-21T15:50:22.546 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:22.546 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.546 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:22.546 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150872+0000", 2024-09-21T15:50:22.546 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.644e-06 2024-09-21T15:50:22.546 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.546 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:22.547 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150879+0000", 2024-09-21T15:50:22.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3509999999999998e-06 2024-09-21T15:50:22.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.547 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:22.547 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150884+0000", 2024-09-21T15:50:22.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9150000000000004e-06 2024-09-21T15:50:22.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.547 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:22.547 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201487+0000", 2024-09-21T15:50:22.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050603276000000003 2024-09-21T15:50:22.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.548 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.548 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:22.548 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201557+0000", 2024-09-21T15:50:22.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9847999999999999e-05 2024-09-21T15:50:22.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.548 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.548 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:22.548 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201624+0000", 2024-09-21T15:50:22.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6730999999999995e-05 2024-09-21T15:50:22.548 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.548 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:22.548 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.548 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.549 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62897 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 389120~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:22.549 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.151044+0000", 2024-09-21T15:50:22.549 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 45.324921197000002, 2024-09-21T15:50:22.549 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050720531999999999, 2024-09-21T15:50:22.549 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:22.549 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:22.549 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:22.549 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:22.549 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:22.549 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62897 2024-09-21T15:50:22.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.549 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:22.549 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.549 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:22.550 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151044+0000", 2024-09-21T15:50:22.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.550 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.550 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:22.550 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151044+0000", 2024-09-21T15:50:22.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.550 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.550 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:22.550 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150888+0000", 2024-09-21T15:50:22.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9998431 2024-09-21T15:50:22.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.550 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:22.551 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151048+0000", 2024-09-21T15:50:22.551 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00016077299999999999 2024-09-21T15:50:22.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:22.551 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151058+0000", 2024-09-21T15:50:22.551 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.465e-06 2024-09-21T15:50:22.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:22.551 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151064+0000", 2024-09-21T15:50:22.551 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3829999999999997e-06 2024-09-21T15:50:22.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.552 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:22.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201648+0000", 2024-09-21T15:50:22.552 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050583597000000001 2024-09-21T15:50:22.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.552 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:22.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201711+0000", 2024-09-21T15:50:22.552 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3616999999999998e-05 2024-09-21T15:50:22.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.552 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:22.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201765+0000", 2024-09-21T15:50:22.552 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3464000000000002e-05 2024-09-21T15:50:22.552 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.552 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:22.553 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.553 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.553 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.553 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:118351 3.4 3:25980f86:::rbd_data.1133545d3fa6.000000000000005a:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-09-21T15:50:22.553 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:58.975795+0000", 2024-09-21T15:50:22.553 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.500170563000001, 2024-09-21T15:50:22.553 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.99016351599999997, 2024-09-21T15:50:22.553 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:22.553 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:22.553 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:22.553 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:22.553 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:22.553 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 118351 2024-09-21T15:50:22.553 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.553 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:22.554 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.554 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:22.554 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975795+0000", 2024-09-21T15:50:22.554 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.554 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.554 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.554 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:22.554 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975795+0000", 2024-09-21T15:50:22.554 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.554 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.554 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.554 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:22.554 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975793+0000", 2024-09-21T15:50:22.554 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:50:22.554 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.554 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.555 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:22.555 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975800+0000", 2024-09-21T15:50:22.555 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8220000000000001e-06 2024-09-21T15:50:22.555 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.555 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.555 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:22.555 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975814+0000", 2024-09-21T15:50:22.555 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3893000000000001e-05 2024-09-21T15:50:22.555 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.555 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.555 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:22.555 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975821+0000", 2024-09-21T15:50:22.555 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1319999999999996e-06 2024-09-21T15:50:22.555 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.555 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.555 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:22.556 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975878+0000", 2024-09-21T15:50:22.556 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7238999999999999e-05 2024-09-21T15:50:22.556 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.556 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.556 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:50:22.556 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975889+0000", 2024-09-21T15:50:22.556 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0509000000000001e-05 2024-09-21T15:50:22.556 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.556 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.556 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:22.556 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.964573+0000", 2024-09-21T15:50:22.556 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.98868396300000005 2024-09-21T15:50:22.556 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.556 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.556 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:22.556 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.964672+0000", 2024-09-21T15:50:22.557 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9601999999999999e-05 2024-09-21T15:50:22.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.557 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.557 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-09-21T15:50:22.557 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.964758+0000", 2024-09-21T15:50:22.557 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5537000000000005e-05 2024-09-21T15:50:22.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.557 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.557 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:22.557 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965847+0000", 2024-09-21T15:50:22.557 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001089404 2024-09-21T15:50:22.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.557 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.557 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:22.557 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965890+0000", 2024-09-21T15:50:22.557 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2521999999999997e-05 2024-09-21T15:50:22.558 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.558 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.558 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:22.558 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965934+0000", 2024-09-21T15:50:22.558 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4489e-05 2024-09-21T15:50:22.558 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.558 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.558 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:22.558 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965959+0000", 2024-09-21T15:50:22.558 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4301999999999999e-05 2024-09-21T15:50:22.558 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.558 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:22.558 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.558 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.558 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.558 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:118353 3.4 3:23841e18:::rbd_data.1133545d3fa6.0000000000000061:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-09-21T15:50:22.559 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:58.975836+0000", 2024-09-21T15:50:22.559 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.500129063999999, 2024-09-21T15:50:22.559 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.99077746700000002, 2024-09-21T15:50:22.559 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:22.559 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:22.559 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:22.559 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:22.559 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:22.559 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 118353 2024-09-21T15:50:22.559 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.559 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:22.559 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.559 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:22.559 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975836+0000", 2024-09-21T15:50:22.559 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.560 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.560 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:22.560 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975836+0000", 2024-09-21T15:50:22.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.560 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.560 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:22.560 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975836+0000", 2024-09-21T15:50:22.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:22.560 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.560 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:22.560 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975839+0000", 2024-09-21T15:50:22.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6129999999999998e-06 2024-09-21T15:50:22.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.561 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.561 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:22.561 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975844+0000", 2024-09-21T15:50:22.561 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5130000000000004e-06 2024-09-21T15:50:22.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.561 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.561 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:22.561 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975850+0000", 2024-09-21T15:50:22.561 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3179999999999997e-06 2024-09-21T15:50:22.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.561 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.561 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:22.561 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975915+0000", 2024-09-21T15:50:22.561 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5168999999999999e-05 2024-09-21T15:50:22.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.562 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.562 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:50:22.562 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975922+0000", 2024-09-21T15:50:22.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2749999999999998e-06 2024-09-21T15:50:22.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.562 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.562 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:22.562 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965431+0000", 2024-09-21T15:50:22.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.98950864699999996 2024-09-21T15:50:22.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.562 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.562 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:22.562 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965535+0000", 2024-09-21T15:50:22.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000103949 2024-09-21T15:50:22.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.563 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-09-21T15:50:22.563 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965607+0000", 2024-09-21T15:50:22.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2478000000000006e-05 2024-09-21T15:50:22.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.563 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:22.563 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.966205+0000", 2024-09-21T15:50:22.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00059775200000000001 2024-09-21T15:50:22.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.563 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:22.563 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.966562+0000", 2024-09-21T15:50:22.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00035732199999999998 2024-09-21T15:50:22.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:22.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.966593+0000", 2024-09-21T15:50:22.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0642999999999998e-05 2024-09-21T15:50:22.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:22.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.966614+0000", 2024-09-21T15:50:22.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1149999999999999e-05 2024-09-21T15:50:22.564 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.564 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:22.564 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.564 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.564 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:22.564 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:22.566 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:50:22.566 INFO:teuthology.orchestra.run.smithi089.stdout:5691 writesame 0xd7831d4 thru 0xd784eee data_size 0x1(0x1d1b bytes) 2024-09-21T15:50:22.567 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:22.564+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:22.577 INFO:teuthology.orchestra.run.smithi089.stdout:5692 trunc from 0xd784eef to 0x293b1f8 2024-09-21T15:50:22.606 INFO:teuthology.orchestra.run.smithi089.stdout:5694 read 0x1068822 thru 0x1068f39 (0x718 bytes) 2024-09-21T15:50:22.616 INFO:teuthology.orchestra.run.smithi089.stdout:5695 write 0x634ae15 thru 0x6358f4f (0xe13b bytes) 2024-09-21T15:50:22.629 INFO:teuthology.orchestra.run.smithi089.stdout:5697 read 0x38bb45a thru 0x38c7ce5 (0xc88c bytes) 2024-09-21T15:50:22.638 INFO:teuthology.orchestra.run.smithi089.stdout:5698 write 0x96a4f44 thru 0x96ac7ed (0x78aa bytes) 2024-09-21T15:50:22.651 INFO:teuthology.orchestra.run.smithi089.stdout:5699 read 0x2715a4d thru 0x27224d1 (0xca85 bytes) 2024-09-21T15:50:22.651 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:22.651 INFO:teuthology.orchestra.run.smithi089.stdout:5702 writesame 0x8a0bf86 thru 0x8a10df1 data_size 0x2a(0x4e6c bytes) 2024-09-21T15:50:22.664 INFO:teuthology.orchestra.run.smithi089.stdout:5703 punch from 0x1c8371f to 0x1c850ae, (0x198f bytes) 2024-09-21T15:50:22.664 INFO:teuthology.orchestra.run.smithi089.stdout:5708 punch from 0x1dc196e to 0x1dc76cd, (0x5d5f bytes) 2024-09-21T15:50:22.664 INFO:teuthology.orchestra.run.smithi089.stdout:5709 read 0x2b0bf49 thru 0x2b1979b (0xd853 bytes) 2024-09-21T15:50:22.664 INFO:teuthology.orchestra.run.smithi089.stdout:5710 read 0x2953ada thru 0x2959afd (0x6024 bytes) 2024-09-21T15:50:22.664 INFO:teuthology.orchestra.run.smithi089.stdout:5711 punch from 0x4f3d3ac to 0x4f41900, (0x4554 bytes) 2024-09-21T15:50:22.664 INFO:teuthology.orchestra.run.smithi089.stdout:5712 read 0x252568b thru 0x2532bbc (0xd532 bytes) 2024-09-21T15:50:22.664 INFO:teuthology.orchestra.run.smithi089.stdout:5713 writesame 0x23e50a5 thru 0x23ecfe8 data_size 0xb4(0x7f44 bytes) 2024-09-21T15:50:22.666 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:22.665+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:22.681 INFO:teuthology.orchestra.run.smithi089.stdout:5714 writesame 0x392bd3a thru 0x393992e data_size 0x1(0xdbf5 bytes) 2024-09-21T15:50:22.686 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:22.687 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:22.687 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:50:22.687 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:22.696 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:50:22.704 INFO:teuthology.orchestra.run.smithi089.stdout:5715 writesame 0x2dbdaee thru 0x2dc9326 data_size 0x1(0xb839 bytes) 2024-09-21T15:50:22.721 INFO:teuthology.orchestra.run.smithi089.stdout:5716 read 0x8e8ca13 thru 0x8e9b20a (0xe7f8 bytes) 2024-09-21T15:50:22.721 INFO:teuthology.orchestra.run.smithi089.stdout:5718 read 0x88d9313 thru 0x88ddff0 (0x4cde bytes) 2024-09-21T15:50:22.730 INFO:teuthology.orchestra.run.smithi089.stdout:5720 trunc from 0x96ac7ee to 0x2340ea9 2024-09-21T15:50:22.767 INFO:teuthology.orchestra.run.smithi089.stdout:5721 clone 41 order 22 su 65536 sc 13 2024-09-21T15:50:22.767 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:22.766+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:22.811 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:22.811 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:22.811 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:50:22.812 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:50:22.812 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:22.820 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:50:22.904 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:22.903+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:22.965 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:22.965 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:50:22.965 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:50:22.965 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:50:22.965 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.965 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:984 3.0 e27/22 3:025e53e7:::rbd_data.1133f3dfbd6b.0000000000000008:head v 27'21, mlcod=27'19)", 2024-09-21T15:50:22.965 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.619393+0000", 2024-09-21T15:50:22.965 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 70.343631275999996, 2024-09-21T15:50:22.965 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003401837, 2024-09-21T15:50:22.965 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:22.965 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:22.966 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:22.966 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.966 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:22.966 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619393+0000", 2024-09-21T15:50:22.966 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.966 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.966 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.966 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:22.966 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619393+0000", 2024-09-21T15:50:22.966 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.966 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.966 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.966 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:22.967 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619392+0000", 2024-09-21T15:50:22.967 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:22.967 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.967 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.967 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:22.967 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619439+0000", 2024-09-21T15:50:22.967 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7170999999999999e-05 2024-09-21T15:50:22.967 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.967 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.967 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:22.967 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619452+0000", 2024-09-21T15:50:22.967 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2986999999999999e-05 2024-09-21T15:50:22.967 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.967 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.967 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:22.968 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619459+0000", 2024-09-21T15:50:22.968 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6109999999999999e-06 2024-09-21T15:50:22.968 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.968 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.968 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:22.968 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619518+0000", 2024-09-21T15:50:22.968 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9117999999999999e-05 2024-09-21T15:50:22.968 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.968 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.968 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:22.968 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619532+0000", 2024-09-21T15:50:22.968 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4622e-05 2024-09-21T15:50:22.968 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.968 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.968 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:22.969 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.622760+0000", 2024-09-21T15:50:22.969 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032278839999999999 2024-09-21T15:50:22.969 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.969 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.969 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:22.969 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.622794+0000", 2024-09-21T15:50:22.969 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4144999999999999e-05 2024-09-21T15:50:22.969 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.969 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:22.969 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.969 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.969 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.969 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:4957 3.9 e29/22 3:95c5ed95:::rbd_data.11335fa2fe69.0000000000000018:head v 29'4, mlcod=26'2)", 2024-09-21T15:50:22.969 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:14.453951+0000", 2024-09-21T15:50:22.969 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 68.509072844000002, 2024-09-21T15:50:22.970 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0043804559999999996, 2024-09-21T15:50:22.970 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:22.970 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:22.970 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:22.970 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.970 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:22.970 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.453951+0000", 2024-09-21T15:50:22.970 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.970 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.970 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.970 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:22.970 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.453951+0000", 2024-09-21T15:50:22.970 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.971 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.971 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.971 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:22.971 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.453950+0000", 2024-09-21T15:50:22.971 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:22.971 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.971 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.971 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:22.971 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454495+0000", 2024-09-21T15:50:22.971 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00054437799999999998 2024-09-21T15:50:22.971 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.971 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.971 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:22.971 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454508+0000", 2024-09-21T15:50:22.971 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3475000000000001e-05 2024-09-21T15:50:22.972 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.972 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.972 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:22.972 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454515+0000", 2024-09-21T15:50:22.972 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8990000000000004e-06 2024-09-21T15:50:22.972 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.972 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.972 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:22.972 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454581+0000", 2024-09-21T15:50:22.972 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5987000000000004e-05 2024-09-21T15:50:22.972 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.972 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.972 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:22.972 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454609+0000", 2024-09-21T15:50:22.972 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7466000000000001e-05 2024-09-21T15:50:22.972 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.973 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.973 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:22.973 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.458303+0000", 2024-09-21T15:50:22.973 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003693979 2024-09-21T15:50:22.973 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.973 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.973 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:22.973 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.458331+0000", 2024-09-21T15:50:22.973 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8932999999999999e-05 2024-09-21T15:50:22.973 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.973 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:22.973 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.973 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.973 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.973 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13420 3.0 e31/22 3:038b8840:::rbd_data.1133dc5adfdf.0000000000000035:head v 31'49, mlcod=31'47)", 2024-09-21T15:50:22.974 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.209233+0000", 2024-09-21T15:50:22.974 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 65.753791227999997, 2024-09-21T15:50:22.974 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0042624960000000002, 2024-09-21T15:50:22.974 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:22.974 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:22.974 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:22.974 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.974 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:22.974 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209233+0000", 2024-09-21T15:50:22.974 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.974 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.974 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.974 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:22.974 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209233+0000", 2024-09-21T15:50:22.974 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.975 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.975 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.975 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:22.975 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209232+0000", 2024-09-21T15:50:22.975 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:22.975 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.975 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.975 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:22.975 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209251+0000", 2024-09-21T15:50:22.975 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9304e-05 2024-09-21T15:50:22.975 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.975 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.975 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:22.975 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209265+0000", 2024-09-21T15:50:22.976 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3532e-05 2024-09-21T15:50:22.976 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.976 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.976 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:22.976 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209272+0000", 2024-09-21T15:50:22.976 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.289e-06 2024-09-21T15:50:22.976 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.976 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.976 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:22.976 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209321+0000", 2024-09-21T15:50:22.976 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8773999999999998e-05 2024-09-21T15:50:22.976 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.976 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.976 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:22.976 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209341+0000", 2024-09-21T15:50:22.977 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0387999999999999e-05 2024-09-21T15:50:22.977 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.977 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.977 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:22.977 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.213464+0000", 2024-09-21T15:50:22.977 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0041232450000000002 2024-09-21T15:50:22.977 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.977 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.977 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:22.977 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.213495+0000", 2024-09-21T15:50:22.977 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0793999999999997e-05 2024-09-21T15:50:22.977 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.977 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:22.977 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.977 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.978 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.978 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13445 3.0 e31/22 3:038b8840:::rbd_data.1133dc5adfdf.0000000000000035:head v 31'50, mlcod=31'48)", 2024-09-21T15:50:22.978 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.232497+0000", 2024-09-21T15:50:22.978 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 65.730527215999999, 2024-09-21T15:50:22.978 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0040493559999999996, 2024-09-21T15:50:22.978 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:22.978 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:22.978 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:22.978 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.978 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:22.978 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232497+0000", 2024-09-21T15:50:22.978 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.978 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.979 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.979 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:22.979 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232497+0000", 2024-09-21T15:50:22.979 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.979 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.979 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.979 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:22.979 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232496+0000", 2024-09-21T15:50:22.979 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:22.979 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.979 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.979 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:22.979 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232503+0000", 2024-09-21T15:50:22.979 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7320000000000001e-06 2024-09-21T15:50:22.979 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.980 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.980 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:22.980 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232512+0000", 2024-09-21T15:50:22.980 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9259999999999995e-06 2024-09-21T15:50:22.980 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.980 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.980 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:22.980 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232520+0000", 2024-09-21T15:50:22.980 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3699999999999997e-06 2024-09-21T15:50:22.980 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.980 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.980 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:22.980 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232564+0000", 2024-09-21T15:50:22.980 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4128999999999997e-05 2024-09-21T15:50:22.980 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.981 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.981 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:22.981 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232585+0000", 2024-09-21T15:50:22.981 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0704999999999999e-05 2024-09-21T15:50:22.981 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.981 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.981 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:22.981 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.236507+0000", 2024-09-21T15:50:22.981 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0039218730000000002 2024-09-21T15:50:22.981 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.981 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.981 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:22.981 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.236546+0000", 2024-09-21T15:50:22.981 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9399e-05 2024-09-21T15:50:22.982 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.982 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:22.982 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.982 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.982 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.982 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13452 3.9 e31/22 3:93f5396a:::rbd_data.1133dc5adfdf.0000000000000078:head v 31'50, mlcod=31'47)", 2024-09-21T15:50:22.982 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.237263+0000", 2024-09-21T15:50:22.982 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 65.725760510000001, 2024-09-21T15:50:22.982 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033768180000000002, 2024-09-21T15:50:22.982 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:22.982 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:22.982 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:22.982 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.982 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:22.983 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237263+0000", 2024-09-21T15:50:22.983 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.983 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.983 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.983 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:22.983 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237263+0000", 2024-09-21T15:50:22.983 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.983 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.983 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.983 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:22.983 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237263+0000", 2024-09-21T15:50:22.983 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:22.983 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.983 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.983 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:22.984 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237269+0000", 2024-09-21T15:50:22.984 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.472e-06 2024-09-21T15:50:22.984 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.984 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.984 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:22.984 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237282+0000", 2024-09-21T15:50:22.984 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2788e-05 2024-09-21T15:50:22.984 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.984 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.984 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:22.984 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237288+0000", 2024-09-21T15:50:22.984 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4899999999999997e-06 2024-09-21T15:50:22.984 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.984 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.985 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:22.985 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237331+0000", 2024-09-21T15:50:22.985 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2425000000000001e-05 2024-09-21T15:50:22.985 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.985 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.985 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:22.985 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237349+0000", 2024-09-21T15:50:22.985 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8405000000000001e-05 2024-09-21T15:50:22.985 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.985 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.985 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:22.985 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.240610+0000", 2024-09-21T15:50:22.985 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032605379999999999 2024-09-21T15:50:22.985 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.985 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.986 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:22.986 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.240640+0000", 2024-09-21T15:50:22.986 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0448000000000001e-05 2024-09-21T15:50:22.986 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.986 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:22.986 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.986 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.986 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.986 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13832 3.0 e31/22 3:0792efd2:::rbd_data.1133dc5adfdf.0000000000000090:head v 31'58, mlcod=31'56)", 2024-09-21T15:50:22.986 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.527530+0000", 2024-09-21T15:50:22.986 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 65.435493690000001, 2024-09-21T15:50:22.986 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035256530000000001, 2024-09-21T15:50:22.986 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:22.986 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:22.987 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:22.987 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.987 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:22.987 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527530+0000", 2024-09-21T15:50:22.987 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.987 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.987 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.987 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:22.987 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527530+0000", 2024-09-21T15:50:22.987 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.987 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.987 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.987 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:22.987 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527529+0000", 2024-09-21T15:50:22.987 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:50:22.987 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.988 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.988 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:22.988 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527550+0000", 2024-09-21T15:50:22.988 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1387999999999999e-05 2024-09-21T15:50:22.988 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.988 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.988 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:22.988 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527568+0000", 2024-09-21T15:50:22.988 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8068e-05 2024-09-21T15:50:22.988 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.988 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.988 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:22.988 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527577+0000", 2024-09-21T15:50:22.988 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9919999999999996e-06 2024-09-21T15:50:22.989 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.989 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.989 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:22.989 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527626+0000", 2024-09-21T15:50:22.989 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8597000000000002e-05 2024-09-21T15:50:22.989 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.989 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.989 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:22.989 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527652+0000", 2024-09-21T15:50:22.989 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5993e-05 2024-09-21T15:50:22.989 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.989 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.989 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:22.989 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.531024+0000", 2024-09-21T15:50:22.989 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033721910000000001 2024-09-21T15:50:22.990 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.990 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.990 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:22.990 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.531056+0000", 2024-09-21T15:50:22.990 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1628000000000003e-05 2024-09-21T15:50:22.990 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.990 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:22.990 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.990 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.990 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.990 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13839 3.0 e31/22 3:03c4fb32:::rbd_data.1133dc5adfdf.0000000000000099:head v 31'59, mlcod=31'56)", 2024-09-21T15:50:22.990 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.528097+0000", 2024-09-21T15:50:22.990 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 65.434927091999995, 2024-09-21T15:50:22.990 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0057228840000000001, 2024-09-21T15:50:22.991 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:22.991 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:22.991 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:22.991 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.991 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:22.991 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528097+0000", 2024-09-21T15:50:22.991 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.991 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.991 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.991 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:22.991 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528097+0000", 2024-09-21T15:50:22.991 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.991 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.991 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.991 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:22.992 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528096+0000", 2024-09-21T15:50:22.992 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:22.992 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.992 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.992 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:22.992 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528105+0000", 2024-09-21T15:50:22.992 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7350000000000002e-06 2024-09-21T15:50:22.992 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.992 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.992 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:22.992 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528117+0000", 2024-09-21T15:50:22.992 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1909000000000001e-05 2024-09-21T15:50:22.992 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.993 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.993 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:22.993 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528124+0000", 2024-09-21T15:50:22.993 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8419999999999999e-06 2024-09-21T15:50:22.993 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.993 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.993 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:22.993 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528168+0000", 2024-09-21T15:50:22.993 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4301999999999997e-05 2024-09-21T15:50:22.993 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.993 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.993 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:22.993 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528191+0000", 2024-09-21T15:50:22.993 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2622000000000001e-05 2024-09-21T15:50:22.993 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.994 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.994 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:22.994 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.533780+0000", 2024-09-21T15:50:22.994 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0055897519999999999 2024-09-21T15:50:22.994 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.994 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.994 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:22.994 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.533820+0000", 2024-09-21T15:50:22.994 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9366999999999998e-05 2024-09-21T15:50:22.994 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.994 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:22.994 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.994 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.994 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.995 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13884 3.9 e31/22 3:93003736:::rbd_data.1133dc5adfdf.00000000000000a1:head v 31'54, mlcod=31'52)", 2024-09-21T15:50:22.995 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.562433+0000", 2024-09-21T15:50:22.995 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 65.400590343999994, 2024-09-21T15:50:22.995 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.005162313, 2024-09-21T15:50:22.995 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:22.995 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:22.995 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:22.995 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.995 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:22.995 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562433+0000", 2024-09-21T15:50:22.995 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.995 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.995 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.996 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:22.996 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562433+0000", 2024-09-21T15:50:22.996 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:22.996 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.996 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.996 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:22.996 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562432+0000", 2024-09-21T15:50:22.996 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:22.996 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.996 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.996 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:22.996 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562467+0000", 2024-09-21T15:50:22.996 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4885999999999998e-05 2024-09-21T15:50:22.996 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.996 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.997 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:22.997 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562481+0000", 2024-09-21T15:50:22.997 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3604000000000001e-05 2024-09-21T15:50:22.997 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.997 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.997 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:22.997 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562489+0000", 2024-09-21T15:50:22.997 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1829999999999994e-06 2024-09-21T15:50:22.997 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.997 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.997 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:22.997 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562536+0000", 2024-09-21T15:50:22.997 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6830999999999999e-05 2024-09-21T15:50:22.997 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.998 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.998 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:22.998 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562559+0000", 2024-09-21T15:50:22.998 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2765999999999999e-05 2024-09-21T15:50:22.998 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.998 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.998 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:22.998 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.567566+0000", 2024-09-21T15:50:22.998 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0050073219999999998 2024-09-21T15:50:22.998 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.998 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.998 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:22.998 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.567596+0000", 2024-09-21T15:50:22.998 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9802999999999999e-05 2024-09-21T15:50:22.999 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.999 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:22.999 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:22.999 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:22.999 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.999 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:22340 3.9 e34/22 3:910a5abe:::rbd_data.1133896adcfd.0000000000000023:head v 34'92, mlcod=34'90)", 2024-09-21T15:50:22.999 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:21.628025+0000", 2024-09-21T15:50:22.999 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 61.334998964, 2024-09-21T15:50:22.999 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046196010000000001, 2024-09-21T15:50:22.999 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:22.999 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:22.999 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:22.999 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:22.999 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:22.999 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628025+0000", 2024-09-21T15:50:23.000 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:23.000 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.000 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.000 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:23.000 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628025+0000", 2024-09-21T15:50:23.000 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:23.000 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.000 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.000 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:23.000 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628024+0000", 2024-09-21T15:50:23.000 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:23.000 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.000 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.000 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:23.000 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628034+0000", 2024-09-21T15:50:23.001 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3470000000000005e-06 2024-09-21T15:50:23.001 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.001 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.001 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:23.001 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628043+0000", 2024-09-21T15:50:23.001 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9450000000000005e-06 2024-09-21T15:50:23.001 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.001 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.001 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:23.001 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628048+0000", 2024-09-21T15:50:23.001 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8080000000000004e-06 2024-09-21T15:50:23.001 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.001 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.001 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:23.002 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628081+0000", 2024-09-21T15:50:23.002 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2738999999999999e-05 2024-09-21T15:50:23.002 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.002 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.002 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:23.002 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628094+0000", 2024-09-21T15:50:23.002 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3273e-05 2024-09-21T15:50:23.002 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.002 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.002 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:23.002 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.632616+0000", 2024-09-21T15:50:23.002 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045217269999999997 2024-09-21T15:50:23.002 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.002 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.003 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:23.003 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.632644+0000", 2024-09-21T15:50:23.003 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8469000000000001e-05 2024-09-21T15:50:23.003 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:23.003 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:23.003 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:23.003 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.003 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.003 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:22417 3.4 e34/22 3:25310a10:::rbd_data.1133896adcfd.0000000000000075:head v 34'48, mlcod=34'46)", 2024-09-21T15:50:23.003 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:21.704297+0000", 2024-09-21T15:50:23.003 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 61.258726866000003, 2024-09-21T15:50:23.003 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0038101799999999998, 2024-09-21T15:50:23.003 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:23.003 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:23.003 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:23.004 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.004 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:23.004 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704297+0000", 2024-09-21T15:50:23.004 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:23.004 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.004 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.004 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:23.004 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704297+0000", 2024-09-21T15:50:23.004 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:23.004 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.004 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.004 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:23.004 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704296+0000", 2024-09-21T15:50:23.004 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:50:23.004 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.005 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.005 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:23.005 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704308+0000", 2024-09-21T15:50:23.005 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2559e-05 2024-09-21T15:50:23.005 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.005 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.005 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:23.005 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704337+0000", 2024-09-21T15:50:23.005 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8960000000000001e-05 2024-09-21T15:50:23.005 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.005 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.005 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:23.005 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704345+0000", 2024-09-21T15:50:23.005 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3059999999999998e-06 2024-09-21T15:50:23.006 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.006 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.006 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:23.006 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704408+0000", 2024-09-21T15:50:23.006 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2156000000000006e-05 2024-09-21T15:50:23.006 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.006 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.006 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:23.006 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704430+0000", 2024-09-21T15:50:23.006 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1931000000000001e-05 2024-09-21T15:50:23.006 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.006 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.006 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:23.006 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.708073+0000", 2024-09-21T15:50:23.007 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003643028 2024-09-21T15:50:23.007 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.007 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.007 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:23.007 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.708107+0000", 2024-09-21T15:50:23.007 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4613999999999998e-05 2024-09-21T15:50:23.007 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:23.007 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:23.007 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:23.007 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.007 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.007 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:25278 3.4 e38/22 3:2251cc1c:::rbd_data.1133ebd6d3ba.0000000000000024:head v 38'51, mlcod=38'49)", 2024-09-21T15:50:23.007 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.317337+0000", 2024-09-21T15:50:23.007 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.645687105999997, 2024-09-21T15:50:23.007 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053904110000000003, 2024-09-21T15:50:23.008 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:23.008 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:23.008 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:23.008 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.008 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:23.008 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317337+0000", 2024-09-21T15:50:23.008 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:23.008 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.008 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.008 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:23.008 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317337+0000", 2024-09-21T15:50:23.008 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:23.008 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.008 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.009 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:23.009 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317335+0000", 2024-09-21T15:50:23.009 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:50:23.009 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.009 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.009 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:23.009 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317347+0000", 2024-09-21T15:50:23.009 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.19e-05 2024-09-21T15:50:23.009 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.009 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.009 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:23.009 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317369+0000", 2024-09-21T15:50:23.009 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2232e-05 2024-09-21T15:50:23.009 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.009 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.010 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:23.010 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317381+0000", 2024-09-21T15:50:23.010 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2231e-05 2024-09-21T15:50:23.010 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.010 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.010 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:23.010 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317425+0000", 2024-09-21T15:50:23.010 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3625000000000003e-05 2024-09-21T15:50:23.010 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.010 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.010 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:23.010 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317449+0000", 2024-09-21T15:50:23.010 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4301999999999999e-05 2024-09-21T15:50:23.010 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.011 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.011 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:23.011 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.322688+0000", 2024-09-21T15:50:23.011 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.005238609 2024-09-21T15:50:23.011 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.011 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.011 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:23.011 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.322727+0000", 2024-09-21T15:50:23.011 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9201e-05 2024-09-21T15:50:23.011 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:23.011 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:23.011 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:23.011 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.011 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.012 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:25286 3.0 e38/22 3:0248fb60:::rbd_data.1133ebd6d3ba.0000000000000066:head v 38'109, mlcod=34'107)", 2024-09-21T15:50:23.012 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.317638+0000", 2024-09-21T15:50:23.012 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.645386227000003, 2024-09-21T15:50:23.012 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0054074370000000002, 2024-09-21T15:50:23.012 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:23.012 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:23.012 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:23.012 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.012 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:23.012 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317638+0000", 2024-09-21T15:50:23.012 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:23.012 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.012 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.012 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:23.012 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317638+0000", 2024-09-21T15:50:23.013 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:23.013 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.013 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.013 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:23.013 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317637+0000", 2024-09-21T15:50:23.013 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:23.013 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.013 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.013 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:23.013 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317643+0000", 2024-09-21T15:50:23.013 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5609999999999998e-06 2024-09-21T15:50:23.013 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.013 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.013 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:23.013 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317653+0000", 2024-09-21T15:50:23.014 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0705e-05 2024-09-21T15:50:23.014 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.014 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.014 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:23.014 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317660+0000", 2024-09-21T15:50:23.014 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2099999999999998e-06 2024-09-21T15:50:23.014 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.014 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.014 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:23.014 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317704+0000", 2024-09-21T15:50:23.014 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4659000000000001e-05 2024-09-21T15:50:23.014 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.014 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.014 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:23.015 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317725+0000", 2024-09-21T15:50:23.015 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0537e-05 2024-09-21T15:50:23.015 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.015 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.015 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:23.015 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323010+0000", 2024-09-21T15:50:23.015 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052853420000000002 2024-09-21T15:50:23.015 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.015 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.015 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:23.015 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323045+0000", 2024-09-21T15:50:23.015 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4860000000000002e-05 2024-09-21T15:50:23.015 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:23.015 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:23.016 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:23.016 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.016 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.016 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:25280 3.4 e38/22 3:27fca38c:::rbd_data.1133ebd6d3ba.0000000000000026:head v 38'52, mlcod=38'49)", 2024-09-21T15:50:23.016 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.317684+0000", 2024-09-21T15:50:23.016 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 56.645340331, 2024-09-21T15:50:23.016 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053570809999999996, 2024-09-21T15:50:23.016 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:23.016 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:23.016 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:23.016 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.016 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:23.016 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317684+0000", 2024-09-21T15:50:23.016 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:23.016 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.017 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.017 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:23.017 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317684+0000", 2024-09-21T15:50:23.017 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:23.017 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.017 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.017 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:23.017 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317683+0000", 2024-09-21T15:50:23.017 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:23.017 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.017 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.017 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:23.017 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317688+0000", 2024-09-21T15:50:23.017 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5009999999999998e-06 2024-09-21T15:50:23.017 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.018 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.018 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:23.018 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317694+0000", 2024-09-21T15:50:23.018 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3579999999999999e-06 2024-09-21T15:50:23.018 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.018 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.018 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:23.018 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317698+0000", 2024-09-21T15:50:23.018 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4150000000000001e-06 2024-09-21T15:50:23.018 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.018 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.018 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:23.018 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317738+0000", 2024-09-21T15:50:23.019 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9575e-05 2024-09-21T15:50:23.019 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.019 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.019 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:23.019 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317762+0000", 2024-09-21T15:50:23.019 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4173999999999999e-05 2024-09-21T15:50:23.019 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.019 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.019 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:23.019 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323005+0000", 2024-09-21T15:50:23.019 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00524333 2024-09-21T15:50:23.019 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.019 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.019 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:23.020 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323041+0000", 2024-09-21T15:50:23.020 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5139999999999999e-05 2024-09-21T15:50:23.020 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:23.020 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:23.020 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:23.020 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.020 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.020 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:34837 3.0 e43/22 3:038527c8:::rbd_header.1133fed1a840:head v 43'130, mlcod=43'128)", 2024-09-21T15:50:23.020 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:50:23.020 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 51.979135632999999, 2024-09-21T15:50:23.020 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045776519999999998, 2024-09-21T15:50:23.020 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:23.020 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:23.020 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:23.020 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.021 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:23.021 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:50:23.021 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:23.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.021 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.021 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:23.021 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:50:23.021 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:23.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.021 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.021 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:23.021 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:50:23.021 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:23.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.022 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.022 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:23.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983892+0000", 2024-09-21T15:50:23.022 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8450000000000002e-06 2024-09-21T15:50:23.022 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.022 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.022 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:23.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983902+0000", 2024-09-21T15:50:23.022 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0299999999999999e-06 2024-09-21T15:50:23.022 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.022 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.022 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:23.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983906+0000", 2024-09-21T15:50:23.022 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2300000000000002e-06 2024-09-21T15:50:23.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.023 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.023 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:23.023 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983940+0000", 2024-09-21T15:50:23.023 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4563000000000001e-05 2024-09-21T15:50:23.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.023 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.023 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:23.023 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983960+0000", 2024-09-21T15:50:23.023 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9199999999999999e-05 2024-09-21T15:50:23.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.023 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.023 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:23.023 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.988430+0000", 2024-09-21T15:50:23.024 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0044709149999999998 2024-09-21T15:50:23.024 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.024 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.024 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:23.024 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.988466+0000", 2024-09-21T15:50:23.024 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5420000000000003e-05 2024-09-21T15:50:23.024 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:23.024 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:23.024 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:23.024 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.024 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.024 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "pg_update_log_missing(3.0 epoch 50/22 rep_tid 498 entries 50'171 (0'0) error 3:5df4a5a0:::rbd_data.11339e87a32a.000000000000008e:head by client.4403.0:80207 0.000000 -2 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 47'167)", 2024-09-21T15:50:23.024 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:50:23.024 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 42.021262759999999, 2024-09-21T15:50:23.024 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0047253119999999997, 2024-09-21T15:50:23.025 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:23.025 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-09-21T15:50:23.025 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:23.025 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.025 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:23.025 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:50:23.025 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:23.025 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.025 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.025 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:23.025 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:50:23.025 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:23.025 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.025 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.025 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:23.026 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:50:23.026 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:23.026 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.026 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.026 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:23.026 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941768+0000", 2024-09-21T15:50:23.026 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7309999999999993e-06 2024-09-21T15:50:23.026 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.026 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.026 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:23.026 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941780+0000", 2024-09-21T15:50:23.026 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1399999999999999e-05 2024-09-21T15:50:23.026 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.026 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.026 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:23.027 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941789+0000", 2024-09-21T15:50:23.027 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5230000000000001e-06 2024-09-21T15:50:23.027 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.027 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.027 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:23.027 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941841+0000", 2024-09-21T15:50:23.027 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1863999999999997e-05 2024-09-21T15:50:23.027 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.027 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.027 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:23.027 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.946486+0000", 2024-09-21T15:50:23.027 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046452539999999997 2024-09-21T15:50:23.027 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:23.027 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:23.027 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:23.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.028 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.028 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:82086 3.0 e50/22 3:0114bc4d:::rbd_data.11334c14b8fd.0000000000000069:head v 50'178, mlcod=50'176)", 2024-09-21T15:50:23.028 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:41.155591+0000", 2024-09-21T15:50:23.028 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 41.807432448, 2024-09-21T15:50:23.028 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003363497, 2024-09-21T15:50:23.028 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:23.028 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:23.028 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:23.028 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.028 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:23.028 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.155591+0000", 2024-09-21T15:50:23.028 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:23.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.028 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.029 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:23.029 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.155591+0000", 2024-09-21T15:50:23.029 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:23.029 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.029 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.029 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:23.029 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.155590+0000", 2024-09-21T15:50:23.029 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:50:23.029 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.029 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.029 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:23.029 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.155606+0000", 2024-09-21T15:50:23.029 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6075000000000001e-05 2024-09-21T15:50:23.029 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.029 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.030 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:23.030 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.155623+0000", 2024-09-21T15:50:23.030 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6838e-05 2024-09-21T15:50:23.030 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.030 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.030 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:23.030 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.155630+0000", 2024-09-21T15:50:23.030 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.627e-06 2024-09-21T15:50:23.030 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.030 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.030 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:23.030 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.155677+0000", 2024-09-21T15:50:23.030 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6669999999999999e-05 2024-09-21T15:50:23.030 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.031 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.031 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:23.031 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.155698+0000", 2024-09-21T15:50:23.031 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1330000000000001e-05 2024-09-21T15:50:23.031 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.031 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.031 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:23.031 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.158918+0000", 2024-09-21T15:50:23.031 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032198600000000002 2024-09-21T15:50:23.031 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.031 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.031 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:23.031 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.158955+0000", 2024-09-21T15:50:23.031 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6647000000000003e-05 2024-09-21T15:50:23.031 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:23.032 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:23.032 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:23.032 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.032 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.032 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:82470 3.4 e50/22 3:21ad7dfe:::rbd_data.11334c14b8fd.00000000000003a7:head v 50'190, mlcod=46'188)", 2024-09-21T15:50:23.032 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:41.291787+0000", 2024-09-21T15:50:23.032 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 41.671236823000001, 2024-09-21T15:50:23.032 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033489969999999998, 2024-09-21T15:50:23.032 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:23.032 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:23.032 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:23.032 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.032 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:23.032 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.291787+0000", 2024-09-21T15:50:23.033 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:23.033 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.033 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.033 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:23.033 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.291787+0000", 2024-09-21T15:50:23.033 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:23.033 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.033 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.033 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:23.033 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.291786+0000", 2024-09-21T15:50:23.033 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:23.033 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.033 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.033 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:23.034 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.291795+0000", 2024-09-21T15:50:23.034 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7830000000000004e-06 2024-09-21T15:50:23.034 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.034 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:23.034 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.291814+0000", 2024-09-21T15:50:23.034 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9117999999999999e-05 2024-09-21T15:50:23.034 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.034 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:23.034 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.291822+0000", 2024-09-21T15:50:23.034 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0669999999999992e-06 2024-09-21T15:50:23.034 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.034 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:23.034 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.291866+0000", 2024-09-21T15:50:23.035 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4669000000000002e-05 2024-09-21T15:50:23.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.035 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:23.035 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.291888+0000", 2024-09-21T15:50:23.035 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1087999999999999e-05 2024-09-21T15:50:23.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.035 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:23.035 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.295104+0000", 2024-09-21T15:50:23.035 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032164910000000001 2024-09-21T15:50:23.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.035 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:23.035 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.295136+0000", 2024-09-21T15:50:23.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1931999999999999e-05 2024-09-21T15:50:23.036 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:23.036 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:23.036 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:23.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.036 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "pg_update_log_missing(3.0 epoch 53/22 rep_tid 585 entries 53'201 (0'0) error 3:6a1d4890:::rbd_data.1133e079cae4.0000000000000004:head by client.4403.0:101089 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 53'198)", 2024-09-21T15:50:23.036 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:44.832827+0000", 2024-09-21T15:50:23.036 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 38.130196951999999, 2024-09-21T15:50:23.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053552929999999997, 2024-09-21T15:50:23.036 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:23.036 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-09-21T15:50:23.036 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:23.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.036 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:23.037 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832827+0000", 2024-09-21T15:50:23.037 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:23.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.037 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:23.037 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832827+0000", 2024-09-21T15:50:23.037 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:23.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.037 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:23.037 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832826+0000", 2024-09-21T15:50:23.037 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:23.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.038 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:23.038 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832835+0000", 2024-09-21T15:50:23.038 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4509999999999992e-06 2024-09-21T15:50:23.038 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.038 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.038 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:23.038 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832854+0000", 2024-09-21T15:50:23.038 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9352999999999999e-05 2024-09-21T15:50:23.038 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.038 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.038 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:23.038 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832866+0000", 2024-09-21T15:50:23.038 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1766000000000001e-05 2024-09-21T15:50:23.038 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.038 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.039 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:23.039 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.833091+0000", 2024-09-21T15:50:23.039 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00022490300000000001 2024-09-21T15:50:23.039 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.039 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.039 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:23.039 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.838182+0000", 2024-09-21T15:50:23.039 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0050915420000000001 2024-09-21T15:50:23.039 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:23.039 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:23.039 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:23.039 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.039 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.039 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:102822 3.4 e54/22 3:2729ba03:::rbd_object_map.113336ae1075:head v 54'208, mlcod=54'205)", 2024-09-21T15:50:23.039 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:46.249540+0000", 2024-09-21T15:50:23.040 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 36.713483719999999, 2024-09-21T15:50:23.040 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052654859999999998, 2024-09-21T15:50:23.040 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:23.040 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:23.040 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:23.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.040 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:23.040 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249540+0000", 2024-09-21T15:50:23.040 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:23.040 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.040 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:23.040 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249540+0000", 2024-09-21T15:50:23.040 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:23.041 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.041 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.041 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:23.041 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249539+0000", 2024-09-21T15:50:23.041 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:23.041 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.041 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.041 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:23.041 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249546+0000", 2024-09-21T15:50:23.041 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7290000000000003e-06 2024-09-21T15:50:23.041 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.041 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.041 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:23.041 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249558+0000", 2024-09-21T15:50:23.041 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1844e-05 2024-09-21T15:50:23.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.042 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.042 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:23.042 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249565+0000", 2024-09-21T15:50:23.042 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8210000000000004e-06 2024-09-21T15:50:23.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.042 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.042 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:23.042 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249602+0000", 2024-09-21T15:50:23.042 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7057000000000001e-05 2024-09-21T15:50:23.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.042 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.042 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:23.042 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249622+0000", 2024-09-21T15:50:23.042 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9678999999999999e-05 2024-09-21T15:50:23.043 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.043 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.043 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:23.043 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254783+0000", 2024-09-21T15:50:23.043 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0051611449999999998 2024-09-21T15:50:23.043 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.043 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.043 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:23.043 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254806+0000", 2024-09-21T15:50:23.043 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2853e-05 2024-09-21T15:50:23.043 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:23.043 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:23.043 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:23.043 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.043 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.044 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:102823 3.4 e54/22 3:2729ba03:::rbd_object_map.113336ae1075:head v 54'209, mlcod=54'205)", 2024-09-21T15:50:23.044 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:46.250318+0000", 2024-09-21T15:50:23.044 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 36.712706134999998, 2024-09-21T15:50:23.044 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045013539999999999, 2024-09-21T15:50:23.044 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:23.044 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:23.044 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:23.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.044 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:23.044 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250318+0000", 2024-09-21T15:50:23.044 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:23.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.044 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:23.044 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250318+0000", 2024-09-21T15:50:23.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:23.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:23.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250317+0000", 2024-09-21T15:50:23.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:23.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:23.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250322+0000", 2024-09-21T15:50:23.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.074e-06 2024-09-21T15:50:23.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:23.046 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250330+0000", 2024-09-21T15:50:23.046 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4730000000000003e-06 2024-09-21T15:50:23.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.046 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:23.046 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250335+0000", 2024-09-21T15:50:23.046 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8019999999999997e-06 2024-09-21T15:50:23.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.046 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:23.046 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250375+0000", 2024-09-21T15:50:23.046 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9459999999999998e-05 2024-09-21T15:50:23.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.047 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:23.047 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250388+0000", 2024-09-21T15:50:23.047 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3288999999999999e-05 2024-09-21T15:50:23.047 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.047 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.047 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:23.047 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254807+0000", 2024-09-21T15:50:23.047 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0044192220000000004 2024-09-21T15:50:23.047 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:23.047 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:23.047 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:23.047 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254819+0000", 2024-09-21T15:50:23.047 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1603999999999999e-05 2024-09-21T15:50:23.047 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:23.047 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:23.048 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:23.048 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:23.048 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:23.048 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:23.049 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:50:23.050 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:23.048+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:23.150 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:23.149+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:23.200 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:23.200 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:23.200 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:50:23.201 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:23.210 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:50:23.236 INFO:teuthology.orchestra.run.smithi089.stdout:checking clone #39, image image_client.0-clone39 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent40 2024-09-21T15:50:23.251 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:23.250+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:23.257 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:50:23.292 INFO:teuthology.orchestra.run.smithi089.stdout:5722 write 0x34f0128 thru 0x34f2830 (0x2709 bytes) 2024-09-21T15:50:23.298 INFO:teuthology.orchestra.run.smithi089.stdout:5723 clone 42 order 23 su 65536 sc 14 2024-09-21T15:50:23.320 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:23.320 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:23.320 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:50:23.320 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:50:23.320 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:23.329 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:50:23.364 INFO:teuthology.orchestra.run.smithi089.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T15:50:23.364 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:23.362+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:23.380 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:50:23.380 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:50:23.396 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:50:23.397 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:50:23.412 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:50:23.413 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:50:23.429 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:50:23.429 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:50:23.445 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:50:23.446 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:50:23.452 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:23.451+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:23.461 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:50:23.462 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:50:23.468 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:23.468 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:23.476 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:50:23.478 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:50:23.478 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:50:23.481 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T15:50:23.482 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T15:50:23.553 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:23.552+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:23.577 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:23.577 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:23.585 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:50:23.654 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:23.653+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:23.686 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:23.687 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:23.695 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:50:23.779 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:23.778+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:23.836 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:23.836 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:23.845 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:50:23.886 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:23.885+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:23.955 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:23.954+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:23.987 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:23.987 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:23.996 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:50:24.080 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:24.079+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:24.097 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [0] live_osds: [1, 2, 3, 4, 5, 6, 7] 2024-09-21T15:50:24.098 INFO:tasks.thrashosds.thrasher:Reviving osd 0 2024-09-21T15:50:24.098 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-21T15:50:24.098 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-21T15:50:24.100 INFO:tasks.ceph.osd.0:Started 2024-09-21T15:50:24.101 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:50:24.116 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T15:50:24.141 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:24.141 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:24.150 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:50:24.178 INFO:teuthology.orchestra.run.smithi089.stdout:flattening image image_client.0-clone41 2024-09-21T15:50:24.195 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:24.190+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:24.234 INFO:teuthology.orchestra.run.smithi089.stderr:no valid command found; 10 closest matches: 2024-09-21T15:50:24.234 INFO:teuthology.orchestra.run.smithi089.stderr:0 2024-09-21T15:50:24.234 INFO:teuthology.orchestra.run.smithi089.stderr:1 2024-09-21T15:50:24.234 INFO:teuthology.orchestra.run.smithi089.stderr:2 2024-09-21T15:50:24.234 INFO:teuthology.orchestra.run.smithi089.stderr:abort 2024-09-21T15:50:24.235 INFO:teuthology.orchestra.run.smithi089.stderr:assert 2024-09-21T15:50:24.235 INFO:teuthology.orchestra.run.smithi089.stderr:config diff 2024-09-21T15:50:24.235 INFO:teuthology.orchestra.run.smithi089.stderr:config diff get 2024-09-21T15:50:24.235 INFO:teuthology.orchestra.run.smithi089.stderr:config get 2024-09-21T15:50:24.235 INFO:teuthology.orchestra.run.smithi089.stderr:config help [] 2024-09-21T15:50:24.235 INFO:teuthology.orchestra.run.smithi089.stderr:config set ... 2024-09-21T15:50:24.235 INFO:teuthology.orchestra.run.smithi089.stderr:admin_socket: invalid command 2024-09-21T15:50:24.238 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T15:50:24.238 INFO:tasks.ceph.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2024-09-21T15:50:24.258 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:24.257+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:24.338 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:24.338 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:24.347 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T15:50:24.359 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:24.358+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:24.395 INFO:teuthology.orchestra.run.smithi089.stdout:checking clone #40, image image_client.0-clone40 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent41 2024-09-21T15:50:24.445 INFO:teuthology.orchestra.run.smithi089.stdout:5724 punch from 0x20ef2a5 to 0x20f9055, (0x9db0 bytes) 2024-09-21T15:50:24.449 INFO:teuthology.orchestra.run.smithi089.stdout:5725 read 0x342c530 thru 0x3432b24 (0x65f5 bytes) 2024-09-21T15:50:24.449 INFO:teuthology.orchestra.run.smithi089.stdout:5727 write 0x2fb6d22 thru 0x2fc5460 (0xe73f bytes) 2024-09-21T15:50:24.456 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:24.456 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:24.456 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:50:24.455+0000 7f7448e0e640 -1 Falling back to public interface 2024-09-21T15:50:24.460 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:24.458+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:24.467 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T15:50:24.473 INFO:teuthology.orchestra.run.smithi089.stdout:5730 trunc from 0x34f2831 to 0x321a6 2024-09-21T15:50:24.506 INFO:teuthology.orchestra.run.smithi089.stdout:5733 clone 43 order 23 su 2097152 sc 2 2024-09-21T15:50:24.560 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:24.559+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:24.608 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:24.609 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:24.617 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T15:50:24.701 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:24.700+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:24.758 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:24.758 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:24.762 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:24.761+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:24.767 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T15:50:24.863 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:24.862+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:24.909 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:24.909 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:24.917 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T15:50:24.965 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:24.964+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:25.059 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:25.059 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:25.065 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:25.065+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:25.067 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T15:50:25.114 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is unset 2024-09-21T15:50:25.166 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:25.165+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:25.175 INFO:teuthology.orchestra.run.smithi089.stdout:truncating image image_client.0-clone42 from 0x321a6 (overlap 0x321a6) to 0x0 2024-09-21T15:50:25.175 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:25.175 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:25.185 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:50:25.223 INFO:teuthology.orchestra.run.smithi089.stdout:checking clone #41, image image_client.0-clone41 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent42 2024-09-21T15:50:25.267 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:25.266+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:25.302 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:25.302 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:25.316 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:50:25.323 INFO:teuthology.orchestra.run.smithi089.stdout:5734 write 0xbf9c367 thru 0xbf9d29b (0xf35 bytes) 2024-09-21T15:50:25.334 INFO:teuthology.orchestra.run.smithi089.stdout:5735 trunc from 0xbf9d29c to 0x543ec17 2024-09-21T15:50:25.356 INFO:teuthology.orchestra.run.smithi089.stdout:5736 write 0xa454e1b thru 0xa45bdc0 (0x6fa6 bytes) 2024-09-21T15:50:25.359 INFO:teuthology.orchestra.run.smithi089.stdout:5737 read 0xa13973a thru 0xa13e3e2 (0x4ca9 bytes) 2024-09-21T15:50:25.364 INFO:teuthology.orchestra.run.smithi089.stdout:5738 write 0xb0ef81b thru 0xb0f38f2 (0x40d8 bytes) 2024-09-21T15:50:25.367 INFO:teuthology.orchestra.run.smithi089.stdout:5739 write 0x2de7c53 thru 0x2df73e1 (0xf78f bytes) 2024-09-21T15:50:25.367 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:25.366+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:25.371 INFO:teuthology.orchestra.run.smithi089.stdout:5740 writesame 0x99b2b89 thru 0x99b537d data_size 0x35(0x27f5 bytes) 2024-09-21T15:50:25.374 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:25.374 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:25.374 INFO:teuthology.orchestra.run.smithi089.stdout:5744 read 0x4453e49 thru 0x4460158 (0xc310 bytes) 2024-09-21T15:50:25.375 INFO:teuthology.orchestra.run.smithi089.stdout:5745 writesame 0x7f7450c thru 0x7f7e2bc data_size 0x49(0x9db1 bytes) 2024-09-21T15:50:25.377 INFO:teuthology.orchestra.run.smithi089.stdout:5746 writesame 0x3afdf00 thru 0x3b0c210 data_size 0x1(0xe311 bytes) 2024-09-21T15:50:25.405 INFO:teuthology.orchestra.run.smithi089.stdout:5747 write 0xb13ac9d thru 0xb140985 (0x5ce9 bytes) 2024-09-21T15:50:25.406 INFO:teuthology.orchestra.run.smithi089.stdout:5749 punch from 0x62f6cff to 0x62f7421, (0x722 bytes) 2024-09-21T15:50:25.406 INFO:teuthology.orchestra.run.smithi089.stdout:5750 writesame 0x35edf0b thru 0x35f8cd3 data_size 0x1(0xadc9 bytes) 2024-09-21T15:50:25.424 INFO:teuthology.orchestra.run.smithi089.stdout:5751 write 0x876a89c thru 0x8770934 (0x6099 bytes) 2024-09-21T15:50:25.427 INFO:teuthology.orchestra.run.smithi089.stdout:5752 read 0x587099 thru 0x58c257 (0x51bf bytes) 2024-09-21T15:50:25.432 INFO:teuthology.orchestra.run.smithi089.stdout:5753 writesame 0xbcdb902 thru 0xbceb765 data_size 0x86(0xfe64 bytes) 2024-09-21T15:50:25.433 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:25.434 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:25.434 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:25.439 INFO:teuthology.orchestra.run.smithi089.stdout:5757 trunc from 0xbceb766 to 0x6ee31a5 2024-09-21T15:50:25.446 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:50:25.451 INFO:teuthology.orchestra.run.smithi089.stdout:5758 read 0x3b15c37 thru 0x3b25543 (0xf90d bytes) 2024-09-21T15:50:25.452 INFO:teuthology.orchestra.run.smithi089.stdout:5759 punch from 0x3309225 to 0x330f7e4, (0x65bf bytes) 2024-09-21T15:50:25.452 INFO:teuthology.orchestra.run.smithi089.stdout:5761 read 0x432bf7d thru 0x432d87e (0x1902 bytes) 2024-09-21T15:50:25.453 INFO:teuthology.orchestra.run.smithi089.stdout:5762 trunc from 0x6ee31a5 to 0x675166b 2024-09-21T15:50:25.457 INFO:teuthology.orchestra.run.smithi089.stdout:5763 punch from 0x46b1d8 to 0x472cdf, (0x7b07 bytes) 2024-09-21T15:50:25.457 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:25.468 INFO:teuthology.orchestra.run.smithi089.stdout:5765 writesame 0xb3eff27 thru 0xb3f7117 data_size 0x15(0x71f1 bytes) 2024-09-21T15:50:25.469 INFO:teuthology.orchestra.run.smithi089.stdout:5766 read 0x611404e thru 0x6116ad7 (0x2a8a bytes) 2024-09-21T15:50:25.469 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:25.467+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:25.472 INFO:teuthology.orchestra.run.smithi089.stdout:5767 trunc from 0xb3f7118 to 0xe5ad124 2024-09-21T15:50:25.477 INFO:teuthology.orchestra.run.smithi089.stdout:5769 punch from 0xb824bf7 to 0xb83094d, (0xbd56 bytes) 2024-09-21T15:50:25.477 INFO:teuthology.orchestra.run.smithi089.stdout:5770 write 0xb785a62 thru 0xb786302 (0x8a1 bytes) 2024-09-21T15:50:25.483 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:50:25.483 INFO:teuthology.orchestra.run.smithi089.stdout:5771 writesame 0x33a6a74 thru 0x33b1819 data_size 0x5d(0xada6 bytes) 2024-09-21T15:50:25.485 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:25.485 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:25.485 INFO:teuthology.orchestra.run.smithi089.stdout:5776 read 0xb8c6ed1 thru 0xb8d4885 (0xd9b5 bytes) 2024-09-21T15:50:25.485 INFO:teuthology.orchestra.run.smithi089.stdout:5778 punch from 0x6b5cefc to 0x6b68b35, (0xbc39 bytes) 2024-09-21T15:50:25.486 INFO:teuthology.orchestra.run.smithi089.stdout:5779 writesame 0x2e0c52 thru 0x2f09ff data_size 0x13(0xfdae bytes) 2024-09-21T15:50:25.488 INFO:teuthology.orchestra.run.smithi089.stdout:5781 write 0x7213893 thru 0x7219fff (0x676d bytes) 2024-09-21T15:50:25.493 INFO:teuthology.orchestra.run.smithi089.stdout:5783 read 0x5bb722a thru 0x5bc0f11 (0x9ce8 bytes) 2024-09-21T15:50:25.493 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:25.494 INFO:teuthology.orchestra.run.smithi089.stdout:5786 read 0x39da4a0 thru 0x39e5754 (0xb2b5 bytes) 2024-09-21T15:50:25.494 INFO:teuthology.orchestra.run.smithi089.stdout:5789 punch from 0x428d242 to 0x429cdc6, (0xfb84 bytes) 2024-09-21T15:50:25.494 INFO:teuthology.orchestra.run.smithi089.stdout:5791 write 0x9c65601 thru 0x9c73e8a (0xe88a bytes) 2024-09-21T15:50:25.512 INFO:teuthology.orchestra.run.smithi089.stdout:5793 trunc from 0xe5ad124 to 0x2f90220 2024-09-21T15:50:25.530 INFO:teuthology.orchestra.run.smithi089.stdout:5794 trunc from 0x2f90220 to 0x14fd5c1 2024-09-21T15:50:25.539 INFO:teuthology.orchestra.run.smithi089.stdout:5795 read 0x7baf94 thru 0x7c85ed (0xd65a bytes) 2024-09-21T15:50:25.540 INFO:teuthology.orchestra.run.smithi089.stdout:5796 read 0x13d8c8a thru 0x13e551a (0xc891 bytes) 2024-09-21T15:50:25.540 INFO:teuthology.orchestra.run.smithi089.stdout:5797 writesame 0x27ab59 thru 0x2832bf data_size 0x1(0x8767 bytes) 2024-09-21T15:50:25.560 INFO:teuthology.orchestra.run.smithi089.stdout:5799 write 0x255ceab thru 0x2565778 (0x88ce bytes) 2024-09-21T15:50:25.563 INFO:teuthology.orchestra.run.smithi089.stdout:5800 read 0x1eaf366 thru 0x1eb6c82 (0x791d bytes) 2024-09-21T15:50:25.563 INFO:teuthology.orchestra.run.smithi089.stdout:5801 read 0x157ada1 thru 0x1580a39 (0x5c99 bytes) 2024-09-21T15:50:25.563 INFO:teuthology.orchestra.run.smithi089.stdout:5802 punch from 0xa2da37 to 0xa2e994, (0xf5d bytes) 2024-09-21T15:50:25.563 INFO:teuthology.orchestra.run.smithi089.stdout:5804 punch from 0x3d1a17 to 0x3d6361, (0x494a bytes) 2024-09-21T15:50:25.570 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:25.568+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:25.572 INFO:teuthology.orchestra.run.smithi089.stdout:5805 trunc from 0x2565779 to 0x92b1d02 2024-09-21T15:50:25.577 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:25.577 INFO:teuthology.orchestra.run.smithi089.stdout:5809 read 0x10731ba thru 0x107ee64 (0xbcab bytes) 2024-09-21T15:50:25.583 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:25.583 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:25.592 INFO:teuthology.orchestra.run.smithi089.stdout:5810 write 0xdf81a6c thru 0xdf8eeef (0xd484 bytes) 2024-09-21T15:50:25.594 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:50:25.594 INFO:teuthology.orchestra.run.smithi089.stdout:5811 read 0x219dae2 thru 0x219dc9e (0x1bd bytes) 2024-09-21T15:50:25.597 INFO:teuthology.orchestra.run.smithi089.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T15:50:25.597 INFO:teuthology.orchestra.run.smithi089.stdout:5812 trunc from 0xdf8eef0 to 0xba5eb00 2024-09-21T15:50:25.605 INFO:teuthology.orchestra.run.smithi089.stdout:5813 write 0x2777880 thru 0x2781e12 (0xa593 bytes) 2024-09-21T15:50:25.608 INFO:teuthology.orchestra.run.smithi089.stdout:5814 write 0x2feec8b thru 0x2ff35b3 (0x4929 bytes) 2024-09-21T15:50:25.609 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:25.610 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:25.615 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:50:25.615 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:50:25.618 INFO:teuthology.orchestra.run.smithi089.stdout:5818 trunc from 0xba5eb00 to 0x48663f5 2024-09-21T15:50:25.632 INFO:teuthology.orchestra.run.smithi089.stdout:5819 writesame 0x71428b8 thru 0x714cd99 data_size 0x86(0xa4e2 bytes) 2024-09-21T15:50:25.632 INFO:teuthology.orchestra.run.smithi089.stdout:5820 read 0x5ce2092 thru 0x5cf0e55 (0xedc4 bytes) 2024-09-21T15:50:25.634 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:50:25.634 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:50:25.635 INFO:teuthology.orchestra.run.smithi089.stdout:5821 trunc from 0x714cd9a to 0x96384fc 2024-09-21T15:50:25.640 INFO:teuthology.orchestra.run.smithi089.stdout:5823 read 0x398fedd thru 0x3991a82 (0x1ba6 bytes) 2024-09-21T15:50:25.640 INFO:teuthology.orchestra.run.smithi089.stdout:5825 writesame 0x7017f01 thru 0x702466e data_size 0x86(0xc76e bytes) 2024-09-21T15:50:25.647 INFO:teuthology.orchestra.run.smithi089.stdout:5826 write 0x9df41f6 thru 0x9dfaed8 (0x6ce3 bytes) 2024-09-21T15:50:25.650 INFO:teuthology.orchestra.run.smithi089.stdout:5828 clone 44 order 20 su 524288 sc 12 2024-09-21T15:50:25.652 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:50:25.652 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:50:25.668 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:50:25.668 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:50:25.670 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:25.669+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:25.685 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:50:25.685 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:50:25.701 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:50:25.701 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:50:25.706 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:25.706 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:25.706 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:50:25.706 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:25.715 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:50:25.718 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:50:25.718 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:50:25.721 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T15:50:25.721 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T15:50:25.770 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:25.769+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:25.820 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:25.820 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:25.820 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:50:25.820 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:50:25.820 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:25.829 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:50:25.872 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:25.870+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:25.972 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:25.972 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:50:25.973 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:50:25.973 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:50:25.973 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.973 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:984 3.0 e27/22 3:025e53e7:::rbd_data.1133f3dfbd6b.0000000000000008:head v 27'21, mlcod=27'19)", 2024-09-21T15:50:25.973 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.619393+0000", 2024-09-21T15:50:25.973 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 73.351219706999998, 2024-09-21T15:50:25.973 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003401837, 2024-09-21T15:50:25.973 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:25.973 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:25.973 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:25.973 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.973 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:25.973 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619393+0000", 2024-09-21T15:50:25.974 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:25.974 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.974 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.974 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:25.974 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619393+0000", 2024-09-21T15:50:25.974 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:25.974 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.974 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.974 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:25.974 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619392+0000", 2024-09-21T15:50:25.974 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:25.974 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.974 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.974 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:25.975 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619439+0000", 2024-09-21T15:50:25.975 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7170999999999999e-05 2024-09-21T15:50:25.975 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.975 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.975 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:25.975 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619452+0000", 2024-09-21T15:50:25.975 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2986999999999999e-05 2024-09-21T15:50:25.975 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.975 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.975 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:25.975 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619459+0000", 2024-09-21T15:50:25.975 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6109999999999999e-06 2024-09-21T15:50:25.975 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.976 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.976 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:25.976 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619518+0000", 2024-09-21T15:50:25.976 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9117999999999999e-05 2024-09-21T15:50:25.976 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.976 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.976 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:25.976 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619532+0000", 2024-09-21T15:50:25.976 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4622e-05 2024-09-21T15:50:25.976 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.976 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.976 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:25.976 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.622760+0000", 2024-09-21T15:50:25.976 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032278839999999999 2024-09-21T15:50:25.976 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.977 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.977 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:25.977 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.622794+0000", 2024-09-21T15:50:25.977 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4144999999999999e-05 2024-09-21T15:50:25.977 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:25.977 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:25.977 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:25.977 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.977 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.977 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:4957 3.9 e29/22 3:95c5ed95:::rbd_data.11335fa2fe69.0000000000000018:head v 29'4, mlcod=26'2)", 2024-09-21T15:50:25.977 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:14.453951+0000", 2024-09-21T15:50:25.977 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 71.516661275000004, 2024-09-21T15:50:25.977 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0043804559999999996, 2024-09-21T15:50:25.977 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:25.978 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:25.978 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:25.978 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.978 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:25.978 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.453951+0000", 2024-09-21T15:50:25.978 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:25.978 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.978 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.978 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:25.978 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.453951+0000", 2024-09-21T15:50:25.978 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:25.978 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.978 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.978 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:25.979 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.453950+0000", 2024-09-21T15:50:25.979 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:25.979 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.979 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.979 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:25.979 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454495+0000", 2024-09-21T15:50:25.979 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00054437799999999998 2024-09-21T15:50:25.979 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.979 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.979 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:25.979 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454508+0000", 2024-09-21T15:50:25.979 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3475000000000001e-05 2024-09-21T15:50:25.979 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.979 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.980 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:25.980 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454515+0000", 2024-09-21T15:50:25.980 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8990000000000004e-06 2024-09-21T15:50:25.980 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.980 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.980 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:25.980 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454581+0000", 2024-09-21T15:50:25.980 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5987000000000004e-05 2024-09-21T15:50:25.980 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.980 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.980 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:25.980 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454609+0000", 2024-09-21T15:50:25.980 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7466000000000001e-05 2024-09-21T15:50:25.980 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.980 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.981 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:25.981 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.458303+0000", 2024-09-21T15:50:25.981 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003693979 2024-09-21T15:50:25.981 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.981 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.981 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:25.981 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.458331+0000", 2024-09-21T15:50:25.981 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8932999999999999e-05 2024-09-21T15:50:25.981 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:25.981 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:25.981 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:25.981 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.981 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.981 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13420 3.0 e31/22 3:038b8840:::rbd_data.1133dc5adfdf.0000000000000035:head v 31'49, mlcod=31'47)", 2024-09-21T15:50:25.982 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.209233+0000", 2024-09-21T15:50:25.982 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 68.761379658999999, 2024-09-21T15:50:25.982 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0042624960000000002, 2024-09-21T15:50:25.982 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:25.982 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:25.982 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:25.982 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.982 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:25.982 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209233+0000", 2024-09-21T15:50:25.982 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:25.982 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.982 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.982 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:25.982 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209233+0000", 2024-09-21T15:50:25.982 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:25.983 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.983 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.983 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:25.983 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209232+0000", 2024-09-21T15:50:25.983 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:25.983 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.983 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.983 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:25.983 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209251+0000", 2024-09-21T15:50:25.983 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9304e-05 2024-09-21T15:50:25.983 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.983 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.983 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:25.983 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209265+0000", 2024-09-21T15:50:25.984 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3532e-05 2024-09-21T15:50:25.984 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.984 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.984 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:25.984 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209272+0000", 2024-09-21T15:50:25.984 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.289e-06 2024-09-21T15:50:25.984 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.984 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.984 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:25.984 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209321+0000", 2024-09-21T15:50:25.984 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8773999999999998e-05 2024-09-21T15:50:25.984 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.984 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.984 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:25.984 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209341+0000", 2024-09-21T15:50:25.985 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0387999999999999e-05 2024-09-21T15:50:25.985 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.985 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.985 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:25.985 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.213464+0000", 2024-09-21T15:50:25.985 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0041232450000000002 2024-09-21T15:50:25.985 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.985 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.985 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:25.985 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.213495+0000", 2024-09-21T15:50:25.985 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0793999999999997e-05 2024-09-21T15:50:25.985 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:25.985 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:25.985 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:25.986 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.986 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.986 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13445 3.0 e31/22 3:038b8840:::rbd_data.1133dc5adfdf.0000000000000035:head v 31'50, mlcod=31'48)", 2024-09-21T15:50:25.986 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.232497+0000", 2024-09-21T15:50:25.986 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 68.738115647000001, 2024-09-21T15:50:25.986 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0040493559999999996, 2024-09-21T15:50:25.986 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:25.986 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:25.986 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:25.986 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.986 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:25.986 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232497+0000", 2024-09-21T15:50:25.987 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:25.987 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.987 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.987 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:25.987 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232497+0000", 2024-09-21T15:50:25.987 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:25.987 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.987 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.987 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:25.987 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232496+0000", 2024-09-21T15:50:25.987 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:25.987 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.987 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.987 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:25.988 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232503+0000", 2024-09-21T15:50:25.988 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7320000000000001e-06 2024-09-21T15:50:25.988 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.988 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.988 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:25.988 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232512+0000", 2024-09-21T15:50:25.988 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9259999999999995e-06 2024-09-21T15:50:25.988 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.988 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.988 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:25.988 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232520+0000", 2024-09-21T15:50:25.988 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3699999999999997e-06 2024-09-21T15:50:25.988 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.988 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.989 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:25.989 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232564+0000", 2024-09-21T15:50:25.989 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4128999999999997e-05 2024-09-21T15:50:25.989 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.989 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.989 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:25.989 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232585+0000", 2024-09-21T15:50:25.989 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0704999999999999e-05 2024-09-21T15:50:25.989 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.989 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.989 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:25.989 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.236507+0000", 2024-09-21T15:50:25.989 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0039218730000000002 2024-09-21T15:50:25.989 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.989 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.990 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:25.990 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.236546+0000", 2024-09-21T15:50:25.990 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9399e-05 2024-09-21T15:50:25.990 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:25.990 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:25.990 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:25.990 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.990 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.990 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13452 3.9 e31/22 3:93f5396a:::rbd_data.1133dc5adfdf.0000000000000078:head v 31'50, mlcod=31'47)", 2024-09-21T15:50:25.990 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.237263+0000", 2024-09-21T15:50:25.990 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 68.733348941000003, 2024-09-21T15:50:25.990 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033768180000000002, 2024-09-21T15:50:25.990 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:25.990 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:25.990 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:25.991 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.991 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:25.991 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237263+0000", 2024-09-21T15:50:25.991 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:25.991 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.991 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.991 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:25.991 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237263+0000", 2024-09-21T15:50:25.991 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:25.991 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.991 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.991 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:25.992 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237263+0000", 2024-09-21T15:50:25.992 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:25.992 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.992 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.992 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:25.992 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237269+0000", 2024-09-21T15:50:25.992 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.472e-06 2024-09-21T15:50:25.992 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.992 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.992 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:25.992 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237282+0000", 2024-09-21T15:50:25.992 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2788e-05 2024-09-21T15:50:25.992 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.992 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.992 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:25.993 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237288+0000", 2024-09-21T15:50:25.993 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4899999999999997e-06 2024-09-21T15:50:25.993 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.993 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.993 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:25.993 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237331+0000", 2024-09-21T15:50:25.993 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2425000000000001e-05 2024-09-21T15:50:25.993 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.993 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.993 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:25.993 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237349+0000", 2024-09-21T15:50:25.993 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8405000000000001e-05 2024-09-21T15:50:25.993 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.993 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.994 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:25.994 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.240610+0000", 2024-09-21T15:50:25.994 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032605379999999999 2024-09-21T15:50:25.994 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.994 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.994 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:25.994 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.240640+0000", 2024-09-21T15:50:25.994 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0448000000000001e-05 2024-09-21T15:50:25.994 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:25.994 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:25.994 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:25.994 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.994 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.994 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13832 3.0 e31/22 3:0792efd2:::rbd_data.1133dc5adfdf.0000000000000090:head v 31'58, mlcod=31'56)", 2024-09-21T15:50:25.995 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.527530+0000", 2024-09-21T15:50:25.995 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 68.443082121000003, 2024-09-21T15:50:25.995 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035256530000000001, 2024-09-21T15:50:25.995 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:25.995 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:25.995 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:25.995 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.995 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:25.995 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527530+0000", 2024-09-21T15:50:25.995 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:25.995 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.995 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.995 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:25.995 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527530+0000", 2024-09-21T15:50:25.996 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:25.996 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.996 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.996 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:25.996 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527529+0000", 2024-09-21T15:50:25.996 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:50:25.996 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.996 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.996 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:25.996 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527550+0000", 2024-09-21T15:50:25.996 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1387999999999999e-05 2024-09-21T15:50:25.996 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.996 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.996 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:25.997 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527568+0000", 2024-09-21T15:50:25.997 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8068e-05 2024-09-21T15:50:25.997 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.997 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.997 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:25.997 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527577+0000", 2024-09-21T15:50:25.997 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9919999999999996e-06 2024-09-21T15:50:25.997 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.997 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.997 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:25.997 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527626+0000", 2024-09-21T15:50:25.997 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8597000000000002e-05 2024-09-21T15:50:25.997 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.997 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.998 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:25.998 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527652+0000", 2024-09-21T15:50:25.998 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5993e-05 2024-09-21T15:50:25.998 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.998 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.998 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:25.998 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.531024+0000", 2024-09-21T15:50:25.998 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033721910000000001 2024-09-21T15:50:25.998 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.998 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.998 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:25.998 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.531056+0000", 2024-09-21T15:50:25.998 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1628000000000003e-05 2024-09-21T15:50:25.998 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:25.999 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:25.999 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:25.999 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:25.999 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.999 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13839 3.0 e31/22 3:03c4fb32:::rbd_data.1133dc5adfdf.0000000000000099:head v 31'59, mlcod=31'56)", 2024-09-21T15:50:25.999 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.528097+0000", 2024-09-21T15:50:25.999 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 68.442515522999997, 2024-09-21T15:50:25.999 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0057228840000000001, 2024-09-21T15:50:25.999 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:25.999 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:25.999 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:25.999 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:25.999 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:25.999 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528097+0000", 2024-09-21T15:50:25.999 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.000 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.000 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.000 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:26.000 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528097+0000", 2024-09-21T15:50:26.000 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.000 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.000 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.000 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:26.000 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528096+0000", 2024-09-21T15:50:26.000 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:26.000 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.000 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.000 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:26.000 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528105+0000", 2024-09-21T15:50:26.001 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7350000000000002e-06 2024-09-21T15:50:26.001 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.001 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.001 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:26.001 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528117+0000", 2024-09-21T15:50:26.001 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1909000000000001e-05 2024-09-21T15:50:26.001 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.001 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.001 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:26.001 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528124+0000", 2024-09-21T15:50:26.001 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8419999999999999e-06 2024-09-21T15:50:26.001 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.001 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.001 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:26.001 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528168+0000", 2024-09-21T15:50:26.002 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4301999999999997e-05 2024-09-21T15:50:26.002 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.002 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.002 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:26.002 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528191+0000", 2024-09-21T15:50:26.002 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2622000000000001e-05 2024-09-21T15:50:26.002 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.002 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.002 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:26.002 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.533780+0000", 2024-09-21T15:50:26.002 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0055897519999999999 2024-09-21T15:50:26.002 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.002 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.002 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:26.002 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.533820+0000", 2024-09-21T15:50:26.003 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9366999999999998e-05 2024-09-21T15:50:26.003 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.003 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:26.003 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.003 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.003 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.003 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13884 3.9 e31/22 3:93003736:::rbd_data.1133dc5adfdf.00000000000000a1:head v 31'54, mlcod=31'52)", 2024-09-21T15:50:26.003 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.562433+0000", 2024-09-21T15:50:26.003 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 68.408178774999996, 2024-09-21T15:50:26.003 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.005162313, 2024-09-21T15:50:26.003 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:26.003 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:26.003 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:26.003 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.004 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:26.004 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562433+0000", 2024-09-21T15:50:26.004 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.004 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.004 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.004 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:26.004 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562433+0000", 2024-09-21T15:50:26.004 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.004 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.004 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.004 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:26.004 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562432+0000", 2024-09-21T15:50:26.004 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:26.005 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.005 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.005 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:26.005 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562467+0000", 2024-09-21T15:50:26.005 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4885999999999998e-05 2024-09-21T15:50:26.005 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.005 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.005 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:26.005 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562481+0000", 2024-09-21T15:50:26.005 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3604000000000001e-05 2024-09-21T15:50:26.005 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.005 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.005 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:26.005 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562489+0000", 2024-09-21T15:50:26.006 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1829999999999994e-06 2024-09-21T15:50:26.006 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.006 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.006 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:26.006 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562536+0000", 2024-09-21T15:50:26.006 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6830999999999999e-05 2024-09-21T15:50:26.006 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.006 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.006 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:26.006 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562559+0000", 2024-09-21T15:50:26.006 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2765999999999999e-05 2024-09-21T15:50:26.006 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.006 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.006 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:26.007 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.567566+0000", 2024-09-21T15:50:26.007 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0050073219999999998 2024-09-21T15:50:26.007 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.007 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.007 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:26.007 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.567596+0000", 2024-09-21T15:50:26.007 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9802999999999999e-05 2024-09-21T15:50:26.007 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.007 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:26.007 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.007 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.007 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.007 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:22340 3.9 e34/22 3:910a5abe:::rbd_data.1133896adcfd.0000000000000023:head v 34'92, mlcod=34'90)", 2024-09-21T15:50:26.007 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:21.628025+0000", 2024-09-21T15:50:26.007 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 64.342587394999995, 2024-09-21T15:50:26.008 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046196010000000001, 2024-09-21T15:50:26.008 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:26.008 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:26.008 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:26.008 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.008 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:26.008 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628025+0000", 2024-09-21T15:50:26.008 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.008 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.008 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.008 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:26.008 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628025+0000", 2024-09-21T15:50:26.008 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.008 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.009 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.009 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:26.009 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628024+0000", 2024-09-21T15:50:26.009 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:26.009 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.009 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.009 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:26.009 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628034+0000", 2024-09-21T15:50:26.009 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3470000000000005e-06 2024-09-21T15:50:26.009 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.009 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.009 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:26.009 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628043+0000", 2024-09-21T15:50:26.009 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9450000000000005e-06 2024-09-21T15:50:26.009 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.010 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.010 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:26.010 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628048+0000", 2024-09-21T15:50:26.010 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8080000000000004e-06 2024-09-21T15:50:26.010 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.010 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.010 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:26.010 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628081+0000", 2024-09-21T15:50:26.010 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2738999999999999e-05 2024-09-21T15:50:26.010 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.010 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.010 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:26.010 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628094+0000", 2024-09-21T15:50:26.010 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3273e-05 2024-09-21T15:50:26.011 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.011 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.011 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:26.011 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.632616+0000", 2024-09-21T15:50:26.011 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045217269999999997 2024-09-21T15:50:26.011 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.011 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.011 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:26.011 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.632644+0000", 2024-09-21T15:50:26.011 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8469000000000001e-05 2024-09-21T15:50:26.011 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.011 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:26.011 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.011 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.011 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.012 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:22417 3.4 e34/22 3:25310a10:::rbd_data.1133896adcfd.0000000000000075:head v 34'48, mlcod=34'46)", 2024-09-21T15:50:26.012 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:21.704297+0000", 2024-09-21T15:50:26.012 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 64.266315297000006, 2024-09-21T15:50:26.012 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0038101799999999998, 2024-09-21T15:50:26.012 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:26.012 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:26.012 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:26.012 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.012 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:26.012 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704297+0000", 2024-09-21T15:50:26.012 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.012 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.012 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.012 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:26.013 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704297+0000", 2024-09-21T15:50:26.013 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.013 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.013 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.013 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:26.013 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704296+0000", 2024-09-21T15:50:26.013 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:50:26.013 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.013 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.013 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:26.013 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704308+0000", 2024-09-21T15:50:26.013 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2559e-05 2024-09-21T15:50:26.013 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.013 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.013 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:26.014 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704337+0000", 2024-09-21T15:50:26.014 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8960000000000001e-05 2024-09-21T15:50:26.014 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.014 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.014 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:26.014 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704345+0000", 2024-09-21T15:50:26.014 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3059999999999998e-06 2024-09-21T15:50:26.014 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.014 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.014 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:26.014 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704408+0000", 2024-09-21T15:50:26.014 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2156000000000006e-05 2024-09-21T15:50:26.014 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.014 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.015 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:26.015 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704430+0000", 2024-09-21T15:50:26.015 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1931000000000001e-05 2024-09-21T15:50:26.015 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.015 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.015 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:26.015 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.708073+0000", 2024-09-21T15:50:26.015 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003643028 2024-09-21T15:50:26.015 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.015 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.015 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:26.015 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.708107+0000", 2024-09-21T15:50:26.015 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4613999999999998e-05 2024-09-21T15:50:26.015 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.015 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:26.016 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.016 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.016 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.016 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:25278 3.4 e38/22 3:2251cc1c:::rbd_data.1133ebd6d3ba.0000000000000024:head v 38'51, mlcod=38'49)", 2024-09-21T15:50:26.016 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.317337+0000", 2024-09-21T15:50:26.016 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 59.653275536999999, 2024-09-21T15:50:26.016 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053904110000000003, 2024-09-21T15:50:26.016 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:26.016 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:26.016 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:26.016 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.016 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:26.016 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317337+0000", 2024-09-21T15:50:26.016 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.016 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.017 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.017 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:26.017 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317337+0000", 2024-09-21T15:50:26.017 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.017 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.017 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.017 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:26.017 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317335+0000", 2024-09-21T15:50:26.017 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:50:26.017 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.017 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.017 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:26.017 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317347+0000", 2024-09-21T15:50:26.017 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.19e-05 2024-09-21T15:50:26.018 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.018 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.018 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:26.018 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317369+0000", 2024-09-21T15:50:26.018 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2232e-05 2024-09-21T15:50:26.018 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.018 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.018 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:26.018 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317381+0000", 2024-09-21T15:50:26.018 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2231e-05 2024-09-21T15:50:26.018 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.018 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.018 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:26.018 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317425+0000", 2024-09-21T15:50:26.018 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3625000000000003e-05 2024-09-21T15:50:26.019 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.019 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.019 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:26.019 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317449+0000", 2024-09-21T15:50:26.019 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4301999999999999e-05 2024-09-21T15:50:26.019 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.019 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.019 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:26.019 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.322688+0000", 2024-09-21T15:50:26.019 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.005238609 2024-09-21T15:50:26.019 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.019 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.019 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:26.019 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.322727+0000", 2024-09-21T15:50:26.019 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9201e-05 2024-09-21T15:50:26.019 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.020 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:26.020 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.020 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.020 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.020 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:25286 3.0 e38/22 3:0248fb60:::rbd_data.1133ebd6d3ba.0000000000000066:head v 38'109, mlcod=34'107)", 2024-09-21T15:50:26.020 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.317638+0000", 2024-09-21T15:50:26.020 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 59.652974657999998, 2024-09-21T15:50:26.020 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0054074370000000002, 2024-09-21T15:50:26.020 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:26.020 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:26.020 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:26.020 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.020 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:26.020 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317638+0000", 2024-09-21T15:50:26.020 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.021 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.021 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:26.021 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317638+0000", 2024-09-21T15:50:26.021 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.021 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.021 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:26.021 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317637+0000", 2024-09-21T15:50:26.021 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:26.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.021 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.021 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:26.021 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317643+0000", 2024-09-21T15:50:26.022 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5609999999999998e-06 2024-09-21T15:50:26.022 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.022 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.022 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:26.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317653+0000", 2024-09-21T15:50:26.022 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0705e-05 2024-09-21T15:50:26.022 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.022 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.022 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:26.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317660+0000", 2024-09-21T15:50:26.022 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2099999999999998e-06 2024-09-21T15:50:26.022 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.022 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.023 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:26.023 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317704+0000", 2024-09-21T15:50:26.023 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4659000000000001e-05 2024-09-21T15:50:26.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.023 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.023 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:26.023 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317725+0000", 2024-09-21T15:50:26.023 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0537e-05 2024-09-21T15:50:26.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.023 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.023 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:26.023 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323010+0000", 2024-09-21T15:50:26.023 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052853420000000002 2024-09-21T15:50:26.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.023 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.024 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:26.024 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323045+0000", 2024-09-21T15:50:26.024 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4860000000000002e-05 2024-09-21T15:50:26.024 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.024 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:26.024 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.024 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.024 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.024 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:25280 3.4 e38/22 3:27fca38c:::rbd_data.1133ebd6d3ba.0000000000000026:head v 38'52, mlcod=38'49)", 2024-09-21T15:50:26.024 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.317684+0000", 2024-09-21T15:50:26.024 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 59.652928762000002, 2024-09-21T15:50:26.024 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053570809999999996, 2024-09-21T15:50:26.024 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:26.024 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:26.025 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:26.025 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.025 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:26.025 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317684+0000", 2024-09-21T15:50:26.025 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.025 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.025 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.025 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:26.025 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317684+0000", 2024-09-21T15:50:26.025 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.025 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.025 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.025 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:26.026 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317683+0000", 2024-09-21T15:50:26.026 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:26.026 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.026 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.026 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:26.026 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317688+0000", 2024-09-21T15:50:26.026 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5009999999999998e-06 2024-09-21T15:50:26.026 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.026 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.026 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:26.026 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317694+0000", 2024-09-21T15:50:26.026 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3579999999999999e-06 2024-09-21T15:50:26.026 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.026 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.027 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:26.027 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317698+0000", 2024-09-21T15:50:26.027 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4150000000000001e-06 2024-09-21T15:50:26.027 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.027 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.027 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:26.027 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317738+0000", 2024-09-21T15:50:26.027 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9575e-05 2024-09-21T15:50:26.027 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.027 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.027 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:26.027 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317762+0000", 2024-09-21T15:50:26.027 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4173999999999999e-05 2024-09-21T15:50:26.027 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.028 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.028 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:26.028 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323005+0000", 2024-09-21T15:50:26.028 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00524333 2024-09-21T15:50:26.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.028 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.028 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:26.028 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323041+0000", 2024-09-21T15:50:26.028 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5139999999999999e-05 2024-09-21T15:50:26.028 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.028 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:26.028 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.029 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.029 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:34837 3.0 e43/22 3:038527c8:::rbd_header.1133fed1a840:head v 43'130, mlcod=43'128)", 2024-09-21T15:50:26.029 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:50:26.029 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 54.986724064000001, 2024-09-21T15:50:26.029 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045776519999999998, 2024-09-21T15:50:26.029 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:26.029 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:26.029 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:26.029 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.029 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:26.029 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:50:26.029 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.029 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.029 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.030 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:26.030 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:50:26.030 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.030 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.030 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.030 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:26.030 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:50:26.030 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:26.030 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.030 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.030 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:26.030 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983892+0000", 2024-09-21T15:50:26.030 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8450000000000002e-06 2024-09-21T15:50:26.031 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.031 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.031 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:26.031 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983902+0000", 2024-09-21T15:50:26.031 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0299999999999999e-06 2024-09-21T15:50:26.031 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.031 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.031 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:26.031 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983906+0000", 2024-09-21T15:50:26.031 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2300000000000002e-06 2024-09-21T15:50:26.031 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.031 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.031 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:26.031 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983940+0000", 2024-09-21T15:50:26.031 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4563000000000001e-05 2024-09-21T15:50:26.032 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.032 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.032 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:26.032 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983960+0000", 2024-09-21T15:50:26.032 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9199999999999999e-05 2024-09-21T15:50:26.032 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.032 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.032 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:26.032 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.988430+0000", 2024-09-21T15:50:26.032 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0044709149999999998 2024-09-21T15:50:26.032 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.032 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.032 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:26.032 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.988466+0000", 2024-09-21T15:50:26.033 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5420000000000003e-05 2024-09-21T15:50:26.033 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.033 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:26.033 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.033 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.033 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.033 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "pg_update_log_missing(3.0 epoch 50/22 rep_tid 498 entries 50'171 (0'0) error 3:5df4a5a0:::rbd_data.11339e87a32a.000000000000008e:head by client.4403.0:80207 0.000000 -2 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 47'167)", 2024-09-21T15:50:26.033 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:50:26.033 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 45.028851191000001, 2024-09-21T15:50:26.033 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0047253119999999997, 2024-09-21T15:50:26.033 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:26.033 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-09-21T15:50:26.033 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:26.033 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.033 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:26.034 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:50:26.034 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.034 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.034 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:26.034 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:50:26.034 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.034 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.034 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:26.034 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:50:26.034 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:26.034 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.035 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:26.035 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941768+0000", 2024-09-21T15:50:26.035 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7309999999999993e-06 2024-09-21T15:50:26.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.035 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:26.035 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941780+0000", 2024-09-21T15:50:26.035 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1399999999999999e-05 2024-09-21T15:50:26.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.035 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:26.035 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941789+0000", 2024-09-21T15:50:26.035 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5230000000000001e-06 2024-09-21T15:50:26.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.036 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:26.036 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941841+0000", 2024-09-21T15:50:26.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1863999999999997e-05 2024-09-21T15:50:26.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.036 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:26.036 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.946486+0000", 2024-09-21T15:50:26.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046452539999999997 2024-09-21T15:50:26.036 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.036 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:26.036 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.036 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:82086 3.0 e50/22 3:0114bc4d:::rbd_data.11334c14b8fd.0000000000000069:head v 50'178, mlcod=50'176)", 2024-09-21T15:50:26.037 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:41.155591+0000", 2024-09-21T15:50:26.037 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 44.815020879000002, 2024-09-21T15:50:26.037 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003363497, 2024-09-21T15:50:26.037 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:26.037 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:26.037 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:26.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.037 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:26.037 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.155591+0000", 2024-09-21T15:50:26.037 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.038 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:26.038 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.155591+0000", 2024-09-21T15:50:26.038 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.038 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.038 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.038 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:26.038 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.155590+0000", 2024-09-21T15:50:26.038 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:50:26.038 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.038 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.038 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:26.038 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.155606+0000", 2024-09-21T15:50:26.038 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6075000000000001e-05 2024-09-21T15:50:26.038 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.039 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.039 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:26.039 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.155623+0000", 2024-09-21T15:50:26.039 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6838e-05 2024-09-21T15:50:26.039 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.039 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.039 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:26.039 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.155630+0000", 2024-09-21T15:50:26.039 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.627e-06 2024-09-21T15:50:26.039 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.039 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.039 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:26.039 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.155677+0000", 2024-09-21T15:50:26.039 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6669999999999999e-05 2024-09-21T15:50:26.040 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.040 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:26.040 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.155698+0000", 2024-09-21T15:50:26.040 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1330000000000001e-05 2024-09-21T15:50:26.040 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.040 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:26.040 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.158918+0000", 2024-09-21T15:50:26.040 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032198600000000002 2024-09-21T15:50:26.040 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.040 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:26.040 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.158955+0000", 2024-09-21T15:50:26.040 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6647000000000003e-05 2024-09-21T15:50:26.041 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.041 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:26.041 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.041 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.041 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.041 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:82470 3.4 e50/22 3:21ad7dfe:::rbd_data.11334c14b8fd.00000000000003a7:head v 50'190, mlcod=46'188)", 2024-09-21T15:50:26.041 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:41.291787+0000", 2024-09-21T15:50:26.041 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 44.678825254000003, 2024-09-21T15:50:26.041 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033489969999999998, 2024-09-21T15:50:26.041 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:26.041 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:26.042 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:26.042 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.042 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:26.042 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.291787+0000", 2024-09-21T15:50:26.042 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.042 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.042 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:26.042 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.291787+0000", 2024-09-21T15:50:26.042 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.042 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.042 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:26.043 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.291786+0000", 2024-09-21T15:50:26.043 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:26.043 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.043 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.043 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:26.043 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.291795+0000", 2024-09-21T15:50:26.043 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7830000000000004e-06 2024-09-21T15:50:26.043 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.043 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.043 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:26.043 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.291814+0000", 2024-09-21T15:50:26.043 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9117999999999999e-05 2024-09-21T15:50:26.043 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.043 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.044 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:26.044 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.291822+0000", 2024-09-21T15:50:26.044 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0669999999999992e-06 2024-09-21T15:50:26.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.044 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:26.044 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.291866+0000", 2024-09-21T15:50:26.044 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4669000000000002e-05 2024-09-21T15:50:26.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.044 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:26.044 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.291888+0000", 2024-09-21T15:50:26.044 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1087999999999999e-05 2024-09-21T15:50:26.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:26.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.295104+0000", 2024-09-21T15:50:26.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032164910000000001 2024-09-21T15:50:26.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:26.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:41.295136+0000", 2024-09-21T15:50:26.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1931999999999999e-05 2024-09-21T15:50:26.045 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.045 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:26.045 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.045 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "pg_update_log_missing(3.0 epoch 53/22 rep_tid 585 entries 53'201 (0'0) error 3:6a1d4890:::rbd_data.1133e079cae4.0000000000000004:head by client.4403.0:101089 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 53'198)", 2024-09-21T15:50:26.046 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:44.832827+0000", 2024-09-21T15:50:26.046 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 41.137785383000001, 2024-09-21T15:50:26.046 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053552929999999997, 2024-09-21T15:50:26.046 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:26.046 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-09-21T15:50:26.046 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:26.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.046 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:26.046 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832827+0000", 2024-09-21T15:50:26.046 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.046 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:26.046 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832827+0000", 2024-09-21T15:50:26.047 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.047 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.047 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.047 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:26.047 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832826+0000", 2024-09-21T15:50:26.047 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:26.047 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.047 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.047 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:26.047 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832835+0000", 2024-09-21T15:50:26.047 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4509999999999992e-06 2024-09-21T15:50:26.047 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.047 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.047 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:26.047 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832854+0000", 2024-09-21T15:50:26.048 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9352999999999999e-05 2024-09-21T15:50:26.048 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.048 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.048 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:26.048 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832866+0000", 2024-09-21T15:50:26.048 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1766000000000001e-05 2024-09-21T15:50:26.048 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.048 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.048 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:26.048 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.833091+0000", 2024-09-21T15:50:26.048 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00022490300000000001 2024-09-21T15:50:26.048 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.048 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.049 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:26.049 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.838182+0000", 2024-09-21T15:50:26.049 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0050915420000000001 2024-09-21T15:50:26.049 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.049 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:26.049 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.049 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.049 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.049 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:102822 3.4 e54/22 3:2729ba03:::rbd_object_map.113336ae1075:head v 54'208, mlcod=54'205)", 2024-09-21T15:50:26.049 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:46.249540+0000", 2024-09-21T15:50:26.049 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 39.721072151000001, 2024-09-21T15:50:26.049 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052654859999999998, 2024-09-21T15:50:26.049 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:26.049 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:26.049 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:26.050 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.050 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:26.050 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249540+0000", 2024-09-21T15:50:26.050 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.050 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.050 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.050 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:26.050 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249540+0000", 2024-09-21T15:50:26.050 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.050 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.050 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.050 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:26.050 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249539+0000", 2024-09-21T15:50:26.050 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:26.050 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.051 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.051 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:26.051 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249546+0000", 2024-09-21T15:50:26.051 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7290000000000003e-06 2024-09-21T15:50:26.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.051 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.051 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:26.051 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249558+0000", 2024-09-21T15:50:26.051 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1844e-05 2024-09-21T15:50:26.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.051 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.051 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:26.051 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249565+0000", 2024-09-21T15:50:26.051 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8210000000000004e-06 2024-09-21T15:50:26.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.052 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.052 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:26.052 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249602+0000", 2024-09-21T15:50:26.052 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7057000000000001e-05 2024-09-21T15:50:26.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.052 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.052 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:26.052 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249622+0000", 2024-09-21T15:50:26.052 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9678999999999999e-05 2024-09-21T15:50:26.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.052 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.052 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:26.052 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254783+0000", 2024-09-21T15:50:26.052 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0051611449999999998 2024-09-21T15:50:26.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.053 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.053 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:26.053 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254806+0000", 2024-09-21T15:50:26.053 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2853e-05 2024-09-21T15:50:26.053 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.053 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:26.053 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.053 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.053 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.053 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:102823 3.4 e54/22 3:2729ba03:::rbd_object_map.113336ae1075:head v 54'209, mlcod=54'205)", 2024-09-21T15:50:26.053 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:46.250318+0000", 2024-09-21T15:50:26.053 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 39.720294566, 2024-09-21T15:50:26.053 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045013539999999999, 2024-09-21T15:50:26.053 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:26.054 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:26.054 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:26.054 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.054 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:26.054 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250318+0000", 2024-09-21T15:50:26.054 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.054 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.054 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.054 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:26.054 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250318+0000", 2024-09-21T15:50:26.054 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.054 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.054 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.054 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:26.054 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250317+0000", 2024-09-21T15:50:26.055 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:26.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.055 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.055 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:26.055 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250322+0000", 2024-09-21T15:50:26.055 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.074e-06 2024-09-21T15:50:26.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.055 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.055 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:26.055 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250330+0000", 2024-09-21T15:50:26.055 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4730000000000003e-06 2024-09-21T15:50:26.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.055 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.055 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:26.056 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250335+0000", 2024-09-21T15:50:26.056 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8019999999999997e-06 2024-09-21T15:50:26.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.056 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.056 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:26.056 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250375+0000", 2024-09-21T15:50:26.056 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9459999999999998e-05 2024-09-21T15:50:26.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.056 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.056 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:26.056 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250388+0000", 2024-09-21T15:50:26.056 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3288999999999999e-05 2024-09-21T15:50:26.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.056 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.057 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:26.057 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254807+0000", 2024-09-21T15:50:26.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0044192220000000004 2024-09-21T15:50:26.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.057 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:26.057 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254819+0000", 2024-09-21T15:50:26.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1603999999999999e-05 2024-09-21T15:50:26.057 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.057 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:26.057 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.057 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.057 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:26.057 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:26.058 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:25.971+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:26.058 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:50:26.073 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:26.072+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:26.174 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:26.172+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:26.195 INFO:teuthology.orchestra.run.smithi089.stdout:truncating image image_client.0-clone43 from 0x9dfaed9 (overlap 0x321a6) to 0x28151 2024-09-21T15:50:26.212 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:26.212 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:26.213 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:50:26.213 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:26.222 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:50:26.256 INFO:teuthology.orchestra.run.smithi089.stdout:checking clone #42, image image_client.0-clone42 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent43 2024-09-21T15:50:26.257 INFO:teuthology.orchestra.run.smithi089.stdout:5829 read 0x988c77e thru 0x989ae85 (0xe708 bytes) 2024-09-21T15:50:26.269 INFO:teuthology.orchestra.run.smithi089.stdout:5830 trunc from 0x9dfaed9 to 0x21ba46 2024-09-21T15:50:26.275 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:26.273+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:26.288 INFO:teuthology.orchestra.run.smithi089.stdout:5831 read 0x5add0 thru 0x68f6e (0xe19f bytes) 2024-09-21T15:50:26.297 INFO:teuthology.orchestra.run.smithi089.stdout:5832 write 0x3cb06ad thru 0x3cbc6e6 (0xc03a bytes) 2024-09-21T15:50:26.300 INFO:teuthology.orchestra.run.smithi089.stdout:5833 read 0x3b44ee2 thru 0x3b53d97 (0xeeb6 bytes) 2024-09-21T15:50:26.300 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:26.300 INFO:teuthology.orchestra.run.smithi089.stdout:5836 read 0x2783dbd thru 0x2784f98 (0x11dc bytes) 2024-09-21T15:50:26.301 INFO:teuthology.orchestra.run.smithi089.stdout:5837 trunc from 0x3cbc6e7 to 0x3206906 2024-09-21T15:50:26.311 INFO:teuthology.orchestra.run.smithi089.stdout:5838 write 0x1253bc4 thru 0x1260f47 (0xd384 bytes) 2024-09-21T15:50:26.331 INFO:teuthology.orchestra.run.smithi089.stdout:5839 writesame 0xcff0f59 thru 0xcff8370 data_size 0x14(0x7418 bytes) 2024-09-21T15:50:26.331 INFO:teuthology.orchestra.run.smithi089.stdout:5840 read 0xcfa9191 thru 0xcfb0bc4 (0x7a34 bytes) 2024-09-21T15:50:26.331 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:26.331 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:26.331 INFO:teuthology.orchestra.run.smithi089.stdout:5843 write 0xc61c97f thru 0xc62727a (0xa8fc bytes) 2024-09-21T15:50:26.332 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:26.332 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:26.332 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:50:26.332 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:50:26.332 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:26.334 INFO:teuthology.orchestra.run.smithi089.stdout:5844 writesame 0x97dc925 thru 0x97dca86 data_size 0x2(0x162 bytes) 2024-09-21T15:50:26.337 INFO:teuthology.orchestra.run.smithi089.stdout:5845 punch from 0x15b21c3 to 0x15c15d6, (0xf413 bytes) 2024-09-21T15:50:26.338 INFO:teuthology.orchestra.run.smithi089.stdout:5847 read 0x1dd2d77 thru 0x1dda80e (0x7a98 bytes) 2024-09-21T15:50:26.338 INFO:teuthology.orchestra.run.smithi089.stdout:5848 read 0xcadaeeb thru 0xcaeab98 (0xfcae bytes) 2024-09-21T15:50:26.338 INFO:teuthology.orchestra.run.smithi089.stdout:5849 writesame 0xa48e0b5 thru 0xa4914cc data_size 0x4(0x3418 bytes) 2024-09-21T15:50:26.342 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:50:26.342 INFO:teuthology.orchestra.run.smithi089.stdout:5850 read 0x8c14318 thru 0x8c2354c (0xf235 bytes) 2024-09-21T15:50:26.342 INFO:teuthology.orchestra.run.smithi089.stdout:5851 punch from 0xc753809 to 0xc7616fc, (0xdef3 bytes) 2024-09-21T15:50:26.343 INFO:teuthology.orchestra.run.smithi089.stdout:5852 trunc from 0xcff8371 to 0xbea7ae1 2024-09-21T15:50:26.353 INFO:teuthology.orchestra.run.smithi089.stdout:5855 write 0xb3f81d6 thru 0xb4076f8 (0xf523 bytes) 2024-09-21T15:50:26.369 INFO:teuthology.orchestra.run.smithi089.stdout:5857 trunc from 0xbea7ae1 to 0x205a1b6 2024-09-21T15:50:26.375 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:26.374+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:26.392 INFO:teuthology.orchestra.run.smithi089.stdout:5859 write 0x7bcc9e5 thru 0x7bd275b (0x5d77 bytes) 2024-09-21T15:50:26.395 INFO:teuthology.orchestra.run.smithi089.stdout:5861 writesame 0x565845e thru 0x5662658 data_size 0x1(0xa1fb bytes) 2024-09-21T15:50:26.417 INFO:teuthology.orchestra.run.smithi089.stdout:5862 read 0x6452dde thru 0x645d113 (0xa336 bytes) 2024-09-21T15:50:26.417 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:26.417 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:26.427 INFO:teuthology.orchestra.run.smithi089.stdout:5865 writesame 0xbbc7255 thru 0xbbd5342 data_size 0x3f(0xe0ee bytes) 2024-09-21T15:50:26.427 INFO:teuthology.orchestra.run.smithi089.stdout:5866 read 0x15c76a9 thru 0x15cfa7b (0x83d3 bytes) 2024-09-21T15:50:26.427 INFO:teuthology.orchestra.run.smithi089.stdout:5867 read 0x28a7409 thru 0x28a8437 (0x102f bytes) 2024-09-21T15:50:26.434 INFO:teuthology.orchestra.run.smithi089.stdout:5868 write 0xdd89e7b thru 0xdd8ab9f (0xd25 bytes) 2024-09-21T15:50:26.447 INFO:teuthology.orchestra.run.smithi089.stdout:5871 trunc from 0xdd8aba0 to 0x5a7bdc4 2024-09-21T15:50:26.456 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:26.456 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:50:26.456 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:50:26.456 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:50:26.456 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.457 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:14051 3.8 3:13437810:::rbd_data.1133dc5adfdf.0000000000000039:head [write 716800~796] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-21T15:50:26.457 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:50:26.457 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 68.759175455999994, 2024-09-21T15:50:26.457 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.98064002699999997, 2024-09-21T15:50:26.457 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:26.457 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:26.457 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:26.457 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:26.457 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:26.457 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 14051 2024-09-21T15:50:26.457 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.457 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:26.457 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.458 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:26.458 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:50:26.458 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.458 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.458 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.458 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:26.458 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:50:26.458 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.458 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.458 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.458 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:26.458 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:50:26.458 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:26.458 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.459 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.459 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:26.459 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695131+0000", 2024-09-21T15:50:26.459 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4680000000000003e-06 2024-09-21T15:50:26.459 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.459 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.459 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:26.459 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695139+0000", 2024-09-21T15:50:26.459 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9400000000000002e-06 2024-09-21T15:50:26.459 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.459 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.459 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:26.459 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695143+0000", 2024-09-21T15:50:26.459 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9899999999999999e-06 2024-09-21T15:50:26.460 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.460 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.460 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:26.460 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695190+0000", 2024-09-21T15:50:26.460 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6458000000000001e-05 2024-09-21T15:50:26.460 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.460 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.460 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:50:26.460 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695201+0000", 2024-09-21T15:50:26.460 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0964999999999999e-05 2024-09-21T15:50:26.460 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.460 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.460 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:26.460 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.669490+0000", 2024-09-21T15:50:26.460 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.97428980499999995 2024-09-21T15:50:26.461 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.461 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.461 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-09-21T15:50:26.461 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.670033+0000", 2024-09-21T15:50:26.461 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00054252599999999999 2024-09-21T15:50:26.461 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.461 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.461 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:26.461 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.671095+0000", 2024-09-21T15:50:26.461 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001062183 2024-09-21T15:50:26.461 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.461 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.461 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:26.461 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.671186+0000", 2024-09-21T15:50:26.462 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1191000000000002e-05 2024-09-21T15:50:26.462 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.462 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.462 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-09-21T15:50:26.462 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.671346+0000", 2024-09-21T15:50:26.462 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015980399999999999 2024-09-21T15:50:26.462 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.462 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.462 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:26.462 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.672221+0000", 2024-09-21T15:50:26.462 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00087471999999999995 2024-09-21T15:50:26.462 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.462 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.462 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:26.463 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.675619+0000", 2024-09-21T15:50:26.463 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00339858 2024-09-21T15:50:26.463 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.463 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.463 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:26.463 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.675670+0000", 2024-09-21T15:50:26.463 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0664000000000001e-05 2024-09-21T15:50:26.463 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.463 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.463 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:26.463 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.675768+0000", 2024-09-21T15:50:26.463 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7415000000000002e-05 2024-09-21T15:50:26.463 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.463 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:26.463 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.464 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.464 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.464 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116235 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 761856~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:26.464 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048806+0000", 2024-09-21T15:50:26.464 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 30.405496572000001, 2024-09-21T15:50:26.464 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056818556999999999, 2024-09-21T15:50:26.464 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:26.464 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:26.464 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:26.464 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:26.464 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:26.464 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116235 2024-09-21T15:50:26.464 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.464 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:26.464 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.465 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:26.465 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048806+0000", 2024-09-21T15:50:26.465 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.465 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.465 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.465 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:26.465 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048806+0000", 2024-09-21T15:50:26.465 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.465 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.465 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.465 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:26.465 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048806+0000", 2024-09-21T15:50:26.465 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:26.465 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.465 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.466 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:26.466 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048807+0000", 2024-09-21T15:50:26.466 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.001e-06 2024-09-21T15:50:26.466 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.466 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.466 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:26.466 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048810+0000", 2024-09-21T15:50:26.466 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.711e-06 2024-09-21T15:50:26.466 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.466 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.466 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:26.466 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048812+0000", 2024-09-21T15:50:26.466 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2290000000000002e-06 2024-09-21T15:50:26.466 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.467 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.467 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:26.467 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105495+0000", 2024-09-21T15:50:26.467 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056682679999999999 2024-09-21T15:50:26.467 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.467 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.467 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:26.467 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105565+0000", 2024-09-21T15:50:26.467 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9868000000000003e-05 2024-09-21T15:50:26.467 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.467 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.467 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:26.467 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105625+0000", 2024-09-21T15:50:26.467 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0268e-05 2024-09-21T15:50:26.467 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.468 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:26.468 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.468 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.468 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.468 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116236 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 768000~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:26.468 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048814+0000", 2024-09-21T15:50:26.468 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 30.405488908999999, 2024-09-21T15:50:26.468 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056959587999999998, 2024-09-21T15:50:26.468 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:26.468 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:26.468 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:26.468 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:26.468 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:26.468 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116236 2024-09-21T15:50:26.468 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.468 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:26.469 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.469 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:26.469 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048814+0000", 2024-09-21T15:50:26.469 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.469 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.469 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.469 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:26.469 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048814+0000", 2024-09-21T15:50:26.469 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.469 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.469 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.469 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:26.469 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048814+0000", 2024-09-21T15:50:26.469 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:26.469 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.469 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.470 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:26.470 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048815+0000", 2024-09-21T15:50:26.470 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7999999999999993e-07 2024-09-21T15:50:26.470 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.470 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.470 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:26.470 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048817+0000", 2024-09-21T15:50:26.470 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6620000000000001e-06 2024-09-21T15:50:26.470 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.470 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.470 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:26.470 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048820+0000", 2024-09-21T15:50:26.470 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2800000000000002e-06 2024-09-21T15:50:26.470 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.471 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.471 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:26.471 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105647+0000", 2024-09-21T15:50:26.471 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056827253000000001 2024-09-21T15:50:26.471 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.471 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.471 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:26.471 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105716+0000", 2024-09-21T15:50:26.471 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8761000000000001e-05 2024-09-21T15:50:26.471 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.471 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.471 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:26.471 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105774+0000", 2024-09-21T15:50:26.471 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7855000000000001e-05 2024-09-21T15:50:26.471 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.472 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:26.472 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.472 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.472 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.472 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116237 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 774144~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:26.472 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048822+0000", 2024-09-21T15:50:26.472 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 30.405481223999999, 2024-09-21T15:50:26.472 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057114028999999997, 2024-09-21T15:50:26.472 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:26.472 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:26.472 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:26.472 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:26.472 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:26.472 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116237 2024-09-21T15:50:26.473 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.473 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:26.473 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.473 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:26.473 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048822+0000", 2024-09-21T15:50:26.473 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.473 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.473 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.473 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:26.473 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048822+0000", 2024-09-21T15:50:26.473 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.474 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.474 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.474 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:26.474 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048822+0000", 2024-09-21T15:50:26.474 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:26.474 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.474 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.474 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:26.474 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048823+0000", 2024-09-21T15:50:26.474 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9699999999999994e-07 2024-09-21T15:50:26.474 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.474 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.474 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:26.474 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048825+0000", 2024-09-21T15:50:26.475 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.65e-06 2024-09-21T15:50:26.475 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.475 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.475 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:26.475 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048828+0000", 2024-09-21T15:50:26.475 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8890000000000002e-06 2024-09-21T15:50:26.475 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.475 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.475 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:26.475 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105802+0000", 2024-09-21T15:50:26.475 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056974415 2024-09-21T15:50:26.475 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.475 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.476 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:26.476 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105871+0000", 2024-09-21T15:50:26.476 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8727999999999997e-05 2024-09-21T15:50:26.476 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.476 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.476 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:26.476 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105936+0000", 2024-09-21T15:50:26.476 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4554999999999994e-05 2024-09-21T15:50:26.476 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.476 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:26.476 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.476 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.476 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.476 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116238 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 780288~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:26.476 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048830+0000", 2024-09-21T15:50:26.477 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 30.405472965000001, 2024-09-21T15:50:26.477 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057296574000000003, 2024-09-21T15:50:26.477 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:26.477 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:26.477 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:26.477 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:26.477 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:26.477 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116238 2024-09-21T15:50:26.477 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.477 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:26.477 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.477 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:26.477 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048830+0000", 2024-09-21T15:50:26.478 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.478 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.478 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.478 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:26.478 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048830+0000", 2024-09-21T15:50:26.478 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.478 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.478 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.478 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:26.478 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048830+0000", 2024-09-21T15:50:26.478 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:26.478 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.478 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.478 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:26.478 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048831+0000", 2024-09-21T15:50:26.479 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0160000000000001e-06 2024-09-21T15:50:26.479 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.479 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.479 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:26.479 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048834+0000", 2024-09-21T15:50:26.479 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.813e-06 2024-09-21T15:50:26.479 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.479 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.479 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:26.479 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048836+0000", 2024-09-21T15:50:26.479 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6790000000000001e-06 2024-09-21T15:50:26.479 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.479 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.480 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:26.480 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105961+0000", 2024-09-21T15:50:26.480 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057124705999999997 2024-09-21T15:50:26.480 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.480 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.480 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:26.480 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106042+0000", 2024-09-21T15:50:26.480 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0820000000000002e-05 2024-09-21T15:50:26.480 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.480 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.480 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:26.480 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106127+0000", 2024-09-21T15:50:26.480 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4746000000000005e-05 2024-09-21T15:50:26.480 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.481 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:26.481 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.481 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.481 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.481 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116324 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 524288~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:26.481 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048838+0000", 2024-09-21T15:50:26.481 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 30.405464655999999, 2024-09-21T15:50:26.481 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057477185, 2024-09-21T15:50:26.481 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:26.481 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:26.481 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:26.481 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:26.481 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:26.481 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116324 2024-09-21T15:50:26.482 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.482 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:26.482 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.482 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:26.482 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048838+0000", 2024-09-21T15:50:26.482 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.482 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.482 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.482 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:26.482 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048838+0000", 2024-09-21T15:50:26.482 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.482 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.482 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.482 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:26.483 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048838+0000", 2024-09-21T15:50:26.483 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:26.483 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.483 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.483 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:26.483 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048842+0000", 2024-09-21T15:50:26.483 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4829999999999999e-06 2024-09-21T15:50:26.483 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.483 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.483 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:26.483 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048846+0000", 2024-09-21T15:50:26.483 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0470000000000004e-06 2024-09-21T15:50:26.484 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.484 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.484 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:26.484 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048849+0000", 2024-09-21T15:50:26.484 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3689999999999998e-06 2024-09-21T15:50:26.484 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.484 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.484 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:26.484 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106157+0000", 2024-09-21T15:50:26.484 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057307942000000001 2024-09-21T15:50:26.484 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.484 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.484 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:26.484 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106237+0000", 2024-09-21T15:50:26.485 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0320000000000003e-05 2024-09-21T15:50:26.485 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.485 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.485 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:26.485 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106315+0000", 2024-09-21T15:50:26.485 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8227999999999997e-05 2024-09-21T15:50:26.485 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.485 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:26.485 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.485 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.485 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.485 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116325 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 530432~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:26.485 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048851+0000", 2024-09-21T15:50:26.485 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 30.405451484, 2024-09-21T15:50:26.485 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057581036000000002, 2024-09-21T15:50:26.486 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:26.486 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:26.486 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:26.486 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:26.486 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:26.486 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116325 2024-09-21T15:50:26.486 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.486 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:26.486 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.486 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:26.486 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048851+0000", 2024-09-21T15:50:26.486 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.486 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.486 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.487 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:26.487 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048851+0000", 2024-09-21T15:50:26.487 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.487 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.487 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:26.487 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048851+0000", 2024-09-21T15:50:26.487 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:26.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.487 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.487 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:26.487 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048852+0000", 2024-09-21T15:50:26.487 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.077e-06 2024-09-21T15:50:26.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.488 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.488 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:26.488 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048855+0000", 2024-09-21T15:50:26.488 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9859999999999999e-06 2024-09-21T15:50:26.488 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.488 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.488 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:26.488 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048858+0000", 2024-09-21T15:50:26.488 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3810000000000002e-06 2024-09-21T15:50:26.488 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.488 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.488 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:26.488 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106335+0000", 2024-09-21T15:50:26.488 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057477473000000001 2024-09-21T15:50:26.489 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.489 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.489 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:26.489 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106393+0000", 2024-09-21T15:50:26.489 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8050000000000002e-05 2024-09-21T15:50:26.489 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.489 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.489 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:26.489 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106432+0000", 2024-09-21T15:50:26.489 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9274999999999999e-05 2024-09-21T15:50:26.489 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.489 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:26.489 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.489 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.489 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.489 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116326 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 536576~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:26.490 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048860+0000", 2024-09-21T15:50:26.490 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 30.405443432999999, 2024-09-21T15:50:26.490 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057774197999999999, 2024-09-21T15:50:26.490 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:26.490 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:26.490 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:26.490 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:26.490 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:26.490 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116326 2024-09-21T15:50:26.490 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.490 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:26.490 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.490 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:26.490 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048860+0000", 2024-09-21T15:50:26.491 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.491 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:26.491 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048860+0000", 2024-09-21T15:50:26.491 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.491 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:26.491 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048859+0000", 2024-09-21T15:50:26.491 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:26.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.491 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:26.491 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048860+0000", 2024-09-21T15:50:26.492 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.02e-06 2024-09-21T15:50:26.492 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.492 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.492 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:26.492 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048863+0000", 2024-09-21T15:50:26.492 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1039999999999998e-06 2024-09-21T15:50:26.492 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.492 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.492 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:26.492 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048867+0000", 2024-09-21T15:50:26.492 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.563e-06 2024-09-21T15:50:26.492 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.492 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.492 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:26.493 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106465+0000", 2024-09-21T15:50:26.493 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057597789000000003 2024-09-21T15:50:26.493 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.493 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.493 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:26.493 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106552+0000", 2024-09-21T15:50:26.493 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7602000000000006e-05 2024-09-21T15:50:26.493 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.493 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.493 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:26.493 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106634+0000", 2024-09-21T15:50:26.493 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1321000000000003e-05 2024-09-21T15:50:26.493 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.493 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:26.493 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.493 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.494 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.494 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116327 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 542720~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:26.494 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048870+0000", 2024-09-21T15:50:26.494 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 30.405432561000001, 2024-09-21T15:50:26.494 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057883148000000002, 2024-09-21T15:50:26.494 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:26.494 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:26.494 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:26.494 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:26.494 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:26.494 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116327 2024-09-21T15:50:26.494 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.494 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:26.494 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.495 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:26.495 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048870+0000", 2024-09-21T15:50:26.495 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.495 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.495 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.495 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:26.495 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048870+0000", 2024-09-21T15:50:26.495 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.495 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.495 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.495 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:26.495 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048870+0000", 2024-09-21T15:50:26.495 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:26.495 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.495 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.495 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:26.496 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048872+0000", 2024-09-21T15:50:26.496 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.824e-06 2024-09-21T15:50:26.496 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.496 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.496 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:26.496 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048879+0000", 2024-09-21T15:50:26.496 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6749999999999996e-06 2024-09-21T15:50:26.496 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.496 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.496 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:26.496 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048883+0000", 2024-09-21T15:50:26.496 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2200000000000003e-06 2024-09-21T15:50:26.496 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.496 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.497 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:26.497 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106663+0000", 2024-09-21T15:50:26.497 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057780049 2024-09-21T15:50:26.497 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.497 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.497 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:26.497 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106711+0000", 2024-09-21T15:50:26.497 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8433999999999997e-05 2024-09-21T15:50:26.497 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.497 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.497 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:26.497 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106754+0000", 2024-09-21T15:50:26.497 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2236e-05 2024-09-21T15:50:26.498 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.498 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:26.498 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.498 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.498 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.498 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116328 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 548864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:26.498 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048887+0000", 2024-09-21T15:50:26.498 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 30.405416040999999, 2024-09-21T15:50:26.498 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057981984, 2024-09-21T15:50:26.498 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:26.498 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:26.498 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:26.498 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:26.499 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:26.499 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116328 2024-09-21T15:50:26.499 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.499 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:26.499 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.499 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:26.499 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048887+0000", 2024-09-21T15:50:26.499 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.499 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.499 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.499 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:26.499 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048887+0000", 2024-09-21T15:50:26.499 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.499 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.499 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.500 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:26.500 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048887+0000", 2024-09-21T15:50:26.500 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:26.500 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.500 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.500 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:26.500 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048891+0000", 2024-09-21T15:50:26.500 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.843e-06 2024-09-21T15:50:26.500 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.500 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.500 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:26.500 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048897+0000", 2024-09-21T15:50:26.500 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5419999999999996e-06 2024-09-21T15:50:26.500 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.500 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.500 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:26.501 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048901+0000", 2024-09-21T15:50:26.501 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4689999999999999e-06 2024-09-21T15:50:26.501 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.501 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.501 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:26.501 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106780+0000", 2024-09-21T15:50:26.501 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057878259000000001 2024-09-21T15:50:26.501 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.501 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.501 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:26.501 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106827+0000", 2024-09-21T15:50:26.501 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7098000000000002e-05 2024-09-21T15:50:26.501 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.501 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.502 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:26.502 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106869+0000", 2024-09-21T15:50:26.502 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2129999999999998e-05 2024-09-21T15:50:26.502 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.502 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:26.502 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.502 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.502 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.502 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116329 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 555008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:26.502 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048905+0000", 2024-09-21T15:50:26.502 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 30.405397931, 2024-09-21T15:50:26.502 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058115554999999999, 2024-09-21T15:50:26.502 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:26.502 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:26.503 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:26.503 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:26.503 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:26.503 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116329 2024-09-21T15:50:26.503 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.503 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:26.503 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.503 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:26.503 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048905+0000", 2024-09-21T15:50:26.503 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.503 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.503 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.503 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:26.503 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048905+0000", 2024-09-21T15:50:26.503 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.504 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.504 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.504 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:26.504 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048905+0000", 2024-09-21T15:50:26.504 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:26.504 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.504 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.504 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:26.504 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048907+0000", 2024-09-21T15:50:26.504 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.086e-06 2024-09-21T15:50:26.504 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.504 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.504 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:26.504 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048912+0000", 2024-09-21T15:50:26.504 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2830000000000001e-06 2024-09-21T15:50:26.505 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.505 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.505 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:26.505 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048916+0000", 2024-09-21T15:50:26.505 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.98e-06 2024-09-21T15:50:26.505 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.505 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.505 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:26.505 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106907+0000", 2024-09-21T15:50:26.505 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057990719000000003 2024-09-21T15:50:26.505 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.505 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.505 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:26.505 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106969+0000", 2024-09-21T15:50:26.505 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1789000000000001e-05 2024-09-21T15:50:26.506 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.506 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.506 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:26.506 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107021+0000", 2024-09-21T15:50:26.506 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2030999999999997e-05 2024-09-21T15:50:26.506 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.506 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:26.506 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.506 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.506 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.506 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116330 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 561152~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:26.506 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048922+0000", 2024-09-21T15:50:26.506 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 30.405380773000001, 2024-09-21T15:50:26.506 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058242343000000002, 2024-09-21T15:50:26.507 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:26.507 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:26.507 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:26.507 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:26.507 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:26.507 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116330 2024-09-21T15:50:26.507 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.507 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:26.507 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.507 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:26.507 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048922+0000", 2024-09-21T15:50:26.507 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.507 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.507 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.508 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:26.508 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048922+0000", 2024-09-21T15:50:26.508 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.508 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.508 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.508 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:26.508 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048922+0000", 2024-09-21T15:50:26.508 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:26.508 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.508 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.508 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:26.508 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048924+0000", 2024-09-21T15:50:26.508 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1660000000000001e-06 2024-09-21T15:50:26.508 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.509 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.509 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:26.509 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048930+0000", 2024-09-21T15:50:26.509 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4529999999999999e-06 2024-09-21T15:50:26.509 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.509 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.509 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:26.509 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048934+0000", 2024-09-21T15:50:26.509 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0269999999999997e-06 2024-09-21T15:50:26.509 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.509 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.509 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:26.509 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107047+0000", 2024-09-21T15:50:26.509 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058112913000000002 2024-09-21T15:50:26.509 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.510 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.510 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:26.510 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107095+0000", 2024-09-21T15:50:26.510 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7778000000000002e-05 2024-09-21T15:50:26.510 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.510 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.510 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:26.510 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107165+0000", 2024-09-21T15:50:26.510 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9370999999999997e-05 2024-09-21T15:50:26.510 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.510 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:26.510 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.510 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.510 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.510 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116331 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 567296~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:26.511 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.049440+0000", 2024-09-21T15:50:26.511 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 30.404862679000001, 2024-09-21T15:50:26.511 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057878485, 2024-09-21T15:50:26.511 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:26.511 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:26.511 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:26.511 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:26.511 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:26.511 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116331 2024-09-21T15:50:26.511 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.511 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:26.511 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.511 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:26.512 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049440+0000", 2024-09-21T15:50:26.512 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.512 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.512 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.512 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:26.512 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049440+0000", 2024-09-21T15:50:26.512 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.512 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.512 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.512 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:26.512 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048939+0000", 2024-09-21T15:50:26.512 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9994984 2024-09-21T15:50:26.512 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.512 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.512 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:26.513 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049444+0000", 2024-09-21T15:50:26.513 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00050557200000000003 2024-09-21T15:50:26.513 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.513 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.513 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:26.513 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049460+0000", 2024-09-21T15:50:26.513 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6120000000000002e-05 2024-09-21T15:50:26.513 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.513 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.513 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:26.513 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049468+0000", 2024-09-21T15:50:26.513 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4209999999999998e-06 2024-09-21T15:50:26.513 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.513 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.514 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:26.514 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107189+0000", 2024-09-21T15:50:26.514 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057721019999999998 2024-09-21T15:50:26.514 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.514 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.514 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:26.514 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107254+0000", 2024-09-21T15:50:26.514 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5265000000000006e-05 2024-09-21T15:50:26.514 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.514 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.514 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:26.514 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107319+0000", 2024-09-21T15:50:26.514 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4784999999999997e-05 2024-09-21T15:50:26.514 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.515 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:26.515 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.515 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.515 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.515 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116332 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 573440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:26.515 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.049474+0000", 2024-09-21T15:50:26.515 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 30.404829417999998, 2024-09-21T15:50:26.515 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058013809, 2024-09-21T15:50:26.515 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:26.515 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:26.515 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:26.515 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:26.515 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:26.515 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116332 2024-09-21T15:50:26.516 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.516 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:26.516 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.516 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:26.516 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049474+0000", 2024-09-21T15:50:26.516 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.516 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.516 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.516 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:26.516 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049474+0000", 2024-09-21T15:50:26.516 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.516 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.516 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.516 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:26.517 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049473+0000", 2024-09-21T15:50:26.517 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:26.517 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.517 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.517 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:26.517 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049476+0000", 2024-09-21T15:50:26.517 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6249999999999999e-06 2024-09-21T15:50:26.517 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.517 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.517 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:26.517 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049482+0000", 2024-09-21T15:50:26.517 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0959999999999997e-06 2024-09-21T15:50:26.517 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.517 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.518 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:26.518 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049487+0000", 2024-09-21T15:50:26.518 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5949999999999998e-06 2024-09-21T15:50:26.518 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.518 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.518 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:26.518 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107344+0000", 2024-09-21T15:50:26.518 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057856594999999997 2024-09-21T15:50:26.518 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.518 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.518 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:26.518 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107422+0000", 2024-09-21T15:50:26.518 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7582000000000004e-05 2024-09-21T15:50:26.518 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.519 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.519 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:26.519 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107487+0000", 2024-09-21T15:50:26.519 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5723999999999996e-05 2024-09-21T15:50:26.519 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.519 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:26.519 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.519 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.519 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.519 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116333 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 579584~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:26.519 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.049993+0000", 2024-09-21T15:50:26.519 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 30.404309635000001, 2024-09-21T15:50:26.519 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057663537000000001, 2024-09-21T15:50:26.519 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:26.520 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:26.520 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:26.520 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:26.520 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:26.520 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116333 2024-09-21T15:50:26.520 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.520 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:26.520 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.520 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:26.520 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049993+0000", 2024-09-21T15:50:26.520 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.520 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.520 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.520 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:26.521 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049993+0000", 2024-09-21T15:50:26.521 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.521 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.521 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.521 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:26.521 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049491+0000", 2024-09-21T15:50:26.521 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9994979 2024-09-21T15:50:26.521 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.521 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.521 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:26.521 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049998+0000", 2024-09-21T15:50:26.521 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00050654200000000002 2024-09-21T15:50:26.521 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.521 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.521 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:26.522 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050010+0000", 2024-09-21T15:50:26.522 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1790999999999999e-05 2024-09-21T15:50:26.522 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.522 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.522 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:26.522 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050016+0000", 2024-09-21T15:50:26.522 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2179999999999996e-06 2024-09-21T15:50:26.522 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.522 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.522 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:26.522 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107513+0000", 2024-09-21T15:50:26.522 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057496927000000003 2024-09-21T15:50:26.522 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.522 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.523 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:26.523 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107590+0000", 2024-09-21T15:50:26.523 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7547999999999997e-05 2024-09-21T15:50:26.523 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.523 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.523 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:26.523 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107657+0000", 2024-09-21T15:50:26.523 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6630000000000004e-05 2024-09-21T15:50:26.523 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.523 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:26.523 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.523 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.523 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.523 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116334 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 585728~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:26.523 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.050091+0000", 2024-09-21T15:50:26.524 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 30.404212007000002, 2024-09-21T15:50:26.524 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057734355000000001, 2024-09-21T15:50:26.524 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:26.524 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:26.524 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:26.524 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:26.524 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:26.524 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116334 2024-09-21T15:50:26.524 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.524 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:26.524 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.524 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:26.524 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050091+0000", 2024-09-21T15:50:26.524 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.525 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.525 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.525 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:26.525 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050091+0000", 2024-09-21T15:50:26.525 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.525 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.525 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.525 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:26.525 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050021+0000", 2024-09-21T15:50:26.525 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999304 2024-09-21T15:50:26.525 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.525 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.525 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:26.525 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050095+0000", 2024-09-21T15:50:26.526 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4158000000000004e-05 2024-09-21T15:50:26.526 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.526 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.526 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:26.526 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050103+0000", 2024-09-21T15:50:26.526 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0770000000000002e-06 2024-09-21T15:50:26.526 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.526 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.526 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:26.526 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050108+0000", 2024-09-21T15:50:26.526 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6330000000000001e-06 2024-09-21T15:50:26.526 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.526 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.526 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:26.527 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107681+0000", 2024-09-21T15:50:26.527 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057572996000000001 2024-09-21T15:50:26.527 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.527 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.527 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:26.527 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107759+0000", 2024-09-21T15:50:26.527 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7368999999999997e-05 2024-09-21T15:50:26.527 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.527 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.527 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:26.527 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107825+0000", 2024-09-21T15:50:26.527 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6783000000000001e-05 2024-09-21T15:50:26.527 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.527 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:26.528 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.528 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.528 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.528 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116335 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 591872~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:26.528 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.050542+0000", 2024-09-21T15:50:26.528 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 30.403761421999999, 2024-09-21T15:50:26.528 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057466455999999999, 2024-09-21T15:50:26.528 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:26.528 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:26.528 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:26.528 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:26.528 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:26.528 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116335 2024-09-21T15:50:26.528 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.529 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:26.529 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.529 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:26.529 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050542+0000", 2024-09-21T15:50:26.529 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.529 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.529 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.529 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:26.529 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050542+0000", 2024-09-21T15:50:26.529 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.529 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.529 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.529 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:26.529 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050113+0000", 2024-09-21T15:50:26.529 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9995713 2024-09-21T15:50:26.530 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.530 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.530 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:26.530 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050546+0000", 2024-09-21T15:50:26.530 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00043328699999999997 2024-09-21T15:50:26.530 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.530 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.530 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:26.530 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050557+0000", 2024-09-21T15:50:26.530 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1011e-05 2024-09-21T15:50:26.530 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.530 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.530 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:26.530 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050564+0000", 2024-09-21T15:50:26.531 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1130000000000003e-06 2024-09-21T15:50:26.531 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.531 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.531 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:26.531 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107852+0000", 2024-09-21T15:50:26.531 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057287350000000001 2024-09-21T15:50:26.531 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.531 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.531 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:26.531 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107928+0000", 2024-09-21T15:50:26.531 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6668000000000005e-05 2024-09-21T15:50:26.531 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.531 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.531 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:26.531 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108008+0000", 2024-09-21T15:50:26.532 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9783999999999994e-05 2024-09-21T15:50:26.532 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.532 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:26.532 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.532 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.532 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.532 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116336 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 598016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:26.532 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.050834+0000", 2024-09-21T15:50:26.532 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 30.403469156, 2024-09-21T15:50:26.532 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057353572999999998, 2024-09-21T15:50:26.532 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:26.532 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:26.532 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:26.532 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:26.532 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:26.532 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116336 2024-09-21T15:50:26.533 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.533 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:26.533 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.533 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:26.533 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050834+0000", 2024-09-21T15:50:26.533 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.533 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.533 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.533 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:26.533 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050834+0000", 2024-09-21T15:50:26.533 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.533 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.533 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.533 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:26.533 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050571+0000", 2024-09-21T15:50:26.533 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9997373 2024-09-21T15:50:26.534 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.534 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.534 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:26.534 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050842+0000", 2024-09-21T15:50:26.534 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00027029099999999998 2024-09-21T15:50:26.534 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.534 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.534 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:26.534 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050853+0000", 2024-09-21T15:50:26.534 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1532000000000001e-05 2024-09-21T15:50:26.534 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.534 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.534 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:26.534 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050863+0000", 2024-09-21T15:50:26.535 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4239999999999999e-06 2024-09-21T15:50:26.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.535 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.535 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:26.535 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108045+0000", 2024-09-21T15:50:26.535 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057182415 2024-09-21T15:50:26.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.535 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.535 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:26.535 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108123+0000", 2024-09-21T15:50:26.535 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8275000000000005e-05 2024-09-21T15:50:26.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.535 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.535 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:26.535 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108187+0000", 2024-09-21T15:50:26.535 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4152000000000005e-05 2024-09-21T15:50:26.536 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.536 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:26.536 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.536 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.536 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.536 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116337 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 604160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:26.536 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.051192+0000", 2024-09-21T15:50:26.536 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 30.403110527999999, 2024-09-21T15:50:26.536 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057175981000000001, 2024-09-21T15:50:26.536 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:26.536 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:26.536 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:26.536 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:26.536 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:26.536 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116337 2024-09-21T15:50:26.536 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.537 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:26.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.537 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:26.537 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051192+0000", 2024-09-21T15:50:26.537 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.537 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:26.537 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051192+0000", 2024-09-21T15:50:26.537 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.537 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:26.537 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050870+0000", 2024-09-21T15:50:26.537 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9996777 2024-09-21T15:50:26.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.538 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:26.538 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051197+0000", 2024-09-21T15:50:26.538 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00032656299999999998 2024-09-21T15:50:26.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.538 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.538 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:26.538 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051209+0000", 2024-09-21T15:50:26.538 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2636000000000001e-05 2024-09-21T15:50:26.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.538 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.538 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:26.538 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051217+0000", 2024-09-21T15:50:26.538 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5989999999999996e-06 2024-09-21T15:50:26.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.538 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.539 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:26.539 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108224+0000", 2024-09-21T15:50:26.539 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057007622000000001 2024-09-21T15:50:26.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.539 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:26.539 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108299+0000", 2024-09-21T15:50:26.539 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5099999999999996e-05 2024-09-21T15:50:26.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.539 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:26.539 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108368+0000", 2024-09-21T15:50:26.539 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8918000000000007e-05 2024-09-21T15:50:26.539 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.539 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:26.540 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.540 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.540 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.540 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116338 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 610304~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:26.540 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.051583+0000", 2024-09-21T15:50:26.540 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 30.402720075000001, 2024-09-21T15:50:26.540 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056967716000000002, 2024-09-21T15:50:26.540 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:26.540 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:26.540 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:26.540 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:26.540 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:26.540 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116338 2024-09-21T15:50:26.540 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.541 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:26.541 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.541 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:26.541 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051583+0000", 2024-09-21T15:50:26.541 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.541 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.541 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:26.541 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051583+0000", 2024-09-21T15:50:26.541 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:26.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.541 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.541 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:26.542 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051223+0000", 2024-09-21T15:50:26.542 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.99964 2024-09-21T15:50:26.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.542 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:26.542 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051587+0000", 2024-09-21T15:50:26.542 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00036352999999999998 2024-09-21T15:50:26.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.542 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:26.542 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051597+0000", 2024-09-21T15:50:26.542 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.893e-06 2024-09-21T15:50:26.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.542 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:26.543 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051603+0000", 2024-09-21T15:50:26.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0519999999999998e-06 2024-09-21T15:50:26.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.543 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.543 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:26.543 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108394+0000", 2024-09-21T15:50:26.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056791531999999999 2024-09-21T15:50:26.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.543 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.543 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:26.543 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108480+0000", 2024-09-21T15:50:26.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.577e-05 2024-09-21T15:50:26.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:26.544 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:26.544 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:26.544 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108551+0000", 2024-09-21T15:50:26.544 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0722000000000005e-05 2024-09-21T15:50:26.544 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.544 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:26.544 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.544 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:26.544 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:26.544 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:26.545 INFO:teuthology.orchestra.run.smithi089.stdout:5872 read 0x12d2221 thru 0x12dc39b (0xa17b bytes) 2024-09-21T15:50:26.545 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:26.545 INFO:teuthology.orchestra.run.smithi089.stdout:5874 read 0x78cd46 thru 0x795029 (0x82e4 bytes) 2024-09-21T15:50:26.545 INFO:teuthology.orchestra.run.smithi089.stdout:5875 write 0x34b3170 thru 0x34bc699 (0x952a bytes) 2024-09-21T15:50:26.546 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:50:26.548 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:26.548 INFO:teuthology.orchestra.run.smithi089.stdout:5877 read 0x57bcf93 thru 0x57c98be (0xc92c bytes) 2024-09-21T15:50:26.548 INFO:teuthology.orchestra.run.smithi089.stdout:5878 write 0x7de8198 thru 0x7df062d (0x8496 bytes) 2024-09-21T15:50:26.549 INFO:teuthology.orchestra.run.smithi089.stdout:5879 writesame 0x7d6bd60 thru 0x7d724ef data_size 0x8(0x6790 bytes) 2024-09-21T15:50:26.549 INFO:teuthology.orchestra.run.smithi089.stdout:5880 read 0xe65586 thru 0xe6a74b (0x51c6 bytes) 2024-09-21T15:50:26.549 INFO:teuthology.orchestra.run.smithi089.stdout:5881 read 0x16c89f8 thru 0x16cf1ce (0x67d7 bytes) 2024-09-21T15:50:26.549 INFO:teuthology.orchestra.run.smithi089.stdout:5882 read 0x19a09 thru 0x23592 (0x9b8a bytes) 2024-09-21T15:50:26.549 INFO:teuthology.orchestra.run.smithi089.stdout:5883 punch from 0x1a6c1ce to 0x1a74e2f, (0x8c61 bytes) 2024-09-21T15:50:26.549 INFO:teuthology.orchestra.run.smithi089.stdout:5885 read 0x28e7bfe thru 0x28e8b62 (0xf65 bytes) 2024-09-21T15:50:26.549 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:26.549 INFO:teuthology.orchestra.run.smithi089.stdout:5888 read 0x27d849 thru 0x288332 (0xaaea bytes) 2024-09-21T15:50:26.549 INFO:teuthology.orchestra.run.smithi089.stdout:5889 read 0x3e3deea thru 0x3e466e1 (0x87f8 bytes) 2024-09-21T15:50:26.549 INFO:teuthology.orchestra.run.smithi089.stdout:5891 write 0x55a3f8a thru 0x55a4db5 (0xe2c bytes) 2024-09-21T15:50:26.549 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:26.549 INFO:teuthology.orchestra.run.smithi089.stdout:5893 punch from 0x3e0f65e to 0x3e172be, (0x7c60 bytes) 2024-09-21T15:50:26.549 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:26.549 INFO:teuthology.orchestra.run.smithi089.stdout:5898 trunc from 0x7df062e to 0xa1368fd 2024-09-21T15:50:26.549 INFO:teuthology.orchestra.run.smithi089.stdout:5899 punch from 0x62250b5 to 0x6225e63, (0xdae bytes) 2024-09-21T15:50:26.550 INFO:teuthology.orchestra.run.smithi089.stdout:5900 read 0x26d608d thru 0x26df203 (0x9177 bytes) 2024-09-21T15:50:26.550 INFO:teuthology.orchestra.run.smithi089.stdout:5901 write 0xead96f8 thru 0xeae6efa (0xd803 bytes) 2024-09-21T15:50:26.550 INFO:teuthology.orchestra.run.smithi089.stdout:5902 punch from 0x65a83e5 to 0x65ae72b, (0x6346 bytes) 2024-09-21T15:50:26.550 INFO:teuthology.orchestra.run.smithi089.stdout:5903 trunc from 0xeae6efb to 0x4e2afc2 2024-09-21T15:50:26.550 INFO:teuthology.orchestra.run.smithi089.stdout:5904 read 0xf86845 thru 0xf88325 (0x1ae1 bytes) 2024-09-21T15:50:26.550 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:26.544+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:26.551 INFO:teuthology.orchestra.run.smithi089.stdout:5905 trunc from 0x4e2afc2 to 0xbde600d 2024-09-21T15:50:26.555 INFO:teuthology.orchestra.run.smithi089.stdout:5906 punch from 0x4cfa149 to 0x4d02e3b, (0x8cf2 bytes) 2024-09-21T15:50:26.555 INFO:teuthology.orchestra.run.smithi089.stdout:5907 writesame 0x5ecee45 thru 0x5ed147b data_size 0x3(0x2637 bytes) 2024-09-21T15:50:26.559 INFO:teuthology.orchestra.run.smithi089.stdout:5908 write 0x56bd83 thru 0x57b393 (0xf611 bytes) 2024-09-21T15:50:26.574 INFO:teuthology.orchestra.run.smithi089.stdout:5911 writesame 0xd29e162 thru 0xd2a49ea data_size 0x1(0x6889 bytes) 2024-09-21T15:50:26.574 INFO:teuthology.orchestra.run.smithi089.stdout:5912 read 0x7408e2e thru 0x74095bf (0x792 bytes) 2024-09-21T15:50:26.574 INFO:teuthology.orchestra.run.smithi089.stdout:5915 read 0x4c44e4d thru 0x4c506c8 (0xb87c bytes) 2024-09-21T15:50:26.574 INFO:teuthology.orchestra.run.smithi089.stdout:5919 punch from 0xb0bacec to 0xb0c557f, (0xa893 bytes) 2024-09-21T15:50:26.574 INFO:teuthology.orchestra.run.smithi089.stdout:5920 punch from 0x336375f to 0x3364b7b, (0x141c bytes) 2024-09-21T15:50:26.574 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:26.574 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:26.585 INFO:teuthology.orchestra.run.smithi089.stdout:5924 write 0xdd29c80 thru 0xdd2e279 (0x45fa bytes) 2024-09-21T15:50:26.591 INFO:teuthology.orchestra.run.smithi089.stdout:5925 read 0xba39714 thru 0xba43d74 (0xa661 bytes) 2024-09-21T15:50:26.591 INFO:teuthology.orchestra.run.smithi089.stdout:5929 write 0xa71185a thru 0xa7207c7 (0xef6e bytes) 2024-09-21T15:50:26.598 INFO:teuthology.orchestra.run.smithi089.stdout:5930 punch from 0xa17b5d8 to 0xa185827, (0xa24f bytes) 2024-09-21T15:50:26.598 INFO:teuthology.orchestra.run.smithi089.stdout:5932 read 0x37e9a0b thru 0x37f2f79 (0x956f bytes) 2024-09-21T15:50:26.598 INFO:teuthology.orchestra.run.smithi089.stdout:5933 write 0x9f2004a thru 0x9f2c98f (0xc946 bytes) 2024-09-21T15:50:26.603 INFO:teuthology.orchestra.run.smithi089.stdout:5935 write 0xd36a967 thru 0xd378705 (0xdd9f bytes) 2024-09-21T15:50:26.610 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:26.610 INFO:teuthology.orchestra.run.smithi089.stdout:5937 write 0xd21bf2d thru 0xd2298a7 (0xd97b bytes) 2024-09-21T15:50:26.633 INFO:teuthology.orchestra.run.smithi089.stdout:5938 trunc from 0xdd2e27a to 0x29f73f 2024-09-21T15:50:26.646 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:26.645+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:26.660 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:26.660 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:26.660 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:50:26.660 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:26.667 INFO:teuthology.orchestra.run.smithi089.stdout:5940 trunc from 0x29f73f to 0x4a662ac 2024-09-21T15:50:26.669 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:50:26.673 INFO:teuthology.orchestra.run.smithi089.stdout:5941 read 0x38b100e thru 0x38c0ff3 (0xffe6 bytes) 2024-09-21T15:50:26.674 INFO:teuthology.orchestra.run.smithi089.stdout:5942 punch from 0x498585f to 0x4989c67, (0x4408 bytes) 2024-09-21T15:50:26.674 INFO:teuthology.orchestra.run.smithi089.stdout:5944 write 0x92b1b6 thru 0x9325ec (0x7437 bytes) 2024-09-21T15:50:26.688 INFO:teuthology.orchestra.run.smithi089.stdout:5946 writesame 0x8d54e09 thru 0x8d5dbed data_size 0x5(0x8de5 bytes) 2024-09-21T15:50:26.688 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:26.688 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:26.688 INFO:teuthology.orchestra.run.smithi089.stdout:5949 punch from 0x5ad9e3b to 0x5ae911e, (0xf2e3 bytes) 2024-09-21T15:50:26.688 INFO:teuthology.orchestra.run.smithi089.stdout:5951 read 0x43ee51e thru 0x43f88f2 (0xa3d5 bytes) 2024-09-21T15:50:26.688 INFO:teuthology.orchestra.run.smithi089.stdout:5952 punch from 0x5af82fb to 0x5b04bf2, (0xc8f7 bytes) 2024-09-21T15:50:26.696 INFO:teuthology.orchestra.run.smithi089.stdout:5953 writesame 0xbfa4c6d thru 0xbfa7d4d data_size 0x3(0x30e1 bytes) 2024-09-21T15:50:26.704 INFO:teuthology.orchestra.run.smithi089.stdout:5954 trunc from 0xbfa7d4e to 0x5419ab7 2024-09-21T15:50:26.716 INFO:teuthology.orchestra.run.smithi089.stdout:5956 trunc from 0x5419ab7 to 0x42ed6d9 2024-09-21T15:50:26.726 INFO:teuthology.orchestra.run.smithi089.stdout:5958 read 0x399d010 thru 0x39a61fd (0x91ee bytes) 2024-09-21T15:50:26.726 INFO:teuthology.orchestra.run.smithi089.stdout:5960 write 0x1736a3d thru 0x1745c3d (0xf201 bytes) 2024-09-21T15:50:26.740 INFO:teuthology.orchestra.run.smithi089.stdout:5961 write 0x5b707f2 thru 0x5b749df (0x41ee bytes) 2024-09-21T15:50:26.744 INFO:teuthology.orchestra.run.smithi089.stdout:5962 read 0x52eafd7 thru 0x52f954d (0xe577 bytes) 2024-09-21T15:50:26.744 INFO:teuthology.orchestra.run.smithi089.stdout:5963 read 0x2900f84 thru 0x290e42c (0xd4a9 bytes) 2024-09-21T15:50:26.744 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:26.747 INFO:teuthology.orchestra.run.smithi089.stdout:5966 trunc from 0x5b749e0 to 0x898b3ed 2024-09-21T15:50:26.753 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:26.752+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:26.754 INFO:teuthology.orchestra.run.smithi089.stdout:5967 read 0x1d2422f thru 0x1d28e8e (0x4c60 bytes) 2024-09-21T15:50:26.754 INFO:teuthology.orchestra.run.smithi089.stdout:5969 read 0x3b0c882 thru 0x3b12013 (0x5792 bytes) 2024-09-21T15:50:26.754 INFO:teuthology.orchestra.run.smithi089.stdout:5970 write 0x353dbc1 thru 0x354b0fe (0xd53e bytes) 2024-09-21T15:50:26.758 INFO:teuthology.orchestra.run.smithi089.stdout:5971 read 0x876d35b thru 0x876fa4c (0x26f2 bytes) 2024-09-21T15:50:26.758 INFO:teuthology.orchestra.run.smithi089.stdout:5972 read 0x977d3d thru 0x97bf4a (0x420e bytes) 2024-09-21T15:50:26.759 INFO:teuthology.orchestra.run.smithi089.stdout:5973 read 0x28dcbdc thru 0x28e5bcf (0x8ff4 bytes) 2024-09-21T15:50:26.759 INFO:teuthology.orchestra.run.smithi089.stdout:5974 read 0x7aa8f83 thru 0x7ab2cd0 (0x9d4e bytes) 2024-09-21T15:50:26.770 INFO:teuthology.orchestra.run.smithi089.stdout:5976 write 0xbddc543 thru 0xbdeb174 (0xec32 bytes) 2024-09-21T15:50:26.774 INFO:teuthology.orchestra.run.smithi089.stdout:5977 punch from 0x66dce5b to 0x66dea65, (0x1c0a bytes) 2024-09-21T15:50:26.774 INFO:teuthology.orchestra.run.smithi089.stdout:5979 write 0x9b08171 thru 0x9b15631 (0xd4c1 bytes) 2024-09-21T15:50:26.778 INFO:teuthology.orchestra.run.smithi089.stdout:5980 punch from 0x24a8f12 to 0x24b397c, (0xaa6a bytes) 2024-09-21T15:50:26.778 INFO:teuthology.orchestra.run.smithi089.stdout:5981 read 0xa7b2653 thru 0xa7b8926 (0x62d4 bytes) 2024-09-21T15:50:26.778 INFO:teuthology.orchestra.run.smithi089.stdout:5982 read 0x7768432 thru 0x77744c8 (0xc097 bytes) 2024-09-21T15:50:26.778 INFO:teuthology.orchestra.run.smithi089.stdout:5985 read 0x72510a1 thru 0x725e512 (0xd472 bytes) 2024-09-21T15:50:26.778 INFO:teuthology.orchestra.run.smithi089.stdout:5986 read 0xbbd740d thru 0xbbdaa37 (0x362b bytes) 2024-09-21T15:50:26.778 INFO:teuthology.orchestra.run.smithi089.stdout:5987 read 0xb64b181 thru 0xb65ae7d (0xfcfd bytes) 2024-09-21T15:50:26.778 INFO:teuthology.orchestra.run.smithi089.stdout:5988 read 0xa63548c thru 0xa63611a (0xc8f bytes) 2024-09-21T15:50:26.778 INFO:teuthology.orchestra.run.smithi089.stdout:5990 write 0x15fbadc thru 0x1603808 (0x7d2d bytes) 2024-09-21T15:50:26.784 INFO:teuthology.orchestra.run.smithi089.stdout:5991 read 0x8eff8e1 thru 0x8f0ad2f (0xb44f bytes) 2024-09-21T15:50:26.784 INFO:teuthology.orchestra.run.smithi089.stdout:5992 read 0x1740e3 thru 0x17be49 (0x7d67 bytes) 2024-09-21T15:50:26.785 INFO:teuthology.orchestra.run.smithi089.stdout:compare and write disabled 2024-09-21T15:50:26.785 INFO:teuthology.orchestra.run.smithi089.stdout:5994 writesame 0x5c592a5 thru 0x5c6463f data_size 0x1(0xb39b bytes) 2024-09-21T15:50:26.816 INFO:teuthology.orchestra.run.smithi089.stdout:5996 writesame 0x974b19d thru 0x974eba1 data_size 0x1(0x3a05 bytes) 2024-09-21T15:50:26.828 INFO:teuthology.orchestra.run.smithi089.stdout:5997 read 0x48b7a75 thru 0x48bd02d (0x55b9 bytes) 2024-09-21T15:50:26.828 INFO:teuthology.orchestra.run.smithi089.stdout:5998 read 0x761caf0 thru 0x761ce41 (0x352 bytes) 2024-09-21T15:50:26.828 INFO:teuthology.orchestra.run.smithi089.stdout:6000 read 0xb80c27f thru 0xb816ae0 (0xa862 bytes) 2024-09-21T15:50:26.829 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:26.829 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:26.829 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:50:26.829 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:50:26.829 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:26.838 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:50:26.848 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:26.847+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:26.858 INFO:teuthology.orchestra.run.smithi089.stdout:checking clone #43, image image_client.0-clone43 against file /home/ubuntu/cephtest/archive/fsx-image_client.0-parent44 2024-09-21T15:50:26.949 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:26.948+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:26.989 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:26.990 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:50:26.990 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:50:26.990 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:50:26.990 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:26.990 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155031 3.1c 3.8612741c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:26.990 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448748+0000", 2024-09-21T15:50:26.990 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.539356358, 2024-09-21T15:50:26.990 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064010286999999999, 2024-09-21T15:50:26.990 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:26.990 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:26.990 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:26.990 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:26.990 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:26.990 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155031 2024-09-21T15:50:26.991 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:26.991 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:26.991 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:26.991 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:26.991 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448748+0000", 2024-09-21T15:50:26.991 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:26.991 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:26.991 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:26.991 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:26.991 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448748+0000", 2024-09-21T15:50:26.991 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:26.991 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:26.991 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:26.991 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:26.991 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448747+0000", 2024-09-21T15:50:26.992 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:26.992 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:26.992 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:26.992 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:26.992 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448753+0000", 2024-09-21T15:50:26.992 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.755e-06 2024-09-21T15:50:26.992 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:26.992 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:26.992 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:26.992 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448763+0000", 2024-09-21T15:50:26.992 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7659999999999993e-06 2024-09-21T15:50:26.992 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:26.992 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:26.992 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:26.992 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448769+0000", 2024-09-21T15:50:26.992 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6060000000000002e-06 2024-09-21T15:50:26.993 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:26.993 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:26.993 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:26.993 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512750+0000", 2024-09-21T15:50:26.993 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.063981298000000006 2024-09-21T15:50:26.993 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:26.993 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:26.993 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:26.993 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512758+0000", 2024-09-21T15:50:26.993 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1669999999999999e-06 2024-09-21T15:50:26.993 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:26.993 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:26.993 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:26.993 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:26.993 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:26.993 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155091 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:26.994 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448916+0000", 2024-09-21T15:50:26.994 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.539188513, 2024-09-21T15:50:26.994 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064006309999999997, 2024-09-21T15:50:26.994 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:26.994 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:26.994 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:26.994 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:26.994 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:26.994 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155091 2024-09-21T15:50:26.994 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:26.994 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:26.994 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:26.994 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:26.994 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448916+0000", 2024-09-21T15:50:26.995 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:26.995 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:26.995 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:26.995 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:26.995 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448916+0000", 2024-09-21T15:50:26.995 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:26.995 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:26.995 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:26.995 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:26.995 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448915+0000", 2024-09-21T15:50:26.995 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:26.995 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:26.995 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:26.995 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:26.996 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448917+0000", 2024-09-21T15:50:26.996 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2400000000000002e-06 2024-09-21T15:50:26.996 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:26.996 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:26.996 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:26.996 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448923+0000", 2024-09-21T15:50:26.996 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0580000000000003e-06 2024-09-21T15:50:26.996 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:26.996 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:26.996 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:26.996 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448927+0000", 2024-09-21T15:50:26.996 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.031e-06 2024-09-21T15:50:26.996 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:26.996 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:26.996 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:26.996 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512916+0000", 2024-09-21T15:50:26.997 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.063988456999999999 2024-09-21T15:50:26.997 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:26.997 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:26.997 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:26.997 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512922+0000", 2024-09-21T15:50:26.997 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9009999999999999e-06 2024-09-21T15:50:26.997 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:26.997 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:26.997 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:26.997 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:26.997 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:26.997 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155092 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:26.997 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448931+0000", 2024-09-21T15:50:26.997 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.5391727309999998, 2024-09-21T15:50:26.997 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064022793999999994, 2024-09-21T15:50:26.998 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:26.998 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:26.998 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:26.998 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:26.998 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:26.998 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155092 2024-09-21T15:50:26.998 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:26.998 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:26.998 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:26.998 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:26.998 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448931+0000", 2024-09-21T15:50:26.998 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:26.998 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:26.998 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:26.998 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:26.998 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448931+0000", 2024-09-21T15:50:26.999 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:26.999 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:26.999 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:26.999 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:26.999 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448931+0000", 2024-09-21T15:50:26.999 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:26.999 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:26.999 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:26.999 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:26.999 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448933+0000", 2024-09-21T15:50:26.999 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1799999999999999e-06 2024-09-21T15:50:26.999 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:26.999 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:26.999 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:26.999 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448939+0000", 2024-09-21T15:50:27.000 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.604e-06 2024-09-21T15:50:27.000 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.000 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.000 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:27.000 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448943+0000", 2024-09-21T15:50:27.000 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0740000000000003e-06 2024-09-21T15:50:27.000 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.000 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.000 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:27.000 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512946+0000", 2024-09-21T15:50:27.000 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064003612000000001 2024-09-21T15:50:27.000 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.000 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.000 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:27.000 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512954+0000", 2024-09-21T15:50:27.000 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6690000000000006e-06 2024-09-21T15:50:27.000 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.001 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:27.001 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.001 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.001 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.001 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155093 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:27.001 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448947+0000", 2024-09-21T15:50:27.001 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.5391573569999997, 2024-09-21T15:50:27.001 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064020717000000005, 2024-09-21T15:50:27.001 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:27.001 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:27.001 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:27.001 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:27.001 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:27.001 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155093 2024-09-21T15:50:27.001 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.001 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:27.002 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.002 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:27.002 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448947+0000", 2024-09-21T15:50:27.002 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.002 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.002 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.002 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:27.002 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448947+0000", 2024-09-21T15:50:27.002 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.002 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.002 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.002 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:27.002 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448946+0000", 2024-09-21T15:50:27.002 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:27.002 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.003 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.003 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:27.003 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448949+0000", 2024-09-21T15:50:27.003 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3860000000000001e-06 2024-09-21T15:50:27.003 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.003 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.003 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:27.003 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448955+0000", 2024-09-21T15:50:27.003 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.823e-06 2024-09-21T15:50:27.003 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.003 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.003 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:27.003 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448959+0000", 2024-09-21T15:50:27.003 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2799999999999997e-06 2024-09-21T15:50:27.003 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.004 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.004 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:27.004 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512964+0000", 2024-09-21T15:50:27.004 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064004741000000004 2024-09-21T15:50:27.004 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.004 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.004 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:27.004 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512967+0000", 2024-09-21T15:50:27.004 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.839e-06 2024-09-21T15:50:27.004 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.004 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:27.004 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.004 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.004 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.004 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155094 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:27.005 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448962+0000", 2024-09-21T15:50:27.005 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.5391417340000002, 2024-09-21T15:50:27.005 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064037378000000006, 2024-09-21T15:50:27.005 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:27.005 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:27.005 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:27.005 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:27.005 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:27.005 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155094 2024-09-21T15:50:27.005 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.005 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:27.005 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.005 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:27.005 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448962+0000", 2024-09-21T15:50:27.005 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.005 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.006 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.006 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:27.006 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448962+0000", 2024-09-21T15:50:27.006 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.006 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.006 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.006 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:27.006 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448962+0000", 2024-09-21T15:50:27.006 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:27.006 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.006 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.006 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:27.006 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448968+0000", 2024-09-21T15:50:27.006 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9710000000000001e-06 2024-09-21T15:50:27.006 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.006 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.007 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:27.007 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448974+0000", 2024-09-21T15:50:27.007 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7230000000000001e-06 2024-09-21T15:50:27.007 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.007 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.007 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:27.007 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448978+0000", 2024-09-21T15:50:27.007 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3749999999999996e-06 2024-09-21T15:50:27.007 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.007 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.007 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:27.007 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512992+0000", 2024-09-21T15:50:27.007 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064013525000000002 2024-09-21T15:50:27.007 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.007 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.007 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:27.008 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513000+0000", 2024-09-21T15:50:27.008 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0830000000000004e-06 2024-09-21T15:50:27.008 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.008 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:27.008 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.008 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.008 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.008 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155095 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:27.008 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448982+0000", 2024-09-21T15:50:27.008 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.539122452, 2024-09-21T15:50:27.008 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064041904999999996, 2024-09-21T15:50:27.008 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:27.008 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:27.008 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:27.008 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:27.009 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:27.009 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155095 2024-09-21T15:50:27.009 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.009 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:27.009 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.009 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:27.009 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448982+0000", 2024-09-21T15:50:27.009 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.009 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.009 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.009 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:27.009 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448982+0000", 2024-09-21T15:50:27.009 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.009 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.009 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.009 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:27.010 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448981+0000", 2024-09-21T15:50:27.010 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:27.010 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.010 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.010 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:27.010 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448983+0000", 2024-09-21T15:50:27.010 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.905e-06 2024-09-21T15:50:27.010 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.010 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.010 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:27.010 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448989+0000", 2024-09-21T15:50:27.010 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.536e-06 2024-09-21T15:50:27.010 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.010 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.010 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:27.010 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448993+0000", 2024-09-21T15:50:27.011 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2390000000000004e-06 2024-09-21T15:50:27.011 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.011 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.011 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:27.011 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513015+0000", 2024-09-21T15:50:27.011 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064022081999999994 2024-09-21T15:50:27.011 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.011 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.011 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:27.011 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513023+0000", 2024-09-21T15:50:27.011 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4500000000000004e-06 2024-09-21T15:50:27.011 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.011 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:27.011 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.011 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.012 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.012 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155096 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:27.012 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448996+0000", 2024-09-21T15:50:27.012 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.5391075509999999, 2024-09-21T15:50:27.012 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064060868000000007, 2024-09-21T15:50:27.012 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:27.012 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:27.012 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:27.012 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:27.012 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:27.012 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155096 2024-09-21T15:50:27.012 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.012 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:27.012 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.012 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:27.012 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448996+0000", 2024-09-21T15:50:27.012 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.013 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.013 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.013 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:27.013 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448996+0000", 2024-09-21T15:50:27.013 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.013 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.013 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.013 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:27.013 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448996+0000", 2024-09-21T15:50:27.013 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:27.013 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.013 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.013 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:27.013 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448998+0000", 2024-09-21T15:50:27.013 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8780000000000001e-06 2024-09-21T15:50:27.014 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.014 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.014 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:27.014 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449003+0000", 2024-09-21T15:50:27.014 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4099999999999999e-06 2024-09-21T15:50:27.014 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.014 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.014 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:27.014 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449008+0000", 2024-09-21T15:50:27.014 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.335e-06 2024-09-21T15:50:27.014 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.014 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.014 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:27.014 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513049+0000", 2024-09-21T15:50:27.014 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064041059999999997 2024-09-21T15:50:27.015 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.015 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.015 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:27.015 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513057+0000", 2024-09-21T15:50:27.015 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5050000000000007e-06 2024-09-21T15:50:27.015 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.015 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:27.015 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.015 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.015 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.015 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155097 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:27.015 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449019+0000", 2024-09-21T15:50:27.015 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.5390851300000001, 2024-09-21T15:50:27.015 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064052488000000005, 2024-09-21T15:50:27.015 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:27.015 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:27.016 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:27.016 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:27.016 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:27.016 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155097 2024-09-21T15:50:27.016 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.016 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:27.016 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.016 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:27.016 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449019+0000", 2024-09-21T15:50:27.016 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.016 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.016 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.016 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:27.016 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449019+0000", 2024-09-21T15:50:27.016 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.016 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.016 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.017 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:27.017 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449019+0000", 2024-09-21T15:50:27.017 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:27.017 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.017 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.017 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:27.017 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449021+0000", 2024-09-21T15:50:27.017 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.244e-06 2024-09-21T15:50:27.017 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.017 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.017 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:27.017 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449027+0000", 2024-09-21T15:50:27.017 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8619999999999999e-06 2024-09-21T15:50:27.017 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.018 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.018 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:27.018 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449031+0000", 2024-09-21T15:50:27.018 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1359999999999999e-06 2024-09-21T15:50:27.018 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.018 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.018 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:27.018 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513067+0000", 2024-09-21T15:50:27.018 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064036391999999998 2024-09-21T15:50:27.018 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.018 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.018 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:27.018 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513071+0000", 2024-09-21T15:50:27.018 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1989999999999999e-06 2024-09-21T15:50:27.018 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.019 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:27.019 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.019 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.019 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.019 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155098 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:27.019 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449035+0000", 2024-09-21T15:50:27.019 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.5390695330000002, 2024-09-21T15:50:27.019 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064071761000000005, 2024-09-21T15:50:27.019 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:27.019 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:27.019 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:27.019 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:27.019 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:27.019 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155098 2024-09-21T15:50:27.019 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.019 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:27.019 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.020 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:27.020 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449035+0000", 2024-09-21T15:50:27.020 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.020 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.020 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.020 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:27.020 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449035+0000", 2024-09-21T15:50:27.020 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.020 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.020 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.020 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:27.020 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449034+0000", 2024-09-21T15:50:27.020 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:27.020 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.020 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.020 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:27.021 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449036+0000", 2024-09-21T15:50:27.021 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9989999999999998e-06 2024-09-21T15:50:27.021 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.021 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.021 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:27.021 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449042+0000", 2024-09-21T15:50:27.021 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6010000000000002e-06 2024-09-21T15:50:27.021 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.021 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.021 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:27.021 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449047+0000", 2024-09-21T15:50:27.021 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3600000000000004e-06 2024-09-21T15:50:27.021 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.021 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.021 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:27.021 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513098+0000", 2024-09-21T15:50:27.021 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064050544000000001 2024-09-21T15:50:27.022 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.022 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.022 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:27.022 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513106+0000", 2024-09-21T15:50:27.022 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5650000000000002e-06 2024-09-21T15:50:27.022 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.022 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:27.022 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.022 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.022 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.022 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155099 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:27.022 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449051+0000", 2024-09-21T15:50:27.022 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.5390529920000002, 2024-09-21T15:50:27.022 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064069388000000005, 2024-09-21T15:50:27.022 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:27.022 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:27.023 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:27.023 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:27.023 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:27.023 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155099 2024-09-21T15:50:27.023 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.023 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:27.023 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.023 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:27.023 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449051+0000", 2024-09-21T15:50:27.023 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.023 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.023 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.023 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:27.023 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449051+0000", 2024-09-21T15:50:27.023 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.023 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.024 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.024 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:27.024 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449051+0000", 2024-09-21T15:50:27.024 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:27.024 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.024 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.024 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:27.024 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449053+0000", 2024-09-21T15:50:27.024 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2620000000000001e-06 2024-09-21T15:50:27.024 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.024 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.024 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:27.024 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449058+0000", 2024-09-21T15:50:27.024 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1680000000000003e-06 2024-09-21T15:50:27.024 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.025 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.025 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:27.025 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449062+0000", 2024-09-21T15:50:27.025 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9979999999999997e-06 2024-09-21T15:50:27.025 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.025 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.025 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:27.025 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513116+0000", 2024-09-21T15:50:27.025 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064053761000000001 2024-09-21T15:50:27.025 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.025 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.025 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:27.025 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513120+0000", 2024-09-21T15:50:27.025 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5399999999999997e-06 2024-09-21T15:50:27.025 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.025 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:27.025 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.026 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.026 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.026 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155100 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:27.026 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449101+0000", 2024-09-21T15:50:27.026 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.5390032370000002, 2024-09-21T15:50:27.026 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064079585999999994, 2024-09-21T15:50:27.026 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:27.026 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:27.026 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:27.026 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:27.026 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:27.026 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155100 2024-09-21T15:50:27.026 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.026 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:27.026 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.027 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:27.027 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449101+0000", 2024-09-21T15:50:27.027 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.027 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.027 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.027 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:27.027 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449101+0000", 2024-09-21T15:50:27.027 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.027 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.027 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.027 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:27.027 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449100+0000", 2024-09-21T15:50:27.027 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-21T15:50:27.027 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.027 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.028 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:27.028 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449105+0000", 2024-09-21T15:50:27.028 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6700000000000002e-06 2024-09-21T15:50:27.028 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.028 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.028 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:27.028 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449117+0000", 2024-09-21T15:50:27.028 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1841000000000001e-05 2024-09-21T15:50:27.028 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.028 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.028 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:27.028 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449124+0000", 2024-09-21T15:50:27.028 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4660000000000002e-06 2024-09-21T15:50:27.028 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.028 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.029 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:27.029 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513163+0000", 2024-09-21T15:50:27.029 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064039053999999998 2024-09-21T15:50:27.029 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.029 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.029 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:27.029 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513180+0000", 2024-09-21T15:50:27.029 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7333000000000002e-05 2024-09-21T15:50:27.029 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.029 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:27.029 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.029 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.029 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.029 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155101 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:27.029 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449166+0000", 2024-09-21T15:50:27.030 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.5389385319999995, 2024-09-21T15:50:27.030 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064027639999999997, 2024-09-21T15:50:27.030 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:27.030 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:27.030 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:27.030 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:27.030 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:27.030 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155101 2024-09-21T15:50:27.030 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.030 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:27.030 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.030 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:27.030 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449166+0000", 2024-09-21T15:50:27.030 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.030 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.031 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.031 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:27.031 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449166+0000", 2024-09-21T15:50:27.031 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.031 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.031 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.031 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:27.031 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449165+0000", 2024-09-21T15:50:27.031 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:27.031 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.031 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.031 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:27.031 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449168+0000", 2024-09-21T15:50:27.031 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5099999999999999e-06 2024-09-21T15:50:27.031 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.031 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.031 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:27.032 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449177+0000", 2024-09-21T15:50:27.032 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6659999999999997e-06 2024-09-21T15:50:27.032 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.032 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.032 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:27.032 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449182+0000", 2024-09-21T15:50:27.032 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1320000000000002e-06 2024-09-21T15:50:27.032 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.032 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.032 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:27.032 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513189+0000", 2024-09-21T15:50:27.032 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064007231999999997 2024-09-21T15:50:27.032 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.032 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.032 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:27.032 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513193+0000", 2024-09-21T15:50:27.033 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.692e-06 2024-09-21T15:50:27.033 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.033 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:27.033 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.033 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.033 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.033 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155134 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:27.033 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449187+0000", 2024-09-21T15:50:27.033 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.5389169440000003, 2024-09-21T15:50:27.033 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064031331999999996, 2024-09-21T15:50:27.033 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:27.033 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:27.033 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:27.033 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:27.033 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:27.033 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155134 2024-09-21T15:50:27.033 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.034 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:27.034 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.034 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:27.034 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449187+0000", 2024-09-21T15:50:27.034 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.034 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.034 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.034 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:27.034 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449187+0000", 2024-09-21T15:50:27.034 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.034 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.034 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.034 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:27.034 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449187+0000", 2024-09-21T15:50:27.034 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:27.035 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.035 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.035 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:27.035 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449189+0000", 2024-09-21T15:50:27.035 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5469999999999999e-06 2024-09-21T15:50:27.035 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.035 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.035 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:27.035 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449196+0000", 2024-09-21T15:50:27.035 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7009999999999999e-06 2024-09-21T15:50:27.035 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.035 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.035 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:27.035 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449202+0000", 2024-09-21T15:50:27.035 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7409999999999998e-06 2024-09-21T15:50:27.036 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.036 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.036 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:27.036 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513214+0000", 2024-09-21T15:50:27.036 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064012301999999993 2024-09-21T15:50:27.036 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.036 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.036 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:27.036 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513218+0000", 2024-09-21T15:50:27.036 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5709999999999999e-06 2024-09-21T15:50:27.036 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.036 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:27.036 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.036 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.036 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.037 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155135 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:27.037 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449207+0000", 2024-09-21T15:50:27.037 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.5388974339999999, 2024-09-21T15:50:27.037 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064029249999999996, 2024-09-21T15:50:27.037 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:27.037 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:27.037 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:27.037 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:27.037 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:27.037 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155135 2024-09-21T15:50:27.037 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.037 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:27.037 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.037 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:27.037 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449207+0000", 2024-09-21T15:50:27.037 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.038 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.038 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.038 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:27.038 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449207+0000", 2024-09-21T15:50:27.038 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.038 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.038 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.038 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:27.038 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449206+0000", 2024-09-21T15:50:27.038 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:27.038 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.038 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.038 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:27.039 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449209+0000", 2024-09-21T15:50:27.039 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5150000000000001e-06 2024-09-21T15:50:27.039 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.039 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.039 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:27.039 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449215+0000", 2024-09-21T15:50:27.039 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9329999999999998e-06 2024-09-21T15:50:27.039 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.039 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.039 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:27.039 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449220+0000", 2024-09-21T15:50:27.039 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0819999999999998e-06 2024-09-21T15:50:27.039 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.039 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.039 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:27.040 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513232+0000", 2024-09-21T15:50:27.040 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064012052 2024-09-21T15:50:27.040 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.040 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.040 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:27.040 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513236+0000", 2024-09-21T15:50:27.040 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2080000000000002e-06 2024-09-21T15:50:27.040 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.040 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:27.040 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.040 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.040 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.040 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155136 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:27.040 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449224+0000", 2024-09-21T15:50:27.040 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.5388798699999997, 2024-09-21T15:50:27.040 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064053025999999999, 2024-09-21T15:50:27.040 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:27.041 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:27.041 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:27.041 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:27.041 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:27.041 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155136 2024-09-21T15:50:27.041 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.041 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:27.041 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.041 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:27.041 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449224+0000", 2024-09-21T15:50:27.041 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.041 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.041 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.041 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:27.041 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449224+0000", 2024-09-21T15:50:27.042 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.042 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.042 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.042 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:27.042 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449224+0000", 2024-09-21T15:50:27.042 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:27.042 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.042 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.042 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:27.042 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449226+0000", 2024-09-21T15:50:27.042 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2699999999999999e-06 2024-09-21T15:50:27.042 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.042 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.042 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:27.042 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449233+0000", 2024-09-21T15:50:27.042 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4289999999999998e-06 2024-09-21T15:50:27.043 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.043 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.043 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:27.043 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449238+0000", 2024-09-21T15:50:27.043 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0730000000000004e-06 2024-09-21T15:50:27.043 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.043 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.043 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:27.043 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513269+0000", 2024-09-21T15:50:27.043 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064030999000000005 2024-09-21T15:50:27.043 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.043 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.043 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:27.043 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513277+0000", 2024-09-21T15:50:27.043 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5520000000000005e-06 2024-09-21T15:50:27.044 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.044 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:27.044 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.044 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.044 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.044 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155137 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:27.044 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449263+0000", 2024-09-21T15:50:27.044 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.5388407380000002, 2024-09-21T15:50:27.044 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.06402803, 2024-09-21T15:50:27.044 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:27.044 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:27.044 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:27.044 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:27.044 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:27.045 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155137 2024-09-21T15:50:27.045 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.045 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:27.045 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.045 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:27.045 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449263+0000", 2024-09-21T15:50:27.045 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.045 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.045 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.045 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:27.045 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449263+0000", 2024-09-21T15:50:27.045 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.045 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.045 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.045 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:27.045 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449262+0000", 2024-09-21T15:50:27.046 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-21T15:50:27.046 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.046 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.046 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:27.046 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449267+0000", 2024-09-21T15:50:27.046 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.933e-06 2024-09-21T15:50:27.046 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.046 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.046 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:27.046 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449279+0000", 2024-09-21T15:50:27.046 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.181e-05 2024-09-21T15:50:27.046 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.047 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.047 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:27.047 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449286+0000", 2024-09-21T15:50:27.047 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4749999999999998e-06 2024-09-21T15:50:27.047 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.047 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.047 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:27.047 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513287+0000", 2024-09-21T15:50:27.047 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064001369000000002 2024-09-21T15:50:27.047 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.047 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.047 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:27.047 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513291+0000", 2024-09-21T15:50:27.048 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3000000000000003e-06 2024-09-21T15:50:27.048 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.048 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:27.048 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.048 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.048 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.048 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155138 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:27.048 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449299+0000", 2024-09-21T15:50:27.048 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.5388052459999999, 2024-09-21T15:50:27.048 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064028359000000007, 2024-09-21T15:50:27.048 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:27.048 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:27.048 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:27.048 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:27.048 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:27.049 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155138 2024-09-21T15:50:27.049 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.049 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:27.049 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.049 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:27.049 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449299+0000", 2024-09-21T15:50:27.049 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.049 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.049 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.049 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:27.049 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449299+0000", 2024-09-21T15:50:27.049 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.049 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.049 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.049 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:27.050 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449298+0000", 2024-09-21T15:50:27.050 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:27.050 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.050 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.050 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:27.050 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449301+0000", 2024-09-21T15:50:27.050 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0230000000000001e-06 2024-09-21T15:50:27.050 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.050 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.050 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:27.050 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449308+0000", 2024-09-21T15:50:27.050 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3119999999999999e-06 2024-09-21T15:50:27.050 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.050 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.050 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:27.051 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449313+0000", 2024-09-21T15:50:27.051 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8980000000000004e-06 2024-09-21T15:50:27.051 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.051 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.051 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:27.051 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513318+0000", 2024-09-21T15:50:27.051 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064005612000000003 2024-09-21T15:50:27.051 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.051 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.051 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:27.051 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513327+0000", 2024-09-21T15:50:27.051 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0100000000000001e-06 2024-09-21T15:50:27.051 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.051 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:27.051 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.051 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.052 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.052 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155142 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:27.052 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449392+0000", 2024-09-21T15:50:27.052 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.5387124879999998, 2024-09-21T15:50:27.052 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064012464000000005, 2024-09-21T15:50:27.052 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:27.052 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:27.052 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:27.052 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:27.052 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:27.052 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155142 2024-09-21T15:50:27.052 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.052 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:27.052 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.052 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:27.052 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449392+0000", 2024-09-21T15:50:27.053 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.053 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.053 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.053 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:27.053 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449392+0000", 2024-09-21T15:50:27.053 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.053 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.053 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.053 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:27.053 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449391+0000", 2024-09-21T15:50:27.053 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:27.053 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.053 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.053 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:27.053 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449393+0000", 2024-09-21T15:50:27.054 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.26e-06 2024-09-21T15:50:27.054 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.054 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.054 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:27.054 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449399+0000", 2024-09-21T15:50:27.054 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6269999999999997e-06 2024-09-21T15:50:27.054 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.054 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.054 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:27.054 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449403+0000", 2024-09-21T15:50:27.054 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3000000000000003e-06 2024-09-21T15:50:27.054 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.054 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.054 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:27.055 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513396+0000", 2024-09-21T15:50:27.055 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.063992568999999999 2024-09-21T15:50:27.055 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.055 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.055 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:27.055 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513404+0000", 2024-09-21T15:50:27.055 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1100000000000003e-06 2024-09-21T15:50:27.055 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.055 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:27.055 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.055 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.055 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.055 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155143 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:27.055 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449407+0000", 2024-09-21T15:50:27.055 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.5386969019999999, 2024-09-21T15:50:27.056 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064009916, 2024-09-21T15:50:27.056 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:27.056 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:27.056 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:27.056 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:27.056 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:27.056 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155143 2024-09-21T15:50:27.056 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.056 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:27.056 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.056 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:27.056 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449407+0000", 2024-09-21T15:50:27.056 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.056 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.057 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.057 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:27.057 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449407+0000", 2024-09-21T15:50:27.057 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.057 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.057 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.057 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:27.057 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449407+0000", 2024-09-21T15:50:27.057 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:27.057 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.057 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.057 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:27.057 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449409+0000", 2024-09-21T15:50:27.057 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1090000000000001e-06 2024-09-21T15:50:27.057 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.058 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.058 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:27.058 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449413+0000", 2024-09-21T15:50:27.058 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6650000000000001e-06 2024-09-21T15:50:27.058 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.058 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.058 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:27.058 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449415+0000", 2024-09-21T15:50:27.058 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5459999999999998e-06 2024-09-21T15:50:27.058 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.058 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.058 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:27.058 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513413+0000", 2024-09-21T15:50:27.058 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.063998037999999993 2024-09-21T15:50:27.059 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.059 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.059 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:27.059 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513417+0000", 2024-09-21T15:50:27.059 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8789999999999996e-06 2024-09-21T15:50:27.059 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.059 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:27.059 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.059 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.059 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.059 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155144 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:27.059 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449427+0000", 2024-09-21T15:50:27.059 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 4.538676894, 2024-09-21T15:50:27.060 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064022531999999993, 2024-09-21T15:50:27.060 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:27.060 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:27.060 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:27.060 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:27.060 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:27.060 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155144 2024-09-21T15:50:27.060 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.060 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:27.060 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.060 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:27.060 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449427+0000", 2024-09-21T15:50:27.060 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.060 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.060 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.060 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:27.060 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449427+0000", 2024-09-21T15:50:27.061 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.061 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.061 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.061 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:27.061 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449427+0000", 2024-09-21T15:50:27.061 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:27.061 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.061 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.061 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:27.061 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449430+0000", 2024-09-21T15:50:27.061 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3579999999999999e-06 2024-09-21T15:50:27.061 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.061 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.061 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:27.061 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449437+0000", 2024-09-21T15:50:27.062 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4580000000000004e-06 2024-09-21T15:50:27.062 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.062 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.062 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:27.062 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449443+0000", 2024-09-21T15:50:27.062 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5609999999999998e-06 2024-09-21T15:50:27.062 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.062 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.062 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:27.062 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513441+0000", 2024-09-21T15:50:27.062 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.063998136999999997 2024-09-21T15:50:27.062 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.062 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.062 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:27.063 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513450+0000", 2024-09-21T15:50:27.063 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4950000000000008e-06 2024-09-21T15:50:27.063 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.063 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:27.063 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.063 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.063 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:27.063 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:27.064 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:50:27.065 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:27.063+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:27.131 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T15:50:27.174 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:27.173+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:27.216 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:27.216 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:27.216 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:50:27.216 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:27.225 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:50:27.268 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:27.267+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:27.327 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:27.327 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:27.327 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:50:27.327 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:50:27.327 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:27.336 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:50:27.367 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:27.366+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:27.467 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:27.466+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:27.480 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:27.480 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:50:27.480 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:50:27.480 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:50:27.480 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.480 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120111 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 321536~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:27.480 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572216+0000", 2024-09-21T15:50:27.480 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 23.906067746000002, 2024-09-21T15:50:27.480 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054657399000000002, 2024-09-21T15:50:27.480 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:27.480 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:27.480 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:27.481 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:27.481 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:27.481 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120111 2024-09-21T15:50:27.481 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.481 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:27.481 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.481 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:27.481 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572216+0000", 2024-09-21T15:50:27.481 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.481 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.481 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.481 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:27.482 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572216+0000", 2024-09-21T15:50:27.482 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.482 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.482 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.482 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:27.482 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572216+0000", 2024-09-21T15:50:27.482 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:27.482 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.482 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.482 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:27.482 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572218+0000", 2024-09-21T15:50:27.482 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1840000000000002e-06 2024-09-21T15:50:27.482 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.482 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.482 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:27.483 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572224+0000", 2024-09-21T15:50:27.483 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7880000000000003e-06 2024-09-21T15:50:27.483 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.483 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.483 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:27.483 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572229+0000", 2024-09-21T15:50:27.483 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8239999999999999e-06 2024-09-21T15:50:27.483 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.483 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.483 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:27.483 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626680+0000", 2024-09-21T15:50:27.483 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054451431000000002 2024-09-21T15:50:27.483 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.483 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.483 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:27.483 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626755+0000", 2024-09-21T15:50:27.484 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4641000000000006e-05 2024-09-21T15:50:27.484 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.484 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.484 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:27.484 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626874+0000", 2024-09-21T15:50:27.484 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00011886 2024-09-21T15:50:27.484 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.484 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:27.484 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.484 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.484 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.484 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120112 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 342016~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:27.484 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572233+0000", 2024-09-21T15:50:27.484 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 23.906051378000001, 2024-09-21T15:50:27.485 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054823997999999999, 2024-09-21T15:50:27.485 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:27.485 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:27.485 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:27.485 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:27.485 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:27.485 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120112 2024-09-21T15:50:27.485 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.485 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:27.485 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.485 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:27.485 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572233+0000", 2024-09-21T15:50:27.485 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.485 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.485 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.485 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:27.486 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572233+0000", 2024-09-21T15:50:27.486 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.486 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.486 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.486 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:27.486 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572232+0000", 2024-09-21T15:50:27.486 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:27.486 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.486 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.486 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:27.486 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572234+0000", 2024-09-21T15:50:27.486 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0660000000000002e-06 2024-09-21T15:50:27.486 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.486 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.486 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:27.487 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572242+0000", 2024-09-21T15:50:27.487 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0430000000000002e-06 2024-09-21T15:50:27.487 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.487 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.487 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:27.487 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572246+0000", 2024-09-21T15:50:27.487 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9080000000000003e-06 2024-09-21T15:50:27.487 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.487 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.487 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:27.487 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626916+0000", 2024-09-21T15:50:27.487 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054669482999999998 2024-09-21T15:50:27.487 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.487 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.487 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:27.487 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626977+0000", 2024-09-21T15:50:27.488 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0597000000000003e-05 2024-09-21T15:50:27.488 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.488 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.488 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:27.488 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627057+0000", 2024-09-21T15:50:27.488 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0258000000000003e-05 2024-09-21T15:50:27.488 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.488 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:27.488 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.488 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.488 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.488 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120113 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 362496~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:27.488 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572253+0000", 2024-09-21T15:50:27.488 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 23.906031392999999, 2024-09-21T15:50:27.488 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.05503044, 2024-09-21T15:50:27.489 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:27.489 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:27.489 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:27.489 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:27.489 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:27.489 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120113 2024-09-21T15:50:27.489 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.489 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:27.489 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.489 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:27.489 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572253+0000", 2024-09-21T15:50:27.489 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.489 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.489 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.489 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:27.489 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572253+0000", 2024-09-21T15:50:27.489 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.490 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.490 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.490 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:27.490 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572252+0000", 2024-09-21T15:50:27.490 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:27.490 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.490 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.490 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:27.490 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572258+0000", 2024-09-21T15:50:27.490 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6169999999999998e-06 2024-09-21T15:50:27.490 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.490 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.490 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:27.490 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572266+0000", 2024-09-21T15:50:27.491 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9410000000000007e-06 2024-09-21T15:50:27.491 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.491 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.491 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:27.491 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572272+0000", 2024-09-21T15:50:27.491 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.241e-06 2024-09-21T15:50:27.491 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.491 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.491 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:27.491 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627087+0000", 2024-09-21T15:50:27.491 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054814850999999998 2024-09-21T15:50:27.491 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.491 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.491 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:27.491 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627162+0000", 2024-09-21T15:50:27.492 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5012999999999995e-05 2024-09-21T15:50:27.492 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.492 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.492 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:27.492 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627283+0000", 2024-09-21T15:50:27.492 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012113 2024-09-21T15:50:27.492 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.492 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:27.492 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.492 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.492 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.492 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120114 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 382976~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:27.492 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572277+0000", 2024-09-21T15:50:27.492 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 23.906007375000002, 2024-09-21T15:50:27.492 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055184395999999997, 2024-09-21T15:50:27.492 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:27.493 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:27.493 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:27.493 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:27.493 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:27.493 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120114 2024-09-21T15:50:27.493 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.493 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:27.493 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.493 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:27.493 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572277+0000", 2024-09-21T15:50:27.493 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.493 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.493 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.493 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:27.493 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572277+0000", 2024-09-21T15:50:27.493 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.494 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.494 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.494 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:27.494 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572276+0000", 2024-09-21T15:50:27.494 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:27.494 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.494 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.494 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:27.494 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572279+0000", 2024-09-21T15:50:27.494 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.373e-06 2024-09-21T15:50:27.494 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.494 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.494 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:27.494 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572285+0000", 2024-09-21T15:50:27.494 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9359999999999996e-06 2024-09-21T15:50:27.495 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.495 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.495 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:27.495 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572289+0000", 2024-09-21T15:50:27.495 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.634e-06 2024-09-21T15:50:27.495 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.495 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.495 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:27.495 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627311+0000", 2024-09-21T15:50:27.495 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055021211 2024-09-21T15:50:27.495 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.495 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.495 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:27.495 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627360+0000", 2024-09-21T15:50:27.495 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9203999999999998e-05 2024-09-21T15:50:27.496 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.496 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.496 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:27.497 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627461+0000", 2024-09-21T15:50:27.497 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000101469 2024-09-21T15:50:27.497 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.497 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:27.497 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.497 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.497 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.497 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120115 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 399360~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:27.497 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572294+0000", 2024-09-21T15:50:27.497 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 23.905990068000001, 2024-09-21T15:50:27.497 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055400092999999997, 2024-09-21T15:50:27.497 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:27.497 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:27.498 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:27.498 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:27.498 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:27.498 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120115 2024-09-21T15:50:27.498 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.498 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:27.498 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.498 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:27.498 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572294+0000", 2024-09-21T15:50:27.498 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.498 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.499 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.499 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:27.499 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572294+0000", 2024-09-21T15:50:27.499 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.499 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.499 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.499 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:27.500 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572294+0000", 2024-09-21T15:50:27.500 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:27.500 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.500 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.500 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:27.500 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572296+0000", 2024-09-21T15:50:27.500 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1100000000000001e-06 2024-09-21T15:50:27.500 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.500 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.500 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:27.500 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572303+0000", 2024-09-21T15:50:27.500 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8580000000000002e-06 2024-09-21T15:50:27.500 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.501 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.501 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:27.501 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572314+0000", 2024-09-21T15:50:27.501 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.101e-05 2024-09-21T15:50:27.501 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.501 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.501 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:27.501 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627496+0000", 2024-09-21T15:50:27.501 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055182522999999997 2024-09-21T15:50:27.501 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.501 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.501 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:27.501 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627568+0000", 2024-09-21T15:50:27.502 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1569999999999994e-05 2024-09-21T15:50:27.502 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.502 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.502 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:27.502 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627694+0000", 2024-09-21T15:50:27.502 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000126372 2024-09-21T15:50:27.502 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.502 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:27.502 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.502 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.502 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.502 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120116 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 419840~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:27.502 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572318+0000", 2024-09-21T15:50:27.502 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 23.905965877, 2024-09-21T15:50:27.503 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055539416000000001, 2024-09-21T15:50:27.503 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:27.503 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:27.503 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:27.503 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:27.503 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:27.503 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120116 2024-09-21T15:50:27.503 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.503 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:27.503 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.503 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:27.503 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572318+0000", 2024-09-21T15:50:27.503 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.504 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.504 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.504 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:27.504 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572318+0000", 2024-09-21T15:50:27.504 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.504 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.504 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.504 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:27.504 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572318+0000", 2024-09-21T15:50:27.504 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:27.504 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.504 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.504 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:27.504 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572322+0000", 2024-09-21T15:50:27.504 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4839999999999998e-06 2024-09-21T15:50:27.505 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.505 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.505 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:27.505 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572329+0000", 2024-09-21T15:50:27.505 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0959999999999997e-06 2024-09-21T15:50:27.505 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.505 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.505 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:27.505 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572333+0000", 2024-09-21T15:50:27.505 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9359999999999998e-06 2024-09-21T15:50:27.505 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.505 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.505 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:27.505 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627726+0000", 2024-09-21T15:50:27.506 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055392829999999997 2024-09-21T15:50:27.506 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.506 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.506 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:27.506 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627779+0000", 2024-09-21T15:50:27.506 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2185999999999998e-05 2024-09-21T15:50:27.506 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.506 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.506 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:27.506 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627858+0000", 2024-09-21T15:50:27.506 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9190999999999994e-05 2024-09-21T15:50:27.506 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.506 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:27.506 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.506 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.507 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.507 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120117 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 440320~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:27.507 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572338+0000", 2024-09-21T15:50:27.507 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 23.905946564000001, 2024-09-21T15:50:27.507 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055680888999999997, 2024-09-21T15:50:27.507 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:27.507 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:27.507 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:27.507 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:27.507 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:27.507 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120117 2024-09-21T15:50:27.507 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.508 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:27.508 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.508 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:27.508 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572338+0000", 2024-09-21T15:50:27.508 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.508 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.508 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.508 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:27.508 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572338+0000", 2024-09-21T15:50:27.508 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.508 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.508 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.508 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:27.508 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572337+0000", 2024-09-21T15:50:27.509 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:27.509 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.509 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.509 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:27.509 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572339+0000", 2024-09-21T15:50:27.509 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9379999999999999e-06 2024-09-21T15:50:27.509 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.509 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.509 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:27.509 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572349+0000", 2024-09-21T15:50:27.509 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7780000000000002e-06 2024-09-21T15:50:27.509 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.509 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.509 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:27.509 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572354+0000", 2024-09-21T15:50:27.510 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8740000000000003e-06 2024-09-21T15:50:27.510 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.510 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.510 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:27.510 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627885+0000", 2024-09-21T15:50:27.510 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055531448999999997 2024-09-21T15:50:27.510 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.510 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.510 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:27.510 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627935+0000", 2024-09-21T15:50:27.510 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9348999999999998e-05 2024-09-21T15:50:27.510 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.510 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.510 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:27.511 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628018+0000", 2024-09-21T15:50:27.511 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3851999999999996e-05 2024-09-21T15:50:27.511 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.511 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:27.511 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.511 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.511 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.511 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120118 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 460800~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:27.511 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572359+0000", 2024-09-21T15:50:27.511 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 23.905925322000002, 2024-09-21T15:50:27.511 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055951714, 2024-09-21T15:50:27.511 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:27.511 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:27.512 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:27.512 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:27.512 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:27.512 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120118 2024-09-21T15:50:27.512 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.512 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:27.512 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.512 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:27.512 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572359+0000", 2024-09-21T15:50:27.512 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.512 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.512 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.512 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:27.512 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572359+0000", 2024-09-21T15:50:27.512 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.513 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.513 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.513 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:27.513 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572358+0000", 2024-09-21T15:50:27.513 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:27.513 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.513 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.513 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:27.513 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572361+0000", 2024-09-21T15:50:27.513 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.26e-06 2024-09-21T15:50:27.513 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.513 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.513 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:27.513 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572367+0000", 2024-09-21T15:50:27.514 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3640000000000004e-06 2024-09-21T15:50:27.514 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.514 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.514 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:27.514 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572376+0000", 2024-09-21T15:50:27.514 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2310000000000002e-06 2024-09-21T15:50:27.514 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.514 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.514 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:27.514 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628096+0000", 2024-09-21T15:50:27.514 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055719369999999997 2024-09-21T15:50:27.514 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.514 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.514 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:27.515 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628178+0000", 2024-09-21T15:50:27.515 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2667e-05 2024-09-21T15:50:27.515 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.515 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.515 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:27.515 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628311+0000", 2024-09-21T15:50:27.515 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000132175 2024-09-21T15:50:27.515 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.515 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:27.515 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.515 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.515 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.515 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120119 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 481280~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:27.515 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572381+0000", 2024-09-21T15:50:27.515 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 23.905902877999999, 2024-09-21T15:50:27.515 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056088791999999998, 2024-09-21T15:50:27.516 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:27.516 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:27.516 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:27.516 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:27.516 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:27.516 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120119 2024-09-21T15:50:27.516 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.516 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:27.516 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.516 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:27.516 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572381+0000", 2024-09-21T15:50:27.516 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.516 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.516 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.516 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:27.516 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572381+0000", 2024-09-21T15:50:27.517 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.517 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.517 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.517 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:27.517 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572381+0000", 2024-09-21T15:50:27.517 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:27.517 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.517 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.517 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:27.517 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572383+0000", 2024-09-21T15:50:27.517 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0619999999999999e-06 2024-09-21T15:50:27.517 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.517 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.517 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:27.517 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572391+0000", 2024-09-21T15:50:27.518 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8769999999999993e-06 2024-09-21T15:50:27.518 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.518 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.518 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:27.518 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572399+0000", 2024-09-21T15:50:27.518 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6240000000000008e-06 2024-09-21T15:50:27.518 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.518 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.518 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:27.518 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628344+0000", 2024-09-21T15:50:27.518 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055944828000000002 2024-09-21T15:50:27.518 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.518 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.518 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:27.518 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628393+0000", 2024-09-21T15:50:27.518 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8653000000000003e-05 2024-09-21T15:50:27.519 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.519 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.519 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:27.519 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628470+0000", 2024-09-21T15:50:27.519 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7121999999999998e-05 2024-09-21T15:50:27.519 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.519 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:27.519 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.519 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.519 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.519 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120120 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 501760~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:27.519 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572418+0000", 2024-09-21T15:50:27.519 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 23.905866379999999, 2024-09-21T15:50:27.519 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056209138999999998, 2024-09-21T15:50:27.519 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:27.519 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:27.520 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:27.520 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:27.520 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:27.520 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120120 2024-09-21T15:50:27.520 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.520 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:27.520 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.520 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:27.520 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572418+0000", 2024-09-21T15:50:27.520 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.520 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.520 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.520 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:27.520 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572418+0000", 2024-09-21T15:50:27.520 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.520 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.521 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.521 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:27.521 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572417+0000", 2024-09-21T15:50:27.521 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:27.521 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.521 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.521 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:27.521 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572420+0000", 2024-09-21T15:50:27.521 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5270000000000001e-06 2024-09-21T15:50:27.521 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.521 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.521 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:27.521 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572426+0000", 2024-09-21T15:50:27.521 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3380000000000001e-06 2024-09-21T15:50:27.521 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.522 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.522 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:27.522 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572431+0000", 2024-09-21T15:50:27.522 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.865e-06 2024-09-21T15:50:27.522 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.522 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.522 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:27.522 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628500+0000", 2024-09-21T15:50:27.522 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056068917000000003 2024-09-21T15:50:27.522 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.522 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.522 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:27.522 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628542+0000", 2024-09-21T15:50:27.522 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1761999999999998e-05 2024-09-21T15:50:27.522 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.523 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.523 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:27.523 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628627+0000", 2024-09-21T15:50:27.523 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5247999999999994e-05 2024-09-21T15:50:27.523 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.523 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:27.523 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.523 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.523 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.523 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120121 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 522240~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:27.523 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572630+0000", 2024-09-21T15:50:27.523 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 23.905653994000001, 2024-09-21T15:50:27.523 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056207209000000001, 2024-09-21T15:50:27.523 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:27.523 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:27.524 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:27.524 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:27.524 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:27.524 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120121 2024-09-21T15:50:27.524 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.524 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:27.524 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.524 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:27.524 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572630+0000", 2024-09-21T15:50:27.524 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.524 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.524 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.524 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:27.524 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572630+0000", 2024-09-21T15:50:27.524 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.524 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.525 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.525 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:27.525 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572435+0000", 2024-09-21T15:50:27.525 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998045 2024-09-21T15:50:27.525 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.525 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.525 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:27.525 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572633+0000", 2024-09-21T15:50:27.525 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000198786 2024-09-21T15:50:27.525 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.525 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.525 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:27.525 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572641+0000", 2024-09-21T15:50:27.525 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9449999999999993e-06 2024-09-21T15:50:27.525 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.525 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.526 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:27.526 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572648+0000", 2024-09-21T15:50:27.526 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1929999999999998e-06 2024-09-21T15:50:27.526 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.526 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.526 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:27.526 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628669+0000", 2024-09-21T15:50:27.526 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056021690999999998 2024-09-21T15:50:27.526 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.526 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.526 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:27.526 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628751+0000", 2024-09-21T15:50:27.526 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1710000000000002e-05 2024-09-21T15:50:27.526 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.526 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.526 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:27.527 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628837+0000", 2024-09-21T15:50:27.527 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6383000000000003e-05 2024-09-21T15:50:27.527 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.527 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:27.527 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.527 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.527 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.527 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120122 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 538624~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:27.527 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.573471+0000", 2024-09-21T15:50:27.527 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 23.904812749000001, 2024-09-21T15:50:27.527 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055563935000000002, 2024-09-21T15:50:27.527 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:27.527 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:27.527 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:27.527 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:27.528 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:27.528 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120122 2024-09-21T15:50:27.528 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.528 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:27.528 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.528 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:27.528 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573471+0000", 2024-09-21T15:50:27.528 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.528 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.528 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.528 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:27.528 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573471+0000", 2024-09-21T15:50:27.528 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.528 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.528 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.529 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:27.529 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572653+0000", 2024-09-21T15:50:27.529 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9991813 2024-09-21T15:50:27.529 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.529 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.529 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:27.529 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573475+0000", 2024-09-21T15:50:27.529 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00082196500000000002 2024-09-21T15:50:27.529 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.529 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.529 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:27.529 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573483+0000", 2024-09-21T15:50:27.529 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5939999999999994e-06 2024-09-21T15:50:27.529 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.529 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.529 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:27.529 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573489+0000", 2024-09-21T15:50:27.530 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1689999999999997e-06 2024-09-21T15:50:27.530 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.530 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.530 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:27.530 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628882+0000", 2024-09-21T15:50:27.530 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055392230000000001 2024-09-21T15:50:27.530 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.530 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.530 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:27.530 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628934+0000", 2024-09-21T15:50:27.530 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1870000000000003e-05 2024-09-21T15:50:27.530 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.530 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.530 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:27.531 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629035+0000", 2024-09-21T15:50:27.531 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000101753 2024-09-21T15:50:27.531 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.531 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:27.531 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.531 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.531 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.531 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120123 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 559104~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:27.531 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.573495+0000", 2024-09-21T15:50:27.531 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 23.904789050000002, 2024-09-21T15:50:27.531 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055704765000000003, 2024-09-21T15:50:27.531 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:27.531 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:27.531 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:27.532 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:27.532 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:27.532 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120123 2024-09-21T15:50:27.532 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.532 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:27.532 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.532 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:27.532 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573495+0000", 2024-09-21T15:50:27.532 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.532 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.532 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.532 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:27.532 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573495+0000", 2024-09-21T15:50:27.533 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.533 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.533 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.533 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:27.533 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573495+0000", 2024-09-21T15:50:27.533 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:27.533 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.533 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.533 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:27.533 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573497+0000", 2024-09-21T15:50:27.533 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3420000000000002e-06 2024-09-21T15:50:27.533 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.533 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.533 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:27.533 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573503+0000", 2024-09-21T15:50:27.534 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7030000000000003e-06 2024-09-21T15:50:27.534 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.534 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.534 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:27.534 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573511+0000", 2024-09-21T15:50:27.534 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0790000000000001e-06 2024-09-21T15:50:27.534 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.534 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.534 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:27.534 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629067+0000", 2024-09-21T15:50:27.534 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055556573999999997 2024-09-21T15:50:27.534 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.534 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.534 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:27.534 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629118+0000", 2024-09-21T15:50:27.535 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1053000000000001e-05 2024-09-21T15:50:27.535 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.535 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.535 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:27.535 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629200+0000", 2024-09-21T15:50:27.535 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1452999999999994e-05 2024-09-21T15:50:27.535 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.535 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:27.535 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.535 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.535 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.535 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120124 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 579584~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:27.535 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.574014+0000", 2024-09-21T15:50:27.535 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 23.904270011000001, 2024-09-21T15:50:27.535 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055440284999999999, 2024-09-21T15:50:27.535 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:27.536 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:27.536 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:27.536 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:27.536 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:27.536 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120124 2024-09-21T15:50:27.536 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.536 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:27.536 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.536 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:27.536 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574014+0000", 2024-09-21T15:50:27.536 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.536 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.536 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.536 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:27.536 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574014+0000", 2024-09-21T15:50:27.536 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.537 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.537 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.537 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:27.537 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573515+0000", 2024-09-21T15:50:27.537 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9995008 2024-09-21T15:50:27.537 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.537 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.537 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:27.537 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574017+0000", 2024-09-21T15:50:27.537 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00050241100000000003 2024-09-21T15:50:27.537 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.537 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.537 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:27.537 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574027+0000", 2024-09-21T15:50:27.537 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4700000000000008e-06 2024-09-21T15:50:27.538 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.538 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.538 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:27.538 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574033+0000", 2024-09-21T15:50:27.538 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1489999999999999e-06 2024-09-21T15:50:27.538 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.538 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.538 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:27.538 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629238+0000", 2024-09-21T15:50:27.538 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055204880999999997 2024-09-21T15:50:27.538 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.538 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.538 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:27.538 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629317+0000", 2024-09-21T15:50:27.538 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8942000000000004e-05 2024-09-21T15:50:27.539 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.539 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.539 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:27.539 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629454+0000", 2024-09-21T15:50:27.539 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013767099999999999 2024-09-21T15:50:27.539 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.539 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:27.539 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.539 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.539 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.539 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120125 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 600064~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:27.539 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.574482+0000", 2024-09-21T15:50:27.539 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 23.903801712, 2024-09-21T15:50:27.539 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055243608, 2024-09-21T15:50:27.539 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:27.539 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:27.540 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:27.540 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:27.540 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:27.540 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120125 2024-09-21T15:50:27.540 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.540 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:27.540 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.540 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:27.540 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574482+0000", 2024-09-21T15:50:27.540 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.540 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.540 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.540 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:27.540 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574482+0000", 2024-09-21T15:50:27.540 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.541 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.541 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.541 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:27.541 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574038+0000", 2024-09-21T15:50:27.541 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9995556 2024-09-21T15:50:27.541 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.541 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.541 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:27.541 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574497+0000", 2024-09-21T15:50:27.541 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00045854400000000002 2024-09-21T15:50:27.541 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.541 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.541 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:27.541 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574509+0000", 2024-09-21T15:50:27.541 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2486e-05 2024-09-21T15:50:27.542 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.542 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.542 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:27.542 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574516+0000", 2024-09-21T15:50:27.542 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2269999999999996e-06 2024-09-21T15:50:27.542 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.542 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.542 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:27.542 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629497+0000", 2024-09-21T15:50:27.542 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054980785999999997 2024-09-21T15:50:27.542 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.542 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.542 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:27.542 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629580+0000", 2024-09-21T15:50:27.542 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2528000000000007e-05 2024-09-21T15:50:27.543 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.543 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.543 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:27.543 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629726+0000", 2024-09-21T15:50:27.543 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014630399999999999 2024-09-21T15:50:27.543 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.543 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:27.543 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.543 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.543 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.543 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120126 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 620544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:27.543 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.575315+0000", 2024-09-21T15:50:27.543 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 23.902968682000001, 2024-09-21T15:50:27.543 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054709559999999997, 2024-09-21T15:50:27.543 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:27.544 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:27.544 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:27.544 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:27.544 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:27.544 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120126 2024-09-21T15:50:27.544 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.544 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:27.544 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.544 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:27.544 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575315+0000", 2024-09-21T15:50:27.544 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.544 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.544 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.544 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:27.544 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575315+0000", 2024-09-21T15:50:27.545 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.545 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.545 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.545 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:27.545 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574522+0000", 2024-09-21T15:50:27.545 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9992065 2024-09-21T15:50:27.545 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.545 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.545 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:27.545 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575319+0000", 2024-09-21T15:50:27.545 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00079626800000000004 2024-09-21T15:50:27.545 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.545 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.545 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:27.546 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575329+0000", 2024-09-21T15:50:27.546 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.028e-05 2024-09-21T15:50:27.546 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.546 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.546 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:27.546 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575335+0000", 2024-09-21T15:50:27.546 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9290000000000003e-06 2024-09-21T15:50:27.546 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.546 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.546 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:27.546 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629806+0000", 2024-09-21T15:50:27.546 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054471558000000003 2024-09-21T15:50:27.546 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.546 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.546 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:27.547 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629885+0000", 2024-09-21T15:50:27.547 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8563e-05 2024-09-21T15:50:27.547 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.547 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.547 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:27.547 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630025+0000", 2024-09-21T15:50:27.547 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014019 2024-09-21T15:50:27.547 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.547 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:27.547 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.547 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.547 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.547 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120127 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 641024~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:27.547 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.575340+0000", 2024-09-21T15:50:27.547 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 23.902943974999999, 2024-09-21T15:50:27.547 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054895589000000002, 2024-09-21T15:50:27.548 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:27.548 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:27.548 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:27.548 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:27.548 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:27.548 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120127 2024-09-21T15:50:27.548 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.548 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:27.548 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.548 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:27.548 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575340+0000", 2024-09-21T15:50:27.548 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.548 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.548 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.548 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:27.548 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575340+0000", 2024-09-21T15:50:27.549 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.549 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.549 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.549 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:27.549 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575340+0000", 2024-09-21T15:50:27.549 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:27.549 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.549 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.549 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:27.549 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575342+0000", 2024-09-21T15:50:27.549 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5600000000000001e-06 2024-09-21T15:50:27.549 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.549 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.549 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:27.549 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575351+0000", 2024-09-21T15:50:27.550 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2349999999999999e-06 2024-09-21T15:50:27.550 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.550 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.550 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:27.550 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575356+0000", 2024-09-21T15:50:27.550 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4600000000000002e-06 2024-09-21T15:50:27.550 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.550 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.550 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:27.550 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630053+0000", 2024-09-21T15:50:27.550 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054697190999999999 2024-09-21T15:50:27.550 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.550 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.551 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:27.551 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630128+0000", 2024-09-21T15:50:27.551 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4777000000000006e-05 2024-09-21T15:50:27.551 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.551 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.551 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:27.551 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630236+0000", 2024-09-21T15:50:27.551 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000107837 2024-09-21T15:50:27.551 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.551 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:27.551 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.551 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.551 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.551 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120128 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 657408~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:27.551 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.575461+0000", 2024-09-21T15:50:27.552 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 23.902823395999999, 2024-09-21T15:50:27.552 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054945121999999999, 2024-09-21T15:50:27.552 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:27.552 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:27.552 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:27.552 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:27.552 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:27.552 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120128 2024-09-21T15:50:27.552 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.552 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:27.552 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.552 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:27.552 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575461+0000", 2024-09-21T15:50:27.552 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.552 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.552 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.553 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:27.553 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575461+0000", 2024-09-21T15:50:27.553 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.553 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.553 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.553 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:27.553 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575360+0000", 2024-09-21T15:50:27.553 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998994 2024-09-21T15:50:27.553 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.553 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.553 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:27.553 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575464+0000", 2024-09-21T15:50:27.553 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000103397 2024-09-21T15:50:27.553 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.553 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.553 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:27.553 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575472+0000", 2024-09-21T15:50:27.554 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3189999999999995e-06 2024-09-21T15:50:27.554 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.554 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.554 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:27.554 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575478+0000", 2024-09-21T15:50:27.554 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2779999999999999e-06 2024-09-21T15:50:27.554 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.554 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.554 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:27.554 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630268+0000", 2024-09-21T15:50:27.554 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054789545000000002 2024-09-21T15:50:27.554 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.554 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.554 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:27.554 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630315+0000", 2024-09-21T15:50:27.555 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7244999999999999e-05 2024-09-21T15:50:27.555 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.555 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.555 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:27.555 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630406+0000", 2024-09-21T15:50:27.555 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0811999999999997e-05 2024-09-21T15:50:27.555 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.555 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:27.555 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.555 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.555 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.555 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120129 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 677888~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:27.555 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.576016+0000", 2024-09-21T15:50:27.555 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 23.902267852000001, 2024-09-21T15:50:27.555 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054602823000000002, 2024-09-21T15:50:27.556 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:27.556 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:27.556 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:27.556 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:27.556 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:27.556 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120129 2024-09-21T15:50:27.556 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.556 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:27.556 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.556 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:27.556 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576016+0000", 2024-09-21T15:50:27.556 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.556 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.556 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.556 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:27.556 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576016+0000", 2024-09-21T15:50:27.556 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.557 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.557 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.557 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:27.557 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575483+0000", 2024-09-21T15:50:27.557 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9994669 2024-09-21T15:50:27.557 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.557 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.557 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:27.557 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576020+0000", 2024-09-21T15:50:27.557 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00053657900000000003 2024-09-21T15:50:27.557 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.557 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.557 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:27.557 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576030+0000", 2024-09-21T15:50:27.557 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0326000000000001e-05 2024-09-21T15:50:27.557 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.558 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.558 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:27.558 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576037+0000", 2024-09-21T15:50:27.558 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6490000000000002e-06 2024-09-21T15:50:27.558 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.558 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.558 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:27.558 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630446+0000", 2024-09-21T15:50:27.558 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054409698999999999 2024-09-21T15:50:27.558 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.558 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.558 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:27.558 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630524+0000", 2024-09-21T15:50:27.558 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7422999999999994e-05 2024-09-21T15:50:27.558 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.558 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.559 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:27.559 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630619+0000", 2024-09-21T15:50:27.559 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5332000000000001e-05 2024-09-21T15:50:27.559 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.559 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:27.559 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.559 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.559 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.559 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120130 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 698368~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:27.559 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.576043+0000", 2024-09-21T15:50:27.559 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 23.902241481000001, 2024-09-21T15:50:27.559 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054781798, 2024-09-21T15:50:27.559 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:27.559 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:27.559 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:27.559 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:27.560 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:27.560 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120130 2024-09-21T15:50:27.560 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.560 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:27.560 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.560 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:27.560 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576043+0000", 2024-09-21T15:50:27.560 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.560 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.560 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.560 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:27.560 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576043+0000", 2024-09-21T15:50:27.560 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:27.560 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.561 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.561 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:27.561 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576042+0000", 2024-09-21T15:50:27.561 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:27.561 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.561 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.561 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:27.561 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576044+0000", 2024-09-21T15:50:27.561 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0999999999999998e-06 2024-09-21T15:50:27.561 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.561 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.561 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:27.561 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576051+0000", 2024-09-21T15:50:27.561 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4779999999999996e-06 2024-09-21T15:50:27.561 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.562 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.562 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:27.562 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576058+0000", 2024-09-21T15:50:27.562 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2180000000000002e-06 2024-09-21T15:50:27.562 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.562 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.562 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:27.562 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630661+0000", 2024-09-21T15:50:27.562 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054603240999999997 2024-09-21T15:50:27.562 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.562 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.562 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:27.562 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630747+0000", 2024-09-21T15:50:27.562 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6046999999999998e-05 2024-09-21T15:50:27.562 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:27.562 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:27.563 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:27.563 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630824+0000", 2024-09-21T15:50:27.563 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7119999999999993e-05 2024-09-21T15:50:27.563 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.563 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:27.563 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.563 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:27.563 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:27.563 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:27.564 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T15:50:27.605 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:27.604+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:27.669 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:27.668+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:27.707 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:27.707 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:27.707 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:50:27.707 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:27.715 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T15:50:27.723 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:50:27.769 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:27.768+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:27.817 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:27.817 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:27.817 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:50:27.817 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:50:27.817 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:27.825 INFO:teuthology.orchestra.run.smithi089.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T15:50:27.826 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T15:50:27.841 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:50:27.841 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:50:27.857 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:50:27.857 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:50:27.869 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:27.868+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:27.873 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:50:27.873 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:50:27.889 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:50:27.890 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:50:27.906 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:50:27.907 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:50:27.927 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:50:27.927 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:50:27.956 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:50:27.956 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:50:27.960 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T15:50:27.961 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T15:50:27.967 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:50:27.965+0000 7f7448e0e640 -1 osd.0 67 log_to_monitors true 2024-09-21T15:50:27.969 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:27.969 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:27.970 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:27.969+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:27.978 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T15:50:28.071 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:28.069+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:28.080 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:28.080 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:28.088 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T15:50:28.133 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is unset 2024-09-21T15:50:28.143 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:50:28.142+0000 7f743c4a5640 -1 osd.0 67 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T15:50:28.171 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:28.170+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:28.195 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:28.196 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:28.204 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T15:50:28.289 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:28.288+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:28.348 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:28.348 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:28.356 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:50:28.400 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:28.399+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:28.460 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:28.460 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:28.469 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:50:28.510 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:28.510+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:28.575 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:28.574+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:28.611 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:28.611 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:28.620 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:50:28.676 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:28.675+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:28.722 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:28.722 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:28.731 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:50:28.776 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:28.775+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:28.832 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:28.833 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:28.841 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:50:28.878 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:28.877+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:28.979 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:28.978+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:28.986 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:28.986 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:28.995 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:50:29.080 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:29.079+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:29.141 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:29.141 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:29.151 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:50:29.194 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:29.193+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:29.240 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:50:29.253 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:29.254 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:29.263 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:50:29.304 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:29.303+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:29.345 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:29.345 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:50:29.345 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:29.345 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160505 3.b 3.3a96e94b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-21T15:50:29.345 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:29.138810+0000", 2024-09-21T15:50:29.345 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.205860656, 2024-09-21T15:50:29.345 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20588010200000001, 2024-09-21T15:50:29.345 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:29.345 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "waiting for peered", 2024-09-21T15:50:29.346 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:29.346 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:29.346 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:29.346 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160505 2024-09-21T15:50:29.346 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:29.346 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:29.346 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:29.346 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:29.346 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138810+0000", 2024-09-21T15:50:29.346 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:29.346 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:29.346 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:29.346 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:29.346 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138810+0000", 2024-09-21T15:50:29.347 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:29.347 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:29.347 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:29.347 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:29.347 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138808+0000", 2024-09-21T15:50:29.347 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:50:29.347 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:29.347 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:29.347 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:29.347 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138815+0000", 2024-09-21T15:50:29.347 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6549999999999998e-06 2024-09-21T15:50:29.347 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:29.347 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:29.347 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:29.347 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138825+0000", 2024-09-21T15:50:29.348 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9790000000000005e-06 2024-09-21T15:50:29.348 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:29.348 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:29.348 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:29.348 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138834+0000", 2024-09-21T15:50:29.348 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5179999999999993e-06 2024-09-21T15:50:29.348 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:29.348 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:29.348 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:29.348 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.140973+0000", 2024-09-21T15:50:29.348 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0021392049999999999 2024-09-21T15:50:29.348 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:29.348 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:29.348 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:50:29.348 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.140981+0000", 2024-09-21T15:50:29.348 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8909999999999995e-06 2024-09-21T15:50:29.348 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:29.349 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:29.349 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:29.349 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:29.349 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:29.349 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160506 3.b 3.3a96e94b (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-21T15:50:29.349 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:29.138844+0000", 2024-09-21T15:50:29.349 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.20582576899999999, 2024-09-21T15:50:29.349 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.20589158499999999, 2024-09-21T15:50:29.349 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:29.349 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "waiting for peered", 2024-09-21T15:50:29.349 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:29.349 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:29.349 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:29.349 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160506 2024-09-21T15:50:29.349 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:29.350 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:29.350 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:29.350 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:29.350 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138844+0000", 2024-09-21T15:50:29.350 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:29.350 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:29.350 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:29.350 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:29.350 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138844+0000", 2024-09-21T15:50:29.350 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:29.350 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:29.350 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:29.350 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:29.351 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138844+0000", 2024-09-21T15:50:29.351 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:29.351 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:29.351 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:29.351 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:29.351 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138848+0000", 2024-09-21T15:50:29.351 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.76e-06 2024-09-21T15:50:29.351 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:29.351 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:29.351 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:29.351 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138854+0000", 2024-09-21T15:50:29.351 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3890000000000002e-06 2024-09-21T15:50:29.351 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:29.351 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:29.351 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:29.352 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138860+0000", 2024-09-21T15:50:29.352 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6409999999999999e-06 2024-09-21T15:50:29.352 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:29.352 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:29.352 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:29.352 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.141015+0000", 2024-09-21T15:50:29.352 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0021553779999999999 2024-09-21T15:50:29.352 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:29.352 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:29.352 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:50:29.352 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.141021+0000", 2024-09-21T15:50:29.352 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7629999999999997e-06 2024-09-21T15:50:29.352 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:29.352 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:29.352 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:29.353 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:29.353 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-21T15:50:29.353 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 2 2024-09-21T15:50:29.353 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:29.354 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-21T15:50:29.395 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:29.394+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:29.408 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:29.408 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:29.419 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:50:29.504 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:50:29.503+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:50:29.567 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:29.567 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:29.576 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:50:29.620 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:29.619+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:29.684 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:50:29.682+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:50:29.689 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:29.689 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:29.698 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:50:29.784 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:50:29.784+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:50:29.840 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:29.841 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:29.849 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:50:29.892 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:29.891+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:29.951 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:29.951 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:29.959 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:50:29.962 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:50:30.006 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:30.004+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:30.080 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-09-21T15:50:30.081 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_max_backfills = '' osd_pg_delete_cost = '' (not observed, change may require restart) osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-09-21T15:50:30.087 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:30.085+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:30.097 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:50:30.097 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:50:30.102 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:30.102 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:30.111 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:50:30.112 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:50:30.112 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:50:30.127 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:50:30.128 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:50:30.142 INFO:teuthology.orchestra.run.smithi089.stderr:reweighted osd.0 to 1 (10000) 2024-09-21T15:50:30.149 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T15:50:30.150 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:50:30.150 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:50:30.161 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-21T15:50:30.170 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:50:30.170 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:50:30.192 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:50:30.193 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:50:30.193 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:30.190+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:30.211 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:50:30.212 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:50:30.220 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:30.220 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:30.220 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:50:30.221 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:50:30.221 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:30.229 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:50:30.289 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:30.287+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:30.333 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:30.334 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:50:30.334 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:50:30.334 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:50:30.334 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.334 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120111 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 321536~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:30.334 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572216+0000", 2024-09-21T15:50:30.334 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 26.759698369999999, 2024-09-21T15:50:30.334 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054657399000000002, 2024-09-21T15:50:30.334 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:30.334 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:30.334 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:30.334 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:30.335 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:30.335 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120111 2024-09-21T15:50:30.335 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.335 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:30.335 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.335 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:30.335 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572216+0000", 2024-09-21T15:50:30.335 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.335 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.335 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.335 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:30.336 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572216+0000", 2024-09-21T15:50:30.336 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.336 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.336 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.336 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:30.336 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572216+0000", 2024-09-21T15:50:30.336 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:30.336 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.336 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.336 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:30.336 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572218+0000", 2024-09-21T15:50:30.337 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1840000000000002e-06 2024-09-21T15:50:30.337 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.337 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.337 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:30.337 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572224+0000", 2024-09-21T15:50:30.337 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7880000000000003e-06 2024-09-21T15:50:30.337 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.337 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.337 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:30.337 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572229+0000", 2024-09-21T15:50:30.337 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8239999999999999e-06 2024-09-21T15:50:30.337 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.337 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.337 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:30.338 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626680+0000", 2024-09-21T15:50:30.338 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054451431000000002 2024-09-21T15:50:30.338 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.338 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.338 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:30.338 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626755+0000", 2024-09-21T15:50:30.338 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4641000000000006e-05 2024-09-21T15:50:30.338 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.338 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.338 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:30.338 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626874+0000", 2024-09-21T15:50:30.338 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00011886 2024-09-21T15:50:30.338 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.339 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:30.339 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.339 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.339 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.339 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120112 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 342016~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:30.339 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572233+0000", 2024-09-21T15:50:30.339 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 26.759682002000002, 2024-09-21T15:50:30.339 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054823997999999999, 2024-09-21T15:50:30.339 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:30.339 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:30.339 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:30.339 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:30.339 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:30.339 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120112 2024-09-21T15:50:30.339 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.340 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:30.340 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.340 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:30.340 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572233+0000", 2024-09-21T15:50:30.340 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.340 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.340 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.340 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:30.340 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572233+0000", 2024-09-21T15:50:30.340 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.340 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.340 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.340 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:30.340 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572232+0000", 2024-09-21T15:50:30.341 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:30.341 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.341 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.341 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:30.341 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572234+0000", 2024-09-21T15:50:30.341 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0660000000000002e-06 2024-09-21T15:50:30.341 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.341 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.341 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:30.341 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572242+0000", 2024-09-21T15:50:30.341 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0430000000000002e-06 2024-09-21T15:50:30.341 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.341 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.341 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:30.342 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572246+0000", 2024-09-21T15:50:30.342 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9080000000000003e-06 2024-09-21T15:50:30.342 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.342 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.342 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:30.342 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626916+0000", 2024-09-21T15:50:30.342 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054669482999999998 2024-09-21T15:50:30.342 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.342 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.342 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:30.342 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626977+0000", 2024-09-21T15:50:30.342 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0597000000000003e-05 2024-09-21T15:50:30.342 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.342 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.342 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:30.343 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627057+0000", 2024-09-21T15:50:30.343 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0258000000000003e-05 2024-09-21T15:50:30.343 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.343 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:30.343 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.343 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.343 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.343 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120113 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 362496~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:30.343 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572253+0000", 2024-09-21T15:50:30.343 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 26.759662017, 2024-09-21T15:50:30.343 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.05503044, 2024-09-21T15:50:30.343 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:30.343 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:30.343 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:30.343 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:30.344 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:30.344 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120113 2024-09-21T15:50:30.344 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.344 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:30.344 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.344 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:30.344 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572253+0000", 2024-09-21T15:50:30.344 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.344 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.344 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.344 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:30.344 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572253+0000", 2024-09-21T15:50:30.344 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.344 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.344 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.345 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:30.345 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572252+0000", 2024-09-21T15:50:30.345 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:30.345 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.345 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.345 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:30.345 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572258+0000", 2024-09-21T15:50:30.345 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6169999999999998e-06 2024-09-21T15:50:30.345 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.345 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.345 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:30.345 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572266+0000", 2024-09-21T15:50:30.345 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9410000000000007e-06 2024-09-21T15:50:30.345 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.345 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.345 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:30.346 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572272+0000", 2024-09-21T15:50:30.346 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.241e-06 2024-09-21T15:50:30.346 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.346 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.346 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:30.346 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627087+0000", 2024-09-21T15:50:30.346 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054814850999999998 2024-09-21T15:50:30.346 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.346 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.346 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:30.346 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627162+0000", 2024-09-21T15:50:30.347 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5012999999999995e-05 2024-09-21T15:50:30.347 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.347 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.347 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:30.347 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627283+0000", 2024-09-21T15:50:30.347 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012113 2024-09-21T15:50:30.347 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.348 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:30.348 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.348 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.348 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.348 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120114 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 382976~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:30.348 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572277+0000", 2024-09-21T15:50:30.348 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 26.759637998999999, 2024-09-21T15:50:30.348 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055184395999999997, 2024-09-21T15:50:30.349 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:30.349 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:30.349 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:30.349 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:30.349 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:30.349 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120114 2024-09-21T15:50:30.349 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.349 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:30.350 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.350 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:30.350 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572277+0000", 2024-09-21T15:50:30.350 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.350 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.350 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.350 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:30.350 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572277+0000", 2024-09-21T15:50:30.351 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.351 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.351 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.351 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:30.351 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572276+0000", 2024-09-21T15:50:30.351 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:30.351 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.352 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.352 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:30.352 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572279+0000", 2024-09-21T15:50:30.352 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.373e-06 2024-09-21T15:50:30.352 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.352 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.352 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:30.353 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572285+0000", 2024-09-21T15:50:30.353 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9359999999999996e-06 2024-09-21T15:50:30.353 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.353 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.353 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:30.353 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572289+0000", 2024-09-21T15:50:30.353 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.634e-06 2024-09-21T15:50:30.353 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.353 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.353 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:30.354 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627311+0000", 2024-09-21T15:50:30.354 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055021211 2024-09-21T15:50:30.354 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.354 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.354 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:30.354 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627360+0000", 2024-09-21T15:50:30.354 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9203999999999998e-05 2024-09-21T15:50:30.354 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.354 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.354 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:30.354 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627461+0000", 2024-09-21T15:50:30.354 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000101469 2024-09-21T15:50:30.355 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.355 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:30.355 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.355 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.355 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.355 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120115 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 399360~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:30.355 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572294+0000", 2024-09-21T15:50:30.355 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 26.759620691999999, 2024-09-21T15:50:30.355 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055400092999999997, 2024-09-21T15:50:30.355 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:30.355 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:30.355 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:30.355 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:30.355 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:30.355 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120115 2024-09-21T15:50:30.356 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.356 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:30.356 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.356 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:30.356 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572294+0000", 2024-09-21T15:50:30.356 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.356 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.356 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.356 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:30.356 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572294+0000", 2024-09-21T15:50:30.356 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.356 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.356 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.356 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:30.356 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572294+0000", 2024-09-21T15:50:30.357 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:30.357 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.357 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.357 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:30.357 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572296+0000", 2024-09-21T15:50:30.357 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1100000000000001e-06 2024-09-21T15:50:30.357 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.357 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.357 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:30.357 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572303+0000", 2024-09-21T15:50:30.357 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8580000000000002e-06 2024-09-21T15:50:30.357 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.357 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.357 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:30.357 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572314+0000", 2024-09-21T15:50:30.357 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.101e-05 2024-09-21T15:50:30.357 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.358 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.358 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:30.358 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627496+0000", 2024-09-21T15:50:30.358 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055182522999999997 2024-09-21T15:50:30.358 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.358 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.358 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:30.358 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627568+0000", 2024-09-21T15:50:30.358 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1569999999999994e-05 2024-09-21T15:50:30.358 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.358 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.358 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:30.358 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627694+0000", 2024-09-21T15:50:30.358 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000126372 2024-09-21T15:50:30.358 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.359 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:30.359 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.359 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.359 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.359 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120116 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 419840~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:30.359 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572318+0000", 2024-09-21T15:50:30.359 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 26.759596501000001, 2024-09-21T15:50:30.359 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055539416000000001, 2024-09-21T15:50:30.359 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:30.359 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:30.359 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:30.359 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:30.359 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:30.359 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120116 2024-09-21T15:50:30.359 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.360 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:30.360 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.360 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:30.360 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572318+0000", 2024-09-21T15:50:30.360 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.360 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.360 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.360 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:30.360 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572318+0000", 2024-09-21T15:50:30.360 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.360 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.360 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.360 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:30.360 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572318+0000", 2024-09-21T15:50:30.360 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:30.361 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.361 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.361 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:30.361 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572322+0000", 2024-09-21T15:50:30.361 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4839999999999998e-06 2024-09-21T15:50:30.361 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.361 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.361 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:30.361 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572329+0000", 2024-09-21T15:50:30.361 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0959999999999997e-06 2024-09-21T15:50:30.361 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.361 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.361 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:30.361 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572333+0000", 2024-09-21T15:50:30.361 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9359999999999998e-06 2024-09-21T15:50:30.362 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.362 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.362 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:30.362 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627726+0000", 2024-09-21T15:50:30.362 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055392829999999997 2024-09-21T15:50:30.362 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.362 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.362 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:30.362 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627779+0000", 2024-09-21T15:50:30.362 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2185999999999998e-05 2024-09-21T15:50:30.362 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.362 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.362 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:30.362 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627858+0000", 2024-09-21T15:50:30.363 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9190999999999994e-05 2024-09-21T15:50:30.363 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.363 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:30.363 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.363 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.363 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.363 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120117 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 440320~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:30.363 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572338+0000", 2024-09-21T15:50:30.363 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 26.759577188000002, 2024-09-21T15:50:30.363 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055680888999999997, 2024-09-21T15:50:30.363 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:30.363 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:30.363 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:30.363 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:30.363 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:30.364 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120117 2024-09-21T15:50:30.364 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.364 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:30.364 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.364 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:30.364 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572338+0000", 2024-09-21T15:50:30.364 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.364 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.364 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.364 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:30.364 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572338+0000", 2024-09-21T15:50:30.364 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.364 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.364 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.364 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:30.365 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572337+0000", 2024-09-21T15:50:30.365 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:30.365 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.365 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.365 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:30.365 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572339+0000", 2024-09-21T15:50:30.365 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9379999999999999e-06 2024-09-21T15:50:30.365 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.365 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.365 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:30.365 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572349+0000", 2024-09-21T15:50:30.365 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7780000000000002e-06 2024-09-21T15:50:30.365 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.365 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.365 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:30.366 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572354+0000", 2024-09-21T15:50:30.366 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8740000000000003e-06 2024-09-21T15:50:30.366 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.366 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.366 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:30.366 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627885+0000", 2024-09-21T15:50:30.366 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055531448999999997 2024-09-21T15:50:30.366 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.366 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.366 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:30.366 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627935+0000", 2024-09-21T15:50:30.366 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9348999999999998e-05 2024-09-21T15:50:30.366 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.367 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.367 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:30.367 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628018+0000", 2024-09-21T15:50:30.367 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3851999999999996e-05 2024-09-21T15:50:30.367 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.367 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:30.367 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.367 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.367 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.367 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120118 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 460800~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:30.367 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572359+0000", 2024-09-21T15:50:30.367 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 26.759555945999999, 2024-09-21T15:50:30.367 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055951714, 2024-09-21T15:50:30.367 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:30.367 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:30.367 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:30.368 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:30.368 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:30.368 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120118 2024-09-21T15:50:30.368 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.368 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:30.368 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.368 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:30.368 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572359+0000", 2024-09-21T15:50:30.368 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.368 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.368 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.368 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:30.368 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572359+0000", 2024-09-21T15:50:30.368 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.369 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.369 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.369 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:30.369 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572358+0000", 2024-09-21T15:50:30.369 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:30.369 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.369 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.369 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:30.369 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572361+0000", 2024-09-21T15:50:30.369 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.26e-06 2024-09-21T15:50:30.369 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.369 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.369 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:30.369 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572367+0000", 2024-09-21T15:50:30.369 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3640000000000004e-06 2024-09-21T15:50:30.370 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.370 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.370 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:30.370 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572376+0000", 2024-09-21T15:50:30.370 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2310000000000002e-06 2024-09-21T15:50:30.370 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.370 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.370 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:30.370 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628096+0000", 2024-09-21T15:50:30.370 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055719369999999997 2024-09-21T15:50:30.370 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.370 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.370 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:30.371 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628178+0000", 2024-09-21T15:50:30.371 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2667e-05 2024-09-21T15:50:30.371 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.371 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.371 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:30.371 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628311+0000", 2024-09-21T15:50:30.371 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000132175 2024-09-21T15:50:30.371 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.371 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:30.371 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.371 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.371 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.371 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120119 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 481280~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:30.371 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572381+0000", 2024-09-21T15:50:30.371 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 26.759533502, 2024-09-21T15:50:30.372 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056088791999999998, 2024-09-21T15:50:30.372 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:30.372 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:30.372 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:30.372 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:30.372 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:30.372 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120119 2024-09-21T15:50:30.372 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.372 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:30.372 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.372 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:30.372 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572381+0000", 2024-09-21T15:50:30.372 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.372 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.372 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.372 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:30.373 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572381+0000", 2024-09-21T15:50:30.373 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.373 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.373 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.373 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:30.373 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572381+0000", 2024-09-21T15:50:30.373 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:30.373 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.373 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.373 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:30.373 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572383+0000", 2024-09-21T15:50:30.373 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0619999999999999e-06 2024-09-21T15:50:30.373 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.373 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.373 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:30.373 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572391+0000", 2024-09-21T15:50:30.374 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8769999999999993e-06 2024-09-21T15:50:30.374 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.374 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.374 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:30.374 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572399+0000", 2024-09-21T15:50:30.374 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6240000000000008e-06 2024-09-21T15:50:30.374 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.374 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.374 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:30.374 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628344+0000", 2024-09-21T15:50:30.374 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055944828000000002 2024-09-21T15:50:30.374 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.374 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.374 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:30.374 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628393+0000", 2024-09-21T15:50:30.375 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8653000000000003e-05 2024-09-21T15:50:30.375 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.375 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.375 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:30.375 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628470+0000", 2024-09-21T15:50:30.375 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7121999999999998e-05 2024-09-21T15:50:30.375 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.375 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:30.375 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.375 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.375 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.376 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120120 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 501760~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:30.376 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572418+0000", 2024-09-21T15:50:30.376 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 26.759497004, 2024-09-21T15:50:30.376 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056209138999999998, 2024-09-21T15:50:30.376 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:30.376 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:30.376 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:30.376 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:30.376 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:30.376 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120120 2024-09-21T15:50:30.376 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.376 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:30.376 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.377 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:30.377 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572418+0000", 2024-09-21T15:50:30.377 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.377 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.377 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.377 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:30.377 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572418+0000", 2024-09-21T15:50:30.377 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.377 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.377 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.377 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:30.377 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572417+0000", 2024-09-21T15:50:30.378 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:30.378 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.378 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.378 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:30.378 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572420+0000", 2024-09-21T15:50:30.378 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5270000000000001e-06 2024-09-21T15:50:30.378 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.378 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.378 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:30.378 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572426+0000", 2024-09-21T15:50:30.378 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3380000000000001e-06 2024-09-21T15:50:30.378 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.378 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.378 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:30.379 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572431+0000", 2024-09-21T15:50:30.379 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.865e-06 2024-09-21T15:50:30.379 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.379 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.379 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:30.379 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628500+0000", 2024-09-21T15:50:30.379 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056068917000000003 2024-09-21T15:50:30.379 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.379 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.379 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:30.379 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628542+0000", 2024-09-21T15:50:30.379 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1761999999999998e-05 2024-09-21T15:50:30.379 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.380 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.380 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:30.380 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628627+0000", 2024-09-21T15:50:30.380 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5247999999999994e-05 2024-09-21T15:50:30.380 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.380 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:30.380 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.380 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.380 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.380 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120121 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 522240~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:30.380 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572630+0000", 2024-09-21T15:50:30.380 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 26.759284617999999, 2024-09-21T15:50:30.380 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056207209000000001, 2024-09-21T15:50:30.380 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:30.381 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:30.381 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:30.381 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:30.381 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:30.381 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120121 2024-09-21T15:50:30.381 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.381 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:30.381 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.381 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:30.381 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572630+0000", 2024-09-21T15:50:30.381 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.381 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.381 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.381 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:30.381 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572630+0000", 2024-09-21T15:50:30.382 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.382 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.382 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.382 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:30.382 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572435+0000", 2024-09-21T15:50:30.382 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998045 2024-09-21T15:50:30.382 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.382 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.382 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:30.382 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572633+0000", 2024-09-21T15:50:30.382 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000198786 2024-09-21T15:50:30.382 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.383 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.383 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:30.383 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572641+0000", 2024-09-21T15:50:30.383 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9449999999999993e-06 2024-09-21T15:50:30.383 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.383 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.383 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:30.383 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572648+0000", 2024-09-21T15:50:30.383 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1929999999999998e-06 2024-09-21T15:50:30.383 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.383 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.383 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:30.383 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628669+0000", 2024-09-21T15:50:30.384 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056021690999999998 2024-09-21T15:50:30.384 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.384 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.384 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:30.384 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628751+0000", 2024-09-21T15:50:30.384 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1710000000000002e-05 2024-09-21T15:50:30.384 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.384 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.384 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:30.384 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628837+0000", 2024-09-21T15:50:30.384 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6383000000000003e-05 2024-09-21T15:50:30.384 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.384 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:30.384 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.385 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.385 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.385 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120122 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 538624~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:30.385 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.573471+0000", 2024-09-21T15:50:30.385 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 26.758443372999999, 2024-09-21T15:50:30.385 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055563935000000002, 2024-09-21T15:50:30.385 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:30.385 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:30.385 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:30.385 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:30.385 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:30.385 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120122 2024-09-21T15:50:30.385 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.385 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:30.386 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.386 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:30.386 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573471+0000", 2024-09-21T15:50:30.386 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.386 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.386 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.386 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:30.386 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573471+0000", 2024-09-21T15:50:30.386 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.386 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.386 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.386 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:30.386 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572653+0000", 2024-09-21T15:50:30.386 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9991813 2024-09-21T15:50:30.387 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.387 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.387 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:30.387 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573475+0000", 2024-09-21T15:50:30.387 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00082196500000000002 2024-09-21T15:50:30.387 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.387 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.387 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:30.387 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573483+0000", 2024-09-21T15:50:30.387 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5939999999999994e-06 2024-09-21T15:50:30.387 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.387 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.387 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:30.387 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573489+0000", 2024-09-21T15:50:30.388 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1689999999999997e-06 2024-09-21T15:50:30.388 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.388 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.388 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:30.388 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628882+0000", 2024-09-21T15:50:30.388 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055392230000000001 2024-09-21T15:50:30.388 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.388 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.388 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:30.388 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628934+0000", 2024-09-21T15:50:30.388 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1870000000000003e-05 2024-09-21T15:50:30.388 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.389 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.389 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:30.389 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629035+0000", 2024-09-21T15:50:30.389 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000101753 2024-09-21T15:50:30.389 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.389 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:30.389 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.389 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.389 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.389 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120123 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 559104~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:30.389 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.573495+0000", 2024-09-21T15:50:30.389 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 26.758419673999999, 2024-09-21T15:50:30.389 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055704765000000003, 2024-09-21T15:50:30.389 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:30.389 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:30.390 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:30.390 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:30.390 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:30.390 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120123 2024-09-21T15:50:30.390 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.390 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:30.390 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.390 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:30.390 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573495+0000", 2024-09-21T15:50:30.390 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.390 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.390 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.390 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:30.390 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573495+0000", 2024-09-21T15:50:30.390 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.390 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.391 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.391 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:30.391 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573495+0000", 2024-09-21T15:50:30.391 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:30.391 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.391 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.391 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:30.391 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573497+0000", 2024-09-21T15:50:30.391 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3420000000000002e-06 2024-09-21T15:50:30.391 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.391 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.391 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:30.391 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573503+0000", 2024-09-21T15:50:30.391 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7030000000000003e-06 2024-09-21T15:50:30.392 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.392 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.392 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:30.392 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573511+0000", 2024-09-21T15:50:30.392 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0790000000000001e-06 2024-09-21T15:50:30.392 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.392 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.392 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:30.392 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629067+0000", 2024-09-21T15:50:30.392 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055556573999999997 2024-09-21T15:50:30.392 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.392 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.392 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:30.392 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629118+0000", 2024-09-21T15:50:30.392 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1053000000000001e-05 2024-09-21T15:50:30.392 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.393 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.393 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:30.393 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629200+0000", 2024-09-21T15:50:30.393 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1452999999999994e-05 2024-09-21T15:50:30.393 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.393 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:30.393 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.393 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.393 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.393 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120124 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 579584~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:30.393 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.574014+0000", 2024-09-21T15:50:30.393 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 26.757900634999999, 2024-09-21T15:50:30.393 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055440284999999999, 2024-09-21T15:50:30.393 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:30.394 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:30.394 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:30.394 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:30.394 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:30.394 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120124 2024-09-21T15:50:30.394 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.394 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:30.394 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.394 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:30.394 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574014+0000", 2024-09-21T15:50:30.394 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.394 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.394 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.394 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:30.394 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574014+0000", 2024-09-21T15:50:30.395 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.395 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.395 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.395 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:30.395 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573515+0000", 2024-09-21T15:50:30.395 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9995008 2024-09-21T15:50:30.395 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.395 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.395 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:30.395 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574017+0000", 2024-09-21T15:50:30.395 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00050241100000000003 2024-09-21T15:50:30.395 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.395 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.395 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:30.395 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574027+0000", 2024-09-21T15:50:30.395 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4700000000000008e-06 2024-09-21T15:50:30.396 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.396 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.396 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:30.396 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574033+0000", 2024-09-21T15:50:30.396 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1489999999999999e-06 2024-09-21T15:50:30.396 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.396 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.396 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:30.396 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629238+0000", 2024-09-21T15:50:30.396 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055204880999999997 2024-09-21T15:50:30.396 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.396 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.396 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:30.396 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629317+0000", 2024-09-21T15:50:30.396 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8942000000000004e-05 2024-09-21T15:50:30.396 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.397 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.397 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:30.397 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629454+0000", 2024-09-21T15:50:30.397 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013767099999999999 2024-09-21T15:50:30.397 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.397 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:30.397 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.397 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.397 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.397 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120125 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 600064~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:30.397 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.574482+0000", 2024-09-21T15:50:30.397 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 26.757432336000001, 2024-09-21T15:50:30.397 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055243608, 2024-09-21T15:50:30.397 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:30.398 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:30.398 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:30.398 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:30.398 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:30.398 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120125 2024-09-21T15:50:30.398 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.398 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:30.398 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.398 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:30.398 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574482+0000", 2024-09-21T15:50:30.398 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.398 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.398 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.398 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:30.399 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574482+0000", 2024-09-21T15:50:30.399 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.399 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.399 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.399 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:30.399 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574038+0000", 2024-09-21T15:50:30.399 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9995556 2024-09-21T15:50:30.399 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.399 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.399 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:30.399 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574497+0000", 2024-09-21T15:50:30.399 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00045854400000000002 2024-09-21T15:50:30.399 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.399 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.399 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:30.400 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574509+0000", 2024-09-21T15:50:30.400 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2486e-05 2024-09-21T15:50:30.400 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.400 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.400 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:30.400 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574516+0000", 2024-09-21T15:50:30.400 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2269999999999996e-06 2024-09-21T15:50:30.400 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.400 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.400 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:30.400 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629497+0000", 2024-09-21T15:50:30.400 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054980785999999997 2024-09-21T15:50:30.400 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.400 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.400 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:30.401 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629580+0000", 2024-09-21T15:50:30.401 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2528000000000007e-05 2024-09-21T15:50:30.401 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.401 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.401 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:30.401 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629726+0000", 2024-09-21T15:50:30.401 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014630399999999999 2024-09-21T15:50:30.401 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.401 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:30.401 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.401 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.401 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.401 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120126 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 620544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:30.401 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.575315+0000", 2024-09-21T15:50:30.401 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 26.756599305999998, 2024-09-21T15:50:30.402 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054709559999999997, 2024-09-21T15:50:30.402 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:30.402 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:30.402 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:30.402 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:30.402 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:30.402 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120126 2024-09-21T15:50:30.402 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.402 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:30.402 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.402 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:30.402 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575315+0000", 2024-09-21T15:50:30.402 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.402 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.402 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.403 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:30.403 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575315+0000", 2024-09-21T15:50:30.403 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.403 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.403 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.403 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:30.403 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574522+0000", 2024-09-21T15:50:30.403 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9992065 2024-09-21T15:50:30.403 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.403 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.403 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:30.403 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575319+0000", 2024-09-21T15:50:30.403 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00079626800000000004 2024-09-21T15:50:30.403 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.403 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.403 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:30.404 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575329+0000", 2024-09-21T15:50:30.404 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.028e-05 2024-09-21T15:50:30.404 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.404 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.404 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:30.404 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575335+0000", 2024-09-21T15:50:30.404 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9290000000000003e-06 2024-09-21T15:50:30.404 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.404 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.404 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:30.404 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629806+0000", 2024-09-21T15:50:30.404 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054471558000000003 2024-09-21T15:50:30.404 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.404 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.404 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:30.404 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629885+0000", 2024-09-21T15:50:30.405 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8563e-05 2024-09-21T15:50:30.405 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.405 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.405 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:30.405 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630025+0000", 2024-09-21T15:50:30.405 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014019 2024-09-21T15:50:30.405 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.405 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:30.405 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.405 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.405 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.405 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120127 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 641024~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:30.405 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.575340+0000", 2024-09-21T15:50:30.405 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 26.756574599, 2024-09-21T15:50:30.406 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054895589000000002, 2024-09-21T15:50:30.406 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:30.406 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:30.406 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:30.406 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:30.406 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:30.406 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120127 2024-09-21T15:50:30.406 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.406 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:30.406 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.406 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:30.406 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575340+0000", 2024-09-21T15:50:30.406 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.406 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.407 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.407 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:30.407 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575340+0000", 2024-09-21T15:50:30.407 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.407 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.407 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.407 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:30.407 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575340+0000", 2024-09-21T15:50:30.407 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:30.407 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.407 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.407 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:30.407 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575342+0000", 2024-09-21T15:50:30.407 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5600000000000001e-06 2024-09-21T15:50:30.407 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.407 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.408 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:30.408 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575351+0000", 2024-09-21T15:50:30.408 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2349999999999999e-06 2024-09-21T15:50:30.408 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.408 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.408 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:30.408 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575356+0000", 2024-09-21T15:50:30.408 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4600000000000002e-06 2024-09-21T15:50:30.408 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.408 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.408 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:30.408 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630053+0000", 2024-09-21T15:50:30.408 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054697190999999999 2024-09-21T15:50:30.408 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.409 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.409 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:30.409 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630128+0000", 2024-09-21T15:50:30.409 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4777000000000006e-05 2024-09-21T15:50:30.409 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.409 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.409 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:30.409 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630236+0000", 2024-09-21T15:50:30.409 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000107837 2024-09-21T15:50:30.409 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.409 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:30.409 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.409 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.409 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.409 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120128 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 657408~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:30.410 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.575461+0000", 2024-09-21T15:50:30.410 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 26.75645402, 2024-09-21T15:50:30.410 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054945121999999999, 2024-09-21T15:50:30.410 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:30.410 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:30.410 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:30.410 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:30.410 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:30.410 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120128 2024-09-21T15:50:30.410 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.410 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:30.410 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.410 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:30.410 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575461+0000", 2024-09-21T15:50:30.411 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.411 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.411 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.411 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:30.411 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575461+0000", 2024-09-21T15:50:30.411 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.411 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.411 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.411 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:30.411 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575360+0000", 2024-09-21T15:50:30.411 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998994 2024-09-21T15:50:30.411 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.411 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.411 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:30.412 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575464+0000", 2024-09-21T15:50:30.412 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000103397 2024-09-21T15:50:30.412 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.412 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.412 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:30.412 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575472+0000", 2024-09-21T15:50:30.412 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3189999999999995e-06 2024-09-21T15:50:30.412 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.412 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.412 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:30.412 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575478+0000", 2024-09-21T15:50:30.412 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2779999999999999e-06 2024-09-21T15:50:30.412 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.412 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.412 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:30.413 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630268+0000", 2024-09-21T15:50:30.413 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054789545000000002 2024-09-21T15:50:30.413 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.413 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.413 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:30.413 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630315+0000", 2024-09-21T15:50:30.413 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7244999999999999e-05 2024-09-21T15:50:30.413 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.413 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.413 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:30.413 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630406+0000", 2024-09-21T15:50:30.413 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0811999999999997e-05 2024-09-21T15:50:30.413 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.413 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:30.413 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.413 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.414 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.414 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120129 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 677888~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:30.414 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.576016+0000", 2024-09-21T15:50:30.414 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 26.755898475999999, 2024-09-21T15:50:30.414 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054602823000000002, 2024-09-21T15:50:30.414 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:30.414 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:30.414 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:30.414 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:30.414 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:30.414 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120129 2024-09-21T15:50:30.414 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.414 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:30.414 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.414 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:30.415 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576016+0000", 2024-09-21T15:50:30.415 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.415 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.415 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.415 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:30.415 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576016+0000", 2024-09-21T15:50:30.415 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.415 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.415 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.415 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:30.415 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575483+0000", 2024-09-21T15:50:30.415 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9994669 2024-09-21T15:50:30.415 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.415 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.415 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:30.416 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576020+0000", 2024-09-21T15:50:30.416 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00053657900000000003 2024-09-21T15:50:30.416 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.416 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.416 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:30.416 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576030+0000", 2024-09-21T15:50:30.416 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0326000000000001e-05 2024-09-21T15:50:30.416 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.416 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.416 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:30.416 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576037+0000", 2024-09-21T15:50:30.416 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6490000000000002e-06 2024-09-21T15:50:30.416 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.416 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.416 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:30.416 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630446+0000", 2024-09-21T15:50:30.417 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054409698999999999 2024-09-21T15:50:30.417 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.417 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.417 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:30.417 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630524+0000", 2024-09-21T15:50:30.417 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7422999999999994e-05 2024-09-21T15:50:30.417 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.417 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.417 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:30.417 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630619+0000", 2024-09-21T15:50:30.417 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5332000000000001e-05 2024-09-21T15:50:30.418 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.418 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:30.418 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.418 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.418 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120130 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 698368~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:30.418 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.576043+0000", 2024-09-21T15:50:30.418 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 26.755872105000002, 2024-09-21T15:50:30.418 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054781798, 2024-09-21T15:50:30.418 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:30.418 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:30.418 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:30.418 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:30.419 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:30.419 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120130 2024-09-21T15:50:30.419 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.419 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:30.419 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.419 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:30.419 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576043+0000", 2024-09-21T15:50:30.419 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.419 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.419 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.419 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:30.419 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576043+0000", 2024-09-21T15:50:30.419 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.419 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.419 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.420 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:30.420 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576042+0000", 2024-09-21T15:50:30.420 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:30.420 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.420 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.420 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:30.420 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576044+0000", 2024-09-21T15:50:30.420 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0999999999999998e-06 2024-09-21T15:50:30.420 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.420 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.420 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:30.420 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576051+0000", 2024-09-21T15:50:30.420 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4779999999999996e-06 2024-09-21T15:50:30.420 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.420 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.420 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:30.421 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576058+0000", 2024-09-21T15:50:30.421 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2180000000000002e-06 2024-09-21T15:50:30.421 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.421 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.421 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:30.421 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630661+0000", 2024-09-21T15:50:30.421 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054603240999999997 2024-09-21T15:50:30.421 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.421 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.421 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:30.421 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630747+0000", 2024-09-21T15:50:30.421 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6046999999999998e-05 2024-09-21T15:50:30.421 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.421 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.421 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:30.422 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630824+0000", 2024-09-21T15:50:30.422 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7119999999999993e-05 2024-09-21T15:50:30.422 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.422 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:30.422 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.422 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.422 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:30.422 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:30.423 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T15:50:30.424 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:30.421+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:30.524 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:50:30.523+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:50:30.525 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:30.525 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:30.526 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:50:30.526 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:30.534 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T15:50:30.625 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:30.624+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:30.635 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:30.636 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:30.636 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:50:30.636 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:50:30.636 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:30.644 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T15:50:30.728 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:30.727+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:30.787 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:30.788 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:50:30.788 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:50:30.788 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:50:30.788 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.788 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63140 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 876544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:30.788 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.133649+0000", 2024-09-21T15:50:30.788 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 53.652342673, 2024-09-21T15:50:30.788 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053950201000000003, 2024-09-21T15:50:30.788 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:30.788 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:30.788 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:30.788 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:30.789 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:30.789 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63140 2024-09-21T15:50:30.789 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.789 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:30.789 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.789 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:30.789 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133649+0000", 2024-09-21T15:50:30.789 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.789 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.789 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.789 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:30.789 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133649+0000", 2024-09-21T15:50:30.790 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.790 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.790 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.790 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:30.790 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133648+0000", 2024-09-21T15:50:30.790 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:30.790 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.790 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.790 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:30.790 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133651+0000", 2024-09-21T15:50:30.790 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1549999999999999e-06 2024-09-21T15:50:30.790 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.790 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.791 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:30.791 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133660+0000", 2024-09-21T15:50:30.791 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3899999999999993e-06 2024-09-21T15:50:30.791 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.791 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.791 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:30.791 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133669+0000", 2024-09-21T15:50:30.791 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3219999999999999e-06 2024-09-21T15:50:30.791 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.791 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.791 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:30.791 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187409+0000", 2024-09-21T15:50:30.791 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053740257 2024-09-21T15:50:30.792 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.792 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.792 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:30.792 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187492+0000", 2024-09-21T15:50:30.792 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2409000000000003e-05 2024-09-21T15:50:30.792 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.792 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.792 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:30.792 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187599+0000", 2024-09-21T15:50:30.792 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000107334 2024-09-21T15:50:30.792 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.792 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:30.792 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.792 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.792 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.793 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63141 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 884736~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:30.793 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.133843+0000", 2024-09-21T15:50:30.793 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 53.652148554, 2024-09-21T15:50:30.793 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053954015000000001, 2024-09-21T15:50:30.793 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:30.793 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:30.793 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:30.793 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:30.793 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:30.793 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63141 2024-09-21T15:50:30.793 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.793 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:30.793 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.793 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:30.794 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133843+0000", 2024-09-21T15:50:30.794 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.794 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.794 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.794 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:30.794 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133843+0000", 2024-09-21T15:50:30.794 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.794 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.794 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.794 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:30.794 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133673+0000", 2024-09-21T15:50:30.794 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998302 2024-09-21T15:50:30.794 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.794 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.795 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:30.795 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133846+0000", 2024-09-21T15:50:30.795 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000172762 2024-09-21T15:50:30.795 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.795 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.795 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:30.795 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133855+0000", 2024-09-21T15:50:30.795 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2940000000000006e-06 2024-09-21T15:50:30.795 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.795 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.795 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:30.795 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133863+0000", 2024-09-21T15:50:30.795 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9740000000000002e-06 2024-09-21T15:50:30.795 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.796 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.796 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:30.796 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187630+0000", 2024-09-21T15:50:30.796 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053767211000000002 2024-09-21T15:50:30.796 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.796 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.796 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:30.796 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187707+0000", 2024-09-21T15:50:30.796 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7446000000000004e-05 2024-09-21T15:50:30.796 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.796 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.796 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:30.796 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187797+0000", 2024-09-21T15:50:30.796 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.9858000000000006e-05 2024-09-21T15:50:30.796 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.797 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:30.797 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.797 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.797 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.797 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63142 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 892928~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:30.797 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134046+0000", 2024-09-21T15:50:30.797 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 53.651945611999999, 2024-09-21T15:50:30.797 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054017849999999999, 2024-09-21T15:50:30.797 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:30.797 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:30.797 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:30.797 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:30.797 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:30.797 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63142 2024-09-21T15:50:30.797 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.798 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:30.798 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.798 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:30.798 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134046+0000", 2024-09-21T15:50:30.798 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.798 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.798 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.798 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:30.798 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134046+0000", 2024-09-21T15:50:30.798 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.798 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.798 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.798 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:30.799 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133867+0000", 2024-09-21T15:50:30.799 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998217 2024-09-21T15:50:30.799 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.799 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.799 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:30.799 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134049+0000", 2024-09-21T15:50:30.799 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000181695 2024-09-21T15:50:30.799 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.799 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.799 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:30.799 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134057+0000", 2024-09-21T15:50:30.799 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.303e-06 2024-09-21T15:50:30.799 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.799 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.799 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:30.800 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134063+0000", 2024-09-21T15:50:30.800 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9959999999999999e-06 2024-09-21T15:50:30.800 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.800 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.800 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:30.800 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187849+0000", 2024-09-21T15:50:30.800 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053785571999999997 2024-09-21T15:50:30.800 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.800 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.800 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:30.800 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187954+0000", 2024-09-21T15:50:30.800 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010524899999999999 2024-09-21T15:50:30.800 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.800 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.800 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:30.801 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188064+0000", 2024-09-21T15:50:30.801 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000109549 2024-09-21T15:50:30.801 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.801 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:30.801 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.801 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.801 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.801 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63143 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 901120~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:30.801 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134239+0000", 2024-09-21T15:50:30.801 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 53.651753018999997, 2024-09-21T15:50:30.801 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054026058000000002, 2024-09-21T15:50:30.801 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:30.801 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:30.801 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:30.801 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:30.802 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:30.802 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63143 2024-09-21T15:50:30.802 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.802 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:30.802 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.802 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:30.802 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134239+0000", 2024-09-21T15:50:30.802 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.802 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.802 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.802 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:30.802 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134239+0000", 2024-09-21T15:50:30.802 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.802 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.803 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.803 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:30.803 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134068+0000", 2024-09-21T15:50:30.803 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998298 2024-09-21T15:50:30.803 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.803 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.803 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:30.803 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134242+0000", 2024-09-21T15:50:30.803 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00017414199999999999 2024-09-21T15:50:30.803 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.803 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.803 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:30.803 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134264+0000", 2024-09-21T15:50:30.803 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1552e-05 2024-09-21T15:50:30.804 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.804 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.804 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:30.804 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134270+0000", 2024-09-21T15:50:30.804 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6939999999999998e-06 2024-09-21T15:50:30.804 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.804 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.804 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:30.804 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188108+0000", 2024-09-21T15:50:30.804 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.05383719 2024-09-21T15:50:30.804 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.804 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.804 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:30.804 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188186+0000", 2024-09-21T15:50:30.804 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8318000000000005e-05 2024-09-21T15:50:30.804 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.804 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.805 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:30.805 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188265+0000", 2024-09-21T15:50:30.805 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8597000000000007e-05 2024-09-21T15:50:30.805 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.805 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:30.805 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.805 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.805 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.805 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63144 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 909312~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:30.805 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134432+0000", 2024-09-21T15:50:30.805 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 53.651559462999998, 2024-09-21T15:50:30.805 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053968953, 2024-09-21T15:50:30.805 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:30.805 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:30.805 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:30.805 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:30.806 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:30.806 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63144 2024-09-21T15:50:30.806 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.806 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:30.806 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.806 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:30.806 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134432+0000", 2024-09-21T15:50:30.806 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.806 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.806 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.806 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:30.806 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134432+0000", 2024-09-21T15:50:30.806 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.806 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.807 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.807 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:30.807 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134276+0000", 2024-09-21T15:50:30.807 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998441 2024-09-21T15:50:30.807 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.807 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.807 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:30.807 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134436+0000", 2024-09-21T15:50:30.807 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000159591 2024-09-21T15:50:30.807 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.807 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.807 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:30.807 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134445+0000", 2024-09-21T15:50:30.807 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6780000000000006e-06 2024-09-21T15:50:30.808 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.808 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.808 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:30.808 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134452+0000", 2024-09-21T15:50:30.808 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6830000000000008e-06 2024-09-21T15:50:30.808 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.808 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.808 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:30.808 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188282+0000", 2024-09-21T15:50:30.808 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053829252000000001 2024-09-21T15:50:30.808 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.808 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.808 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:30.808 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188328+0000", 2024-09-21T15:50:30.808 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6734999999999999e-05 2024-09-21T15:50:30.809 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.809 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.809 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:30.809 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188401+0000", 2024-09-21T15:50:30.809 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2731000000000005e-05 2024-09-21T15:50:30.809 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.809 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:30.809 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.809 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.809 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.809 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63145 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 860160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:30.809 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134736+0000", 2024-09-21T15:50:30.809 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 53.651255677999998, 2024-09-21T15:50:30.809 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053865960999999997, 2024-09-21T15:50:30.810 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:30.810 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:30.810 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:30.810 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:30.810 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:30.810 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63145 2024-09-21T15:50:30.810 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.810 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:30.810 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.810 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:30.810 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134736+0000", 2024-09-21T15:50:30.810 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.810 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.810 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.811 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:30.811 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134736+0000", 2024-09-21T15:50:30.811 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.811 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.811 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.811 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:30.811 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134458+0000", 2024-09-21T15:50:30.811 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999722 2024-09-21T15:50:30.811 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.811 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.811 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:30.811 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134741+0000", 2024-09-21T15:50:30.811 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00028245599999999999 2024-09-21T15:50:30.811 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.811 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.812 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:30.812 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134753+0000", 2024-09-21T15:50:30.812 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2548000000000001e-05 2024-09-21T15:50:30.812 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.812 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.812 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:30.812 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134760+0000", 2024-09-21T15:50:30.812 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4189999999999996e-06 2024-09-21T15:50:30.812 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.812 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.812 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:30.812 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188426+0000", 2024-09-21T15:50:30.812 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053665894999999998 2024-09-21T15:50:30.812 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.812 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.812 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:30.813 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188502+0000", 2024-09-21T15:50:30.813 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5791000000000007e-05 2024-09-21T15:50:30.813 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.813 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.813 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:30.813 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188602+0000", 2024-09-21T15:50:30.813 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9667999999999995e-05 2024-09-21T15:50:30.813 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.813 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:30.813 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.813 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.813 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.813 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63148 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 933888~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:30.813 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.135289+0000", 2024-09-21T15:50:30.813 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 53.650702723000002, 2024-09-21T15:50:30.814 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053948488000000003, 2024-09-21T15:50:30.814 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:30.814 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:30.814 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:30.814 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:30.814 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:30.814 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63148 2024-09-21T15:50:30.814 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.814 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:30.814 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.814 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:30.814 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135289+0000", 2024-09-21T15:50:30.814 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.814 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.815 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.815 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:30.815 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135289+0000", 2024-09-21T15:50:30.815 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.815 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.815 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.815 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:30.815 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135288+0000", 2024-09-21T15:50:30.815 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:30.815 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.815 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.815 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:30.815 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135296+0000", 2024-09-21T15:50:30.815 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5399999999999998e-06 2024-09-21T15:50:30.815 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.816 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.816 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:30.816 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135303+0000", 2024-09-21T15:50:30.816 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0500000000000003e-06 2024-09-21T15:50:30.816 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.816 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.816 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:30.816 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135308+0000", 2024-09-21T15:50:30.816 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9749999999999998e-06 2024-09-21T15:50:30.816 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.816 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.816 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:30.816 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189066+0000", 2024-09-21T15:50:30.816 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053758501 2024-09-21T15:50:30.816 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.817 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.817 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:30.817 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189145+0000", 2024-09-21T15:50:30.817 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9017000000000006e-05 2024-09-21T15:50:30.817 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.817 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.817 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:30.817 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189237+0000", 2024-09-21T15:50:30.817 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1906999999999999e-05 2024-09-21T15:50:30.817 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.817 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:30.817 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.817 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.817 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.817 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63149 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 942080~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:30.818 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.135430+0000", 2024-09-21T15:50:30.818 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 53.650561394, 2024-09-21T15:50:30.818 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054005434999999997, 2024-09-21T15:50:30.818 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:30.818 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:30.818 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:30.818 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:30.818 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:30.818 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63149 2024-09-21T15:50:30.818 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.818 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:30.818 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.818 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:30.818 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135430+0000", 2024-09-21T15:50:30.818 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.819 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.819 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.819 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:30.819 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135430+0000", 2024-09-21T15:50:30.819 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.819 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.819 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.819 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:30.819 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135313+0000", 2024-09-21T15:50:30.819 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998827 2024-09-21T15:50:30.819 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.819 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.819 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:30.819 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135434+0000", 2024-09-21T15:50:30.819 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000120876 2024-09-21T15:50:30.820 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.820 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.820 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:30.820 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135443+0000", 2024-09-21T15:50:30.820 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3340000000000008e-06 2024-09-21T15:50:30.820 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.820 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.820 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:30.820 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135449+0000", 2024-09-21T15:50:30.820 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1580000000000002e-06 2024-09-21T15:50:30.820 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.820 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.820 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:30.820 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189267+0000", 2024-09-21T15:50:30.820 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053817775999999998 2024-09-21T15:50:30.821 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.821 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.821 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:30.821 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189336+0000", 2024-09-21T15:50:30.821 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9485000000000003e-05 2024-09-21T15:50:30.821 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.821 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.821 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:30.821 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189436+0000", 2024-09-21T15:50:30.821 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9277000000000005e-05 2024-09-21T15:50:30.821 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.821 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:30.821 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.821 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.821 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.821 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63150 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 950272~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:30.822 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.135618+0000", 2024-09-21T15:50:30.822 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 53.650373649000002, 2024-09-21T15:50:30.822 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054027393, 2024-09-21T15:50:30.822 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:30.822 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:30.822 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:30.822 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:30.822 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:30.822 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63150 2024-09-21T15:50:30.822 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.822 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:30.822 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.822 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:30.822 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135618+0000", 2024-09-21T15:50:30.822 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.823 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.823 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.823 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:30.823 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135618+0000", 2024-09-21T15:50:30.823 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.823 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.823 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.823 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:30.823 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135454+0000", 2024-09-21T15:50:30.823 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998364 2024-09-21T15:50:30.823 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.823 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.823 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:30.823 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135621+0000", 2024-09-21T15:50:30.823 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016714099999999999 2024-09-21T15:50:30.823 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.824 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.824 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:30.824 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135631+0000", 2024-09-21T15:50:30.824 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5340000000000005e-06 2024-09-21T15:50:30.824 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.824 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.824 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:30.824 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135638+0000", 2024-09-21T15:50:30.824 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9280000000000004e-06 2024-09-21T15:50:30.824 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.824 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.824 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:30.825 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189463+0000", 2024-09-21T15:50:30.825 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053825681 2024-09-21T15:50:30.825 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.825 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.825 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:30.825 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189546+0000", 2024-09-21T15:50:30.825 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2278000000000001e-05 2024-09-21T15:50:30.825 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.825 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.825 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:30.825 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189645+0000", 2024-09-21T15:50:30.825 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9543999999999994e-05 2024-09-21T15:50:30.825 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.825 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:30.825 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.826 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.826 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.826 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63152 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 966656~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:30.826 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.136029+0000", 2024-09-21T15:50:30.826 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 53.649963057000001, 2024-09-21T15:50:30.826 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054024823999999999, 2024-09-21T15:50:30.826 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:30.826 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:30.826 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:30.826 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:30.826 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:30.826 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63152 2024-09-21T15:50:30.826 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.826 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:30.826 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.827 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:30.827 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136029+0000", 2024-09-21T15:50:30.827 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.827 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.827 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.827 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:30.827 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136029+0000", 2024-09-21T15:50:30.827 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.827 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.827 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.827 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:30.827 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136028+0000", 2024-09-21T15:50:30.827 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:30.827 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.827 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.827 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:30.828 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136031+0000", 2024-09-21T15:50:30.828 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.033e-06 2024-09-21T15:50:30.828 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.828 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.828 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:30.828 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136039+0000", 2024-09-21T15:50:30.828 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7200000000000006e-06 2024-09-21T15:50:30.828 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.828 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.828 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:30.828 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136051+0000", 2024-09-21T15:50:30.828 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1987000000000001e-05 2024-09-21T15:50:30.828 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.828 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.828 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:30.829 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189876+0000", 2024-09-21T15:50:30.829 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053825264999999997 2024-09-21T15:50:30.829 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.829 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.829 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:30.829 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189955+0000", 2024-09-21T15:50:30.829 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9296999999999997e-05 2024-09-21T15:50:30.829 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.829 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.829 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:30.829 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190053+0000", 2024-09-21T15:50:30.829 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.8067999999999997e-05 2024-09-21T15:50:30.829 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.829 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:30.830 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.830 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.830 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.830 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63153 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 974848~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:30.830 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.136311+0000", 2024-09-21T15:50:30.830 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 53.649680543999999, 2024-09-21T15:50:30.830 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053933825999999997, 2024-09-21T15:50:30.830 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:30.830 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:30.830 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:30.830 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:30.830 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:30.831 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63153 2024-09-21T15:50:30.831 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.831 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:30.831 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.831 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:30.831 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136311+0000", 2024-09-21T15:50:30.831 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.831 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.831 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.831 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:30.831 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136311+0000", 2024-09-21T15:50:30.831 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.831 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.831 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.831 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:30.831 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136057+0000", 2024-09-21T15:50:30.832 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9997463 2024-09-21T15:50:30.832 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.832 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.832 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:30.832 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136316+0000", 2024-09-21T15:50:30.832 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00025872299999999998 2024-09-21T15:50:30.832 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.832 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.832 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:30.832 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136326+0000", 2024-09-21T15:50:30.832 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0552e-05 2024-09-21T15:50:30.832 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.832 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.832 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:30.832 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136334+0000", 2024-09-21T15:50:30.833 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1409999999999999e-06 2024-09-21T15:50:30.833 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.833 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.833 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:30.833 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190080+0000", 2024-09-21T15:50:30.833 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053745933000000003 2024-09-21T15:50:30.833 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.833 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.833 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:30.833 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190157+0000", 2024-09-21T15:50:30.833 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7412999999999999e-05 2024-09-21T15:50:30.833 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.833 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.833 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:30.833 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190245+0000", 2024-09-21T15:50:30.834 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7904000000000004e-05 2024-09-21T15:50:30.834 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.834 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:30.834 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.834 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.834 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.834 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63154 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 983040~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:30.834 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.136472+0000", 2024-09-21T15:50:30.834 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 53.649519193000003, 2024-09-21T15:50:30.834 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053968448000000002, 2024-09-21T15:50:30.834 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:30.834 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:30.834 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:30.834 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:30.834 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:30.835 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63154 2024-09-21T15:50:30.835 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.835 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:30.835 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.835 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:30.835 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136472+0000", 2024-09-21T15:50:30.835 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.835 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.835 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.835 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:30.835 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136472+0000", 2024-09-21T15:50:30.835 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.835 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.835 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.836 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:30.836 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136341+0000", 2024-09-21T15:50:30.836 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998684 2024-09-21T15:50:30.836 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.836 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.836 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:30.836 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136476+0000", 2024-09-21T15:50:30.836 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013528100000000001 2024-09-21T15:50:30.836 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.836 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.836 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:30.836 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136484+0000", 2024-09-21T15:50:30.836 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4760000000000001e-06 2024-09-21T15:50:30.836 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.836 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.836 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:30.837 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136490+0000", 2024-09-21T15:50:30.837 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8640000000000001e-06 2024-09-21T15:50:30.837 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.837 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.837 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:30.837 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190271+0000", 2024-09-21T15:50:30.837 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053781223000000003 2024-09-21T15:50:30.837 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.837 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.837 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:30.837 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190348+0000", 2024-09-21T15:50:30.837 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7058999999999995e-05 2024-09-21T15:50:30.837 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.837 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.837 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:30.837 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190441+0000", 2024-09-21T15:50:30.837 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2937999999999997e-05 2024-09-21T15:50:30.838 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.838 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:30.838 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.838 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.838 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.838 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63157 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1015808~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:30.838 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137112+0000", 2024-09-21T15:50:30.838 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 53.648879905999998, 2024-09-21T15:50:30.838 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053932345, 2024-09-21T15:50:30.838 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:30.838 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:30.838 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:30.838 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:30.838 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:30.839 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63157 2024-09-21T15:50:30.839 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.839 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:30.839 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.839 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:30.839 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137112+0000", 2024-09-21T15:50:30.839 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.839 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.839 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.839 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:30.839 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137112+0000", 2024-09-21T15:50:30.839 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.839 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.839 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.840 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:30.840 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137072+0000", 2024-09-21T15:50:30.840 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999604 2024-09-21T15:50:30.840 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.840 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.840 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:30.840 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137115+0000", 2024-09-21T15:50:30.840 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2379999999999997e-05 2024-09-21T15:50:30.840 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.840 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.840 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:30.840 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137121+0000", 2024-09-21T15:50:30.840 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6499999999999999e-06 2024-09-21T15:50:30.840 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.840 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.840 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:30.841 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137126+0000", 2024-09-21T15:50:30.841 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0769999999999998e-06 2024-09-21T15:50:30.841 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.841 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.841 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:30.841 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190868+0000", 2024-09-21T15:50:30.841 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053741555000000003 2024-09-21T15:50:30.841 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.841 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.841 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:30.841 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190945+0000", 2024-09-21T15:50:30.841 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7171999999999999e-05 2024-09-21T15:50:30.841 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.841 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.841 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:30.842 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191044+0000", 2024-09-21T15:50:30.842 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9031999999999996e-05 2024-09-21T15:50:30.842 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.842 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:30.842 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.842 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.842 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.842 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63158 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1007616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:30.842 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137293+0000", 2024-09-21T15:50:30.842 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 53.648698619000001, 2024-09-21T15:50:30.842 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054005642, 2024-09-21T15:50:30.842 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:30.842 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:30.842 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:30.842 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:30.842 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:30.843 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63158 2024-09-21T15:50:30.843 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.843 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:30.843 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.843 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:30.843 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137293+0000", 2024-09-21T15:50:30.843 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.843 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.843 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.843 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:30.843 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137293+0000", 2024-09-21T15:50:30.843 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.843 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.843 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.843 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:30.844 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137131+0000", 2024-09-21T15:50:30.844 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998379 2024-09-21T15:50:30.844 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.844 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.844 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:30.844 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137296+0000", 2024-09-21T15:50:30.844 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016568999999999999 2024-09-21T15:50:30.844 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.844 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.844 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:30.844 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137306+0000", 2024-09-21T15:50:30.844 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.6029999999999994e-06 2024-09-21T15:50:30.844 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.844 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.844 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:30.844 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137312+0000", 2024-09-21T15:50:30.845 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3210000000000001e-06 2024-09-21T15:50:30.845 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.845 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.845 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:30.845 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191087+0000", 2024-09-21T15:50:30.845 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053774686000000002 2024-09-21T15:50:30.845 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.845 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.845 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:30.845 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191175+0000", 2024-09-21T15:50:30.845 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8106e-05 2024-09-21T15:50:30.845 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.845 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.845 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:30.845 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191299+0000", 2024-09-21T15:50:30.845 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012343599999999999 2024-09-21T15:50:30.846 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.846 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:30.846 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.846 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.846 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.846 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63159 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1024000~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:30.846 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137568+0000", 2024-09-21T15:50:30.846 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 53.648423346999998, 2024-09-21T15:50:30.846 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053962999999999997, 2024-09-21T15:50:30.846 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:30.846 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:30.846 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:30.846 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:30.846 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:30.846 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63159 2024-09-21T15:50:30.846 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.847 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:30.847 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.847 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:30.847 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137568+0000", 2024-09-21T15:50:30.847 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.847 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.847 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.847 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:30.847 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137568+0000", 2024-09-21T15:50:30.847 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.847 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.847 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.847 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:30.847 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137317+0000", 2024-09-21T15:50:30.847 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9997492 2024-09-21T15:50:30.847 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.848 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.848 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:30.848 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137572+0000", 2024-09-21T15:50:30.848 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00025491499999999998 2024-09-21T15:50:30.848 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.848 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.848 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:30.848 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137582+0000", 2024-09-21T15:50:30.848 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4490000000000005e-06 2024-09-21T15:50:30.848 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.848 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.848 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:30.848 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137589+0000", 2024-09-21T15:50:30.848 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5580000000000003e-06 2024-09-21T15:50:30.848 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.848 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.848 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:30.849 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191344+0000", 2024-09-21T15:50:30.849 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053754272999999998 2024-09-21T15:50:30.849 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.849 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.849 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:30.849 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191429+0000", 2024-09-21T15:50:30.849 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.509e-05 2024-09-21T15:50:30.849 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.849 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.849 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:30.849 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191531+0000", 2024-09-21T15:50:30.849 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000102519 2024-09-21T15:50:30.849 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.849 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:30.849 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.849 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.850 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.850 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63160 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1032192~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:30.850 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137771+0000", 2024-09-21T15:50:30.850 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 53.648220813000002, 2024-09-21T15:50:30.850 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053957818999999997, 2024-09-21T15:50:30.850 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:30.850 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:30.850 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:30.850 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:30.850 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:30.850 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63160 2024-09-21T15:50:30.850 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.850 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:30.850 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.850 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:30.851 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137771+0000", 2024-09-21T15:50:30.851 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.851 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.851 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.851 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:30.851 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137771+0000", 2024-09-21T15:50:30.851 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.851 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.851 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.851 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:30.851 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137596+0000", 2024-09-21T15:50:30.851 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998255 2024-09-21T15:50:30.852 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.852 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.852 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:30.852 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137780+0000", 2024-09-21T15:50:30.852 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00018363900000000001 2024-09-21T15:50:30.852 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.852 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.852 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:30.852 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137791+0000", 2024-09-21T15:50:30.852 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.096e-05 2024-09-21T15:50:30.852 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.852 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.852 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:30.852 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137798+0000", 2024-09-21T15:50:30.852 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1169999999999998e-06 2024-09-21T15:50:30.853 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.853 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.853 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:30.853 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191560+0000", 2024-09-21T15:50:30.853 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053762163000000002 2024-09-21T15:50:30.853 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.853 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.853 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:30.853 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191634+0000", 2024-09-21T15:50:30.853 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4381999999999994e-05 2024-09-21T15:50:30.853 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.853 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.853 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:30.853 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191729+0000", 2024-09-21T15:50:30.853 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4098000000000006e-05 2024-09-21T15:50:30.853 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.854 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:30.854 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.854 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.854 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.854 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63161 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1040384~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:30.854 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.138018+0000", 2024-09-21T15:50:30.854 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 53.647974058999999, 2024-09-21T15:50:30.854 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053905657000000003, 2024-09-21T15:50:30.854 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:30.854 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:30.854 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:30.854 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:30.854 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:30.854 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63161 2024-09-21T15:50:30.854 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.855 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:30.855 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.855 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:30.855 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138018+0000", 2024-09-21T15:50:30.855 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.855 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.855 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.855 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:30.855 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138018+0000", 2024-09-21T15:50:30.855 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.855 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.855 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.855 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:30.855 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137804+0000", 2024-09-21T15:50:30.855 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9997859 2024-09-21T15:50:30.855 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.856 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.856 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:30.856 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138021+0000", 2024-09-21T15:50:30.856 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00021769899999999999 2024-09-21T15:50:30.856 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.856 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.856 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:30.856 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138043+0000", 2024-09-21T15:50:30.856 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1627000000000001e-05 2024-09-21T15:50:30.856 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.856 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.856 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:30.856 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138054+0000", 2024-09-21T15:50:30.856 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1321999999999999e-05 2024-09-21T15:50:30.857 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.857 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.857 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:30.857 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191755+0000", 2024-09-21T15:50:30.857 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053700354999999998 2024-09-21T15:50:30.857 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.857 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.857 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:30.857 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191830+0000", 2024-09-21T15:50:30.857 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5320000000000004e-05 2024-09-21T15:50:30.857 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.857 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.857 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:30.857 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191923+0000", 2024-09-21T15:50:30.858 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3282999999999994e-05 2024-09-21T15:50:30.858 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.858 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:30.858 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.858 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.858 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.858 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63176 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 108544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:30.858 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.140861+0000", 2024-09-21T15:50:30.858 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 53.645130739999999, 2024-09-21T15:50:30.858 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053935394999999997, 2024-09-21T15:50:30.858 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:30.858 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:30.858 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:30.858 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:30.858 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:30.858 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63176 2024-09-21T15:50:30.859 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.859 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:30.859 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.859 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:30.859 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140861+0000", 2024-09-21T15:50:30.859 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.859 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.859 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.859 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:30.859 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140861+0000", 2024-09-21T15:50:30.859 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.859 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.859 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.859 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:30.859 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140860+0000", 2024-09-21T15:50:30.859 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:30.859 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.860 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.860 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:30.860 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140863+0000", 2024-09-21T15:50:30.860 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8710000000000001e-06 2024-09-21T15:50:30.860 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.860 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.860 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:30.860 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140871+0000", 2024-09-21T15:50:30.860 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8089999999999993e-06 2024-09-21T15:50:30.860 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.860 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.860 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:30.860 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140877+0000", 2024-09-21T15:50:30.860 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5969999999999999e-06 2024-09-21T15:50:30.860 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.860 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.861 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:30.861 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.194608+0000", 2024-09-21T15:50:30.861 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053731749000000002 2024-09-21T15:50:30.861 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.861 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.861 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:30.861 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.194697+0000", 2024-09-21T15:50:30.861 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8504000000000005e-05 2024-09-21T15:50:30.861 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.861 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.861 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:30.861 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.194796+0000", 2024-09-21T15:50:30.861 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9414999999999996e-05 2024-09-21T15:50:30.861 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.861 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:30.861 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.862 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.862 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.862 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63182 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 157696~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:30.862 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:50:30.862 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 53.643858340000001, 2024-09-21T15:50:30.862 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053958825000000002, 2024-09-21T15:50:30.862 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:30.862 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:30.862 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:30.862 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:30.862 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:30.862 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63182 2024-09-21T15:50:30.862 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.862 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:30.862 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.862 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:30.863 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:50:30.863 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.863 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.863 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.863 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:30.863 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:50:30.863 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.863 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.863 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.863 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:30.863 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:50:30.863 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:30.863 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.863 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.863 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:30.864 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142136+0000", 2024-09-21T15:50:30.864 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8169999999999999e-06 2024-09-21T15:50:30.864 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.864 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.864 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:30.864 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:50:30.864 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3752e-05 2024-09-21T15:50:30.864 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.864 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.864 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:30.864 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142155+0000", 2024-09-21T15:50:30.864 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1859999999999997e-06 2024-09-21T15:50:30.864 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.864 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.864 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:30.865 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.195914+0000", 2024-09-21T15:50:30.865 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053758528999999999 2024-09-21T15:50:30.865 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.865 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.865 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:30.865 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.195995+0000", 2024-09-21T15:50:30.865 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0983999999999996e-05 2024-09-21T15:50:30.865 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.865 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.865 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:30.865 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.196092+0000", 2024-09-21T15:50:30.865 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7069000000000002e-05 2024-09-21T15:50:30.865 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.865 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:30.865 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.865 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.866 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.866 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63193 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 239616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:30.866 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:50:30.866 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 53.641770755000003, 2024-09-21T15:50:30.866 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053956577999999998, 2024-09-21T15:50:30.866 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:30.866 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:30.866 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:30.866 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:30.866 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:30.866 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63193 2024-09-21T15:50:30.866 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.867 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:30.867 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.867 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:30.867 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:50:30.867 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.867 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.867 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.867 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:30.867 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:50:30.867 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:30.867 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.867 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.867 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:30.867 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:50:30.867 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:30.868 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.868 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.868 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:30.868 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144223+0000", 2024-09-21T15:50:30.868 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0990000000000001e-06 2024-09-21T15:50:30.868 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.868 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.868 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:30.868 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144230+0000", 2024-09-21T15:50:30.868 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9070000000000001e-06 2024-09-21T15:50:30.868 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.868 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.868 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:30.868 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144234+0000", 2024-09-21T15:50:30.869 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9010000000000002e-06 2024-09-21T15:50:30.869 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.869 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.869 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:30.869 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.198004+0000", 2024-09-21T15:50:30.869 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053769539999999998 2024-09-21T15:50:30.869 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.869 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.869 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:30.869 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.198083+0000", 2024-09-21T15:50:30.869 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9531999999999997e-05 2024-09-21T15:50:30.869 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:30.869 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:30.869 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:30.869 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.198177+0000", 2024-09-21T15:50:30.869 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3906000000000005e-05 2024-09-21T15:50:30.870 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.870 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:30.870 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.870 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:30.870 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:30.870 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:30.871 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T15:50:30.872 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:50:30.870+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:50:30.972 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:30.971+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:31.012 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:31.012 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:31.012 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:50:31.012 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:31.021 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T15:50:31.073 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:50:31.071+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:50:31.122 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:31.122 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:31.122 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:50:31.122 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:50:31.122 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:31.131 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T15:50:31.141 INFO:teuthology.orchestra.run.smithi089.stderr:reweighted osd.1 to 1 (10000) 2024-09-21T15:50:31.144 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is set 2024-09-21T15:50:31.166 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-21T15:50:31.174 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:31.173+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:31.243 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:31.243 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:50:31.243 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:50:31.243 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:50:31.243 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.243 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4209.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:50:31.243 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:50:31.243 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 100.049646482, 2024-09-21T15:50:31.243 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.19452909900000001, 2024-09-21T15:50:31.243 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:31.243 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:31.244 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:31.244 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4209", 2024-09-21T15:50:31.244 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/4045143050", 2024-09-21T15:50:31.244 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1 2024-09-21T15:50:31.244 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.244 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:31.244 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.244 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:31.244 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:50:31.244 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:31.244 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.244 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.244 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:31.244 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:50:31.245 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:31.245 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.245 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.245 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:31.245 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191161+0000", 2024-09-21T15:50:31.245 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-21T15:50:31.245 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.245 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.245 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:31.245 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191169+0000", 2024-09-21T15:50:31.245 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0090000000000007e-06 2024-09-21T15:50:31.245 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.245 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.245 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:31.245 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191180+0000", 2024-09-21T15:50:31.245 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0728e-05 2024-09-21T15:50:31.245 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.246 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.246 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:31.246 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191187+0000", 2024-09-21T15:50:31.246 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9009999999999994e-06 2024-09-21T15:50:31.246 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.246 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.246 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:31.246 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191238+0000", 2024-09-21T15:50:31.246 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0287000000000002e-05 2024-09-21T15:50:31.246 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.246 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.246 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-21T15:50:31.246 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191245+0000", 2024-09-21T15:50:31.246 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0709999999999997e-06 2024-09-21T15:50:31.246 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.247 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.247 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:31.247 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380309+0000", 2024-09-21T15:50:31.247 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.189064277 2024-09-21T15:50:31.247 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.247 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.247 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:31.247 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380454+0000", 2024-09-21T15:50:31.247 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014490799999999999 2024-09-21T15:50:31.247 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.247 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.247 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0", 2024-09-21T15:50:31.247 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380610+0000", 2024-09-21T15:50:31.247 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015650199999999999 2024-09-21T15:50:31.247 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.248 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.248 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:50:31.248 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.381160+0000", 2024-09-21T15:50:31.248 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00054903799999999996 2024-09-21T15:50:31.248 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.248 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.248 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:31.248 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385591+0000", 2024-09-21T15:50:31.248 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.004431272 2024-09-21T15:50:31.248 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.248 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.248 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:50:31.248 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385651+0000", 2024-09-21T15:50:31.248 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0455999999999999e-05 2024-09-21T15:50:31.248 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.248 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.249 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:31.249 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385691+0000", 2024-09-21T15:50:31.249 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0234999999999997e-05 2024-09-21T15:50:31.249 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:31.249 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:31.249 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:31.249 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.249 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.249 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64034 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1083392~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:31.249 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167717+0000", 2024-09-21T15:50:31.249 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 54.073092152000001, 2024-09-21T15:50:31.249 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048233316999999998, 2024-09-21T15:50:31.249 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:31.250 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:31.250 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:31.250 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:31.250 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:31.250 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64034 2024-09-21T15:50:31.250 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.250 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:31.250 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.250 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:31.250 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167717+0000", 2024-09-21T15:50:31.250 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:31.250 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.250 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.250 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:31.250 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167717+0000", 2024-09-21T15:50:31.251 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:31.251 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.251 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.251 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:31.251 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167716+0000", 2024-09-21T15:50:31.251 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:31.251 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.251 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.251 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:31.251 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167718+0000", 2024-09-21T15:50:31.251 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.165e-06 2024-09-21T15:50:31.251 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.251 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.251 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:31.251 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167725+0000", 2024-09-21T15:50:31.252 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3380000000000001e-06 2024-09-21T15:50:31.252 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.252 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.252 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:31.252 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167730+0000", 2024-09-21T15:50:31.252 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8010000000000003e-06 2024-09-21T15:50:31.252 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.252 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.252 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:31.252 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215807+0000", 2024-09-21T15:50:31.252 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048077823999999998 2024-09-21T15:50:31.252 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.252 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.252 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:31.253 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215884+0000", 2024-09-21T15:50:31.253 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6283000000000001e-05 2024-09-21T15:50:31.253 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.253 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.253 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:31.253 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215950+0000", 2024-09-21T15:50:31.253 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6290000000000004e-05 2024-09-21T15:50:31.253 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:31.253 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:31.253 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:31.253 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.253 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.253 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64035 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1091584~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:31.253 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:50:31.253 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 54.073074939000001, 2024-09-21T15:50:31.254 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048412117999999997, 2024-09-21T15:50:31.254 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:31.254 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:31.254 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:31.254 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:31.254 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:31.254 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64035 2024-09-21T15:50:31.254 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.254 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:31.254 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.254 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:31.254 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:50:31.254 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:31.254 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.255 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.255 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:31.255 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:50:31.255 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:31.255 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.255 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.255 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:31.255 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:50:31.255 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:31.255 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.255 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.255 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:31.255 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167746+0000", 2024-09-21T15:50:31.255 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2031000000000001e-05 2024-09-21T15:50:31.255 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.256 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.256 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:31.256 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167753+0000", 2024-09-21T15:50:31.256 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0620000000000003e-06 2024-09-21T15:50:31.256 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.256 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.256 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:31.256 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167759+0000", 2024-09-21T15:50:31.256 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.596e-06 2024-09-21T15:50:31.256 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.256 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.256 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:31.256 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215977+0000", 2024-09-21T15:50:31.256 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048217318000000002 2024-09-21T15:50:31.256 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.256 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.257 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:31.257 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216076+0000", 2024-09-21T15:50:31.257 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9846000000000006e-05 2024-09-21T15:50:31.257 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.257 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.257 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:31.257 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216146+0000", 2024-09-21T15:50:31.257 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9578000000000004e-05 2024-09-21T15:50:31.257 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:31.257 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:31.257 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:31.257 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.257 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.257 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64036 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1099776~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:31.257 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167853+0000", 2024-09-21T15:50:31.257 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 54.072956326000003, 2024-09-21T15:50:31.258 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048450675999999998, 2024-09-21T15:50:31.258 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:31.258 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:31.258 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:31.258 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:31.258 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:31.258 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64036 2024-09-21T15:50:31.258 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.258 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:31.258 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.258 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:31.258 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167853+0000", 2024-09-21T15:50:31.258 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:31.258 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.258 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.259 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:31.259 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167853+0000", 2024-09-21T15:50:31.259 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:31.259 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.259 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.259 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:31.259 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167852+0000", 2024-09-21T15:50:31.259 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:31.259 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.259 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.259 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:31.259 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167855+0000", 2024-09-21T15:50:31.259 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5640000000000001e-06 2024-09-21T15:50:31.259 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.259 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.260 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:31.260 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167864+0000", 2024-09-21T15:50:31.260 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8140000000000007e-06 2024-09-21T15:50:31.260 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.260 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.260 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:31.260 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167871+0000", 2024-09-21T15:50:31.260 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5729999999999996e-06 2024-09-21T15:50:31.260 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.260 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.260 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:31.260 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216174+0000", 2024-09-21T15:50:31.260 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048303437999999997 2024-09-21T15:50:31.260 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.260 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.261 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:31.261 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216250+0000", 2024-09-21T15:50:31.261 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5661999999999995e-05 2024-09-21T15:50:31.261 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.261 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.261 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:31.261 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216303+0000", 2024-09-21T15:50:31.261 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3273000000000003e-05 2024-09-21T15:50:31.261 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:31.261 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:31.261 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:31.261 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.261 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.261 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64037 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1771520~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:31.262 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:50:31.262 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 54.072932782999999, 2024-09-21T15:50:31.262 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048588771000000003, 2024-09-21T15:50:31.262 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:31.262 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:31.262 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:31.262 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:31.262 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:31.262 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64037 2024-09-21T15:50:31.262 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.262 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:31.262 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.262 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:31.262 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:50:31.262 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:31.262 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.263 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.263 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:31.263 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:50:31.263 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:31.263 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.263 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.263 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:31.263 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:50:31.263 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:31.263 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.263 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.263 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:31.263 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167878+0000", 2024-09-21T15:50:31.263 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3e-06 2024-09-21T15:50:31.263 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.264 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.264 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:31.264 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167884+0000", 2024-09-21T15:50:31.264 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3269999999999997e-06 2024-09-21T15:50:31.264 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.264 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.264 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:31.264 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167889+0000", 2024-09-21T15:50:31.264 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9269999999999996e-06 2024-09-21T15:50:31.264 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.264 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.264 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:31.264 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216331+0000", 2024-09-21T15:50:31.264 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048442102000000001 2024-09-21T15:50:31.264 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.265 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.265 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:31.265 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216393+0000", 2024-09-21T15:50:31.265 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1132999999999999e-05 2024-09-21T15:50:31.265 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.265 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.265 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:31.265 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216465+0000", 2024-09-21T15:50:31.265 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2339e-05 2024-09-21T15:50:31.265 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:31.265 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:31.265 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:31.265 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.265 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.265 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64038 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1779712~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:31.265 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:50:31.266 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 54.072914951999998, 2024-09-21T15:50:31.266 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048749795999999998, 2024-09-21T15:50:31.266 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:31.266 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:31.266 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:31.266 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:31.266 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:31.266 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64038 2024-09-21T15:50:31.266 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.266 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:31.266 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.266 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:31.266 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:50:31.266 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:31.266 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.267 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.267 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:31.267 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:50:31.267 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:31.267 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.267 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.267 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:31.267 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:50:31.267 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:31.267 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.267 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.267 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:31.267 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167896+0000", 2024-09-21T15:50:31.268 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2419999999999999e-06 2024-09-21T15:50:31.268 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.268 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.268 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:31.268 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167902+0000", 2024-09-21T15:50:31.268 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0859999999999998e-06 2024-09-21T15:50:31.268 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.268 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.268 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:31.268 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167907+0000", 2024-09-21T15:50:31.268 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9679999999999997e-06 2024-09-21T15:50:31.268 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.268 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.268 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:31.268 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216506+0000", 2024-09-21T15:50:31.269 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048599374000000001 2024-09-21T15:50:31.269 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.269 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.269 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:31.269 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216579+0000", 2024-09-21T15:50:31.269 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2801999999999999e-05 2024-09-21T15:50:31.269 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.269 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.269 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:31.269 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216644+0000", 2024-09-21T15:50:31.269 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4715000000000006e-05 2024-09-21T15:50:31.269 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:31.269 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:31.269 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:31.269 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.269 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.270 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64039 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1107968~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:31.270 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:50:31.270 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 54.072897597000001, 2024-09-21T15:50:31.270 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048911370000000003, 2024-09-21T15:50:31.270 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:31.270 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:31.270 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:31.270 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:31.270 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:31.270 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64039 2024-09-21T15:50:31.270 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.270 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:31.270 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.270 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:31.270 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:50:31.271 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:31.271 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.271 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.271 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:31.271 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:50:31.271 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:31.271 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.271 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.271 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:31.271 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:50:31.271 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:31.271 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.272 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.272 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:31.272 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167913+0000", 2024-09-21T15:50:31.272 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0839999999999999e-06 2024-09-21T15:50:31.272 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.272 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.272 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:31.272 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167921+0000", 2024-09-21T15:50:31.272 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8310000000000001e-06 2024-09-21T15:50:31.272 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.272 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.272 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:31.272 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167926+0000", 2024-09-21T15:50:31.272 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0270000000000003e-06 2024-09-21T15:50:31.272 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.273 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.273 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:31.273 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216680+0000", 2024-09-21T15:50:31.273 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048753890000000001 2024-09-21T15:50:31.273 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.273 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.273 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:31.273 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216757+0000", 2024-09-21T15:50:31.273 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6848999999999996e-05 2024-09-21T15:50:31.273 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.273 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.273 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:31.273 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216823+0000", 2024-09-21T15:50:31.273 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6068000000000005e-05 2024-09-21T15:50:31.274 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:31.274 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:31.274 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:31.274 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.274 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.274 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64040 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1787904~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:31.274 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167931+0000", 2024-09-21T15:50:31.274 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 54.072878365000001, 2024-09-21T15:50:31.274 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049010561000000001, 2024-09-21T15:50:31.274 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:31.274 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:31.274 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:31.274 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:31.274 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:31.274 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64040 2024-09-21T15:50:31.275 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.275 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:31.275 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.275 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:31.275 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167931+0000", 2024-09-21T15:50:31.275 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:31.275 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.275 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.275 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:31.275 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167931+0000", 2024-09-21T15:50:31.275 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:31.275 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.275 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.275 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:31.275 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167930+0000", 2024-09-21T15:50:31.276 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:31.276 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.276 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.276 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:31.276 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167932+0000", 2024-09-21T15:50:31.276 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.198e-06 2024-09-21T15:50:31.276 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.276 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.276 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:31.276 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167938+0000", 2024-09-21T15:50:31.276 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.105e-06 2024-09-21T15:50:31.276 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.276 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.276 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:31.276 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167943+0000", 2024-09-21T15:50:31.276 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9429999999999999e-06 2024-09-21T15:50:31.277 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.277 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.277 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:31.277 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216848+0000", 2024-09-21T15:50:31.277 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048905200000000003 2024-09-21T15:50:31.277 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.277 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.277 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:31.277 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216903+0000", 2024-09-21T15:50:31.277 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4793999999999997e-05 2024-09-21T15:50:31.277 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.277 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.277 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:31.277 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216941+0000", 2024-09-21T15:50:31.277 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7790999999999998e-05 2024-09-21T15:50:31.277 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:31.278 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:31.278 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:31.278 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.278 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.278 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64041 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1796096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:31.278 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:50:31.278 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 54.072861394, 2024-09-21T15:50:31.278 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049181457999999997, 2024-09-21T15:50:31.278 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:31.278 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:31.278 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:31.278 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:31.278 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:31.278 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64041 2024-09-21T15:50:31.278 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.279 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:31.279 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.279 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:31.279 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:50:31.279 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:31.279 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.279 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.279 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:31.279 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:50:31.279 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:31.279 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.279 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.279 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:31.279 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:50:31.280 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:31.280 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.280 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.280 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:31.280 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167949+0000", 2024-09-21T15:50:31.280 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0200000000000001e-06 2024-09-21T15:50:31.280 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.280 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.280 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:31.280 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167955+0000", 2024-09-21T15:50:31.280 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5990000000000001e-06 2024-09-21T15:50:31.280 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.280 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.280 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:31.280 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167959+0000", 2024-09-21T15:50:31.281 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5589999999999999e-06 2024-09-21T15:50:31.281 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.281 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.281 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:31.281 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216972+0000", 2024-09-21T15:50:31.281 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049012574000000003 2024-09-21T15:50:31.281 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.281 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.281 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:31.281 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217057+0000", 2024-09-21T15:50:31.281 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4820000000000004e-05 2024-09-21T15:50:31.281 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.281 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.282 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:31.282 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217129+0000", 2024-09-21T15:50:31.282 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2186000000000003e-05 2024-09-21T15:50:31.282 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:31.282 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:31.282 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:31.282 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.282 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.282 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64042 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1804288~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:31.282 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167964+0000", 2024-09-21T15:50:31.282 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 54.072845102999999, 2024-09-21T15:50:31.282 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049333372, 2024-09-21T15:50:31.282 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:31.282 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:31.283 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:31.283 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:31.283 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:31.283 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64042 2024-09-21T15:50:31.283 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.283 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:31.283 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.283 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:31.283 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167964+0000", 2024-09-21T15:50:31.283 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:31.283 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.283 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.283 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:31.284 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167964+0000", 2024-09-21T15:50:31.284 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:31.284 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.284 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.284 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:31.284 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167963+0000", 2024-09-21T15:50:31.284 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:31.284 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.284 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.284 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:31.284 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167966+0000", 2024-09-21T15:50:31.284 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1119999999999999e-06 2024-09-21T15:50:31.284 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.284 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.284 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:31.285 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167975+0000", 2024-09-21T15:50:31.285 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1130000000000007e-06 2024-09-21T15:50:31.285 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.285 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.285 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:31.285 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167980+0000", 2024-09-21T15:50:31.285 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9599999999999999e-06 2024-09-21T15:50:31.285 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.285 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.285 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:31.285 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217158+0000", 2024-09-21T15:50:31.285 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049178569999999998 2024-09-21T15:50:31.285 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.285 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.286 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:31.286 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217232+0000", 2024-09-21T15:50:31.286 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3589000000000003e-05 2024-09-21T15:50:31.286 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.286 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.286 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:31.286 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217297+0000", 2024-09-21T15:50:31.286 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5339999999999994e-05 2024-09-21T15:50:31.286 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:31.286 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:31.286 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:31.286 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.286 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.286 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64043 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1812480~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:31.286 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.168085+0000", 2024-09-21T15:50:31.286 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 54.072724035999997, 2024-09-21T15:50:31.287 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049365157999999999, 2024-09-21T15:50:31.287 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:31.287 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:31.287 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:31.287 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:31.287 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:31.287 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64043 2024-09-21T15:50:31.287 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.287 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:31.287 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.287 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:31.287 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168085+0000", 2024-09-21T15:50:31.287 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:31.287 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.287 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.287 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:31.288 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168085+0000", 2024-09-21T15:50:31.288 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:31.288 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.288 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.288 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:31.288 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168084+0000", 2024-09-21T15:50:31.288 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:31.288 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.288 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.288 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:31.288 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168088+0000", 2024-09-21T15:50:31.288 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5420000000000001e-06 2024-09-21T15:50:31.288 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.288 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.288 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:31.288 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168097+0000", 2024-09-21T15:50:31.289 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5410000000000006e-06 2024-09-21T15:50:31.289 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.289 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.289 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:31.289 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168104+0000", 2024-09-21T15:50:31.289 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6719999999999998e-06 2024-09-21T15:50:31.289 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.289 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.289 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:31.289 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217323+0000", 2024-09-21T15:50:31.289 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049218857999999997 2024-09-21T15:50:31.289 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.290 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.290 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:31.290 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217397+0000", 2024-09-21T15:50:31.290 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3932999999999998e-05 2024-09-21T15:50:31.290 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.290 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.290 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:31.290 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217450+0000", 2024-09-21T15:50:31.290 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3168000000000003e-05 2024-09-21T15:50:31.290 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:31.290 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:31.290 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:31.290 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.291 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.291 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155255 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 323584~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:31.291 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.440996+0000", 2024-09-21T15:50:31.291 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 8.7998125090000006, 2024-09-21T15:50:31.291 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048233848000000003, 2024-09-21T15:50:31.291 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:31.291 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:31.291 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:31.291 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:31.291 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:31.291 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155255 2024-09-21T15:50:31.291 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.292 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:31.292 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.292 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:31.292 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.440996+0000", 2024-09-21T15:50:31.292 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:31.292 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.292 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.292 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:31.292 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.440996+0000", 2024-09-21T15:50:31.292 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:31.292 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.292 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.292 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:31.292 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.440996+0000", 2024-09-21T15:50:31.293 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:31.293 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.293 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.293 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:31.293 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441004+0000", 2024-09-21T15:50:31.293 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8460000000000008e-06 2024-09-21T15:50:31.293 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.293 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.293 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:31.293 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441018+0000", 2024-09-21T15:50:31.293 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.42e-05 2024-09-21T15:50:31.293 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.293 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.293 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:31.293 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441023+0000", 2024-09-21T15:50:31.294 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0019999999999997e-06 2024-09-21T15:50:31.294 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.294 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.294 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:31.294 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489060+0000", 2024-09-21T15:50:31.294 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048036660000000002 2024-09-21T15:50:31.294 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.294 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.294 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:31.294 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489107+0000", 2024-09-21T15:50:31.294 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7153000000000001e-05 2024-09-21T15:50:31.294 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.294 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.294 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:31.294 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489230+0000", 2024-09-21T15:50:31.295 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012328600000000001 2024-09-21T15:50:31.295 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:31.295 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:31.295 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:31.295 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.295 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.295 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155256 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 356352~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:31.295 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441027+0000", 2024-09-21T15:50:31.295 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 8.7997814220000006, 2024-09-21T15:50:31.295 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048677606999999998, 2024-09-21T15:50:31.295 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:31.295 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:31.295 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:31.295 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:31.296 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:31.296 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155256 2024-09-21T15:50:31.296 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.296 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:31.296 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.296 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:31.296 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441027+0000", 2024-09-21T15:50:31.296 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:31.296 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.296 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.296 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:31.296 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441027+0000", 2024-09-21T15:50:31.296 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:31.296 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.297 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.297 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:31.297 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441027+0000", 2024-09-21T15:50:31.297 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:31.297 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.297 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.297 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:31.297 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441029+0000", 2024-09-21T15:50:31.297 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.39e-06 2024-09-21T15:50:31.297 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.297 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.297 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:31.297 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441036+0000", 2024-09-21T15:50:31.297 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1129999999999997e-06 2024-09-21T15:50:31.298 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.298 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.298 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:31.298 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441040+0000", 2024-09-21T15:50:31.298 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8459999999999999e-06 2024-09-21T15:50:31.298 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.298 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.298 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:31.298 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489244+0000", 2024-09-21T15:50:31.298 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048203751000000003 2024-09-21T15:50:31.298 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.298 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.299 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:31.299 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489288+0000", 2024-09-21T15:50:31.299 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4153999999999998e-05 2024-09-21T15:50:31.299 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.299 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.299 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:31.299 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489705+0000", 2024-09-21T15:50:31.299 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000416773 2024-09-21T15:50:31.299 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:31.299 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:31.299 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:31.299 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.299 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.299 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155257 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 389120~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:31.299 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441045+0000", 2024-09-21T15:50:31.300 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 8.7997643310000004, 2024-09-21T15:50:31.300 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049000325999999997, 2024-09-21T15:50:31.300 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:31.300 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:31.300 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:31.300 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:31.300 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:31.300 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155257 2024-09-21T15:50:31.300 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.300 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:31.300 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.300 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:31.300 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441045+0000", 2024-09-21T15:50:31.300 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:31.300 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.300 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.301 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:31.301 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441045+0000", 2024-09-21T15:50:31.301 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:31.301 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.301 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.301 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:31.301 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441044+0000", 2024-09-21T15:50:31.301 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:31.301 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.301 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.301 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:31.301 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441046+0000", 2024-09-21T15:50:31.301 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1909999999999999e-06 2024-09-21T15:50:31.301 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.301 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.302 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:31.302 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441054+0000", 2024-09-21T15:50:31.302 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3900000000000004e-06 2024-09-21T15:50:31.302 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.302 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.302 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:31.302 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441058+0000", 2024-09-21T15:50:31.302 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4619999999999998e-06 2024-09-21T15:50:31.302 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.302 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.302 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:31.302 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489741+0000", 2024-09-21T15:50:31.302 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048682660000000003 2024-09-21T15:50:31.302 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.303 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.303 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:31.303 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489813+0000", 2024-09-21T15:50:31.303 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2051999999999994e-05 2024-09-21T15:50:31.303 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.303 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.303 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:31.303 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490045+0000", 2024-09-21T15:50:31.303 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00023191299999999999 2024-09-21T15:50:31.303 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:31.303 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:31.303 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:31.303 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.303 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.304 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155258 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 421888~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:31.304 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441062+0000", 2024-09-21T15:50:31.304 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 8.7997467890000003, 2024-09-21T15:50:31.304 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049339651999999998, 2024-09-21T15:50:31.304 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:31.304 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:31.304 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:31.304 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:31.304 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:31.304 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155258 2024-09-21T15:50:31.304 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.304 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:31.304 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.304 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:31.305 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441062+0000", 2024-09-21T15:50:31.305 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:31.305 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.305 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.305 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:31.305 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441062+0000", 2024-09-21T15:50:31.305 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:31.305 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.305 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.305 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:31.305 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441062+0000", 2024-09-21T15:50:31.305 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:31.305 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.305 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.305 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:31.305 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441064+0000", 2024-09-21T15:50:31.306 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.967e-06 2024-09-21T15:50:31.306 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.306 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.306 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:31.306 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441069+0000", 2024-09-21T15:50:31.306 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1529999999999996e-06 2024-09-21T15:50:31.306 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.306 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.306 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:31.306 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441073+0000", 2024-09-21T15:50:31.306 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3850000000000004e-06 2024-09-21T15:50:31.306 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.306 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.306 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:31.307 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490082+0000", 2024-09-21T15:50:31.307 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.04900885 2024-09-21T15:50:31.307 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.307 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.307 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:31.307 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490166+0000", 2024-09-21T15:50:31.307 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3640999999999994e-05 2024-09-21T15:50:31.307 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.307 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.307 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:31.307 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490402+0000", 2024-09-21T15:50:31.307 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000235978 2024-09-21T15:50:31.307 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:31.307 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:31.307 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:31.307 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.308 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.308 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155259 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 454656~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:31.308 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441086+0000", 2024-09-21T15:50:31.308 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 8.7997229840000006, 2024-09-21T15:50:31.308 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049624397000000001, 2024-09-21T15:50:31.308 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:31.308 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:31.308 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:31.308 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:31.308 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:31.308 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155259 2024-09-21T15:50:31.308 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.308 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:31.308 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.308 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:31.309 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441086+0000", 2024-09-21T15:50:31.309 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:31.309 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.309 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.309 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:31.309 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441086+0000", 2024-09-21T15:50:31.309 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:31.309 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.309 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.309 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:31.309 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441086+0000", 2024-09-21T15:50:31.309 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:31.309 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.310 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.310 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:31.310 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441088+0000", 2024-09-21T15:50:31.310 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2630000000000002e-06 2024-09-21T15:50:31.310 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.310 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.310 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:31.310 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441093+0000", 2024-09-21T15:50:31.310 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5010000000000004e-06 2024-09-21T15:50:31.310 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.310 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.310 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:31.310 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441098+0000", 2024-09-21T15:50:31.310 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.707e-06 2024-09-21T15:50:31.311 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.311 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.311 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:31.311 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490447+0000", 2024-09-21T15:50:31.311 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049349125000000001 2024-09-21T15:50:31.311 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.311 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.311 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:31.311 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490504+0000", 2024-09-21T15:50:31.311 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6521000000000003e-05 2024-09-21T15:50:31.311 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.311 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.311 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:31.311 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490710+0000", 2024-09-21T15:50:31.311 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00020663999999999999 2024-09-21T15:50:31.312 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:31.312 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:31.312 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:31.312 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.312 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.312 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155260 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 487424~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:31.312 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441160+0000", 2024-09-21T15:50:31.312 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 8.7996490220000005, 2024-09-21T15:50:31.312 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049876810000000001, 2024-09-21T15:50:31.312 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:31.312 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:31.312 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:31.312 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:31.312 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:31.312 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155260 2024-09-21T15:50:31.312 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.313 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:31.313 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.313 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:31.313 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441160+0000", 2024-09-21T15:50:31.313 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:31.313 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.313 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.313 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:31.313 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441160+0000", 2024-09-21T15:50:31.313 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:31.313 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.313 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.313 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:31.313 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441159+0000", 2024-09-21T15:50:31.313 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:31.313 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.314 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.314 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:31.314 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441164+0000", 2024-09-21T15:50:31.314 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4059999999999998e-06 2024-09-21T15:50:31.314 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.314 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.314 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:31.314 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441175+0000", 2024-09-21T15:50:31.314 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1095e-05 2024-09-21T15:50:31.314 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.314 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.314 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:31.314 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441182+0000", 2024-09-21T15:50:31.314 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2139999999999999e-06 2024-09-21T15:50:31.315 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.315 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.315 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:31.315 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490751+0000", 2024-09-21T15:50:31.315 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049569230999999998 2024-09-21T15:50:31.315 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.315 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.315 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:31.315 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490837+0000", 2024-09-21T15:50:31.315 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6017e-05 2024-09-21T15:50:31.315 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.315 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.315 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:31.315 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491037+0000", 2024-09-21T15:50:31.315 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00019952300000000001 2024-09-21T15:50:31.316 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:31.316 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:31.316 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:31.316 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.316 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.316 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155261 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 520192~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:31.316 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441187+0000", 2024-09-21T15:50:31.316 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 8.7996218390000003, 2024-09-21T15:50:31.316 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.050124931999999997, 2024-09-21T15:50:31.316 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:31.316 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:31.316 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:31.316 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:31.316 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:31.316 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155261 2024-09-21T15:50:31.317 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.317 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:31.317 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.317 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:31.317 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441187+0000", 2024-09-21T15:50:31.317 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:31.317 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.317 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.317 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:31.317 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441187+0000", 2024-09-21T15:50:31.317 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:31.317 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.317 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.317 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:31.317 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441187+0000", 2024-09-21T15:50:31.318 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:31.318 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.318 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.318 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:31.318 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441189+0000", 2024-09-21T15:50:31.318 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3649999999999998e-06 2024-09-21T15:50:31.318 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.318 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.318 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:31.318 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441196+0000", 2024-09-21T15:50:31.318 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5379999999999999e-06 2024-09-21T15:50:31.318 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.318 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.318 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:31.318 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441200+0000", 2024-09-21T15:50:31.319 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4719999999999997e-06 2024-09-21T15:50:31.319 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.319 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.319 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:31.319 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491070+0000", 2024-09-21T15:50:31.319 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049869915000000001 2024-09-21T15:50:31.319 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.319 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.319 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:31.319 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491146+0000", 2024-09-21T15:50:31.319 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6589999999999997e-05 2024-09-21T15:50:31.319 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.319 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.319 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:31.319 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491312+0000", 2024-09-21T15:50:31.319 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000165475 2024-09-21T15:50:31.320 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:31.320 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:31.320 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:31.320 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.320 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.320 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155308 3.11 3:8b5183e5:::rbd_data.11335aef9442.0000000000000000:2a [read 356352~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:31.320 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441369+0000", 2024-09-21T15:50:31.320 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 8.7994399909999998, 2024-09-21T15:50:31.320 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.050090672000000003, 2024-09-21T15:50:31.320 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:31.320 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:31.320 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:31.320 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:31.320 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:31.320 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155308 2024-09-21T15:50:31.321 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.321 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:31.321 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.321 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:31.321 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441369+0000", 2024-09-21T15:50:31.321 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:31.321 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.321 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.321 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:31.321 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441369+0000", 2024-09-21T15:50:31.321 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:31.321 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.321 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.321 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:31.321 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441368+0000", 2024-09-21T15:50:31.321 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:31.322 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.322 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.322 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:31.322 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441371+0000", 2024-09-21T15:50:31.322 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1100000000000001e-06 2024-09-21T15:50:31.322 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.322 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.322 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:31.322 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441377+0000", 2024-09-21T15:50:31.322 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5649999999999998e-06 2024-09-21T15:50:31.322 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.322 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.322 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:31.322 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441382+0000", 2024-09-21T15:50:31.322 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5929999999999999e-06 2024-09-21T15:50:31.323 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.323 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.323 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:31.323 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491349+0000", 2024-09-21T15:50:31.323 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049967122000000003 2024-09-21T15:50:31.323 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.323 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.323 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:31.323 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491400+0000", 2024-09-21T15:50:31.323 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1589999999999999e-05 2024-09-21T15:50:31.323 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.323 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.323 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:31.323 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491460+0000", 2024-09-21T15:50:31.323 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9070000000000002e-05 2024-09-21T15:50:31.324 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:31.324 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:31.324 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:31.324 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.324 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.324 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155309 3.11 3:8b5183e5:::rbd_data.11335aef9442.0000000000000000:2a [read 389120~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:31.324 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441386+0000", 2024-09-21T15:50:31.324 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 8.7994226419999997, 2024-09-21T15:50:31.324 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.050184777, 2024-09-21T15:50:31.324 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:31.324 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:31.324 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:31.324 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:31.324 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:31.325 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155309 2024-09-21T15:50:31.325 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.325 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:31.325 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.325 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:31.325 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441386+0000", 2024-09-21T15:50:31.325 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:31.325 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.325 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.325 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:31.325 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441386+0000", 2024-09-21T15:50:31.325 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:31.326 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.326 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.326 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:31.326 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441386+0000", 2024-09-21T15:50:31.326 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:31.326 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.326 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.326 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:31.326 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441388+0000", 2024-09-21T15:50:31.326 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4420000000000001e-06 2024-09-21T15:50:31.326 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.326 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.326 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:31.327 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441395+0000", 2024-09-21T15:50:31.327 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6189999999999996e-06 2024-09-21T15:50:31.327 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.327 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.327 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:31.327 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441400+0000", 2024-09-21T15:50:31.327 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.118e-06 2024-09-21T15:50:31.327 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.327 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.327 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:31.327 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491473+0000", 2024-09-21T15:50:31.327 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.050072783000000003 2024-09-21T15:50:31.327 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.327 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.327 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:31.327 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491518+0000", 2024-09-21T15:50:31.328 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4854000000000001e-05 2024-09-21T15:50:31.328 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:31.328 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:31.328 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:31.328 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491571+0000", 2024-09-21T15:50:31.328 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3358e-05 2024-09-21T15:50:31.328 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:31.328 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:31.328 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:31.328 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:31.328 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:31.328 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:31.329 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:50:31.330 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:50:31.328+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:50:31.430 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:31.429+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:31.442 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:31.442 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:31.442 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:50:31.442 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:31.452 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:50:31.531 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:31.530+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:31.562 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:31.562 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:31.562 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:50:31.562 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:50:31.562 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:31.572 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:50:31.631 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:50:31.630+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:50:31.681 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:31.681 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:50:31.681 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:50:31.681 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:50:31.681 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.681 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160505 3.b 3:d297695c:::rbd_header.11338bae974a:head [watch reconnect cookie 140381478598448 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-21T15:50:31.681 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:29.138810+0000", 2024-09-21T15:50:31.681 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 2.5399236419999998, 2024-09-21T15:50:31.681 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0172598319999999, 2024-09-21T15:50:31.681 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:31.681 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:31.682 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:31.682 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:31.682 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:31.682 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160505 2024-09-21T15:50:31.682 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.682 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:31.682 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.682 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:31.682 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138810+0000", 2024-09-21T15:50:31.682 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:31.682 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.682 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.682 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:31.683 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138810+0000", 2024-09-21T15:50:31.683 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:31.683 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.683 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.683 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:31.683 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138808+0000", 2024-09-21T15:50:31.683 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:50:31.683 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.683 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.683 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:31.683 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138815+0000", 2024-09-21T15:50:31.684 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6549999999999998e-06 2024-09-21T15:50:31.684 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.684 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.684 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:31.684 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138825+0000", 2024-09-21T15:50:31.684 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9790000000000005e-06 2024-09-21T15:50:31.684 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.684 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.684 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:31.684 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138834+0000", 2024-09-21T15:50:31.684 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5179999999999993e-06 2024-09-21T15:50:31.684 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.684 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.684 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:31.684 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.140973+0000", 2024-09-21T15:50:31.685 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0021392049999999999 2024-09-21T15:50:31.685 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.685 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.685 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:50:31.685 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.140981+0000", 2024-09-21T15:50:31.685 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8909999999999995e-06 2024-09-21T15:50:31.685 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.685 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.685 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:31.685 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.150457+0000", 2024-09-21T15:50:31.685 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.009475645 2024-09-21T15:50:31.686 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.686 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.686 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for missing object", 2024-09-21T15:50:31.686 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.150589+0000", 2024-09-21T15:50:31.686 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000131779 2024-09-21T15:50:31.686 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.686 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.686 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:31.686 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.151723+0000", 2024-09-21T15:50:31.686 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0011345979999999999 2024-09-21T15:50:31.686 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.686 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.686 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-09-21T15:50:31.686 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.151872+0000", 2024-09-21T15:50:31.687 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000149021 2024-09-21T15:50:31.687 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.687 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.687 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:31.687 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.152563+0000", 2024-09-21T15:50:31.687 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00069051500000000005 2024-09-21T15:50:31.687 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.687 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.687 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:31.687 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.152673+0000", 2024-09-21T15:50:31.687 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000109795 2024-09-21T15:50:31.687 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.687 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.687 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:31.688 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.156069+0000", 2024-09-21T15:50:31.688 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003396736 2024-09-21T15:50:31.688 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:31.688 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:31.688 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:31.688 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.688 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.688 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160506 3.b 3:d297695c:::rbd_header.11338bae974a:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-21T15:50:31.688 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:29.138844+0000", 2024-09-21T15:50:31.688 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 2.5398887549999998, 2024-09-21T15:50:31.688 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0174592680000001, 2024-09-21T15:50:31.688 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:31.688 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:31.688 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:31.688 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:31.688 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:31.689 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160506 2024-09-21T15:50:31.689 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.689 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:31.689 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.689 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:31.689 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138844+0000", 2024-09-21T15:50:31.689 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:31.689 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.689 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.689 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:31.689 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138844+0000", 2024-09-21T15:50:31.689 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:31.689 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.689 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.689 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:31.690 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138844+0000", 2024-09-21T15:50:31.690 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:31.690 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.690 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.690 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:31.690 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138848+0000", 2024-09-21T15:50:31.690 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.76e-06 2024-09-21T15:50:31.690 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.690 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.690 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:31.690 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138854+0000", 2024-09-21T15:50:31.690 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3890000000000002e-06 2024-09-21T15:50:31.690 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.691 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.691 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:31.691 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138860+0000", 2024-09-21T15:50:31.691 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6409999999999999e-06 2024-09-21T15:50:31.691 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.691 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.691 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:31.691 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.141015+0000", 2024-09-21T15:50:31.691 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0021553779999999999 2024-09-21T15:50:31.691 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.691 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.691 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:50:31.692 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.141021+0000", 2024-09-21T15:50:31.692 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7629999999999997e-06 2024-09-21T15:50:31.692 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.692 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.692 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:31.692 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.150858+0000", 2024-09-21T15:50:31.692 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0098372200000001 2024-09-21T15:50:31.692 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.692 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.692 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for missing object", 2024-09-21T15:50:31.692 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.150899+0000", 2024-09-21T15:50:31.692 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1112000000000002e-05 2024-09-21T15:50:31.692 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.692 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.693 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:31.693 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.152069+0000", 2024-09-21T15:50:31.693 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001170307 2024-09-21T15:50:31.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.693 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.693 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-09-21T15:50:31.693 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.152192+0000", 2024-09-21T15:50:31.693 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012289700000000001 2024-09-21T15:50:31.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.693 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.693 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:31.693 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.153362+0000", 2024-09-21T15:50:31.693 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0011700549999999999 2024-09-21T15:50:31.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.693 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.694 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:31.694 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.153457+0000", 2024-09-21T15:50:31.694 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4687999999999999e-05 2024-09-21T15:50:31.694 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.694 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.694 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:50:31.694 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.153893+0000", 2024-09-21T15:50:31.694 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000436363 2024-09-21T15:50:31.694 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.694 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.694 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:31.694 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.156134+0000", 2024-09-21T15:50:31.694 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0022410009999999998 2024-09-21T15:50:31.694 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.694 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.695 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:31.695 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.156148+0000", 2024-09-21T15:50:31.695 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3574000000000001e-05 2024-09-21T15:50:31.695 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.695 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.695 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:31.695 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.156214+0000", 2024-09-21T15:50:31.695 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6438000000000003e-05 2024-09-21T15:50:31.695 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.695 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.695 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:31.695 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.156304+0000", 2024-09-21T15:50:31.695 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9278999999999997e-05 2024-09-21T15:50:31.695 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:31.695 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:31.696 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:31.696 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.696 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.696 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "MOSDPGPush(3.b 94/93 [PushOp(3:d40b7a9f:::rbd_data.11333a557424.000000000000001d:head, version: 70'684, data_included: [235520~12288,286720~16384,339968~12288,393216~10240], data_size: 51200, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d40b7a9f:::rbd_data.11333a557424.000000000000001d:head@70'684, size: 403456, copy_subset: [0~403456], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:403456, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:50:31.696 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:30.152698+0000", 2024-09-21T15:50:31.696 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.5260356289999999, 2024-09-21T15:50:31.696 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003011138, 2024-09-21T15:50:31.696 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:31.696 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:31.696 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:31.696 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.696 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:31.696 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.152698+0000", 2024-09-21T15:50:31.696 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:31.696 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.696 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.697 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:31.697 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.152698+0000", 2024-09-21T15:50:31.697 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:31.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.697 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.697 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:31.697 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.152697+0000", 2024-09-21T15:50:31.697 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:31.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.697 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.697 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:31.697 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.152727+0000", 2024-09-21T15:50:31.697 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9989e-05 2024-09-21T15:50:31.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.697 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.698 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:31.698 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.152747+0000", 2024-09-21T15:50:31.698 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0514e-05 2024-09-21T15:50:31.698 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.698 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.698 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:31.698 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.152775+0000", 2024-09-21T15:50:31.698 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7934000000000001e-05 2024-09-21T15:50:31.698 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.698 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.698 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:31.698 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.154532+0000", 2024-09-21T15:50:31.698 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0017566400000000001 2024-09-21T15:50:31.698 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.698 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.699 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:31.699 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.154546+0000", 2024-09-21T15:50:31.699 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3995000000000001e-05 2024-09-21T15:50:31.699 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.699 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.699 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:31.699 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.155709+0000", 2024-09-21T15:50:31.699 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001162819 2024-09-21T15:50:31.699 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:31.699 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:31.699 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:31.699 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.700 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.700 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160512 3.b 3:d297695c:::rbd_header.11338bae974a:head [watch unwatch cookie 140381478598448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T15:50:31.700 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:30.161362+0000", 2024-09-21T15:50:31.700 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.5173709469999999, 2024-09-21T15:50:31.700 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0074105029999999997, 2024-09-21T15:50:31.700 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:31.700 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:31.700 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:31.700 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:31.700 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:31.700 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160512 2024-09-21T15:50:31.700 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.700 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:31.700 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.700 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:31.701 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161362+0000", 2024-09-21T15:50:31.701 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:31.701 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.701 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.701 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:31.701 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161362+0000", 2024-09-21T15:50:31.701 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:31.701 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.701 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.701 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:31.701 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161362+0000", 2024-09-21T15:50:31.701 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:31.701 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.701 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.702 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:31.702 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161365+0000", 2024-09-21T15:50:31.702 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8669999999999998e-06 2024-09-21T15:50:31.702 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.702 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.702 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:31.702 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161371+0000", 2024-09-21T15:50:31.702 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8009999999999998e-06 2024-09-21T15:50:31.702 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.702 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.702 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:31.702 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161376+0000", 2024-09-21T15:50:31.702 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0340000000000004e-06 2024-09-21T15:50:31.702 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.702 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.703 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:31.703 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161762+0000", 2024-09-21T15:50:31.703 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038575099999999999 2024-09-21T15:50:31.703 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.703 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.703 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:31.703 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161838+0000", 2024-09-21T15:50:31.703 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5697000000000004e-05 2024-09-21T15:50:31.703 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.703 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.703 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:50:31.703 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161952+0000", 2024-09-21T15:50:31.703 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000114034 2024-09-21T15:50:31.704 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.704 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.704 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:31.704 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.163727+0000", 2024-09-21T15:50:31.704 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0017749560000000001 2024-09-21T15:50:31.704 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.704 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.704 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:31.704 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168545+0000", 2024-09-21T15:50:31.704 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0048186080000000003 2024-09-21T15:50:31.704 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.704 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.704 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:31.704 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168593+0000", 2024-09-21T15:50:31.704 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7549999999999997e-05 2024-09-21T15:50:31.704 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.705 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.705 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:31.705 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168773+0000", 2024-09-21T15:50:31.705 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00017976600000000001 2024-09-21T15:50:31.705 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:31.705 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:31.705 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:31.705 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.705 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.705 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160513 3.b 3:d297695c:::rbd_header.11338bae974a:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-09-21T15:50:31.705 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:30.161755+0000", 2024-09-21T15:50:31.705 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.5169780159999999, 2024-09-21T15:50:31.705 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0072250300000000003, 2024-09-21T15:50:31.705 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:31.705 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:31.706 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:31.706 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:31.706 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:31.706 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160513 2024-09-21T15:50:31.706 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.706 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:31.706 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.706 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:31.706 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161755+0000", 2024-09-21T15:50:31.706 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:31.706 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.706 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.706 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:31.706 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161755+0000", 2024-09-21T15:50:31.707 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:31.707 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.707 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.707 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:31.707 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161754+0000", 2024-09-21T15:50:31.707 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:31.707 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.707 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.707 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:31.707 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161759+0000", 2024-09-21T15:50:31.707 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9699999999999998e-06 2024-09-21T15:50:31.707 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.707 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.707 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:31.707 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161768+0000", 2024-09-21T15:50:31.707 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9949999999999994e-06 2024-09-21T15:50:31.708 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.708 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.708 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:31.708 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161774+0000", 2024-09-21T15:50:31.708 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1290000000000001e-06 2024-09-21T15:50:31.708 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.708 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.708 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:31.708 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.162239+0000", 2024-09-21T15:50:31.708 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00046406600000000001 2024-09-21T15:50:31.708 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.708 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.708 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-09-21T15:50:31.708 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.162296+0000", 2024-09-21T15:50:31.708 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7122e-05 2024-09-21T15:50:31.709 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.709 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.709 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:31.709 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168802+0000", 2024-09-21T15:50:31.709 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0065061700000000004 2024-09-21T15:50:31.709 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.709 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.709 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:31.709 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168877+0000", 2024-09-21T15:50:31.709 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5214000000000002e-05 2024-09-21T15:50:31.709 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.709 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.709 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:31.709 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168980+0000", 2024-09-21T15:50:31.709 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000103182 2024-09-21T15:50:31.710 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:31.710 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:31.710 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:31.710 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.710 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.710 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160527 3.b 3:d297695c:::rbd_header.11338bae974a:head [watch watch cookie 140381476148256] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T15:50:31.710 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:30.182756+0000", 2024-09-21T15:50:31.710 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.4959774349999999, 2024-09-21T15:50:31.710 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0054202579999999998, 2024-09-21T15:50:31.710 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:31.710 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:31.710 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:31.710 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:31.710 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:31.710 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160527 2024-09-21T15:50:31.710 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.711 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:31.711 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.711 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:31.711 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182756+0000", 2024-09-21T15:50:31.711 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:31.711 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.711 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.711 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:31.711 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182756+0000", 2024-09-21T15:50:31.711 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:31.711 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.711 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.711 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:31.711 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182755+0000", 2024-09-21T15:50:31.712 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:31.712 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.712 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.712 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:31.712 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182760+0000", 2024-09-21T15:50:31.712 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4269999999999999e-06 2024-09-21T15:50:31.712 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.712 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.712 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:31.712 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182771+0000", 2024-09-21T15:50:31.712 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0312999999999999e-05 2024-09-21T15:50:31.712 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.712 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.712 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:31.712 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182778+0000", 2024-09-21T15:50:31.713 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5700000000000004e-06 2024-09-21T15:50:31.713 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.713 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.713 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:31.713 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.184745+0000", 2024-09-21T15:50:31.713 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0019667170000000002 2024-09-21T15:50:31.713 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.713 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.713 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:31.713 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.184867+0000", 2024-09-21T15:50:31.713 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000122504 2024-09-21T15:50:31.713 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.713 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.713 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:50:31.714 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.185135+0000", 2024-09-21T15:50:31.714 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000267135 2024-09-21T15:50:31.714 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.714 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.714 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:31.714 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.187721+0000", 2024-09-21T15:50:31.714 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002586891 2024-09-21T15:50:31.714 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.714 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.714 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:31.714 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.187874+0000", 2024-09-21T15:50:31.714 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000152726 2024-09-21T15:50:31.714 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.714 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.715 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:31.715 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.187951+0000", 2024-09-21T15:50:31.715 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7132000000000006e-05 2024-09-21T15:50:31.715 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.715 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.715 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:31.715 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.188176+0000", 2024-09-21T15:50:31.715 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00022477900000000001 2024-09-21T15:50:31.715 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:31.715 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:31.715 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:31.715 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.715 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.715 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160529 3.b 3:d297695c:::rbd_header.11338bae974a:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T15:50:31.715 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:30.188676+0000", 2024-09-21T15:50:31.715 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.490056992, 2024-09-21T15:50:31.716 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035442500000000001, 2024-09-21T15:50:31.716 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:31.716 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:31.716 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:31.716 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:31.716 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:31.716 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160529 2024-09-21T15:50:31.716 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.716 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:31.716 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.716 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:31.716 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.188676+0000", 2024-09-21T15:50:31.716 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:31.716 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.716 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.717 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:31.717 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.188676+0000", 2024-09-21T15:50:31.717 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:31.717 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.717 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.717 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:31.717 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.188676+0000", 2024-09-21T15:50:31.717 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:31.717 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.717 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.717 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:31.717 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.188679+0000", 2024-09-21T15:50:31.717 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7790000000000002e-06 2024-09-21T15:50:31.717 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.717 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.717 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:31.718 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.188687+0000", 2024-09-21T15:50:31.718 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1300000000000003e-06 2024-09-21T15:50:31.718 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.718 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.718 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:31.718 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.188691+0000", 2024-09-21T15:50:31.718 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3069999999999996e-06 2024-09-21T15:50:31.718 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.718 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.718 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:31.718 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.188829+0000", 2024-09-21T15:50:31.718 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013851799999999999 2024-09-21T15:50:31.718 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.718 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.718 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:31.719 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.188942+0000", 2024-09-21T15:50:31.719 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011276400000000001 2024-09-21T15:50:31.719 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.719 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.719 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:50:31.719 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.189208+0000", 2024-09-21T15:50:31.719 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00026575500000000001 2024-09-21T15:50:31.719 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.719 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.719 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:31.719 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.190170+0000", 2024-09-21T15:50:31.719 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00096190299999999996 2024-09-21T15:50:31.719 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.719 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.720 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:31.720 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.192075+0000", 2024-09-21T15:50:31.720 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001905672 2024-09-21T15:50:31.720 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.720 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.720 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:31.720 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.192113+0000", 2024-09-21T15:50:31.720 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7410000000000003e-05 2024-09-21T15:50:31.720 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.720 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.720 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:31.720 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.192220+0000", 2024-09-21T15:50:31.720 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000107611 2024-09-21T15:50:31.720 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:31.721 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:31.721 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:31.721 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.721 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.721 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160554 3.b 3:d297695c:::rbd_header.11338bae974a:head [watch watch cookie 140381476148256] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-21T15:50:31.721 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.171599+0000", 2024-09-21T15:50:31.721 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.507134643, 2024-09-21T15:50:31.721 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035069400000000001, 2024-09-21T15:50:31.721 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:31.721 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:31.721 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:31.721 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:31.721 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:31.721 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160554 2024-09-21T15:50:31.721 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.722 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:31.722 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.722 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:31.722 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.171599+0000", 2024-09-21T15:50:31.722 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:31.722 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.722 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.722 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:31.722 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.171599+0000", 2024-09-21T15:50:31.722 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:31.722 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.722 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.722 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:31.722 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.171598+0000", 2024-09-21T15:50:31.722 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:31.723 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.723 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.723 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:31.723 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.171602+0000", 2024-09-21T15:50:31.723 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4109999999999998e-06 2024-09-21T15:50:31.723 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.723 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.723 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:31.723 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.171612+0000", 2024-09-21T15:50:31.723 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.961e-06 2024-09-21T15:50:31.723 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.723 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.723 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:31.723 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.171620+0000", 2024-09-21T15:50:31.723 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4610000000000002e-06 2024-09-21T15:50:31.724 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.724 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.724 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:31.724 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.171646+0000", 2024-09-21T15:50:31.724 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6455999999999999e-05 2024-09-21T15:50:31.724 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.724 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.724 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:31.724 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.171703+0000", 2024-09-21T15:50:31.724 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7411000000000003e-05 2024-09-21T15:50:31.724 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.724 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.724 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:50:31.724 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.171800+0000", 2024-09-21T15:50:31.724 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6825999999999998e-05 2024-09-21T15:50:31.725 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.725 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.725 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:31.725 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.175002+0000", 2024-09-21T15:50:31.725 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032012339999999999 2024-09-21T15:50:31.725 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.725 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.725 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:31.725 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.175016+0000", 2024-09-21T15:50:31.725 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4191e-05 2024-09-21T15:50:31.725 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.725 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.725 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:31.725 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.175046+0000", 2024-09-21T15:50:31.725 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.061e-05 2024-09-21T15:50:31.726 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.726 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.726 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:31.726 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.175106+0000", 2024-09-21T15:50:31.726 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9175999999999997e-05 2024-09-21T15:50:31.726 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:31.726 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:31.726 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:31.726 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.726 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.726 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:160568 3.18 e95/94 3:1feb8588:::rbd_id.image_client.0-clone41:head v 95'405, mlcod=0'0)", 2024-09-21T15:50:31.726 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.186644+0000", 2024-09-21T15:50:31.726 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.49208878900000003, 2024-09-21T15:50:31.726 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032131130000000001, 2024-09-21T15:50:31.726 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:31.727 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:31.727 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:31.727 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.727 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:31.727 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.186644+0000", 2024-09-21T15:50:31.727 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:31.727 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.727 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.727 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:31.727 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.186644+0000", 2024-09-21T15:50:31.727 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:31.727 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.727 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.727 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:31.727 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.186644+0000", 2024-09-21T15:50:31.727 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:31.728 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.728 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.728 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:31.728 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.186650+0000", 2024-09-21T15:50:31.728 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4659999999999996e-06 2024-09-21T15:50:31.728 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.728 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.728 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:31.728 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.186660+0000", 2024-09-21T15:50:31.728 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6709999999999994e-06 2024-09-21T15:50:31.728 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.728 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.728 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:31.728 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.186666+0000", 2024-09-21T15:50:31.728 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6779999999999997e-06 2024-09-21T15:50:31.729 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.729 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.729 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:31.729 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.186713+0000", 2024-09-21T15:50:31.729 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7580999999999997e-05 2024-09-21T15:50:31.729 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.729 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.729 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:31.729 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.186728+0000", 2024-09-21T15:50:31.729 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5003e-05 2024-09-21T15:50:31.729 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.729 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.729 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:31.729 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.189825+0000", 2024-09-21T15:50:31.729 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0030967070000000002 2024-09-21T15:50:31.730 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.730 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.730 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:31.730 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.189858+0000", 2024-09-21T15:50:31.730 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2657999999999998e-05 2024-09-21T15:50:31.730 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:31.730 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:31.730 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:31.730 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.730 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.730 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160586 3.b 3:d297695c:::rbd_header.11338bae974a:head [watch watch cookie 140381476029296] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-21T15:50:31.730 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.201587+0000", 2024-09-21T15:50:31.730 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.477146075, 2024-09-21T15:50:31.730 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0040745900000000003, 2024-09-21T15:50:31.730 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:31.731 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:31.731 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:31.731 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:31.731 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:31.731 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160586 2024-09-21T15:50:31.731 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.731 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:31.731 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.731 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:31.731 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.201587+0000", 2024-09-21T15:50:31.731 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:31.731 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.731 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.731 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:31.732 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.201587+0000", 2024-09-21T15:50:31.732 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:31.732 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.732 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.732 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:31.732 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.201586+0000", 2024-09-21T15:50:31.732 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:31.732 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.732 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.732 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:31.732 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.201592+0000", 2024-09-21T15:50:31.732 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7910000000000001e-06 2024-09-21T15:50:31.732 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.732 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.733 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:31.733 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.201603+0000", 2024-09-21T15:50:31.733 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0736e-05 2024-09-21T15:50:31.733 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.733 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.733 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:31.733 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.201609+0000", 2024-09-21T15:50:31.733 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9739999999999999e-06 2024-09-21T15:50:31.733 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.733 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.733 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:31.733 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.201643+0000", 2024-09-21T15:50:31.733 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4572e-05 2024-09-21T15:50:31.733 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.733 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.734 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:31.734 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.201726+0000", 2024-09-21T15:50:31.734 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.2763999999999996e-05 2024-09-21T15:50:31.734 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.734 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.734 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:50:31.734 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.201832+0000", 2024-09-21T15:50:31.734 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105931 2024-09-21T15:50:31.734 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.734 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.734 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:31.734 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.202743+0000", 2024-09-21T15:50:31.734 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00091128699999999999 2024-09-21T15:50:31.735 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.735 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.735 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:31.735 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.205540+0000", 2024-09-21T15:50:31.735 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0027967199999999999 2024-09-21T15:50:31.735 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.735 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.735 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:31.735 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.205598+0000", 2024-09-21T15:50:31.735 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7513999999999999e-05 2024-09-21T15:50:31.735 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.735 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.735 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:31.735 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.205662+0000", 2024-09-21T15:50:31.736 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4148999999999999e-05 2024-09-21T15:50:31.736 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:31.736 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:31.736 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:31.736 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.736 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.736 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160605 3.b 3:d1863919:::rbd_data.11338bae974a.0000000000000008:head [delete] snapc 2d=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-21T15:50:31.736 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.213794+0000", 2024-09-21T15:50:31.736 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.464939555, 2024-09-21T15:50:31.736 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046152490000000001, 2024-09-21T15:50:31.736 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:31.736 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:31.736 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:31.736 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:31.737 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:31.737 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160605 2024-09-21T15:50:31.737 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.737 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:31.737 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.737 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:31.737 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.213794+0000", 2024-09-21T15:50:31.737 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:31.737 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.737 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.737 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:31.737 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.213794+0000", 2024-09-21T15:50:31.737 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:31.737 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.738 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.738 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:31.738 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.213793+0000", 2024-09-21T15:50:31.738 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:31.738 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.738 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.738 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:31.738 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.213799+0000", 2024-09-21T15:50:31.738 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9959999999999999e-06 2024-09-21T15:50:31.738 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.738 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.738 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:31.738 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.213815+0000", 2024-09-21T15:50:31.738 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5880999999999999e-05 2024-09-21T15:50:31.739 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.739 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.739 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:31.739 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.213838+0000", 2024-09-21T15:50:31.739 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3671000000000001e-05 2024-09-21T15:50:31.739 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.739 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.739 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:31.739 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.213887+0000", 2024-09-21T15:50:31.739 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8865000000000001e-05 2024-09-21T15:50:31.739 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.739 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.739 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:31.739 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.213962+0000", 2024-09-21T15:50:31.740 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4592999999999995e-05 2024-09-21T15:50:31.740 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.740 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.740 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:50:31.740 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.214088+0000", 2024-09-21T15:50:31.740 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012666199999999999 2024-09-21T15:50:31.740 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.740 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.740 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:31.740 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216917+0000", 2024-09-21T15:50:31.740 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0028284059999999999 2024-09-21T15:50:31.740 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.740 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.740 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:31.740 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.218319+0000", 2024-09-21T15:50:31.741 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001402613 2024-09-21T15:50:31.741 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.741 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.741 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:31.741 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.218372+0000", 2024-09-21T15:50:31.741 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2636000000000002e-05 2024-09-21T15:50:31.741 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.741 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.741 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:31.741 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.218409+0000", 2024-09-21T15:50:31.741 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6942e-05 2024-09-21T15:50:31.741 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:31.741 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:31.741 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:31.741 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.741 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.742 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:160602 3.12 e95/94 3:49a61a92:::rbd_data.11338bae974a.0000000000000005:head v 95'344, mlcod=0'0)", 2024-09-21T15:50:31.742 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.215034+0000", 2024-09-21T15:50:31.742 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.46369913800000001, 2024-09-21T15:50:31.742 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045544870000000003, 2024-09-21T15:50:31.742 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:31.742 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:31.742 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:31.742 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.742 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:31.742 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215034+0000", 2024-09-21T15:50:31.742 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:31.742 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.742 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.742 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:31.743 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215034+0000", 2024-09-21T15:50:31.743 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:31.743 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.743 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.743 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:31.743 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215034+0000", 2024-09-21T15:50:31.743 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:31.743 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.743 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.743 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:31.743 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215039+0000", 2024-09-21T15:50:31.743 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1549999999999998e-06 2024-09-21T15:50:31.743 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.743 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.743 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:31.744 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215050+0000", 2024-09-21T15:50:31.744 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1392e-05 2024-09-21T15:50:31.744 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.744 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.744 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:31.744 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215057+0000", 2024-09-21T15:50:31.744 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7399999999999998e-06 2024-09-21T15:50:31.744 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.744 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.744 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:31.744 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215187+0000", 2024-09-21T15:50:31.744 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013010100000000001 2024-09-21T15:50:31.744 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.744 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.745 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:31.745 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215208+0000", 2024-09-21T15:50:31.745 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1013999999999999e-05 2024-09-21T15:50:31.745 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.745 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.745 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:31.745 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.219549+0000", 2024-09-21T15:50:31.745 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0043404990000000003 2024-09-21T15:50:31.745 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.745 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.745 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:31.745 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.219589+0000", 2024-09-21T15:50:31.745 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9878e-05 2024-09-21T15:50:31.745 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:31.745 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:31.745 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:31.746 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.746 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.746 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:160597 3.1 e95/94 3:84154dfc:::rbd_data.11338bae974a.0000000000000000:head v 95'451, mlcod=0'0)", 2024-09-21T15:50:31.746 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.215080+0000", 2024-09-21T15:50:31.746 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.46365368200000001, 2024-09-21T15:50:31.746 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045596819999999998, 2024-09-21T15:50:31.746 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:31.746 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:31.746 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:31.746 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.746 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:31.746 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215080+0000", 2024-09-21T15:50:31.746 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:31.746 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.747 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.747 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:31.747 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215080+0000", 2024-09-21T15:50:31.747 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:31.747 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.747 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.747 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:31.747 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215079+0000", 2024-09-21T15:50:31.747 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:31.747 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.747 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.747 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:31.747 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215083+0000", 2024-09-21T15:50:31.747 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2570000000000001e-06 2024-09-21T15:50:31.747 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.748 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.748 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:31.748 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215091+0000", 2024-09-21T15:50:31.748 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7829999999999999e-06 2024-09-21T15:50:31.748 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.748 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.748 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:31.748 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215096+0000", 2024-09-21T15:50:31.748 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0370000000000002e-06 2024-09-21T15:50:31.748 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.748 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.748 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:31.748 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216055+0000", 2024-09-21T15:50:31.748 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00095892299999999998 2024-09-21T15:50:31.748 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.748 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.749 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:31.749 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216078+0000", 2024-09-21T15:50:31.749 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2376000000000001e-05 2024-09-21T15:50:31.749 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.749 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.749 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:31.749 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.219598+0000", 2024-09-21T15:50:31.749 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035209299999999998 2024-09-21T15:50:31.749 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.749 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.749 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:31.749 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.219639+0000", 2024-09-21T15:50:31.749 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0766000000000002e-05 2024-09-21T15:50:31.749 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:31.749 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:31.750 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:31.750 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.750 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.750 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:160601 3.17 e95/94 3:e9d872b2:::rbd_data.11338bae974a.0000000000000004:head v 95'912, mlcod=0'0)", 2024-09-21T15:50:31.750 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.215668+0000", 2024-09-21T15:50:31.750 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.463065065, 2024-09-21T15:50:31.750 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0038923640000000002, 2024-09-21T15:50:31.750 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:31.750 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:31.750 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:31.750 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.750 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:31.750 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215668+0000", 2024-09-21T15:50:31.750 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:31.750 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.750 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.751 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:31.751 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215668+0000", 2024-09-21T15:50:31.751 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:31.751 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.751 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.751 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:31.751 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215668+0000", 2024-09-21T15:50:31.751 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:31.751 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.751 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.751 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:31.751 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215672+0000", 2024-09-21T15:50:31.751 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7680000000000002e-06 2024-09-21T15:50:31.752 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.752 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.752 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:31.752 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215682+0000", 2024-09-21T15:50:31.752 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0774000000000001e-05 2024-09-21T15:50:31.752 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.752 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.752 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:31.752 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215689+0000", 2024-09-21T15:50:31.752 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4670000000000001e-06 2024-09-21T15:50:31.752 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.752 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.752 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:31.752 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215740+0000", 2024-09-21T15:50:31.752 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1100000000000002e-05 2024-09-21T15:50:31.753 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.753 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.753 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:31.753 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215761+0000", 2024-09-21T15:50:31.753 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1112e-05 2024-09-21T15:50:31.753 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.753 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.753 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:31.753 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.219520+0000", 2024-09-21T15:50:31.753 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0037590739999999998 2024-09-21T15:50:31.753 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.753 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.753 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:31.753 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.219561+0000", 2024-09-21T15:50:31.753 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0329e-05 2024-09-21T15:50:31.754 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:31.754 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:31.754 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:31.754 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.754 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.754 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:160608 3.12 e95/94 3:48a47e1c:::rbd_data.11338bae974a.000000000000000b:head v 95'345, mlcod=0'0)", 2024-09-21T15:50:31.754 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.216934+0000", 2024-09-21T15:50:31.754 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.461799297, 2024-09-21T15:50:31.754 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053335759999999996, 2024-09-21T15:50:31.754 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:31.754 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:31.754 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:31.754 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.754 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:31.754 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216934+0000", 2024-09-21T15:50:31.755 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:31.755 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.755 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.755 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:31.755 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216934+0000", 2024-09-21T15:50:31.755 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:31.755 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.755 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.755 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:31.755 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216933+0000", 2024-09-21T15:50:31.755 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:31.755 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.755 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.755 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:31.755 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216939+0000", 2024-09-21T15:50:31.755 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6459999999999998e-06 2024-09-21T15:50:31.756 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.756 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.756 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:31.756 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216947+0000", 2024-09-21T15:50:31.756 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.2479999999999996e-06 2024-09-21T15:50:31.756 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.756 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.756 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:31.756 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216951+0000", 2024-09-21T15:50:31.756 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3760000000000001e-06 2024-09-21T15:50:31.756 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.756 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.756 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:31.756 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216998+0000", 2024-09-21T15:50:31.756 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6244e-05 2024-09-21T15:50:31.756 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.757 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.757 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:31.757 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.217019+0000", 2024-09-21T15:50:31.757 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1191000000000001e-05 2024-09-21T15:50:31.757 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.757 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.757 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:31.757 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.222230+0000", 2024-09-21T15:50:31.757 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052114350000000004 2024-09-21T15:50:31.757 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.757 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.757 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:31.757 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.222267+0000", 2024-09-21T15:50:31.757 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7169999999999998e-05 2024-09-21T15:50:31.757 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:31.757 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:31.758 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:31.758 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.758 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.758 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:160610 3.3 e95/94 3:c238de8e:::rbd_data.11338bae974a.000000000000000f:head v 95'632, mlcod=71'310)", 2024-09-21T15:50:31.758 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.218864+0000", 2024-09-21T15:50:31.758 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.45986908799999998, 2024-09-21T15:50:31.758 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0034014280000000002, 2024-09-21T15:50:31.758 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:31.758 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:31.758 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:31.758 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.758 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:31.758 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.218864+0000", 2024-09-21T15:50:31.758 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:31.758 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.759 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.759 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:31.759 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.218864+0000", 2024-09-21T15:50:31.759 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:31.759 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.759 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.759 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:31.759 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.218863+0000", 2024-09-21T15:50:31.759 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:31.759 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.759 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.759 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:31.759 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.218870+0000", 2024-09-21T15:50:31.759 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2559999999999999e-06 2024-09-21T15:50:31.759 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.760 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.760 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:31.760 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.218880+0000", 2024-09-21T15:50:31.760 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.009e-05 2024-09-21T15:50:31.760 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.760 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.760 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:31.760 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.218885+0000", 2024-09-21T15:50:31.760 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0039999999999999e-06 2024-09-21T15:50:31.760 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.760 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.760 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:31.760 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.218919+0000", 2024-09-21T15:50:31.761 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4195e-05 2024-09-21T15:50:31.761 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.761 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.761 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:31.761 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.218941+0000", 2024-09-21T15:50:31.761 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2266e-05 2024-09-21T15:50:31.761 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.761 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.761 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:31.761 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.222241+0000", 2024-09-21T15:50:31.761 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032993889999999998 2024-09-21T15:50:31.761 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.761 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.761 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:31.761 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.222266+0000", 2024-09-21T15:50:31.761 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4929000000000001e-05 2024-09-21T15:50:31.762 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:31.762 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:31.762 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:31.762 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.762 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.762 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160614 3.b 3:d297695c:::rbd_header.11338bae974a:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-21T15:50:31.762 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.227565+0000", 2024-09-21T15:50:31.762 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.451168401, 2024-09-21T15:50:31.762 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0027157330000000001, 2024-09-21T15:50:31.762 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:31.762 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:31.762 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:31.762 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:31.762 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:31.762 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160614 2024-09-21T15:50:31.763 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.763 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:31.763 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.763 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:31.763 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.227565+0000", 2024-09-21T15:50:31.763 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:31.763 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.763 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.763 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:31.763 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.227565+0000", 2024-09-21T15:50:31.763 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:31.763 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.763 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.763 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:31.763 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.227563+0000", 2024-09-21T15:50:31.763 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:50:31.764 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.764 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.764 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:31.764 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.227570+0000", 2024-09-21T15:50:31.764 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.409e-06 2024-09-21T15:50:31.764 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.764 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.764 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:31.764 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.227582+0000", 2024-09-21T15:50:31.764 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2454e-05 2024-09-21T15:50:31.764 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.764 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.764 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:31.764 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.227590+0000", 2024-09-21T15:50:31.765 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5070000000000003e-06 2024-09-21T15:50:31.765 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.765 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.765 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:31.765 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.227628+0000", 2024-09-21T15:50:31.765 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8699999999999999e-05 2024-09-21T15:50:31.765 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.765 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.765 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:31.765 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.227689+0000", 2024-09-21T15:50:31.765 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0210000000000001e-05 2024-09-21T15:50:31.765 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.765 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.765 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:50:31.765 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.227806+0000", 2024-09-21T15:50:31.765 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000117198 2024-09-21T15:50:31.766 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.766 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.766 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:31.766 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.228924+0000", 2024-09-21T15:50:31.766 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001118665 2024-09-21T15:50:31.766 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.766 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.766 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:31.766 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.230205+0000", 2024-09-21T15:50:31.767 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00128053 2024-09-21T15:50:31.767 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.767 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.767 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:31.767 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.230237+0000", 2024-09-21T15:50:31.767 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1693000000000003e-05 2024-09-21T15:50:31.767 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.767 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.767 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:31.767 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.230281+0000", 2024-09-21T15:50:31.767 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3909000000000003e-05 2024-09-21T15:50:31.767 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:31.767 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:31.767 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:31.767 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.768 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.768 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160618 3.b 3:d297695c:::rbd_header.11338bae974a:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-21T15:50:31.768 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.233457+0000", 2024-09-21T15:50:31.768 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.44527645999999999, 2024-09-21T15:50:31.768 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.005085198, 2024-09-21T15:50:31.768 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:31.768 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:31.768 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:31.768 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:31.768 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:31.768 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160618 2024-09-21T15:50:31.768 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.768 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:31.768 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.768 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:31.768 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233457+0000", 2024-09-21T15:50:31.769 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:31.769 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.769 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.769 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:31.769 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233457+0000", 2024-09-21T15:50:31.769 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:31.769 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.769 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.769 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:31.769 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233456+0000", 2024-09-21T15:50:31.769 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:31.769 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.769 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.769 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:31.769 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233463+0000", 2024-09-21T15:50:31.769 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9310000000000002e-06 2024-09-21T15:50:31.770 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.770 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.770 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:31.770 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233473+0000", 2024-09-21T15:50:31.770 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0519e-05 2024-09-21T15:50:31.770 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.770 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.770 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:31.770 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233481+0000", 2024-09-21T15:50:31.770 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4009999999999997e-06 2024-09-21T15:50:31.770 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.770 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.770 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:31.770 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233513+0000", 2024-09-21T15:50:31.770 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1328999999999998e-05 2024-09-21T15:50:31.771 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.771 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.771 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:31.771 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233568+0000", 2024-09-21T15:50:31.771 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5606999999999998e-05 2024-09-21T15:50:31.771 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.771 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.771 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:50:31.771 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233670+0000", 2024-09-21T15:50:31.771 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010180599999999999 2024-09-21T15:50:31.771 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.771 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.771 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:31.771 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.234761+0000", 2024-09-21T15:50:31.771 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010905909999999999 2024-09-21T15:50:31.772 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.772 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.772 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:31.772 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.238442+0000", 2024-09-21T15:50:31.772 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0036809429999999999 2024-09-21T15:50:31.772 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.772 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.772 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:31.772 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.238498+0000", 2024-09-21T15:50:31.772 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6116999999999998e-05 2024-09-21T15:50:31.772 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.772 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.772 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:31.772 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.238542+0000", 2024-09-21T15:50:31.772 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4134000000000001e-05 2024-09-21T15:50:31.772 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:31.773 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:31.773 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:31.773 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.773 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.773 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160660 3.b 3:d7701915:::rbd_object_map.1133fd7483d4.000000000000002c:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95)", 2024-09-21T15:50:31.773 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.274362+0000", 2024-09-21T15:50:31.773 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.40437076300000002, 2024-09-21T15:50:31.773 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0028700079999999998, 2024-09-21T15:50:31.773 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:31.773 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:31.773 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:31.773 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:31.773 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:31.773 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160660 2024-09-21T15:50:31.774 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.774 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:31.774 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.774 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:31.774 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.274362+0000", 2024-09-21T15:50:31.774 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:31.774 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.774 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.774 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:31.774 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.274362+0000", 2024-09-21T15:50:31.774 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:31.774 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.774 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.775 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:31.775 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.274361+0000", 2024-09-21T15:50:31.775 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:31.775 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.775 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.775 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:31.775 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.274368+0000", 2024-09-21T15:50:31.775 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4500000000000001e-06 2024-09-21T15:50:31.775 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.775 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.775 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:31.775 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.274378+0000", 2024-09-21T15:50:31.775 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0309000000000001e-05 2024-09-21T15:50:31.775 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.775 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.775 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:31.776 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.274383+0000", 2024-09-21T15:50:31.776 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7570000000000004e-06 2024-09-21T15:50:31.776 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.776 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.776 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:31.776 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.274438+0000", 2024-09-21T15:50:31.776 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5238999999999997e-05 2024-09-21T15:50:31.776 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.776 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.776 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:31.776 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.274540+0000", 2024-09-21T15:50:31.776 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101411 2024-09-21T15:50:31.776 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.776 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.776 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:31.776 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.277232+0000", 2024-09-21T15:50:31.777 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0026929359999999999 2024-09-21T15:50:31.777 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:31.777 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:31.777 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:31.777 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.777 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.777 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160662 3.b 3:d7701915:::rbd_object_map.1133fd7483d4.000000000000002c:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-21T15:50:31.777 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.279369+0000", 2024-09-21T15:50:31.777 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.39936453, 2024-09-21T15:50:31.777 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0056948040000000004, 2024-09-21T15:50:31.777 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:31.777 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:31.777 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:31.777 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:31.777 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:31.777 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160662 2024-09-21T15:50:31.778 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.778 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:31.778 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.778 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:31.778 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279369+0000", 2024-09-21T15:50:31.778 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:31.778 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.778 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.778 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:31.778 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279369+0000", 2024-09-21T15:50:31.778 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:31.778 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.778 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.778 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:31.778 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279368+0000", 2024-09-21T15:50:31.779 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:31.779 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.779 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.779 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:31.779 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279373+0000", 2024-09-21T15:50:31.779 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6529999999999999e-06 2024-09-21T15:50:31.779 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.779 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.779 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:31.779 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279384+0000", 2024-09-21T15:50:31.779 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0256e-05 2024-09-21T15:50:31.779 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.779 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.779 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:31.779 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279388+0000", 2024-09-21T15:50:31.780 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5630000000000002e-06 2024-09-21T15:50:31.780 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.780 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.780 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:31.780 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279424+0000", 2024-09-21T15:50:31.780 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5985999999999997e-05 2024-09-21T15:50:31.780 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.780 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.780 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:31.780 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279478+0000", 2024-09-21T15:50:31.780 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3944999999999999e-05 2024-09-21T15:50:31.780 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.780 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.780 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:50:31.781 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279575+0000", 2024-09-21T15:50:31.781 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6817999999999994e-05 2024-09-21T15:50:31.781 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.781 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.781 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:31.781 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.282391+0000", 2024-09-21T15:50:31.781 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0028156700000000002 2024-09-21T15:50:31.781 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.781 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.781 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:31.781 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.284977+0000", 2024-09-21T15:50:31.781 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0025859640000000001 2024-09-21T15:50:31.781 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.781 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.782 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:31.782 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.285021+0000", 2024-09-21T15:50:31.782 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4415000000000001e-05 2024-09-21T15:50:31.782 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:31.782 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:31.782 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:31.782 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.285063+0000", 2024-09-21T15:50:31.782 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2447000000000002e-05 2024-09-21T15:50:31.782 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:31.782 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:31.782 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:31.782 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:31.782 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:31.782 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:31.784 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:50:31.784 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:31.782+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:31.885 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:31.883+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:31.939 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:31.939 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:31.939 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:50:31.939 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:31.948 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:50:31.985 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:31.984+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:32.053 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:32.054 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:32.054 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:50:32.054 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:50:32.054 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:32.062 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:50:32.106 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:32.105+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:32.156 INFO:teuthology.orchestra.run.smithi089.stderr:reweighted osd.2 to 1 (10000) 2024-09-21T15:50:32.174 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:32.174 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:50:32.174 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:50:32.174 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:50:32.174 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.174 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62833 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 192512~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:32.174 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:50:32.174 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 55.029777617000001, 2024-09-21T15:50:32.175 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050387007999999997, 2024-09-21T15:50:32.175 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:32.175 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:32.175 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:32.175 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:32.175 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:32.175 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62833 2024-09-21T15:50:32.175 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.175 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:32.175 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.175 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:32.175 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:50:32.175 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:32.175 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.176 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.176 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:32.176 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:50:32.176 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:32.176 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.176 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.176 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:32.176 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:50:32.176 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:32.176 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.176 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.176 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:32.176 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142150+0000", 2024-09-21T15:50:32.176 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1999999999999999e-06 2024-09-21T15:50:32.176 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.177 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.177 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:32.177 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142153+0000", 2024-09-21T15:50:32.177 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7410000000000001e-06 2024-09-21T15:50:32.177 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.177 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.177 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:32.177 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142156+0000", 2024-09-21T15:50:32.177 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5739999999999998e-06 2024-09-21T15:50:32.177 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.177 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.177 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:32.177 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.192358+0000", 2024-09-21T15:50:32.177 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050202452000000002 2024-09-21T15:50:32.178 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.178 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.178 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:32.178 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.192435+0000", 2024-09-21T15:50:32.178 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6738999999999998e-05 2024-09-21T15:50:32.178 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.178 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.178 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:32.178 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.192536+0000", 2024-09-21T15:50:32.178 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101508 2024-09-21T15:50:32.178 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:32.178 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:32.178 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:32.178 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.178 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.179 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62849 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 208896~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:32.179 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.144181+0000", 2024-09-21T15:50:32.179 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 55.027745545000002, 2024-09-21T15:50:32.179 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050460390000000001, 2024-09-21T15:50:32.179 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:32.179 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:32.179 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:32.179 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:32.179 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:32.179 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62849 2024-09-21T15:50:32.179 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.179 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:32.179 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.179 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:32.179 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144181+0000", 2024-09-21T15:50:32.180 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:32.180 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.180 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.180 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:32.180 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144181+0000", 2024-09-21T15:50:32.180 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:32.180 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.180 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.180 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:32.180 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144181+0000", 2024-09-21T15:50:32.180 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:32.180 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.180 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.180 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:32.180 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144183+0000", 2024-09-21T15:50:32.180 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3860000000000001e-06 2024-09-21T15:50:32.181 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.181 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.181 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:32.181 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144189+0000", 2024-09-21T15:50:32.181 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0599999999999996e-06 2024-09-21T15:50:32.181 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.181 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.181 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:32.181 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144194+0000", 2024-09-21T15:50:32.181 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4379999999999997e-06 2024-09-21T15:50:32.181 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.181 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.181 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:32.181 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.194457+0000", 2024-09-21T15:50:32.182 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050263317000000002 2024-09-21T15:50:32.182 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.182 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.182 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:32.182 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.194548+0000", 2024-09-21T15:50:32.182 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0785000000000006e-05 2024-09-21T15:50:32.182 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.182 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.182 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:32.182 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.194642+0000", 2024-09-21T15:50:32.182 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3837000000000003e-05 2024-09-21T15:50:32.182 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:32.182 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:32.182 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:32.183 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.183 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.183 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62862 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 954368~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:32.183 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.146176+0000", 2024-09-21T15:50:32.183 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 55.025750481999999, 2024-09-21T15:50:32.183 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050442901999999998, 2024-09-21T15:50:32.183 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:32.183 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:32.183 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:32.183 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:32.183 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:32.183 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62862 2024-09-21T15:50:32.183 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.183 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:32.184 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.184 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:32.184 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146176+0000", 2024-09-21T15:50:32.184 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:32.184 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.184 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.184 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:32.184 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146176+0000", 2024-09-21T15:50:32.184 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:32.184 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.184 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.184 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:32.184 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146176+0000", 2024-09-21T15:50:32.184 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:32.184 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.184 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.185 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:32.185 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146178+0000", 2024-09-21T15:50:32.185 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4739999999999999e-06 2024-09-21T15:50:32.185 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.185 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.185 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:32.185 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146186+0000", 2024-09-21T15:50:32.185 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4989999999999997e-06 2024-09-21T15:50:32.185 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.185 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.185 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:32.185 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146191+0000", 2024-09-21T15:50:32.185 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7999999999999998e-06 2024-09-21T15:50:32.185 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.185 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.185 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:32.186 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196442+0000", 2024-09-21T15:50:32.186 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050251665000000001 2024-09-21T15:50:32.186 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.186 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.186 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:32.186 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196518+0000", 2024-09-21T15:50:32.186 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5677000000000001e-05 2024-09-21T15:50:32.186 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.186 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.186 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:32.186 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196619+0000", 2024-09-21T15:50:32.186 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101247 2024-09-21T15:50:32.186 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:32.186 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:32.187 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:32.187 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.187 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.187 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62863 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 937984~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:32.187 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.146195+0000", 2024-09-21T15:50:32.187 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 55.025732290999997, 2024-09-21T15:50:32.187 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050642305999999998, 2024-09-21T15:50:32.187 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:32.187 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:32.187 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:32.187 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:32.187 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:32.187 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62863 2024-09-21T15:50:32.187 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.188 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:32.188 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.188 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:32.188 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146195+0000", 2024-09-21T15:50:32.188 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:32.188 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.188 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.188 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:32.188 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146195+0000", 2024-09-21T15:50:32.188 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:32.188 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.188 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.188 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:32.189 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146194+0000", 2024-09-21T15:50:32.189 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:32.189 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.189 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.189 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:32.189 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146196+0000", 2024-09-21T15:50:32.189 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1900000000000002e-06 2024-09-21T15:50:32.189 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.189 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.189 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:32.189 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146202+0000", 2024-09-21T15:50:32.189 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.57e-06 2024-09-21T15:50:32.189 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.189 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.189 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:32.190 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146206+0000", 2024-09-21T15:50:32.190 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5079999999999999e-06 2024-09-21T15:50:32.190 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.190 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.190 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:32.190 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196655+0000", 2024-09-21T15:50:32.190 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050448973000000001 2024-09-21T15:50:32.190 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.190 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.190 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:32.190 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196730+0000", 2024-09-21T15:50:32.190 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4728999999999996e-05 2024-09-21T15:50:32.190 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.190 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.190 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:32.191 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196837+0000", 2024-09-21T15:50:32.191 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000106735 2024-09-21T15:50:32.191 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:32.191 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:32.191 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:32.191 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.191 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.191 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62872 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 290816~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:32.191 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148149+0000", 2024-09-21T15:50:32.191 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 55.023777615, 2024-09-21T15:50:32.191 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.05046991, 2024-09-21T15:50:32.191 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:32.191 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:32.191 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:32.192 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:32.192 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:32.192 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62872 2024-09-21T15:50:32.192 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.192 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:32.192 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.192 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:32.192 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148149+0000", 2024-09-21T15:50:32.192 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:32.192 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.192 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.192 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:32.192 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148149+0000", 2024-09-21T15:50:32.193 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:32.193 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.193 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.193 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:32.193 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148149+0000", 2024-09-21T15:50:32.193 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:32.193 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.194 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.194 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:32.194 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148157+0000", 2024-09-21T15:50:32.194 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4749999999999993e-06 2024-09-21T15:50:32.194 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.194 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.194 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:32.194 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148163+0000", 2024-09-21T15:50:32.194 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8250000000000001e-06 2024-09-21T15:50:32.194 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.194 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.194 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:32.195 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148168+0000", 2024-09-21T15:50:32.195 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4730000000000002e-06 2024-09-21T15:50:32.195 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.195 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.195 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:32.195 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198471+0000", 2024-09-21T15:50:32.195 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050302974 2024-09-21T15:50:32.195 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.195 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.195 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:32.195 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198555+0000", 2024-09-21T15:50:32.195 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4783999999999993e-05 2024-09-21T15:50:32.196 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.196 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.196 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:32.196 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198619+0000", 2024-09-21T15:50:32.196 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3751000000000007e-05 2024-09-21T15:50:32.196 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:32.196 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:32.196 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:32.196 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.196 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.196 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62873 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 315392~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:32.196 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148172+0000", 2024-09-21T15:50:32.196 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 55.023755324, 2024-09-21T15:50:32.197 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050707998999999997, 2024-09-21T15:50:32.197 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:32.197 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:32.197 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:32.197 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:32.197 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:32.197 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62873 2024-09-21T15:50:32.197 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.197 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:32.197 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.197 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:32.197 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148172+0000", 2024-09-21T15:50:32.197 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:32.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:32.198 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148172+0000", 2024-09-21T15:50:32.198 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:32.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:32.198 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148171+0000", 2024-09-21T15:50:32.198 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:32.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:32.199 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148173+0000", 2024-09-21T15:50:32.199 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2340000000000001e-06 2024-09-21T15:50:32.199 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.199 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.199 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:32.199 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148179+0000", 2024-09-21T15:50:32.199 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9340000000000003e-06 2024-09-21T15:50:32.199 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.199 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.199 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:32.199 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148186+0000", 2024-09-21T15:50:32.199 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2349999999999996e-06 2024-09-21T15:50:32.199 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.199 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.200 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:32.200 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198684+0000", 2024-09-21T15:50:32.200 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050498890999999997 2024-09-21T15:50:32.200 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.200 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.200 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:32.200 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198778+0000", 2024-09-21T15:50:32.200 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3134999999999995e-05 2024-09-21T15:50:32.200 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.200 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.200 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:32.200 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198880+0000", 2024-09-21T15:50:32.200 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101906 2024-09-21T15:50:32.201 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:32.201 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:32.201 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:32.201 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.201 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.201 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62874 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 323584~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:32.201 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148211+0000", 2024-09-21T15:50:32.201 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 55.023715564, 2024-09-21T15:50:32.201 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050887766000000001, 2024-09-21T15:50:32.201 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:32.201 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:32.202 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:32.202 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:32.202 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:32.202 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62874 2024-09-21T15:50:32.202 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.202 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:32.202 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.202 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:32.202 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148211+0000", 2024-09-21T15:50:32.202 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:32.202 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.202 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.203 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:32.203 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148211+0000", 2024-09-21T15:50:32.203 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:32.203 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.203 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.203 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:32.203 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148211+0000", 2024-09-21T15:50:32.203 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:32.203 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.203 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.203 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:32.203 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148213+0000", 2024-09-21T15:50:32.204 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6860000000000002e-06 2024-09-21T15:50:32.204 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.204 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.204 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:32.204 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148220+0000", 2024-09-21T15:50:32.204 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4740000000000002e-06 2024-09-21T15:50:32.204 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.204 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.204 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:32.204 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148226+0000", 2024-09-21T15:50:32.204 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0120000000000002e-06 2024-09-21T15:50:32.204 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.205 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.205 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:32.205 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198940+0000", 2024-09-21T15:50:32.205 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050714412 2024-09-21T15:50:32.205 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.205 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.205 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:32.205 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198995+0000", 2024-09-21T15:50:32.205 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4863e-05 2024-09-21T15:50:32.205 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.205 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.205 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:32.206 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.199099+0000", 2024-09-21T15:50:32.206 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010383299999999999 2024-09-21T15:50:32.206 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:32.206 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:32.206 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:32.206 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.206 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.206 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62877 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 995328~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:32.206 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148230+0000", 2024-09-21T15:50:32.206 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 55.023696704000002, 2024-09-21T15:50:32.206 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.051108949000000001, 2024-09-21T15:50:32.206 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:32.207 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:32.207 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:32.207 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:32.207 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:32.207 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62877 2024-09-21T15:50:32.207 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.207 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:32.207 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.207 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:32.207 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148230+0000", 2024-09-21T15:50:32.207 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:32.207 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.207 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.208 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:32.208 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148230+0000", 2024-09-21T15:50:32.208 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:32.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.208 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.208 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:32.208 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148230+0000", 2024-09-21T15:50:32.208 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:32.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.208 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.208 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:32.208 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148232+0000", 2024-09-21T15:50:32.208 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1220000000000002e-06 2024-09-21T15:50:32.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.209 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.209 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:32.209 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148238+0000", 2024-09-21T15:50:32.209 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.789e-06 2024-09-21T15:50:32.209 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.209 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.209 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:32.209 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148242+0000", 2024-09-21T15:50:32.209 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7369999999999997e-06 2024-09-21T15:50:32.209 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.209 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.209 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:32.209 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.199201+0000", 2024-09-21T15:50:32.209 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050959088 2024-09-21T15:50:32.210 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.210 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.210 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:32.210 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.199285+0000", 2024-09-21T15:50:32.210 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3147999999999997e-05 2024-09-21T15:50:32.210 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.210 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.210 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:32.210 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.199339+0000", 2024-09-21T15:50:32.210 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4484999999999998e-05 2024-09-21T15:50:32.210 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:32.210 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:32.210 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:32.211 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.211 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.211 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62886 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 1044480~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:32.211 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150138+0000", 2024-09-21T15:50:32.211 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 55.021789036000001, 2024-09-21T15:50:32.211 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050345527000000001, 2024-09-21T15:50:32.211 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:32.211 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:32.211 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:32.211 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:32.211 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:32.211 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62886 2024-09-21T15:50:32.211 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.211 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:32.212 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.212 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:32.212 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150138+0000", 2024-09-21T15:50:32.212 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:32.212 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.212 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.212 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:32.212 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150138+0000", 2024-09-21T15:50:32.212 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:32.212 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.212 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.212 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:32.212 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150137+0000", 2024-09-21T15:50:32.213 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:32.213 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.213 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.213 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:32.213 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150140+0000", 2024-09-21T15:50:32.213 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2460000000000002e-06 2024-09-21T15:50:32.213 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.213 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.213 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:32.213 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150145+0000", 2024-09-21T15:50:32.213 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6520000000000003e-06 2024-09-21T15:50:32.213 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.213 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.214 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:32.214 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150150+0000", 2024-09-21T15:50:32.214 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5310000000000003e-06 2024-09-21T15:50:32.214 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.214 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.214 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:32.214 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200356+0000", 2024-09-21T15:50:32.214 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050205759000000003 2024-09-21T15:50:32.214 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.214 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.214 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:32.214 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200415+0000", 2024-09-21T15:50:32.215 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9719999999999997e-05 2024-09-21T15:50:32.215 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.215 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.215 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:32.215 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200483+0000", 2024-09-21T15:50:32.215 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7959000000000004e-05 2024-09-21T15:50:32.215 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:32.215 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:32.215 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:32.215 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.215 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.215 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62888 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 331776~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:32.215 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150154+0000", 2024-09-21T15:50:32.216 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 55.021773109999998, 2024-09-21T15:50:32.216 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050483169000000001, 2024-09-21T15:50:32.216 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:32.216 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:32.216 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:32.216 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:32.216 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:32.216 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62888 2024-09-21T15:50:32.216 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.216 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:32.216 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.216 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:32.216 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150154+0000", 2024-09-21T15:50:32.217 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:32.217 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.217 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.217 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:32.217 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150154+0000", 2024-09-21T15:50:32.217 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:32.217 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.217 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.217 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:32.217 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150153+0000", 2024-09-21T15:50:32.217 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:32.217 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.218 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.218 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:32.218 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150155+0000", 2024-09-21T15:50:32.218 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9920000000000002e-06 2024-09-21T15:50:32.218 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.218 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.218 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:32.218 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150161+0000", 2024-09-21T15:50:32.218 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5439999999999998e-06 2024-09-21T15:50:32.218 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.218 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.218 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:32.219 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150165+0000", 2024-09-21T15:50:32.219 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3460000000000004e-06 2024-09-21T15:50:32.219 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.219 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.219 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:32.219 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200502+0000", 2024-09-21T15:50:32.219 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050337150999999997 2024-09-21T15:50:32.219 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.219 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.219 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:32.219 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200575+0000", 2024-09-21T15:50:32.219 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2250999999999997e-05 2024-09-21T15:50:32.219 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.219 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.220 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:32.220 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200637+0000", 2024-09-21T15:50:32.220 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2195999999999999e-05 2024-09-21T15:50:32.220 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:32.220 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:32.220 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:32.220 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.220 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.220 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62889 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 339968~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:32.220 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150169+0000", 2024-09-21T15:50:32.220 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 55.021757610999998, 2024-09-21T15:50:32.220 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050629678999999997, 2024-09-21T15:50:32.220 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:32.220 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:32.221 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:32.221 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:32.221 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:32.221 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62889 2024-09-21T15:50:32.221 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.221 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:32.221 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.221 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:32.221 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150169+0000", 2024-09-21T15:50:32.221 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:32.221 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.221 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.221 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:32.221 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150169+0000", 2024-09-21T15:50:32.222 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:32.222 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.222 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.222 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:32.222 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150169+0000", 2024-09-21T15:50:32.222 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:32.222 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.222 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.222 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:32.222 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150171+0000", 2024-09-21T15:50:32.222 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3910000000000001e-06 2024-09-21T15:50:32.223 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.223 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.223 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:32.223 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150177+0000", 2024-09-21T15:50:32.223 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4079999999999997e-06 2024-09-21T15:50:32.223 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.223 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.223 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:32.223 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150182+0000", 2024-09-21T15:50:32.223 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3890000000000004e-06 2024-09-21T15:50:32.223 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.224 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.224 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:32.224 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200664+0000", 2024-09-21T15:50:32.224 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050482212999999998 2024-09-21T15:50:32.224 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.224 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.224 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:32.224 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200738+0000", 2024-09-21T15:50:32.224 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4114000000000002e-05 2024-09-21T15:50:32.224 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.224 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.224 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:32.225 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200799+0000", 2024-09-21T15:50:32.225 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0503e-05 2024-09-21T15:50:32.225 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:32.225 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:32.225 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:32.225 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.225 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.225 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62891 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 348160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:32.225 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150186+0000", 2024-09-21T15:50:32.225 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 55.021740469000001, 2024-09-21T15:50:32.225 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050794552, 2024-09-21T15:50:32.225 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:32.226 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:32.226 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:32.226 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:32.226 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:32.226 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62891 2024-09-21T15:50:32.226 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.226 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:32.226 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.226 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:32.226 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150186+0000", 2024-09-21T15:50:32.226 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:32.226 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.227 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.227 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:32.227 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150186+0000", 2024-09-21T15:50:32.227 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:32.227 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.227 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.227 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:32.227 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150186+0000", 2024-09-21T15:50:32.227 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:32.228 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.228 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.228 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:32.228 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150193+0000", 2024-09-21T15:50:32.228 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6830000000000002e-06 2024-09-21T15:50:32.228 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.228 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.228 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:32.228 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150199+0000", 2024-09-21T15:50:32.228 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9859999999999999e-06 2024-09-21T15:50:32.228 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.228 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.229 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:32.229 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150203+0000", 2024-09-21T15:50:32.229 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5920000000000002e-06 2024-09-21T15:50:32.229 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.229 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.229 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:32.229 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200835+0000", 2024-09-21T15:50:32.229 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050631783 2024-09-21T15:50:32.229 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.229 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.229 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:32.230 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200894+0000", 2024-09-21T15:50:32.230 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8560999999999997e-05 2024-09-21T15:50:32.230 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.230 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.230 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:32.230 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200981+0000", 2024-09-21T15:50:32.230 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7305999999999994e-05 2024-09-21T15:50:32.230 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:32.230 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:32.230 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:32.230 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.230 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.231 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62892 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 356352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:32.231 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150208+0000", 2024-09-21T15:50:32.231 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 55.021719081000001, 2024-09-21T15:50:32.231 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050951769000000001, 2024-09-21T15:50:32.231 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:32.231 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:32.231 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:32.231 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:32.231 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:32.231 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62892 2024-09-21T15:50:32.231 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.232 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:32.232 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.232 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:32.232 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150208+0000", 2024-09-21T15:50:32.232 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:32.232 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.232 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.232 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:32.232 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150208+0000", 2024-09-21T15:50:32.232 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:32.232 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.232 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.233 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:32.233 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150207+0000", 2024-09-21T15:50:32.233 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:32.233 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.233 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.233 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:32.233 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150210+0000", 2024-09-21T15:50:32.233 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.238e-06 2024-09-21T15:50:32.233 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.233 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.233 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:32.234 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150215+0000", 2024-09-21T15:50:32.234 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6350000000000003e-06 2024-09-21T15:50:32.234 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.234 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.234 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:32.234 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150220+0000", 2024-09-21T15:50:32.234 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.544e-06 2024-09-21T15:50:32.234 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.234 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.234 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:32.234 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201014+0000", 2024-09-21T15:50:32.235 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050793932999999999 2024-09-21T15:50:32.235 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.235 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.235 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:32.235 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201092+0000", 2024-09-21T15:50:32.235 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7961999999999997e-05 2024-09-21T15:50:32.235 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.235 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.235 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:32.235 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201160+0000", 2024-09-21T15:50:32.235 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7836000000000006e-05 2024-09-21T15:50:32.236 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:32.236 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:32.236 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:32.236 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.236 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.236 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62894 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 364544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:32.236 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150820+0000", 2024-09-21T15:50:32.236 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 55.021106881000001, 2024-09-21T15:50:32.236 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050489196, 2024-09-21T15:50:32.236 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:32.236 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:32.236 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:32.236 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:32.237 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:32.237 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62894 2024-09-21T15:50:32.237 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.237 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:32.237 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.237 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:32.237 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150820+0000", 2024-09-21T15:50:32.237 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:32.237 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.237 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.237 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:32.238 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150820+0000", 2024-09-21T15:50:32.238 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:32.238 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.238 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.238 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:32.238 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150223+0000", 2024-09-21T15:50:32.238 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9994035 2024-09-21T15:50:32.238 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.238 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.238 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:32.238 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150825+0000", 2024-09-21T15:50:32.238 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00060137099999999996 2024-09-21T15:50:32.239 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.239 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.239 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:32.239 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150837+0000", 2024-09-21T15:50:32.239 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2503999999999999e-05 2024-09-21T15:50:32.239 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.239 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.239 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:32.239 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150844+0000", 2024-09-21T15:50:32.239 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6900000000000003e-06 2024-09-21T15:50:32.239 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.239 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.239 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:32.240 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201187+0000", 2024-09-21T15:50:32.240 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050342973999999999 2024-09-21T15:50:32.240 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.240 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.240 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:32.240 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201250+0000", 2024-09-21T15:50:32.240 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3274000000000004e-05 2024-09-21T15:50:32.240 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.240 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.240 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:32.240 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201309+0000", 2024-09-21T15:50:32.240 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9058000000000003e-05 2024-09-21T15:50:32.241 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:32.241 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:32.241 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:32.241 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.241 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.241 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62895 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 380928~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:32.241 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150851+0000", 2024-09-21T15:50:32.241 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 55.021076284999999, 2024-09-21T15:50:32.241 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050611097000000001, 2024-09-21T15:50:32.241 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:32.241 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:32.241 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:32.242 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:32.242 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:32.242 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62895 2024-09-21T15:50:32.242 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.242 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:32.242 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.242 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:32.242 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150851+0000", 2024-09-21T15:50:32.242 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:32.242 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.242 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.242 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:32.242 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150851+0000", 2024-09-21T15:50:32.243 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:32.243 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.243 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.243 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:32.243 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150850+0000", 2024-09-21T15:50:32.243 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:32.243 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.243 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.243 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:32.243 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150853+0000", 2024-09-21T15:50:32.243 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1039999999999998e-06 2024-09-21T15:50:32.243 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.243 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.244 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:32.244 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150860+0000", 2024-09-21T15:50:32.244 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3180000000000001e-06 2024-09-21T15:50:32.244 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.244 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.244 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:32.244 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150866+0000", 2024-09-21T15:50:32.244 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1599999999999997e-06 2024-09-21T15:50:32.244 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.244 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.244 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:32.244 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201332+0000", 2024-09-21T15:50:32.244 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050466100999999999 2024-09-21T15:50:32.245 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.245 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.245 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:32.245 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201392+0000", 2024-09-21T15:50:32.245 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0053999999999998e-05 2024-09-21T15:50:32.245 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.245 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.245 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:32.245 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201462+0000", 2024-09-21T15:50:32.245 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9861000000000001e-05 2024-09-21T15:50:32.245 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:32.245 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:32.245 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:32.246 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.246 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.246 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62896 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 372736~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:32.246 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150870+0000", 2024-09-21T15:50:32.246 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 55.021056504999997, 2024-09-21T15:50:32.246 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050753292999999998, 2024-09-21T15:50:32.246 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:32.246 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:32.246 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:32.246 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:32.246 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:32.246 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62896 2024-09-21T15:50:32.246 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.247 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:32.247 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.247 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:32.247 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150870+0000", 2024-09-21T15:50:32.247 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:32.247 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.247 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.247 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:32.247 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150870+0000", 2024-09-21T15:50:32.247 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:32.247 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.247 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.247 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:32.247 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150870+0000", 2024-09-21T15:50:32.248 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:32.248 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.248 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.248 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:32.248 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150872+0000", 2024-09-21T15:50:32.248 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.644e-06 2024-09-21T15:50:32.248 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.248 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.248 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:32.248 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150879+0000", 2024-09-21T15:50:32.248 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3509999999999998e-06 2024-09-21T15:50:32.248 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.248 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.249 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:32.249 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150884+0000", 2024-09-21T15:50:32.249 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9150000000000004e-06 2024-09-21T15:50:32.249 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.249 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.249 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:32.249 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201487+0000", 2024-09-21T15:50:32.249 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050603276000000003 2024-09-21T15:50:32.249 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.249 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.249 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:32.249 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201557+0000", 2024-09-21T15:50:32.249 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9847999999999999e-05 2024-09-21T15:50:32.249 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.250 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.250 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:32.250 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201624+0000", 2024-09-21T15:50:32.250 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6730999999999995e-05 2024-09-21T15:50:32.250 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:32.250 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:32.250 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:32.250 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.250 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.250 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62897 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 389120~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:32.250 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.151044+0000", 2024-09-21T15:50:32.250 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 55.020882465, 2024-09-21T15:50:32.251 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050720531999999999, 2024-09-21T15:50:32.251 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:32.251 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:32.251 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:32.251 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:32.251 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:32.251 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62897 2024-09-21T15:50:32.251 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.251 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:32.251 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.251 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:32.251 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151044+0000", 2024-09-21T15:50:32.251 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:32.252 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.252 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.252 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:32.252 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151044+0000", 2024-09-21T15:50:32.252 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:32.252 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.252 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.252 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:32.252 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150888+0000", 2024-09-21T15:50:32.252 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9998431 2024-09-21T15:50:32.252 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.252 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.252 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:32.253 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151048+0000", 2024-09-21T15:50:32.253 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00016077299999999999 2024-09-21T15:50:32.253 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.253 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.253 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:32.253 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151058+0000", 2024-09-21T15:50:32.253 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.465e-06 2024-09-21T15:50:32.253 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.253 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.253 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:32.253 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151064+0000", 2024-09-21T15:50:32.253 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3829999999999997e-06 2024-09-21T15:50:32.254 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.254 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.254 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:32.254 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201648+0000", 2024-09-21T15:50:32.254 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050583597000000001 2024-09-21T15:50:32.254 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.254 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.254 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:32.254 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201711+0000", 2024-09-21T15:50:32.254 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3616999999999998e-05 2024-09-21T15:50:32.254 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.255 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.255 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:32.255 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201765+0000", 2024-09-21T15:50:32.255 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3464000000000002e-05 2024-09-21T15:50:32.255 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:32.255 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:32.255 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:32.255 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.255 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.255 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:118351 3.4 3:25980f86:::rbd_data.1133545d3fa6.000000000000005a:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-09-21T15:50:32.255 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:58.975795+0000", 2024-09-21T15:50:32.255 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 33.196131831000002, 2024-09-21T15:50:32.255 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.99016351599999997, 2024-09-21T15:50:32.256 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:32.256 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:32.256 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:32.256 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:32.256 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:32.256 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 118351 2024-09-21T15:50:32.256 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.256 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:32.256 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.256 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:32.256 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975795+0000", 2024-09-21T15:50:32.256 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:32.256 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.256 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.257 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:32.257 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975795+0000", 2024-09-21T15:50:32.257 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:32.257 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.257 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.257 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:32.257 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975793+0000", 2024-09-21T15:50:32.257 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:50:32.257 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.257 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.257 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:32.257 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975800+0000", 2024-09-21T15:50:32.257 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8220000000000001e-06 2024-09-21T15:50:32.257 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.258 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.258 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:32.258 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975814+0000", 2024-09-21T15:50:32.258 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3893000000000001e-05 2024-09-21T15:50:32.258 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.258 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.258 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:32.258 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975821+0000", 2024-09-21T15:50:32.258 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1319999999999996e-06 2024-09-21T15:50:32.258 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.258 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.259 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:32.259 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975878+0000", 2024-09-21T15:50:32.259 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7238999999999999e-05 2024-09-21T15:50:32.259 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.259 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.259 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:50:32.259 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975889+0000", 2024-09-21T15:50:32.259 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0509000000000001e-05 2024-09-21T15:50:32.259 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.259 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.259 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:32.259 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.964573+0000", 2024-09-21T15:50:32.259 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.98868396300000005 2024-09-21T15:50:32.259 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.260 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.260 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:32.260 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.964672+0000", 2024-09-21T15:50:32.260 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9601999999999999e-05 2024-09-21T15:50:32.260 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.260 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.260 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-09-21T15:50:32.260 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.964758+0000", 2024-09-21T15:50:32.260 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5537000000000005e-05 2024-09-21T15:50:32.260 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.260 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.260 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:32.260 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965847+0000", 2024-09-21T15:50:32.260 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001089404 2024-09-21T15:50:32.261 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.261 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.261 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:32.261 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965890+0000", 2024-09-21T15:50:32.261 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2521999999999997e-05 2024-09-21T15:50:32.261 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.261 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.261 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:32.261 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965934+0000", 2024-09-21T15:50:32.261 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4489e-05 2024-09-21T15:50:32.261 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.261 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.261 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:32.261 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965959+0000", 2024-09-21T15:50:32.262 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4301999999999999e-05 2024-09-21T15:50:32.262 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:32.262 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:32.262 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:32.262 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.262 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.262 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:118353 3.4 3:23841e18:::rbd_data.1133545d3fa6.0000000000000061:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-09-21T15:50:32.262 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:58.975836+0000", 2024-09-21T15:50:32.262 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 33.196090331999997, 2024-09-21T15:50:32.262 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.99077746700000002, 2024-09-21T15:50:32.262 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:32.262 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:32.262 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:32.262 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:32.263 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:32.263 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 118353 2024-09-21T15:50:32.263 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.263 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:32.263 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.263 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:32.263 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975836+0000", 2024-09-21T15:50:32.263 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:32.263 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.263 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.263 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:32.263 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975836+0000", 2024-09-21T15:50:32.263 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:32.263 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.264 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.264 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:32.264 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975836+0000", 2024-09-21T15:50:32.264 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:32.264 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.264 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.264 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:32.264 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975839+0000", 2024-09-21T15:50:32.264 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6129999999999998e-06 2024-09-21T15:50:32.264 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.264 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.264 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:32.265 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975844+0000", 2024-09-21T15:50:32.265 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5130000000000004e-06 2024-09-21T15:50:32.265 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.265 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.265 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:32.265 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975850+0000", 2024-09-21T15:50:32.265 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3179999999999997e-06 2024-09-21T15:50:32.265 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.265 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.265 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:32.265 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975915+0000", 2024-09-21T15:50:32.265 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5168999999999999e-05 2024-09-21T15:50:32.266 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.266 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.266 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:50:32.266 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975922+0000", 2024-09-21T15:50:32.266 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2749999999999998e-06 2024-09-21T15:50:32.266 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.266 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.266 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:32.266 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965431+0000", 2024-09-21T15:50:32.266 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.98950864699999996 2024-09-21T15:50:32.266 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.266 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.266 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:32.267 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965535+0000", 2024-09-21T15:50:32.267 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000103949 2024-09-21T15:50:32.267 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.267 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.267 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-09-21T15:50:32.267 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965607+0000", 2024-09-21T15:50:32.267 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2478000000000006e-05 2024-09-21T15:50:32.267 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.267 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.267 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:32.267 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.966205+0000", 2024-09-21T15:50:32.267 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00059775200000000001 2024-09-21T15:50:32.267 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.267 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.268 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:32.268 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.966562+0000", 2024-09-21T15:50:32.268 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00035732199999999998 2024-09-21T15:50:32.268 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.268 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.268 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:32.268 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.966593+0000", 2024-09-21T15:50:32.268 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0642999999999998e-05 2024-09-21T15:50:32.268 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.268 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.268 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:32.268 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.966614+0000", 2024-09-21T15:50:32.268 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1149999999999999e-05 2024-09-21T15:50:32.268 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:32.269 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:32.269 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:32.269 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.269 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.269 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160535 3.6 3:67cae2ed:::rbd_mirroring:head [list-watchers] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-09-21T15:50:32.269 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:30.195408+0000", 2024-09-21T15:50:32.269 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.9765192499999999, 2024-09-21T15:50:32.269 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.96025944900000004, 2024-09-21T15:50:32.269 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:32.269 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-09-21T15:50:32.269 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:32.269 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:32.269 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:32.269 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160535 2024-09-21T15:50:32.270 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.270 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:32.270 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.270 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:32.270 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195408+0000", 2024-09-21T15:50:32.270 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:32.270 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.270 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.270 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:32.270 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195408+0000", 2024-09-21T15:50:32.270 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:32.270 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.270 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.271 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:32.271 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195406+0000", 2024-09-21T15:50:32.271 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:50:32.271 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.271 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.271 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:32.271 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195414+0000", 2024-09-21T15:50:32.271 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.2870000000000004e-06 2024-09-21T15:50:32.271 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.271 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.271 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:32.271 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195432+0000", 2024-09-21T15:50:32.271 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8110000000000001e-05 2024-09-21T15:50:32.271 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.272 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.272 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:32.272 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195441+0000", 2024-09-21T15:50:32.272 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7679999999999997e-06 2024-09-21T15:50:32.272 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.272 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.272 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:32.272 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195505+0000", 2024-09-21T15:50:32.272 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3769000000000006e-05 2024-09-21T15:50:32.272 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.272 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.272 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:50:32.272 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195518+0000", 2024-09-21T15:50:32.272 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3341e-05 2024-09-21T15:50:32.273 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.273 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.273 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:32.273 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.155509+0000", 2024-09-21T15:50:32.273 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.959990393 2024-09-21T15:50:32.273 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:32.273 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:32.273 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:32.273 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.155667+0000", 2024-09-21T15:50:32.273 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000158505 2024-09-21T15:50:32.273 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:32.273 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:32.273 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:32.273 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:32.274 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:32.274 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:32.275 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-21T15:50:32.276 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:50:32.276 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:50:32.317 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:32.315+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:32.378 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:32.377+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:32.425 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:32.425 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:32.426 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:50:32.426 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:32.435 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:50:32.444 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-09-21T15:50:32.444 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:50:32.462 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:50:32.463 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:50:32.479 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:32.478+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:32.493 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:50:32.494 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:50:32.509 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:50:32.510 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:50:32.526 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:50:32.526 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:50:32.542 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:50:32.543 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:50:32.549 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:32.549 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:32.558 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:50:32.559 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:50:32.560 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:50:32.576 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:50:32.576 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:50:32.579 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:32.578+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:32.662 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:32.662 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:32.668 INFO:teuthology.orchestra.run.smithi089.stderr:reweighted osd.3 to 1 (10000) 2024-09-21T15:50:32.675 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:50:32.680 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:32.679+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:32.690 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-21T15:50:32.780 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:32.779+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:32.792 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:32.792 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:32.801 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:50:32.882 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:32.880+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:32.945 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:32.946 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:32.955 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:50:32.998 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:32.997+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:33.058 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:33.058 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:33.066 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:50:33.110 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:33.109+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:33.170 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T15:50:33.180 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:33.180 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:33.190 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:50:33.214 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:50:33.213+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:50:33.285 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:33.284+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:33.333 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:33.333 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:33.342 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:50:33.425 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:33.424+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:33.484 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:33.485 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:33.487 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:33.486+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:33.493 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:50:33.588 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:50:33.586+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:50:33.636 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:33.636 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:33.645 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:50:33.669 INFO:teuthology.orchestra.run.smithi089.stderr:reweighted osd.4 to 1 (10000) 2024-09-21T15:50:33.669 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is set 2024-09-21T15:50:33.684 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-21T15:50:33.725 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:33.724+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:33.748 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:33.748 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:33.756 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:50:33.791 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:33.790+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:33.858 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:33.858 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:33.867 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T15:50:33.910 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:33.909+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:33.969 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:33.969 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:33.978 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T15:50:34.019 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:34.018+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:34.093 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:34.092+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:34.120 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:34.120 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:34.129 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T15:50:34.194 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:34.193+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:34.237 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:34.237 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:34.246 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T15:50:34.295 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:34.294+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:34.348 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:34.348 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:34.356 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T15:50:34.400 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:34.399+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:34.458 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:34.458 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:34.466 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T15:50:34.507 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:34.506+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:34.588 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:50:34.598 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:34.597+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:34.608 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:34.609 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:34.617 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:50:34.672 INFO:teuthology.orchestra.run.smithi089.stderr:reweighted osd.5 to 1 (10000) 2024-09-21T15:50:34.694 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-09-21T15:50:34.699 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:34.698+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:34.717 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-09-21T15:50:34.718 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:50:34.736 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:50:34.736 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:50:34.750 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:34.750 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:34.750 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:50:34.750 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:34.753 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:50:34.753 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:50:34.759 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:50:34.769 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:50:34.769 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:50:34.785 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:50:34.785 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:50:34.799 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:34.798+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:34.802 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:50:34.802 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:50:34.818 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:50:34.819 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:50:34.839 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:50:34.839 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:50:34.875 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:34.876 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:34.876 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:50:34.876 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:50:34.876 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:34.885 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:50:34.900 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:34.899+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:35.001 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:35.000+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:35.049 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:35.049 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:50:35.049 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:50:35.049 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:50:35.049 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.049 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160505 3.b 3:d297695c:::rbd_header.11338bae974a:head [watch reconnect cookie 140381478598448 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-21T15:50:35.049 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:29.138810+0000", 2024-09-21T15:50:35.049 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.8897718179999998, 2024-09-21T15:50:35.050 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0172598319999999, 2024-09-21T15:50:35.050 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:35.050 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:35.050 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:35.050 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:35.050 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:35.050 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160505 2024-09-21T15:50:35.050 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.050 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:35.050 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.050 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:35.050 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138810+0000", 2024-09-21T15:50:35.051 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.051 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.051 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:35.051 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138810+0000", 2024-09-21T15:50:35.051 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.051 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.051 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:35.051 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138808+0000", 2024-09-21T15:50:35.051 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:50:35.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.051 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.051 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:35.051 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138815+0000", 2024-09-21T15:50:35.052 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6549999999999998e-06 2024-09-21T15:50:35.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.052 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.052 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:35.052 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138825+0000", 2024-09-21T15:50:35.052 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9790000000000005e-06 2024-09-21T15:50:35.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.052 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.052 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:35.052 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138834+0000", 2024-09-21T15:50:35.052 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5179999999999993e-06 2024-09-21T15:50:35.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.052 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.052 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:35.053 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.140973+0000", 2024-09-21T15:50:35.053 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0021392049999999999 2024-09-21T15:50:35.053 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.053 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.053 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:50:35.053 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.140981+0000", 2024-09-21T15:50:35.053 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8909999999999995e-06 2024-09-21T15:50:35.053 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.053 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.053 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:35.053 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.150457+0000", 2024-09-21T15:50:35.053 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.009475645 2024-09-21T15:50:35.053 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.053 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.054 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for missing object", 2024-09-21T15:50:35.054 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.150589+0000", 2024-09-21T15:50:35.054 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000131779 2024-09-21T15:50:35.054 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.054 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.054 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:35.054 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.151723+0000", 2024-09-21T15:50:35.054 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0011345979999999999 2024-09-21T15:50:35.054 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.054 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.054 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-09-21T15:50:35.054 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.151872+0000", 2024-09-21T15:50:35.054 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000149021 2024-09-21T15:50:35.054 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.054 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.055 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:35.055 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.152563+0000", 2024-09-21T15:50:35.055 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00069051500000000005 2024-09-21T15:50:35.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.055 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.055 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:35.055 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.152673+0000", 2024-09-21T15:50:35.055 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000109795 2024-09-21T15:50:35.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.055 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.055 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:35.055 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.156069+0000", 2024-09-21T15:50:35.055 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003396736 2024-09-21T15:50:35.055 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.056 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:35.056 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.056 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.056 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160506 3.b 3:d297695c:::rbd_header.11338bae974a:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-21T15:50:35.056 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:29.138844+0000", 2024-09-21T15:50:35.056 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.8897369309999998, 2024-09-21T15:50:35.056 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0174592680000001, 2024-09-21T15:50:35.056 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:35.056 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:35.056 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:35.056 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:35.056 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:35.056 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160506 2024-09-21T15:50:35.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.057 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:35.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.057 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:35.057 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138844+0000", 2024-09-21T15:50:35.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.057 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:35.057 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138844+0000", 2024-09-21T15:50:35.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.057 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:35.057 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138844+0000", 2024-09-21T15:50:35.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:35.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.058 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:35.058 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138848+0000", 2024-09-21T15:50:35.058 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.76e-06 2024-09-21T15:50:35.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.058 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:35.058 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138854+0000", 2024-09-21T15:50:35.058 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3890000000000002e-06 2024-09-21T15:50:35.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.058 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:35.058 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138860+0000", 2024-09-21T15:50:35.058 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6409999999999999e-06 2024-09-21T15:50:35.059 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.059 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.059 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:35.059 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.141015+0000", 2024-09-21T15:50:35.059 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0021553779999999999 2024-09-21T15:50:35.059 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.059 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.059 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:50:35.059 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.141021+0000", 2024-09-21T15:50:35.059 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7629999999999997e-06 2024-09-21T15:50:35.059 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.059 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.059 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:35.059 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.150858+0000", 2024-09-21T15:50:35.060 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0098372200000001 2024-09-21T15:50:35.060 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.060 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.060 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for missing object", 2024-09-21T15:50:35.060 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.150899+0000", 2024-09-21T15:50:35.060 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1112000000000002e-05 2024-09-21T15:50:35.060 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.060 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.060 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:35.060 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.152069+0000", 2024-09-21T15:50:35.060 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001170307 2024-09-21T15:50:35.060 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.060 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.060 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-09-21T15:50:35.060 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.152192+0000", 2024-09-21T15:50:35.061 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012289700000000001 2024-09-21T15:50:35.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.061 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:35.061 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.153362+0000", 2024-09-21T15:50:35.061 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0011700549999999999 2024-09-21T15:50:35.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.061 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:35.061 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.153457+0000", 2024-09-21T15:50:35.061 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4687999999999999e-05 2024-09-21T15:50:35.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.061 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:50:35.061 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.153893+0000", 2024-09-21T15:50:35.062 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000436363 2024-09-21T15:50:35.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.062 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:35.062 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.156134+0000", 2024-09-21T15:50:35.062 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0022410009999999998 2024-09-21T15:50:35.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.062 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:35.062 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.156148+0000", 2024-09-21T15:50:35.062 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3574000000000001e-05 2024-09-21T15:50:35.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.062 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:35.062 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.156214+0000", 2024-09-21T15:50:35.062 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6438000000000003e-05 2024-09-21T15:50:35.063 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.063 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.063 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:35.063 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.156304+0000", 2024-09-21T15:50:35.063 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9278999999999997e-05 2024-09-21T15:50:35.063 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.063 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:35.063 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.063 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.063 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.063 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "MOSDPGPush(3.b 94/93 [PushOp(3:d40b7a9f:::rbd_data.11333a557424.000000000000001d:head, version: 70'684, data_included: [235520~12288,286720~16384,339968~12288,393216~10240], data_size: 51200, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(3:d40b7a9f:::rbd_data.11333a557424.000000000000001d:head@70'684, size: 403456, copy_subset: [0~403456], clone_subset: {}, snapset: 0=[]:{}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:403456, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:false, error:false))])", 2024-09-21T15:50:35.063 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:30.152698+0000", 2024-09-21T15:50:35.063 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.875883805, 2024-09-21T15:50:35.063 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003011138, 2024-09-21T15:50:35.064 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:35.064 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:35.064 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:35.064 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.064 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:35.064 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.152698+0000", 2024-09-21T15:50:35.064 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.064 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.064 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.064 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:35.064 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.152698+0000", 2024-09-21T15:50:35.064 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.064 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.064 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.064 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:35.065 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.152697+0000", 2024-09-21T15:50:35.065 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:35.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.065 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:35.065 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.152727+0000", 2024-09-21T15:50:35.065 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9989e-05 2024-09-21T15:50:35.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.065 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:35.065 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.152747+0000", 2024-09-21T15:50:35.065 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0514e-05 2024-09-21T15:50:35.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.066 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:35.066 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.152775+0000", 2024-09-21T15:50:35.066 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7934000000000001e-05 2024-09-21T15:50:35.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.066 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.066 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:35.066 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.154532+0000", 2024-09-21T15:50:35.066 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0017566400000000001 2024-09-21T15:50:35.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.066 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.066 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:35.066 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.154546+0000", 2024-09-21T15:50:35.067 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3995000000000001e-05 2024-09-21T15:50:35.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.067 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:35.067 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.155709+0000", 2024-09-21T15:50:35.067 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001162819 2024-09-21T15:50:35.067 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.067 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:35.067 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.067 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160512 3.b 3:d297695c:::rbd_header.11338bae974a:head [watch unwatch cookie 140381478598448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T15:50:35.067 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:30.161362+0000", 2024-09-21T15:50:35.067 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.8672191229999999, 2024-09-21T15:50:35.067 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0074105029999999997, 2024-09-21T15:50:35.068 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:35.068 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:35.068 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:35.068 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:35.068 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:35.068 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160512 2024-09-21T15:50:35.068 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.068 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:35.068 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.068 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:35.068 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161362+0000", 2024-09-21T15:50:35.068 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.068 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.068 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.069 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:35.069 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161362+0000", 2024-09-21T15:50:35.069 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.069 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:35.069 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161362+0000", 2024-09-21T15:50:35.069 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:35.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.069 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:35.069 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161365+0000", 2024-09-21T15:50:35.069 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8669999999999998e-06 2024-09-21T15:50:35.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.070 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:35.070 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161371+0000", 2024-09-21T15:50:35.070 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8009999999999998e-06 2024-09-21T15:50:35.070 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.070 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:35.070 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161376+0000", 2024-09-21T15:50:35.070 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0340000000000004e-06 2024-09-21T15:50:35.070 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.070 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:35.070 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161762+0000", 2024-09-21T15:50:35.070 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038575099999999999 2024-09-21T15:50:35.070 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.070 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:35.071 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161838+0000", 2024-09-21T15:50:35.071 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5697000000000004e-05 2024-09-21T15:50:35.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.071 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:50:35.071 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161952+0000", 2024-09-21T15:50:35.071 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000114034 2024-09-21T15:50:35.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.071 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:35.071 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.163727+0000", 2024-09-21T15:50:35.071 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0017749560000000001 2024-09-21T15:50:35.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.072 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:35.072 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168545+0000", 2024-09-21T15:50:35.072 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0048186080000000003 2024-09-21T15:50:35.072 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.072 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.072 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:35.072 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168593+0000", 2024-09-21T15:50:35.072 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7549999999999997e-05 2024-09-21T15:50:35.072 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.072 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.072 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:35.072 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168773+0000", 2024-09-21T15:50:35.072 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00017976600000000001 2024-09-21T15:50:35.072 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.072 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:35.073 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.073 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.073 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.073 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160513 3.b 3:d297695c:::rbd_header.11338bae974a:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-09-21T15:50:35.073 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:30.161755+0000", 2024-09-21T15:50:35.073 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.8668261919999996, 2024-09-21T15:50:35.073 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0072250300000000003, 2024-09-21T15:50:35.073 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:35.073 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:35.073 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:35.073 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:35.073 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:35.073 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160513 2024-09-21T15:50:35.073 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.074 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:35.074 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.074 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:35.074 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161755+0000", 2024-09-21T15:50:35.074 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.074 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.074 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.074 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:35.074 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161755+0000", 2024-09-21T15:50:35.074 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.074 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.074 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.074 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:35.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161754+0000", 2024-09-21T15:50:35.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:35.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:35.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161759+0000", 2024-09-21T15:50:35.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9699999999999998e-06 2024-09-21T15:50:35.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:35.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161768+0000", 2024-09-21T15:50:35.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9949999999999994e-06 2024-09-21T15:50:35.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.076 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:35.076 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161774+0000", 2024-09-21T15:50:35.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1290000000000001e-06 2024-09-21T15:50:35.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.076 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:35.076 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.162239+0000", 2024-09-21T15:50:35.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00046406600000000001 2024-09-21T15:50:35.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.077 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.077 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-09-21T15:50:35.077 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.162296+0000", 2024-09-21T15:50:35.077 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7122e-05 2024-09-21T15:50:35.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.077 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.077 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:35.077 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168802+0000", 2024-09-21T15:50:35.077 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0065061700000000004 2024-09-21T15:50:35.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.077 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.077 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:35.077 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168877+0000", 2024-09-21T15:50:35.077 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5214000000000002e-05 2024-09-21T15:50:35.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.078 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:35.078 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168980+0000", 2024-09-21T15:50:35.078 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000103182 2024-09-21T15:50:35.078 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.078 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:35.078 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.078 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.078 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160527 3.b 3:d297695c:::rbd_header.11338bae974a:head [watch watch cookie 140381476148256] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T15:50:35.078 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:30.182756+0000", 2024-09-21T15:50:35.078 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.8458256110000004, 2024-09-21T15:50:35.078 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0054202579999999998, 2024-09-21T15:50:35.079 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:35.079 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:35.079 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:35.079 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:35.079 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:35.079 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160527 2024-09-21T15:50:35.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.079 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:35.079 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.079 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:35.079 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182756+0000", 2024-09-21T15:50:35.079 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.079 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.080 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:35.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182756+0000", 2024-09-21T15:50:35.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.080 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:35.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182755+0000", 2024-09-21T15:50:35.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:35.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.080 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:35.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182760+0000", 2024-09-21T15:50:35.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4269999999999999e-06 2024-09-21T15:50:35.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.081 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:35.081 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182771+0000", 2024-09-21T15:50:35.081 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0312999999999999e-05 2024-09-21T15:50:35.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.081 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:35.081 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182778+0000", 2024-09-21T15:50:35.081 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5700000000000004e-06 2024-09-21T15:50:35.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.081 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:35.081 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.184745+0000", 2024-09-21T15:50:35.081 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0019667170000000002 2024-09-21T15:50:35.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.082 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.082 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:35.082 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.184867+0000", 2024-09-21T15:50:35.082 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000122504 2024-09-21T15:50:35.082 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.082 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.082 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:50:35.082 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.185135+0000", 2024-09-21T15:50:35.082 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000267135 2024-09-21T15:50:35.082 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.082 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.082 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:35.082 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.187721+0000", 2024-09-21T15:50:35.082 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002586891 2024-09-21T15:50:35.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.083 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:35.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.187874+0000", 2024-09-21T15:50:35.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000152726 2024-09-21T15:50:35.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.083 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:35.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.187951+0000", 2024-09-21T15:50:35.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7132000000000006e-05 2024-09-21T15:50:35.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.083 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:35.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.188176+0000", 2024-09-21T15:50:35.084 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00022477900000000001 2024-09-21T15:50:35.084 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.084 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:35.084 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.084 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160529 3.b 3:d297695c:::rbd_header.11338bae974a:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T15:50:35.084 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:30.188676+0000", 2024-09-21T15:50:35.084 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.8399051679999996, 2024-09-21T15:50:35.084 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035442500000000001, 2024-09-21T15:50:35.084 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:35.084 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:35.084 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:35.084 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:35.085 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:35.085 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160529 2024-09-21T15:50:35.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.085 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:35.085 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.085 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:35.085 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.188676+0000", 2024-09-21T15:50:35.085 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.085 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.085 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:35.085 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.188676+0000", 2024-09-21T15:50:35.085 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.085 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.086 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:35.086 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.188676+0000", 2024-09-21T15:50:35.086 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:35.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.086 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:35.086 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.188679+0000", 2024-09-21T15:50:35.086 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7790000000000002e-06 2024-09-21T15:50:35.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.086 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:35.086 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.188687+0000", 2024-09-21T15:50:35.086 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1300000000000003e-06 2024-09-21T15:50:35.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.087 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:35.087 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.188691+0000", 2024-09-21T15:50:35.087 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3069999999999996e-06 2024-09-21T15:50:35.087 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.087 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.087 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:35.087 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.188829+0000", 2024-09-21T15:50:35.087 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013851799999999999 2024-09-21T15:50:35.087 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.087 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.087 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:35.087 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.188942+0000", 2024-09-21T15:50:35.087 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011276400000000001 2024-09-21T15:50:35.087 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.087 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.088 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:50:35.088 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.189208+0000", 2024-09-21T15:50:35.088 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00026575500000000001 2024-09-21T15:50:35.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.088 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:35.088 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.190170+0000", 2024-09-21T15:50:35.088 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00096190299999999996 2024-09-21T15:50:35.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.088 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:35.088 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.192075+0000", 2024-09-21T15:50:35.088 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001905672 2024-09-21T15:50:35.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.089 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.089 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:35.089 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.192113+0000", 2024-09-21T15:50:35.089 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7410000000000003e-05 2024-09-21T15:50:35.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.089 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.089 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:35.089 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.192220+0000", 2024-09-21T15:50:35.089 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000107611 2024-09-21T15:50:35.089 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.089 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:35.089 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.089 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.089 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160554 3.b 3:d297695c:::rbd_header.11338bae974a:head [watch watch cookie 140381476148256] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-21T15:50:35.089 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.171599+0000", 2024-09-21T15:50:35.089 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.8569828189999997, 2024-09-21T15:50:35.090 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035069400000000001, 2024-09-21T15:50:35.090 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:35.090 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:35.090 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:35.090 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:35.090 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:35.090 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160554 2024-09-21T15:50:35.090 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.090 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:35.090 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.090 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:35.090 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.171599+0000", 2024-09-21T15:50:35.090 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.090 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.091 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:35.091 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.171599+0000", 2024-09-21T15:50:35.091 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.091 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:35.091 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.171598+0000", 2024-09-21T15:50:35.091 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:35.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.091 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:35.091 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.171602+0000", 2024-09-21T15:50:35.091 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4109999999999998e-06 2024-09-21T15:50:35.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.092 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:35.092 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.171612+0000", 2024-09-21T15:50:35.092 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.961e-06 2024-09-21T15:50:35.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.092 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:35.092 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.171620+0000", 2024-09-21T15:50:35.092 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4610000000000002e-06 2024-09-21T15:50:35.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.092 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:35.092 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.171646+0000", 2024-09-21T15:50:35.092 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6455999999999999e-05 2024-09-21T15:50:35.093 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.093 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.093 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:35.093 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.171703+0000", 2024-09-21T15:50:35.093 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7411000000000003e-05 2024-09-21T15:50:35.093 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.093 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.093 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:50:35.093 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.171800+0000", 2024-09-21T15:50:35.093 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6825999999999998e-05 2024-09-21T15:50:35.093 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.093 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.093 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:35.093 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.175002+0000", 2024-09-21T15:50:35.093 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032012339999999999 2024-09-21T15:50:35.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.094 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.094 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:35.094 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.175016+0000", 2024-09-21T15:50:35.094 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4191e-05 2024-09-21T15:50:35.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.094 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.094 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:35.094 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.175046+0000", 2024-09-21T15:50:35.094 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.061e-05 2024-09-21T15:50:35.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.094 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.094 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:35.094 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.175106+0000", 2024-09-21T15:50:35.095 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9175999999999997e-05 2024-09-21T15:50:35.095 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.095 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:35.095 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.095 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.095 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:160568 3.18 e95/94 3:1feb8588:::rbd_id.image_client.0-clone41:head v 95'405, mlcod=0'0)", 2024-09-21T15:50:35.095 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.186644+0000", 2024-09-21T15:50:35.095 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.8419369649999999, 2024-09-21T15:50:35.095 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032131130000000001, 2024-09-21T15:50:35.095 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:35.095 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:35.095 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:35.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.095 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:35.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.186644+0000", 2024-09-21T15:50:35.096 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:35.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.186644+0000", 2024-09-21T15:50:35.096 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:35.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.186644+0000", 2024-09-21T15:50:35.096 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:35.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:35.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.186650+0000", 2024-09-21T15:50:35.096 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4659999999999996e-06 2024-09-21T15:50:35.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.097 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:35.097 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.186660+0000", 2024-09-21T15:50:35.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6709999999999994e-06 2024-09-21T15:50:35.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.097 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:35.097 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.186666+0000", 2024-09-21T15:50:35.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6779999999999997e-06 2024-09-21T15:50:35.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.097 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:35.097 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.186713+0000", 2024-09-21T15:50:35.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7580999999999997e-05 2024-09-21T15:50:35.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:35.098 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.186728+0000", 2024-09-21T15:50:35.098 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5003e-05 2024-09-21T15:50:35.098 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:35.098 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.189825+0000", 2024-09-21T15:50:35.098 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0030967070000000002 2024-09-21T15:50:35.098 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:35.098 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.189858+0000", 2024-09-21T15:50:35.098 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2657999999999998e-05 2024-09-21T15:50:35.098 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.098 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:35.099 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.099 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160586 3.b 3:d297695c:::rbd_header.11338bae974a:head [watch watch cookie 140381476029296] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-21T15:50:35.099 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.201587+0000", 2024-09-21T15:50:35.099 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.8269942509999999, 2024-09-21T15:50:35.099 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0040745900000000003, 2024-09-21T15:50:35.099 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:35.099 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:35.099 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:35.099 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:35.099 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:35.099 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160586 2024-09-21T15:50:35.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.100 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:35.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.100 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:35.100 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.201587+0000", 2024-09-21T15:50:35.100 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.100 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:35.100 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.201587+0000", 2024-09-21T15:50:35.100 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.100 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:35.100 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.201586+0000", 2024-09-21T15:50:35.100 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:35.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.101 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:35.101 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.201592+0000", 2024-09-21T15:50:35.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7910000000000001e-06 2024-09-21T15:50:35.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.101 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:35.101 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.201603+0000", 2024-09-21T15:50:35.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0736e-05 2024-09-21T15:50:35.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.101 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:35.101 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.201609+0000", 2024-09-21T15:50:35.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9739999999999999e-06 2024-09-21T15:50:35.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.102 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:35.102 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.201643+0000", 2024-09-21T15:50:35.102 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4572e-05 2024-09-21T15:50:35.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.102 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:35.102 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.201726+0000", 2024-09-21T15:50:35.102 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.2763999999999996e-05 2024-09-21T15:50:35.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.102 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:50:35.102 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.201832+0000", 2024-09-21T15:50:35.102 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105931 2024-09-21T15:50:35.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.103 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:35.103 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.202743+0000", 2024-09-21T15:50:35.103 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00091128699999999999 2024-09-21T15:50:35.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.103 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:35.103 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.205540+0000", 2024-09-21T15:50:35.103 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0027967199999999999 2024-09-21T15:50:35.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.104 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:35.104 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.205598+0000", 2024-09-21T15:50:35.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7513999999999999e-05 2024-09-21T15:50:35.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.104 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:35.104 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.205662+0000", 2024-09-21T15:50:35.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4148999999999999e-05 2024-09-21T15:50:35.104 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.104 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:35.104 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.104 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160605 3.b 3:d1863919:::rbd_data.11338bae974a.0000000000000008:head [delete] snapc 2d=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-21T15:50:35.105 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.213794+0000", 2024-09-21T15:50:35.105 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.814787731, 2024-09-21T15:50:35.105 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046152490000000001, 2024-09-21T15:50:35.105 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:35.105 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:35.105 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:35.105 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:35.105 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:35.105 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160605 2024-09-21T15:50:35.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.105 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:35.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.105 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:35.106 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.213794+0000", 2024-09-21T15:50:35.106 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.106 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:35.106 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.213794+0000", 2024-09-21T15:50:35.106 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.106 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:35.106 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.213793+0000", 2024-09-21T15:50:35.106 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:35.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.106 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:35.107 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.213799+0000", 2024-09-21T15:50:35.107 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9959999999999999e-06 2024-09-21T15:50:35.107 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.107 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.107 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:35.107 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.213815+0000", 2024-09-21T15:50:35.107 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5880999999999999e-05 2024-09-21T15:50:35.107 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.107 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.107 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:35.107 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.213838+0000", 2024-09-21T15:50:35.107 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3671000000000001e-05 2024-09-21T15:50:35.107 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.107 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.107 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:35.107 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.213887+0000", 2024-09-21T15:50:35.108 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8865000000000001e-05 2024-09-21T15:50:35.108 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.108 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.108 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:35.108 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.213962+0000", 2024-09-21T15:50:35.108 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4592999999999995e-05 2024-09-21T15:50:35.108 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.108 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.108 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:50:35.108 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.214088+0000", 2024-09-21T15:50:35.108 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012666199999999999 2024-09-21T15:50:35.108 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.108 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.108 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:35.108 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216917+0000", 2024-09-21T15:50:35.109 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0028284059999999999 2024-09-21T15:50:35.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.109 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:35.109 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.218319+0000", 2024-09-21T15:50:35.109 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001402613 2024-09-21T15:50:35.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.109 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:35.109 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.218372+0000", 2024-09-21T15:50:35.109 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2636000000000002e-05 2024-09-21T15:50:35.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.109 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:35.110 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.218409+0000", 2024-09-21T15:50:35.110 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6942e-05 2024-09-21T15:50:35.110 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.110 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:35.110 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.110 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.110 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.110 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:160602 3.12 e95/94 3:49a61a92:::rbd_data.11338bae974a.0000000000000005:head v 95'344, mlcod=0'0)", 2024-09-21T15:50:35.110 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.215034+0000", 2024-09-21T15:50:35.110 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.813547314, 2024-09-21T15:50:35.110 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045544870000000003, 2024-09-21T15:50:35.110 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:35.110 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:35.110 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:35.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.111 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:35.111 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215034+0000", 2024-09-21T15:50:35.111 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.111 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:35.111 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215034+0000", 2024-09-21T15:50:35.111 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.111 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:35.111 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215034+0000", 2024-09-21T15:50:35.111 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:35.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.112 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:35.112 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215039+0000", 2024-09-21T15:50:35.112 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1549999999999998e-06 2024-09-21T15:50:35.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.112 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:35.112 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215050+0000", 2024-09-21T15:50:35.112 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1392e-05 2024-09-21T15:50:35.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.112 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:35.112 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215057+0000", 2024-09-21T15:50:35.112 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7399999999999998e-06 2024-09-21T15:50:35.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.112 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:35.113 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215187+0000", 2024-09-21T15:50:35.113 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013010100000000001 2024-09-21T15:50:35.113 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.113 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:35.113 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215208+0000", 2024-09-21T15:50:35.113 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1013999999999999e-05 2024-09-21T15:50:35.113 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.113 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:35.113 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.219549+0000", 2024-09-21T15:50:35.113 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0043404990000000003 2024-09-21T15:50:35.113 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.113 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:35.113 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.219589+0000", 2024-09-21T15:50:35.114 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9878e-05 2024-09-21T15:50:35.114 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.114 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:35.114 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.114 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.114 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:160597 3.1 e95/94 3:84154dfc:::rbd_data.11338bae974a.0000000000000000:head v 95'451, mlcod=0'0)", 2024-09-21T15:50:35.114 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.215080+0000", 2024-09-21T15:50:35.114 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.813501858, 2024-09-21T15:50:35.114 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045596819999999998, 2024-09-21T15:50:35.114 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:35.114 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:35.114 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:35.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:35.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215080+0000", 2024-09-21T15:50:35.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:35.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215080+0000", 2024-09-21T15:50:35.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:35.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215079+0000", 2024-09-21T15:50:35.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:35.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.116 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:35.116 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215083+0000", 2024-09-21T15:50:35.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2570000000000001e-06 2024-09-21T15:50:35.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.116 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:35.116 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215091+0000", 2024-09-21T15:50:35.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7829999999999999e-06 2024-09-21T15:50:35.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.116 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:35.116 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215096+0000", 2024-09-21T15:50:35.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0370000000000002e-06 2024-09-21T15:50:35.117 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.117 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.117 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:35.117 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216055+0000", 2024-09-21T15:50:35.117 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00095892299999999998 2024-09-21T15:50:35.117 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.117 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.117 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:35.117 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216078+0000", 2024-09-21T15:50:35.117 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2376000000000001e-05 2024-09-21T15:50:35.117 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.117 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.117 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:35.117 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.219598+0000", 2024-09-21T15:50:35.118 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035209299999999998 2024-09-21T15:50:35.118 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.118 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.118 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:35.118 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.219639+0000", 2024-09-21T15:50:35.118 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0766000000000002e-05 2024-09-21T15:50:35.118 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.118 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:35.118 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.118 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.118 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.118 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:160601 3.17 e95/94 3:e9d872b2:::rbd_data.11338bae974a.0000000000000004:head v 95'912, mlcod=0'0)", 2024-09-21T15:50:35.118 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.215668+0000", 2024-09-21T15:50:35.119 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.812913241, 2024-09-21T15:50:35.119 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0038923640000000002, 2024-09-21T15:50:35.119 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:35.119 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:35.119 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:35.119 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.119 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:35.119 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215668+0000", 2024-09-21T15:50:35.119 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.119 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.119 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.119 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:35.119 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215668+0000", 2024-09-21T15:50:35.119 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.120 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.120 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:35.120 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215668+0000", 2024-09-21T15:50:35.120 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:35.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.120 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.120 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:35.120 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215672+0000", 2024-09-21T15:50:35.120 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7680000000000002e-06 2024-09-21T15:50:35.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.120 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.120 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:35.121 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215682+0000", 2024-09-21T15:50:35.121 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0774000000000001e-05 2024-09-21T15:50:35.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.121 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:35.121 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215689+0000", 2024-09-21T15:50:35.121 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4670000000000001e-06 2024-09-21T15:50:35.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.121 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:35.121 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215740+0000", 2024-09-21T15:50:35.121 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1100000000000002e-05 2024-09-21T15:50:35.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.122 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.122 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:35.122 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215761+0000", 2024-09-21T15:50:35.122 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1112e-05 2024-09-21T15:50:35.122 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.122 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.122 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:35.122 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.219520+0000", 2024-09-21T15:50:35.122 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0037590739999999998 2024-09-21T15:50:35.122 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.122 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.122 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:35.122 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.219561+0000", 2024-09-21T15:50:35.122 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0329e-05 2024-09-21T15:50:35.123 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.123 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:35.123 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.123 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.123 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:160608 3.12 e95/94 3:48a47e1c:::rbd_data.11338bae974a.000000000000000b:head v 95'345, mlcod=0'0)", 2024-09-21T15:50:35.123 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.216934+0000", 2024-09-21T15:50:35.123 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.8116474729999998, 2024-09-21T15:50:35.123 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053335759999999996, 2024-09-21T15:50:35.123 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:35.123 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:35.123 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:35.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.123 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:35.124 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216934+0000", 2024-09-21T15:50:35.124 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.124 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.124 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.124 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:35.124 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216934+0000", 2024-09-21T15:50:35.124 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.124 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.124 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.124 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:35.124 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216933+0000", 2024-09-21T15:50:35.124 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:35.124 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.124 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.124 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:35.125 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216939+0000", 2024-09-21T15:50:35.125 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6459999999999998e-06 2024-09-21T15:50:35.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.125 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.125 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:35.125 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216947+0000", 2024-09-21T15:50:35.125 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.2479999999999996e-06 2024-09-21T15:50:35.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.125 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.125 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:35.125 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216951+0000", 2024-09-21T15:50:35.125 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3760000000000001e-06 2024-09-21T15:50:35.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.125 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.126 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:35.126 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216998+0000", 2024-09-21T15:50:35.126 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6244e-05 2024-09-21T15:50:35.126 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.126 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.126 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:35.126 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.217019+0000", 2024-09-21T15:50:35.126 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1191000000000001e-05 2024-09-21T15:50:35.126 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.126 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.126 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:35.126 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.222230+0000", 2024-09-21T15:50:35.126 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052114350000000004 2024-09-21T15:50:35.126 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.127 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.127 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:35.127 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.222267+0000", 2024-09-21T15:50:35.127 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7169999999999998e-05 2024-09-21T15:50:35.127 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.127 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:35.127 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.127 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.127 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.127 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:160610 3.3 e95/94 3:c238de8e:::rbd_data.11338bae974a.000000000000000f:head v 95'632, mlcod=71'310)", 2024-09-21T15:50:35.127 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.218864+0000", 2024-09-21T15:50:35.127 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.8097172640000001, 2024-09-21T15:50:35.128 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0034014280000000002, 2024-09-21T15:50:35.128 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:35.128 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:35.128 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:35.128 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.128 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:35.128 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.218864+0000", 2024-09-21T15:50:35.128 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.129 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.129 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.129 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:35.129 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.218864+0000", 2024-09-21T15:50:35.129 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.130 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.130 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.131 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:35.131 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.218863+0000", 2024-09-21T15:50:35.131 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:35.131 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.131 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.131 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:35.131 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.218870+0000", 2024-09-21T15:50:35.132 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2559999999999999e-06 2024-09-21T15:50:35.132 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.132 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.132 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:35.132 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.218880+0000", 2024-09-21T15:50:35.132 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.009e-05 2024-09-21T15:50:35.132 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.132 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.132 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:35.132 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.218885+0000", 2024-09-21T15:50:35.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0039999999999999e-06 2024-09-21T15:50:35.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.133 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.133 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:35.133 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.218919+0000", 2024-09-21T15:50:35.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4195e-05 2024-09-21T15:50:35.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.133 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.133 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:35.133 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.218941+0000", 2024-09-21T15:50:35.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2266e-05 2024-09-21T15:50:35.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.134 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.134 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:35.134 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.222241+0000", 2024-09-21T15:50:35.134 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032993889999999998 2024-09-21T15:50:35.134 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.134 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.134 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:35.134 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.222266+0000", 2024-09-21T15:50:35.134 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4929000000000001e-05 2024-09-21T15:50:35.134 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.134 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:35.134 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.134 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.134 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.135 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160614 3.b 3:d297695c:::rbd_header.11338bae974a:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-21T15:50:35.135 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.227565+0000", 2024-09-21T15:50:35.135 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.801016577, 2024-09-21T15:50:35.135 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0027157330000000001, 2024-09-21T15:50:35.135 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:35.135 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:35.135 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:35.135 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:35.135 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:35.135 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160614 2024-09-21T15:50:35.135 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.135 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:35.135 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.136 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:35.136 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.227565+0000", 2024-09-21T15:50:35.136 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.136 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.136 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.136 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:35.136 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.227565+0000", 2024-09-21T15:50:35.136 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.136 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.136 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.136 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:35.136 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.227563+0000", 2024-09-21T15:50:35.136 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:50:35.136 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.136 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.137 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:35.137 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.227570+0000", 2024-09-21T15:50:35.137 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.409e-06 2024-09-21T15:50:35.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.137 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:35.137 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.227582+0000", 2024-09-21T15:50:35.137 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2454e-05 2024-09-21T15:50:35.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.137 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:35.137 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.227590+0000", 2024-09-21T15:50:35.137 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5070000000000003e-06 2024-09-21T15:50:35.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.138 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.138 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:35.138 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.227628+0000", 2024-09-21T15:50:35.138 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8699999999999999e-05 2024-09-21T15:50:35.138 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.138 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.138 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:35.138 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.227689+0000", 2024-09-21T15:50:35.138 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0210000000000001e-05 2024-09-21T15:50:35.138 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.138 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.138 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:50:35.138 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.227806+0000", 2024-09-21T15:50:35.138 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000117198 2024-09-21T15:50:35.138 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.139 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:35.139 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.228924+0000", 2024-09-21T15:50:35.139 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001118665 2024-09-21T15:50:35.139 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.139 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:35.139 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.230205+0000", 2024-09-21T15:50:35.139 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00128053 2024-09-21T15:50:35.139 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.139 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:35.139 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.230237+0000", 2024-09-21T15:50:35.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1693000000000003e-05 2024-09-21T15:50:35.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.140 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:35.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.230281+0000", 2024-09-21T15:50:35.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3909000000000003e-05 2024-09-21T15:50:35.140 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.140 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:35.140 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.140 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160618 3.b 3:d297695c:::rbd_header.11338bae974a:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-21T15:50:35.140 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.233457+0000", 2024-09-21T15:50:35.140 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.7951246360000002, 2024-09-21T15:50:35.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.005085198, 2024-09-21T15:50:35.140 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:35.140 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:35.141 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:35.141 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:35.141 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:35.141 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160618 2024-09-21T15:50:35.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.141 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:35.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.141 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:35.141 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233457+0000", 2024-09-21T15:50:35.141 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.141 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:35.141 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233457+0000", 2024-09-21T15:50:35.142 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.142 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.142 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.142 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:35.142 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233456+0000", 2024-09-21T15:50:35.142 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:35.142 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.142 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.142 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:35.142 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233463+0000", 2024-09-21T15:50:35.142 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9310000000000002e-06 2024-09-21T15:50:35.142 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.142 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.142 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:35.142 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233473+0000", 2024-09-21T15:50:35.143 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0519e-05 2024-09-21T15:50:35.143 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.143 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.143 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:35.143 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233481+0000", 2024-09-21T15:50:35.143 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4009999999999997e-06 2024-09-21T15:50:35.143 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.143 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.143 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:35.143 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233513+0000", 2024-09-21T15:50:35.143 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1328999999999998e-05 2024-09-21T15:50:35.143 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.143 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.143 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:35.143 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233568+0000", 2024-09-21T15:50:35.144 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5606999999999998e-05 2024-09-21T15:50:35.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.144 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.144 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:50:35.144 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233670+0000", 2024-09-21T15:50:35.144 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010180599999999999 2024-09-21T15:50:35.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.144 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.144 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:35.144 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.234761+0000", 2024-09-21T15:50:35.144 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010905909999999999 2024-09-21T15:50:35.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.144 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.145 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:35.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.238442+0000", 2024-09-21T15:50:35.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0036809429999999999 2024-09-21T15:50:35.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.145 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:35.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.238498+0000", 2024-09-21T15:50:35.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6116999999999998e-05 2024-09-21T15:50:35.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.145 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:35.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.238542+0000", 2024-09-21T15:50:35.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4134000000000001e-05 2024-09-21T15:50:35.145 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.146 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:35.146 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.146 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.146 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.146 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160660 3.b 3:d7701915:::rbd_object_map.1133fd7483d4.000000000000002c:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95)", 2024-09-21T15:50:35.146 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.274362+0000", 2024-09-21T15:50:35.146 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.7542189390000003, 2024-09-21T15:50:35.146 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0028700079999999998, 2024-09-21T15:50:35.146 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:35.146 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:35.146 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:35.146 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:35.146 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:35.147 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160660 2024-09-21T15:50:35.147 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.147 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:35.147 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.147 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:35.147 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.274362+0000", 2024-09-21T15:50:35.147 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.147 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.147 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.147 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:35.147 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.274362+0000", 2024-09-21T15:50:35.147 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.147 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.147 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.147 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:35.147 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.274361+0000", 2024-09-21T15:50:35.148 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:35.148 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.148 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.148 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:35.148 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.274368+0000", 2024-09-21T15:50:35.148 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4500000000000001e-06 2024-09-21T15:50:35.148 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.148 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.148 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:35.148 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.274378+0000", 2024-09-21T15:50:35.148 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0309000000000001e-05 2024-09-21T15:50:35.148 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.148 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.148 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:35.148 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.274383+0000", 2024-09-21T15:50:35.148 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7570000000000004e-06 2024-09-21T15:50:35.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.149 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:35.149 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.274438+0000", 2024-09-21T15:50:35.149 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5238999999999997e-05 2024-09-21T15:50:35.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.149 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:35.149 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.274540+0000", 2024-09-21T15:50:35.149 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101411 2024-09-21T15:50:35.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.149 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:35.149 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.277232+0000", 2024-09-21T15:50:35.149 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0026929359999999999 2024-09-21T15:50:35.149 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.150 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:35.150 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.150 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.150 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.150 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160662 3.b 3:d7701915:::rbd_object_map.1133fd7483d4.000000000000002c:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-21T15:50:35.150 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.279369+0000", 2024-09-21T15:50:35.150 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.7492127059999998, 2024-09-21T15:50:35.150 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0056948040000000004, 2024-09-21T15:50:35.150 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:35.150 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:35.150 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:35.150 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:35.150 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:35.150 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160662 2024-09-21T15:50:35.151 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.151 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:35.151 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.151 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:35.151 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279369+0000", 2024-09-21T15:50:35.151 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.151 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.151 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.151 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:35.151 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279369+0000", 2024-09-21T15:50:35.151 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.151 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.152 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.152 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:35.152 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279368+0000", 2024-09-21T15:50:35.152 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:35.152 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.152 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.152 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:35.152 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279373+0000", 2024-09-21T15:50:35.152 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6529999999999999e-06 2024-09-21T15:50:35.152 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.152 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.152 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:35.152 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279384+0000", 2024-09-21T15:50:35.152 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0256e-05 2024-09-21T15:50:35.152 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.152 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.153 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:35.153 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279388+0000", 2024-09-21T15:50:35.153 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5630000000000002e-06 2024-09-21T15:50:35.153 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.153 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.153 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:35.153 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279424+0000", 2024-09-21T15:50:35.153 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5985999999999997e-05 2024-09-21T15:50:35.153 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.153 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.153 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:35.153 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279478+0000", 2024-09-21T15:50:35.153 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3944999999999999e-05 2024-09-21T15:50:35.153 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.153 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.154 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:50:35.154 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279575+0000", 2024-09-21T15:50:35.154 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6817999999999994e-05 2024-09-21T15:50:35.154 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.154 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.154 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:35.154 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.282391+0000", 2024-09-21T15:50:35.154 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0028156700000000002 2024-09-21T15:50:35.154 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.154 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.154 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:35.154 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.284977+0000", 2024-09-21T15:50:35.154 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0025859640000000001 2024-09-21T15:50:35.155 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.155 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.155 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:35.155 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.285021+0000", 2024-09-21T15:50:35.155 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4415000000000001e-05 2024-09-21T15:50:35.155 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.155 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.155 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:35.155 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.285063+0000", 2024-09-21T15:50:35.155 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2447000000000002e-05 2024-09-21T15:50:35.155 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.155 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:35.155 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.155 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.155 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:35.156 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:35.156 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:50:35.157 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:35.157+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:35.259 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:35.258+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:35.259 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:35.259 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:50:35.259 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.259 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160974 3.4 3.ae24bb44 (undecoded) ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-21T15:50:35.259 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:34.748093+0000", 2024-09-21T15:50:35.259 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.51052849499999997, 2024-09-21T15:50:35.260 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.51054845900000001, 2024-09-21T15:50:35.260 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:35.260 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "waiting for peered", 2024-09-21T15:50:35.260 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:35.260 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:35.260 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:35.260 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160974 2024-09-21T15:50:35.260 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.260 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:35.260 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.260 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:35.260 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748093+0000", 2024-09-21T15:50:35.260 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.260 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.260 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.261 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:35.261 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748093+0000", 2024-09-21T15:50:35.261 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.261 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.261 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.261 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:35.261 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748091+0000", 2024-09-21T15:50:35.261 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:50:35.261 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.261 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.261 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:35.261 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748101+0000", 2024-09-21T15:50:35.261 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5950000000000005e-06 2024-09-21T15:50:35.261 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.261 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.261 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:35.262 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748122+0000", 2024-09-21T15:50:35.262 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1611e-05 2024-09-21T15:50:35.262 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.262 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.262 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:35.262 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748133+0000", 2024-09-21T15:50:35.262 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0224000000000001e-05 2024-09-21T15:50:35.262 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.262 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.262 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:35.262 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748185+0000", 2024-09-21T15:50:35.262 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2005000000000001e-05 2024-09-21T15:50:35.263 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.263 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.263 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:50:35.263 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748194+0000", 2024-09-21T15:50:35.263 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0180000000000007e-06 2024-09-21T15:50:35.263 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.263 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:35.263 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.263 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.263 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-21T15:50:35.263 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 1 2024-09-21T15:50:35.263 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:35.267 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:50:35.359 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:35.358+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:35.371 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:35.371 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:35.371 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:50:35.371 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:50:35.371 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:35.380 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:50:35.460 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:35.459+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:35.484 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:35.484 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:50:35.484 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:50:35.484 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:50:35.484 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.485 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62833 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 192512~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:35.485 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:50:35.485 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 58.340318029999999, 2024-09-21T15:50:35.485 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050387007999999997, 2024-09-21T15:50:35.485 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:35.485 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:35.485 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:35.485 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:35.485 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:35.485 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62833 2024-09-21T15:50:35.485 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.485 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:35.486 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.486 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:35.486 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:50:35.486 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.486 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.486 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.486 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:35.486 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:50:35.486 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.486 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.486 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.486 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:35.486 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:50:35.486 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:35.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.487 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.487 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:35.487 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142150+0000", 2024-09-21T15:50:35.487 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1999999999999999e-06 2024-09-21T15:50:35.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.487 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.487 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:35.487 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142153+0000", 2024-09-21T15:50:35.487 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7410000000000001e-06 2024-09-21T15:50:35.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.487 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.487 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:35.487 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142156+0000", 2024-09-21T15:50:35.488 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5739999999999998e-06 2024-09-21T15:50:35.488 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.488 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.488 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:35.488 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.192358+0000", 2024-09-21T15:50:35.488 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050202452000000002 2024-09-21T15:50:35.488 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.488 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.488 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:35.488 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.192435+0000", 2024-09-21T15:50:35.488 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6738999999999998e-05 2024-09-21T15:50:35.488 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.488 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.488 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:35.488 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.192536+0000", 2024-09-21T15:50:35.489 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101508 2024-09-21T15:50:35.489 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.489 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:35.489 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.489 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.489 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.489 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62849 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 208896~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:35.489 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.144181+0000", 2024-09-21T15:50:35.489 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 58.338285958, 2024-09-21T15:50:35.489 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050460390000000001, 2024-09-21T15:50:35.489 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:35.489 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:35.489 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:35.489 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:35.489 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:35.490 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62849 2024-09-21T15:50:35.490 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.490 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:35.490 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.490 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:35.490 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144181+0000", 2024-09-21T15:50:35.490 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.490 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.490 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.490 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:35.490 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144181+0000", 2024-09-21T15:50:35.490 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.490 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.490 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.491 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:35.491 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144181+0000", 2024-09-21T15:50:35.491 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:35.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.491 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:35.491 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144183+0000", 2024-09-21T15:50:35.491 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3860000000000001e-06 2024-09-21T15:50:35.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.491 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:35.491 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144189+0000", 2024-09-21T15:50:35.491 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0599999999999996e-06 2024-09-21T15:50:35.491 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.491 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.492 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:35.492 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144194+0000", 2024-09-21T15:50:35.492 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4379999999999997e-06 2024-09-21T15:50:35.492 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.492 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.492 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:35.492 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.194457+0000", 2024-09-21T15:50:35.492 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050263317000000002 2024-09-21T15:50:35.492 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.492 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.492 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:35.492 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.194548+0000", 2024-09-21T15:50:35.492 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0785000000000006e-05 2024-09-21T15:50:35.492 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.493 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.493 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:35.493 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.194642+0000", 2024-09-21T15:50:35.493 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3837000000000003e-05 2024-09-21T15:50:35.493 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.493 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:35.493 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.493 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.493 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.493 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62862 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 954368~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:35.493 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.146176+0000", 2024-09-21T15:50:35.493 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 58.336290894999998, 2024-09-21T15:50:35.493 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050442901999999998, 2024-09-21T15:50:35.493 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:35.494 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:35.494 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:35.494 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:35.494 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:35.494 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62862 2024-09-21T15:50:35.494 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.494 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:35.494 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.494 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:35.494 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146176+0000", 2024-09-21T15:50:35.494 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.494 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.494 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.494 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:35.495 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146176+0000", 2024-09-21T15:50:35.495 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.495 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.495 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.495 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:35.495 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146176+0000", 2024-09-21T15:50:35.495 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:35.495 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.495 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.495 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:35.495 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146178+0000", 2024-09-21T15:50:35.495 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4739999999999999e-06 2024-09-21T15:50:35.495 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.496 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.496 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:35.496 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146186+0000", 2024-09-21T15:50:35.496 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4989999999999997e-06 2024-09-21T15:50:35.496 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.496 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.496 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:35.497 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146191+0000", 2024-09-21T15:50:35.497 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7999999999999998e-06 2024-09-21T15:50:35.497 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.497 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.497 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:35.497 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196442+0000", 2024-09-21T15:50:35.497 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050251665000000001 2024-09-21T15:50:35.498 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.498 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.498 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:35.498 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196518+0000", 2024-09-21T15:50:35.498 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5677000000000001e-05 2024-09-21T15:50:35.498 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.498 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.499 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:35.499 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196619+0000", 2024-09-21T15:50:35.499 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101247 2024-09-21T15:50:35.499 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.499 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:35.499 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.499 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.499 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.499 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62863 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 937984~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:35.500 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.146195+0000", 2024-09-21T15:50:35.500 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 58.336272704000002, 2024-09-21T15:50:35.500 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050642305999999998, 2024-09-21T15:50:35.500 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:35.500 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:35.500 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:35.500 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:35.500 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:35.501 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62863 2024-09-21T15:50:35.501 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.501 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:35.501 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.501 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:35.501 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146195+0000", 2024-09-21T15:50:35.501 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.502 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.502 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.502 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:35.502 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146195+0000", 2024-09-21T15:50:35.502 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.502 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.503 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.503 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:35.503 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146194+0000", 2024-09-21T15:50:35.503 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:35.503 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.503 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.503 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:35.503 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146196+0000", 2024-09-21T15:50:35.503 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1900000000000002e-06 2024-09-21T15:50:35.503 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.504 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.504 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:35.504 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146202+0000", 2024-09-21T15:50:35.504 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.57e-06 2024-09-21T15:50:35.504 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.504 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.504 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:35.504 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146206+0000", 2024-09-21T15:50:35.504 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5079999999999999e-06 2024-09-21T15:50:35.504 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.504 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.504 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:35.504 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196655+0000", 2024-09-21T15:50:35.505 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050448973000000001 2024-09-21T15:50:35.505 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.505 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.505 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:35.505 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196730+0000", 2024-09-21T15:50:35.505 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4728999999999996e-05 2024-09-21T15:50:35.505 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.505 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.505 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:35.505 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196837+0000", 2024-09-21T15:50:35.505 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000106735 2024-09-21T15:50:35.505 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.505 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:35.506 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.506 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.506 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.506 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62872 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 290816~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:35.506 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148149+0000", 2024-09-21T15:50:35.506 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 58.334318027999998, 2024-09-21T15:50:35.506 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.05046991, 2024-09-21T15:50:35.506 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:35.506 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:35.506 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:35.506 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:35.506 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:35.506 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62872 2024-09-21T15:50:35.507 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.507 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:35.507 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.507 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:35.507 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148149+0000", 2024-09-21T15:50:35.507 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.507 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.507 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.507 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:35.507 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148149+0000", 2024-09-21T15:50:35.507 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.507 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.507 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.507 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:35.508 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148149+0000", 2024-09-21T15:50:35.508 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:35.508 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.508 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.508 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:35.508 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148157+0000", 2024-09-21T15:50:35.508 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4749999999999993e-06 2024-09-21T15:50:35.508 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.508 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.508 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:35.508 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148163+0000", 2024-09-21T15:50:35.508 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8250000000000001e-06 2024-09-21T15:50:35.509 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.509 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.509 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:35.509 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148168+0000", 2024-09-21T15:50:35.509 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4730000000000002e-06 2024-09-21T15:50:35.509 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.509 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.509 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:35.509 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198471+0000", 2024-09-21T15:50:35.509 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050302974 2024-09-21T15:50:35.509 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.509 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.509 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:35.509 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198555+0000", 2024-09-21T15:50:35.509 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4783999999999993e-05 2024-09-21T15:50:35.509 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.510 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.510 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:35.510 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198619+0000", 2024-09-21T15:50:35.510 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3751000000000007e-05 2024-09-21T15:50:35.510 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.510 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:35.510 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.510 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.510 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.510 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62873 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 315392~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:35.510 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148172+0000", 2024-09-21T15:50:35.510 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 58.334295736999998, 2024-09-21T15:50:35.510 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050707998999999997, 2024-09-21T15:50:35.510 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:35.510 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:35.511 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:35.511 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:35.511 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:35.511 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62873 2024-09-21T15:50:35.511 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.511 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:35.511 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.511 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:35.511 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148172+0000", 2024-09-21T15:50:35.511 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.511 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.511 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.511 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:35.511 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148172+0000", 2024-09-21T15:50:35.511 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.511 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.512 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.512 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:35.512 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148171+0000", 2024-09-21T15:50:35.512 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:35.512 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.512 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.512 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:35.512 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148173+0000", 2024-09-21T15:50:35.512 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2340000000000001e-06 2024-09-21T15:50:35.512 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.512 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.512 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:35.513 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148179+0000", 2024-09-21T15:50:35.513 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9340000000000003e-06 2024-09-21T15:50:35.513 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.513 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.513 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:35.513 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148186+0000", 2024-09-21T15:50:35.513 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2349999999999996e-06 2024-09-21T15:50:35.513 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.513 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.513 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:35.513 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198684+0000", 2024-09-21T15:50:35.513 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050498890999999997 2024-09-21T15:50:35.513 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.513 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.513 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:35.514 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198778+0000", 2024-09-21T15:50:35.514 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3134999999999995e-05 2024-09-21T15:50:35.514 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.514 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.514 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:35.514 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198880+0000", 2024-09-21T15:50:35.514 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101906 2024-09-21T15:50:35.514 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.514 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:35.514 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.514 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.514 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.514 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62874 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 323584~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:35.515 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148211+0000", 2024-09-21T15:50:35.515 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 58.334255976999998, 2024-09-21T15:50:35.515 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050887766000000001, 2024-09-21T15:50:35.515 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:35.515 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:35.515 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:35.515 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:35.515 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:35.515 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62874 2024-09-21T15:50:35.515 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.515 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:35.515 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.515 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:35.515 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148211+0000", 2024-09-21T15:50:35.515 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.516 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.516 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.516 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:35.516 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148211+0000", 2024-09-21T15:50:35.516 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.516 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.516 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.516 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:35.516 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148211+0000", 2024-09-21T15:50:35.516 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:35.516 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.516 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.516 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:35.516 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148213+0000", 2024-09-21T15:50:35.517 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6860000000000002e-06 2024-09-21T15:50:35.517 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.517 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.517 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:35.517 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148220+0000", 2024-09-21T15:50:35.517 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4740000000000002e-06 2024-09-21T15:50:35.517 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.517 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.517 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:35.517 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148226+0000", 2024-09-21T15:50:35.517 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0120000000000002e-06 2024-09-21T15:50:35.517 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.517 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.517 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:35.517 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198940+0000", 2024-09-21T15:50:35.517 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050714412 2024-09-21T15:50:35.518 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.518 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.518 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:35.518 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198995+0000", 2024-09-21T15:50:35.518 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4863e-05 2024-09-21T15:50:35.518 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.518 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.518 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:35.518 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.199099+0000", 2024-09-21T15:50:35.518 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010383299999999999 2024-09-21T15:50:35.518 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.518 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:35.518 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.518 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.519 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.519 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62877 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 995328~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:35.519 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148230+0000", 2024-09-21T15:50:35.519 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 58.334237117000001, 2024-09-21T15:50:35.519 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.051108949000000001, 2024-09-21T15:50:35.519 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:35.519 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:35.519 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:35.519 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:35.519 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:35.519 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62877 2024-09-21T15:50:35.519 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.519 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:35.519 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.519 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:35.519 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148230+0000", 2024-09-21T15:50:35.520 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.520 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.520 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.520 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:35.520 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148230+0000", 2024-09-21T15:50:35.520 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.520 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.520 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.520 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:35.520 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148230+0000", 2024-09-21T15:50:35.520 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:35.520 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.520 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.520 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:35.520 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148232+0000", 2024-09-21T15:50:35.521 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1220000000000002e-06 2024-09-21T15:50:35.521 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.521 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.521 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:35.521 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148238+0000", 2024-09-21T15:50:35.521 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.789e-06 2024-09-21T15:50:35.521 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.521 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.521 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:35.521 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148242+0000", 2024-09-21T15:50:35.521 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7369999999999997e-06 2024-09-21T15:50:35.521 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.521 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.521 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:35.522 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.199201+0000", 2024-09-21T15:50:35.522 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050959088 2024-09-21T15:50:35.522 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.522 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.522 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:35.522 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.199285+0000", 2024-09-21T15:50:35.522 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3147999999999997e-05 2024-09-21T15:50:35.522 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.522 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.522 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:35.522 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.199339+0000", 2024-09-21T15:50:35.522 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4484999999999998e-05 2024-09-21T15:50:35.522 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.522 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:35.523 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.523 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.523 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.523 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62886 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 1044480~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:35.523 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150138+0000", 2024-09-21T15:50:35.523 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 58.332329448999999, 2024-09-21T15:50:35.523 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050345527000000001, 2024-09-21T15:50:35.523 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:35.523 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:35.523 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:35.523 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:35.523 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:35.523 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62886 2024-09-21T15:50:35.523 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.524 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:35.524 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.524 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:35.524 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150138+0000", 2024-09-21T15:50:35.524 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.524 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.524 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.524 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:35.524 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150138+0000", 2024-09-21T15:50:35.524 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.524 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.524 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.524 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:35.524 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150137+0000", 2024-09-21T15:50:35.524 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:35.525 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.525 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.525 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:35.525 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150140+0000", 2024-09-21T15:50:35.525 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2460000000000002e-06 2024-09-21T15:50:35.525 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.525 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.525 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:35.525 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150145+0000", 2024-09-21T15:50:35.525 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6520000000000003e-06 2024-09-21T15:50:35.525 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.525 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.525 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:35.525 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150150+0000", 2024-09-21T15:50:35.525 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5310000000000003e-06 2024-09-21T15:50:35.526 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.526 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.526 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:35.526 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200356+0000", 2024-09-21T15:50:35.526 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050205759000000003 2024-09-21T15:50:35.526 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.526 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.526 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:35.526 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200415+0000", 2024-09-21T15:50:35.526 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9719999999999997e-05 2024-09-21T15:50:35.526 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.526 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.526 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:35.526 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200483+0000", 2024-09-21T15:50:35.527 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7959000000000004e-05 2024-09-21T15:50:35.527 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.527 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:35.527 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.527 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.527 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.527 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62888 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 331776~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:35.527 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150154+0000", 2024-09-21T15:50:35.527 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 58.332313523000003, 2024-09-21T15:50:35.527 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050483169000000001, 2024-09-21T15:50:35.527 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:35.527 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:35.527 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:35.527 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:35.527 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:35.528 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62888 2024-09-21T15:50:35.528 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.528 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:35.528 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.528 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:35.528 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150154+0000", 2024-09-21T15:50:35.528 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.528 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.528 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.528 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:35.528 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150154+0000", 2024-09-21T15:50:35.528 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.528 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.528 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.529 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:35.529 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150153+0000", 2024-09-21T15:50:35.529 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:35.529 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.529 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.529 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:35.529 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150155+0000", 2024-09-21T15:50:35.529 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9920000000000002e-06 2024-09-21T15:50:35.529 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.529 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.529 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:35.529 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150161+0000", 2024-09-21T15:50:35.529 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5439999999999998e-06 2024-09-21T15:50:35.530 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.530 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.530 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:35.530 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150165+0000", 2024-09-21T15:50:35.530 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3460000000000004e-06 2024-09-21T15:50:35.530 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.530 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.530 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:35.530 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200502+0000", 2024-09-21T15:50:35.530 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050337150999999997 2024-09-21T15:50:35.530 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.530 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.531 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:35.531 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200575+0000", 2024-09-21T15:50:35.531 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2250999999999997e-05 2024-09-21T15:50:35.531 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.531 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.531 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:35.531 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200637+0000", 2024-09-21T15:50:35.531 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2195999999999999e-05 2024-09-21T15:50:35.531 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.531 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:35.531 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.531 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.532 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.532 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62889 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 339968~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:35.532 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150169+0000", 2024-09-21T15:50:35.532 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 58.332298024000004, 2024-09-21T15:50:35.532 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050629678999999997, 2024-09-21T15:50:35.532 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:35.532 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:35.532 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:35.532 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:35.532 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:35.532 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62889 2024-09-21T15:50:35.532 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.532 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:35.532 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.533 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:35.533 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150169+0000", 2024-09-21T15:50:35.533 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.533 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.533 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.533 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:35.533 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150169+0000", 2024-09-21T15:50:35.533 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.533 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.533 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.533 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:35.533 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150169+0000", 2024-09-21T15:50:35.533 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:35.534 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.534 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.534 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:35.534 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150171+0000", 2024-09-21T15:50:35.534 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3910000000000001e-06 2024-09-21T15:50:35.534 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.534 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.534 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:35.534 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150177+0000", 2024-09-21T15:50:35.534 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4079999999999997e-06 2024-09-21T15:50:35.534 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.534 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.534 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:35.535 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150182+0000", 2024-09-21T15:50:35.535 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3890000000000004e-06 2024-09-21T15:50:35.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.535 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.535 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:35.535 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200664+0000", 2024-09-21T15:50:35.535 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050482212999999998 2024-09-21T15:50:35.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.535 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.535 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:35.535 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200738+0000", 2024-09-21T15:50:35.535 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4114000000000002e-05 2024-09-21T15:50:35.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.535 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.536 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:35.536 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200799+0000", 2024-09-21T15:50:35.536 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0503e-05 2024-09-21T15:50:35.536 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.536 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:35.536 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.536 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.536 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.536 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62891 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 348160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:35.536 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150186+0000", 2024-09-21T15:50:35.536 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 58.332280881999999, 2024-09-21T15:50:35.536 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050794552, 2024-09-21T15:50:35.536 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:35.536 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:35.537 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:35.537 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:35.537 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:35.537 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62891 2024-09-21T15:50:35.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.537 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:35.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.537 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:35.537 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150186+0000", 2024-09-21T15:50:35.537 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.537 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:35.537 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150186+0000", 2024-09-21T15:50:35.538 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.538 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.538 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:35.538 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150186+0000", 2024-09-21T15:50:35.538 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:35.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.538 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.538 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:35.538 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150193+0000", 2024-09-21T15:50:35.538 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6830000000000002e-06 2024-09-21T15:50:35.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.538 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.538 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:35.538 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150199+0000", 2024-09-21T15:50:35.539 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9859999999999999e-06 2024-09-21T15:50:35.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.539 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:35.539 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150203+0000", 2024-09-21T15:50:35.539 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5920000000000002e-06 2024-09-21T15:50:35.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.539 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:35.539 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200835+0000", 2024-09-21T15:50:35.539 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050631783 2024-09-21T15:50:35.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.539 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:35.539 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200894+0000", 2024-09-21T15:50:35.540 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8560999999999997e-05 2024-09-21T15:50:35.540 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.540 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.540 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:35.540 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200981+0000", 2024-09-21T15:50:35.540 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7305999999999994e-05 2024-09-21T15:50:35.540 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.540 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:35.540 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.540 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.540 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.540 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62892 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 356352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:35.540 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150208+0000", 2024-09-21T15:50:35.540 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 58.332259493999999, 2024-09-21T15:50:35.540 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050951769000000001, 2024-09-21T15:50:35.541 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:35.541 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:35.541 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:35.541 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:35.541 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:35.541 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62892 2024-09-21T15:50:35.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.541 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:35.541 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.541 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:35.541 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150208+0000", 2024-09-21T15:50:35.541 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.541 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.541 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:35.541 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150208+0000", 2024-09-21T15:50:35.542 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.542 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:35.542 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150207+0000", 2024-09-21T15:50:35.542 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:35.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.542 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:35.542 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150210+0000", 2024-09-21T15:50:35.542 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.238e-06 2024-09-21T15:50:35.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.542 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:35.542 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150215+0000", 2024-09-21T15:50:35.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6350000000000003e-06 2024-09-21T15:50:35.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.543 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.543 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:35.543 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150220+0000", 2024-09-21T15:50:35.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.544e-06 2024-09-21T15:50:35.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.543 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.543 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:35.543 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201014+0000", 2024-09-21T15:50:35.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050793932999999999 2024-09-21T15:50:35.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.543 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.543 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:35.544 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201092+0000", 2024-09-21T15:50:35.544 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7961999999999997e-05 2024-09-21T15:50:35.544 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.544 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.544 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:35.544 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201160+0000", 2024-09-21T15:50:35.544 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7836000000000006e-05 2024-09-21T15:50:35.544 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.544 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:35.544 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.544 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.544 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.544 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62894 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 364544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:35.544 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150820+0000", 2024-09-21T15:50:35.544 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 58.331647294, 2024-09-21T15:50:35.544 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050489196, 2024-09-21T15:50:35.545 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:35.545 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:35.545 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:35.545 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:35.545 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:35.545 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62894 2024-09-21T15:50:35.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.545 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:35.545 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.545 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:35.545 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150820+0000", 2024-09-21T15:50:35.545 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.545 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.545 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:35.545 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150820+0000", 2024-09-21T15:50:35.546 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.546 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.546 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:35.546 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150223+0000", 2024-09-21T15:50:35.546 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9994035 2024-09-21T15:50:35.546 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.546 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:35.546 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150825+0000", 2024-09-21T15:50:35.546 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00060137099999999996 2024-09-21T15:50:35.546 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.547 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:35.547 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150837+0000", 2024-09-21T15:50:35.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2503999999999999e-05 2024-09-21T15:50:35.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.547 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:35.547 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150844+0000", 2024-09-21T15:50:35.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6900000000000003e-06 2024-09-21T15:50:35.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.547 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:35.547 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201187+0000", 2024-09-21T15:50:35.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050342973999999999 2024-09-21T15:50:35.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.548 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.548 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:35.548 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201250+0000", 2024-09-21T15:50:35.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3274000000000004e-05 2024-09-21T15:50:35.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.548 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.548 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:35.548 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201309+0000", 2024-09-21T15:50:35.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9058000000000003e-05 2024-09-21T15:50:35.548 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.548 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:35.548 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.549 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.549 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62895 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 380928~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:35.549 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150851+0000", 2024-09-21T15:50:35.549 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 58.331616697999998, 2024-09-21T15:50:35.549 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050611097000000001, 2024-09-21T15:50:35.549 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:35.549 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:35.549 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:35.549 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:35.549 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:35.549 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62895 2024-09-21T15:50:35.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.549 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:35.549 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.549 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:35.550 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150851+0000", 2024-09-21T15:50:35.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.550 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.550 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:35.550 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150851+0000", 2024-09-21T15:50:35.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.550 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.550 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:35.550 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150850+0000", 2024-09-21T15:50:35.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:35.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:35.551 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150853+0000", 2024-09-21T15:50:35.551 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1039999999999998e-06 2024-09-21T15:50:35.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:35.551 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150860+0000", 2024-09-21T15:50:35.551 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3180000000000001e-06 2024-09-21T15:50:35.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:35.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150866+0000", 2024-09-21T15:50:35.552 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1599999999999997e-06 2024-09-21T15:50:35.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.552 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:35.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201332+0000", 2024-09-21T15:50:35.552 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050466100999999999 2024-09-21T15:50:35.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.552 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:35.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201392+0000", 2024-09-21T15:50:35.552 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0053999999999998e-05 2024-09-21T15:50:35.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.552 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:35.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201462+0000", 2024-09-21T15:50:35.553 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9861000000000001e-05 2024-09-21T15:50:35.553 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.553 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:35.553 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.553 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.553 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.553 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62896 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 372736~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:35.553 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150870+0000", 2024-09-21T15:50:35.553 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 58.331596918000002, 2024-09-21T15:50:35.553 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050753292999999998, 2024-09-21T15:50:35.553 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:35.553 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:35.553 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:35.554 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:35.554 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:35.554 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62896 2024-09-21T15:50:35.554 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.554 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:35.554 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.554 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:35.554 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150870+0000", 2024-09-21T15:50:35.554 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.554 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.554 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.555 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:35.555 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150870+0000", 2024-09-21T15:50:35.555 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.555 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.555 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.555 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:35.555 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150870+0000", 2024-09-21T15:50:35.555 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:35.555 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.555 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.555 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:35.555 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150872+0000", 2024-09-21T15:50:35.555 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.644e-06 2024-09-21T15:50:35.555 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.555 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.556 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:35.556 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150879+0000", 2024-09-21T15:50:35.556 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3509999999999998e-06 2024-09-21T15:50:35.556 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.556 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.556 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:35.556 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150884+0000", 2024-09-21T15:50:35.556 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9150000000000004e-06 2024-09-21T15:50:35.556 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.556 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.556 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:35.556 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201487+0000", 2024-09-21T15:50:35.556 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050603276000000003 2024-09-21T15:50:35.556 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.556 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.556 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:35.557 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201557+0000", 2024-09-21T15:50:35.557 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9847999999999999e-05 2024-09-21T15:50:35.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.557 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.557 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:35.557 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201624+0000", 2024-09-21T15:50:35.557 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6730999999999995e-05 2024-09-21T15:50:35.557 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.557 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:35.557 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.557 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.557 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62897 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 389120~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:35.557 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.151044+0000", 2024-09-21T15:50:35.557 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 58.331422877999998, 2024-09-21T15:50:35.557 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050720531999999999, 2024-09-21T15:50:35.558 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:35.558 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:35.558 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:35.558 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:35.558 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:35.558 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62897 2024-09-21T15:50:35.558 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.558 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:35.558 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.558 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:35.558 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151044+0000", 2024-09-21T15:50:35.558 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.558 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.558 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.558 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:35.559 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151044+0000", 2024-09-21T15:50:35.559 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.559 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.559 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.559 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:35.559 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150888+0000", 2024-09-21T15:50:35.559 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9998431 2024-09-21T15:50:35.559 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.559 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.559 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:35.559 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151048+0000", 2024-09-21T15:50:35.559 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00016077299999999999 2024-09-21T15:50:35.559 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.559 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.559 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:35.560 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151058+0000", 2024-09-21T15:50:35.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.465e-06 2024-09-21T15:50:35.560 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.560 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:35.560 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151064+0000", 2024-09-21T15:50:35.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3829999999999997e-06 2024-09-21T15:50:35.560 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.560 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:35.560 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201648+0000", 2024-09-21T15:50:35.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050583597000000001 2024-09-21T15:50:35.560 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.560 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:35.560 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201711+0000", 2024-09-21T15:50:35.561 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3616999999999998e-05 2024-09-21T15:50:35.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.561 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.561 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:35.561 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201765+0000", 2024-09-21T15:50:35.561 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3464000000000002e-05 2024-09-21T15:50:35.561 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.561 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:35.561 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.561 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.561 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:118351 3.4 3:25980f86:::rbd_data.1133545d3fa6.000000000000005a:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-09-21T15:50:35.561 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:58.975795+0000", 2024-09-21T15:50:35.561 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 36.506672244000001, 2024-09-21T15:50:35.561 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.99016351599999997, 2024-09-21T15:50:35.561 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:35.562 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:35.562 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:35.562 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:35.562 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:35.562 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 118351 2024-09-21T15:50:35.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.562 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:35.562 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.562 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:35.562 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975795+0000", 2024-09-21T15:50:35.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.562 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.562 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:35.562 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975795+0000", 2024-09-21T15:50:35.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.563 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:35.563 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975793+0000", 2024-09-21T15:50:35.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:50:35.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.563 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:35.563 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975800+0000", 2024-09-21T15:50:35.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8220000000000001e-06 2024-09-21T15:50:35.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:35.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975814+0000", 2024-09-21T15:50:35.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3893000000000001e-05 2024-09-21T15:50:35.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:35.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975821+0000", 2024-09-21T15:50:35.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1319999999999996e-06 2024-09-21T15:50:35.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:35.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975878+0000", 2024-09-21T15:50:35.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7238999999999999e-05 2024-09-21T15:50:35.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.565 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:50:35.565 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975889+0000", 2024-09-21T15:50:35.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0509000000000001e-05 2024-09-21T15:50:35.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.565 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:35.565 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.964573+0000", 2024-09-21T15:50:35.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.98868396300000005 2024-09-21T15:50:35.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.565 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:35.565 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.964672+0000", 2024-09-21T15:50:35.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9601999999999999e-05 2024-09-21T15:50:35.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.566 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-09-21T15:50:35.566 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.964758+0000", 2024-09-21T15:50:35.566 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5537000000000005e-05 2024-09-21T15:50:35.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.566 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:35.566 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965847+0000", 2024-09-21T15:50:35.566 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001089404 2024-09-21T15:50:35.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.566 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:35.566 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965890+0000", 2024-09-21T15:50:35.566 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2521999999999997e-05 2024-09-21T15:50:35.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.567 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.567 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:35.567 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965934+0000", 2024-09-21T15:50:35.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4489e-05 2024-09-21T15:50:35.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.567 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.567 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:35.567 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965959+0000", 2024-09-21T15:50:35.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4301999999999999e-05 2024-09-21T15:50:35.567 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.567 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:35.567 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.567 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.567 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:118353 3.4 3:23841e18:::rbd_data.1133545d3fa6.0000000000000061:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-09-21T15:50:35.568 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:58.975836+0000", 2024-09-21T15:50:35.568 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 36.506630745000002, 2024-09-21T15:50:35.568 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.99077746700000002, 2024-09-21T15:50:35.568 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:35.568 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:35.568 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:35.568 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:35.568 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:35.568 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 118353 2024-09-21T15:50:35.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.568 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:35.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.568 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:35.568 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975836+0000", 2024-09-21T15:50:35.568 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.569 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:35.569 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975836+0000", 2024-09-21T15:50:35.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.569 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:35.569 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975836+0000", 2024-09-21T15:50:35.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:35.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.569 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:35.569 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975839+0000", 2024-09-21T15:50:35.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6129999999999998e-06 2024-09-21T15:50:35.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:35.570 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975844+0000", 2024-09-21T15:50:35.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5130000000000004e-06 2024-09-21T15:50:35.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:35.570 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975850+0000", 2024-09-21T15:50:35.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3179999999999997e-06 2024-09-21T15:50:35.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:35.570 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975915+0000", 2024-09-21T15:50:35.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5168999999999999e-05 2024-09-21T15:50:35.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:50:35.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975922+0000", 2024-09-21T15:50:35.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2749999999999998e-06 2024-09-21T15:50:35.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:35.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965431+0000", 2024-09-21T15:50:35.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.98950864699999996 2024-09-21T15:50:35.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:35.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965535+0000", 2024-09-21T15:50:35.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000103949 2024-09-21T15:50:35.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-09-21T15:50:35.572 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965607+0000", 2024-09-21T15:50:35.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2478000000000006e-05 2024-09-21T15:50:35.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.572 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:35.572 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.966205+0000", 2024-09-21T15:50:35.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00059775200000000001 2024-09-21T15:50:35.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.572 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:35.572 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.966562+0000", 2024-09-21T15:50:35.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00035732199999999998 2024-09-21T15:50:35.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.572 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:35.572 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.966593+0000", 2024-09-21T15:50:35.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0642999999999998e-05 2024-09-21T15:50:35.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:35.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.966614+0000", 2024-09-21T15:50:35.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1149999999999999e-05 2024-09-21T15:50:35.573 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.573 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:35.573 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.573 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160535 3.6 3:67cae2ed:::rbd_mirroring:head [list-watchers] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-09-21T15:50:35.573 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:30.195408+0000", 2024-09-21T15:50:35.573 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.287059663, 2024-09-21T15:50:35.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.96025944900000004, 2024-09-21T15:50:35.573 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:35.574 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-09-21T15:50:35.574 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:35.574 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:35.574 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:35.574 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160535 2024-09-21T15:50:35.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.574 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:35.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.574 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:35.574 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195408+0000", 2024-09-21T15:50:35.574 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.574 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:35.574 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195408+0000", 2024-09-21T15:50:35.574 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:35.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:35.575 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195406+0000", 2024-09-21T15:50:35.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:50:35.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:35.575 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195414+0000", 2024-09-21T15:50:35.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.2870000000000004e-06 2024-09-21T15:50:35.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:35.575 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195432+0000", 2024-09-21T15:50:35.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8110000000000001e-05 2024-09-21T15:50:35.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.576 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:35.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195441+0000", 2024-09-21T15:50:35.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7679999999999997e-06 2024-09-21T15:50:35.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.576 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:35.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195505+0000", 2024-09-21T15:50:35.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3769000000000006e-05 2024-09-21T15:50:35.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.576 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:50:35.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195518+0000", 2024-09-21T15:50:35.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3341e-05 2024-09-21T15:50:35.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:35.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.155509+0000", 2024-09-21T15:50:35.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.959990393 2024-09-21T15:50:35.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:35.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:35.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:35.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.155667+0000", 2024-09-21T15:50:35.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000158505 2024-09-21T15:50:35.577 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.577 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:35.577 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.577 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:35.578 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:35.578 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:35.579 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:50:35.579 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:50:35.577+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:50:35.677 INFO:teuthology.orchestra.run.smithi089.stderr:reweighted osd.6 to 1 (10000) 2024-09-21T15:50:35.679 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:35.678+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:35.686 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T15:50:35.693 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-09-21T15:50:35.729 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:35.729 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:35.729 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:50:35.729 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:35.739 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:50:35.780 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:35.779+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:35.881 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:35.880+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:35.889 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:35.889 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:35.889 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:50:35.889 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:50:35.889 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:35.898 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:50:35.983 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:35.982+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:36.044 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:36.044 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:50:36.044 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:50:36.044 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:50:36.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.044 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:984 3.0 e27/22 3:025e53e7:::rbd_data.1133f3dfbd6b.0000000000000008:head v 27'21, mlcod=27'19)", 2024-09-21T15:50:36.045 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.619393+0000", 2024-09-21T15:50:36.045 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 83.423269671, 2024-09-21T15:50:36.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003401837, 2024-09-21T15:50:36.045 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:36.045 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:36.045 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:36.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:36.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619393+0000", 2024-09-21T15:50:36.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:36.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619393+0000", 2024-09-21T15:50:36.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.046 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:36.046 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619392+0000", 2024-09-21T15:50:36.046 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:36.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.046 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:36.046 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619439+0000", 2024-09-21T15:50:36.046 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7170999999999999e-05 2024-09-21T15:50:36.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.046 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:36.046 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619452+0000", 2024-09-21T15:50:36.047 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2986999999999999e-05 2024-09-21T15:50:36.047 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.047 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.047 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:36.047 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619459+0000", 2024-09-21T15:50:36.047 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6109999999999999e-06 2024-09-21T15:50:36.047 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.047 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.047 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:36.047 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619518+0000", 2024-09-21T15:50:36.047 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9117999999999999e-05 2024-09-21T15:50:36.047 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.047 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.047 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:36.048 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619532+0000", 2024-09-21T15:50:36.048 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4622e-05 2024-09-21T15:50:36.048 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.048 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.048 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:36.048 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.622760+0000", 2024-09-21T15:50:36.048 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032278839999999999 2024-09-21T15:50:36.048 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.048 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.048 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:36.048 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.622794+0000", 2024-09-21T15:50:36.048 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4144999999999999e-05 2024-09-21T15:50:36.048 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.048 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:36.048 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.049 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.049 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.049 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:4957 3.9 e29/22 3:95c5ed95:::rbd_data.11335fa2fe69.0000000000000018:head v 29'4, mlcod=26'2)", 2024-09-21T15:50:36.049 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:14.453951+0000", 2024-09-21T15:50:36.049 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 81.588711239000006, 2024-09-21T15:50:36.049 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0043804559999999996, 2024-09-21T15:50:36.049 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:36.049 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:36.049 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:36.049 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.049 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:36.049 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.453951+0000", 2024-09-21T15:50:36.049 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.049 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.049 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.050 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:36.050 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.453951+0000", 2024-09-21T15:50:36.050 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.050 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.050 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.050 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:36.050 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.453950+0000", 2024-09-21T15:50:36.050 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:36.050 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.050 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.050 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:36.050 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454495+0000", 2024-09-21T15:50:36.051 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00054437799999999998 2024-09-21T15:50:36.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.051 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.051 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:36.051 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454508+0000", 2024-09-21T15:50:36.051 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3475000000000001e-05 2024-09-21T15:50:36.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.051 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.051 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:36.051 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454515+0000", 2024-09-21T15:50:36.051 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8990000000000004e-06 2024-09-21T15:50:36.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.051 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.051 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:36.051 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454581+0000", 2024-09-21T15:50:36.052 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5987000000000004e-05 2024-09-21T15:50:36.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.052 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.052 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:36.052 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454609+0000", 2024-09-21T15:50:36.052 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7466000000000001e-05 2024-09-21T15:50:36.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.052 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.052 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:36.052 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.458303+0000", 2024-09-21T15:50:36.052 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003693979 2024-09-21T15:50:36.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.052 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.052 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:36.052 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.458331+0000", 2024-09-21T15:50:36.053 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8932999999999999e-05 2024-09-21T15:50:36.053 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.053 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:36.053 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.053 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.053 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.053 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13420 3.0 e31/22 3:038b8840:::rbd_data.1133dc5adfdf.0000000000000035:head v 31'49, mlcod=31'47)", 2024-09-21T15:50:36.053 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.209233+0000", 2024-09-21T15:50:36.053 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 78.833429623000001, 2024-09-21T15:50:36.053 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0042624960000000002, 2024-09-21T15:50:36.053 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:36.053 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:36.053 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:36.053 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.054 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:36.054 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209233+0000", 2024-09-21T15:50:36.054 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.054 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.054 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.054 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:36.054 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209233+0000", 2024-09-21T15:50:36.054 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.054 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.054 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.054 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:36.054 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209232+0000", 2024-09-21T15:50:36.054 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:36.054 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.054 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.055 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:36.055 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209251+0000", 2024-09-21T15:50:36.055 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9304e-05 2024-09-21T15:50:36.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.055 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.055 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:36.055 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209265+0000", 2024-09-21T15:50:36.055 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3532e-05 2024-09-21T15:50:36.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.055 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.055 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:36.055 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209272+0000", 2024-09-21T15:50:36.055 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.289e-06 2024-09-21T15:50:36.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.055 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.055 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:36.056 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209321+0000", 2024-09-21T15:50:36.056 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8773999999999998e-05 2024-09-21T15:50:36.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.056 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.056 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:36.056 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209341+0000", 2024-09-21T15:50:36.056 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0387999999999999e-05 2024-09-21T15:50:36.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.056 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.056 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:36.056 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.213464+0000", 2024-09-21T15:50:36.056 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0041232450000000002 2024-09-21T15:50:36.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.057 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:36.057 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.213495+0000", 2024-09-21T15:50:36.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0793999999999997e-05 2024-09-21T15:50:36.057 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.057 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:36.057 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.057 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13445 3.0 e31/22 3:038b8840:::rbd_data.1133dc5adfdf.0000000000000035:head v 31'50, mlcod=31'48)", 2024-09-21T15:50:36.057 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.232497+0000", 2024-09-21T15:50:36.057 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 78.810165611000002, 2024-09-21T15:50:36.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0040493559999999996, 2024-09-21T15:50:36.057 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:36.057 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:36.057 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:36.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.058 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:36.058 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232497+0000", 2024-09-21T15:50:36.058 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.058 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:36.058 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232497+0000", 2024-09-21T15:50:36.058 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.058 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:36.058 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232496+0000", 2024-09-21T15:50:36.058 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:36.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.059 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:36.059 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232503+0000", 2024-09-21T15:50:36.059 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7320000000000001e-06 2024-09-21T15:50:36.059 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.059 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.059 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:36.059 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232512+0000", 2024-09-21T15:50:36.059 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9259999999999995e-06 2024-09-21T15:50:36.059 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.059 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.059 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:36.059 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232520+0000", 2024-09-21T15:50:36.059 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3699999999999997e-06 2024-09-21T15:50:36.059 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.059 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.060 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:36.060 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232564+0000", 2024-09-21T15:50:36.060 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4128999999999997e-05 2024-09-21T15:50:36.060 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.060 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.060 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:36.060 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232585+0000", 2024-09-21T15:50:36.060 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0704999999999999e-05 2024-09-21T15:50:36.060 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.060 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.060 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:36.060 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.236507+0000", 2024-09-21T15:50:36.060 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0039218730000000002 2024-09-21T15:50:36.060 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.060 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.061 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:36.061 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.236546+0000", 2024-09-21T15:50:36.061 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9399e-05 2024-09-21T15:50:36.061 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.061 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:36.061 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.061 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13452 3.9 e31/22 3:93f5396a:::rbd_data.1133dc5adfdf.0000000000000078:head v 31'50, mlcod=31'47)", 2024-09-21T15:50:36.061 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.237263+0000", 2024-09-21T15:50:36.061 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 78.805398905000004, 2024-09-21T15:50:36.061 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033768180000000002, 2024-09-21T15:50:36.062 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:36.062 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:36.062 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:36.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.062 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:36.062 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237263+0000", 2024-09-21T15:50:36.062 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.062 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:36.062 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237263+0000", 2024-09-21T15:50:36.062 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.062 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:36.063 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237263+0000", 2024-09-21T15:50:36.063 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:36.063 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.063 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.063 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:36.063 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237269+0000", 2024-09-21T15:50:36.063 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.472e-06 2024-09-21T15:50:36.063 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.063 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.063 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:36.063 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237282+0000", 2024-09-21T15:50:36.063 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2788e-05 2024-09-21T15:50:36.063 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.063 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.063 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:36.064 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237288+0000", 2024-09-21T15:50:36.064 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4899999999999997e-06 2024-09-21T15:50:36.064 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.064 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.064 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:36.064 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237331+0000", 2024-09-21T15:50:36.064 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2425000000000001e-05 2024-09-21T15:50:36.064 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.064 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.064 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:36.064 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237349+0000", 2024-09-21T15:50:36.064 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8405000000000001e-05 2024-09-21T15:50:36.064 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.064 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.064 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:36.064 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.240610+0000", 2024-09-21T15:50:36.065 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032605379999999999 2024-09-21T15:50:36.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.065 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:36.065 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.240640+0000", 2024-09-21T15:50:36.065 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0448000000000001e-05 2024-09-21T15:50:36.065 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.065 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:36.065 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.065 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13832 3.0 e31/22 3:0792efd2:::rbd_data.1133dc5adfdf.0000000000000090:head v 31'58, mlcod=31'56)", 2024-09-21T15:50:36.065 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.527530+0000", 2024-09-21T15:50:36.065 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 78.515132085000005, 2024-09-21T15:50:36.066 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035256530000000001, 2024-09-21T15:50:36.066 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:36.066 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:36.066 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:36.066 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.066 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:36.066 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527530+0000", 2024-09-21T15:50:36.066 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.066 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.066 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:36.066 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527530+0000", 2024-09-21T15:50:36.066 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.067 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:36.067 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527529+0000", 2024-09-21T15:50:36.067 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:50:36.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.067 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:36.067 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527550+0000", 2024-09-21T15:50:36.067 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1387999999999999e-05 2024-09-21T15:50:36.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.067 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:36.067 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527568+0000", 2024-09-21T15:50:36.067 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8068e-05 2024-09-21T15:50:36.068 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.068 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.068 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:36.068 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527577+0000", 2024-09-21T15:50:36.068 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9919999999999996e-06 2024-09-21T15:50:36.068 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.068 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.068 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:36.068 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527626+0000", 2024-09-21T15:50:36.068 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8597000000000002e-05 2024-09-21T15:50:36.068 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.068 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.068 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:36.068 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527652+0000", 2024-09-21T15:50:36.068 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5993e-05 2024-09-21T15:50:36.068 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.068 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.069 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:36.069 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.531024+0000", 2024-09-21T15:50:36.069 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033721910000000001 2024-09-21T15:50:36.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.069 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:36.069 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.531056+0000", 2024-09-21T15:50:36.069 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1628000000000003e-05 2024-09-21T15:50:36.069 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.069 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:36.069 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.069 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13839 3.0 e31/22 3:03c4fb32:::rbd_data.1133dc5adfdf.0000000000000099:head v 31'59, mlcod=31'56)", 2024-09-21T15:50:36.069 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.528097+0000", 2024-09-21T15:50:36.069 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 78.514565486999999, 2024-09-21T15:50:36.070 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0057228840000000001, 2024-09-21T15:50:36.070 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:36.070 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:36.070 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:36.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.070 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:36.070 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528097+0000", 2024-09-21T15:50:36.070 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.070 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.070 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:36.070 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528097+0000", 2024-09-21T15:50:36.070 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.071 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:36.071 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528096+0000", 2024-09-21T15:50:36.071 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:36.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.071 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:36.071 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528105+0000", 2024-09-21T15:50:36.071 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7350000000000002e-06 2024-09-21T15:50:36.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.071 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:36.071 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528117+0000", 2024-09-21T15:50:36.071 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1909000000000001e-05 2024-09-21T15:50:36.072 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.072 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.072 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:36.072 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528124+0000", 2024-09-21T15:50:36.072 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8419999999999999e-06 2024-09-21T15:50:36.072 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.072 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.072 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:36.072 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528168+0000", 2024-09-21T15:50:36.072 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4301999999999997e-05 2024-09-21T15:50:36.072 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.072 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.072 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:36.072 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528191+0000", 2024-09-21T15:50:36.072 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2622000000000001e-05 2024-09-21T15:50:36.073 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.073 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.073 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:36.073 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.533780+0000", 2024-09-21T15:50:36.073 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0055897519999999999 2024-09-21T15:50:36.073 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.073 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.073 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:36.073 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.533820+0000", 2024-09-21T15:50:36.073 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9366999999999998e-05 2024-09-21T15:50:36.073 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.073 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:36.073 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.073 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.073 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.074 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13884 3.9 e31/22 3:93003736:::rbd_data.1133dc5adfdf.00000000000000a1:head v 31'54, mlcod=31'52)", 2024-09-21T15:50:36.074 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.562433+0000", 2024-09-21T15:50:36.074 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 78.480228738999998, 2024-09-21T15:50:36.074 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.005162313, 2024-09-21T15:50:36.074 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:36.074 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:36.074 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:36.074 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.074 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:36.074 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562433+0000", 2024-09-21T15:50:36.074 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.074 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.074 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:36.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562433+0000", 2024-09-21T15:50:36.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:36.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562432+0000", 2024-09-21T15:50:36.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:36.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:36.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562467+0000", 2024-09-21T15:50:36.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4885999999999998e-05 2024-09-21T15:50:36.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.076 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:36.076 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562481+0000", 2024-09-21T15:50:36.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3604000000000001e-05 2024-09-21T15:50:36.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.076 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:36.076 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562489+0000", 2024-09-21T15:50:36.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1829999999999994e-06 2024-09-21T15:50:36.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.076 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:36.076 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562536+0000", 2024-09-21T15:50:36.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6830999999999999e-05 2024-09-21T15:50:36.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.077 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.077 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:36.077 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562559+0000", 2024-09-21T15:50:36.077 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2765999999999999e-05 2024-09-21T15:50:36.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.077 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.077 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:36.077 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.567566+0000", 2024-09-21T15:50:36.077 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0050073219999999998 2024-09-21T15:50:36.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.077 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.077 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:36.078 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.567596+0000", 2024-09-21T15:50:36.078 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9802999999999999e-05 2024-09-21T15:50:36.078 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.078 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:36.078 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.078 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.078 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:22340 3.9 e34/22 3:910a5abe:::rbd_data.1133896adcfd.0000000000000023:head v 34'92, mlcod=34'90)", 2024-09-21T15:50:36.078 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:21.628025+0000", 2024-09-21T15:50:36.078 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 74.414637358999997, 2024-09-21T15:50:36.078 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046196010000000001, 2024-09-21T15:50:36.078 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:36.078 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:36.079 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:36.079 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.079 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:36.079 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628025+0000", 2024-09-21T15:50:36.079 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.079 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.079 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:36.079 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628025+0000", 2024-09-21T15:50:36.079 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.079 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.079 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:36.079 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628024+0000", 2024-09-21T15:50:36.079 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:36.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.080 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:36.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628034+0000", 2024-09-21T15:50:36.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3470000000000005e-06 2024-09-21T15:50:36.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.080 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:36.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628043+0000", 2024-09-21T15:50:36.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9450000000000005e-06 2024-09-21T15:50:36.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.080 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:36.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628048+0000", 2024-09-21T15:50:36.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8080000000000004e-06 2024-09-21T15:50:36.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.081 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:36.081 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628081+0000", 2024-09-21T15:50:36.081 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2738999999999999e-05 2024-09-21T15:50:36.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.081 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:36.081 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628094+0000", 2024-09-21T15:50:36.081 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3273e-05 2024-09-21T15:50:36.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.081 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:36.081 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.632616+0000", 2024-09-21T15:50:36.081 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045217269999999997 2024-09-21T15:50:36.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.082 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:36.082 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.632644+0000", 2024-09-21T15:50:36.082 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8469000000000001e-05 2024-09-21T15:50:36.082 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.082 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:36.082 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.082 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.082 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.082 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:22417 3.4 e34/22 3:25310a10:::rbd_data.1133896adcfd.0000000000000075:head v 34'48, mlcod=34'46)", 2024-09-21T15:50:36.082 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:21.704297+0000", 2024-09-21T15:50:36.082 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 74.338365261000007, 2024-09-21T15:50:36.082 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0038101799999999998, 2024-09-21T15:50:36.082 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:36.083 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:36.083 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:36.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.083 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:36.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704297+0000", 2024-09-21T15:50:36.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.083 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:36.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704297+0000", 2024-09-21T15:50:36.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.083 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:36.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704296+0000", 2024-09-21T15:50:36.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:50:36.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.084 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:36.084 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704308+0000", 2024-09-21T15:50:36.084 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2559e-05 2024-09-21T15:50:36.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.084 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:36.084 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704337+0000", 2024-09-21T15:50:36.084 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8960000000000001e-05 2024-09-21T15:50:36.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.084 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:36.084 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704345+0000", 2024-09-21T15:50:36.084 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3059999999999998e-06 2024-09-21T15:50:36.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.085 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.085 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:36.085 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704408+0000", 2024-09-21T15:50:36.085 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2156000000000006e-05 2024-09-21T15:50:36.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.085 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.085 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:36.085 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704430+0000", 2024-09-21T15:50:36.085 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1931000000000001e-05 2024-09-21T15:50:36.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.085 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.085 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:36.085 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.708073+0000", 2024-09-21T15:50:36.085 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003643028 2024-09-21T15:50:36.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.086 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:36.086 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.708107+0000", 2024-09-21T15:50:36.086 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4613999999999998e-05 2024-09-21T15:50:36.086 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.086 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:36.086 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.086 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:25278 3.4 e38/22 3:2251cc1c:::rbd_data.1133ebd6d3ba.0000000000000024:head v 38'51, mlcod=38'49)", 2024-09-21T15:50:36.086 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.317337+0000", 2024-09-21T15:50:36.086 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 69.725325501, 2024-09-21T15:50:36.086 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053904110000000003, 2024-09-21T15:50:36.086 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:36.086 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:36.087 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:36.087 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.087 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:36.087 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317337+0000", 2024-09-21T15:50:36.087 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.087 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.087 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.087 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:36.087 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317337+0000", 2024-09-21T15:50:36.087 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.087 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.087 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.087 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:36.087 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317335+0000", 2024-09-21T15:50:36.087 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:50:36.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.088 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:36.088 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317347+0000", 2024-09-21T15:50:36.088 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.19e-05 2024-09-21T15:50:36.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.088 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:36.088 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317369+0000", 2024-09-21T15:50:36.088 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2232e-05 2024-09-21T15:50:36.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.088 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:36.088 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317381+0000", 2024-09-21T15:50:36.088 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2231e-05 2024-09-21T15:50:36.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.089 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.089 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:36.089 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317425+0000", 2024-09-21T15:50:36.089 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3625000000000003e-05 2024-09-21T15:50:36.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.089 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.089 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:36.089 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317449+0000", 2024-09-21T15:50:36.089 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4301999999999999e-05 2024-09-21T15:50:36.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.089 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.089 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:36.089 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.322688+0000", 2024-09-21T15:50:36.089 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.005238609 2024-09-21T15:50:36.090 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.090 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.090 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:36.090 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.322727+0000", 2024-09-21T15:50:36.090 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9201e-05 2024-09-21T15:50:36.090 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.090 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:36.090 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.090 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.090 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.090 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:25286 3.0 e38/22 3:0248fb60:::rbd_data.1133ebd6d3ba.0000000000000066:head v 38'109, mlcod=34'107)", 2024-09-21T15:50:36.090 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.317638+0000", 2024-09-21T15:50:36.090 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 69.725024622000007, 2024-09-21T15:50:36.090 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0054074370000000002, 2024-09-21T15:50:36.091 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:36.091 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:36.091 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:36.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.091 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:36.091 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317638+0000", 2024-09-21T15:50:36.091 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.091 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:36.091 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317638+0000", 2024-09-21T15:50:36.091 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.091 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:36.091 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317637+0000", 2024-09-21T15:50:36.092 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:36.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.092 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:36.092 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317643+0000", 2024-09-21T15:50:36.092 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5609999999999998e-06 2024-09-21T15:50:36.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.092 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:36.092 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317653+0000", 2024-09-21T15:50:36.092 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0705e-05 2024-09-21T15:50:36.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.092 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:36.092 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317660+0000", 2024-09-21T15:50:36.093 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2099999999999998e-06 2024-09-21T15:50:36.093 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.093 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.093 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:36.093 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317704+0000", 2024-09-21T15:50:36.093 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4659000000000001e-05 2024-09-21T15:50:36.093 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.093 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.093 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:36.093 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317725+0000", 2024-09-21T15:50:36.093 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0537e-05 2024-09-21T15:50:36.093 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.093 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.093 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:36.093 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323010+0000", 2024-09-21T15:50:36.093 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052853420000000002 2024-09-21T15:50:36.093 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.094 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.094 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:36.094 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323045+0000", 2024-09-21T15:50:36.094 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4860000000000002e-05 2024-09-21T15:50:36.094 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.094 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:36.094 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.094 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.094 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:25280 3.4 e38/22 3:27fca38c:::rbd_data.1133ebd6d3ba.0000000000000026:head v 38'52, mlcod=38'49)", 2024-09-21T15:50:36.094 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.317684+0000", 2024-09-21T15:50:36.094 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 69.724978726000003, 2024-09-21T15:50:36.094 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053570809999999996, 2024-09-21T15:50:36.095 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:36.095 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:36.095 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:36.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.095 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:36.095 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317684+0000", 2024-09-21T15:50:36.095 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.095 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.095 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:36.095 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317684+0000", 2024-09-21T15:50:36.095 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.095 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:36.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317683+0000", 2024-09-21T15:50:36.096 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:36.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:36.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317688+0000", 2024-09-21T15:50:36.096 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5009999999999998e-06 2024-09-21T15:50:36.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:36.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317694+0000", 2024-09-21T15:50:36.096 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3579999999999999e-06 2024-09-21T15:50:36.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.097 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:36.097 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317698+0000", 2024-09-21T15:50:36.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4150000000000001e-06 2024-09-21T15:50:36.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.097 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:36.097 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317738+0000", 2024-09-21T15:50:36.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9575e-05 2024-09-21T15:50:36.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.097 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:36.097 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317762+0000", 2024-09-21T15:50:36.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4173999999999999e-05 2024-09-21T15:50:36.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:36.098 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323005+0000", 2024-09-21T15:50:36.098 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00524333 2024-09-21T15:50:36.098 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:36.098 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323041+0000", 2024-09-21T15:50:36.098 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5139999999999999e-05 2024-09-21T15:50:36.098 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.098 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:36.098 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.098 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.099 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:34837 3.0 e43/22 3:038527c8:::rbd_header.1133fed1a840:head v 43'130, mlcod=43'128)", 2024-09-21T15:50:36.099 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:50:36.099 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 65.058774028000002, 2024-09-21T15:50:36.099 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045776519999999998, 2024-09-21T15:50:36.099 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:36.099 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:36.099 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:36.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.099 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:36.099 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:50:36.099 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.100 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:36.100 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:50:36.100 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.100 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:36.100 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:50:36.100 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:36.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.101 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:36.101 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983892+0000", 2024-09-21T15:50:36.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8450000000000002e-06 2024-09-21T15:50:36.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.101 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:36.101 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983902+0000", 2024-09-21T15:50:36.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0299999999999999e-06 2024-09-21T15:50:36.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.101 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:36.101 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983906+0000", 2024-09-21T15:50:36.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2300000000000002e-06 2024-09-21T15:50:36.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.102 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:36.102 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983940+0000", 2024-09-21T15:50:36.102 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4563000000000001e-05 2024-09-21T15:50:36.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.102 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:36.102 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983960+0000", 2024-09-21T15:50:36.102 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9199999999999999e-05 2024-09-21T15:50:36.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.102 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:36.103 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.988430+0000", 2024-09-21T15:50:36.103 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0044709149999999998 2024-09-21T15:50:36.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.103 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:36.103 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.988466+0000", 2024-09-21T15:50:36.103 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5420000000000003e-05 2024-09-21T15:50:36.103 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.103 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:36.103 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.103 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "pg_update_log_missing(3.0 epoch 50/22 rep_tid 498 entries 50'171 (0'0) error 3:5df4a5a0:::rbd_data.11339e87a32a.000000000000008e:head by client.4403.0:80207 0.000000 -2 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 47'167)", 2024-09-21T15:50:36.103 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:50:36.104 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 55.100901155000003, 2024-09-21T15:50:36.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0047253119999999997, 2024-09-21T15:50:36.104 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:36.104 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-09-21T15:50:36.104 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:36.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.104 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:36.104 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:50:36.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.104 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:36.104 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:50:36.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.105 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:36.105 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:50:36.105 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:36.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.105 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:36.105 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941768+0000", 2024-09-21T15:50:36.105 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7309999999999993e-06 2024-09-21T15:50:36.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.105 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:36.105 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941780+0000", 2024-09-21T15:50:36.105 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1399999999999999e-05 2024-09-21T15:50:36.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.106 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:36.106 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941789+0000", 2024-09-21T15:50:36.106 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5230000000000001e-06 2024-09-21T15:50:36.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.106 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:36.106 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941841+0000", 2024-09-21T15:50:36.106 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1863999999999997e-05 2024-09-21T15:50:36.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.106 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:36.106 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.946486+0000", 2024-09-21T15:50:36.107 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046452539999999997 2024-09-21T15:50:36.107 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.107 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:36.107 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.107 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.107 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.107 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "pg_update_log_missing(3.0 epoch 53/22 rep_tid 585 entries 53'201 (0'0) error 3:6a1d4890:::rbd_data.1133e079cae4.0000000000000004:head by client.4403.0:101089 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 53'198)", 2024-09-21T15:50:36.107 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:44.832827+0000", 2024-09-21T15:50:36.107 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 51.209835347000002, 2024-09-21T15:50:36.107 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053552929999999997, 2024-09-21T15:50:36.107 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:36.107 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-09-21T15:50:36.107 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:36.107 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.107 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:36.108 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832827+0000", 2024-09-21T15:50:36.108 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.108 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.108 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.108 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:36.108 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832827+0000", 2024-09-21T15:50:36.108 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.108 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.108 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.108 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:36.108 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832826+0000", 2024-09-21T15:50:36.108 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:36.108 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.108 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.108 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:36.109 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832835+0000", 2024-09-21T15:50:36.109 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4509999999999992e-06 2024-09-21T15:50:36.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.109 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:36.109 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832854+0000", 2024-09-21T15:50:36.109 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9352999999999999e-05 2024-09-21T15:50:36.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.109 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:36.109 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832866+0000", 2024-09-21T15:50:36.109 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1766000000000001e-05 2024-09-21T15:50:36.110 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.110 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.110 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:36.110 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.833091+0000", 2024-09-21T15:50:36.110 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00022490300000000001 2024-09-21T15:50:36.110 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.110 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.110 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:36.110 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.838182+0000", 2024-09-21T15:50:36.110 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0050915420000000001 2024-09-21T15:50:36.110 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.110 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:36.110 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.110 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.111 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:102822 3.4 e54/22 3:2729ba03:::rbd_object_map.113336ae1075:head v 54'208, mlcod=54'205)", 2024-09-21T15:50:36.111 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:46.249540+0000", 2024-09-21T15:50:36.111 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 49.793122115000003, 2024-09-21T15:50:36.111 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052654859999999998, 2024-09-21T15:50:36.111 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:36.111 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:36.111 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:36.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.111 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:36.111 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249540+0000", 2024-09-21T15:50:36.111 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.112 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:36.112 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249540+0000", 2024-09-21T15:50:36.112 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.112 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:36.112 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249539+0000", 2024-09-21T15:50:36.112 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:36.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.112 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:36.112 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249546+0000", 2024-09-21T15:50:36.113 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7290000000000003e-06 2024-09-21T15:50:36.113 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.113 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:36.113 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249558+0000", 2024-09-21T15:50:36.113 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1844e-05 2024-09-21T15:50:36.113 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.113 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:36.113 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249565+0000", 2024-09-21T15:50:36.113 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8210000000000004e-06 2024-09-21T15:50:36.113 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.113 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:36.114 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249602+0000", 2024-09-21T15:50:36.114 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7057000000000001e-05 2024-09-21T15:50:36.114 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.114 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:36.114 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249622+0000", 2024-09-21T15:50:36.114 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9678999999999999e-05 2024-09-21T15:50:36.114 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.114 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:36.114 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254783+0000", 2024-09-21T15:50:36.114 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0051611449999999998 2024-09-21T15:50:36.114 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:36.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254806+0000", 2024-09-21T15:50:36.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2853e-05 2024-09-21T15:50:36.115 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.115 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:36.115 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.115 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:102823 3.4 e54/22 3:2729ba03:::rbd_object_map.113336ae1075:head v 54'209, mlcod=54'205)", 2024-09-21T15:50:36.115 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:46.250318+0000", 2024-09-21T15:50:36.115 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 49.792344530000001, 2024-09-21T15:50:36.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045013539999999999, 2024-09-21T15:50:36.115 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:36.116 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:36.116 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:36.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.116 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:36.116 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250318+0000", 2024-09-21T15:50:36.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.116 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:36.116 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250318+0000", 2024-09-21T15:50:36.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.116 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:36.117 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250317+0000", 2024-09-21T15:50:36.117 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:36.117 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.117 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.117 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:36.117 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250322+0000", 2024-09-21T15:50:36.117 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.074e-06 2024-09-21T15:50:36.117 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.117 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.117 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:36.117 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250330+0000", 2024-09-21T15:50:36.117 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4730000000000003e-06 2024-09-21T15:50:36.117 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.117 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.118 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:36.118 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250335+0000", 2024-09-21T15:50:36.118 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8019999999999997e-06 2024-09-21T15:50:36.118 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.118 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.118 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:36.118 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250375+0000", 2024-09-21T15:50:36.118 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9459999999999998e-05 2024-09-21T15:50:36.118 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.118 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.118 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:36.119 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250388+0000", 2024-09-21T15:50:36.119 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3288999999999999e-05 2024-09-21T15:50:36.119 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.119 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.119 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:36.119 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254807+0000", 2024-09-21T15:50:36.119 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0044192220000000004 2024-09-21T15:50:36.119 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.119 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.119 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:36.119 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254819+0000", 2024-09-21T15:50:36.119 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1603999999999999e-05 2024-09-21T15:50:36.119 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.119 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:36.119 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.120 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.120 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:160603 3.9 e95/22 3:9171211b:::rbd_data.11338bae974a.0000000000000006:head v 95'343, mlcod=92'340)", 2024-09-21T15:50:36.120 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.215621+0000", 2024-09-21T15:50:36.120 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.8270414989999999, 2024-09-21T15:50:36.120 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0041276250000000002, 2024-09-21T15:50:36.120 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:36.120 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:36.120 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:36.120 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.120 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:36.120 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215621+0000", 2024-09-21T15:50:36.120 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.121 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:36.121 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215621+0000", 2024-09-21T15:50:36.121 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.121 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:36.121 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215620+0000", 2024-09-21T15:50:36.121 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:36.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.121 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:36.121 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215627+0000", 2024-09-21T15:50:36.121 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8850000000000001e-06 2024-09-21T15:50:36.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.122 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.122 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:36.122 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215803+0000", 2024-09-21T15:50:36.122 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000176422 2024-09-21T15:50:36.122 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.122 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.122 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:36.122 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215814+0000", 2024-09-21T15:50:36.122 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0135e-05 2024-09-21T15:50:36.122 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.122 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.122 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:36.122 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215866+0000", 2024-09-21T15:50:36.122 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2327999999999998e-05 2024-09-21T15:50:36.123 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.123 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:36.123 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215887+0000", 2024-09-21T15:50:36.123 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1449000000000001e-05 2024-09-21T15:50:36.123 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.123 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:36.123 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.219719+0000", 2024-09-21T15:50:36.123 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0038312630000000001 2024-09-21T15:50:36.123 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.123 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:36.123 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.219748+0000", 2024-09-21T15:50:36.123 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9737999999999999e-05 2024-09-21T15:50:36.124 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.124 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:36.124 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.124 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.124 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.124 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:160974 3.4 e100/99 3:22dd2475:::rbd_id.image_client.0-clone39:head v 100'593, mlcod=0'0)", 2024-09-21T15:50:36.124 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:35.683900+0000", 2024-09-21T15:50:36.124 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.35876206500000002, 2024-09-21T15:50:36.124 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0034160029999999999, 2024-09-21T15:50:36.124 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:36.124 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:36.124 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:36.124 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.124 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:36.124 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.683900+0000", 2024-09-21T15:50:36.125 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.125 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.125 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:36.125 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.683900+0000", 2024-09-21T15:50:36.125 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.125 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.125 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:36.125 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.683900+0000", 2024-09-21T15:50:36.125 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:36.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.125 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.125 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:36.126 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.683906+0000", 2024-09-21T15:50:36.126 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1759999999999998e-06 2024-09-21T15:50:36.126 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.126 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.126 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:36.126 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.683915+0000", 2024-09-21T15:50:36.126 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4320000000000005e-06 2024-09-21T15:50:36.126 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.126 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.126 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:36.126 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.683923+0000", 2024-09-21T15:50:36.126 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.259e-06 2024-09-21T15:50:36.126 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.126 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.127 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:36.127 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.683961+0000", 2024-09-21T15:50:36.127 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7014000000000002e-05 2024-09-21T15:50:36.127 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.127 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.127 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:36.127 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.683982+0000", 2024-09-21T15:50:36.127 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.145e-05 2024-09-21T15:50:36.127 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.127 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.127 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:36.127 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.687265+0000", 2024-09-21T15:50:36.127 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003283252 2024-09-21T15:50:36.127 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.127 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.128 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:36.128 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.687316+0000", 2024-09-21T15:50:36.128 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0963e-05 2024-09-21T15:50:36.128 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.128 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:36.128 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.128 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.128 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:36.128 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:36.129 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:50:36.130 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:50:36.128+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:50:36.231 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:36.230+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:36.272 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:36.272 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:36.272 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:50:36.272 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:36.280 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:50:36.332 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:36.331+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:36.383 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:36.383 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:36.383 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:50:36.383 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:50:36.383 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:36.392 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:50:36.435 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:50:36.434+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:50:36.495 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:36.495 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:50:36.495 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:50:36.495 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:50:36.495 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.495 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:14051 3.8 3:13437810:::rbd_data.1133dc5adfdf.0000000000000039:head [write 716800~796] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-21T15:50:36.496 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:50:36.496 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 78.798553706000007, 2024-09-21T15:50:36.496 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.98064002699999997, 2024-09-21T15:50:36.496 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:36.496 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:36.496 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:36.496 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:36.496 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:36.496 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 14051 2024-09-21T15:50:36.496 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.496 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:36.496 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.496 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:36.496 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:50:36.497 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.497 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.497 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.497 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:36.497 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:50:36.497 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.497 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.497 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.497 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:36.497 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:50:36.497 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:36.497 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.498 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.498 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:36.498 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695131+0000", 2024-09-21T15:50:36.498 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4680000000000003e-06 2024-09-21T15:50:36.498 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.498 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.498 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:36.498 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695139+0000", 2024-09-21T15:50:36.498 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9400000000000002e-06 2024-09-21T15:50:36.498 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.498 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.498 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:36.498 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695143+0000", 2024-09-21T15:50:36.499 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9899999999999999e-06 2024-09-21T15:50:36.499 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.499 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.499 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:36.499 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695190+0000", 2024-09-21T15:50:36.499 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6458000000000001e-05 2024-09-21T15:50:36.499 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.499 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.499 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:50:36.499 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695201+0000", 2024-09-21T15:50:36.499 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0964999999999999e-05 2024-09-21T15:50:36.499 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.499 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.499 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:36.499 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.669490+0000", 2024-09-21T15:50:36.500 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.97428980499999995 2024-09-21T15:50:36.500 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.500 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.500 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-09-21T15:50:36.500 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.670033+0000", 2024-09-21T15:50:36.500 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00054252599999999999 2024-09-21T15:50:36.500 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.500 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.500 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:36.500 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.671095+0000", 2024-09-21T15:50:36.500 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001062183 2024-09-21T15:50:36.500 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.500 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.500 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:36.500 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.671186+0000", 2024-09-21T15:50:36.501 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1191000000000002e-05 2024-09-21T15:50:36.501 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.501 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.501 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-09-21T15:50:36.501 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.671346+0000", 2024-09-21T15:50:36.501 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015980399999999999 2024-09-21T15:50:36.501 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.501 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.501 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:36.501 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.672221+0000", 2024-09-21T15:50:36.501 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00087471999999999995 2024-09-21T15:50:36.501 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.501 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.501 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:36.502 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.675619+0000", 2024-09-21T15:50:36.502 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00339858 2024-09-21T15:50:36.502 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.502 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.502 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:36.502 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.675670+0000", 2024-09-21T15:50:36.502 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0664000000000001e-05 2024-09-21T15:50:36.502 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.502 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.502 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:36.502 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.675768+0000", 2024-09-21T15:50:36.502 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7415000000000002e-05 2024-09-21T15:50:36.503 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.503 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:36.503 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.503 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.503 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.503 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116235 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 761856~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:36.503 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048806+0000", 2024-09-21T15:50:36.503 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 40.444874822000003, 2024-09-21T15:50:36.503 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056818556999999999, 2024-09-21T15:50:36.503 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:36.503 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:36.503 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:36.503 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:36.503 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:36.503 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116235 2024-09-21T15:50:36.503 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.504 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:36.504 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.504 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:36.504 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048806+0000", 2024-09-21T15:50:36.504 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.504 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.504 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.504 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:36.504 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048806+0000", 2024-09-21T15:50:36.504 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.504 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.504 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.504 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:36.504 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048806+0000", 2024-09-21T15:50:36.505 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:36.505 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.505 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.505 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:36.505 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048807+0000", 2024-09-21T15:50:36.505 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.001e-06 2024-09-21T15:50:36.505 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.505 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.505 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:36.505 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048810+0000", 2024-09-21T15:50:36.505 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.711e-06 2024-09-21T15:50:36.505 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.505 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.505 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:36.505 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048812+0000", 2024-09-21T15:50:36.505 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2290000000000002e-06 2024-09-21T15:50:36.506 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.506 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.506 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:36.506 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105495+0000", 2024-09-21T15:50:36.506 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056682679999999999 2024-09-21T15:50:36.506 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.506 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.506 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:36.506 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105565+0000", 2024-09-21T15:50:36.506 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9868000000000003e-05 2024-09-21T15:50:36.506 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.506 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.506 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:36.506 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105625+0000", 2024-09-21T15:50:36.506 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0268e-05 2024-09-21T15:50:36.507 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.507 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:36.507 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.507 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.507 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.507 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116236 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 768000~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:36.507 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048814+0000", 2024-09-21T15:50:36.507 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 40.444867158999998, 2024-09-21T15:50:36.507 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056959587999999998, 2024-09-21T15:50:36.507 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:36.507 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:36.507 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:36.507 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:36.507 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:36.507 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116236 2024-09-21T15:50:36.508 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.508 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:36.508 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.508 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:36.508 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048814+0000", 2024-09-21T15:50:36.508 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.508 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.508 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.508 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:36.508 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048814+0000", 2024-09-21T15:50:36.508 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.508 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.508 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.508 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:36.508 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048814+0000", 2024-09-21T15:50:36.509 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:36.509 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.509 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.509 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:36.509 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048815+0000", 2024-09-21T15:50:36.509 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7999999999999993e-07 2024-09-21T15:50:36.509 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.509 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.509 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:36.509 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048817+0000", 2024-09-21T15:50:36.509 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6620000000000001e-06 2024-09-21T15:50:36.510 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.510 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.510 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:36.510 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048820+0000", 2024-09-21T15:50:36.510 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2800000000000002e-06 2024-09-21T15:50:36.510 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.510 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.510 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:36.510 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105647+0000", 2024-09-21T15:50:36.510 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056827253000000001 2024-09-21T15:50:36.510 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.510 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.510 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:36.511 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105716+0000", 2024-09-21T15:50:36.511 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8761000000000001e-05 2024-09-21T15:50:36.511 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.511 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.511 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:36.511 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105774+0000", 2024-09-21T15:50:36.511 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7855000000000001e-05 2024-09-21T15:50:36.511 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.511 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:36.511 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.511 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.511 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.511 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116237 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 774144~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:36.511 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048822+0000", 2024-09-21T15:50:36.511 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 40.444859473999998, 2024-09-21T15:50:36.512 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057114028999999997, 2024-09-21T15:50:36.512 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:36.512 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:36.512 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:36.512 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:36.512 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:36.512 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116237 2024-09-21T15:50:36.512 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.512 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:36.512 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.512 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:36.512 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048822+0000", 2024-09-21T15:50:36.512 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.512 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.513 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.513 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:36.513 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048822+0000", 2024-09-21T15:50:36.513 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.513 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.513 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.513 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:36.513 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048822+0000", 2024-09-21T15:50:36.513 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:36.513 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.513 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.513 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:36.513 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048823+0000", 2024-09-21T15:50:36.513 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9699999999999994e-07 2024-09-21T15:50:36.514 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.514 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.514 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:36.514 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048825+0000", 2024-09-21T15:50:36.514 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.65e-06 2024-09-21T15:50:36.514 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.514 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.514 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:36.514 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048828+0000", 2024-09-21T15:50:36.514 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8890000000000002e-06 2024-09-21T15:50:36.514 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.514 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.514 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:36.514 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105802+0000", 2024-09-21T15:50:36.515 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056974415 2024-09-21T15:50:36.515 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.515 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.515 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:36.515 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105871+0000", 2024-09-21T15:50:36.515 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8727999999999997e-05 2024-09-21T15:50:36.515 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.515 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.515 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:36.515 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105936+0000", 2024-09-21T15:50:36.515 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4554999999999994e-05 2024-09-21T15:50:36.515 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.516 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:36.516 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.516 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.516 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.516 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116238 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 780288~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:36.516 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048830+0000", 2024-09-21T15:50:36.516 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 40.444851215, 2024-09-21T15:50:36.516 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057296574000000003, 2024-09-21T15:50:36.516 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:36.516 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:36.516 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:36.516 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:36.516 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:36.516 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116238 2024-09-21T15:50:36.516 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.517 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:36.517 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.517 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:36.517 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048830+0000", 2024-09-21T15:50:36.517 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.517 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.517 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.517 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:36.517 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048830+0000", 2024-09-21T15:50:36.517 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.517 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.517 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.517 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:36.518 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048830+0000", 2024-09-21T15:50:36.518 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:36.518 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.518 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.518 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:36.518 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048831+0000", 2024-09-21T15:50:36.518 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0160000000000001e-06 2024-09-21T15:50:36.518 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.518 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.518 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:36.518 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048834+0000", 2024-09-21T15:50:36.518 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.813e-06 2024-09-21T15:50:36.518 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.518 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.519 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:36.519 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048836+0000", 2024-09-21T15:50:36.519 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6790000000000001e-06 2024-09-21T15:50:36.519 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.519 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.519 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:36.519 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105961+0000", 2024-09-21T15:50:36.519 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057124705999999997 2024-09-21T15:50:36.519 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.519 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.519 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:36.519 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106042+0000", 2024-09-21T15:50:36.519 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0820000000000002e-05 2024-09-21T15:50:36.519 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.519 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.520 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:36.520 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106127+0000", 2024-09-21T15:50:36.520 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4746000000000005e-05 2024-09-21T15:50:36.520 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.520 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:36.520 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.520 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.520 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.520 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116324 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 524288~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:36.520 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048838+0000", 2024-09-21T15:50:36.520 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 40.444842905999998, 2024-09-21T15:50:36.521 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057477185, 2024-09-21T15:50:36.521 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:36.521 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:36.521 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:36.521 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:36.521 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:36.521 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116324 2024-09-21T15:50:36.521 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.521 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:36.521 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.521 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:36.521 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048838+0000", 2024-09-21T15:50:36.521 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.521 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.522 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.522 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:36.522 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048838+0000", 2024-09-21T15:50:36.522 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.522 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.522 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.522 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:36.522 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048838+0000", 2024-09-21T15:50:36.522 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:36.522 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.522 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.522 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:36.522 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048842+0000", 2024-09-21T15:50:36.522 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4829999999999999e-06 2024-09-21T15:50:36.523 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.523 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.523 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:36.523 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048846+0000", 2024-09-21T15:50:36.523 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0470000000000004e-06 2024-09-21T15:50:36.523 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.523 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.523 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:36.523 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048849+0000", 2024-09-21T15:50:36.523 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3689999999999998e-06 2024-09-21T15:50:36.523 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.523 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.523 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:36.523 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106157+0000", 2024-09-21T15:50:36.524 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057307942000000001 2024-09-21T15:50:36.524 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.524 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.524 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:36.524 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106237+0000", 2024-09-21T15:50:36.524 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0320000000000003e-05 2024-09-21T15:50:36.524 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.524 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.524 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:36.524 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106315+0000", 2024-09-21T15:50:36.524 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8227999999999997e-05 2024-09-21T15:50:36.525 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.525 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:36.525 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.525 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.525 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.525 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116325 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 530432~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:36.525 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048851+0000", 2024-09-21T15:50:36.525 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 40.444829734000002, 2024-09-21T15:50:36.525 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057581036000000002, 2024-09-21T15:50:36.525 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:36.525 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:36.525 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:36.526 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:36.526 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:36.526 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116325 2024-09-21T15:50:36.526 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.526 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:36.526 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.526 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:36.526 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048851+0000", 2024-09-21T15:50:36.526 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.526 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.526 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.526 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:36.526 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048851+0000", 2024-09-21T15:50:36.526 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.527 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.527 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.527 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:36.527 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048851+0000", 2024-09-21T15:50:36.527 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:36.527 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.527 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.527 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:36.527 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048852+0000", 2024-09-21T15:50:36.527 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.077e-06 2024-09-21T15:50:36.527 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.527 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.527 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:36.527 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048855+0000", 2024-09-21T15:50:36.528 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9859999999999999e-06 2024-09-21T15:50:36.528 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.528 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.528 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:36.528 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048858+0000", 2024-09-21T15:50:36.528 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3810000000000002e-06 2024-09-21T15:50:36.528 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.528 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.528 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:36.528 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106335+0000", 2024-09-21T15:50:36.528 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057477473000000001 2024-09-21T15:50:36.528 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.528 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.528 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:36.528 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106393+0000", 2024-09-21T15:50:36.529 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8050000000000002e-05 2024-09-21T15:50:36.529 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.529 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.529 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:36.529 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106432+0000", 2024-09-21T15:50:36.529 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9274999999999999e-05 2024-09-21T15:50:36.529 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.529 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:36.529 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.529 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.529 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.529 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116326 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 536576~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:36.529 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048860+0000", 2024-09-21T15:50:36.529 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 40.444821683000001, 2024-09-21T15:50:36.529 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057774197999999999, 2024-09-21T15:50:36.530 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:36.530 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:36.530 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:36.530 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:36.530 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:36.530 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116326 2024-09-21T15:50:36.530 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.530 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:36.530 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.530 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:36.530 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048860+0000", 2024-09-21T15:50:36.530 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.530 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.530 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.531 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:36.531 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048860+0000", 2024-09-21T15:50:36.531 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.531 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.531 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.531 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:36.531 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048859+0000", 2024-09-21T15:50:36.531 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:36.531 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.531 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.531 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:36.531 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048860+0000", 2024-09-21T15:50:36.531 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.02e-06 2024-09-21T15:50:36.531 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.532 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.532 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:36.532 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048863+0000", 2024-09-21T15:50:36.532 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1039999999999998e-06 2024-09-21T15:50:36.532 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.532 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.532 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:36.532 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048867+0000", 2024-09-21T15:50:36.532 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.563e-06 2024-09-21T15:50:36.532 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.532 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.532 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:36.532 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106465+0000", 2024-09-21T15:50:36.532 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057597789000000003 2024-09-21T15:50:36.532 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.533 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.533 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:36.533 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106552+0000", 2024-09-21T15:50:36.533 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7602000000000006e-05 2024-09-21T15:50:36.533 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.533 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.533 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:36.533 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106634+0000", 2024-09-21T15:50:36.533 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1321000000000003e-05 2024-09-21T15:50:36.533 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.533 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:36.533 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.533 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.534 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.534 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116327 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 542720~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:36.534 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048870+0000", 2024-09-21T15:50:36.534 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 40.444810810999996, 2024-09-21T15:50:36.534 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057883148000000002, 2024-09-21T15:50:36.534 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:36.534 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:36.534 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:36.534 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:36.534 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:36.534 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116327 2024-09-21T15:50:36.534 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.534 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:36.534 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.535 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:36.535 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048870+0000", 2024-09-21T15:50:36.535 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.535 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.535 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:36.535 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048870+0000", 2024-09-21T15:50:36.535 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.535 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.535 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:36.535 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048870+0000", 2024-09-21T15:50:36.536 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:36.536 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.536 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.536 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:36.536 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048872+0000", 2024-09-21T15:50:36.536 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.824e-06 2024-09-21T15:50:36.536 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.536 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.536 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:36.536 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048879+0000", 2024-09-21T15:50:36.536 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6749999999999996e-06 2024-09-21T15:50:36.536 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.536 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.536 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:36.536 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048883+0000", 2024-09-21T15:50:36.536 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2200000000000003e-06 2024-09-21T15:50:36.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.537 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:36.537 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106663+0000", 2024-09-21T15:50:36.537 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057780049 2024-09-21T15:50:36.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.537 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:36.537 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106711+0000", 2024-09-21T15:50:36.537 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8433999999999997e-05 2024-09-21T15:50:36.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.537 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:36.537 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106754+0000", 2024-09-21T15:50:36.537 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2236e-05 2024-09-21T15:50:36.538 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.538 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:36.538 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.538 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.538 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116328 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 548864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:36.538 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048887+0000", 2024-09-21T15:50:36.538 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 40.444794291000001, 2024-09-21T15:50:36.538 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057981984, 2024-09-21T15:50:36.538 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:36.538 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:36.538 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:36.538 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:36.539 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:36.539 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116328 2024-09-21T15:50:36.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.539 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:36.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.539 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:36.539 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048887+0000", 2024-09-21T15:50:36.539 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.539 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:36.539 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048887+0000", 2024-09-21T15:50:36.539 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.540 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.540 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:36.540 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048887+0000", 2024-09-21T15:50:36.540 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:36.540 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.540 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.540 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:36.540 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048891+0000", 2024-09-21T15:50:36.540 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.843e-06 2024-09-21T15:50:36.540 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.540 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.540 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:36.540 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048897+0000", 2024-09-21T15:50:36.540 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5419999999999996e-06 2024-09-21T15:50:36.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.541 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.541 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:36.541 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048901+0000", 2024-09-21T15:50:36.541 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4689999999999999e-06 2024-09-21T15:50:36.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.541 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.541 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:36.541 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106780+0000", 2024-09-21T15:50:36.541 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057878259000000001 2024-09-21T15:50:36.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.541 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.541 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:36.541 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106827+0000", 2024-09-21T15:50:36.542 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7098000000000002e-05 2024-09-21T15:50:36.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.542 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:36.542 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106869+0000", 2024-09-21T15:50:36.542 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2129999999999998e-05 2024-09-21T15:50:36.542 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.542 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:36.542 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.542 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116329 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 555008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:36.543 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048905+0000", 2024-09-21T15:50:36.543 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 40.444776181000002, 2024-09-21T15:50:36.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058115554999999999, 2024-09-21T15:50:36.543 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:36.543 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:36.543 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:36.543 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:36.543 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:36.543 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116329 2024-09-21T15:50:36.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.543 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:36.543 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.543 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:36.543 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048905+0000", 2024-09-21T15:50:36.544 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.544 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.544 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.544 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:36.544 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048905+0000", 2024-09-21T15:50:36.544 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.544 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.544 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.544 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:36.544 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048905+0000", 2024-09-21T15:50:36.544 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:36.544 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.544 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.544 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:36.544 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048907+0000", 2024-09-21T15:50:36.545 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.086e-06 2024-09-21T15:50:36.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.545 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.545 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:36.545 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048912+0000", 2024-09-21T15:50:36.545 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2830000000000001e-06 2024-09-21T15:50:36.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.545 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.545 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:36.545 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048916+0000", 2024-09-21T15:50:36.545 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.98e-06 2024-09-21T15:50:36.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.545 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.545 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:36.545 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106907+0000", 2024-09-21T15:50:36.546 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057990719000000003 2024-09-21T15:50:36.546 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.546 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:36.546 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106969+0000", 2024-09-21T15:50:36.546 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1789000000000001e-05 2024-09-21T15:50:36.546 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.546 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:36.546 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107021+0000", 2024-09-21T15:50:36.546 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2030999999999997e-05 2024-09-21T15:50:36.546 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.546 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:36.547 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.547 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116330 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 561152~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:36.547 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048922+0000", 2024-09-21T15:50:36.547 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 40.444759023000003, 2024-09-21T15:50:36.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058242343000000002, 2024-09-21T15:50:36.547 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:36.547 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:36.547 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:36.547 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:36.547 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:36.547 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116330 2024-09-21T15:50:36.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.548 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:36.548 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.548 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:36.548 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048922+0000", 2024-09-21T15:50:36.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.548 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.548 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:36.548 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048922+0000", 2024-09-21T15:50:36.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.548 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.548 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:36.549 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048922+0000", 2024-09-21T15:50:36.549 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:36.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.549 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.549 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:36.549 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048924+0000", 2024-09-21T15:50:36.549 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1660000000000001e-06 2024-09-21T15:50:36.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.549 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.549 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:36.549 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048930+0000", 2024-09-21T15:50:36.549 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4529999999999999e-06 2024-09-21T15:50:36.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.549 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.549 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:36.549 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048934+0000", 2024-09-21T15:50:36.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0269999999999997e-06 2024-09-21T15:50:36.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.550 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.550 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:36.550 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107047+0000", 2024-09-21T15:50:36.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058112913000000002 2024-09-21T15:50:36.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.550 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.550 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:36.550 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107095+0000", 2024-09-21T15:50:36.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7778000000000002e-05 2024-09-21T15:50:36.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.550 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:36.551 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107165+0000", 2024-09-21T15:50:36.551 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9370999999999997e-05 2024-09-21T15:50:36.551 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.551 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:36.551 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.551 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116331 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 567296~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:36.551 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.049440+0000", 2024-09-21T15:50:36.551 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 40.444240929000003, 2024-09-21T15:50:36.551 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057878485, 2024-09-21T15:50:36.551 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:36.551 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:36.552 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:36.552 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:36.552 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:36.552 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116331 2024-09-21T15:50:36.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.552 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:36.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.552 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:36.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049440+0000", 2024-09-21T15:50:36.552 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.552 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:36.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049440+0000", 2024-09-21T15:50:36.553 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.553 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.553 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.553 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:36.553 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048939+0000", 2024-09-21T15:50:36.553 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9994984 2024-09-21T15:50:36.553 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.553 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.553 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:36.553 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049444+0000", 2024-09-21T15:50:36.553 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00050557200000000003 2024-09-21T15:50:36.553 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.553 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.553 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:36.553 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049460+0000", 2024-09-21T15:50:36.554 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6120000000000002e-05 2024-09-21T15:50:36.554 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.554 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.554 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:36.554 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049468+0000", 2024-09-21T15:50:36.554 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4209999999999998e-06 2024-09-21T15:50:36.554 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.554 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.554 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:36.554 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107189+0000", 2024-09-21T15:50:36.555 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057721019999999998 2024-09-21T15:50:36.555 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.555 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.555 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:36.555 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107254+0000", 2024-09-21T15:50:36.555 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5265000000000006e-05 2024-09-21T15:50:36.555 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.555 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.555 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:36.555 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107319+0000", 2024-09-21T15:50:36.555 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4784999999999997e-05 2024-09-21T15:50:36.555 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.555 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:36.555 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.556 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.556 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.556 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116332 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 573440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:36.556 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.049474+0000", 2024-09-21T15:50:36.556 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 40.444207667999997, 2024-09-21T15:50:36.556 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058013809, 2024-09-21T15:50:36.556 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:36.556 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:36.556 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:36.556 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:36.556 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:36.556 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116332 2024-09-21T15:50:36.556 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.556 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:36.556 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.556 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:36.557 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049474+0000", 2024-09-21T15:50:36.557 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.557 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.557 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:36.557 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049474+0000", 2024-09-21T15:50:36.557 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.557 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.557 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:36.557 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049473+0000", 2024-09-21T15:50:36.557 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:36.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.557 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.558 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:36.558 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049476+0000", 2024-09-21T15:50:36.558 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6249999999999999e-06 2024-09-21T15:50:36.558 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.558 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.558 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:36.558 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049482+0000", 2024-09-21T15:50:36.558 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0959999999999997e-06 2024-09-21T15:50:36.558 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.558 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.558 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:36.558 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049487+0000", 2024-09-21T15:50:36.559 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5949999999999998e-06 2024-09-21T15:50:36.559 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.559 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.559 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:36.559 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107344+0000", 2024-09-21T15:50:36.559 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057856594999999997 2024-09-21T15:50:36.559 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.559 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.559 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:36.559 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107422+0000", 2024-09-21T15:50:36.559 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7582000000000004e-05 2024-09-21T15:50:36.559 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.559 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.560 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:36.560 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107487+0000", 2024-09-21T15:50:36.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5723999999999996e-05 2024-09-21T15:50:36.560 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.560 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:36.560 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.560 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.560 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116333 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 579584~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:36.560 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.049993+0000", 2024-09-21T15:50:36.560 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 40.443687885000003, 2024-09-21T15:50:36.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057663537000000001, 2024-09-21T15:50:36.560 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:36.560 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:36.560 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:36.561 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:36.561 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:36.561 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116333 2024-09-21T15:50:36.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.561 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:36.561 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.561 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:36.561 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049993+0000", 2024-09-21T15:50:36.561 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.561 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.561 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:36.561 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049993+0000", 2024-09-21T15:50:36.561 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.562 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.562 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:36.562 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049491+0000", 2024-09-21T15:50:36.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9994979 2024-09-21T15:50:36.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.562 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.562 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:36.562 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049998+0000", 2024-09-21T15:50:36.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00050654200000000002 2024-09-21T15:50:36.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.562 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.562 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:36.563 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050010+0000", 2024-09-21T15:50:36.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1790999999999999e-05 2024-09-21T15:50:36.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.563 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:36.563 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050016+0000", 2024-09-21T15:50:36.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2179999999999996e-06 2024-09-21T15:50:36.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.563 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:36.563 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107513+0000", 2024-09-21T15:50:36.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057496927000000003 2024-09-21T15:50:36.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.563 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:36.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107590+0000", 2024-09-21T15:50:36.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7547999999999997e-05 2024-09-21T15:50:36.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:36.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107657+0000", 2024-09-21T15:50:36.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6630000000000004e-05 2024-09-21T15:50:36.564 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.564 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:36.564 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.564 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116334 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 585728~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:36.564 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.050091+0000", 2024-09-21T15:50:36.564 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 40.443590256999997, 2024-09-21T15:50:36.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057734355000000001, 2024-09-21T15:50:36.565 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:36.565 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:36.565 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:36.565 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:36.565 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:36.565 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116334 2024-09-21T15:50:36.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.565 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:36.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.565 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:36.565 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050091+0000", 2024-09-21T15:50:36.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.566 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:36.566 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050091+0000", 2024-09-21T15:50:36.566 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.566 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:36.566 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050021+0000", 2024-09-21T15:50:36.566 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999304 2024-09-21T15:50:36.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.566 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:36.567 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050095+0000", 2024-09-21T15:50:36.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4158000000000004e-05 2024-09-21T15:50:36.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.567 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.567 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:36.567 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050103+0000", 2024-09-21T15:50:36.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0770000000000002e-06 2024-09-21T15:50:36.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.567 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.567 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:36.567 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050108+0000", 2024-09-21T15:50:36.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6330000000000001e-06 2024-09-21T15:50:36.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.568 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:36.568 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107681+0000", 2024-09-21T15:50:36.568 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057572996000000001 2024-09-21T15:50:36.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.568 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:36.568 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107759+0000", 2024-09-21T15:50:36.568 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7368999999999997e-05 2024-09-21T15:50:36.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.568 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:36.568 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107825+0000", 2024-09-21T15:50:36.568 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6783000000000001e-05 2024-09-21T15:50:36.568 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.569 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:36.569 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.569 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116335 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 591872~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:36.569 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.050542+0000", 2024-09-21T15:50:36.569 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 40.443139672000001, 2024-09-21T15:50:36.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057466455999999999, 2024-09-21T15:50:36.569 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:36.569 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:36.569 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:36.569 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:36.569 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:36.570 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116335 2024-09-21T15:50:36.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.570 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:36.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:36.570 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050542+0000", 2024-09-21T15:50:36.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:36.570 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050542+0000", 2024-09-21T15:50:36.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:36.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050113+0000", 2024-09-21T15:50:36.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9995713 2024-09-21T15:50:36.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:36.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050546+0000", 2024-09-21T15:50:36.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00043328699999999997 2024-09-21T15:50:36.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:36.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050557+0000", 2024-09-21T15:50:36.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1011e-05 2024-09-21T15:50:36.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.572 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:36.572 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050564+0000", 2024-09-21T15:50:36.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1130000000000003e-06 2024-09-21T15:50:36.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.572 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:36.572 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107852+0000", 2024-09-21T15:50:36.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057287350000000001 2024-09-21T15:50:36.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.572 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:36.572 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107928+0000", 2024-09-21T15:50:36.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6668000000000005e-05 2024-09-21T15:50:36.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.573 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:36.573 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108008+0000", 2024-09-21T15:50:36.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9783999999999994e-05 2024-09-21T15:50:36.573 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.573 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:36.573 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.573 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116336 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 598016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:36.573 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.050834+0000", 2024-09-21T15:50:36.573 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 40.442847405999999, 2024-09-21T15:50:36.574 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057353572999999998, 2024-09-21T15:50:36.574 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:36.574 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:36.574 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:36.574 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:36.574 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:36.574 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116336 2024-09-21T15:50:36.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.574 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:36.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.574 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:36.574 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050834+0000", 2024-09-21T15:50:36.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:36.575 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050834+0000", 2024-09-21T15:50:36.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:36.575 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050571+0000", 2024-09-21T15:50:36.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9997373 2024-09-21T15:50:36.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:36.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050842+0000", 2024-09-21T15:50:36.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00027029099999999998 2024-09-21T15:50:36.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.576 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:36.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050853+0000", 2024-09-21T15:50:36.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1532000000000001e-05 2024-09-21T15:50:36.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.576 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:36.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050863+0000", 2024-09-21T15:50:36.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4239999999999999e-06 2024-09-21T15:50:36.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.576 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:36.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108045+0000", 2024-09-21T15:50:36.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057182415 2024-09-21T15:50:36.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:36.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108123+0000", 2024-09-21T15:50:36.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8275000000000005e-05 2024-09-21T15:50:36.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:36.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108187+0000", 2024-09-21T15:50:36.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4152000000000005e-05 2024-09-21T15:50:36.577 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.577 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:36.578 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.578 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116337 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 604160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:36.578 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.051192+0000", 2024-09-21T15:50:36.578 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 40.442488777999998, 2024-09-21T15:50:36.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057175981000000001, 2024-09-21T15:50:36.578 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:36.578 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:36.578 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:36.578 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:36.578 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:36.578 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116337 2024-09-21T15:50:36.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.579 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:36.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:36.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051192+0000", 2024-09-21T15:50:36.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:36.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051192+0000", 2024-09-21T15:50:36.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:36.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050870+0000", 2024-09-21T15:50:36.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9996777 2024-09-21T15:50:36.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:36.580 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051197+0000", 2024-09-21T15:50:36.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00032656299999999998 2024-09-21T15:50:36.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:36.580 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051209+0000", 2024-09-21T15:50:36.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2636000000000001e-05 2024-09-21T15:50:36.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:36.580 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051217+0000", 2024-09-21T15:50:36.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5989999999999996e-06 2024-09-21T15:50:36.581 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.581 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:36.581 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108224+0000", 2024-09-21T15:50:36.581 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057007622000000001 2024-09-21T15:50:36.581 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.581 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:36.581 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108299+0000", 2024-09-21T15:50:36.581 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5099999999999996e-05 2024-09-21T15:50:36.581 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.581 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:36.581 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108368+0000", 2024-09-21T15:50:36.581 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8918000000000007e-05 2024-09-21T15:50:36.582 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.582 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:36.582 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.582 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.582 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116338 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 610304~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:36.582 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.051583+0000", 2024-09-21T15:50:36.582 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 40.442098325000003, 2024-09-21T15:50:36.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056967716000000002, 2024-09-21T15:50:36.582 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:36.582 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:36.582 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:36.582 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:36.582 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:36.583 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116338 2024-09-21T15:50:36.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.583 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:36.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:36.583 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051583+0000", 2024-09-21T15:50:36.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:36.583 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051583+0000", 2024-09-21T15:50:36.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:36.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:36.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051223+0000", 2024-09-21T15:50:36.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.99964 2024-09-21T15:50:36.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:36.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051587+0000", 2024-09-21T15:50:36.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00036352999999999998 2024-09-21T15:50:36.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:36.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051597+0000", 2024-09-21T15:50:36.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.893e-06 2024-09-21T15:50:36.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:36.585 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051603+0000", 2024-09-21T15:50:36.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0519999999999998e-06 2024-09-21T15:50:36.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.585 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:36.585 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108394+0000", 2024-09-21T15:50:36.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056791531999999999 2024-09-21T15:50:36.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.585 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:36.585 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108480+0000", 2024-09-21T15:50:36.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.577e-05 2024-09-21T15:50:36.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:36.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:36.585 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:36.586 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108551+0000", 2024-09-21T15:50:36.586 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0722000000000005e-05 2024-09-21T15:50:36.586 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.586 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:36.586 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.586 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:36.586 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:36.586 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:36.587 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:50:36.588 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:36.587+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:36.688 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:36.687+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:36.688 INFO:teuthology.orchestra.run.smithi089.stderr:reweighted osd.7 to 1 (10000) 2024-09-21T15:50:36.689 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is unset 2024-09-21T15:50:36.707 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-21T15:50:36.707 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T15:50:36.709 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:36.709 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:50:36.709 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.709 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(client.4403.0:161124 3.1 e101/96 3:82ccd16f:::rbd_header.11335aef9442:head v 101'462, mlcod=101'460)", 2024-09-21T15:50:36.709 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:36.708808+0000", 2024-09-21T15:50:36.709 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.000196548, 2024-09-21T15:50:36.709 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000220433, 2024-09-21T15:50:36.709 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:36.710 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:36.710 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:36.710 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.710 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:36.710 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:36.708808+0000", 2024-09-21T15:50:36.710 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:36.710 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.710 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.710 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:36.710 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:36.708808+0000", 2024-09-21T15:50:36.710 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:36.710 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.710 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.710 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:36.710 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:36.708807+0000", 2024-09-21T15:50:36.711 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:36.711 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.711 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.711 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:36.711 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:36.708813+0000", 2024-09-21T15:50:36.711 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4130000000000003e-06 2024-09-21T15:50:36.711 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.711 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.711 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:36.711 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:36.708826+0000", 2024-09-21T15:50:36.711 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2162e-05 2024-09-21T15:50:36.711 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.711 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.711 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:36.711 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:36.708832+0000", 2024-09-21T15:50:36.712 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1709999999999999e-06 2024-09-21T15:50:36.712 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.712 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.712 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:36.712 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:36.708864+0000", 2024-09-21T15:50:36.712 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.2632999999999997e-05 2024-09-21T15:50:36.712 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.712 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.712 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:36.712 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:36.708883+0000", 2024-09-21T15:50:36.712 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8229000000000001e-05 2024-09-21T15:50:36.712 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:36.712 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:36.712 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:36.712 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.713 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.713 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_repop(osd.7.0:4464 3.11 e101/22 3:8b5183e5:::rbd_data.11335aef9442.0000000000000000:2a v 101'922, mlcod=99'919)", 2024-09-21T15:50:36.713 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:36.699934+0000", 2024-09-21T15:50:36.713 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 0.009070247, 2024-09-21T15:50:36.713 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0091320919999999996, 2024-09-21T15:50:36.713 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:36.713 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:36.713 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:36.713 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.713 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:36.713 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:36.699934+0000", 2024-09-21T15:50:36.713 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:36.713 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.713 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.713 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:36.713 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:36.699934+0000", 2024-09-21T15:50:36.714 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:36.714 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.714 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.714 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:36.714 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:36.699933+0000", 2024-09-21T15:50:36.714 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:36.714 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.714 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.714 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:36.714 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:36.699940+0000", 2024-09-21T15:50:36.714 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4949999999999996e-06 2024-09-21T15:50:36.714 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.715 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.715 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:36.715 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:36.699954+0000", 2024-09-21T15:50:36.715 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.4333e-05 2024-09-21T15:50:36.715 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.715 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.715 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:36.715 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:36.699961+0000", 2024-09-21T15:50:36.715 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6359999999999997e-06 2024-09-21T15:50:36.715 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.715 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.715 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:36.715 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:36.700057+0000", 2024-09-21T15:50:36.715 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5693999999999996e-05 2024-09-21T15:50:36.715 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.715 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.716 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:36.716 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:36.700088+0000", 2024-09-21T15:50:36.716 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1680999999999997e-05 2024-09-21T15:50:36.716 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:36.716 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:36.716 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:36.716 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:36.716 INFO:teuthology.orchestra.run.smithi100.stdout: ], 2024-09-21T15:50:36.716 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 2 2024-09-21T15:50:36.716 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:36.719 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:50:36.791 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:36.790+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:36.850 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:50:36.864 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:36.865 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:36.865 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:50:36.865 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:50:36.865 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:36.873 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:50:36.917 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:36.916+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:36.973 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-09-21T15:50:36.973 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:50:36.976 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:36.976 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:50:36.977 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:50:36.977 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:50:36.977 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.977 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155031 3.1c 3.8612741c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:36.977 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448748+0000", 2024-09-21T15:50:36.977 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 14.526192872999999, 2024-09-21T15:50:36.977 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064010286999999999, 2024-09-21T15:50:36.977 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:36.977 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:36.977 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:36.977 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:36.977 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:36.977 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155031 2024-09-21T15:50:36.977 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.977 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:36.978 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.978 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:36.978 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448748+0000", 2024-09-21T15:50:36.978 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:36.978 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.978 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.978 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:36.978 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448748+0000", 2024-09-21T15:50:36.978 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:36.978 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.978 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.978 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:36.978 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448747+0000", 2024-09-21T15:50:36.979 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:36.979 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.979 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.979 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:36.979 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448753+0000", 2024-09-21T15:50:36.979 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.755e-06 2024-09-21T15:50:36.979 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.979 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.979 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:36.979 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448763+0000", 2024-09-21T15:50:36.979 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7659999999999993e-06 2024-09-21T15:50:36.979 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.979 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.979 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:36.980 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448769+0000", 2024-09-21T15:50:36.980 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6060000000000002e-06 2024-09-21T15:50:36.980 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.980 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.980 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:36.980 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512750+0000", 2024-09-21T15:50:36.980 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.063981298000000006 2024-09-21T15:50:36.980 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.980 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.980 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:36.980 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512758+0000", 2024-09-21T15:50:36.980 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1669999999999999e-06 2024-09-21T15:50:36.980 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:36.980 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:36.981 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:36.981 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.981 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.981 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155091 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:36.981 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448916+0000", 2024-09-21T15:50:36.981 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 14.526025027999999, 2024-09-21T15:50:36.981 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064006309999999997, 2024-09-21T15:50:36.981 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:36.981 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:36.981 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:36.981 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:36.981 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:36.981 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155091 2024-09-21T15:50:36.981 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.982 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:36.982 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.982 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:36.982 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448916+0000", 2024-09-21T15:50:36.982 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:36.982 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.982 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.982 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:36.982 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448916+0000", 2024-09-21T15:50:36.982 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:36.982 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.982 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.982 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:36.982 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448915+0000", 2024-09-21T15:50:36.983 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:36.983 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.983 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.983 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:36.983 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448917+0000", 2024-09-21T15:50:36.983 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2400000000000002e-06 2024-09-21T15:50:36.983 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.983 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.983 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:36.983 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448923+0000", 2024-09-21T15:50:36.983 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0580000000000003e-06 2024-09-21T15:50:36.983 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.983 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.984 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:36.984 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448927+0000", 2024-09-21T15:50:36.984 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.031e-06 2024-09-21T15:50:36.984 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.984 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.984 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:36.984 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512916+0000", 2024-09-21T15:50:36.984 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.063988456999999999 2024-09-21T15:50:36.984 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.984 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.984 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:36.984 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512922+0000", 2024-09-21T15:50:36.984 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9009999999999999e-06 2024-09-21T15:50:36.984 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:36.985 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:36.985 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:36.985 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.985 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.985 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155092 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:36.985 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448931+0000", 2024-09-21T15:50:36.985 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 14.526009245999999, 2024-09-21T15:50:36.985 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064022793999999994, 2024-09-21T15:50:36.985 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:36.985 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:36.985 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:36.985 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:36.985 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:36.985 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155092 2024-09-21T15:50:36.985 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.986 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:36.986 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.986 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:36.986 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448931+0000", 2024-09-21T15:50:36.986 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:36.986 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.986 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.986 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:36.986 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448931+0000", 2024-09-21T15:50:36.986 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:36.986 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.986 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.986 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:36.986 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448931+0000", 2024-09-21T15:50:36.987 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:36.987 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.987 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.987 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:36.987 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448933+0000", 2024-09-21T15:50:36.987 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1799999999999999e-06 2024-09-21T15:50:36.987 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.987 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.987 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:36.987 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448939+0000", 2024-09-21T15:50:36.987 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.604e-06 2024-09-21T15:50:36.987 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.987 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.988 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:36.988 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448943+0000", 2024-09-21T15:50:36.988 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0740000000000003e-06 2024-09-21T15:50:36.988 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.988 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.988 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:36.988 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512946+0000", 2024-09-21T15:50:36.988 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064003612000000001 2024-09-21T15:50:36.988 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.988 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.988 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:36.988 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512954+0000", 2024-09-21T15:50:36.988 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6690000000000006e-06 2024-09-21T15:50:36.988 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:36.989 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:36.989 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:36.989 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.989 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.989 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155093 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:36.989 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448947+0000", 2024-09-21T15:50:36.989 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 14.525993872000001, 2024-09-21T15:50:36.989 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064020717000000005, 2024-09-21T15:50:36.989 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:36.989 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:36.989 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:36.989 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:36.989 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:36.989 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155093 2024-09-21T15:50:36.990 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.990 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:36.990 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.990 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:36.990 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448947+0000", 2024-09-21T15:50:36.990 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:36.990 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.990 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.990 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:36.990 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448947+0000", 2024-09-21T15:50:36.990 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:36.990 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.990 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.990 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:36.991 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448946+0000", 2024-09-21T15:50:36.991 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:36.991 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.991 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.991 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:36.991 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448949+0000", 2024-09-21T15:50:36.991 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3860000000000001e-06 2024-09-21T15:50:36.991 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.991 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.991 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:36.991 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448955+0000", 2024-09-21T15:50:36.991 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.823e-06 2024-09-21T15:50:36.991 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.991 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.991 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:36.992 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448959+0000", 2024-09-21T15:50:36.992 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2799999999999997e-06 2024-09-21T15:50:36.992 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.992 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.992 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:36.992 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512964+0000", 2024-09-21T15:50:36.992 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064004741000000004 2024-09-21T15:50:36.992 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.992 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.992 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:36.992 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512967+0000", 2024-09-21T15:50:36.992 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.839e-06 2024-09-21T15:50:36.992 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:36.992 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:36.993 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:36.993 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.993 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.993 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155094 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:36.993 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448962+0000", 2024-09-21T15:50:36.993 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 14.525978249, 2024-09-21T15:50:36.993 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064037378000000006, 2024-09-21T15:50:36.993 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:36.993 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:36.993 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:36.993 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:36.993 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:36.993 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155094 2024-09-21T15:50:36.993 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.993 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:36.993 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.994 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:36.994 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448962+0000", 2024-09-21T15:50:36.994 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:36.994 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.994 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.994 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:36.994 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448962+0000", 2024-09-21T15:50:36.994 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:36.994 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.994 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.994 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:36.994 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448962+0000", 2024-09-21T15:50:36.994 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:36.994 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.995 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.995 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:36.995 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448968+0000", 2024-09-21T15:50:36.995 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9710000000000001e-06 2024-09-21T15:50:36.995 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.995 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.995 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:36.995 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448974+0000", 2024-09-21T15:50:36.995 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7230000000000001e-06 2024-09-21T15:50:36.995 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.995 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.995 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:36.995 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448978+0000", 2024-09-21T15:50:36.995 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3749999999999996e-06 2024-09-21T15:50:36.995 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.996 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.996 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:36.996 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512992+0000", 2024-09-21T15:50:36.996 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064013525000000002 2024-09-21T15:50:36.996 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.996 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.996 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:36.996 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513000+0000", 2024-09-21T15:50:36.996 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0830000000000004e-06 2024-09-21T15:50:36.996 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:36.996 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:36.996 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:36.996 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.996 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.996 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155095 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:36.997 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448982+0000", 2024-09-21T15:50:36.997 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 14.525958966999999, 2024-09-21T15:50:36.997 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064041904999999996, 2024-09-21T15:50:36.997 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:36.997 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:36.997 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:36.997 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:36.997 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:36.997 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155095 2024-09-21T15:50:36.997 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.997 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:36.997 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.997 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:36.997 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448982+0000", 2024-09-21T15:50:36.997 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:36.998 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.998 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.998 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:36.998 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448982+0000", 2024-09-21T15:50:36.998 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:36.998 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.998 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.998 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:36.998 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448981+0000", 2024-09-21T15:50:36.998 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:36.998 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.998 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.998 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:36.999 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448983+0000", 2024-09-21T15:50:36.999 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.905e-06 2024-09-21T15:50:36.999 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.999 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.999 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:36.999 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448989+0000", 2024-09-21T15:50:36.999 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.536e-06 2024-09-21T15:50:36.999 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.999 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:36.999 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:36.999 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448993+0000", 2024-09-21T15:50:36.999 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2390000000000004e-06 2024-09-21T15:50:36.999 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:36.999 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.000 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:37.000 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513015+0000", 2024-09-21T15:50:37.000 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064022081999999994 2024-09-21T15:50:37.000 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.000 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.000 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:37.000 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513023+0000", 2024-09-21T15:50:37.000 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4500000000000004e-06 2024-09-21T15:50:37.000 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:37.000 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:37.000 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:37.000 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.000 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.000 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155096 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:37.000 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448996+0000", 2024-09-21T15:50:37.001 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 14.525944065999999, 2024-09-21T15:50:37.001 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064060868000000007, 2024-09-21T15:50:37.001 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:37.001 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:37.001 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:37.001 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:37.001 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:37.001 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155096 2024-09-21T15:50:37.001 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.001 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:37.001 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.001 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:37.001 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448996+0000", 2024-09-21T15:50:37.001 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:37.001 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.001 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.002 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:37.002 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448996+0000", 2024-09-21T15:50:37.002 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:37.002 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.002 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.002 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:37.002 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448996+0000", 2024-09-21T15:50:37.002 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:37.002 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.002 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.002 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:37.002 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448998+0000", 2024-09-21T15:50:37.002 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8780000000000001e-06 2024-09-21T15:50:37.003 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.003 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.003 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:37.003 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449003+0000", 2024-09-21T15:50:37.003 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4099999999999999e-06 2024-09-21T15:50:37.003 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.003 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.003 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:37.003 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449008+0000", 2024-09-21T15:50:37.003 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.335e-06 2024-09-21T15:50:37.003 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.003 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.003 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:37.003 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513049+0000", 2024-09-21T15:50:37.004 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064041059999999997 2024-09-21T15:50:37.004 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.004 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.004 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:37.004 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513057+0000", 2024-09-21T15:50:37.004 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5050000000000007e-06 2024-09-21T15:50:37.004 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:37.004 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:37.004 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:37.004 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.004 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.004 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155097 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:37.004 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449019+0000", 2024-09-21T15:50:37.004 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 14.525921645, 2024-09-21T15:50:37.005 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064052488000000005, 2024-09-21T15:50:37.005 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:37.005 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:37.005 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:37.005 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:37.005 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:37.005 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155097 2024-09-21T15:50:37.005 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.005 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:37.005 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.005 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:37.005 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449019+0000", 2024-09-21T15:50:37.005 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:37.006 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.006 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.006 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:37.006 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449019+0000", 2024-09-21T15:50:37.006 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:37.006 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.006 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.006 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:37.006 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449019+0000", 2024-09-21T15:50:37.006 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:37.006 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.006 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.006 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:37.006 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449021+0000", 2024-09-21T15:50:37.007 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.244e-06 2024-09-21T15:50:37.007 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.007 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.007 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:37.007 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449027+0000", 2024-09-21T15:50:37.007 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8619999999999999e-06 2024-09-21T15:50:37.007 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.007 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.007 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:37.007 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449031+0000", 2024-09-21T15:50:37.007 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1359999999999999e-06 2024-09-21T15:50:37.007 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.007 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.007 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:37.008 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513067+0000", 2024-09-21T15:50:37.008 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064036391999999998 2024-09-21T15:50:37.008 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.008 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.008 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:37.008 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513071+0000", 2024-09-21T15:50:37.008 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1989999999999999e-06 2024-09-21T15:50:37.008 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:37.008 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:37.008 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:37.008 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.008 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.008 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155098 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:37.008 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449035+0000", 2024-09-21T15:50:37.008 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 14.525906048, 2024-09-21T15:50:37.008 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064071761000000005, 2024-09-21T15:50:37.009 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:37.009 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:37.009 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:37.009 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:37.009 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:37.009 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155098 2024-09-21T15:50:37.009 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.009 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:37.009 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.009 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:37.009 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449035+0000", 2024-09-21T15:50:37.009 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:37.009 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.009 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.010 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:37.010 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449035+0000", 2024-09-21T15:50:37.010 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:37.010 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.010 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.010 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:37.010 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449034+0000", 2024-09-21T15:50:37.010 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:37.010 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.010 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.010 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:37.010 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449036+0000", 2024-09-21T15:50:37.010 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9989999999999998e-06 2024-09-21T15:50:37.010 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.010 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.011 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:37.011 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449042+0000", 2024-09-21T15:50:37.011 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6010000000000002e-06 2024-09-21T15:50:37.011 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.011 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.011 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:37.011 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449047+0000", 2024-09-21T15:50:37.011 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3600000000000004e-06 2024-09-21T15:50:37.011 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.011 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.011 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:37.011 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513098+0000", 2024-09-21T15:50:37.011 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064050544000000001 2024-09-21T15:50:37.011 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.012 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.012 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:37.012 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513106+0000", 2024-09-21T15:50:37.012 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5650000000000002e-06 2024-09-21T15:50:37.012 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:37.012 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:37.012 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:37.012 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.012 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.012 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155099 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:37.012 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449051+0000", 2024-09-21T15:50:37.012 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 14.525889507, 2024-09-21T15:50:37.012 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064069388000000005, 2024-09-21T15:50:37.012 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:37.012 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:37.013 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:37.013 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:37.013 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:37.013 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155099 2024-09-21T15:50:37.013 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.013 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:37.013 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.013 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:37.013 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449051+0000", 2024-09-21T15:50:37.013 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:37.013 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.013 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.013 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:37.013 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449051+0000", 2024-09-21T15:50:37.013 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:37.014 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.014 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.014 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:37.014 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449051+0000", 2024-09-21T15:50:37.014 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:37.014 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.014 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.014 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:37.014 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449053+0000", 2024-09-21T15:50:37.014 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2620000000000001e-06 2024-09-21T15:50:37.014 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.014 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.014 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:37.015 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449058+0000", 2024-09-21T15:50:37.015 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1680000000000003e-06 2024-09-21T15:50:37.015 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.015 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.015 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:37.015 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449062+0000", 2024-09-21T15:50:37.015 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9979999999999997e-06 2024-09-21T15:50:37.015 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.015 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.015 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:37.015 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513116+0000", 2024-09-21T15:50:37.015 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064053761000000001 2024-09-21T15:50:37.015 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.015 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.016 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:37.016 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513120+0000", 2024-09-21T15:50:37.016 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5399999999999997e-06 2024-09-21T15:50:37.016 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:37.016 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:37.016 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:37.016 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.016 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.016 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155100 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:37.016 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449101+0000", 2024-09-21T15:50:37.016 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 14.525839752, 2024-09-21T15:50:37.016 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064079585999999994, 2024-09-21T15:50:37.017 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:37.017 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:37.017 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:37.017 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:37.017 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:37.017 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155100 2024-09-21T15:50:37.017 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.017 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:37.017 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.017 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:37.017 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449101+0000", 2024-09-21T15:50:37.017 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:37.017 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.017 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.017 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:37.018 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449101+0000", 2024-09-21T15:50:37.018 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:37.018 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.018 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.018 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:37.018 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449100+0000", 2024-09-21T15:50:37.018 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-21T15:50:37.018 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.018 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.018 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:37.018 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449105+0000", 2024-09-21T15:50:37.018 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6700000000000002e-06 2024-09-21T15:50:37.018 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.019 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.019 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:37.019 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449117+0000", 2024-09-21T15:50:37.019 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1841000000000001e-05 2024-09-21T15:50:37.019 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.019 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.019 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:37.019 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449124+0000", 2024-09-21T15:50:37.019 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4660000000000002e-06 2024-09-21T15:50:37.019 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.019 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.019 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:37.020 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513163+0000", 2024-09-21T15:50:37.020 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064039053999999998 2024-09-21T15:50:37.020 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.020 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.020 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:37.020 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513180+0000", 2024-09-21T15:50:37.020 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7333000000000002e-05 2024-09-21T15:50:37.020 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:37.020 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:37.020 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:37.020 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.020 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.020 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155101 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:37.020 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449166+0000", 2024-09-21T15:50:37.021 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 14.525775047, 2024-09-21T15:50:37.021 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064027639999999997, 2024-09-21T15:50:37.021 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:37.021 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:37.021 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:37.021 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:37.021 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:37.021 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155101 2024-09-21T15:50:37.021 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.021 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:37.021 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.021 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:37.021 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449166+0000", 2024-09-21T15:50:37.021 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:37.021 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.022 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.022 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:37.022 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449166+0000", 2024-09-21T15:50:37.022 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:37.022 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.022 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.022 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:37.022 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449165+0000", 2024-09-21T15:50:37.022 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:37.022 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.022 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.022 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:37.022 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449168+0000", 2024-09-21T15:50:37.022 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5099999999999999e-06 2024-09-21T15:50:37.023 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.023 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.023 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:37.023 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449177+0000", 2024-09-21T15:50:37.023 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6659999999999997e-06 2024-09-21T15:50:37.023 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.023 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.023 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:37.023 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449182+0000", 2024-09-21T15:50:37.023 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1320000000000002e-06 2024-09-21T15:50:37.023 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.023 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.024 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:37.024 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513189+0000", 2024-09-21T15:50:37.024 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064007231999999997 2024-09-21T15:50:37.024 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.024 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.024 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:37.024 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513193+0000", 2024-09-21T15:50:37.024 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.692e-06 2024-09-21T15:50:37.024 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:37.024 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:37.024 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:37.024 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.024 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.024 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155134 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:37.024 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449187+0000", 2024-09-21T15:50:37.024 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 14.525753459000001, 2024-09-21T15:50:37.025 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064031331999999996, 2024-09-21T15:50:37.025 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:37.025 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:37.025 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:37.025 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:37.025 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:37.025 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155134 2024-09-21T15:50:37.025 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.025 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:37.025 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.025 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:37.025 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449187+0000", 2024-09-21T15:50:37.025 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:37.026 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.026 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.026 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:37.026 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449187+0000", 2024-09-21T15:50:37.026 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:37.026 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.026 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.026 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:37.026 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449187+0000", 2024-09-21T15:50:37.026 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:37.026 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.026 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.026 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:37.026 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449189+0000", 2024-09-21T15:50:37.026 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5469999999999999e-06 2024-09-21T15:50:37.027 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.027 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.027 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:37.027 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449196+0000", 2024-09-21T15:50:37.027 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7009999999999999e-06 2024-09-21T15:50:37.027 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.027 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.027 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:37.027 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449202+0000", 2024-09-21T15:50:37.027 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7409999999999998e-06 2024-09-21T15:50:37.027 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.027 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.027 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:37.027 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513214+0000", 2024-09-21T15:50:37.028 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064012301999999993 2024-09-21T15:50:37.028 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.028 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.028 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:37.028 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513218+0000", 2024-09-21T15:50:37.028 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5709999999999999e-06 2024-09-21T15:50:37.028 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:37.028 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:37.028 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:37.028 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.028 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.028 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155135 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:37.028 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449207+0000", 2024-09-21T15:50:37.028 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 14.525733948999999, 2024-09-21T15:50:37.028 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064029249999999996, 2024-09-21T15:50:37.028 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:37.029 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:37.029 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:37.029 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:37.029 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:37.029 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155135 2024-09-21T15:50:37.029 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.029 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:37.029 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.029 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:37.029 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449207+0000", 2024-09-21T15:50:37.029 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:37.029 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.029 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.029 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:37.029 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449207+0000", 2024-09-21T15:50:37.030 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:37.030 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.030 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.030 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:37.030 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449206+0000", 2024-09-21T15:50:37.030 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:37.030 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.030 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.030 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:37.030 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449209+0000", 2024-09-21T15:50:37.030 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5150000000000001e-06 2024-09-21T15:50:37.030 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.030 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.030 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:37.031 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449215+0000", 2024-09-21T15:50:37.031 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9329999999999998e-06 2024-09-21T15:50:37.031 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.031 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.031 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:37.031 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449220+0000", 2024-09-21T15:50:37.031 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0819999999999998e-06 2024-09-21T15:50:37.031 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.031 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.031 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:37.031 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513232+0000", 2024-09-21T15:50:37.031 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064012052 2024-09-21T15:50:37.031 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.031 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.031 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:37.032 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513236+0000", 2024-09-21T15:50:37.032 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2080000000000002e-06 2024-09-21T15:50:37.032 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:37.032 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:37.032 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:37.032 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.032 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.032 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155136 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:37.032 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449224+0000", 2024-09-21T15:50:37.032 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 14.525716384999999, 2024-09-21T15:50:37.032 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064053025999999999, 2024-09-21T15:50:37.032 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:37.032 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:37.032 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:37.032 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:37.033 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:37.033 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155136 2024-09-21T15:50:37.033 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.033 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:37.033 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.033 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:37.033 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449224+0000", 2024-09-21T15:50:37.033 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:37.033 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.033 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.033 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:37.033 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449224+0000", 2024-09-21T15:50:37.033 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:37.033 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.033 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.033 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:37.034 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449224+0000", 2024-09-21T15:50:37.034 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:37.034 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.034 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.034 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:37.034 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449226+0000", 2024-09-21T15:50:37.034 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2699999999999999e-06 2024-09-21T15:50:37.034 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.034 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.034 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:37.034 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449233+0000", 2024-09-21T15:50:37.034 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4289999999999998e-06 2024-09-21T15:50:37.034 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.035 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.035 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:37.035 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449238+0000", 2024-09-21T15:50:37.035 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0730000000000004e-06 2024-09-21T15:50:37.035 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.035 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.035 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:37.035 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513269+0000", 2024-09-21T15:50:37.035 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064030999000000005 2024-09-21T15:50:37.035 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.035 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.035 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:37.035 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513277+0000", 2024-09-21T15:50:37.035 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5520000000000005e-06 2024-09-21T15:50:37.035 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:37.036 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:37.036 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:37.036 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.036 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.036 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155137 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:37.036 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449263+0000", 2024-09-21T15:50:37.036 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 14.525677253, 2024-09-21T15:50:37.036 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.06402803, 2024-09-21T15:50:37.036 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:37.036 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:37.036 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:37.036 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:37.036 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:37.036 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155137 2024-09-21T15:50:37.037 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.037 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:37.037 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.037 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:37.037 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449263+0000", 2024-09-21T15:50:37.037 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:37.037 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.037 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.037 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:37.037 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449263+0000", 2024-09-21T15:50:37.037 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:37.037 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.037 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.037 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:37.038 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449262+0000", 2024-09-21T15:50:37.038 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-21T15:50:37.038 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.038 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.038 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:37.038 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449267+0000", 2024-09-21T15:50:37.038 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.933e-06 2024-09-21T15:50:37.038 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.038 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.038 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:37.038 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449279+0000", 2024-09-21T15:50:37.038 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.181e-05 2024-09-21T15:50:37.038 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.039 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.039 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:37.039 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449286+0000", 2024-09-21T15:50:37.039 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4749999999999998e-06 2024-09-21T15:50:37.039 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.039 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.039 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:37.039 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513287+0000", 2024-09-21T15:50:37.039 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064001369000000002 2024-09-21T15:50:37.039 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.039 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.040 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:37.040 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513291+0000", 2024-09-21T15:50:37.040 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3000000000000003e-06 2024-09-21T15:50:37.040 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:37.040 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:37.040 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:37.040 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.040 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.040 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155138 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:37.040 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449299+0000", 2024-09-21T15:50:37.040 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 14.525641760999999, 2024-09-21T15:50:37.040 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064028359000000007, 2024-09-21T15:50:37.040 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:37.040 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:37.040 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:37.041 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:37.041 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:37.041 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155138 2024-09-21T15:50:37.041 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.041 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:37.041 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.041 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:37.041 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449299+0000", 2024-09-21T15:50:37.041 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:37.041 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.041 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.041 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:37.041 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449299+0000", 2024-09-21T15:50:37.041 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:37.041 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.041 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.042 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:37.042 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449298+0000", 2024-09-21T15:50:37.042 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:37.042 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.042 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.042 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:37.042 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449301+0000", 2024-09-21T15:50:37.042 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0230000000000001e-06 2024-09-21T15:50:37.042 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.042 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.042 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:37.042 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449308+0000", 2024-09-21T15:50:37.042 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3119999999999999e-06 2024-09-21T15:50:37.042 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.043 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.043 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:37.043 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449313+0000", 2024-09-21T15:50:37.043 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8980000000000004e-06 2024-09-21T15:50:37.043 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.043 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.043 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:37.043 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513318+0000", 2024-09-21T15:50:37.043 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064005612000000003 2024-09-21T15:50:37.043 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.043 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.043 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:37.043 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513327+0000", 2024-09-21T15:50:37.043 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0100000000000001e-06 2024-09-21T15:50:37.043 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:37.044 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:37.044 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:37.044 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.044 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.044 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155142 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:37.044 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449392+0000", 2024-09-21T15:50:37.044 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 14.525549003, 2024-09-21T15:50:37.044 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064012464000000005, 2024-09-21T15:50:37.044 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:37.044 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:37.044 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:37.044 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:37.044 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:37.044 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155142 2024-09-21T15:50:37.044 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.044 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:37.045 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.045 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:37.045 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449392+0000", 2024-09-21T15:50:37.045 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:37.045 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.045 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.045 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:37.045 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449392+0000", 2024-09-21T15:50:37.045 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:37.045 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.045 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.045 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:37.045 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449391+0000", 2024-09-21T15:50:37.045 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:37.046 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.046 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.046 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:37.046 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449393+0000", 2024-09-21T15:50:37.046 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.26e-06 2024-09-21T15:50:37.046 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.046 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.046 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:37.046 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449399+0000", 2024-09-21T15:50:37.046 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6269999999999997e-06 2024-09-21T15:50:37.046 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.046 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.046 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:37.046 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449403+0000", 2024-09-21T15:50:37.047 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3000000000000003e-06 2024-09-21T15:50:37.047 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.047 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.047 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:37.047 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513396+0000", 2024-09-21T15:50:37.047 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.063992568999999999 2024-09-21T15:50:37.047 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.047 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.047 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:37.047 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513404+0000", 2024-09-21T15:50:37.047 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1100000000000003e-06 2024-09-21T15:50:37.047 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:37.047 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:37.047 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:37.047 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.048 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.048 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155143 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:37.048 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449407+0000", 2024-09-21T15:50:37.048 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 14.525533417, 2024-09-21T15:50:37.048 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064009916, 2024-09-21T15:50:37.048 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:37.048 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:37.048 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:37.048 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:37.048 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:37.048 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155143 2024-09-21T15:50:37.048 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.048 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:37.048 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.048 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:37.049 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449407+0000", 2024-09-21T15:50:37.049 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:37.049 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.049 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.049 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:37.049 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449407+0000", 2024-09-21T15:50:37.049 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:37.049 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.049 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.049 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:37.049 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449407+0000", 2024-09-21T15:50:37.049 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:37.049 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.049 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.049 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:37.050 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449409+0000", 2024-09-21T15:50:37.050 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1090000000000001e-06 2024-09-21T15:50:37.050 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.050 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.050 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:37.050 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449413+0000", 2024-09-21T15:50:37.050 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6650000000000001e-06 2024-09-21T15:50:37.050 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.050 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.050 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:37.050 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449415+0000", 2024-09-21T15:50:37.050 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5459999999999998e-06 2024-09-21T15:50:37.050 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.050 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.051 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:37.051 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513413+0000", 2024-09-21T15:50:37.051 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.063998037999999993 2024-09-21T15:50:37.051 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.051 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.051 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:37.051 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513417+0000", 2024-09-21T15:50:37.051 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8789999999999996e-06 2024-09-21T15:50:37.051 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:37.051 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:37.051 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:37.051 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.051 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.052 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155144 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:37.052 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449427+0000", 2024-09-21T15:50:37.052 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 14.525513409, 2024-09-21T15:50:37.052 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064022531999999993, 2024-09-21T15:50:37.052 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:37.052 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:37.052 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:37.052 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:37.052 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:37.052 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155144 2024-09-21T15:50:37.052 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.052 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:37.052 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.052 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:37.052 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449427+0000", 2024-09-21T15:50:37.052 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:37.053 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.053 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.053 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:37.053 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449427+0000", 2024-09-21T15:50:37.053 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:37.053 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.053 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.053 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:37.053 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449427+0000", 2024-09-21T15:50:37.053 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:37.053 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.053 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.053 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:37.053 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449430+0000", 2024-09-21T15:50:37.053 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3579999999999999e-06 2024-09-21T15:50:37.054 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.054 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.054 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:37.054 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449437+0000", 2024-09-21T15:50:37.054 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4580000000000004e-06 2024-09-21T15:50:37.054 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.054 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.054 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:37.054 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449443+0000", 2024-09-21T15:50:37.054 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5609999999999998e-06 2024-09-21T15:50:37.054 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.054 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.054 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:37.054 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513441+0000", 2024-09-21T15:50:37.054 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.063998136999999997 2024-09-21T15:50:37.055 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:37.055 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:37.055 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:37.055 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513450+0000", 2024-09-21T15:50:37.055 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4950000000000008e-06 2024-09-21T15:50:37.055 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:37.055 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:37.055 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:37.055 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:37.055 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:37.055 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:37.056 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:50:37.057 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:50:37.057 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:50:37.057 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:50:37.058 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:50:37.058 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:50:37.058 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:50:37.059 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:50:37.059 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:50:37.060 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:50:37.061 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-09-21T15:50:37.062 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":147,"stamp":"2024-09-21T15:50:36.682835+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1484996922,"num_objects":2240,"num_object_clones":292,"num_object_copies":4480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2240,"num_whiteouts":1256,"num_read":196955,"num_read_kb":833474,"num_write":21498,"num_write_kb":305140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":841,"num_bytes_recovered":47753679,"num_keys_recovered":165,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17552,"ondisk_log_size":17552,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":85,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":744772,"kb_used_data":369740,"kb_used_omap":24,"kb_used_meta":374887,"kb_avail":754229948,"statfs":{"total":773094113280,"available":772331466752,"internally_reserved":0,"allocated":378613760,"data_stored":512270573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25118,"internal_metadata":383884770},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-62160017,"num_objects":-89,"num_object_clones":-21,"num_object_copies":-178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-554,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-89,"num_whiteouts":-8,"num_read":1318,"num_read_kb":9321,"num_write":286,"num_write_kb":834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":691,"num_bytes_recovered":38465827,"num_keys_recovered":141,"num_objects_omap":-1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.018044"},"pg_stats":[{"pgid":"3.1f","version":"97'368","reported_seq":1751,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-21T15:50:32.682121+0000","last_change":"2024-09-21T15:50:32.682121+0000","last_active":"2024-09-21T15:50:32.682121+0000","last_peered":"2024-09-21T15:50:32.682121+0000","last_clean":"2024-09-21T15:50:32.682121+0000","last_became_active":"2024-09-21T15:50:32.670964+0000","last_became_peered":"2024-09-21T15:50:32.670964+0000","last_unstale":"2024-09-21T15:50:32.682121+0000","last_undegraded":"2024-09-21T15:50:32.682121+0000","last_fullsized":"2024-09-21T15:50:32.682121+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":368,"log_dups_size":0,"ondisk_log_size":368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:09:09.918864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0091502430000000006,"stat_sum":{"num_bytes":7925545,"num_objects":51,"num_object_clones":5,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":32,"num_read":2166,"num_read_kb":10210,"num_write":412,"num_write_kb":5950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":1974433,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"4"}]},{"pgid":"3.1e","version":"97'478","reported_seq":5380,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.670183+0000","last_change":"2024-09-21T15:50:34.670386+0000","last_active":"2024-09-21T15:50:34.670183+0000","last_peered":"2024-09-21T15:50:34.670183+0000","last_clean":"2024-09-21T15:50:34.670183+0000","last_became_active":"2024-09-21T15:50:20.087949+0000","last_became_peered":"2024-09-21T15:50:20.087949+0000","last_unstale":"2024-09-21T15:50:34.670183+0000","last_undegraded":"2024-09-21T15:50:34.670183+0000","last_fullsized":"2024-09-21T15:50:34.670183+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":478,"log_dups_size":0,"ondisk_log_size":478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:26:31.786747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036498599999999998,"stat_sum":{"num_bytes":53784778,"num_objects":82,"num_object_clones":10,"num_object_copies":164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":45,"num_read":6836,"num_read_kb":25379,"num_write":492,"num_write_kb":4338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"3.1d","version":"97'545","reported_seq":9776,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.673773+0000","last_change":"2024-09-21T15:50:34.679434+0000","last_active":"2024-09-21T15:50:34.673773+0000","last_peered":"2024-09-21T15:50:34.673773+0000","last_clean":"2024-09-21T15:50:34.673773+0000","last_became_active":"2024-09-21T15:49:39.833206+0000","last_became_peered":"2024-09-21T15:49:39.833206+0000","last_unstale":"2024-09-21T15:50:34.673773+0000","last_undegraded":"2024-09-21T15:50:34.673773+0000","last_fullsized":"2024-09-21T15:50:34.673773+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":545,"log_dups_size":0,"ondisk_log_size":545,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:36:14.082318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057857509999999996,"stat_sum":{"num_bytes":72397601,"num_objects":88,"num_object_clones":16,"num_object_copies":176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88,"num_whiteouts":46,"num_read":10437,"num_read_kb":28908,"num_write":627,"num_write_kb":8866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"3.1c","version":"99'471","reported_seq":3905,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.745138+0000","last_change":"2024-09-21T15:50:34.688353+0000","last_active":"2024-09-21T15:50:34.745138+0000","last_peered":"2024-09-21T15:50:34.745138+0000","last_clean":"2024-09-21T15:50:34.745138+0000","last_became_active":"2024-09-21T15:49:39.833113+0000","last_became_peered":"2024-09-21T15:49:39.833113+0000","last_unstale":"2024-09-21T15:50:34.745138+0000","last_undegraded":"2024-09-21T15:50:34.745138+0000","last_fullsized":"2024-09-21T15:50:34.745138+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":471,"log_dups_size":0,"ondisk_log_size":471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:43:57.592707+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.014658071999999999,"stat_sum":{"num_bytes":16748346,"num_objects":62,"num_object_clones":7,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":32,"num_read":4719,"num_read_kb":16862,"num_write":512,"num_write_kb":4777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"3.1b","version":"97'436","reported_seq":4008,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-21T15:50:32.756744+0000","last_change":"2024-09-21T15:50:32.689144+0000","last_active":"2024-09-21T15:50:32.756744+0000","last_peered":"2024-09-21T15:50:32.756744+0000","last_clean":"2024-09-21T15:50:32.756744+0000","last_became_active":"2024-09-21T15:50:32.674250+0000","last_became_peered":"2024-09-21T15:50:32.674250+0000","last_unstale":"2024-09-21T15:50:32.756744+0000","last_undegraded":"2024-09-21T15:50:32.756744+0000","last_fullsized":"2024-09-21T15:50:32.756744+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":436,"log_dups_size":0,"ondisk_log_size":436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:54:57.399818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01221852,"stat_sum":{"num_bytes":57746387,"num_objects":76,"num_object_clones":5,"num_object_copies":152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":42,"num_read":4612,"num_read_kb":29920,"num_write":444,"num_write_kb":13501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":4759000,"num_keys_recovered":12,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"4"}]},{"pgid":"3.1a","version":"86'348","reported_seq":6175,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.716068+0000","last_change":"2024-09-21T15:50:34.688560+0000","last_active":"2024-09-21T15:50:34.716068+0000","last_peered":"2024-09-21T15:50:34.716068+0000","last_clean":"2024-09-21T15:50:34.716068+0000","last_became_active":"2024-09-21T15:49:06.693903+0000","last_became_peered":"2024-09-21T15:49:06.693903+0000","last_unstale":"2024-09-21T15:50:34.716068+0000","last_undegraded":"2024-09-21T15:50:34.716068+0000","last_fullsized":"2024-09-21T15:50:34.716068+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:41:45.432893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014798159999999999,"stat_sum":{"num_bytes":52938660,"num_objects":68,"num_object_clones":9,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":36,"num_read":6525,"num_read_kb":31121,"num_write":367,"num_write_kb":9464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"3.19","version":"99'409","reported_seq":8031,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.689444+0000","last_change":"2024-09-21T15:50:34.689718+0000","last_active":"2024-09-21T15:50:34.689444+0000","last_peered":"2024-09-21T15:50:34.689444+0000","last_clean":"2024-09-21T15:50:34.689444+0000","last_became_active":"2024-09-21T15:49:06.691222+0000","last_became_peered":"2024-09-21T15:49:06.691222+0000","last_unstale":"2024-09-21T15:50:34.689444+0000","last_undegraded":"2024-09-21T15:50:34.689444+0000","last_fullsized":"2024-09-21T15:50:34.689444+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":409,"log_dups_size":0,"ondisk_log_size":409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:31:29.545866+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.015870318000000001,"stat_sum":{"num_bytes":65843420,"num_objects":71,"num_object_clones":5,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":36,"num_read":8232,"num_read_kb":39547,"num_write":475,"num_write_kb":7403,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"3.18","version":"97'407","reported_seq":3449,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-21T15:50:32.767353+0000","last_change":"2024-09-21T15:50:32.688342+0000","last_active":"2024-09-21T15:50:32.767353+0000","last_peered":"2024-09-21T15:50:32.767353+0000","last_clean":"2024-09-21T15:50:32.767353+0000","last_became_active":"2024-09-21T15:50:32.672791+0000","last_became_peered":"2024-09-21T15:50:32.672791+0000","last_unstale":"2024-09-21T15:50:32.767353+0000","last_undegraded":"2024-09-21T15:50:32.767353+0000","last_fullsized":"2024-09-21T15:50:32.767353+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":407,"log_dups_size":0,"ondisk_log_size":407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:07:19.540473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013942115999999999,"stat_sum":{"num_bytes":48735247,"num_objects":67,"num_object_clones":9,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":30,"num_read":5193,"num_read_kb":30076,"num_write":442,"num_write_kb":11051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":2076323,"num_keys_recovered":11,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"4"}]},{"pgid":"3.17","version":"97'917","reported_seq":8775,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-21T15:50:32.775257+0000","last_change":"2024-09-21T15:50:32.684436+0000","last_active":"2024-09-21T15:50:32.775257+0000","last_peered":"2024-09-21T15:50:32.775257+0000","last_clean":"2024-09-21T15:50:32.775257+0000","last_became_active":"2024-09-21T15:50:32.668771+0000","last_became_peered":"2024-09-21T15:50:32.668771+0000","last_unstale":"2024-09-21T15:50:32.775257+0000","last_undegraded":"2024-09-21T15:50:32.775257+0000","last_fullsized":"2024-09-21T15:50:32.775257+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":917,"log_dups_size":0,"ondisk_log_size":917,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:39:43.554150+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.01371901,"stat_sum":{"num_bytes":23526347,"num_objects":74,"num_object_clones":9,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":43,"num_read":11967,"num_read_kb":20847,"num_write":1212,"num_write_kb":8234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":3319185,"num_keys_recovered":34,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"4"}]},{"pgid":"3.16","version":"90'468","reported_seq":6313,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.669833+0000","last_change":"2024-09-21T15:50:34.670140+0000","last_active":"2024-09-21T15:50:34.669833+0000","last_peered":"2024-09-21T15:50:34.669833+0000","last_clean":"2024-09-21T15:50:34.669833+0000","last_became_active":"2024-09-21T15:49:06.688808+0000","last_became_peered":"2024-09-21T15:49:06.688808+0000","last_unstale":"2024-09-21T15:50:34.669833+0000","last_undegraded":"2024-09-21T15:50:34.669833+0000","last_fullsized":"2024-09-21T15:50:34.669833+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":468,"log_dups_size":0,"ondisk_log_size":468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:43:02.012254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045747499999999999,"stat_sum":{"num_bytes":96553494,"num_objects":69,"num_object_clones":9,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":38,"num_read":7324,"num_read_kb":49627,"num_write":491,"num_write_kb":13812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"3.15","version":"99'483","reported_seq":3566,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.694048+0000","last_change":"2024-09-21T15:50:34.694265+0000","last_active":"2024-09-21T15:50:34.694048+0000","last_peered":"2024-09-21T15:50:34.694048+0000","last_clean":"2024-09-21T15:50:34.694048+0000","last_became_active":"2024-09-21T15:49:06.691091+0000","last_became_peered":"2024-09-21T15:49:06.691091+0000","last_unstale":"2024-09-21T15:50:34.694048+0000","last_undegraded":"2024-09-21T15:50:34.694048+0000","last_fullsized":"2024-09-21T15:50:34.694048+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":483,"log_dups_size":0,"ondisk_log_size":483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:13:12.258161+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0056158889999999998,"stat_sum":{"num_bytes":32455197,"num_objects":64,"num_object_clones":11,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":37,"num_read":4033,"num_read_kb":27130,"num_write":613,"num_write_kb":17935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"3.14","version":"99'1134","reported_seq":4056,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.691334+0000","last_change":"2024-09-21T15:50:34.691584+0000","last_active":"2024-09-21T15:50:34.691334+0000","last_peered":"2024-09-21T15:50:34.691334+0000","last_clean":"2024-09-21T15:50:34.691334+0000","last_became_active":"2024-09-21T15:49:06.688803+0000","last_became_peered":"2024-09-21T15:49:06.688803+0000","last_unstale":"2024-09-21T15:50:34.691334+0000","last_undegraded":"2024-09-21T15:50:34.691334+0000","last_fullsized":"2024-09-21T15:50:34.691334+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":1134,"log_dups_size":0,"ondisk_log_size":1134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:18:47.662631+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.017866338999999998,"stat_sum":{"num_bytes":69136666,"num_objects":55,"num_object_clones":4,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":28,"num_read":7641,"num_read_kb":18141,"num_write":1856,"num_write_kb":6175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"2.7","version":"0'0","reported_seq":95,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-21T15:50:33.667873+0000","last_change":"2024-09-21T15:48:59.666431+0000","last_active":"2024-09-21T15:50:33.667873+0000","last_peered":"2024-09-21T15:50:33.667873+0000","last_clean":"2024-09-21T15:50:33.667873+0000","last_became_active":"2024-09-21T15:48:57.583665+0000","last_became_peered":"2024-09-21T15:48:57.583665+0000","last_unstale":"2024-09-21T15:50:33.667873+0000","last_undegraded":"2024-09-21T15:50:33.667873+0000","last_fullsized":"2024-09-21T15:50:33.667873+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:52:09.228964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031098399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"97'353","reported_seq":2734,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-21T15:50:32.769944+0000","last_change":"2024-09-21T15:50:32.684589+0000","last_active":"2024-09-21T15:50:32.769944+0000","last_peered":"2024-09-21T15:50:32.769944+0000","last_clean":"2024-09-21T15:50:32.769944+0000","last_became_active":"2024-09-21T15:50:32.671424+0000","last_became_peered":"2024-09-21T15:50:32.671424+0000","last_unstale":"2024-09-21T15:50:32.769944+0000","last_undegraded":"2024-09-21T15:50:32.769944+0000","last_fullsized":"2024-09-21T15:50:32.769944+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:04:14.554732+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.011297845000000001,"stat_sum":{"num_bytes":14549265,"num_objects":67,"num_object_clones":10,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":44,"num_read":2599,"num_read_kb":12929,"num_write":362,"num_write_kb":8387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":4494388,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"4"}]},{"pgid":"2.6","version":"0'0","reported_seq":95,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-21T15:50:33.667962+0000","last_change":"2024-09-21T15:48:59.589790+0000","last_active":"2024-09-21T15:50:33.667962+0000","last_peered":"2024-09-21T15:50:33.667962+0000","last_clean":"2024-09-21T15:50:33.667962+0000","last_became_active":"2024-09-21T15:48:57.582121+0000","last_became_peered":"2024-09-21T15:48:57.582121+0000","last_unstale":"2024-09-21T15:50:33.667962+0000","last_undegraded":"2024-09-21T15:50:33.667962+0000","last_fullsized":"2024-09-21T15:50:33.667962+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:33:47.592624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039654700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"97'516","reported_seq":7087,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.689003+0000","last_change":"2024-09-21T15:50:34.689275+0000","last_active":"2024-09-21T15:50:34.689003+0000","last_peered":"2024-09-21T15:50:34.689003+0000","last_clean":"2024-09-21T15:50:34.689003+0000","last_became_active":"2024-09-21T15:50:20.087935+0000","last_became_peered":"2024-09-21T15:50:20.087935+0000","last_unstale":"2024-09-21T15:50:34.689003+0000","last_undegraded":"2024-09-21T15:50:34.689003+0000","last_fullsized":"2024-09-21T15:50:34.689003+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":516,"log_dups_size":0,"ondisk_log_size":516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:27:45.090104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062972600000000005,"stat_sum":{"num_bytes":34633616,"num_objects":75,"num_object_clones":11,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75,"num_whiteouts":44,"num_read":8167,"num_read_kb":21012,"num_write":508,"num_write_kb":5391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"2.5","version":"0'0","reported_seq":108,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.149704+0000","last_change":"2024-09-21T15:50:30.146743+0000","last_active":"2024-09-21T15:50:34.149704+0000","last_peered":"2024-09-21T15:50:34.149704+0000","last_clean":"2024-09-21T15:50:34.149704+0000","last_became_active":"2024-09-21T15:50:30.146428+0000","last_became_peered":"2024-09-21T15:50:30.146428+0000","last_unstale":"2024-09-21T15:50:34.149704+0000","last_undegraded":"2024-09-21T15:50:34.149704+0000","last_fullsized":"2024-09-21T15:50:34.149704+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:58:38.689807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036074600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"98'592","reported_seq":8284,"reported_epoch":99,"state":"peering","last_fresh":"2024-09-21T15:50:34.674329+0000","last_change":"2024-09-21T15:50:34.674592+0000","last_active":"2024-09-21T15:50:34.674329+0000","last_peered":"2024-09-21T15:50:34.567209+0000","last_clean":"2024-09-21T15:50:31.216782+0000","last_became_active":"2024-09-21T15:50:34.144000+0000","last_became_peered":"2024-09-21T15:50:34.144000+0000","last_unstale":"2024-09-21T15:50:34.674329+0000","last_undegraded":"2024-09-21T15:50:34.674329+0000","last_fullsized":"2024-09-21T15:50:34.674329+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":592,"log_dups_size":0,"ondisk_log_size":592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:56:17.362656+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0066134189999999997,"stat_sum":{"num_bytes":36608631,"num_objects":68,"num_object_clones":13,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":34,"num_read":10413,"num_read_kb":27152,"num_write":790,"num_write_kb":10996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":175,"num_bytes_recovered":14134268,"num_keys_recovered":36,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,5],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2d","length":"3"}]},{"pgid":"2.4","version":"0'0","reported_seq":108,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-21T15:50:33.667911+0000","last_change":"2024-09-21T15:50:30.146654+0000","last_active":"2024-09-21T15:50:33.667911+0000","last_peered":"2024-09-21T15:50:33.667911+0000","last_clean":"2024-09-21T15:50:33.667911+0000","last_became_active":"2024-09-21T15:50:30.146273+0000","last_became_peered":"2024-09-21T15:50:30.146273+0000","last_unstale":"2024-09-21T15:50:33.667911+0000","last_undegraded":"2024-09-21T15:50:33.667911+0000","last_fullsized":"2024-09-21T15:50:33.667911+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:21:48.178519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040525700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"99'517","reported_seq":4959,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.682802+0000","last_change":"2024-09-21T15:50:34.683079+0000","last_active":"2024-09-21T15:50:34.682802+0000","last_peered":"2024-09-21T15:50:34.682802+0000","last_clean":"2024-09-21T15:50:34.682802+0000","last_became_active":"2024-09-21T15:49:39.835880+0000","last_became_peered":"2024-09-21T15:49:39.835880+0000","last_unstale":"2024-09-21T15:50:34.682802+0000","last_undegraded":"2024-09-21T15:50:34.682802+0000","last_fullsized":"2024-09-21T15:50:34.682802+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":517,"log_dups_size":0,"ondisk_log_size":517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:28:03.488776+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0062742270000000003,"stat_sum":{"num_bytes":75150008,"num_objects":84,"num_object_clones":13,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84,"num_whiteouts":43,"num_read":5511,"num_read_kb":33759,"num_write":629,"num_write_kb":16321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"3.3","version":"99'643","reported_seq":3291,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.683173+0000","last_change":"2024-09-21T15:50:34.683173+0000","last_active":"2024-09-21T15:50:34.683173+0000","last_peered":"2024-09-21T15:50:34.683173+0000","last_clean":"2024-09-21T15:50:34.683173+0000","last_became_active":"2024-09-21T15:50:32.677456+0000","last_became_peered":"2024-09-21T15:50:32.677456+0000","last_unstale":"2024-09-21T15:50:34.683173+0000","last_undegraded":"2024-09-21T15:50:34.683173+0000","last_fullsized":"2024-09-21T15:50:34.683173+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":643,"log_dups_size":0,"ondisk_log_size":643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:46:31.194345+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0095847789999999999,"stat_sum":{"num_bytes":23664738,"num_objects":79,"num_object_clones":14,"num_object_copies":158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79,"num_whiteouts":47,"num_read":4317,"num_read_kb":18605,"num_write":713,"num_write_kb":7264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":632176,"num_keys_recovered":24,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"2.2","version":"20'2","reported_seq":74,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-21T15:50:33.667901+0000","last_change":"2024-09-21T15:49:39.834640+0000","last_active":"2024-09-21T15:50:33.667901+0000","last_peered":"2024-09-21T15:50:33.667901+0000","last_clean":"2024-09-21T15:50:33.667901+0000","last_became_active":"2024-09-21T15:49:39.834207+0000","last_became_peered":"2024-09-21T15:49:39.834207+0000","last_unstale":"2024-09-21T15:50:33.667901+0000","last_undegraded":"2024-09-21T15:50:33.667901+0000","last_fullsized":"2024-09-21T15:50:33.667901+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:02:54.255716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"98'346","reported_seq":4438,"reported_epoch":99,"state":"peering","last_fresh":"2024-09-21T15:50:34.674453+0000","last_change":"2024-09-21T15:50:34.674621+0000","last_active":"2024-09-21T15:50:34.674453+0000","last_peered":"2024-09-21T15:50:34.339949+0000","last_clean":"2024-09-21T15:50:31.142696+0000","last_became_active":"2024-09-21T15:50:34.143212+0000","last_became_peered":"2024-09-21T15:50:34.143212+0000","last_unstale":"2024-09-21T15:50:34.674453+0000","last_undegraded":"2024-09-21T15:50:34.674453+0000","last_fullsized":"2024-09-21T15:50:34.674453+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":346,"log_dups_size":0,"ondisk_log_size":346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:49:25.944216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052199600000000005,"stat_sum":{"num_bytes":20675160,"num_objects":57,"num_object_clones":4,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":35,"num_read":4043,"num_read_kb":23169,"num_write":345,"num_write_kb":4139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":187,"num_bytes_recovered":4305276,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,6],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2d","length":"3"}]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.679630+0000","last_change":"2024-09-21T15:50:32.672133+0000","last_active":"2024-09-21T15:50:34.679630+0000","last_peered":"2024-09-21T15:50:34.679630+0000","last_clean":"2024-09-21T15:50:34.679630+0000","last_became_active":"2024-09-21T15:50:32.671865+0000","last_became_peered":"2024-09-21T15:50:32.671865+0000","last_unstale":"2024-09-21T15:50:34.679630+0000","last_undegraded":"2024-09-21T15:50:34.679630+0000","last_fullsized":"2024-09-21T15:50:34.679630+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:55:23.999144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"97'453","reported_seq":2000,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-21T15:50:32.689058+0000","last_change":"2024-09-21T15:50:32.689245+0000","last_active":"2024-09-21T15:50:32.689058+0000","last_peered":"2024-09-21T15:50:32.689058+0000","last_clean":"2024-09-21T15:50:32.689058+0000","last_became_active":"2024-09-21T15:50:32.671413+0000","last_became_peered":"2024-09-21T15:50:32.671413+0000","last_unstale":"2024-09-21T15:50:32.689058+0000","last_undegraded":"2024-09-21T15:50:32.689058+0000","last_fullsized":"2024-09-21T15:50:32.689058+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":453,"log_dups_size":0,"ondisk_log_size":453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:04:19.509402+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.01529492,"stat_sum":{"num_bytes":26803270,"num_objects":67,"num_object_clones":5,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":42,"num_read":3001,"num_read_kb":15971,"num_write":510,"num_write_kb":5955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":1696547,"num_keys_recovered":12,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"4"}]},{"pgid":"2.0","version":"0'0","reported_seq":95,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.149531+0000","last_change":"2024-09-21T15:49:00.114730+0000","last_active":"2024-09-21T15:50:34.149531+0000","last_peered":"2024-09-21T15:50:34.149531+0000","last_clean":"2024-09-21T15:50:34.149531+0000","last_became_active":"2024-09-21T15:48:57.585525+0000","last_became_peered":"2024-09-21T15:48:57.585525+0000","last_unstale":"2024-09-21T15:50:34.149531+0000","last_undegraded":"2024-09-21T15:50:34.149531+0000","last_fullsized":"2024-09-21T15:50:34.149531+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:35:35.786526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031691100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"99'365","reported_seq":2203,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.679026+0000","last_change":"2024-09-21T15:50:34.679434+0000","last_active":"2024-09-21T15:50:34.679026+0000","last_peered":"2024-09-21T15:50:34.679026+0000","last_clean":"2024-09-21T15:50:34.679026+0000","last_became_active":"2024-09-21T15:50:20.087681+0000","last_became_peered":"2024-09-21T15:50:20.087681+0000","last_unstale":"2024-09-21T15:50:34.679026+0000","last_undegraded":"2024-09-21T15:50:34.679026+0000","last_fullsized":"2024-09-21T15:50:34.679026+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":365,"log_dups_size":0,"ondisk_log_size":365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:07:05.059684+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0098461269999999997,"stat_sum":{"num_bytes":71749570,"num_objects":65,"num_object_clones":6,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":34,"num_read":2107,"num_read_kb":34423,"num_write":369,"num_write_kb":12608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"2.3","version":"18'1","reported_seq":67,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.135566+0000","last_change":"2024-09-21T15:49:39.834423+0000","last_active":"2024-09-21T15:50:34.135566+0000","last_peered":"2024-09-21T15:50:34.135566+0000","last_clean":"2024-09-21T15:50:34.135566+0000","last_became_active":"2024-09-21T15:49:39.833860+0000","last_became_peered":"2024-09-21T15:49:39.833860+0000","last_unstale":"2024-09-21T15:50:34.135566+0000","last_undegraded":"2024-09-21T15:50:34.135566+0000","last_fullsized":"2024-09-21T15:50:34.135566+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:11:43.630472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":256,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.149567+0000","last_change":"2024-09-21T15:50:30.147280+0000","last_active":"2024-09-21T15:50:34.149567+0000","last_peered":"2024-09-21T15:50:34.149567+0000","last_clean":"2024-09-21T15:50:34.149567+0000","last_became_active":"2024-09-21T15:50:30.146987+0000","last_became_peered":"2024-09-21T15:50:30.146987+0000","last_unstale":"2024-09-21T15:50:34.149567+0000","last_undegraded":"2024-09-21T15:50:34.149567+0000","last_fullsized":"2024-09-21T15:50:34.149567+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:50.161884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:50.161884+0000","last_clean_scrub_stamp":"2024-09-21T15:48:50.161884+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:26:32.853650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"97'451","reported_seq":6968,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.676782+0000","last_change":"2024-09-21T15:50:34.676960+0000","last_active":"2024-09-21T15:50:34.676782+0000","last_peered":"2024-09-21T15:50:34.676782+0000","last_clean":"2024-09-21T15:50:34.676782+0000","last_became_active":"2024-09-21T15:49:18.669143+0000","last_became_peered":"2024-09-21T15:49:18.669143+0000","last_unstale":"2024-09-21T15:50:34.676782+0000","last_undegraded":"2024-09-21T15:50:34.676782+0000","last_fullsized":"2024-09-21T15:50:34.676782+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":451,"log_dups_size":0,"ondisk_log_size":451,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:31:03.684897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048745400000000001,"stat_sum":{"num_bytes":42105276,"num_objects":76,"num_object_clones":8,"num_object_copies":152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":39,"num_read":6996,"num_read_kb":29381,"num_write":489,"num_write_kb":4779,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":296439,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"3.9","version":"95'343","reported_seq":2877,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.679521+0000","last_change":"2024-09-21T15:50:34.679797+0000","last_active":"2024-09-21T15:50:34.679521+0000","last_peered":"2024-09-21T15:50:34.679521+0000","last_clean":"2024-09-21T15:50:34.679521+0000","last_became_active":"2024-09-21T15:49:07.210506+0000","last_became_peered":"2024-09-21T15:49:07.210506+0000","last_unstale":"2024-09-21T15:50:34.679521+0000","last_undegraded":"2024-09-21T15:50:34.679521+0000","last_fullsized":"2024-09-21T15:50:34.679521+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":343,"log_dups_size":0,"ondisk_log_size":343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:55:18.046738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0061377159999999997,"stat_sum":{"num_bytes":43732741,"num_objects":65,"num_object_clones":4,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":34,"num_read":2674,"num_read_kb":28217,"num_write":414,"num_write_kb":12085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"3.a","version":"99'476","reported_seq":11781,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.682550+0000","last_change":"2024-09-21T15:50:34.682748+0000","last_active":"2024-09-21T15:50:34.682550+0000","last_peered":"2024-09-21T15:50:34.682550+0000","last_clean":"2024-09-21T15:50:34.682550+0000","last_became_active":"2024-09-21T15:49:06.694202+0000","last_became_peered":"2024-09-21T15:49:06.694202+0000","last_unstale":"2024-09-21T15:50:34.682550+0000","last_undegraded":"2024-09-21T15:50:34.682550+0000","last_fullsized":"2024-09-21T15:50:34.682550+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":476,"log_dups_size":0,"ondisk_log_size":476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:02:05.237705+0000","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0.012832926999999999,"stat_sum":{"num_bytes":70190032,"num_objects":67,"num_object_clones":10,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":34,"num_read":12417,"num_read_kb":63115,"num_write":489,"num_write_kb":14195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"3.b","version":"97'803","reported_seq":3095,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-21T15:50:32.775526+0000","last_change":"2024-09-21T15:50:32.161720+0000","last_active":"2024-09-21T15:50:32.775526+0000","last_peered":"2024-09-21T15:50:32.775526+0000","last_clean":"2024-09-21T15:50:32.775526+0000","last_became_active":"2024-09-21T15:50:30.149210+0000","last_became_peered":"2024-09-21T15:50:30.149210+0000","last_unstale":"2024-09-21T15:50:32.775526+0000","last_undegraded":"2024-09-21T15:50:32.775526+0000","last_fullsized":"2024-09-21T15:50:32.775526+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":803,"log_dups_size":0,"ondisk_log_size":803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:51:30.431071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003604112,"stat_sum":{"num_bytes":40581743,"num_objects":60,"num_object_clones":10,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":33,"num_read":5903,"num_read_kb":20139,"num_write":1214,"num_write_kb":7745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":5514164,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"4"}]},{"pgid":"3.c","version":"91'638","reported_seq":6249,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.676261+0000","last_change":"2024-09-21T15:50:34.676449+0000","last_active":"2024-09-21T15:50:34.676261+0000","last_peered":"2024-09-21T15:50:34.676261+0000","last_clean":"2024-09-21T15:50:34.676261+0000","last_became_active":"2024-09-21T15:49:39.835282+0000","last_became_peered":"2024-09-21T15:49:39.835282+0000","last_unstale":"2024-09-21T15:50:34.676261+0000","last_undegraded":"2024-09-21T15:50:34.676261+0000","last_fullsized":"2024-09-21T15:50:34.676261+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":638,"log_dups_size":0,"ondisk_log_size":638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:51:40.356906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049830499999999999,"stat_sum":{"num_bytes":25638002,"num_objects":72,"num_object_clones":7,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":42,"num_read":8360,"num_read_kb":25266,"num_write":787,"num_write_kb":4983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"3.d","version":"99'814","reported_seq":3865,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.697392+0000","last_change":"2024-09-21T15:50:34.697527+0000","last_active":"2024-09-21T15:50:34.697392+0000","last_peered":"2024-09-21T15:50:34.697392+0000","last_clean":"2024-09-21T15:50:34.697392+0000","last_became_active":"2024-09-21T15:49:06.689203+0000","last_became_peered":"2024-09-21T15:49:06.689203+0000","last_unstale":"2024-09-21T15:50:34.697392+0000","last_undegraded":"2024-09-21T15:50:34.697392+0000","last_fullsized":"2024-09-21T15:50:34.697392+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":814,"log_dups_size":0,"ondisk_log_size":814,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:22:17.578359+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.008723026,"stat_sum":{"num_bytes":78437702,"num_objects":80,"num_object_clones":12,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":44,"num_read":6904,"num_read_kb":27775,"num_write":1093,"num_write_kb":14798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"3.e","version":"97'469","reported_seq":7670,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.688800+0000","last_change":"2024-09-21T15:50:34.688968+0000","last_active":"2024-09-21T15:50:34.688800+0000","last_peered":"2024-09-21T15:50:34.688800+0000","last_clean":"2024-09-21T15:50:34.688800+0000","last_became_active":"2024-09-21T15:49:06.690101+0000","last_became_peered":"2024-09-21T15:49:06.690101+0000","last_unstale":"2024-09-21T15:50:34.688800+0000","last_undegraded":"2024-09-21T15:50:34.688800+0000","last_fullsized":"2024-09-21T15:50:34.688800+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":469,"log_dups_size":0,"ondisk_log_size":469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:43:30.659104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000516691,"stat_sum":{"num_bytes":116442302,"num_objects":82,"num_object_clones":19,"num_object_copies":164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":44,"num_read":8382,"num_read_kb":42678,"num_write":458,"num_write_kb":16261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"3.f","version":"99'605","reported_seq":3370,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.741468+0000","last_change":"2024-09-21T15:50:34.694457+0000","last_active":"2024-09-21T15:50:34.741468+0000","last_peered":"2024-09-21T15:50:34.741468+0000","last_clean":"2024-09-21T15:50:34.741468+0000","last_became_active":"2024-09-21T15:49:06.694250+0000","last_became_peered":"2024-09-21T15:49:06.694250+0000","last_unstale":"2024-09-21T15:50:34.741468+0000","last_undegraded":"2024-09-21T15:50:34.741468+0000","last_fullsized":"2024-09-21T15:50:34.741468+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":605,"log_dups_size":0,"ondisk_log_size":605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:21:37.777769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056822970000000002,"stat_sum":{"num_bytes":18103525,"num_objects":77,"num_object_clones":8,"num_object_copies":154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77,"num_whiteouts":50,"num_read":4619,"num_read_kb":15654,"num_write":810,"num_write_kb":5051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"3.10","version":"99'431","reported_seq":2797,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.679243+0000","last_change":"2024-09-21T15:50:34.679406+0000","last_active":"2024-09-21T15:50:34.679243+0000","last_peered":"2024-09-21T15:50:34.679243+0000","last_clean":"2024-09-21T15:50:34.679243+0000","last_became_active":"2024-09-21T15:50:32.676147+0000","last_became_peered":"2024-09-21T15:50:32.676147+0000","last_unstale":"2024-09-21T15:50:34.679243+0000","last_undegraded":"2024-09-21T15:50:34.679243+0000","last_fullsized":"2024-09-21T15:50:34.679243+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":431,"log_dups_size":0,"ondisk_log_size":431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:44:18.821310+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0093129760000000006,"stat_sum":{"num_bytes":13246714,"num_objects":57,"num_object_clones":4,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":44,"num_read":2952,"num_read_kb":11672,"num_write":356,"num_write_kb":4465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":1191778,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"3.11","version":"99'919","reported_seq":3784,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.696783+0000","last_change":"2024-09-21T15:50:34.697143+0000","last_active":"2024-09-21T15:50:34.696783+0000","last_peered":"2024-09-21T15:50:34.696783+0000","last_clean":"2024-09-21T15:50:34.696783+0000","last_became_active":"2024-09-21T15:49:06.688798+0000","last_became_peered":"2024-09-21T15:49:06.688798+0000","last_unstale":"2024-09-21T15:50:34.696783+0000","last_undegraded":"2024-09-21T15:50:34.696783+0000","last_fullsized":"2024-09-21T15:50:34.696783+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":919,"log_dups_size":0,"ondisk_log_size":919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:48:33.679133+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.008774479,"stat_sum":{"num_bytes":59874290,"num_objects":65,"num_object_clones":9,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":35,"num_read":6119,"num_read_kb":16744,"num_write":1280,"num_write_kb":8388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"3.12","version":"97'352","reported_seq":2735,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-21T15:50:32.688878+0000","last_change":"2024-09-21T15:50:32.688878+0000","last_active":"2024-09-21T15:50:32.688878+0000","last_peered":"2024-09-21T15:50:32.688878+0000","last_clean":"2024-09-21T15:50:32.688878+0000","last_became_active":"2024-09-21T15:50:32.671686+0000","last_became_peered":"2024-09-21T15:50:32.671686+0000","last_unstale":"2024-09-21T15:50:32.688878+0000","last_undegraded":"2024-09-21T15:50:32.688878+0000","last_fullsized":"2024-09-21T15:50:32.688878+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:40:53.968477+0000","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.015262016999999999,"stat_sum":{"num_bytes":39845746,"num_objects":64,"num_object_clones":8,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":41,"num_read":2441,"num_read_kb":11662,"num_write":386,"num_write_kb":13839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":3359702,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"4"}]},{"pgid":"3.13","version":"97'882","reported_seq":7104,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.688516+0000","last_change":"2024-09-21T15:50:34.688748+0000","last_active":"2024-09-21T15:50:34.688516+0000","last_peered":"2024-09-21T15:50:34.688516+0000","last_clean":"2024-09-21T15:50:34.688516+0000","last_became_active":"2024-09-21T15:49:06.690128+0000","last_became_peered":"2024-09-21T15:49:06.690128+0000","last_unstale":"2024-09-21T15:50:34.688516+0000","last_undegraded":"2024-09-21T15:50:34.688516+0000","last_fullsized":"2024-09-21T15:50:34.688516+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":882,"log_dups_size":0,"ondisk_log_size":882,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:11.763959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036025,"stat_sum":{"num_bytes":34582516,"num_objects":82,"num_object_clones":18,"num_object_copies":164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":48,"num_read":9229,"num_read_kb":26283,"num_write":1392,"num_write_kb":13566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2b","length":"5"}]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1484406535,"num_objects":2236,"num_object_clones":292,"num_object_copies":4472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2236,"num_whiteouts":1256,"num_read":196839,"num_read_kb":833374,"num_write":21337,"num_write_kb":302722,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":841,"num_bytes_recovered":47753679,"num_keys_recovered":165,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":377659392,"data_stored":506957447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5973,"internal_metadata":0},"log_size":17432,"ondisk_log_size":17432,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574872,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":127844,"kb_used_data":56532,"kb_used_omap":1,"kb_used_meta":71294,"kb_avail":94243996,"statfs":{"total":96636764160,"available":96505851904,"internally_reserved":0,"allocated":57888768,"data_stored":75307892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":73005515},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.34499999999999997}]},{"osd":1,"last update":"Sat Sep 21 15:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.37},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.38}]},{"osd":2,"last update":"Sat Sep 21 15:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.433}]},{"osd":3,"last update":"Sat Sep 21 15:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.36099999999999999}]},{"osd":4,"last update":"Sat Sep 21 15:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.317}]},{"osd":5,"last update":"Sat Sep 21 15:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.40600000000000003}]},{"osd":6,"last update":"Sat Sep 21 15:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.39700000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574875,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59612,"kb_used_data":24392,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94312228,"statfs":{"total":96636764160,"available":96575721472,"internally_reserved":0,"allocated":24977408,"data_stored":37618825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.42299999999999999}]},{"osd":1,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.38}]},{"osd":2,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.40400000000000003}]},{"osd":3,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.39200000000000002}]},{"osd":4,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.27100000000000002}]},{"osd":5,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.313}]},{"osd":7,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.37},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.41399999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574874,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":112016,"kb_used_data":62848,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94259824,"statfs":{"total":96636764160,"available":96522059776,"internally_reserved":0,"allocated":64356352,"data_stored":86513493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.48499999999999999}]},{"osd":1,"last update":"Sat Sep 21 15:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.52200000000000002}]},{"osd":2,"last update":"Sat Sep 21 15:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.50800000000000001}]},{"osd":3,"last update":"Sat Sep 21 15:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.375},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.6140000000000001,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.54000000000000004}]},{"osd":4,"last update":"Sat Sep 21 15:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":5.6399999999999997,"5min":5.6399999999999997,"15min":5.6399999999999997},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.27300000000000002}]},{"osd":6,"last update":"Sat Sep 21 15:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.55200000000000005}]},{"osd":7,"last update":"Sat Sep 21 15:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.39400000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574873,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":152052,"kb_used_data":80744,"kb_used_omap":1,"kb_used_meta":71294,"kb_avail":94219788,"statfs":{"total":96636764160,"available":96481062912,"internally_reserved":0,"allocated":82681856,"data_stored":113666860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73005510},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:49:52 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.32700000000000001}]},{"osd":1,"last update":"Sat Sep 21 15:49:52 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.217},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.38300000000000001}]},{"osd":2,"last update":"Sat Sep 21 15:49:52 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.437,"15min":0.437},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.31}]},{"osd":3,"last update":"Sat Sep 21 15:49:52 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.39600000000000002}]},{"osd":5,"last update":"Sat Sep 21 15:49:52 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":2.4489999999999998,"5min":2.4489999999999998,"15min":2.4489999999999998},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.44800000000000001}]},{"osd":6,"last update":"Sat Sep 21 15:49:52 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":2.7709999999999999,"5min":2.7709999999999999,"15min":2.7709999999999999},"last":0.34899999999999998}]},{"osd":7,"last update":"Sat Sep 21 15:49:52 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.8069999999999999,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.374}]}]},{"osd":3,"up_from":13,"seq":55834574874,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":109508,"kb_used_data":60344,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94262332,"statfs":{"total":96636764160,"available":96524627968,"internally_reserved":0,"allocated":61792256,"data_stored":85566797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50330056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.218},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.442,"5min":0.442,"15min":0.442},"last":0.20100000000000001}]},{"osd":1,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.28399999999999997}]},{"osd":2,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.25600000000000001}]},{"osd":4,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.371}]},{"osd":5,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.34200000000000003}]},{"osd":6,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.442}]},{"osd":7,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.38900000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574874,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38152,"kb_used_data":7036,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94333688,"statfs":{"total":96636764160,"available":96597696512,"internally_reserved":0,"allocated":7204864,"data_stored":6531679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.23200000000000001}]},{"osd":1,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":1.621,"5min":1.621,"15min":1.621},"last":0.19900000000000001},{"interface":"front","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.27100000000000002}]},{"osd":3,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":11.56,"5min":11.56,"15min":11.56},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.32300000000000001}]},{"osd":4,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.653,"5min":1.653,"15min":1.653},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":2.4660000000000002,"5min":2.4660000000000002,"15min":2.4660000000000002},"last":0.40699999999999997}]},{"osd":5,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.37},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.309}]},{"osd":6,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.436}]},{"osd":7,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.39500000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574873,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63648,"kb_used_data":28432,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94308192,"statfs":{"total":96636764160,"available":96571588608,"internally_reserved":0,"allocated":29114368,"data_stored":40956490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.315,"15min":0.315},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"last":0.31900000000000001}]},{"osd":2,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":5.0670000000000002,"5min":5.0670000000000002,"15min":5.0670000000000002},"last":0.28999999999999998}]},{"osd":3,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":16.032,"5min":16.032,"15min":16.032},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.309}]},{"osd":4,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2.2269999999999999,"5min":2.2269999999999999,"15min":2.2269999999999999},"last":0.46200000000000002}]},{"osd":5,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":2.2890000000000001,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":4.6989999999999998,"5min":4.6989999999999998,"15min":4.6989999999999998},"last":0.35899999999999999}]},{"osd":6,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":2.262,"5min":2.262,"15min":2.262},"last":0.41099999999999998}]},{"osd":7,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":4.742,"5min":4.742,"15min":4.742},"last":0.374},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.49099999999999999}]}]},{"osd":0,"up_from":93,"seq":399431958530,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81940,"kb_used_data":49412,"kb_used_omap":13,"kb_used_meta":32498,"kb_avail":94289900,"statfs":{"total":96636764160,"available":96552857600,"internally_reserved":0,"allocated":50597888,"data_stored":66108537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13968,"internal_metadata":33278320},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23300000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48852992,"data_stored":64979533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5973,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":27947008,"data_stored":38839043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":15609856,"data_stored":21056570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":60628992,"data_stored":82925081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":81522688,"data_stored":108334153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":63152128,"data_stored":84879777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":23814144,"data_stored":34452821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":56131584,"data_stored":71490469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T15:50:37.065 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T15:50:37.066 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:50:37.066 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:50:37.066 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:37.055+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:37.073 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:50:37.074 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:50:37.090 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:50:37.090 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:50:37.158 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:37.156+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:37.199 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:37.199 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:37.208 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:50:37.259 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:37.258+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:37.346 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:50:37.347 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-09-21T15:50:37.351 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:37.351 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:37.358 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":147,"stamp":"2024-09-21T15:50:36.682835+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1484996922,"num_objects":2240,"num_object_clones":292,"num_object_copies":4480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2240,"num_whiteouts":1256,"num_read":196955,"num_read_kb":833474,"num_write":21498,"num_write_kb":305140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":841,"num_bytes_recovered":47753679,"num_keys_recovered":165,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17552,"ondisk_log_size":17552,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":85,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":744772,"kb_used_data":369740,"kb_used_omap":24,"kb_used_meta":374887,"kb_avail":754229948,"statfs":{"total":773094113280,"available":772331466752,"internally_reserved":0,"allocated":378613760,"data_stored":512270573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25118,"internal_metadata":383884770},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-62160017,"num_objects":-89,"num_object_clones":-21,"num_object_copies":-178,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-554,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-89,"num_whiteouts":-8,"num_read":1318,"num_read_kb":9321,"num_write":286,"num_write_kb":834,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":691,"num_bytes_recovered":38465827,"num_keys_recovered":141,"num_objects_omap":-1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.018044"},"pg_stats":[{"pgid":"3.1f","version":"97'368","reported_seq":1751,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-21T15:50:32.682121+0000","last_change":"2024-09-21T15:50:32.682121+0000","last_active":"2024-09-21T15:50:32.682121+0000","last_peered":"2024-09-21T15:50:32.682121+0000","last_clean":"2024-09-21T15:50:32.682121+0000","last_became_active":"2024-09-21T15:50:32.670964+0000","last_became_peered":"2024-09-21T15:50:32.670964+0000","last_unstale":"2024-09-21T15:50:32.682121+0000","last_undegraded":"2024-09-21T15:50:32.682121+0000","last_fullsized":"2024-09-21T15:50:32.682121+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":368,"log_dups_size":0,"ondisk_log_size":368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:09:09.918864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0091502430000000006,"stat_sum":{"num_bytes":7925545,"num_objects":51,"num_object_clones":5,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":32,"num_read":2166,"num_read_kb":10210,"num_write":412,"num_write_kb":5950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":1974433,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"4"}]},{"pgid":"3.1e","version":"97'478","reported_seq":5380,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.670183+0000","last_change":"2024-09-21T15:50:34.670386+0000","last_active":"2024-09-21T15:50:34.670183+0000","last_peered":"2024-09-21T15:50:34.670183+0000","last_clean":"2024-09-21T15:50:34.670183+0000","last_became_active":"2024-09-21T15:50:20.087949+0000","last_became_peered":"2024-09-21T15:50:20.087949+0000","last_unstale":"2024-09-21T15:50:34.670183+0000","last_undegraded":"2024-09-21T15:50:34.670183+0000","last_fullsized":"2024-09-21T15:50:34.670183+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":478,"log_dups_size":0,"ondisk_log_size":478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:26:31.786747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036498599999999998,"stat_sum":{"num_bytes":53784778,"num_objects":82,"num_object_clones":10,"num_object_copies":164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":45,"num_read":6836,"num_read_kb":25379,"num_write":492,"num_write_kb":4338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"3.1d","version":"97'545","reported_seq":9776,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.673773+0000","last_change":"2024-09-21T15:50:34.679434+0000","last_active":"2024-09-21T15:50:34.673773+0000","last_peered":"2024-09-21T15:50:34.673773+0000","last_clean":"2024-09-21T15:50:34.673773+0000","last_became_active":"2024-09-21T15:49:39.833206+0000","last_became_peered":"2024-09-21T15:49:39.833206+0000","last_unstale":"2024-09-21T15:50:34.673773+0000","last_undegraded":"2024-09-21T15:50:34.673773+0000","last_fullsized":"2024-09-21T15:50:34.673773+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":545,"log_dups_size":0,"ondisk_log_size":545,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:36:14.082318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057857509999999996,"stat_sum":{"num_bytes":72397601,"num_objects":88,"num_object_clones":16,"num_object_copies":176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88,"num_whiteouts":46,"num_read":10437,"num_read_kb":28908,"num_write":627,"num_write_kb":8866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"3.1c","version":"99'471","reported_seq":3905,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.745138+0000","last_change":"2024-09-21T15:50:34.688353+0000","last_active":"2024-09-21T15:50:34.745138+0000","last_peered":"2024-09-21T15:50:34.745138+0000","last_clean":"2024-09-21T15:50:34.745138+0000","last_became_active":"2024-09-21T15:49:39.833113+0000","last_became_peered":"2024-09-21T15:49:39.833113+0000","last_unstale":"2024-09-21T15:50:34.745138+0000","last_undegraded":"2024-09-21T15:50:34.745138+0000","last_fullsized":"2024-09-21T15:50:34.745138+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":471,"log_dups_size":0,"ondisk_log_size":471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:43:57.592707+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.014658071999999999,"stat_sum":{"num_bytes":16748346,"num_objects":62,"num_object_clones":7,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":32,"num_read":4719,"num_read_kb":16862,"num_write":512,"num_write_kb":4777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"3.1b","version":"97'436","reported_seq":4008,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-21T15:50:32.756744+0000","last_change":"2024-09-21T15:50:32.689144+0000","last_active":"2024-09-21T15:50:32.756744+0000","last_peered":"2024-09-21T15:50:32.756744+0000","last_clean":"2024-09-21T15:50:32.756744+0000","last_became_active":"2024-09-21T15:50:32.674250+0000","last_became_peered":"2024-09-21T15:50:32.674250+0000","last_unstale":"2024-09-21T15:50:32.756744+0000","last_undegraded":"2024-09-21T15:50:32.756744+0000","last_fullsized":"2024-09-21T15:50:32.756744+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":436,"log_dups_size":0,"ondisk_log_size":436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:54:57.399818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01221852,"stat_sum":{"num_bytes":57746387,"num_objects":76,"num_object_clones":5,"num_object_copies":152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":42,"num_read":4612,"num_read_kb":29920,"num_write":444,"num_write_kb":13501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":4759000,"num_keys_recovered":12,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"4"}]},{"pgid":"3.1a","version":"86'348","reported_seq":6175,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.716068+0000","last_change":"2024-09-21T15:50:34.688560+0000","last_active":"2024-09-21T15:50:34.716068+0000","last_peered":"2024-09-21T15:50:34.716068+0000","last_clean":"2024-09-21T15:50:34.716068+0000","last_became_active":"2024-09-21T15:49:06.693903+0000","last_became_peered":"2024-09-21T15:49:06.693903+0000","last_unstale":"2024-09-21T15:50:34.716068+0000","last_undegraded":"2024-09-21T15:50:34.716068+0000","last_fullsized":"2024-09-21T15:50:34.716068+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:41:45.432893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014798159999999999,"stat_sum":{"num_bytes":52938660,"num_objects":68,"num_object_clones":9,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":36,"num_read":6525,"num_read_kb":31121,"num_write":367,"num_write_kb":9464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"3.19","version":"99'409","reported_seq":8031,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.689444+0000","last_change":"2024-09-21T15:50:34.689718+0000","last_active":"2024-09-21T15:50:34.689444+0000","last_peered":"2024-09-21T15:50:34.689444+0000","last_clean":"2024-09-21T15:50:34.689444+0000","last_became_active":"2024-09-21T15:49:06.691222+0000","last_became_peered":"2024-09-21T15:49:06.691222+0000","last_unstale":"2024-09-21T15:50:34.689444+0000","last_undegraded":"2024-09-21T15:50:34.689444+0000","last_fullsized":"2024-09-21T15:50:34.689444+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":409,"log_dups_size":0,"ondisk_log_size":409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:31:29.545866+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.015870318000000001,"stat_sum":{"num_bytes":65843420,"num_objects":71,"num_object_clones":5,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":36,"num_read":8232,"num_read_kb":39547,"num_write":475,"num_write_kb":7403,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"3.18","version":"97'407","reported_seq":3449,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-21T15:50:32.767353+0000","last_change":"2024-09-21T15:50:32.688342+0000","last_active":"2024-09-21T15:50:32.767353+0000","last_peered":"2024-09-21T15:50:32.767353+0000","last_clean":"2024-09-21T15:50:32.767353+0000","last_became_active":"2024-09-21T15:50:32.672791+0000","last_became_peered":"2024-09-21T15:50:32.672791+0000","last_unstale":"2024-09-21T15:50:32.767353+0000","last_undegraded":"2024-09-21T15:50:32.767353+0000","last_fullsized":"2024-09-21T15:50:32.767353+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":407,"log_dups_size":0,"ondisk_log_size":407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:07:19.540473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013942115999999999,"stat_sum":{"num_bytes":48735247,"num_objects":67,"num_object_clones":9,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":30,"num_read":5193,"num_read_kb":30076,"num_write":442,"num_write_kb":11051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":2076323,"num_keys_recovered":11,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"4"}]},{"pgid":"3.17","version":"97'917","reported_seq":8775,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-21T15:50:32.775257+0000","last_change":"2024-09-21T15:50:32.684436+0000","last_active":"2024-09-21T15:50:32.775257+0000","last_peered":"2024-09-21T15:50:32.775257+0000","last_clean":"2024-09-21T15:50:32.775257+0000","last_became_active":"2024-09-21T15:50:32.668771+0000","last_became_peered":"2024-09-21T15:50:32.668771+0000","last_unstale":"2024-09-21T15:50:32.775257+0000","last_undegraded":"2024-09-21T15:50:32.775257+0000","last_fullsized":"2024-09-21T15:50:32.775257+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":917,"log_dups_size":0,"ondisk_log_size":917,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:39:43.554150+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.01371901,"stat_sum":{"num_bytes":23526347,"num_objects":74,"num_object_clones":9,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":43,"num_read":11967,"num_read_kb":20847,"num_write":1212,"num_write_kb":8234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":3319185,"num_keys_recovered":34,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"4"}]},{"pgid":"3.16","version":"90'468","reported_seq":6313,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.669833+0000","last_change":"2024-09-21T15:50:34.670140+0000","last_active":"2024-09-21T15:50:34.669833+0000","last_peered":"2024-09-21T15:50:34.669833+0000","last_clean":"2024-09-21T15:50:34.669833+0000","last_became_active":"2024-09-21T15:49:06.688808+0000","last_became_peered":"2024-09-21T15:49:06.688808+0000","last_unstale":"2024-09-21T15:50:34.669833+0000","last_undegraded":"2024-09-21T15:50:34.669833+0000","last_fullsized":"2024-09-21T15:50:34.669833+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":468,"log_dups_size":0,"ondisk_log_size":468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:43:02.012254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045747499999999999,"stat_sum":{"num_bytes":96553494,"num_objects":69,"num_object_clones":9,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":38,"num_read":7324,"num_read_kb":49627,"num_write":491,"num_write_kb":13812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"3.15","version":"99'483","reported_seq":3566,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.694048+0000","last_change":"2024-09-21T15:50:34.694265+0000","last_active":"2024-09-21T15:50:34.694048+0000","last_peered":"2024-09-21T15:50:34.694048+0000","last_clean":"2024-09-21T15:50:34.694048+0000","last_became_active":"2024-09-21T15:49:06.691091+0000","last_became_peered":"2024-09-21T15:49:06.691091+0000","last_unstale":"2024-09-21T15:50:34.694048+0000","last_undegraded":"2024-09-21T15:50:34.694048+0000","last_fullsized":"2024-09-21T15:50:34.694048+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":483,"log_dups_size":0,"ondisk_log_size":483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:13:12.258161+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0056158889999999998,"stat_sum":{"num_bytes":32455197,"num_objects":64,"num_object_clones":11,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":37,"num_read":4033,"num_read_kb":27130,"num_write":613,"num_write_kb":17935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"3.14","version":"99'1134","reported_seq":4056,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.691334+0000","last_change":"2024-09-21T15:50:34.691584+0000","last_active":"2024-09-21T15:50:34.691334+0000","last_peered":"2024-09-21T15:50:34.691334+0000","last_clean":"2024-09-21T15:50:34.691334+0000","last_became_active":"2024-09-21T15:49:06.688803+0000","last_became_peered":"2024-09-21T15:49:06.688803+0000","last_unstale":"2024-09-21T15:50:34.691334+0000","last_undegraded":"2024-09-21T15:50:34.691334+0000","last_fullsized":"2024-09-21T15:50:34.691334+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":1134,"log_dups_size":0,"ondisk_log_size":1134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:18:47.662631+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.017866338999999998,"stat_sum":{"num_bytes":69136666,"num_objects":55,"num_object_clones":4,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":28,"num_read":7641,"num_read_kb":18141,"num_write":1856,"num_write_kb":6175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"2.7","version":"0'0","reported_seq":95,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-21T15:50:33.667873+0000","last_change":"2024-09-21T15:48:59.666431+0000","last_active":"2024-09-21T15:50:33.667873+0000","last_peered":"2024-09-21T15:50:33.667873+0000","last_clean":"2024-09-21T15:50:33.667873+0000","last_became_active":"2024-09-21T15:48:57.583665+0000","last_became_peered":"2024-09-21T15:48:57.583665+0000","last_unstale":"2024-09-21T15:50:33.667873+0000","last_undegraded":"2024-09-21T15:50:33.667873+0000","last_fullsized":"2024-09-21T15:50:33.667873+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:52:09.228964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031098399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"97'353","reported_seq":2734,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-21T15:50:32.769944+0000","last_change":"2024-09-21T15:50:32.684589+0000","last_active":"2024-09-21T15:50:32.769944+0000","last_peered":"2024-09-21T15:50:32.769944+0000","last_clean":"2024-09-21T15:50:32.769944+0000","last_became_active":"2024-09-21T15:50:32.671424+0000","last_became_peered":"2024-09-21T15:50:32.671424+0000","last_unstale":"2024-09-21T15:50:32.769944+0000","last_undegraded":"2024-09-21T15:50:32.769944+0000","last_fullsized":"2024-09-21T15:50:32.769944+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:04:14.554732+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.011297845000000001,"stat_sum":{"num_bytes":14549265,"num_objects":67,"num_object_clones":10,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":44,"num_read":2599,"num_read_kb":12929,"num_write":362,"num_write_kb":8387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":4494388,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"4"}]},{"pgid":"2.6","version":"0'0","reported_seq":95,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-21T15:50:33.667962+0000","last_change":"2024-09-21T15:48:59.589790+0000","last_active":"2024-09-21T15:50:33.667962+0000","last_peered":"2024-09-21T15:50:33.667962+0000","last_clean":"2024-09-21T15:50:33.667962+0000","last_became_active":"2024-09-21T15:48:57.582121+0000","last_became_peered":"2024-09-21T15:48:57.582121+0000","last_unstale":"2024-09-21T15:50:33.667962+0000","last_undegraded":"2024-09-21T15:50:33.667962+0000","last_fullsized":"2024-09-21T15:50:33.667962+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:33:47.592624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039654700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"97'516","reported_seq":7087,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.689003+0000","last_change":"2024-09-21T15:50:34.689275+0000","last_active":"2024-09-21T15:50:34.689003+0000","last_peered":"2024-09-21T15:50:34.689003+0000","last_clean":"2024-09-21T15:50:34.689003+0000","last_became_active":"2024-09-21T15:50:20.087935+0000","last_became_peered":"2024-09-21T15:50:20.087935+0000","last_unstale":"2024-09-21T15:50:34.689003+0000","last_undegraded":"2024-09-21T15:50:34.689003+0000","last_fullsized":"2024-09-21T15:50:34.689003+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":516,"log_dups_size":0,"ondisk_log_size":516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:27:45.090104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062972600000000005,"stat_sum":{"num_bytes":34633616,"num_objects":75,"num_object_clones":11,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75,"num_whiteouts":44,"num_read":8167,"num_read_kb":21012,"num_write":508,"num_write_kb":5391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"2.5","version":"0'0","reported_seq":108,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.149704+0000","last_change":"2024-09-21T15:50:30.146743+0000","last_active":"2024-09-21T15:50:34.149704+0000","last_peered":"2024-09-21T15:50:34.149704+0000","last_clean":"2024-09-21T15:50:34.149704+0000","last_became_active":"2024-09-21T15:50:30.146428+0000","last_became_peered":"2024-09-21T15:50:30.146428+0000","last_unstale":"2024-09-21T15:50:34.149704+0000","last_undegraded":"2024-09-21T15:50:34.149704+0000","last_fullsized":"2024-09-21T15:50:34.149704+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:58:38.689807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036074600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"98'592","reported_seq":8284,"reported_epoch":99,"state":"peering","last_fresh":"2024-09-21T15:50:34.674329+0000","last_change":"2024-09-21T15:50:34.674592+0000","last_active":"2024-09-21T15:50:34.674329+0000","last_peered":"2024-09-21T15:50:34.567209+0000","last_clean":"2024-09-21T15:50:31.216782+0000","last_became_active":"2024-09-21T15:50:34.144000+0000","last_became_peered":"2024-09-21T15:50:34.144000+0000","last_unstale":"2024-09-21T15:50:34.674329+0000","last_undegraded":"2024-09-21T15:50:34.674329+0000","last_fullsized":"2024-09-21T15:50:34.674329+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":592,"log_dups_size":0,"ondisk_log_size":592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:56:17.362656+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0066134189999999997,"stat_sum":{"num_bytes":36608631,"num_objects":68,"num_object_clones":13,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":34,"num_read":10413,"num_read_kb":27152,"num_write":790,"num_write_kb":10996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":175,"num_bytes_recovered":14134268,"num_keys_recovered":36,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,5],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2d","length":"3"}]},{"pgid":"2.4","version":"0'0","reported_seq":108,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-21T15:50:33.667911+0000","last_change":"2024-09-21T15:50:30.146654+0000","last_active":"2024-09-21T15:50:33.667911+0000","last_peered":"2024-09-21T15:50:33.667911+0000","last_clean":"2024-09-21T15:50:33.667911+0000","last_became_active":"2024-09-21T15:50:30.146273+0000","last_became_peered":"2024-09-21T15:50:30.146273+0000","last_unstale":"2024-09-21T15:50:33.667911+0000","last_undegraded":"2024-09-21T15:50:33.667911+0000","last_fullsized":"2024-09-21T15:50:33.667911+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:21:48.178519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040525700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"99'517","reported_seq":4959,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.682802+0000","last_change":"2024-09-21T15:50:34.683079+0000","last_active":"2024-09-21T15:50:34.682802+0000","last_peered":"2024-09-21T15:50:34.682802+0000","last_clean":"2024-09-21T15:50:34.682802+0000","last_became_active":"2024-09-21T15:49:39.835880+0000","last_became_peered":"2024-09-21T15:49:39.835880+0000","last_unstale":"2024-09-21T15:50:34.682802+0000","last_undegraded":"2024-09-21T15:50:34.682802+0000","last_fullsized":"2024-09-21T15:50:34.682802+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":517,"log_dups_size":0,"ondisk_log_size":517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:28:03.488776+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0062742270000000003,"stat_sum":{"num_bytes":75150008,"num_objects":84,"num_object_clones":13,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84,"num_whiteouts":43,"num_read":5511,"num_read_kb":33759,"num_write":629,"num_write_kb":16321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"3.3","version":"99'643","reported_seq":3291,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.683173+0000","last_change":"2024-09-21T15:50:34.683173+0000","last_active":"2024-09-21T15:50:34.683173+0000","last_peered":"2024-09-21T15:50:34.683173+0000","last_clean":"2024-09-21T15:50:34.683173+0000","last_became_active":"2024-09-21T15:50:32.677456+0000","last_became_peered":"2024-09-21T15:50:32.677456+0000","last_unstale":"2024-09-21T15:50:34.683173+0000","last_undegraded":"2024-09-21T15:50:34.683173+0000","last_fullsized":"2024-09-21T15:50:34.683173+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":643,"log_dups_size":0,"ondisk_log_size":643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:46:31.194345+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0095847789999999999,"stat_sum":{"num_bytes":23664738,"num_objects":79,"num_object_clones":14,"num_object_copies":158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79,"num_whiteouts":47,"num_read":4317,"num_read_kb":18605,"num_write":713,"num_write_kb":7264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":632176,"num_keys_recovered":24,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"2.2","version":"20'2","reported_seq":74,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-21T15:50:33.667901+0000","last_change":"2024-09-21T15:49:39.834640+0000","last_active":"2024-09-21T15:50:33.667901+0000","last_peered":"2024-09-21T15:50:33.667901+0000","last_clean":"2024-09-21T15:50:33.667901+0000","last_became_active":"2024-09-21T15:49:39.834207+0000","last_became_peered":"2024-09-21T15:49:39.834207+0000","last_unstale":"2024-09-21T15:50:33.667901+0000","last_undegraded":"2024-09-21T15:50:33.667901+0000","last_fullsized":"2024-09-21T15:50:33.667901+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:02:54.255716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"98'346","reported_seq":4438,"reported_epoch":99,"state":"peering","last_fresh":"2024-09-21T15:50:34.674453+0000","last_change":"2024-09-21T15:50:34.674621+0000","last_active":"2024-09-21T15:50:34.674453+0000","last_peered":"2024-09-21T15:50:34.339949+0000","last_clean":"2024-09-21T15:50:31.142696+0000","last_became_active":"2024-09-21T15:50:34.143212+0000","last_became_peered":"2024-09-21T15:50:34.143212+0000","last_unstale":"2024-09-21T15:50:34.674453+0000","last_undegraded":"2024-09-21T15:50:34.674453+0000","last_fullsized":"2024-09-21T15:50:34.674453+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":346,"log_dups_size":0,"ondisk_log_size":346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:49:25.944216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052199600000000005,"stat_sum":{"num_bytes":20675160,"num_objects":57,"num_object_clones":4,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":35,"num_read":4043,"num_read_kb":23169,"num_write":345,"num_write_kb":4139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":187,"num_bytes_recovered":4305276,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,6],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2d","length":"3"}]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.679630+0000","last_change":"2024-09-21T15:50:32.672133+0000","last_active":"2024-09-21T15:50:34.679630+0000","last_peered":"2024-09-21T15:50:34.679630+0000","last_clean":"2024-09-21T15:50:34.679630+0000","last_became_active":"2024-09-21T15:50:32.671865+0000","last_became_peered":"2024-09-21T15:50:32.671865+0000","last_unstale":"2024-09-21T15:50:34.679630+0000","last_undegraded":"2024-09-21T15:50:34.679630+0000","last_fullsized":"2024-09-21T15:50:34.679630+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:55:23.999144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"97'453","reported_seq":2000,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-21T15:50:32.689058+0000","last_change":"2024-09-21T15:50:32.689245+0000","last_active":"2024-09-21T15:50:32.689058+0000","last_peered":"2024-09-21T15:50:32.689058+0000","last_clean":"2024-09-21T15:50:32.689058+0000","last_became_active":"2024-09-21T15:50:32.671413+0000","last_became_peered":"2024-09-21T15:50:32.671413+0000","last_unstale":"2024-09-21T15:50:32.689058+0000","last_undegraded":"2024-09-21T15:50:32.689058+0000","last_fullsized":"2024-09-21T15:50:32.689058+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":453,"log_dups_size":0,"ondisk_log_size":453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:04:19.509402+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.01529492,"stat_sum":{"num_bytes":26803270,"num_objects":67,"num_object_clones":5,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":42,"num_read":3001,"num_read_kb":15971,"num_write":510,"num_write_kb":5955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":1696547,"num_keys_recovered":12,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"4"}]},{"pgid":"2.0","version":"0'0","reported_seq":95,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.149531+0000","last_change":"2024-09-21T15:49:00.114730+0000","last_active":"2024-09-21T15:50:34.149531+0000","last_peered":"2024-09-21T15:50:34.149531+0000","last_clean":"2024-09-21T15:50:34.149531+0000","last_became_active":"2024-09-21T15:48:57.585525+0000","last_became_peered":"2024-09-21T15:48:57.585525+0000","last_unstale":"2024-09-21T15:50:34.149531+0000","last_undegraded":"2024-09-21T15:50:34.149531+0000","last_fullsized":"2024-09-21T15:50:34.149531+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:35:35.786526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031691100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"99'365","reported_seq":2203,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.679026+0000","last_change":"2024-09-21T15:50:34.679434+0000","last_active":"2024-09-21T15:50:34.679026+0000","last_peered":"2024-09-21T15:50:34.679026+0000","last_clean":"2024-09-21T15:50:34.679026+0000","last_became_active":"2024-09-21T15:50:20.087681+0000","last_became_peered":"2024-09-21T15:50:20.087681+0000","last_unstale":"2024-09-21T15:50:34.679026+0000","last_undegraded":"2024-09-21T15:50:34.679026+0000","last_fullsized":"2024-09-21T15:50:34.679026+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":365,"log_dups_size":0,"ondisk_log_size":365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:07:05.059684+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0098461269999999997,"stat_sum":{"num_bytes":71749570,"num_objects":65,"num_object_clones":6,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":34,"num_read":2107,"num_read_kb":34423,"num_write":369,"num_write_kb":12608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"2.3","version":"18'1","reported_seq":67,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.135566+0000","last_change":"2024-09-21T15:49:39.834423+0000","last_active":"2024-09-21T15:50:34.135566+0000","last_peered":"2024-09-21T15:50:34.135566+0000","last_clean":"2024-09-21T15:50:34.135566+0000","last_became_active":"2024-09-21T15:49:39.833860+0000","last_became_peered":"2024-09-21T15:49:39.833860+0000","last_unstale":"2024-09-21T15:50:34.135566+0000","last_undegraded":"2024-09-21T15:50:34.135566+0000","last_fullsized":"2024-09-21T15:50:34.135566+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:11:43.630472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":256,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.149567+0000","last_change":"2024-09-21T15:50:30.147280+0000","last_active":"2024-09-21T15:50:34.149567+0000","last_peered":"2024-09-21T15:50:34.149567+0000","last_clean":"2024-09-21T15:50:34.149567+0000","last_became_active":"2024-09-21T15:50:30.146987+0000","last_became_peered":"2024-09-21T15:50:30.146987+0000","last_unstale":"2024-09-21T15:50:34.149567+0000","last_undegraded":"2024-09-21T15:50:34.149567+0000","last_fullsized":"2024-09-21T15:50:34.149567+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:50.161884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:50.161884+0000","last_clean_scrub_stamp":"2024-09-21T15:48:50.161884+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:26:32.853650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"97'451","reported_seq":6968,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.676782+0000","last_change":"2024-09-21T15:50:34.676960+0000","last_active":"2024-09-21T15:50:34.676782+0000","last_peered":"2024-09-21T15:50:34.676782+0000","last_clean":"2024-09-21T15:50:34.676782+0000","last_became_active":"2024-09-21T15:49:18.669143+0000","last_became_peered":"2024-09-21T15:49:18.669143+0000","last_unstale":"2024-09-21T15:50:34.676782+0000","last_undegraded":"2024-09-21T15:50:34.676782+0000","last_fullsized":"2024-09-21T15:50:34.676782+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":451,"log_dups_size":0,"ondisk_log_size":451,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:31:03.684897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048745400000000001,"stat_sum":{"num_bytes":42105276,"num_objects":76,"num_object_clones":8,"num_object_copies":152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":39,"num_read":6996,"num_read_kb":29381,"num_write":489,"num_write_kb":4779,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":296439,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"3.9","version":"95'343","reported_seq":2877,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.679521+0000","last_change":"2024-09-21T15:50:34.679797+0000","last_active":"2024-09-21T15:50:34.679521+0000","last_peered":"2024-09-21T15:50:34.679521+0000","last_clean":"2024-09-21T15:50:34.679521+0000","last_became_active":"2024-09-21T15:49:07.210506+0000","last_became_peered":"2024-09-21T15:49:07.210506+0000","last_unstale":"2024-09-21T15:50:34.679521+0000","last_undegraded":"2024-09-21T15:50:34.679521+0000","last_fullsized":"2024-09-21T15:50:34.679521+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":343,"log_dups_size":0,"ondisk_log_size":343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:55:18.046738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0061377159999999997,"stat_sum":{"num_bytes":43732741,"num_objects":65,"num_object_clones":4,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":34,"num_read":2674,"num_read_kb":28217,"num_write":414,"num_write_kb":12085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"3.a","version":"99'476","reported_seq":11781,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.682550+0000","last_change":"2024-09-21T15:50:34.682748+0000","last_active":"2024-09-21T15:50:34.682550+0000","last_peered":"2024-09-21T15:50:34.682550+0000","last_clean":"2024-09-21T15:50:34.682550+0000","last_became_active":"2024-09-21T15:49:06.694202+0000","last_became_peered":"2024-09-21T15:49:06.694202+0000","last_unstale":"2024-09-21T15:50:34.682550+0000","last_undegraded":"2024-09-21T15:50:34.682550+0000","last_fullsized":"2024-09-21T15:50:34.682550+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":476,"log_dups_size":0,"ondisk_log_size":476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:02:05.237705+0000","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0.012832926999999999,"stat_sum":{"num_bytes":70190032,"num_objects":67,"num_object_clones":10,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":34,"num_read":12417,"num_read_kb":63115,"num_write":489,"num_write_kb":14195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"3.b","version":"97'803","reported_seq":3095,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-21T15:50:32.775526+0000","last_change":"2024-09-21T15:50:32.161720+0000","last_active":"2024-09-21T15:50:32.775526+0000","last_peered":"2024-09-21T15:50:32.775526+0000","last_clean":"2024-09-21T15:50:32.775526+0000","last_became_active":"2024-09-21T15:50:30.149210+0000","last_became_peered":"2024-09-21T15:50:30.149210+0000","last_unstale":"2024-09-21T15:50:32.775526+0000","last_undegraded":"2024-09-21T15:50:32.775526+0000","last_fullsized":"2024-09-21T15:50:32.775526+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":803,"log_dups_size":0,"ondisk_log_size":803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:51:30.431071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003604112,"stat_sum":{"num_bytes":40581743,"num_objects":60,"num_object_clones":10,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":33,"num_read":5903,"num_read_kb":20139,"num_write":1214,"num_write_kb":7745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":5514164,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"4"}]},{"pgid":"3.c","version":"91'638","reported_seq":6249,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.676261+0000","last_change":"2024-09-21T15:50:34.676449+0000","last_active":"2024-09-21T15:50:34.676261+0000","last_peered":"2024-09-21T15:50:34.676261+0000","last_clean":"2024-09-21T15:50:34.676261+0000","last_became_active":"2024-09-21T15:49:39.835282+0000","last_became_peered":"2024-09-21T15:49:39.835282+0000","last_unstale":"2024-09-21T15:50:34.676261+0000","last_undegraded":"2024-09-21T15:50:34.676261+0000","last_fullsized":"2024-09-21T15:50:34.676261+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":638,"log_dups_size":0,"ondisk_log_size":638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:51:40.356906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049830499999999999,"stat_sum":{"num_bytes":25638002,"num_objects":72,"num_object_clones":7,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":42,"num_read":8360,"num_read_kb":25266,"num_write":787,"num_write_kb":4983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"3.d","version":"99'814","reported_seq":3865,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.697392+0000","last_change":"2024-09-21T15:50:34.697527+0000","last_active":"2024-09-21T15:50:34.697392+0000","last_peered":"2024-09-21T15:50:34.697392+0000","last_clean":"2024-09-21T15:50:34.697392+0000","last_became_active":"2024-09-21T15:49:06.689203+0000","last_became_peered":"2024-09-21T15:49:06.689203+0000","last_unstale":"2024-09-21T15:50:34.697392+0000","last_undegraded":"2024-09-21T15:50:34.697392+0000","last_fullsized":"2024-09-21T15:50:34.697392+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":814,"log_dups_size":0,"ondisk_log_size":814,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:22:17.578359+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.008723026,"stat_sum":{"num_bytes":78437702,"num_objects":80,"num_object_clones":12,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":44,"num_read":6904,"num_read_kb":27775,"num_write":1093,"num_write_kb":14798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"3.e","version":"97'469","reported_seq":7670,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.688800+0000","last_change":"2024-09-21T15:50:34.688968+0000","last_active":"2024-09-21T15:50:34.688800+0000","last_peered":"2024-09-21T15:50:34.688800+0000","last_clean":"2024-09-21T15:50:34.688800+0000","last_became_active":"2024-09-21T15:49:06.690101+0000","last_became_peered":"2024-09-21T15:49:06.690101+0000","last_unstale":"2024-09-21T15:50:34.688800+0000","last_undegraded":"2024-09-21T15:50:34.688800+0000","last_fullsized":"2024-09-21T15:50:34.688800+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":469,"log_dups_size":0,"ondisk_log_size":469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:43:30.659104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000516691,"stat_sum":{"num_bytes":116442302,"num_objects":82,"num_object_clones":19,"num_object_copies":164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":44,"num_read":8382,"num_read_kb":42678,"num_write":458,"num_write_kb":16261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"3.f","version":"99'605","reported_seq":3370,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.741468+0000","last_change":"2024-09-21T15:50:34.694457+0000","last_active":"2024-09-21T15:50:34.741468+0000","last_peered":"2024-09-21T15:50:34.741468+0000","last_clean":"2024-09-21T15:50:34.741468+0000","last_became_active":"2024-09-21T15:49:06.694250+0000","last_became_peered":"2024-09-21T15:49:06.694250+0000","last_unstale":"2024-09-21T15:50:34.741468+0000","last_undegraded":"2024-09-21T15:50:34.741468+0000","last_fullsized":"2024-09-21T15:50:34.741468+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":605,"log_dups_size":0,"ondisk_log_size":605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:21:37.777769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056822970000000002,"stat_sum":{"num_bytes":18103525,"num_objects":77,"num_object_clones":8,"num_object_copies":154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77,"num_whiteouts":50,"num_read":4619,"num_read_kb":15654,"num_write":810,"num_write_kb":5051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"3.10","version":"99'431","reported_seq":2797,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.679243+0000","last_change":"2024-09-21T15:50:34.679406+0000","last_active":"2024-09-21T15:50:34.679243+0000","last_peered":"2024-09-21T15:50:34.679243+0000","last_clean":"2024-09-21T15:50:34.679243+0000","last_became_active":"2024-09-21T15:50:32.676147+0000","last_became_peered":"2024-09-21T15:50:32.676147+0000","last_unstale":"2024-09-21T15:50:34.679243+0000","last_undegraded":"2024-09-21T15:50:34.679243+0000","last_fullsized":"2024-09-21T15:50:34.679243+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":431,"log_dups_size":0,"ondisk_log_size":431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:44:18.821310+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0093129760000000006,"stat_sum":{"num_bytes":13246714,"num_objects":57,"num_object_clones":4,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":44,"num_read":2952,"num_read_kb":11672,"num_write":356,"num_write_kb":4465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":1191778,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"3.11","version":"99'919","reported_seq":3784,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.696783+0000","last_change":"2024-09-21T15:50:34.697143+0000","last_active":"2024-09-21T15:50:34.696783+0000","last_peered":"2024-09-21T15:50:34.696783+0000","last_clean":"2024-09-21T15:50:34.696783+0000","last_became_active":"2024-09-21T15:49:06.688798+0000","last_became_peered":"2024-09-21T15:49:06.688798+0000","last_unstale":"2024-09-21T15:50:34.696783+0000","last_undegraded":"2024-09-21T15:50:34.696783+0000","last_fullsized":"2024-09-21T15:50:34.696783+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":919,"log_dups_size":0,"ondisk_log_size":919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:48:33.679133+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.008774479,"stat_sum":{"num_bytes":59874290,"num_objects":65,"num_object_clones":9,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":35,"num_read":6119,"num_read_kb":16744,"num_write":1280,"num_write_kb":8388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"3.12","version":"97'352","reported_seq":2735,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-21T15:50:32.688878+0000","last_change":"2024-09-21T15:50:32.688878+0000","last_active":"2024-09-21T15:50:32.688878+0000","last_peered":"2024-09-21T15:50:32.688878+0000","last_clean":"2024-09-21T15:50:32.688878+0000","last_became_active":"2024-09-21T15:50:32.671686+0000","last_became_peered":"2024-09-21T15:50:32.671686+0000","last_unstale":"2024-09-21T15:50:32.688878+0000","last_undegraded":"2024-09-21T15:50:32.688878+0000","last_fullsized":"2024-09-21T15:50:32.688878+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:40:53.968477+0000","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.015262016999999999,"stat_sum":{"num_bytes":39845746,"num_objects":64,"num_object_clones":8,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":41,"num_read":2441,"num_read_kb":11662,"num_write":386,"num_write_kb":13839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":3359702,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"4"}]},{"pgid":"3.13","version":"97'882","reported_seq":7104,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.688516+0000","last_change":"2024-09-21T15:50:34.688748+0000","last_active":"2024-09-21T15:50:34.688516+0000","last_peered":"2024-09-21T15:50:34.688516+0000","last_clean":"2024-09-21T15:50:34.688516+0000","last_became_active":"2024-09-21T15:49:06.690128+0000","last_became_peered":"2024-09-21T15:49:06.690128+0000","last_unstale":"2024-09-21T15:50:34.688516+0000","last_undegraded":"2024-09-21T15:50:34.688516+0000","last_fullsized":"2024-09-21T15:50:34.688516+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":882,"log_dups_size":0,"ondisk_log_size":882,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:11.763959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036025,"stat_sum":{"num_bytes":34582516,"num_objects":82,"num_object_clones":18,"num_object_copies":164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":48,"num_read":9229,"num_read_kb":26283,"num_write":1392,"num_write_kb":13566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2b","length":"5"}]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1484406535,"num_objects":2236,"num_object_clones":292,"num_object_copies":4472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2236,"num_whiteouts":1256,"num_read":196839,"num_read_kb":833374,"num_write":21337,"num_write_kb":302722,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":841,"num_bytes_recovered":47753679,"num_keys_recovered":165,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":377659392,"data_stored":506957447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5973,"internal_metadata":0},"log_size":17432,"ondisk_log_size":17432,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574872,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":127844,"kb_used_data":56532,"kb_used_omap":1,"kb_used_meta":71294,"kb_avail":94243996,"statfs":{"total":96636764160,"available":96505851904,"internally_reserved":0,"allocated":57888768,"data_stored":75307892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":73005515},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.34499999999999997}]},{"osd":1,"last update":"Sat Sep 21 15:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.37},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.38}]},{"osd":2,"last update":"Sat Sep 21 15:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.433}]},{"osd":3,"last update":"Sat Sep 21 15:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.36099999999999999}]},{"osd":4,"last update":"Sat Sep 21 15:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.317}]},{"osd":5,"last update":"Sat Sep 21 15:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.40600000000000003}]},{"osd":6,"last update":"Sat Sep 21 15:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.39700000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574875,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59612,"kb_used_data":24392,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94312228,"statfs":{"total":96636764160,"available":96575721472,"internally_reserved":0,"allocated":24977408,"data_stored":37618825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.42299999999999999}]},{"osd":1,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.38}]},{"osd":2,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.40400000000000003}]},{"osd":3,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.39200000000000002}]},{"osd":4,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.27100000000000002}]},{"osd":5,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.313}]},{"osd":7,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.37},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.41399999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574874,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":112016,"kb_used_data":62848,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94259824,"statfs":{"total":96636764160,"available":96522059776,"internally_reserved":0,"allocated":64356352,"data_stored":86513493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.48499999999999999}]},{"osd":1,"last update":"Sat Sep 21 15:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.52200000000000002}]},{"osd":2,"last update":"Sat Sep 21 15:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.50800000000000001}]},{"osd":3,"last update":"Sat Sep 21 15:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.375},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.6140000000000001,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.54000000000000004}]},{"osd":4,"last update":"Sat Sep 21 15:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":5.6399999999999997,"5min":5.6399999999999997,"15min":5.6399999999999997},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.27300000000000002}]},{"osd":6,"last update":"Sat Sep 21 15:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.55200000000000005}]},{"osd":7,"last update":"Sat Sep 21 15:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.39400000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574873,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":152052,"kb_used_data":80744,"kb_used_omap":1,"kb_used_meta":71294,"kb_avail":94219788,"statfs":{"total":96636764160,"available":96481062912,"internally_reserved":0,"allocated":82681856,"data_stored":113666860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73005510},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:49:52 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.32700000000000001}]},{"osd":1,"last update":"Sat Sep 21 15:49:52 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.217},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.38300000000000001}]},{"osd":2,"last update":"Sat Sep 21 15:49:52 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.437,"15min":0.437},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.31}]},{"osd":3,"last update":"Sat Sep 21 15:49:52 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.39600000000000002}]},{"osd":5,"last update":"Sat Sep 21 15:49:52 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":2.4489999999999998,"5min":2.4489999999999998,"15min":2.4489999999999998},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.44800000000000001}]},{"osd":6,"last update":"Sat Sep 21 15:49:52 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":2.7709999999999999,"5min":2.7709999999999999,"15min":2.7709999999999999},"last":0.34899999999999998}]},{"osd":7,"last update":"Sat Sep 21 15:49:52 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.8069999999999999,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.374}]}]},{"osd":3,"up_from":13,"seq":55834574874,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":109508,"kb_used_data":60344,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94262332,"statfs":{"total":96636764160,"available":96524627968,"internally_reserved":0,"allocated":61792256,"data_stored":85566797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50330056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.218},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.442,"5min":0.442,"15min":0.442},"last":0.20100000000000001}]},{"osd":1,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.28399999999999997}]},{"osd":2,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.25600000000000001}]},{"osd":4,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.371}]},{"osd":5,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.34200000000000003}]},{"osd":6,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.442}]},{"osd":7,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.38900000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574874,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38152,"kb_used_data":7036,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94333688,"statfs":{"total":96636764160,"available":96597696512,"internally_reserved":0,"allocated":7204864,"data_stored":6531679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.23200000000000001}]},{"osd":1,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":1.621,"5min":1.621,"15min":1.621},"last":0.19900000000000001},{"interface":"front","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.27100000000000002}]},{"osd":3,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":11.56,"5min":11.56,"15min":11.56},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.32300000000000001}]},{"osd":4,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.653,"5min":1.653,"15min":1.653},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":2.4660000000000002,"5min":2.4660000000000002,"15min":2.4660000000000002},"last":0.40699999999999997}]},{"osd":5,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.37},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.309}]},{"osd":6,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.436}]},{"osd":7,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.39500000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574873,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63648,"kb_used_data":28432,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94308192,"statfs":{"total":96636764160,"available":96571588608,"internally_reserved":0,"allocated":29114368,"data_stored":40956490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.315,"15min":0.315},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"last":0.31900000000000001}]},{"osd":2,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":5.0670000000000002,"5min":5.0670000000000002,"15min":5.0670000000000002},"last":0.28999999999999998}]},{"osd":3,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":16.032,"5min":16.032,"15min":16.032},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.309}]},{"osd":4,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2.2269999999999999,"5min":2.2269999999999999,"15min":2.2269999999999999},"last":0.46200000000000002}]},{"osd":5,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":2.2890000000000001,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":4.6989999999999998,"5min":4.6989999999999998,"15min":4.6989999999999998},"last":0.35899999999999999}]},{"osd":6,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":2.262,"5min":2.262,"15min":2.262},"last":0.41099999999999998}]},{"osd":7,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":4.742,"5min":4.742,"15min":4.742},"last":0.374},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.49099999999999999}]}]},{"osd":0,"up_from":93,"seq":399431958530,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81940,"kb_used_data":49412,"kb_used_omap":13,"kb_used_meta":32498,"kb_avail":94289900,"statfs":{"total":96636764160,"available":96552857600,"internally_reserved":0,"allocated":50597888,"data_stored":66108537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13968,"internal_metadata":33278320},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23300000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48852992,"data_stored":64979533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5973,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":27947008,"data_stored":38839043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":15609856,"data_stored":21056570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":60628992,"data_stored":82925081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":81522688,"data_stored":108334153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":63152128,"data_stored":84879777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":23814144,"data_stored":34452821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":56131584,"data_stored":71490469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T15:50:37.361 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-09-21T15:50:37.362 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:50:37.363 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:37.361+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:37.463 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:37.462+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:37.505 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:37.505 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:37.514 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T15:50:37.563 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:37.562+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:37.655 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:37.655 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:37.664 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T15:50:37.665 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:37.663+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:37.739 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:50:37.739 INFO:teuthology.orchestra.run.smithi089.stdout:{"fsid":"4a9b64ea-3aaf-4a40-8ce8-02163a0f6e12","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"nodeep-scrub flag(s) set","count":12},"muted":false}},"mutes":[]},"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":119,"monmap":{"epoch":1,"min_mon_release_name":"reef","num_mons":3},"osdmap":{"epoch":102,"num_osds":8,"num_up_osds":8,"osd_up_since":1726933829,"num_in_osds":8,"osd_in_since":1726933720,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":39},{"state_name":"peering","count":2}],"num_pgs":41,"num_pools":3,"num_objects":2240,"data_bytes":1484996922,"bytes_used":762646528,"bytes_avail":772331466752,"bytes_total":773094113280,"inactive_pgs_ratio":0.048780485987663269,"recovering_objects_per_sec":184,"recovering_bytes_per_sec":10245071,"recovering_keys_per_sec":37,"num_objects_recovered":691,"num_bytes_recovered":38465827,"num_keys_recovered":141,"read_bytes_sec":2542157,"write_bytes_sec":227460,"read_op_per_sec":351,"write_op_per_sec":76},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["iostat","nfs","restful"],"services":{}},"servicemap":{"epoch":3,"modified":"2024-09-21T15:49:51.899551+0000","services":{}},"progress_events":{}} 2024-09-21T15:50:37.752 INFO:tasks.ceph.ceph_manager.ceph:making progress, resetting timeout 2024-09-21T15:50:37.752 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T15:50:37.764 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:37.763+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:37.783 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:37.783 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:37.792 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T15:50:37.865 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:37.864+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:37.935 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:37.935 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:37.943 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T15:50:37.966 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:50:37.965+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:50:38.046 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:38.046 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:38.054 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T15:50:38.065 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:50:38.066 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-09-21T15:50:38.067 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:38.066+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:38.077 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":149,"stamp":"2024-09-21T15:50:37.906918+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1484996922,"num_objects":2240,"num_object_clones":292,"num_object_copies":4480,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2240,"num_whiteouts":1256,"num_read":196955,"num_read_kb":833474,"num_write":21498,"num_write_kb":305140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":841,"num_bytes_recovered":47753679,"num_keys_recovered":165,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17552,"ondisk_log_size":17552,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":85,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":744772,"kb_used_data":369740,"kb_used_omap":24,"kb_used_meta":374887,"kb_avail":754229948,"statfs":{"total":773094113280,"available":772331466752,"internally_reserved":0,"allocated":378613760,"data_stored":512270573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25118,"internal_metadata":383884770},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-38957205,"num_objects":-61,"num_object_clones":-15,"num_object_copies":-122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-61,"num_whiteouts":-8,"num_read":374,"num_read_kb":317,"num_write":95,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":334,"num_bytes_recovered":11272085,"num_keys_recovered":60,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000510"},"pg_stats":[{"pgid":"3.1f","version":"97'368","reported_seq":1751,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-21T15:50:32.682121+0000","last_change":"2024-09-21T15:50:32.682121+0000","last_active":"2024-09-21T15:50:32.682121+0000","last_peered":"2024-09-21T15:50:32.682121+0000","last_clean":"2024-09-21T15:50:32.682121+0000","last_became_active":"2024-09-21T15:50:32.670964+0000","last_became_peered":"2024-09-21T15:50:32.670964+0000","last_unstale":"2024-09-21T15:50:32.682121+0000","last_undegraded":"2024-09-21T15:50:32.682121+0000","last_fullsized":"2024-09-21T15:50:32.682121+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":368,"log_dups_size":0,"ondisk_log_size":368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:09:09.918864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0091502430000000006,"stat_sum":{"num_bytes":7925545,"num_objects":51,"num_object_clones":5,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":32,"num_read":2166,"num_read_kb":10210,"num_write":412,"num_write_kb":5950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":1974433,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"4"}]},{"pgid":"3.1e","version":"97'478","reported_seq":5380,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.670183+0000","last_change":"2024-09-21T15:50:34.670386+0000","last_active":"2024-09-21T15:50:34.670183+0000","last_peered":"2024-09-21T15:50:34.670183+0000","last_clean":"2024-09-21T15:50:34.670183+0000","last_became_active":"2024-09-21T15:50:20.087949+0000","last_became_peered":"2024-09-21T15:50:20.087949+0000","last_unstale":"2024-09-21T15:50:34.670183+0000","last_undegraded":"2024-09-21T15:50:34.670183+0000","last_fullsized":"2024-09-21T15:50:34.670183+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":478,"log_dups_size":0,"ondisk_log_size":478,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:26:31.786747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036498599999999998,"stat_sum":{"num_bytes":53784778,"num_objects":82,"num_object_clones":10,"num_object_copies":164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":45,"num_read":6836,"num_read_kb":25379,"num_write":492,"num_write_kb":4338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"3.1d","version":"97'545","reported_seq":9776,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.673773+0000","last_change":"2024-09-21T15:50:34.679434+0000","last_active":"2024-09-21T15:50:34.673773+0000","last_peered":"2024-09-21T15:50:34.673773+0000","last_clean":"2024-09-21T15:50:34.673773+0000","last_became_active":"2024-09-21T15:49:39.833206+0000","last_became_peered":"2024-09-21T15:49:39.833206+0000","last_unstale":"2024-09-21T15:50:34.673773+0000","last_undegraded":"2024-09-21T15:50:34.673773+0000","last_fullsized":"2024-09-21T15:50:34.673773+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":545,"log_dups_size":0,"ondisk_log_size":545,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:36:14.082318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057857509999999996,"stat_sum":{"num_bytes":72397601,"num_objects":88,"num_object_clones":16,"num_object_copies":176,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":88,"num_whiteouts":46,"num_read":10437,"num_read_kb":28908,"num_write":627,"num_write_kb":8866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"3.1c","version":"99'471","reported_seq":3905,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.745138+0000","last_change":"2024-09-21T15:50:34.688353+0000","last_active":"2024-09-21T15:50:34.745138+0000","last_peered":"2024-09-21T15:50:34.745138+0000","last_clean":"2024-09-21T15:50:34.745138+0000","last_became_active":"2024-09-21T15:49:39.833113+0000","last_became_peered":"2024-09-21T15:49:39.833113+0000","last_unstale":"2024-09-21T15:50:34.745138+0000","last_undegraded":"2024-09-21T15:50:34.745138+0000","last_fullsized":"2024-09-21T15:50:34.745138+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":471,"log_dups_size":0,"ondisk_log_size":471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:43:57.592707+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.014658071999999999,"stat_sum":{"num_bytes":16748346,"num_objects":62,"num_object_clones":7,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":32,"num_read":4719,"num_read_kb":16862,"num_write":512,"num_write_kb":4777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"3.1b","version":"97'436","reported_seq":4008,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-21T15:50:32.756744+0000","last_change":"2024-09-21T15:50:32.689144+0000","last_active":"2024-09-21T15:50:32.756744+0000","last_peered":"2024-09-21T15:50:32.756744+0000","last_clean":"2024-09-21T15:50:32.756744+0000","last_became_active":"2024-09-21T15:50:32.674250+0000","last_became_peered":"2024-09-21T15:50:32.674250+0000","last_unstale":"2024-09-21T15:50:32.756744+0000","last_undegraded":"2024-09-21T15:50:32.756744+0000","last_fullsized":"2024-09-21T15:50:32.756744+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":436,"log_dups_size":0,"ondisk_log_size":436,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:54:57.399818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.01221852,"stat_sum":{"num_bytes":57746387,"num_objects":76,"num_object_clones":5,"num_object_copies":152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":42,"num_read":4612,"num_read_kb":29920,"num_write":444,"num_write_kb":13501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":4759000,"num_keys_recovered":12,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"4"}]},{"pgid":"3.1a","version":"86'348","reported_seq":6175,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.716068+0000","last_change":"2024-09-21T15:50:34.688560+0000","last_active":"2024-09-21T15:50:34.716068+0000","last_peered":"2024-09-21T15:50:34.716068+0000","last_clean":"2024-09-21T15:50:34.716068+0000","last_became_active":"2024-09-21T15:49:06.693903+0000","last_became_peered":"2024-09-21T15:49:06.693903+0000","last_unstale":"2024-09-21T15:50:34.716068+0000","last_undegraded":"2024-09-21T15:50:34.716068+0000","last_fullsized":"2024-09-21T15:50:34.716068+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:41:45.432893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.014798159999999999,"stat_sum":{"num_bytes":52938660,"num_objects":68,"num_object_clones":9,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":36,"num_read":6525,"num_read_kb":31121,"num_write":367,"num_write_kb":9464,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"3.19","version":"99'409","reported_seq":8031,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.689444+0000","last_change":"2024-09-21T15:50:34.689718+0000","last_active":"2024-09-21T15:50:34.689444+0000","last_peered":"2024-09-21T15:50:34.689444+0000","last_clean":"2024-09-21T15:50:34.689444+0000","last_became_active":"2024-09-21T15:49:06.691222+0000","last_became_peered":"2024-09-21T15:49:06.691222+0000","last_unstale":"2024-09-21T15:50:34.689444+0000","last_undegraded":"2024-09-21T15:50:34.689444+0000","last_fullsized":"2024-09-21T15:50:34.689444+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":409,"log_dups_size":0,"ondisk_log_size":409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:31:29.545866+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.015870318000000001,"stat_sum":{"num_bytes":65843420,"num_objects":71,"num_object_clones":5,"num_object_copies":142,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":36,"num_read":8232,"num_read_kb":39547,"num_write":475,"num_write_kb":7403,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"3.18","version":"97'407","reported_seq":3449,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-21T15:50:32.767353+0000","last_change":"2024-09-21T15:50:32.688342+0000","last_active":"2024-09-21T15:50:32.767353+0000","last_peered":"2024-09-21T15:50:32.767353+0000","last_clean":"2024-09-21T15:50:32.767353+0000","last_became_active":"2024-09-21T15:50:32.672791+0000","last_became_peered":"2024-09-21T15:50:32.672791+0000","last_unstale":"2024-09-21T15:50:32.767353+0000","last_undegraded":"2024-09-21T15:50:32.767353+0000","last_fullsized":"2024-09-21T15:50:32.767353+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":407,"log_dups_size":0,"ondisk_log_size":407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:07:19.540473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013942115999999999,"stat_sum":{"num_bytes":48735247,"num_objects":67,"num_object_clones":9,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":30,"num_read":5193,"num_read_kb":30076,"num_write":442,"num_write_kb":11051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":2076323,"num_keys_recovered":11,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"4"}]},{"pgid":"3.17","version":"97'917","reported_seq":8775,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-21T15:50:32.775257+0000","last_change":"2024-09-21T15:50:32.684436+0000","last_active":"2024-09-21T15:50:32.775257+0000","last_peered":"2024-09-21T15:50:32.775257+0000","last_clean":"2024-09-21T15:50:32.775257+0000","last_became_active":"2024-09-21T15:50:32.668771+0000","last_became_peered":"2024-09-21T15:50:32.668771+0000","last_unstale":"2024-09-21T15:50:32.775257+0000","last_undegraded":"2024-09-21T15:50:32.775257+0000","last_fullsized":"2024-09-21T15:50:32.775257+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":917,"log_dups_size":0,"ondisk_log_size":917,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:39:43.554150+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.01371901,"stat_sum":{"num_bytes":23526347,"num_objects":74,"num_object_clones":9,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":43,"num_read":11967,"num_read_kb":20847,"num_write":1212,"num_write_kb":8234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":3319185,"num_keys_recovered":34,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"4"}]},{"pgid":"3.16","version":"90'468","reported_seq":6313,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.669833+0000","last_change":"2024-09-21T15:50:34.670140+0000","last_active":"2024-09-21T15:50:34.669833+0000","last_peered":"2024-09-21T15:50:34.669833+0000","last_clean":"2024-09-21T15:50:34.669833+0000","last_became_active":"2024-09-21T15:49:06.688808+0000","last_became_peered":"2024-09-21T15:49:06.688808+0000","last_unstale":"2024-09-21T15:50:34.669833+0000","last_undegraded":"2024-09-21T15:50:34.669833+0000","last_fullsized":"2024-09-21T15:50:34.669833+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":468,"log_dups_size":0,"ondisk_log_size":468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:43:02.012254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00045747499999999999,"stat_sum":{"num_bytes":96553494,"num_objects":69,"num_object_clones":9,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":38,"num_read":7324,"num_read_kb":49627,"num_write":491,"num_write_kb":13812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"3.15","version":"99'483","reported_seq":3566,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.694048+0000","last_change":"2024-09-21T15:50:34.694265+0000","last_active":"2024-09-21T15:50:34.694048+0000","last_peered":"2024-09-21T15:50:34.694048+0000","last_clean":"2024-09-21T15:50:34.694048+0000","last_became_active":"2024-09-21T15:49:06.691091+0000","last_became_peered":"2024-09-21T15:49:06.691091+0000","last_unstale":"2024-09-21T15:50:34.694048+0000","last_undegraded":"2024-09-21T15:50:34.694048+0000","last_fullsized":"2024-09-21T15:50:34.694048+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":483,"log_dups_size":0,"ondisk_log_size":483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:13:12.258161+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0056158889999999998,"stat_sum":{"num_bytes":32455197,"num_objects":64,"num_object_clones":11,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":37,"num_read":4033,"num_read_kb":27130,"num_write":613,"num_write_kb":17935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"3.14","version":"99'1134","reported_seq":4056,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.691334+0000","last_change":"2024-09-21T15:50:34.691584+0000","last_active":"2024-09-21T15:50:34.691334+0000","last_peered":"2024-09-21T15:50:34.691334+0000","last_clean":"2024-09-21T15:50:34.691334+0000","last_became_active":"2024-09-21T15:49:06.688803+0000","last_became_peered":"2024-09-21T15:49:06.688803+0000","last_unstale":"2024-09-21T15:50:34.691334+0000","last_undegraded":"2024-09-21T15:50:34.691334+0000","last_fullsized":"2024-09-21T15:50:34.691334+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":1134,"log_dups_size":0,"ondisk_log_size":1134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:18:47.662631+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.017866338999999998,"stat_sum":{"num_bytes":69136666,"num_objects":55,"num_object_clones":4,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":28,"num_read":7641,"num_read_kb":18141,"num_write":1856,"num_write_kb":6175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"2.7","version":"0'0","reported_seq":95,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-21T15:50:33.667873+0000","last_change":"2024-09-21T15:48:59.666431+0000","last_active":"2024-09-21T15:50:33.667873+0000","last_peered":"2024-09-21T15:50:33.667873+0000","last_clean":"2024-09-21T15:50:33.667873+0000","last_became_active":"2024-09-21T15:48:57.583665+0000","last_became_peered":"2024-09-21T15:48:57.583665+0000","last_unstale":"2024-09-21T15:50:33.667873+0000","last_undegraded":"2024-09-21T15:50:33.667873+0000","last_fullsized":"2024-09-21T15:50:33.667873+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:52:09.228964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031098399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"97'353","reported_seq":2734,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-21T15:50:32.769944+0000","last_change":"2024-09-21T15:50:32.684589+0000","last_active":"2024-09-21T15:50:32.769944+0000","last_peered":"2024-09-21T15:50:32.769944+0000","last_clean":"2024-09-21T15:50:32.769944+0000","last_became_active":"2024-09-21T15:50:32.671424+0000","last_became_peered":"2024-09-21T15:50:32.671424+0000","last_unstale":"2024-09-21T15:50:32.769944+0000","last_undegraded":"2024-09-21T15:50:32.769944+0000","last_fullsized":"2024-09-21T15:50:32.769944+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:04:14.554732+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.011297845000000001,"stat_sum":{"num_bytes":14549265,"num_objects":67,"num_object_clones":10,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":44,"num_read":2599,"num_read_kb":12929,"num_write":362,"num_write_kb":8387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":4494388,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"4"}]},{"pgid":"2.6","version":"0'0","reported_seq":95,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-21T15:50:33.667962+0000","last_change":"2024-09-21T15:48:59.589790+0000","last_active":"2024-09-21T15:50:33.667962+0000","last_peered":"2024-09-21T15:50:33.667962+0000","last_clean":"2024-09-21T15:50:33.667962+0000","last_became_active":"2024-09-21T15:48:57.582121+0000","last_became_peered":"2024-09-21T15:48:57.582121+0000","last_unstale":"2024-09-21T15:50:33.667962+0000","last_undegraded":"2024-09-21T15:50:33.667962+0000","last_fullsized":"2024-09-21T15:50:33.667962+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:33:47.592624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039654700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"97'516","reported_seq":7087,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.689003+0000","last_change":"2024-09-21T15:50:34.689275+0000","last_active":"2024-09-21T15:50:34.689003+0000","last_peered":"2024-09-21T15:50:34.689003+0000","last_clean":"2024-09-21T15:50:34.689003+0000","last_became_active":"2024-09-21T15:50:20.087935+0000","last_became_peered":"2024-09-21T15:50:20.087935+0000","last_unstale":"2024-09-21T15:50:34.689003+0000","last_undegraded":"2024-09-21T15:50:34.689003+0000","last_fullsized":"2024-09-21T15:50:34.689003+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":516,"log_dups_size":0,"ondisk_log_size":516,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:27:45.090104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062972600000000005,"stat_sum":{"num_bytes":34633616,"num_objects":75,"num_object_clones":11,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75,"num_whiteouts":44,"num_read":8167,"num_read_kb":21012,"num_write":508,"num_write_kb":5391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"2.5","version":"0'0","reported_seq":108,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.149704+0000","last_change":"2024-09-21T15:50:30.146743+0000","last_active":"2024-09-21T15:50:34.149704+0000","last_peered":"2024-09-21T15:50:34.149704+0000","last_clean":"2024-09-21T15:50:34.149704+0000","last_became_active":"2024-09-21T15:50:30.146428+0000","last_became_peered":"2024-09-21T15:50:30.146428+0000","last_unstale":"2024-09-21T15:50:34.149704+0000","last_undegraded":"2024-09-21T15:50:34.149704+0000","last_fullsized":"2024-09-21T15:50:34.149704+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:58:38.689807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036074600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"98'592","reported_seq":8284,"reported_epoch":99,"state":"peering","last_fresh":"2024-09-21T15:50:34.674329+0000","last_change":"2024-09-21T15:50:34.674592+0000","last_active":"2024-09-21T15:50:34.674329+0000","last_peered":"2024-09-21T15:50:34.567209+0000","last_clean":"2024-09-21T15:50:31.216782+0000","last_became_active":"2024-09-21T15:50:34.144000+0000","last_became_peered":"2024-09-21T15:50:34.144000+0000","last_unstale":"2024-09-21T15:50:34.674329+0000","last_undegraded":"2024-09-21T15:50:34.674329+0000","last_fullsized":"2024-09-21T15:50:34.674329+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":592,"log_dups_size":0,"ondisk_log_size":592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:56:17.362656+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0066134189999999997,"stat_sum":{"num_bytes":36608631,"num_objects":68,"num_object_clones":13,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":34,"num_read":10413,"num_read_kb":27152,"num_write":790,"num_write_kb":10996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":175,"num_bytes_recovered":14134268,"num_keys_recovered":36,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,5],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2d","length":"3"}]},{"pgid":"2.4","version":"0'0","reported_seq":108,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-21T15:50:33.667911+0000","last_change":"2024-09-21T15:50:30.146654+0000","last_active":"2024-09-21T15:50:33.667911+0000","last_peered":"2024-09-21T15:50:33.667911+0000","last_clean":"2024-09-21T15:50:33.667911+0000","last_became_active":"2024-09-21T15:50:30.146273+0000","last_became_peered":"2024-09-21T15:50:30.146273+0000","last_unstale":"2024-09-21T15:50:33.667911+0000","last_undegraded":"2024-09-21T15:50:33.667911+0000","last_fullsized":"2024-09-21T15:50:33.667911+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:21:48.178519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040525700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"99'517","reported_seq":4959,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.682802+0000","last_change":"2024-09-21T15:50:34.683079+0000","last_active":"2024-09-21T15:50:34.682802+0000","last_peered":"2024-09-21T15:50:34.682802+0000","last_clean":"2024-09-21T15:50:34.682802+0000","last_became_active":"2024-09-21T15:49:39.835880+0000","last_became_peered":"2024-09-21T15:49:39.835880+0000","last_unstale":"2024-09-21T15:50:34.682802+0000","last_undegraded":"2024-09-21T15:50:34.682802+0000","last_fullsized":"2024-09-21T15:50:34.682802+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":517,"log_dups_size":0,"ondisk_log_size":517,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:28:03.488776+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0062742270000000003,"stat_sum":{"num_bytes":75150008,"num_objects":84,"num_object_clones":13,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84,"num_whiteouts":43,"num_read":5511,"num_read_kb":33759,"num_write":629,"num_write_kb":16321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"3.3","version":"99'643","reported_seq":3291,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.683173+0000","last_change":"2024-09-21T15:50:34.683173+0000","last_active":"2024-09-21T15:50:34.683173+0000","last_peered":"2024-09-21T15:50:34.683173+0000","last_clean":"2024-09-21T15:50:34.683173+0000","last_became_active":"2024-09-21T15:50:32.677456+0000","last_became_peered":"2024-09-21T15:50:32.677456+0000","last_unstale":"2024-09-21T15:50:34.683173+0000","last_undegraded":"2024-09-21T15:50:34.683173+0000","last_fullsized":"2024-09-21T15:50:34.683173+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":643,"log_dups_size":0,"ondisk_log_size":643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:46:31.194345+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0095847789999999999,"stat_sum":{"num_bytes":23664738,"num_objects":79,"num_object_clones":14,"num_object_copies":158,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":79,"num_whiteouts":47,"num_read":4317,"num_read_kb":18605,"num_write":713,"num_write_kb":7264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":632176,"num_keys_recovered":24,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"2.2","version":"20'2","reported_seq":74,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-21T15:50:33.667901+0000","last_change":"2024-09-21T15:49:39.834640+0000","last_active":"2024-09-21T15:50:33.667901+0000","last_peered":"2024-09-21T15:50:33.667901+0000","last_clean":"2024-09-21T15:50:33.667901+0000","last_became_active":"2024-09-21T15:49:39.834207+0000","last_became_peered":"2024-09-21T15:49:39.834207+0000","last_unstale":"2024-09-21T15:50:33.667901+0000","last_undegraded":"2024-09-21T15:50:33.667901+0000","last_fullsized":"2024-09-21T15:50:33.667901+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:02:54.255716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"98'346","reported_seq":4438,"reported_epoch":99,"state":"peering","last_fresh":"2024-09-21T15:50:34.674453+0000","last_change":"2024-09-21T15:50:34.674621+0000","last_active":"2024-09-21T15:50:34.674453+0000","last_peered":"2024-09-21T15:50:34.339949+0000","last_clean":"2024-09-21T15:50:31.142696+0000","last_became_active":"2024-09-21T15:50:34.143212+0000","last_became_peered":"2024-09-21T15:50:34.143212+0000","last_unstale":"2024-09-21T15:50:34.674453+0000","last_undegraded":"2024-09-21T15:50:34.674453+0000","last_fullsized":"2024-09-21T15:50:34.674453+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":346,"log_dups_size":0,"ondisk_log_size":346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:49:25.944216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052199600000000005,"stat_sum":{"num_bytes":20675160,"num_objects":57,"num_object_clones":4,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":35,"num_read":4043,"num_read_kb":23169,"num_write":345,"num_write_kb":4139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":187,"num_bytes_recovered":4305276,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,6],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2d","length":"3"}]},{"pgid":"2.1","version":"0'0","reported_seq":12,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.679630+0000","last_change":"2024-09-21T15:50:32.672133+0000","last_active":"2024-09-21T15:50:34.679630+0000","last_peered":"2024-09-21T15:50:34.679630+0000","last_clean":"2024-09-21T15:50:34.679630+0000","last_became_active":"2024-09-21T15:50:32.671865+0000","last_became_peered":"2024-09-21T15:50:32.671865+0000","last_unstale":"2024-09-21T15:50:34.679630+0000","last_undegraded":"2024-09-21T15:50:34.679630+0000","last_fullsized":"2024-09-21T15:50:34.679630+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:55:23.999144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"97'453","reported_seq":2000,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-21T15:50:32.689058+0000","last_change":"2024-09-21T15:50:32.689245+0000","last_active":"2024-09-21T15:50:32.689058+0000","last_peered":"2024-09-21T15:50:32.689058+0000","last_clean":"2024-09-21T15:50:32.689058+0000","last_became_active":"2024-09-21T15:50:32.671413+0000","last_became_peered":"2024-09-21T15:50:32.671413+0000","last_unstale":"2024-09-21T15:50:32.689058+0000","last_undegraded":"2024-09-21T15:50:32.689058+0000","last_fullsized":"2024-09-21T15:50:32.689058+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":453,"log_dups_size":0,"ondisk_log_size":453,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:04:19.509402+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.01529492,"stat_sum":{"num_bytes":26803270,"num_objects":67,"num_object_clones":5,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":42,"num_read":3001,"num_read_kb":15971,"num_write":510,"num_write_kb":5955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":1696547,"num_keys_recovered":12,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"4"}]},{"pgid":"2.0","version":"0'0","reported_seq":95,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.149531+0000","last_change":"2024-09-21T15:49:00.114730+0000","last_active":"2024-09-21T15:50:34.149531+0000","last_peered":"2024-09-21T15:50:34.149531+0000","last_clean":"2024-09-21T15:50:34.149531+0000","last_became_active":"2024-09-21T15:48:57.585525+0000","last_became_peered":"2024-09-21T15:48:57.585525+0000","last_unstale":"2024-09-21T15:50:34.149531+0000","last_undegraded":"2024-09-21T15:50:34.149531+0000","last_fullsized":"2024-09-21T15:50:34.149531+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:35:35.786526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031691100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"99'365","reported_seq":2203,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.679026+0000","last_change":"2024-09-21T15:50:34.679434+0000","last_active":"2024-09-21T15:50:34.679026+0000","last_peered":"2024-09-21T15:50:34.679026+0000","last_clean":"2024-09-21T15:50:34.679026+0000","last_became_active":"2024-09-21T15:50:20.087681+0000","last_became_peered":"2024-09-21T15:50:20.087681+0000","last_unstale":"2024-09-21T15:50:34.679026+0000","last_undegraded":"2024-09-21T15:50:34.679026+0000","last_fullsized":"2024-09-21T15:50:34.679026+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":365,"log_dups_size":0,"ondisk_log_size":365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:07:05.059684+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0098461269999999997,"stat_sum":{"num_bytes":71749570,"num_objects":65,"num_object_clones":6,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":34,"num_read":2107,"num_read_kb":34423,"num_write":369,"num_write_kb":12608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"2.3","version":"18'1","reported_seq":67,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.135566+0000","last_change":"2024-09-21T15:49:39.834423+0000","last_active":"2024-09-21T15:50:34.135566+0000","last_peered":"2024-09-21T15:50:34.135566+0000","last_clean":"2024-09-21T15:50:34.135566+0000","last_became_active":"2024-09-21T15:49:39.833860+0000","last_became_peered":"2024-09-21T15:49:39.833860+0000","last_unstale":"2024-09-21T15:50:34.135566+0000","last_undegraded":"2024-09-21T15:50:34.135566+0000","last_fullsized":"2024-09-21T15:50:34.135566+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:11:43.630472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":256,"reported_epoch":98,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.149567+0000","last_change":"2024-09-21T15:50:30.147280+0000","last_active":"2024-09-21T15:50:34.149567+0000","last_peered":"2024-09-21T15:50:34.149567+0000","last_clean":"2024-09-21T15:50:34.149567+0000","last_became_active":"2024-09-21T15:50:30.146987+0000","last_became_peered":"2024-09-21T15:50:30.146987+0000","last_unstale":"2024-09-21T15:50:34.149567+0000","last_undegraded":"2024-09-21T15:50:34.149567+0000","last_fullsized":"2024-09-21T15:50:34.149567+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:50.161884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:50.161884+0000","last_clean_scrub_stamp":"2024-09-21T15:48:50.161884+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:26:32.853650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"97'451","reported_seq":6968,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.676782+0000","last_change":"2024-09-21T15:50:34.676960+0000","last_active":"2024-09-21T15:50:34.676782+0000","last_peered":"2024-09-21T15:50:34.676782+0000","last_clean":"2024-09-21T15:50:34.676782+0000","last_became_active":"2024-09-21T15:49:18.669143+0000","last_became_peered":"2024-09-21T15:49:18.669143+0000","last_unstale":"2024-09-21T15:50:34.676782+0000","last_undegraded":"2024-09-21T15:50:34.676782+0000","last_fullsized":"2024-09-21T15:50:34.676782+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":451,"log_dups_size":0,"ondisk_log_size":451,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:31:03.684897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048745400000000001,"stat_sum":{"num_bytes":42105276,"num_objects":76,"num_object_clones":8,"num_object_copies":152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":39,"num_read":6996,"num_read_kb":29381,"num_write":489,"num_write_kb":4779,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":296439,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"3.9","version":"95'343","reported_seq":2877,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.679521+0000","last_change":"2024-09-21T15:50:34.679797+0000","last_active":"2024-09-21T15:50:34.679521+0000","last_peered":"2024-09-21T15:50:34.679521+0000","last_clean":"2024-09-21T15:50:34.679521+0000","last_became_active":"2024-09-21T15:49:07.210506+0000","last_became_peered":"2024-09-21T15:49:07.210506+0000","last_unstale":"2024-09-21T15:50:34.679521+0000","last_undegraded":"2024-09-21T15:50:34.679521+0000","last_fullsized":"2024-09-21T15:50:34.679521+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":343,"log_dups_size":0,"ondisk_log_size":343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:55:18.046738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0061377159999999997,"stat_sum":{"num_bytes":43732741,"num_objects":65,"num_object_clones":4,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":34,"num_read":2674,"num_read_kb":28217,"num_write":414,"num_write_kb":12085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"3.a","version":"99'476","reported_seq":11781,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.682550+0000","last_change":"2024-09-21T15:50:34.682748+0000","last_active":"2024-09-21T15:50:34.682550+0000","last_peered":"2024-09-21T15:50:34.682550+0000","last_clean":"2024-09-21T15:50:34.682550+0000","last_became_active":"2024-09-21T15:49:06.694202+0000","last_became_peered":"2024-09-21T15:49:06.694202+0000","last_unstale":"2024-09-21T15:50:34.682550+0000","last_undegraded":"2024-09-21T15:50:34.682550+0000","last_fullsized":"2024-09-21T15:50:34.682550+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":476,"log_dups_size":0,"ondisk_log_size":476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:02:05.237705+0000","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0.012832926999999999,"stat_sum":{"num_bytes":70190032,"num_objects":67,"num_object_clones":10,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":34,"num_read":12417,"num_read_kb":63115,"num_write":489,"num_write_kb":14195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"3.b","version":"97'803","reported_seq":3095,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-21T15:50:32.775526+0000","last_change":"2024-09-21T15:50:32.161720+0000","last_active":"2024-09-21T15:50:32.775526+0000","last_peered":"2024-09-21T15:50:32.775526+0000","last_clean":"2024-09-21T15:50:32.775526+0000","last_became_active":"2024-09-21T15:50:30.149210+0000","last_became_peered":"2024-09-21T15:50:30.149210+0000","last_unstale":"2024-09-21T15:50:32.775526+0000","last_undegraded":"2024-09-21T15:50:32.775526+0000","last_fullsized":"2024-09-21T15:50:32.775526+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":803,"log_dups_size":0,"ondisk_log_size":803,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:51:30.431071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003604112,"stat_sum":{"num_bytes":40581743,"num_objects":60,"num_object_clones":10,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":33,"num_read":5903,"num_read_kb":20139,"num_write":1214,"num_write_kb":7745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":5514164,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"4"}]},{"pgid":"3.c","version":"91'638","reported_seq":6249,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.676261+0000","last_change":"2024-09-21T15:50:34.676449+0000","last_active":"2024-09-21T15:50:34.676261+0000","last_peered":"2024-09-21T15:50:34.676261+0000","last_clean":"2024-09-21T15:50:34.676261+0000","last_became_active":"2024-09-21T15:49:39.835282+0000","last_became_peered":"2024-09-21T15:49:39.835282+0000","last_unstale":"2024-09-21T15:50:34.676261+0000","last_undegraded":"2024-09-21T15:50:34.676261+0000","last_fullsized":"2024-09-21T15:50:34.676261+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":638,"log_dups_size":0,"ondisk_log_size":638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:51:40.356906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049830499999999999,"stat_sum":{"num_bytes":25638002,"num_objects":72,"num_object_clones":7,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":42,"num_read":8360,"num_read_kb":25266,"num_write":787,"num_write_kb":4983,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"3.d","version":"99'814","reported_seq":3865,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.697392+0000","last_change":"2024-09-21T15:50:34.697527+0000","last_active":"2024-09-21T15:50:34.697392+0000","last_peered":"2024-09-21T15:50:34.697392+0000","last_clean":"2024-09-21T15:50:34.697392+0000","last_became_active":"2024-09-21T15:49:06.689203+0000","last_became_peered":"2024-09-21T15:49:06.689203+0000","last_unstale":"2024-09-21T15:50:34.697392+0000","last_undegraded":"2024-09-21T15:50:34.697392+0000","last_fullsized":"2024-09-21T15:50:34.697392+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":814,"log_dups_size":0,"ondisk_log_size":814,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:22:17.578359+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.008723026,"stat_sum":{"num_bytes":78437702,"num_objects":80,"num_object_clones":12,"num_object_copies":160,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":80,"num_whiteouts":44,"num_read":6904,"num_read_kb":27775,"num_write":1093,"num_write_kb":14798,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"3.e","version":"97'469","reported_seq":7670,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.688800+0000","last_change":"2024-09-21T15:50:34.688968+0000","last_active":"2024-09-21T15:50:34.688800+0000","last_peered":"2024-09-21T15:50:34.688800+0000","last_clean":"2024-09-21T15:50:34.688800+0000","last_became_active":"2024-09-21T15:49:06.690101+0000","last_became_peered":"2024-09-21T15:49:06.690101+0000","last_unstale":"2024-09-21T15:50:34.688800+0000","last_undegraded":"2024-09-21T15:50:34.688800+0000","last_fullsized":"2024-09-21T15:50:34.688800+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":469,"log_dups_size":0,"ondisk_log_size":469,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:43:30.659104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000516691,"stat_sum":{"num_bytes":116442302,"num_objects":82,"num_object_clones":19,"num_object_copies":164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":44,"num_read":8382,"num_read_kb":42678,"num_write":458,"num_write_kb":16261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"3.f","version":"99'605","reported_seq":3370,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.741468+0000","last_change":"2024-09-21T15:50:34.694457+0000","last_active":"2024-09-21T15:50:34.741468+0000","last_peered":"2024-09-21T15:50:34.741468+0000","last_clean":"2024-09-21T15:50:34.741468+0000","last_became_active":"2024-09-21T15:49:06.694250+0000","last_became_peered":"2024-09-21T15:49:06.694250+0000","last_unstale":"2024-09-21T15:50:34.741468+0000","last_undegraded":"2024-09-21T15:50:34.741468+0000","last_fullsized":"2024-09-21T15:50:34.741468+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":605,"log_dups_size":0,"ondisk_log_size":605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:21:37.777769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0056822970000000002,"stat_sum":{"num_bytes":18103525,"num_objects":77,"num_object_clones":8,"num_object_copies":154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77,"num_whiteouts":50,"num_read":4619,"num_read_kb":15654,"num_write":810,"num_write_kb":5051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"3.10","version":"99'431","reported_seq":2797,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.679243+0000","last_change":"2024-09-21T15:50:34.679406+0000","last_active":"2024-09-21T15:50:34.679243+0000","last_peered":"2024-09-21T15:50:34.679243+0000","last_clean":"2024-09-21T15:50:34.679243+0000","last_became_active":"2024-09-21T15:50:32.676147+0000","last_became_peered":"2024-09-21T15:50:32.676147+0000","last_unstale":"2024-09-21T15:50:34.679243+0000","last_undegraded":"2024-09-21T15:50:34.679243+0000","last_fullsized":"2024-09-21T15:50:34.679243+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":431,"log_dups_size":0,"ondisk_log_size":431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:44:18.821310+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0093129760000000006,"stat_sum":{"num_bytes":13246714,"num_objects":57,"num_object_clones":4,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":44,"num_read":2952,"num_read_kb":11672,"num_write":356,"num_write_kb":4465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":1191778,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"3.11","version":"99'919","reported_seq":3784,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.696783+0000","last_change":"2024-09-21T15:50:34.697143+0000","last_active":"2024-09-21T15:50:34.696783+0000","last_peered":"2024-09-21T15:50:34.696783+0000","last_clean":"2024-09-21T15:50:34.696783+0000","last_became_active":"2024-09-21T15:49:06.688798+0000","last_became_peered":"2024-09-21T15:49:06.688798+0000","last_unstale":"2024-09-21T15:50:34.696783+0000","last_undegraded":"2024-09-21T15:50:34.696783+0000","last_fullsized":"2024-09-21T15:50:34.696783+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":919,"log_dups_size":0,"ondisk_log_size":919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:48:33.679133+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.008774479,"stat_sum":{"num_bytes":59874290,"num_objects":65,"num_object_clones":9,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":35,"num_read":6119,"num_read_kb":16744,"num_write":1280,"num_write_kb":8388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2b","length":"5"}]},{"pgid":"3.12","version":"97'352","reported_seq":2735,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-09-21T15:50:32.688878+0000","last_change":"2024-09-21T15:50:32.688878+0000","last_active":"2024-09-21T15:50:32.688878+0000","last_peered":"2024-09-21T15:50:32.688878+0000","last_clean":"2024-09-21T15:50:32.688878+0000","last_became_active":"2024-09-21T15:50:32.671686+0000","last_became_peered":"2024-09-21T15:50:32.671686+0000","last_unstale":"2024-09-21T15:50:32.688878+0000","last_undegraded":"2024-09-21T15:50:32.688878+0000","last_fullsized":"2024-09-21T15:50:32.688878+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:40:53.968477+0000","scrub_duration":0,"objects_trimmed":6,"snaptrim_duration":0.015262016999999999,"stat_sum":{"num_bytes":39845746,"num_objects":64,"num_object_clones":8,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":41,"num_read":2441,"num_read_kb":11662,"num_write":386,"num_write_kb":13839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":3359702,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c","length":"4"}]},{"pgid":"3.13","version":"97'882","reported_seq":7104,"reported_epoch":99,"state":"active+clean","last_fresh":"2024-09-21T15:50:34.688516+0000","last_change":"2024-09-21T15:50:34.688748+0000","last_active":"2024-09-21T15:50:34.688516+0000","last_peered":"2024-09-21T15:50:34.688516+0000","last_clean":"2024-09-21T15:50:34.688516+0000","last_became_active":"2024-09-21T15:49:06.690128+0000","last_became_peered":"2024-09-21T15:49:06.690128+0000","last_unstale":"2024-09-21T15:50:34.688516+0000","last_undegraded":"2024-09-21T15:50:34.688516+0000","last_fullsized":"2024-09-21T15:50:34.688516+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":882,"log_dups_size":0,"ondisk_log_size":882,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:11.763959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036025,"stat_sum":{"num_bytes":34582516,"num_objects":82,"num_object_clones":18,"num_object_copies":164,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":48,"num_read":9229,"num_read_kb":26283,"num_write":1392,"num_write_kb":13566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2b","length":"5"}]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1484406535,"num_objects":2236,"num_object_clones":292,"num_object_copies":4472,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2236,"num_whiteouts":1256,"num_read":196839,"num_read_kb":833374,"num_write":21337,"num_write_kb":302722,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":841,"num_bytes_recovered":47753679,"num_keys_recovered":165,"num_objects_omap":44,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":377659392,"data_stored":506957447,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5973,"internal_metadata":0},"log_size":17432,"ondisk_log_size":17432,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574872,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":127844,"kb_used_data":56532,"kb_used_omap":1,"kb_used_meta":71294,"kb_avail":94243996,"statfs":{"total":96636764160,"available":96505851904,"internally_reserved":0,"allocated":57888768,"data_stored":75307892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":73005515},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.34499999999999997}]},{"osd":1,"last update":"Sat Sep 21 15:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.37},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.38}]},{"osd":2,"last update":"Sat Sep 21 15:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.433}]},{"osd":3,"last update":"Sat Sep 21 15:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.36099999999999999}]},{"osd":4,"last update":"Sat Sep 21 15:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.317}]},{"osd":5,"last update":"Sat Sep 21 15:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.40600000000000003}]},{"osd":6,"last update":"Sat Sep 21 15:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.39700000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574875,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59612,"kb_used_data":24392,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94312228,"statfs":{"total":96636764160,"available":96575721472,"internally_reserved":0,"allocated":24977408,"data_stored":37618825,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.42299999999999999}]},{"osd":1,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.38}]},{"osd":2,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.40400000000000003}]},{"osd":3,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.39200000000000002}]},{"osd":4,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.27100000000000002}]},{"osd":5,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.313}]},{"osd":7,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.37},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.41399999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574874,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":112016,"kb_used_data":62848,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94259824,"statfs":{"total":96636764160,"available":96522059776,"internally_reserved":0,"allocated":64356352,"data_stored":86513493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.48499999999999999}]},{"osd":1,"last update":"Sat Sep 21 15:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.52200000000000002}]},{"osd":2,"last update":"Sat Sep 21 15:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.50800000000000001}]},{"osd":3,"last update":"Sat Sep 21 15:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.375},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.6140000000000001,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.54000000000000004}]},{"osd":4,"last update":"Sat Sep 21 15:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":5.6399999999999997,"5min":5.6399999999999997,"15min":5.6399999999999997},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.27300000000000002}]},{"osd":6,"last update":"Sat Sep 21 15:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.55200000000000005}]},{"osd":7,"last update":"Sat Sep 21 15:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.39400000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574873,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":152052,"kb_used_data":80744,"kb_used_omap":1,"kb_used_meta":71294,"kb_avail":94219788,"statfs":{"total":96636764160,"available":96481062912,"internally_reserved":0,"allocated":82681856,"data_stored":113666860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":73005510},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:49:52 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.32700000000000001}]},{"osd":1,"last update":"Sat Sep 21 15:49:52 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.217},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.38300000000000001}]},{"osd":2,"last update":"Sat Sep 21 15:49:52 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.437,"15min":0.437},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.31}]},{"osd":3,"last update":"Sat Sep 21 15:49:52 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.39600000000000002}]},{"osd":5,"last update":"Sat Sep 21 15:49:52 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":2.4489999999999998,"5min":2.4489999999999998,"15min":2.4489999999999998},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.44800000000000001}]},{"osd":6,"last update":"Sat Sep 21 15:49:52 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":2.7709999999999999,"5min":2.7709999999999999,"15min":2.7709999999999999},"last":0.34899999999999998}]},{"osd":7,"last update":"Sat Sep 21 15:49:52 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.8069999999999999,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.374}]}]},{"osd":3,"up_from":13,"seq":55834574874,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":109508,"kb_used_data":60344,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94262332,"statfs":{"total":96636764160,"available":96524627968,"internally_reserved":0,"allocated":61792256,"data_stored":85566797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50330056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.218},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.442,"5min":0.442,"15min":0.442},"last":0.20100000000000001}]},{"osd":1,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.23799999999999999},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.28399999999999997}]},{"osd":2,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.25600000000000001}]},{"osd":4,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.371}]},{"osd":5,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.34200000000000003}]},{"osd":6,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.442}]},{"osd":7,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.38900000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574874,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":38152,"kb_used_data":7036,"kb_used_omap":1,"kb_used_meta":31102,"kb_avail":94333688,"statfs":{"total":96636764160,"available":96597696512,"internally_reserved":0,"allocated":7204864,"data_stored":6531679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31848902},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.25800000000000001},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.23200000000000001}]},{"osd":1,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":1.621,"5min":1.621,"15min":1.621},"last":0.19900000000000001},{"interface":"front","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.27100000000000002}]},{"osd":3,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":11.56,"5min":11.56,"15min":11.56},"last":0.24399999999999999},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.32300000000000001}]},{"osd":4,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.653,"5min":1.653,"15min":1.653},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":2.4660000000000002,"5min":2.4660000000000002,"15min":2.4660000000000002},"last":0.40699999999999997}]},{"osd":5,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.37},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.309}]},{"osd":6,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.436}]},{"osd":7,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.39500000000000002}]}]},{"osd":1,"up_from":13,"seq":55834574873,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":63648,"kb_used_data":28432,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94308192,"statfs":{"total":96636764160,"available":96571588608,"internally_reserved":0,"allocated":29114368,"data_stored":40956490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.315,"15min":0.315},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"last":0.31900000000000001}]},{"osd":2,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":5.0670000000000002,"5min":5.0670000000000002,"15min":5.0670000000000002},"last":0.28999999999999998}]},{"osd":3,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":16.032,"5min":16.032,"15min":16.032},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.309}]},{"osd":4,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2.2269999999999999,"5min":2.2269999999999999,"15min":2.2269999999999999},"last":0.46200000000000002}]},{"osd":5,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":2.2890000000000001,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":4.6989999999999998,"5min":4.6989999999999998,"15min":4.6989999999999998},"last":0.35899999999999999}]},{"osd":6,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":2.262,"5min":2.262,"15min":2.262},"last":0.41099999999999998}]},{"osd":7,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":4.742,"5min":4.742,"15min":4.742},"last":0.374},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.49099999999999999}]}]},{"osd":0,"up_from":93,"seq":399431958530,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":81940,"kb_used_data":49412,"kb_used_omap":13,"kb_used_meta":32498,"kb_avail":94289900,"statfs":{"total":96636764160,"available":96552857600,"internally_reserved":0,"allocated":50597888,"data_stored":66108537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13968,"internal_metadata":33278320},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.23300000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.432}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44800000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53900000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":48852992,"data_stored":64979533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5973,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":27947008,"data_stored":38839043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":15609856,"data_stored":21056570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":60628992,"data_stored":82925081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":81522688,"data_stored":108334153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":63152128,"data_stored":84879777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":23814144,"data_stored":34452821,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":56131584,"data_stored":71490469,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T15:50:38.155 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:38.155 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:38.164 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T15:50:38.207 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:38.206+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:38.265 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:38.266 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:38.269 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:38.268+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:38.274 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:50:38.370 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:38.369+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:38.378 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:38.378 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:38.387 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:50:38.470 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:38.469+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:38.488 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:38.488 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:38.497 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:50:38.571 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:38.570+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:38.599 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:38.599 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:38.607 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:50:38.672 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:38.671+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:38.709 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T15:50:38.717 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:38.717 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:38.726 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:50:38.772 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:38.771+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:38.873 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:38.871+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:38.882 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:38.882 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:38.891 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:50:38.974 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:38.973+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:39.034 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:39.035 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:39.043 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:50:39.087 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:39.086+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:39.102 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:50:39.147 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:39.148 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:39.158 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:50:39.201 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:39.200+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:39.235 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-09-21T15:50:39.235 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:50:39.252 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:50:39.252 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:50:39.267 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:50:39.267 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:50:39.272 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:39.272 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:39.272 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:50:39.272 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:50:39.272 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:39.275 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:39.275+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:39.281 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:50:39.283 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:50:39.284 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:50:39.300 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:50:39.301 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:50:39.317 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:50:39.317 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:50:39.334 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:50:39.334 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:50:39.350 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:50:39.350 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:50:39.389 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:39.389 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:50:39.389 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:50:39.389 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:50:39.389 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.389 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:984 3.0 e27/22 3:025e53e7:::rbd_data.1133f3dfbd6b.0000000000000008:head v 27'21, mlcod=27'19)", 2024-09-21T15:50:39.389 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.619393+0000", 2024-09-21T15:50:39.389 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 86.767768079000007, 2024-09-21T15:50:39.389 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003401837, 2024-09-21T15:50:39.389 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:39.389 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:39.389 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:39.389 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.389 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:39.390 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619393+0000", 2024-09-21T15:50:39.390 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.390 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.390 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.390 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:39.390 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619393+0000", 2024-09-21T15:50:39.390 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.390 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.390 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.390 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:39.390 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619392+0000", 2024-09-21T15:50:39.390 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:39.390 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.390 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.390 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:39.391 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619439+0000", 2024-09-21T15:50:39.391 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7170999999999999e-05 2024-09-21T15:50:39.391 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.391 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.391 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:39.391 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619452+0000", 2024-09-21T15:50:39.391 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2986999999999999e-05 2024-09-21T15:50:39.391 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.391 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.391 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:39.391 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619459+0000", 2024-09-21T15:50:39.391 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6109999999999999e-06 2024-09-21T15:50:39.391 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.391 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.392 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:39.392 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619518+0000", 2024-09-21T15:50:39.392 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9117999999999999e-05 2024-09-21T15:50:39.392 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.392 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.392 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:39.392 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619532+0000", 2024-09-21T15:50:39.392 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4622e-05 2024-09-21T15:50:39.392 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.392 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.392 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:39.392 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.622760+0000", 2024-09-21T15:50:39.392 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032278839999999999 2024-09-21T15:50:39.392 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.392 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.393 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:39.393 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.622794+0000", 2024-09-21T15:50:39.393 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4144999999999999e-05 2024-09-21T15:50:39.393 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.393 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:39.393 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.393 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.393 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.393 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:4957 3.9 e29/22 3:95c5ed95:::rbd_data.11335fa2fe69.0000000000000018:head v 29'4, mlcod=26'2)", 2024-09-21T15:50:39.393 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:14.453951+0000", 2024-09-21T15:50:39.393 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 84.933209646999998, 2024-09-21T15:50:39.393 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0043804559999999996, 2024-09-21T15:50:39.393 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:39.393 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:39.393 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:39.394 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.394 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:39.394 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.453951+0000", 2024-09-21T15:50:39.394 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.394 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.394 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.394 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:39.394 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.453951+0000", 2024-09-21T15:50:39.394 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.394 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.394 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.394 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:39.394 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.453950+0000", 2024-09-21T15:50:39.394 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:39.395 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.395 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.395 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:39.395 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454495+0000", 2024-09-21T15:50:39.395 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00054437799999999998 2024-09-21T15:50:39.395 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.395 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.395 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:39.395 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454508+0000", 2024-09-21T15:50:39.395 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3475000000000001e-05 2024-09-21T15:50:39.395 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.395 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.395 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:39.395 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454515+0000", 2024-09-21T15:50:39.395 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8990000000000004e-06 2024-09-21T15:50:39.396 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.396 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.396 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:39.396 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454581+0000", 2024-09-21T15:50:39.396 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5987000000000004e-05 2024-09-21T15:50:39.396 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.396 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.396 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:39.396 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454609+0000", 2024-09-21T15:50:39.396 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7466000000000001e-05 2024-09-21T15:50:39.396 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.396 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.396 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:39.396 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.458303+0000", 2024-09-21T15:50:39.396 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003693979 2024-09-21T15:50:39.396 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.397 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.397 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:39.397 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.458331+0000", 2024-09-21T15:50:39.397 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8932999999999999e-05 2024-09-21T15:50:39.397 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.397 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:39.397 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.397 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.397 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.397 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13420 3.0 e31/22 3:038b8840:::rbd_data.1133dc5adfdf.0000000000000035:head v 31'49, mlcod=31'47)", 2024-09-21T15:50:39.397 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.209233+0000", 2024-09-21T15:50:39.397 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 82.177928030999993, 2024-09-21T15:50:39.397 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0042624960000000002, 2024-09-21T15:50:39.397 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:39.397 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:39.398 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:39.398 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.398 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:39.398 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209233+0000", 2024-09-21T15:50:39.398 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.398 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.398 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.398 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:39.398 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209233+0000", 2024-09-21T15:50:39.398 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.398 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.398 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.398 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:39.398 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209232+0000", 2024-09-21T15:50:39.399 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:39.399 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.399 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.399 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:39.399 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209251+0000", 2024-09-21T15:50:39.399 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9304e-05 2024-09-21T15:50:39.399 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.399 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.399 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:39.399 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209265+0000", 2024-09-21T15:50:39.399 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3532e-05 2024-09-21T15:50:39.399 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.399 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.399 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:39.399 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209272+0000", 2024-09-21T15:50:39.399 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.289e-06 2024-09-21T15:50:39.400 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.400 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.400 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:39.400 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209321+0000", 2024-09-21T15:50:39.400 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8773999999999998e-05 2024-09-21T15:50:39.400 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.400 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.400 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:39.400 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209341+0000", 2024-09-21T15:50:39.400 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0387999999999999e-05 2024-09-21T15:50:39.400 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.400 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.400 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:39.400 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.213464+0000", 2024-09-21T15:50:39.400 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0041232450000000002 2024-09-21T15:50:39.401 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.401 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.401 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:39.401 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.213495+0000", 2024-09-21T15:50:39.401 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0793999999999997e-05 2024-09-21T15:50:39.401 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.401 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:39.401 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.401 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.401 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.401 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13445 3.0 e31/22 3:038b8840:::rbd_data.1133dc5adfdf.0000000000000035:head v 31'50, mlcod=31'48)", 2024-09-21T15:50:39.401 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.232497+0000", 2024-09-21T15:50:39.401 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 82.154664018999995, 2024-09-21T15:50:39.401 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0040493559999999996, 2024-09-21T15:50:39.401 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:39.401 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:39.402 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:39.402 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.402 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:39.402 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232497+0000", 2024-09-21T15:50:39.402 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.402 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.402 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.402 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:39.402 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232497+0000", 2024-09-21T15:50:39.402 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.402 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.402 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.402 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:39.402 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232496+0000", 2024-09-21T15:50:39.402 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:39.403 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.403 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.403 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:39.403 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232503+0000", 2024-09-21T15:50:39.403 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7320000000000001e-06 2024-09-21T15:50:39.403 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.403 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.403 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:39.403 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232512+0000", 2024-09-21T15:50:39.403 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9259999999999995e-06 2024-09-21T15:50:39.403 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.403 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.403 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:39.403 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232520+0000", 2024-09-21T15:50:39.403 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3699999999999997e-06 2024-09-21T15:50:39.404 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.404 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.404 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:39.404 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232564+0000", 2024-09-21T15:50:39.404 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4128999999999997e-05 2024-09-21T15:50:39.404 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.404 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.404 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:39.404 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232585+0000", 2024-09-21T15:50:39.404 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0704999999999999e-05 2024-09-21T15:50:39.404 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.404 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.404 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:39.404 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.236507+0000", 2024-09-21T15:50:39.404 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0039218730000000002 2024-09-21T15:50:39.404 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.405 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.405 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:39.405 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.236546+0000", 2024-09-21T15:50:39.405 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9399e-05 2024-09-21T15:50:39.405 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.405 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:39.405 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.405 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.405 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.405 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13452 3.9 e31/22 3:93f5396a:::rbd_data.1133dc5adfdf.0000000000000078:head v 31'50, mlcod=31'47)", 2024-09-21T15:50:39.405 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.237263+0000", 2024-09-21T15:50:39.405 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 82.149897312999997, 2024-09-21T15:50:39.405 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033768180000000002, 2024-09-21T15:50:39.405 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:39.405 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:39.406 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:39.406 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.406 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:39.406 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237263+0000", 2024-09-21T15:50:39.406 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.406 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.406 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.406 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:39.406 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237263+0000", 2024-09-21T15:50:39.406 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.406 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.406 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.406 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:39.406 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237263+0000", 2024-09-21T15:50:39.407 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:39.407 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.407 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.407 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:39.407 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237269+0000", 2024-09-21T15:50:39.407 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.472e-06 2024-09-21T15:50:39.407 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.407 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.407 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:39.407 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237282+0000", 2024-09-21T15:50:39.407 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2788e-05 2024-09-21T15:50:39.407 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.407 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.407 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:39.407 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237288+0000", 2024-09-21T15:50:39.409 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4899999999999997e-06 2024-09-21T15:50:39.409 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.409 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.409 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:39.410 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237331+0000", 2024-09-21T15:50:39.410 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2425000000000001e-05 2024-09-21T15:50:39.410 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.410 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.410 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:39.410 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.237349+0000", 2024-09-21T15:50:39.410 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8405000000000001e-05 2024-09-21T15:50:39.410 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.410 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.410 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:39.410 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.240610+0000", 2024-09-21T15:50:39.410 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032605379999999999 2024-09-21T15:50:39.410 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.410 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.411 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:39.411 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.240640+0000", 2024-09-21T15:50:39.411 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0448000000000001e-05 2024-09-21T15:50:39.411 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.411 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:39.411 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.411 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.411 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.411 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13832 3.0 e31/22 3:0792efd2:::rbd_data.1133dc5adfdf.0000000000000090:head v 31'58, mlcod=31'56)", 2024-09-21T15:50:39.411 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.527530+0000", 2024-09-21T15:50:39.411 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 81.859630492999997, 2024-09-21T15:50:39.411 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035256530000000001, 2024-09-21T15:50:39.411 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:39.412 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:39.412 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:39.412 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.412 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:39.412 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527530+0000", 2024-09-21T15:50:39.412 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.412 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.412 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.412 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:39.412 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527530+0000", 2024-09-21T15:50:39.412 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.412 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.412 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.412 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:39.412 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527529+0000", 2024-09-21T15:50:39.413 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:50:39.413 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.413 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.413 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:39.413 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527550+0000", 2024-09-21T15:50:39.413 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1387999999999999e-05 2024-09-21T15:50:39.413 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.413 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.413 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:39.413 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527568+0000", 2024-09-21T15:50:39.413 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8068e-05 2024-09-21T15:50:39.413 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.413 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.413 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:39.414 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527577+0000", 2024-09-21T15:50:39.414 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9919999999999996e-06 2024-09-21T15:50:39.414 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.414 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.414 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:39.414 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527626+0000", 2024-09-21T15:50:39.414 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8597000000000002e-05 2024-09-21T15:50:39.414 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.414 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.414 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:39.414 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527652+0000", 2024-09-21T15:50:39.414 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5993e-05 2024-09-21T15:50:39.414 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.414 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.415 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:39.415 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.531024+0000", 2024-09-21T15:50:39.415 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033721910000000001 2024-09-21T15:50:39.415 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.415 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.415 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:39.415 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.531056+0000", 2024-09-21T15:50:39.415 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1628000000000003e-05 2024-09-21T15:50:39.415 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.415 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:39.415 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.415 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.415 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.415 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13839 3.0 e31/22 3:03c4fb32:::rbd_data.1133dc5adfdf.0000000000000099:head v 31'59, mlcod=31'56)", 2024-09-21T15:50:39.415 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.528097+0000", 2024-09-21T15:50:39.415 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 81.859063895000006, 2024-09-21T15:50:39.416 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0057228840000000001, 2024-09-21T15:50:39.416 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:39.416 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:39.416 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:39.416 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.416 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:39.416 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528097+0000", 2024-09-21T15:50:39.416 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.416 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.416 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.416 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:39.416 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528097+0000", 2024-09-21T15:50:39.416 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.416 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.417 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.417 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:39.417 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528096+0000", 2024-09-21T15:50:39.417 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:39.417 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.417 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.417 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:39.417 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528105+0000", 2024-09-21T15:50:39.417 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7350000000000002e-06 2024-09-21T15:50:39.417 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.417 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.417 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:39.417 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528117+0000", 2024-09-21T15:50:39.417 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1909000000000001e-05 2024-09-21T15:50:39.418 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.418 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.418 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:39.418 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528124+0000", 2024-09-21T15:50:39.418 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8419999999999999e-06 2024-09-21T15:50:39.418 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.418 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.418 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:39.418 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528168+0000", 2024-09-21T15:50:39.418 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4301999999999997e-05 2024-09-21T15:50:39.418 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.418 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.418 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:39.418 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528191+0000", 2024-09-21T15:50:39.419 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2622000000000001e-05 2024-09-21T15:50:39.419 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.419 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.419 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:39.419 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.533780+0000", 2024-09-21T15:50:39.419 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0055897519999999999 2024-09-21T15:50:39.419 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.419 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.419 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:39.419 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.533820+0000", 2024-09-21T15:50:39.419 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9366999999999998e-05 2024-09-21T15:50:39.419 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.419 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:39.419 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.420 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.420 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.420 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13884 3.9 e31/22 3:93003736:::rbd_data.1133dc5adfdf.00000000000000a1:head v 31'54, mlcod=31'52)", 2024-09-21T15:50:39.420 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.562433+0000", 2024-09-21T15:50:39.420 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 81.824727147000004, 2024-09-21T15:50:39.420 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.005162313, 2024-09-21T15:50:39.420 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:39.420 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:39.420 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:39.420 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.420 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:39.420 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562433+0000", 2024-09-21T15:50:39.420 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.420 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.420 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.421 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:39.421 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562433+0000", 2024-09-21T15:50:39.421 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.421 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.421 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.421 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:39.421 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562432+0000", 2024-09-21T15:50:39.421 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:39.421 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.421 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.421 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:39.421 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562467+0000", 2024-09-21T15:50:39.421 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4885999999999998e-05 2024-09-21T15:50:39.421 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.421 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.422 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:39.422 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562481+0000", 2024-09-21T15:50:39.422 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3604000000000001e-05 2024-09-21T15:50:39.422 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.422 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.422 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:39.422 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562489+0000", 2024-09-21T15:50:39.422 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1829999999999994e-06 2024-09-21T15:50:39.422 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.422 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.422 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:39.422 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562536+0000", 2024-09-21T15:50:39.422 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6830999999999999e-05 2024-09-21T15:50:39.423 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.423 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.423 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:39.423 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562559+0000", 2024-09-21T15:50:39.423 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2765999999999999e-05 2024-09-21T15:50:39.423 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.423 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.423 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:39.423 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.567566+0000", 2024-09-21T15:50:39.423 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0050073219999999998 2024-09-21T15:50:39.423 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.423 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.423 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:39.423 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.567596+0000", 2024-09-21T15:50:39.424 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9802999999999999e-05 2024-09-21T15:50:39.424 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.424 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:39.424 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.424 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.424 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.424 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:22340 3.9 e34/22 3:910a5abe:::rbd_data.1133896adcfd.0000000000000023:head v 34'92, mlcod=34'90)", 2024-09-21T15:50:39.424 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:21.628025+0000", 2024-09-21T15:50:39.424 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 77.759135767000004, 2024-09-21T15:50:39.424 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046196010000000001, 2024-09-21T15:50:39.424 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:39.424 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:39.424 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:39.424 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.424 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:39.425 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628025+0000", 2024-09-21T15:50:39.425 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.425 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.425 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.425 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:39.425 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628025+0000", 2024-09-21T15:50:39.425 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.425 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.425 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.425 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:39.425 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628024+0000", 2024-09-21T15:50:39.425 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:39.425 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.425 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.425 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:39.426 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628034+0000", 2024-09-21T15:50:39.426 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3470000000000005e-06 2024-09-21T15:50:39.426 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.426 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.426 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:39.426 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628043+0000", 2024-09-21T15:50:39.426 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9450000000000005e-06 2024-09-21T15:50:39.426 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.426 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.426 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:39.426 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628048+0000", 2024-09-21T15:50:39.426 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8080000000000004e-06 2024-09-21T15:50:39.427 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.427 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.427 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:39.427 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628081+0000", 2024-09-21T15:50:39.427 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2738999999999999e-05 2024-09-21T15:50:39.427 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.427 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.427 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:39.427 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628094+0000", 2024-09-21T15:50:39.427 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3273e-05 2024-09-21T15:50:39.427 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.427 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.427 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:39.427 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.632616+0000", 2024-09-21T15:50:39.427 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045217269999999997 2024-09-21T15:50:39.428 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.428 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.428 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:39.428 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.632644+0000", 2024-09-21T15:50:39.428 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8469000000000001e-05 2024-09-21T15:50:39.428 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.428 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:39.428 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.428 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.428 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.428 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:22417 3.4 e34/22 3:25310a10:::rbd_data.1133896adcfd.0000000000000075:head v 34'48, mlcod=34'46)", 2024-09-21T15:50:39.428 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:21.704297+0000", 2024-09-21T15:50:39.428 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 77.682863669, 2024-09-21T15:50:39.428 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0038101799999999998, 2024-09-21T15:50:39.428 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:39.429 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:39.429 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:39.429 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.429 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:39.429 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704297+0000", 2024-09-21T15:50:39.429 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.429 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.429 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.429 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:39.429 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704297+0000", 2024-09-21T15:50:39.429 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.429 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.429 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.429 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:39.429 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704296+0000", 2024-09-21T15:50:39.429 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:50:39.430 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.430 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.430 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:39.430 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704308+0000", 2024-09-21T15:50:39.430 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2559e-05 2024-09-21T15:50:39.430 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.430 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.430 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:39.430 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704337+0000", 2024-09-21T15:50:39.430 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8960000000000001e-05 2024-09-21T15:50:39.430 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.430 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.430 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:39.431 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704345+0000", 2024-09-21T15:50:39.431 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3059999999999998e-06 2024-09-21T15:50:39.431 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.431 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.431 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:39.431 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704408+0000", 2024-09-21T15:50:39.431 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2156000000000006e-05 2024-09-21T15:50:39.431 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.431 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.431 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:39.431 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704430+0000", 2024-09-21T15:50:39.431 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1931000000000001e-05 2024-09-21T15:50:39.431 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.431 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.432 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:39.432 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.708073+0000", 2024-09-21T15:50:39.432 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003643028 2024-09-21T15:50:39.432 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.432 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.432 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:39.432 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.708107+0000", 2024-09-21T15:50:39.432 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4613999999999998e-05 2024-09-21T15:50:39.432 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.432 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:39.432 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.432 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.432 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.432 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:25278 3.4 e38/22 3:2251cc1c:::rbd_data.1133ebd6d3ba.0000000000000024:head v 38'51, mlcod=38'49)", 2024-09-21T15:50:39.432 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.317337+0000", 2024-09-21T15:50:39.433 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 73.069823908999993, 2024-09-21T15:50:39.433 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053904110000000003, 2024-09-21T15:50:39.433 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:39.433 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:39.433 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:39.433 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.433 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:39.433 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317337+0000", 2024-09-21T15:50:39.433 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.433 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.433 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.433 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:39.433 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317337+0000", 2024-09-21T15:50:39.433 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.434 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.434 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.434 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:39.434 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317335+0000", 2024-09-21T15:50:39.434 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:50:39.434 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.434 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.434 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:39.434 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317347+0000", 2024-09-21T15:50:39.434 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.19e-05 2024-09-21T15:50:39.434 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.434 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.434 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:39.434 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317369+0000", 2024-09-21T15:50:39.435 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2232e-05 2024-09-21T15:50:39.435 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.435 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.435 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:39.435 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317381+0000", 2024-09-21T15:50:39.435 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2231e-05 2024-09-21T15:50:39.435 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.435 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.435 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:39.435 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317425+0000", 2024-09-21T15:50:39.435 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3625000000000003e-05 2024-09-21T15:50:39.435 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.435 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.435 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:39.435 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317449+0000", 2024-09-21T15:50:39.436 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4301999999999999e-05 2024-09-21T15:50:39.436 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.436 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.436 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:39.436 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.322688+0000", 2024-09-21T15:50:39.436 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.005238609 2024-09-21T15:50:39.436 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.436 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.436 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:39.436 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.322727+0000", 2024-09-21T15:50:39.436 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9201e-05 2024-09-21T15:50:39.436 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.436 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:39.436 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.436 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.437 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.437 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:25286 3.0 e38/22 3:0248fb60:::rbd_data.1133ebd6d3ba.0000000000000066:head v 38'109, mlcod=34'107)", 2024-09-21T15:50:39.437 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.317638+0000", 2024-09-21T15:50:39.437 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 73.069523029999999, 2024-09-21T15:50:39.437 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0054074370000000002, 2024-09-21T15:50:39.437 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:39.437 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:39.437 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:39.437 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.437 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:39.437 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317638+0000", 2024-09-21T15:50:39.437 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.437 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.437 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.437 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:39.438 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317638+0000", 2024-09-21T15:50:39.438 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.438 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.438 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.438 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:39.438 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317637+0000", 2024-09-21T15:50:39.438 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:39.438 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.438 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.438 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:39.438 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317643+0000", 2024-09-21T15:50:39.438 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5609999999999998e-06 2024-09-21T15:50:39.438 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.438 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.439 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:39.439 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317653+0000", 2024-09-21T15:50:39.439 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0705e-05 2024-09-21T15:50:39.439 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.439 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.439 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:39.439 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317660+0000", 2024-09-21T15:50:39.439 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2099999999999998e-06 2024-09-21T15:50:39.439 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.439 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.439 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:39.439 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317704+0000", 2024-09-21T15:50:39.439 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4659000000000001e-05 2024-09-21T15:50:39.439 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.440 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.440 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:39.440 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317725+0000", 2024-09-21T15:50:39.440 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0537e-05 2024-09-21T15:50:39.440 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.440 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.440 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:39.440 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323010+0000", 2024-09-21T15:50:39.440 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052853420000000002 2024-09-21T15:50:39.440 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.440 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.440 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:39.440 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323045+0000", 2024-09-21T15:50:39.440 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4860000000000002e-05 2024-09-21T15:50:39.440 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.441 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:39.441 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.441 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.441 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.441 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:25280 3.4 e38/22 3:27fca38c:::rbd_data.1133ebd6d3ba.0000000000000026:head v 38'52, mlcod=38'49)", 2024-09-21T15:50:39.441 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.317684+0000", 2024-09-21T15:50:39.441 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 73.069477133999996, 2024-09-21T15:50:39.441 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053570809999999996, 2024-09-21T15:50:39.441 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:39.441 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:39.441 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:39.441 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.441 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:39.441 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317684+0000", 2024-09-21T15:50:39.442 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.442 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.442 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.442 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:39.442 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317684+0000", 2024-09-21T15:50:39.442 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.442 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.442 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.442 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:39.442 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317683+0000", 2024-09-21T15:50:39.442 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:39.442 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.442 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.442 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:39.442 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317688+0000", 2024-09-21T15:50:39.443 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5009999999999998e-06 2024-09-21T15:50:39.443 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.443 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.443 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:39.443 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317694+0000", 2024-09-21T15:50:39.443 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3579999999999999e-06 2024-09-21T15:50:39.443 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.443 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.443 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:39.443 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317698+0000", 2024-09-21T15:50:39.443 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4150000000000001e-06 2024-09-21T15:50:39.443 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.443 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.443 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:39.444 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317738+0000", 2024-09-21T15:50:39.444 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9575e-05 2024-09-21T15:50:39.444 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.444 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.444 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:39.444 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317762+0000", 2024-09-21T15:50:39.444 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4173999999999999e-05 2024-09-21T15:50:39.444 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.444 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.444 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:39.444 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323005+0000", 2024-09-21T15:50:39.444 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00524333 2024-09-21T15:50:39.444 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.444 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.445 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:39.445 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323041+0000", 2024-09-21T15:50:39.445 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5139999999999999e-05 2024-09-21T15:50:39.445 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.445 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:39.445 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.445 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.445 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.445 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:34837 3.0 e43/22 3:038527c8:::rbd_header.1133fed1a840:head v 43'130, mlcod=43'128)", 2024-09-21T15:50:39.445 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:50:39.445 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 68.403272435999995, 2024-09-21T15:50:39.445 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045776519999999998, 2024-09-21T15:50:39.445 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:39.445 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:39.445 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:39.445 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.446 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:39.446 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:50:39.446 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.446 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.446 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.446 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:39.446 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:50:39.446 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.446 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.446 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.446 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:39.446 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:50:39.446 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:39.447 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.447 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.447 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:39.447 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983892+0000", 2024-09-21T15:50:39.447 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8450000000000002e-06 2024-09-21T15:50:39.447 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.447 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.447 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:39.447 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983902+0000", 2024-09-21T15:50:39.447 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0299999999999999e-06 2024-09-21T15:50:39.447 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.447 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.447 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:39.447 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983906+0000", 2024-09-21T15:50:39.447 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2300000000000002e-06 2024-09-21T15:50:39.448 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.448 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.448 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:39.448 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983940+0000", 2024-09-21T15:50:39.448 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4563000000000001e-05 2024-09-21T15:50:39.448 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.448 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.448 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:39.448 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983960+0000", 2024-09-21T15:50:39.448 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9199999999999999e-05 2024-09-21T15:50:39.448 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.448 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.448 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:39.448 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.988430+0000", 2024-09-21T15:50:39.449 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0044709149999999998 2024-09-21T15:50:39.449 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.449 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.449 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:39.449 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.988466+0000", 2024-09-21T15:50:39.449 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5420000000000003e-05 2024-09-21T15:50:39.449 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.449 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:39.449 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.449 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.449 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.449 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "pg_update_log_missing(3.0 epoch 50/22 rep_tid 498 entries 50'171 (0'0) error 3:5df4a5a0:::rbd_data.11339e87a32a.000000000000008e:head by client.4403.0:80207 0.000000 -2 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 47'167)", 2024-09-21T15:50:39.449 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:50:39.449 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 58.445399563000002, 2024-09-21T15:50:39.450 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0047253119999999997, 2024-09-21T15:50:39.450 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:39.450 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-09-21T15:50:39.450 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:39.450 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.450 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:39.450 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:50:39.450 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.450 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.450 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.450 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:39.450 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:50:39.450 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.450 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.450 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.451 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:39.451 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:50:39.451 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:39.451 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.451 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.451 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:39.451 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941768+0000", 2024-09-21T15:50:39.451 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7309999999999993e-06 2024-09-21T15:50:39.451 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.451 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.451 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:39.451 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941780+0000", 2024-09-21T15:50:39.451 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1399999999999999e-05 2024-09-21T15:50:39.451 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.451 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.452 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:39.452 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941789+0000", 2024-09-21T15:50:39.452 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5230000000000001e-06 2024-09-21T15:50:39.452 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.452 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.452 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:39.452 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941841+0000", 2024-09-21T15:50:39.452 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1863999999999997e-05 2024-09-21T15:50:39.452 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.452 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.452 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:39.452 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.946486+0000", 2024-09-21T15:50:39.452 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046452539999999997 2024-09-21T15:50:39.452 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.453 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:39.453 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.453 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.453 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.453 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "pg_update_log_missing(3.0 epoch 53/22 rep_tid 585 entries 53'201 (0'0) error 3:6a1d4890:::rbd_data.1133e079cae4.0000000000000004:head by client.4403.0:101089 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 53'198)", 2024-09-21T15:50:39.453 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:44.832827+0000", 2024-09-21T15:50:39.453 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 54.554333755000002, 2024-09-21T15:50:39.453 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053552929999999997, 2024-09-21T15:50:39.453 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:39.453 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-09-21T15:50:39.453 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:39.453 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.453 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:39.453 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832827+0000", 2024-09-21T15:50:39.454 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.454 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.454 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.454 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:39.454 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832827+0000", 2024-09-21T15:50:39.454 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.454 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.454 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.454 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:39.454 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832826+0000", 2024-09-21T15:50:39.454 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:39.454 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.454 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.454 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:39.455 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832835+0000", 2024-09-21T15:50:39.455 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4509999999999992e-06 2024-09-21T15:50:39.455 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.455 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.455 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:39.455 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832854+0000", 2024-09-21T15:50:39.455 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9352999999999999e-05 2024-09-21T15:50:39.455 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.455 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.455 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:39.455 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832866+0000", 2024-09-21T15:50:39.455 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1766000000000001e-05 2024-09-21T15:50:39.455 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.455 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.455 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:39.456 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.833091+0000", 2024-09-21T15:50:39.456 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00022490300000000001 2024-09-21T15:50:39.456 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.456 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.456 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:39.456 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.838182+0000", 2024-09-21T15:50:39.456 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0050915420000000001 2024-09-21T15:50:39.456 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.456 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:39.456 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.456 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.456 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.456 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:102822 3.4 e54/22 3:2729ba03:::rbd_object_map.113336ae1075:head v 54'208, mlcod=54'205)", 2024-09-21T15:50:39.456 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:46.249540+0000", 2024-09-21T15:50:39.456 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 53.137620523000002, 2024-09-21T15:50:39.457 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052654859999999998, 2024-09-21T15:50:39.457 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:39.457 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:39.457 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:39.457 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.457 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:39.457 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249540+0000", 2024-09-21T15:50:39.457 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.457 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.457 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.457 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:39.457 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249540+0000", 2024-09-21T15:50:39.457 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.457 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.457 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.458 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:39.458 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249539+0000", 2024-09-21T15:50:39.458 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:39.458 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.458 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.458 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:39.458 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249546+0000", 2024-09-21T15:50:39.458 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7290000000000003e-06 2024-09-21T15:50:39.458 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.458 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.458 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:39.458 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249558+0000", 2024-09-21T15:50:39.458 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1844e-05 2024-09-21T15:50:39.458 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.459 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.459 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:39.459 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249565+0000", 2024-09-21T15:50:39.459 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8210000000000004e-06 2024-09-21T15:50:39.459 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.459 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.459 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:39.459 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249602+0000", 2024-09-21T15:50:39.459 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7057000000000001e-05 2024-09-21T15:50:39.459 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.459 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.459 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:39.459 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249622+0000", 2024-09-21T15:50:39.459 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9678999999999999e-05 2024-09-21T15:50:39.459 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.460 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.460 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:39.460 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254783+0000", 2024-09-21T15:50:39.460 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0051611449999999998 2024-09-21T15:50:39.460 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.460 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.460 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:39.460 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254806+0000", 2024-09-21T15:50:39.460 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2853e-05 2024-09-21T15:50:39.460 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.460 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:39.460 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.460 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.460 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.461 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:102823 3.4 e54/22 3:2729ba03:::rbd_object_map.113336ae1075:head v 54'209, mlcod=54'205)", 2024-09-21T15:50:39.461 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:46.250318+0000", 2024-09-21T15:50:39.461 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 53.136842938000001, 2024-09-21T15:50:39.461 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045013539999999999, 2024-09-21T15:50:39.461 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:39.461 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:39.461 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:39.461 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.461 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:39.461 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250318+0000", 2024-09-21T15:50:39.461 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.461 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.461 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.461 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:39.461 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250318+0000", 2024-09-21T15:50:39.462 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.462 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.462 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.462 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:39.462 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250317+0000", 2024-09-21T15:50:39.462 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:39.462 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.462 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.462 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:39.462 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250322+0000", 2024-09-21T15:50:39.462 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.074e-06 2024-09-21T15:50:39.462 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.462 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.462 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:39.463 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250330+0000", 2024-09-21T15:50:39.463 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4730000000000003e-06 2024-09-21T15:50:39.463 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.463 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.463 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:39.463 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250335+0000", 2024-09-21T15:50:39.463 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8019999999999997e-06 2024-09-21T15:50:39.463 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.463 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.463 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:39.463 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250375+0000", 2024-09-21T15:50:39.463 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9459999999999998e-05 2024-09-21T15:50:39.463 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.463 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.464 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:39.464 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250388+0000", 2024-09-21T15:50:39.464 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3288999999999999e-05 2024-09-21T15:50:39.464 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.464 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.464 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:39.464 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254807+0000", 2024-09-21T15:50:39.464 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0044192220000000004 2024-09-21T15:50:39.464 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.464 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.464 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:39.464 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254819+0000", 2024-09-21T15:50:39.464 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1603999999999999e-05 2024-09-21T15:50:39.464 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.464 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:39.465 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.465 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.465 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.465 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:160603 3.9 e95/22 3:9171211b:::rbd_data.11338bae974a.0000000000000006:head v 95'343, mlcod=92'340)", 2024-09-21T15:50:39.465 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.215621+0000", 2024-09-21T15:50:39.465 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.1715399069999997, 2024-09-21T15:50:39.465 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0041276250000000002, 2024-09-21T15:50:39.465 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:39.465 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:39.465 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:39.465 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.465 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:39.465 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215621+0000", 2024-09-21T15:50:39.465 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.466 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.466 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.466 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:39.466 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215621+0000", 2024-09-21T15:50:39.466 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.466 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.466 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.466 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:39.466 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215620+0000", 2024-09-21T15:50:39.466 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:39.466 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.466 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.466 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:39.466 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215627+0000", 2024-09-21T15:50:39.467 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8850000000000001e-06 2024-09-21T15:50:39.467 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.467 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.467 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:39.467 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215803+0000", 2024-09-21T15:50:39.467 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000176422 2024-09-21T15:50:39.467 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.467 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.467 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:39.467 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215814+0000", 2024-09-21T15:50:39.467 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0135e-05 2024-09-21T15:50:39.467 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.467 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.467 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:39.467 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215866+0000", 2024-09-21T15:50:39.468 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2327999999999998e-05 2024-09-21T15:50:39.468 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.468 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.468 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:39.468 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215887+0000", 2024-09-21T15:50:39.468 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1449000000000001e-05 2024-09-21T15:50:39.468 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.468 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.468 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:39.468 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.219719+0000", 2024-09-21T15:50:39.468 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0038312630000000001 2024-09-21T15:50:39.468 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.468 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.468 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:39.468 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.219748+0000", 2024-09-21T15:50:39.469 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9737999999999999e-05 2024-09-21T15:50:39.469 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.469 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:39.469 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.469 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.469 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.469 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:160974 3.4 e100/99 3:22dd2475:::rbd_id.image_client.0-clone39:head v 100'593, mlcod=0'0)", 2024-09-21T15:50:39.469 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:35.683900+0000", 2024-09-21T15:50:39.469 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.7032604730000003, 2024-09-21T15:50:39.469 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0034160029999999999, 2024-09-21T15:50:39.469 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:39.469 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:39.469 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:39.469 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.469 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:39.470 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.683900+0000", 2024-09-21T15:50:39.470 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.470 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.470 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.470 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:39.470 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.683900+0000", 2024-09-21T15:50:39.470 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.470 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.470 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.470 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:39.470 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.683900+0000", 2024-09-21T15:50:39.470 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:39.470 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.470 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.471 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:39.471 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.683906+0000", 2024-09-21T15:50:39.471 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1759999999999998e-06 2024-09-21T15:50:39.471 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.471 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.471 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:39.471 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.683915+0000", 2024-09-21T15:50:39.471 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4320000000000005e-06 2024-09-21T15:50:39.471 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.471 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.471 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:39.471 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.683923+0000", 2024-09-21T15:50:39.471 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.259e-06 2024-09-21T15:50:39.471 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.472 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.472 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:39.472 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.683961+0000", 2024-09-21T15:50:39.472 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7014000000000002e-05 2024-09-21T15:50:39.472 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.472 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.472 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:39.472 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.683982+0000", 2024-09-21T15:50:39.472 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.145e-05 2024-09-21T15:50:39.472 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.472 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.472 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:39.472 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.687265+0000", 2024-09-21T15:50:39.472 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003283252 2024-09-21T15:50:39.472 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.473 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.473 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:39.473 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.687316+0000", 2024-09-21T15:50:39.473 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0963e-05 2024-09-21T15:50:39.473 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.473 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:39.473 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.473 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.473 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:39.473 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:39.473 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:39.387+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:39.474 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:50:39.515 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:39.514+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:39.578 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:39.577+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:39.616 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:39.616 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:39.616 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:50:39.616 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:39.625 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:50:39.678 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:39.677+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:39.700 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is unset 2024-09-21T15:50:39.736 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:39.736 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:39.736 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:50:39.737 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:50:39.737 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:39.746 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:50:39.779 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:39.778+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:39.880 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:39.878+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:39.891 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:39.891 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:50:39.891 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:50:39.891 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:50:39.892 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.892 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:14051 3.8 3:13437810:::rbd_data.1133dc5adfdf.0000000000000039:head [write 716800~796] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-21T15:50:39.892 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:50:39.892 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 82.194561258999997, 2024-09-21T15:50:39.892 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.98064002699999997, 2024-09-21T15:50:39.892 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:39.892 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:39.892 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:39.892 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:39.892 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:39.892 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 14051 2024-09-21T15:50:39.893 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.893 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:39.893 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.893 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:39.893 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:50:39.893 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.893 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.893 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.893 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:39.893 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:50:39.893 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.893 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.893 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.893 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:39.893 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:50:39.894 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:39.894 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.894 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.894 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:39.894 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695131+0000", 2024-09-21T15:50:39.894 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4680000000000003e-06 2024-09-21T15:50:39.894 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.894 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.894 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:39.894 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695139+0000", 2024-09-21T15:50:39.894 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9400000000000002e-06 2024-09-21T15:50:39.894 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.894 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.895 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:39.895 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695143+0000", 2024-09-21T15:50:39.895 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9899999999999999e-06 2024-09-21T15:50:39.895 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.895 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.895 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:39.895 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695190+0000", 2024-09-21T15:50:39.895 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6458000000000001e-05 2024-09-21T15:50:39.895 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.895 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.895 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:50:39.895 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695201+0000", 2024-09-21T15:50:39.895 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0964999999999999e-05 2024-09-21T15:50:39.895 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.895 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.896 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:39.896 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.669490+0000", 2024-09-21T15:50:39.896 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.97428980499999995 2024-09-21T15:50:39.896 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.896 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.896 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-09-21T15:50:39.896 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.670033+0000", 2024-09-21T15:50:39.896 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00054252599999999999 2024-09-21T15:50:39.896 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.896 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.896 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:39.896 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.671095+0000", 2024-09-21T15:50:39.896 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001062183 2024-09-21T15:50:39.896 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.896 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.897 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:39.897 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.671186+0000", 2024-09-21T15:50:39.897 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1191000000000002e-05 2024-09-21T15:50:39.897 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.897 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.897 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-09-21T15:50:39.897 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.671346+0000", 2024-09-21T15:50:39.897 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015980399999999999 2024-09-21T15:50:39.897 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.897 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.897 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:39.897 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.672221+0000", 2024-09-21T15:50:39.897 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00087471999999999995 2024-09-21T15:50:39.897 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.897 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.897 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:39.898 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.675619+0000", 2024-09-21T15:50:39.898 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00339858 2024-09-21T15:50:39.898 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.898 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.898 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:39.898 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.675670+0000", 2024-09-21T15:50:39.898 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0664000000000001e-05 2024-09-21T15:50:39.898 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.898 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.899 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:39.899 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.675768+0000", 2024-09-21T15:50:39.899 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7415000000000002e-05 2024-09-21T15:50:39.899 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.899 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:39.899 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.899 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.900 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.900 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116235 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 761856~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:39.900 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048806+0000", 2024-09-21T15:50:39.900 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 43.840882375, 2024-09-21T15:50:39.900 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056818556999999999, 2024-09-21T15:50:39.900 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:39.900 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:39.900 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:39.901 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:39.901 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:39.901 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116235 2024-09-21T15:50:39.901 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.901 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:39.901 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.901 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:39.901 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048806+0000", 2024-09-21T15:50:39.901 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.902 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.902 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.902 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:39.902 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048806+0000", 2024-09-21T15:50:39.902 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.902 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.902 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.902 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:39.902 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048806+0000", 2024-09-21T15:50:39.903 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:39.903 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.903 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.903 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:39.903 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048807+0000", 2024-09-21T15:50:39.903 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.001e-06 2024-09-21T15:50:39.903 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.904 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.904 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:39.904 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048810+0000", 2024-09-21T15:50:39.904 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.711e-06 2024-09-21T15:50:39.904 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.904 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.904 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:39.904 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048812+0000", 2024-09-21T15:50:39.905 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2290000000000002e-06 2024-09-21T15:50:39.905 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.905 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.905 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:39.905 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105495+0000", 2024-09-21T15:50:39.905 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056682679999999999 2024-09-21T15:50:39.905 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.905 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.905 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:39.905 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105565+0000", 2024-09-21T15:50:39.905 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9868000000000003e-05 2024-09-21T15:50:39.905 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.905 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.905 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:39.906 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105625+0000", 2024-09-21T15:50:39.906 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0268e-05 2024-09-21T15:50:39.906 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.906 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:39.906 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.906 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.906 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.906 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116236 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 768000~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:39.906 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048814+0000", 2024-09-21T15:50:39.906 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 43.840874712000002, 2024-09-21T15:50:39.906 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056959587999999998, 2024-09-21T15:50:39.906 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:39.906 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:39.907 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:39.907 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:39.907 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:39.907 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116236 2024-09-21T15:50:39.907 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.907 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:39.907 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.907 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:39.907 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048814+0000", 2024-09-21T15:50:39.907 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.907 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.907 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.907 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:39.907 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048814+0000", 2024-09-21T15:50:39.907 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.907 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.907 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.908 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:39.908 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048814+0000", 2024-09-21T15:50:39.908 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:39.908 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.908 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.908 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:39.908 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048815+0000", 2024-09-21T15:50:39.908 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7999999999999993e-07 2024-09-21T15:50:39.908 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.908 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.908 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:39.908 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048817+0000", 2024-09-21T15:50:39.908 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6620000000000001e-06 2024-09-21T15:50:39.908 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.908 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.908 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:39.909 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048820+0000", 2024-09-21T15:50:39.909 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2800000000000002e-06 2024-09-21T15:50:39.909 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.909 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.909 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:39.909 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105647+0000", 2024-09-21T15:50:39.909 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056827253000000001 2024-09-21T15:50:39.909 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.909 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.909 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:39.909 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105716+0000", 2024-09-21T15:50:39.909 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8761000000000001e-05 2024-09-21T15:50:39.909 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.909 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.909 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:39.909 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105774+0000", 2024-09-21T15:50:39.909 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7855000000000001e-05 2024-09-21T15:50:39.910 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.910 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:39.910 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.910 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.910 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.910 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116237 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 774144~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:39.910 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048822+0000", 2024-09-21T15:50:39.910 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 43.840867027000002, 2024-09-21T15:50:39.910 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057114028999999997, 2024-09-21T15:50:39.910 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:39.910 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:39.910 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:39.910 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:39.910 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:39.911 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116237 2024-09-21T15:50:39.911 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.911 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:39.911 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.911 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:39.911 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048822+0000", 2024-09-21T15:50:39.911 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.911 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.911 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.911 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:39.911 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048822+0000", 2024-09-21T15:50:39.911 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.911 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.911 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.911 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:39.911 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048822+0000", 2024-09-21T15:50:39.911 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:39.912 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.912 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.912 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:39.912 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048823+0000", 2024-09-21T15:50:39.912 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9699999999999994e-07 2024-09-21T15:50:39.912 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.912 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.912 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:39.912 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048825+0000", 2024-09-21T15:50:39.912 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.65e-06 2024-09-21T15:50:39.912 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.912 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.912 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:39.912 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048828+0000", 2024-09-21T15:50:39.912 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8890000000000002e-06 2024-09-21T15:50:39.912 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.913 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.913 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:39.913 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105802+0000", 2024-09-21T15:50:39.913 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056974415 2024-09-21T15:50:39.913 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.913 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.913 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:39.913 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105871+0000", 2024-09-21T15:50:39.913 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8727999999999997e-05 2024-09-21T15:50:39.913 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.913 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.913 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:39.913 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105936+0000", 2024-09-21T15:50:39.913 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4554999999999994e-05 2024-09-21T15:50:39.913 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.914 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:39.914 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.914 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.914 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.914 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116238 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 780288~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:39.914 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048830+0000", 2024-09-21T15:50:39.914 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 43.840858767999997, 2024-09-21T15:50:39.914 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057296574000000003, 2024-09-21T15:50:39.914 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:39.914 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:39.914 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:39.914 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:39.914 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:39.914 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116238 2024-09-21T15:50:39.915 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.915 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:39.915 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.915 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:39.915 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048830+0000", 2024-09-21T15:50:39.915 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.915 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.915 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.915 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:39.915 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048830+0000", 2024-09-21T15:50:39.915 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.915 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.915 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.915 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:39.915 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048830+0000", 2024-09-21T15:50:39.916 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:39.916 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.916 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.916 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:39.916 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048831+0000", 2024-09-21T15:50:39.916 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0160000000000001e-06 2024-09-21T15:50:39.916 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.916 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.916 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:39.916 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048834+0000", 2024-09-21T15:50:39.916 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.813e-06 2024-09-21T15:50:39.916 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.916 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.917 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:39.917 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048836+0000", 2024-09-21T15:50:39.917 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6790000000000001e-06 2024-09-21T15:50:39.917 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.917 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.917 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:39.917 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105961+0000", 2024-09-21T15:50:39.917 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057124705999999997 2024-09-21T15:50:39.917 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.917 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.917 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:39.917 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106042+0000", 2024-09-21T15:50:39.917 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0820000000000002e-05 2024-09-21T15:50:39.917 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.917 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.918 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:39.918 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106127+0000", 2024-09-21T15:50:39.918 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4746000000000005e-05 2024-09-21T15:50:39.918 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.918 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:39.918 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.918 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.918 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.918 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116324 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 524288~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:39.918 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048838+0000", 2024-09-21T15:50:39.918 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 43.840850459000002, 2024-09-21T15:50:39.918 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057477185, 2024-09-21T15:50:39.918 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:39.919 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:39.919 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:39.919 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:39.919 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:39.919 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116324 2024-09-21T15:50:39.919 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.919 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:39.919 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.919 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:39.919 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048838+0000", 2024-09-21T15:50:39.919 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.919 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.919 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.919 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:39.919 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048838+0000", 2024-09-21T15:50:39.920 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.920 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.920 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.920 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:39.920 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048838+0000", 2024-09-21T15:50:39.920 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:39.920 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.920 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.920 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:39.920 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048842+0000", 2024-09-21T15:50:39.920 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4829999999999999e-06 2024-09-21T15:50:39.920 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.920 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.920 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:39.920 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048846+0000", 2024-09-21T15:50:39.920 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0470000000000004e-06 2024-09-21T15:50:39.921 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.921 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.921 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:39.921 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048849+0000", 2024-09-21T15:50:39.921 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3689999999999998e-06 2024-09-21T15:50:39.921 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.921 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.921 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:39.921 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106157+0000", 2024-09-21T15:50:39.921 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057307942000000001 2024-09-21T15:50:39.921 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.921 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.921 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:39.921 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106237+0000", 2024-09-21T15:50:39.922 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0320000000000003e-05 2024-09-21T15:50:39.922 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.922 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.922 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:39.922 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106315+0000", 2024-09-21T15:50:39.922 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8227999999999997e-05 2024-09-21T15:50:39.922 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.922 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:39.922 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.922 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.922 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.922 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116325 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 530432~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:39.923 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048851+0000", 2024-09-21T15:50:39.923 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 43.840837286999999, 2024-09-21T15:50:39.923 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057581036000000002, 2024-09-21T15:50:39.923 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:39.923 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:39.923 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:39.923 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:39.923 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:39.923 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116325 2024-09-21T15:50:39.923 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.923 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:39.923 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.923 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:39.923 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048851+0000", 2024-09-21T15:50:39.923 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.924 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.924 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.924 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:39.924 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048851+0000", 2024-09-21T15:50:39.924 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.924 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.924 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.924 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:39.924 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048851+0000", 2024-09-21T15:50:39.924 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:39.924 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.924 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.924 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:39.924 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048852+0000", 2024-09-21T15:50:39.924 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.077e-06 2024-09-21T15:50:39.925 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.925 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.925 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:39.925 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048855+0000", 2024-09-21T15:50:39.925 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9859999999999999e-06 2024-09-21T15:50:39.925 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.925 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.925 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:39.925 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048858+0000", 2024-09-21T15:50:39.925 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3810000000000002e-06 2024-09-21T15:50:39.925 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.925 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.925 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:39.925 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106335+0000", 2024-09-21T15:50:39.925 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057477473000000001 2024-09-21T15:50:39.925 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.926 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.926 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:39.926 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106393+0000", 2024-09-21T15:50:39.926 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8050000000000002e-05 2024-09-21T15:50:39.926 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.926 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.926 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:39.926 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106432+0000", 2024-09-21T15:50:39.926 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9274999999999999e-05 2024-09-21T15:50:39.926 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.926 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:39.926 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.926 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.926 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.926 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116326 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 536576~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:39.926 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048860+0000", 2024-09-21T15:50:39.927 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 43.840829235999998, 2024-09-21T15:50:39.927 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057774197999999999, 2024-09-21T15:50:39.927 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:39.927 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:39.927 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:39.927 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:39.927 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:39.927 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116326 2024-09-21T15:50:39.927 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.927 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:39.927 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.927 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:39.927 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048860+0000", 2024-09-21T15:50:39.927 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.928 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.928 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.928 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:39.928 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048860+0000", 2024-09-21T15:50:39.928 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.928 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.928 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.928 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:39.928 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048859+0000", 2024-09-21T15:50:39.928 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:39.928 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.928 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.929 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:39.929 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048860+0000", 2024-09-21T15:50:39.929 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.02e-06 2024-09-21T15:50:39.929 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.929 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.929 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:39.929 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048863+0000", 2024-09-21T15:50:39.929 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1039999999999998e-06 2024-09-21T15:50:39.929 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.929 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.929 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:39.929 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048867+0000", 2024-09-21T15:50:39.929 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.563e-06 2024-09-21T15:50:39.929 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.929 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.929 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:39.929 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106465+0000", 2024-09-21T15:50:39.930 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057597789000000003 2024-09-21T15:50:39.930 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.930 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.930 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:39.930 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106552+0000", 2024-09-21T15:50:39.930 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7602000000000006e-05 2024-09-21T15:50:39.930 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.930 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.930 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:39.930 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106634+0000", 2024-09-21T15:50:39.930 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1321000000000003e-05 2024-09-21T15:50:39.930 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.930 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:39.930 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.930 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.930 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.931 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116327 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 542720~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:39.931 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048870+0000", 2024-09-21T15:50:39.931 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 43.840818364, 2024-09-21T15:50:39.931 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057883148000000002, 2024-09-21T15:50:39.931 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:39.931 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:39.931 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:39.931 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:39.931 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:39.931 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116327 2024-09-21T15:50:39.931 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.931 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:39.931 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.931 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:39.931 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048870+0000", 2024-09-21T15:50:39.931 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.932 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.932 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.932 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:39.932 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048870+0000", 2024-09-21T15:50:39.932 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.932 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.932 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.932 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:39.932 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048870+0000", 2024-09-21T15:50:39.932 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:39.932 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.932 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.932 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:39.932 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048872+0000", 2024-09-21T15:50:39.932 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.824e-06 2024-09-21T15:50:39.933 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.933 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.933 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:39.933 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048879+0000", 2024-09-21T15:50:39.933 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6749999999999996e-06 2024-09-21T15:50:39.933 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.933 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.933 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:39.933 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048883+0000", 2024-09-21T15:50:39.933 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2200000000000003e-06 2024-09-21T15:50:39.933 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.933 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.933 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:39.933 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106663+0000", 2024-09-21T15:50:39.933 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057780049 2024-09-21T15:50:39.933 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.934 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.934 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:39.934 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106711+0000", 2024-09-21T15:50:39.934 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8433999999999997e-05 2024-09-21T15:50:39.934 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.934 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.934 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:39.934 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106754+0000", 2024-09-21T15:50:39.934 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2236e-05 2024-09-21T15:50:39.934 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.934 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:39.934 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.935 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.935 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.935 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116328 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 548864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:39.935 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048887+0000", 2024-09-21T15:50:39.935 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 43.840801843999998, 2024-09-21T15:50:39.935 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057981984, 2024-09-21T15:50:39.935 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:39.935 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:39.935 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:39.935 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:39.935 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:39.935 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116328 2024-09-21T15:50:39.935 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.935 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:39.935 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.935 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:39.936 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048887+0000", 2024-09-21T15:50:39.936 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.936 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.936 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.936 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:39.936 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048887+0000", 2024-09-21T15:50:39.936 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.936 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.936 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.936 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:39.936 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048887+0000", 2024-09-21T15:50:39.936 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:39.936 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.936 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.936 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:39.936 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048891+0000", 2024-09-21T15:50:39.936 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.843e-06 2024-09-21T15:50:39.937 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.937 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.937 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:39.937 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048897+0000", 2024-09-21T15:50:39.937 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5419999999999996e-06 2024-09-21T15:50:39.937 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.937 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.937 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:39.937 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048901+0000", 2024-09-21T15:50:39.937 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4689999999999999e-06 2024-09-21T15:50:39.937 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.937 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.937 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:39.937 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106780+0000", 2024-09-21T15:50:39.937 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057878259000000001 2024-09-21T15:50:39.937 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.937 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.938 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:39.938 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106827+0000", 2024-09-21T15:50:39.938 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7098000000000002e-05 2024-09-21T15:50:39.938 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.938 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.938 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:39.938 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106869+0000", 2024-09-21T15:50:39.938 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2129999999999998e-05 2024-09-21T15:50:39.938 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.938 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:39.938 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.938 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.938 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.938 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116329 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 555008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:39.939 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048905+0000", 2024-09-21T15:50:39.939 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 43.840783733999999, 2024-09-21T15:50:39.939 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058115554999999999, 2024-09-21T15:50:39.939 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:39.939 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:39.939 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:39.939 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:39.939 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:39.939 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116329 2024-09-21T15:50:39.939 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.939 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:39.939 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.939 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:39.939 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048905+0000", 2024-09-21T15:50:39.939 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.939 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.940 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.940 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:39.940 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048905+0000", 2024-09-21T15:50:39.940 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.940 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.940 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.940 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:39.940 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048905+0000", 2024-09-21T15:50:39.940 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:39.940 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.940 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.940 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:39.940 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048907+0000", 2024-09-21T15:50:39.940 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.086e-06 2024-09-21T15:50:39.940 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.941 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.941 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:39.941 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048912+0000", 2024-09-21T15:50:39.941 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2830000000000001e-06 2024-09-21T15:50:39.941 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.941 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.941 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:39.941 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048916+0000", 2024-09-21T15:50:39.941 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.98e-06 2024-09-21T15:50:39.941 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.941 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.941 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:39.941 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106907+0000", 2024-09-21T15:50:39.941 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057990719000000003 2024-09-21T15:50:39.941 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.941 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.942 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:39.942 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106969+0000", 2024-09-21T15:50:39.942 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1789000000000001e-05 2024-09-21T15:50:39.942 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.942 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.942 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:39.942 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107021+0000", 2024-09-21T15:50:39.942 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2030999999999997e-05 2024-09-21T15:50:39.942 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.942 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:39.942 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.942 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.942 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.942 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116330 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 561152~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:39.942 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048922+0000", 2024-09-21T15:50:39.942 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 43.840766576, 2024-09-21T15:50:39.943 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058242343000000002, 2024-09-21T15:50:39.943 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:39.943 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:39.943 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:39.943 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:39.943 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:39.943 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116330 2024-09-21T15:50:39.943 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.943 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:39.943 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.943 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:39.943 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048922+0000", 2024-09-21T15:50:39.943 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.943 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.943 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.943 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:39.943 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048922+0000", 2024-09-21T15:50:39.944 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.944 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.944 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.944 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:39.944 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048922+0000", 2024-09-21T15:50:39.944 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:39.944 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.944 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.944 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:39.944 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048924+0000", 2024-09-21T15:50:39.944 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1660000000000001e-06 2024-09-21T15:50:39.944 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.944 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.944 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:39.944 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048930+0000", 2024-09-21T15:50:39.944 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4529999999999999e-06 2024-09-21T15:50:39.944 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.945 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.945 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:39.945 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048934+0000", 2024-09-21T15:50:39.945 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0269999999999997e-06 2024-09-21T15:50:39.945 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.945 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.945 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:39.945 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107047+0000", 2024-09-21T15:50:39.945 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058112913000000002 2024-09-21T15:50:39.945 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.945 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.945 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:39.945 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107095+0000", 2024-09-21T15:50:39.945 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7778000000000002e-05 2024-09-21T15:50:39.945 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.945 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.946 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:39.946 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107165+0000", 2024-09-21T15:50:39.946 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9370999999999997e-05 2024-09-21T15:50:39.946 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.946 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:39.946 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.946 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.946 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.946 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116331 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 567296~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:39.946 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.049440+0000", 2024-09-21T15:50:39.946 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 43.840248482, 2024-09-21T15:50:39.946 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057878485, 2024-09-21T15:50:39.946 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:39.946 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:39.947 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:39.947 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:39.947 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:39.947 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116331 2024-09-21T15:50:39.947 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.947 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:39.947 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.947 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:39.947 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049440+0000", 2024-09-21T15:50:39.947 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.947 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.947 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.947 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:39.947 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049440+0000", 2024-09-21T15:50:39.947 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.947 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.948 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.948 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:39.948 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048939+0000", 2024-09-21T15:50:39.948 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9994984 2024-09-21T15:50:39.948 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.948 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.948 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:39.948 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049444+0000", 2024-09-21T15:50:39.948 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00050557200000000003 2024-09-21T15:50:39.948 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.948 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.948 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:39.948 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049460+0000", 2024-09-21T15:50:39.948 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6120000000000002e-05 2024-09-21T15:50:39.948 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.948 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.949 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:39.949 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049468+0000", 2024-09-21T15:50:39.949 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4209999999999998e-06 2024-09-21T15:50:39.949 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.949 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.949 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:39.949 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107189+0000", 2024-09-21T15:50:39.949 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057721019999999998 2024-09-21T15:50:39.949 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.949 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.949 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:39.949 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107254+0000", 2024-09-21T15:50:39.949 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5265000000000006e-05 2024-09-21T15:50:39.949 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.949 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.949 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:39.949 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107319+0000", 2024-09-21T15:50:39.950 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4784999999999997e-05 2024-09-21T15:50:39.950 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.950 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:39.950 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.950 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.950 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.950 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116332 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 573440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:39.950 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.049474+0000", 2024-09-21T15:50:39.950 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 43.840215221000001, 2024-09-21T15:50:39.950 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058013809, 2024-09-21T15:50:39.950 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:39.950 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:39.950 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:39.950 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:39.950 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:39.950 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116332 2024-09-21T15:50:39.950 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.951 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:39.951 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.951 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:39.951 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049474+0000", 2024-09-21T15:50:39.951 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.951 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.951 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.951 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:39.951 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049474+0000", 2024-09-21T15:50:39.951 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.951 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.951 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.951 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:39.951 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049473+0000", 2024-09-21T15:50:39.951 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:39.951 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.952 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.952 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:39.952 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049476+0000", 2024-09-21T15:50:39.952 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6249999999999999e-06 2024-09-21T15:50:39.952 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.952 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.952 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:39.952 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049482+0000", 2024-09-21T15:50:39.952 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0959999999999997e-06 2024-09-21T15:50:39.952 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.952 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.952 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:39.952 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049487+0000", 2024-09-21T15:50:39.952 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5949999999999998e-06 2024-09-21T15:50:39.952 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.952 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.953 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:39.953 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107344+0000", 2024-09-21T15:50:39.953 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057856594999999997 2024-09-21T15:50:39.953 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.953 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.953 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:39.953 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107422+0000", 2024-09-21T15:50:39.953 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7582000000000004e-05 2024-09-21T15:50:39.953 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.953 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.953 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:39.953 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107487+0000", 2024-09-21T15:50:39.953 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5723999999999996e-05 2024-09-21T15:50:39.953 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.953 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:39.953 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.954 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.954 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.954 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116333 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 579584~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:39.954 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.049993+0000", 2024-09-21T15:50:39.954 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 43.839695438, 2024-09-21T15:50:39.954 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057663537000000001, 2024-09-21T15:50:39.954 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:39.954 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:39.954 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:39.954 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:39.954 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:39.954 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116333 2024-09-21T15:50:39.954 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.954 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:39.954 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.955 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:39.955 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049993+0000", 2024-09-21T15:50:39.955 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.955 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.955 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.955 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:39.955 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049993+0000", 2024-09-21T15:50:39.955 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.955 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.955 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.955 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:39.955 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049491+0000", 2024-09-21T15:50:39.955 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9994979 2024-09-21T15:50:39.955 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.955 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.955 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:39.956 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049998+0000", 2024-09-21T15:50:39.956 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00050654200000000002 2024-09-21T15:50:39.956 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.956 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.956 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:39.956 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050010+0000", 2024-09-21T15:50:39.956 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1790999999999999e-05 2024-09-21T15:50:39.956 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.956 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.956 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:39.956 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050016+0000", 2024-09-21T15:50:39.956 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2179999999999996e-06 2024-09-21T15:50:39.956 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.956 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.956 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:39.956 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107513+0000", 2024-09-21T15:50:39.956 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057496927000000003 2024-09-21T15:50:39.957 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.957 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.957 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:39.957 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107590+0000", 2024-09-21T15:50:39.957 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7547999999999997e-05 2024-09-21T15:50:39.957 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.957 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.957 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:39.957 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107657+0000", 2024-09-21T15:50:39.957 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6630000000000004e-05 2024-09-21T15:50:39.957 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.957 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:39.957 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.957 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.957 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.957 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116334 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 585728~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:39.958 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.050091+0000", 2024-09-21T15:50:39.958 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 43.839597810000001, 2024-09-21T15:50:39.958 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057734355000000001, 2024-09-21T15:50:39.958 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:39.958 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:39.958 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:39.958 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:39.958 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:39.958 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116334 2024-09-21T15:50:39.958 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.958 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:39.958 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.958 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:39.958 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050091+0000", 2024-09-21T15:50:39.959 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.959 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.959 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.959 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:39.959 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050091+0000", 2024-09-21T15:50:39.959 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.959 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.959 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.959 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:39.959 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050021+0000", 2024-09-21T15:50:39.959 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999304 2024-09-21T15:50:39.959 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.959 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.959 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:39.959 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050095+0000", 2024-09-21T15:50:39.959 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4158000000000004e-05 2024-09-21T15:50:39.959 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.960 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.960 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:39.960 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050103+0000", 2024-09-21T15:50:39.960 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0770000000000002e-06 2024-09-21T15:50:39.960 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.960 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.960 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:39.960 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050108+0000", 2024-09-21T15:50:39.960 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6330000000000001e-06 2024-09-21T15:50:39.960 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.960 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.960 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:39.960 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107681+0000", 2024-09-21T15:50:39.960 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057572996000000001 2024-09-21T15:50:39.960 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.960 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.961 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:39.961 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107759+0000", 2024-09-21T15:50:39.961 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7368999999999997e-05 2024-09-21T15:50:39.961 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.961 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.961 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:39.961 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107825+0000", 2024-09-21T15:50:39.961 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6783000000000001e-05 2024-09-21T15:50:39.961 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.961 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:39.961 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.961 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.961 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.961 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116335 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 591872~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:39.961 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.050542+0000", 2024-09-21T15:50:39.961 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 43.839147224999998, 2024-09-21T15:50:39.962 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057466455999999999, 2024-09-21T15:50:39.962 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:39.962 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:39.962 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:39.962 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:39.962 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:39.962 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116335 2024-09-21T15:50:39.962 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.962 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:39.962 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.962 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:39.962 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050542+0000", 2024-09-21T15:50:39.962 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.962 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.962 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.962 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:39.962 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050542+0000", 2024-09-21T15:50:39.963 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.963 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.963 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.963 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:39.963 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050113+0000", 2024-09-21T15:50:39.963 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9995713 2024-09-21T15:50:39.963 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.963 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.963 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:39.963 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050546+0000", 2024-09-21T15:50:39.963 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00043328699999999997 2024-09-21T15:50:39.963 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.963 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.963 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:39.963 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050557+0000", 2024-09-21T15:50:39.963 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1011e-05 2024-09-21T15:50:39.964 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.964 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.964 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:39.964 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050564+0000", 2024-09-21T15:50:39.964 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1130000000000003e-06 2024-09-21T15:50:39.964 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.964 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.964 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:39.964 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107852+0000", 2024-09-21T15:50:39.964 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057287350000000001 2024-09-21T15:50:39.964 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.964 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.964 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:39.964 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107928+0000", 2024-09-21T15:50:39.964 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6668000000000005e-05 2024-09-21T15:50:39.965 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.965 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.965 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:39.965 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108008+0000", 2024-09-21T15:50:39.965 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9783999999999994e-05 2024-09-21T15:50:39.965 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.965 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:39.965 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.965 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.965 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.965 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116336 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 598016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:39.965 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.050834+0000", 2024-09-21T15:50:39.965 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 43.838854959000003, 2024-09-21T15:50:39.965 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057353572999999998, 2024-09-21T15:50:39.965 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:39.966 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:39.966 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:39.966 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:39.966 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:39.966 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116336 2024-09-21T15:50:39.966 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.966 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:39.966 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.966 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:39.966 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050834+0000", 2024-09-21T15:50:39.966 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.966 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.966 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.966 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:39.966 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050834+0000", 2024-09-21T15:50:39.967 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.967 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.967 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.967 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:39.967 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050571+0000", 2024-09-21T15:50:39.967 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9997373 2024-09-21T15:50:39.967 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.967 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.967 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:39.967 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050842+0000", 2024-09-21T15:50:39.967 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00027029099999999998 2024-09-21T15:50:39.967 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.967 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.967 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:39.967 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050853+0000", 2024-09-21T15:50:39.968 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1532000000000001e-05 2024-09-21T15:50:39.968 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.968 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.968 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:39.968 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050863+0000", 2024-09-21T15:50:39.968 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4239999999999999e-06 2024-09-21T15:50:39.968 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.968 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.968 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:39.968 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108045+0000", 2024-09-21T15:50:39.968 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057182415 2024-09-21T15:50:39.968 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.968 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.968 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:39.968 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108123+0000", 2024-09-21T15:50:39.968 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8275000000000005e-05 2024-09-21T15:50:39.969 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.969 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.969 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:39.969 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108187+0000", 2024-09-21T15:50:39.969 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4152000000000005e-05 2024-09-21T15:50:39.969 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.969 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:39.969 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.969 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.969 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.969 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116337 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 604160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:39.969 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.051192+0000", 2024-09-21T15:50:39.969 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 43.838496331000002, 2024-09-21T15:50:39.969 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057175981000000001, 2024-09-21T15:50:39.969 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:39.969 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:39.970 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:39.970 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:39.970 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:39.970 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116337 2024-09-21T15:50:39.970 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.970 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:39.970 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.970 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:39.970 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051192+0000", 2024-09-21T15:50:39.970 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.970 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.970 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.970 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:39.970 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051192+0000", 2024-09-21T15:50:39.970 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.971 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.971 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.971 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:39.971 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050870+0000", 2024-09-21T15:50:39.971 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9996777 2024-09-21T15:50:39.971 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.971 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.971 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:39.971 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051197+0000", 2024-09-21T15:50:39.971 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00032656299999999998 2024-09-21T15:50:39.971 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.971 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.971 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:39.971 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051209+0000", 2024-09-21T15:50:39.971 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2636000000000001e-05 2024-09-21T15:50:39.972 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.972 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.972 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:39.972 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051217+0000", 2024-09-21T15:50:39.972 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5989999999999996e-06 2024-09-21T15:50:39.972 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.972 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.972 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:39.972 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108224+0000", 2024-09-21T15:50:39.972 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057007622000000001 2024-09-21T15:50:39.972 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.972 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.972 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:39.972 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108299+0000", 2024-09-21T15:50:39.972 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5099999999999996e-05 2024-09-21T15:50:39.972 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.973 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.973 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:39.973 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108368+0000", 2024-09-21T15:50:39.973 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8918000000000007e-05 2024-09-21T15:50:39.973 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.973 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:39.973 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.973 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.973 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.973 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116338 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 610304~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:39.973 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.051583+0000", 2024-09-21T15:50:39.973 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 43.838105878, 2024-09-21T15:50:39.973 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056967716000000002, 2024-09-21T15:50:39.973 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:39.973 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:39.973 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:39.974 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:39.974 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:39.974 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116338 2024-09-21T15:50:39.974 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.974 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:39.974 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.974 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:39.974 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051583+0000", 2024-09-21T15:50:39.974 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.974 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.974 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.974 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:39.974 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051583+0000", 2024-09-21T15:50:39.974 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:39.974 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.974 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.975 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:39.975 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051223+0000", 2024-09-21T15:50:39.975 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.99964 2024-09-21T15:50:39.975 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.975 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.975 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:39.975 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051587+0000", 2024-09-21T15:50:39.975 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00036352999999999998 2024-09-21T15:50:39.975 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.975 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.975 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:39.975 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051597+0000", 2024-09-21T15:50:39.975 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.893e-06 2024-09-21T15:50:39.975 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.975 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.976 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:39.976 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051603+0000", 2024-09-21T15:50:39.976 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0519999999999998e-06 2024-09-21T15:50:39.976 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.976 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.976 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:39.976 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108394+0000", 2024-09-21T15:50:39.976 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056791531999999999 2024-09-21T15:50:39.976 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.976 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.976 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:39.976 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108480+0000", 2024-09-21T15:50:39.976 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.577e-05 2024-09-21T15:50:39.976 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:39.976 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:39.977 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:39.977 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108551+0000", 2024-09-21T15:50:39.977 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0722000000000005e-05 2024-09-21T15:50:39.977 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.977 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:39.977 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.977 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:39.977 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:39.977 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:39.978 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:50:39.980 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:39.979+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:40.080 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:40.079+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:40.086 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:40.086 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:40.086 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:50:40.086 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:40.095 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:50:40.182 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:40.180+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:40.237 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:40.238 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:40.238 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:50:40.238 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:50:40.238 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:40.246 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:50:40.283 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:40.281+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:40.350 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:40.350 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:50:40.350 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:50:40.350 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:50:40.350 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.351 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155031 3.1c 3.8612741c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:40.351 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448748+0000", 2024-09-21T15:50:40.351 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 17.899979909999999, 2024-09-21T15:50:40.351 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064010286999999999, 2024-09-21T15:50:40.351 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:40.351 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:40.351 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:40.351 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:40.351 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:40.351 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155031 2024-09-21T15:50:40.351 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.351 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:40.351 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.351 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:40.351 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448748+0000", 2024-09-21T15:50:40.352 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.352 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.352 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.352 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:40.352 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448748+0000", 2024-09-21T15:50:40.352 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.352 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.352 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.352 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:40.352 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448747+0000", 2024-09-21T15:50:40.352 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:40.352 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.352 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.352 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:40.352 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448753+0000", 2024-09-21T15:50:40.353 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.755e-06 2024-09-21T15:50:40.353 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.353 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.353 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:40.353 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448763+0000", 2024-09-21T15:50:40.353 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7659999999999993e-06 2024-09-21T15:50:40.353 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.353 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.353 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:40.353 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448769+0000", 2024-09-21T15:50:40.353 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6060000000000002e-06 2024-09-21T15:50:40.353 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.353 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.353 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:40.354 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512750+0000", 2024-09-21T15:50:40.354 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.063981298000000006 2024-09-21T15:50:40.354 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.354 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.354 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:40.354 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512758+0000", 2024-09-21T15:50:40.354 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1669999999999999e-06 2024-09-21T15:50:40.354 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.354 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:40.354 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.354 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.354 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.354 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155091 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:40.354 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448916+0000", 2024-09-21T15:50:40.355 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 17.899812064999999, 2024-09-21T15:50:40.355 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064006309999999997, 2024-09-21T15:50:40.355 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:40.355 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:40.355 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:40.355 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:40.355 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:40.355 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155091 2024-09-21T15:50:40.355 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.355 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:40.355 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.355 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:40.355 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448916+0000", 2024-09-21T15:50:40.355 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.356 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.356 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.356 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:40.356 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448916+0000", 2024-09-21T15:50:40.356 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.356 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.356 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.356 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:40.356 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448915+0000", 2024-09-21T15:50:40.356 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:40.356 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.356 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.356 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:40.356 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448917+0000", 2024-09-21T15:50:40.357 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2400000000000002e-06 2024-09-21T15:50:40.357 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.357 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.357 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:40.357 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448923+0000", 2024-09-21T15:50:40.357 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0580000000000003e-06 2024-09-21T15:50:40.357 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.357 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.357 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:40.357 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448927+0000", 2024-09-21T15:50:40.357 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.031e-06 2024-09-21T15:50:40.357 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.357 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.357 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:40.358 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512916+0000", 2024-09-21T15:50:40.358 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.063988456999999999 2024-09-21T15:50:40.358 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.358 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.358 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:40.358 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512922+0000", 2024-09-21T15:50:40.358 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9009999999999999e-06 2024-09-21T15:50:40.358 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.358 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:40.358 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.358 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.358 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.358 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155092 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:40.358 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448931+0000", 2024-09-21T15:50:40.359 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 17.899796283000001, 2024-09-21T15:50:40.359 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064022793999999994, 2024-09-21T15:50:40.359 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:40.359 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:40.359 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:40.359 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:40.359 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:40.359 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155092 2024-09-21T15:50:40.359 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.359 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:40.359 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.359 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:40.359 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448931+0000", 2024-09-21T15:50:40.359 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.359 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.360 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.360 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:40.360 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448931+0000", 2024-09-21T15:50:40.360 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.360 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.360 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.360 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:40.360 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448931+0000", 2024-09-21T15:50:40.360 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:40.360 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.360 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.360 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:40.360 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448933+0000", 2024-09-21T15:50:40.360 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1799999999999999e-06 2024-09-21T15:50:40.360 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.361 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.361 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:40.361 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448939+0000", 2024-09-21T15:50:40.361 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.604e-06 2024-09-21T15:50:40.361 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.361 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.361 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:40.361 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448943+0000", 2024-09-21T15:50:40.361 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0740000000000003e-06 2024-09-21T15:50:40.361 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.361 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.361 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:40.361 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512946+0000", 2024-09-21T15:50:40.361 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064003612000000001 2024-09-21T15:50:40.361 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.362 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.362 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:40.362 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512954+0000", 2024-09-21T15:50:40.362 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6690000000000006e-06 2024-09-21T15:50:40.362 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.362 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:40.362 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.362 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.362 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.362 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155093 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:40.362 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448947+0000", 2024-09-21T15:50:40.362 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 17.899780909, 2024-09-21T15:50:40.362 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064020717000000005, 2024-09-21T15:50:40.362 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:40.363 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:40.363 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:40.363 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:40.363 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:40.363 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155093 2024-09-21T15:50:40.363 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.363 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:40.363 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.363 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:40.363 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448947+0000", 2024-09-21T15:50:40.363 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.363 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.363 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.363 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:40.363 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448947+0000", 2024-09-21T15:50:40.363 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.364 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.364 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.364 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:40.364 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448946+0000", 2024-09-21T15:50:40.364 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:40.364 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.364 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.364 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:40.364 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448949+0000", 2024-09-21T15:50:40.364 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3860000000000001e-06 2024-09-21T15:50:40.364 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.364 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.364 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:40.364 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448955+0000", 2024-09-21T15:50:40.364 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.823e-06 2024-09-21T15:50:40.365 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.365 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.365 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:40.365 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448959+0000", 2024-09-21T15:50:40.365 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2799999999999997e-06 2024-09-21T15:50:40.365 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.365 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.365 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:40.365 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512964+0000", 2024-09-21T15:50:40.365 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064004741000000004 2024-09-21T15:50:40.365 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.365 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.365 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:40.365 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512967+0000", 2024-09-21T15:50:40.365 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.839e-06 2024-09-21T15:50:40.366 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.366 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:40.366 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.366 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.366 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.366 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155094 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:40.366 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448962+0000", 2024-09-21T15:50:40.366 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 17.899765286000001, 2024-09-21T15:50:40.366 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064037378000000006, 2024-09-21T15:50:40.366 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:40.366 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:40.366 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:40.366 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:40.366 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:40.366 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155094 2024-09-21T15:50:40.367 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.367 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:40.367 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.367 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:40.367 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448962+0000", 2024-09-21T15:50:40.367 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.367 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.367 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.367 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:40.367 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448962+0000", 2024-09-21T15:50:40.367 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.367 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.367 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.367 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:40.367 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448962+0000", 2024-09-21T15:50:40.368 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:40.368 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.368 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.368 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:40.368 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448968+0000", 2024-09-21T15:50:40.368 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9710000000000001e-06 2024-09-21T15:50:40.368 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.368 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.368 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:40.368 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448974+0000", 2024-09-21T15:50:40.368 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7230000000000001e-06 2024-09-21T15:50:40.368 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.368 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.368 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:40.368 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448978+0000", 2024-09-21T15:50:40.368 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3749999999999996e-06 2024-09-21T15:50:40.369 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.369 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.369 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:40.369 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512992+0000", 2024-09-21T15:50:40.369 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064013525000000002 2024-09-21T15:50:40.369 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.369 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.369 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:40.369 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513000+0000", 2024-09-21T15:50:40.369 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0830000000000004e-06 2024-09-21T15:50:40.369 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.369 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:40.369 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.369 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.369 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.369 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155095 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:40.370 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448982+0000", 2024-09-21T15:50:40.370 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 17.899746004000001, 2024-09-21T15:50:40.370 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064041904999999996, 2024-09-21T15:50:40.370 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:40.370 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:40.370 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:40.370 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:40.370 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:40.370 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155095 2024-09-21T15:50:40.370 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.370 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:40.370 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.370 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:40.370 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448982+0000", 2024-09-21T15:50:40.370 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.371 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.371 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.371 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:40.371 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448982+0000", 2024-09-21T15:50:40.371 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.371 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.371 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.371 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:40.371 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448981+0000", 2024-09-21T15:50:40.371 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:40.371 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.371 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.371 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:40.371 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448983+0000", 2024-09-21T15:50:40.371 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.905e-06 2024-09-21T15:50:40.372 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.372 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.372 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:40.372 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448989+0000", 2024-09-21T15:50:40.372 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.536e-06 2024-09-21T15:50:40.372 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.372 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.372 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:40.372 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448993+0000", 2024-09-21T15:50:40.372 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2390000000000004e-06 2024-09-21T15:50:40.372 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.372 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.372 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:40.372 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513015+0000", 2024-09-21T15:50:40.372 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064022081999999994 2024-09-21T15:50:40.373 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.373 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.373 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:40.373 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513023+0000", 2024-09-21T15:50:40.373 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4500000000000004e-06 2024-09-21T15:50:40.373 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.373 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:40.373 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.373 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.373 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.373 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155096 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:40.373 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448996+0000", 2024-09-21T15:50:40.373 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 17.899731103000001, 2024-09-21T15:50:40.373 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064060868000000007, 2024-09-21T15:50:40.373 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:40.373 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:40.373 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:40.374 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:40.374 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:40.374 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155096 2024-09-21T15:50:40.374 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.374 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:40.374 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.374 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:40.374 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448996+0000", 2024-09-21T15:50:40.374 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.374 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.374 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.374 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:40.374 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448996+0000", 2024-09-21T15:50:40.374 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.374 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.375 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.375 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:40.375 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448996+0000", 2024-09-21T15:50:40.375 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:40.375 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.375 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.375 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:40.375 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448998+0000", 2024-09-21T15:50:40.375 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8780000000000001e-06 2024-09-21T15:50:40.375 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.375 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.375 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:40.375 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449003+0000", 2024-09-21T15:50:40.375 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4099999999999999e-06 2024-09-21T15:50:40.375 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.376 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.376 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:40.376 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449008+0000", 2024-09-21T15:50:40.376 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.335e-06 2024-09-21T15:50:40.376 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.376 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.376 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:40.376 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513049+0000", 2024-09-21T15:50:40.376 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064041059999999997 2024-09-21T15:50:40.376 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.376 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.376 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:40.376 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513057+0000", 2024-09-21T15:50:40.376 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5050000000000007e-06 2024-09-21T15:50:40.376 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.377 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:40.377 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.377 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.377 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.377 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155097 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:40.377 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449019+0000", 2024-09-21T15:50:40.377 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 17.899708682, 2024-09-21T15:50:40.377 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064052488000000005, 2024-09-21T15:50:40.377 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:40.377 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:40.377 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:40.377 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:40.377 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:40.377 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155097 2024-09-21T15:50:40.378 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.378 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:40.378 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.378 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:40.378 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449019+0000", 2024-09-21T15:50:40.378 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.378 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.378 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.378 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:40.378 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449019+0000", 2024-09-21T15:50:40.378 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.378 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.378 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.378 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:40.378 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449019+0000", 2024-09-21T15:50:40.378 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:40.379 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.379 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.379 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:40.379 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449021+0000", 2024-09-21T15:50:40.379 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.244e-06 2024-09-21T15:50:40.379 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.379 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.379 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:40.379 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449027+0000", 2024-09-21T15:50:40.379 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8619999999999999e-06 2024-09-21T15:50:40.379 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.379 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.379 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:40.379 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449031+0000", 2024-09-21T15:50:40.379 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1359999999999999e-06 2024-09-21T15:50:40.379 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.380 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.380 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:40.380 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513067+0000", 2024-09-21T15:50:40.380 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064036391999999998 2024-09-21T15:50:40.380 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.380 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.380 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:40.380 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513071+0000", 2024-09-21T15:50:40.380 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1989999999999999e-06 2024-09-21T15:50:40.380 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.380 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:40.380 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.380 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.380 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.380 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155098 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:40.380 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449035+0000", 2024-09-21T15:50:40.380 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 17.899693084999999, 2024-09-21T15:50:40.381 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064071761000000005, 2024-09-21T15:50:40.381 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:40.381 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:40.381 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:40.381 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:40.381 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:40.381 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155098 2024-09-21T15:50:40.381 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.381 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:40.381 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.381 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:40.381 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449035+0000", 2024-09-21T15:50:40.381 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.381 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.381 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.381 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:40.382 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449035+0000", 2024-09-21T15:50:40.382 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.382 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.382 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.382 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:40.382 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449034+0000", 2024-09-21T15:50:40.382 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:40.382 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.382 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.382 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:40.382 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449036+0000", 2024-09-21T15:50:40.382 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9989999999999998e-06 2024-09-21T15:50:40.382 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.382 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.382 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:40.383 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449042+0000", 2024-09-21T15:50:40.383 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6010000000000002e-06 2024-09-21T15:50:40.383 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.383 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.383 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:40.383 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449047+0000", 2024-09-21T15:50:40.383 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3600000000000004e-06 2024-09-21T15:50:40.383 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.383 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.383 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:40.383 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513098+0000", 2024-09-21T15:50:40.383 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064050544000000001 2024-09-21T15:50:40.383 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.383 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.384 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:40.384 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513106+0000", 2024-09-21T15:50:40.384 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5650000000000002e-06 2024-09-21T15:50:40.384 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.384 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:40.384 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.384 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.384 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.384 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155099 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:40.384 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449051+0000", 2024-09-21T15:50:40.384 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 17.899676543999998, 2024-09-21T15:50:40.384 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064069388000000005, 2024-09-21T15:50:40.384 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:40.385 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:40.385 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:40.385 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:40.385 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:40.385 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155099 2024-09-21T15:50:40.385 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.385 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:40.385 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.385 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:40.385 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449051+0000", 2024-09-21T15:50:40.385 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.385 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.385 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.385 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:40.385 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449051+0000", 2024-09-21T15:50:40.386 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.386 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.386 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.386 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:40.386 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449051+0000", 2024-09-21T15:50:40.386 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:40.386 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.386 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.386 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:40.386 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449053+0000", 2024-09-21T15:50:40.386 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2620000000000001e-06 2024-09-21T15:50:40.386 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.386 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.386 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:40.386 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449058+0000", 2024-09-21T15:50:40.387 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1680000000000003e-06 2024-09-21T15:50:40.387 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.387 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.387 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:40.387 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449062+0000", 2024-09-21T15:50:40.387 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9979999999999997e-06 2024-09-21T15:50:40.387 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.387 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.387 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:40.387 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513116+0000", 2024-09-21T15:50:40.387 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064053761000000001 2024-09-21T15:50:40.387 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.387 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.387 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:40.387 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513120+0000", 2024-09-21T15:50:40.387 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5399999999999997e-06 2024-09-21T15:50:40.388 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.388 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:40.388 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.388 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.388 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.388 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155100 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:40.388 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449101+0000", 2024-09-21T15:50:40.388 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 17.899626788999999, 2024-09-21T15:50:40.388 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064079585999999994, 2024-09-21T15:50:40.388 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:40.388 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:40.388 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:40.388 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:40.388 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:40.389 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155100 2024-09-21T15:50:40.389 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.389 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:40.389 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.389 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:40.389 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449101+0000", 2024-09-21T15:50:40.389 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.389 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.389 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.389 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:40.389 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449101+0000", 2024-09-21T15:50:40.389 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.389 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.389 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.389 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:40.390 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449100+0000", 2024-09-21T15:50:40.390 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-21T15:50:40.390 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.390 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.390 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:40.390 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449105+0000", 2024-09-21T15:50:40.390 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6700000000000002e-06 2024-09-21T15:50:40.390 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.390 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.390 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:40.390 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449117+0000", 2024-09-21T15:50:40.391 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1841000000000001e-05 2024-09-21T15:50:40.391 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.391 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.391 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:40.391 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449124+0000", 2024-09-21T15:50:40.391 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4660000000000002e-06 2024-09-21T15:50:40.392 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.392 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.392 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:40.392 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513163+0000", 2024-09-21T15:50:40.392 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064039053999999998 2024-09-21T15:50:40.392 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.392 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.392 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:40.392 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513180+0000", 2024-09-21T15:50:40.392 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7333000000000002e-05 2024-09-21T15:50:40.392 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.392 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:40.392 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.393 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.393 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.393 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155101 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:40.393 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449166+0000", 2024-09-21T15:50:40.393 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 17.899562083999999, 2024-09-21T15:50:40.393 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064027639999999997, 2024-09-21T15:50:40.393 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:40.393 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:40.393 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:40.393 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:40.393 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:40.393 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155101 2024-09-21T15:50:40.393 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.393 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:40.393 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.394 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:40.394 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449166+0000", 2024-09-21T15:50:40.394 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.394 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.394 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.394 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:40.394 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449166+0000", 2024-09-21T15:50:40.394 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.394 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.394 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.394 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:40.394 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449165+0000", 2024-09-21T15:50:40.395 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:40.395 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.395 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.395 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:40.395 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449168+0000", 2024-09-21T15:50:40.395 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5099999999999999e-06 2024-09-21T15:50:40.395 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.395 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.395 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:40.395 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449177+0000", 2024-09-21T15:50:40.395 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6659999999999997e-06 2024-09-21T15:50:40.395 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.395 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.395 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:40.395 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449182+0000", 2024-09-21T15:50:40.396 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1320000000000002e-06 2024-09-21T15:50:40.396 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.396 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.396 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:40.396 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513189+0000", 2024-09-21T15:50:40.396 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064007231999999997 2024-09-21T15:50:40.396 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.396 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.396 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:40.396 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513193+0000", 2024-09-21T15:50:40.396 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.692e-06 2024-09-21T15:50:40.396 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.396 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:40.396 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.397 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.397 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.397 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155134 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:40.397 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449187+0000", 2024-09-21T15:50:40.397 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 17.899540496, 2024-09-21T15:50:40.397 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064031331999999996, 2024-09-21T15:50:40.397 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:40.397 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:40.397 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:40.397 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:40.397 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:40.397 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155134 2024-09-21T15:50:40.397 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.398 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:40.398 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.398 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:40.398 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449187+0000", 2024-09-21T15:50:40.398 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.398 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.398 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.398 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:40.398 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449187+0000", 2024-09-21T15:50:40.398 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.398 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.398 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.398 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:40.398 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449187+0000", 2024-09-21T15:50:40.399 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:40.399 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.399 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.399 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:40.399 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449189+0000", 2024-09-21T15:50:40.399 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5469999999999999e-06 2024-09-21T15:50:40.399 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.399 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.399 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:40.399 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449196+0000", 2024-09-21T15:50:40.399 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7009999999999999e-06 2024-09-21T15:50:40.399 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.399 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.399 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:40.400 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449202+0000", 2024-09-21T15:50:40.400 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7409999999999998e-06 2024-09-21T15:50:40.400 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.400 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.400 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:40.400 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513214+0000", 2024-09-21T15:50:40.400 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064012301999999993 2024-09-21T15:50:40.400 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.400 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.400 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:40.400 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513218+0000", 2024-09-21T15:50:40.400 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5709999999999999e-06 2024-09-21T15:50:40.400 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.400 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:40.400 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.401 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.401 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.401 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155135 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:40.401 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449207+0000", 2024-09-21T15:50:40.401 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 17.899520985999999, 2024-09-21T15:50:40.401 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064029249999999996, 2024-09-21T15:50:40.401 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:40.401 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:40.401 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:40.401 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:40.401 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:40.401 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155135 2024-09-21T15:50:40.401 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.401 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:40.401 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.402 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:40.402 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449207+0000", 2024-09-21T15:50:40.402 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.402 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.402 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.402 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:40.402 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449207+0000", 2024-09-21T15:50:40.402 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.402 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.402 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.402 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:40.402 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449206+0000", 2024-09-21T15:50:40.403 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:40.403 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.403 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.403 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:40.403 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449209+0000", 2024-09-21T15:50:40.403 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5150000000000001e-06 2024-09-21T15:50:40.403 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.403 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.403 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:40.403 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449215+0000", 2024-09-21T15:50:40.403 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9329999999999998e-06 2024-09-21T15:50:40.403 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.403 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.403 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:40.403 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449220+0000", 2024-09-21T15:50:40.404 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0819999999999998e-06 2024-09-21T15:50:40.404 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.404 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.404 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:40.404 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513232+0000", 2024-09-21T15:50:40.404 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064012052 2024-09-21T15:50:40.404 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.404 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.404 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:40.404 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513236+0000", 2024-09-21T15:50:40.404 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2080000000000002e-06 2024-09-21T15:50:40.404 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.404 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:40.404 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.404 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.405 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.405 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155136 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:40.405 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449224+0000", 2024-09-21T15:50:40.405 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 17.899503421999999, 2024-09-21T15:50:40.405 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064053025999999999, 2024-09-21T15:50:40.405 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:40.405 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:40.405 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:40.405 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:40.405 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:40.405 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155136 2024-09-21T15:50:40.405 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.405 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:40.406 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.406 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:40.406 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449224+0000", 2024-09-21T15:50:40.406 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.406 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.406 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.406 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:40.406 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449224+0000", 2024-09-21T15:50:40.406 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.406 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.406 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.406 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:40.406 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449224+0000", 2024-09-21T15:50:40.407 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:40.407 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.407 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.407 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:40.407 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449226+0000", 2024-09-21T15:50:40.407 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2699999999999999e-06 2024-09-21T15:50:40.407 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.407 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.407 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:40.407 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449233+0000", 2024-09-21T15:50:40.407 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4289999999999998e-06 2024-09-21T15:50:40.407 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.407 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.407 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:40.408 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449238+0000", 2024-09-21T15:50:40.408 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0730000000000004e-06 2024-09-21T15:50:40.408 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.408 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.408 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:40.408 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513269+0000", 2024-09-21T15:50:40.408 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064030999000000005 2024-09-21T15:50:40.408 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.408 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.408 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:40.408 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513277+0000", 2024-09-21T15:50:40.408 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5520000000000005e-06 2024-09-21T15:50:40.408 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.408 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:40.409 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.409 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.409 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.409 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155137 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:40.409 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449263+0000", 2024-09-21T15:50:40.409 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 17.899464290000001, 2024-09-21T15:50:40.409 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.06402803, 2024-09-21T15:50:40.409 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:40.409 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:40.409 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:40.409 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:40.409 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:40.409 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155137 2024-09-21T15:50:40.409 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.409 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:40.410 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.410 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:40.410 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449263+0000", 2024-09-21T15:50:40.410 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.410 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.410 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.410 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:40.410 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449263+0000", 2024-09-21T15:50:40.410 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.410 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.410 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.410 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:40.410 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449262+0000", 2024-09-21T15:50:40.410 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-21T15:50:40.410 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.411 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.411 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:40.411 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449267+0000", 2024-09-21T15:50:40.411 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.933e-06 2024-09-21T15:50:40.411 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.411 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.411 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:40.411 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449279+0000", 2024-09-21T15:50:40.411 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.181e-05 2024-09-21T15:50:40.411 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.411 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.411 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:40.411 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449286+0000", 2024-09-21T15:50:40.411 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4749999999999998e-06 2024-09-21T15:50:40.411 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.412 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.412 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:40.412 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513287+0000", 2024-09-21T15:50:40.412 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064001369000000002 2024-09-21T15:50:40.412 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.412 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.412 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:40.412 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513291+0000", 2024-09-21T15:50:40.412 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3000000000000003e-06 2024-09-21T15:50:40.412 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.412 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:40.412 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.412 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.412 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.412 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155138 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:40.412 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449299+0000", 2024-09-21T15:50:40.413 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 17.899428797999999, 2024-09-21T15:50:40.413 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064028359000000007, 2024-09-21T15:50:40.413 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:40.413 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:40.413 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:40.413 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:40.413 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:40.413 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155138 2024-09-21T15:50:40.413 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.413 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:40.413 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.413 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:40.413 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449299+0000", 2024-09-21T15:50:40.413 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.413 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.414 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.414 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:40.414 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449299+0000", 2024-09-21T15:50:40.414 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.414 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.414 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.414 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:40.414 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449298+0000", 2024-09-21T15:50:40.414 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:40.414 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.414 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.414 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:40.415 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449301+0000", 2024-09-21T15:50:40.415 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0230000000000001e-06 2024-09-21T15:50:40.415 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.415 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.415 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:40.415 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449308+0000", 2024-09-21T15:50:40.415 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3119999999999999e-06 2024-09-21T15:50:40.415 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.415 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.416 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:40.416 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449313+0000", 2024-09-21T15:50:40.416 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8980000000000004e-06 2024-09-21T15:50:40.416 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.416 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.416 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:40.416 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513318+0000", 2024-09-21T15:50:40.416 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064005612000000003 2024-09-21T15:50:40.416 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.416 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.416 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:40.416 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513327+0000", 2024-09-21T15:50:40.416 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0100000000000001e-06 2024-09-21T15:50:40.416 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.417 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:40.417 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.417 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.417 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.417 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155142 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:40.417 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449392+0000", 2024-09-21T15:50:40.417 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 17.899336040000001, 2024-09-21T15:50:40.417 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064012464000000005, 2024-09-21T15:50:40.417 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:40.417 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:40.417 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:40.417 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:40.417 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:40.417 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155142 2024-09-21T15:50:40.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.418 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:40.418 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:40.418 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449392+0000", 2024-09-21T15:50:40.418 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.418 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:40.418 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449392+0000", 2024-09-21T15:50:40.418 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.418 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:40.419 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449391+0000", 2024-09-21T15:50:40.419 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:40.419 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.419 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.419 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:40.419 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449393+0000", 2024-09-21T15:50:40.419 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.26e-06 2024-09-21T15:50:40.419 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.419 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.419 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:40.419 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449399+0000", 2024-09-21T15:50:40.419 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6269999999999997e-06 2024-09-21T15:50:40.419 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.419 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.419 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:40.420 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449403+0000", 2024-09-21T15:50:40.420 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3000000000000003e-06 2024-09-21T15:50:40.420 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.420 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.420 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:40.420 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513396+0000", 2024-09-21T15:50:40.420 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.063992568999999999 2024-09-21T15:50:40.420 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.420 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.420 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:40.420 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513404+0000", 2024-09-21T15:50:40.420 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1100000000000003e-06 2024-09-21T15:50:40.420 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.420 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:40.420 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.421 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.421 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.421 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155143 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:40.421 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449407+0000", 2024-09-21T15:50:40.421 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 17.899320454000001, 2024-09-21T15:50:40.421 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064009916, 2024-09-21T15:50:40.421 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:40.421 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:40.421 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:40.421 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:40.421 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:40.421 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155143 2024-09-21T15:50:40.421 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.421 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:40.421 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.421 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:40.422 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449407+0000", 2024-09-21T15:50:40.422 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.422 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.422 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.422 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:40.422 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449407+0000", 2024-09-21T15:50:40.422 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.422 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.422 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.422 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:40.422 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449407+0000", 2024-09-21T15:50:40.422 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:40.422 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.422 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.423 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:40.423 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449409+0000", 2024-09-21T15:50:40.423 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1090000000000001e-06 2024-09-21T15:50:40.423 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.423 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.423 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:40.423 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449413+0000", 2024-09-21T15:50:40.423 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6650000000000001e-06 2024-09-21T15:50:40.423 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.423 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.423 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:40.423 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449415+0000", 2024-09-21T15:50:40.423 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5459999999999998e-06 2024-09-21T15:50:40.424 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.424 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.424 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:40.424 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513413+0000", 2024-09-21T15:50:40.424 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.063998037999999993 2024-09-21T15:50:40.424 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.424 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.424 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:40.424 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513417+0000", 2024-09-21T15:50:40.424 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8789999999999996e-06 2024-09-21T15:50:40.424 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.424 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:40.424 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.424 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.424 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.424 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155144 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:40.425 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449427+0000", 2024-09-21T15:50:40.425 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 17.899300446000002, 2024-09-21T15:50:40.425 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064022531999999993, 2024-09-21T15:50:40.425 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:40.425 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:40.425 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:40.425 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:40.425 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:40.425 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155144 2024-09-21T15:50:40.425 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.425 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:40.425 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.425 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:40.425 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449427+0000", 2024-09-21T15:50:40.426 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.426 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.426 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.426 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:40.426 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449427+0000", 2024-09-21T15:50:40.426 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.426 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.426 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.426 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:40.426 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449427+0000", 2024-09-21T15:50:40.426 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:40.426 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.426 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.426 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:40.426 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449430+0000", 2024-09-21T15:50:40.427 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3579999999999999e-06 2024-09-21T15:50:40.427 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.427 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.427 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:40.427 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449437+0000", 2024-09-21T15:50:40.427 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4580000000000004e-06 2024-09-21T15:50:40.427 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.427 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.427 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:40.427 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449443+0000", 2024-09-21T15:50:40.427 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5609999999999998e-06 2024-09-21T15:50:40.427 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.427 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.427 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:40.427 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513441+0000", 2024-09-21T15:50:40.428 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.063998136999999997 2024-09-21T15:50:40.428 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.428 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.428 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:40.428 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513450+0000", 2024-09-21T15:50:40.428 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4950000000000008e-06 2024-09-21T15:50:40.428 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.428 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:40.428 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.428 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.428 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:40.428 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:40.429 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:50:40.430 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:40.428+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:40.530 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:40.529+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:40.531 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:40.531 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:40.531 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:50:40.531 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:40.539 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:50:40.631 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:40.629+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:40.641 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:40.642 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:40.642 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:50:40.642 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:50:40.642 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:40.651 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:50:40.735 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:40.734+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:40.805 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:40.805 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:50:40.805 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:50:40.805 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:50:40.805 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.806 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120111 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 321536~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:40.806 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572216+0000", 2024-09-21T15:50:40.806 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 37.231130118000003, 2024-09-21T15:50:40.806 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054657399000000002, 2024-09-21T15:50:40.806 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:40.806 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:40.806 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:40.806 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:40.806 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:40.806 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120111 2024-09-21T15:50:40.806 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.806 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:40.806 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.806 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:40.807 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572216+0000", 2024-09-21T15:50:40.807 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.807 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.807 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.807 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:40.807 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572216+0000", 2024-09-21T15:50:40.807 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.807 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.807 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.807 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:40.807 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572216+0000", 2024-09-21T15:50:40.807 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:40.807 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.807 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.808 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:40.808 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572218+0000", 2024-09-21T15:50:40.808 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1840000000000002e-06 2024-09-21T15:50:40.808 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.808 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.808 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:40.808 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572224+0000", 2024-09-21T15:50:40.808 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7880000000000003e-06 2024-09-21T15:50:40.808 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.808 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.808 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:40.808 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572229+0000", 2024-09-21T15:50:40.808 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8239999999999999e-06 2024-09-21T15:50:40.808 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.808 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.809 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:40.809 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626680+0000", 2024-09-21T15:50:40.809 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054451431000000002 2024-09-21T15:50:40.809 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.809 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.809 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:40.809 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626755+0000", 2024-09-21T15:50:40.809 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4641000000000006e-05 2024-09-21T15:50:40.809 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.809 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.809 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:40.809 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626874+0000", 2024-09-21T15:50:40.809 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00011886 2024-09-21T15:50:40.809 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.809 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:40.810 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.810 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.810 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.810 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120112 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 342016~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:40.810 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572233+0000", 2024-09-21T15:50:40.810 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 37.231113749999999, 2024-09-21T15:50:40.810 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054823997999999999, 2024-09-21T15:50:40.810 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:40.810 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:40.810 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:40.810 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:40.810 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:40.810 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120112 2024-09-21T15:50:40.810 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.811 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:40.811 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.811 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:40.811 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572233+0000", 2024-09-21T15:50:40.811 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.811 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.811 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.811 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:40.811 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572233+0000", 2024-09-21T15:50:40.811 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.811 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.811 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.811 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:40.811 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572232+0000", 2024-09-21T15:50:40.812 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:40.812 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.812 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.812 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:40.812 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572234+0000", 2024-09-21T15:50:40.812 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0660000000000002e-06 2024-09-21T15:50:40.812 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.812 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.812 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:40.812 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572242+0000", 2024-09-21T15:50:40.812 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0430000000000002e-06 2024-09-21T15:50:40.812 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.812 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.812 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:40.812 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572246+0000", 2024-09-21T15:50:40.813 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9080000000000003e-06 2024-09-21T15:50:40.813 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.813 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.813 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:40.813 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626916+0000", 2024-09-21T15:50:40.813 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054669482999999998 2024-09-21T15:50:40.813 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.813 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.813 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:40.813 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626977+0000", 2024-09-21T15:50:40.813 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0597000000000003e-05 2024-09-21T15:50:40.813 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.813 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.813 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:40.813 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627057+0000", 2024-09-21T15:50:40.814 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0258000000000003e-05 2024-09-21T15:50:40.814 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.814 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:40.814 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.814 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.814 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.814 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120113 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 362496~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:40.814 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572253+0000", 2024-09-21T15:50:40.814 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 37.231093764999997, 2024-09-21T15:50:40.814 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.05503044, 2024-09-21T15:50:40.814 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:40.814 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:40.814 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:40.814 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:40.814 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:40.815 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120113 2024-09-21T15:50:40.815 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.815 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:40.815 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.815 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:40.815 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572253+0000", 2024-09-21T15:50:40.815 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.815 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.815 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.815 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:40.815 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572253+0000", 2024-09-21T15:50:40.815 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.815 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.815 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.815 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:40.816 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572252+0000", 2024-09-21T15:50:40.816 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:40.816 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.816 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.816 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:40.816 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572258+0000", 2024-09-21T15:50:40.816 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6169999999999998e-06 2024-09-21T15:50:40.816 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.816 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.816 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:40.816 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572266+0000", 2024-09-21T15:50:40.816 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9410000000000007e-06 2024-09-21T15:50:40.816 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.816 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.816 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:40.816 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572272+0000", 2024-09-21T15:50:40.817 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.241e-06 2024-09-21T15:50:40.817 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.817 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.817 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:40.817 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627087+0000", 2024-09-21T15:50:40.817 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054814850999999998 2024-09-21T15:50:40.817 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.817 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.817 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:40.817 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627162+0000", 2024-09-21T15:50:40.817 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5012999999999995e-05 2024-09-21T15:50:40.817 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.818 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.818 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:40.818 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627283+0000", 2024-09-21T15:50:40.818 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012113 2024-09-21T15:50:40.818 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.818 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:40.818 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.818 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.818 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.818 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120114 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 382976~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:40.818 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572277+0000", 2024-09-21T15:50:40.818 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 37.231069746999999, 2024-09-21T15:50:40.818 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055184395999999997, 2024-09-21T15:50:40.818 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:40.819 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:40.819 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:40.819 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:40.819 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:40.819 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120114 2024-09-21T15:50:40.819 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.819 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:40.819 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.819 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:40.819 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572277+0000", 2024-09-21T15:50:40.819 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.819 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.819 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.819 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:40.819 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572277+0000", 2024-09-21T15:50:40.820 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.820 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.820 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.820 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:40.820 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572276+0000", 2024-09-21T15:50:40.820 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:40.820 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.820 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.820 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:40.820 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572279+0000", 2024-09-21T15:50:40.820 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.373e-06 2024-09-21T15:50:40.820 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.820 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.820 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:40.821 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572285+0000", 2024-09-21T15:50:40.821 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9359999999999996e-06 2024-09-21T15:50:40.821 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.821 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.821 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:40.821 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572289+0000", 2024-09-21T15:50:40.821 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.634e-06 2024-09-21T15:50:40.821 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.821 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.821 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:40.821 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627311+0000", 2024-09-21T15:50:40.821 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055021211 2024-09-21T15:50:40.821 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.821 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.821 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:40.822 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627360+0000", 2024-09-21T15:50:40.822 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9203999999999998e-05 2024-09-21T15:50:40.822 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.822 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.822 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:40.822 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627461+0000", 2024-09-21T15:50:40.822 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000101469 2024-09-21T15:50:40.822 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.822 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:40.822 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.822 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.822 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.822 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120115 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 399360~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:40.822 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572294+0000", 2024-09-21T15:50:40.823 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 37.231052439999999, 2024-09-21T15:50:40.823 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055400092999999997, 2024-09-21T15:50:40.823 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:40.823 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:40.823 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:40.823 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:40.823 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:40.823 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120115 2024-09-21T15:50:40.823 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.823 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:40.823 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.823 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:40.823 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572294+0000", 2024-09-21T15:50:40.823 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.824 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.824 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.824 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:40.824 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572294+0000", 2024-09-21T15:50:40.824 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.824 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.824 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.824 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:40.824 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572294+0000", 2024-09-21T15:50:40.824 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:40.824 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.824 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.824 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:40.824 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572296+0000", 2024-09-21T15:50:40.824 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1100000000000001e-06 2024-09-21T15:50:40.824 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.825 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.825 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:40.825 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572303+0000", 2024-09-21T15:50:40.825 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8580000000000002e-06 2024-09-21T15:50:40.825 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.825 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.825 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:40.825 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572314+0000", 2024-09-21T15:50:40.825 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.101e-05 2024-09-21T15:50:40.825 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.825 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.825 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:40.825 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627496+0000", 2024-09-21T15:50:40.825 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055182522999999997 2024-09-21T15:50:40.826 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.826 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.826 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:40.826 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627568+0000", 2024-09-21T15:50:40.826 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1569999999999994e-05 2024-09-21T15:50:40.826 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.826 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.826 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:40.826 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627694+0000", 2024-09-21T15:50:40.826 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000126372 2024-09-21T15:50:40.826 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.826 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:40.826 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.826 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.827 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.827 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120116 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 419840~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:40.827 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572318+0000", 2024-09-21T15:50:40.827 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 37.231028248999998, 2024-09-21T15:50:40.827 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055539416000000001, 2024-09-21T15:50:40.827 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:40.827 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:40.827 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:40.827 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:40.827 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:40.827 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120116 2024-09-21T15:50:40.827 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.827 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:40.827 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.827 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:40.828 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572318+0000", 2024-09-21T15:50:40.828 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.828 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.828 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.828 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:40.828 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572318+0000", 2024-09-21T15:50:40.828 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.828 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.828 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.828 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:40.828 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572318+0000", 2024-09-21T15:50:40.828 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:40.828 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.828 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.829 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:40.829 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572322+0000", 2024-09-21T15:50:40.829 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4839999999999998e-06 2024-09-21T15:50:40.829 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.829 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.829 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:40.829 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572329+0000", 2024-09-21T15:50:40.829 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0959999999999997e-06 2024-09-21T15:50:40.829 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.829 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.829 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:40.829 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572333+0000", 2024-09-21T15:50:40.830 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9359999999999998e-06 2024-09-21T15:50:40.830 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.830 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.830 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:40.830 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627726+0000", 2024-09-21T15:50:40.830 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055392829999999997 2024-09-21T15:50:40.830 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.830 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.830 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:40.830 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627779+0000", 2024-09-21T15:50:40.830 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2185999999999998e-05 2024-09-21T15:50:40.830 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.830 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.830 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:40.830 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627858+0000", 2024-09-21T15:50:40.831 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9190999999999994e-05 2024-09-21T15:50:40.831 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.831 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:40.831 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.831 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.831 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.831 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120117 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 440320~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:40.831 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572338+0000", 2024-09-21T15:50:40.831 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 37.231008936000002, 2024-09-21T15:50:40.831 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055680888999999997, 2024-09-21T15:50:40.831 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:40.831 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:40.831 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:40.831 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:40.832 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:40.832 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120117 2024-09-21T15:50:40.832 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.832 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:40.832 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.832 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:40.832 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572338+0000", 2024-09-21T15:50:40.832 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.832 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.832 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.832 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:40.832 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572338+0000", 2024-09-21T15:50:40.832 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.832 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.833 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.833 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:40.833 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572337+0000", 2024-09-21T15:50:40.833 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:40.833 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.833 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.833 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:40.833 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572339+0000", 2024-09-21T15:50:40.833 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9379999999999999e-06 2024-09-21T15:50:40.833 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.833 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.833 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:40.833 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572349+0000", 2024-09-21T15:50:40.833 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7780000000000002e-06 2024-09-21T15:50:40.834 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.834 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.834 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:40.834 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572354+0000", 2024-09-21T15:50:40.834 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8740000000000003e-06 2024-09-21T15:50:40.834 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.834 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.834 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:40.834 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627885+0000", 2024-09-21T15:50:40.834 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055531448999999997 2024-09-21T15:50:40.834 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.834 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.834 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:40.835 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627935+0000", 2024-09-21T15:50:40.835 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9348999999999998e-05 2024-09-21T15:50:40.835 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.835 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.835 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:40.835 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628018+0000", 2024-09-21T15:50:40.835 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3851999999999996e-05 2024-09-21T15:50:40.835 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.835 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:40.835 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.835 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.835 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.835 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120118 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 460800~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:40.836 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572359+0000", 2024-09-21T15:50:40.836 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 37.230987694, 2024-09-21T15:50:40.836 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055951714, 2024-09-21T15:50:40.836 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:40.836 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:40.836 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:40.836 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:40.836 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:40.836 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120118 2024-09-21T15:50:40.836 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.836 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:40.836 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.836 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:40.837 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572359+0000", 2024-09-21T15:50:40.837 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.837 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.837 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.837 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:40.837 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572359+0000", 2024-09-21T15:50:40.837 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.837 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.837 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.837 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:40.837 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572358+0000", 2024-09-21T15:50:40.837 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:40.837 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.838 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.838 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:40.838 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572361+0000", 2024-09-21T15:50:40.838 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.26e-06 2024-09-21T15:50:40.838 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.838 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.838 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:40.838 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572367+0000", 2024-09-21T15:50:40.838 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3640000000000004e-06 2024-09-21T15:50:40.838 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.838 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.838 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:40.838 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572376+0000", 2024-09-21T15:50:40.838 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2310000000000002e-06 2024-09-21T15:50:40.839 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.839 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.839 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:40.839 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628096+0000", 2024-09-21T15:50:40.839 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055719369999999997 2024-09-21T15:50:40.839 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.839 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.839 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:40.839 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628178+0000", 2024-09-21T15:50:40.839 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2667e-05 2024-09-21T15:50:40.839 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.839 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.839 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:40.839 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628311+0000", 2024-09-21T15:50:40.839 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000132175 2024-09-21T15:50:40.840 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.840 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:40.840 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.840 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.840 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.840 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120119 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 481280~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:40.840 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572381+0000", 2024-09-21T15:50:40.840 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 37.230965249999997, 2024-09-21T15:50:40.840 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056088791999999998, 2024-09-21T15:50:40.840 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:40.840 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:40.840 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:40.840 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:40.840 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:40.840 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120119 2024-09-21T15:50:40.841 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.841 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:40.841 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.841 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:40.841 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572381+0000", 2024-09-21T15:50:40.841 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.841 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.841 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.841 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:40.841 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572381+0000", 2024-09-21T15:50:40.841 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.841 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.841 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.841 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:40.841 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572381+0000", 2024-09-21T15:50:40.842 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:40.842 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.842 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.842 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:40.842 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572383+0000", 2024-09-21T15:50:40.842 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0619999999999999e-06 2024-09-21T15:50:40.842 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.842 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.842 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:40.842 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572391+0000", 2024-09-21T15:50:40.842 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8769999999999993e-06 2024-09-21T15:50:40.842 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.842 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.842 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:40.842 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572399+0000", 2024-09-21T15:50:40.843 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6240000000000008e-06 2024-09-21T15:50:40.843 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.843 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.843 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:40.843 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628344+0000", 2024-09-21T15:50:40.843 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055944828000000002 2024-09-21T15:50:40.843 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.843 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.843 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:40.843 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628393+0000", 2024-09-21T15:50:40.843 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8653000000000003e-05 2024-09-21T15:50:40.843 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.843 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.843 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:40.843 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628470+0000", 2024-09-21T15:50:40.844 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7121999999999998e-05 2024-09-21T15:50:40.844 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.844 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:40.844 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.844 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.844 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.844 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120120 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 501760~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:40.844 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572418+0000", 2024-09-21T15:50:40.844 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 37.230928751999997, 2024-09-21T15:50:40.844 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056209138999999998, 2024-09-21T15:50:40.844 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:40.844 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:40.844 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:40.844 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:40.845 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:40.845 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120120 2024-09-21T15:50:40.845 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.845 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:40.845 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.845 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:40.845 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572418+0000", 2024-09-21T15:50:40.845 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.845 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.845 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.845 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:40.845 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572418+0000", 2024-09-21T15:50:40.845 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.845 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.846 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.846 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:40.846 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572417+0000", 2024-09-21T15:50:40.846 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:40.846 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.846 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.846 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:40.846 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572420+0000", 2024-09-21T15:50:40.846 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5270000000000001e-06 2024-09-21T15:50:40.846 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.846 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.846 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:40.846 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572426+0000", 2024-09-21T15:50:40.847 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3380000000000001e-06 2024-09-21T15:50:40.847 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.847 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.847 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:40.847 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572431+0000", 2024-09-21T15:50:40.847 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.865e-06 2024-09-21T15:50:40.847 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.847 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.847 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:40.847 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628500+0000", 2024-09-21T15:50:40.847 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056068917000000003 2024-09-21T15:50:40.847 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.847 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.847 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:40.848 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628542+0000", 2024-09-21T15:50:40.848 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1761999999999998e-05 2024-09-21T15:50:40.848 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.848 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.848 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:40.848 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628627+0000", 2024-09-21T15:50:40.848 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5247999999999994e-05 2024-09-21T15:50:40.848 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.849 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:40.849 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.849 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.849 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.849 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120121 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 522240~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:40.849 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572630+0000", 2024-09-21T15:50:40.849 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 37.230716366000003, 2024-09-21T15:50:40.850 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056207209000000001, 2024-09-21T15:50:40.850 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:40.850 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:40.850 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:40.850 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:40.850 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:40.850 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120121 2024-09-21T15:50:40.851 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.851 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:40.851 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.851 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:40.851 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572630+0000", 2024-09-21T15:50:40.851 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.851 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.852 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.852 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:40.852 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572630+0000", 2024-09-21T15:50:40.852 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.852 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.852 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.852 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:40.852 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572435+0000", 2024-09-21T15:50:40.852 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998045 2024-09-21T15:50:40.853 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.853 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.853 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:40.853 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572633+0000", 2024-09-21T15:50:40.853 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000198786 2024-09-21T15:50:40.853 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.853 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.853 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:40.854 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572641+0000", 2024-09-21T15:50:40.854 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9449999999999993e-06 2024-09-21T15:50:40.854 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.854 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.854 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:40.854 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572648+0000", 2024-09-21T15:50:40.855 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1929999999999998e-06 2024-09-21T15:50:40.855 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.855 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.855 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:40.855 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628669+0000", 2024-09-21T15:50:40.855 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056021690999999998 2024-09-21T15:50:40.855 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.856 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.856 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:40.856 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628751+0000", 2024-09-21T15:50:40.856 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1710000000000002e-05 2024-09-21T15:50:40.856 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.856 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.856 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:40.856 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628837+0000", 2024-09-21T15:50:40.856 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6383000000000003e-05 2024-09-21T15:50:40.856 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.856 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:40.856 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.856 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.857 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.857 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120122 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 538624~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:40.857 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.573471+0000", 2024-09-21T15:50:40.857 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 37.229875120999999, 2024-09-21T15:50:40.857 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055563935000000002, 2024-09-21T15:50:40.857 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:40.857 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:40.857 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:40.857 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:40.857 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:40.857 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120122 2024-09-21T15:50:40.857 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.857 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:40.858 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.858 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:40.858 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573471+0000", 2024-09-21T15:50:40.858 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.858 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.858 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.858 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:40.858 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573471+0000", 2024-09-21T15:50:40.858 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.858 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.858 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.858 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:40.858 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572653+0000", 2024-09-21T15:50:40.859 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9991813 2024-09-21T15:50:40.859 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.859 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.859 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:40.859 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573475+0000", 2024-09-21T15:50:40.859 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00082196500000000002 2024-09-21T15:50:40.859 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.859 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.859 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:40.859 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573483+0000", 2024-09-21T15:50:40.859 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5939999999999994e-06 2024-09-21T15:50:40.859 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.859 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.859 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:40.859 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573489+0000", 2024-09-21T15:50:40.859 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1689999999999997e-06 2024-09-21T15:50:40.860 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.860 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.860 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:40.860 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628882+0000", 2024-09-21T15:50:40.860 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055392230000000001 2024-09-21T15:50:40.860 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.860 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.860 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:40.860 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628934+0000", 2024-09-21T15:50:40.860 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1870000000000003e-05 2024-09-21T15:50:40.860 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.860 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.860 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:40.860 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629035+0000", 2024-09-21T15:50:40.860 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000101753 2024-09-21T15:50:40.860 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.861 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:40.861 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.861 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.861 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.861 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120123 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 559104~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:40.861 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.573495+0000", 2024-09-21T15:50:40.861 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 37.229851422000003, 2024-09-21T15:50:40.861 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055704765000000003, 2024-09-21T15:50:40.861 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:40.861 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:40.861 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:40.861 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:40.861 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:40.861 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120123 2024-09-21T15:50:40.861 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.861 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:40.862 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.862 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:40.862 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573495+0000", 2024-09-21T15:50:40.862 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.862 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.862 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.862 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:40.862 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573495+0000", 2024-09-21T15:50:40.862 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.862 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.862 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.862 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:40.862 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573495+0000", 2024-09-21T15:50:40.862 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:40.862 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.863 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.863 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:40.863 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573497+0000", 2024-09-21T15:50:40.863 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3420000000000002e-06 2024-09-21T15:50:40.863 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.863 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.863 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:40.863 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573503+0000", 2024-09-21T15:50:40.863 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7030000000000003e-06 2024-09-21T15:50:40.863 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.863 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.863 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:40.863 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573511+0000", 2024-09-21T15:50:40.864 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0790000000000001e-06 2024-09-21T15:50:40.864 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.864 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.864 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:40.864 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629067+0000", 2024-09-21T15:50:40.864 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055556573999999997 2024-09-21T15:50:40.864 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.864 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.864 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:40.864 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629118+0000", 2024-09-21T15:50:40.864 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1053000000000001e-05 2024-09-21T15:50:40.864 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.864 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.864 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:40.864 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629200+0000", 2024-09-21T15:50:40.865 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1452999999999994e-05 2024-09-21T15:50:40.865 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.865 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:40.865 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.865 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.865 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.865 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120124 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 579584~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:40.865 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.574014+0000", 2024-09-21T15:50:40.865 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 37.229332382999999, 2024-09-21T15:50:40.865 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055440284999999999, 2024-09-21T15:50:40.865 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:40.865 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:40.865 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:40.865 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:40.866 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:40.866 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120124 2024-09-21T15:50:40.866 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.866 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:40.866 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.866 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:40.866 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574014+0000", 2024-09-21T15:50:40.866 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.866 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.866 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.866 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:40.866 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574014+0000", 2024-09-21T15:50:40.867 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.867 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.867 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.867 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:40.867 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573515+0000", 2024-09-21T15:50:40.867 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9995008 2024-09-21T15:50:40.867 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.867 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.867 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:40.867 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574017+0000", 2024-09-21T15:50:40.867 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00050241100000000003 2024-09-21T15:50:40.867 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.867 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.867 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:40.868 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574027+0000", 2024-09-21T15:50:40.868 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4700000000000008e-06 2024-09-21T15:50:40.868 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.868 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.868 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:40.868 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574033+0000", 2024-09-21T15:50:40.868 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1489999999999999e-06 2024-09-21T15:50:40.868 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.868 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.868 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:40.868 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629238+0000", 2024-09-21T15:50:40.868 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055204880999999997 2024-09-21T15:50:40.868 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.868 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.868 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:40.869 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629317+0000", 2024-09-21T15:50:40.869 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8942000000000004e-05 2024-09-21T15:50:40.869 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.869 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.869 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:40.869 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629454+0000", 2024-09-21T15:50:40.869 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013767099999999999 2024-09-21T15:50:40.869 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.869 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:40.869 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.869 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.869 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.869 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120125 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 600064~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:40.869 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.574482+0000", 2024-09-21T15:50:40.870 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 37.228864084000001, 2024-09-21T15:50:40.870 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055243608, 2024-09-21T15:50:40.870 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:40.870 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:40.870 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:40.870 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:40.870 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:40.870 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120125 2024-09-21T15:50:40.870 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.870 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:40.870 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.870 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:40.870 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574482+0000", 2024-09-21T15:50:40.870 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.871 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.871 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.871 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:40.871 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574482+0000", 2024-09-21T15:50:40.871 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.871 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.871 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.871 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:40.871 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574038+0000", 2024-09-21T15:50:40.871 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9995556 2024-09-21T15:50:40.871 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.871 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.871 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:40.872 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574497+0000", 2024-09-21T15:50:40.872 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00045854400000000002 2024-09-21T15:50:40.872 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.872 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.872 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:40.872 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574509+0000", 2024-09-21T15:50:40.872 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2486e-05 2024-09-21T15:50:40.872 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.872 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.872 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:40.872 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574516+0000", 2024-09-21T15:50:40.872 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2269999999999996e-06 2024-09-21T15:50:40.872 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.872 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.872 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:40.872 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629497+0000", 2024-09-21T15:50:40.873 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054980785999999997 2024-09-21T15:50:40.873 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.873 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.873 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:40.873 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629580+0000", 2024-09-21T15:50:40.873 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2528000000000007e-05 2024-09-21T15:50:40.873 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.873 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.873 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:40.873 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629726+0000", 2024-09-21T15:50:40.873 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014630399999999999 2024-09-21T15:50:40.873 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.873 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:40.873 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.873 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.874 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.874 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120126 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 620544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:40.874 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.575315+0000", 2024-09-21T15:50:40.874 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 37.228031053999999, 2024-09-21T15:50:40.874 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054709559999999997, 2024-09-21T15:50:40.874 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:40.874 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:40.874 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:40.874 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:40.874 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:40.874 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120126 2024-09-21T15:50:40.874 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.874 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:40.874 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.875 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:40.875 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575315+0000", 2024-09-21T15:50:40.875 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.875 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.875 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.875 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:40.875 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575315+0000", 2024-09-21T15:50:40.875 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.875 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.875 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.875 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:40.875 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574522+0000", 2024-09-21T15:50:40.875 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9992065 2024-09-21T15:50:40.875 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.875 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.876 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:40.876 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575319+0000", 2024-09-21T15:50:40.876 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00079626800000000004 2024-09-21T15:50:40.876 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.876 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.876 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:40.876 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575329+0000", 2024-09-21T15:50:40.876 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.028e-05 2024-09-21T15:50:40.876 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.876 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.876 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:40.876 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575335+0000", 2024-09-21T15:50:40.876 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9290000000000003e-06 2024-09-21T15:50:40.876 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.876 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.876 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:40.877 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629806+0000", 2024-09-21T15:50:40.877 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054471558000000003 2024-09-21T15:50:40.877 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.877 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.877 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:40.877 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629885+0000", 2024-09-21T15:50:40.877 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8563e-05 2024-09-21T15:50:40.877 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.877 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.877 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:40.877 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630025+0000", 2024-09-21T15:50:40.877 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014019 2024-09-21T15:50:40.877 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.877 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:40.877 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.877 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.878 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.878 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120127 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 641024~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:40.878 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.575340+0000", 2024-09-21T15:50:40.878 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 37.228006346999997, 2024-09-21T15:50:40.878 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054895589000000002, 2024-09-21T15:50:40.878 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:40.878 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:40.878 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:40.878 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:40.878 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:40.878 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120127 2024-09-21T15:50:40.878 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.878 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:40.878 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.879 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:40.879 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575340+0000", 2024-09-21T15:50:40.879 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.879 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.879 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.879 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:40.879 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575340+0000", 2024-09-21T15:50:40.879 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.879 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.879 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.879 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:40.879 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575340+0000", 2024-09-21T15:50:40.879 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:40.879 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.879 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.880 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:40.880 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575342+0000", 2024-09-21T15:50:40.880 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5600000000000001e-06 2024-09-21T15:50:40.880 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.880 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.880 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:40.880 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575351+0000", 2024-09-21T15:50:40.880 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2349999999999999e-06 2024-09-21T15:50:40.880 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.880 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.880 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:40.880 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575356+0000", 2024-09-21T15:50:40.880 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4600000000000002e-06 2024-09-21T15:50:40.880 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.880 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.880 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:40.880 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630053+0000", 2024-09-21T15:50:40.881 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054697190999999999 2024-09-21T15:50:40.881 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.881 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.881 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:40.881 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630128+0000", 2024-09-21T15:50:40.881 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4777000000000006e-05 2024-09-21T15:50:40.881 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.881 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.881 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:40.881 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630236+0000", 2024-09-21T15:50:40.881 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000107837 2024-09-21T15:50:40.881 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.881 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:40.881 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.882 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.882 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.882 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120128 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 657408~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:40.882 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.575461+0000", 2024-09-21T15:50:40.882 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 37.227885768, 2024-09-21T15:50:40.882 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054945121999999999, 2024-09-21T15:50:40.882 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:40.882 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:40.882 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:40.882 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:40.882 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:40.883 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120128 2024-09-21T15:50:40.883 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.883 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:40.883 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.883 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:40.883 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575461+0000", 2024-09-21T15:50:40.883 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.883 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.883 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.883 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:40.883 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575461+0000", 2024-09-21T15:50:40.883 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.883 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.883 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.884 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:40.884 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575360+0000", 2024-09-21T15:50:40.884 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998994 2024-09-21T15:50:40.884 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.884 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.884 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:40.884 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575464+0000", 2024-09-21T15:50:40.884 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000103397 2024-09-21T15:50:40.884 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.884 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.884 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:40.884 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575472+0000", 2024-09-21T15:50:40.884 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3189999999999995e-06 2024-09-21T15:50:40.884 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.885 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.885 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:40.885 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575478+0000", 2024-09-21T15:50:40.885 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2779999999999999e-06 2024-09-21T15:50:40.885 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.885 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.885 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:40.885 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630268+0000", 2024-09-21T15:50:40.885 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054789545000000002 2024-09-21T15:50:40.885 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.886 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.886 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:40.886 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630315+0000", 2024-09-21T15:50:40.886 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7244999999999999e-05 2024-09-21T15:50:40.886 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.886 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.886 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:40.886 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630406+0000", 2024-09-21T15:50:40.886 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0811999999999997e-05 2024-09-21T15:50:40.886 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.886 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:40.886 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.886 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.886 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.887 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120129 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 677888~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:40.887 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.576016+0000", 2024-09-21T15:50:40.887 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 37.227330223999999, 2024-09-21T15:50:40.887 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054602823000000002, 2024-09-21T15:50:40.887 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:40.887 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:40.887 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:40.887 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:40.887 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:40.887 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120129 2024-09-21T15:50:40.887 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.887 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:40.887 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.887 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:40.888 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576016+0000", 2024-09-21T15:50:40.888 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.888 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.888 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.888 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:40.888 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576016+0000", 2024-09-21T15:50:40.888 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.888 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.888 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.888 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:40.888 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575483+0000", 2024-09-21T15:50:40.888 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9994669 2024-09-21T15:50:40.888 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.888 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.888 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:40.888 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576020+0000", 2024-09-21T15:50:40.889 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00053657900000000003 2024-09-21T15:50:40.889 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.889 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.889 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:40.889 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576030+0000", 2024-09-21T15:50:40.889 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0326000000000001e-05 2024-09-21T15:50:40.889 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.889 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.889 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:40.889 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576037+0000", 2024-09-21T15:50:40.889 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6490000000000002e-06 2024-09-21T15:50:40.889 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.889 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.889 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:40.890 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630446+0000", 2024-09-21T15:50:40.890 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054409698999999999 2024-09-21T15:50:40.890 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.890 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.890 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:40.890 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630524+0000", 2024-09-21T15:50:40.890 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7422999999999994e-05 2024-09-21T15:50:40.890 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.890 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.890 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:40.890 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630619+0000", 2024-09-21T15:50:40.890 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5332000000000001e-05 2024-09-21T15:50:40.890 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.890 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:40.890 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.891 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.891 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.891 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120130 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 698368~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:40.891 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.576043+0000", 2024-09-21T15:50:40.891 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 37.227303853000002, 2024-09-21T15:50:40.891 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054781798, 2024-09-21T15:50:40.891 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:40.891 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:40.891 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:40.891 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:40.891 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:40.891 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120130 2024-09-21T15:50:40.891 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.891 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:40.891 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.892 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:40.892 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576043+0000", 2024-09-21T15:50:40.892 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.892 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.892 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.892 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:40.892 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576043+0000", 2024-09-21T15:50:40.892 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:40.892 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.892 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.892 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:40.892 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576042+0000", 2024-09-21T15:50:40.892 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:40.892 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.892 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.893 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:40.893 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576044+0000", 2024-09-21T15:50:40.893 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0999999999999998e-06 2024-09-21T15:50:40.893 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.893 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.893 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:40.893 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576051+0000", 2024-09-21T15:50:40.893 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4779999999999996e-06 2024-09-21T15:50:40.893 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.893 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.893 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:40.893 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576058+0000", 2024-09-21T15:50:40.893 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2180000000000002e-06 2024-09-21T15:50:40.893 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.893 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.893 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:40.894 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630661+0000", 2024-09-21T15:50:40.894 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054603240999999997 2024-09-21T15:50:40.894 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.894 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.894 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:40.894 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630747+0000", 2024-09-21T15:50:40.894 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6046999999999998e-05 2024-09-21T15:50:40.894 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:40.894 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:40.894 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:40.894 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630824+0000", 2024-09-21T15:50:40.894 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7119999999999993e-05 2024-09-21T15:50:40.894 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.894 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:40.895 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.895 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:40.895 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:40.895 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:40.896 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T15:50:40.896 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:40.895+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:40.996 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:40.995+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:41.038 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:41.038 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:41.038 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:50:41.039 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:41.047 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T15:50:41.080 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T15:50:41.131 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:41.131+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:41.190 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:41.190 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:41.190 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:50:41.190 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:50:41.190 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:41.199 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T15:50:41.199 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:41.197+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:41.299 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:41.298+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:41.343 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:41.343 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:50:41.343 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:50:41.343 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:50:41.343 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.343 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63140 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 876544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:41.343 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.133649+0000", 2024-09-21T15:50:41.343 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 64.207406562000003, 2024-09-21T15:50:41.343 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053950201000000003, 2024-09-21T15:50:41.343 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:41.343 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:41.344 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:41.344 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:41.344 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:41.344 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63140 2024-09-21T15:50:41.344 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.344 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:41.344 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.344 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:41.344 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133649+0000", 2024-09-21T15:50:41.344 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:41.344 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.344 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.344 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:41.344 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133649+0000", 2024-09-21T15:50:41.345 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:41.345 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.345 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.345 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:41.345 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133648+0000", 2024-09-21T15:50:41.345 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:41.345 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.345 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.345 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:41.345 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133651+0000", 2024-09-21T15:50:41.345 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1549999999999999e-06 2024-09-21T15:50:41.345 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.345 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.345 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:41.346 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133660+0000", 2024-09-21T15:50:41.346 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3899999999999993e-06 2024-09-21T15:50:41.346 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.346 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.346 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:41.346 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133669+0000", 2024-09-21T15:50:41.346 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3219999999999999e-06 2024-09-21T15:50:41.346 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.346 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.346 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:41.346 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187409+0000", 2024-09-21T15:50:41.346 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053740257 2024-09-21T15:50:41.346 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.346 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.347 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:41.347 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187492+0000", 2024-09-21T15:50:41.347 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2409000000000003e-05 2024-09-21T15:50:41.347 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.347 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.347 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:41.347 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187599+0000", 2024-09-21T15:50:41.347 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000107334 2024-09-21T15:50:41.347 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:41.347 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:41.347 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:41.347 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.347 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.348 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63141 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 884736~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:41.348 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.133843+0000", 2024-09-21T15:50:41.348 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 64.207212443000003, 2024-09-21T15:50:41.348 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053954015000000001, 2024-09-21T15:50:41.348 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:41.348 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:41.348 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:41.348 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:41.348 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:41.348 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63141 2024-09-21T15:50:41.348 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.348 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:41.348 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.348 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:41.348 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133843+0000", 2024-09-21T15:50:41.349 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:41.349 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.349 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.349 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:41.349 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133843+0000", 2024-09-21T15:50:41.349 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:41.349 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.349 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.349 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:41.349 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133673+0000", 2024-09-21T15:50:41.349 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998302 2024-09-21T15:50:41.349 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.349 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.349 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:41.349 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133846+0000", 2024-09-21T15:50:41.350 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000172762 2024-09-21T15:50:41.350 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.350 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.350 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:41.350 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133855+0000", 2024-09-21T15:50:41.350 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2940000000000006e-06 2024-09-21T15:50:41.350 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.350 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.350 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:41.350 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133863+0000", 2024-09-21T15:50:41.350 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9740000000000002e-06 2024-09-21T15:50:41.350 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.350 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.350 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:41.351 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187630+0000", 2024-09-21T15:50:41.351 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053767211000000002 2024-09-21T15:50:41.351 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.351 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.351 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:41.351 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187707+0000", 2024-09-21T15:50:41.351 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7446000000000004e-05 2024-09-21T15:50:41.351 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.351 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.351 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:41.351 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187797+0000", 2024-09-21T15:50:41.351 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.9858000000000006e-05 2024-09-21T15:50:41.351 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:41.352 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:41.352 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:41.352 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.352 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.352 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63142 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 892928~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:41.352 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134046+0000", 2024-09-21T15:50:41.352 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 64.207009501000002, 2024-09-21T15:50:41.352 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054017849999999999, 2024-09-21T15:50:41.352 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:41.352 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:41.352 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:41.352 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:41.352 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:41.353 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63142 2024-09-21T15:50:41.353 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.353 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:41.353 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.353 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:41.353 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134046+0000", 2024-09-21T15:50:41.353 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:41.353 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.353 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.353 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:41.353 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134046+0000", 2024-09-21T15:50:41.353 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:41.353 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.353 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.354 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:41.354 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133867+0000", 2024-09-21T15:50:41.354 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998217 2024-09-21T15:50:41.354 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.354 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.354 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:41.354 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134049+0000", 2024-09-21T15:50:41.354 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000181695 2024-09-21T15:50:41.354 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.354 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.354 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:41.354 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134057+0000", 2024-09-21T15:50:41.354 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.303e-06 2024-09-21T15:50:41.354 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.354 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.355 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:41.355 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134063+0000", 2024-09-21T15:50:41.355 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9959999999999999e-06 2024-09-21T15:50:41.355 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.355 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.355 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:41.355 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187849+0000", 2024-09-21T15:50:41.355 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053785571999999997 2024-09-21T15:50:41.355 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.355 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.355 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:41.355 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187954+0000", 2024-09-21T15:50:41.355 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010524899999999999 2024-09-21T15:50:41.355 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.355 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.356 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:41.356 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188064+0000", 2024-09-21T15:50:41.356 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000109549 2024-09-21T15:50:41.356 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:41.356 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:41.356 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:41.356 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.356 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.356 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63143 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 901120~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:41.356 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134239+0000", 2024-09-21T15:50:41.356 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 64.206816907999993, 2024-09-21T15:50:41.356 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054026058000000002, 2024-09-21T15:50:41.356 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:41.356 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:41.356 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:41.357 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:41.357 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:41.357 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63143 2024-09-21T15:50:41.357 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.357 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:41.357 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.357 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:41.357 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134239+0000", 2024-09-21T15:50:41.357 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:41.357 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.357 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.357 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:41.357 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134239+0000", 2024-09-21T15:50:41.357 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:41.357 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.358 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.358 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:41.358 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134068+0000", 2024-09-21T15:50:41.358 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998298 2024-09-21T15:50:41.358 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.358 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.358 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:41.358 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134242+0000", 2024-09-21T15:50:41.358 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00017414199999999999 2024-09-21T15:50:41.358 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.358 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.358 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:41.358 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134264+0000", 2024-09-21T15:50:41.358 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1552e-05 2024-09-21T15:50:41.359 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.359 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.359 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:41.359 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134270+0000", 2024-09-21T15:50:41.359 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6939999999999998e-06 2024-09-21T15:50:41.359 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.359 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.359 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:41.359 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188108+0000", 2024-09-21T15:50:41.359 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.05383719 2024-09-21T15:50:41.359 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.359 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.359 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:41.359 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188186+0000", 2024-09-21T15:50:41.359 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8318000000000005e-05 2024-09-21T15:50:41.360 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.360 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.360 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:41.360 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188265+0000", 2024-09-21T15:50:41.360 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8597000000000007e-05 2024-09-21T15:50:41.360 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:41.360 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:41.360 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:41.360 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.360 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.360 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63144 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 909312~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:41.360 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134432+0000", 2024-09-21T15:50:41.360 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 64.206623351999994, 2024-09-21T15:50:41.360 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053968953, 2024-09-21T15:50:41.361 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:41.361 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:41.361 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:41.361 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:41.361 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:41.361 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63144 2024-09-21T15:50:41.361 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.361 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:41.361 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.361 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:41.361 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134432+0000", 2024-09-21T15:50:41.361 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:41.361 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.361 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.362 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:41.362 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134432+0000", 2024-09-21T15:50:41.362 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:41.362 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.362 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.362 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:41.362 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134276+0000", 2024-09-21T15:50:41.362 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998441 2024-09-21T15:50:41.362 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.362 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.362 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:41.362 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134436+0000", 2024-09-21T15:50:41.362 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000159591 2024-09-21T15:50:41.363 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.363 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.363 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:41.363 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134445+0000", 2024-09-21T15:50:41.363 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6780000000000006e-06 2024-09-21T15:50:41.363 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.363 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.363 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:41.363 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134452+0000", 2024-09-21T15:50:41.363 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6830000000000008e-06 2024-09-21T15:50:41.363 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.363 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.363 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:41.363 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188282+0000", 2024-09-21T15:50:41.364 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053829252000000001 2024-09-21T15:50:41.364 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.364 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.364 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:41.364 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188328+0000", 2024-09-21T15:50:41.364 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6734999999999999e-05 2024-09-21T15:50:41.364 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.364 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.364 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:41.364 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188401+0000", 2024-09-21T15:50:41.364 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2731000000000005e-05 2024-09-21T15:50:41.364 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:41.364 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:41.364 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:41.364 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.365 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.365 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63145 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 860160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:41.365 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134736+0000", 2024-09-21T15:50:41.365 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 64.206319566999994, 2024-09-21T15:50:41.365 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053865960999999997, 2024-09-21T15:50:41.365 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:41.365 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:41.365 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:41.365 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:41.365 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:41.365 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63145 2024-09-21T15:50:41.365 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.365 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:41.365 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.366 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:41.366 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134736+0000", 2024-09-21T15:50:41.366 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:41.366 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.366 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.366 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:41.366 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134736+0000", 2024-09-21T15:50:41.366 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:41.366 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.366 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.366 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:41.366 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134458+0000", 2024-09-21T15:50:41.366 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999722 2024-09-21T15:50:41.366 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.367 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.367 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:41.367 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134741+0000", 2024-09-21T15:50:41.367 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00028245599999999999 2024-09-21T15:50:41.367 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.367 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.367 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:41.367 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134753+0000", 2024-09-21T15:50:41.367 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2548000000000001e-05 2024-09-21T15:50:41.367 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.367 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.367 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:41.367 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134760+0000", 2024-09-21T15:50:41.367 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4189999999999996e-06 2024-09-21T15:50:41.367 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.368 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.368 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:41.368 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188426+0000", 2024-09-21T15:50:41.368 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053665894999999998 2024-09-21T15:50:41.368 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.368 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.368 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:41.368 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188502+0000", 2024-09-21T15:50:41.368 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5791000000000007e-05 2024-09-21T15:50:41.368 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.368 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.368 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:41.368 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188602+0000", 2024-09-21T15:50:41.368 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9667999999999995e-05 2024-09-21T15:50:41.368 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:41.369 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:41.369 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:41.369 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.369 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.369 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63148 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 933888~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:41.369 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.135289+0000", 2024-09-21T15:50:41.369 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 64.205766612000005, 2024-09-21T15:50:41.369 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053948488000000003, 2024-09-21T15:50:41.369 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:41.369 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:41.369 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:41.369 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:41.369 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:41.369 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63148 2024-09-21T15:50:41.370 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.370 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:41.370 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.370 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:41.370 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135289+0000", 2024-09-21T15:50:41.370 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:41.370 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.370 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.370 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:41.370 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135289+0000", 2024-09-21T15:50:41.370 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:41.370 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.370 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.371 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:41.371 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135288+0000", 2024-09-21T15:50:41.371 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:41.371 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.371 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.371 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:41.371 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135296+0000", 2024-09-21T15:50:41.371 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5399999999999998e-06 2024-09-21T15:50:41.371 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.371 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.371 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:41.371 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135303+0000", 2024-09-21T15:50:41.371 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0500000000000003e-06 2024-09-21T15:50:41.371 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.372 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.372 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:41.372 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135308+0000", 2024-09-21T15:50:41.372 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9749999999999998e-06 2024-09-21T15:50:41.372 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.372 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.372 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:41.372 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189066+0000", 2024-09-21T15:50:41.372 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053758501 2024-09-21T15:50:41.372 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.372 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.372 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:41.372 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189145+0000", 2024-09-21T15:50:41.372 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9017000000000006e-05 2024-09-21T15:50:41.373 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.373 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.373 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:41.373 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189237+0000", 2024-09-21T15:50:41.373 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1906999999999999e-05 2024-09-21T15:50:41.373 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:41.373 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:41.373 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:41.373 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.373 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.373 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63149 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 942080~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:41.373 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.135430+0000", 2024-09-21T15:50:41.373 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 64.205625283000003, 2024-09-21T15:50:41.373 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054005434999999997, 2024-09-21T15:50:41.373 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:41.374 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:41.374 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:41.374 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:41.374 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:41.374 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63149 2024-09-21T15:50:41.374 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.374 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:41.374 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.374 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:41.374 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135430+0000", 2024-09-21T15:50:41.374 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:41.374 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.374 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.374 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:41.375 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135430+0000", 2024-09-21T15:50:41.375 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:41.375 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.375 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.375 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:41.375 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135313+0000", 2024-09-21T15:50:41.375 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998827 2024-09-21T15:50:41.375 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.375 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.375 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:41.375 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135434+0000", 2024-09-21T15:50:41.375 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000120876 2024-09-21T15:50:41.375 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.376 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.376 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:41.376 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135443+0000", 2024-09-21T15:50:41.376 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3340000000000008e-06 2024-09-21T15:50:41.376 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.376 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.376 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:41.376 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135449+0000", 2024-09-21T15:50:41.376 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1580000000000002e-06 2024-09-21T15:50:41.376 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.376 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.376 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:41.376 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189267+0000", 2024-09-21T15:50:41.376 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053817775999999998 2024-09-21T15:50:41.377 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.377 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.377 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:41.377 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189336+0000", 2024-09-21T15:50:41.377 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9485000000000003e-05 2024-09-21T15:50:41.377 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.377 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.377 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:41.377 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189436+0000", 2024-09-21T15:50:41.377 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9277000000000005e-05 2024-09-21T15:50:41.377 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:41.377 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:41.377 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:41.377 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.378 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.378 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63150 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 950272~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:41.378 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.135618+0000", 2024-09-21T15:50:41.378 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 64.205437537999998, 2024-09-21T15:50:41.378 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054027393, 2024-09-21T15:50:41.378 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:41.378 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:41.378 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:41.378 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:41.378 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:41.378 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63150 2024-09-21T15:50:41.378 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.378 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:41.378 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.378 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:41.379 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135618+0000", 2024-09-21T15:50:41.379 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:41.379 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.379 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.379 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:41.379 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135618+0000", 2024-09-21T15:50:41.379 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:41.379 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.379 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.379 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:41.379 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135454+0000", 2024-09-21T15:50:41.379 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998364 2024-09-21T15:50:41.379 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.379 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.380 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:41.380 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135621+0000", 2024-09-21T15:50:41.380 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016714099999999999 2024-09-21T15:50:41.380 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.380 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.380 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:41.380 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135631+0000", 2024-09-21T15:50:41.380 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5340000000000005e-06 2024-09-21T15:50:41.380 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.380 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.380 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:41.380 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135638+0000", 2024-09-21T15:50:41.380 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9280000000000004e-06 2024-09-21T15:50:41.380 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.380 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.381 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:41.381 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189463+0000", 2024-09-21T15:50:41.381 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053825681 2024-09-21T15:50:41.381 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.381 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.381 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:41.381 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189546+0000", 2024-09-21T15:50:41.381 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2278000000000001e-05 2024-09-21T15:50:41.381 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.381 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.381 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:41.381 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189645+0000", 2024-09-21T15:50:41.381 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9543999999999994e-05 2024-09-21T15:50:41.381 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:41.381 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:41.382 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:41.382 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.382 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.382 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63152 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 966656~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:41.382 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.136029+0000", 2024-09-21T15:50:41.382 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 64.205026946000004, 2024-09-21T15:50:41.382 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054024823999999999, 2024-09-21T15:50:41.382 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:41.382 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:41.382 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:41.382 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:41.382 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:41.382 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63152 2024-09-21T15:50:41.382 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.382 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:41.383 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.383 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:41.383 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136029+0000", 2024-09-21T15:50:41.383 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:41.383 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.383 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.383 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:41.383 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136029+0000", 2024-09-21T15:50:41.383 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:41.383 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.383 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.383 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:41.383 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136028+0000", 2024-09-21T15:50:41.384 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:41.384 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.384 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.384 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:41.384 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136031+0000", 2024-09-21T15:50:41.384 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.033e-06 2024-09-21T15:50:41.384 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.384 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.384 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:41.384 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136039+0000", 2024-09-21T15:50:41.384 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7200000000000006e-06 2024-09-21T15:50:41.384 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.384 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.384 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:41.385 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136051+0000", 2024-09-21T15:50:41.385 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1987000000000001e-05 2024-09-21T15:50:41.385 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.385 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.385 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:41.385 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189876+0000", 2024-09-21T15:50:41.385 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053825264999999997 2024-09-21T15:50:41.385 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.385 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.385 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:41.385 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189955+0000", 2024-09-21T15:50:41.385 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9296999999999997e-05 2024-09-21T15:50:41.385 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.385 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.385 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:41.386 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190053+0000", 2024-09-21T15:50:41.386 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.8067999999999997e-05 2024-09-21T15:50:41.386 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:41.386 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:41.386 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:41.386 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.386 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.386 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63153 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 974848~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:41.386 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.136311+0000", 2024-09-21T15:50:41.386 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 64.204744433000002, 2024-09-21T15:50:41.386 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053933825999999997, 2024-09-21T15:50:41.386 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:41.386 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:41.386 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:41.387 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:41.387 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:41.387 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63153 2024-09-21T15:50:41.387 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.387 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:41.387 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.387 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:41.387 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136311+0000", 2024-09-21T15:50:41.387 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:41.387 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.387 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.387 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:41.387 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136311+0000", 2024-09-21T15:50:41.387 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:41.388 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.388 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.388 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:41.388 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136057+0000", 2024-09-21T15:50:41.388 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9997463 2024-09-21T15:50:41.388 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.388 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.388 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:41.388 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136316+0000", 2024-09-21T15:50:41.388 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00025872299999999998 2024-09-21T15:50:41.388 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.388 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.388 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:41.388 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136326+0000", 2024-09-21T15:50:41.388 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0552e-05 2024-09-21T15:50:41.389 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.389 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.389 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:41.389 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136334+0000", 2024-09-21T15:50:41.389 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1409999999999999e-06 2024-09-21T15:50:41.389 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.389 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.389 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:41.389 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190080+0000", 2024-09-21T15:50:41.389 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053745933000000003 2024-09-21T15:50:41.389 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.389 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.389 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:41.389 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190157+0000", 2024-09-21T15:50:41.390 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7412999999999999e-05 2024-09-21T15:50:41.390 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.390 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.390 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:41.390 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190245+0000", 2024-09-21T15:50:41.390 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7904000000000004e-05 2024-09-21T15:50:41.390 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:41.390 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:41.390 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:41.390 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.390 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.390 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63154 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 983040~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:41.390 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.136472+0000", 2024-09-21T15:50:41.390 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 64.204583081999999, 2024-09-21T15:50:41.390 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053968448000000002, 2024-09-21T15:50:41.391 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:41.391 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:41.391 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:41.391 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:41.391 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:41.391 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63154 2024-09-21T15:50:41.391 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.391 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:41.391 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.391 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:41.391 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136472+0000", 2024-09-21T15:50:41.391 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:41.391 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.391 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.392 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:41.392 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136472+0000", 2024-09-21T15:50:41.392 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:41.392 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.392 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.392 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:41.392 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136341+0000", 2024-09-21T15:50:41.392 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998684 2024-09-21T15:50:41.392 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.392 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.392 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:41.392 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136476+0000", 2024-09-21T15:50:41.392 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013528100000000001 2024-09-21T15:50:41.392 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.393 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.393 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:41.393 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136484+0000", 2024-09-21T15:50:41.393 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4760000000000001e-06 2024-09-21T15:50:41.393 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.393 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.393 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:41.393 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136490+0000", 2024-09-21T15:50:41.393 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8640000000000001e-06 2024-09-21T15:50:41.393 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.393 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.393 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:41.393 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190271+0000", 2024-09-21T15:50:41.393 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053781223000000003 2024-09-21T15:50:41.394 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.394 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.394 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:41.394 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190348+0000", 2024-09-21T15:50:41.394 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7058999999999995e-05 2024-09-21T15:50:41.394 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.394 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.394 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:41.394 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190441+0000", 2024-09-21T15:50:41.394 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2937999999999997e-05 2024-09-21T15:50:41.394 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:41.394 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:41.394 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:41.394 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.394 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.395 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63157 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1015808~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:41.395 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137112+0000", 2024-09-21T15:50:41.395 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 64.203943795000001, 2024-09-21T15:50:41.395 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053932345, 2024-09-21T15:50:41.395 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:41.395 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:41.395 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:41.395 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:41.395 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:41.395 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63157 2024-09-21T15:50:41.395 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.395 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:41.395 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.395 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:41.395 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137112+0000", 2024-09-21T15:50:41.396 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:41.396 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.396 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.396 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:41.396 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137112+0000", 2024-09-21T15:50:41.396 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:41.396 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.396 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.396 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:41.396 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137072+0000", 2024-09-21T15:50:41.396 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999604 2024-09-21T15:50:41.396 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.396 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.396 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:41.397 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137115+0000", 2024-09-21T15:50:41.397 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2379999999999997e-05 2024-09-21T15:50:41.397 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.397 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.397 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:41.397 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137121+0000", 2024-09-21T15:50:41.397 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6499999999999999e-06 2024-09-21T15:50:41.397 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.397 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.397 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:41.397 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137126+0000", 2024-09-21T15:50:41.397 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0769999999999998e-06 2024-09-21T15:50:41.398 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.398 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.398 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:41.398 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190868+0000", 2024-09-21T15:50:41.398 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053741555000000003 2024-09-21T15:50:41.398 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.398 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.398 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:41.398 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190945+0000", 2024-09-21T15:50:41.398 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7171999999999999e-05 2024-09-21T15:50:41.398 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.398 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.398 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:41.399 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191044+0000", 2024-09-21T15:50:41.399 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9031999999999996e-05 2024-09-21T15:50:41.399 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:41.399 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:41.399 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:41.399 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.399 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.399 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63158 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1007616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:41.399 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137293+0000", 2024-09-21T15:50:41.399 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 64.203762507999997, 2024-09-21T15:50:41.399 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054005642, 2024-09-21T15:50:41.399 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:41.400 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:41.400 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:41.400 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:41.400 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:41.400 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63158 2024-09-21T15:50:41.400 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.400 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:41.400 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.400 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:41.400 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137293+0000", 2024-09-21T15:50:41.400 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:41.400 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.400 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.400 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:41.400 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137293+0000", 2024-09-21T15:50:41.401 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:41.401 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.401 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.401 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:41.401 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137131+0000", 2024-09-21T15:50:41.401 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998379 2024-09-21T15:50:41.401 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.401 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.401 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:41.401 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137296+0000", 2024-09-21T15:50:41.401 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016568999999999999 2024-09-21T15:50:41.401 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.401 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.401 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:41.402 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137306+0000", 2024-09-21T15:50:41.402 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.6029999999999994e-06 2024-09-21T15:50:41.402 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.402 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.402 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:41.402 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137312+0000", 2024-09-21T15:50:41.402 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3210000000000001e-06 2024-09-21T15:50:41.402 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.402 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.402 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:41.402 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191087+0000", 2024-09-21T15:50:41.402 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053774686000000002 2024-09-21T15:50:41.402 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.402 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.403 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:41.403 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191175+0000", 2024-09-21T15:50:41.403 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8106e-05 2024-09-21T15:50:41.403 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.403 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.403 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:41.403 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191299+0000", 2024-09-21T15:50:41.403 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012343599999999999 2024-09-21T15:50:41.403 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:41.403 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:41.403 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:41.403 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.403 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.403 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63159 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1024000~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:41.404 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137568+0000", 2024-09-21T15:50:41.404 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 64.203487236000001, 2024-09-21T15:50:41.404 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053962999999999997, 2024-09-21T15:50:41.404 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:41.404 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:41.404 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:41.404 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:41.404 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:41.404 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63159 2024-09-21T15:50:41.404 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.405 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:41.405 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.405 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:41.405 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137568+0000", 2024-09-21T15:50:41.405 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:41.405 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.405 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.405 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:41.405 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137568+0000", 2024-09-21T15:50:41.405 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:41.405 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.405 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.406 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:41.406 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137317+0000", 2024-09-21T15:50:41.406 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9997492 2024-09-21T15:50:41.406 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.406 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.406 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:41.406 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137572+0000", 2024-09-21T15:50:41.406 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00025491499999999998 2024-09-21T15:50:41.406 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.406 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.406 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:41.406 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137582+0000", 2024-09-21T15:50:41.406 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4490000000000005e-06 2024-09-21T15:50:41.407 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.407 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.407 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:41.407 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137589+0000", 2024-09-21T15:50:41.407 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5580000000000003e-06 2024-09-21T15:50:41.407 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.407 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.407 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:41.407 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191344+0000", 2024-09-21T15:50:41.407 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053754272999999998 2024-09-21T15:50:41.407 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.407 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.407 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:41.407 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191429+0000", 2024-09-21T15:50:41.408 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.509e-05 2024-09-21T15:50:41.408 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.408 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.408 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:41.408 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191531+0000", 2024-09-21T15:50:41.408 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000102519 2024-09-21T15:50:41.408 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:41.408 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:41.408 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:41.408 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.408 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.408 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63160 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1032192~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:41.408 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137771+0000", 2024-09-21T15:50:41.408 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 64.203284702000005, 2024-09-21T15:50:41.408 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053957818999999997, 2024-09-21T15:50:41.409 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:41.409 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:41.409 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:41.409 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:41.409 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:41.409 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63160 2024-09-21T15:50:41.409 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.409 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:41.409 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.409 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:41.409 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137771+0000", 2024-09-21T15:50:41.409 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:41.409 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.409 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.410 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:41.410 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137771+0000", 2024-09-21T15:50:41.410 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:41.410 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.410 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.410 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:41.410 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137596+0000", 2024-09-21T15:50:41.410 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998255 2024-09-21T15:50:41.410 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.410 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.410 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:41.410 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137780+0000", 2024-09-21T15:50:41.410 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00018363900000000001 2024-09-21T15:50:41.410 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.411 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.411 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:41.411 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137791+0000", 2024-09-21T15:50:41.411 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.096e-05 2024-09-21T15:50:41.411 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.411 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.411 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:41.411 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137798+0000", 2024-09-21T15:50:41.411 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1169999999999998e-06 2024-09-21T15:50:41.411 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.411 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.411 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:41.411 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191560+0000", 2024-09-21T15:50:41.411 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053762163000000002 2024-09-21T15:50:41.411 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.412 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.412 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:41.412 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191634+0000", 2024-09-21T15:50:41.412 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4381999999999994e-05 2024-09-21T15:50:41.412 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.412 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.412 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:41.412 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191729+0000", 2024-09-21T15:50:41.412 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4098000000000006e-05 2024-09-21T15:50:41.412 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:41.412 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:41.412 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:41.412 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.413 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.413 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63161 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1040384~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:41.413 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.138018+0000", 2024-09-21T15:50:41.413 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 64.203037948000002, 2024-09-21T15:50:41.413 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053905657000000003, 2024-09-21T15:50:41.413 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:41.413 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:41.413 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:41.413 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:41.413 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:41.413 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63161 2024-09-21T15:50:41.413 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.413 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:41.413 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.414 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:41.414 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138018+0000", 2024-09-21T15:50:41.414 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:41.414 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.414 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.414 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:41.414 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138018+0000", 2024-09-21T15:50:41.414 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:41.414 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.414 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.414 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:41.414 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137804+0000", 2024-09-21T15:50:41.414 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9997859 2024-09-21T15:50:41.414 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.415 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.415 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:41.415 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138021+0000", 2024-09-21T15:50:41.415 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00021769899999999999 2024-09-21T15:50:41.415 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.415 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.415 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:41.415 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138043+0000", 2024-09-21T15:50:41.415 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1627000000000001e-05 2024-09-21T15:50:41.415 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.415 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.415 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:41.415 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138054+0000", 2024-09-21T15:50:41.415 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1321999999999999e-05 2024-09-21T15:50:41.415 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.416 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.416 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:41.416 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191755+0000", 2024-09-21T15:50:41.416 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053700354999999998 2024-09-21T15:50:41.416 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.416 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.416 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:41.416 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191830+0000", 2024-09-21T15:50:41.416 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5320000000000004e-05 2024-09-21T15:50:41.416 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.416 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.416 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:41.416 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191923+0000", 2024-09-21T15:50:41.416 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3282999999999994e-05 2024-09-21T15:50:41.417 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:41.417 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:41.417 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:41.417 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.417 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.417 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63176 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 108544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:41.417 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.140861+0000", 2024-09-21T15:50:41.417 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 64.200194628999995, 2024-09-21T15:50:41.417 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053935394999999997, 2024-09-21T15:50:41.417 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:41.417 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:41.417 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:41.417 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:41.417 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:41.418 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63176 2024-09-21T15:50:41.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.418 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:41.418 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:41.418 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140861+0000", 2024-09-21T15:50:41.418 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:41.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.418 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:41.418 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140861+0000", 2024-09-21T15:50:41.418 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:41.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.418 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.419 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:41.419 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140860+0000", 2024-09-21T15:50:41.419 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:41.419 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.419 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.419 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:41.419 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140863+0000", 2024-09-21T15:50:41.419 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8710000000000001e-06 2024-09-21T15:50:41.419 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.419 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.419 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:41.419 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140871+0000", 2024-09-21T15:50:41.419 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8089999999999993e-06 2024-09-21T15:50:41.419 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.419 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.420 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:41.420 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140877+0000", 2024-09-21T15:50:41.420 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5969999999999999e-06 2024-09-21T15:50:41.420 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.420 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.420 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:41.420 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.194608+0000", 2024-09-21T15:50:41.420 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053731749000000002 2024-09-21T15:50:41.420 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.420 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.420 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:41.420 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.194697+0000", 2024-09-21T15:50:41.421 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8504000000000005e-05 2024-09-21T15:50:41.421 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.421 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.421 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:41.421 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.194796+0000", 2024-09-21T15:50:41.421 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9414999999999996e-05 2024-09-21T15:50:41.421 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:41.421 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:41.421 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:41.421 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.421 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.421 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63182 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 157696~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:41.421 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:50:41.421 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 64.198922229000004, 2024-09-21T15:50:41.421 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053958825000000002, 2024-09-21T15:50:41.422 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:41.422 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:41.422 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:41.422 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:41.422 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:41.422 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63182 2024-09-21T15:50:41.422 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.422 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:41.422 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.422 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:41.422 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:50:41.422 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:41.422 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.423 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.423 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:41.423 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:50:41.423 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:41.423 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.423 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.423 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:41.423 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:50:41.423 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:41.423 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.423 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.423 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:41.423 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142136+0000", 2024-09-21T15:50:41.423 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8169999999999999e-06 2024-09-21T15:50:41.423 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.424 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.424 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:41.424 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:50:41.424 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3752e-05 2024-09-21T15:50:41.424 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.424 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.424 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:41.424 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142155+0000", 2024-09-21T15:50:41.424 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1859999999999997e-06 2024-09-21T15:50:41.424 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.424 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.424 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:41.424 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.195914+0000", 2024-09-21T15:50:41.424 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053758528999999999 2024-09-21T15:50:41.425 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.425 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.425 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:41.425 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.195995+0000", 2024-09-21T15:50:41.425 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0983999999999996e-05 2024-09-21T15:50:41.425 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.425 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.425 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:41.425 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.196092+0000", 2024-09-21T15:50:41.425 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7069000000000002e-05 2024-09-21T15:50:41.425 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:41.425 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:41.425 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:41.425 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.425 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.426 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63193 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 239616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:41.426 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:50:41.426 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 64.196834644000006, 2024-09-21T15:50:41.426 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053956577999999998, 2024-09-21T15:50:41.426 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:41.426 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:41.426 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:41.426 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:41.426 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:41.426 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63193 2024-09-21T15:50:41.426 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.426 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:41.426 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.427 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:41.427 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:50:41.427 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:41.427 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.427 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.427 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:41.427 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:50:41.427 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:41.427 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.427 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.427 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:41.427 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:50:41.427 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:41.427 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.428 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.428 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:41.428 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144223+0000", 2024-09-21T15:50:41.428 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0990000000000001e-06 2024-09-21T15:50:41.428 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.428 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.428 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:41.428 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144230+0000", 2024-09-21T15:50:41.428 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9070000000000001e-06 2024-09-21T15:50:41.428 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.428 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.428 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:41.428 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144234+0000", 2024-09-21T15:50:41.428 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9010000000000002e-06 2024-09-21T15:50:41.428 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.429 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.429 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:41.429 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.198004+0000", 2024-09-21T15:50:41.429 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053769539999999998 2024-09-21T15:50:41.429 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.429 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.429 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:41.429 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.198083+0000", 2024-09-21T15:50:41.429 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9531999999999997e-05 2024-09-21T15:50:41.429 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:41.429 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:41.429 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:41.429 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.198177+0000", 2024-09-21T15:50:41.429 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3906000000000005e-05 2024-09-21T15:50:41.430 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:41.430 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:41.430 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:41.430 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:41.430 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:41.430 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:41.430 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:50:41.433 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T15:50:41.433 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:50:41.434 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-09-21T15:50:41.435 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":153,"stamp":"2024-09-21T15:50:40.702059+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1345918076,"num_objects":2105,"num_object_clones":245,"num_object_copies":4210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2105,"num_whiteouts":1224,"num_read":198207,"num_read_kb":834540,"num_write":21708,"num_write_kb":305196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":841,"num_bytes_recovered":47753679,"num_keys_recovered":165,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17877,"ondisk_log_size":17877,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":699584,"kb_used_data":316296,"kb_used_omap":24,"kb_used_meta":383079,"kb_avail":754275136,"statfs":{"total":773094113280,"available":772377739264,"internally_reserved":0,"allocated":323887104,"data_stored":424597741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25118,"internal_metadata":392273378},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-139078846,"num_objects":-135,"num_object_clones":-47,"num_object_copies":-270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-135,"num_whiteouts":-32,"num_read":1252,"num_read_kb":1066,"num_write":210,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.019224"},"pg_stats":[{"pgid":"3.1f","version":"102'373","reported_seq":1777,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T15:50:37.680975+0000","last_change":"2024-09-21T15:50:37.680975+0000","last_active":"2024-09-21T15:50:37.680975+0000","last_peered":"2024-09-21T15:50:37.680975+0000","last_clean":"2024-09-21T15:50:37.680975+0000","last_became_active":"2024-09-21T15:50:32.670964+0000","last_became_peered":"2024-09-21T15:50:32.670964+0000","last_unstale":"2024-09-21T15:50:37.680975+0000","last_undegraded":"2024-09-21T15:50:37.680975+0000","last_fullsized":"2024-09-21T15:50:37.680975+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":373,"log_dups_size":0,"ondisk_log_size":373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:09:09.918864+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.011407993999999999,"stat_sum":{"num_bytes":5107497,"num_objects":47,"num_object_clones":3,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":31,"num_read":2172,"num_read_kb":10213,"num_write":413,"num_write_kb":5950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":1974433,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"29","length":"4"}]},{"pgid":"3.1e","version":"103'483","reported_seq":5409,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-21T15:50:39.773593+0000","last_change":"2024-09-21T15:50:39.773790+0000","last_active":"2024-09-21T15:50:39.773593+0000","last_peered":"2024-09-21T15:50:39.773593+0000","last_clean":"2024-09-21T15:50:39.773593+0000","last_became_active":"2024-09-21T15:50:20.087949+0000","last_became_peered":"2024-09-21T15:50:20.087949+0000","last_unstale":"2024-09-21T15:50:39.773593+0000","last_undegraded":"2024-09-21T15:50:39.773593+0000","last_fullsized":"2024-09-21T15:50:39.773593+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":483,"log_dups_size":0,"ondisk_log_size":483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:26:31.786747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000310413,"stat_sum":{"num_bytes":53219311,"num_objects":77,"num_object_clones":10,"num_object_copies":154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77,"num_whiteouts":45,"num_read":6840,"num_read_kb":25381,"num_write":497,"num_write_kb":4338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"27","length":"6"}]},{"pgid":"3.1d","version":"103'547","reported_seq":9800,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-21T15:50:39.702196+0000","last_change":"2024-09-21T15:50:39.702399+0000","last_active":"2024-09-21T15:50:39.702196+0000","last_peered":"2024-09-21T15:50:39.702196+0000","last_clean":"2024-09-21T15:50:39.702196+0000","last_became_active":"2024-09-21T15:49:39.833206+0000","last_became_peered":"2024-09-21T15:49:39.833206+0000","last_unstale":"2024-09-21T15:50:39.702196+0000","last_undegraded":"2024-09-21T15:50:39.702196+0000","last_fullsized":"2024-09-21T15:50:39.702196+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":547,"log_dups_size":0,"ondisk_log_size":547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:36:14.082318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035473200000000002,"stat_sum":{"num_bytes":71988001,"num_objects":86,"num_object_clones":16,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":46,"num_read":10437,"num_read_kb":28908,"num_write":629,"num_write_kb":8866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"27","length":"6"}]},{"pgid":"3.1c","version":"104'510","reported_seq":4082,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-21T15:50:39.731602+0000","last_change":"2024-09-21T15:50:39.702815+0000","last_active":"2024-09-21T15:50:39.731602+0000","last_peered":"2024-09-21T15:50:39.731602+0000","last_clean":"2024-09-21T15:50:39.731602+0000","last_became_active":"2024-09-21T15:49:39.833113+0000","last_became_peered":"2024-09-21T15:49:39.833113+0000","last_unstale":"2024-09-21T15:50:39.731602+0000","last_undegraded":"2024-09-21T15:50:39.731602+0000","last_fullsized":"2024-09-21T15:50:39.731602+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":510,"log_dups_size":0,"ondisk_log_size":510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:43:57.592707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071529000000000004,"stat_sum":{"num_bytes":11366202,"num_objects":55,"num_object_clones":5,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":31,"num_read":4982,"num_read_kb":17094,"num_write":544,"num_write_kb":4783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"27","length":"6"}]},{"pgid":"3.1b","version":"100'441","reported_seq":4033,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T15:50:37.676929+0000","last_change":"2024-09-21T15:50:37.677124+0000","last_active":"2024-09-21T15:50:37.676929+0000","last_peered":"2024-09-21T15:50:37.676929+0000","last_clean":"2024-09-21T15:50:37.676929+0000","last_became_active":"2024-09-21T15:50:32.674250+0000","last_became_peered":"2024-09-21T15:50:32.674250+0000","last_unstale":"2024-09-21T15:50:37.676929+0000","last_undegraded":"2024-09-21T15:50:37.676929+0000","last_fullsized":"2024-09-21T15:50:37.676929+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":441,"log_dups_size":0,"ondisk_log_size":441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:54:57.399818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0066613599999999999,"stat_sum":{"num_bytes":54600659,"num_objects":74,"num_object_clones":5,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":42,"num_read":4613,"num_read_kb":29921,"num_write":447,"num_write_kb":13501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":4759000,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"29","length":"4"}]},{"pgid":"3.1a","version":"103'371","reported_seq":6335,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-21T15:50:39.804248+0000","last_change":"2024-09-21T15:50:39.702513+0000","last_active":"2024-09-21T15:50:39.804248+0000","last_peered":"2024-09-21T15:50:39.804248+0000","last_clean":"2024-09-21T15:50:39.804248+0000","last_became_active":"2024-09-21T15:49:06.693903+0000","last_became_peered":"2024-09-21T15:49:06.693903+0000","last_unstale":"2024-09-21T15:50:39.804248+0000","last_undegraded":"2024-09-21T15:50:39.804248+0000","last_fullsized":"2024-09-21T15:50:39.804248+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":371,"log_dups_size":0,"ondisk_log_size":371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:41:45.432893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000465484,"stat_sum":{"num_bytes":52938660,"num_objects":67,"num_object_clones":9,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":36,"num_read":6761,"num_read_kb":31330,"num_write":384,"num_write_kb":9470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"27","length":"6"}]},{"pgid":"3.19","version":"103'412","reported_seq":8058,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-21T15:50:39.699393+0000","last_change":"2024-09-21T15:50:39.699605+0000","last_active":"2024-09-21T15:50:39.699393+0000","last_peered":"2024-09-21T15:50:39.699393+0000","last_clean":"2024-09-21T15:50:39.699393+0000","last_became_active":"2024-09-21T15:49:06.691222+0000","last_became_peered":"2024-09-21T15:49:06.691222+0000","last_unstale":"2024-09-21T15:50:39.699393+0000","last_undegraded":"2024-09-21T15:50:39.699393+0000","last_fullsized":"2024-09-21T15:50:39.699393+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":412,"log_dups_size":0,"ondisk_log_size":412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:31:29.545866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000340033,"stat_sum":{"num_bytes":62034124,"num_objects":68,"num_object_clones":5,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":36,"num_read":8235,"num_read_kb":39548,"num_write":478,"num_write_kb":7403,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"27","length":"6"}]},{"pgid":"3.18","version":"97'407","reported_seq":3467,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T15:50:37.669493+0000","last_change":"2024-09-21T15:50:37.669931+0000","last_active":"2024-09-21T15:50:37.669493+0000","last_peered":"2024-09-21T15:50:37.669493+0000","last_clean":"2024-09-21T15:50:37.669493+0000","last_became_active":"2024-09-21T15:50:32.672791+0000","last_became_peered":"2024-09-21T15:50:32.672791+0000","last_unstale":"2024-09-21T15:50:37.669493+0000","last_undegraded":"2024-09-21T15:50:37.669493+0000","last_fullsized":"2024-09-21T15:50:37.669493+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":407,"log_dups_size":0,"ondisk_log_size":407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:07:19.540473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010436410000000001,"stat_sum":{"num_bytes":48735247,"num_objects":67,"num_object_clones":9,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":30,"num_read":5193,"num_read_kb":30076,"num_write":442,"num_write_kb":11051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":2076323,"num_keys_recovered":11,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"29","length":"4"}]},{"pgid":"3.17","version":"102'924","reported_seq":8830,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T15:50:37.772305+0000","last_change":"2024-09-21T15:50:37.674917+0000","last_active":"2024-09-21T15:50:37.772305+0000","last_peered":"2024-09-21T15:50:37.772305+0000","last_clean":"2024-09-21T15:50:37.772305+0000","last_became_active":"2024-09-21T15:50:32.668771+0000","last_became_peered":"2024-09-21T15:50:32.668771+0000","last_unstale":"2024-09-21T15:50:37.772305+0000","last_undegraded":"2024-09-21T15:50:37.772305+0000","last_fullsized":"2024-09-21T15:50:37.772305+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":924,"log_dups_size":0,"ondisk_log_size":924,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:39:43.554150+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0055834279999999997,"stat_sum":{"num_bytes":15502015,"num_objects":68,"num_object_clones":6,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":41,"num_read":12072,"num_read_kb":20937,"num_write":1213,"num_write_kb":8234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":3319185,"num_keys_recovered":34,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"29","length":"4"}]},{"pgid":"3.16","version":"104'475","reported_seq":6341,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-21T15:50:39.758595+0000","last_change":"2024-09-21T15:50:39.758595+0000","last_active":"2024-09-21T15:50:39.758595+0000","last_peered":"2024-09-21T15:50:39.758595+0000","last_clean":"2024-09-21T15:50:39.758595+0000","last_became_active":"2024-09-21T15:49:06.688808+0000","last_became_peered":"2024-09-21T15:49:06.688808+0000","last_unstale":"2024-09-21T15:50:39.758595+0000","last_undegraded":"2024-09-21T15:50:39.758595+0000","last_fullsized":"2024-09-21T15:50:39.758595+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":475,"log_dups_size":0,"ondisk_log_size":475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:43:02.012254+0000","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0.0036253359999999998,"stat_sum":{"num_bytes":81763516,"num_objects":63,"num_object_clones":6,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":36,"num_read":7324,"num_read_kb":49627,"num_write":492,"num_write_kb":13812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"27","length":"6"}]},{"pgid":"3.15","version":"102'490","reported_seq":3594,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-21T15:50:39.748785+0000","last_change":"2024-09-21T15:50:39.749098+0000","last_active":"2024-09-21T15:50:39.748785+0000","last_peered":"2024-09-21T15:50:39.748785+0000","last_clean":"2024-09-21T15:50:39.748785+0000","last_became_active":"2024-09-21T15:49:06.691091+0000","last_became_peered":"2024-09-21T15:49:06.691091+0000","last_unstale":"2024-09-21T15:50:39.748785+0000","last_undegraded":"2024-09-21T15:50:39.748785+0000","last_fullsized":"2024-09-21T15:50:39.748785+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":490,"log_dups_size":0,"ondisk_log_size":490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:13:12.258161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00141209,"stat_sum":{"num_bytes":25828601,"num_objects":58,"num_object_clones":8,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":35,"num_read":4033,"num_read_kb":27130,"num_write":614,"num_write_kb":17935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"27","length":"6"}]},{"pgid":"3.14","version":"102'1148","reported_seq":4096,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-21T15:50:39.702920+0000","last_change":"2024-09-21T15:50:39.703344+0000","last_active":"2024-09-21T15:50:39.702920+0000","last_peered":"2024-09-21T15:50:39.702920+0000","last_clean":"2024-09-21T15:50:39.702920+0000","last_became_active":"2024-09-21T15:49:06.688803+0000","last_became_peered":"2024-09-21T15:49:06.688803+0000","last_unstale":"2024-09-21T15:50:39.702920+0000","last_undegraded":"2024-09-21T15:50:39.702920+0000","last_fullsized":"2024-09-21T15:50:39.702920+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":1148,"log_dups_size":0,"ondisk_log_size":1148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:18:47.662631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001174029,"stat_sum":{"num_bytes":63762684,"num_objects":51,"num_object_clones":3,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":28,"num_read":7660,"num_read_kb":18155,"num_write":1867,"num_write_kb":6175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"27","length":"6"}]},{"pgid":"2.7","version":"0'0","reported_seq":100,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-21T15:50:38.790164+0000","last_change":"2024-09-21T15:48:59.666431+0000","last_active":"2024-09-21T15:50:38.790164+0000","last_peered":"2024-09-21T15:50:38.790164+0000","last_clean":"2024-09-21T15:50:38.790164+0000","last_became_active":"2024-09-21T15:48:57.583665+0000","last_became_peered":"2024-09-21T15:48:57.583665+0000","last_unstale":"2024-09-21T15:50:38.790164+0000","last_undegraded":"2024-09-21T15:50:38.790164+0000","last_fullsized":"2024-09-21T15:50:38.790164+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:52:09.228964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031098399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"102'363","reported_seq":2761,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T15:50:37.675954+0000","last_change":"2024-09-21T15:50:37.675954+0000","last_active":"2024-09-21T15:50:37.675954+0000","last_peered":"2024-09-21T15:50:37.675954+0000","last_clean":"2024-09-21T15:50:37.675954+0000","last_became_active":"2024-09-21T15:50:32.671424+0000","last_became_peered":"2024-09-21T15:50:32.671424+0000","last_unstale":"2024-09-21T15:50:37.675954+0000","last_undegraded":"2024-09-21T15:50:37.675954+0000","last_fullsized":"2024-09-21T15:50:37.675954+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":363,"log_dups_size":0,"ondisk_log_size":363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:04:14.554732+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.0069010080000000001,"stat_sum":{"num_bytes":11403537,"num_objects":61,"num_object_clones":7,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":41,"num_read":2599,"num_read_kb":12929,"num_write":362,"num_write_kb":8387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":4494388,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"29","length":"4"}]},{"pgid":"2.6","version":"0'0","reported_seq":100,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-21T15:50:38.697723+0000","last_change":"2024-09-21T15:48:59.589790+0000","last_active":"2024-09-21T15:50:38.697723+0000","last_peered":"2024-09-21T15:50:38.697723+0000","last_clean":"2024-09-21T15:50:38.697723+0000","last_became_active":"2024-09-21T15:48:57.582121+0000","last_became_peered":"2024-09-21T15:48:57.582121+0000","last_unstale":"2024-09-21T15:50:38.697723+0000","last_undegraded":"2024-09-21T15:50:38.697723+0000","last_fullsized":"2024-09-21T15:50:38.697723+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:33:47.592624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039654700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"103'522","reported_seq":7114,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-21T15:50:39.749134+0000","last_change":"2024-09-21T15:50:39.749257+0000","last_active":"2024-09-21T15:50:39.749134+0000","last_peered":"2024-09-21T15:50:39.749134+0000","last_clean":"2024-09-21T15:50:39.749134+0000","last_became_active":"2024-09-21T15:50:20.087935+0000","last_became_peered":"2024-09-21T15:50:20.087935+0000","last_unstale":"2024-09-21T15:50:39.749134+0000","last_undegraded":"2024-09-21T15:50:39.749134+0000","last_fullsized":"2024-09-21T15:50:39.749134+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":522,"log_dups_size":0,"ondisk_log_size":522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:27:45.090104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015282539999999999,"stat_sum":{"num_bytes":33576848,"num_objects":69,"num_object_clones":8,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":41,"num_read":8167,"num_read_kb":21012,"num_write":508,"num_write_kb":5391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"27","length":"6"}]},{"pgid":"2.5","version":"0'0","reported_seq":113,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-21T15:50:38.692676+0000","last_change":"2024-09-21T15:50:30.146743+0000","last_active":"2024-09-21T15:50:38.692676+0000","last_peered":"2024-09-21T15:50:38.692676+0000","last_clean":"2024-09-21T15:50:38.692676+0000","last_became_active":"2024-09-21T15:50:30.146428+0000","last_became_peered":"2024-09-21T15:50:30.146428+0000","last_unstale":"2024-09-21T15:50:38.692676+0000","last_undegraded":"2024-09-21T15:50:38.692676+0000","last_fullsized":"2024-09-21T15:50:38.692676+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:58:38.689807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036074600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"104'615","reported_seq":8435,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-21T15:50:39.789080+0000","last_change":"2024-09-21T15:50:39.699669+0000","last_active":"2024-09-21T15:50:39.789080+0000","last_peered":"2024-09-21T15:50:39.789080+0000","last_clean":"2024-09-21T15:50:39.789080+0000","last_became_active":"2024-09-21T15:50:35.682788+0000","last_became_peered":"2024-09-21T15:50:35.682788+0000","last_unstale":"2024-09-21T15:50:39.789080+0000","last_undegraded":"2024-09-21T15:50:39.789080+0000","last_fullsized":"2024-09-21T15:50:39.789080+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":615,"log_dups_size":0,"ondisk_log_size":615,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:20:40.695690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035933100000000001,"stat_sum":{"num_bytes":36608615,"num_objects":66,"num_object_clones":13,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":34,"num_read":10649,"num_read_kb":27356,"num_write":808,"num_write_kb":11002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":175,"num_bytes_recovered":14134268,"num_keys_recovered":36,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"27","length":"6"}]},{"pgid":"2.4","version":"0'0","reported_seq":113,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-21T15:50:38.697660+0000","last_change":"2024-09-21T15:50:30.146654+0000","last_active":"2024-09-21T15:50:38.697660+0000","last_peered":"2024-09-21T15:50:38.697660+0000","last_clean":"2024-09-21T15:50:38.697660+0000","last_became_active":"2024-09-21T15:50:30.146273+0000","last_became_peered":"2024-09-21T15:50:30.146273+0000","last_unstale":"2024-09-21T15:50:38.697660+0000","last_undegraded":"2024-09-21T15:50:38.697660+0000","last_fullsized":"2024-09-21T15:50:38.697660+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:21:48.178519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040525700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"104'527","reported_seq":4991,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-21T15:50:39.776218+0000","last_change":"2024-09-21T15:50:39.776352+0000","last_active":"2024-09-21T15:50:39.776218+0000","last_peered":"2024-09-21T15:50:39.776218+0000","last_clean":"2024-09-21T15:50:39.776218+0000","last_became_active":"2024-09-21T15:49:39.835880+0000","last_became_peered":"2024-09-21T15:49:39.835880+0000","last_unstale":"2024-09-21T15:50:39.776218+0000","last_undegraded":"2024-09-21T15:50:39.776218+0000","last_fullsized":"2024-09-21T15:50:39.776218+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":527,"log_dups_size":0,"ondisk_log_size":527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:28:03.488776+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0032615970000000002,"stat_sum":{"num_bytes":65511075,"num_objects":75,"num_object_clones":9,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75,"num_whiteouts":40,"num_read":5513,"num_read_kb":33760,"num_write":631,"num_write_kb":16321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"27","length":"6"}]},{"pgid":"3.3","version":"104'678","reported_seq":3378,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-21T15:50:39.818869+0000","last_change":"2024-09-21T15:50:39.703294+0000","last_active":"2024-09-21T15:50:39.818869+0000","last_peered":"2024-09-21T15:50:39.818869+0000","last_clean":"2024-09-21T15:50:39.818869+0000","last_became_active":"2024-09-21T15:50:32.677456+0000","last_became_peered":"2024-09-21T15:50:32.677456+0000","last_unstale":"2024-09-21T15:50:39.818869+0000","last_undegraded":"2024-09-21T15:50:39.818869+0000","last_fullsized":"2024-09-21T15:50:39.818869+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":678,"log_dups_size":0,"ondisk_log_size":678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:46:31.194345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00118261,"stat_sum":{"num_bytes":23140359,"num_objects":74,"num_object_clones":13,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":46,"num_read":4388,"num_read_kb":18663,"num_write":747,"num_write_kb":7282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":632176,"num_keys_recovered":24,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"27","length":"6"}]},{"pgid":"2.2","version":"20'2","reported_seq":79,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-21T15:50:38.697561+0000","last_change":"2024-09-21T15:49:39.834640+0000","last_active":"2024-09-21T15:50:38.697561+0000","last_peered":"2024-09-21T15:50:38.697561+0000","last_clean":"2024-09-21T15:50:38.697561+0000","last_became_active":"2024-09-21T15:49:39.834207+0000","last_became_peered":"2024-09-21T15:49:39.834207+0000","last_unstale":"2024-09-21T15:50:38.697561+0000","last_undegraded":"2024-09-21T15:50:38.697561+0000","last_fullsized":"2024-09-21T15:50:38.697561+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:02:54.255716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"98'346","reported_seq":4475,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-21T15:50:39.699652+0000","last_change":"2024-09-21T15:50:39.699821+0000","last_active":"2024-09-21T15:50:39.699652+0000","last_peered":"2024-09-21T15:50:39.699652+0000","last_clean":"2024-09-21T15:50:39.699652+0000","last_became_active":"2024-09-21T15:50:35.682001+0000","last_became_peered":"2024-09-21T15:50:35.682001+0000","last_unstale":"2024-09-21T15:50:39.699652+0000","last_undegraded":"2024-09-21T15:50:39.699652+0000","last_fullsized":"2024-09-21T15:50:39.699652+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":346,"log_dups_size":0,"ondisk_log_size":346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:09:15.325457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043887400000000001,"stat_sum":{"num_bytes":20675160,"num_objects":57,"num_object_clones":4,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":35,"num_read":4043,"num_read_kb":23169,"num_write":345,"num_write_kb":4139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":187,"num_bytes_recovered":4305276,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"27","length":"6"}]},{"pgid":"2.1","version":"0'0","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-21T15:50:38.703626+0000","last_change":"2024-09-21T15:50:32.672133+0000","last_active":"2024-09-21T15:50:38.703626+0000","last_peered":"2024-09-21T15:50:38.703626+0000","last_clean":"2024-09-21T15:50:38.703626+0000","last_became_active":"2024-09-21T15:50:32.671865+0000","last_became_peered":"2024-09-21T15:50:32.671865+0000","last_unstale":"2024-09-21T15:50:38.703626+0000","last_undegraded":"2024-09-21T15:50:38.703626+0000","last_fullsized":"2024-09-21T15:50:38.703626+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:55:23.999144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"101'477","reported_seq":2124,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T15:50:37.676120+0000","last_change":"2024-09-21T15:50:37.676722+0000","last_active":"2024-09-21T15:50:37.676120+0000","last_peered":"2024-09-21T15:50:37.676120+0000","last_clean":"2024-09-21T15:50:37.676120+0000","last_became_active":"2024-09-21T15:50:32.671413+0000","last_became_peered":"2024-09-21T15:50:32.671413+0000","last_unstale":"2024-09-21T15:50:37.676120+0000","last_undegraded":"2024-09-21T15:50:37.676120+0000","last_fullsized":"2024-09-21T15:50:37.676120+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:04:19.509402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0071081549999999997,"stat_sum":{"num_bytes":26803231,"num_objects":65,"num_object_clones":5,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":42,"num_read":3205,"num_read_kb":16147,"num_write":528,"num_write_kb":5961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":1696547,"num_keys_recovered":12,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"29","length":"4"}]},{"pgid":"2.0","version":"0'0","reported_seq":100,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-21T15:50:38.692395+0000","last_change":"2024-09-21T15:49:00.114730+0000","last_active":"2024-09-21T15:50:38.692395+0000","last_peered":"2024-09-21T15:50:38.692395+0000","last_clean":"2024-09-21T15:50:38.692395+0000","last_became_active":"2024-09-21T15:48:57.585525+0000","last_became_peered":"2024-09-21T15:48:57.585525+0000","last_unstale":"2024-09-21T15:50:38.692395+0000","last_undegraded":"2024-09-21T15:50:38.692395+0000","last_fullsized":"2024-09-21T15:50:38.692395+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:35:35.786526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031691100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"103'369","reported_seq":2229,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-21T15:50:39.755047+0000","last_change":"2024-09-21T15:50:39.755413+0000","last_active":"2024-09-21T15:50:39.755047+0000","last_peered":"2024-09-21T15:50:39.755047+0000","last_clean":"2024-09-21T15:50:39.755047+0000","last_became_active":"2024-09-21T15:50:20.087681+0000","last_became_peered":"2024-09-21T15:50:20.087681+0000","last_unstale":"2024-09-21T15:50:39.755047+0000","last_undegraded":"2024-09-21T15:50:39.755047+0000","last_fullsized":"2024-09-21T15:50:39.755047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":369,"log_dups_size":0,"ondisk_log_size":369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:07:05.059684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049546400000000004,"stat_sum":{"num_bytes":67416002,"num_objects":61,"num_object_clones":5,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":33,"num_read":2107,"num_read_kb":34423,"num_write":371,"num_write_kb":12608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"27","length":"6"}]},{"pgid":"2.3","version":"18'1","reported_seq":72,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-21T15:50:38.703633+0000","last_change":"2024-09-21T15:49:39.834423+0000","last_active":"2024-09-21T15:50:38.703633+0000","last_peered":"2024-09-21T15:50:38.703633+0000","last_clean":"2024-09-21T15:50:38.703633+0000","last_became_active":"2024-09-21T15:49:39.833860+0000","last_became_peered":"2024-09-21T15:49:39.833860+0000","last_unstale":"2024-09-21T15:50:38.703633+0000","last_undegraded":"2024-09-21T15:50:38.703633+0000","last_fullsized":"2024-09-21T15:50:38.703633+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:11:43.630472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":261,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-21T15:50:38.692410+0000","last_change":"2024-09-21T15:50:30.147280+0000","last_active":"2024-09-21T15:50:38.692410+0000","last_peered":"2024-09-21T15:50:38.692410+0000","last_clean":"2024-09-21T15:50:38.692410+0000","last_became_active":"2024-09-21T15:50:30.146987+0000","last_became_peered":"2024-09-21T15:50:30.146987+0000","last_unstale":"2024-09-21T15:50:38.692410+0000","last_undegraded":"2024-09-21T15:50:38.692410+0000","last_fullsized":"2024-09-21T15:50:38.692410+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:50.161884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:50.161884+0000","last_clean_scrub_stamp":"2024-09-21T15:48:50.161884+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:26:32.853650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"103'466","reported_seq":7009,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-21T15:50:39.774229+0000","last_change":"2024-09-21T15:50:39.774410+0000","last_active":"2024-09-21T15:50:39.774229+0000","last_peered":"2024-09-21T15:50:39.774229+0000","last_clean":"2024-09-21T15:50:39.774229+0000","last_became_active":"2024-09-21T15:49:18.669143+0000","last_became_peered":"2024-09-21T15:49:18.669143+0000","last_unstale":"2024-09-21T15:50:39.774229+0000","last_undegraded":"2024-09-21T15:50:39.774229+0000","last_fullsized":"2024-09-21T15:50:39.774229+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":466,"log_dups_size":0,"ondisk_log_size":466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:31:03.684897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012451039999999999,"stat_sum":{"num_bytes":41140696,"num_objects":72,"num_object_clones":7,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":38,"num_read":7027,"num_read_kb":29405,"num_write":503,"num_write_kb":4783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":296439,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"27","length":"6"}]},{"pgid":"3.9","version":"103'346","reported_seq":2904,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-21T15:50:39.702467+0000","last_change":"2024-09-21T15:50:39.702848+0000","last_active":"2024-09-21T15:50:39.702467+0000","last_peered":"2024-09-21T15:50:39.702467+0000","last_clean":"2024-09-21T15:50:39.702467+0000","last_became_active":"2024-09-21T15:49:07.210506+0000","last_became_peered":"2024-09-21T15:49:07.210506+0000","last_unstale":"2024-09-21T15:50:39.702467+0000","last_undegraded":"2024-09-21T15:50:39.702467+0000","last_fullsized":"2024-09-21T15:50:39.702467+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":346,"log_dups_size":0,"ondisk_log_size":346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:55:18.046738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077619199999999998,"stat_sum":{"num_bytes":43716341,"num_objects":62,"num_object_clones":3,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":33,"num_read":2677,"num_read_kb":28218,"num_write":415,"num_write_kb":12085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"27","length":"6"}]},{"pgid":"3.a","version":"104'488","reported_seq":11814,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-21T15:50:39.781233+0000","last_change":"2024-09-21T15:50:39.781233+0000","last_active":"2024-09-21T15:50:39.781233+0000","last_peered":"2024-09-21T15:50:39.781233+0000","last_clean":"2024-09-21T15:50:39.781233+0000","last_became_active":"2024-09-21T15:49:06.694202+0000","last_became_peered":"2024-09-21T15:49:06.694202+0000","last_unstale":"2024-09-21T15:50:39.781233+0000","last_undegraded":"2024-09-21T15:50:39.781233+0000","last_fullsized":"2024-09-21T15:50:39.781233+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":488,"log_dups_size":0,"ondisk_log_size":488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:02:05.237705+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.0077539449999999999,"stat_sum":{"num_bytes":53937104,"num_objects":62,"num_object_clones":8,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":34,"num_read":12422,"num_read_kb":63120,"num_write":497,"num_write_kb":14195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"27","length":"6"}]},{"pgid":"3.b","version":"101'808","reported_seq":3118,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T15:50:37.676760+0000","last_change":"2024-09-21T15:50:37.676904+0000","last_active":"2024-09-21T15:50:37.676760+0000","last_peered":"2024-09-21T15:50:37.676760+0000","last_clean":"2024-09-21T15:50:37.676760+0000","last_became_active":"2024-09-21T15:50:30.149210+0000","last_became_peered":"2024-09-21T15:50:30.149210+0000","last_unstale":"2024-09-21T15:50:37.676760+0000","last_undegraded":"2024-09-21T15:50:37.676760+0000","last_fullsized":"2024-09-21T15:50:37.676760+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":808,"log_dups_size":0,"ondisk_log_size":808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:51:30.431071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0067440240000000004,"stat_sum":{"num_bytes":32234670,"num_objects":56,"num_object_clones":8,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":32,"num_read":5903,"num_read_kb":20139,"num_write":1215,"num_write_kb":7745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":5514164,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"29","length":"4"}]},{"pgid":"3.c","version":"104'643","reported_seq":6279,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-21T15:50:39.816061+0000","last_change":"2024-09-21T15:50:39.773314+0000","last_active":"2024-09-21T15:50:39.816061+0000","last_peered":"2024-09-21T15:50:39.816061+0000","last_clean":"2024-09-21T15:50:39.816061+0000","last_became_active":"2024-09-21T15:49:39.835282+0000","last_became_peered":"2024-09-21T15:49:39.835282+0000","last_unstale":"2024-09-21T15:50:39.816061+0000","last_undegraded":"2024-09-21T15:50:39.816061+0000","last_fullsized":"2024-09-21T15:50:39.816061+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":643,"log_dups_size":0,"ondisk_log_size":643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:51:40.356906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026967099999999997,"stat_sum":{"num_bytes":24065080,"num_objects":69,"num_object_clones":6,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":41,"num_read":8370,"num_read_kb":25272,"num_write":790,"num_write_kb":4984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"27","length":"6"}]},{"pgid":"3.d","version":"104'837","reported_seq":3915,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-21T15:50:39.748944+0000","last_change":"2024-09-21T15:50:39.748944+0000","last_active":"2024-09-21T15:50:39.748944+0000","last_peered":"2024-09-21T15:50:39.748944+0000","last_clean":"2024-09-21T15:50:39.748944+0000","last_became_active":"2024-09-21T15:49:06.689203+0000","last_became_peered":"2024-09-21T15:49:06.689203+0000","last_unstale":"2024-09-21T15:50:39.748944+0000","last_undegraded":"2024-09-21T15:50:39.748944+0000","last_fullsized":"2024-09-21T15:50:39.748944+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":837,"log_dups_size":0,"ondisk_log_size":837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:22:17.578359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001401926,"stat_sum":{"num_bytes":65898393,"num_objects":70,"num_object_clones":9,"num_object_copies":140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70,"num_whiteouts":43,"num_read":6938,"num_read_kb":27800,"num_write":1112,"num_write_kb":14803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"27","length":"6"}]},{"pgid":"3.e","version":"103'484","reported_seq":7704,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-21T15:50:39.748983+0000","last_change":"2024-09-21T15:50:39.749218+0000","last_active":"2024-09-21T15:50:39.748983+0000","last_peered":"2024-09-21T15:50:39.748983+0000","last_clean":"2024-09-21T15:50:39.748983+0000","last_became_active":"2024-09-21T15:49:06.690101+0000","last_became_peered":"2024-09-21T15:49:06.690101+0000","last_unstale":"2024-09-21T15:50:39.748983+0000","last_undegraded":"2024-09-21T15:50:39.748983+0000","last_fullsized":"2024-09-21T15:50:39.748983+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":484,"log_dups_size":0,"ondisk_log_size":484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:43:30.659104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001511624,"stat_sum":{"num_bytes":104751931,"num_objects":69,"num_object_clones":13,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":40,"num_read":8382,"num_read_kb":42678,"num_write":461,"num_write_kb":16261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"27","length":"6"}]},{"pgid":"3.f","version":"100'610","reported_seq":3400,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-21T15:50:39.800086+0000","last_change":"2024-09-21T15:50:39.749454+0000","last_active":"2024-09-21T15:50:39.800086+0000","last_peered":"2024-09-21T15:50:39.800086+0000","last_clean":"2024-09-21T15:50:39.800086+0000","last_became_active":"2024-09-21T15:49:06.694250+0000","last_became_peered":"2024-09-21T15:49:06.694250+0000","last_unstale":"2024-09-21T15:50:39.800086+0000","last_undegraded":"2024-09-21T15:50:39.800086+0000","last_fullsized":"2024-09-21T15:50:39.800086+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":610,"log_dups_size":0,"ondisk_log_size":610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:21:37.777769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015581709999999999,"stat_sum":{"num_bytes":18103488,"num_objects":76,"num_object_clones":8,"num_object_copies":152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":50,"num_read":4638,"num_read_kb":15668,"num_write":817,"num_write_kb":5055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"27","length":"6"}]},{"pgid":"3.10","version":"100'432","reported_seq":2820,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-21T15:50:39.776606+0000","last_change":"2024-09-21T15:50:39.776999+0000","last_active":"2024-09-21T15:50:39.776606+0000","last_peered":"2024-09-21T15:50:39.776606+0000","last_clean":"2024-09-21T15:50:39.776606+0000","last_became_active":"2024-09-21T15:50:32.676147+0000","last_became_peered":"2024-09-21T15:50:32.676147+0000","last_unstale":"2024-09-21T15:50:39.776606+0000","last_undegraded":"2024-09-21T15:50:39.776606+0000","last_fullsized":"2024-09-21T15:50:39.776606+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":432,"log_dups_size":0,"ondisk_log_size":432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:44:18.821310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033896400000000002,"stat_sum":{"num_bytes":9591471,"num_objects":56,"num_object_clones":4,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":44,"num_read":2952,"num_read_kb":11672,"num_write":357,"num_write_kb":4465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":1191778,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"27","length":"6"}]},{"pgid":"3.11","version":"102'925","reported_seq":3812,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-21T15:50:39.747597+0000","last_change":"2024-09-21T15:50:39.748029+0000","last_active":"2024-09-21T15:50:39.747597+0000","last_peered":"2024-09-21T15:50:39.747597+0000","last_clean":"2024-09-21T15:50:39.747597+0000","last_became_active":"2024-09-21T15:49:06.688798+0000","last_became_peered":"2024-09-21T15:49:06.688798+0000","last_unstale":"2024-09-21T15:50:39.747597+0000","last_undegraded":"2024-09-21T15:50:39.747597+0000","last_fullsized":"2024-09-21T15:50:39.747597+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":925,"log_dups_size":0,"ondisk_log_size":925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:48:33.679133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055274399999999998,"stat_sum":{"num_bytes":47370639,"num_objects":60,"num_object_clones":7,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":34,"num_read":6119,"num_read_kb":16744,"num_write":1282,"num_write_kb":8388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"27","length":"6"}]},{"pgid":"3.12","version":"97'352","reported_seq":2753,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T15:50:37.670058+0000","last_change":"2024-09-21T15:50:37.670498+0000","last_active":"2024-09-21T15:50:37.670058+0000","last_peered":"2024-09-21T15:50:37.670058+0000","last_clean":"2024-09-21T15:50:37.670058+0000","last_became_active":"2024-09-21T15:50:32.671686+0000","last_became_peered":"2024-09-21T15:50:32.671686+0000","last_unstale":"2024-09-21T15:50:37.670058+0000","last_undegraded":"2024-09-21T15:50:37.670058+0000","last_fullsized":"2024-09-21T15:50:37.670058+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:40:53.968477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013049859999999999,"stat_sum":{"num_bytes":39845746,"num_objects":64,"num_object_clones":8,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":41,"num_read":2441,"num_read_kb":11662,"num_write":386,"num_write_kb":13839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":3359702,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"29","length":"4"}]},{"pgid":"3.13","version":"102'888","reported_seq":7131,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-21T15:50:39.747684+0000","last_change":"2024-09-21T15:50:39.748043+0000","last_active":"2024-09-21T15:50:39.747684+0000","last_peered":"2024-09-21T15:50:39.747684+0000","last_clean":"2024-09-21T15:50:39.747684+0000","last_became_active":"2024-09-21T15:49:06.690128+0000","last_became_peered":"2024-09-21T15:49:06.690128+0000","last_unstale":"2024-09-21T15:50:39.747684+0000","last_undegraded":"2024-09-21T15:50:39.747684+0000","last_fullsized":"2024-09-21T15:50:39.747684+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":888,"log_dups_size":0,"ondisk_log_size":888,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:11.763959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055121900000000004,"stat_sum":{"num_bytes":32690786,"num_objects":76,"num_object_clones":15,"num_object_copies":152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":45,"num_read":9229,"num_read_kb":26283,"num_write":1392,"num_write_kb":13566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"27","length":"6"}]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1345327689,"num_objects":2101,"num_object_clones":245,"num_object_copies":4202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2101,"num_whiteouts":1224,"num_read":198091,"num_read_kb":834440,"num_write":21547,"num_write_kb":302778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":841,"num_bytes_recovered":47753679,"num_keys_recovered":165,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":295682048,"data_stored":384608964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5973,"internal_metadata":0},"log_size":17757,"ondisk_log_size":17757,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574873,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":115768,"kb_used_data":44440,"kb_used_omap":1,"kb_used_meta":71294,"kb_avail":94256072,"statfs":{"total":96636764160,"available":96518217728,"internally_reserved":0,"allocated":45506560,"data_stored":54694203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":73005515},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.47899999999999998}]},{"osd":1,"last update":"Sat Sep 21 15:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.39400000000000002}]},{"osd":2,"last update":"Sat Sep 21 15:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.371}]},{"osd":3,"last update":"Sat Sep 21 15:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.40600000000000003}]},{"osd":4,"last update":"Sat Sep 21 15:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.27000000000000002}]},{"osd":5,"last update":"Sat Sep 21 15:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.434},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.42399999999999999}]},{"osd":6,"last update":"Sat Sep 21 15:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.46300000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574876,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54912,"kb_used_data":19696,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94316928,"statfs":{"total":96636764160,"available":96580534272,"internally_reserved":0,"allocated":20168704,"data_stored":30225630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.36699999999999999}]},{"osd":1,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.40500000000000003}]},{"osd":2,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.45500000000000002}]},{"osd":3,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.48899999999999999}]},{"osd":4,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.435},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.311}]},{"osd":5,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.32800000000000001}]},{"osd":7,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.498},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.379}]}]},{"osd":5,"up_from":13,"seq":55834574875,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95336,"kb_used_data":46152,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94276504,"statfs":{"total":96636764160,"available":96539140096,"internally_reserved":0,"allocated":47259648,"data_stored":58835974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.41299999999999998}]},{"osd":1,"last update":"Sat Sep 21 15:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.38400000000000001}]},{"osd":2,"last update":"Sat Sep 21 15:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.42799999999999999}]},{"osd":3,"last update":"Sat Sep 21 15:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.443},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.6140000000000001,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.39700000000000002}]},{"osd":4,"last update":"Sat Sep 21 15:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":5.6399999999999997,"5min":5.6399999999999997,"15min":5.6399999999999997},"last":0.316},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.27700000000000002}]},{"osd":6,"last update":"Sat Sep 21 15:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.34599999999999997}]},{"osd":7,"last update":"Sat Sep 21 15:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.373},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.29699999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574874,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":144188,"kb_used_data":68776,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94227652,"statfs":{"total":96636764160,"available":96489115648,"internally_reserved":0,"allocated":70426624,"data_stored":92385875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77199814},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:49:52 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.39000000000000001}]},{"osd":1,"last update":"Sat Sep 21 15:49:52 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.48799999999999999}]},{"osd":2,"last update":"Sat Sep 21 15:49:52 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.437,"15min":0.437},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.54000000000000004}]},{"osd":3,"last update":"Sat Sep 21 15:49:52 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.57499999999999996}]},{"osd":5,"last update":"Sat Sep 21 15:49:52 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":2.4489999999999998,"5min":2.4489999999999998,"15min":2.4489999999999998},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.46400000000000002}]},{"osd":6,"last update":"Sat Sep 21 15:49:52 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":2.7709999999999999,"5min":2.7709999999999999,"15min":2.7709999999999999},"last":0.5}]},{"osd":7,"last update":"Sat Sep 21 15:49:52 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.8069999999999999,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.45400000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574875,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":104120,"kb_used_data":54928,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94267720,"statfs":{"total":96636764160,"available":96530145280,"internally_reserved":0,"allocated":56246272,"data_stored":76314126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50330056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.442,"5min":0.442,"15min":0.442},"last":0.27600000000000002}]},{"osd":1,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.28999999999999998}]},{"osd":2,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.35999999999999999}]},{"osd":4,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.498},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.51800000000000002}]},{"osd":5,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.41399999999999998}]},{"osd":6,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.372}]},{"osd":7,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.45200000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574875,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51636,"kb_used_data":16424,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94320204,"statfs":{"total":96636764160,"available":96583888896,"internally_reserved":0,"allocated":16818176,"data_stored":21623934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.39900000000000002}]},{"osd":1,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":1.621,"5min":1.621,"15min":1.621},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.27600000000000002}]},{"osd":3,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":11.56,"5min":11.56,"15min":11.56},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.36399999999999999}]},{"osd":4,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.653,"5min":1.653,"15min":1.653},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":2.4660000000000002,"5min":2.4660000000000002,"15min":2.4660000000000002},"last":0.64200000000000002}]},{"osd":5,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.65400000000000003}]},{"osd":6,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.626},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.49199999999999999}]},{"osd":7,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.55600000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574874,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60112,"kb_used_data":24900,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94311728,"statfs":{"total":96636764160,"available":96575209472,"internally_reserved":0,"allocated":25497600,"data_stored":35744602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.315,"15min":0.315},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"last":0.20599999999999999}]},{"osd":2,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":5.0670000000000002,"5min":5.0670000000000002,"15min":5.0670000000000002},"last":0.30299999999999999}]},{"osd":3,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":16.032,"5min":16.032,"15min":16.032},"last":0.219},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.315}]},{"osd":4,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2.2269999999999999,"5min":2.2269999999999999,"15min":2.2269999999999999},"last":0.39900000000000002}]},{"osd":5,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":2.2890000000000001,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":4.6989999999999998,"5min":4.6989999999999998,"15min":4.6989999999999998},"last":0.434}]},{"osd":6,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.373},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":2.262,"5min":2.262,"15min":2.262},"last":0.443}]},{"osd":7,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":4.742,"5min":4.742,"15min":4.742},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.47099999999999997}]}]},{"osd":0,"up_from":93,"seq":399431958531,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73512,"kb_used_data":40980,"kb_used_omap":13,"kb_used_meta":32498,"kb_avail":94298328,"statfs":{"total":96636764160,"available":96561487872,"internally_reserved":0,"allocated":41963520,"data_stored":54773397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13968,"internal_metadata":33278320},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40157184,"data_stored":53613697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5973,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24268800,"data_stored":35168995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":15593472,"data_stored":21048346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53026816,"data_stored":71791765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":64872448,"data_stored":83182490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":41746432,"data_stored":49783990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":14618624,"data_stored":21021038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":41398272,"data_stored":48998643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T15:50:41.438 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-21T15:50:41.439 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:41.430+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:41.439 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T15:50:41.439 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T15:50:41.533 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:41.531+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:41.576 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:41.576 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:41.577 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:50:41.577 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:41.585 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T15:50:41.597 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-09-21T15:50:41.597 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:50:41.613 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:50:41.614 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:50:41.629 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:50:41.629 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:50:41.634 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:41.632+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:41.645 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:50:41.646 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:50:41.662 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:50:41.662 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:50:41.677 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:50:41.678 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:50:41.687 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:41.688 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:41.688 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:50:41.688 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:50:41.688 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:41.694 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:50:41.694 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:50:41.697 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T15:50:41.713 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:50:41.713 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:50:41.717 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T15:50:41.741 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:41.740+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:41.771 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:50:41.772 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-09-21T15:50:41.785 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":154,"stamp":"2024-09-21T15:50:41.703395+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1345918076,"num_objects":2105,"num_object_clones":245,"num_object_copies":4210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2105,"num_whiteouts":1224,"num_read":198207,"num_read_kb":834540,"num_write":21708,"num_write_kb":305196,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":841,"num_bytes_recovered":47753679,"num_keys_recovered":165,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17877,"ondisk_log_size":17877,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":699584,"kb_used_data":316296,"kb_used_omap":24,"kb_used_meta":383079,"kb_avail":754275136,"statfs":{"total":773094113280,"available":772377739264,"internally_reserved":0,"allocated":323887104,"data_stored":424597741,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25118,"internal_metadata":392273378},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-139078846,"num_objects":-135,"num_object_clones":-47,"num_object_copies":-270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-135,"num_whiteouts":-32,"num_read":1252,"num_read_kb":1066,"num_write":210,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.036334"},"pg_stats":[{"pgid":"3.1f","version":"102'373","reported_seq":1777,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T15:50:37.680975+0000","last_change":"2024-09-21T15:50:37.680975+0000","last_active":"2024-09-21T15:50:37.680975+0000","last_peered":"2024-09-21T15:50:37.680975+0000","last_clean":"2024-09-21T15:50:37.680975+0000","last_became_active":"2024-09-21T15:50:32.670964+0000","last_became_peered":"2024-09-21T15:50:32.670964+0000","last_unstale":"2024-09-21T15:50:37.680975+0000","last_undegraded":"2024-09-21T15:50:37.680975+0000","last_fullsized":"2024-09-21T15:50:37.680975+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":373,"log_dups_size":0,"ondisk_log_size":373,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:09:09.918864+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.011407993999999999,"stat_sum":{"num_bytes":5107497,"num_objects":47,"num_object_clones":3,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":31,"num_read":2172,"num_read_kb":10213,"num_write":413,"num_write_kb":5950,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":1974433,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"29","length":"4"}]},{"pgid":"3.1e","version":"103'483","reported_seq":5409,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-21T15:50:39.773593+0000","last_change":"2024-09-21T15:50:39.773790+0000","last_active":"2024-09-21T15:50:39.773593+0000","last_peered":"2024-09-21T15:50:39.773593+0000","last_clean":"2024-09-21T15:50:39.773593+0000","last_became_active":"2024-09-21T15:50:20.087949+0000","last_became_peered":"2024-09-21T15:50:20.087949+0000","last_unstale":"2024-09-21T15:50:39.773593+0000","last_undegraded":"2024-09-21T15:50:39.773593+0000","last_fullsized":"2024-09-21T15:50:39.773593+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":483,"log_dups_size":0,"ondisk_log_size":483,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:26:31.786747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000310413,"stat_sum":{"num_bytes":53219311,"num_objects":77,"num_object_clones":10,"num_object_copies":154,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77,"num_whiteouts":45,"num_read":6840,"num_read_kb":25381,"num_write":497,"num_write_kb":4338,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"27","length":"6"}]},{"pgid":"3.1d","version":"103'547","reported_seq":9800,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-21T15:50:39.702196+0000","last_change":"2024-09-21T15:50:39.702399+0000","last_active":"2024-09-21T15:50:39.702196+0000","last_peered":"2024-09-21T15:50:39.702196+0000","last_clean":"2024-09-21T15:50:39.702196+0000","last_became_active":"2024-09-21T15:49:39.833206+0000","last_became_peered":"2024-09-21T15:49:39.833206+0000","last_unstale":"2024-09-21T15:50:39.702196+0000","last_undegraded":"2024-09-21T15:50:39.702196+0000","last_fullsized":"2024-09-21T15:50:39.702196+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":547,"log_dups_size":0,"ondisk_log_size":547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:36:14.082318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035473200000000002,"stat_sum":{"num_bytes":71988001,"num_objects":86,"num_object_clones":16,"num_object_copies":172,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":86,"num_whiteouts":46,"num_read":10437,"num_read_kb":28908,"num_write":629,"num_write_kb":8866,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"27","length":"6"}]},{"pgid":"3.1c","version":"104'510","reported_seq":4082,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-21T15:50:39.731602+0000","last_change":"2024-09-21T15:50:39.702815+0000","last_active":"2024-09-21T15:50:39.731602+0000","last_peered":"2024-09-21T15:50:39.731602+0000","last_clean":"2024-09-21T15:50:39.731602+0000","last_became_active":"2024-09-21T15:49:39.833113+0000","last_became_peered":"2024-09-21T15:49:39.833113+0000","last_unstale":"2024-09-21T15:50:39.731602+0000","last_undegraded":"2024-09-21T15:50:39.731602+0000","last_fullsized":"2024-09-21T15:50:39.731602+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":510,"log_dups_size":0,"ondisk_log_size":510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:43:57.592707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00071529000000000004,"stat_sum":{"num_bytes":11366202,"num_objects":55,"num_object_clones":5,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":31,"num_read":4982,"num_read_kb":17094,"num_write":544,"num_write_kb":4783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"27","length":"6"}]},{"pgid":"3.1b","version":"100'441","reported_seq":4033,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T15:50:37.676929+0000","last_change":"2024-09-21T15:50:37.677124+0000","last_active":"2024-09-21T15:50:37.676929+0000","last_peered":"2024-09-21T15:50:37.676929+0000","last_clean":"2024-09-21T15:50:37.676929+0000","last_became_active":"2024-09-21T15:50:32.674250+0000","last_became_peered":"2024-09-21T15:50:32.674250+0000","last_unstale":"2024-09-21T15:50:37.676929+0000","last_undegraded":"2024-09-21T15:50:37.676929+0000","last_fullsized":"2024-09-21T15:50:37.676929+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":441,"log_dups_size":0,"ondisk_log_size":441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:54:57.399818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0066613599999999999,"stat_sum":{"num_bytes":54600659,"num_objects":74,"num_object_clones":5,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":42,"num_read":4613,"num_read_kb":29921,"num_write":447,"num_write_kb":13501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":4759000,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"29","length":"4"}]},{"pgid":"3.1a","version":"103'371","reported_seq":6335,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-21T15:50:39.804248+0000","last_change":"2024-09-21T15:50:39.702513+0000","last_active":"2024-09-21T15:50:39.804248+0000","last_peered":"2024-09-21T15:50:39.804248+0000","last_clean":"2024-09-21T15:50:39.804248+0000","last_became_active":"2024-09-21T15:49:06.693903+0000","last_became_peered":"2024-09-21T15:49:06.693903+0000","last_unstale":"2024-09-21T15:50:39.804248+0000","last_undegraded":"2024-09-21T15:50:39.804248+0000","last_fullsized":"2024-09-21T15:50:39.804248+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":371,"log_dups_size":0,"ondisk_log_size":371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:41:45.432893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000465484,"stat_sum":{"num_bytes":52938660,"num_objects":67,"num_object_clones":9,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":36,"num_read":6761,"num_read_kb":31330,"num_write":384,"num_write_kb":9470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"27","length":"6"}]},{"pgid":"3.19","version":"103'412","reported_seq":8058,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-21T15:50:39.699393+0000","last_change":"2024-09-21T15:50:39.699605+0000","last_active":"2024-09-21T15:50:39.699393+0000","last_peered":"2024-09-21T15:50:39.699393+0000","last_clean":"2024-09-21T15:50:39.699393+0000","last_became_active":"2024-09-21T15:49:06.691222+0000","last_became_peered":"2024-09-21T15:49:06.691222+0000","last_unstale":"2024-09-21T15:50:39.699393+0000","last_undegraded":"2024-09-21T15:50:39.699393+0000","last_fullsized":"2024-09-21T15:50:39.699393+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":412,"log_dups_size":0,"ondisk_log_size":412,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:31:29.545866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000340033,"stat_sum":{"num_bytes":62034124,"num_objects":68,"num_object_clones":5,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":36,"num_read":8235,"num_read_kb":39548,"num_write":478,"num_write_kb":7403,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"27","length":"6"}]},{"pgid":"3.18","version":"97'407","reported_seq":3467,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T15:50:37.669493+0000","last_change":"2024-09-21T15:50:37.669931+0000","last_active":"2024-09-21T15:50:37.669493+0000","last_peered":"2024-09-21T15:50:37.669493+0000","last_clean":"2024-09-21T15:50:37.669493+0000","last_became_active":"2024-09-21T15:50:32.672791+0000","last_became_peered":"2024-09-21T15:50:32.672791+0000","last_unstale":"2024-09-21T15:50:37.669493+0000","last_undegraded":"2024-09-21T15:50:37.669493+0000","last_fullsized":"2024-09-21T15:50:37.669493+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":407,"log_dups_size":0,"ondisk_log_size":407,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:07:19.540473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010436410000000001,"stat_sum":{"num_bytes":48735247,"num_objects":67,"num_object_clones":9,"num_object_copies":134,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":30,"num_read":5193,"num_read_kb":30076,"num_write":442,"num_write_kb":11051,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":2076323,"num_keys_recovered":11,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"29","length":"4"}]},{"pgid":"3.17","version":"102'924","reported_seq":8830,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T15:50:37.772305+0000","last_change":"2024-09-21T15:50:37.674917+0000","last_active":"2024-09-21T15:50:37.772305+0000","last_peered":"2024-09-21T15:50:37.772305+0000","last_clean":"2024-09-21T15:50:37.772305+0000","last_became_active":"2024-09-21T15:50:32.668771+0000","last_became_peered":"2024-09-21T15:50:32.668771+0000","last_unstale":"2024-09-21T15:50:37.772305+0000","last_undegraded":"2024-09-21T15:50:37.772305+0000","last_fullsized":"2024-09-21T15:50:37.772305+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":924,"log_dups_size":0,"ondisk_log_size":924,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:39:43.554150+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0055834279999999997,"stat_sum":{"num_bytes":15502015,"num_objects":68,"num_object_clones":6,"num_object_copies":136,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":68,"num_whiteouts":41,"num_read":12072,"num_read_kb":20937,"num_write":1213,"num_write_kb":8234,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":3319185,"num_keys_recovered":34,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"29","length":"4"}]},{"pgid":"3.16","version":"104'475","reported_seq":6341,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-21T15:50:39.758595+0000","last_change":"2024-09-21T15:50:39.758595+0000","last_active":"2024-09-21T15:50:39.758595+0000","last_peered":"2024-09-21T15:50:39.758595+0000","last_clean":"2024-09-21T15:50:39.758595+0000","last_became_active":"2024-09-21T15:49:06.688808+0000","last_became_peered":"2024-09-21T15:49:06.688808+0000","last_unstale":"2024-09-21T15:50:39.758595+0000","last_undegraded":"2024-09-21T15:50:39.758595+0000","last_fullsized":"2024-09-21T15:50:39.758595+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":475,"log_dups_size":0,"ondisk_log_size":475,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:43:02.012254+0000","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0.0036253359999999998,"stat_sum":{"num_bytes":81763516,"num_objects":63,"num_object_clones":6,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":63,"num_whiteouts":36,"num_read":7324,"num_read_kb":49627,"num_write":492,"num_write_kb":13812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"27","length":"6"}]},{"pgid":"3.15","version":"102'490","reported_seq":3594,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-21T15:50:39.748785+0000","last_change":"2024-09-21T15:50:39.749098+0000","last_active":"2024-09-21T15:50:39.748785+0000","last_peered":"2024-09-21T15:50:39.748785+0000","last_clean":"2024-09-21T15:50:39.748785+0000","last_became_active":"2024-09-21T15:49:06.691091+0000","last_became_peered":"2024-09-21T15:49:06.691091+0000","last_unstale":"2024-09-21T15:50:39.748785+0000","last_undegraded":"2024-09-21T15:50:39.748785+0000","last_fullsized":"2024-09-21T15:50:39.748785+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":490,"log_dups_size":0,"ondisk_log_size":490,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:13:12.258161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00141209,"stat_sum":{"num_bytes":25828601,"num_objects":58,"num_object_clones":8,"num_object_copies":116,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":58,"num_whiteouts":35,"num_read":4033,"num_read_kb":27130,"num_write":614,"num_write_kb":17935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"27","length":"6"}]},{"pgid":"3.14","version":"102'1148","reported_seq":4096,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-21T15:50:39.702920+0000","last_change":"2024-09-21T15:50:39.703344+0000","last_active":"2024-09-21T15:50:39.702920+0000","last_peered":"2024-09-21T15:50:39.702920+0000","last_clean":"2024-09-21T15:50:39.702920+0000","last_became_active":"2024-09-21T15:49:06.688803+0000","last_became_peered":"2024-09-21T15:49:06.688803+0000","last_unstale":"2024-09-21T15:50:39.702920+0000","last_undegraded":"2024-09-21T15:50:39.702920+0000","last_fullsized":"2024-09-21T15:50:39.702920+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":1148,"log_dups_size":0,"ondisk_log_size":1148,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:18:47.662631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001174029,"stat_sum":{"num_bytes":63762684,"num_objects":51,"num_object_clones":3,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":28,"num_read":7660,"num_read_kb":18155,"num_write":1867,"num_write_kb":6175,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"27","length":"6"}]},{"pgid":"2.7","version":"0'0","reported_seq":100,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-21T15:50:38.790164+0000","last_change":"2024-09-21T15:48:59.666431+0000","last_active":"2024-09-21T15:50:38.790164+0000","last_peered":"2024-09-21T15:50:38.790164+0000","last_clean":"2024-09-21T15:50:38.790164+0000","last_became_active":"2024-09-21T15:48:57.583665+0000","last_became_peered":"2024-09-21T15:48:57.583665+0000","last_unstale":"2024-09-21T15:50:38.790164+0000","last_undegraded":"2024-09-21T15:50:38.790164+0000","last_fullsized":"2024-09-21T15:50:38.790164+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:52:09.228964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031098399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"102'363","reported_seq":2761,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T15:50:37.675954+0000","last_change":"2024-09-21T15:50:37.675954+0000","last_active":"2024-09-21T15:50:37.675954+0000","last_peered":"2024-09-21T15:50:37.675954+0000","last_clean":"2024-09-21T15:50:37.675954+0000","last_became_active":"2024-09-21T15:50:32.671424+0000","last_became_peered":"2024-09-21T15:50:32.671424+0000","last_unstale":"2024-09-21T15:50:37.675954+0000","last_undegraded":"2024-09-21T15:50:37.675954+0000","last_fullsized":"2024-09-21T15:50:37.675954+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":363,"log_dups_size":0,"ondisk_log_size":363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:04:14.554732+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.0069010080000000001,"stat_sum":{"num_bytes":11403537,"num_objects":61,"num_object_clones":7,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":41,"num_read":2599,"num_read_kb":12929,"num_write":362,"num_write_kb":8387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":4494388,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"29","length":"4"}]},{"pgid":"2.6","version":"0'0","reported_seq":100,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-21T15:50:38.697723+0000","last_change":"2024-09-21T15:48:59.589790+0000","last_active":"2024-09-21T15:50:38.697723+0000","last_peered":"2024-09-21T15:50:38.697723+0000","last_clean":"2024-09-21T15:50:38.697723+0000","last_became_active":"2024-09-21T15:48:57.582121+0000","last_became_peered":"2024-09-21T15:48:57.582121+0000","last_unstale":"2024-09-21T15:50:38.697723+0000","last_undegraded":"2024-09-21T15:50:38.697723+0000","last_fullsized":"2024-09-21T15:50:38.697723+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:33:47.592624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039654700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"103'522","reported_seq":7114,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-21T15:50:39.749134+0000","last_change":"2024-09-21T15:50:39.749257+0000","last_active":"2024-09-21T15:50:39.749134+0000","last_peered":"2024-09-21T15:50:39.749134+0000","last_clean":"2024-09-21T15:50:39.749134+0000","last_became_active":"2024-09-21T15:50:20.087935+0000","last_became_peered":"2024-09-21T15:50:20.087935+0000","last_unstale":"2024-09-21T15:50:39.749134+0000","last_undegraded":"2024-09-21T15:50:39.749134+0000","last_fullsized":"2024-09-21T15:50:39.749134+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":522,"log_dups_size":0,"ondisk_log_size":522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:27:45.090104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015282539999999999,"stat_sum":{"num_bytes":33576848,"num_objects":69,"num_object_clones":8,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":41,"num_read":8167,"num_read_kb":21012,"num_write":508,"num_write_kb":5391,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"27","length":"6"}]},{"pgid":"2.5","version":"0'0","reported_seq":113,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-21T15:50:38.692676+0000","last_change":"2024-09-21T15:50:30.146743+0000","last_active":"2024-09-21T15:50:38.692676+0000","last_peered":"2024-09-21T15:50:38.692676+0000","last_clean":"2024-09-21T15:50:38.692676+0000","last_became_active":"2024-09-21T15:50:30.146428+0000","last_became_peered":"2024-09-21T15:50:30.146428+0000","last_unstale":"2024-09-21T15:50:38.692676+0000","last_undegraded":"2024-09-21T15:50:38.692676+0000","last_fullsized":"2024-09-21T15:50:38.692676+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:58:38.689807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036074600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"104'615","reported_seq":8435,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-21T15:50:39.789080+0000","last_change":"2024-09-21T15:50:39.699669+0000","last_active":"2024-09-21T15:50:39.789080+0000","last_peered":"2024-09-21T15:50:39.789080+0000","last_clean":"2024-09-21T15:50:39.789080+0000","last_became_active":"2024-09-21T15:50:35.682788+0000","last_became_peered":"2024-09-21T15:50:35.682788+0000","last_unstale":"2024-09-21T15:50:39.789080+0000","last_undegraded":"2024-09-21T15:50:39.789080+0000","last_fullsized":"2024-09-21T15:50:39.789080+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":615,"log_dups_size":0,"ondisk_log_size":615,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:20:40.695690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035933100000000001,"stat_sum":{"num_bytes":36608615,"num_objects":66,"num_object_clones":13,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":34,"num_read":10649,"num_read_kb":27356,"num_write":808,"num_write_kb":11002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":175,"num_bytes_recovered":14134268,"num_keys_recovered":36,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"27","length":"6"}]},{"pgid":"2.4","version":"0'0","reported_seq":113,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-21T15:50:38.697660+0000","last_change":"2024-09-21T15:50:30.146654+0000","last_active":"2024-09-21T15:50:38.697660+0000","last_peered":"2024-09-21T15:50:38.697660+0000","last_clean":"2024-09-21T15:50:38.697660+0000","last_became_active":"2024-09-21T15:50:30.146273+0000","last_became_peered":"2024-09-21T15:50:30.146273+0000","last_unstale":"2024-09-21T15:50:38.697660+0000","last_undegraded":"2024-09-21T15:50:38.697660+0000","last_fullsized":"2024-09-21T15:50:38.697660+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:21:48.178519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040525700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"104'527","reported_seq":4991,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-21T15:50:39.776218+0000","last_change":"2024-09-21T15:50:39.776352+0000","last_active":"2024-09-21T15:50:39.776218+0000","last_peered":"2024-09-21T15:50:39.776218+0000","last_clean":"2024-09-21T15:50:39.776218+0000","last_became_active":"2024-09-21T15:49:39.835880+0000","last_became_peered":"2024-09-21T15:49:39.835880+0000","last_unstale":"2024-09-21T15:50:39.776218+0000","last_undegraded":"2024-09-21T15:50:39.776218+0000","last_fullsized":"2024-09-21T15:50:39.776218+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":527,"log_dups_size":0,"ondisk_log_size":527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:28:03.488776+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0032615970000000002,"stat_sum":{"num_bytes":65511075,"num_objects":75,"num_object_clones":9,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":75,"num_whiteouts":40,"num_read":5513,"num_read_kb":33760,"num_write":631,"num_write_kb":16321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"27","length":"6"}]},{"pgid":"3.3","version":"104'678","reported_seq":3378,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-21T15:50:39.818869+0000","last_change":"2024-09-21T15:50:39.703294+0000","last_active":"2024-09-21T15:50:39.818869+0000","last_peered":"2024-09-21T15:50:39.818869+0000","last_clean":"2024-09-21T15:50:39.818869+0000","last_became_active":"2024-09-21T15:50:32.677456+0000","last_became_peered":"2024-09-21T15:50:32.677456+0000","last_unstale":"2024-09-21T15:50:39.818869+0000","last_undegraded":"2024-09-21T15:50:39.818869+0000","last_fullsized":"2024-09-21T15:50:39.818869+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":678,"log_dups_size":0,"ondisk_log_size":678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:46:31.194345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00118261,"stat_sum":{"num_bytes":23140359,"num_objects":74,"num_object_clones":13,"num_object_copies":148,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":74,"num_whiteouts":46,"num_read":4388,"num_read_kb":18663,"num_write":747,"num_write_kb":7282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":632176,"num_keys_recovered":24,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"27","length":"6"}]},{"pgid":"2.2","version":"20'2","reported_seq":79,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-21T15:50:38.697561+0000","last_change":"2024-09-21T15:49:39.834640+0000","last_active":"2024-09-21T15:50:38.697561+0000","last_peered":"2024-09-21T15:50:38.697561+0000","last_clean":"2024-09-21T15:50:38.697561+0000","last_became_active":"2024-09-21T15:49:39.834207+0000","last_became_peered":"2024-09-21T15:49:39.834207+0000","last_unstale":"2024-09-21T15:50:38.697561+0000","last_undegraded":"2024-09-21T15:50:38.697561+0000","last_fullsized":"2024-09-21T15:50:38.697561+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:02:54.255716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"98'346","reported_seq":4475,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-21T15:50:39.699652+0000","last_change":"2024-09-21T15:50:39.699821+0000","last_active":"2024-09-21T15:50:39.699652+0000","last_peered":"2024-09-21T15:50:39.699652+0000","last_clean":"2024-09-21T15:50:39.699652+0000","last_became_active":"2024-09-21T15:50:35.682001+0000","last_became_peered":"2024-09-21T15:50:35.682001+0000","last_unstale":"2024-09-21T15:50:39.699652+0000","last_undegraded":"2024-09-21T15:50:39.699652+0000","last_fullsized":"2024-09-21T15:50:39.699652+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":346,"log_dups_size":0,"ondisk_log_size":346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:09:15.325457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00043887400000000001,"stat_sum":{"num_bytes":20675160,"num_objects":57,"num_object_clones":4,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":35,"num_read":4043,"num_read_kb":23169,"num_write":345,"num_write_kb":4139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":187,"num_bytes_recovered":4305276,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"27","length":"6"}]},{"pgid":"2.1","version":"0'0","reported_seq":17,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-21T15:50:38.703626+0000","last_change":"2024-09-21T15:50:32.672133+0000","last_active":"2024-09-21T15:50:38.703626+0000","last_peered":"2024-09-21T15:50:38.703626+0000","last_clean":"2024-09-21T15:50:38.703626+0000","last_became_active":"2024-09-21T15:50:32.671865+0000","last_became_peered":"2024-09-21T15:50:32.671865+0000","last_unstale":"2024-09-21T15:50:38.703626+0000","last_undegraded":"2024-09-21T15:50:38.703626+0000","last_fullsized":"2024-09-21T15:50:38.703626+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:55:23.999144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"101'477","reported_seq":2124,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T15:50:37.676120+0000","last_change":"2024-09-21T15:50:37.676722+0000","last_active":"2024-09-21T15:50:37.676120+0000","last_peered":"2024-09-21T15:50:37.676120+0000","last_clean":"2024-09-21T15:50:37.676120+0000","last_became_active":"2024-09-21T15:50:32.671413+0000","last_became_peered":"2024-09-21T15:50:32.671413+0000","last_unstale":"2024-09-21T15:50:37.676120+0000","last_undegraded":"2024-09-21T15:50:37.676120+0000","last_fullsized":"2024-09-21T15:50:37.676120+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":477,"log_dups_size":0,"ondisk_log_size":477,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:04:19.509402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0071081549999999997,"stat_sum":{"num_bytes":26803231,"num_objects":65,"num_object_clones":5,"num_object_copies":130,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":65,"num_whiteouts":42,"num_read":3205,"num_read_kb":16147,"num_write":528,"num_write_kb":5961,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":1696547,"num_keys_recovered":12,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"29","length":"4"}]},{"pgid":"2.0","version":"0'0","reported_seq":100,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-21T15:50:38.692395+0000","last_change":"2024-09-21T15:49:00.114730+0000","last_active":"2024-09-21T15:50:38.692395+0000","last_peered":"2024-09-21T15:50:38.692395+0000","last_clean":"2024-09-21T15:50:38.692395+0000","last_became_active":"2024-09-21T15:48:57.585525+0000","last_became_peered":"2024-09-21T15:48:57.585525+0000","last_unstale":"2024-09-21T15:50:38.692395+0000","last_undegraded":"2024-09-21T15:50:38.692395+0000","last_fullsized":"2024-09-21T15:50:38.692395+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:35:35.786526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031691100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"103'369","reported_seq":2229,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-21T15:50:39.755047+0000","last_change":"2024-09-21T15:50:39.755413+0000","last_active":"2024-09-21T15:50:39.755047+0000","last_peered":"2024-09-21T15:50:39.755047+0000","last_clean":"2024-09-21T15:50:39.755047+0000","last_became_active":"2024-09-21T15:50:20.087681+0000","last_became_peered":"2024-09-21T15:50:20.087681+0000","last_unstale":"2024-09-21T15:50:39.755047+0000","last_undegraded":"2024-09-21T15:50:39.755047+0000","last_fullsized":"2024-09-21T15:50:39.755047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":369,"log_dups_size":0,"ondisk_log_size":369,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:07:05.059684+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049546400000000004,"stat_sum":{"num_bytes":67416002,"num_objects":61,"num_object_clones":5,"num_object_copies":122,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":33,"num_read":2107,"num_read_kb":34423,"num_write":371,"num_write_kb":12608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"27","length":"6"}]},{"pgid":"2.3","version":"18'1","reported_seq":72,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-21T15:50:38.703633+0000","last_change":"2024-09-21T15:49:39.834423+0000","last_active":"2024-09-21T15:50:38.703633+0000","last_peered":"2024-09-21T15:50:38.703633+0000","last_clean":"2024-09-21T15:50:38.703633+0000","last_became_active":"2024-09-21T15:49:39.833860+0000","last_became_peered":"2024-09-21T15:49:39.833860+0000","last_unstale":"2024-09-21T15:50:38.703633+0000","last_undegraded":"2024-09-21T15:50:38.703633+0000","last_fullsized":"2024-09-21T15:50:38.703633+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:11:43.630472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":261,"reported_epoch":103,"state":"active+clean","last_fresh":"2024-09-21T15:50:38.692410+0000","last_change":"2024-09-21T15:50:30.147280+0000","last_active":"2024-09-21T15:50:38.692410+0000","last_peered":"2024-09-21T15:50:38.692410+0000","last_clean":"2024-09-21T15:50:38.692410+0000","last_became_active":"2024-09-21T15:50:30.146987+0000","last_became_peered":"2024-09-21T15:50:30.146987+0000","last_unstale":"2024-09-21T15:50:38.692410+0000","last_undegraded":"2024-09-21T15:50:38.692410+0000","last_fullsized":"2024-09-21T15:50:38.692410+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:50.161884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:50.161884+0000","last_clean_scrub_stamp":"2024-09-21T15:48:50.161884+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:26:32.853650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"103'466","reported_seq":7009,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-21T15:50:39.774229+0000","last_change":"2024-09-21T15:50:39.774410+0000","last_active":"2024-09-21T15:50:39.774229+0000","last_peered":"2024-09-21T15:50:39.774229+0000","last_clean":"2024-09-21T15:50:39.774229+0000","last_became_active":"2024-09-21T15:49:18.669143+0000","last_became_peered":"2024-09-21T15:49:18.669143+0000","last_unstale":"2024-09-21T15:50:39.774229+0000","last_undegraded":"2024-09-21T15:50:39.774229+0000","last_fullsized":"2024-09-21T15:50:39.774229+0000","mapping_epoch":32,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":33,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":466,"log_dups_size":0,"ondisk_log_size":466,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:31:03.684897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012451039999999999,"stat_sum":{"num_bytes":41140696,"num_objects":72,"num_object_clones":7,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":72,"num_whiteouts":38,"num_read":7027,"num_read_kb":29405,"num_write":503,"num_write_kb":4783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":296439,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"27","length":"6"}]},{"pgid":"3.9","version":"103'346","reported_seq":2904,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-21T15:50:39.702467+0000","last_change":"2024-09-21T15:50:39.702848+0000","last_active":"2024-09-21T15:50:39.702467+0000","last_peered":"2024-09-21T15:50:39.702467+0000","last_clean":"2024-09-21T15:50:39.702467+0000","last_became_active":"2024-09-21T15:49:07.210506+0000","last_became_peered":"2024-09-21T15:49:07.210506+0000","last_unstale":"2024-09-21T15:50:39.702467+0000","last_undegraded":"2024-09-21T15:50:39.702467+0000","last_fullsized":"2024-09-21T15:50:39.702467+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":346,"log_dups_size":0,"ondisk_log_size":346,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:55:18.046738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00077619199999999998,"stat_sum":{"num_bytes":43716341,"num_objects":62,"num_object_clones":3,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":33,"num_read":2677,"num_read_kb":28218,"num_write":415,"num_write_kb":12085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"27","length":"6"}]},{"pgid":"3.a","version":"104'488","reported_seq":11814,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-21T15:50:39.781233+0000","last_change":"2024-09-21T15:50:39.781233+0000","last_active":"2024-09-21T15:50:39.781233+0000","last_peered":"2024-09-21T15:50:39.781233+0000","last_clean":"2024-09-21T15:50:39.781233+0000","last_became_active":"2024-09-21T15:49:06.694202+0000","last_became_peered":"2024-09-21T15:49:06.694202+0000","last_unstale":"2024-09-21T15:50:39.781233+0000","last_undegraded":"2024-09-21T15:50:39.781233+0000","last_fullsized":"2024-09-21T15:50:39.781233+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":488,"log_dups_size":0,"ondisk_log_size":488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:02:05.237705+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.0077539449999999999,"stat_sum":{"num_bytes":53937104,"num_objects":62,"num_object_clones":8,"num_object_copies":124,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":34,"num_read":12422,"num_read_kb":63120,"num_write":497,"num_write_kb":14195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"27","length":"6"}]},{"pgid":"3.b","version":"101'808","reported_seq":3118,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T15:50:37.676760+0000","last_change":"2024-09-21T15:50:37.676904+0000","last_active":"2024-09-21T15:50:37.676760+0000","last_peered":"2024-09-21T15:50:37.676760+0000","last_clean":"2024-09-21T15:50:37.676760+0000","last_became_active":"2024-09-21T15:50:30.149210+0000","last_became_peered":"2024-09-21T15:50:30.149210+0000","last_unstale":"2024-09-21T15:50:37.676760+0000","last_undegraded":"2024-09-21T15:50:37.676760+0000","last_fullsized":"2024-09-21T15:50:37.676760+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":808,"log_dups_size":0,"ondisk_log_size":808,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:51:30.431071+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0067440240000000004,"stat_sum":{"num_bytes":32234670,"num_objects":56,"num_object_clones":8,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":32,"num_read":5903,"num_read_kb":20139,"num_write":1215,"num_write_kb":7745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":5514164,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"29","length":"4"}]},{"pgid":"3.c","version":"104'643","reported_seq":6279,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-21T15:50:39.816061+0000","last_change":"2024-09-21T15:50:39.773314+0000","last_active":"2024-09-21T15:50:39.816061+0000","last_peered":"2024-09-21T15:50:39.816061+0000","last_clean":"2024-09-21T15:50:39.816061+0000","last_became_active":"2024-09-21T15:49:39.835282+0000","last_became_peered":"2024-09-21T15:49:39.835282+0000","last_unstale":"2024-09-21T15:50:39.816061+0000","last_undegraded":"2024-09-21T15:50:39.816061+0000","last_fullsized":"2024-09-21T15:50:39.816061+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":643,"log_dups_size":0,"ondisk_log_size":643,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:51:40.356906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00026967099999999997,"stat_sum":{"num_bytes":24065080,"num_objects":69,"num_object_clones":6,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":41,"num_read":8370,"num_read_kb":25272,"num_write":790,"num_write_kb":4984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"27","length":"6"}]},{"pgid":"3.d","version":"104'837","reported_seq":3915,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-21T15:50:39.748944+0000","last_change":"2024-09-21T15:50:39.748944+0000","last_active":"2024-09-21T15:50:39.748944+0000","last_peered":"2024-09-21T15:50:39.748944+0000","last_clean":"2024-09-21T15:50:39.748944+0000","last_became_active":"2024-09-21T15:49:06.689203+0000","last_became_peered":"2024-09-21T15:49:06.689203+0000","last_unstale":"2024-09-21T15:50:39.748944+0000","last_undegraded":"2024-09-21T15:50:39.748944+0000","last_fullsized":"2024-09-21T15:50:39.748944+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":837,"log_dups_size":0,"ondisk_log_size":837,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:22:17.578359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001401926,"stat_sum":{"num_bytes":65898393,"num_objects":70,"num_object_clones":9,"num_object_copies":140,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":70,"num_whiteouts":43,"num_read":6938,"num_read_kb":27800,"num_write":1112,"num_write_kb":14803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"27","length":"6"}]},{"pgid":"3.e","version":"103'484","reported_seq":7704,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-21T15:50:39.748983+0000","last_change":"2024-09-21T15:50:39.749218+0000","last_active":"2024-09-21T15:50:39.748983+0000","last_peered":"2024-09-21T15:50:39.748983+0000","last_clean":"2024-09-21T15:50:39.748983+0000","last_became_active":"2024-09-21T15:49:06.690101+0000","last_became_peered":"2024-09-21T15:49:06.690101+0000","last_unstale":"2024-09-21T15:50:39.748983+0000","last_undegraded":"2024-09-21T15:50:39.748983+0000","last_fullsized":"2024-09-21T15:50:39.748983+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":484,"log_dups_size":0,"ondisk_log_size":484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:43:30.659104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001511624,"stat_sum":{"num_bytes":104751931,"num_objects":69,"num_object_clones":13,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":69,"num_whiteouts":40,"num_read":8382,"num_read_kb":42678,"num_write":461,"num_write_kb":16261,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"27","length":"6"}]},{"pgid":"3.f","version":"100'610","reported_seq":3400,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-21T15:50:39.800086+0000","last_change":"2024-09-21T15:50:39.749454+0000","last_active":"2024-09-21T15:50:39.800086+0000","last_peered":"2024-09-21T15:50:39.800086+0000","last_clean":"2024-09-21T15:50:39.800086+0000","last_became_active":"2024-09-21T15:49:06.694250+0000","last_became_peered":"2024-09-21T15:49:06.694250+0000","last_unstale":"2024-09-21T15:50:39.800086+0000","last_undegraded":"2024-09-21T15:50:39.800086+0000","last_fullsized":"2024-09-21T15:50:39.800086+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":610,"log_dups_size":0,"ondisk_log_size":610,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:21:37.777769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0015581709999999999,"stat_sum":{"num_bytes":18103488,"num_objects":76,"num_object_clones":8,"num_object_copies":152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":50,"num_read":4638,"num_read_kb":15668,"num_write":817,"num_write_kb":5055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"27","length":"6"}]},{"pgid":"3.10","version":"100'432","reported_seq":2820,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-21T15:50:39.776606+0000","last_change":"2024-09-21T15:50:39.776999+0000","last_active":"2024-09-21T15:50:39.776606+0000","last_peered":"2024-09-21T15:50:39.776606+0000","last_clean":"2024-09-21T15:50:39.776606+0000","last_became_active":"2024-09-21T15:50:32.676147+0000","last_became_peered":"2024-09-21T15:50:32.676147+0000","last_unstale":"2024-09-21T15:50:39.776606+0000","last_undegraded":"2024-09-21T15:50:39.776606+0000","last_fullsized":"2024-09-21T15:50:39.776606+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":432,"log_dups_size":0,"ondisk_log_size":432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:44:18.821310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033896400000000002,"stat_sum":{"num_bytes":9591471,"num_objects":56,"num_object_clones":4,"num_object_copies":112,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":44,"num_read":2952,"num_read_kb":11672,"num_write":357,"num_write_kb":4465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":1191778,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"27","length":"6"}]},{"pgid":"3.11","version":"102'925","reported_seq":3812,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-21T15:50:39.747597+0000","last_change":"2024-09-21T15:50:39.748029+0000","last_active":"2024-09-21T15:50:39.747597+0000","last_peered":"2024-09-21T15:50:39.747597+0000","last_clean":"2024-09-21T15:50:39.747597+0000","last_became_active":"2024-09-21T15:49:06.688798+0000","last_became_peered":"2024-09-21T15:49:06.688798+0000","last_unstale":"2024-09-21T15:50:39.747597+0000","last_undegraded":"2024-09-21T15:50:39.747597+0000","last_fullsized":"2024-09-21T15:50:39.747597+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":925,"log_dups_size":0,"ondisk_log_size":925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:48:33.679133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055274399999999998,"stat_sum":{"num_bytes":47370639,"num_objects":60,"num_object_clones":7,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":34,"num_read":6119,"num_read_kb":16744,"num_write":1282,"num_write_kb":8388,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"27","length":"6"}]},{"pgid":"3.12","version":"97'352","reported_seq":2753,"reported_epoch":102,"state":"active+clean","last_fresh":"2024-09-21T15:50:37.670058+0000","last_change":"2024-09-21T15:50:37.670498+0000","last_active":"2024-09-21T15:50:37.670058+0000","last_peered":"2024-09-21T15:50:37.670058+0000","last_clean":"2024-09-21T15:50:37.670058+0000","last_became_active":"2024-09-21T15:50:32.671686+0000","last_became_peered":"2024-09-21T15:50:32.671686+0000","last_unstale":"2024-09-21T15:50:37.670058+0000","last_undegraded":"2024-09-21T15:50:37.670058+0000","last_fullsized":"2024-09-21T15:50:37.670058+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":352,"log_dups_size":0,"ondisk_log_size":352,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:40:53.968477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013049859999999999,"stat_sum":{"num_bytes":39845746,"num_objects":64,"num_object_clones":8,"num_object_copies":128,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":41,"num_read":2441,"num_read_kb":11662,"num_write":386,"num_write_kb":13839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":3359702,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"29","length":"4"}]},{"pgid":"3.13","version":"102'888","reported_seq":7131,"reported_epoch":104,"state":"active+clean","last_fresh":"2024-09-21T15:50:39.747684+0000","last_change":"2024-09-21T15:50:39.748043+0000","last_active":"2024-09-21T15:50:39.747684+0000","last_peered":"2024-09-21T15:50:39.747684+0000","last_clean":"2024-09-21T15:50:39.747684+0000","last_became_active":"2024-09-21T15:49:06.690128+0000","last_became_peered":"2024-09-21T15:49:06.690128+0000","last_unstale":"2024-09-21T15:50:39.747684+0000","last_undegraded":"2024-09-21T15:50:39.747684+0000","last_fullsized":"2024-09-21T15:50:39.747684+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":888,"log_dups_size":0,"ondisk_log_size":888,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:11.763959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055121900000000004,"stat_sum":{"num_bytes":32690786,"num_objects":76,"num_object_clones":15,"num_object_copies":152,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":76,"num_whiteouts":45,"num_read":9229,"num_read_kb":26283,"num_write":1392,"num_write_kb":13566,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"27","length":"6"}]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1345327689,"num_objects":2101,"num_object_clones":245,"num_object_copies":4202,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2101,"num_whiteouts":1224,"num_read":198091,"num_read_kb":834440,"num_write":21547,"num_write_kb":302778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":841,"num_bytes_recovered":47753679,"num_keys_recovered":165,"num_objects_omap":38,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":295682048,"data_stored":384608964,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5973,"internal_metadata":0},"log_size":17757,"ondisk_log_size":17757,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574873,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":115768,"kb_used_data":44440,"kb_used_omap":1,"kb_used_meta":71294,"kb_avail":94256072,"statfs":{"total":96636764160,"available":96518217728,"internally_reserved":0,"allocated":45506560,"data_stored":54694203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":73005515},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.47899999999999998}]},{"osd":1,"last update":"Sat Sep 21 15:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":1.04,"5min":1.04,"15min":1.04},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.39400000000000002}]},{"osd":2,"last update":"Sat Sep 21 15:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.96499999999999997,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.371}]},{"osd":3,"last update":"Sat Sep 21 15:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.40600000000000003}]},{"osd":4,"last update":"Sat Sep 21 15:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.97599999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.27000000000000002}]},{"osd":5,"last update":"Sat Sep 21 15:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.434},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.42399999999999999}]},{"osd":6,"last update":"Sat Sep 21 15:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.46300000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574876,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54912,"kb_used_data":19696,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94316928,"statfs":{"total":96636764160,"available":96580534272,"internally_reserved":0,"allocated":20168704,"data_stored":30225630,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":36043207},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.36699999999999999}]},{"osd":1,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.40500000000000003}]},{"osd":2,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.45500000000000002}]},{"osd":3,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.84399999999999997,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.48899999999999999}]},{"osd":4,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.41799999999999998,"15min":0.41799999999999998},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.435},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.311}]},{"osd":5,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.32800000000000001}]},{"osd":7,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.30599999999999999},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.498},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.379}]}]},{"osd":5,"up_from":13,"seq":55834574875,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95336,"kb_used_data":46152,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94276504,"statfs":{"total":96636764160,"available":96539140096,"internally_reserved":0,"allocated":47259648,"data_stored":58835974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":50330054},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.41299999999999998}]},{"osd":1,"last update":"Sat Sep 21 15:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.91300000000000003,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.38400000000000001}]},{"osd":2,"last update":"Sat Sep 21 15:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":1.046,"5min":1.046,"15min":1.046},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.47599999999999998,"15min":0.47599999999999998},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.42799999999999999}]},{"osd":3,"last update":"Sat Sep 21 15:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.79600000000000004,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.443},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":1.6140000000000001,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.39700000000000002}]},{"osd":4,"last update":"Sat Sep 21 15:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":5.6399999999999997,"5min":5.6399999999999997,"15min":5.6399999999999997},"last":0.316},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.27700000000000002}]},{"osd":6,"last update":"Sat Sep 21 15:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.34599999999999997}]},{"osd":7,"last update":"Sat Sep 21 15:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.373},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.29699999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574874,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":144188,"kb_used_data":68776,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94227652,"statfs":{"total":96636764160,"available":96489115648,"internally_reserved":0,"allocated":70426624,"data_stored":92385875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":77199814},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:49:52 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.39000000000000001}]},{"osd":1,"last update":"Sat Sep 21 15:49:52 2024","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46100000000000002,"15min":0.46100000000000002},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.48799999999999999}]},{"osd":2,"last update":"Sat Sep 21 15:49:52 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.437,"15min":0.437},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.54000000000000004}]},{"osd":3,"last update":"Sat Sep 21 15:49:52 2024","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.45100000000000001,"15min":0.45100000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.57499999999999996}]},{"osd":5,"last update":"Sat Sep 21 15:49:52 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":2.4489999999999998,"5min":2.4489999999999998,"15min":2.4489999999999998},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.42599999999999999,"5min":0.42599999999999999,"15min":0.42599999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.46400000000000002}]},{"osd":6,"last update":"Sat Sep 21 15:49:52 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":2.7709999999999999,"5min":2.7709999999999999,"15min":2.7709999999999999},"last":0.5}]},{"osd":7,"last update":"Sat Sep 21 15:49:52 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":1.8069999999999999,"5min":1.8069999999999999,"15min":1.8069999999999999},"last":0.45400000000000001}]}]},{"osd":3,"up_from":13,"seq":55834574875,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":104120,"kb_used_data":54928,"kb_used_omap":1,"kb_used_meta":49150,"kb_avail":94267720,"statfs":{"total":96636764160,"available":96530145280,"internally_reserved":0,"allocated":56246272,"data_stored":76314126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":50330056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.442,"5min":0.442,"15min":0.442},"last":0.27600000000000002}]},{"osd":1,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.28999999999999998}]},{"osd":2,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.35999999999999999}]},{"osd":4,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.498},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.41599999999999998,"5min":0.41599999999999998,"15min":0.41599999999999998},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.51800000000000002}]},{"osd":5,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.41399999999999998}]},{"osd":6,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.36899999999999999},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.372}]},{"osd":7,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.90700000000000003,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.45200000000000001}]}]},{"osd":2,"up_from":13,"seq":55834574875,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":51636,"kb_used_data":16424,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94320204,"statfs":{"total":96636764160,"available":96583888896,"internally_reserved":0,"allocated":16818176,"data_stored":21623934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.39900000000000002}]},{"osd":1,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":1.621,"5min":1.621,"15min":1.621},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.27600000000000002}]},{"osd":3,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":11.56,"5min":11.56,"15min":11.56},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.80900000000000005,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.36399999999999999}]},{"osd":4,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.653,"5min":1.653,"15min":1.653},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":2.4660000000000002,"5min":2.4660000000000002,"15min":2.4660000000000002},"last":0.64200000000000002}]},{"osd":5,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.84699999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.54600000000000004,"15min":0.54600000000000004},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.92000000000000004,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.65400000000000003}]},{"osd":6,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.0109999999999999,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.626},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.49199999999999999}]},{"osd":7,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":1.089,"5min":1.089,"15min":1.089},"last":0.55600000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574874,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":60112,"kb_used_data":24900,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94311728,"statfs":{"total":96636764160,"available":96575209472,"internally_reserved":0,"allocated":25497600,"data_stored":35744602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.315,"15min":0.315},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"last":0.20599999999999999}]},{"osd":2,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.17799999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":5.0670000000000002,"5min":5.0670000000000002,"15min":5.0670000000000002},"last":0.30299999999999999}]},{"osd":3,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":1.0700000000000001,"5min":1.0700000000000001,"15min":1.0700000000000001},"min":{"1min":0.252,"5min":0.252,"15min":0.252},"max":{"1min":16.032,"5min":16.032,"15min":16.032},"last":0.219},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.315}]},{"osd":4,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.76900000000000002,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":2.2269999999999999,"5min":2.2269999999999999,"15min":2.2269999999999999},"last":0.39900000000000002}]},{"osd":5,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":2.2890000000000001,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":4.6989999999999998,"5min":4.6989999999999998,"15min":4.6989999999999998},"last":0.434}]},{"osd":6,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.373},{"interface":"front","average":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":2.262,"5min":2.262,"15min":2.262},"last":0.443}]},{"osd":7,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":4.742,"5min":4.742,"15min":4.742},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.89500000000000002,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.47099999999999997}]}]},{"osd":0,"up_from":93,"seq":399431958531,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":73512,"kb_used_data":40980,"kb_used_omap":13,"kb_used_meta":32498,"kb_avail":94298328,"statfs":{"total":96636764160,"available":96561487872,"internally_reserved":0,"allocated":41963520,"data_stored":54773397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13968,"internal_metadata":33278320},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29499999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32500000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57999999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.503}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40157184,"data_stored":53613697,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5973,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":24268800,"data_stored":35168995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":15593472,"data_stored":21048346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":53026816,"data_stored":71791765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":64872448,"data_stored":83182490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":41746432,"data_stored":49783990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":14618624,"data_stored":21021038,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":41398272,"data_stored":48998643,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T15:50:41.812 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:41.812 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:41.821 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:50:41.866 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:50:41.863+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:50:41.929 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:41.929 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:41.936 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:41.935+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:41.938 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:50:42.037 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:42.036+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:42.081 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:42.081 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:42.090 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:50:42.138 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:42.137+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:42.232 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:42.232 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:42.238 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:42.238+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:42.240 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:50:42.340 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:42.338+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:42.342 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:42.342 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:42.350 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:50:42.441 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:50:42.439+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:50:42.453 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:42.453 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:42.462 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:50:42.546 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:42.545+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:42.606 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:42.606 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:42.614 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:50:42.642 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:42.641+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:42.717 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is set 2024-09-21T15:50:42.743 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:42.742+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:42.770 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:42.770 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:42.779 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:50:42.865 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:50:42.864+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:50:42.923 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:42.923 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:42.932 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:50:42.973 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:42.972+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:43.045 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:43.043+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:43.074 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:43.074 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:43.082 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:50:43.146 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:50:43.145+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:50:43.184 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:43.184 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:43.193 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:50:43.277 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:43.276+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:43.335 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:43.335 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:43.344 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:50:43.348 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:43.346+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:43.449 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:43.448+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:43.486 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:43.486 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:43.495 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:50:43.550 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:43.549+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:43.608 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:43.609 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:43.617 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:50:43.658 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:43.657+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:43.725 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:50:43.751 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:43.750+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:43.769 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:43.769 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:43.778 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:50:43.852 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-09-21T15:50:43.852 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:50:43.863 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:43.862+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:43.868 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:50:43.869 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:50:43.884 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:50:43.884 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:50:43.900 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:50:43.900 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:50:43.916 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:50:43.916 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:50:43.924 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:43.924 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:50:43.924 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:50:43.924 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:50:43.924 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.924 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155031 3.1c 3.8612741c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:43.924 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448748+0000", 2024-09-21T15:50:43.924 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 21.473595109000001, 2024-09-21T15:50:43.924 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064010286999999999, 2024-09-21T15:50:43.924 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:43.924 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:43.924 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:43.924 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:43.924 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:43.925 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155031 2024-09-21T15:50:43.925 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.925 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:43.925 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.925 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:43.925 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448748+0000", 2024-09-21T15:50:43.925 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:43.925 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.925 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.925 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:43.925 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448748+0000", 2024-09-21T15:50:43.925 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:43.925 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.925 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.925 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:43.926 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448747+0000", 2024-09-21T15:50:43.926 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:43.926 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.926 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.926 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:43.926 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448753+0000", 2024-09-21T15:50:43.926 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.755e-06 2024-09-21T15:50:43.926 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.926 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.926 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:43.926 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448763+0000", 2024-09-21T15:50:43.926 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7659999999999993e-06 2024-09-21T15:50:43.926 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.927 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.927 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:43.927 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448769+0000", 2024-09-21T15:50:43.927 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6060000000000002e-06 2024-09-21T15:50:43.927 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.927 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.927 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:43.927 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512750+0000", 2024-09-21T15:50:43.927 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.063981298000000006 2024-09-21T15:50:43.927 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.927 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.927 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:43.927 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512758+0000", 2024-09-21T15:50:43.927 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1669999999999999e-06 2024-09-21T15:50:43.927 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:43.927 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:43.928 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:43.928 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.928 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.928 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155091 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:43.928 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448916+0000", 2024-09-21T15:50:43.928 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 21.473427264000001, 2024-09-21T15:50:43.928 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064006309999999997, 2024-09-21T15:50:43.928 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:43.928 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:43.928 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:43.928 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:43.928 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:43.928 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155091 2024-09-21T15:50:43.928 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.928 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:43.929 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.929 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:43.929 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448916+0000", 2024-09-21T15:50:43.929 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:43.929 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.929 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.929 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:43.929 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448916+0000", 2024-09-21T15:50:43.929 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:43.929 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.929 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.929 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:43.929 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448915+0000", 2024-09-21T15:50:43.929 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:43.929 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.930 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.930 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:43.930 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448917+0000", 2024-09-21T15:50:43.930 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2400000000000002e-06 2024-09-21T15:50:43.930 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.930 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.930 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:43.930 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448923+0000", 2024-09-21T15:50:43.930 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0580000000000003e-06 2024-09-21T15:50:43.930 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.930 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.930 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:43.930 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448927+0000", 2024-09-21T15:50:43.931 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.031e-06 2024-09-21T15:50:43.931 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.931 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.931 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:43.931 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512916+0000", 2024-09-21T15:50:43.931 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.063988456999999999 2024-09-21T15:50:43.931 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.931 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.931 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:43.931 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512922+0000", 2024-09-21T15:50:43.931 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9009999999999999e-06 2024-09-21T15:50:43.931 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:43.931 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:43.931 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:43.931 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.932 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.932 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155092 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:43.932 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448931+0000", 2024-09-21T15:50:43.932 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 21.473411481999999, 2024-09-21T15:50:43.932 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064022793999999994, 2024-09-21T15:50:43.932 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:43.932 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:43.932 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:43.932 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:43.932 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:43.932 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155092 2024-09-21T15:50:43.932 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.932 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:43.932 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.932 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:43.932 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448931+0000", 2024-09-21T15:50:43.933 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:43.933 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.933 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.933 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:43.933 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448931+0000", 2024-09-21T15:50:43.933 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:43.933 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.933 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.933 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:43.933 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448931+0000", 2024-09-21T15:50:43.933 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:43.933 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.933 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.933 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:43.933 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448933+0000", 2024-09-21T15:50:43.934 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1799999999999999e-06 2024-09-21T15:50:43.934 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.934 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.934 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:43.934 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448939+0000", 2024-09-21T15:50:43.934 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.604e-06 2024-09-21T15:50:43.934 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.934 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.934 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:43.934 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448943+0000", 2024-09-21T15:50:43.934 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0740000000000003e-06 2024-09-21T15:50:43.934 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.934 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.934 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:43.935 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512946+0000", 2024-09-21T15:50:43.935 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064003612000000001 2024-09-21T15:50:43.935 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.935 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.935 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:43.935 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512954+0000", 2024-09-21T15:50:43.935 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6690000000000006e-06 2024-09-21T15:50:43.935 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:43.935 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:43.935 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:43.935 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.935 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.935 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155093 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:43.935 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448947+0000", 2024-09-21T15:50:43.935 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 21.473396107999999, 2024-09-21T15:50:43.936 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064020717000000005, 2024-09-21T15:50:43.936 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:43.936 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:43.936 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:43.936 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:43.936 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:43.936 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155093 2024-09-21T15:50:43.936 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.936 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:43.936 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.936 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:43.936 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448947+0000", 2024-09-21T15:50:43.936 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:43.936 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.936 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.936 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:43.937 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448947+0000", 2024-09-21T15:50:43.937 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:43.937 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.937 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.937 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:43.937 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448946+0000", 2024-09-21T15:50:43.937 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:43.937 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.937 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.937 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:43.937 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448949+0000", 2024-09-21T15:50:43.937 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3860000000000001e-06 2024-09-21T15:50:43.937 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.937 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.937 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:43.938 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448955+0000", 2024-09-21T15:50:43.938 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.823e-06 2024-09-21T15:50:43.938 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.938 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.938 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:43.938 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448959+0000", 2024-09-21T15:50:43.938 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2799999999999997e-06 2024-09-21T15:50:43.938 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.938 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.938 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:43.938 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512964+0000", 2024-09-21T15:50:43.938 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064004741000000004 2024-09-21T15:50:43.938 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.938 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.938 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:43.939 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512967+0000", 2024-09-21T15:50:43.939 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.839e-06 2024-09-21T15:50:43.939 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:43.939 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:43.939 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:43.939 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.939 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.939 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155094 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:43.939 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448962+0000", 2024-09-21T15:50:43.939 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 21.473380485, 2024-09-21T15:50:43.939 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064037378000000006, 2024-09-21T15:50:43.939 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:43.939 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:43.939 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:43.939 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:43.939 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:43.940 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155094 2024-09-21T15:50:43.940 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.940 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:43.940 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.940 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:43.940 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448962+0000", 2024-09-21T15:50:43.940 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:43.940 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.940 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.940 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:43.940 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448962+0000", 2024-09-21T15:50:43.940 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:43.940 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.940 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.941 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:43.941 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448962+0000", 2024-09-21T15:50:43.941 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:43.941 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.941 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.941 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:43.941 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448968+0000", 2024-09-21T15:50:43.941 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9710000000000001e-06 2024-09-21T15:50:43.941 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.941 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.941 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:43.941 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448974+0000", 2024-09-21T15:50:43.941 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7230000000000001e-06 2024-09-21T15:50:43.941 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.941 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.941 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:43.941 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448978+0000", 2024-09-21T15:50:43.942 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3749999999999996e-06 2024-09-21T15:50:43.942 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.942 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.942 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:43.942 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512992+0000", 2024-09-21T15:50:43.942 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064013525000000002 2024-09-21T15:50:43.942 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.942 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.942 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:43.942 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513000+0000", 2024-09-21T15:50:43.942 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0830000000000004e-06 2024-09-21T15:50:43.942 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:43.942 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:43.942 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:43.942 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.942 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.943 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155095 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:43.943 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448982+0000", 2024-09-21T15:50:43.943 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 21.473361203, 2024-09-21T15:50:43.943 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064041904999999996, 2024-09-21T15:50:43.943 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:43.943 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:43.943 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:43.943 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:43.943 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:43.943 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155095 2024-09-21T15:50:43.943 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.943 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:43.943 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.943 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:43.943 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448982+0000", 2024-09-21T15:50:43.943 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:43.944 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.944 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.944 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:43.944 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448982+0000", 2024-09-21T15:50:43.944 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:43.944 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.944 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.944 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:43.944 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448981+0000", 2024-09-21T15:50:43.944 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:43.944 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.944 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.944 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:43.944 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448983+0000", 2024-09-21T15:50:43.944 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.905e-06 2024-09-21T15:50:43.944 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.945 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.945 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:43.945 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448989+0000", 2024-09-21T15:50:43.945 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.536e-06 2024-09-21T15:50:43.945 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.945 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.945 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:43.945 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448993+0000", 2024-09-21T15:50:43.945 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2390000000000004e-06 2024-09-21T15:50:43.945 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.945 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.945 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:43.945 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513015+0000", 2024-09-21T15:50:43.945 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064022081999999994 2024-09-21T15:50:43.945 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.945 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.946 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:43.946 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513023+0000", 2024-09-21T15:50:43.946 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4500000000000004e-06 2024-09-21T15:50:43.946 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:43.946 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:43.946 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:43.946 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.946 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.946 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155096 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:43.946 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448996+0000", 2024-09-21T15:50:43.946 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 21.473346301999999, 2024-09-21T15:50:43.946 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064060868000000007, 2024-09-21T15:50:43.946 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:43.946 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:43.946 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:43.947 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:43.947 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:43.947 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155096 2024-09-21T15:50:43.947 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.947 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:43.947 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.947 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:43.947 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448996+0000", 2024-09-21T15:50:43.947 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:43.947 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.947 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.947 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:43.947 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448996+0000", 2024-09-21T15:50:43.947 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:43.947 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.947 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.948 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:43.948 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448996+0000", 2024-09-21T15:50:43.948 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:43.948 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.948 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.948 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:43.948 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448998+0000", 2024-09-21T15:50:43.948 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8780000000000001e-06 2024-09-21T15:50:43.948 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.948 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.948 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:43.948 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449003+0000", 2024-09-21T15:50:43.948 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4099999999999999e-06 2024-09-21T15:50:43.948 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.949 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.949 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:43.949 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449008+0000", 2024-09-21T15:50:43.949 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.335e-06 2024-09-21T15:50:43.949 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.949 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.949 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:43.949 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513049+0000", 2024-09-21T15:50:43.949 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064041059999999997 2024-09-21T15:50:43.949 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.949 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.949 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:43.949 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513057+0000", 2024-09-21T15:50:43.949 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5050000000000007e-06 2024-09-21T15:50:43.949 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:43.949 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:43.949 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:43.950 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.950 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.950 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155097 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:43.950 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449019+0000", 2024-09-21T15:50:43.950 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 21.473323880999999, 2024-09-21T15:50:43.950 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064052488000000005, 2024-09-21T15:50:43.950 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:43.950 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:43.950 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:43.950 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:43.950 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:43.950 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155097 2024-09-21T15:50:43.950 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.950 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:43.951 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.951 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:43.951 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449019+0000", 2024-09-21T15:50:43.951 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:43.951 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.951 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.951 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:43.951 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449019+0000", 2024-09-21T15:50:43.951 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:43.951 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.951 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.951 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:43.951 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449019+0000", 2024-09-21T15:50:43.951 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:43.951 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.951 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.951 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:43.952 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449021+0000", 2024-09-21T15:50:43.952 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.244e-06 2024-09-21T15:50:43.952 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.952 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.952 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:43.952 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449027+0000", 2024-09-21T15:50:43.952 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8619999999999999e-06 2024-09-21T15:50:43.952 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.952 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.952 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:43.952 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449031+0000", 2024-09-21T15:50:43.952 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1359999999999999e-06 2024-09-21T15:50:43.952 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.952 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.953 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:43.953 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513067+0000", 2024-09-21T15:50:43.953 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064036391999999998 2024-09-21T15:50:43.953 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.953 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.953 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:43.953 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513071+0000", 2024-09-21T15:50:43.953 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1989999999999999e-06 2024-09-21T15:50:43.953 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:43.953 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:43.953 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:43.953 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.953 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.953 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155098 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:43.954 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449035+0000", 2024-09-21T15:50:43.954 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 21.473308284000002, 2024-09-21T15:50:43.954 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064071761000000005, 2024-09-21T15:50:43.954 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:43.954 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:43.954 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:43.954 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:43.954 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:43.954 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155098 2024-09-21T15:50:43.954 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.954 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:43.954 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.954 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:43.954 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449035+0000", 2024-09-21T15:50:43.955 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:43.955 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.955 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.955 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:43.955 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449035+0000", 2024-09-21T15:50:43.955 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:43.955 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.955 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.955 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:43.955 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449034+0000", 2024-09-21T15:50:43.955 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:43.955 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.955 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.955 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:43.955 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449036+0000", 2024-09-21T15:50:43.956 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9989999999999998e-06 2024-09-21T15:50:43.956 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.956 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.956 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:43.956 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449042+0000", 2024-09-21T15:50:43.956 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6010000000000002e-06 2024-09-21T15:50:43.956 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.956 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.956 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:43.956 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449047+0000", 2024-09-21T15:50:43.956 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3600000000000004e-06 2024-09-21T15:50:43.956 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.956 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.956 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:43.956 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513098+0000", 2024-09-21T15:50:43.957 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064050544000000001 2024-09-21T15:50:43.957 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.957 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.957 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:43.957 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513106+0000", 2024-09-21T15:50:43.957 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5650000000000002e-06 2024-09-21T15:50:43.957 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:43.957 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:43.957 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:43.957 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.957 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.957 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155099 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:43.957 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449051+0000", 2024-09-21T15:50:43.957 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 21.473291743000001, 2024-09-21T15:50:43.957 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064069388000000005, 2024-09-21T15:50:43.958 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:43.958 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:43.958 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:43.958 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:43.958 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:43.958 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155099 2024-09-21T15:50:43.958 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.958 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:43.958 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.958 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:43.958 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449051+0000", 2024-09-21T15:50:43.958 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:43.958 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.958 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.959 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:43.959 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449051+0000", 2024-09-21T15:50:43.959 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:43.959 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.959 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.959 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:43.959 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449051+0000", 2024-09-21T15:50:43.959 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:43.959 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.959 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.959 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:43.959 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449053+0000", 2024-09-21T15:50:43.959 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2620000000000001e-06 2024-09-21T15:50:43.959 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.959 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.960 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:43.960 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449058+0000", 2024-09-21T15:50:43.960 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1680000000000003e-06 2024-09-21T15:50:43.960 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.960 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.960 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:43.960 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449062+0000", 2024-09-21T15:50:43.960 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9979999999999997e-06 2024-09-21T15:50:43.960 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.960 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.960 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:43.960 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513116+0000", 2024-09-21T15:50:43.960 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064053761000000001 2024-09-21T15:50:43.960 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.960 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.960 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:43.961 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513120+0000", 2024-09-21T15:50:43.961 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5399999999999997e-06 2024-09-21T15:50:43.961 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:43.961 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:43.961 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:43.961 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.961 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.961 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155100 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:43.961 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449101+0000", 2024-09-21T15:50:43.961 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 21.473241988000002, 2024-09-21T15:50:43.961 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064079585999999994, 2024-09-21T15:50:43.961 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:43.961 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:43.961 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:43.961 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:43.961 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:43.962 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155100 2024-09-21T15:50:43.962 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.962 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:43.962 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.962 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:43.962 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449101+0000", 2024-09-21T15:50:43.962 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:43.962 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.962 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.962 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:43.962 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449101+0000", 2024-09-21T15:50:43.962 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:43.962 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.962 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.963 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:43.963 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449100+0000", 2024-09-21T15:50:43.963 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-21T15:50:43.963 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.963 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.963 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:43.963 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449105+0000", 2024-09-21T15:50:43.963 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6700000000000002e-06 2024-09-21T15:50:43.963 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.963 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.963 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:43.963 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449117+0000", 2024-09-21T15:50:43.963 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1841000000000001e-05 2024-09-21T15:50:43.963 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.963 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.964 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:43.964 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449124+0000", 2024-09-21T15:50:43.964 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4660000000000002e-06 2024-09-21T15:50:43.964 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.964 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.964 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:43.964 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513163+0000", 2024-09-21T15:50:43.964 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064039053999999998 2024-09-21T15:50:43.964 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.964 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.964 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:43.964 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513180+0000", 2024-09-21T15:50:43.964 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7333000000000002e-05 2024-09-21T15:50:43.964 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:43.964 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:43.965 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:43.965 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.965 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.965 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155101 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:43.965 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449166+0000", 2024-09-21T15:50:43.965 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 21.473177282999998, 2024-09-21T15:50:43.965 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064027639999999997, 2024-09-21T15:50:43.965 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:43.965 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:43.965 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:43.965 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:43.965 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:43.965 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155101 2024-09-21T15:50:43.965 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.966 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:43.966 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.966 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:43.966 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449166+0000", 2024-09-21T15:50:43.966 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:43.966 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.966 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.966 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:43.966 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449166+0000", 2024-09-21T15:50:43.966 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:43.966 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.966 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.966 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:43.967 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449165+0000", 2024-09-21T15:50:43.967 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:43.967 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.967 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.967 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:43.967 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449168+0000", 2024-09-21T15:50:43.967 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5099999999999999e-06 2024-09-21T15:50:43.967 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.967 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.967 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:43.967 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449177+0000", 2024-09-21T15:50:43.967 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6659999999999997e-06 2024-09-21T15:50:43.967 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.967 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.967 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:43.968 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449182+0000", 2024-09-21T15:50:43.968 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1320000000000002e-06 2024-09-21T15:50:43.968 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.968 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.968 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:43.968 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513189+0000", 2024-09-21T15:50:43.968 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064007231999999997 2024-09-21T15:50:43.968 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.968 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.968 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:43.968 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513193+0000", 2024-09-21T15:50:43.968 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.692e-06 2024-09-21T15:50:43.968 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:43.968 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:43.968 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:43.969 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.969 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.969 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155134 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:43.969 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449187+0000", 2024-09-21T15:50:43.969 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 21.473155694999999, 2024-09-21T15:50:43.969 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064031331999999996, 2024-09-21T15:50:43.969 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:43.969 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:43.969 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:43.969 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:43.969 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:43.969 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155134 2024-09-21T15:50:43.969 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.969 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:43.970 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.970 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:43.970 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449187+0000", 2024-09-21T15:50:43.970 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:43.970 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.970 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.970 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:43.970 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449187+0000", 2024-09-21T15:50:43.970 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:43.970 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.970 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.970 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:43.970 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449187+0000", 2024-09-21T15:50:43.971 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:43.971 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.971 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.971 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:43.971 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449189+0000", 2024-09-21T15:50:43.971 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5469999999999999e-06 2024-09-21T15:50:43.971 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.971 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.971 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:43.971 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449196+0000", 2024-09-21T15:50:43.971 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7009999999999999e-06 2024-09-21T15:50:43.971 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.971 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.971 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:43.972 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449202+0000", 2024-09-21T15:50:43.972 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7409999999999998e-06 2024-09-21T15:50:43.972 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.972 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.972 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:43.972 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513214+0000", 2024-09-21T15:50:43.972 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064012301999999993 2024-09-21T15:50:43.972 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.972 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.972 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:43.972 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513218+0000", 2024-09-21T15:50:43.972 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5709999999999999e-06 2024-09-21T15:50:43.972 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:43.972 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:43.972 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:43.972 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.973 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.973 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155135 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:43.973 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449207+0000", 2024-09-21T15:50:43.973 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 21.473136185000001, 2024-09-21T15:50:43.973 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064029249999999996, 2024-09-21T15:50:43.973 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:43.973 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:43.973 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:43.973 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:43.973 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:43.973 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155135 2024-09-21T15:50:43.973 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.973 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:43.973 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.973 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:43.973 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449207+0000", 2024-09-21T15:50:43.974 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:43.974 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.974 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.974 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:43.974 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449207+0000", 2024-09-21T15:50:43.974 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:43.974 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.974 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.974 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:43.974 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449206+0000", 2024-09-21T15:50:43.974 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:43.974 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.974 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.974 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:43.975 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449209+0000", 2024-09-21T15:50:43.975 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5150000000000001e-06 2024-09-21T15:50:43.975 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.975 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.975 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:43.975 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449215+0000", 2024-09-21T15:50:43.975 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9329999999999998e-06 2024-09-21T15:50:43.975 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.975 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.975 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:43.975 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449220+0000", 2024-09-21T15:50:43.975 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0819999999999998e-06 2024-09-21T15:50:43.975 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.975 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.976 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:43.976 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513232+0000", 2024-09-21T15:50:43.976 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064012052 2024-09-21T15:50:43.976 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.976 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.976 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:43.976 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513236+0000", 2024-09-21T15:50:43.976 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2080000000000002e-06 2024-09-21T15:50:43.976 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:43.976 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:43.976 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:43.976 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.976 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.976 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155136 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:43.976 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449224+0000", 2024-09-21T15:50:43.977 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 21.473118621000001, 2024-09-21T15:50:43.977 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064053025999999999, 2024-09-21T15:50:43.977 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:43.977 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:43.977 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:43.977 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:43.977 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:43.977 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155136 2024-09-21T15:50:43.977 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.977 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:43.977 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.977 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:43.977 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449224+0000", 2024-09-21T15:50:43.977 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:43.977 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.977 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.978 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:43.978 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449224+0000", 2024-09-21T15:50:43.978 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:43.978 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.978 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.978 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:43.978 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449224+0000", 2024-09-21T15:50:43.978 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:43.978 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.978 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.978 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:43.978 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449226+0000", 2024-09-21T15:50:43.978 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2699999999999999e-06 2024-09-21T15:50:43.978 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.978 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.978 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:43.979 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449233+0000", 2024-09-21T15:50:43.979 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4289999999999998e-06 2024-09-21T15:50:43.979 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.979 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.979 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:43.979 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449238+0000", 2024-09-21T15:50:43.979 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0730000000000004e-06 2024-09-21T15:50:43.979 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.979 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.979 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:43.979 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513269+0000", 2024-09-21T15:50:43.979 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064030999000000005 2024-09-21T15:50:43.979 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.979 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.979 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:43.980 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513277+0000", 2024-09-21T15:50:43.980 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5520000000000005e-06 2024-09-21T15:50:43.980 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:43.980 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:43.980 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:43.980 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.980 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.980 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155137 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:43.980 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449263+0000", 2024-09-21T15:50:43.980 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 21.473079489, 2024-09-21T15:50:43.980 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.06402803, 2024-09-21T15:50:43.980 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:43.980 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:43.981 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:43.981 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:43.981 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:43.981 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155137 2024-09-21T15:50:43.981 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.981 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:43.981 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.981 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:43.981 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449263+0000", 2024-09-21T15:50:43.981 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:43.981 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.981 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.981 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:43.981 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449263+0000", 2024-09-21T15:50:43.982 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:43.982 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.982 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.982 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:43.982 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449262+0000", 2024-09-21T15:50:43.982 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-21T15:50:43.982 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.982 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.982 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:43.982 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449267+0000", 2024-09-21T15:50:43.982 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.933e-06 2024-09-21T15:50:43.982 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.982 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.983 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:43.983 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449279+0000", 2024-09-21T15:50:43.983 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.181e-05 2024-09-21T15:50:43.983 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.983 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.983 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:43.983 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449286+0000", 2024-09-21T15:50:43.983 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4749999999999998e-06 2024-09-21T15:50:43.983 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.983 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.983 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:43.983 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513287+0000", 2024-09-21T15:50:43.983 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064001369000000002 2024-09-21T15:50:43.983 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.983 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.983 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:43.984 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513291+0000", 2024-09-21T15:50:43.984 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3000000000000003e-06 2024-09-21T15:50:43.984 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:43.984 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:43.984 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:43.984 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.984 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.984 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155138 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:43.984 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449299+0000", 2024-09-21T15:50:43.984 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 21.473043997000001, 2024-09-21T15:50:43.984 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064028359000000007, 2024-09-21T15:50:43.984 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:43.984 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:43.984 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:43.985 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:43.985 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:43.985 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155138 2024-09-21T15:50:43.985 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.985 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:43.985 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.985 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:43.985 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449299+0000", 2024-09-21T15:50:43.985 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:43.985 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.985 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.985 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:43.985 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449299+0000", 2024-09-21T15:50:43.985 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:43.986 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.986 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.986 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:43.986 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449298+0000", 2024-09-21T15:50:43.986 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:43.986 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.986 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.986 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:43.986 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449301+0000", 2024-09-21T15:50:43.986 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0230000000000001e-06 2024-09-21T15:50:43.986 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.986 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.986 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:43.986 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449308+0000", 2024-09-21T15:50:43.986 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3119999999999999e-06 2024-09-21T15:50:43.986 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.987 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.987 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:43.987 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449313+0000", 2024-09-21T15:50:43.987 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8980000000000004e-06 2024-09-21T15:50:43.987 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.987 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.987 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:43.987 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513318+0000", 2024-09-21T15:50:43.987 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064005612000000003 2024-09-21T15:50:43.987 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.987 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.987 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:43.987 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513327+0000", 2024-09-21T15:50:43.987 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0100000000000001e-06 2024-09-21T15:50:43.987 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:43.988 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:43.988 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:43.988 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.988 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.988 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155142 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:43.988 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449392+0000", 2024-09-21T15:50:43.988 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 21.472951239, 2024-09-21T15:50:43.988 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064012464000000005, 2024-09-21T15:50:43.988 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:43.988 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:43.988 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:43.988 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:43.988 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:43.988 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155142 2024-09-21T15:50:43.988 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.988 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:43.989 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.989 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:43.989 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449392+0000", 2024-09-21T15:50:43.989 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:43.989 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.989 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.989 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:43.989 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449392+0000", 2024-09-21T15:50:43.989 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:43.989 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.989 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.989 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:43.989 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449391+0000", 2024-09-21T15:50:43.989 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:43.989 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.989 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.990 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:43.990 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449393+0000", 2024-09-21T15:50:43.990 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.26e-06 2024-09-21T15:50:43.990 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.990 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.990 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:43.990 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449399+0000", 2024-09-21T15:50:43.990 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6269999999999997e-06 2024-09-21T15:50:43.990 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.990 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.990 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:43.990 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449403+0000", 2024-09-21T15:50:43.990 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3000000000000003e-06 2024-09-21T15:50:43.990 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.990 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.991 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:43.991 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513396+0000", 2024-09-21T15:50:43.991 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.063992568999999999 2024-09-21T15:50:43.991 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.991 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.991 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:43.991 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513404+0000", 2024-09-21T15:50:43.991 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1100000000000003e-06 2024-09-21T15:50:43.991 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:43.991 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:43.991 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:43.991 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.991 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.991 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155143 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:43.991 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449407+0000", 2024-09-21T15:50:43.992 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 21.472935653, 2024-09-21T15:50:43.992 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064009916, 2024-09-21T15:50:43.992 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:43.992 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:43.992 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:43.992 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:43.992 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:43.992 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155143 2024-09-21T15:50:43.992 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.992 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:43.992 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.992 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:43.992 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449407+0000", 2024-09-21T15:50:43.992 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:43.993 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.993 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.993 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:43.993 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449407+0000", 2024-09-21T15:50:43.993 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:43.993 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.993 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.993 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:43.993 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449407+0000", 2024-09-21T15:50:43.993 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:43.993 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.993 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.993 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:43.993 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449409+0000", 2024-09-21T15:50:43.994 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1090000000000001e-06 2024-09-21T15:50:43.994 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.994 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.994 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:43.994 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449413+0000", 2024-09-21T15:50:43.994 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6650000000000001e-06 2024-09-21T15:50:43.994 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.994 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.994 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:43.994 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449415+0000", 2024-09-21T15:50:43.994 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5459999999999998e-06 2024-09-21T15:50:43.994 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.994 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.994 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:43.994 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513413+0000", 2024-09-21T15:50:43.995 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.063998037999999993 2024-09-21T15:50:43.995 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.995 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.995 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:43.995 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513417+0000", 2024-09-21T15:50:43.995 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8789999999999996e-06 2024-09-21T15:50:43.995 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:43.995 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:43.995 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:43.995 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.995 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.995 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155144 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:43.995 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449427+0000", 2024-09-21T15:50:43.996 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 21.472915645, 2024-09-21T15:50:43.996 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064022531999999993, 2024-09-21T15:50:43.996 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:43.996 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:43.996 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:43.996 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:43.996 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:43.996 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155144 2024-09-21T15:50:43.996 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.996 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:43.996 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.996 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:43.996 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449427+0000", 2024-09-21T15:50:43.996 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:43.996 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.997 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.997 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:43.997 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449427+0000", 2024-09-21T15:50:43.997 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:43.997 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.997 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.997 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:43.997 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449427+0000", 2024-09-21T15:50:43.997 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:43.997 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.997 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.997 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:43.997 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449430+0000", 2024-09-21T15:50:43.997 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3579999999999999e-06 2024-09-21T15:50:43.997 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.997 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.998 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:43.998 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449437+0000", 2024-09-21T15:50:43.998 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4580000000000004e-06 2024-09-21T15:50:43.998 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.998 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.998 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:43.998 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449443+0000", 2024-09-21T15:50:43.998 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5609999999999998e-06 2024-09-21T15:50:43.998 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.998 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.998 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:43.998 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513441+0000", 2024-09-21T15:50:43.998 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.063998136999999997 2024-09-21T15:50:43.998 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:43.998 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:43.999 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:43.999 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513450+0000", 2024-09-21T15:50:43.999 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4950000000000008e-06 2024-09-21T15:50:43.999 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:43.999 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:43.999 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:43.999 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:43.999 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:43.999 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:44.000 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:50:44.000 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:50:44.001 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:50:44.001 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:50:44.001 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:50:44.002 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:50:44.002 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:50:44.003 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:50:43.999+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:50:44.101 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:44.100+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:44.144 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:44.144 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:44.144 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:50:44.144 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:44.152 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:50:44.202 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:44.200+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:44.254 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:44.254 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:44.254 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:50:44.254 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:50:44.254 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:44.263 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:50:44.304 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:44.303+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:44.403 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:44.402+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:44.407 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:44.407 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:50:44.407 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:50:44.407 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:50:44.408 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.408 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120111 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 321536~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:44.408 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572216+0000", 2024-09-21T15:50:44.408 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 40.833459167000001, 2024-09-21T15:50:44.408 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054657399000000002, 2024-09-21T15:50:44.408 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:44.408 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:44.408 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:44.408 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:44.408 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:44.408 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120111 2024-09-21T15:50:44.409 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.409 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:44.409 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.409 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:44.409 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572216+0000", 2024-09-21T15:50:44.409 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.409 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.409 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.409 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:44.409 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572216+0000", 2024-09-21T15:50:44.409 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.409 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.409 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.409 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:44.410 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572216+0000", 2024-09-21T15:50:44.410 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:44.410 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.410 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.410 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:44.410 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572218+0000", 2024-09-21T15:50:44.410 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1840000000000002e-06 2024-09-21T15:50:44.410 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.410 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.410 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:44.410 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572224+0000", 2024-09-21T15:50:44.410 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7880000000000003e-06 2024-09-21T15:50:44.410 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.410 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.411 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:44.411 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572229+0000", 2024-09-21T15:50:44.411 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8239999999999999e-06 2024-09-21T15:50:44.411 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.411 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.411 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:44.411 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626680+0000", 2024-09-21T15:50:44.411 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054451431000000002 2024-09-21T15:50:44.411 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.411 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.411 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:44.411 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626755+0000", 2024-09-21T15:50:44.411 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4641000000000006e-05 2024-09-21T15:50:44.411 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.412 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.412 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:44.412 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626874+0000", 2024-09-21T15:50:44.412 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00011886 2024-09-21T15:50:44.412 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.412 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:44.412 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.412 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.412 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.412 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120112 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 342016~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:44.412 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572233+0000", 2024-09-21T15:50:44.412 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 40.833442798999997, 2024-09-21T15:50:44.413 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054823997999999999, 2024-09-21T15:50:44.413 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:44.413 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:44.413 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:44.413 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:44.413 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:44.413 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120112 2024-09-21T15:50:44.413 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.413 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:44.413 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.413 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:44.413 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572233+0000", 2024-09-21T15:50:44.413 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.414 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.414 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.414 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:44.414 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572233+0000", 2024-09-21T15:50:44.414 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.414 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.414 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.414 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:44.414 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572232+0000", 2024-09-21T15:50:44.414 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:44.414 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.414 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.414 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:44.415 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572234+0000", 2024-09-21T15:50:44.415 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0660000000000002e-06 2024-09-21T15:50:44.415 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.415 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.415 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:44.415 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572242+0000", 2024-09-21T15:50:44.415 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0430000000000002e-06 2024-09-21T15:50:44.415 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.415 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.415 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:44.415 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572246+0000", 2024-09-21T15:50:44.415 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9080000000000003e-06 2024-09-21T15:50:44.415 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.415 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.415 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:44.416 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626916+0000", 2024-09-21T15:50:44.416 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054669482999999998 2024-09-21T15:50:44.416 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.416 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.416 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:44.416 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626977+0000", 2024-09-21T15:50:44.416 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0597000000000003e-05 2024-09-21T15:50:44.416 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.416 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.416 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:44.416 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627057+0000", 2024-09-21T15:50:44.416 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0258000000000003e-05 2024-09-21T15:50:44.416 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.416 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:44.416 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.417 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.417 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.417 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120113 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 362496~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:44.417 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572253+0000", 2024-09-21T15:50:44.417 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 40.833422814000002, 2024-09-21T15:50:44.417 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.05503044, 2024-09-21T15:50:44.417 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:44.417 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:44.417 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:44.417 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:44.417 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:44.417 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120113 2024-09-21T15:50:44.417 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.417 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:44.418 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:44.418 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572253+0000", 2024-09-21T15:50:44.418 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.418 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:44.418 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572253+0000", 2024-09-21T15:50:44.418 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.418 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:44.418 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572252+0000", 2024-09-21T15:50:44.418 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:44.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.419 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.419 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:44.419 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572258+0000", 2024-09-21T15:50:44.419 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6169999999999998e-06 2024-09-21T15:50:44.419 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.419 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.419 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:44.419 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572266+0000", 2024-09-21T15:50:44.419 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9410000000000007e-06 2024-09-21T15:50:44.419 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.419 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.419 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:44.419 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572272+0000", 2024-09-21T15:50:44.419 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.241e-06 2024-09-21T15:50:44.420 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.420 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.420 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:44.420 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627087+0000", 2024-09-21T15:50:44.420 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054814850999999998 2024-09-21T15:50:44.420 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.420 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.420 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:44.420 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627162+0000", 2024-09-21T15:50:44.420 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5012999999999995e-05 2024-09-21T15:50:44.420 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.420 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.420 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:44.420 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627283+0000", 2024-09-21T15:50:44.420 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012113 2024-09-21T15:50:44.421 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.421 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:44.421 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.421 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.421 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.421 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120114 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 382976~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:44.421 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572277+0000", 2024-09-21T15:50:44.421 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 40.833398795999997, 2024-09-21T15:50:44.421 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055184395999999997, 2024-09-21T15:50:44.421 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:44.421 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:44.421 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:44.421 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:44.421 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:44.422 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120114 2024-09-21T15:50:44.422 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.422 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:44.422 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.422 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:44.422 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572277+0000", 2024-09-21T15:50:44.422 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.422 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.422 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.422 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:44.422 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572277+0000", 2024-09-21T15:50:44.422 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.422 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.422 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.423 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:44.423 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572276+0000", 2024-09-21T15:50:44.423 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:44.423 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.423 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.423 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:44.423 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572279+0000", 2024-09-21T15:50:44.423 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.373e-06 2024-09-21T15:50:44.423 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.423 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.423 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:44.423 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572285+0000", 2024-09-21T15:50:44.423 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9359999999999996e-06 2024-09-21T15:50:44.424 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.424 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.424 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:44.424 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572289+0000", 2024-09-21T15:50:44.424 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.634e-06 2024-09-21T15:50:44.424 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.424 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.424 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:44.424 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627311+0000", 2024-09-21T15:50:44.424 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055021211 2024-09-21T15:50:44.424 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.424 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.424 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:44.424 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627360+0000", 2024-09-21T15:50:44.425 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9203999999999998e-05 2024-09-21T15:50:44.425 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.425 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.425 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:44.425 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627461+0000", 2024-09-21T15:50:44.425 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000101469 2024-09-21T15:50:44.425 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.425 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:44.425 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.425 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.425 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.425 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120115 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 399360~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:44.425 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572294+0000", 2024-09-21T15:50:44.425 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 40.833381488999997, 2024-09-21T15:50:44.426 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055400092999999997, 2024-09-21T15:50:44.426 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:44.426 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:44.426 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:44.426 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:44.426 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:44.426 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120115 2024-09-21T15:50:44.426 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.426 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:44.426 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.426 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:44.426 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572294+0000", 2024-09-21T15:50:44.426 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.427 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.427 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.427 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:44.427 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572294+0000", 2024-09-21T15:50:44.427 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.427 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.427 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.427 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:44.427 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572294+0000", 2024-09-21T15:50:44.427 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:44.427 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.427 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.427 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:44.427 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572296+0000", 2024-09-21T15:50:44.428 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1100000000000001e-06 2024-09-21T15:50:44.428 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.428 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.428 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:44.428 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572303+0000", 2024-09-21T15:50:44.428 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8580000000000002e-06 2024-09-21T15:50:44.428 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.428 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.428 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:44.428 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572314+0000", 2024-09-21T15:50:44.428 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.101e-05 2024-09-21T15:50:44.428 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.428 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.428 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:44.428 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627496+0000", 2024-09-21T15:50:44.429 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055182522999999997 2024-09-21T15:50:44.429 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.429 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.429 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:44.429 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627568+0000", 2024-09-21T15:50:44.429 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1569999999999994e-05 2024-09-21T15:50:44.429 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.429 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.429 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:44.429 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627694+0000", 2024-09-21T15:50:44.429 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000126372 2024-09-21T15:50:44.429 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.429 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:44.429 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.430 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.430 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.430 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120116 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 419840~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:44.430 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572318+0000", 2024-09-21T15:50:44.430 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 40.833357298000003, 2024-09-21T15:50:44.430 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055539416000000001, 2024-09-21T15:50:44.430 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:44.430 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:44.430 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:44.430 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:44.430 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:44.430 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120116 2024-09-21T15:50:44.430 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.430 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:44.431 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.431 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:44.431 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572318+0000", 2024-09-21T15:50:44.431 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.431 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.431 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.431 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:44.431 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572318+0000", 2024-09-21T15:50:44.431 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.431 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.431 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.431 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:44.431 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572318+0000", 2024-09-21T15:50:44.431 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:44.432 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.432 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.432 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:44.432 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572322+0000", 2024-09-21T15:50:44.432 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4839999999999998e-06 2024-09-21T15:50:44.432 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.432 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.432 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:44.432 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572329+0000", 2024-09-21T15:50:44.432 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0959999999999997e-06 2024-09-21T15:50:44.432 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.432 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.432 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:44.433 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572333+0000", 2024-09-21T15:50:44.433 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9359999999999998e-06 2024-09-21T15:50:44.433 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.433 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.433 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:44.433 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627726+0000", 2024-09-21T15:50:44.433 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055392829999999997 2024-09-21T15:50:44.433 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.433 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.433 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:44.433 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627779+0000", 2024-09-21T15:50:44.433 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2185999999999998e-05 2024-09-21T15:50:44.433 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.433 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.434 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:44.434 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627858+0000", 2024-09-21T15:50:44.434 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9190999999999994e-05 2024-09-21T15:50:44.434 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.434 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:44.434 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.434 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.434 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.434 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120117 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 440320~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:44.434 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572338+0000", 2024-09-21T15:50:44.434 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 40.833337985, 2024-09-21T15:50:44.434 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055680888999999997, 2024-09-21T15:50:44.434 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:44.434 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:44.435 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:44.435 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:44.435 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:44.435 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120117 2024-09-21T15:50:44.435 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.435 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:44.435 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.435 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:44.435 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572338+0000", 2024-09-21T15:50:44.435 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.435 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.435 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.435 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:44.435 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572338+0000", 2024-09-21T15:50:44.436 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.436 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.436 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.436 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:44.436 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572337+0000", 2024-09-21T15:50:44.436 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:44.436 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.436 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.436 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:44.436 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572339+0000", 2024-09-21T15:50:44.436 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9379999999999999e-06 2024-09-21T15:50:44.436 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.436 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.436 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:44.437 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572349+0000", 2024-09-21T15:50:44.437 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7780000000000002e-06 2024-09-21T15:50:44.437 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.437 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.437 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:44.437 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572354+0000", 2024-09-21T15:50:44.437 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8740000000000003e-06 2024-09-21T15:50:44.437 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.437 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.437 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:44.437 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627885+0000", 2024-09-21T15:50:44.437 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055531448999999997 2024-09-21T15:50:44.437 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.437 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.438 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:44.438 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627935+0000", 2024-09-21T15:50:44.438 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9348999999999998e-05 2024-09-21T15:50:44.438 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.438 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.438 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:44.438 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628018+0000", 2024-09-21T15:50:44.438 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3851999999999996e-05 2024-09-21T15:50:44.438 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.438 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:44.438 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.438 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.438 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.438 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120118 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 460800~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:44.439 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572359+0000", 2024-09-21T15:50:44.439 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 40.833316742999997, 2024-09-21T15:50:44.439 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055951714, 2024-09-21T15:50:44.439 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:44.439 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:44.439 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:44.439 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:44.439 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:44.439 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120118 2024-09-21T15:50:44.439 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.439 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:44.439 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.439 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:44.439 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572359+0000", 2024-09-21T15:50:44.440 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.440 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.440 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.440 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:44.440 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572359+0000", 2024-09-21T15:50:44.440 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.440 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.440 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.440 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:44.440 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572358+0000", 2024-09-21T15:50:44.440 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:44.440 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.440 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.440 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:44.441 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572361+0000", 2024-09-21T15:50:44.441 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.26e-06 2024-09-21T15:50:44.441 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.441 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.441 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:44.441 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572367+0000", 2024-09-21T15:50:44.441 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3640000000000004e-06 2024-09-21T15:50:44.441 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.441 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.441 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:44.441 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572376+0000", 2024-09-21T15:50:44.442 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2310000000000002e-06 2024-09-21T15:50:44.442 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.442 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.442 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:44.442 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628096+0000", 2024-09-21T15:50:44.442 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055719369999999997 2024-09-21T15:50:44.442 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.442 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.442 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:44.442 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628178+0000", 2024-09-21T15:50:44.442 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2667e-05 2024-09-21T15:50:44.442 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.443 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.443 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:44.443 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628311+0000", 2024-09-21T15:50:44.443 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000132175 2024-09-21T15:50:44.443 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.443 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:44.443 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.443 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.443 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.443 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120119 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 481280~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:44.443 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572381+0000", 2024-09-21T15:50:44.443 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 40.833294299000002, 2024-09-21T15:50:44.443 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056088791999999998, 2024-09-21T15:50:44.444 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:44.444 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:44.444 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:44.444 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:44.444 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:44.444 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120119 2024-09-21T15:50:44.444 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.444 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:44.444 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.444 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:44.444 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572381+0000", 2024-09-21T15:50:44.444 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.444 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.444 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.444 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:44.445 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572381+0000", 2024-09-21T15:50:44.445 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.445 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.445 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.445 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:44.445 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572381+0000", 2024-09-21T15:50:44.445 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:44.445 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.445 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.445 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:44.445 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572383+0000", 2024-09-21T15:50:44.445 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0619999999999999e-06 2024-09-21T15:50:44.445 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.445 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.445 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:44.446 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572391+0000", 2024-09-21T15:50:44.446 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8769999999999993e-06 2024-09-21T15:50:44.446 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.446 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.446 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:44.446 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572399+0000", 2024-09-21T15:50:44.446 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6240000000000008e-06 2024-09-21T15:50:44.446 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.446 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.446 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:44.446 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628344+0000", 2024-09-21T15:50:44.446 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055944828000000002 2024-09-21T15:50:44.446 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.446 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.446 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:44.447 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628393+0000", 2024-09-21T15:50:44.447 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8653000000000003e-05 2024-09-21T15:50:44.447 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.447 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.447 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:44.447 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628470+0000", 2024-09-21T15:50:44.447 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7121999999999998e-05 2024-09-21T15:50:44.447 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.447 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:44.447 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.447 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.447 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.447 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120120 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 501760~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:44.447 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572418+0000", 2024-09-21T15:50:44.447 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 40.833257801000002, 2024-09-21T15:50:44.448 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056209138999999998, 2024-09-21T15:50:44.448 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:44.448 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:44.448 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:44.448 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:44.448 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:44.448 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120120 2024-09-21T15:50:44.448 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.448 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:44.448 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.448 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:44.448 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572418+0000", 2024-09-21T15:50:44.448 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.448 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.448 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.448 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:44.449 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572418+0000", 2024-09-21T15:50:44.449 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.449 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.449 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.449 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:44.449 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572417+0000", 2024-09-21T15:50:44.449 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:44.449 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.449 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.449 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:44.449 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572420+0000", 2024-09-21T15:50:44.449 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5270000000000001e-06 2024-09-21T15:50:44.449 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.449 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.450 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:44.450 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572426+0000", 2024-09-21T15:50:44.450 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3380000000000001e-06 2024-09-21T15:50:44.450 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.450 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.450 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:44.450 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572431+0000", 2024-09-21T15:50:44.450 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.865e-06 2024-09-21T15:50:44.450 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.450 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.450 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:44.450 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628500+0000", 2024-09-21T15:50:44.450 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056068917000000003 2024-09-21T15:50:44.450 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.451 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.451 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:44.451 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628542+0000", 2024-09-21T15:50:44.451 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1761999999999998e-05 2024-09-21T15:50:44.451 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.451 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.451 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:44.451 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628627+0000", 2024-09-21T15:50:44.451 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5247999999999994e-05 2024-09-21T15:50:44.451 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.451 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:44.451 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.451 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.451 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.451 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120121 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 522240~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:44.452 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572630+0000", 2024-09-21T15:50:44.452 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 40.833045415000001, 2024-09-21T15:50:44.452 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056207209000000001, 2024-09-21T15:50:44.452 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:44.452 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:44.452 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:44.452 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:44.452 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:44.452 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120121 2024-09-21T15:50:44.452 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.452 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:44.452 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.452 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:44.452 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572630+0000", 2024-09-21T15:50:44.452 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.453 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.453 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.453 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:44.453 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572630+0000", 2024-09-21T15:50:44.453 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.453 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.453 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.453 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:44.453 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572435+0000", 2024-09-21T15:50:44.453 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998045 2024-09-21T15:50:44.453 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.453 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.453 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:44.453 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572633+0000", 2024-09-21T15:50:44.453 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000198786 2024-09-21T15:50:44.454 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.454 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.454 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:44.454 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572641+0000", 2024-09-21T15:50:44.454 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9449999999999993e-06 2024-09-21T15:50:44.454 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.454 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.454 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:44.454 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572648+0000", 2024-09-21T15:50:44.454 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1929999999999998e-06 2024-09-21T15:50:44.454 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.454 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.454 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:44.455 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628669+0000", 2024-09-21T15:50:44.455 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056021690999999998 2024-09-21T15:50:44.455 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.455 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.455 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:44.455 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628751+0000", 2024-09-21T15:50:44.455 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1710000000000002e-05 2024-09-21T15:50:44.455 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.455 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.455 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:44.455 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628837+0000", 2024-09-21T15:50:44.455 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6383000000000003e-05 2024-09-21T15:50:44.455 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.455 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:44.455 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.456 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.456 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.456 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120122 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 538624~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:44.456 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.573471+0000", 2024-09-21T15:50:44.456 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 40.832204169999997, 2024-09-21T15:50:44.456 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055563935000000002, 2024-09-21T15:50:44.456 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:44.456 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:44.456 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:44.456 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:44.456 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:44.456 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120122 2024-09-21T15:50:44.456 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.456 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:44.457 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.457 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:44.457 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573471+0000", 2024-09-21T15:50:44.457 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.457 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.457 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.457 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:44.457 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573471+0000", 2024-09-21T15:50:44.457 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.457 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.457 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.457 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:44.457 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572653+0000", 2024-09-21T15:50:44.457 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9991813 2024-09-21T15:50:44.458 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.458 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.458 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:44.458 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573475+0000", 2024-09-21T15:50:44.458 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00082196500000000002 2024-09-21T15:50:44.458 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.458 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.458 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:44.458 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573483+0000", 2024-09-21T15:50:44.458 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5939999999999994e-06 2024-09-21T15:50:44.458 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.458 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.458 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:44.459 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573489+0000", 2024-09-21T15:50:44.459 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1689999999999997e-06 2024-09-21T15:50:44.459 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.459 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.459 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:44.459 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628882+0000", 2024-09-21T15:50:44.459 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055392230000000001 2024-09-21T15:50:44.459 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.459 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.459 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:44.459 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628934+0000", 2024-09-21T15:50:44.459 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1870000000000003e-05 2024-09-21T15:50:44.459 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.459 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.459 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:44.460 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629035+0000", 2024-09-21T15:50:44.460 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000101753 2024-09-21T15:50:44.460 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.460 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:44.460 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.460 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.460 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.460 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120123 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 559104~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:44.460 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.573495+0000", 2024-09-21T15:50:44.460 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 40.832180471000001, 2024-09-21T15:50:44.460 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055704765000000003, 2024-09-21T15:50:44.460 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:44.460 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:44.461 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:44.461 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:44.461 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:44.461 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120123 2024-09-21T15:50:44.461 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.461 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:44.461 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.461 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:44.461 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573495+0000", 2024-09-21T15:50:44.461 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.461 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.461 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.461 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:44.461 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573495+0000", 2024-09-21T15:50:44.462 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.462 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.462 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.462 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:44.462 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573495+0000", 2024-09-21T15:50:44.462 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:44.462 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.462 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.462 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:44.462 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573497+0000", 2024-09-21T15:50:44.462 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3420000000000002e-06 2024-09-21T15:50:44.462 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.462 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.462 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:44.463 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573503+0000", 2024-09-21T15:50:44.463 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7030000000000003e-06 2024-09-21T15:50:44.463 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.463 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.463 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:44.463 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573511+0000", 2024-09-21T15:50:44.463 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0790000000000001e-06 2024-09-21T15:50:44.463 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.463 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.463 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:44.463 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629067+0000", 2024-09-21T15:50:44.463 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055556573999999997 2024-09-21T15:50:44.463 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.463 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.464 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:44.464 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629118+0000", 2024-09-21T15:50:44.464 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1053000000000001e-05 2024-09-21T15:50:44.464 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.464 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.464 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:44.464 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629200+0000", 2024-09-21T15:50:44.464 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1452999999999994e-05 2024-09-21T15:50:44.464 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.464 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:44.464 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.464 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.464 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.464 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120124 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 579584~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:44.465 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.574014+0000", 2024-09-21T15:50:44.465 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 40.831661431999997, 2024-09-21T15:50:44.465 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055440284999999999, 2024-09-21T15:50:44.465 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:44.465 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:44.465 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:44.465 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:44.465 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:44.465 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120124 2024-09-21T15:50:44.465 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.465 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:44.465 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.465 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:44.465 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574014+0000", 2024-09-21T15:50:44.466 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.466 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.466 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.466 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:44.466 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574014+0000", 2024-09-21T15:50:44.466 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.466 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.466 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.466 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:44.466 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573515+0000", 2024-09-21T15:50:44.466 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9995008 2024-09-21T15:50:44.466 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.466 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.466 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:44.467 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574017+0000", 2024-09-21T15:50:44.467 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00050241100000000003 2024-09-21T15:50:44.467 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.467 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.467 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:44.467 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574027+0000", 2024-09-21T15:50:44.467 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4700000000000008e-06 2024-09-21T15:50:44.467 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.467 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.467 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:44.467 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574033+0000", 2024-09-21T15:50:44.467 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1489999999999999e-06 2024-09-21T15:50:44.467 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.467 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.468 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:44.468 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629238+0000", 2024-09-21T15:50:44.468 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055204880999999997 2024-09-21T15:50:44.468 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.468 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.468 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:44.468 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629317+0000", 2024-09-21T15:50:44.468 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8942000000000004e-05 2024-09-21T15:50:44.468 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.468 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.468 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:44.468 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629454+0000", 2024-09-21T15:50:44.468 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013767099999999999 2024-09-21T15:50:44.468 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.468 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:44.469 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.469 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.469 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.469 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120125 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 600064~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:44.469 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.574482+0000", 2024-09-21T15:50:44.469 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 40.831193132999999, 2024-09-21T15:50:44.469 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055243608, 2024-09-21T15:50:44.469 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:44.469 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:44.469 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:44.469 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:44.469 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:44.469 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120125 2024-09-21T15:50:44.469 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.469 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:44.470 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.470 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:44.470 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574482+0000", 2024-09-21T15:50:44.470 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.470 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.470 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.470 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:44.470 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574482+0000", 2024-09-21T15:50:44.470 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.470 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.470 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.470 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:44.470 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574038+0000", 2024-09-21T15:50:44.470 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9995556 2024-09-21T15:50:44.470 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.471 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.471 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:44.471 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574497+0000", 2024-09-21T15:50:44.471 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00045854400000000002 2024-09-21T15:50:44.471 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.471 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.471 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:44.471 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574509+0000", 2024-09-21T15:50:44.471 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2486e-05 2024-09-21T15:50:44.471 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.471 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.471 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:44.471 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574516+0000", 2024-09-21T15:50:44.471 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2269999999999996e-06 2024-09-21T15:50:44.471 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.472 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.472 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:44.472 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629497+0000", 2024-09-21T15:50:44.472 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054980785999999997 2024-09-21T15:50:44.472 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.472 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.472 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:44.472 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629580+0000", 2024-09-21T15:50:44.472 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2528000000000007e-05 2024-09-21T15:50:44.472 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.472 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.472 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:44.472 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629726+0000", 2024-09-21T15:50:44.472 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014630399999999999 2024-09-21T15:50:44.472 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.473 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:44.473 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.473 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.473 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.473 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120126 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 620544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:44.473 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.575315+0000", 2024-09-21T15:50:44.473 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 40.830360102999997, 2024-09-21T15:50:44.473 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054709559999999997, 2024-09-21T15:50:44.473 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:44.473 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:44.473 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:44.473 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:44.473 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:44.473 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120126 2024-09-21T15:50:44.473 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.473 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:44.474 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.474 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:44.474 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575315+0000", 2024-09-21T15:50:44.474 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.474 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.474 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.474 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:44.474 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575315+0000", 2024-09-21T15:50:44.474 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.474 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.474 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.474 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:44.474 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574522+0000", 2024-09-21T15:50:44.474 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9992065 2024-09-21T15:50:44.475 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.475 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.475 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:44.475 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575319+0000", 2024-09-21T15:50:44.475 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00079626800000000004 2024-09-21T15:50:44.475 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.475 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.475 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:44.475 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575329+0000", 2024-09-21T15:50:44.475 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.028e-05 2024-09-21T15:50:44.475 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.475 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.475 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:44.475 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575335+0000", 2024-09-21T15:50:44.475 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9290000000000003e-06 2024-09-21T15:50:44.476 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.476 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.476 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:44.476 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629806+0000", 2024-09-21T15:50:44.476 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054471558000000003 2024-09-21T15:50:44.476 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.476 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.476 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:44.476 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629885+0000", 2024-09-21T15:50:44.476 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8563e-05 2024-09-21T15:50:44.476 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.476 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.476 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:44.476 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630025+0000", 2024-09-21T15:50:44.476 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014019 2024-09-21T15:50:44.477 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.477 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:44.477 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.477 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.477 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.477 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120127 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 641024~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:44.477 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.575340+0000", 2024-09-21T15:50:44.477 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 40.830335396000002, 2024-09-21T15:50:44.477 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054895589000000002, 2024-09-21T15:50:44.477 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:44.477 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:44.477 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:44.477 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:44.477 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:44.477 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120127 2024-09-21T15:50:44.478 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.478 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:44.478 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.478 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:44.478 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575340+0000", 2024-09-21T15:50:44.478 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.478 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.478 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.478 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:44.478 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575340+0000", 2024-09-21T15:50:44.478 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.478 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.478 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.478 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:44.479 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575340+0000", 2024-09-21T15:50:44.479 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:44.479 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.479 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.479 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:44.479 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575342+0000", 2024-09-21T15:50:44.479 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5600000000000001e-06 2024-09-21T15:50:44.479 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.479 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.479 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:44.479 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575351+0000", 2024-09-21T15:50:44.479 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2349999999999999e-06 2024-09-21T15:50:44.479 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.479 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.480 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:44.480 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575356+0000", 2024-09-21T15:50:44.480 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4600000000000002e-06 2024-09-21T15:50:44.480 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.480 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.480 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:44.480 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630053+0000", 2024-09-21T15:50:44.480 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054697190999999999 2024-09-21T15:50:44.480 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.480 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.480 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:44.480 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630128+0000", 2024-09-21T15:50:44.480 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4777000000000006e-05 2024-09-21T15:50:44.480 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.481 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.481 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:44.481 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630236+0000", 2024-09-21T15:50:44.481 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000107837 2024-09-21T15:50:44.481 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.481 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:44.481 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.481 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.481 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.481 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120128 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 657408~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:44.481 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.575461+0000", 2024-09-21T15:50:44.481 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 40.830214816999998, 2024-09-21T15:50:44.481 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054945121999999999, 2024-09-21T15:50:44.481 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:44.481 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:44.482 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:44.482 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:44.482 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:44.482 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120128 2024-09-21T15:50:44.482 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.482 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:44.482 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.482 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:44.482 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575461+0000", 2024-09-21T15:50:44.482 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.482 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.482 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.482 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:44.483 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575461+0000", 2024-09-21T15:50:44.483 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.483 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.483 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.483 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:44.483 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575360+0000", 2024-09-21T15:50:44.483 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998994 2024-09-21T15:50:44.483 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.483 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.483 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:44.483 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575464+0000", 2024-09-21T15:50:44.483 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000103397 2024-09-21T15:50:44.483 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.483 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.483 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:44.484 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575472+0000", 2024-09-21T15:50:44.484 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3189999999999995e-06 2024-09-21T15:50:44.484 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.484 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.484 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:44.484 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575478+0000", 2024-09-21T15:50:44.484 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2779999999999999e-06 2024-09-21T15:50:44.484 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.484 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.484 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:44.484 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630268+0000", 2024-09-21T15:50:44.484 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054789545000000002 2024-09-21T15:50:44.484 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.484 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.484 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:44.485 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630315+0000", 2024-09-21T15:50:44.485 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7244999999999999e-05 2024-09-21T15:50:44.485 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.485 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.485 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:44.485 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630406+0000", 2024-09-21T15:50:44.485 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0811999999999997e-05 2024-09-21T15:50:44.485 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.485 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:44.485 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.485 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.485 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.485 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120129 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 677888~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:44.485 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.576016+0000", 2024-09-21T15:50:44.486 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 40.829659272999997, 2024-09-21T15:50:44.486 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054602823000000002, 2024-09-21T15:50:44.486 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:44.486 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:44.486 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:44.486 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:44.486 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:44.486 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120129 2024-09-21T15:50:44.486 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.486 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:44.486 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.486 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:44.486 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576016+0000", 2024-09-21T15:50:44.486 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.486 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.487 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.487 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:44.487 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576016+0000", 2024-09-21T15:50:44.487 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.487 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.487 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.487 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:44.487 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575483+0000", 2024-09-21T15:50:44.487 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9994669 2024-09-21T15:50:44.487 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.487 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.487 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:44.487 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576020+0000", 2024-09-21T15:50:44.487 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00053657900000000003 2024-09-21T15:50:44.487 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.488 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.488 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:44.488 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576030+0000", 2024-09-21T15:50:44.488 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0326000000000001e-05 2024-09-21T15:50:44.488 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.488 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.488 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:44.488 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576037+0000", 2024-09-21T15:50:44.488 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6490000000000002e-06 2024-09-21T15:50:44.488 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.488 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.488 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:44.488 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630446+0000", 2024-09-21T15:50:44.488 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054409698999999999 2024-09-21T15:50:44.488 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.488 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.489 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:44.489 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630524+0000", 2024-09-21T15:50:44.489 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7422999999999994e-05 2024-09-21T15:50:44.489 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.489 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.489 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:44.489 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630619+0000", 2024-09-21T15:50:44.489 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5332000000000001e-05 2024-09-21T15:50:44.489 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.489 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:44.489 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.489 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.489 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.489 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120130 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 698368~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:44.489 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.576043+0000", 2024-09-21T15:50:44.490 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 40.829632902, 2024-09-21T15:50:44.490 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054781798, 2024-09-21T15:50:44.490 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:44.490 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:44.490 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:44.490 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:44.490 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:44.490 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120130 2024-09-21T15:50:44.490 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.490 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:44.490 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.490 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:44.490 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576043+0000", 2024-09-21T15:50:44.490 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.491 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.491 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.491 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:44.491 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576043+0000", 2024-09-21T15:50:44.491 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.491 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.491 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.491 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:44.491 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576042+0000", 2024-09-21T15:50:44.491 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:44.491 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.491 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.491 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:44.491 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576044+0000", 2024-09-21T15:50:44.491 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0999999999999998e-06 2024-09-21T15:50:44.492 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.492 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.492 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:44.492 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576051+0000", 2024-09-21T15:50:44.492 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4779999999999996e-06 2024-09-21T15:50:44.492 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.492 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.492 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:44.492 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576058+0000", 2024-09-21T15:50:44.492 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2180000000000002e-06 2024-09-21T15:50:44.492 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.492 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.492 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:44.492 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630661+0000", 2024-09-21T15:50:44.492 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054603240999999997 2024-09-21T15:50:44.492 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.493 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.493 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:44.493 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630747+0000", 2024-09-21T15:50:44.493 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6046999999999998e-05 2024-09-21T15:50:44.493 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.493 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.493 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:44.493 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630824+0000", 2024-09-21T15:50:44.493 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7119999999999993e-05 2024-09-21T15:50:44.493 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.493 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:44.493 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.493 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.493 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:44.493 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:44.494 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T15:50:44.504 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:44.503+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:44.605 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:44.604+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:44.638 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:44.638 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:44.638 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:50:44.638 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:44.647 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T15:50:44.706 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:50:44.705+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:50:44.737 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T15:50:44.794 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:44.795 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:44.795 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:50:44.795 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:50:44.795 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:44.804 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T15:50:44.806 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:44.805+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:44.907 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:44.906+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:44.913 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:44.913 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:50:44.913 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:50:44.913 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:50:44.913 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.913 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63140 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 876544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:44.914 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.133649+0000", 2024-09-21T15:50:44.914 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 67.778041884000004, 2024-09-21T15:50:44.914 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053950201000000003, 2024-09-21T15:50:44.914 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:44.914 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:44.914 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:44.914 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:44.914 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:44.914 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63140 2024-09-21T15:50:44.914 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.914 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:44.914 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.914 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:44.915 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133649+0000", 2024-09-21T15:50:44.915 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.915 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.915 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.915 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:44.915 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133649+0000", 2024-09-21T15:50:44.915 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.915 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.915 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.915 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:44.915 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133648+0000", 2024-09-21T15:50:44.915 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:44.915 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.916 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.916 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:44.916 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133651+0000", 2024-09-21T15:50:44.916 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1549999999999999e-06 2024-09-21T15:50:44.916 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.916 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.916 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:44.916 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133660+0000", 2024-09-21T15:50:44.916 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3899999999999993e-06 2024-09-21T15:50:44.916 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.916 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.916 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:44.916 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133669+0000", 2024-09-21T15:50:44.916 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3219999999999999e-06 2024-09-21T15:50:44.916 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.916 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.916 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:44.917 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187409+0000", 2024-09-21T15:50:44.917 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053740257 2024-09-21T15:50:44.917 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.917 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.917 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:44.917 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187492+0000", 2024-09-21T15:50:44.917 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2409000000000003e-05 2024-09-21T15:50:44.917 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.917 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.917 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:44.917 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187599+0000", 2024-09-21T15:50:44.917 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000107334 2024-09-21T15:50:44.917 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.917 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:44.917 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.917 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.918 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.918 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63141 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 884736~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:44.918 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.133843+0000", 2024-09-21T15:50:44.918 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 67.777847765000004, 2024-09-21T15:50:44.918 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053954015000000001, 2024-09-21T15:50:44.918 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:44.918 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:44.918 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:44.918 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:44.918 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:44.918 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63141 2024-09-21T15:50:44.918 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.918 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:44.918 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.918 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:44.918 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133843+0000", 2024-09-21T15:50:44.919 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.919 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.919 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.919 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:44.919 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133843+0000", 2024-09-21T15:50:44.919 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.919 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.919 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.919 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:44.919 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133673+0000", 2024-09-21T15:50:44.919 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998302 2024-09-21T15:50:44.919 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.919 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.919 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:44.919 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133846+0000", 2024-09-21T15:50:44.920 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000172762 2024-09-21T15:50:44.920 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.920 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.920 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:44.920 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133855+0000", 2024-09-21T15:50:44.920 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2940000000000006e-06 2024-09-21T15:50:44.920 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.920 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.920 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:44.920 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133863+0000", 2024-09-21T15:50:44.920 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9740000000000002e-06 2024-09-21T15:50:44.920 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.920 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.920 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:44.920 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187630+0000", 2024-09-21T15:50:44.921 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053767211000000002 2024-09-21T15:50:44.921 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.921 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.921 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:44.921 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187707+0000", 2024-09-21T15:50:44.921 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7446000000000004e-05 2024-09-21T15:50:44.921 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.921 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.921 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:44.921 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187797+0000", 2024-09-21T15:50:44.921 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.9858000000000006e-05 2024-09-21T15:50:44.921 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.921 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:44.921 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.922 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.922 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.922 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63142 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 892928~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:44.922 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134046+0000", 2024-09-21T15:50:44.922 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 67.777644823000003, 2024-09-21T15:50:44.922 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054017849999999999, 2024-09-21T15:50:44.922 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:44.922 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:44.922 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:44.922 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:44.922 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:44.922 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63142 2024-09-21T15:50:44.922 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.922 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:44.922 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.923 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:44.923 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134046+0000", 2024-09-21T15:50:44.923 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.923 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.923 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.923 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:44.923 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134046+0000", 2024-09-21T15:50:44.923 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.923 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.923 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.923 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:44.923 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133867+0000", 2024-09-21T15:50:44.923 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998217 2024-09-21T15:50:44.923 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.924 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.924 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:44.924 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134049+0000", 2024-09-21T15:50:44.924 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000181695 2024-09-21T15:50:44.924 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.924 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.924 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:44.924 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134057+0000", 2024-09-21T15:50:44.924 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.303e-06 2024-09-21T15:50:44.924 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.924 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.924 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:44.924 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134063+0000", 2024-09-21T15:50:44.924 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9959999999999999e-06 2024-09-21T15:50:44.925 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.925 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.925 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:44.925 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187849+0000", 2024-09-21T15:50:44.925 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053785571999999997 2024-09-21T15:50:44.925 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.925 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.925 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:44.925 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187954+0000", 2024-09-21T15:50:44.925 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010524899999999999 2024-09-21T15:50:44.925 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.925 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.925 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:44.925 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188064+0000", 2024-09-21T15:50:44.925 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000109549 2024-09-21T15:50:44.926 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.926 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:44.926 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.926 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.926 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.926 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63143 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 901120~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:44.926 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134239+0000", 2024-09-21T15:50:44.926 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 67.777452229999994, 2024-09-21T15:50:44.926 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054026058000000002, 2024-09-21T15:50:44.926 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:44.926 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:44.926 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:44.926 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:44.926 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:44.926 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63143 2024-09-21T15:50:44.927 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.927 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:44.927 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.927 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:44.927 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134239+0000", 2024-09-21T15:50:44.927 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.927 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.927 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.927 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:44.927 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134239+0000", 2024-09-21T15:50:44.927 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.927 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.927 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.927 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:44.927 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134068+0000", 2024-09-21T15:50:44.928 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998298 2024-09-21T15:50:44.928 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.928 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.928 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:44.928 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134242+0000", 2024-09-21T15:50:44.928 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00017414199999999999 2024-09-21T15:50:44.928 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.928 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.928 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:44.928 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134264+0000", 2024-09-21T15:50:44.928 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1552e-05 2024-09-21T15:50:44.928 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.928 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.928 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:44.928 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134270+0000", 2024-09-21T15:50:44.928 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6939999999999998e-06 2024-09-21T15:50:44.929 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.929 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.929 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:44.929 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188108+0000", 2024-09-21T15:50:44.929 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.05383719 2024-09-21T15:50:44.929 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.929 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.929 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:44.929 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188186+0000", 2024-09-21T15:50:44.929 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8318000000000005e-05 2024-09-21T15:50:44.929 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.929 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.929 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:44.929 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188265+0000", 2024-09-21T15:50:44.929 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8597000000000007e-05 2024-09-21T15:50:44.929 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.930 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:44.930 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.930 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.930 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.930 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63144 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 909312~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:44.930 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134432+0000", 2024-09-21T15:50:44.930 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 67.777258673999995, 2024-09-21T15:50:44.930 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053968953, 2024-09-21T15:50:44.930 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:44.930 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:44.930 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:44.930 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:44.930 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:44.930 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63144 2024-09-21T15:50:44.931 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.931 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:44.931 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.931 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:44.931 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134432+0000", 2024-09-21T15:50:44.931 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.931 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.931 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.931 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:44.931 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134432+0000", 2024-09-21T15:50:44.931 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.931 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.931 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.931 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:44.932 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134276+0000", 2024-09-21T15:50:44.932 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998441 2024-09-21T15:50:44.932 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.932 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.932 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:44.932 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134436+0000", 2024-09-21T15:50:44.932 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000159591 2024-09-21T15:50:44.932 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.932 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.932 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:44.932 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134445+0000", 2024-09-21T15:50:44.932 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6780000000000006e-06 2024-09-21T15:50:44.932 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.932 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.932 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:44.932 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134452+0000", 2024-09-21T15:50:44.933 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6830000000000008e-06 2024-09-21T15:50:44.933 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.933 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.933 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:44.933 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188282+0000", 2024-09-21T15:50:44.933 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053829252000000001 2024-09-21T15:50:44.933 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.933 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.933 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:44.933 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188328+0000", 2024-09-21T15:50:44.933 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6734999999999999e-05 2024-09-21T15:50:44.933 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.933 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.933 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:44.933 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188401+0000", 2024-09-21T15:50:44.934 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2731000000000005e-05 2024-09-21T15:50:44.934 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.934 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:44.934 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.934 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.934 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.934 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63145 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 860160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:44.934 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134736+0000", 2024-09-21T15:50:44.934 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 67.776954888999995, 2024-09-21T15:50:44.934 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053865960999999997, 2024-09-21T15:50:44.934 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:44.934 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:44.934 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:44.934 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:44.934 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:44.935 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63145 2024-09-21T15:50:44.935 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.935 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:44.935 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.935 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:44.935 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134736+0000", 2024-09-21T15:50:44.935 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.935 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.935 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.935 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:44.935 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134736+0000", 2024-09-21T15:50:44.935 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.935 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.935 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.935 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:44.935 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134458+0000", 2024-09-21T15:50:44.936 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999722 2024-09-21T15:50:44.936 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.936 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.936 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:44.936 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134741+0000", 2024-09-21T15:50:44.936 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00028245599999999999 2024-09-21T15:50:44.936 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.936 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.936 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:44.936 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134753+0000", 2024-09-21T15:50:44.936 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2548000000000001e-05 2024-09-21T15:50:44.936 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.936 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.936 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:44.936 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134760+0000", 2024-09-21T15:50:44.936 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4189999999999996e-06 2024-09-21T15:50:44.937 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.937 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.937 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:44.937 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188426+0000", 2024-09-21T15:50:44.937 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053665894999999998 2024-09-21T15:50:44.937 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.937 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.937 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:44.937 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188502+0000", 2024-09-21T15:50:44.937 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5791000000000007e-05 2024-09-21T15:50:44.937 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.937 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.937 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:44.937 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188602+0000", 2024-09-21T15:50:44.937 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9667999999999995e-05 2024-09-21T15:50:44.937 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.937 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:44.938 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.938 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.938 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.938 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63148 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 933888~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:44.938 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.135289+0000", 2024-09-21T15:50:44.938 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 67.776401934000006, 2024-09-21T15:50:44.938 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053948488000000003, 2024-09-21T15:50:44.938 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:44.938 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:44.938 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:44.938 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:44.938 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:44.938 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63148 2024-09-21T15:50:44.938 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.938 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:44.938 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.939 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:44.939 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135289+0000", 2024-09-21T15:50:44.939 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.939 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.939 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.939 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:44.939 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135289+0000", 2024-09-21T15:50:44.939 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.939 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.939 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.939 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:44.939 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135288+0000", 2024-09-21T15:50:44.939 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:44.939 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.939 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.940 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:44.940 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135296+0000", 2024-09-21T15:50:44.940 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5399999999999998e-06 2024-09-21T15:50:44.940 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.940 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.940 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:44.940 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135303+0000", 2024-09-21T15:50:44.940 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0500000000000003e-06 2024-09-21T15:50:44.940 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.940 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.940 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:44.940 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135308+0000", 2024-09-21T15:50:44.940 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9749999999999998e-06 2024-09-21T15:50:44.940 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.940 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.940 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:44.941 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189066+0000", 2024-09-21T15:50:44.941 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053758501 2024-09-21T15:50:44.941 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.941 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.941 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:44.941 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189145+0000", 2024-09-21T15:50:44.941 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9017000000000006e-05 2024-09-21T15:50:44.941 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.941 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.941 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:44.941 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189237+0000", 2024-09-21T15:50:44.941 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1906999999999999e-05 2024-09-21T15:50:44.941 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.941 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:44.941 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.942 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.942 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.942 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63149 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 942080~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:44.942 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.135430+0000", 2024-09-21T15:50:44.942 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 67.776260605000004, 2024-09-21T15:50:44.942 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054005434999999997, 2024-09-21T15:50:44.942 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:44.942 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:44.942 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:44.942 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:44.942 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:44.942 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63149 2024-09-21T15:50:44.942 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.942 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:44.942 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.943 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:44.943 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135430+0000", 2024-09-21T15:50:44.943 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.943 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.943 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.943 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:44.943 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135430+0000", 2024-09-21T15:50:44.943 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.943 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.943 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.943 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:44.943 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135313+0000", 2024-09-21T15:50:44.943 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998827 2024-09-21T15:50:44.943 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.943 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.944 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:44.944 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135434+0000", 2024-09-21T15:50:44.944 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000120876 2024-09-21T15:50:44.944 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.944 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.944 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:44.944 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135443+0000", 2024-09-21T15:50:44.944 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3340000000000008e-06 2024-09-21T15:50:44.944 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.944 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.944 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:44.944 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135449+0000", 2024-09-21T15:50:44.944 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1580000000000002e-06 2024-09-21T15:50:44.944 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.944 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.944 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:44.945 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189267+0000", 2024-09-21T15:50:44.945 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053817775999999998 2024-09-21T15:50:44.945 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.945 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.945 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:44.945 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189336+0000", 2024-09-21T15:50:44.945 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9485000000000003e-05 2024-09-21T15:50:44.945 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.945 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.945 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:44.945 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189436+0000", 2024-09-21T15:50:44.945 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9277000000000005e-05 2024-09-21T15:50:44.945 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.946 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:44.946 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.946 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.946 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.946 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63150 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 950272~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:44.946 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.135618+0000", 2024-09-21T15:50:44.946 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 67.776072859999999, 2024-09-21T15:50:44.946 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054027393, 2024-09-21T15:50:44.946 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:44.946 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:44.946 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:44.946 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:44.946 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:44.946 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63150 2024-09-21T15:50:44.946 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.946 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:44.947 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.947 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:44.947 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135618+0000", 2024-09-21T15:50:44.947 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.947 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.947 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.947 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:44.947 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135618+0000", 2024-09-21T15:50:44.947 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.947 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.947 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.947 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:44.947 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135454+0000", 2024-09-21T15:50:44.947 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998364 2024-09-21T15:50:44.947 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.948 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.948 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:44.948 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135621+0000", 2024-09-21T15:50:44.948 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016714099999999999 2024-09-21T15:50:44.948 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.948 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.948 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:44.948 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135631+0000", 2024-09-21T15:50:44.948 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5340000000000005e-06 2024-09-21T15:50:44.948 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.948 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.948 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:44.948 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135638+0000", 2024-09-21T15:50:44.948 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9280000000000004e-06 2024-09-21T15:50:44.948 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.948 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.949 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:44.949 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189463+0000", 2024-09-21T15:50:44.949 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053825681 2024-09-21T15:50:44.949 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.949 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.949 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:44.949 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189546+0000", 2024-09-21T15:50:44.949 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2278000000000001e-05 2024-09-21T15:50:44.949 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.949 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.949 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:44.949 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189645+0000", 2024-09-21T15:50:44.949 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9543999999999994e-05 2024-09-21T15:50:44.949 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.949 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:44.949 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.950 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.950 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.950 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63152 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 966656~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:44.950 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.136029+0000", 2024-09-21T15:50:44.950 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 67.775662268000005, 2024-09-21T15:50:44.950 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054024823999999999, 2024-09-21T15:50:44.950 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:44.950 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:44.950 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:44.950 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:44.950 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:44.950 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63152 2024-09-21T15:50:44.950 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.950 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:44.950 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.951 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:44.951 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136029+0000", 2024-09-21T15:50:44.951 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.951 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.951 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.951 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:44.951 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136029+0000", 2024-09-21T15:50:44.951 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.951 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.951 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.951 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:44.951 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136028+0000", 2024-09-21T15:50:44.951 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:44.951 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.951 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.952 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:44.952 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136031+0000", 2024-09-21T15:50:44.952 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.033e-06 2024-09-21T15:50:44.952 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.952 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.952 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:44.952 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136039+0000", 2024-09-21T15:50:44.952 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7200000000000006e-06 2024-09-21T15:50:44.952 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.952 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.952 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:44.952 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136051+0000", 2024-09-21T15:50:44.952 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1987000000000001e-05 2024-09-21T15:50:44.952 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.952 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.952 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:44.953 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189876+0000", 2024-09-21T15:50:44.953 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053825264999999997 2024-09-21T15:50:44.953 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.953 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.953 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:44.953 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189955+0000", 2024-09-21T15:50:44.953 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9296999999999997e-05 2024-09-21T15:50:44.953 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.953 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.953 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:44.953 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190053+0000", 2024-09-21T15:50:44.953 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.8067999999999997e-05 2024-09-21T15:50:44.953 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.953 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:44.953 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.953 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.954 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.954 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63153 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 974848~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:44.954 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.136311+0000", 2024-09-21T15:50:44.954 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 67.775379755000003, 2024-09-21T15:50:44.954 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053933825999999997, 2024-09-21T15:50:44.954 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:44.954 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:44.954 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:44.954 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:44.954 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:44.954 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63153 2024-09-21T15:50:44.954 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.954 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:44.954 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.954 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:44.955 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136311+0000", 2024-09-21T15:50:44.955 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.955 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.955 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.955 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:44.955 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136311+0000", 2024-09-21T15:50:44.955 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.955 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.955 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.955 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:44.955 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136057+0000", 2024-09-21T15:50:44.955 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9997463 2024-09-21T15:50:44.955 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.956 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.956 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:44.956 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136316+0000", 2024-09-21T15:50:44.956 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00025872299999999998 2024-09-21T15:50:44.956 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.956 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.956 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:44.956 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136326+0000", 2024-09-21T15:50:44.956 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0552e-05 2024-09-21T15:50:44.956 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.956 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.956 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:44.956 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136334+0000", 2024-09-21T15:50:44.956 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1409999999999999e-06 2024-09-21T15:50:44.956 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.957 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.957 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:44.957 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190080+0000", 2024-09-21T15:50:44.957 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053745933000000003 2024-09-21T15:50:44.957 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.957 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.957 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:44.957 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190157+0000", 2024-09-21T15:50:44.957 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7412999999999999e-05 2024-09-21T15:50:44.957 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.957 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.957 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:44.957 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190245+0000", 2024-09-21T15:50:44.957 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7904000000000004e-05 2024-09-21T15:50:44.957 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.957 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:44.958 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.958 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.958 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.958 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63154 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 983040~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:44.958 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.136472+0000", 2024-09-21T15:50:44.958 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 67.775218404, 2024-09-21T15:50:44.958 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053968448000000002, 2024-09-21T15:50:44.958 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:44.958 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:44.958 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:44.958 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:44.958 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:44.958 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63154 2024-09-21T15:50:44.959 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.959 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:44.959 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.959 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:44.959 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136472+0000", 2024-09-21T15:50:44.959 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.959 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.959 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.959 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:44.959 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136472+0000", 2024-09-21T15:50:44.959 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.959 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.959 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.959 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:44.959 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136341+0000", 2024-09-21T15:50:44.959 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998684 2024-09-21T15:50:44.960 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.960 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.960 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:44.960 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136476+0000", 2024-09-21T15:50:44.960 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013528100000000001 2024-09-21T15:50:44.960 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.960 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.960 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:44.960 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136484+0000", 2024-09-21T15:50:44.960 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4760000000000001e-06 2024-09-21T15:50:44.960 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.960 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.960 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:44.960 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136490+0000", 2024-09-21T15:50:44.960 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8640000000000001e-06 2024-09-21T15:50:44.961 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.961 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.961 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:44.961 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190271+0000", 2024-09-21T15:50:44.961 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053781223000000003 2024-09-21T15:50:44.961 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.961 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.961 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:44.961 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190348+0000", 2024-09-21T15:50:44.961 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7058999999999995e-05 2024-09-21T15:50:44.961 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.961 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.961 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:44.961 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190441+0000", 2024-09-21T15:50:44.961 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2937999999999997e-05 2024-09-21T15:50:44.962 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.962 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:44.962 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.962 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.962 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.962 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63157 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1015808~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:44.962 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137112+0000", 2024-09-21T15:50:44.962 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 67.774579117000002, 2024-09-21T15:50:44.962 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053932345, 2024-09-21T15:50:44.962 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:44.962 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:44.962 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:44.962 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:44.962 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:44.962 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63157 2024-09-21T15:50:44.963 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.963 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:44.963 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.963 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:44.963 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137112+0000", 2024-09-21T15:50:44.963 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.963 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.963 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.963 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:44.963 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137112+0000", 2024-09-21T15:50:44.963 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.963 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.963 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.963 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:44.963 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137072+0000", 2024-09-21T15:50:44.963 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999604 2024-09-21T15:50:44.964 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.964 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.964 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:44.964 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137115+0000", 2024-09-21T15:50:44.964 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2379999999999997e-05 2024-09-21T15:50:44.964 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.964 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.964 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:44.964 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137121+0000", 2024-09-21T15:50:44.964 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6499999999999999e-06 2024-09-21T15:50:44.964 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.964 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.964 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:44.964 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137126+0000", 2024-09-21T15:50:44.964 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0769999999999998e-06 2024-09-21T15:50:44.965 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.965 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.965 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:44.965 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190868+0000", 2024-09-21T15:50:44.965 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053741555000000003 2024-09-21T15:50:44.965 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.965 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.965 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:44.965 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190945+0000", 2024-09-21T15:50:44.965 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7171999999999999e-05 2024-09-21T15:50:44.965 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.965 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.965 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:44.965 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191044+0000", 2024-09-21T15:50:44.965 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9031999999999996e-05 2024-09-21T15:50:44.966 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.966 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:44.966 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.966 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.966 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.966 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63158 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1007616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:44.966 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137293+0000", 2024-09-21T15:50:44.966 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 67.774397829999998, 2024-09-21T15:50:44.966 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054005642, 2024-09-21T15:50:44.966 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:44.966 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:44.966 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:44.966 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:44.966 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:44.966 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63158 2024-09-21T15:50:44.967 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.967 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:44.967 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.967 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:44.967 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137293+0000", 2024-09-21T15:50:44.967 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.967 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.967 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.967 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:44.967 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137293+0000", 2024-09-21T15:50:44.967 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.967 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.967 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.967 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:44.967 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137131+0000", 2024-09-21T15:50:44.967 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998379 2024-09-21T15:50:44.968 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.968 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.968 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:44.968 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137296+0000", 2024-09-21T15:50:44.968 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016568999999999999 2024-09-21T15:50:44.968 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.968 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.968 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:44.968 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137306+0000", 2024-09-21T15:50:44.968 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.6029999999999994e-06 2024-09-21T15:50:44.968 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.968 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.968 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:44.968 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137312+0000", 2024-09-21T15:50:44.968 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3210000000000001e-06 2024-09-21T15:50:44.969 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.969 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.969 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:44.969 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191087+0000", 2024-09-21T15:50:44.969 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053774686000000002 2024-09-21T15:50:44.969 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.969 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.969 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:44.969 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191175+0000", 2024-09-21T15:50:44.969 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8106e-05 2024-09-21T15:50:44.969 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.969 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.969 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:44.969 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191299+0000", 2024-09-21T15:50:44.969 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012343599999999999 2024-09-21T15:50:44.970 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.970 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:44.970 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.970 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.970 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.970 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63159 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1024000~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:44.970 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137568+0000", 2024-09-21T15:50:44.970 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 67.774122558000002, 2024-09-21T15:50:44.970 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053962999999999997, 2024-09-21T15:50:44.970 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:44.970 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:44.970 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:44.970 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:44.971 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:44.971 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63159 2024-09-21T15:50:44.971 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.971 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:44.971 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.971 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:44.971 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137568+0000", 2024-09-21T15:50:44.971 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.971 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.971 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.971 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:44.971 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137568+0000", 2024-09-21T15:50:44.971 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.971 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.971 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.972 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:44.972 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137317+0000", 2024-09-21T15:50:44.972 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9997492 2024-09-21T15:50:44.972 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.972 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.972 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:44.972 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137572+0000", 2024-09-21T15:50:44.972 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00025491499999999998 2024-09-21T15:50:44.972 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.972 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.972 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:44.972 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137582+0000", 2024-09-21T15:50:44.972 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4490000000000005e-06 2024-09-21T15:50:44.972 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.973 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.973 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:44.973 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137589+0000", 2024-09-21T15:50:44.973 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5580000000000003e-06 2024-09-21T15:50:44.973 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.973 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.973 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:44.973 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191344+0000", 2024-09-21T15:50:44.973 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053754272999999998 2024-09-21T15:50:44.973 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.973 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.973 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:44.973 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191429+0000", 2024-09-21T15:50:44.973 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.509e-05 2024-09-21T15:50:44.974 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.974 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.974 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:44.974 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191531+0000", 2024-09-21T15:50:44.974 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000102519 2024-09-21T15:50:44.974 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.974 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:44.974 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.974 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.974 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.974 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63160 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1032192~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:44.974 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137771+0000", 2024-09-21T15:50:44.974 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 67.773920024000006, 2024-09-21T15:50:44.974 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053957818999999997, 2024-09-21T15:50:44.975 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:44.975 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:44.975 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:44.975 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:44.975 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:44.975 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63160 2024-09-21T15:50:44.975 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.975 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:44.975 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.975 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:44.975 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137771+0000", 2024-09-21T15:50:44.975 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.975 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.976 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.976 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:44.976 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137771+0000", 2024-09-21T15:50:44.976 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.976 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.976 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.976 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:44.976 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137596+0000", 2024-09-21T15:50:44.976 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998255 2024-09-21T15:50:44.976 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.976 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.976 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:44.976 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137780+0000", 2024-09-21T15:50:44.976 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00018363900000000001 2024-09-21T15:50:44.976 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.977 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.977 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:44.977 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137791+0000", 2024-09-21T15:50:44.977 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.096e-05 2024-09-21T15:50:44.977 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.977 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.977 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:44.977 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137798+0000", 2024-09-21T15:50:44.977 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1169999999999998e-06 2024-09-21T15:50:44.977 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.977 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.977 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:44.977 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191560+0000", 2024-09-21T15:50:44.977 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053762163000000002 2024-09-21T15:50:44.977 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.977 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.978 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:44.978 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191634+0000", 2024-09-21T15:50:44.978 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4381999999999994e-05 2024-09-21T15:50:44.978 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.978 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.978 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:44.978 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191729+0000", 2024-09-21T15:50:44.978 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4098000000000006e-05 2024-09-21T15:50:44.978 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.978 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:44.978 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.978 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.978 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.978 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63161 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1040384~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:44.978 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.138018+0000", 2024-09-21T15:50:44.978 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 67.773673270000003, 2024-09-21T15:50:44.979 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053905657000000003, 2024-09-21T15:50:44.979 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:44.979 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:44.979 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:44.979 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:44.979 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:44.979 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63161 2024-09-21T15:50:44.979 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.979 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:44.979 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.979 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:44.979 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138018+0000", 2024-09-21T15:50:44.979 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.979 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.979 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.979 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:44.980 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138018+0000", 2024-09-21T15:50:44.980 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.980 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.980 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.980 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:44.980 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137804+0000", 2024-09-21T15:50:44.980 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9997859 2024-09-21T15:50:44.980 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.980 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.980 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:44.980 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138021+0000", 2024-09-21T15:50:44.980 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00021769899999999999 2024-09-21T15:50:44.980 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.980 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.980 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:44.981 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138043+0000", 2024-09-21T15:50:44.981 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1627000000000001e-05 2024-09-21T15:50:44.981 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.981 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.981 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:44.981 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138054+0000", 2024-09-21T15:50:44.981 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1321999999999999e-05 2024-09-21T15:50:44.981 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.981 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.981 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:44.981 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191755+0000", 2024-09-21T15:50:44.981 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053700354999999998 2024-09-21T15:50:44.981 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.981 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.981 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:44.982 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191830+0000", 2024-09-21T15:50:44.982 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5320000000000004e-05 2024-09-21T15:50:44.982 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.982 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.982 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:44.982 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191923+0000", 2024-09-21T15:50:44.982 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3282999999999994e-05 2024-09-21T15:50:44.982 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.982 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:44.982 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.982 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.982 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.982 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63176 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 108544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:44.983 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.140861+0000", 2024-09-21T15:50:44.983 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 67.770829950999996, 2024-09-21T15:50:44.983 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053935394999999997, 2024-09-21T15:50:44.983 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:44.983 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:44.983 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:44.983 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:44.983 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:44.983 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63176 2024-09-21T15:50:44.983 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.983 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:44.983 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.983 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:44.983 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140861+0000", 2024-09-21T15:50:44.984 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.984 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.984 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.984 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:44.984 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140861+0000", 2024-09-21T15:50:44.984 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.984 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.984 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.984 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:44.984 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140860+0000", 2024-09-21T15:50:44.984 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:44.984 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.984 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.984 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:44.985 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140863+0000", 2024-09-21T15:50:44.985 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8710000000000001e-06 2024-09-21T15:50:44.985 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.985 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.985 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:44.985 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140871+0000", 2024-09-21T15:50:44.985 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8089999999999993e-06 2024-09-21T15:50:44.985 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.985 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.985 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:44.985 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140877+0000", 2024-09-21T15:50:44.985 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5969999999999999e-06 2024-09-21T15:50:44.985 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.985 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.985 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:44.986 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.194608+0000", 2024-09-21T15:50:44.986 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053731749000000002 2024-09-21T15:50:44.986 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.986 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.986 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:44.986 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.194697+0000", 2024-09-21T15:50:44.986 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8504000000000005e-05 2024-09-21T15:50:44.986 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.986 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.986 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:44.986 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.194796+0000", 2024-09-21T15:50:44.986 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9414999999999996e-05 2024-09-21T15:50:44.986 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.986 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:44.986 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.987 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.987 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.987 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63182 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 157696~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:44.987 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:50:44.987 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 67.769557551000005, 2024-09-21T15:50:44.987 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053958825000000002, 2024-09-21T15:50:44.987 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:44.987 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:44.987 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:44.987 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:44.987 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:44.987 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63182 2024-09-21T15:50:44.987 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.987 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:44.987 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.988 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:44.988 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:50:44.988 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.988 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.988 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.988 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:44.988 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:50:44.988 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.988 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.988 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.988 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:44.988 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:50:44.988 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:44.988 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.988 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.989 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:44.989 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142136+0000", 2024-09-21T15:50:44.989 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8169999999999999e-06 2024-09-21T15:50:44.989 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.989 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.989 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:44.989 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:50:44.989 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3752e-05 2024-09-21T15:50:44.989 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.989 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.989 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:44.989 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142155+0000", 2024-09-21T15:50:44.989 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1859999999999997e-06 2024-09-21T15:50:44.990 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.990 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.990 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:44.990 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.195914+0000", 2024-09-21T15:50:44.990 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053758528999999999 2024-09-21T15:50:44.990 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.990 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.990 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:44.990 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.195995+0000", 2024-09-21T15:50:44.990 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0983999999999996e-05 2024-09-21T15:50:44.990 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.990 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.990 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:44.990 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.196092+0000", 2024-09-21T15:50:44.990 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7069000000000002e-05 2024-09-21T15:50:44.991 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.991 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:44.991 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.991 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.991 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.991 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63193 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 239616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:44.991 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:50:44.991 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 67.767469965999993, 2024-09-21T15:50:44.991 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053956577999999998, 2024-09-21T15:50:44.991 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:44.991 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:44.991 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:44.991 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:44.991 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:44.991 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63193 2024-09-21T15:50:44.992 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.992 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:44.992 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.992 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:44.992 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:50:44.992 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.992 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.992 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.992 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:44.992 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:50:44.992 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:44.992 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.992 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.992 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:44.992 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:50:44.992 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:44.993 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.993 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.993 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:44.993 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144223+0000", 2024-09-21T15:50:44.993 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0990000000000001e-06 2024-09-21T15:50:44.993 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.993 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.993 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:44.993 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144230+0000", 2024-09-21T15:50:44.993 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9070000000000001e-06 2024-09-21T15:50:44.993 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.993 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.993 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:44.993 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144234+0000", 2024-09-21T15:50:44.994 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9010000000000002e-06 2024-09-21T15:50:44.994 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.994 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.994 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:44.994 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.198004+0000", 2024-09-21T15:50:44.994 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053769539999999998 2024-09-21T15:50:44.994 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.994 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.994 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:44.994 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.198083+0000", 2024-09-21T15:50:44.994 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9531999999999997e-05 2024-09-21T15:50:44.994 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:44.994 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:44.994 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:44.995 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.198177+0000", 2024-09-21T15:50:44.995 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3906000000000005e-05 2024-09-21T15:50:44.995 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.995 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:44.995 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.995 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:44.995 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:44.995 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:44.996 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T15:50:45.008 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:45.007+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:45.109 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:45.108+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:45.140 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:45.140 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:45.140 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:50:45.141 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:45.149 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T15:50:45.233 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:45.232+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:45.291 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:45.291 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:45.291 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:50:45.291 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:50:45.291 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:45.300 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T15:50:45.311 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:45.310+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:45.412 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:45.411+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:45.443 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:45.444 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:50:45.444 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:50:45.444 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:50:45.444 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.444 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4209.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:50:45.444 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:50:45.444 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 114.250954916, 2024-09-21T15:50:45.444 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.19452909900000001, 2024-09-21T15:50:45.444 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:45.444 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:45.444 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:45.444 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4209", 2024-09-21T15:50:45.445 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/4045143050", 2024-09-21T15:50:45.445 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1 2024-09-21T15:50:45.445 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.445 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:45.445 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.445 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:45.445 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:50:45.445 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:45.445 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.445 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.445 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:45.445 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:50:45.445 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:45.445 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.445 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.446 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:45.446 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191161+0000", 2024-09-21T15:50:45.446 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-21T15:50:45.446 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.446 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.446 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:45.446 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191169+0000", 2024-09-21T15:50:45.446 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0090000000000007e-06 2024-09-21T15:50:45.446 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.446 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.446 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:45.446 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191180+0000", 2024-09-21T15:50:45.446 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0728e-05 2024-09-21T15:50:45.446 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.446 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.446 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:45.447 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191187+0000", 2024-09-21T15:50:45.447 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9009999999999994e-06 2024-09-21T15:50:45.447 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.447 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.447 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:45.447 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191238+0000", 2024-09-21T15:50:45.447 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0287000000000002e-05 2024-09-21T15:50:45.447 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.447 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.447 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-21T15:50:45.447 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191245+0000", 2024-09-21T15:50:45.447 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0709999999999997e-06 2024-09-21T15:50:45.447 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.447 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.447 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:45.447 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380309+0000", 2024-09-21T15:50:45.448 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.189064277 2024-09-21T15:50:45.448 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.448 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.448 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:45.448 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380454+0000", 2024-09-21T15:50:45.448 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014490799999999999 2024-09-21T15:50:45.448 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.448 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.448 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0", 2024-09-21T15:50:45.448 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380610+0000", 2024-09-21T15:50:45.448 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015650199999999999 2024-09-21T15:50:45.448 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.448 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.448 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:50:45.448 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.381160+0000", 2024-09-21T15:50:45.448 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00054903799999999996 2024-09-21T15:50:45.448 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.449 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.449 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:45.449 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385591+0000", 2024-09-21T15:50:45.449 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.004431272 2024-09-21T15:50:45.449 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.449 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.449 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:50:45.449 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385651+0000", 2024-09-21T15:50:45.449 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0455999999999999e-05 2024-09-21T15:50:45.449 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.449 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.449 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:45.449 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385691+0000", 2024-09-21T15:50:45.449 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0234999999999997e-05 2024-09-21T15:50:45.449 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:45.449 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:45.450 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:45.450 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.450 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.450 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64034 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1083392~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:45.450 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167717+0000", 2024-09-21T15:50:45.450 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 68.274400585999999, 2024-09-21T15:50:45.450 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048233316999999998, 2024-09-21T15:50:45.450 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:45.450 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:45.450 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:45.450 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:45.450 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:45.450 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64034 2024-09-21T15:50:45.450 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.450 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:45.450 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.451 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:45.451 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167717+0000", 2024-09-21T15:50:45.451 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:45.451 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.451 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.451 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:45.451 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167717+0000", 2024-09-21T15:50:45.451 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:45.451 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.451 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.451 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:45.451 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167716+0000", 2024-09-21T15:50:45.451 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:45.451 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.451 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.451 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:45.452 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167718+0000", 2024-09-21T15:50:45.452 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.165e-06 2024-09-21T15:50:45.452 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.452 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.452 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:45.452 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167725+0000", 2024-09-21T15:50:45.452 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3380000000000001e-06 2024-09-21T15:50:45.452 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.452 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.452 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:45.452 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167730+0000", 2024-09-21T15:50:45.452 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8010000000000003e-06 2024-09-21T15:50:45.452 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.452 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.452 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:45.452 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215807+0000", 2024-09-21T15:50:45.452 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048077823999999998 2024-09-21T15:50:45.453 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.453 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.453 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:45.453 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215884+0000", 2024-09-21T15:50:45.453 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6283000000000001e-05 2024-09-21T15:50:45.453 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.453 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.453 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:45.453 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215950+0000", 2024-09-21T15:50:45.453 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6290000000000004e-05 2024-09-21T15:50:45.453 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:45.453 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:45.453 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:45.453 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.453 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.453 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64035 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1091584~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:45.453 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:50:45.454 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 68.274383373000006, 2024-09-21T15:50:45.454 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048412117999999997, 2024-09-21T15:50:45.454 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:45.454 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:45.454 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:45.454 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:45.454 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:45.454 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64035 2024-09-21T15:50:45.454 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.454 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:45.454 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.454 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:45.454 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:50:45.454 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:45.454 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.454 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.455 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:45.455 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:50:45.455 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:45.455 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.455 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.455 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:45.455 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:50:45.455 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:45.455 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.455 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.455 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:45.455 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167746+0000", 2024-09-21T15:50:45.455 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2031000000000001e-05 2024-09-21T15:50:45.455 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.455 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.456 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:45.456 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167753+0000", 2024-09-21T15:50:45.456 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0620000000000003e-06 2024-09-21T15:50:45.456 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.456 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.456 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:45.456 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167759+0000", 2024-09-21T15:50:45.456 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.596e-06 2024-09-21T15:50:45.456 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.456 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.456 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:45.456 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215977+0000", 2024-09-21T15:50:45.456 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048217318000000002 2024-09-21T15:50:45.456 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.456 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.456 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:45.457 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216076+0000", 2024-09-21T15:50:45.457 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9846000000000006e-05 2024-09-21T15:50:45.457 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.457 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.457 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:45.457 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216146+0000", 2024-09-21T15:50:45.457 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9578000000000004e-05 2024-09-21T15:50:45.457 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:45.457 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:45.457 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:45.457 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.457 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.457 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64036 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1099776~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:45.457 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167853+0000", 2024-09-21T15:50:45.457 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 68.274264759999994, 2024-09-21T15:50:45.457 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048450675999999998, 2024-09-21T15:50:45.457 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:45.458 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:45.458 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:45.458 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:45.458 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:45.458 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64036 2024-09-21T15:50:45.458 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.458 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:45.458 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.458 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:45.458 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167853+0000", 2024-09-21T15:50:45.458 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:45.458 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.458 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.458 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:45.458 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167853+0000", 2024-09-21T15:50:45.458 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:45.459 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.459 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.459 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:45.459 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167852+0000", 2024-09-21T15:50:45.459 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:45.459 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.459 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.459 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:45.459 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167855+0000", 2024-09-21T15:50:45.459 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5640000000000001e-06 2024-09-21T15:50:45.459 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.459 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.459 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:45.459 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167864+0000", 2024-09-21T15:50:45.459 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8140000000000007e-06 2024-09-21T15:50:45.459 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.459 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.460 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:45.460 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167871+0000", 2024-09-21T15:50:45.460 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5729999999999996e-06 2024-09-21T15:50:45.460 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.460 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.460 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:45.460 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216174+0000", 2024-09-21T15:50:45.460 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048303437999999997 2024-09-21T15:50:45.460 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.460 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.460 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:45.460 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216250+0000", 2024-09-21T15:50:45.460 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5661999999999995e-05 2024-09-21T15:50:45.460 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.460 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.460 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:45.460 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216303+0000", 2024-09-21T15:50:45.461 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3273000000000003e-05 2024-09-21T15:50:45.461 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:45.461 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:45.461 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:45.461 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.461 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.461 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64037 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1771520~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:45.461 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:50:45.461 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 68.274241216999997, 2024-09-21T15:50:45.461 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048588771000000003, 2024-09-21T15:50:45.461 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:45.461 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:45.461 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:45.461 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:45.461 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:45.461 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64037 2024-09-21T15:50:45.461 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.462 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:45.462 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.462 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:45.462 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:50:45.462 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:45.462 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.462 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.462 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:45.462 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:50:45.462 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:45.462 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.462 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.462 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:45.462 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:50:45.462 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:45.463 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.463 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.463 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:45.463 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167878+0000", 2024-09-21T15:50:45.463 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3e-06 2024-09-21T15:50:45.463 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.463 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.463 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:45.463 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167884+0000", 2024-09-21T15:50:45.463 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3269999999999997e-06 2024-09-21T15:50:45.463 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.463 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.463 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:45.463 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167889+0000", 2024-09-21T15:50:45.463 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9269999999999996e-06 2024-09-21T15:50:45.463 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.463 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.464 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:45.464 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216331+0000", 2024-09-21T15:50:45.464 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048442102000000001 2024-09-21T15:50:45.464 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.464 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.464 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:45.464 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216393+0000", 2024-09-21T15:50:45.464 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1132999999999999e-05 2024-09-21T15:50:45.464 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.464 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.464 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:45.464 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216465+0000", 2024-09-21T15:50:45.464 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2339e-05 2024-09-21T15:50:45.464 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:45.464 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:45.464 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:45.465 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.465 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.465 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64038 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1779712~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:45.465 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:50:45.465 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 68.274223386000003, 2024-09-21T15:50:45.465 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048749795999999998, 2024-09-21T15:50:45.465 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:45.465 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:45.465 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:45.465 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:45.465 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:45.465 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64038 2024-09-21T15:50:45.465 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.465 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:45.465 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.466 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:45.466 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:50:45.466 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:45.466 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.466 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.466 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:45.466 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:50:45.466 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:45.466 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.466 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.466 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:45.466 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:50:45.466 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:45.466 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.466 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.467 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:45.467 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167896+0000", 2024-09-21T15:50:45.467 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2419999999999999e-06 2024-09-21T15:50:45.467 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.467 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.467 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:45.467 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167902+0000", 2024-09-21T15:50:45.467 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0859999999999998e-06 2024-09-21T15:50:45.467 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.467 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.467 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:45.467 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167907+0000", 2024-09-21T15:50:45.467 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9679999999999997e-06 2024-09-21T15:50:45.467 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.467 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.467 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:45.468 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216506+0000", 2024-09-21T15:50:45.468 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048599374000000001 2024-09-21T15:50:45.468 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.468 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.468 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:45.468 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216579+0000", 2024-09-21T15:50:45.468 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2801999999999999e-05 2024-09-21T15:50:45.468 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.468 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.468 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:45.468 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216644+0000", 2024-09-21T15:50:45.468 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4715000000000006e-05 2024-09-21T15:50:45.468 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:45.468 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:45.468 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:45.468 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.469 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.469 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64039 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1107968~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:45.469 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:50:45.469 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 68.274206031000006, 2024-09-21T15:50:45.469 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048911370000000003, 2024-09-21T15:50:45.469 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:45.469 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:45.469 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:45.469 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:45.469 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:45.469 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64039 2024-09-21T15:50:45.469 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.469 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:45.469 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.469 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:45.469 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:50:45.470 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:45.470 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.470 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.470 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:45.470 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:50:45.470 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:45.470 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.470 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.470 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:45.470 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:50:45.470 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:45.470 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.470 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.470 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:45.470 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167913+0000", 2024-09-21T15:50:45.470 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0839999999999999e-06 2024-09-21T15:50:45.470 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.471 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.471 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:45.471 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167921+0000", 2024-09-21T15:50:45.471 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8310000000000001e-06 2024-09-21T15:50:45.471 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.471 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.471 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:45.471 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167926+0000", 2024-09-21T15:50:45.471 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0270000000000003e-06 2024-09-21T15:50:45.471 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.471 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.471 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:45.471 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216680+0000", 2024-09-21T15:50:45.471 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048753890000000001 2024-09-21T15:50:45.471 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.471 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.472 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:45.472 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216757+0000", 2024-09-21T15:50:45.472 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6848999999999996e-05 2024-09-21T15:50:45.472 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.472 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.472 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:45.472 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216823+0000", 2024-09-21T15:50:45.472 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6068000000000005e-05 2024-09-21T15:50:45.472 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:45.472 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:45.472 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:45.472 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.472 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.472 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64040 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1787904~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:45.472 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167931+0000", 2024-09-21T15:50:45.472 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 68.274186799000006, 2024-09-21T15:50:45.472 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049010561000000001, 2024-09-21T15:50:45.473 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:45.473 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:45.473 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:45.473 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:45.473 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:45.473 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64040 2024-09-21T15:50:45.473 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.473 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:45.473 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.473 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:45.473 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167931+0000", 2024-09-21T15:50:45.473 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:45.473 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.473 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.473 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:45.473 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167931+0000", 2024-09-21T15:50:45.474 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:45.474 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.474 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.474 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:45.474 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167930+0000", 2024-09-21T15:50:45.474 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:45.474 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.474 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.474 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:45.474 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167932+0000", 2024-09-21T15:50:45.474 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.198e-06 2024-09-21T15:50:45.474 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.474 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.474 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:45.475 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167938+0000", 2024-09-21T15:50:45.475 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.105e-06 2024-09-21T15:50:45.475 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.475 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.475 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:45.475 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167943+0000", 2024-09-21T15:50:45.475 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9429999999999999e-06 2024-09-21T15:50:45.475 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.475 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.475 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:45.475 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216848+0000", 2024-09-21T15:50:45.475 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048905200000000003 2024-09-21T15:50:45.475 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.475 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.475 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:45.475 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216903+0000", 2024-09-21T15:50:45.476 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4793999999999997e-05 2024-09-21T15:50:45.476 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.476 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.476 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:45.476 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216941+0000", 2024-09-21T15:50:45.476 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7790999999999998e-05 2024-09-21T15:50:45.476 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:45.476 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:45.476 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:45.476 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.476 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.476 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64041 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1796096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:45.476 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:50:45.476 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 68.274169827999998, 2024-09-21T15:50:45.476 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049181457999999997, 2024-09-21T15:50:45.476 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:45.477 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:45.477 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:45.477 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:45.477 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:45.477 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64041 2024-09-21T15:50:45.477 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.477 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:45.477 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.477 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:45.477 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:50:45.477 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:45.477 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.477 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.477 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:45.477 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:50:45.477 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:45.477 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.478 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.478 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:45.478 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:50:45.478 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:45.478 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.478 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.478 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:45.478 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167949+0000", 2024-09-21T15:50:45.478 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0200000000000001e-06 2024-09-21T15:50:45.478 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.478 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.478 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:45.478 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167955+0000", 2024-09-21T15:50:45.478 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5990000000000001e-06 2024-09-21T15:50:45.478 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.478 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.479 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:45.479 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167959+0000", 2024-09-21T15:50:45.479 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5589999999999999e-06 2024-09-21T15:50:45.479 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.479 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.479 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:45.479 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216972+0000", 2024-09-21T15:50:45.479 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049012574000000003 2024-09-21T15:50:45.479 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.479 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.479 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:45.479 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217057+0000", 2024-09-21T15:50:45.479 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4820000000000004e-05 2024-09-21T15:50:45.479 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.479 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.479 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:45.479 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217129+0000", 2024-09-21T15:50:45.480 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2186000000000003e-05 2024-09-21T15:50:45.480 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:45.480 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:45.480 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:45.480 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.480 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.480 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64042 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1804288~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:45.480 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167964+0000", 2024-09-21T15:50:45.480 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 68.274153537000004, 2024-09-21T15:50:45.480 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049333372, 2024-09-21T15:50:45.480 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:45.480 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:45.480 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:45.480 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:45.480 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:45.481 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64042 2024-09-21T15:50:45.481 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.481 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:45.481 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.481 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:45.481 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167964+0000", 2024-09-21T15:50:45.481 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:45.481 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.481 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.481 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:45.481 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167964+0000", 2024-09-21T15:50:45.481 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:45.481 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.481 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.481 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:45.481 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167963+0000", 2024-09-21T15:50:45.481 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:45.482 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.482 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.482 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:45.482 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167966+0000", 2024-09-21T15:50:45.482 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1119999999999999e-06 2024-09-21T15:50:45.482 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.482 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.482 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:45.482 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167975+0000", 2024-09-21T15:50:45.482 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1130000000000007e-06 2024-09-21T15:50:45.482 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.482 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.482 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:45.482 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167980+0000", 2024-09-21T15:50:45.483 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9599999999999999e-06 2024-09-21T15:50:45.483 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.483 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.483 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:45.483 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217158+0000", 2024-09-21T15:50:45.483 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049178569999999998 2024-09-21T15:50:45.483 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.483 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.483 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:45.483 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217232+0000", 2024-09-21T15:50:45.483 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3589000000000003e-05 2024-09-21T15:50:45.483 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.483 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.483 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:45.483 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217297+0000", 2024-09-21T15:50:45.484 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5339999999999994e-05 2024-09-21T15:50:45.484 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:45.484 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:45.484 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:45.484 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.484 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.484 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64043 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1812480~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:45.484 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.168085+0000", 2024-09-21T15:50:45.484 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 68.274032469999995, 2024-09-21T15:50:45.484 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049365157999999999, 2024-09-21T15:50:45.484 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:45.484 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:45.484 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:45.484 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:45.484 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:45.484 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64043 2024-09-21T15:50:45.485 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.485 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:45.485 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.485 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:45.485 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168085+0000", 2024-09-21T15:50:45.485 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:45.485 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.485 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.485 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:45.485 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168085+0000", 2024-09-21T15:50:45.485 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:45.485 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.485 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.485 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:45.485 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168084+0000", 2024-09-21T15:50:45.485 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:45.486 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.486 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.486 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:45.486 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168088+0000", 2024-09-21T15:50:45.486 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5420000000000001e-06 2024-09-21T15:50:45.486 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.486 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.486 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:45.486 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168097+0000", 2024-09-21T15:50:45.486 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5410000000000006e-06 2024-09-21T15:50:45.486 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.486 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.486 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:45.486 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168104+0000", 2024-09-21T15:50:45.486 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6719999999999998e-06 2024-09-21T15:50:45.487 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.487 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.487 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:45.487 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217323+0000", 2024-09-21T15:50:45.487 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049218857999999997 2024-09-21T15:50:45.487 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.487 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.487 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:45.487 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217397+0000", 2024-09-21T15:50:45.487 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3932999999999998e-05 2024-09-21T15:50:45.487 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.487 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.487 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:45.487 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217450+0000", 2024-09-21T15:50:45.487 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3168000000000003e-05 2024-09-21T15:50:45.487 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:45.487 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:45.488 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:45.488 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.488 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.488 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155255 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 323584~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:45.488 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.440996+0000", 2024-09-21T15:50:45.488 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 23.001120943, 2024-09-21T15:50:45.488 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048233848000000003, 2024-09-21T15:50:45.488 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:45.488 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:45.488 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:45.488 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:45.488 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:45.488 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155255 2024-09-21T15:50:45.488 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.488 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:45.488 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.489 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:45.489 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.440996+0000", 2024-09-21T15:50:45.489 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:45.489 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.489 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.489 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:45.489 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.440996+0000", 2024-09-21T15:50:45.489 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:45.489 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.489 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.489 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:45.489 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.440996+0000", 2024-09-21T15:50:45.489 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:45.489 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.489 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.489 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:45.490 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441004+0000", 2024-09-21T15:50:45.490 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8460000000000008e-06 2024-09-21T15:50:45.490 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.490 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.490 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:45.490 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441018+0000", 2024-09-21T15:50:45.490 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.42e-05 2024-09-21T15:50:45.490 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.490 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.490 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:45.490 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441023+0000", 2024-09-21T15:50:45.490 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0019999999999997e-06 2024-09-21T15:50:45.490 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.490 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.490 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:45.490 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489060+0000", 2024-09-21T15:50:45.490 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048036660000000002 2024-09-21T15:50:45.491 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.491 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.491 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:45.491 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489107+0000", 2024-09-21T15:50:45.491 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7153000000000001e-05 2024-09-21T15:50:45.491 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.491 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.491 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:45.491 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489230+0000", 2024-09-21T15:50:45.491 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012328600000000001 2024-09-21T15:50:45.491 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:45.491 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:45.491 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:45.492 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.492 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.492 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155256 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 356352~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:45.492 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441027+0000", 2024-09-21T15:50:45.492 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 23.001089856, 2024-09-21T15:50:45.492 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048677606999999998, 2024-09-21T15:50:45.492 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:45.492 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:45.492 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:45.492 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:45.492 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:45.492 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155256 2024-09-21T15:50:45.492 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.492 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:45.492 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.493 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:45.493 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441027+0000", 2024-09-21T15:50:45.493 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:45.493 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.493 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.493 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:45.493 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441027+0000", 2024-09-21T15:50:45.493 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:45.493 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.493 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.493 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:45.493 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441027+0000", 2024-09-21T15:50:45.493 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:45.493 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.494 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.494 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:45.494 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441029+0000", 2024-09-21T15:50:45.494 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.39e-06 2024-09-21T15:50:45.494 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.494 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.494 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:45.494 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441036+0000", 2024-09-21T15:50:45.494 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1129999999999997e-06 2024-09-21T15:50:45.494 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.494 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.494 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:45.494 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441040+0000", 2024-09-21T15:50:45.494 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8459999999999999e-06 2024-09-21T15:50:45.495 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.495 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.495 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:45.495 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489244+0000", 2024-09-21T15:50:45.495 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048203751000000003 2024-09-21T15:50:45.495 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.495 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.495 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:45.495 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489288+0000", 2024-09-21T15:50:45.495 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4153999999999998e-05 2024-09-21T15:50:45.495 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.495 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.495 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:45.495 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489705+0000", 2024-09-21T15:50:45.495 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000416773 2024-09-21T15:50:45.495 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:45.496 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:45.496 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:45.496 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.496 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.496 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155257 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 389120~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:45.496 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441045+0000", 2024-09-21T15:50:45.496 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 23.001072765, 2024-09-21T15:50:45.496 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049000325999999997, 2024-09-21T15:50:45.496 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:45.496 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:45.496 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:45.496 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:45.496 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:45.496 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155257 2024-09-21T15:50:45.496 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.496 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:45.497 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.497 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:45.497 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441045+0000", 2024-09-21T15:50:45.497 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:45.497 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.497 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.497 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:45.497 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441045+0000", 2024-09-21T15:50:45.497 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:45.497 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.497 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.497 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:45.497 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441044+0000", 2024-09-21T15:50:45.497 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:45.497 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.497 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.497 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:45.498 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441046+0000", 2024-09-21T15:50:45.498 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1909999999999999e-06 2024-09-21T15:50:45.498 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.498 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.498 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:45.498 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441054+0000", 2024-09-21T15:50:45.498 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3900000000000004e-06 2024-09-21T15:50:45.498 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.498 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.498 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:45.498 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441058+0000", 2024-09-21T15:50:45.498 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4619999999999998e-06 2024-09-21T15:50:45.498 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.499 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.499 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:45.499 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489741+0000", 2024-09-21T15:50:45.499 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048682660000000003 2024-09-21T15:50:45.499 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.499 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.499 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:45.499 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489813+0000", 2024-09-21T15:50:45.499 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2051999999999994e-05 2024-09-21T15:50:45.499 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.499 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.499 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:45.499 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490045+0000", 2024-09-21T15:50:45.499 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00023191299999999999 2024-09-21T15:50:45.499 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:45.499 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:45.500 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:45.500 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.500 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.500 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155258 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 421888~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:45.500 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441062+0000", 2024-09-21T15:50:45.500 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 23.001055223000002, 2024-09-21T15:50:45.500 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049339651999999998, 2024-09-21T15:50:45.500 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:45.500 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:45.500 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:45.500 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:45.500 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:45.500 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155258 2024-09-21T15:50:45.500 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.500 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:45.500 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.500 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:45.501 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441062+0000", 2024-09-21T15:50:45.501 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:45.501 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.501 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.501 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:45.501 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441062+0000", 2024-09-21T15:50:45.501 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:45.501 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.501 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.501 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:45.501 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441062+0000", 2024-09-21T15:50:45.501 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:45.501 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.501 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.501 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:45.501 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441064+0000", 2024-09-21T15:50:45.502 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.967e-06 2024-09-21T15:50:45.502 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.502 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.502 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:45.502 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441069+0000", 2024-09-21T15:50:45.502 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1529999999999996e-06 2024-09-21T15:50:45.502 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.502 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.502 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:45.502 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441073+0000", 2024-09-21T15:50:45.502 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3850000000000004e-06 2024-09-21T15:50:45.502 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.502 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.502 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:45.502 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490082+0000", 2024-09-21T15:50:45.502 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.04900885 2024-09-21T15:50:45.502 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.503 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.503 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:45.503 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490166+0000", 2024-09-21T15:50:45.503 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3640999999999994e-05 2024-09-21T15:50:45.503 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.503 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.503 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:45.503 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490402+0000", 2024-09-21T15:50:45.503 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000235978 2024-09-21T15:50:45.503 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:45.503 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:45.503 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:45.503 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.503 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.503 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155259 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 454656~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:45.503 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441086+0000", 2024-09-21T15:50:45.504 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 23.001031418, 2024-09-21T15:50:45.504 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049624397000000001, 2024-09-21T15:50:45.504 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:45.504 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:45.504 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:45.504 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:45.504 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:45.504 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155259 2024-09-21T15:50:45.504 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.504 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:45.504 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.504 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:45.504 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441086+0000", 2024-09-21T15:50:45.505 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:45.505 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.505 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.505 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:45.505 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441086+0000", 2024-09-21T15:50:45.505 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:45.505 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.505 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.505 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:45.505 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441086+0000", 2024-09-21T15:50:45.505 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:45.505 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.505 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.505 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:45.506 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441088+0000", 2024-09-21T15:50:45.506 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2630000000000002e-06 2024-09-21T15:50:45.506 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.506 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.506 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:45.506 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441093+0000", 2024-09-21T15:50:45.506 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5010000000000004e-06 2024-09-21T15:50:45.506 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.506 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.506 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:45.506 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441098+0000", 2024-09-21T15:50:45.506 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.707e-06 2024-09-21T15:50:45.506 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.506 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.506 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:45.506 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490447+0000", 2024-09-21T15:50:45.507 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049349125000000001 2024-09-21T15:50:45.507 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.507 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.507 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:45.507 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490504+0000", 2024-09-21T15:50:45.507 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6521000000000003e-05 2024-09-21T15:50:45.507 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.507 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.507 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:45.507 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490710+0000", 2024-09-21T15:50:45.507 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00020663999999999999 2024-09-21T15:50:45.507 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:45.507 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:45.507 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:45.507 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.507 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.507 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155260 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 487424~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:45.508 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441160+0000", 2024-09-21T15:50:45.508 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 23.000957455999998, 2024-09-21T15:50:45.508 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049876810000000001, 2024-09-21T15:50:45.508 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:45.508 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:45.508 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:45.508 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:45.508 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:45.508 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155260 2024-09-21T15:50:45.508 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.508 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:45.508 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.508 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:45.508 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441160+0000", 2024-09-21T15:50:45.508 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:45.508 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.508 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.509 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:45.509 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441160+0000", 2024-09-21T15:50:45.509 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:45.509 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.509 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.509 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:45.509 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441159+0000", 2024-09-21T15:50:45.509 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:45.509 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.509 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.509 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:45.509 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441164+0000", 2024-09-21T15:50:45.509 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4059999999999998e-06 2024-09-21T15:50:45.509 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.509 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.509 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:45.510 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441175+0000", 2024-09-21T15:50:45.510 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1095e-05 2024-09-21T15:50:45.510 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.510 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.510 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:45.510 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441182+0000", 2024-09-21T15:50:45.510 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2139999999999999e-06 2024-09-21T15:50:45.510 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.510 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.510 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:45.510 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490751+0000", 2024-09-21T15:50:45.510 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049569230999999998 2024-09-21T15:50:45.510 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.510 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.510 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:45.511 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490837+0000", 2024-09-21T15:50:45.511 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6017e-05 2024-09-21T15:50:45.511 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.511 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.511 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:45.511 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491037+0000", 2024-09-21T15:50:45.511 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00019952300000000001 2024-09-21T15:50:45.511 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:45.511 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:45.511 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:45.511 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.511 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.511 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155261 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 520192~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:45.511 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441187+0000", 2024-09-21T15:50:45.511 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 23.000930273000002, 2024-09-21T15:50:45.511 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.050124931999999997, 2024-09-21T15:50:45.511 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:45.512 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:45.512 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:45.512 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:45.512 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:45.512 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155261 2024-09-21T15:50:45.512 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.512 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:45.512 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.512 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:45.512 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441187+0000", 2024-09-21T15:50:45.512 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:45.512 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.512 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.512 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:45.512 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441187+0000", 2024-09-21T15:50:45.512 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:45.513 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.513 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.513 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:45.513 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441187+0000", 2024-09-21T15:50:45.513 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:45.513 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.513 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.513 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:45.513 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441189+0000", 2024-09-21T15:50:45.513 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3649999999999998e-06 2024-09-21T15:50:45.513 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.513 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.513 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:45.513 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441196+0000", 2024-09-21T15:50:45.513 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5379999999999999e-06 2024-09-21T15:50:45.513 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.514 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.514 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:45.514 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441200+0000", 2024-09-21T15:50:45.514 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4719999999999997e-06 2024-09-21T15:50:45.514 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.514 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.514 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:45.514 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491070+0000", 2024-09-21T15:50:45.514 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049869915000000001 2024-09-21T15:50:45.514 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.514 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.514 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:45.515 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491146+0000", 2024-09-21T15:50:45.515 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6589999999999997e-05 2024-09-21T15:50:45.515 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.515 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.515 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:45.515 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491312+0000", 2024-09-21T15:50:45.515 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000165475 2024-09-21T15:50:45.515 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:45.515 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:45.515 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:45.515 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.515 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.515 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155308 3.11 3:8b5183e5:::rbd_data.11335aef9442.0000000000000000:2a [read 356352~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:45.515 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441369+0000", 2024-09-21T15:50:45.516 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 23.000748425000001, 2024-09-21T15:50:45.516 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.050090672000000003, 2024-09-21T15:50:45.516 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:45.516 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:45.516 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:45.516 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:45.516 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:45.516 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155308 2024-09-21T15:50:45.516 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.516 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:45.516 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.516 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:45.516 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441369+0000", 2024-09-21T15:50:45.516 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:45.516 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.517 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.517 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:45.517 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441369+0000", 2024-09-21T15:50:45.517 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:45.517 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.517 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.517 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:45.517 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441368+0000", 2024-09-21T15:50:45.517 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:45.517 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.517 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.517 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:45.517 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441371+0000", 2024-09-21T15:50:45.517 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1100000000000001e-06 2024-09-21T15:50:45.517 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.517 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.518 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:45.518 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441377+0000", 2024-09-21T15:50:45.518 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5649999999999998e-06 2024-09-21T15:50:45.518 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.518 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.518 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:45.518 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441382+0000", 2024-09-21T15:50:45.518 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5929999999999999e-06 2024-09-21T15:50:45.518 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.518 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.518 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:45.518 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491349+0000", 2024-09-21T15:50:45.518 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049967122000000003 2024-09-21T15:50:45.518 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.518 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.519 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:45.519 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491400+0000", 2024-09-21T15:50:45.519 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1589999999999999e-05 2024-09-21T15:50:45.519 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.519 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.519 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:45.519 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491460+0000", 2024-09-21T15:50:45.519 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9070000000000002e-05 2024-09-21T15:50:45.519 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:45.519 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:45.519 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:45.519 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.519 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.519 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155309 3.11 3:8b5183e5:::rbd_data.11335aef9442.0000000000000000:2a [read 389120~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:45.519 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441386+0000", 2024-09-21T15:50:45.519 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 23.000731076000001, 2024-09-21T15:50:45.519 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.050184777, 2024-09-21T15:50:45.520 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:45.520 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:45.520 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:45.520 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:45.520 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:45.520 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155309 2024-09-21T15:50:45.520 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.520 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:45.520 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.520 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:45.520 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441386+0000", 2024-09-21T15:50:45.520 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:45.520 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.520 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.520 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:45.520 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441386+0000", 2024-09-21T15:50:45.520 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:45.521 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.521 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.521 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:45.521 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441386+0000", 2024-09-21T15:50:45.521 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:45.521 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.521 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.521 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:45.521 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441388+0000", 2024-09-21T15:50:45.521 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4420000000000001e-06 2024-09-21T15:50:45.521 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.521 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.521 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:45.521 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441395+0000", 2024-09-21T15:50:45.521 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6189999999999996e-06 2024-09-21T15:50:45.521 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.522 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.522 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:45.522 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441400+0000", 2024-09-21T15:50:45.522 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.118e-06 2024-09-21T15:50:45.522 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.522 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.522 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:45.522 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491473+0000", 2024-09-21T15:50:45.522 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.050072783000000003 2024-09-21T15:50:45.522 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.522 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.522 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:45.522 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491518+0000", 2024-09-21T15:50:45.522 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4854000000000001e-05 2024-09-21T15:50:45.522 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:45.522 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:45.523 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:45.523 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491571+0000", 2024-09-21T15:50:45.523 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3358e-05 2024-09-21T15:50:45.523 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:45.523 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:45.523 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:45.523 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:45.523 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:45.523 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:45.524 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:50:45.524 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:45.523+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:45.625 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:45.624+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:45.629 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:45.629 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:45.629 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:50:45.629 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:45.637 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:50:45.726 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:45.725+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:45.738 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is set 2024-09-21T15:50:45.741 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:45.741 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:45.741 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:50:45.741 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:50:45.741 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:45.752 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:50:45.836 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:45.835+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:45.905 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:45.905 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:50:45.905 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:50:45.905 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:50:45.905 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.905 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160505 3.b 3:d297695c:::rbd_header.11338bae974a:head [watch reconnect cookie 140381478598448 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-21T15:50:45.905 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:29.138810+0000", 2024-09-21T15:50:45.906 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 16.764120212999998, 2024-09-21T15:50:45.906 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0172598319999999, 2024-09-21T15:50:45.906 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:45.906 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:45.906 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:45.906 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:45.906 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:45.906 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160505 2024-09-21T15:50:45.906 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.906 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:45.906 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.906 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:45.906 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138810+0000", 2024-09-21T15:50:45.907 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:45.907 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.907 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.907 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:45.907 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138810+0000", 2024-09-21T15:50:45.907 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:45.907 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.907 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.907 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:45.907 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138808+0000", 2024-09-21T15:50:45.907 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:50:45.907 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.907 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.907 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:45.908 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138815+0000", 2024-09-21T15:50:45.908 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6549999999999998e-06 2024-09-21T15:50:45.908 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.908 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.908 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:45.908 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138825+0000", 2024-09-21T15:50:45.908 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9790000000000005e-06 2024-09-21T15:50:45.908 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.908 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.908 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:45.908 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138834+0000", 2024-09-21T15:50:45.908 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5179999999999993e-06 2024-09-21T15:50:45.908 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.908 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.908 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:45.909 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.140973+0000", 2024-09-21T15:50:45.909 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0021392049999999999 2024-09-21T15:50:45.909 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.909 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.909 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:50:45.909 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.140981+0000", 2024-09-21T15:50:45.909 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8909999999999995e-06 2024-09-21T15:50:45.909 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.909 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.909 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:45.909 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.150457+0000", 2024-09-21T15:50:45.909 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.009475645 2024-09-21T15:50:45.909 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.909 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.909 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for missing object", 2024-09-21T15:50:45.910 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.150589+0000", 2024-09-21T15:50:45.910 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000131779 2024-09-21T15:50:45.910 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.910 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.910 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:45.910 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.151723+0000", 2024-09-21T15:50:45.910 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0011345979999999999 2024-09-21T15:50:45.910 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.910 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.910 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-09-21T15:50:45.910 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.151872+0000", 2024-09-21T15:50:45.910 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000149021 2024-09-21T15:50:45.910 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.911 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.911 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:45.911 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.152563+0000", 2024-09-21T15:50:45.911 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00069051500000000005 2024-09-21T15:50:45.911 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.911 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.911 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:45.911 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.152673+0000", 2024-09-21T15:50:45.911 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000109795 2024-09-21T15:50:45.911 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.911 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.911 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:45.911 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.156069+0000", 2024-09-21T15:50:45.911 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003396736 2024-09-21T15:50:45.911 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:45.912 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:45.912 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:45.912 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.912 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.912 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160506 3.b 3:d297695c:::rbd_header.11338bae974a:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-21T15:50:45.912 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:29.138844+0000", 2024-09-21T15:50:45.912 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 16.764085326, 2024-09-21T15:50:45.912 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0174592680000001, 2024-09-21T15:50:45.912 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:45.912 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:45.912 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:45.912 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:45.912 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:45.912 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160506 2024-09-21T15:50:45.912 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.913 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:45.913 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.913 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:45.913 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138844+0000", 2024-09-21T15:50:45.913 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:45.913 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.913 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.913 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:45.913 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138844+0000", 2024-09-21T15:50:45.913 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:45.913 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.913 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.913 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:45.913 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138844+0000", 2024-09-21T15:50:45.913 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:45.914 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.914 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.914 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:45.914 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138848+0000", 2024-09-21T15:50:45.914 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.76e-06 2024-09-21T15:50:45.914 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.914 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.914 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:45.914 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138854+0000", 2024-09-21T15:50:45.914 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3890000000000002e-06 2024-09-21T15:50:45.914 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.914 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.914 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:45.914 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138860+0000", 2024-09-21T15:50:45.915 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6409999999999999e-06 2024-09-21T15:50:45.915 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.915 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.915 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:45.915 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.141015+0000", 2024-09-21T15:50:45.915 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0021553779999999999 2024-09-21T15:50:45.915 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.915 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.915 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:50:45.915 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.141021+0000", 2024-09-21T15:50:45.915 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7629999999999997e-06 2024-09-21T15:50:45.915 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.915 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.915 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:45.915 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.150858+0000", 2024-09-21T15:50:45.916 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0098372200000001 2024-09-21T15:50:45.916 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.916 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.916 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for missing object", 2024-09-21T15:50:45.916 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.150899+0000", 2024-09-21T15:50:45.916 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1112000000000002e-05 2024-09-21T15:50:45.916 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.916 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.916 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:45.916 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.152069+0000", 2024-09-21T15:50:45.916 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001170307 2024-09-21T15:50:45.916 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.916 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.916 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-09-21T15:50:45.916 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.152192+0000", 2024-09-21T15:50:45.917 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012289700000000001 2024-09-21T15:50:45.917 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.917 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.917 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:45.917 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.153362+0000", 2024-09-21T15:50:45.917 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0011700549999999999 2024-09-21T15:50:45.917 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.917 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.917 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:45.917 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.153457+0000", 2024-09-21T15:50:45.917 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4687999999999999e-05 2024-09-21T15:50:45.917 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.917 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.917 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:50:45.918 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.153893+0000", 2024-09-21T15:50:45.918 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000436363 2024-09-21T15:50:45.918 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.918 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.918 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:45.918 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.156134+0000", 2024-09-21T15:50:45.918 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0022410009999999998 2024-09-21T15:50:45.918 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.918 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.918 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:45.918 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.156148+0000", 2024-09-21T15:50:45.918 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3574000000000001e-05 2024-09-21T15:50:45.918 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.919 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.919 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:45.919 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.156214+0000", 2024-09-21T15:50:45.919 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6438000000000003e-05 2024-09-21T15:50:45.919 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.919 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.919 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:45.919 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.156304+0000", 2024-09-21T15:50:45.919 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9278999999999997e-05 2024-09-21T15:50:45.919 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:45.919 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:45.919 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:45.919 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.919 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.920 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160512 3.b 3:d297695c:::rbd_header.11338bae974a:head [watch unwatch cookie 140381478598448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T15:50:45.920 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:30.161362+0000", 2024-09-21T15:50:45.920 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 15.741567518, 2024-09-21T15:50:45.920 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0074105029999999997, 2024-09-21T15:50:45.920 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:45.920 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:45.920 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:45.920 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:45.920 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:45.920 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160512 2024-09-21T15:50:45.920 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.920 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:45.920 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.920 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:45.920 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161362+0000", 2024-09-21T15:50:45.921 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:45.921 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.921 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.921 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:45.921 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161362+0000", 2024-09-21T15:50:45.921 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:45.921 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.921 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.921 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:45.921 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161362+0000", 2024-09-21T15:50:45.921 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:45.921 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.921 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.921 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:45.921 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161365+0000", 2024-09-21T15:50:45.922 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8669999999999998e-06 2024-09-21T15:50:45.922 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.922 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.922 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:45.922 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161371+0000", 2024-09-21T15:50:45.922 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8009999999999998e-06 2024-09-21T15:50:45.922 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.922 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.922 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:45.922 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161376+0000", 2024-09-21T15:50:45.922 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0340000000000004e-06 2024-09-21T15:50:45.922 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.922 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.922 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:45.923 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161762+0000", 2024-09-21T15:50:45.923 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038575099999999999 2024-09-21T15:50:45.923 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.923 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.923 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:45.923 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161838+0000", 2024-09-21T15:50:45.923 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5697000000000004e-05 2024-09-21T15:50:45.923 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.923 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.923 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:50:45.923 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161952+0000", 2024-09-21T15:50:45.923 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000114034 2024-09-21T15:50:45.923 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.923 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.923 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:45.923 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.163727+0000", 2024-09-21T15:50:45.924 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0017749560000000001 2024-09-21T15:50:45.924 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.924 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.924 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:45.924 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168545+0000", 2024-09-21T15:50:45.924 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0048186080000000003 2024-09-21T15:50:45.924 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.924 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.924 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:45.924 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168593+0000", 2024-09-21T15:50:45.924 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7549999999999997e-05 2024-09-21T15:50:45.924 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.924 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.924 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:45.925 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168773+0000", 2024-09-21T15:50:45.925 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00017976600000000001 2024-09-21T15:50:45.925 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:45.925 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:45.925 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:45.925 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.925 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.925 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160513 3.b 3:d297695c:::rbd_header.11338bae974a:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-09-21T15:50:45.925 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:30.161755+0000", 2024-09-21T15:50:45.925 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 15.741174587, 2024-09-21T15:50:45.925 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0072250300000000003, 2024-09-21T15:50:45.925 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:45.925 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:45.925 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:45.925 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:45.926 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:45.926 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160513 2024-09-21T15:50:45.926 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.926 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:45.926 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.926 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:45.926 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161755+0000", 2024-09-21T15:50:45.926 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:45.926 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.926 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.926 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:45.926 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161755+0000", 2024-09-21T15:50:45.926 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:45.926 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.926 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.927 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:45.927 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161754+0000", 2024-09-21T15:50:45.927 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:45.927 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.927 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.927 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:45.927 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161759+0000", 2024-09-21T15:50:45.927 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9699999999999998e-06 2024-09-21T15:50:45.927 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.927 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.927 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:45.927 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161768+0000", 2024-09-21T15:50:45.927 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9949999999999994e-06 2024-09-21T15:50:45.927 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.927 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.927 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:45.928 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161774+0000", 2024-09-21T15:50:45.928 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1290000000000001e-06 2024-09-21T15:50:45.928 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.928 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.928 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:45.928 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.162239+0000", 2024-09-21T15:50:45.928 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00046406600000000001 2024-09-21T15:50:45.928 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.928 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.928 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-09-21T15:50:45.928 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.162296+0000", 2024-09-21T15:50:45.928 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7122e-05 2024-09-21T15:50:45.928 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.928 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.928 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:45.928 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168802+0000", 2024-09-21T15:50:45.929 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0065061700000000004 2024-09-21T15:50:45.929 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.929 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.929 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:45.929 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168877+0000", 2024-09-21T15:50:45.929 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5214000000000002e-05 2024-09-21T15:50:45.929 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.929 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.929 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:45.929 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168980+0000", 2024-09-21T15:50:45.929 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000103182 2024-09-21T15:50:45.929 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:45.929 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:45.929 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:45.929 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.929 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.930 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160527 3.b 3:d297695c:::rbd_header.11338bae974a:head [watch watch cookie 140381476148256] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T15:50:45.930 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:30.182756+0000", 2024-09-21T15:50:45.930 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 15.720174006000001, 2024-09-21T15:50:45.930 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0054202579999999998, 2024-09-21T15:50:45.930 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:45.930 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:45.930 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:45.930 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:45.930 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:45.930 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160527 2024-09-21T15:50:45.930 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.930 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:45.930 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.930 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:45.930 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182756+0000", 2024-09-21T15:50:45.931 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:45.931 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.931 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.931 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:45.931 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182756+0000", 2024-09-21T15:50:45.931 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:45.931 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.931 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.931 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:45.931 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182755+0000", 2024-09-21T15:50:45.931 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:45.931 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.931 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.931 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:45.931 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182760+0000", 2024-09-21T15:50:45.931 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4269999999999999e-06 2024-09-21T15:50:45.932 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.932 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.932 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:45.932 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182771+0000", 2024-09-21T15:50:45.932 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0312999999999999e-05 2024-09-21T15:50:45.932 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.932 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.932 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:45.932 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182778+0000", 2024-09-21T15:50:45.932 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5700000000000004e-06 2024-09-21T15:50:45.932 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.932 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.932 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:45.932 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.184745+0000", 2024-09-21T15:50:45.932 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0019667170000000002 2024-09-21T15:50:45.933 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.933 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.933 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:45.933 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.184867+0000", 2024-09-21T15:50:45.933 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000122504 2024-09-21T15:50:45.933 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.933 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.933 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:50:45.933 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.185135+0000", 2024-09-21T15:50:45.933 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000267135 2024-09-21T15:50:45.933 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.933 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.933 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:45.933 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.187721+0000", 2024-09-21T15:50:45.933 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002586891 2024-09-21T15:50:45.933 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.934 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.934 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:45.934 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.187874+0000", 2024-09-21T15:50:45.934 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000152726 2024-09-21T15:50:45.934 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.934 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.934 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:45.934 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.187951+0000", 2024-09-21T15:50:45.934 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7132000000000006e-05 2024-09-21T15:50:45.934 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.934 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.934 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:45.934 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.188176+0000", 2024-09-21T15:50:45.934 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00022477900000000001 2024-09-21T15:50:45.935 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:45.935 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:45.935 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:45.935 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.935 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.935 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160586 3.b 3:d297695c:::rbd_header.11338bae974a:head [watch watch cookie 140381476029296] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-21T15:50:45.935 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.201587+0000", 2024-09-21T15:50:45.935 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.701342646000001, 2024-09-21T15:50:45.935 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0040745900000000003, 2024-09-21T15:50:45.935 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:45.935 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:45.935 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:45.935 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:45.935 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:45.936 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160586 2024-09-21T15:50:45.936 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.936 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:45.936 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.936 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:45.936 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.201587+0000", 2024-09-21T15:50:45.936 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:45.936 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.936 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.936 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:45.936 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.201587+0000", 2024-09-21T15:50:45.936 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:45.936 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.936 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.936 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:45.936 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.201586+0000", 2024-09-21T15:50:45.936 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:45.937 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.937 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.937 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:45.937 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.201592+0000", 2024-09-21T15:50:45.937 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7910000000000001e-06 2024-09-21T15:50:45.937 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.937 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.937 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:45.937 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.201603+0000", 2024-09-21T15:50:45.937 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0736e-05 2024-09-21T15:50:45.937 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.937 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.937 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:45.937 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.201609+0000", 2024-09-21T15:50:45.937 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9739999999999999e-06 2024-09-21T15:50:45.937 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.938 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.938 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:45.938 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.201643+0000", 2024-09-21T15:50:45.938 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4572e-05 2024-09-21T15:50:45.938 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.938 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.938 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:45.938 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.201726+0000", 2024-09-21T15:50:45.938 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.2763999999999996e-05 2024-09-21T15:50:45.938 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.938 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.938 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:50:45.938 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.201832+0000", 2024-09-21T15:50:45.938 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105931 2024-09-21T15:50:45.938 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.939 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.939 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:45.939 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.202743+0000", 2024-09-21T15:50:45.939 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00091128699999999999 2024-09-21T15:50:45.939 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.939 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.939 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:45.939 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.205540+0000", 2024-09-21T15:50:45.939 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0027967199999999999 2024-09-21T15:50:45.939 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.939 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.939 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:45.939 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.205598+0000", 2024-09-21T15:50:45.939 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7513999999999999e-05 2024-09-21T15:50:45.939 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.939 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.940 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:45.940 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.205662+0000", 2024-09-21T15:50:45.940 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4148999999999999e-05 2024-09-21T15:50:45.940 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:45.940 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:45.940 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:45.940 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.940 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.940 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160605 3.b 3:d1863919:::rbd_data.11338bae974a.0000000000000008:head [delete] snapc 2d=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-21T15:50:45.940 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.213794+0000", 2024-09-21T15:50:45.940 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.689136125999999, 2024-09-21T15:50:45.940 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046152490000000001, 2024-09-21T15:50:45.940 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:45.940 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:45.940 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:45.940 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:45.941 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:45.941 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160605 2024-09-21T15:50:45.941 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.941 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:45.941 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.941 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:45.941 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.213794+0000", 2024-09-21T15:50:45.941 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:45.941 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.941 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.941 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:45.941 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.213794+0000", 2024-09-21T15:50:45.941 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:45.941 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.941 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.941 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:45.942 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.213793+0000", 2024-09-21T15:50:45.942 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:45.942 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.942 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.942 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:45.942 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.213799+0000", 2024-09-21T15:50:45.942 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9959999999999999e-06 2024-09-21T15:50:45.942 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.942 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.942 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:45.942 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.213815+0000", 2024-09-21T15:50:45.942 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5880999999999999e-05 2024-09-21T15:50:45.942 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.942 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.942 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:45.942 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.213838+0000", 2024-09-21T15:50:45.943 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3671000000000001e-05 2024-09-21T15:50:45.943 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.943 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.943 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:45.943 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.213887+0000", 2024-09-21T15:50:45.943 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8865000000000001e-05 2024-09-21T15:50:45.943 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.943 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.943 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:45.943 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.213962+0000", 2024-09-21T15:50:45.943 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4592999999999995e-05 2024-09-21T15:50:45.943 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.943 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.943 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:50:45.943 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.214088+0000", 2024-09-21T15:50:45.944 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012666199999999999 2024-09-21T15:50:45.944 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.944 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.944 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:45.944 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216917+0000", 2024-09-21T15:50:45.944 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0028284059999999999 2024-09-21T15:50:45.944 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.944 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.944 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:45.944 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.218319+0000", 2024-09-21T15:50:45.944 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001402613 2024-09-21T15:50:45.944 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.944 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.944 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:45.944 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.218372+0000", 2024-09-21T15:50:45.945 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2636000000000002e-05 2024-09-21T15:50:45.945 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.945 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.945 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:45.945 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.218409+0000", 2024-09-21T15:50:45.945 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6942e-05 2024-09-21T15:50:45.945 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:45.945 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:45.945 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:45.945 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.945 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.945 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:160602 3.12 e95/94 3:49a61a92:::rbd_data.11338bae974a.0000000000000005:head v 95'344, mlcod=0'0)", 2024-09-21T15:50:45.945 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.215034+0000", 2024-09-21T15:50:45.945 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.687895708999999, 2024-09-21T15:50:45.945 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045544870000000003, 2024-09-21T15:50:45.946 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:45.946 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:45.946 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:45.946 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.946 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:45.946 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215034+0000", 2024-09-21T15:50:45.946 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:45.946 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.946 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.946 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:45.946 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215034+0000", 2024-09-21T15:50:45.946 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:45.946 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.946 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.946 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:45.946 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215034+0000", 2024-09-21T15:50:45.947 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:45.947 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.947 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.947 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:45.947 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215039+0000", 2024-09-21T15:50:45.947 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1549999999999998e-06 2024-09-21T15:50:45.947 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.947 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.947 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:45.947 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215050+0000", 2024-09-21T15:50:45.947 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1392e-05 2024-09-21T15:50:45.947 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.947 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.947 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:45.948 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215057+0000", 2024-09-21T15:50:45.948 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7399999999999998e-06 2024-09-21T15:50:45.948 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.948 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.948 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:45.948 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215187+0000", 2024-09-21T15:50:45.948 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013010100000000001 2024-09-21T15:50:45.948 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.948 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.948 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:45.948 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215208+0000", 2024-09-21T15:50:45.948 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1013999999999999e-05 2024-09-21T15:50:45.948 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.948 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.948 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:45.948 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.219549+0000", 2024-09-21T15:50:45.949 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0043404990000000003 2024-09-21T15:50:45.949 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.949 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.949 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:45.949 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.219589+0000", 2024-09-21T15:50:45.949 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9878e-05 2024-09-21T15:50:45.949 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:45.949 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:45.949 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:45.949 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.949 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.949 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:160597 3.1 e95/94 3:84154dfc:::rbd_data.11338bae974a.0000000000000000:head v 95'451, mlcod=0'0)", 2024-09-21T15:50:45.949 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.215080+0000", 2024-09-21T15:50:45.949 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.687850253000001, 2024-09-21T15:50:45.949 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045596819999999998, 2024-09-21T15:50:45.950 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:45.950 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:45.950 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:45.950 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.950 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:45.950 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215080+0000", 2024-09-21T15:50:45.950 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:45.950 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.950 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.950 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:45.950 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215080+0000", 2024-09-21T15:50:45.950 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:45.950 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.950 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.950 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:45.951 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215079+0000", 2024-09-21T15:50:45.951 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:45.951 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.951 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.951 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:45.951 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215083+0000", 2024-09-21T15:50:45.951 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2570000000000001e-06 2024-09-21T15:50:45.951 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.951 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.951 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:45.951 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215091+0000", 2024-09-21T15:50:45.951 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7829999999999999e-06 2024-09-21T15:50:45.951 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.951 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.951 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:45.952 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215096+0000", 2024-09-21T15:50:45.952 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0370000000000002e-06 2024-09-21T15:50:45.952 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.952 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.952 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:45.952 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216055+0000", 2024-09-21T15:50:45.952 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00095892299999999998 2024-09-21T15:50:45.952 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.952 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.952 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:45.952 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216078+0000", 2024-09-21T15:50:45.952 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2376000000000001e-05 2024-09-21T15:50:45.952 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.952 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.953 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:45.953 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.219598+0000", 2024-09-21T15:50:45.953 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035209299999999998 2024-09-21T15:50:45.953 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.953 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.953 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:45.953 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.219639+0000", 2024-09-21T15:50:45.953 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0766000000000002e-05 2024-09-21T15:50:45.953 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:45.953 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:45.953 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:45.953 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.953 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.953 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:160601 3.17 e95/94 3:e9d872b2:::rbd_data.11338bae974a.0000000000000004:head v 95'912, mlcod=0'0)", 2024-09-21T15:50:45.954 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.215668+0000", 2024-09-21T15:50:45.954 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.687261636000001, 2024-09-21T15:50:45.954 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0038923640000000002, 2024-09-21T15:50:45.954 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:45.954 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:45.954 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:45.954 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.954 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:45.954 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215668+0000", 2024-09-21T15:50:45.954 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:45.954 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.954 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.954 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:45.954 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215668+0000", 2024-09-21T15:50:45.954 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:45.955 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.955 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.955 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:45.955 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215668+0000", 2024-09-21T15:50:45.955 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:45.955 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.955 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.955 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:45.955 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215672+0000", 2024-09-21T15:50:45.955 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7680000000000002e-06 2024-09-21T15:50:45.955 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.955 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.955 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:45.955 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215682+0000", 2024-09-21T15:50:45.956 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0774000000000001e-05 2024-09-21T15:50:45.956 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.956 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.956 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:45.956 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215689+0000", 2024-09-21T15:50:45.956 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4670000000000001e-06 2024-09-21T15:50:45.956 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.956 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.956 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:45.956 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215740+0000", 2024-09-21T15:50:45.956 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1100000000000002e-05 2024-09-21T15:50:45.956 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.956 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.956 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:45.957 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215761+0000", 2024-09-21T15:50:45.957 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1112e-05 2024-09-21T15:50:45.957 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.957 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.957 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:45.957 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.219520+0000", 2024-09-21T15:50:45.957 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0037590739999999998 2024-09-21T15:50:45.957 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.957 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.957 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:45.957 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.219561+0000", 2024-09-21T15:50:45.957 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0329e-05 2024-09-21T15:50:45.957 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:45.957 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:45.958 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:45.958 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.958 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.958 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:160608 3.12 e95/94 3:48a47e1c:::rbd_data.11338bae974a.000000000000000b:head v 95'345, mlcod=0'0)", 2024-09-21T15:50:45.958 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.216934+0000", 2024-09-21T15:50:45.958 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.685995867999999, 2024-09-21T15:50:45.958 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053335759999999996, 2024-09-21T15:50:45.958 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:45.958 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:45.958 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:45.958 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.958 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:45.958 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216934+0000", 2024-09-21T15:50:45.959 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:45.959 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.959 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.959 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:45.959 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216934+0000", 2024-09-21T15:50:45.959 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:45.959 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.959 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.959 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:45.959 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216933+0000", 2024-09-21T15:50:45.959 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:45.959 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.959 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.959 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:45.959 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216939+0000", 2024-09-21T15:50:45.959 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6459999999999998e-06 2024-09-21T15:50:45.960 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.960 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.960 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:45.960 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216947+0000", 2024-09-21T15:50:45.960 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.2479999999999996e-06 2024-09-21T15:50:45.960 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.960 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.960 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:45.960 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216951+0000", 2024-09-21T15:50:45.960 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3760000000000001e-06 2024-09-21T15:50:45.960 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.960 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.960 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:45.960 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216998+0000", 2024-09-21T15:50:45.961 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6244e-05 2024-09-21T15:50:45.961 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.961 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.961 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:45.961 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.217019+0000", 2024-09-21T15:50:45.961 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1191000000000001e-05 2024-09-21T15:50:45.961 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.961 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.961 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:45.961 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.222230+0000", 2024-09-21T15:50:45.961 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052114350000000004 2024-09-21T15:50:45.961 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.961 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.961 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:45.961 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.222267+0000", 2024-09-21T15:50:45.962 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7169999999999998e-05 2024-09-21T15:50:45.962 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:45.962 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:45.962 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:45.962 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.962 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.962 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160618 3.b 3:d297695c:::rbd_header.11338bae974a:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-21T15:50:45.962 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.233457+0000", 2024-09-21T15:50:45.962 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.669473031000001, 2024-09-21T15:50:45.962 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.005085198, 2024-09-21T15:50:45.962 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:45.962 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:45.962 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:45.962 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:45.962 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:45.963 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160618 2024-09-21T15:50:45.963 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.963 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:45.963 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.963 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:45.963 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233457+0000", 2024-09-21T15:50:45.963 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:45.963 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.963 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.963 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:45.963 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233457+0000", 2024-09-21T15:50:45.963 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:45.963 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.963 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.963 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:45.964 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233456+0000", 2024-09-21T15:50:45.964 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:45.964 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.964 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.964 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:45.964 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233463+0000", 2024-09-21T15:50:45.964 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9310000000000002e-06 2024-09-21T15:50:45.964 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.964 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.964 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:45.964 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233473+0000", 2024-09-21T15:50:45.964 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0519e-05 2024-09-21T15:50:45.964 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.965 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.965 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:45.965 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233481+0000", 2024-09-21T15:50:45.965 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4009999999999997e-06 2024-09-21T15:50:45.965 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.965 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.965 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:45.965 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233513+0000", 2024-09-21T15:50:45.965 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1328999999999998e-05 2024-09-21T15:50:45.965 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.965 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.965 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:45.965 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233568+0000", 2024-09-21T15:50:45.965 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5606999999999998e-05 2024-09-21T15:50:45.965 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.965 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.965 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:50:45.966 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233670+0000", 2024-09-21T15:50:45.966 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010180599999999999 2024-09-21T15:50:45.966 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.966 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.966 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:45.966 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.234761+0000", 2024-09-21T15:50:45.966 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010905909999999999 2024-09-21T15:50:45.966 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.966 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.966 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:45.966 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.238442+0000", 2024-09-21T15:50:45.966 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0036809429999999999 2024-09-21T15:50:45.966 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.966 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.966 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:45.967 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.238498+0000", 2024-09-21T15:50:45.967 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6116999999999998e-05 2024-09-21T15:50:45.967 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.967 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.967 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:45.967 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.238542+0000", 2024-09-21T15:50:45.967 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4134000000000001e-05 2024-09-21T15:50:45.967 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:45.967 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:45.967 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:45.967 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.967 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.967 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160662 3.b 3:d7701915:::rbd_object_map.1133fd7483d4.000000000000002c:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-21T15:50:45.967 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.279369+0000", 2024-09-21T15:50:45.967 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.623561101, 2024-09-21T15:50:45.967 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0056948040000000004, 2024-09-21T15:50:45.968 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:45.968 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:45.968 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:45.968 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:45.968 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:45.968 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160662 2024-09-21T15:50:45.968 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.968 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:45.968 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.968 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:45.968 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279369+0000", 2024-09-21T15:50:45.968 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:45.968 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.968 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.968 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:45.968 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279369+0000", 2024-09-21T15:50:45.969 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:45.969 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.969 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.969 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:45.969 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279368+0000", 2024-09-21T15:50:45.969 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:45.969 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.969 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.969 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:45.969 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279373+0000", 2024-09-21T15:50:45.969 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6529999999999999e-06 2024-09-21T15:50:45.969 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.969 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.969 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:45.969 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279384+0000", 2024-09-21T15:50:45.969 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0256e-05 2024-09-21T15:50:45.970 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.970 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.970 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:45.970 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279388+0000", 2024-09-21T15:50:45.970 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5630000000000002e-06 2024-09-21T15:50:45.970 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.970 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.970 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:45.970 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279424+0000", 2024-09-21T15:50:45.970 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5985999999999997e-05 2024-09-21T15:50:45.970 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.970 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.970 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:45.970 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279478+0000", 2024-09-21T15:50:45.970 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3944999999999999e-05 2024-09-21T15:50:45.970 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.971 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.971 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:50:45.971 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279575+0000", 2024-09-21T15:50:45.971 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6817999999999994e-05 2024-09-21T15:50:45.971 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.971 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.971 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:45.971 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.282391+0000", 2024-09-21T15:50:45.971 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0028156700000000002 2024-09-21T15:50:45.971 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.971 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.971 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:45.971 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.284977+0000", 2024-09-21T15:50:45.971 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0025859640000000001 2024-09-21T15:50:45.972 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.972 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.972 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:45.972 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.285021+0000", 2024-09-21T15:50:45.972 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4415000000000001e-05 2024-09-21T15:50:45.972 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.972 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.972 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:45.972 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.285063+0000", 2024-09-21T15:50:45.972 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2447000000000002e-05 2024-09-21T15:50:45.972 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:45.972 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:45.972 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:45.972 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.972 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.972 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:161118 3.1 3:82ccd16f:::rbd_header.11335aef9442:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-21T15:50:45.972 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:36.688929+0000", 2024-09-21T15:50:45.973 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.2140008430000009, 2024-09-21T15:50:45.973 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0040062300000000004, 2024-09-21T15:50:45.973 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:45.973 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:45.973 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:45.973 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:45.973 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:45.973 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 161118 2024-09-21T15:50:45.973 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.973 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:45.973 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.973 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:45.973 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.688929+0000", 2024-09-21T15:50:45.973 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:45.973 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.973 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.973 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:45.974 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.688929+0000", 2024-09-21T15:50:45.974 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:45.974 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.974 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.974 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:45.974 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.688927+0000", 2024-09-21T15:50:45.974 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:50:45.974 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.974 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.974 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:45.974 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.688938+0000", 2024-09-21T15:50:45.974 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1466e-05 2024-09-21T15:50:45.974 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.974 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.975 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:45.975 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.688960+0000", 2024-09-21T15:50:45.975 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.128e-05 2024-09-21T15:50:45.975 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.975 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.975 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:45.975 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.688969+0000", 2024-09-21T15:50:45.975 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3400000000000004e-06 2024-09-21T15:50:45.975 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.975 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.975 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:45.975 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.689036+0000", 2024-09-21T15:50:45.975 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6506000000000003e-05 2024-09-21T15:50:45.975 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.975 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.976 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op must wait for map", 2024-09-21T15:50:45.976 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.689045+0000", 2024-09-21T15:50:45.976 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9129999999999998e-06 2024-09-21T15:50:45.976 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.976 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.976 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:45.976 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.690856+0000", 2024-09-21T15:50:45.976 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0018102960000000001 2024-09-21T15:50:45.976 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.976 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.976 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:45.976 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.690946+0000", 2024-09-21T15:50:45.976 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0265000000000004e-05 2024-09-21T15:50:45.976 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.976 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.976 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-09-21T15:50:45.977 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.691301+0000", 2024-09-21T15:50:45.977 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00035520799999999998 2024-09-21T15:50:45.977 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.977 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.977 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:45.977 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.692521+0000", 2024-09-21T15:50:45.977 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012194930000000001 2024-09-21T15:50:45.977 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.977 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.977 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:45.977 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.692849+0000", 2024-09-21T15:50:45.977 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00032818000000000002 2024-09-21T15:50:45.977 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.977 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.977 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:45.977 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.692897+0000", 2024-09-21T15:50:45.978 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8260999999999998e-05 2024-09-21T15:50:45.978 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.978 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.978 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:45.978 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.692935+0000", 2024-09-21T15:50:45.978 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8022999999999998e-05 2024-09-21T15:50:45.978 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:45.978 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:45.978 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:45.978 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.978 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.978 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:161122 3.1 3:82ccd16f:::rbd_header.11335aef9442:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-21T15:50:45.978 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:36.703081+0000", 2024-09-21T15:50:45.978 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.1998489380000006, 2024-09-21T15:50:45.978 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0051212200000000001, 2024-09-21T15:50:45.978 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:45.979 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:45.979 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:45.979 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:45.979 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:45.979 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 161122 2024-09-21T15:50:45.979 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.979 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:45.979 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.979 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:45.979 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.703081+0000", 2024-09-21T15:50:45.979 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:45.979 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.979 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.979 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:45.979 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.703081+0000", 2024-09-21T15:50:45.979 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:45.980 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.980 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.980 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:45.980 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.703079+0000", 2024-09-21T15:50:45.980 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-09-21T15:50:45.980 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.980 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.980 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:45.980 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.703091+0000", 2024-09-21T15:50:45.980 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2584e-05 2024-09-21T15:50:45.980 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.980 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.980 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:45.980 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.703107+0000", 2024-09-21T15:50:45.980 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6313000000000001e-05 2024-09-21T15:50:45.981 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.981 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.981 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:45.981 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.703119+0000", 2024-09-21T15:50:45.981 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1404e-05 2024-09-21T15:50:45.981 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.981 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.981 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:45.981 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.703176+0000", 2024-09-21T15:50:45.981 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6740999999999998e-05 2024-09-21T15:50:45.981 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.981 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.981 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:45.981 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.703281+0000", 2024-09-21T15:50:45.981 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010508 2024-09-21T15:50:45.981 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.982 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.982 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-09-21T15:50:45.982 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.703483+0000", 2024-09-21T15:50:45.982 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00020257399999999999 2024-09-21T15:50:45.982 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.982 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.982 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:45.982 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.705657+0000", 2024-09-21T15:50:45.982 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0021735610000000001 2024-09-21T15:50:45.982 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.982 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.982 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:45.982 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.708144+0000", 2024-09-21T15:50:45.982 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002487122 2024-09-21T15:50:45.983 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.983 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.983 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:45.983 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.708177+0000", 2024-09-21T15:50:45.983 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2574999999999999e-05 2024-09-21T15:50:45.983 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.983 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.983 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:45.983 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.708202+0000", 2024-09-21T15:50:45.983 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5513000000000001e-05 2024-09-21T15:50:45.983 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:45.983 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:45.983 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:45.983 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.984 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.984 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:161219 3.17 3:e87cb47d:::rbd_data.11335aef9442.000000000000000a:head [delete] snapc 2a=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-21T15:50:45.984 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:36.771292+0000", 2024-09-21T15:50:45.984 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.1316372890000004, 2024-09-21T15:50:45.984 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0058540529999999997, 2024-09-21T15:50:45.984 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:45.984 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:45.984 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:45.984 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:45.984 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:45.984 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 161219 2024-09-21T15:50:45.984 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.984 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:45.984 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.984 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:45.985 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771292+0000", 2024-09-21T15:50:45.985 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:45.985 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.985 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.985 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:45.985 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771292+0000", 2024-09-21T15:50:45.985 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:45.985 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.985 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.985 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:45.985 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771292+0000", 2024-09-21T15:50:45.985 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:45.985 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.985 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.985 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:45.986 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771298+0000", 2024-09-21T15:50:45.986 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9519999999999999e-06 2024-09-21T15:50:45.986 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.986 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.986 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:45.986 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771308+0000", 2024-09-21T15:50:45.986 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0601000000000001e-05 2024-09-21T15:50:45.986 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.986 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.986 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:45.986 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771314+0000", 2024-09-21T15:50:45.986 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.913e-06 2024-09-21T15:50:45.986 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.986 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.987 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:45.987 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771346+0000", 2024-09-21T15:50:45.987 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1711000000000002e-05 2024-09-21T15:50:45.987 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.987 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.987 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:45.987 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771399+0000", 2024-09-21T15:50:45.987 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2883000000000002e-05 2024-09-21T15:50:45.987 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.987 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.987 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-09-21T15:50:45.987 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771470+0000", 2024-09-21T15:50:45.987 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0934999999999998e-05 2024-09-21T15:50:45.987 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.987 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.988 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:45.988 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.772234+0000", 2024-09-21T15:50:45.988 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00076485400000000003 2024-09-21T15:50:45.988 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.988 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.988 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:45.988 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.777083+0000", 2024-09-21T15:50:45.988 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0048486329999999998 2024-09-21T15:50:45.988 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.988 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.988 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:45.988 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.777111+0000", 2024-09-21T15:50:45.988 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8399999999999999e-05 2024-09-21T15:50:45.988 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.989 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.989 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:45.989 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.777147+0000", 2024-09-21T15:50:45.989 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5049000000000003e-05 2024-09-21T15:50:45.989 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:45.989 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:45.989 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:45.989 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.989 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.989 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:161762 3.17 3:ef707828:::rbd_header.11338db70e6f:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T15:50:45.989 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:40.705291+0000", 2024-09-21T15:50:45.989 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.1976392389999999, 2024-09-21T15:50:45.989 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0048878120000000001, 2024-09-21T15:50:45.989 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:45.989 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:45.990 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:45.990 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:45.990 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:45.990 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 161762 2024-09-21T15:50:45.990 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.990 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:45.990 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.990 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:45.990 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.705291+0000", 2024-09-21T15:50:45.990 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:45.990 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.990 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.990 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:45.990 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.705291+0000", 2024-09-21T15:50:45.991 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:45.991 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.991 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.991 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:45.991 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.705289+0000", 2024-09-21T15:50:45.991 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:50:45.991 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.991 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.991 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:45.991 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.705296+0000", 2024-09-21T15:50:45.991 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1999999999999997e-06 2024-09-21T15:50:45.991 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.991 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.991 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:45.991 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.705309+0000", 2024-09-21T15:50:45.992 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2723e-05 2024-09-21T15:50:45.992 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.992 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.992 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:45.992 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.705318+0000", 2024-09-21T15:50:45.992 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4890000000000001e-06 2024-09-21T15:50:45.992 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.992 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.992 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:45.992 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.705370+0000", 2024-09-21T15:50:45.992 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1493000000000003e-05 2024-09-21T15:50:45.992 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.992 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.992 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:45.992 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.705480+0000", 2024-09-21T15:50:45.993 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011059 2024-09-21T15:50:45.993 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.993 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.993 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-09-21T15:50:45.993 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.705800+0000", 2024-09-21T15:50:45.993 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00031958399999999998 2024-09-21T15:50:45.993 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.993 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.993 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:45.993 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.707140+0000", 2024-09-21T15:50:45.993 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001339954 2024-09-21T15:50:45.993 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.993 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.993 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:45.993 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.710103+0000", 2024-09-21T15:50:45.993 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029629109999999999 2024-09-21T15:50:45.994 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.994 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.994 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:45.994 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.710136+0000", 2024-09-21T15:50:45.994 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3300000000000003e-05 2024-09-21T15:50:45.994 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.994 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.994 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:45.994 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.710178+0000", 2024-09-21T15:50:45.994 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2172999999999998e-05 2024-09-21T15:50:45.994 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:45.994 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:45.994 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:45.994 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.994 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.995 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:161853 3.1b 3:da34e1d8:::rbd_data.11338db70e6f.0000000000000001:head [delete] snapc 26=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T15:50:45.995 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:40.782229+0000", 2024-09-21T15:50:45.995 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.1207010139999998, 2024-09-21T15:50:45.995 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0040918660000000004, 2024-09-21T15:50:45.995 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:45.995 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:45.995 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:45.995 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:45.995 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:45.995 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 161853 2024-09-21T15:50:45.995 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.995 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:45.995 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.995 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:45.996 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782229+0000", 2024-09-21T15:50:45.996 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:45.996 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.996 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.996 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:45.996 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782229+0000", 2024-09-21T15:50:45.996 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:45.996 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.996 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.996 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:45.996 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782228+0000", 2024-09-21T15:50:45.996 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:45.996 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.996 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.997 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:45.997 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782233+0000", 2024-09-21T15:50:45.997 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2549999999999997e-06 2024-09-21T15:50:45.997 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.997 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.997 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:45.997 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782246+0000", 2024-09-21T15:50:45.997 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2563e-05 2024-09-21T15:50:45.997 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.997 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.997 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:45.997 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782252+0000", 2024-09-21T15:50:45.997 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.331e-06 2024-09-21T15:50:45.997 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.998 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.998 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:45.998 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782306+0000", 2024-09-21T15:50:45.998 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4078e-05 2024-09-21T15:50:45.998 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.998 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.998 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:45.998 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782474+0000", 2024-09-21T15:50:45.998 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000167713 2024-09-21T15:50:45.998 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.998 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.998 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-09-21T15:50:45.998 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782574+0000", 2024-09-21T15:50:45.998 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010028 2024-09-21T15:50:45.999 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.999 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.999 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:45.999 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.784621+0000", 2024-09-21T15:50:45.999 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0020471320000000001 2024-09-21T15:50:45.999 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.999 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.999 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:45.999 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786240+0000", 2024-09-21T15:50:45.999 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0016184330000000001 2024-09-21T15:50:45.999 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:45.999 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:45.999 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:45.999 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786286+0000", 2024-09-21T15:50:46.000 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6245999999999997e-05 2024-09-21T15:50:46.000 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:46.000 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:46.000 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:46.000 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786321+0000", 2024-09-21T15:50:46.000 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4739000000000001e-05 2024-09-21T15:50:46.000 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:46.000 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:46.000 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:46.000 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:46.000 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:46.000 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:161859 3.17 3:ee7ab022:::rbd_data.11338db70e6f.0000000000000007:head [delete] snapc 26=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T15:50:46.000 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:40.782264+0000", 2024-09-21T15:50:46.000 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.1206654299999999, 2024-09-21T15:50:46.000 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052663939999999998, 2024-09-21T15:50:46.001 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:46.001 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:46.001 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:46.001 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:46.001 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:46.001 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 161859 2024-09-21T15:50:46.001 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:46.001 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:46.001 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:46.001 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:46.001 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782264+0000", 2024-09-21T15:50:46.001 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:46.001 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:46.001 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:46.002 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:46.002 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782264+0000", 2024-09-21T15:50:46.002 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:46.002 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:46.002 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:46.002 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:46.002 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782264+0000", 2024-09-21T15:50:46.002 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:46.002 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:46.002 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:46.002 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:46.002 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782266+0000", 2024-09-21T15:50:46.002 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4590000000000001e-06 2024-09-21T15:50:46.002 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:46.003 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:46.003 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:46.003 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782275+0000", 2024-09-21T15:50:46.003 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5450000000000003e-06 2024-09-21T15:50:46.003 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:46.003 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:46.003 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:46.003 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782281+0000", 2024-09-21T15:50:46.003 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7180000000000001e-06 2024-09-21T15:50:46.003 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:46.003 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:46.003 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:46.003 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782312+0000", 2024-09-21T15:50:46.003 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1059000000000002e-05 2024-09-21T15:50:46.003 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:46.004 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:46.004 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:46.004 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782470+0000", 2024-09-21T15:50:46.004 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000157818 2024-09-21T15:50:46.004 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:46.004 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:46.004 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-09-21T15:50:46.004 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782574+0000", 2024-09-21T15:50:46.004 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000104536 2024-09-21T15:50:46.004 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:46.004 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:46.004 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:46.004 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.783499+0000", 2024-09-21T15:50:46.004 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00092479599999999997 2024-09-21T15:50:46.004 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:46.005 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:46.005 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:46.005 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.787472+0000", 2024-09-21T15:50:46.005 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0039730319999999996 2024-09-21T15:50:46.005 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:46.005 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:46.005 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:46.005 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.787507+0000", 2024-09-21T15:50:46.005 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5552000000000001e-05 2024-09-21T15:50:46.005 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:46.005 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:46.005 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:46.005 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.787531+0000", 2024-09-21T15:50:46.005 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3281999999999998e-05 2024-09-21T15:50:46.006 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:46.006 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:46.006 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:46.006 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:46.006 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:46.006 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:162390 3.17 3:ea8605c7:::rbd_header.1133fc941201:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-09-21T15:50:46.006 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:44.740587+0000", 2024-09-21T15:50:46.006 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.1623428339999999, 2024-09-21T15:50:46.006 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052218530000000003, 2024-09-21T15:50:46.006 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:46.006 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:46.006 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:46.006 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:46.006 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:46.007 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 162390 2024-09-21T15:50:46.007 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:46.007 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:46.007 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:46.007 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:46.007 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.740587+0000", 2024-09-21T15:50:46.007 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:46.007 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:46.007 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:46.007 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:46.007 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.740587+0000", 2024-09-21T15:50:46.007 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:46.007 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:46.007 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:46.007 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:46.007 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.740585+0000", 2024-09-21T15:50:46.008 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:50:46.008 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:46.008 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:46.008 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:46.008 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.740595+0000", 2024-09-21T15:50:46.008 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7839999999999998e-06 2024-09-21T15:50:46.008 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:46.008 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:46.008 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:46.008 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.740609+0000", 2024-09-21T15:50:46.008 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.384e-05 2024-09-21T15:50:46.008 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:46.008 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:46.008 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:46.008 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.740617+0000", 2024-09-21T15:50:46.008 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7560000000000005e-06 2024-09-21T15:50:46.008 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:46.009 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:46.009 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:46.009 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.740671+0000", 2024-09-21T15:50:46.009 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4073999999999998e-05 2024-09-21T15:50:46.009 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:46.009 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:46.009 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:46.009 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.740799+0000", 2024-09-21T15:50:46.009 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000128057 2024-09-21T15:50:46.009 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:46.009 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:46.009 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-09-21T15:50:46.009 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.741258+0000", 2024-09-21T15:50:46.009 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00045892900000000001 2024-09-21T15:50:46.009 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:46.009 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:46.009 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:46.010 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.742101+0000", 2024-09-21T15:50:46.010 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00084276499999999999 2024-09-21T15:50:46.010 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:46.010 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:46.010 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:46.010 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.745743+0000", 2024-09-21T15:50:46.010 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0036415280000000002 2024-09-21T15:50:46.010 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:46.010 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:46.010 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:46.010 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.745781+0000", 2024-09-21T15:50:46.010 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8398000000000001e-05 2024-09-21T15:50:46.010 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:46.010 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:46.010 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:46.011 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.745809+0000", 2024-09-21T15:50:46.011 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7691999999999998e-05 2024-09-21T15:50:46.011 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:46.011 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:46.011 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:46.011 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:46.011 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:46.011 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:46.011 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:50:46.013 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:46.011+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:46.013 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:50:46.112 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:46.111+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:46.157 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:46.157 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:46.157 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:50:46.157 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:46.167 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:50:46.172 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-09-21T15:50:46.173 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:50:46.189 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:50:46.190 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:50:46.205 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:50:46.205 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:50:46.213 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:50:46.212+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:50:46.222 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:50:46.223 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:50:46.239 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:50:46.239 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:50:46.255 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:50:46.255 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:50:46.271 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:50:46.271 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:50:46.271 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:46.271 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:46.280 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:50:46.287 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:50:46.287 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:50:46.339 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:50:46.338+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:50:46.389 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:46.389 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:46.397 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:50:46.438 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:46.437+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:46.517 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:46.516+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:46.538 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:46.539 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:46.547 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:50:46.618 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:50:46.616+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:50:46.649 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:46.649 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:46.657 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:50:46.719 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:46.718+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:46.788 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-09-21T15:50:46.789 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T15:50:46.789 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-21T15:50:46.789 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-21T15:50:46.812 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:46.812 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:46.820 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:46.818+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:46.823 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:50:46.920 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:46.919+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:46.969 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:46.970 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:46.979 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:50:47.020 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:47.019+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:47.079 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:50:47.079 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:47.079 INFO:teuthology.orchestra.run.smithi089.stdout: "epoch": 111, 2024-09-21T15:50:47.079 INFO:teuthology.orchestra.run.smithi089.stdout: "fsid": "4a9b64ea-3aaf-4a40-8ce8-02163a0f6e12", 2024-09-21T15:50:47.079 INFO:teuthology.orchestra.run.smithi089.stdout: "created": "2024-09-21T15:48:37.867501+0000", 2024-09-21T15:50:47.079 INFO:teuthology.orchestra.run.smithi089.stdout: "modified": "2024-09-21T15:50:46.739408+0000", 2024-09-21T15:50:47.080 INFO:teuthology.orchestra.run.smithi089.stdout: "last_up_change": "2024-09-21T15:50:29.130795+0000", 2024-09-21T15:50:47.080 INFO:teuthology.orchestra.run.smithi089.stdout: "last_in_change": "2024-09-21T15:48:40.108789+0000", 2024-09-21T15:50:47.080 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T15:50:47.080 INFO:teuthology.orchestra.run.smithi089.stdout: "flags_num": 5806080, 2024-09-21T15:50:47.080 INFO:teuthology.orchestra.run.smithi089.stdout: "flags_set": [ 2024-09-21T15:50:47.080 INFO:teuthology.orchestra.run.smithi089.stdout: "nodeep-scrub", 2024-09-21T15:50:47.080 INFO:teuthology.orchestra.run.smithi089.stdout: "noscrub", 2024-09-21T15:50:47.080 INFO:teuthology.orchestra.run.smithi089.stdout: "pglog_hardlimit", 2024-09-21T15:50:47.080 INFO:teuthology.orchestra.run.smithi089.stdout: "purged_snapdirs", 2024-09-21T15:50:47.080 INFO:teuthology.orchestra.run.smithi089.stdout: "recovery_deletes", 2024-09-21T15:50:47.080 INFO:teuthology.orchestra.run.smithi089.stdout: "sortbitwise" 2024-09-21T15:50:47.080 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-21T15:50:47.080 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_version": 5, 2024-09-21T15:50:47.080 INFO:teuthology.orchestra.run.smithi089.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T15:50:47.080 INFO:teuthology.orchestra.run.smithi089.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T15:50:47.080 INFO:teuthology.orchestra.run.smithi089.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T15:50:47.081 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_snapshot": "", 2024-09-21T15:50:47.081 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_max": 3, 2024-09-21T15:50:47.081 INFO:teuthology.orchestra.run.smithi089.stdout: "max_osd": 8, 2024-09-21T15:50:47.081 INFO:teuthology.orchestra.run.smithi089.stdout: "require_min_compat_client": "luminous", 2024-09-21T15:50:47.081 INFO:teuthology.orchestra.run.smithi089.stdout: "min_compat_client": "luminous", 2024-09-21T15:50:47.081 INFO:teuthology.orchestra.run.smithi089.stdout: "require_osd_release": "reef", 2024-09-21T15:50:47.081 INFO:teuthology.orchestra.run.smithi089.stdout: "allow_crimson": false, 2024-09-21T15:50:47.081 INFO:teuthology.orchestra.run.smithi089.stdout: "pools": [ 2024-09-21T15:50:47.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.081 INFO:teuthology.orchestra.run.smithi089.stdout: "pool": 1, 2024-09-21T15:50:47.081 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_name": ".mgr", 2024-09-21T15:50:47.081 INFO:teuthology.orchestra.run.smithi089.stdout: "create_time": "2024-09-21T15:48:49.921079+0000", 2024-09-21T15:50:47.081 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-21T15:50:47.081 INFO:teuthology.orchestra.run.smithi089.stdout: "flags_names": "hashpspool", 2024-09-21T15:50:47.081 INFO:teuthology.orchestra.run.smithi089.stdout: "type": 1, 2024-09-21T15:50:47.081 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 2, 2024-09-21T15:50:47.082 INFO:teuthology.orchestra.run.smithi089.stdout: "min_size": 1, 2024-09-21T15:50:47.082 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_rule": 0, 2024-09-21T15:50:47.082 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_count": 0, 2024-09-21T15:50:47.082 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_target": 0, 2024-09-21T15:50:47.082 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T15:50:47.082 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T15:50:47.082 INFO:teuthology.orchestra.run.smithi089.stdout: "object_hash": 2, 2024-09-21T15:50:47.082 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_autoscale_mode": "off", 2024-09-21T15:50:47.082 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num": 1, 2024-09-21T15:50:47.082 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num": 1, 2024-09-21T15:50:47.082 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num_target": 1, 2024-09-21T15:50:47.082 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_target": 1, 2024-09-21T15:50:47.082 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_pending": 1, 2024-09-21T15:50:47.082 INFO:teuthology.orchestra.run.smithi089.stdout: "last_pg_merge_meta": { 2024-09-21T15:50:47.082 INFO:teuthology.orchestra.run.smithi089.stdout: "source_pgid": "0.0", 2024-09-21T15:50:47.083 INFO:teuthology.orchestra.run.smithi089.stdout: "ready_epoch": 0, 2024-09-21T15:50:47.083 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_started": 0, 2024-09-21T15:50:47.083 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_clean": 0, 2024-09-21T15:50:47.083 INFO:teuthology.orchestra.run.smithi089.stdout: "source_version": "0'0", 2024-09-21T15:50:47.083 INFO:teuthology.orchestra.run.smithi089.stdout: "target_version": "0'0" 2024-09-21T15:50:47.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.083 INFO:teuthology.orchestra.run.smithi089.stdout: "last_change": "16", 2024-09-21T15:50:47.083 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend": "0", 2024-09-21T15:50:47.083 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T15:50:47.083 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T15:50:47.083 INFO:teuthology.orchestra.run.smithi089.stdout: "auid": 0, 2024-09-21T15:50:47.083 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_mode": "selfmanaged", 2024-09-21T15:50:47.083 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_seq": 0, 2024-09-21T15:50:47.083 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_epoch": 0, 2024-09-21T15:50:47.083 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_snaps": [], 2024-09-21T15:50:47.083 INFO:teuthology.orchestra.run.smithi089.stdout: "removed_snaps": "[]", 2024-09-21T15:50:47.083 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_bytes": 0, 2024-09-21T15:50:47.084 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_objects": 0, 2024-09-21T15:50:47.084 INFO:teuthology.orchestra.run.smithi089.stdout: "tiers": [], 2024-09-21T15:50:47.084 INFO:teuthology.orchestra.run.smithi089.stdout: "tier_of": -1, 2024-09-21T15:50:47.084 INFO:teuthology.orchestra.run.smithi089.stdout: "read_tier": -1, 2024-09-21T15:50:47.084 INFO:teuthology.orchestra.run.smithi089.stdout: "write_tier": -1, 2024-09-21T15:50:47.084 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_mode": "none", 2024-09-21T15:50:47.084 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_bytes": 0, 2024-09-21T15:50:47.084 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_objects": 0, 2024-09-21T15:50:47.084 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T15:50:47.084 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T15:50:47.084 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T15:50:47.084 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_flush_age": 0, 2024-09-21T15:50:47.084 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_evict_age": 0, 2024-09-21T15:50:47.084 INFO:teuthology.orchestra.run.smithi089.stdout: "erasure_code_profile": "", 2024-09-21T15:50:47.084 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_params": { 2024-09-21T15:50:47.084 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "none" 2024-09-21T15:50:47.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.085 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_period": 0, 2024-09-21T15:50:47.085 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_count": 0, 2024-09-21T15:50:47.085 INFO:teuthology.orchestra.run.smithi089.stdout: "use_gmt_hitset": true, 2024-09-21T15:50:47.085 INFO:teuthology.orchestra.run.smithi089.stdout: "min_read_recency_for_promote": 0, 2024-09-21T15:50:47.085 INFO:teuthology.orchestra.run.smithi089.stdout: "min_write_recency_for_promote": 0, 2024-09-21T15:50:47.085 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T15:50:47.085 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_search_last_n": 0, 2024-09-21T15:50:47.085 INFO:teuthology.orchestra.run.smithi089.stdout: "grade_table": [], 2024-09-21T15:50:47.085 INFO:teuthology.orchestra.run.smithi089.stdout: "stripe_width": 0, 2024-09-21T15:50:47.085 INFO:teuthology.orchestra.run.smithi089.stdout: "expected_num_objects": 0, 2024-09-21T15:50:47.085 INFO:teuthology.orchestra.run.smithi089.stdout: "fast_read": false, 2024-09-21T15:50:47.085 INFO:teuthology.orchestra.run.smithi089.stdout: "options": { 2024-09-21T15:50:47.085 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_max": 32, 2024-09-21T15:50:47.085 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_min": 1 2024-09-21T15:50:47.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.086 INFO:teuthology.orchestra.run.smithi089.stdout: "application_metadata": { 2024-09-21T15:50:47.086 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": {} 2024-09-21T15:50:47.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.086 INFO:teuthology.orchestra.run.smithi089.stdout: "read_balance": { 2024-09-21T15:50:47.086 INFO:teuthology.orchestra.run.smithi089.stdout: "score_acting": 8, 2024-09-21T15:50:47.086 INFO:teuthology.orchestra.run.smithi089.stdout: "score_stable": 8, 2024-09-21T15:50:47.086 INFO:teuthology.orchestra.run.smithi089.stdout: "optimal_score": 0.25, 2024-09-21T15:50:47.086 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_acting": 2, 2024-09-21T15:50:47.086 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_stable": 2, 2024-09-21T15:50:47.086 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity_weighted": 0.86229705810546875, 2024-09-21T15:50:47.086 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity": 0.86000001430511475, 2024-09-21T15:50:47.086 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity_weighted": 0.86000001430511475 2024-09-21T15:50:47.086 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:47.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.087 INFO:teuthology.orchestra.run.smithi089.stdout: "pool": 2, 2024-09-21T15:50:47.087 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_name": "rbd", 2024-09-21T15:50:47.087 INFO:teuthology.orchestra.run.smithi089.stdout: "create_time": "2024-09-21T15:48:55.812982+0000", 2024-09-21T15:50:47.087 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 8193, 2024-09-21T15:50:47.087 INFO:teuthology.orchestra.run.smithi089.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T15:50:47.087 INFO:teuthology.orchestra.run.smithi089.stdout: "type": 1, 2024-09-21T15:50:47.087 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 2, 2024-09-21T15:50:47.087 INFO:teuthology.orchestra.run.smithi089.stdout: "min_size": 1, 2024-09-21T15:50:47.087 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_rule": 0, 2024-09-21T15:50:47.087 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_count": 0, 2024-09-21T15:50:47.087 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_target": 0, 2024-09-21T15:50:47.087 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T15:50:47.087 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T15:50:47.087 INFO:teuthology.orchestra.run.smithi089.stdout: "object_hash": 2, 2024-09-21T15:50:47.088 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_autoscale_mode": "off", 2024-09-21T15:50:47.088 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num": 8, 2024-09-21T15:50:47.088 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num": 8, 2024-09-21T15:50:47.088 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num_target": 8, 2024-09-21T15:50:47.088 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_target": 8, 2024-09-21T15:50:47.088 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_pending": 8, 2024-09-21T15:50:47.088 INFO:teuthology.orchestra.run.smithi089.stdout: "last_pg_merge_meta": { 2024-09-21T15:50:47.088 INFO:teuthology.orchestra.run.smithi089.stdout: "source_pgid": "0.0", 2024-09-21T15:50:47.088 INFO:teuthology.orchestra.run.smithi089.stdout: "ready_epoch": 0, 2024-09-21T15:50:47.088 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_started": 0, 2024-09-21T15:50:47.088 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_clean": 0, 2024-09-21T15:50:47.088 INFO:teuthology.orchestra.run.smithi089.stdout: "source_version": "0'0", 2024-09-21T15:50:47.088 INFO:teuthology.orchestra.run.smithi089.stdout: "target_version": "0'0" 2024-09-21T15:50:47.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.088 INFO:teuthology.orchestra.run.smithi089.stdout: "last_change": "20", 2024-09-21T15:50:47.088 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend": "0", 2024-09-21T15:50:47.089 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T15:50:47.089 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T15:50:47.089 INFO:teuthology.orchestra.run.smithi089.stdout: "auid": 0, 2024-09-21T15:50:47.089 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_mode": "selfmanaged", 2024-09-21T15:50:47.089 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_seq": 3, 2024-09-21T15:50:47.089 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_epoch": 20, 2024-09-21T15:50:47.089 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_snaps": [], 2024-09-21T15:50:47.089 INFO:teuthology.orchestra.run.smithi089.stdout: "removed_snaps": "[]", 2024-09-21T15:50:47.089 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_bytes": 0, 2024-09-21T15:50:47.089 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_objects": 0, 2024-09-21T15:50:47.089 INFO:teuthology.orchestra.run.smithi089.stdout: "tiers": [], 2024-09-21T15:50:47.089 INFO:teuthology.orchestra.run.smithi089.stdout: "tier_of": -1, 2024-09-21T15:50:47.089 INFO:teuthology.orchestra.run.smithi089.stdout: "read_tier": -1, 2024-09-21T15:50:47.089 INFO:teuthology.orchestra.run.smithi089.stdout: "write_tier": -1, 2024-09-21T15:50:47.089 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_mode": "none", 2024-09-21T15:50:47.090 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_bytes": 0, 2024-09-21T15:50:47.090 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_objects": 0, 2024-09-21T15:50:47.090 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T15:50:47.090 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T15:50:47.090 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T15:50:47.090 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_flush_age": 0, 2024-09-21T15:50:47.090 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_evict_age": 0, 2024-09-21T15:50:47.090 INFO:teuthology.orchestra.run.smithi089.stdout: "erasure_code_profile": "", 2024-09-21T15:50:47.090 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_params": { 2024-09-21T15:50:47.090 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "none" 2024-09-21T15:50:47.090 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.090 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_period": 0, 2024-09-21T15:50:47.090 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_count": 0, 2024-09-21T15:50:47.090 INFO:teuthology.orchestra.run.smithi089.stdout: "use_gmt_hitset": true, 2024-09-21T15:50:47.090 INFO:teuthology.orchestra.run.smithi089.stdout: "min_read_recency_for_promote": 0, 2024-09-21T15:50:47.091 INFO:teuthology.orchestra.run.smithi089.stdout: "min_write_recency_for_promote": 0, 2024-09-21T15:50:47.091 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T15:50:47.091 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_search_last_n": 0, 2024-09-21T15:50:47.091 INFO:teuthology.orchestra.run.smithi089.stdout: "grade_table": [], 2024-09-21T15:50:47.091 INFO:teuthology.orchestra.run.smithi089.stdout: "stripe_width": 0, 2024-09-21T15:50:47.091 INFO:teuthology.orchestra.run.smithi089.stdout: "expected_num_objects": 0, 2024-09-21T15:50:47.091 INFO:teuthology.orchestra.run.smithi089.stdout: "fast_read": false, 2024-09-21T15:50:47.091 INFO:teuthology.orchestra.run.smithi089.stdout: "options": {}, 2024-09-21T15:50:47.091 INFO:teuthology.orchestra.run.smithi089.stdout: "application_metadata": { 2024-09-21T15:50:47.091 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd": {} 2024-09-21T15:50:47.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.091 INFO:teuthology.orchestra.run.smithi089.stdout: "read_balance": { 2024-09-21T15:50:47.091 INFO:teuthology.orchestra.run.smithi089.stdout: "score_acting": 2.9900000095367432, 2024-09-21T15:50:47.091 INFO:teuthology.orchestra.run.smithi089.stdout: "score_stable": 2.9900000095367432, 2024-09-21T15:50:47.092 INFO:teuthology.orchestra.run.smithi089.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T15:50:47.092 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_acting": 2.630000114440918, 2024-09-21T15:50:47.092 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_stable": 2.630000114440918, 2024-09-21T15:50:47.092 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity_weighted": 0.86229705810546875, 2024-09-21T15:50:47.092 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity": 0.86000001430511475, 2024-09-21T15:50:47.092 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity_weighted": 0.86000001430511475 2024-09-21T15:50:47.092 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:47.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.092 INFO:teuthology.orchestra.run.smithi089.stdout: "pool": 3, 2024-09-21T15:50:47.092 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_name": "pool_client.0", 2024-09-21T15:50:47.092 INFO:teuthology.orchestra.run.smithi089.stdout: "create_time": "2024-09-21T15:49:05.523888+0000", 2024-09-21T15:50:47.092 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 8193, 2024-09-21T15:50:47.092 INFO:teuthology.orchestra.run.smithi089.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T15:50:47.092 INFO:teuthology.orchestra.run.smithi089.stdout: "type": 1, 2024-09-21T15:50:47.092 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 2, 2024-09-21T15:50:47.093 INFO:teuthology.orchestra.run.smithi089.stdout: "min_size": 1, 2024-09-21T15:50:47.093 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_rule": 0, 2024-09-21T15:50:47.093 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_count": 0, 2024-09-21T15:50:47.093 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_target": 0, 2024-09-21T15:50:47.093 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T15:50:47.093 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T15:50:47.093 INFO:teuthology.orchestra.run.smithi089.stdout: "object_hash": 2, 2024-09-21T15:50:47.093 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_autoscale_mode": "off", 2024-09-21T15:50:47.093 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num": 32, 2024-09-21T15:50:47.093 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num": 32, 2024-09-21T15:50:47.093 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num_target": 32, 2024-09-21T15:50:47.093 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_target": 32, 2024-09-21T15:50:47.093 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_pending": 32, 2024-09-21T15:50:47.093 INFO:teuthology.orchestra.run.smithi089.stdout: "last_pg_merge_meta": { 2024-09-21T15:50:47.093 INFO:teuthology.orchestra.run.smithi089.stdout: "source_pgid": "0.0", 2024-09-21T15:50:47.094 INFO:teuthology.orchestra.run.smithi089.stdout: "ready_epoch": 0, 2024-09-21T15:50:47.094 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_started": 0, 2024-09-21T15:50:47.094 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_clean": 0, 2024-09-21T15:50:47.094 INFO:teuthology.orchestra.run.smithi089.stdout: "source_version": "0'0", 2024-09-21T15:50:47.094 INFO:teuthology.orchestra.run.smithi089.stdout: "target_version": "0'0" 2024-09-21T15:50:47.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.094 INFO:teuthology.orchestra.run.smithi089.stdout: "last_change": "111", 2024-09-21T15:50:47.094 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend": "0", 2024-09-21T15:50:47.094 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T15:50:47.094 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T15:50:47.094 INFO:teuthology.orchestra.run.smithi089.stdout: "auid": 0, 2024-09-21T15:50:47.094 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_mode": "selfmanaged", 2024-09-21T15:50:47.094 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_seq": 63, 2024-09-21T15:50:47.094 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_epoch": 111, 2024-09-21T15:50:47.095 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_snaps": [], 2024-09-21T15:50:47.095 INFO:teuthology.orchestra.run.smithi089.stdout: "removed_snaps": "[]", 2024-09-21T15:50:47.095 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_bytes": 0, 2024-09-21T15:50:47.095 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_objects": 0, 2024-09-21T15:50:47.095 INFO:teuthology.orchestra.run.smithi089.stdout: "tiers": [], 2024-09-21T15:50:47.095 INFO:teuthology.orchestra.run.smithi089.stdout: "tier_of": -1, 2024-09-21T15:50:47.095 INFO:teuthology.orchestra.run.smithi089.stdout: "read_tier": -1, 2024-09-21T15:50:47.095 INFO:teuthology.orchestra.run.smithi089.stdout: "write_tier": -1, 2024-09-21T15:50:47.095 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_mode": "none", 2024-09-21T15:50:47.095 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_bytes": 0, 2024-09-21T15:50:47.095 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_objects": 0, 2024-09-21T15:50:47.095 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T15:50:47.095 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T15:50:47.095 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T15:50:47.096 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_flush_age": 0, 2024-09-21T15:50:47.096 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_evict_age": 0, 2024-09-21T15:50:47.096 INFO:teuthology.orchestra.run.smithi089.stdout: "erasure_code_profile": "", 2024-09-21T15:50:47.096 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_params": { 2024-09-21T15:50:47.096 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "none" 2024-09-21T15:50:47.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.096 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_period": 0, 2024-09-21T15:50:47.096 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_count": 0, 2024-09-21T15:50:47.096 INFO:teuthology.orchestra.run.smithi089.stdout: "use_gmt_hitset": true, 2024-09-21T15:50:47.096 INFO:teuthology.orchestra.run.smithi089.stdout: "min_read_recency_for_promote": 0, 2024-09-21T15:50:47.096 INFO:teuthology.orchestra.run.smithi089.stdout: "min_write_recency_for_promote": 0, 2024-09-21T15:50:47.096 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T15:50:47.096 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_search_last_n": 0, 2024-09-21T15:50:47.096 INFO:teuthology.orchestra.run.smithi089.stdout: "grade_table": [], 2024-09-21T15:50:47.096 INFO:teuthology.orchestra.run.smithi089.stdout: "stripe_width": 0, 2024-09-21T15:50:47.097 INFO:teuthology.orchestra.run.smithi089.stdout: "expected_num_objects": 0, 2024-09-21T15:50:47.097 INFO:teuthology.orchestra.run.smithi089.stdout: "fast_read": false, 2024-09-21T15:50:47.097 INFO:teuthology.orchestra.run.smithi089.stdout: "options": {}, 2024-09-21T15:50:47.097 INFO:teuthology.orchestra.run.smithi089.stdout: "application_metadata": { 2024-09-21T15:50:47.097 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd": {} 2024-09-21T15:50:47.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.097 INFO:teuthology.orchestra.run.smithi089.stdout: "read_balance": { 2024-09-21T15:50:47.097 INFO:teuthology.orchestra.run.smithi089.stdout: "score_acting": 2, 2024-09-21T15:50:47.097 INFO:teuthology.orchestra.run.smithi089.stdout: "score_stable": 2, 2024-09-21T15:50:47.097 INFO:teuthology.orchestra.run.smithi089.stdout: "optimal_score": 1, 2024-09-21T15:50:47.097 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_acting": 2, 2024-09-21T15:50:47.097 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_stable": 2, 2024-09-21T15:50:47.097 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity_weighted": 0.86229705810546875, 2024-09-21T15:50:47.097 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity": 0.86000001430511475, 2024-09-21T15:50:47.097 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity_weighted": 0.86000001430511475 2024-09-21T15:50:47.097 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:47.097 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:47.098 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-21T15:50:47.098 INFO:teuthology.orchestra.run.smithi089.stdout: "osds": [ 2024-09-21T15:50:47.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.098 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 0, 2024-09-21T15:50:47.098 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "7b3fabfa-3fe6-47c7-9d0c-a6036f738771", 2024-09-21T15:50:47.098 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-09-21T15:50:47.098 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-09-21T15:50:47.098 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-09-21T15:50:47.098 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-09-21T15:50:47.098 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 13, 2024-09-21T15:50:47.098 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 67, 2024-09-21T15:50:47.098 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 93, 2024-09-21T15:50:47.098 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 96, 2024-09-21T15:50:47.098 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 69, 2024-09-21T15:50:47.098 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-09-21T15:50:47.099 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-09-21T15:50:47.099 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:50:47.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.099 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:50:47.099 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6800", 2024-09-21T15:50:47.099 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 1222314186 2024-09-21T15:50:47.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.099 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:50:47.099 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6801", 2024-09-21T15:50:47.099 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 1222314186 2024-09-21T15:50:47.099 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:47.099 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:47.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.099 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-09-21T15:50:47.099 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:50:47.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.100 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:50:47.100 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6803", 2024-09-21T15:50:47.100 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 1222314186 2024-09-21T15:50:47.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.100 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:50:47.100 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6805", 2024-09-21T15:50:47.100 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 1222314186 2024-09-21T15:50:47.100 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:47.100 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:47.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.100 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-09-21T15:50:47.100 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:50:47.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.100 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:50:47.101 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6811", 2024-09-21T15:50:47.101 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 1222314186 2024-09-21T15:50:47.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.101 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:50:47.101 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6813", 2024-09-21T15:50:47.101 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 1222314186 2024-09-21T15:50:47.101 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:47.101 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:47.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.101 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-09-21T15:50:47.101 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:50:47.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.101 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:50:47.101 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6807", 2024-09-21T15:50:47.101 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 1222314186 2024-09-21T15:50:47.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.102 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:50:47.102 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6809", 2024-09-21T15:50:47.102 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 1222314186 2024-09-21T15:50:47.102 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:47.102 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:47.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.102 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.89:6801/1222314186", 2024-09-21T15:50:47.102 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.89:6805/1222314186", 2024-09-21T15:50:47.102 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.89:6813/1222314186", 2024-09-21T15:50:47.102 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.89:6809/1222314186", 2024-09-21T15:50:47.102 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-09-21T15:50:47.102 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-09-21T15:50:47.102 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-09-21T15:50:47.102 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:47.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.103 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 1, 2024-09-21T15:50:47.103 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "767ad1bc-84ef-4102-83f5-aacda0917ddb", 2024-09-21T15:50:47.103 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-09-21T15:50:47.103 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-09-21T15:50:47.103 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-09-21T15:50:47.103 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-09-21T15:50:47.103 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-09-21T15:50:47.103 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-09-21T15:50:47.103 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-09-21T15:50:47.103 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 99, 2024-09-21T15:50:47.103 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-09-21T15:50:47.103 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-09-21T15:50:47.103 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-09-21T15:50:47.103 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:50:47.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.104 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:50:47.104 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6816", 2024-09-21T15:50:47.104 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3187095367 2024-09-21T15:50:47.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.104 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:50:47.104 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6817", 2024-09-21T15:50:47.104 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3187095367 2024-09-21T15:50:47.104 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:47.104 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:47.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.104 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-09-21T15:50:47.104 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:50:47.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.104 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:50:47.105 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6818", 2024-09-21T15:50:47.105 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3187095367 2024-09-21T15:50:47.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.105 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:50:47.105 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6819", 2024-09-21T15:50:47.105 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3187095367 2024-09-21T15:50:47.105 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:47.105 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:47.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.105 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-09-21T15:50:47.105 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:50:47.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.105 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:50:47.105 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6822", 2024-09-21T15:50:47.106 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3187095367 2024-09-21T15:50:47.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.106 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:50:47.106 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6824", 2024-09-21T15:50:47.106 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3187095367 2024-09-21T15:50:47.106 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:47.106 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:47.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.106 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-09-21T15:50:47.106 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:50:47.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.106 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:50:47.106 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6820", 2024-09-21T15:50:47.106 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3187095367 2024-09-21T15:50:47.107 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.107 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.107 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:50:47.107 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6821", 2024-09-21T15:50:47.107 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3187095367 2024-09-21T15:50:47.107 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:47.107 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:47.107 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.107 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.89:6817/3187095367", 2024-09-21T15:50:47.107 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.89:6819/3187095367", 2024-09-21T15:50:47.107 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.89:6824/3187095367", 2024-09-21T15:50:47.107 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.89:6821/3187095367", 2024-09-21T15:50:47.107 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-09-21T15:50:47.107 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-09-21T15:50:47.107 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-09-21T15:50:47.107 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:47.108 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.108 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.108 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 2, 2024-09-21T15:50:47.108 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "1dbd0d50-aae9-42a7-90a1-aa92697778ae", 2024-09-21T15:50:47.108 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-09-21T15:50:47.108 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-09-21T15:50:47.108 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-09-21T15:50:47.108 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-09-21T15:50:47.108 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-09-21T15:50:47.108 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-09-21T15:50:47.108 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-09-21T15:50:47.108 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 96, 2024-09-21T15:50:47.108 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-09-21T15:50:47.108 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-09-21T15:50:47.108 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-09-21T15:50:47.108 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:50:47.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.109 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:50:47.109 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6802", 2024-09-21T15:50:47.109 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 4098625737 2024-09-21T15:50:47.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.109 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:50:47.109 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6804", 2024-09-21T15:50:47.109 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 4098625737 2024-09-21T15:50:47.109 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:47.109 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:47.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.109 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-09-21T15:50:47.109 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:50:47.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.109 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:50:47.110 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6806", 2024-09-21T15:50:47.110 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 4098625737 2024-09-21T15:50:47.110 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.110 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.110 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:50:47.110 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6808", 2024-09-21T15:50:47.110 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 4098625737 2024-09-21T15:50:47.110 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:47.110 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:47.110 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.110 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-09-21T15:50:47.110 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:50:47.110 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.110 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:50:47.111 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6814", 2024-09-21T15:50:47.111 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 4098625737 2024-09-21T15:50:47.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.111 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:50:47.111 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6815", 2024-09-21T15:50:47.111 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 4098625737 2024-09-21T15:50:47.111 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:47.111 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:47.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.111 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-09-21T15:50:47.111 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:50:47.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.111 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:50:47.111 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6810", 2024-09-21T15:50:47.112 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 4098625737 2024-09-21T15:50:47.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.112 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:50:47.112 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6812", 2024-09-21T15:50:47.112 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 4098625737 2024-09-21T15:50:47.112 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:47.112 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:47.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.112 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.89:6804/4098625737", 2024-09-21T15:50:47.112 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.89:6808/4098625737", 2024-09-21T15:50:47.112 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.89:6815/4098625737", 2024-09-21T15:50:47.112 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.89:6812/4098625737", 2024-09-21T15:50:47.112 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-09-21T15:50:47.112 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-09-21T15:50:47.112 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-09-21T15:50:47.113 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:47.113 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.113 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 3, 2024-09-21T15:50:47.113 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "f37cd295-2bac-4df4-a99b-abedc0978dce", 2024-09-21T15:50:47.113 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-09-21T15:50:47.113 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-09-21T15:50:47.113 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-09-21T15:50:47.113 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 0.6057586669921875, 2024-09-21T15:50:47.113 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-09-21T15:50:47.113 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-09-21T15:50:47.113 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-09-21T15:50:47.113 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 95, 2024-09-21T15:50:47.113 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-09-21T15:50:47.113 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-09-21T15:50:47.113 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-09-21T15:50:47.113 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:50:47.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.114 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:50:47.114 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6823", 2024-09-21T15:50:47.114 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 1886828410 2024-09-21T15:50:47.114 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.114 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:50:47.114 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6825", 2024-09-21T15:50:47.114 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 1886828410 2024-09-21T15:50:47.114 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:47.114 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:47.114 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.114 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-09-21T15:50:47.114 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:50:47.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.115 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:50:47.115 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6826", 2024-09-21T15:50:47.115 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 1886828410 2024-09-21T15:50:47.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.115 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:50:47.115 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6827", 2024-09-21T15:50:47.115 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 1886828410 2024-09-21T15:50:47.115 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:47.115 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:47.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.115 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-09-21T15:50:47.115 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:50:47.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.115 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:50:47.115 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6830", 2024-09-21T15:50:47.116 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 1886828410 2024-09-21T15:50:47.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.116 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:50:47.116 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6831", 2024-09-21T15:50:47.116 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 1886828410 2024-09-21T15:50:47.116 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:47.116 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:47.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.116 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-09-21T15:50:47.116 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:50:47.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.116 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:50:47.116 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6828", 2024-09-21T15:50:47.116 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 1886828410 2024-09-21T15:50:47.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.117 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:50:47.117 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6829", 2024-09-21T15:50:47.117 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 1886828410 2024-09-21T15:50:47.117 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:47.117 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:47.117 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.117 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.89:6825/1886828410", 2024-09-21T15:50:47.117 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.89:6827/1886828410", 2024-09-21T15:50:47.117 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.89:6831/1886828410", 2024-09-21T15:50:47.117 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.89:6829/1886828410", 2024-09-21T15:50:47.117 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-09-21T15:50:47.117 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-09-21T15:50:47.117 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-09-21T15:50:47.117 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:47.117 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.117 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.118 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 4, 2024-09-21T15:50:47.118 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "2eeb351d-d51c-4fc3-884e-debecd995aea", 2024-09-21T15:50:47.118 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-09-21T15:50:47.118 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-09-21T15:50:47.118 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-09-21T15:50:47.118 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-09-21T15:50:47.118 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-09-21T15:50:47.118 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-09-21T15:50:47.118 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-09-21T15:50:47.118 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 96, 2024-09-21T15:50:47.118 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-09-21T15:50:47.118 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-09-21T15:50:47.118 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-09-21T15:50:47.118 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:50:47.118 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.119 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:50:47.119 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6817", 2024-09-21T15:50:47.119 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 22722842 2024-09-21T15:50:47.119 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.119 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.119 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:50:47.119 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6819", 2024-09-21T15:50:47.119 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 22722842 2024-09-21T15:50:47.119 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:47.119 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:47.119 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.119 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-09-21T15:50:47.119 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:50:47.119 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.120 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:50:47.120 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6820", 2024-09-21T15:50:47.120 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 22722842 2024-09-21T15:50:47.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.120 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.120 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:50:47.120 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6822", 2024-09-21T15:50:47.120 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 22722842 2024-09-21T15:50:47.120 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:47.120 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:47.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.120 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-09-21T15:50:47.120 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:50:47.120 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.120 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:50:47.121 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6828", 2024-09-21T15:50:47.121 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 22722842 2024-09-21T15:50:47.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.121 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:50:47.121 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6830", 2024-09-21T15:50:47.121 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 22722842 2024-09-21T15:50:47.121 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:47.121 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:47.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.121 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-09-21T15:50:47.121 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:50:47.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.121 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:50:47.121 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6824", 2024-09-21T15:50:47.122 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 22722842 2024-09-21T15:50:47.122 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.122 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.122 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:50:47.122 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6826", 2024-09-21T15:50:47.122 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 22722842 2024-09-21T15:50:47.122 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:47.122 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:47.122 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.122 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.100:6819/22722842", 2024-09-21T15:50:47.122 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.100:6822/22722842", 2024-09-21T15:50:47.122 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.100:6830/22722842", 2024-09-21T15:50:47.122 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.100:6826/22722842", 2024-09-21T15:50:47.122 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-09-21T15:50:47.122 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-09-21T15:50:47.123 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-09-21T15:50:47.123 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:47.123 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.123 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 5, 2024-09-21T15:50:47.123 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "b5396267-6823-42f1-bd04-bd8f144fa76c", 2024-09-21T15:50:47.123 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-09-21T15:50:47.123 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-09-21T15:50:47.123 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-09-21T15:50:47.123 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 0.2926177978515625, 2024-09-21T15:50:47.123 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-09-21T15:50:47.123 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-09-21T15:50:47.123 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-09-21T15:50:47.124 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 95, 2024-09-21T15:50:47.124 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-09-21T15:50:47.124 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-09-21T15:50:47.124 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-09-21T15:50:47.124 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:50:47.124 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.124 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:50:47.124 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6816", 2024-09-21T15:50:47.124 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 363733545 2024-09-21T15:50:47.124 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.124 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.124 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:50:47.124 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6818", 2024-09-21T15:50:47.124 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 363733545 2024-09-21T15:50:47.124 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:47.125 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:47.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.125 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-09-21T15:50:47.125 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:50:47.125 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.125 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:50:47.125 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6821", 2024-09-21T15:50:47.125 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 363733545 2024-09-21T15:50:47.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.125 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.125 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:50:47.125 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6823", 2024-09-21T15:50:47.125 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 363733545 2024-09-21T15:50:47.125 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:47.125 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:47.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.126 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-09-21T15:50:47.126 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:50:47.126 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.126 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:50:47.126 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6829", 2024-09-21T15:50:47.126 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 363733545 2024-09-21T15:50:47.126 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.126 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.126 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:50:47.126 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6831", 2024-09-21T15:50:47.126 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 363733545 2024-09-21T15:50:47.126 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:47.126 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:47.126 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.126 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-09-21T15:50:47.127 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:50:47.127 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.127 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:50:47.127 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6825", 2024-09-21T15:50:47.127 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 363733545 2024-09-21T15:50:47.127 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.127 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.127 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:50:47.127 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6827", 2024-09-21T15:50:47.127 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 363733545 2024-09-21T15:50:47.127 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:47.127 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:47.127 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.127 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.100:6818/363733545", 2024-09-21T15:50:47.127 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.100:6823/363733545", 2024-09-21T15:50:47.128 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.100:6831/363733545", 2024-09-21T15:50:47.128 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.100:6827/363733545", 2024-09-21T15:50:47.128 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-09-21T15:50:47.128 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-09-21T15:50:47.128 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-09-21T15:50:47.128 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:47.128 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.128 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.128 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 6, 2024-09-21T15:50:47.128 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "955b67e9-73bc-42c1-88e0-3356c105e1ea", 2024-09-21T15:50:47.128 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-09-21T15:50:47.128 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-09-21T15:50:47.128 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-09-21T15:50:47.128 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-09-21T15:50:47.128 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-09-21T15:50:47.129 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-09-21T15:50:47.129 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-09-21T15:50:47.129 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 96, 2024-09-21T15:50:47.129 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-09-21T15:50:47.129 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-09-21T15:50:47.129 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-09-21T15:50:47.129 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:50:47.129 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.129 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:50:47.129 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6808", 2024-09-21T15:50:47.129 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 205694879 2024-09-21T15:50:47.129 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.129 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.129 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:50:47.129 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6809", 2024-09-21T15:50:47.129 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 205694879 2024-09-21T15:50:47.130 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:47.130 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:47.130 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.130 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-09-21T15:50:47.130 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:50:47.130 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.130 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:50:47.130 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6810", 2024-09-21T15:50:47.130 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 205694879 2024-09-21T15:50:47.130 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.130 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.130 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:50:47.130 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6811", 2024-09-21T15:50:47.130 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 205694879 2024-09-21T15:50:47.130 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:47.131 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:47.131 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.131 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-09-21T15:50:47.131 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:50:47.131 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.131 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:50:47.131 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6814", 2024-09-21T15:50:47.131 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 205694879 2024-09-21T15:50:47.131 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.131 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.131 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:50:47.131 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6815", 2024-09-21T15:50:47.131 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 205694879 2024-09-21T15:50:47.131 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:47.131 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:47.131 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.132 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-09-21T15:50:47.132 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:50:47.132 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.132 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:50:47.132 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6812", 2024-09-21T15:50:47.132 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 205694879 2024-09-21T15:50:47.132 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.132 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.132 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:50:47.132 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6813", 2024-09-21T15:50:47.132 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 205694879 2024-09-21T15:50:47.132 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:47.132 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:47.132 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.133 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.100:6809/205694879", 2024-09-21T15:50:47.133 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.100:6811/205694879", 2024-09-21T15:50:47.133 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.100:6815/205694879", 2024-09-21T15:50:47.133 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.100:6813/205694879", 2024-09-21T15:50:47.133 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-09-21T15:50:47.133 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-09-21T15:50:47.133 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-09-21T15:50:47.133 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:47.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.133 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.133 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 7, 2024-09-21T15:50:47.133 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "414ea7f0-138d-404e-bce6-fe8dedcdc938", 2024-09-21T15:50:47.133 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-09-21T15:50:47.133 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-09-21T15:50:47.133 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-09-21T15:50:47.133 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-09-21T15:50:47.134 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-09-21T15:50:47.134 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-09-21T15:50:47.134 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-09-21T15:50:47.134 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 95, 2024-09-21T15:50:47.134 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-09-21T15:50:47.134 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-09-21T15:50:47.134 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-09-21T15:50:47.134 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:50:47.134 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.134 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:50:47.134 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6800", 2024-09-21T15:50:47.134 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3120011840 2024-09-21T15:50:47.134 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.134 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.134 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:50:47.135 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6801", 2024-09-21T15:50:47.135 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3120011840 2024-09-21T15:50:47.135 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:47.135 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:47.135 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.135 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-09-21T15:50:47.135 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:50:47.135 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.135 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:50:47.135 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6802", 2024-09-21T15:50:47.135 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3120011840 2024-09-21T15:50:47.135 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.135 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.135 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:50:47.135 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6803", 2024-09-21T15:50:47.135 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3120011840 2024-09-21T15:50:47.136 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:47.136 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:47.136 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.136 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-09-21T15:50:47.136 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:50:47.136 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.136 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:50:47.136 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6806", 2024-09-21T15:50:47.136 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3120011840 2024-09-21T15:50:47.136 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.136 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.136 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:50:47.136 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6807", 2024-09-21T15:50:47.136 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3120011840 2024-09-21T15:50:47.136 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:47.136 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:47.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.137 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-09-21T15:50:47.137 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:50:47.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.137 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:50:47.137 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6804", 2024-09-21T15:50:47.137 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3120011840 2024-09-21T15:50:47.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.137 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:50:47.137 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6805", 2024-09-21T15:50:47.137 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3120011840 2024-09-21T15:50:47.137 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:47.137 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:47.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.137 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.100:6801/3120011840", 2024-09-21T15:50:47.138 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.100:6803/3120011840", 2024-09-21T15:50:47.138 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.100:6807/3120011840", 2024-09-21T15:50:47.138 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.100:6805/3120011840", 2024-09-21T15:50:47.138 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-09-21T15:50:47.138 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-09-21T15:50:47.138 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-09-21T15:50:47.138 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:47.138 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:47.138 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-21T15:50:47.138 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_xinfo": [ 2024-09-21T15:50:47.138 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.138 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 0, 2024-09-21T15:50:47.138 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-09-21T15:50:47.138 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-09-21T15:50:47.138 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-09-21T15:50:47.139 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-09-21T15:50:47.139 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-09-21T15:50:47.139 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-09-21T15:48:45.230746+0000", 2024-09-21T15:50:47.139 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 69 2024-09-21T15:50:47.139 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.139 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 1, 2024-09-21T15:50:47.139 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-09-21T15:50:47.139 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-09-21T15:50:47.139 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-09-21T15:50:47.139 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-09-21T15:50:47.139 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-09-21T15:50:47.139 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-09-21T15:48:45.303567+0000", 2024-09-21T15:50:47.139 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-09-21T15:50:47.139 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.140 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 2, 2024-09-21T15:50:47.140 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-09-21T15:50:47.140 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-09-21T15:50:47.140 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-09-21T15:50:47.140 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-09-21T15:50:47.140 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-09-21T15:50:47.140 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-09-21T15:48:45.210941+0000", 2024-09-21T15:50:47.140 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-09-21T15:50:47.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.140 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 3, 2024-09-21T15:50:47.140 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-09-21T15:50:47.140 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-09-21T15:50:47.140 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-09-21T15:50:47.141 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-09-21T15:50:47.141 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-09-21T15:50:47.141 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-09-21T15:48:45.218918+0000", 2024-09-21T15:50:47.141 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-09-21T15:50:47.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.141 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 4, 2024-09-21T15:50:47.141 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-09-21T15:50:47.141 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-09-21T15:50:47.141 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-09-21T15:50:47.141 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-09-21T15:50:47.141 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-09-21T15:50:47.141 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-09-21T15:48:45.301781+0000", 2024-09-21T15:50:47.141 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-09-21T15:50:47.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.142 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 5, 2024-09-21T15:50:47.142 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-09-21T15:50:47.142 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-09-21T15:50:47.142 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-09-21T15:50:47.142 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-09-21T15:50:47.142 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-09-21T15:50:47.142 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-09-21T15:48:45.248444+0000", 2024-09-21T15:50:47.142 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-09-21T15:50:47.142 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.142 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.142 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 6, 2024-09-21T15:50:47.142 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-09-21T15:50:47.142 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-09-21T15:50:47.142 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-09-21T15:50:47.142 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-09-21T15:50:47.142 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-09-21T15:50:47.143 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-09-21T15:48:45.275465+0000", 2024-09-21T15:50:47.143 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-09-21T15:50:47.143 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.143 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.143 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 7, 2024-09-21T15:50:47.143 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-09-21T15:50:47.143 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-09-21T15:50:47.143 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-09-21T15:50:47.143 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-09-21T15:50:47.143 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-09-21T15:50:47.143 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-09-21T15:48:45.166209+0000", 2024-09-21T15:50:47.143 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-09-21T15:50:47.143 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:47.143 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-21T15:50:47.143 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_upmap": [], 2024-09-21T15:50:47.144 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_upmap_items": [ 2024-09-21T15:50:47.144 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.144 INFO:teuthology.orchestra.run.smithi089.stdout: "pgid": "3.8", 2024-09-21T15:50:47.144 INFO:teuthology.orchestra.run.smithi089.stdout: "mappings": [ 2024-09-21T15:50:47.144 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.144 INFO:teuthology.orchestra.run.smithi089.stdout: "from": 1, 2024-09-21T15:50:47.144 INFO:teuthology.orchestra.run.smithi089.stdout: "to": 6 2024-09-21T15:50:47.144 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:47.144 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:47.144 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:47.144 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-21T15:50:47.144 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_upmap_primaries": [], 2024-09-21T15:50:47.144 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_temp": [], 2024-09-21T15:50:47.144 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_temp": [], 2024-09-21T15:50:47.144 INFO:teuthology.orchestra.run.smithi089.stdout: "blocklist": {}, 2024-09-21T15:50:47.144 INFO:teuthology.orchestra.run.smithi089.stdout: "range_blocklist": {}, 2024-09-21T15:50:47.144 INFO:teuthology.orchestra.run.smithi089.stdout: "erasure_code_profiles": { 2024-09-21T15:50:47.145 INFO:teuthology.orchestra.run.smithi089.stdout: "default": { 2024-09-21T15:50:47.145 INFO:teuthology.orchestra.run.smithi089.stdout: "crush-failure-domain": "osd", 2024-09-21T15:50:47.145 INFO:teuthology.orchestra.run.smithi089.stdout: "k": "2", 2024-09-21T15:50:47.145 INFO:teuthology.orchestra.run.smithi089.stdout: "m": "1", 2024-09-21T15:50:47.145 INFO:teuthology.orchestra.run.smithi089.stdout: "plugin": "jerasure", 2024-09-21T15:50:47.145 INFO:teuthology.orchestra.run.smithi089.stdout: "technique": "reed_sol_van" 2024-09-21T15:50:47.145 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:47.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:47.145 INFO:teuthology.orchestra.run.smithi089.stdout: "removed_snaps_queue": [ 2024-09-21T15:50:47.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.145 INFO:teuthology.orchestra.run.smithi089.stdout: "pool": 3, 2024-09-21T15:50:47.145 INFO:teuthology.orchestra.run.smithi089.stdout: "snaps": [ 2024-09-21T15:50:47.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.145 INFO:teuthology.orchestra.run.smithi089.stdout: "begin": 32, 2024-09-21T15:50:47.145 INFO:teuthology.orchestra.run.smithi089.stdout: "length": 13 2024-09-21T15:50:47.145 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:47.146 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:47.146 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:47.146 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-21T15:50:47.146 INFO:teuthology.orchestra.run.smithi089.stdout: "new_removed_snaps": [ 2024-09-21T15:50:47.146 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.146 INFO:teuthology.orchestra.run.smithi089.stdout: "pool": 3, 2024-09-21T15:50:47.146 INFO:teuthology.orchestra.run.smithi089.stdout: "snaps": [ 2024-09-21T15:50:47.146 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:47.146 INFO:teuthology.orchestra.run.smithi089.stdout: "begin": 32, 2024-09-21T15:50:47.146 INFO:teuthology.orchestra.run.smithi089.stdout: "length": 1 2024-09-21T15:50:47.146 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:47.146 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:47.146 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:47.146 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-21T15:50:47.146 INFO:teuthology.orchestra.run.smithi089.stdout: "new_purged_snaps": [], 2024-09-21T15:50:47.146 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_node_flags": {}, 2024-09-21T15:50:47.147 INFO:teuthology.orchestra.run.smithi089.stdout: "device_class_flags": {}, 2024-09-21T15:50:47.147 INFO:teuthology.orchestra.run.smithi089.stdout: "stretch_mode": { 2024-09-21T15:50:47.147 INFO:teuthology.orchestra.run.smithi089.stdout: "stretch_mode_enabled": false, 2024-09-21T15:50:47.147 INFO:teuthology.orchestra.run.smithi089.stdout: "stretch_bucket_count": 0, 2024-09-21T15:50:47.147 INFO:teuthology.orchestra.run.smithi089.stdout: "degraded_stretch_mode": 0, 2024-09-21T15:50:47.147 INFO:teuthology.orchestra.run.smithi089.stdout: "recovering_stretch_mode": 0, 2024-09-21T15:50:47.147 INFO:teuthology.orchestra.run.smithi089.stdout: "stretch_mode_bucket": 0 2024-09-21T15:50:47.147 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:47.147 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:47.149 INFO:tasks.thrashosds.thrasher:j is {'epoch': 111, 'fsid': '4a9b64ea-3aaf-4a40-8ce8-02163a0f6e12', 'created': '2024-09-21T15:48:37.867501+0000', 'modified': '2024-09-21T15:50:46.739408+0000', 'last_up_change': '2024-09-21T15:50:29.130795+0000', 'last_in_change': '2024-09-21T15:48:40.108789+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T15:48:49.921079+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8622970581054688, 'average_primary_affinity': 0.8600000143051147, 'average_primary_affinity_weighted': 0.8600000143051147}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-21T15:48:55.812982+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 20, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2.990000009536743, 'score_stable': 2.990000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.630000114440918, 'raw_score_stable': 2.630000114440918, 'primary_affinity_weighted': 0.8622970581054688, 'average_primary_affinity': 0.8600000143051147, 'average_primary_affinity_weighted': 0.8600000143051147}}, {'pool': 3, 'pool_name': 'pool_client.0', 'create_time': '2024-09-21T15:49:05.523888+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '111', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 63, 'snap_epoch': 111, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8622970581054688, 'average_primary_affinity': 0.8600000143051147, 'average_primary_affinity_weighted': 0.8600000143051147}}], 'osds': [{'osd': 0, 'uuid': '7b3fabfa-3fe6-47c7-9d0c-a6036f738771', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 67, 'up_from': 93, 'up_thru': 96, 'down_at': 69, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6800', 'nonce': 1222314186}, {'type': 'v1', 'addr': '172.21.15.89:6801', 'nonce': 1222314186}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6803', 'nonce': 1222314186}, {'type': 'v1', 'addr': '172.21.15.89:6805', 'nonce': 1222314186}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6811', 'nonce': 1222314186}, {'type': 'v1', 'addr': '172.21.15.89:6813', 'nonce': 1222314186}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6807', 'nonce': 1222314186}, {'type': 'v1', 'addr': '172.21.15.89:6809', 'nonce': 1222314186}]}, 'public_addr': '172.21.15.89:6801/1222314186', 'cluster_addr': '172.21.15.89:6805/1222314186', 'heartbeat_back_addr': '172.21.15.89:6813/1222314186', 'heartbeat_front_addr': '172.21.15.89:6809/1222314186', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '767ad1bc-84ef-4102-83f5-aacda0917ddb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 99, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6816', 'nonce': 3187095367}, {'type': 'v1', 'addr': '172.21.15.89:6817', 'nonce': 3187095367}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6818', 'nonce': 3187095367}, {'type': 'v1', 'addr': '172.21.15.89:6819', 'nonce': 3187095367}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6822', 'nonce': 3187095367}, {'type': 'v1', 'addr': '172.21.15.89:6824', 'nonce': 3187095367}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6820', 'nonce': 3187095367}, {'type': 'v1', 'addr': '172.21.15.89:6821', 'nonce': 3187095367}]}, 'public_addr': '172.21.15.89:6817/3187095367', 'cluster_addr': '172.21.15.89:6819/3187095367', 'heartbeat_back_addr': '172.21.15.89:6824/3187095367', 'heartbeat_front_addr': '172.21.15.89:6821/3187095367', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '1dbd0d50-aae9-42a7-90a1-aa92697778ae', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 96, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6802', 'nonce': 4098625737}, {'type': 'v1', 'addr': '172.21.15.89:6804', 'nonce': 4098625737}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6806', 'nonce': 4098625737}, {'type': 'v1', 'addr': '172.21.15.89:6808', 'nonce': 4098625737}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6814', 'nonce': 4098625737}, {'type': 'v1', 'addr': '172.21.15.89:6815', 'nonce': 4098625737}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6810', 'nonce': 4098625737}, {'type': 'v1', 'addr': '172.21.15.89:6812', 'nonce': 4098625737}]}, 'public_addr': '172.21.15.89:6804/4098625737', 'cluster_addr': '172.21.15.89:6808/4098625737', 'heartbeat_back_addr': '172.21.15.89:6815/4098625737', 'heartbeat_front_addr': '172.21.15.89:6812/4098625737', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'f37cd295-2bac-4df4-a99b-abedc0978dce', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.6057586669921875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 95, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6823', 'nonce': 1886828410}, {'type': 'v1', 'addr': '172.21.15.89:6825', 'nonce': 1886828410}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6826', 'nonce': 1886828410}, {'type': 'v1', 'addr': '172.21.15.89:6827', 'nonce': 1886828410}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6830', 'nonce': 1886828410}, {'type': 'v1', 'addr': '172.21.15.89:6831', 'nonce': 1886828410}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6828', 'nonce': 1886828410}, {'type': 'v1', 'addr': '172.21.15.89:6829', 'nonce': 1886828410}]}, 'public_addr': '172.21.15.89:6825/1886828410', 'cluster_addr': '172.21.15.89:6827/1886828410', 'heartbeat_back_addr': '172.21.15.89:6831/1886828410', 'heartbeat_front_addr': '172.21.15.89:6829/1886828410', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '2eeb351d-d51c-4fc3-884e-debecd995aea', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 96, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6817', 'nonce': 22722842}, {'type': 'v1', 'addr': '172.21.15.100:6819', 'nonce': 22722842}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6820', 'nonce': 22722842}, {'type': 'v1', 'addr': '172.21.15.100:6822', 'nonce': 22722842}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6828', 'nonce': 22722842}, {'type': 'v1', 'addr': '172.21.15.100:6830', 'nonce': 22722842}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6824', 'nonce': 22722842}, {'type': 'v1', 'addr': '172.21.15.100:6826', 'nonce': 22722842}]}, 'public_addr': '172.21.15.100:6819/22722842', 'cluster_addr': '172.21.15.100:6822/22722842', 'heartbeat_back_addr': '172.21.15.100:6830/22722842', 'heartbeat_front_addr': '172.21.15.100:6826/22722842', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'b5396267-6823-42f1-bd04-bd8f144fa76c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.2926177978515625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 95, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6816', 'nonce': 363733545}, {'type': 'v1', 'addr': '172.21.15.100:6818', 'nonce': 363733545}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6821', 'nonce': 363733545}, {'type': 'v1', 'addr': '172.21.15.100:6823', 'nonce': 363733545}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6829', 'nonce': 363733545}, {'type': 'v1', 'addr': '172.21.15.100:6831', 'nonce': 363733545}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6825', 'nonce': 363733545}, {'type': 'v1', 'addr': '172.21.15.100:6827', 'nonce': 363733545}]}, 'public_addr': '172.21.15.100:6818/363733545', 'cluster_addr': '172.21.15.100:6823/363733545', 'heartbeat_back_addr': '172.21.15.100:6831/363733545', 'heartbeat_front_addr': '172.21.15.100:6827/363733545', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '955b67e9-73bc-42c1-88e0-3356c105e1ea', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 96, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6808', 'nonce': 205694879}, {'type': 'v1', 'addr': '172.21.15.100:6809', 'nonce': 205694879}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6810', 'nonce': 205694879}, {'type': 'v1', 'addr': '172.21.15.100:6811', 'nonce': 205694879}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6814', 'nonce': 205694879}, {'type': 'v1', 'addr': '172.21.15.100:6815', 'nonce': 205694879}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6812', 'nonce': 205694879}, {'type': 'v1', 'addr': '172.21.15.100:6813', 'nonce': 205694879}]}, 'public_addr': '172.21.15.100:6809/205694879', 'cluster_addr': '172.21.15.100:6811/205694879', 'heartbeat_back_addr': '172.21.15.100:6815/205694879', 'heartbeat_front_addr': '172.21.15.100:6813/205694879', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '414ea7f0-138d-404e-bce6-fe8dedcdc938', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 95, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6800', 'nonce': 3120011840}, {'type': 'v1', 'addr': '172.21.15.100:6801', 'nonce': 3120011840}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6802', 'nonce': 3120011840}, {'type': 'v1', 'addr': '172.21.15.100:6803', 'nonce': 3120011840}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6806', 'nonce': 3120011840}, {'type': 'v1', 'addr': '172.21.15.100:6807', 'nonce': 3120011840}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6804', 'nonce': 3120011840}, {'type': 'v1', 'addr': '172.21.15.100:6805', 'nonce': 3120011840}]}, 'public_addr': '172.21.15.100:6801/3120011840', 'cluster_addr': '172.21.15.100:6803/3120011840', 'heartbeat_back_addr': '172.21.15.100:6807/3120011840', 'heartbeat_front_addr': '172.21.15.100:6805/3120011840', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:48:45.230746+0000', 'dead_epoch': 69}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:48:45.303567+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:48:45.210941+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:48:45.218918+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:48:45.301781+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:48:45.248444+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:48:45.275465+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:48:45.166209+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '3.8', 'mappings': [{'from': 1, 'to': 6}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 3, 'snaps': [{'begin': 32, 'length': 13}]}], 'new_removed_snaps': [{'pool': 3, 'snaps': [{'begin': 32, 'length': 1}]}], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-21T15:50:47.149 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 3.8 2024-09-21T15:50:47.149 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd rm-pg-upmap-items 3.8 2024-09-21T15:50:47.150 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:47.150 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:47.150 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:50:47.151 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:50:47.147+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:50:47.249 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:47.248+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:47.293 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:47.294 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:47.303 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:50:47.350 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:47.349+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:47.410 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:47.410 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:47.418 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:50:47.462 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:47.461+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:47.520 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:47.520 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:47.528 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:50:47.570 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:47.569+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:47.652 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:47.651+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:47.670 INFO:teuthology.orchestra.run.smithi089.stderr:clear 3.8 pg_upmap_items mapping 2024-09-21T15:50:47.676 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:47.676 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:47.687 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:50:47.753 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:47.752+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:47.754 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T15:50:47.832 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:47.832 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:47.840 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:50:47.854 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:47.853+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:47.942 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:47.942 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:47.951 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:50:47.994 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:47.993+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:48.053 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:48.053 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:48.055 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:50:48.054+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:50:48.061 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T15:50:48.157 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:50:48.156+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:50:48.204 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:48.204 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:48.213 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T15:50:48.258 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:48.257+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:48.298 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:50:48.314 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:48.314 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:48.322 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T15:50:48.363 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:48.362+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:48.416 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-09-21T15:50:48.416 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:50:48.432 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:50:48.432 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:50:48.447 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:50:48.448 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:50:48.459 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:48.458+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:48.462 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:50:48.463 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:50:48.464 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:48.464 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:48.472 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T15:50:48.478 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:50:48.478 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:50:48.494 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:50:48.495 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:50:48.511 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:50:48.511 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:50:48.526 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:50:48.527 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:50:48.560 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:48.559+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:48.616 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:48.616 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:48.616 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:50:48.616 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:48.624 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T15:50:48.665 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:48.664+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:48.670 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is unset 2024-09-21T15:50:48.761 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:48.760+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:48.767 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:48.767 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:48.767 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:50:48.767 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:50:48.767 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:48.776 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T15:50:48.862 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:48.861+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:48.922 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:48.922 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:50:48.922 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:50:48.922 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:50:48.922 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.923 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4209.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:50:48.923 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:50:48.923 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 117.729043897, 2024-09-21T15:50:48.923 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.19452909900000001, 2024-09-21T15:50:48.923 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:48.923 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:48.923 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:48.923 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4209", 2024-09-21T15:50:48.923 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/4045143050", 2024-09-21T15:50:48.923 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1 2024-09-21T15:50:48.923 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.923 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:48.923 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.923 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:48.924 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:50:48.924 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:48.924 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.924 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.924 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:48.924 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:50:48.924 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:48.924 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.924 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.924 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:48.924 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191161+0000", 2024-09-21T15:50:48.924 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-21T15:50:48.924 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.924 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.924 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:48.925 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191169+0000", 2024-09-21T15:50:48.925 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0090000000000007e-06 2024-09-21T15:50:48.925 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.925 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.925 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:48.925 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191180+0000", 2024-09-21T15:50:48.925 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0728e-05 2024-09-21T15:50:48.925 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.925 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.925 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:48.925 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191187+0000", 2024-09-21T15:50:48.925 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9009999999999994e-06 2024-09-21T15:50:48.925 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.925 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.925 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:48.925 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191238+0000", 2024-09-21T15:50:48.926 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0287000000000002e-05 2024-09-21T15:50:48.926 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.926 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.926 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-21T15:50:48.926 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191245+0000", 2024-09-21T15:50:48.926 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0709999999999997e-06 2024-09-21T15:50:48.926 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.926 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.926 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:48.926 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380309+0000", 2024-09-21T15:50:48.926 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.189064277 2024-09-21T15:50:48.926 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.926 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.926 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:48.927 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380454+0000", 2024-09-21T15:50:48.927 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014490799999999999 2024-09-21T15:50:48.927 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.927 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.927 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0", 2024-09-21T15:50:48.927 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380610+0000", 2024-09-21T15:50:48.927 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015650199999999999 2024-09-21T15:50:48.927 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.927 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.927 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:50:48.927 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.381160+0000", 2024-09-21T15:50:48.927 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00054903799999999996 2024-09-21T15:50:48.927 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.927 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.928 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:48.928 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385591+0000", 2024-09-21T15:50:48.928 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.004431272 2024-09-21T15:50:48.928 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.928 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.928 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:50:48.928 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385651+0000", 2024-09-21T15:50:48.928 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0455999999999999e-05 2024-09-21T15:50:48.928 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.928 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.928 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:48.928 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385691+0000", 2024-09-21T15:50:48.928 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0234999999999997e-05 2024-09-21T15:50:48.928 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:48.928 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:48.929 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:48.929 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.929 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.929 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64034 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1083392~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:48.929 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167717+0000", 2024-09-21T15:50:48.929 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 71.752489566999998, 2024-09-21T15:50:48.929 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048233316999999998, 2024-09-21T15:50:48.929 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:48.929 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:48.929 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:48.929 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:48.929 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:48.929 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64034 2024-09-21T15:50:48.929 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.929 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:48.930 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.930 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:48.930 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167717+0000", 2024-09-21T15:50:48.930 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:48.930 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.930 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.930 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:48.930 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167717+0000", 2024-09-21T15:50:48.930 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:48.930 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.930 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.930 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:48.930 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167716+0000", 2024-09-21T15:50:48.930 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:48.930 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.931 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.931 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:48.931 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167718+0000", 2024-09-21T15:50:48.931 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.165e-06 2024-09-21T15:50:48.931 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.931 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.931 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:48.931 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167725+0000", 2024-09-21T15:50:48.931 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3380000000000001e-06 2024-09-21T15:50:48.931 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.931 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.931 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:48.931 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167730+0000", 2024-09-21T15:50:48.931 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8010000000000003e-06 2024-09-21T15:50:48.931 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.931 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.932 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:48.932 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215807+0000", 2024-09-21T15:50:48.932 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048077823999999998 2024-09-21T15:50:48.932 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.932 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.932 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:48.932 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215884+0000", 2024-09-21T15:50:48.932 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6283000000000001e-05 2024-09-21T15:50:48.932 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.932 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.932 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:48.932 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215950+0000", 2024-09-21T15:50:48.932 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6290000000000004e-05 2024-09-21T15:50:48.932 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:48.932 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:48.932 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:48.933 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.933 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.933 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64035 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1091584~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:48.933 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:50:48.933 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 71.752472354000005, 2024-09-21T15:50:48.933 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048412117999999997, 2024-09-21T15:50:48.933 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:48.933 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:48.933 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:48.933 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:48.933 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:48.933 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64035 2024-09-21T15:50:48.933 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.933 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:48.933 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.933 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:48.934 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:50:48.934 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:48.934 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.934 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.934 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:48.934 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:50:48.934 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:48.934 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.934 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.934 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:48.934 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:50:48.934 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:48.934 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.934 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.934 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:48.935 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167746+0000", 2024-09-21T15:50:48.935 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2031000000000001e-05 2024-09-21T15:50:48.935 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.935 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.935 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:48.935 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167753+0000", 2024-09-21T15:50:48.935 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0620000000000003e-06 2024-09-21T15:50:48.935 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.935 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.935 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:48.935 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167759+0000", 2024-09-21T15:50:48.935 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.596e-06 2024-09-21T15:50:48.935 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.935 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.935 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:48.936 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215977+0000", 2024-09-21T15:50:48.936 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048217318000000002 2024-09-21T15:50:48.936 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.936 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.936 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:48.936 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216076+0000", 2024-09-21T15:50:48.936 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9846000000000006e-05 2024-09-21T15:50:48.936 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.936 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.936 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:48.936 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216146+0000", 2024-09-21T15:50:48.936 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9578000000000004e-05 2024-09-21T15:50:48.936 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:48.936 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:48.936 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:48.937 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.937 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.937 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64036 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1099776~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:48.937 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167853+0000", 2024-09-21T15:50:48.937 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 71.752353740999993, 2024-09-21T15:50:48.937 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048450675999999998, 2024-09-21T15:50:48.937 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:48.937 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:48.937 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:48.937 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:48.937 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:48.937 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64036 2024-09-21T15:50:48.937 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.937 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:48.938 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.938 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:48.938 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167853+0000", 2024-09-21T15:50:48.938 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:48.938 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.938 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.938 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:48.938 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167853+0000", 2024-09-21T15:50:48.938 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:48.938 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.938 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.938 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:48.938 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167852+0000", 2024-09-21T15:50:48.938 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:48.938 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.939 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.939 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:48.939 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167855+0000", 2024-09-21T15:50:48.939 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5640000000000001e-06 2024-09-21T15:50:48.939 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.939 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.939 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:48.939 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167864+0000", 2024-09-21T15:50:48.939 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8140000000000007e-06 2024-09-21T15:50:48.939 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.939 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.939 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:48.939 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167871+0000", 2024-09-21T15:50:48.939 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5729999999999996e-06 2024-09-21T15:50:48.939 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.940 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.940 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:48.940 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216174+0000", 2024-09-21T15:50:48.940 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048303437999999997 2024-09-21T15:50:48.940 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.940 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.940 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:48.940 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216250+0000", 2024-09-21T15:50:48.940 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5661999999999995e-05 2024-09-21T15:50:48.940 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.940 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.940 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:48.940 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216303+0000", 2024-09-21T15:50:48.940 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3273000000000003e-05 2024-09-21T15:50:48.940 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:48.941 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:48.941 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:48.941 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.941 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.941 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64037 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1771520~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:48.941 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:50:48.941 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 71.752330197999996, 2024-09-21T15:50:48.941 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048588771000000003, 2024-09-21T15:50:48.941 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:48.941 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:48.941 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:48.941 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:48.941 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:48.941 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64037 2024-09-21T15:50:48.941 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.942 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:48.942 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.942 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:48.942 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:50:48.942 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:48.942 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.942 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.942 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:48.942 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:50:48.942 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:48.942 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.942 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.942 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:48.942 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:50:48.943 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:48.943 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.943 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.943 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:48.943 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167878+0000", 2024-09-21T15:50:48.943 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3e-06 2024-09-21T15:50:48.943 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.943 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.943 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:48.943 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167884+0000", 2024-09-21T15:50:48.943 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3269999999999997e-06 2024-09-21T15:50:48.943 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.943 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.943 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:48.943 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167889+0000", 2024-09-21T15:50:48.944 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9269999999999996e-06 2024-09-21T15:50:48.944 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.944 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.944 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:48.944 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216331+0000", 2024-09-21T15:50:48.944 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048442102000000001 2024-09-21T15:50:48.944 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.944 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.944 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:48.944 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216393+0000", 2024-09-21T15:50:48.944 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1132999999999999e-05 2024-09-21T15:50:48.944 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.944 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.944 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:48.944 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216465+0000", 2024-09-21T15:50:48.945 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2339e-05 2024-09-21T15:50:48.945 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:48.945 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:48.945 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:48.945 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.945 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.945 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64038 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1779712~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:48.945 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:50:48.945 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 71.752312367000002, 2024-09-21T15:50:48.945 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048749795999999998, 2024-09-21T15:50:48.945 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:48.945 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:48.945 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:48.945 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:48.945 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:48.946 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64038 2024-09-21T15:50:48.946 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.946 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:48.946 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.946 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:48.946 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:50:48.946 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:48.946 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.946 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.946 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:48.946 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:50:48.946 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:48.946 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.946 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.947 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:48.947 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:50:48.947 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:48.947 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.947 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.947 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:48.947 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167896+0000", 2024-09-21T15:50:48.947 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2419999999999999e-06 2024-09-21T15:50:48.947 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.947 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.947 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:48.947 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167902+0000", 2024-09-21T15:50:48.947 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0859999999999998e-06 2024-09-21T15:50:48.947 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.948 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.948 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:48.948 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167907+0000", 2024-09-21T15:50:48.948 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9679999999999997e-06 2024-09-21T15:50:48.948 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.948 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.948 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:48.948 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216506+0000", 2024-09-21T15:50:48.948 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048599374000000001 2024-09-21T15:50:48.948 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.948 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.948 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:48.948 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216579+0000", 2024-09-21T15:50:48.948 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2801999999999999e-05 2024-09-21T15:50:48.949 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.949 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.949 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:48.949 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216644+0000", 2024-09-21T15:50:48.949 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4715000000000006e-05 2024-09-21T15:50:48.949 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:48.949 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:48.949 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:48.949 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.949 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.949 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64039 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1107968~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:48.949 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:50:48.949 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 71.752295012000005, 2024-09-21T15:50:48.950 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048911370000000003, 2024-09-21T15:50:48.950 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:48.950 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:48.950 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:48.950 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:48.950 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:48.950 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64039 2024-09-21T15:50:48.950 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.950 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:48.950 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.950 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:48.950 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:50:48.950 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:48.950 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.951 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.951 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:48.951 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:50:48.951 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:48.951 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.951 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.951 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:48.951 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:50:48.951 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:48.951 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.951 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.951 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:48.951 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167913+0000", 2024-09-21T15:50:48.951 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0839999999999999e-06 2024-09-21T15:50:48.951 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.952 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.952 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:48.952 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167921+0000", 2024-09-21T15:50:48.952 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8310000000000001e-06 2024-09-21T15:50:48.952 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.952 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.952 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:48.952 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167926+0000", 2024-09-21T15:50:48.952 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0270000000000003e-06 2024-09-21T15:50:48.952 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.952 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.952 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:48.952 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216680+0000", 2024-09-21T15:50:48.952 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048753890000000001 2024-09-21T15:50:48.952 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.953 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.953 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:48.953 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216757+0000", 2024-09-21T15:50:48.953 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6848999999999996e-05 2024-09-21T15:50:48.953 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.953 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.953 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:48.953 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216823+0000", 2024-09-21T15:50:48.953 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6068000000000005e-05 2024-09-21T15:50:48.953 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:48.953 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:48.953 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:48.953 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.953 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.953 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64040 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1787904~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:48.953 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167931+0000", 2024-09-21T15:50:48.954 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 71.752275780000005, 2024-09-21T15:50:48.954 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049010561000000001, 2024-09-21T15:50:48.954 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:48.954 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:48.954 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:48.954 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:48.954 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:48.954 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64040 2024-09-21T15:50:48.954 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.954 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:48.954 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.954 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:48.954 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167931+0000", 2024-09-21T15:50:48.954 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:48.955 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.955 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.955 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:48.955 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167931+0000", 2024-09-21T15:50:48.955 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:48.955 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.955 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.955 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:48.955 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167930+0000", 2024-09-21T15:50:48.955 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:48.955 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.955 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.955 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:48.955 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167932+0000", 2024-09-21T15:50:48.955 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.198e-06 2024-09-21T15:50:48.956 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.956 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.956 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:48.956 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167938+0000", 2024-09-21T15:50:48.956 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.105e-06 2024-09-21T15:50:48.956 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.956 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.956 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:48.956 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167943+0000", 2024-09-21T15:50:48.956 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9429999999999999e-06 2024-09-21T15:50:48.956 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.956 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.956 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:48.956 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216848+0000", 2024-09-21T15:50:48.957 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048905200000000003 2024-09-21T15:50:48.957 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.957 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.957 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:48.957 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216903+0000", 2024-09-21T15:50:48.957 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4793999999999997e-05 2024-09-21T15:50:48.957 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.957 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.957 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:48.957 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216941+0000", 2024-09-21T15:50:48.957 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7790999999999998e-05 2024-09-21T15:50:48.957 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:48.957 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:48.957 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:48.957 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.957 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.958 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64041 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1796096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:48.958 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:50:48.958 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 71.752258808999997, 2024-09-21T15:50:48.958 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049181457999999997, 2024-09-21T15:50:48.958 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:48.958 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:48.958 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:48.958 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:48.958 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:48.958 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64041 2024-09-21T15:50:48.958 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.958 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:48.958 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.958 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:48.959 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:50:48.959 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:48.959 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.959 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.959 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:48.959 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:50:48.959 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:48.959 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.959 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.959 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:48.959 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:50:48.959 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:48.959 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.959 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.959 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:48.960 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167949+0000", 2024-09-21T15:50:48.960 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0200000000000001e-06 2024-09-21T15:50:48.960 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.960 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.960 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:48.960 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167955+0000", 2024-09-21T15:50:48.960 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5990000000000001e-06 2024-09-21T15:50:48.960 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.960 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.960 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:48.960 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167959+0000", 2024-09-21T15:50:48.960 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5589999999999999e-06 2024-09-21T15:50:48.960 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.960 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.961 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:48.961 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216972+0000", 2024-09-21T15:50:48.961 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049012574000000003 2024-09-21T15:50:48.961 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.961 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.961 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:48.961 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217057+0000", 2024-09-21T15:50:48.961 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4820000000000004e-05 2024-09-21T15:50:48.961 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.961 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.961 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:48.961 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217129+0000", 2024-09-21T15:50:48.961 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2186000000000003e-05 2024-09-21T15:50:48.961 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:48.961 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:48.962 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:48.962 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.962 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.962 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64042 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1804288~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:48.962 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167964+0000", 2024-09-21T15:50:48.962 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 71.752242518000003, 2024-09-21T15:50:48.962 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049333372, 2024-09-21T15:50:48.962 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:48.962 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:48.962 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:48.962 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:48.962 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:48.962 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64042 2024-09-21T15:50:48.962 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.963 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:48.963 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.963 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:48.963 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167964+0000", 2024-09-21T15:50:48.963 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:48.963 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.963 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.963 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:48.963 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167964+0000", 2024-09-21T15:50:48.963 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:48.963 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.963 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.963 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:48.963 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167963+0000", 2024-09-21T15:50:48.963 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:48.964 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.964 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.964 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:48.964 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167966+0000", 2024-09-21T15:50:48.964 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1119999999999999e-06 2024-09-21T15:50:48.964 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.964 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.964 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:48.964 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167975+0000", 2024-09-21T15:50:48.964 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1130000000000007e-06 2024-09-21T15:50:48.964 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.964 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.964 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:48.965 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167980+0000", 2024-09-21T15:50:48.965 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9599999999999999e-06 2024-09-21T15:50:48.965 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.965 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.965 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:48.965 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217158+0000", 2024-09-21T15:50:48.965 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049178569999999998 2024-09-21T15:50:48.965 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.965 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.965 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:48.965 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217232+0000", 2024-09-21T15:50:48.965 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3589000000000003e-05 2024-09-21T15:50:48.965 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.965 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.965 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:48.965 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217297+0000", 2024-09-21T15:50:48.966 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5339999999999994e-05 2024-09-21T15:50:48.966 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:48.966 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:48.966 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:48.966 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.966 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.966 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64043 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1812480~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:48.966 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.168085+0000", 2024-09-21T15:50:48.966 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 71.752121450999994, 2024-09-21T15:50:48.966 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049365157999999999, 2024-09-21T15:50:48.966 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:48.966 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:48.966 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:48.966 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:48.966 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:48.967 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64043 2024-09-21T15:50:48.967 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.967 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:48.967 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.967 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:48.967 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168085+0000", 2024-09-21T15:50:48.967 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:48.967 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.967 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.967 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:48.967 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168085+0000", 2024-09-21T15:50:48.967 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:48.967 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.967 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.967 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:48.967 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168084+0000", 2024-09-21T15:50:48.968 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:48.968 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.968 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.968 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:48.968 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168088+0000", 2024-09-21T15:50:48.968 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5420000000000001e-06 2024-09-21T15:50:48.968 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.968 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.968 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:48.968 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168097+0000", 2024-09-21T15:50:48.968 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5410000000000006e-06 2024-09-21T15:50:48.968 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.968 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.969 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:48.969 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168104+0000", 2024-09-21T15:50:48.969 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6719999999999998e-06 2024-09-21T15:50:48.969 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.969 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.969 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:48.969 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217323+0000", 2024-09-21T15:50:48.969 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049218857999999997 2024-09-21T15:50:48.969 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.969 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.969 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:48.969 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217397+0000", 2024-09-21T15:50:48.969 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3932999999999998e-05 2024-09-21T15:50:48.969 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.970 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.970 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:48.970 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217450+0000", 2024-09-21T15:50:48.970 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3168000000000003e-05 2024-09-21T15:50:48.970 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:48.970 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:48.970 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:48.970 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.970 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.970 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155255 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 323584~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:48.970 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.440996+0000", 2024-09-21T15:50:48.970 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 26.479209923999999, 2024-09-21T15:50:48.970 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048233848000000003, 2024-09-21T15:50:48.971 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:48.971 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:48.971 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:48.971 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:48.971 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:48.971 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155255 2024-09-21T15:50:48.971 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.971 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:48.971 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.971 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:48.971 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.440996+0000", 2024-09-21T15:50:48.971 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:48.971 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.971 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.971 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:48.971 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.440996+0000", 2024-09-21T15:50:48.972 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:48.972 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.972 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.972 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:48.972 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.440996+0000", 2024-09-21T15:50:48.972 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:48.972 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.972 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.972 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:48.972 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441004+0000", 2024-09-21T15:50:48.972 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8460000000000008e-06 2024-09-21T15:50:48.972 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.972 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.972 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:48.972 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441018+0000", 2024-09-21T15:50:48.972 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.42e-05 2024-09-21T15:50:48.973 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.973 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.973 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:48.973 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441023+0000", 2024-09-21T15:50:48.973 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0019999999999997e-06 2024-09-21T15:50:48.973 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.973 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.973 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:48.973 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489060+0000", 2024-09-21T15:50:48.973 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048036660000000002 2024-09-21T15:50:48.973 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.973 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.973 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:48.973 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489107+0000", 2024-09-21T15:50:48.973 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7153000000000001e-05 2024-09-21T15:50:48.974 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.974 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.974 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:48.974 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489230+0000", 2024-09-21T15:50:48.974 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012328600000000001 2024-09-21T15:50:48.974 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:48.974 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:48.974 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:48.974 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.974 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.974 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155256 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 356352~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:48.974 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441027+0000", 2024-09-21T15:50:48.974 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 26.479178836999999, 2024-09-21T15:50:48.974 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048677606999999998, 2024-09-21T15:50:48.975 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:48.975 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:48.975 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:48.975 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:48.975 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:48.975 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155256 2024-09-21T15:50:48.975 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.975 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:48.975 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.975 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:48.975 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441027+0000", 2024-09-21T15:50:48.975 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:48.975 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.975 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.975 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:48.976 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441027+0000", 2024-09-21T15:50:48.976 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:48.976 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.976 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.976 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:48.976 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441027+0000", 2024-09-21T15:50:48.977 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:48.977 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.977 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.978 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:48.978 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441029+0000", 2024-09-21T15:50:48.978 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.39e-06 2024-09-21T15:50:48.978 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.978 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.978 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:48.978 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441036+0000", 2024-09-21T15:50:48.978 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1129999999999997e-06 2024-09-21T15:50:48.978 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.978 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.978 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:48.978 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441040+0000", 2024-09-21T15:50:48.978 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8459999999999999e-06 2024-09-21T15:50:48.978 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.979 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.979 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:48.979 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489244+0000", 2024-09-21T15:50:48.979 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048203751000000003 2024-09-21T15:50:48.979 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.979 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.979 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:48.979 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489288+0000", 2024-09-21T15:50:48.979 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4153999999999998e-05 2024-09-21T15:50:48.979 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.979 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.979 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:48.979 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489705+0000", 2024-09-21T15:50:48.979 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000416773 2024-09-21T15:50:48.979 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:48.980 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:48.980 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:48.980 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.980 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.980 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155257 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 389120~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:48.980 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441045+0000", 2024-09-21T15:50:48.980 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 26.479161745999999, 2024-09-21T15:50:48.980 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049000325999999997, 2024-09-21T15:50:48.980 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:48.980 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:48.980 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:48.980 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:48.980 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:48.980 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155257 2024-09-21T15:50:48.980 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.981 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:48.981 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.981 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:48.981 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441045+0000", 2024-09-21T15:50:48.981 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:48.981 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.981 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.981 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:48.981 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441045+0000", 2024-09-21T15:50:48.981 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:48.981 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.981 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.981 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:48.981 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441044+0000", 2024-09-21T15:50:48.981 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:48.982 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.982 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.982 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:48.982 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441046+0000", 2024-09-21T15:50:48.982 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1909999999999999e-06 2024-09-21T15:50:48.982 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.982 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.982 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:48.982 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441054+0000", 2024-09-21T15:50:48.982 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3900000000000004e-06 2024-09-21T15:50:48.982 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.982 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.982 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:48.983 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441058+0000", 2024-09-21T15:50:48.983 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4619999999999998e-06 2024-09-21T15:50:48.983 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.983 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.983 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:48.983 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489741+0000", 2024-09-21T15:50:48.983 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048682660000000003 2024-09-21T15:50:48.983 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.983 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.983 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:48.983 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489813+0000", 2024-09-21T15:50:48.983 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2051999999999994e-05 2024-09-21T15:50:48.983 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.983 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.983 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:48.984 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490045+0000", 2024-09-21T15:50:48.984 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00023191299999999999 2024-09-21T15:50:48.984 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:48.984 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:48.984 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:48.984 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.984 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.984 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155258 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 421888~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:48.984 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441062+0000", 2024-09-21T15:50:48.984 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 26.479144204000001, 2024-09-21T15:50:48.984 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049339651999999998, 2024-09-21T15:50:48.984 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:48.984 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:48.984 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:48.984 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:48.985 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:48.985 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155258 2024-09-21T15:50:48.985 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.985 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:48.985 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.985 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:48.985 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441062+0000", 2024-09-21T15:50:48.985 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:48.985 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.985 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.985 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:48.986 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441062+0000", 2024-09-21T15:50:48.986 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:48.986 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.986 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.986 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:48.986 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441062+0000", 2024-09-21T15:50:48.986 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:48.986 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.986 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.986 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:48.986 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441064+0000", 2024-09-21T15:50:48.986 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.967e-06 2024-09-21T15:50:48.986 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.986 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.987 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:48.987 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441069+0000", 2024-09-21T15:50:48.987 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1529999999999996e-06 2024-09-21T15:50:48.987 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.987 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.987 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:48.987 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441073+0000", 2024-09-21T15:50:48.987 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3850000000000004e-06 2024-09-21T15:50:48.987 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.987 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.987 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:48.987 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490082+0000", 2024-09-21T15:50:48.987 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.04900885 2024-09-21T15:50:48.987 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.988 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.988 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:48.988 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490166+0000", 2024-09-21T15:50:48.988 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3640999999999994e-05 2024-09-21T15:50:48.988 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.988 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.988 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:48.988 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490402+0000", 2024-09-21T15:50:48.988 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000235978 2024-09-21T15:50:48.988 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:48.988 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:48.988 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:48.988 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.988 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.989 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155259 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 454656~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:48.989 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441086+0000", 2024-09-21T15:50:48.989 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 26.479120398999999, 2024-09-21T15:50:48.989 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049624397000000001, 2024-09-21T15:50:48.989 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:48.989 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:48.989 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:48.989 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:48.989 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:48.989 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155259 2024-09-21T15:50:48.989 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.989 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:48.989 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.989 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:48.990 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441086+0000", 2024-09-21T15:50:48.990 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:48.990 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.990 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.990 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:48.990 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441086+0000", 2024-09-21T15:50:48.990 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:48.990 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.990 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.990 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:48.990 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441086+0000", 2024-09-21T15:50:48.990 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:48.990 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.990 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.991 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:48.991 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441088+0000", 2024-09-21T15:50:48.991 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2630000000000002e-06 2024-09-21T15:50:48.991 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.991 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.991 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:48.991 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441093+0000", 2024-09-21T15:50:48.991 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5010000000000004e-06 2024-09-21T15:50:48.991 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.991 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.991 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:48.991 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441098+0000", 2024-09-21T15:50:48.991 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.707e-06 2024-09-21T15:50:48.991 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.991 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.992 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:48.992 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490447+0000", 2024-09-21T15:50:48.992 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049349125000000001 2024-09-21T15:50:48.992 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.992 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.992 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:48.992 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490504+0000", 2024-09-21T15:50:48.992 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6521000000000003e-05 2024-09-21T15:50:48.992 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.992 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.992 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:48.992 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490710+0000", 2024-09-21T15:50:48.992 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00020663999999999999 2024-09-21T15:50:48.992 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:48.993 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:48.993 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:48.993 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.993 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.993 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155260 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 487424~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:48.993 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441160+0000", 2024-09-21T15:50:48.993 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 26.479046437000001, 2024-09-21T15:50:48.993 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049876810000000001, 2024-09-21T15:50:48.993 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:48.993 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:48.993 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:48.993 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:48.993 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:48.993 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155260 2024-09-21T15:50:48.993 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.994 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:48.994 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.994 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:48.994 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441160+0000", 2024-09-21T15:50:48.994 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:48.994 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.994 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.994 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:48.994 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441160+0000", 2024-09-21T15:50:48.994 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:48.994 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.994 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.994 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:48.995 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441159+0000", 2024-09-21T15:50:48.995 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:48.995 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.995 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.995 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:48.995 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441164+0000", 2024-09-21T15:50:48.995 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4059999999999998e-06 2024-09-21T15:50:48.995 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.995 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.995 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:48.995 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441175+0000", 2024-09-21T15:50:48.995 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1095e-05 2024-09-21T15:50:48.995 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.995 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.996 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:48.996 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441182+0000", 2024-09-21T15:50:48.996 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2139999999999999e-06 2024-09-21T15:50:48.996 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.996 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.996 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:48.996 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490751+0000", 2024-09-21T15:50:48.996 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049569230999999998 2024-09-21T15:50:48.996 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.996 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.996 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:48.996 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490837+0000", 2024-09-21T15:50:48.996 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6017e-05 2024-09-21T15:50:48.996 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.996 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.997 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:48.997 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491037+0000", 2024-09-21T15:50:48.997 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00019952300000000001 2024-09-21T15:50:48.997 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:48.997 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:48.997 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:48.997 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.997 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.997 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155261 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 520192~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:48.997 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441187+0000", 2024-09-21T15:50:48.997 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 26.479019254000001, 2024-09-21T15:50:48.997 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.050124931999999997, 2024-09-21T15:50:48.997 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:48.998 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:48.998 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:48.998 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:48.998 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:48.998 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155261 2024-09-21T15:50:48.998 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.998 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:48.998 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.998 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:48.998 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441187+0000", 2024-09-21T15:50:48.998 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:48.998 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.998 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.998 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:48.999 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441187+0000", 2024-09-21T15:50:48.999 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:48.999 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.999 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.999 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:48.999 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441187+0000", 2024-09-21T15:50:48.999 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:48.999 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.999 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.999 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:48.999 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441189+0000", 2024-09-21T15:50:48.999 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3649999999999998e-06 2024-09-21T15:50:48.999 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:48.999 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:48.999 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:49.000 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441196+0000", 2024-09-21T15:50:49.000 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5379999999999999e-06 2024-09-21T15:50:49.000 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:49.000 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:49.000 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:49.000 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441200+0000", 2024-09-21T15:50:49.000 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4719999999999997e-06 2024-09-21T15:50:49.000 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:49.000 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:49.000 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:49.000 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491070+0000", 2024-09-21T15:50:49.000 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049869915000000001 2024-09-21T15:50:49.000 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:49.000 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:49.000 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:49.001 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491146+0000", 2024-09-21T15:50:49.001 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6589999999999997e-05 2024-09-21T15:50:49.001 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:49.001 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:49.001 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:49.001 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491312+0000", 2024-09-21T15:50:49.001 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000165475 2024-09-21T15:50:49.001 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:49.001 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:49.001 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:49.001 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:49.001 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:49.001 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155308 3.11 3:8b5183e5:::rbd_data.11335aef9442.0000000000000000:2a [read 356352~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:49.001 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441369+0000", 2024-09-21T15:50:49.002 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 26.478837406, 2024-09-21T15:50:49.002 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.050090672000000003, 2024-09-21T15:50:49.002 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:49.002 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:49.002 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:49.002 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:49.002 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:49.002 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155308 2024-09-21T15:50:49.002 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:49.002 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:49.002 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:49.002 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:49.002 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441369+0000", 2024-09-21T15:50:49.003 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:49.003 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:49.003 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:49.003 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:49.003 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441369+0000", 2024-09-21T15:50:49.003 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:49.003 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:49.003 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:49.003 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:49.003 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441368+0000", 2024-09-21T15:50:49.003 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:49.003 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:49.003 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:49.003 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:49.004 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441371+0000", 2024-09-21T15:50:49.004 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1100000000000001e-06 2024-09-21T15:50:49.004 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:49.004 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:49.004 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:49.004 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441377+0000", 2024-09-21T15:50:49.004 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5649999999999998e-06 2024-09-21T15:50:49.004 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:49.004 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:49.004 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:49.004 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441382+0000", 2024-09-21T15:50:49.005 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5929999999999999e-06 2024-09-21T15:50:49.005 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:49.005 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:49.005 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:49.005 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491349+0000", 2024-09-21T15:50:49.005 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049967122000000003 2024-09-21T15:50:49.005 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:49.005 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:49.005 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:49.005 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491400+0000", 2024-09-21T15:50:49.005 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1589999999999999e-05 2024-09-21T15:50:49.005 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:49.005 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:49.006 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:49.006 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491460+0000", 2024-09-21T15:50:49.006 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9070000000000002e-05 2024-09-21T15:50:49.006 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:49.006 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:49.006 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:49.006 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:49.006 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:49.006 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155309 3.11 3:8b5183e5:::rbd_data.11335aef9442.0000000000000000:2a [read 389120~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:49.006 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441386+0000", 2024-09-21T15:50:49.006 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 26.478820057, 2024-09-21T15:50:49.006 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.050184777, 2024-09-21T15:50:49.006 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:49.006 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:49.007 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:49.007 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:49.007 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:49.007 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155309 2024-09-21T15:50:49.007 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:49.007 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:49.007 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:49.007 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:49.007 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441386+0000", 2024-09-21T15:50:49.007 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:49.007 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:49.007 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:49.007 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:49.007 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441386+0000", 2024-09-21T15:50:49.007 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:49.008 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:49.008 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:49.008 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:49.008 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441386+0000", 2024-09-21T15:50:49.008 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:49.008 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:49.008 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:49.008 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:49.008 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441388+0000", 2024-09-21T15:50:49.008 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4420000000000001e-06 2024-09-21T15:50:49.008 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:49.008 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:49.008 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:49.008 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441395+0000", 2024-09-21T15:50:49.009 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6189999999999996e-06 2024-09-21T15:50:49.009 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:49.009 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:49.009 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:49.009 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441400+0000", 2024-09-21T15:50:49.009 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.118e-06 2024-09-21T15:50:49.009 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:49.009 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:49.009 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:49.009 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491473+0000", 2024-09-21T15:50:49.009 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.050072783000000003 2024-09-21T15:50:49.009 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:49.009 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:49.009 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:49.009 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491518+0000", 2024-09-21T15:50:49.010 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4854000000000001e-05 2024-09-21T15:50:49.010 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:49.010 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:49.010 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:49.010 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491571+0000", 2024-09-21T15:50:49.010 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3358e-05 2024-09-21T15:50:49.010 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:49.010 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:49.010 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:49.010 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:49.010 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:49.010 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:49.012 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:50:49.012 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:49.010+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:49.112 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:49.111+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:49.114 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:49.114 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:49.115 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:50:49.115 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:49.123 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:50:49.213 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:49.212+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:49.225 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:49.226 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:49.226 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:50:49.226 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:50:49.226 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:49.234 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:50:49.315 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:49.313+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:49.377 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:49.377 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:50:49.377 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:50:49.377 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:50:49.377 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.378 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160505 3.b 3:d297695c:::rbd_header.11338bae974a:head [watch reconnect cookie 140381478598448 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-21T15:50:49.378 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:29.138810+0000", 2024-09-21T15:50:49.378 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 20.236288198, 2024-09-21T15:50:49.378 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0172598319999999, 2024-09-21T15:50:49.378 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:49.378 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:49.378 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:49.378 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:49.378 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:49.378 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160505 2024-09-21T15:50:49.378 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.378 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:49.378 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.379 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:49.379 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138810+0000", 2024-09-21T15:50:49.379 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.379 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.379 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.379 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:49.379 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138810+0000", 2024-09-21T15:50:49.379 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.379 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.379 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.379 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:49.379 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138808+0000", 2024-09-21T15:50:49.379 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:50:49.379 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.380 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.380 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:49.380 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138815+0000", 2024-09-21T15:50:49.380 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6549999999999998e-06 2024-09-21T15:50:49.380 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.380 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.380 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:49.380 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138825+0000", 2024-09-21T15:50:49.380 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9790000000000005e-06 2024-09-21T15:50:49.380 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.380 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.380 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:49.380 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138834+0000", 2024-09-21T15:50:49.380 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5179999999999993e-06 2024-09-21T15:50:49.381 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.381 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.381 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:49.381 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.140973+0000", 2024-09-21T15:50:49.381 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0021392049999999999 2024-09-21T15:50:49.381 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.381 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.381 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:50:49.381 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.140981+0000", 2024-09-21T15:50:49.381 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8909999999999995e-06 2024-09-21T15:50:49.381 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.381 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.381 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:49.381 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.150457+0000", 2024-09-21T15:50:49.381 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.009475645 2024-09-21T15:50:49.382 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.382 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.382 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for missing object", 2024-09-21T15:50:49.382 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.150589+0000", 2024-09-21T15:50:49.382 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000131779 2024-09-21T15:50:49.382 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.382 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.382 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:49.382 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.151723+0000", 2024-09-21T15:50:49.382 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0011345979999999999 2024-09-21T15:50:49.382 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.382 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.382 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-09-21T15:50:49.383 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.151872+0000", 2024-09-21T15:50:49.383 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000149021 2024-09-21T15:50:49.383 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.383 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.383 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:49.383 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.152563+0000", 2024-09-21T15:50:49.383 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00069051500000000005 2024-09-21T15:50:49.383 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.383 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.383 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:49.383 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.152673+0000", 2024-09-21T15:50:49.383 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000109795 2024-09-21T15:50:49.383 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.383 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.383 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:49.384 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.156069+0000", 2024-09-21T15:50:49.384 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003396736 2024-09-21T15:50:49.384 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.384 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:49.384 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.384 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.384 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.384 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160506 3.b 3:d297695c:::rbd_header.11338bae974a:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-21T15:50:49.384 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:29.138844+0000", 2024-09-21T15:50:49.384 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 20.236253310999999, 2024-09-21T15:50:49.384 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0174592680000001, 2024-09-21T15:50:49.384 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:49.384 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:49.384 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:49.385 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:49.385 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:49.385 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160506 2024-09-21T15:50:49.385 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.385 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:49.385 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.385 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:49.385 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138844+0000", 2024-09-21T15:50:49.385 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.385 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.385 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.385 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:49.385 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138844+0000", 2024-09-21T15:50:49.385 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.385 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.386 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.386 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:49.386 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138844+0000", 2024-09-21T15:50:49.386 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:49.386 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.386 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.386 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:49.386 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138848+0000", 2024-09-21T15:50:49.386 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.76e-06 2024-09-21T15:50:49.386 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.386 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.386 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:49.386 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138854+0000", 2024-09-21T15:50:49.387 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3890000000000002e-06 2024-09-21T15:50:49.387 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.387 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.387 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:49.387 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138860+0000", 2024-09-21T15:50:49.387 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6409999999999999e-06 2024-09-21T15:50:49.387 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.387 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.387 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:49.387 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.141015+0000", 2024-09-21T15:50:49.387 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0021553779999999999 2024-09-21T15:50:49.387 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.387 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.387 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:50:49.388 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.141021+0000", 2024-09-21T15:50:49.388 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7629999999999997e-06 2024-09-21T15:50:49.388 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.388 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.388 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:49.388 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.150858+0000", 2024-09-21T15:50:49.388 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0098372200000001 2024-09-21T15:50:49.388 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.388 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.388 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for missing object", 2024-09-21T15:50:49.388 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.150899+0000", 2024-09-21T15:50:49.388 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1112000000000002e-05 2024-09-21T15:50:49.388 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.388 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.389 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:49.389 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.152069+0000", 2024-09-21T15:50:49.389 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001170307 2024-09-21T15:50:49.389 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.389 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.389 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-09-21T15:50:49.389 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.152192+0000", 2024-09-21T15:50:49.389 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012289700000000001 2024-09-21T15:50:49.389 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.389 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.389 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:49.389 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.153362+0000", 2024-09-21T15:50:49.389 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0011700549999999999 2024-09-21T15:50:49.389 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.390 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.390 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:49.390 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.153457+0000", 2024-09-21T15:50:49.390 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4687999999999999e-05 2024-09-21T15:50:49.390 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.390 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.390 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:50:49.390 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.153893+0000", 2024-09-21T15:50:49.390 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000436363 2024-09-21T15:50:49.390 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.390 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.390 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:49.390 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.156134+0000", 2024-09-21T15:50:49.390 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0022410009999999998 2024-09-21T15:50:49.391 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.391 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.391 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:49.391 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.156148+0000", 2024-09-21T15:50:49.391 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3574000000000001e-05 2024-09-21T15:50:49.391 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.391 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.391 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:49.391 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.156214+0000", 2024-09-21T15:50:49.391 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6438000000000003e-05 2024-09-21T15:50:49.391 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.391 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.391 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:49.391 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.156304+0000", 2024-09-21T15:50:49.392 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9278999999999997e-05 2024-09-21T15:50:49.392 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.392 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:49.392 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.392 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.392 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.392 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160512 3.b 3:d297695c:::rbd_header.11338bae974a:head [watch unwatch cookie 140381478598448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T15:50:49.392 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:30.161362+0000", 2024-09-21T15:50:49.392 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 19.213735502999999, 2024-09-21T15:50:49.392 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0074105029999999997, 2024-09-21T15:50:49.392 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:49.392 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:49.392 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:49.393 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:49.393 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:49.393 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160512 2024-09-21T15:50:49.393 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.393 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:49.393 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.393 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:49.393 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161362+0000", 2024-09-21T15:50:49.393 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.393 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.393 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.393 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:49.393 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161362+0000", 2024-09-21T15:50:49.393 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.393 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.393 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.394 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:49.394 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161362+0000", 2024-09-21T15:50:49.394 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:49.394 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.394 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.394 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:49.394 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161365+0000", 2024-09-21T15:50:49.394 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8669999999999998e-06 2024-09-21T15:50:49.394 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.394 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.394 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:49.394 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161371+0000", 2024-09-21T15:50:49.394 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8009999999999998e-06 2024-09-21T15:50:49.394 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.395 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.395 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:49.395 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161376+0000", 2024-09-21T15:50:49.395 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0340000000000004e-06 2024-09-21T15:50:49.395 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.395 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.395 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:49.395 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161762+0000", 2024-09-21T15:50:49.395 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038575099999999999 2024-09-21T15:50:49.395 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.395 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.395 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:49.395 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161838+0000", 2024-09-21T15:50:49.395 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5697000000000004e-05 2024-09-21T15:50:49.395 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.396 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.396 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:50:49.396 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161952+0000", 2024-09-21T15:50:49.396 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000114034 2024-09-21T15:50:49.396 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.396 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.396 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:49.396 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.163727+0000", 2024-09-21T15:50:49.396 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0017749560000000001 2024-09-21T15:50:49.396 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.396 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.396 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:49.396 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168545+0000", 2024-09-21T15:50:49.396 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0048186080000000003 2024-09-21T15:50:49.396 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.396 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.397 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:49.397 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168593+0000", 2024-09-21T15:50:49.397 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7549999999999997e-05 2024-09-21T15:50:49.397 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.397 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.397 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:49.397 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168773+0000", 2024-09-21T15:50:49.397 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00017976600000000001 2024-09-21T15:50:49.397 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.397 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:49.397 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.397 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.397 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.397 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160513 3.b 3:d297695c:::rbd_header.11338bae974a:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-09-21T15:50:49.397 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:30.161755+0000", 2024-09-21T15:50:49.397 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 19.213342571999998, 2024-09-21T15:50:49.398 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0072250300000000003, 2024-09-21T15:50:49.398 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:49.398 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:49.398 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:49.398 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:49.398 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:49.398 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160513 2024-09-21T15:50:49.398 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.398 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:49.398 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.398 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:49.398 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161755+0000", 2024-09-21T15:50:49.398 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.399 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.399 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.399 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:49.399 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161755+0000", 2024-09-21T15:50:49.399 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.399 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.399 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.399 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:49.399 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161754+0000", 2024-09-21T15:50:49.399 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:49.399 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.399 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.399 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:49.399 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161759+0000", 2024-09-21T15:50:49.400 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9699999999999998e-06 2024-09-21T15:50:49.400 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.400 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.400 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:49.400 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161768+0000", 2024-09-21T15:50:49.400 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9949999999999994e-06 2024-09-21T15:50:49.400 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.400 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.400 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:49.400 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161774+0000", 2024-09-21T15:50:49.400 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1290000000000001e-06 2024-09-21T15:50:49.400 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.400 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.400 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:49.400 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.162239+0000", 2024-09-21T15:50:49.401 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00046406600000000001 2024-09-21T15:50:49.401 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.401 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.401 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-09-21T15:50:49.401 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.162296+0000", 2024-09-21T15:50:49.401 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7122e-05 2024-09-21T15:50:49.401 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.401 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.401 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:49.401 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168802+0000", 2024-09-21T15:50:49.401 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0065061700000000004 2024-09-21T15:50:49.401 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.401 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.401 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:49.401 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168877+0000", 2024-09-21T15:50:49.402 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5214000000000002e-05 2024-09-21T15:50:49.402 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.402 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.402 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:49.402 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168980+0000", 2024-09-21T15:50:49.402 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000103182 2024-09-21T15:50:49.402 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.402 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:49.402 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.402 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.402 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.402 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160527 3.b 3:d297695c:::rbd_header.11338bae974a:head [watch watch cookie 140381476148256] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T15:50:49.402 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:30.182756+0000", 2024-09-21T15:50:49.402 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 19.192341990999999, 2024-09-21T15:50:49.402 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0054202579999999998, 2024-09-21T15:50:49.403 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:49.403 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:49.403 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:49.403 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:49.403 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:49.403 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160527 2024-09-21T15:50:49.403 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.403 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:49.403 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.403 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:49.403 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182756+0000", 2024-09-21T15:50:49.403 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.403 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.403 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.403 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:49.404 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182756+0000", 2024-09-21T15:50:49.404 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.404 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.404 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.404 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:49.404 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182755+0000", 2024-09-21T15:50:49.404 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:49.404 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.404 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.404 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:49.404 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182760+0000", 2024-09-21T15:50:49.404 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4269999999999999e-06 2024-09-21T15:50:49.404 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.405 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.405 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:49.405 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182771+0000", 2024-09-21T15:50:49.405 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0312999999999999e-05 2024-09-21T15:50:49.405 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.405 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.405 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:49.405 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182778+0000", 2024-09-21T15:50:49.405 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5700000000000004e-06 2024-09-21T15:50:49.405 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.406 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.406 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:49.406 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.184745+0000", 2024-09-21T15:50:49.406 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0019667170000000002 2024-09-21T15:50:49.406 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.406 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.406 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:49.406 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.184867+0000", 2024-09-21T15:50:49.406 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000122504 2024-09-21T15:50:49.406 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.406 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.406 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:50:49.406 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.185135+0000", 2024-09-21T15:50:49.406 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000267135 2024-09-21T15:50:49.407 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.407 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.407 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:49.407 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.187721+0000", 2024-09-21T15:50:49.407 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002586891 2024-09-21T15:50:49.407 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.407 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.407 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:49.407 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.187874+0000", 2024-09-21T15:50:49.407 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000152726 2024-09-21T15:50:49.407 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.407 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.407 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:49.407 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.187951+0000", 2024-09-21T15:50:49.407 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7132000000000006e-05 2024-09-21T15:50:49.408 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.408 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.408 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:49.408 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.188176+0000", 2024-09-21T15:50:49.408 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00022477900000000001 2024-09-21T15:50:49.408 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.408 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:49.408 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.408 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.408 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.408 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160586 3.b 3:d297695c:::rbd_header.11338bae974a:head [watch watch cookie 140381476029296] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-21T15:50:49.408 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.201587+0000", 2024-09-21T15:50:49.408 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.173510630999999, 2024-09-21T15:50:49.408 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0040745900000000003, 2024-09-21T15:50:49.408 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:49.408 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:49.409 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:49.409 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:49.409 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:49.409 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160586 2024-09-21T15:50:49.409 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.409 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:49.409 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.409 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:49.409 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.201587+0000", 2024-09-21T15:50:49.409 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.409 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.409 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.409 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:49.409 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.201587+0000", 2024-09-21T15:50:49.410 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.410 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.410 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.410 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:49.410 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.201586+0000", 2024-09-21T15:50:49.410 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:49.410 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.410 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.410 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:49.410 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.201592+0000", 2024-09-21T15:50:49.410 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7910000000000001e-06 2024-09-21T15:50:49.410 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.410 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.410 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:49.410 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.201603+0000", 2024-09-21T15:50:49.411 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0736e-05 2024-09-21T15:50:49.411 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.411 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.411 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:49.411 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.201609+0000", 2024-09-21T15:50:49.411 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9739999999999999e-06 2024-09-21T15:50:49.411 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.411 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.411 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:49.411 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.201643+0000", 2024-09-21T15:50:49.411 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4572e-05 2024-09-21T15:50:49.411 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.411 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.411 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:49.411 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.201726+0000", 2024-09-21T15:50:49.411 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.2763999999999996e-05 2024-09-21T15:50:49.412 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.412 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.412 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:50:49.412 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.201832+0000", 2024-09-21T15:50:49.412 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105931 2024-09-21T15:50:49.412 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.412 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.412 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:49.412 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.202743+0000", 2024-09-21T15:50:49.412 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00091128699999999999 2024-09-21T15:50:49.412 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.412 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.412 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:49.412 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.205540+0000", 2024-09-21T15:50:49.412 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0027967199999999999 2024-09-21T15:50:49.412 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.413 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.413 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:49.413 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.205598+0000", 2024-09-21T15:50:49.413 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7513999999999999e-05 2024-09-21T15:50:49.413 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.413 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.413 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:49.413 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.205662+0000", 2024-09-21T15:50:49.413 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4148999999999999e-05 2024-09-21T15:50:49.413 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.413 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:49.413 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.413 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.413 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.413 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160605 3.b 3:d1863919:::rbd_data.11338bae974a.0000000000000008:head [delete] snapc 2d=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-21T15:50:49.413 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.213794+0000", 2024-09-21T15:50:49.414 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.161304111, 2024-09-21T15:50:49.414 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046152490000000001, 2024-09-21T15:50:49.414 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:49.414 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:49.414 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:49.414 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:49.414 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:49.414 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160605 2024-09-21T15:50:49.414 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.414 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:49.414 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.414 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:49.414 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.213794+0000", 2024-09-21T15:50:49.414 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.415 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.415 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.415 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:49.415 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.213794+0000", 2024-09-21T15:50:49.415 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.415 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.415 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.415 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:49.415 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.213793+0000", 2024-09-21T15:50:49.415 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:49.415 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.415 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.415 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:49.416 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.213799+0000", 2024-09-21T15:50:49.416 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9959999999999999e-06 2024-09-21T15:50:49.416 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.416 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.416 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:49.416 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.213815+0000", 2024-09-21T15:50:49.416 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5880999999999999e-05 2024-09-21T15:50:49.416 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.416 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.416 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:49.416 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.213838+0000", 2024-09-21T15:50:49.416 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3671000000000001e-05 2024-09-21T15:50:49.416 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.416 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.417 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:49.417 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.213887+0000", 2024-09-21T15:50:49.417 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8865000000000001e-05 2024-09-21T15:50:49.417 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.417 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.417 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:49.417 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.213962+0000", 2024-09-21T15:50:49.417 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4592999999999995e-05 2024-09-21T15:50:49.417 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.417 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.417 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:50:49.417 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.214088+0000", 2024-09-21T15:50:49.417 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012666199999999999 2024-09-21T15:50:49.417 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.418 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.418 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:49.418 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216917+0000", 2024-09-21T15:50:49.418 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0028284059999999999 2024-09-21T15:50:49.418 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.418 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.418 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:49.418 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.218319+0000", 2024-09-21T15:50:49.418 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001402613 2024-09-21T15:50:49.418 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.418 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.418 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:49.418 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.218372+0000", 2024-09-21T15:50:49.418 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2636000000000002e-05 2024-09-21T15:50:49.419 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.419 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.419 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:49.419 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.218409+0000", 2024-09-21T15:50:49.419 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6942e-05 2024-09-21T15:50:49.419 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.419 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:49.419 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.419 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.419 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.419 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:160602 3.12 e95/94 3:49a61a92:::rbd_data.11338bae974a.0000000000000005:head v 95'344, mlcod=0'0)", 2024-09-21T15:50:49.419 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.215034+0000", 2024-09-21T15:50:49.419 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.160063694000002, 2024-09-21T15:50:49.420 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045544870000000003, 2024-09-21T15:50:49.420 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:49.420 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:49.420 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:49.420 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.420 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:49.420 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215034+0000", 2024-09-21T15:50:49.420 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.420 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.420 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.420 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:49.420 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215034+0000", 2024-09-21T15:50:49.420 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.420 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.421 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.421 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:49.421 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215034+0000", 2024-09-21T15:50:49.421 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:49.421 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.421 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.421 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:49.421 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215039+0000", 2024-09-21T15:50:49.421 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1549999999999998e-06 2024-09-21T15:50:49.421 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.421 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.421 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:49.421 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215050+0000", 2024-09-21T15:50:49.421 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1392e-05 2024-09-21T15:50:49.421 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.422 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.422 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:49.422 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215057+0000", 2024-09-21T15:50:49.422 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7399999999999998e-06 2024-09-21T15:50:49.422 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.422 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.422 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:49.422 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215187+0000", 2024-09-21T15:50:49.422 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013010100000000001 2024-09-21T15:50:49.422 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.422 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.422 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:49.422 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215208+0000", 2024-09-21T15:50:49.422 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1013999999999999e-05 2024-09-21T15:50:49.422 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.423 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.423 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:49.423 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.219549+0000", 2024-09-21T15:50:49.423 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0043404990000000003 2024-09-21T15:50:49.423 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.423 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.423 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:49.423 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.219589+0000", 2024-09-21T15:50:49.423 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9878e-05 2024-09-21T15:50:49.423 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.423 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:49.423 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.423 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.423 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.424 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:160597 3.1 e95/94 3:84154dfc:::rbd_data.11338bae974a.0000000000000000:head v 95'451, mlcod=0'0)", 2024-09-21T15:50:49.424 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.215080+0000", 2024-09-21T15:50:49.424 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.160018237999999, 2024-09-21T15:50:49.424 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045596819999999998, 2024-09-21T15:50:49.424 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:49.424 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:49.424 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:49.424 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.424 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:49.424 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215080+0000", 2024-09-21T15:50:49.424 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.424 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.424 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.424 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:49.424 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215080+0000", 2024-09-21T15:50:49.425 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.425 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.425 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.425 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:49.425 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215079+0000", 2024-09-21T15:50:49.425 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:49.425 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.425 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.425 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:49.425 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215083+0000", 2024-09-21T15:50:49.425 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2570000000000001e-06 2024-09-21T15:50:49.425 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.425 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.426 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:49.426 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215091+0000", 2024-09-21T15:50:49.426 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7829999999999999e-06 2024-09-21T15:50:49.426 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.426 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.426 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:49.426 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215096+0000", 2024-09-21T15:50:49.426 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0370000000000002e-06 2024-09-21T15:50:49.426 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.426 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.426 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:49.426 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216055+0000", 2024-09-21T15:50:49.426 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00095892299999999998 2024-09-21T15:50:49.426 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.427 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.427 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:49.427 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216078+0000", 2024-09-21T15:50:49.427 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2376000000000001e-05 2024-09-21T15:50:49.427 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.427 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.427 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:49.427 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.219598+0000", 2024-09-21T15:50:49.427 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035209299999999998 2024-09-21T15:50:49.427 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.427 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.427 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:49.427 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.219639+0000", 2024-09-21T15:50:49.427 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0766000000000002e-05 2024-09-21T15:50:49.428 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.428 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:49.428 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.429 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.429 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.429 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:160601 3.17 e95/94 3:e9d872b2:::rbd_data.11338bae974a.0000000000000004:head v 95'912, mlcod=0'0)", 2024-09-21T15:50:49.429 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.215668+0000", 2024-09-21T15:50:49.429 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.159429621000001, 2024-09-21T15:50:49.429 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0038923640000000002, 2024-09-21T15:50:49.430 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:49.430 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:49.430 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:49.430 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.430 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:49.430 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215668+0000", 2024-09-21T15:50:49.430 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.430 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.430 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.430 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:49.430 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215668+0000", 2024-09-21T15:50:49.430 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.431 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.431 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.431 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:49.431 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215668+0000", 2024-09-21T15:50:49.431 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:49.431 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.431 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.431 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:49.431 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215672+0000", 2024-09-21T15:50:49.431 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7680000000000002e-06 2024-09-21T15:50:49.431 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.431 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.431 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:49.431 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215682+0000", 2024-09-21T15:50:49.431 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0774000000000001e-05 2024-09-21T15:50:49.431 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.432 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.432 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:49.432 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215689+0000", 2024-09-21T15:50:49.432 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4670000000000001e-06 2024-09-21T15:50:49.432 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.432 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.432 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:49.432 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215740+0000", 2024-09-21T15:50:49.432 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1100000000000002e-05 2024-09-21T15:50:49.432 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.432 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.432 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:49.432 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215761+0000", 2024-09-21T15:50:49.432 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1112e-05 2024-09-21T15:50:49.432 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.433 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.433 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:49.433 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.219520+0000", 2024-09-21T15:50:49.433 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0037590739999999998 2024-09-21T15:50:49.433 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.433 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.433 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:49.433 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.219561+0000", 2024-09-21T15:50:49.433 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0329e-05 2024-09-21T15:50:49.433 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.433 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:49.433 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.433 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.433 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.433 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:160608 3.12 e95/94 3:48a47e1c:::rbd_data.11338bae974a.000000000000000b:head v 95'345, mlcod=0'0)", 2024-09-21T15:50:49.434 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.216934+0000", 2024-09-21T15:50:49.434 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.158163853000001, 2024-09-21T15:50:49.434 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053335759999999996, 2024-09-21T15:50:49.434 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:49.434 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:49.434 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:49.434 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.434 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:49.434 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216934+0000", 2024-09-21T15:50:49.434 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.434 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.434 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.434 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:49.434 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216934+0000", 2024-09-21T15:50:49.434 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.435 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.435 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.435 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:49.435 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216933+0000", 2024-09-21T15:50:49.435 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:49.435 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.435 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.435 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:49.435 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216939+0000", 2024-09-21T15:50:49.435 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6459999999999998e-06 2024-09-21T15:50:49.435 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.435 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.435 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:49.435 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216947+0000", 2024-09-21T15:50:49.435 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.2479999999999996e-06 2024-09-21T15:50:49.436 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.436 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.436 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:49.436 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216951+0000", 2024-09-21T15:50:49.436 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3760000000000001e-06 2024-09-21T15:50:49.436 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.436 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.436 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:49.436 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216998+0000", 2024-09-21T15:50:49.436 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6244e-05 2024-09-21T15:50:49.436 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.436 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.436 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:49.436 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.217019+0000", 2024-09-21T15:50:49.437 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1191000000000001e-05 2024-09-21T15:50:49.437 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.437 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.437 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:49.437 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.222230+0000", 2024-09-21T15:50:49.437 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052114350000000004 2024-09-21T15:50:49.437 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.437 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.437 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:49.437 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.222267+0000", 2024-09-21T15:50:49.437 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7169999999999998e-05 2024-09-21T15:50:49.437 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.437 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:49.437 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.437 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.437 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.438 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160618 3.b 3:d297695c:::rbd_header.11338bae974a:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-21T15:50:49.438 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.233457+0000", 2024-09-21T15:50:49.438 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.141641016000001, 2024-09-21T15:50:49.438 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.005085198, 2024-09-21T15:50:49.438 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:49.438 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:49.438 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:49.438 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:49.438 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:49.438 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160618 2024-09-21T15:50:49.438 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.438 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:49.438 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.438 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:49.438 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233457+0000", 2024-09-21T15:50:49.439 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.439 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.439 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.439 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:49.439 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233457+0000", 2024-09-21T15:50:49.439 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.439 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.439 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.439 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:49.439 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233456+0000", 2024-09-21T15:50:49.439 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:49.439 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.439 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.439 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:49.440 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233463+0000", 2024-09-21T15:50:49.440 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9310000000000002e-06 2024-09-21T15:50:49.440 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.440 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.440 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:49.440 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233473+0000", 2024-09-21T15:50:49.440 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0519e-05 2024-09-21T15:50:49.440 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.440 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.440 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:49.440 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233481+0000", 2024-09-21T15:50:49.440 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4009999999999997e-06 2024-09-21T15:50:49.440 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.440 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.440 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:49.440 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233513+0000", 2024-09-21T15:50:49.441 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1328999999999998e-05 2024-09-21T15:50:49.441 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.441 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.441 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:49.441 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233568+0000", 2024-09-21T15:50:49.441 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5606999999999998e-05 2024-09-21T15:50:49.441 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.441 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.441 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:50:49.441 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233670+0000", 2024-09-21T15:50:49.441 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010180599999999999 2024-09-21T15:50:49.441 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.441 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.441 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:49.441 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.234761+0000", 2024-09-21T15:50:49.441 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010905909999999999 2024-09-21T15:50:49.442 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.442 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.442 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:49.442 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.238442+0000", 2024-09-21T15:50:49.442 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0036809429999999999 2024-09-21T15:50:49.442 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.442 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.442 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:49.442 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.238498+0000", 2024-09-21T15:50:49.442 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6116999999999998e-05 2024-09-21T15:50:49.442 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.442 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.442 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:49.442 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.238542+0000", 2024-09-21T15:50:49.443 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4134000000000001e-05 2024-09-21T15:50:49.443 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.443 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:49.443 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.443 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.443 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.443 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160662 3.b 3:d7701915:::rbd_object_map.1133fd7483d4.000000000000002c:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-21T15:50:49.443 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.279369+0000", 2024-09-21T15:50:49.443 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.095729085999999, 2024-09-21T15:50:49.443 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0056948040000000004, 2024-09-21T15:50:49.443 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:49.443 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:49.443 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:49.443 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:49.443 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:49.444 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160662 2024-09-21T15:50:49.444 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.444 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:49.444 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.444 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:49.444 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279369+0000", 2024-09-21T15:50:49.444 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.444 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.444 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.444 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:49.444 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279369+0000", 2024-09-21T15:50:49.444 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.444 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.444 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.444 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:49.444 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279368+0000", 2024-09-21T15:50:49.445 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:49.445 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.445 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.445 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:49.445 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279373+0000", 2024-09-21T15:50:49.445 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6529999999999999e-06 2024-09-21T15:50:49.445 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.445 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.445 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:49.445 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279384+0000", 2024-09-21T15:50:49.445 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0256e-05 2024-09-21T15:50:49.445 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.445 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.445 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:49.445 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279388+0000", 2024-09-21T15:50:49.446 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5630000000000002e-06 2024-09-21T15:50:49.446 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.446 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.446 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:49.446 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279424+0000", 2024-09-21T15:50:49.446 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5985999999999997e-05 2024-09-21T15:50:49.446 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.446 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.446 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:49.446 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279478+0000", 2024-09-21T15:50:49.446 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3944999999999999e-05 2024-09-21T15:50:49.446 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.446 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.446 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:50:49.447 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279575+0000", 2024-09-21T15:50:49.447 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6817999999999994e-05 2024-09-21T15:50:49.447 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.447 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.447 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:49.447 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.282391+0000", 2024-09-21T15:50:49.447 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0028156700000000002 2024-09-21T15:50:49.447 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.447 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.447 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:49.447 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.284977+0000", 2024-09-21T15:50:49.447 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0025859640000000001 2024-09-21T15:50:49.447 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.447 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.447 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:49.448 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.285021+0000", 2024-09-21T15:50:49.448 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4415000000000001e-05 2024-09-21T15:50:49.448 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.448 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.448 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:49.448 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.285063+0000", 2024-09-21T15:50:49.448 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2447000000000002e-05 2024-09-21T15:50:49.448 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.448 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:49.448 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.448 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.448 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.448 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:161118 3.1 3:82ccd16f:::rbd_header.11335aef9442:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-21T15:50:49.448 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:36.688929+0000", 2024-09-21T15:50:49.449 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.686168828, 2024-09-21T15:50:49.449 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0040062300000000004, 2024-09-21T15:50:49.449 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:49.449 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:49.449 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:49.449 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:49.449 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:49.449 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 161118 2024-09-21T15:50:49.449 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.449 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:49.449 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.449 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:49.449 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.688929+0000", 2024-09-21T15:50:49.449 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.449 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.450 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.450 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:49.450 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.688929+0000", 2024-09-21T15:50:49.450 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.450 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.450 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.450 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:49.450 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.688927+0000", 2024-09-21T15:50:49.450 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:50:49.450 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.450 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.450 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:49.450 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.688938+0000", 2024-09-21T15:50:49.451 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1466e-05 2024-09-21T15:50:49.451 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.451 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.451 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:49.451 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.688960+0000", 2024-09-21T15:50:49.451 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.128e-05 2024-09-21T15:50:49.451 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.451 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.451 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:49.451 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.688969+0000", 2024-09-21T15:50:49.451 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3400000000000004e-06 2024-09-21T15:50:49.451 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.451 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.451 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:49.452 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.689036+0000", 2024-09-21T15:50:49.452 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6506000000000003e-05 2024-09-21T15:50:49.452 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.452 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.452 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op must wait for map", 2024-09-21T15:50:49.452 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.689045+0000", 2024-09-21T15:50:49.452 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9129999999999998e-06 2024-09-21T15:50:49.452 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.452 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.452 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:49.452 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.690856+0000", 2024-09-21T15:50:49.452 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0018102960000000001 2024-09-21T15:50:49.452 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.452 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.452 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:49.452 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.690946+0000", 2024-09-21T15:50:49.453 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0265000000000004e-05 2024-09-21T15:50:49.453 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.453 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.453 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-09-21T15:50:49.453 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.691301+0000", 2024-09-21T15:50:49.453 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00035520799999999998 2024-09-21T15:50:49.453 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.453 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.453 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:49.453 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.692521+0000", 2024-09-21T15:50:49.453 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012194930000000001 2024-09-21T15:50:49.453 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.453 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.453 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:49.454 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.692849+0000", 2024-09-21T15:50:49.454 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00032818000000000002 2024-09-21T15:50:49.454 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.454 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.454 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:49.454 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.692897+0000", 2024-09-21T15:50:49.454 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8260999999999998e-05 2024-09-21T15:50:49.454 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.454 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.454 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:49.454 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.692935+0000", 2024-09-21T15:50:49.454 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8022999999999998e-05 2024-09-21T15:50:49.454 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.454 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:49.455 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.455 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.455 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.455 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:161122 3.1 3:82ccd16f:::rbd_header.11335aef9442:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-21T15:50:49.455 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:36.703081+0000", 2024-09-21T15:50:49.455 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.672016922999999, 2024-09-21T15:50:49.455 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0051212200000000001, 2024-09-21T15:50:49.455 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:49.455 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:49.455 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:49.455 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:49.455 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:49.455 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 161122 2024-09-21T15:50:49.455 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.456 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:49.456 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.456 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:49.456 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.703081+0000", 2024-09-21T15:50:49.456 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.456 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.456 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.456 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:49.456 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.703081+0000", 2024-09-21T15:50:49.456 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.456 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.456 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.456 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:49.456 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.703079+0000", 2024-09-21T15:50:49.456 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-09-21T15:50:49.457 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.457 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.457 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:49.457 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.703091+0000", 2024-09-21T15:50:49.457 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2584e-05 2024-09-21T15:50:49.457 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.457 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.457 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:49.457 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.703107+0000", 2024-09-21T15:50:49.457 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6313000000000001e-05 2024-09-21T15:50:49.457 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.457 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.457 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:49.457 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.703119+0000", 2024-09-21T15:50:49.457 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1404e-05 2024-09-21T15:50:49.458 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.458 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.458 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:49.458 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.703176+0000", 2024-09-21T15:50:49.458 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6740999999999998e-05 2024-09-21T15:50:49.458 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.458 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.458 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:49.458 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.703281+0000", 2024-09-21T15:50:49.458 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010508 2024-09-21T15:50:49.458 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.458 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.458 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-09-21T15:50:49.458 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.703483+0000", 2024-09-21T15:50:49.458 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00020257399999999999 2024-09-21T15:50:49.459 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.459 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.459 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:49.459 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.705657+0000", 2024-09-21T15:50:49.459 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0021735610000000001 2024-09-21T15:50:49.459 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.459 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.459 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:49.459 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.708144+0000", 2024-09-21T15:50:49.459 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002487122 2024-09-21T15:50:49.459 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.459 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.459 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:49.459 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.708177+0000", 2024-09-21T15:50:49.459 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2574999999999999e-05 2024-09-21T15:50:49.460 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.460 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.460 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:49.460 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.708202+0000", 2024-09-21T15:50:49.460 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5513000000000001e-05 2024-09-21T15:50:49.460 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.460 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:49.460 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.460 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.460 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.460 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:161219 3.17 3:e87cb47d:::rbd_data.11335aef9442.000000000000000a:head [delete] snapc 2a=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-21T15:50:49.460 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:36.771292+0000", 2024-09-21T15:50:49.460 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.603805273999999, 2024-09-21T15:50:49.460 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0058540529999999997, 2024-09-21T15:50:49.461 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:49.461 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:49.461 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:49.461 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:49.461 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:49.461 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 161219 2024-09-21T15:50:49.461 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.461 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:49.461 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.461 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:49.461 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771292+0000", 2024-09-21T15:50:49.461 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.461 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.461 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.462 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:49.462 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771292+0000", 2024-09-21T15:50:49.462 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.462 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.462 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.462 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:49.462 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771292+0000", 2024-09-21T15:50:49.462 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:49.462 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.462 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.462 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:49.462 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771298+0000", 2024-09-21T15:50:49.462 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9519999999999999e-06 2024-09-21T15:50:49.462 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.462 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.463 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:49.463 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771308+0000", 2024-09-21T15:50:49.463 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0601000000000001e-05 2024-09-21T15:50:49.463 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.463 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.463 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:49.463 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771314+0000", 2024-09-21T15:50:49.463 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.913e-06 2024-09-21T15:50:49.463 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.463 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.463 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:49.463 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771346+0000", 2024-09-21T15:50:49.463 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1711000000000002e-05 2024-09-21T15:50:49.463 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.463 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.464 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:49.464 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771399+0000", 2024-09-21T15:50:49.464 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2883000000000002e-05 2024-09-21T15:50:49.464 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.464 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.464 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-09-21T15:50:49.464 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771470+0000", 2024-09-21T15:50:49.464 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0934999999999998e-05 2024-09-21T15:50:49.464 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.464 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.464 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:49.464 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.772234+0000", 2024-09-21T15:50:49.464 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00076485400000000003 2024-09-21T15:50:49.464 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.464 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.465 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:49.465 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.777083+0000", 2024-09-21T15:50:49.465 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0048486329999999998 2024-09-21T15:50:49.465 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.465 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.465 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:49.465 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.777111+0000", 2024-09-21T15:50:49.465 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8399999999999999e-05 2024-09-21T15:50:49.465 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.465 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.465 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:49.465 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.777147+0000", 2024-09-21T15:50:49.465 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5049000000000003e-05 2024-09-21T15:50:49.465 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.466 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:49.466 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.466 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.466 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.466 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:161762 3.17 3:ef707828:::rbd_header.11338db70e6f:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T15:50:49.466 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:40.705291+0000", 2024-09-21T15:50:49.466 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.6698072239999995, 2024-09-21T15:50:49.466 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0048878120000000001, 2024-09-21T15:50:49.466 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:49.466 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:49.466 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:49.466 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:49.466 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:49.466 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 161762 2024-09-21T15:50:49.467 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.467 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:49.467 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.467 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:49.467 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.705291+0000", 2024-09-21T15:50:49.467 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.467 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.467 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.467 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:49.467 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.705291+0000", 2024-09-21T15:50:49.467 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.467 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.467 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.467 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:49.467 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.705289+0000", 2024-09-21T15:50:49.468 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:50:49.468 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.468 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.468 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:49.468 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.705296+0000", 2024-09-21T15:50:49.468 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1999999999999997e-06 2024-09-21T15:50:49.468 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.468 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.468 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:49.468 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.705309+0000", 2024-09-21T15:50:49.468 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2723e-05 2024-09-21T15:50:49.468 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.468 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.468 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:49.468 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.705318+0000", 2024-09-21T15:50:49.469 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4890000000000001e-06 2024-09-21T15:50:49.469 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.469 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.469 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:49.469 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.705370+0000", 2024-09-21T15:50:49.469 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1493000000000003e-05 2024-09-21T15:50:49.469 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.469 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.469 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:49.469 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.705480+0000", 2024-09-21T15:50:49.469 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011059 2024-09-21T15:50:49.469 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.469 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.469 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-09-21T15:50:49.470 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.705800+0000", 2024-09-21T15:50:49.470 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00031958399999999998 2024-09-21T15:50:49.470 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.470 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.470 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:49.470 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.707140+0000", 2024-09-21T15:50:49.470 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001339954 2024-09-21T15:50:49.470 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.470 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.470 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:49.470 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.710103+0000", 2024-09-21T15:50:49.470 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029629109999999999 2024-09-21T15:50:49.470 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.470 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.470 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:49.471 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.710136+0000", 2024-09-21T15:50:49.471 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3300000000000003e-05 2024-09-21T15:50:49.471 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.471 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.471 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:49.471 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.710178+0000", 2024-09-21T15:50:49.471 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2172999999999998e-05 2024-09-21T15:50:49.471 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.471 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:49.471 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.471 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.471 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.471 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:161853 3.1b 3:da34e1d8:::rbd_data.11338db70e6f.0000000000000001:head [delete] snapc 26=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T15:50:49.471 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:40.782229+0000", 2024-09-21T15:50:49.472 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.5928689990000002, 2024-09-21T15:50:49.472 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0040918660000000004, 2024-09-21T15:50:49.472 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:49.472 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:49.472 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:49.472 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:49.472 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:49.472 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 161853 2024-09-21T15:50:49.472 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.472 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:49.472 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.472 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:49.472 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782229+0000", 2024-09-21T15:50:49.472 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.472 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.473 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.473 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:49.473 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782229+0000", 2024-09-21T15:50:49.473 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.473 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.473 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.473 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:49.473 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782228+0000", 2024-09-21T15:50:49.473 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:49.473 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.473 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.473 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:49.473 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782233+0000", 2024-09-21T15:50:49.473 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2549999999999997e-06 2024-09-21T15:50:49.474 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.474 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.474 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:49.474 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782246+0000", 2024-09-21T15:50:49.474 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2563e-05 2024-09-21T15:50:49.474 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.474 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.474 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:49.474 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782252+0000", 2024-09-21T15:50:49.474 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.331e-06 2024-09-21T15:50:49.474 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.474 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.474 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:49.474 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782306+0000", 2024-09-21T15:50:49.474 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4078e-05 2024-09-21T15:50:49.475 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.475 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.475 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:49.475 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782474+0000", 2024-09-21T15:50:49.475 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000167713 2024-09-21T15:50:49.475 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.475 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.475 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-09-21T15:50:49.475 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782574+0000", 2024-09-21T15:50:49.475 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010028 2024-09-21T15:50:49.475 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.475 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.475 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:49.475 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.784621+0000", 2024-09-21T15:50:49.476 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0020471320000000001 2024-09-21T15:50:49.476 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.476 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.476 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:49.476 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786240+0000", 2024-09-21T15:50:49.476 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0016184330000000001 2024-09-21T15:50:49.476 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.476 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.476 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:49.476 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786286+0000", 2024-09-21T15:50:49.476 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6245999999999997e-05 2024-09-21T15:50:49.476 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.476 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.476 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:49.477 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786321+0000", 2024-09-21T15:50:49.477 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4739000000000001e-05 2024-09-21T15:50:49.477 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.477 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:49.477 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.477 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.477 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.477 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:161859 3.17 3:ee7ab022:::rbd_data.11338db70e6f.0000000000000007:head [delete] snapc 26=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T15:50:49.477 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:40.782264+0000", 2024-09-21T15:50:49.477 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.5928334149999994, 2024-09-21T15:50:49.477 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052663939999999998, 2024-09-21T15:50:49.477 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:49.477 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:49.478 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:49.478 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:49.478 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:49.478 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 161859 2024-09-21T15:50:49.478 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.478 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:49.478 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.478 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:49.478 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782264+0000", 2024-09-21T15:50:49.478 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.478 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.478 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.478 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:49.478 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782264+0000", 2024-09-21T15:50:49.479 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.479 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.479 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.479 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:49.479 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782264+0000", 2024-09-21T15:50:49.479 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:49.479 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.479 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.479 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:49.479 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782266+0000", 2024-09-21T15:50:49.479 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4590000000000001e-06 2024-09-21T15:50:49.479 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.479 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.479 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:49.480 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782275+0000", 2024-09-21T15:50:49.480 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5450000000000003e-06 2024-09-21T15:50:49.480 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.480 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.480 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:49.480 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782281+0000", 2024-09-21T15:50:49.480 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7180000000000001e-06 2024-09-21T15:50:49.480 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.480 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.480 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:49.480 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782312+0000", 2024-09-21T15:50:49.480 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1059000000000002e-05 2024-09-21T15:50:49.480 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.480 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.481 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:49.481 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782470+0000", 2024-09-21T15:50:49.481 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000157818 2024-09-21T15:50:49.481 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.481 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.481 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-09-21T15:50:49.481 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782574+0000", 2024-09-21T15:50:49.481 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000104536 2024-09-21T15:50:49.481 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.481 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.481 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:49.481 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.783499+0000", 2024-09-21T15:50:49.481 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00092479599999999997 2024-09-21T15:50:49.481 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.482 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.482 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:49.482 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.787472+0000", 2024-09-21T15:50:49.482 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0039730319999999996 2024-09-21T15:50:49.482 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.482 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.482 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:49.482 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.787507+0000", 2024-09-21T15:50:49.482 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5552000000000001e-05 2024-09-21T15:50:49.482 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.482 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.482 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:49.482 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.787531+0000", 2024-09-21T15:50:49.482 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3281999999999998e-05 2024-09-21T15:50:49.483 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.483 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:49.483 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.483 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.483 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.483 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:162390 3.17 3:ea8605c7:::rbd_header.1133fc941201:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-09-21T15:50:49.483 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:44.740587+0000", 2024-09-21T15:50:49.483 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.6345108189999999, 2024-09-21T15:50:49.483 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052218530000000003, 2024-09-21T15:50:49.483 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:49.483 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:49.483 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:49.483 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:49.483 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:49.483 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 162390 2024-09-21T15:50:49.484 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.484 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:49.484 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.484 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:49.484 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.740587+0000", 2024-09-21T15:50:49.484 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.484 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.484 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.484 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:49.484 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.740587+0000", 2024-09-21T15:50:49.484 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.484 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.484 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.484 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:49.484 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.740585+0000", 2024-09-21T15:50:49.485 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:50:49.485 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.485 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.485 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:49.485 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.740595+0000", 2024-09-21T15:50:49.485 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7839999999999998e-06 2024-09-21T15:50:49.485 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.485 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.485 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:49.485 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.740609+0000", 2024-09-21T15:50:49.485 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.384e-05 2024-09-21T15:50:49.485 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.485 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.485 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:49.485 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.740617+0000", 2024-09-21T15:50:49.486 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7560000000000005e-06 2024-09-21T15:50:49.486 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.486 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.486 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:49.486 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.740671+0000", 2024-09-21T15:50:49.486 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4073999999999998e-05 2024-09-21T15:50:49.486 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.486 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.486 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:49.486 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.740799+0000", 2024-09-21T15:50:49.486 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000128057 2024-09-21T15:50:49.486 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.486 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.486 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-09-21T15:50:49.486 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.741258+0000", 2024-09-21T15:50:49.487 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00045892900000000001 2024-09-21T15:50:49.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.487 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.487 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:49.487 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.742101+0000", 2024-09-21T15:50:49.487 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00084276499999999999 2024-09-21T15:50:49.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.487 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.487 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:49.487 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.745743+0000", 2024-09-21T15:50:49.487 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0036415280000000002 2024-09-21T15:50:49.487 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.487 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.487 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:49.487 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.745781+0000", 2024-09-21T15:50:49.488 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8398000000000001e-05 2024-09-21T15:50:49.488 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.488 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.488 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:49.488 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.745809+0000", 2024-09-21T15:50:49.488 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7691999999999998e-05 2024-09-21T15:50:49.488 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.488 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:49.488 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.488 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.488 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:49.488 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:49.490 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:50:49.490 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:49.488+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:49.591 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:49.589+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:49.594 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:49.594 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:49.594 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:50:49.594 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:49.602 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:50:49.691 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:49.690+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:49.711 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:49.711 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:49.711 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:50:49.711 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:50:49.711 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:49.720 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:50:49.804 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:49.803+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:49.884 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:49.884 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:50:49.885 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:50:49.885 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:50:49.885 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.885 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62833 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 192512~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:49.885 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:50:49.885 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 72.740648891999996, 2024-09-21T15:50:49.885 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050387007999999997, 2024-09-21T15:50:49.885 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:49.885 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:49.885 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:49.885 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:49.885 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:49.885 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62833 2024-09-21T15:50:49.885 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.885 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:49.886 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.886 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:49.886 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:50:49.886 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.886 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.886 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.886 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:49.886 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:50:49.886 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.886 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.886 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.886 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:49.886 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:50:49.887 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:49.887 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.887 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.887 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:49.887 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142150+0000", 2024-09-21T15:50:49.887 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1999999999999999e-06 2024-09-21T15:50:49.887 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.887 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.887 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:49.887 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142153+0000", 2024-09-21T15:50:49.887 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7410000000000001e-06 2024-09-21T15:50:49.887 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.887 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.888 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:49.888 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142156+0000", 2024-09-21T15:50:49.888 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5739999999999998e-06 2024-09-21T15:50:49.888 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.888 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.888 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:49.888 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.192358+0000", 2024-09-21T15:50:49.888 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050202452000000002 2024-09-21T15:50:49.888 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.888 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.888 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:49.888 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.192435+0000", 2024-09-21T15:50:49.888 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6738999999999998e-05 2024-09-21T15:50:49.888 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.888 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.889 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:49.889 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.192536+0000", 2024-09-21T15:50:49.889 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101508 2024-09-21T15:50:49.889 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.889 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:49.889 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.889 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.889 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.889 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62849 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 208896~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:49.889 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.144181+0000", 2024-09-21T15:50:49.889 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 72.738616820000004, 2024-09-21T15:50:49.889 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050460390000000001, 2024-09-21T15:50:49.889 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:49.889 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:49.889 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:49.890 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:49.890 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:49.890 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62849 2024-09-21T15:50:49.890 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.890 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:49.890 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.890 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:49.890 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144181+0000", 2024-09-21T15:50:49.890 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.890 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.890 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.890 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:49.890 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144181+0000", 2024-09-21T15:50:49.890 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.891 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.891 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.891 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:49.891 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144181+0000", 2024-09-21T15:50:49.891 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:49.891 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.891 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.891 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:49.891 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144183+0000", 2024-09-21T15:50:49.891 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3860000000000001e-06 2024-09-21T15:50:49.891 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.891 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.891 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:49.891 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144189+0000", 2024-09-21T15:50:49.891 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0599999999999996e-06 2024-09-21T15:50:49.892 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.892 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.892 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:49.892 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144194+0000", 2024-09-21T15:50:49.892 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4379999999999997e-06 2024-09-21T15:50:49.892 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.892 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.892 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:49.892 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.194457+0000", 2024-09-21T15:50:49.892 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050263317000000002 2024-09-21T15:50:49.892 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.892 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.892 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:49.892 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.194548+0000", 2024-09-21T15:50:49.893 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0785000000000006e-05 2024-09-21T15:50:49.893 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.893 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.893 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:49.893 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.194642+0000", 2024-09-21T15:50:49.893 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3837000000000003e-05 2024-09-21T15:50:49.893 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.893 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:49.893 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.893 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.893 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.893 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62862 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 954368~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:49.893 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.146176+0000", 2024-09-21T15:50:49.893 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 72.736621756999995, 2024-09-21T15:50:49.893 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050442901999999998, 2024-09-21T15:50:49.894 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:49.894 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:49.894 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:49.894 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:49.894 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:49.894 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62862 2024-09-21T15:50:49.894 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.894 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:49.894 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.894 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:49.894 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146176+0000", 2024-09-21T15:50:49.894 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.894 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.894 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.895 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:49.895 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146176+0000", 2024-09-21T15:50:49.895 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.895 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.895 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.895 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:49.895 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146176+0000", 2024-09-21T15:50:49.895 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:49.895 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.895 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.895 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:49.895 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146178+0000", 2024-09-21T15:50:49.895 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4739999999999999e-06 2024-09-21T15:50:49.895 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.895 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.896 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:49.896 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146186+0000", 2024-09-21T15:50:49.896 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4989999999999997e-06 2024-09-21T15:50:49.896 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.896 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.896 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:49.896 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146191+0000", 2024-09-21T15:50:49.896 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7999999999999998e-06 2024-09-21T15:50:49.896 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.896 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.896 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:49.896 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196442+0000", 2024-09-21T15:50:49.896 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050251665000000001 2024-09-21T15:50:49.896 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.896 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.896 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:49.897 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196518+0000", 2024-09-21T15:50:49.897 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5677000000000001e-05 2024-09-21T15:50:49.897 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.897 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.897 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:49.897 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196619+0000", 2024-09-21T15:50:49.897 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101247 2024-09-21T15:50:49.897 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.897 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:49.897 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.897 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.897 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.897 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62863 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 937984~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:49.897 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.146195+0000", 2024-09-21T15:50:49.897 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 72.736603565999999, 2024-09-21T15:50:49.897 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050642305999999998, 2024-09-21T15:50:49.898 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:49.898 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:49.898 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:49.898 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:49.898 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:49.898 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62863 2024-09-21T15:50:49.898 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.898 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:49.898 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.898 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:49.898 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146195+0000", 2024-09-21T15:50:49.898 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.898 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.898 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.899 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:49.899 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146195+0000", 2024-09-21T15:50:49.899 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.899 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.899 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.899 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:49.899 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146194+0000", 2024-09-21T15:50:49.899 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:49.899 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.899 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.899 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:49.899 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146196+0000", 2024-09-21T15:50:49.899 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1900000000000002e-06 2024-09-21T15:50:49.899 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.899 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.900 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:49.900 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146202+0000", 2024-09-21T15:50:49.900 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.57e-06 2024-09-21T15:50:49.900 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.900 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.900 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:49.900 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146206+0000", 2024-09-21T15:50:49.900 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5079999999999999e-06 2024-09-21T15:50:49.900 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.900 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.900 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:49.900 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196655+0000", 2024-09-21T15:50:49.900 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050448973000000001 2024-09-21T15:50:49.900 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.901 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.901 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:49.901 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196730+0000", 2024-09-21T15:50:49.901 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4728999999999996e-05 2024-09-21T15:50:49.901 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.901 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.901 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:49.901 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196837+0000", 2024-09-21T15:50:49.901 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000106735 2024-09-21T15:50:49.901 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.901 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:49.901 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.901 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.901 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.902 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62872 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 290816~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:49.902 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148149+0000", 2024-09-21T15:50:49.902 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 72.734648890000003, 2024-09-21T15:50:49.902 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.05046991, 2024-09-21T15:50:49.902 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:49.902 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:49.902 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:49.902 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:49.902 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:49.902 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62872 2024-09-21T15:50:49.902 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.902 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:49.902 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.902 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:49.902 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148149+0000", 2024-09-21T15:50:49.903 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.903 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.903 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.903 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:49.903 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148149+0000", 2024-09-21T15:50:49.903 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.903 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.903 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.903 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:49.903 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148149+0000", 2024-09-21T15:50:49.903 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:49.903 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.903 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.903 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:49.903 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148157+0000", 2024-09-21T15:50:49.904 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4749999999999993e-06 2024-09-21T15:50:49.904 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.904 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.904 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:49.904 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148163+0000", 2024-09-21T15:50:49.904 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8250000000000001e-06 2024-09-21T15:50:49.904 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.904 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.904 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:49.904 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148168+0000", 2024-09-21T15:50:49.904 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4730000000000002e-06 2024-09-21T15:50:49.904 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.904 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.904 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:49.905 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198471+0000", 2024-09-21T15:50:49.905 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050302974 2024-09-21T15:50:49.905 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.905 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.905 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:49.905 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198555+0000", 2024-09-21T15:50:49.905 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4783999999999993e-05 2024-09-21T15:50:49.905 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.905 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.905 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:49.905 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198619+0000", 2024-09-21T15:50:49.905 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3751000000000007e-05 2024-09-21T15:50:49.905 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.906 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:49.906 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.906 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.906 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.906 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62873 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 315392~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:49.906 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148172+0000", 2024-09-21T15:50:49.906 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 72.734626598999995, 2024-09-21T15:50:49.906 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050707998999999997, 2024-09-21T15:50:49.906 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:49.906 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:49.906 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:49.906 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:49.906 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:49.906 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62873 2024-09-21T15:50:49.907 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.907 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:49.907 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.907 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:49.907 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148172+0000", 2024-09-21T15:50:49.907 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.907 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.907 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.907 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:49.907 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148172+0000", 2024-09-21T15:50:49.907 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.907 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.907 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.908 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:49.908 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148171+0000", 2024-09-21T15:50:49.908 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:49.908 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.908 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.908 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:49.908 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148173+0000", 2024-09-21T15:50:49.908 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2340000000000001e-06 2024-09-21T15:50:49.908 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.908 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.908 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:49.908 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148179+0000", 2024-09-21T15:50:49.908 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9340000000000003e-06 2024-09-21T15:50:49.908 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.909 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.909 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:49.909 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148186+0000", 2024-09-21T15:50:49.909 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2349999999999996e-06 2024-09-21T15:50:49.909 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.909 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.909 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:49.909 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198684+0000", 2024-09-21T15:50:49.909 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050498890999999997 2024-09-21T15:50:49.909 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.909 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.909 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:49.909 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198778+0000", 2024-09-21T15:50:49.909 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3134999999999995e-05 2024-09-21T15:50:49.909 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.910 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.910 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:49.910 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198880+0000", 2024-09-21T15:50:49.910 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101906 2024-09-21T15:50:49.910 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.910 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:49.910 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.910 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.910 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.910 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62874 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 323584~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:49.910 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148211+0000", 2024-09-21T15:50:49.910 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 72.734586839000002, 2024-09-21T15:50:49.910 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050887766000000001, 2024-09-21T15:50:49.910 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:49.911 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:49.911 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:49.911 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:49.911 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:49.911 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62874 2024-09-21T15:50:49.911 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.911 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:49.911 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.911 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:49.911 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148211+0000", 2024-09-21T15:50:49.911 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.911 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.911 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.911 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:49.912 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148211+0000", 2024-09-21T15:50:49.912 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.912 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.912 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.912 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:49.912 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148211+0000", 2024-09-21T15:50:49.912 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:49.912 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.912 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.912 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:49.912 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148213+0000", 2024-09-21T15:50:49.912 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6860000000000002e-06 2024-09-21T15:50:49.912 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.912 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.912 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:49.913 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148220+0000", 2024-09-21T15:50:49.913 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4740000000000002e-06 2024-09-21T15:50:49.913 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.913 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.913 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:49.913 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148226+0000", 2024-09-21T15:50:49.913 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0120000000000002e-06 2024-09-21T15:50:49.913 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.913 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.913 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:49.913 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198940+0000", 2024-09-21T15:50:49.913 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050714412 2024-09-21T15:50:49.913 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.913 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.913 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:49.914 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198995+0000", 2024-09-21T15:50:49.914 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4863e-05 2024-09-21T15:50:49.914 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.914 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.914 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:49.914 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.199099+0000", 2024-09-21T15:50:49.914 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010383299999999999 2024-09-21T15:50:49.914 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.914 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:49.914 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.914 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.914 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.914 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62877 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 995328~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:49.914 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148230+0000", 2024-09-21T15:50:49.914 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 72.734567979000005, 2024-09-21T15:50:49.915 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.051108949000000001, 2024-09-21T15:50:49.915 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:49.915 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:49.915 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:49.915 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:49.915 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:49.915 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62877 2024-09-21T15:50:49.915 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.915 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:49.915 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.915 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:49.915 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148230+0000", 2024-09-21T15:50:49.915 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.915 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.915 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.916 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:49.916 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148230+0000", 2024-09-21T15:50:49.916 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.916 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.916 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.916 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:49.916 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148230+0000", 2024-09-21T15:50:49.916 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:49.916 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.916 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.916 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:49.916 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148232+0000", 2024-09-21T15:50:49.916 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1220000000000002e-06 2024-09-21T15:50:49.916 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.917 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.917 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:49.917 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148238+0000", 2024-09-21T15:50:49.917 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.789e-06 2024-09-21T15:50:49.917 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.917 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.917 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:49.917 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148242+0000", 2024-09-21T15:50:49.917 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7369999999999997e-06 2024-09-21T15:50:49.917 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.917 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.917 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:49.917 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.199201+0000", 2024-09-21T15:50:49.917 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050959088 2024-09-21T15:50:49.917 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.917 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.918 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:49.918 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.199285+0000", 2024-09-21T15:50:49.918 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3147999999999997e-05 2024-09-21T15:50:49.918 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.918 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.918 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:49.918 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.199339+0000", 2024-09-21T15:50:49.918 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4484999999999998e-05 2024-09-21T15:50:49.918 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.918 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:49.918 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.918 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.918 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.918 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62888 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 331776~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:49.918 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150154+0000", 2024-09-21T15:50:49.919 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 72.732644385, 2024-09-21T15:50:49.919 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050483169000000001, 2024-09-21T15:50:49.919 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:49.919 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:49.919 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:49.919 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:49.919 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:49.919 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62888 2024-09-21T15:50:49.919 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.919 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:49.919 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.919 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:49.919 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150154+0000", 2024-09-21T15:50:49.919 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.920 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.920 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.920 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:49.920 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150154+0000", 2024-09-21T15:50:49.920 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.920 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.920 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.920 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:49.920 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150153+0000", 2024-09-21T15:50:49.920 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:49.920 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.920 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.920 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:49.920 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150155+0000", 2024-09-21T15:50:49.920 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9920000000000002e-06 2024-09-21T15:50:49.921 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.921 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.921 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:49.921 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150161+0000", 2024-09-21T15:50:49.921 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5439999999999998e-06 2024-09-21T15:50:49.921 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.921 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.921 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:49.921 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150165+0000", 2024-09-21T15:50:49.921 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3460000000000004e-06 2024-09-21T15:50:49.921 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.921 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.921 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:49.921 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200502+0000", 2024-09-21T15:50:49.921 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050337150999999997 2024-09-21T15:50:49.921 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.922 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.922 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:49.922 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200575+0000", 2024-09-21T15:50:49.922 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2250999999999997e-05 2024-09-21T15:50:49.922 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.922 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.922 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:49.922 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200637+0000", 2024-09-21T15:50:49.922 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2195999999999999e-05 2024-09-21T15:50:49.922 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.922 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:49.922 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.922 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.922 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.922 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62889 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 339968~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:49.923 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150169+0000", 2024-09-21T15:50:49.923 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 72.732628886000001, 2024-09-21T15:50:49.923 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050629678999999997, 2024-09-21T15:50:49.923 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:49.923 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:49.923 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:49.923 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:49.923 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:49.923 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62889 2024-09-21T15:50:49.923 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.923 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:49.923 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.923 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:49.923 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150169+0000", 2024-09-21T15:50:49.923 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.923 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.924 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.924 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:49.924 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150169+0000", 2024-09-21T15:50:49.924 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.924 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.924 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.924 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:49.924 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150169+0000", 2024-09-21T15:50:49.924 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:49.924 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.924 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.924 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:49.924 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150171+0000", 2024-09-21T15:50:49.924 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3910000000000001e-06 2024-09-21T15:50:49.924 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.925 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.925 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:49.925 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150177+0000", 2024-09-21T15:50:49.925 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4079999999999997e-06 2024-09-21T15:50:49.925 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.925 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.925 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:49.925 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150182+0000", 2024-09-21T15:50:49.925 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3890000000000004e-06 2024-09-21T15:50:49.925 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.925 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.925 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:49.925 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200664+0000", 2024-09-21T15:50:49.925 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050482212999999998 2024-09-21T15:50:49.925 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.926 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.926 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:49.926 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200738+0000", 2024-09-21T15:50:49.926 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4114000000000002e-05 2024-09-21T15:50:49.926 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.926 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.926 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:49.926 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200799+0000", 2024-09-21T15:50:49.926 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0503e-05 2024-09-21T15:50:49.926 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.926 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:49.926 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.926 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.926 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.927 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62891 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 348160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:49.927 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150186+0000", 2024-09-21T15:50:49.927 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 72.732611743999996, 2024-09-21T15:50:49.927 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050794552, 2024-09-21T15:50:49.927 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:49.927 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:49.927 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:49.927 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:49.927 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:49.927 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62891 2024-09-21T15:50:49.927 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.927 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:49.927 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.927 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:49.927 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150186+0000", 2024-09-21T15:50:49.928 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.928 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.928 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.928 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:49.928 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150186+0000", 2024-09-21T15:50:49.928 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.928 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.928 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.928 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:49.928 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150186+0000", 2024-09-21T15:50:49.928 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:49.928 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.928 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.928 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:49.929 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150193+0000", 2024-09-21T15:50:49.929 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6830000000000002e-06 2024-09-21T15:50:49.929 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.929 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.929 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:49.929 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150199+0000", 2024-09-21T15:50:49.929 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9859999999999999e-06 2024-09-21T15:50:49.929 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.929 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.929 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:49.929 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150203+0000", 2024-09-21T15:50:49.929 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5920000000000002e-06 2024-09-21T15:50:49.929 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.929 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.929 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:49.929 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200835+0000", 2024-09-21T15:50:49.930 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050631783 2024-09-21T15:50:49.930 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.930 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.930 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:49.930 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200894+0000", 2024-09-21T15:50:49.930 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8560999999999997e-05 2024-09-21T15:50:49.930 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.930 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.930 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:49.930 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200981+0000", 2024-09-21T15:50:49.930 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7305999999999994e-05 2024-09-21T15:50:49.930 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.930 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:49.930 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.931 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.931 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.931 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62892 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 356352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:49.931 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150208+0000", 2024-09-21T15:50:49.931 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 72.732590356000003, 2024-09-21T15:50:49.931 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050951769000000001, 2024-09-21T15:50:49.931 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:49.931 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:49.931 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:49.931 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:49.931 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:49.931 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62892 2024-09-21T15:50:49.931 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.932 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:49.932 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.932 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:49.932 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150208+0000", 2024-09-21T15:50:49.932 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.932 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.932 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.932 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:49.932 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150208+0000", 2024-09-21T15:50:49.932 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.932 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.932 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.932 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:49.932 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150207+0000", 2024-09-21T15:50:49.932 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:49.933 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.933 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.933 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:49.933 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150210+0000", 2024-09-21T15:50:49.933 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.238e-06 2024-09-21T15:50:49.933 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.933 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.933 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:49.933 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150215+0000", 2024-09-21T15:50:49.933 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6350000000000003e-06 2024-09-21T15:50:49.933 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.933 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.933 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:49.933 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150220+0000", 2024-09-21T15:50:49.933 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.544e-06 2024-09-21T15:50:49.933 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.934 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.934 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:49.934 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201014+0000", 2024-09-21T15:50:49.934 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050793932999999999 2024-09-21T15:50:49.934 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.934 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.934 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:49.934 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201092+0000", 2024-09-21T15:50:49.934 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7961999999999997e-05 2024-09-21T15:50:49.934 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.934 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.934 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:49.934 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201160+0000", 2024-09-21T15:50:49.934 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7836000000000006e-05 2024-09-21T15:50:49.934 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.935 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:49.935 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.935 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.935 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.935 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62894 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 364544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:49.935 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150820+0000", 2024-09-21T15:50:49.935 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 72.731978155999997, 2024-09-21T15:50:49.935 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050489196, 2024-09-21T15:50:49.935 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:49.935 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:49.935 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:49.935 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:49.935 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:49.935 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62894 2024-09-21T15:50:49.935 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.936 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:49.936 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.936 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:49.936 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150820+0000", 2024-09-21T15:50:49.936 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.936 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.936 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.936 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:49.936 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150820+0000", 2024-09-21T15:50:49.936 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.936 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.936 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.936 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:49.936 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150223+0000", 2024-09-21T15:50:49.936 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9994035 2024-09-21T15:50:49.937 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.937 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.937 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:49.937 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150825+0000", 2024-09-21T15:50:49.937 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00060137099999999996 2024-09-21T15:50:49.937 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.937 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.937 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:49.937 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150837+0000", 2024-09-21T15:50:49.937 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2503999999999999e-05 2024-09-21T15:50:49.937 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.937 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.937 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:49.937 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150844+0000", 2024-09-21T15:50:49.938 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6900000000000003e-06 2024-09-21T15:50:49.938 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.938 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.938 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:49.938 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201187+0000", 2024-09-21T15:50:49.938 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050342973999999999 2024-09-21T15:50:49.938 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.938 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.938 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:49.938 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201250+0000", 2024-09-21T15:50:49.938 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3274000000000004e-05 2024-09-21T15:50:49.938 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.938 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.938 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:49.939 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201309+0000", 2024-09-21T15:50:49.939 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9058000000000003e-05 2024-09-21T15:50:49.939 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.939 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:49.939 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.939 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.939 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.939 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62895 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 380928~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:49.939 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150851+0000", 2024-09-21T15:50:49.939 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 72.731947559999995, 2024-09-21T15:50:49.939 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050611097000000001, 2024-09-21T15:50:49.939 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:49.939 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:49.939 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:49.939 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:49.940 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:49.940 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62895 2024-09-21T15:50:49.940 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.940 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:49.940 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.940 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:49.940 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150851+0000", 2024-09-21T15:50:49.940 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.940 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.940 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.940 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:49.940 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150851+0000", 2024-09-21T15:50:49.940 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.940 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.941 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.941 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:49.941 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150850+0000", 2024-09-21T15:50:49.941 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:49.941 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.941 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.941 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:49.941 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150853+0000", 2024-09-21T15:50:49.941 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1039999999999998e-06 2024-09-21T15:50:49.941 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.941 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.941 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:49.941 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150860+0000", 2024-09-21T15:50:49.941 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3180000000000001e-06 2024-09-21T15:50:49.942 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.942 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.942 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:49.942 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150866+0000", 2024-09-21T15:50:49.942 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1599999999999997e-06 2024-09-21T15:50:49.942 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.942 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.942 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:49.942 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201332+0000", 2024-09-21T15:50:49.942 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050466100999999999 2024-09-21T15:50:49.942 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.942 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.942 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:49.942 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201392+0000", 2024-09-21T15:50:49.943 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0053999999999998e-05 2024-09-21T15:50:49.943 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.943 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.943 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:49.943 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201462+0000", 2024-09-21T15:50:49.943 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9861000000000001e-05 2024-09-21T15:50:49.943 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.943 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:49.943 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.943 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.943 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.943 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62896 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 372736~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:49.943 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150870+0000", 2024-09-21T15:50:49.943 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 72.731927780000007, 2024-09-21T15:50:49.944 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050753292999999998, 2024-09-21T15:50:49.944 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:49.944 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:49.944 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:49.944 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:49.944 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:49.944 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62896 2024-09-21T15:50:49.944 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.944 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:49.944 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.944 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:49.944 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150870+0000", 2024-09-21T15:50:49.944 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.944 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.945 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.945 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:49.945 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150870+0000", 2024-09-21T15:50:49.945 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.945 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.945 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.945 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:49.945 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150870+0000", 2024-09-21T15:50:49.945 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:49.945 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.945 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.945 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:49.945 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150872+0000", 2024-09-21T15:50:49.945 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.644e-06 2024-09-21T15:50:49.945 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.946 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.946 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:49.946 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150879+0000", 2024-09-21T15:50:49.946 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3509999999999998e-06 2024-09-21T15:50:49.946 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.946 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.946 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:49.946 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150884+0000", 2024-09-21T15:50:49.946 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9150000000000004e-06 2024-09-21T15:50:49.946 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.946 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.946 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:49.946 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201487+0000", 2024-09-21T15:50:49.946 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050603276000000003 2024-09-21T15:50:49.946 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.947 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.947 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:49.947 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201557+0000", 2024-09-21T15:50:49.947 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9847999999999999e-05 2024-09-21T15:50:49.947 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.947 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.947 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:49.947 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201624+0000", 2024-09-21T15:50:49.947 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6730999999999995e-05 2024-09-21T15:50:49.947 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.947 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:49.947 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.947 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.947 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.948 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62897 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 389120~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:49.948 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.151044+0000", 2024-09-21T15:50:49.948 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 72.731753740000002, 2024-09-21T15:50:49.948 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050720531999999999, 2024-09-21T15:50:49.948 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:49.948 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:49.948 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:49.948 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:49.948 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:49.948 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62897 2024-09-21T15:50:49.948 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.948 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:49.948 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.948 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:49.949 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151044+0000", 2024-09-21T15:50:49.949 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.949 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.949 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.949 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:49.949 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151044+0000", 2024-09-21T15:50:49.949 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.949 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.949 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.949 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:49.949 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150888+0000", 2024-09-21T15:50:49.949 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9998431 2024-09-21T15:50:49.949 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.949 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.950 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:49.950 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151048+0000", 2024-09-21T15:50:49.950 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00016077299999999999 2024-09-21T15:50:49.950 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.950 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.950 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:49.950 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151058+0000", 2024-09-21T15:50:49.950 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.465e-06 2024-09-21T15:50:49.950 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.950 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.950 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:49.950 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151064+0000", 2024-09-21T15:50:49.950 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3829999999999997e-06 2024-09-21T15:50:49.950 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.951 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.951 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:49.951 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201648+0000", 2024-09-21T15:50:49.951 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050583597000000001 2024-09-21T15:50:49.951 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.951 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.951 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:49.951 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201711+0000", 2024-09-21T15:50:49.951 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3616999999999998e-05 2024-09-21T15:50:49.951 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.951 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.951 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:49.951 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201765+0000", 2024-09-21T15:50:49.951 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3464000000000002e-05 2024-09-21T15:50:49.952 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.952 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:49.952 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.952 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.952 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.952 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:118351 3.4 3:25980f86:::rbd_data.1133545d3fa6.000000000000005a:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-09-21T15:50:49.952 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:58.975795+0000", 2024-09-21T15:50:49.952 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 50.907003105999998, 2024-09-21T15:50:49.952 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.99016351599999997, 2024-09-21T15:50:49.952 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:49.952 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:49.952 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:49.952 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:49.952 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:49.953 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 118351 2024-09-21T15:50:49.953 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.953 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:49.953 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.953 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:49.953 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975795+0000", 2024-09-21T15:50:49.953 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.953 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.953 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.953 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:49.953 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975795+0000", 2024-09-21T15:50:49.953 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.953 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.953 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.954 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:49.954 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975793+0000", 2024-09-21T15:50:49.954 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:50:49.954 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.954 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.954 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:49.954 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975800+0000", 2024-09-21T15:50:49.954 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8220000000000001e-06 2024-09-21T15:50:49.954 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.954 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.954 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:49.954 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975814+0000", 2024-09-21T15:50:49.954 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3893000000000001e-05 2024-09-21T15:50:49.954 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.955 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.955 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:49.955 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975821+0000", 2024-09-21T15:50:49.955 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1319999999999996e-06 2024-09-21T15:50:49.955 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.955 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.955 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:49.955 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975878+0000", 2024-09-21T15:50:49.955 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7238999999999999e-05 2024-09-21T15:50:49.955 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.955 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.955 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:50:49.955 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975889+0000", 2024-09-21T15:50:49.955 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0509000000000001e-05 2024-09-21T15:50:49.955 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.956 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.956 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:49.956 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.964573+0000", 2024-09-21T15:50:49.956 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.98868396300000005 2024-09-21T15:50:49.956 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.956 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.956 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:49.956 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.964672+0000", 2024-09-21T15:50:49.956 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9601999999999999e-05 2024-09-21T15:50:49.956 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.956 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.956 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-09-21T15:50:49.956 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.964758+0000", 2024-09-21T15:50:49.956 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5537000000000005e-05 2024-09-21T15:50:49.957 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.957 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.957 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:49.957 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965847+0000", 2024-09-21T15:50:49.957 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001089404 2024-09-21T15:50:49.957 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.957 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.957 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:49.957 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965890+0000", 2024-09-21T15:50:49.957 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2521999999999997e-05 2024-09-21T15:50:49.957 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.957 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.957 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:49.957 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965934+0000", 2024-09-21T15:50:49.957 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4489e-05 2024-09-21T15:50:49.958 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.958 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.958 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:49.958 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965959+0000", 2024-09-21T15:50:49.958 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4301999999999999e-05 2024-09-21T15:50:49.958 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.958 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:49.958 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.958 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.958 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.958 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:118353 3.4 3:23841e18:::rbd_data.1133545d3fa6.0000000000000061:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-09-21T15:50:49.958 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:58.975836+0000", 2024-09-21T15:50:49.958 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 50.906961607, 2024-09-21T15:50:49.958 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.99077746700000002, 2024-09-21T15:50:49.958 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:49.959 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:49.959 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:49.959 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:49.959 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:49.959 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 118353 2024-09-21T15:50:49.959 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.959 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:49.959 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.959 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:49.959 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975836+0000", 2024-09-21T15:50:49.959 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.959 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.959 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.959 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:49.960 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975836+0000", 2024-09-21T15:50:49.960 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.960 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.960 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.960 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:49.960 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975836+0000", 2024-09-21T15:50:49.960 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:49.960 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.960 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.960 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:49.960 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975839+0000", 2024-09-21T15:50:49.960 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6129999999999998e-06 2024-09-21T15:50:49.960 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.960 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.960 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:49.961 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975844+0000", 2024-09-21T15:50:49.961 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5130000000000004e-06 2024-09-21T15:50:49.961 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.961 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.961 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:49.961 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975850+0000", 2024-09-21T15:50:49.961 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3179999999999997e-06 2024-09-21T15:50:49.961 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.961 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.961 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:49.961 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975915+0000", 2024-09-21T15:50:49.961 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5168999999999999e-05 2024-09-21T15:50:49.961 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.961 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.961 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:50:49.962 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975922+0000", 2024-09-21T15:50:49.962 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2749999999999998e-06 2024-09-21T15:50:49.962 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.962 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.962 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:49.962 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965431+0000", 2024-09-21T15:50:49.962 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.98950864699999996 2024-09-21T15:50:49.962 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.962 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.962 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:49.962 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965535+0000", 2024-09-21T15:50:49.962 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000103949 2024-09-21T15:50:49.962 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.963 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.963 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-09-21T15:50:49.963 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965607+0000", 2024-09-21T15:50:49.963 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2478000000000006e-05 2024-09-21T15:50:49.963 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.963 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.963 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:49.963 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.966205+0000", 2024-09-21T15:50:49.963 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00059775200000000001 2024-09-21T15:50:49.963 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.963 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.963 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:49.963 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.966562+0000", 2024-09-21T15:50:49.963 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00035732199999999998 2024-09-21T15:50:49.963 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.964 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.964 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:49.964 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.966593+0000", 2024-09-21T15:50:49.964 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0642999999999998e-05 2024-09-21T15:50:49.964 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.964 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.964 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:49.964 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.966614+0000", 2024-09-21T15:50:49.964 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1149999999999999e-05 2024-09-21T15:50:49.964 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.964 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:49.964 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.964 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.964 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.965 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160535 3.6 3:67cae2ed:::rbd_mirroring:head [list-watchers] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-09-21T15:50:49.965 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:30.195408+0000", 2024-09-21T15:50:49.965 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 19.687390525000001, 2024-09-21T15:50:49.965 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.96025944900000004, 2024-09-21T15:50:49.965 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:49.965 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-09-21T15:50:49.965 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:49.965 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:49.965 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:49.965 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160535 2024-09-21T15:50:49.965 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.965 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:49.965 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.965 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:49.966 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195408+0000", 2024-09-21T15:50:49.966 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.966 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.966 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.966 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:49.966 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195408+0000", 2024-09-21T15:50:49.966 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.966 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.966 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.966 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:49.966 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195406+0000", 2024-09-21T15:50:49.966 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:50:49.966 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.966 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.967 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:49.967 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195414+0000", 2024-09-21T15:50:49.967 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.2870000000000004e-06 2024-09-21T15:50:49.967 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.967 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.967 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:49.967 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195432+0000", 2024-09-21T15:50:49.967 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8110000000000001e-05 2024-09-21T15:50:49.967 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.967 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.967 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:49.967 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195441+0000", 2024-09-21T15:50:49.967 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7679999999999997e-06 2024-09-21T15:50:49.967 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.968 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.968 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:49.968 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195505+0000", 2024-09-21T15:50:49.968 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3769000000000006e-05 2024-09-21T15:50:49.968 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.968 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.968 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:50:49.968 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195518+0000", 2024-09-21T15:50:49.968 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3341e-05 2024-09-21T15:50:49.968 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.968 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.968 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:49.968 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.155509+0000", 2024-09-21T15:50:49.968 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.959990393 2024-09-21T15:50:49.968 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.968 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.969 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:49.969 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.155667+0000", 2024-09-21T15:50:49.969 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000158505 2024-09-21T15:50:49.969 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.969 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:49.969 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.969 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.969 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.969 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160974 3.4 3:22dd2475:::rbd_id.image_client.0-clone39:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-21T15:50:49.969 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:34.748093+0000", 2024-09-21T15:50:49.969 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 15.134704974, 2024-09-21T15:50:49.969 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.94006199499999998, 2024-09-21T15:50:49.969 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:49.969 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:49.969 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:49.970 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:49.970 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:49.970 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160974 2024-09-21T15:50:49.970 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.970 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:49.970 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.970 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:49.970 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748093+0000", 2024-09-21T15:50:49.970 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.970 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.970 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.970 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:49.970 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748093+0000", 2024-09-21T15:50:49.970 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:49.971 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.971 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.971 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:49.971 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748091+0000", 2024-09-21T15:50:49.971 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:50:49.971 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.971 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.971 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:49.971 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748101+0000", 2024-09-21T15:50:49.971 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5950000000000005e-06 2024-09-21T15:50:49.971 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.971 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.971 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:49.971 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748122+0000", 2024-09-21T15:50:49.972 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1611e-05 2024-09-21T15:50:49.972 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.972 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.972 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:49.972 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748133+0000", 2024-09-21T15:50:49.972 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0224000000000001e-05 2024-09-21T15:50:49.972 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.972 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.972 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:49.972 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748185+0000", 2024-09-21T15:50:49.972 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2005000000000001e-05 2024-09-21T15:50:49.972 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.972 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.972 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:50:49.972 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748194+0000", 2024-09-21T15:50:49.973 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0180000000000007e-06 2024-09-21T15:50:49.973 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.973 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.973 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:49.973 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.683582+0000", 2024-09-21T15:50:49.973 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.93538800499999997 2024-09-21T15:50:49.973 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.973 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.973 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:49.973 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.683711+0000", 2024-09-21T15:50:49.973 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000129746 2024-09-21T15:50:49.973 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.973 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.973 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-09-21T15:50:49.973 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.683838+0000", 2024-09-21T15:50:49.974 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012693200000000001 2024-09-21T15:50:49.974 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.974 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.974 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:49.974 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.687445+0000", 2024-09-21T15:50:49.974 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0036071340000000001 2024-09-21T15:50:49.974 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.974 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.974 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:49.974 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.688064+0000", 2024-09-21T15:50:49.974 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00061857100000000005 2024-09-21T15:50:49.974 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.974 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.974 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:49.974 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.688113+0000", 2024-09-21T15:50:49.975 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9128000000000001e-05 2024-09-21T15:50:49.975 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:49.975 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:49.975 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:49.975 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.688155+0000", 2024-09-21T15:50:49.975 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2026e-05 2024-09-21T15:50:49.975 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.975 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:49.975 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.975 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:49.975 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:49.975 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:49.977 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:50:49.977 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:50:49.975+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:50:50.077 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:50.076+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:50.120 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:50.120 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:50.120 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:50:50.120 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:50.129 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:50:50.178 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:50.177+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:50.230 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:50.230 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:50.230 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:50:50.230 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:50:50.230 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:50.238 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:50:50.279 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:50.278+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:50.341 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:50.341 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:50:50.341 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:50:50.341 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:50:50.342 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.342 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:984 3.0 e27/22 3:025e53e7:::rbd_data.1133f3dfbd6b.0000000000000008:head v 27'21, mlcod=27'19)", 2024-09-21T15:50:50.342 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.619393+0000", 2024-09-21T15:50:50.342 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 97.72046383, 2024-09-21T15:50:50.342 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003401837, 2024-09-21T15:50:50.342 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:50.342 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:50.342 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:50.342 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.342 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:50.342 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619393+0000", 2024-09-21T15:50:50.342 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:50.342 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.342 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.343 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:50.343 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619393+0000", 2024-09-21T15:50:50.343 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:50.343 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.343 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.343 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:50.343 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619392+0000", 2024-09-21T15:50:50.343 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:50.343 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.343 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.343 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:50.343 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619439+0000", 2024-09-21T15:50:50.343 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7170999999999999e-05 2024-09-21T15:50:50.343 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.344 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.344 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:50.344 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619452+0000", 2024-09-21T15:50:50.344 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2986999999999999e-05 2024-09-21T15:50:50.344 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.344 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.344 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:50.344 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619459+0000", 2024-09-21T15:50:50.344 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6109999999999999e-06 2024-09-21T15:50:50.344 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.344 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.344 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:50.344 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619518+0000", 2024-09-21T15:50:50.344 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9117999999999999e-05 2024-09-21T15:50:50.344 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.344 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.345 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:50.345 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619532+0000", 2024-09-21T15:50:50.345 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4622e-05 2024-09-21T15:50:50.345 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.345 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.345 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:50.345 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.622760+0000", 2024-09-21T15:50:50.345 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032278839999999999 2024-09-21T15:50:50.345 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.345 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.345 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:50.345 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.622794+0000", 2024-09-21T15:50:50.345 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4144999999999999e-05 2024-09-21T15:50:50.345 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:50.346 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:50.346 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:50.346 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.346 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.346 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:4957 3.9 e29/22 3:95c5ed95:::rbd_data.11335fa2fe69.0000000000000018:head v 29'4, mlcod=26'2)", 2024-09-21T15:50:50.346 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:14.453951+0000", 2024-09-21T15:50:50.346 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 95.885905398000006, 2024-09-21T15:50:50.346 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0043804559999999996, 2024-09-21T15:50:50.346 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:50.346 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:50.346 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:50.346 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.346 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:50.346 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.453951+0000", 2024-09-21T15:50:50.347 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:50.347 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.347 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.347 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:50.347 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.453951+0000", 2024-09-21T15:50:50.347 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:50.347 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.347 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.347 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:50.347 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.453950+0000", 2024-09-21T15:50:50.347 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:50.347 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.347 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.347 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:50.347 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454495+0000", 2024-09-21T15:50:50.348 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00054437799999999998 2024-09-21T15:50:50.348 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.348 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.348 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:50.348 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454508+0000", 2024-09-21T15:50:50.348 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3475000000000001e-05 2024-09-21T15:50:50.348 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.348 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.348 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:50.348 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454515+0000", 2024-09-21T15:50:50.348 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8990000000000004e-06 2024-09-21T15:50:50.348 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.348 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.348 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:50.349 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454581+0000", 2024-09-21T15:50:50.349 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5987000000000004e-05 2024-09-21T15:50:50.349 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.349 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.349 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:50.349 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454609+0000", 2024-09-21T15:50:50.349 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7466000000000001e-05 2024-09-21T15:50:50.349 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.349 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.349 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:50.349 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.458303+0000", 2024-09-21T15:50:50.349 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003693979 2024-09-21T15:50:50.349 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.349 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.349 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:50.350 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.458331+0000", 2024-09-21T15:50:50.350 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8932999999999999e-05 2024-09-21T15:50:50.350 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:50.350 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:50.350 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:50.350 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.350 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.350 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13420 3.0 e31/22 3:038b8840:::rbd_data.1133dc5adfdf.0000000000000035:head v 31'49, mlcod=31'47)", 2024-09-21T15:50:50.350 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.209233+0000", 2024-09-21T15:50:50.350 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 93.130623782000001, 2024-09-21T15:50:50.350 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0042624960000000002, 2024-09-21T15:50:50.350 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:50.350 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:50.350 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:50.351 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.351 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:50.351 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209233+0000", 2024-09-21T15:50:50.351 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:50.351 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.351 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.351 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:50.351 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209233+0000", 2024-09-21T15:50:50.351 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:50.351 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.351 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.351 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:50.351 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209232+0000", 2024-09-21T15:50:50.351 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:50.352 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.352 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.352 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:50.352 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209251+0000", 2024-09-21T15:50:50.352 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9304e-05 2024-09-21T15:50:50.352 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.352 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.352 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:50.352 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209265+0000", 2024-09-21T15:50:50.352 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3532e-05 2024-09-21T15:50:50.352 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.352 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.352 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:50.352 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209272+0000", 2024-09-21T15:50:50.352 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.289e-06 2024-09-21T15:50:50.352 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.353 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.353 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:50.353 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209321+0000", 2024-09-21T15:50:50.353 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8773999999999998e-05 2024-09-21T15:50:50.353 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.353 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.353 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:50.353 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209341+0000", 2024-09-21T15:50:50.353 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0387999999999999e-05 2024-09-21T15:50:50.353 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.353 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.353 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:50.353 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.213464+0000", 2024-09-21T15:50:50.353 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0041232450000000002 2024-09-21T15:50:50.353 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.354 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.354 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:50.354 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.213495+0000", 2024-09-21T15:50:50.354 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0793999999999997e-05 2024-09-21T15:50:50.354 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:50.354 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:50.354 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:50.354 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.354 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.354 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13445 3.0 e31/22 3:038b8840:::rbd_data.1133dc5adfdf.0000000000000035:head v 31'50, mlcod=31'48)", 2024-09-21T15:50:50.354 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.232497+0000", 2024-09-21T15:50:50.354 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 93.107359770000002, 2024-09-21T15:50:50.354 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0040493559999999996, 2024-09-21T15:50:50.354 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:50.355 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:50.355 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:50.355 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.355 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:50.355 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232497+0000", 2024-09-21T15:50:50.355 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:50.355 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.355 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.355 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:50.355 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232497+0000", 2024-09-21T15:50:50.355 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:50.355 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.356 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.356 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:50.356 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232496+0000", 2024-09-21T15:50:50.356 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:50.356 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.356 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.356 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:50.357 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232503+0000", 2024-09-21T15:50:50.357 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7320000000000001e-06 2024-09-21T15:50:50.357 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.357 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.357 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:50.357 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232512+0000", 2024-09-21T15:50:50.357 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9259999999999995e-06 2024-09-21T15:50:50.357 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.357 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.357 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:50.357 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232520+0000", 2024-09-21T15:50:50.357 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3699999999999997e-06 2024-09-21T15:50:50.357 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.357 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.357 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:50.358 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232564+0000", 2024-09-21T15:50:50.358 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4128999999999997e-05 2024-09-21T15:50:50.358 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.358 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.358 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:50.358 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232585+0000", 2024-09-21T15:50:50.358 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0704999999999999e-05 2024-09-21T15:50:50.358 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.358 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.358 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:50.358 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.236507+0000", 2024-09-21T15:50:50.358 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0039218730000000002 2024-09-21T15:50:50.358 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.359 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.359 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:50.359 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.236546+0000", 2024-09-21T15:50:50.359 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9399e-05 2024-09-21T15:50:50.359 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:50.359 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:50.359 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:50.359 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.359 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.359 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13832 3.0 e31/22 3:0792efd2:::rbd_data.1133dc5adfdf.0000000000000090:head v 31'58, mlcod=31'56)", 2024-09-21T15:50:50.359 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.527530+0000", 2024-09-21T15:50:50.359 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 92.812326244000005, 2024-09-21T15:50:50.359 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035256530000000001, 2024-09-21T15:50:50.359 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:50.360 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:50.360 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:50.360 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.360 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:50.360 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527530+0000", 2024-09-21T15:50:50.360 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:50.360 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.360 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.360 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:50.360 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527530+0000", 2024-09-21T15:50:50.360 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:50.360 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.360 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.360 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:50.360 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527529+0000", 2024-09-21T15:50:50.360 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:50:50.360 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.361 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.361 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:50.361 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527550+0000", 2024-09-21T15:50:50.361 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1387999999999999e-05 2024-09-21T15:50:50.361 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.361 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.361 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:50.361 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527568+0000", 2024-09-21T15:50:50.361 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8068e-05 2024-09-21T15:50:50.361 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.361 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.361 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:50.361 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527577+0000", 2024-09-21T15:50:50.362 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9919999999999996e-06 2024-09-21T15:50:50.362 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.362 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.362 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:50.362 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527626+0000", 2024-09-21T15:50:50.362 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8597000000000002e-05 2024-09-21T15:50:50.362 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.362 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.362 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:50.362 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527652+0000", 2024-09-21T15:50:50.362 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5993e-05 2024-09-21T15:50:50.362 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.362 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.362 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:50.362 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.531024+0000", 2024-09-21T15:50:50.363 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033721910000000001 2024-09-21T15:50:50.363 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.363 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.363 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:50.363 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.531056+0000", 2024-09-21T15:50:50.363 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1628000000000003e-05 2024-09-21T15:50:50.363 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:50.363 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:50.363 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:50.363 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.363 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.363 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13839 3.0 e31/22 3:03c4fb32:::rbd_data.1133dc5adfdf.0000000000000099:head v 31'59, mlcod=31'56)", 2024-09-21T15:50:50.363 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.528097+0000", 2024-09-21T15:50:50.363 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 92.811759645999999, 2024-09-21T15:50:50.364 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0057228840000000001, 2024-09-21T15:50:50.364 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:50.364 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:50.364 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:50.364 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.364 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:50.364 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528097+0000", 2024-09-21T15:50:50.364 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:50.364 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.364 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.364 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:50.364 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528097+0000", 2024-09-21T15:50:50.364 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:50.364 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.364 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.365 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:50.365 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528096+0000", 2024-09-21T15:50:50.365 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:50.365 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.365 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.365 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:50.365 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528105+0000", 2024-09-21T15:50:50.365 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7350000000000002e-06 2024-09-21T15:50:50.365 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.365 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.365 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:50.365 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528117+0000", 2024-09-21T15:50:50.365 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1909000000000001e-05 2024-09-21T15:50:50.365 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.365 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.366 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:50.366 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528124+0000", 2024-09-21T15:50:50.366 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8419999999999999e-06 2024-09-21T15:50:50.366 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.366 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.366 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:50.366 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528168+0000", 2024-09-21T15:50:50.366 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4301999999999997e-05 2024-09-21T15:50:50.366 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.366 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.366 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:50.366 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528191+0000", 2024-09-21T15:50:50.366 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2622000000000001e-05 2024-09-21T15:50:50.366 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.367 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.367 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:50.367 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.533780+0000", 2024-09-21T15:50:50.367 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0055897519999999999 2024-09-21T15:50:50.367 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.367 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.367 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:50.367 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.533820+0000", 2024-09-21T15:50:50.367 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9366999999999998e-05 2024-09-21T15:50:50.367 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:50.367 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:50.367 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:50.367 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.367 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.367 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13884 3.9 e31/22 3:93003736:::rbd_data.1133dc5adfdf.00000000000000a1:head v 31'54, mlcod=31'52)", 2024-09-21T15:50:50.368 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.562433+0000", 2024-09-21T15:50:50.368 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 92.777422897999998, 2024-09-21T15:50:50.368 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.005162313, 2024-09-21T15:50:50.368 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:50.368 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:50.368 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:50.368 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.368 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:50.368 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562433+0000", 2024-09-21T15:50:50.368 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:50.368 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.368 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.368 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:50.368 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562433+0000", 2024-09-21T15:50:50.368 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:50.368 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.369 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.369 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:50.369 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562432+0000", 2024-09-21T15:50:50.369 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:50.369 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.369 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.369 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:50.369 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562467+0000", 2024-09-21T15:50:50.369 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4885999999999998e-05 2024-09-21T15:50:50.369 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.369 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.369 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:50.369 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562481+0000", 2024-09-21T15:50:50.369 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3604000000000001e-05 2024-09-21T15:50:50.369 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.370 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.370 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:50.370 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562489+0000", 2024-09-21T15:50:50.370 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1829999999999994e-06 2024-09-21T15:50:50.370 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.370 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.370 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:50.370 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562536+0000", 2024-09-21T15:50:50.370 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6830999999999999e-05 2024-09-21T15:50:50.370 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.370 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.370 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:50.370 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562559+0000", 2024-09-21T15:50:50.370 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2765999999999999e-05 2024-09-21T15:50:50.371 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.371 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.371 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:50.371 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.567566+0000", 2024-09-21T15:50:50.371 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0050073219999999998 2024-09-21T15:50:50.371 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.371 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.371 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:50.371 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.567596+0000", 2024-09-21T15:50:50.371 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9802999999999999e-05 2024-09-21T15:50:50.371 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:50.371 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:50.371 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:50.371 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.372 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.372 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:22340 3.9 e34/22 3:910a5abe:::rbd_data.1133896adcfd.0000000000000023:head v 34'92, mlcod=34'90)", 2024-09-21T15:50:50.372 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:21.628025+0000", 2024-09-21T15:50:50.372 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 88.711831517999997, 2024-09-21T15:50:50.372 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046196010000000001, 2024-09-21T15:50:50.372 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:50.372 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:50.372 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:50.372 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.372 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:50.372 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628025+0000", 2024-09-21T15:50:50.372 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:50.372 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.372 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.372 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:50.373 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628025+0000", 2024-09-21T15:50:50.373 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:50.373 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.373 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.373 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:50.373 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628024+0000", 2024-09-21T15:50:50.373 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:50.373 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.373 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.373 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:50.373 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628034+0000", 2024-09-21T15:50:50.373 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3470000000000005e-06 2024-09-21T15:50:50.373 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.373 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.374 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:50.374 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628043+0000", 2024-09-21T15:50:50.374 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9450000000000005e-06 2024-09-21T15:50:50.374 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.374 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.374 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:50.374 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628048+0000", 2024-09-21T15:50:50.374 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8080000000000004e-06 2024-09-21T15:50:50.374 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.374 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.374 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:50.374 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628081+0000", 2024-09-21T15:50:50.374 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2738999999999999e-05 2024-09-21T15:50:50.375 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.375 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.375 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:50.375 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628094+0000", 2024-09-21T15:50:50.375 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3273e-05 2024-09-21T15:50:50.375 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.375 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.375 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:50.375 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.632616+0000", 2024-09-21T15:50:50.375 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045217269999999997 2024-09-21T15:50:50.375 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.375 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.375 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:50.375 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.632644+0000", 2024-09-21T15:50:50.375 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8469000000000001e-05 2024-09-21T15:50:50.376 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:50.376 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:50.376 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:50.376 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.376 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.376 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:22417 3.4 e34/22 3:25310a10:::rbd_data.1133896adcfd.0000000000000075:head v 34'48, mlcod=34'46)", 2024-09-21T15:50:50.376 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:21.704297+0000", 2024-09-21T15:50:50.376 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 88.635559420000007, 2024-09-21T15:50:50.376 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0038101799999999998, 2024-09-21T15:50:50.376 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:50.376 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:50.376 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:50.376 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.376 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:50.376 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704297+0000", 2024-09-21T15:50:50.376 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:50.377 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.377 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.377 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:50.377 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704297+0000", 2024-09-21T15:50:50.377 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:50.377 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.377 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.377 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:50.377 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704296+0000", 2024-09-21T15:50:50.377 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:50:50.377 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.377 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.377 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:50.377 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704308+0000", 2024-09-21T15:50:50.377 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2559e-05 2024-09-21T15:50:50.378 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.378 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.378 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:50.378 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704337+0000", 2024-09-21T15:50:50.378 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8960000000000001e-05 2024-09-21T15:50:50.378 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.378 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.378 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:50.378 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704345+0000", 2024-09-21T15:50:50.378 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3059999999999998e-06 2024-09-21T15:50:50.378 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.378 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.378 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:50.378 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704408+0000", 2024-09-21T15:50:50.378 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2156000000000006e-05 2024-09-21T15:50:50.379 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.379 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.379 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:50.379 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704430+0000", 2024-09-21T15:50:50.379 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1931000000000001e-05 2024-09-21T15:50:50.379 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.379 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.379 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:50.379 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.708073+0000", 2024-09-21T15:50:50.379 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003643028 2024-09-21T15:50:50.379 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.379 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.379 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:50.380 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.708107+0000", 2024-09-21T15:50:50.380 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4613999999999998e-05 2024-09-21T15:50:50.380 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:50.380 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:50.380 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:50.380 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.380 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.380 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:25278 3.4 e38/22 3:2251cc1c:::rbd_data.1133ebd6d3ba.0000000000000024:head v 38'51, mlcod=38'49)", 2024-09-21T15:50:50.380 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.317337+0000", 2024-09-21T15:50:50.380 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 84.02251966, 2024-09-21T15:50:50.380 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053904110000000003, 2024-09-21T15:50:50.380 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:50.380 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:50.380 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:50.380 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.381 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:50.381 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317337+0000", 2024-09-21T15:50:50.381 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:50.381 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.381 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.381 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:50.381 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317337+0000", 2024-09-21T15:50:50.381 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:50.381 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.381 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.381 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:50.381 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317335+0000", 2024-09-21T15:50:50.381 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:50:50.381 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.382 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.382 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:50.382 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317347+0000", 2024-09-21T15:50:50.382 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.19e-05 2024-09-21T15:50:50.382 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.382 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.382 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:50.382 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317369+0000", 2024-09-21T15:50:50.382 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2232e-05 2024-09-21T15:50:50.382 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.382 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.382 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:50.382 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317381+0000", 2024-09-21T15:50:50.382 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2231e-05 2024-09-21T15:50:50.383 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.383 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.383 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:50.383 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317425+0000", 2024-09-21T15:50:50.383 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3625000000000003e-05 2024-09-21T15:50:50.383 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.383 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.383 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:50.383 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317449+0000", 2024-09-21T15:50:50.383 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4301999999999999e-05 2024-09-21T15:50:50.383 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.383 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.383 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:50.383 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.322688+0000", 2024-09-21T15:50:50.383 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.005238609 2024-09-21T15:50:50.384 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.384 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.384 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:50.384 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.322727+0000", 2024-09-21T15:50:50.384 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9201e-05 2024-09-21T15:50:50.384 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:50.384 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:50.384 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:50.384 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.384 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.384 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:25286 3.0 e38/22 3:0248fb60:::rbd_data.1133ebd6d3ba.0000000000000066:head v 38'109, mlcod=34'107)", 2024-09-21T15:50:50.384 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.317638+0000", 2024-09-21T15:50:50.384 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 84.022218781000007, 2024-09-21T15:50:50.384 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0054074370000000002, 2024-09-21T15:50:50.384 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:50.385 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:50.385 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:50.385 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.385 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:50.385 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317638+0000", 2024-09-21T15:50:50.385 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:50.385 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.385 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.385 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:50.385 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317638+0000", 2024-09-21T15:50:50.385 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:50.385 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.385 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.385 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:50.385 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317637+0000", 2024-09-21T15:50:50.386 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:50.386 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.386 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.386 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:50.386 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317643+0000", 2024-09-21T15:50:50.386 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5609999999999998e-06 2024-09-21T15:50:50.386 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.386 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.386 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:50.386 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317653+0000", 2024-09-21T15:50:50.386 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0705e-05 2024-09-21T15:50:50.386 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.386 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.386 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:50.387 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317660+0000", 2024-09-21T15:50:50.387 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2099999999999998e-06 2024-09-21T15:50:50.387 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.387 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.387 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:50.387 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317704+0000", 2024-09-21T15:50:50.387 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4659000000000001e-05 2024-09-21T15:50:50.387 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.387 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.387 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:50.387 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317725+0000", 2024-09-21T15:50:50.387 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0537e-05 2024-09-21T15:50:50.387 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.387 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.387 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:50.388 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323010+0000", 2024-09-21T15:50:50.388 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052853420000000002 2024-09-21T15:50:50.388 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.388 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.388 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:50.388 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323045+0000", 2024-09-21T15:50:50.388 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4860000000000002e-05 2024-09-21T15:50:50.388 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:50.388 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:50.388 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:50.388 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.388 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.388 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:25280 3.4 e38/22 3:27fca38c:::rbd_data.1133ebd6d3ba.0000000000000026:head v 38'52, mlcod=38'49)", 2024-09-21T15:50:50.388 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.317684+0000", 2024-09-21T15:50:50.388 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 84.022172885000003, 2024-09-21T15:50:50.389 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053570809999999996, 2024-09-21T15:50:50.389 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:50.389 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:50.389 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:50.389 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.389 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:50.389 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317684+0000", 2024-09-21T15:50:50.389 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:50.389 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.389 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.389 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:50.389 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317684+0000", 2024-09-21T15:50:50.389 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:50.389 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.389 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.390 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:50.390 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317683+0000", 2024-09-21T15:50:50.390 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:50.390 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.390 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.390 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:50.390 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317688+0000", 2024-09-21T15:50:50.390 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5009999999999998e-06 2024-09-21T15:50:50.390 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.390 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.390 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:50.390 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317694+0000", 2024-09-21T15:50:50.390 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3579999999999999e-06 2024-09-21T15:50:50.391 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.391 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.391 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:50.391 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317698+0000", 2024-09-21T15:50:50.391 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4150000000000001e-06 2024-09-21T15:50:50.391 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.391 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.391 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:50.391 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317738+0000", 2024-09-21T15:50:50.391 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9575e-05 2024-09-21T15:50:50.391 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.391 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.391 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:50.391 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317762+0000", 2024-09-21T15:50:50.392 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4173999999999999e-05 2024-09-21T15:50:50.392 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.392 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.392 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:50.392 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323005+0000", 2024-09-21T15:50:50.392 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00524333 2024-09-21T15:50:50.392 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.392 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.392 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:50.392 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323041+0000", 2024-09-21T15:50:50.392 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5139999999999999e-05 2024-09-21T15:50:50.392 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:50.392 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:50.392 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:50.392 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.392 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.393 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:34837 3.0 e43/22 3:038527c8:::rbd_header.1133fed1a840:head v 43'130, mlcod=43'128)", 2024-09-21T15:50:50.393 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:50:50.393 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 79.355968187000002, 2024-09-21T15:50:50.393 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045776519999999998, 2024-09-21T15:50:50.393 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:50.393 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:50.393 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:50.393 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.393 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:50.393 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:50:50.393 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:50.393 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.393 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.393 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:50.393 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:50:50.394 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:50.394 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.394 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.394 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:50.394 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:50:50.394 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:50.394 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.394 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.394 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:50.394 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983892+0000", 2024-09-21T15:50:50.394 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8450000000000002e-06 2024-09-21T15:50:50.394 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.394 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.394 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:50.395 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983902+0000", 2024-09-21T15:50:50.395 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0299999999999999e-06 2024-09-21T15:50:50.395 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.395 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.395 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:50.395 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983906+0000", 2024-09-21T15:50:50.395 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2300000000000002e-06 2024-09-21T15:50:50.395 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.395 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.395 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:50.395 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983940+0000", 2024-09-21T15:50:50.395 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4563000000000001e-05 2024-09-21T15:50:50.396 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.396 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.396 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:50.396 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983960+0000", 2024-09-21T15:50:50.396 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9199999999999999e-05 2024-09-21T15:50:50.396 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.396 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.396 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:50.396 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.988430+0000", 2024-09-21T15:50:50.396 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0044709149999999998 2024-09-21T15:50:50.396 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.396 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.396 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:50.396 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.988466+0000", 2024-09-21T15:50:50.397 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5420000000000003e-05 2024-09-21T15:50:50.397 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:50.397 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:50.397 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:50.397 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.397 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.397 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "pg_update_log_missing(3.0 epoch 50/22 rep_tid 498 entries 50'171 (0'0) error 3:5df4a5a0:::rbd_data.11339e87a32a.000000000000008e:head by client.4403.0:80207 0.000000 -2 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 47'167)", 2024-09-21T15:50:50.397 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:50:50.397 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 69.398095314000003, 2024-09-21T15:50:50.397 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0047253119999999997, 2024-09-21T15:50:50.397 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:50.397 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-09-21T15:50:50.397 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:50.397 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.398 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:50.398 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:50:50.398 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:50.398 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.398 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.398 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:50.398 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:50:50.398 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:50.398 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.398 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.398 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:50.398 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:50:50.398 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:50.398 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.399 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.399 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:50.399 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941768+0000", 2024-09-21T15:50:50.399 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7309999999999993e-06 2024-09-21T15:50:50.399 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.399 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.399 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:50.399 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941780+0000", 2024-09-21T15:50:50.399 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1399999999999999e-05 2024-09-21T15:50:50.399 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.399 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.399 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:50.399 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941789+0000", 2024-09-21T15:50:50.399 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5230000000000001e-06 2024-09-21T15:50:50.400 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.400 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.400 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:50.400 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941841+0000", 2024-09-21T15:50:50.400 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1863999999999997e-05 2024-09-21T15:50:50.400 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.400 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.400 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:50.400 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.946486+0000", 2024-09-21T15:50:50.400 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046452539999999997 2024-09-21T15:50:50.400 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:50.400 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:50.400 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:50.400 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.401 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.401 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "pg_update_log_missing(3.0 epoch 53/22 rep_tid 585 entries 53'201 (0'0) error 3:6a1d4890:::rbd_data.1133e079cae4.0000000000000004:head by client.4403.0:101089 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 53'198)", 2024-09-21T15:50:50.401 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:44.832827+0000", 2024-09-21T15:50:50.401 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 65.507029505999995, 2024-09-21T15:50:50.401 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053552929999999997, 2024-09-21T15:50:50.401 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:50.401 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-09-21T15:50:50.401 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:50.401 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.401 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:50.401 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832827+0000", 2024-09-21T15:50:50.401 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:50.401 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.401 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.401 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:50.402 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832827+0000", 2024-09-21T15:50:50.402 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:50.402 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.402 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.402 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:50.402 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832826+0000", 2024-09-21T15:50:50.402 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:50.402 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.402 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.402 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:50.402 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832835+0000", 2024-09-21T15:50:50.402 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4509999999999992e-06 2024-09-21T15:50:50.402 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.402 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.403 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:50.403 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832854+0000", 2024-09-21T15:50:50.403 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9352999999999999e-05 2024-09-21T15:50:50.403 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.403 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.403 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:50.403 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832866+0000", 2024-09-21T15:50:50.403 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1766000000000001e-05 2024-09-21T15:50:50.403 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.403 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.403 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:50.403 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.833091+0000", 2024-09-21T15:50:50.403 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00022490300000000001 2024-09-21T15:50:50.403 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.404 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.404 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:50.404 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.838182+0000", 2024-09-21T15:50:50.404 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0050915420000000001 2024-09-21T15:50:50.404 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:50.404 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:50.404 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:50.404 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.404 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.404 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:102822 3.4 e54/22 3:2729ba03:::rbd_object_map.113336ae1075:head v 54'208, mlcod=54'205)", 2024-09-21T15:50:50.404 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:46.249540+0000", 2024-09-21T15:50:50.404 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 64.090316274000003, 2024-09-21T15:50:50.404 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052654859999999998, 2024-09-21T15:50:50.404 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:50.404 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:50.405 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:50.405 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.405 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:50.405 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249540+0000", 2024-09-21T15:50:50.405 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:50.405 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.405 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.405 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:50.405 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249540+0000", 2024-09-21T15:50:50.405 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:50.405 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.405 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.405 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:50.405 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249539+0000", 2024-09-21T15:50:50.405 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:50.406 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.406 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.406 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:50.406 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249546+0000", 2024-09-21T15:50:50.406 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7290000000000003e-06 2024-09-21T15:50:50.406 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.406 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.406 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:50.406 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249558+0000", 2024-09-21T15:50:50.406 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1844e-05 2024-09-21T15:50:50.406 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.406 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.406 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:50.406 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249565+0000", 2024-09-21T15:50:50.407 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8210000000000004e-06 2024-09-21T15:50:50.407 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.407 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.407 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:50.407 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249602+0000", 2024-09-21T15:50:50.407 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7057000000000001e-05 2024-09-21T15:50:50.407 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.407 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.407 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:50.407 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249622+0000", 2024-09-21T15:50:50.407 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9678999999999999e-05 2024-09-21T15:50:50.407 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.407 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.408 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:50.408 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254783+0000", 2024-09-21T15:50:50.408 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0051611449999999998 2024-09-21T15:50:50.408 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.408 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.408 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:50.408 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254806+0000", 2024-09-21T15:50:50.408 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2853e-05 2024-09-21T15:50:50.408 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:50.408 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:50.408 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:50.408 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.408 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.408 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:102823 3.4 e54/22 3:2729ba03:::rbd_object_map.113336ae1075:head v 54'209, mlcod=54'205)", 2024-09-21T15:50:50.408 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:46.250318+0000", 2024-09-21T15:50:50.409 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 64.089538688999994, 2024-09-21T15:50:50.409 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045013539999999999, 2024-09-21T15:50:50.409 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:50.409 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:50.409 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:50.409 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.409 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:50.409 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250318+0000", 2024-09-21T15:50:50.409 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:50.409 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.409 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.409 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:50.409 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250318+0000", 2024-09-21T15:50:50.410 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:50.410 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.410 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.410 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:50.410 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250317+0000", 2024-09-21T15:50:50.410 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:50.410 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.410 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.410 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:50.410 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250322+0000", 2024-09-21T15:50:50.410 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.074e-06 2024-09-21T15:50:50.410 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.410 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.411 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:50.411 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250330+0000", 2024-09-21T15:50:50.411 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4730000000000003e-06 2024-09-21T15:50:50.411 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.411 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.411 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:50.411 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250335+0000", 2024-09-21T15:50:50.411 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8019999999999997e-06 2024-09-21T15:50:50.411 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.411 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.411 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:50.411 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250375+0000", 2024-09-21T15:50:50.411 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9459999999999998e-05 2024-09-21T15:50:50.411 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.411 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.412 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:50.412 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250388+0000", 2024-09-21T15:50:50.412 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3288999999999999e-05 2024-09-21T15:50:50.412 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.412 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.412 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:50.412 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254807+0000", 2024-09-21T15:50:50.412 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0044192220000000004 2024-09-21T15:50:50.412 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.412 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.412 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:50.412 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254819+0000", 2024-09-21T15:50:50.412 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1603999999999999e-05 2024-09-21T15:50:50.412 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:50.413 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:50.413 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:50.413 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.413 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.413 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:160603 3.9 e95/22 3:9171211b:::rbd_data.11338bae974a.0000000000000006:head v 95'343, mlcod=92'340)", 2024-09-21T15:50:50.413 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.215621+0000", 2024-09-21T15:50:50.413 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 19.124235658, 2024-09-21T15:50:50.413 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0041276250000000002, 2024-09-21T15:50:50.413 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:50.413 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:50.413 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:50.413 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.413 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:50.413 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215621+0000", 2024-09-21T15:50:50.413 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:50.414 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.414 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.414 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:50.414 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215621+0000", 2024-09-21T15:50:50.414 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:50.414 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.414 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.414 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:50.414 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215620+0000", 2024-09-21T15:50:50.414 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:50.414 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.414 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.414 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:50.415 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215627+0000", 2024-09-21T15:50:50.415 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8850000000000001e-06 2024-09-21T15:50:50.415 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.415 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.415 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:50.415 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215803+0000", 2024-09-21T15:50:50.415 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000176422 2024-09-21T15:50:50.415 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.415 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.415 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:50.415 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215814+0000", 2024-09-21T15:50:50.415 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0135e-05 2024-09-21T15:50:50.415 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.415 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.415 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:50.416 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215866+0000", 2024-09-21T15:50:50.416 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2327999999999998e-05 2024-09-21T15:50:50.416 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.416 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.416 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:50.416 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215887+0000", 2024-09-21T15:50:50.416 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1449000000000001e-05 2024-09-21T15:50:50.416 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.416 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.416 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:50.416 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.219719+0000", 2024-09-21T15:50:50.416 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0038312630000000001 2024-09-21T15:50:50.416 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.416 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.416 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:50.417 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.219748+0000", 2024-09-21T15:50:50.417 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9737999999999999e-05 2024-09-21T15:50:50.417 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:50.417 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:50.417 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:50.417 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.417 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.417 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:160974 3.4 e100/99 3:22dd2475:::rbd_id.image_client.0-clone39:head v 100'593, mlcod=0'0)", 2024-09-21T15:50:50.417 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:35.683900+0000", 2024-09-21T15:50:50.417 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.655956224000001, 2024-09-21T15:50:50.417 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0034160029999999999, 2024-09-21T15:50:50.417 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:50.417 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:50.417 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:50.417 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.418 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:50.418 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.683900+0000", 2024-09-21T15:50:50.418 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:50.418 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.418 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.418 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:50.418 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.683900+0000", 2024-09-21T15:50:50.418 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:50.418 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.418 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.418 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:50.418 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.683900+0000", 2024-09-21T15:50:50.418 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:50.418 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.418 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.419 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:50.419 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.683906+0000", 2024-09-21T15:50:50.419 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1759999999999998e-06 2024-09-21T15:50:50.419 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.419 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.419 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:50.419 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.683915+0000", 2024-09-21T15:50:50.419 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4320000000000005e-06 2024-09-21T15:50:50.419 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.419 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.419 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:50.419 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.683923+0000", 2024-09-21T15:50:50.419 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.259e-06 2024-09-21T15:50:50.419 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.419 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.420 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:50.420 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.683961+0000", 2024-09-21T15:50:50.420 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7014000000000002e-05 2024-09-21T15:50:50.420 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.420 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.420 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:50.420 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.683982+0000", 2024-09-21T15:50:50.420 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.145e-05 2024-09-21T15:50:50.420 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.420 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.420 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:50.420 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.687265+0000", 2024-09-21T15:50:50.420 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003283252 2024-09-21T15:50:50.420 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.420 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.421 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:50.421 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.687316+0000", 2024-09-21T15:50:50.421 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0963e-05 2024-09-21T15:50:50.421 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:50.421 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:50.421 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:50.421 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.421 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.421 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:161865 3.9 e105/22 3:90e88bad:::rbd_data.11338db70e6f.0000000000000015:head v 105'347, mlcod=103'345)", 2024-09-21T15:50:50.421 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:40.786848+0000", 2024-09-21T15:50:50.421 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.5530087560000005, 2024-09-21T15:50:50.421 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0056408860000000003, 2024-09-21T15:50:50.421 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:50.421 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:50.421 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:50.422 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.422 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:50.422 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786848+0000", 2024-09-21T15:50:50.422 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:50.422 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.422 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.422 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:50.422 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786848+0000", 2024-09-21T15:50:50.422 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:50.422 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.422 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.422 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:50.422 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786847+0000", 2024-09-21T15:50:50.422 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:50.422 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.423 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.423 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:50.423 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786851+0000", 2024-09-21T15:50:50.423 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7419999999999999e-06 2024-09-21T15:50:50.423 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.423 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.423 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:50.423 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786872+0000", 2024-09-21T15:50:50.423 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1186e-05 2024-09-21T15:50:50.423 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.423 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.423 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:50.423 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786878+0000", 2024-09-21T15:50:50.423 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0970000000000002e-06 2024-09-21T15:50:50.424 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.424 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.424 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:50.424 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786918+0000", 2024-09-21T15:50:50.424 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.998e-05 2024-09-21T15:50:50.424 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.424 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.424 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:50.424 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786941+0000", 2024-09-21T15:50:50.424 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.245e-05 2024-09-21T15:50:50.424 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.424 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.424 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:50.424 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.792460+0000", 2024-09-21T15:50:50.424 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0055191210000000001 2024-09-21T15:50:50.425 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:50.425 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:50.425 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:50.425 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.792489+0000", 2024-09-21T15:50:50.425 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8552e-05 2024-09-21T15:50:50.425 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:50.425 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:50.425 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:50.425 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:50.425 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:50.425 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:50.426 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:50:50.427 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:50.425+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:50.527 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:50.526+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:50.530 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:50.530 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:50.530 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:50:50.530 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:50.537 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:50:50.539 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:50:50.627 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:50.626+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:50.685 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T15:50:50.688 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:50.688 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:50.688 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:50:50.689 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:50:50.689 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:50.699 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:50:50.708 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-09-21T15:50:50.709 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:50:50.728 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:50.727+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:50.731 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:50:50.731 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:50:50.751 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:50:50.751 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:50:50.776 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:50:50.776 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:50:50.792 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:50:50.792 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:50:50.810 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:50:50.810 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:50:50.826 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:50:50.827 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:50:50.830 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:50.829+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:50.840 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:50.840 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:50.843 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:50:50.843 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:50:50.853 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:50:50.930 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:50.929+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:50.963 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:50.963 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:50.971 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:50:51.031 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:51.029+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:51.073 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:51.074 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:51.082 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:50:51.166 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:51.165+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:51.224 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:51.225 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:51.233 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:50:51.233 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:51.232+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:51.334 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:51.333+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:51.375 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:51.376 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:51.384 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:50:51.435 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:51.434+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:51.484 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:51.485 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:51.493 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:50:51.536 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:50:51.535+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:50:51.595 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:51.596 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:51.604 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T15:50:51.647 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:51.646+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:51.680 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is unset 2024-09-21T15:50:51.708 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:51.708 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:51.717 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T15:50:51.758 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:51.757+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:51.838 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:51.837+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:51.859 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:51.859 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:51.868 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T15:50:51.939 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:51.938+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:52.009 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:52.009 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:52.018 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T15:50:52.039 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:50:52.039+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:50:52.121 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:52.121 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:52.130 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T15:50:52.174 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:52.173+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:52.231 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:52.231 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:52.239 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T15:50:52.241 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:52.239+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:52.342 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:52.340+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:52.382 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:52.382 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:52.390 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:50:52.442 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:52.441+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:52.501 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:52.502 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:52.511 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:50:52.543 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:52.542+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:52.613 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:52.614 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:52.623 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:50:52.667 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:52.666+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:52.692 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-09-21T15:50:52.693 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T15:50:52.693 INFO:tasks.thrashosds.thrasher:Removing osd 3, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T15:50:52.693 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 3 2024-09-21T15:50:52.736 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:52.736 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:52.745 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:50:52.746 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:52.744+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:52.846 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:52.844+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:52.854 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:50:52.894 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:52.895 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:52.904 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:50:52.947 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:52.946+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:52.981 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-09-21T15:50:52.981 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:50:52.998 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:50:52.998 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:50:53.002 INFO:tasks.ceph.mon.a.smithi089.stderr:2024-09-21T15:50:53.000+0000 7f17447bb640 -1 mon.a@0(leader).osd e117 definitely_dead 0 2024-09-21T15:50:53.015 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:50:53.015 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:50:53.031 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:50:53.031 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:50:53.047 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:50:53.047 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:50:53.047 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:53.046+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:53.053 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:53.054 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:53.054 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:50:53.054 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:50:53.054 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:53.063 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:50:53.063 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:50:53.064 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:50:53.080 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:50:53.080 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:50:53.096 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:50:53.097 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:50:53.148 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:53.147+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:53.168 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:53.169 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:50:53.169 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:50:53.169 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:50:53.169 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.169 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62833 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 192512~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:53.169 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:50:53.169 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 76.024773428000003, 2024-09-21T15:50:53.169 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050387007999999997, 2024-09-21T15:50:53.169 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:53.169 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:53.169 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:53.169 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:53.169 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:53.169 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62833 2024-09-21T15:50:53.170 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.170 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:53.170 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.170 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:53.170 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:50:53.170 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.170 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.170 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.170 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:53.170 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:50:53.170 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.170 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.170 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.171 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:53.171 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:50:53.171 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:53.171 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.171 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.171 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:53.171 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142150+0000", 2024-09-21T15:50:53.171 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1999999999999999e-06 2024-09-21T15:50:53.171 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.171 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.171 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:53.171 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142153+0000", 2024-09-21T15:50:53.171 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7410000000000001e-06 2024-09-21T15:50:53.171 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.171 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.172 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:53.172 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142156+0000", 2024-09-21T15:50:53.172 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5739999999999998e-06 2024-09-21T15:50:53.172 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.172 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.172 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:53.172 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.192358+0000", 2024-09-21T15:50:53.172 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050202452000000002 2024-09-21T15:50:53.172 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.172 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.172 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:53.172 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.192435+0000", 2024-09-21T15:50:53.172 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6738999999999998e-05 2024-09-21T15:50:53.172 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.173 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.173 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:53.173 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.192536+0000", 2024-09-21T15:50:53.173 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101508 2024-09-21T15:50:53.173 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.173 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:53.173 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.173 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.173 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.173 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62849 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 208896~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:53.173 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.144181+0000", 2024-09-21T15:50:53.173 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 76.022741355999997, 2024-09-21T15:50:53.173 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050460390000000001, 2024-09-21T15:50:53.173 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:53.173 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:53.173 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:53.174 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:53.174 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:53.174 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62849 2024-09-21T15:50:53.174 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.174 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:53.174 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.174 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:53.174 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144181+0000", 2024-09-21T15:50:53.174 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.174 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.174 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.174 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:53.174 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144181+0000", 2024-09-21T15:50:53.175 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.175 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.175 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.175 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:53.175 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144181+0000", 2024-09-21T15:50:53.175 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:53.175 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.175 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.175 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:53.175 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144183+0000", 2024-09-21T15:50:53.175 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3860000000000001e-06 2024-09-21T15:50:53.175 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.175 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.175 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:53.176 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144189+0000", 2024-09-21T15:50:53.176 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0599999999999996e-06 2024-09-21T15:50:53.176 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.176 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.176 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:53.176 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144194+0000", 2024-09-21T15:50:53.176 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4379999999999997e-06 2024-09-21T15:50:53.176 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.176 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.176 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:53.176 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.194457+0000", 2024-09-21T15:50:53.176 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050263317000000002 2024-09-21T15:50:53.176 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.176 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.176 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:53.177 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.194548+0000", 2024-09-21T15:50:53.177 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0785000000000006e-05 2024-09-21T15:50:53.177 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.177 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.177 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:53.177 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.194642+0000", 2024-09-21T15:50:53.177 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3837000000000003e-05 2024-09-21T15:50:53.177 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.177 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:53.177 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.177 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.177 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.177 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62862 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 954368~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:53.178 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.146176+0000", 2024-09-21T15:50:53.178 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 76.020746293000002, 2024-09-21T15:50:53.178 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050442901999999998, 2024-09-21T15:50:53.178 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:53.178 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:53.178 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:53.178 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:53.178 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:53.178 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62862 2024-09-21T15:50:53.178 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.178 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:53.178 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.178 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:53.178 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146176+0000", 2024-09-21T15:50:53.179 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.179 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.179 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.179 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:53.179 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146176+0000", 2024-09-21T15:50:53.179 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.179 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.179 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.179 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:53.179 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146176+0000", 2024-09-21T15:50:53.179 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:53.179 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.179 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.179 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:53.180 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146178+0000", 2024-09-21T15:50:53.180 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4739999999999999e-06 2024-09-21T15:50:53.180 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.180 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.180 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:53.180 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146186+0000", 2024-09-21T15:50:53.180 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4989999999999997e-06 2024-09-21T15:50:53.180 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.180 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.180 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:53.180 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146191+0000", 2024-09-21T15:50:53.180 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7999999999999998e-06 2024-09-21T15:50:53.180 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.180 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.181 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:53.181 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196442+0000", 2024-09-21T15:50:53.181 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050251665000000001 2024-09-21T15:50:53.181 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.181 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.181 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:53.181 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196518+0000", 2024-09-21T15:50:53.181 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5677000000000001e-05 2024-09-21T15:50:53.181 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.181 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.181 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:53.181 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196619+0000", 2024-09-21T15:50:53.181 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101247 2024-09-21T15:50:53.181 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.182 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:53.182 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.182 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.182 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.182 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62863 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 937984~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:53.182 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.146195+0000", 2024-09-21T15:50:53.182 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 76.020728102000007, 2024-09-21T15:50:53.182 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050642305999999998, 2024-09-21T15:50:53.182 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:53.182 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:53.182 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:53.182 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:53.182 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:53.183 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62863 2024-09-21T15:50:53.183 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.183 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:53.183 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.183 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:53.183 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146195+0000", 2024-09-21T15:50:53.183 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.183 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.183 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.183 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:53.183 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146195+0000", 2024-09-21T15:50:53.183 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.183 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.184 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.184 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:53.184 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146194+0000", 2024-09-21T15:50:53.184 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:53.184 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.184 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.184 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:53.184 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146196+0000", 2024-09-21T15:50:53.184 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1900000000000002e-06 2024-09-21T15:50:53.184 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.184 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.184 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:53.184 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146202+0000", 2024-09-21T15:50:53.184 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.57e-06 2024-09-21T15:50:53.185 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.185 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.185 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:53.185 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146206+0000", 2024-09-21T15:50:53.185 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5079999999999999e-06 2024-09-21T15:50:53.185 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.185 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.185 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:53.185 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196655+0000", 2024-09-21T15:50:53.185 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050448973000000001 2024-09-21T15:50:53.185 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.185 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.185 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:53.185 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196730+0000", 2024-09-21T15:50:53.186 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4728999999999996e-05 2024-09-21T15:50:53.186 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.186 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.186 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:53.186 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196837+0000", 2024-09-21T15:50:53.186 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000106735 2024-09-21T15:50:53.186 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.186 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:53.186 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.186 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.186 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.186 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62872 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 290816~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:53.186 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148149+0000", 2024-09-21T15:50:53.187 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 76.018773425999996, 2024-09-21T15:50:53.187 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.05046991, 2024-09-21T15:50:53.187 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:53.187 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:53.187 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:53.187 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:53.187 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:53.187 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62872 2024-09-21T15:50:53.187 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.187 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:53.187 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.187 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:53.187 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148149+0000", 2024-09-21T15:50:53.187 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.188 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.188 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.188 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:53.188 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148149+0000", 2024-09-21T15:50:53.188 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.188 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.188 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.188 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:53.188 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148149+0000", 2024-09-21T15:50:53.188 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:53.188 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.188 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.188 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:53.189 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148157+0000", 2024-09-21T15:50:53.189 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4749999999999993e-06 2024-09-21T15:50:53.189 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.189 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.189 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:53.189 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148163+0000", 2024-09-21T15:50:53.189 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8250000000000001e-06 2024-09-21T15:50:53.189 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.189 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.189 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:53.189 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148168+0000", 2024-09-21T15:50:53.189 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4730000000000002e-06 2024-09-21T15:50:53.189 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.189 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.190 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:53.190 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198471+0000", 2024-09-21T15:50:53.190 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050302974 2024-09-21T15:50:53.190 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.190 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.190 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:53.190 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198555+0000", 2024-09-21T15:50:53.190 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4783999999999993e-05 2024-09-21T15:50:53.190 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.190 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.190 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:53.190 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198619+0000", 2024-09-21T15:50:53.190 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3751000000000007e-05 2024-09-21T15:50:53.190 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.191 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:53.191 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.191 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.191 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.191 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62873 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 315392~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:53.191 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148172+0000", 2024-09-21T15:50:53.191 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 76.018751135000002, 2024-09-21T15:50:53.191 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050707998999999997, 2024-09-21T15:50:53.191 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:53.191 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:53.191 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:53.191 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:53.191 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:53.191 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62873 2024-09-21T15:50:53.192 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.192 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:53.192 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.192 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:53.192 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148172+0000", 2024-09-21T15:50:53.192 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.192 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.192 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.192 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:53.192 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148172+0000", 2024-09-21T15:50:53.192 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.192 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.192 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.192 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:53.192 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148171+0000", 2024-09-21T15:50:53.193 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:53.193 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.193 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.193 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:53.193 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148173+0000", 2024-09-21T15:50:53.193 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2340000000000001e-06 2024-09-21T15:50:53.193 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.193 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.193 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:53.193 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148179+0000", 2024-09-21T15:50:53.193 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9340000000000003e-06 2024-09-21T15:50:53.193 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.193 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.193 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:53.193 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148186+0000", 2024-09-21T15:50:53.194 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2349999999999996e-06 2024-09-21T15:50:53.194 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.194 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.194 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:53.194 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198684+0000", 2024-09-21T15:50:53.194 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050498890999999997 2024-09-21T15:50:53.194 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.194 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.194 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:53.194 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198778+0000", 2024-09-21T15:50:53.194 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3134999999999995e-05 2024-09-21T15:50:53.194 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.194 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.194 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:53.195 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198880+0000", 2024-09-21T15:50:53.195 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101906 2024-09-21T15:50:53.195 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.195 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:53.195 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.195 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.195 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.195 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62874 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 323584~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:53.195 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148211+0000", 2024-09-21T15:50:53.195 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 76.018711374999995, 2024-09-21T15:50:53.195 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050887766000000001, 2024-09-21T15:50:53.195 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:53.195 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:53.196 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:53.196 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:53.196 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:53.196 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62874 2024-09-21T15:50:53.196 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.196 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:53.196 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.196 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:53.196 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148211+0000", 2024-09-21T15:50:53.196 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.196 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.196 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.196 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:53.196 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148211+0000", 2024-09-21T15:50:53.197 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.197 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.197 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.197 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:53.197 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148211+0000", 2024-09-21T15:50:53.197 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:53.197 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.197 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.197 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:53.197 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148213+0000", 2024-09-21T15:50:53.197 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6860000000000002e-06 2024-09-21T15:50:53.197 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.197 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:53.198 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148220+0000", 2024-09-21T15:50:53.198 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4740000000000002e-06 2024-09-21T15:50:53.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:53.198 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148226+0000", 2024-09-21T15:50:53.198 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0120000000000002e-06 2024-09-21T15:50:53.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:53.198 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198940+0000", 2024-09-21T15:50:53.198 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050714412 2024-09-21T15:50:53.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.199 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.199 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:53.199 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198995+0000", 2024-09-21T15:50:53.199 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4863e-05 2024-09-21T15:50:53.199 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.199 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.199 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:53.199 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.199099+0000", 2024-09-21T15:50:53.199 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010383299999999999 2024-09-21T15:50:53.199 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.199 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:53.199 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.199 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.199 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.199 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62877 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 995328~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:53.200 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148230+0000", 2024-09-21T15:50:53.200 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 76.018692514999998, 2024-09-21T15:50:53.200 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.051108949000000001, 2024-09-21T15:50:53.200 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:53.200 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:53.200 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:53.200 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:53.200 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:53.200 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62877 2024-09-21T15:50:53.200 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.200 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:53.200 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.200 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:53.200 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148230+0000", 2024-09-21T15:50:53.200 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.201 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.201 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.201 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:53.201 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148230+0000", 2024-09-21T15:50:53.201 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.201 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.201 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.201 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:53.201 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148230+0000", 2024-09-21T15:50:53.201 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:53.201 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.201 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.201 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:53.201 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148232+0000", 2024-09-21T15:50:53.201 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1220000000000002e-06 2024-09-21T15:50:53.202 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.202 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.202 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:53.202 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148238+0000", 2024-09-21T15:50:53.202 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.789e-06 2024-09-21T15:50:53.202 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.202 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.202 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:53.202 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148242+0000", 2024-09-21T15:50:53.202 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7369999999999997e-06 2024-09-21T15:50:53.202 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.202 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.202 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:53.202 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.199201+0000", 2024-09-21T15:50:53.203 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050959088 2024-09-21T15:50:53.203 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.203 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.203 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:53.203 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.199285+0000", 2024-09-21T15:50:53.203 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3147999999999997e-05 2024-09-21T15:50:53.203 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.203 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.203 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:53.203 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.199339+0000", 2024-09-21T15:50:53.203 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4484999999999998e-05 2024-09-21T15:50:53.203 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.203 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:53.203 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.203 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.204 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.204 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62888 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 331776~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:53.204 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150154+0000", 2024-09-21T15:50:53.204 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 76.016768920999993, 2024-09-21T15:50:53.204 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050483169000000001, 2024-09-21T15:50:53.204 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:53.204 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:53.204 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:53.204 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:53.204 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:53.204 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62888 2024-09-21T15:50:53.204 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.204 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:53.204 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.205 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:53.205 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150154+0000", 2024-09-21T15:50:53.205 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.205 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.205 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.205 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:53.205 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150154+0000", 2024-09-21T15:50:53.205 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.205 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.205 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.205 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:53.205 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150153+0000", 2024-09-21T15:50:53.205 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:53.205 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.205 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.206 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:53.206 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150155+0000", 2024-09-21T15:50:53.206 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9920000000000002e-06 2024-09-21T15:50:53.206 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.206 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.206 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:53.206 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150161+0000", 2024-09-21T15:50:53.206 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5439999999999998e-06 2024-09-21T15:50:53.206 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.206 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.206 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:53.206 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150165+0000", 2024-09-21T15:50:53.206 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3460000000000004e-06 2024-09-21T15:50:53.206 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.207 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.207 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:53.207 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200502+0000", 2024-09-21T15:50:53.207 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050337150999999997 2024-09-21T15:50:53.207 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.207 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.207 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:53.207 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200575+0000", 2024-09-21T15:50:53.207 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2250999999999997e-05 2024-09-21T15:50:53.207 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.207 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.207 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:53.207 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200637+0000", 2024-09-21T15:50:53.208 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2195999999999999e-05 2024-09-21T15:50:53.208 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.208 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:53.208 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.208 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.208 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62889 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 339968~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:53.208 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150169+0000", 2024-09-21T15:50:53.208 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 76.016753421999994, 2024-09-21T15:50:53.208 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050629678999999997, 2024-09-21T15:50:53.208 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:53.208 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:53.208 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:53.208 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:53.208 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:53.209 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62889 2024-09-21T15:50:53.209 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.209 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:53.209 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.209 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:53.209 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150169+0000", 2024-09-21T15:50:53.209 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.209 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.209 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.209 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:53.209 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150169+0000", 2024-09-21T15:50:53.209 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.209 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.209 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.210 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:53.210 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150169+0000", 2024-09-21T15:50:53.210 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:53.210 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.210 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.210 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:53.210 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150171+0000", 2024-09-21T15:50:53.210 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3910000000000001e-06 2024-09-21T15:50:53.210 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.210 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.210 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:53.210 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150177+0000", 2024-09-21T15:50:53.210 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4079999999999997e-06 2024-09-21T15:50:53.211 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.211 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.211 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:53.211 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150182+0000", 2024-09-21T15:50:53.211 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3890000000000004e-06 2024-09-21T15:50:53.211 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.211 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.211 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:53.211 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200664+0000", 2024-09-21T15:50:53.211 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050482212999999998 2024-09-21T15:50:53.211 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.211 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.211 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:53.211 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200738+0000", 2024-09-21T15:50:53.212 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4114000000000002e-05 2024-09-21T15:50:53.212 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.212 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.212 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:53.212 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200799+0000", 2024-09-21T15:50:53.212 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0503e-05 2024-09-21T15:50:53.212 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.212 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:53.212 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.212 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.212 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.212 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62891 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 348160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:53.212 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150186+0000", 2024-09-21T15:50:53.212 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 76.016736280000003, 2024-09-21T15:50:53.213 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050794552, 2024-09-21T15:50:53.213 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:53.213 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:53.213 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:53.213 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:53.213 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:53.213 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62891 2024-09-21T15:50:53.213 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.213 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:53.213 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.213 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:53.213 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150186+0000", 2024-09-21T15:50:53.213 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.213 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.213 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.214 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:53.214 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150186+0000", 2024-09-21T15:50:53.214 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.214 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.214 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.214 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:53.214 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150186+0000", 2024-09-21T15:50:53.214 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:53.214 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.214 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.214 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:53.214 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150193+0000", 2024-09-21T15:50:53.214 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6830000000000002e-06 2024-09-21T15:50:53.214 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.215 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.215 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:53.215 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150199+0000", 2024-09-21T15:50:53.215 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9859999999999999e-06 2024-09-21T15:50:53.215 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.215 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.215 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:53.215 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150203+0000", 2024-09-21T15:50:53.215 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5920000000000002e-06 2024-09-21T15:50:53.215 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.215 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.215 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:53.215 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200835+0000", 2024-09-21T15:50:53.215 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050631783 2024-09-21T15:50:53.216 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.216 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.216 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:53.216 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200894+0000", 2024-09-21T15:50:53.216 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8560999999999997e-05 2024-09-21T15:50:53.216 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.216 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.216 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:53.216 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200981+0000", 2024-09-21T15:50:53.216 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7305999999999994e-05 2024-09-21T15:50:53.216 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.216 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:53.216 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.216 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.216 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.216 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62892 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 356352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:53.217 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150208+0000", 2024-09-21T15:50:53.217 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 76.016714891999996, 2024-09-21T15:50:53.217 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050951769000000001, 2024-09-21T15:50:53.217 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:53.217 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:53.217 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:53.217 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:53.217 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:53.217 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62892 2024-09-21T15:50:53.217 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.217 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:53.217 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.217 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:53.217 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150208+0000", 2024-09-21T15:50:53.217 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.218 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.218 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.218 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:53.218 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150208+0000", 2024-09-21T15:50:53.218 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.218 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.218 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.218 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:53.218 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150207+0000", 2024-09-21T15:50:53.218 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:53.218 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.218 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.218 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:53.219 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150210+0000", 2024-09-21T15:50:53.219 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.238e-06 2024-09-21T15:50:53.219 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.219 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.219 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:53.219 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150215+0000", 2024-09-21T15:50:53.219 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6350000000000003e-06 2024-09-21T15:50:53.219 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.219 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.219 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:53.219 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150220+0000", 2024-09-21T15:50:53.219 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.544e-06 2024-09-21T15:50:53.219 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.219 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.219 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:53.219 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201014+0000", 2024-09-21T15:50:53.220 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050793932999999999 2024-09-21T15:50:53.220 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.220 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.220 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:53.220 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201092+0000", 2024-09-21T15:50:53.220 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7961999999999997e-05 2024-09-21T15:50:53.220 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.220 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.220 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:53.220 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201160+0000", 2024-09-21T15:50:53.220 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7836000000000006e-05 2024-09-21T15:50:53.220 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.220 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:53.220 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.220 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.220 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.221 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62894 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 364544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:53.221 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150820+0000", 2024-09-21T15:50:53.221 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 76.016102692000004, 2024-09-21T15:50:53.221 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050489196, 2024-09-21T15:50:53.221 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:53.221 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:53.221 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:53.221 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:53.221 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:53.221 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62894 2024-09-21T15:50:53.221 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.221 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:53.221 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.221 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:53.222 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150820+0000", 2024-09-21T15:50:53.222 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.222 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.222 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.222 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:53.222 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150820+0000", 2024-09-21T15:50:53.222 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.222 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.222 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.222 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:53.222 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150223+0000", 2024-09-21T15:50:53.222 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9994035 2024-09-21T15:50:53.222 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.222 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.222 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:53.222 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150825+0000", 2024-09-21T15:50:53.223 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00060137099999999996 2024-09-21T15:50:53.223 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.223 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.223 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:53.223 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150837+0000", 2024-09-21T15:50:53.223 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2503999999999999e-05 2024-09-21T15:50:53.223 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.223 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.223 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:53.223 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150844+0000", 2024-09-21T15:50:53.223 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6900000000000003e-06 2024-09-21T15:50:53.223 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.223 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.224 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:53.224 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201187+0000", 2024-09-21T15:50:53.224 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050342973999999999 2024-09-21T15:50:53.224 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.224 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.224 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:53.224 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201250+0000", 2024-09-21T15:50:53.224 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3274000000000004e-05 2024-09-21T15:50:53.224 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.224 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.224 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:53.224 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201309+0000", 2024-09-21T15:50:53.224 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9058000000000003e-05 2024-09-21T15:50:53.224 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.225 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:53.225 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.225 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.225 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.225 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62895 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 380928~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:53.225 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150851+0000", 2024-09-21T15:50:53.225 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 76.016072096000002, 2024-09-21T15:50:53.225 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050611097000000001, 2024-09-21T15:50:53.225 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:53.225 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:53.225 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:53.225 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:53.225 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:53.225 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62895 2024-09-21T15:50:53.225 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.226 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:53.226 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.226 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:53.226 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150851+0000", 2024-09-21T15:50:53.226 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.226 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.226 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.226 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:53.226 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150851+0000", 2024-09-21T15:50:53.226 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.226 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.226 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.226 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:53.226 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150850+0000", 2024-09-21T15:50:53.226 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:53.227 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.227 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.227 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:53.227 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150853+0000", 2024-09-21T15:50:53.227 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1039999999999998e-06 2024-09-21T15:50:53.227 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.227 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.227 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:53.227 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150860+0000", 2024-09-21T15:50:53.227 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3180000000000001e-06 2024-09-21T15:50:53.227 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.227 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.227 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:53.227 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150866+0000", 2024-09-21T15:50:53.227 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1599999999999997e-06 2024-09-21T15:50:53.228 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.228 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.228 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:53.228 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201332+0000", 2024-09-21T15:50:53.228 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050466100999999999 2024-09-21T15:50:53.228 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.228 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.228 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:53.228 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201392+0000", 2024-09-21T15:50:53.228 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0053999999999998e-05 2024-09-21T15:50:53.228 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.228 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.228 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:53.228 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201462+0000", 2024-09-21T15:50:53.228 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9861000000000001e-05 2024-09-21T15:50:53.228 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.229 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:53.229 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.229 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.229 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.229 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62896 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 372736~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:53.229 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150870+0000", 2024-09-21T15:50:53.229 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 76.016052316, 2024-09-21T15:50:53.229 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050753292999999998, 2024-09-21T15:50:53.229 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:53.229 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:53.229 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:53.229 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:53.229 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:53.229 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62896 2024-09-21T15:50:53.229 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.230 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:53.230 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.230 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:53.230 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150870+0000", 2024-09-21T15:50:53.230 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.230 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.230 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.230 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:53.230 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150870+0000", 2024-09-21T15:50:53.230 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.230 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.230 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.230 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:53.230 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150870+0000", 2024-09-21T15:50:53.230 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:53.231 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.231 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.231 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:53.231 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150872+0000", 2024-09-21T15:50:53.231 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.644e-06 2024-09-21T15:50:53.231 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.231 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.231 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:53.231 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150879+0000", 2024-09-21T15:50:53.231 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3509999999999998e-06 2024-09-21T15:50:53.231 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.231 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.231 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:53.231 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150884+0000", 2024-09-21T15:50:53.231 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9150000000000004e-06 2024-09-21T15:50:53.231 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.232 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.232 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:53.232 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201487+0000", 2024-09-21T15:50:53.232 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050603276000000003 2024-09-21T15:50:53.232 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.232 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.232 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:53.232 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201557+0000", 2024-09-21T15:50:53.232 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9847999999999999e-05 2024-09-21T15:50:53.232 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.232 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.232 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:53.232 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201624+0000", 2024-09-21T15:50:53.232 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6730999999999995e-05 2024-09-21T15:50:53.232 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.233 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:53.233 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.233 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.233 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.233 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62897 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 389120~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:53.233 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.151044+0000", 2024-09-21T15:50:53.233 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 76.015878275999995, 2024-09-21T15:50:53.233 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050720531999999999, 2024-09-21T15:50:53.233 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:53.233 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:53.233 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:53.233 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:53.233 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:53.233 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62897 2024-09-21T15:50:53.234 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.234 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:53.234 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.234 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:53.234 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151044+0000", 2024-09-21T15:50:53.234 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.234 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.234 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.234 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:53.234 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151044+0000", 2024-09-21T15:50:53.234 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.234 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.234 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.234 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:53.234 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150888+0000", 2024-09-21T15:50:53.235 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9998431 2024-09-21T15:50:53.235 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.235 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.235 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:53.235 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151048+0000", 2024-09-21T15:50:53.235 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00016077299999999999 2024-09-21T15:50:53.235 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.235 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.235 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:53.235 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151058+0000", 2024-09-21T15:50:53.235 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.465e-06 2024-09-21T15:50:53.235 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.235 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.235 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:53.235 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151064+0000", 2024-09-21T15:50:53.235 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3829999999999997e-06 2024-09-21T15:50:53.236 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.236 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.236 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:53.236 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201648+0000", 2024-09-21T15:50:53.236 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050583597000000001 2024-09-21T15:50:53.236 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.236 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.236 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:53.236 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201711+0000", 2024-09-21T15:50:53.236 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3616999999999998e-05 2024-09-21T15:50:53.236 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.236 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.236 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:53.236 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201765+0000", 2024-09-21T15:50:53.236 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3464000000000002e-05 2024-09-21T15:50:53.237 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.237 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:53.237 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.237 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.237 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.237 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:118351 3.4 3:25980f86:::rbd_data.1133545d3fa6.000000000000005a:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-09-21T15:50:53.237 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:58.975795+0000", 2024-09-21T15:50:53.237 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 54.191127641999998, 2024-09-21T15:50:53.237 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.99016351599999997, 2024-09-21T15:50:53.237 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:53.237 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:53.237 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:53.237 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:53.237 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:53.237 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 118351 2024-09-21T15:50:53.237 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.238 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:53.238 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.238 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:53.238 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975795+0000", 2024-09-21T15:50:53.238 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.238 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.238 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.238 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:53.238 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975795+0000", 2024-09-21T15:50:53.238 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.238 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.238 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.238 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:53.238 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975793+0000", 2024-09-21T15:50:53.238 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:50:53.239 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.239 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.239 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:53.239 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975800+0000", 2024-09-21T15:50:53.239 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8220000000000001e-06 2024-09-21T15:50:53.239 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.239 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.239 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:53.239 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975814+0000", 2024-09-21T15:50:53.239 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3893000000000001e-05 2024-09-21T15:50:53.239 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.239 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.239 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:53.239 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975821+0000", 2024-09-21T15:50:53.240 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1319999999999996e-06 2024-09-21T15:50:53.240 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.240 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.240 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:53.240 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975878+0000", 2024-09-21T15:50:53.240 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7238999999999999e-05 2024-09-21T15:50:53.240 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.240 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.240 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:50:53.240 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975889+0000", 2024-09-21T15:50:53.240 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0509000000000001e-05 2024-09-21T15:50:53.240 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.241 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.241 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:53.241 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.964573+0000", 2024-09-21T15:50:53.241 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.98868396300000005 2024-09-21T15:50:53.241 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.241 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.241 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:53.241 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.964672+0000", 2024-09-21T15:50:53.241 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9601999999999999e-05 2024-09-21T15:50:53.241 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.241 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.241 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-09-21T15:50:53.241 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.964758+0000", 2024-09-21T15:50:53.242 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5537000000000005e-05 2024-09-21T15:50:53.242 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.242 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.242 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:53.242 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965847+0000", 2024-09-21T15:50:53.242 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001089404 2024-09-21T15:50:53.242 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.242 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.242 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:53.242 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965890+0000", 2024-09-21T15:50:53.242 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2521999999999997e-05 2024-09-21T15:50:53.242 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.243 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.243 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:53.243 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965934+0000", 2024-09-21T15:50:53.243 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4489e-05 2024-09-21T15:50:53.243 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.243 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.243 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:53.243 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965959+0000", 2024-09-21T15:50:53.243 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4301999999999999e-05 2024-09-21T15:50:53.243 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.243 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:53.243 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.243 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.243 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.243 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:118353 3.4 3:23841e18:::rbd_data.1133545d3fa6.0000000000000061:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-09-21T15:50:53.243 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:58.975836+0000", 2024-09-21T15:50:53.244 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 54.191086143, 2024-09-21T15:50:53.244 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.99077746700000002, 2024-09-21T15:50:53.244 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:53.244 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:53.244 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:53.244 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:53.244 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:53.244 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 118353 2024-09-21T15:50:53.244 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.244 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:53.244 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.244 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:53.244 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975836+0000", 2024-09-21T15:50:53.244 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.244 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.245 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.245 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:53.245 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975836+0000", 2024-09-21T15:50:53.245 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.245 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.245 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.245 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:53.245 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975836+0000", 2024-09-21T15:50:53.245 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:53.245 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.245 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.245 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:53.245 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975839+0000", 2024-09-21T15:50:53.245 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6129999999999998e-06 2024-09-21T15:50:53.245 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.245 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.246 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:53.246 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975844+0000", 2024-09-21T15:50:53.246 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5130000000000004e-06 2024-09-21T15:50:53.246 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.246 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.246 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:53.246 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975850+0000", 2024-09-21T15:50:53.246 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3179999999999997e-06 2024-09-21T15:50:53.246 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.246 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.246 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:53.246 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975915+0000", 2024-09-21T15:50:53.246 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5168999999999999e-05 2024-09-21T15:50:53.246 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.247 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.247 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:50:53.247 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975922+0000", 2024-09-21T15:50:53.247 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2749999999999998e-06 2024-09-21T15:50:53.247 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.247 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.247 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:53.247 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965431+0000", 2024-09-21T15:50:53.247 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.98950864699999996 2024-09-21T15:50:53.247 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.247 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.247 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:53.247 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965535+0000", 2024-09-21T15:50:53.247 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000103949 2024-09-21T15:50:53.247 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.247 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.248 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-09-21T15:50:53.248 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965607+0000", 2024-09-21T15:50:53.248 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2478000000000006e-05 2024-09-21T15:50:53.248 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.248 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.248 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:53.248 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.966205+0000", 2024-09-21T15:50:53.248 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00059775200000000001 2024-09-21T15:50:53.248 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.248 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.248 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:53.248 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.966562+0000", 2024-09-21T15:50:53.248 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00035732199999999998 2024-09-21T15:50:53.248 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.249 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.249 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:53.249 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.966593+0000", 2024-09-21T15:50:53.249 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0642999999999998e-05 2024-09-21T15:50:53.249 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.249 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.249 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:53.249 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.966614+0000", 2024-09-21T15:50:53.249 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1149999999999999e-05 2024-09-21T15:50:53.249 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.249 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:53.249 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.249 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.249 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.249 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160535 3.6 3:67cae2ed:::rbd_mirroring:head [list-watchers] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-09-21T15:50:53.249 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:30.195408+0000", 2024-09-21T15:50:53.250 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 22.971515061000002, 2024-09-21T15:50:53.250 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.96025944900000004, 2024-09-21T15:50:53.250 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:53.250 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-09-21T15:50:53.250 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:53.250 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:53.250 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:53.250 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160535 2024-09-21T15:50:53.250 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.250 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:53.250 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.250 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:53.250 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195408+0000", 2024-09-21T15:50:53.251 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.251 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.251 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.251 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:53.251 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195408+0000", 2024-09-21T15:50:53.251 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.251 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.251 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.251 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:53.251 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195406+0000", 2024-09-21T15:50:53.251 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:50:53.251 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.251 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.251 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:53.251 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195414+0000", 2024-09-21T15:50:53.252 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.2870000000000004e-06 2024-09-21T15:50:53.252 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.252 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.252 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:53.252 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195432+0000", 2024-09-21T15:50:53.252 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8110000000000001e-05 2024-09-21T15:50:53.252 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.252 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.252 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:53.252 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195441+0000", 2024-09-21T15:50:53.252 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7679999999999997e-06 2024-09-21T15:50:53.252 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.252 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.252 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:53.252 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195505+0000", 2024-09-21T15:50:53.252 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3769000000000006e-05 2024-09-21T15:50:53.253 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.253 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.253 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:50:53.253 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195518+0000", 2024-09-21T15:50:53.253 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3341e-05 2024-09-21T15:50:53.253 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.253 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.253 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:53.253 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.155509+0000", 2024-09-21T15:50:53.253 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.959990393 2024-09-21T15:50:53.253 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.253 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.253 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:53.253 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.155667+0000", 2024-09-21T15:50:53.253 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000158505 2024-09-21T15:50:53.253 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.254 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:53.254 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.254 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.254 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.254 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160974 3.4 3:22dd2475:::rbd_id.image_client.0-clone39:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-21T15:50:53.254 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:34.748093+0000", 2024-09-21T15:50:53.254 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.418829509999998, 2024-09-21T15:50:53.254 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.94006199499999998, 2024-09-21T15:50:53.254 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:53.254 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:53.254 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:53.254 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:53.254 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:53.254 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160974 2024-09-21T15:50:53.254 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.255 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:53.255 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.255 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:53.255 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748093+0000", 2024-09-21T15:50:53.255 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.255 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.255 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.255 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:53.255 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748093+0000", 2024-09-21T15:50:53.255 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.255 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.255 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.255 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:53.255 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748091+0000", 2024-09-21T15:50:53.255 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:50:53.256 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.256 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.256 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:53.256 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748101+0000", 2024-09-21T15:50:53.256 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5950000000000005e-06 2024-09-21T15:50:53.256 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.256 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.256 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:53.256 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748122+0000", 2024-09-21T15:50:53.256 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1611e-05 2024-09-21T15:50:53.256 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.256 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.256 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:53.256 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748133+0000", 2024-09-21T15:50:53.256 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0224000000000001e-05 2024-09-21T15:50:53.257 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.257 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.257 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:53.257 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748185+0000", 2024-09-21T15:50:53.257 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2005000000000001e-05 2024-09-21T15:50:53.257 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.257 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.257 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:50:53.257 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748194+0000", 2024-09-21T15:50:53.257 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0180000000000007e-06 2024-09-21T15:50:53.257 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.257 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.257 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:53.257 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.683582+0000", 2024-09-21T15:50:53.257 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.93538800499999997 2024-09-21T15:50:53.258 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.258 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.258 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:53.258 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.683711+0000", 2024-09-21T15:50:53.258 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000129746 2024-09-21T15:50:53.258 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.258 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.258 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-09-21T15:50:53.258 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.683838+0000", 2024-09-21T15:50:53.258 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012693200000000001 2024-09-21T15:50:53.258 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.258 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.258 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:53.258 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.687445+0000", 2024-09-21T15:50:53.258 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0036071340000000001 2024-09-21T15:50:53.258 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.259 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.259 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:53.259 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.688064+0000", 2024-09-21T15:50:53.259 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00061857100000000005 2024-09-21T15:50:53.259 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.259 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.259 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:53.259 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.688113+0000", 2024-09-21T15:50:53.259 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9128000000000001e-05 2024-09-21T15:50:53.259 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.259 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.259 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:53.259 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.688155+0000", 2024-09-21T15:50:53.259 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2026e-05 2024-09-21T15:50:53.260 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.260 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:53.260 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.260 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.260 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:53.260 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:53.261 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:50:53.261 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:50:53.260+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:50:53.361 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:53.360+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:53.402 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:53.402 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:53.402 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:50:53.402 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:53.411 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:50:53.462 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:53.461+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:53.513 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:53.513 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:53.513 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:50:53.514 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:50:53.514 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:53.522 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:50:53.563 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:53.562+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:53.665 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:53.665 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:50:53.665 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:50:53.665 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:50:53.665 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.665 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:984 3.0 e27/22 3:025e53e7:::rbd_data.1133f3dfbd6b.0000000000000008:head v 27'21, mlcod=27'19)", 2024-09-21T15:50:53.665 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:12.619393+0000", 2024-09-21T15:50:53.666 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 101.044277603, 2024-09-21T15:50:53.666 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003401837, 2024-09-21T15:50:53.666 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:53.666 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:53.666 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:53.666 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.666 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:53.666 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619393+0000", 2024-09-21T15:50:53.666 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.666 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.666 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.666 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:53.666 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619393+0000", 2024-09-21T15:50:53.666 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.667 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.667 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.667 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:53.667 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619392+0000", 2024-09-21T15:50:53.667 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:53.667 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.667 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.667 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:53.667 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619439+0000", 2024-09-21T15:50:53.667 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7170999999999999e-05 2024-09-21T15:50:53.667 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.667 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.667 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:53.667 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619452+0000", 2024-09-21T15:50:53.668 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2986999999999999e-05 2024-09-21T15:50:53.668 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.668 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.668 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:53.668 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619459+0000", 2024-09-21T15:50:53.668 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6109999999999999e-06 2024-09-21T15:50:53.668 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.668 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.668 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:53.668 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619518+0000", 2024-09-21T15:50:53.668 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9117999999999999e-05 2024-09-21T15:50:53.668 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.668 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.669 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:53.669 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.619532+0000", 2024-09-21T15:50:53.669 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.4622e-05 2024-09-21T15:50:53.669 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.669 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.669 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:53.669 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.622760+0000", 2024-09-21T15:50:53.669 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0032278839999999999 2024-09-21T15:50:53.669 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.669 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.669 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:53.669 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:12.622794+0000", 2024-09-21T15:50:53.669 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4144999999999999e-05 2024-09-21T15:50:53.669 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.670 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:53.670 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.670 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.670 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.670 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:4957 3.9 e29/22 3:95c5ed95:::rbd_data.11335fa2fe69.0000000000000018:head v 29'4, mlcod=26'2)", 2024-09-21T15:50:53.670 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:14.453951+0000", 2024-09-21T15:50:53.670 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 99.209719171000003, 2024-09-21T15:50:53.670 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0043804559999999996, 2024-09-21T15:50:53.670 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:53.670 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:53.670 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:53.670 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.670 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:53.671 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.453951+0000", 2024-09-21T15:50:53.671 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.671 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.671 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.671 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:53.671 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.453951+0000", 2024-09-21T15:50:53.671 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.671 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.671 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.671 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:53.671 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.453950+0000", 2024-09-21T15:50:53.671 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:53.671 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.671 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.672 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:53.672 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454495+0000", 2024-09-21T15:50:53.672 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00054437799999999998 2024-09-21T15:50:53.672 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.672 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.672 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:53.672 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454508+0000", 2024-09-21T15:50:53.672 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3475000000000001e-05 2024-09-21T15:50:53.672 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.672 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.672 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:53.672 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454515+0000", 2024-09-21T15:50:53.672 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8990000000000004e-06 2024-09-21T15:50:53.672 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.673 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.673 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:53.673 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454581+0000", 2024-09-21T15:50:53.673 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5987000000000004e-05 2024-09-21T15:50:53.673 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.673 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.673 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:53.673 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454609+0000", 2024-09-21T15:50:53.673 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7466000000000001e-05 2024-09-21T15:50:53.673 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.673 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.673 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:53.673 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.458303+0000", 2024-09-21T15:50:53.673 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003693979 2024-09-21T15:50:53.673 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.674 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.674 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:53.674 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.458331+0000", 2024-09-21T15:50:53.674 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8932999999999999e-05 2024-09-21T15:50:53.674 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.674 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:53.674 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.674 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.674 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.674 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13420 3.0 e31/22 3:038b8840:::rbd_data.1133dc5adfdf.0000000000000035:head v 31'49, mlcod=31'47)", 2024-09-21T15:50:53.674 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.209233+0000", 2024-09-21T15:50:53.674 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 96.454437554999998, 2024-09-21T15:50:53.674 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0042624960000000002, 2024-09-21T15:50:53.674 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:53.675 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:53.675 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:53.675 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.675 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:53.675 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209233+0000", 2024-09-21T15:50:53.675 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.675 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.675 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.675 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:53.675 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209233+0000", 2024-09-21T15:50:53.675 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.675 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.675 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.675 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:53.675 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209232+0000", 2024-09-21T15:50:53.676 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:53.676 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.676 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.676 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:53.676 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209251+0000", 2024-09-21T15:50:53.676 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9304e-05 2024-09-21T15:50:53.676 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.676 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.676 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:53.676 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209265+0000", 2024-09-21T15:50:53.676 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3532e-05 2024-09-21T15:50:53.676 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.676 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.676 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:53.677 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209272+0000", 2024-09-21T15:50:53.677 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.289e-06 2024-09-21T15:50:53.677 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.677 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.677 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:53.677 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209321+0000", 2024-09-21T15:50:53.677 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8773999999999998e-05 2024-09-21T15:50:53.677 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.677 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.677 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:53.677 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209341+0000", 2024-09-21T15:50:53.677 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0387999999999999e-05 2024-09-21T15:50:53.677 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.677 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.677 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:53.678 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.213464+0000", 2024-09-21T15:50:53.678 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0041232450000000002 2024-09-21T15:50:53.678 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.678 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.678 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:53.678 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.213495+0000", 2024-09-21T15:50:53.678 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0793999999999997e-05 2024-09-21T15:50:53.678 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.678 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:53.678 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.678 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.678 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.678 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13445 3.0 e31/22 3:038b8840:::rbd_data.1133dc5adfdf.0000000000000035:head v 31'50, mlcod=31'48)", 2024-09-21T15:50:53.678 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.232497+0000", 2024-09-21T15:50:53.679 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 96.431173543, 2024-09-21T15:50:53.679 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0040493559999999996, 2024-09-21T15:50:53.679 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:53.679 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:53.679 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:53.679 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.679 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:53.679 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232497+0000", 2024-09-21T15:50:53.679 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.679 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.679 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.679 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:53.679 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232497+0000", 2024-09-21T15:50:53.679 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.680 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.680 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.680 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:53.680 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232496+0000", 2024-09-21T15:50:53.680 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:53.680 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.680 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.680 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:53.680 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232503+0000", 2024-09-21T15:50:53.680 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7320000000000001e-06 2024-09-21T15:50:53.680 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.680 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.680 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:53.680 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232512+0000", 2024-09-21T15:50:53.681 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9259999999999995e-06 2024-09-21T15:50:53.681 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.681 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.681 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:53.681 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232520+0000", 2024-09-21T15:50:53.681 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3699999999999997e-06 2024-09-21T15:50:53.681 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.681 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.681 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:53.681 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232564+0000", 2024-09-21T15:50:53.681 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4128999999999997e-05 2024-09-21T15:50:53.681 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.681 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.681 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:53.681 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232585+0000", 2024-09-21T15:50:53.682 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0704999999999999e-05 2024-09-21T15:50:53.682 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.682 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.682 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:53.682 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.236507+0000", 2024-09-21T15:50:53.682 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0039218730000000002 2024-09-21T15:50:53.682 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.682 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.682 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:53.682 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.236546+0000", 2024-09-21T15:50:53.682 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9399e-05 2024-09-21T15:50:53.682 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.682 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:53.682 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.682 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.683 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.683 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13832 3.0 e31/22 3:0792efd2:::rbd_data.1133dc5adfdf.0000000000000090:head v 31'58, mlcod=31'56)", 2024-09-21T15:50:53.683 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.527530+0000", 2024-09-21T15:50:53.683 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 96.136140017000002, 2024-09-21T15:50:53.683 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035256530000000001, 2024-09-21T15:50:53.683 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:53.683 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:53.683 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:53.683 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.683 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:53.683 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527530+0000", 2024-09-21T15:50:53.683 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.683 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.683 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.684 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:53.684 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527530+0000", 2024-09-21T15:50:53.684 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.684 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.684 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.684 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:53.684 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527529+0000", 2024-09-21T15:50:53.684 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:50:53.684 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.684 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.684 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:53.684 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527550+0000", 2024-09-21T15:50:53.684 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1387999999999999e-05 2024-09-21T15:50:53.684 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.685 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.685 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:53.685 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527568+0000", 2024-09-21T15:50:53.685 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8068e-05 2024-09-21T15:50:53.685 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.685 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.685 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:53.685 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527577+0000", 2024-09-21T15:50:53.685 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9919999999999996e-06 2024-09-21T15:50:53.685 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.685 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.685 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:53.685 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527626+0000", 2024-09-21T15:50:53.685 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8597000000000002e-05 2024-09-21T15:50:53.685 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.686 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.686 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:53.686 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527652+0000", 2024-09-21T15:50:53.686 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5993e-05 2024-09-21T15:50:53.686 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.686 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.686 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:53.686 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.531024+0000", 2024-09-21T15:50:53.686 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033721910000000001 2024-09-21T15:50:53.686 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.686 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.686 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:53.686 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.531056+0000", 2024-09-21T15:50:53.686 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1628000000000003e-05 2024-09-21T15:50:53.686 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.687 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:53.687 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.687 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.687 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.687 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13839 3.0 e31/22 3:03c4fb32:::rbd_data.1133dc5adfdf.0000000000000099:head v 31'59, mlcod=31'56)", 2024-09-21T15:50:53.687 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.528097+0000", 2024-09-21T15:50:53.687 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 96.135573418999996, 2024-09-21T15:50:53.687 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0057228840000000001, 2024-09-21T15:50:53.687 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:53.687 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:53.687 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:53.687 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.687 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:53.687 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528097+0000", 2024-09-21T15:50:53.688 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.688 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.688 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.688 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:53.688 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528097+0000", 2024-09-21T15:50:53.688 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.688 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.688 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.688 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:53.688 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528096+0000", 2024-09-21T15:50:53.688 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:53.688 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.688 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.688 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:53.688 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528105+0000", 2024-09-21T15:50:53.689 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7350000000000002e-06 2024-09-21T15:50:53.689 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.689 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.689 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:53.689 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528117+0000", 2024-09-21T15:50:53.689 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1909000000000001e-05 2024-09-21T15:50:53.689 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.689 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.689 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:53.689 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528124+0000", 2024-09-21T15:50:53.689 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8419999999999999e-06 2024-09-21T15:50:53.689 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.689 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.689 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:53.689 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528168+0000", 2024-09-21T15:50:53.690 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4301999999999997e-05 2024-09-21T15:50:53.690 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.690 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.690 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:53.690 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528191+0000", 2024-09-21T15:50:53.690 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2622000000000001e-05 2024-09-21T15:50:53.690 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.690 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.690 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:53.690 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.533780+0000", 2024-09-21T15:50:53.690 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0055897519999999999 2024-09-21T15:50:53.690 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.690 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.690 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:53.690 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.533820+0000", 2024-09-21T15:50:53.691 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9366999999999998e-05 2024-09-21T15:50:53.691 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.691 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:53.691 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.691 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.691 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.691 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13884 3.9 e31/22 3:93003736:::rbd_data.1133dc5adfdf.00000000000000a1:head v 31'54, mlcod=31'52)", 2024-09-21T15:50:53.691 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.562433+0000", 2024-09-21T15:50:53.691 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 96.101236670999995, 2024-09-21T15:50:53.691 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.005162313, 2024-09-21T15:50:53.691 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:53.691 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:53.691 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:53.691 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.692 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:53.692 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562433+0000", 2024-09-21T15:50:53.692 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.692 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.692 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.692 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:53.692 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562433+0000", 2024-09-21T15:50:53.692 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.692 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.692 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.692 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:53.692 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562432+0000", 2024-09-21T15:50:53.692 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:53.692 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.693 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.693 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:53.693 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562467+0000", 2024-09-21T15:50:53.693 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4885999999999998e-05 2024-09-21T15:50:53.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.693 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.693 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:53.693 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562481+0000", 2024-09-21T15:50:53.693 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3604000000000001e-05 2024-09-21T15:50:53.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.693 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.693 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:53.693 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562489+0000", 2024-09-21T15:50:53.693 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1829999999999994e-06 2024-09-21T15:50:53.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.694 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.694 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:53.694 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562536+0000", 2024-09-21T15:50:53.694 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6830999999999999e-05 2024-09-21T15:50:53.694 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.694 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.694 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:53.694 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562559+0000", 2024-09-21T15:50:53.694 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2765999999999999e-05 2024-09-21T15:50:53.694 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.694 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.694 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:53.694 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.567566+0000", 2024-09-21T15:50:53.694 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0050073219999999998 2024-09-21T15:50:53.695 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.695 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.695 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:53.695 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.567596+0000", 2024-09-21T15:50:53.695 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9802999999999999e-05 2024-09-21T15:50:53.695 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.695 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:53.695 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.695 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.695 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.695 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:22340 3.9 e34/22 3:910a5abe:::rbd_data.1133896adcfd.0000000000000023:head v 34'92, mlcod=34'90)", 2024-09-21T15:50:53.695 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:21.628025+0000", 2024-09-21T15:50:53.695 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 92.035645290999994, 2024-09-21T15:50:53.695 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046196010000000001, 2024-09-21T15:50:53.696 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:53.696 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:53.696 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:53.696 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.696 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:53.696 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628025+0000", 2024-09-21T15:50:53.696 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.696 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.696 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.696 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:53.696 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628025+0000", 2024-09-21T15:50:53.696 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.696 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.696 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.696 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:53.696 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628024+0000", 2024-09-21T15:50:53.697 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:53.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.697 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.697 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:53.697 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628034+0000", 2024-09-21T15:50:53.697 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3470000000000005e-06 2024-09-21T15:50:53.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.697 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.697 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:53.697 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628043+0000", 2024-09-21T15:50:53.697 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9450000000000005e-06 2024-09-21T15:50:53.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.697 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.697 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:53.698 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628048+0000", 2024-09-21T15:50:53.698 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8080000000000004e-06 2024-09-21T15:50:53.698 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.698 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.698 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:53.698 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628081+0000", 2024-09-21T15:50:53.698 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2738999999999999e-05 2024-09-21T15:50:53.698 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.698 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.698 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:53.698 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628094+0000", 2024-09-21T15:50:53.698 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3273e-05 2024-09-21T15:50:53.698 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.698 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.698 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:53.699 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.632616+0000", 2024-09-21T15:50:53.699 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045217269999999997 2024-09-21T15:50:53.699 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.699 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.699 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:53.699 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.632644+0000", 2024-09-21T15:50:53.699 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8469000000000001e-05 2024-09-21T15:50:53.699 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.699 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:53.699 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.699 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.699 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.699 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:22417 3.4 e34/22 3:25310a10:::rbd_data.1133896adcfd.0000000000000075:head v 34'48, mlcod=34'46)", 2024-09-21T15:50:53.699 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:21.704297+0000", 2024-09-21T15:50:53.699 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 91.959373193000005, 2024-09-21T15:50:53.700 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0038101799999999998, 2024-09-21T15:50:53.700 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:53.700 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:53.700 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:53.700 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.700 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:53.700 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704297+0000", 2024-09-21T15:50:53.700 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.700 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.700 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.700 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:53.700 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704297+0000", 2024-09-21T15:50:53.700 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.700 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.701 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.701 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:53.701 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704296+0000", 2024-09-21T15:50:53.701 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:50:53.701 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.701 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.701 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:53.701 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704308+0000", 2024-09-21T15:50:53.701 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2559e-05 2024-09-21T15:50:53.701 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.701 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.701 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:53.701 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704337+0000", 2024-09-21T15:50:53.701 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8960000000000001e-05 2024-09-21T15:50:53.701 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.702 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.702 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:53.702 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704345+0000", 2024-09-21T15:50:53.702 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3059999999999998e-06 2024-09-21T15:50:53.702 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.702 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.702 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:53.702 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704408+0000", 2024-09-21T15:50:53.702 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2156000000000006e-05 2024-09-21T15:50:53.702 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.702 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.702 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:53.702 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704430+0000", 2024-09-21T15:50:53.703 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1931000000000001e-05 2024-09-21T15:50:53.703 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.703 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.703 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:53.703 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.708073+0000", 2024-09-21T15:50:53.703 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003643028 2024-09-21T15:50:53.703 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.703 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.703 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:53.703 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.708107+0000", 2024-09-21T15:50:53.703 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4613999999999998e-05 2024-09-21T15:50:53.703 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.703 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:53.703 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.704 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.704 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.704 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:25278 3.4 e38/22 3:2251cc1c:::rbd_data.1133ebd6d3ba.0000000000000024:head v 38'51, mlcod=38'49)", 2024-09-21T15:50:53.704 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.317337+0000", 2024-09-21T15:50:53.704 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 87.346333432999998, 2024-09-21T15:50:53.704 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053904110000000003, 2024-09-21T15:50:53.704 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:53.704 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:53.704 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:53.704 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.704 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:53.704 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317337+0000", 2024-09-21T15:50:53.704 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.704 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.704 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.705 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:53.705 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317337+0000", 2024-09-21T15:50:53.705 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.705 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.705 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.705 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:53.705 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317335+0000", 2024-09-21T15:50:53.705 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:50:53.705 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.705 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.705 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:53.705 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317347+0000", 2024-09-21T15:50:53.705 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.19e-05 2024-09-21T15:50:53.705 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.705 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.706 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:53.706 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317369+0000", 2024-09-21T15:50:53.706 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2232e-05 2024-09-21T15:50:53.706 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.706 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.706 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:53.706 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317381+0000", 2024-09-21T15:50:53.706 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2231e-05 2024-09-21T15:50:53.706 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.706 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.706 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:53.706 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317425+0000", 2024-09-21T15:50:53.706 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3625000000000003e-05 2024-09-21T15:50:53.707 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.707 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.707 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:53.707 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317449+0000", 2024-09-21T15:50:53.707 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4301999999999999e-05 2024-09-21T15:50:53.707 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.707 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.707 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:53.707 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.322688+0000", 2024-09-21T15:50:53.707 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.005238609 2024-09-21T15:50:53.707 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.707 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.707 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:53.707 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.322727+0000", 2024-09-21T15:50:53.707 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9201e-05 2024-09-21T15:50:53.708 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.708 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:53.708 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.708 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.708 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.708 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:25286 3.0 e38/22 3:0248fb60:::rbd_data.1133ebd6d3ba.0000000000000066:head v 38'109, mlcod=34'107)", 2024-09-21T15:50:53.708 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.317638+0000", 2024-09-21T15:50:53.708 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 87.346032554000004, 2024-09-21T15:50:53.708 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0054074370000000002, 2024-09-21T15:50:53.708 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:53.708 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:53.708 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:53.708 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.708 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:53.709 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317638+0000", 2024-09-21T15:50:53.709 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.709 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.709 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.709 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:53.709 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317638+0000", 2024-09-21T15:50:53.709 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.709 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.709 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.709 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:53.709 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317637+0000", 2024-09-21T15:50:53.709 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:53.709 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.709 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.709 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:53.710 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317643+0000", 2024-09-21T15:50:53.710 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5609999999999998e-06 2024-09-21T15:50:53.710 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.710 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.710 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:53.710 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317653+0000", 2024-09-21T15:50:53.710 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0705e-05 2024-09-21T15:50:53.710 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.710 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.710 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:53.710 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317660+0000", 2024-09-21T15:50:53.710 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2099999999999998e-06 2024-09-21T15:50:53.710 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.710 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.711 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:53.711 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317704+0000", 2024-09-21T15:50:53.711 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4659000000000001e-05 2024-09-21T15:50:53.711 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.711 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.711 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:53.711 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317725+0000", 2024-09-21T15:50:53.711 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0537e-05 2024-09-21T15:50:53.711 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.711 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.711 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:53.711 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323010+0000", 2024-09-21T15:50:53.711 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052853420000000002 2024-09-21T15:50:53.711 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.712 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.712 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:53.712 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323045+0000", 2024-09-21T15:50:53.712 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4860000000000002e-05 2024-09-21T15:50:53.712 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.712 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:53.712 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.712 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.712 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.712 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:25280 3.4 e38/22 3:27fca38c:::rbd_data.1133ebd6d3ba.0000000000000026:head v 38'52, mlcod=38'49)", 2024-09-21T15:50:53.712 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.317684+0000", 2024-09-21T15:50:53.712 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 87.345986658000001, 2024-09-21T15:50:53.712 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053570809999999996, 2024-09-21T15:50:53.712 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:53.712 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:53.713 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:53.713 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.713 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:53.713 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317684+0000", 2024-09-21T15:50:53.713 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.713 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.713 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.713 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:53.713 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317684+0000", 2024-09-21T15:50:53.713 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.713 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.713 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.713 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:53.713 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317683+0000", 2024-09-21T15:50:53.713 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:53.714 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.714 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.714 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:53.714 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317688+0000", 2024-09-21T15:50:53.714 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5009999999999998e-06 2024-09-21T15:50:53.714 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.714 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.714 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:53.714 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317694+0000", 2024-09-21T15:50:53.714 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3579999999999999e-06 2024-09-21T15:50:53.714 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.714 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.714 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:53.714 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317698+0000", 2024-09-21T15:50:53.715 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4150000000000001e-06 2024-09-21T15:50:53.715 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.715 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.715 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:53.715 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317738+0000", 2024-09-21T15:50:53.715 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9575e-05 2024-09-21T15:50:53.715 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.715 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.715 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:53.715 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317762+0000", 2024-09-21T15:50:53.715 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4173999999999999e-05 2024-09-21T15:50:53.715 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.715 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.715 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:53.715 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323005+0000", 2024-09-21T15:50:53.716 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00524333 2024-09-21T15:50:53.716 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.716 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.716 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:53.716 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323041+0000", 2024-09-21T15:50:53.716 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5139999999999999e-05 2024-09-21T15:50:53.716 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.716 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:53.716 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.716 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.716 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.716 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:34837 3.0 e43/22 3:038527c8:::rbd_header.1133fed1a840:head v 43'130, mlcod=43'128)", 2024-09-21T15:50:53.716 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:50:53.716 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 82.67978196, 2024-09-21T15:50:53.716 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045776519999999998, 2024-09-21T15:50:53.716 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:53.717 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:53.717 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:53.717 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.717 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:53.717 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:50:53.717 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.717 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.717 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.717 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:53.717 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:50:53.717 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.717 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.717 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.717 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:53.717 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:50:53.718 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:53.718 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.718 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.718 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:53.718 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983892+0000", 2024-09-21T15:50:53.718 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8450000000000002e-06 2024-09-21T15:50:53.718 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.718 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.718 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:53.718 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983902+0000", 2024-09-21T15:50:53.718 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0299999999999999e-06 2024-09-21T15:50:53.718 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.718 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.718 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:53.718 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983906+0000", 2024-09-21T15:50:53.723 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2300000000000002e-06 2024-09-21T15:50:53.723 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.723 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.723 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:53.723 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983940+0000", 2024-09-21T15:50:53.723 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4563000000000001e-05 2024-09-21T15:50:53.723 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.723 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.723 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:53.723 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983960+0000", 2024-09-21T15:50:53.724 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9199999999999999e-05 2024-09-21T15:50:53.724 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.724 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.724 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:53.724 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.988430+0000", 2024-09-21T15:50:53.724 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0044709149999999998 2024-09-21T15:50:53.724 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.724 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.724 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:53.724 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.988466+0000", 2024-09-21T15:50:53.724 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5420000000000003e-05 2024-09-21T15:50:53.724 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.724 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:53.724 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.724 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.725 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.725 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "pg_update_log_missing(3.0 epoch 50/22 rep_tid 498 entries 50'171 (0'0) error 3:5df4a5a0:::rbd_data.11339e87a32a.000000000000008e:head by client.4403.0:80207 0.000000 -2 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 47'167)", 2024-09-21T15:50:53.725 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:50:53.725 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 72.721909087, 2024-09-21T15:50:53.725 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0047253119999999997, 2024-09-21T15:50:53.725 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:53.725 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-09-21T15:50:53.725 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:53.725 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.725 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:53.725 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:50:53.725 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.725 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.725 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.726 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:53.726 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:50:53.726 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.726 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.726 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.726 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:53.726 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:50:53.726 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:53.726 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.727 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.727 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:53.727 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941768+0000", 2024-09-21T15:50:53.727 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7309999999999993e-06 2024-09-21T15:50:53.727 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.727 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.727 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:53.727 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941780+0000", 2024-09-21T15:50:53.727 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1399999999999999e-05 2024-09-21T15:50:53.727 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.727 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.727 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:53.727 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941789+0000", 2024-09-21T15:50:53.727 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5230000000000001e-06 2024-09-21T15:50:53.727 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.728 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.728 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:53.728 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941841+0000", 2024-09-21T15:50:53.728 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1863999999999997e-05 2024-09-21T15:50:53.728 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.728 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.728 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:53.728 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.946486+0000", 2024-09-21T15:50:53.728 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046452539999999997 2024-09-21T15:50:53.728 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.728 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:53.728 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.728 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.728 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.728 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "pg_update_log_missing(3.0 epoch 53/22 rep_tid 585 entries 53'201 (0'0) error 3:6a1d4890:::rbd_data.1133e079cae4.0000000000000004:head by client.4403.0:101089 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 53'198)", 2024-09-21T15:50:53.728 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:44.832827+0000", 2024-09-21T15:50:53.728 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 68.830843279000007, 2024-09-21T15:50:53.729 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053552929999999997, 2024-09-21T15:50:53.729 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:53.729 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-09-21T15:50:53.729 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:53.729 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.729 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:53.729 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832827+0000", 2024-09-21T15:50:53.729 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.729 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.729 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.729 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:53.729 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832827+0000", 2024-09-21T15:50:53.729 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.729 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.729 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.729 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:53.730 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832826+0000", 2024-09-21T15:50:53.730 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:50:53.730 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.730 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.730 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:53.730 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832835+0000", 2024-09-21T15:50:53.730 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4509999999999992e-06 2024-09-21T15:50:53.730 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.730 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.730 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:53.730 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832854+0000", 2024-09-21T15:50:53.730 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9352999999999999e-05 2024-09-21T15:50:53.730 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.730 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.730 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:53.730 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832866+0000", 2024-09-21T15:50:53.731 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1766000000000001e-05 2024-09-21T15:50:53.731 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.731 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.731 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:53.731 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.833091+0000", 2024-09-21T15:50:53.731 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00022490300000000001 2024-09-21T15:50:53.731 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.731 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.731 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:53.731 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.838182+0000", 2024-09-21T15:50:53.731 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0050915420000000001 2024-09-21T15:50:53.731 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.731 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:53.731 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.731 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.731 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.732 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:102822 3.4 e54/22 3:2729ba03:::rbd_object_map.113336ae1075:head v 54'208, mlcod=54'205)", 2024-09-21T15:50:53.732 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:46.249540+0000", 2024-09-21T15:50:53.732 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 67.414130047, 2024-09-21T15:50:53.732 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052654859999999998, 2024-09-21T15:50:53.732 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:53.732 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:53.732 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:53.732 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.732 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:53.732 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249540+0000", 2024-09-21T15:50:53.732 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.732 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.732 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.732 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:53.732 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249540+0000", 2024-09-21T15:50:53.732 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.733 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.733 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.733 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:53.733 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249539+0000", 2024-09-21T15:50:53.733 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:53.733 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.733 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.733 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:53.733 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249546+0000", 2024-09-21T15:50:53.733 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7290000000000003e-06 2024-09-21T15:50:53.733 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.733 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.733 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:53.733 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249558+0000", 2024-09-21T15:50:53.733 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1844e-05 2024-09-21T15:50:53.733 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.733 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.734 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:53.734 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249565+0000", 2024-09-21T15:50:53.734 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8210000000000004e-06 2024-09-21T15:50:53.734 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.734 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.734 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:53.734 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249602+0000", 2024-09-21T15:50:53.734 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7057000000000001e-05 2024-09-21T15:50:53.734 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.734 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.734 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:53.734 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249622+0000", 2024-09-21T15:50:53.734 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9678999999999999e-05 2024-09-21T15:50:53.734 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.734 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.735 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:53.735 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254783+0000", 2024-09-21T15:50:53.735 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0051611449999999998 2024-09-21T15:50:53.735 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.735 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.735 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:53.735 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254806+0000", 2024-09-21T15:50:53.735 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2853e-05 2024-09-21T15:50:53.735 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.735 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:53.735 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.735 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.735 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.735 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:102823 3.4 e54/22 3:2729ba03:::rbd_object_map.113336ae1075:head v 54'209, mlcod=54'205)", 2024-09-21T15:50:53.735 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:46.250318+0000", 2024-09-21T15:50:53.735 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 67.413352462000006, 2024-09-21T15:50:53.736 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045013539999999999, 2024-09-21T15:50:53.736 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:53.736 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:53.736 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:53.736 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.736 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:53.736 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250318+0000", 2024-09-21T15:50:53.736 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.736 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.736 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.736 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:53.736 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250318+0000", 2024-09-21T15:50:53.736 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.736 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.736 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.736 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:53.737 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250317+0000", 2024-09-21T15:50:53.737 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:53.737 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.737 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.737 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:53.737 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250322+0000", 2024-09-21T15:50:53.737 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.074e-06 2024-09-21T15:50:53.737 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.737 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.737 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:53.737 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250330+0000", 2024-09-21T15:50:53.737 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4730000000000003e-06 2024-09-21T15:50:53.737 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.737 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.737 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:53.737 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250335+0000", 2024-09-21T15:50:53.738 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8019999999999997e-06 2024-09-21T15:50:53.738 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.738 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.738 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:53.738 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250375+0000", 2024-09-21T15:50:53.738 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9459999999999998e-05 2024-09-21T15:50:53.738 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.738 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.738 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:53.738 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250388+0000", 2024-09-21T15:50:53.738 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3288999999999999e-05 2024-09-21T15:50:53.738 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.738 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.738 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:53.738 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254807+0000", 2024-09-21T15:50:53.739 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0044192220000000004 2024-09-21T15:50:53.739 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.739 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.739 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:53.739 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254819+0000", 2024-09-21T15:50:53.739 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1603999999999999e-05 2024-09-21T15:50:53.739 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.739 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:53.739 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.739 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.739 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.739 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:160603 3.9 e95/22 3:9171211b:::rbd_data.11338bae974a.0000000000000006:head v 95'343, mlcod=92'340)", 2024-09-21T15:50:53.739 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.215621+0000", 2024-09-21T15:50:53.739 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 22.448049431000001, 2024-09-21T15:50:53.739 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0041276250000000002, 2024-09-21T15:50:53.740 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:53.740 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:53.740 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:53.740 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.740 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:53.740 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215621+0000", 2024-09-21T15:50:53.740 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.740 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.740 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.740 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:53.740 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215621+0000", 2024-09-21T15:50:53.740 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.740 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.740 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.740 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:53.740 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215620+0000", 2024-09-21T15:50:53.741 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:53.741 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.741 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.741 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:53.741 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215627+0000", 2024-09-21T15:50:53.741 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8850000000000001e-06 2024-09-21T15:50:53.741 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.741 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.741 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:53.741 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215803+0000", 2024-09-21T15:50:53.741 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000176422 2024-09-21T15:50:53.741 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.741 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.741 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:53.741 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215814+0000", 2024-09-21T15:50:53.741 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0135e-05 2024-09-21T15:50:53.741 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.742 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.742 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:53.742 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215866+0000", 2024-09-21T15:50:53.742 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2327999999999998e-05 2024-09-21T15:50:53.742 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.742 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.742 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:53.742 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215887+0000", 2024-09-21T15:50:53.742 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1449000000000001e-05 2024-09-21T15:50:53.742 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.742 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.742 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:53.742 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.219719+0000", 2024-09-21T15:50:53.742 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0038312630000000001 2024-09-21T15:50:53.742 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.742 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.743 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:53.743 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.219748+0000", 2024-09-21T15:50:53.743 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9737999999999999e-05 2024-09-21T15:50:53.743 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.743 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:53.743 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.743 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.743 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.743 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:160974 3.4 e100/99 3:22dd2475:::rbd_id.image_client.0-clone39:head v 100'593, mlcod=0'0)", 2024-09-21T15:50:53.743 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:35.683900+0000", 2024-09-21T15:50:53.743 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.979769996999998, 2024-09-21T15:50:53.743 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0034160029999999999, 2024-09-21T15:50:53.743 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:53.743 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:53.743 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:53.743 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.743 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:53.744 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.683900+0000", 2024-09-21T15:50:53.744 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.744 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.744 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.744 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:53.744 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.683900+0000", 2024-09-21T15:50:53.744 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.744 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.744 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.744 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:53.744 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.683900+0000", 2024-09-21T15:50:53.744 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:53.744 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.744 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.744 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:53.744 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.683906+0000", 2024-09-21T15:50:53.745 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1759999999999998e-06 2024-09-21T15:50:53.745 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.745 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.745 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:53.745 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.683915+0000", 2024-09-21T15:50:53.745 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4320000000000005e-06 2024-09-21T15:50:53.745 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.745 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.745 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:53.745 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.683923+0000", 2024-09-21T15:50:53.745 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.259e-06 2024-09-21T15:50:53.745 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.745 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.745 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:53.745 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.683961+0000", 2024-09-21T15:50:53.745 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7014000000000002e-05 2024-09-21T15:50:53.746 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.746 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.746 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:53.746 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.683982+0000", 2024-09-21T15:50:53.746 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.145e-05 2024-09-21T15:50:53.746 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.746 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.746 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:53.746 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.687265+0000", 2024-09-21T15:50:53.746 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003283252 2024-09-21T15:50:53.746 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.746 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.746 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:53.746 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.687316+0000", 2024-09-21T15:50:53.746 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0963e-05 2024-09-21T15:50:53.746 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.746 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:53.747 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.747 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.747 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.747 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:161865 3.9 e105/22 3:90e88bad:::rbd_data.11338db70e6f.0000000000000015:head v 105'347, mlcod=103'345)", 2024-09-21T15:50:53.747 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:40.786848+0000", 2024-09-21T15:50:53.747 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.876822529, 2024-09-21T15:50:53.747 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0056408860000000003, 2024-09-21T15:50:53.747 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:53.747 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:53.747 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:53.747 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.747 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:53.747 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786848+0000", 2024-09-21T15:50:53.747 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.747 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.747 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.748 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:53.748 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786848+0000", 2024-09-21T15:50:53.748 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:53.748 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.748 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.748 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:53.748 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786847+0000", 2024-09-21T15:50:53.748 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:53.748 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.748 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.748 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:53.748 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786851+0000", 2024-09-21T15:50:53.748 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7419999999999999e-06 2024-09-21T15:50:53.748 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.748 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.748 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:53.749 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786872+0000", 2024-09-21T15:50:53.749 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1186e-05 2024-09-21T15:50:53.749 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.749 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.749 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:53.749 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786878+0000", 2024-09-21T15:50:53.749 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0970000000000002e-06 2024-09-21T15:50:53.749 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.749 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.749 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:53.749 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786918+0000", 2024-09-21T15:50:53.749 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.998e-05 2024-09-21T15:50:53.749 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.749 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.749 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:53.749 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786941+0000", 2024-09-21T15:50:53.749 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.245e-05 2024-09-21T15:50:53.750 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.750 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.750 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:53.750 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.792460+0000", 2024-09-21T15:50:53.750 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0055191210000000001 2024-09-21T15:50:53.750 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:53.750 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:53.750 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:53.750 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.792489+0000", 2024-09-21T15:50:53.750 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8552e-05 2024-09-21T15:50:53.750 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.750 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:53.750 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.750 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:53.750 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:53.751 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:53.751 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:53.663+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:53.752 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:50:53.752 INFO:teuthology.orchestra.run.smithi089.stderr:marked out osd.3. 2024-09-21T15:50:53.753 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T15:50:53.793 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:53.792+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:53.866 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:53.864+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:53.898 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:53.899 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:53.899 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:50:53.899 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:53.908 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:50:53.992 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:53.991+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:54.052 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:54.052 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:54.052 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:50:54.052 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:50:54.052 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:54.060 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:50:54.068 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:54.067+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:54.168 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:54.167+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:54.205 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:54.205 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:50:54.205 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:50:54.205 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:50:54.205 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.205 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:14051 3.8 3:13437810:::rbd_data.1133dc5adfdf.0000000000000039:head [write 716800~796] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-21T15:50:54.205 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:50:54.205 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 96.508315956999994, 2024-09-21T15:50:54.205 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.98064002699999997, 2024-09-21T15:50:54.206 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:54.206 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:54.206 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:54.206 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:54.206 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:54.206 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 14051 2024-09-21T15:50:54.206 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.206 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:54.206 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.206 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:54.206 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:50:54.206 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:54.206 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.207 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.207 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:54.207 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:50:54.207 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:54.207 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.207 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.207 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:54.207 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:50:54.207 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:54.207 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.207 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.207 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:54.207 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695131+0000", 2024-09-21T15:50:54.207 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4680000000000003e-06 2024-09-21T15:50:54.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.208 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.208 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:54.208 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695139+0000", 2024-09-21T15:50:54.208 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9400000000000002e-06 2024-09-21T15:50:54.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.208 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.208 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:54.208 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695143+0000", 2024-09-21T15:50:54.208 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9899999999999999e-06 2024-09-21T15:50:54.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.208 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.208 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:54.208 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695190+0000", 2024-09-21T15:50:54.208 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6458000000000001e-05 2024-09-21T15:50:54.209 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.209 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.209 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:50:54.209 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695201+0000", 2024-09-21T15:50:54.209 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0964999999999999e-05 2024-09-21T15:50:54.209 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.209 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.209 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:54.209 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.669490+0000", 2024-09-21T15:50:54.209 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.97428980499999995 2024-09-21T15:50:54.209 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.209 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.209 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-09-21T15:50:54.209 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.670033+0000", 2024-09-21T15:50:54.210 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00054252599999999999 2024-09-21T15:50:54.210 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.210 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.210 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:54.210 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.671095+0000", 2024-09-21T15:50:54.210 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001062183 2024-09-21T15:50:54.210 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.210 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.210 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:54.210 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.671186+0000", 2024-09-21T15:50:54.210 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1191000000000002e-05 2024-09-21T15:50:54.210 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.210 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.210 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-09-21T15:50:54.211 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.671346+0000", 2024-09-21T15:50:54.211 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015980399999999999 2024-09-21T15:50:54.211 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.211 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.211 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:54.211 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.672221+0000", 2024-09-21T15:50:54.211 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00087471999999999995 2024-09-21T15:50:54.211 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.211 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.211 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:54.211 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.675619+0000", 2024-09-21T15:50:54.211 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00339858 2024-09-21T15:50:54.211 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.211 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.211 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:54.212 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.675670+0000", 2024-09-21T15:50:54.212 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0664000000000001e-05 2024-09-21T15:50:54.212 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.212 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.212 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:54.212 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.675768+0000", 2024-09-21T15:50:54.212 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7415000000000002e-05 2024-09-21T15:50:54.212 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:54.212 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:54.212 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:54.212 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.212 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.212 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116235 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 761856~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:54.212 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048806+0000", 2024-09-21T15:50:54.213 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 58.154637073000004, 2024-09-21T15:50:54.213 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056818556999999999, 2024-09-21T15:50:54.213 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:54.213 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:54.213 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:54.213 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:54.213 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:54.213 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116235 2024-09-21T15:50:54.213 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.213 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:54.213 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.213 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:54.213 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048806+0000", 2024-09-21T15:50:54.213 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:54.213 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.213 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.214 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:54.214 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048806+0000", 2024-09-21T15:50:54.214 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:54.214 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.214 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.214 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:54.214 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048806+0000", 2024-09-21T15:50:54.214 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:54.214 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.214 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.214 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:54.214 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048807+0000", 2024-09-21T15:50:54.214 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.001e-06 2024-09-21T15:50:54.215 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.215 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.215 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:54.215 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048810+0000", 2024-09-21T15:50:54.215 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.711e-06 2024-09-21T15:50:54.215 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.215 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.215 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:54.215 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048812+0000", 2024-09-21T15:50:54.215 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2290000000000002e-06 2024-09-21T15:50:54.215 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.215 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.215 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:54.216 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105495+0000", 2024-09-21T15:50:54.216 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056682679999999999 2024-09-21T15:50:54.216 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.216 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.216 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:54.216 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105565+0000", 2024-09-21T15:50:54.216 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9868000000000003e-05 2024-09-21T15:50:54.216 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.216 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.216 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:54.216 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105625+0000", 2024-09-21T15:50:54.216 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0268e-05 2024-09-21T15:50:54.216 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:54.217 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:54.217 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:54.217 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.217 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.217 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116236 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 768000~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:54.217 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048814+0000", 2024-09-21T15:50:54.217 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 58.154629409999998, 2024-09-21T15:50:54.217 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056959587999999998, 2024-09-21T15:50:54.217 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:54.217 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:54.217 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:54.217 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:54.217 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:54.217 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116236 2024-09-21T15:50:54.217 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.218 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:54.218 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.218 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:54.218 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048814+0000", 2024-09-21T15:50:54.218 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:54.218 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.218 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.218 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:54.218 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048814+0000", 2024-09-21T15:50:54.218 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:54.218 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.218 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.218 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:54.218 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048814+0000", 2024-09-21T15:50:54.218 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:54.219 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.219 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.219 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:54.219 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048815+0000", 2024-09-21T15:50:54.219 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7999999999999993e-07 2024-09-21T15:50:54.219 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.219 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.219 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:54.219 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048817+0000", 2024-09-21T15:50:54.219 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6620000000000001e-06 2024-09-21T15:50:54.219 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.219 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.219 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:54.219 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048820+0000", 2024-09-21T15:50:54.220 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2800000000000002e-06 2024-09-21T15:50:54.220 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.220 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.220 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:54.220 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105647+0000", 2024-09-21T15:50:54.220 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056827253000000001 2024-09-21T15:50:54.220 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.220 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.220 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:54.220 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105716+0000", 2024-09-21T15:50:54.220 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8761000000000001e-05 2024-09-21T15:50:54.220 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.220 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.220 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:54.220 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105774+0000", 2024-09-21T15:50:54.221 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7855000000000001e-05 2024-09-21T15:50:54.221 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:54.221 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:54.221 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:54.221 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.221 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.221 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116237 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 774144~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:54.221 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048822+0000", 2024-09-21T15:50:54.221 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 58.154621724999998, 2024-09-21T15:50:54.221 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057114028999999997, 2024-09-21T15:50:54.221 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:54.221 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:54.221 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:54.221 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:54.222 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:54.222 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116237 2024-09-21T15:50:54.222 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.222 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:54.222 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.222 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:54.222 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048822+0000", 2024-09-21T15:50:54.222 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:54.222 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.222 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.222 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:54.222 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048822+0000", 2024-09-21T15:50:54.222 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:54.222 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.222 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.223 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:54.223 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048822+0000", 2024-09-21T15:50:54.223 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:54.223 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.223 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.223 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:54.223 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048823+0000", 2024-09-21T15:50:54.223 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9699999999999994e-07 2024-09-21T15:50:54.223 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.223 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.223 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:54.223 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048825+0000", 2024-09-21T15:50:54.223 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.65e-06 2024-09-21T15:50:54.223 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.223 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.224 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:54.224 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048828+0000", 2024-09-21T15:50:54.224 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8890000000000002e-06 2024-09-21T15:50:54.224 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.224 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.224 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:54.224 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105802+0000", 2024-09-21T15:50:54.224 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056974415 2024-09-21T15:50:54.224 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.224 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.224 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:54.224 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105871+0000", 2024-09-21T15:50:54.224 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8727999999999997e-05 2024-09-21T15:50:54.225 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.225 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.225 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:54.225 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105936+0000", 2024-09-21T15:50:54.225 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4554999999999994e-05 2024-09-21T15:50:54.225 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:54.225 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:54.225 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:54.225 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.225 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.225 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116238 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 780288~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:54.225 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048830+0000", 2024-09-21T15:50:54.225 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 58.154613466000001, 2024-09-21T15:50:54.225 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057296574000000003, 2024-09-21T15:50:54.225 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:54.226 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:54.226 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:54.226 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:54.226 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:54.226 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116238 2024-09-21T15:50:54.226 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.226 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:54.226 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.226 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:54.226 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048830+0000", 2024-09-21T15:50:54.226 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:54.226 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.226 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.226 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:54.227 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048830+0000", 2024-09-21T15:50:54.227 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:54.227 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.227 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.227 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:54.227 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048830+0000", 2024-09-21T15:50:54.227 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:54.227 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.227 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.227 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:54.227 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048831+0000", 2024-09-21T15:50:54.227 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0160000000000001e-06 2024-09-21T15:50:54.227 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.227 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.227 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:54.228 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048834+0000", 2024-09-21T15:50:54.228 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.813e-06 2024-09-21T15:50:54.228 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.228 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.228 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:54.228 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048836+0000", 2024-09-21T15:50:54.228 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6790000000000001e-06 2024-09-21T15:50:54.228 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.228 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.228 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:54.228 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105961+0000", 2024-09-21T15:50:54.229 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057124705999999997 2024-09-21T15:50:54.229 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.229 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.229 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:54.229 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106042+0000", 2024-09-21T15:50:54.229 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0820000000000002e-05 2024-09-21T15:50:54.229 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.229 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.229 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:54.229 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106127+0000", 2024-09-21T15:50:54.229 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4746000000000005e-05 2024-09-21T15:50:54.229 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:54.229 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:54.229 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:54.229 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.230 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.230 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116324 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 524288~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:54.230 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048838+0000", 2024-09-21T15:50:54.230 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 58.154605156999999, 2024-09-21T15:50:54.230 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057477185, 2024-09-21T15:50:54.230 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:54.230 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:54.230 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:54.230 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:54.230 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:54.230 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116324 2024-09-21T15:50:54.230 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.230 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:54.231 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.231 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:54.231 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048838+0000", 2024-09-21T15:50:54.231 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:54.231 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.231 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.231 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:54.231 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048838+0000", 2024-09-21T15:50:54.231 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:54.231 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.231 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.231 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:54.231 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048838+0000", 2024-09-21T15:50:54.231 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:54.231 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.231 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.231 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:54.232 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048842+0000", 2024-09-21T15:50:54.232 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4829999999999999e-06 2024-09-21T15:50:54.232 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.232 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.232 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:54.232 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048846+0000", 2024-09-21T15:50:54.232 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0470000000000004e-06 2024-09-21T15:50:54.232 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.232 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.232 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:54.232 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048849+0000", 2024-09-21T15:50:54.232 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3689999999999998e-06 2024-09-21T15:50:54.232 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.232 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.232 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:54.233 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106157+0000", 2024-09-21T15:50:54.233 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057307942000000001 2024-09-21T15:50:54.233 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.233 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.233 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:54.233 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106237+0000", 2024-09-21T15:50:54.233 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0320000000000003e-05 2024-09-21T15:50:54.233 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.233 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.233 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:54.233 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106315+0000", 2024-09-21T15:50:54.233 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8227999999999997e-05 2024-09-21T15:50:54.233 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:54.233 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:54.233 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:54.233 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.234 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.234 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116325 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 530432~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:54.234 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048851+0000", 2024-09-21T15:50:54.234 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 58.154591985000003, 2024-09-21T15:50:54.234 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057581036000000002, 2024-09-21T15:50:54.234 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:54.234 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:54.234 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:54.234 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:54.234 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:54.234 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116325 2024-09-21T15:50:54.234 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.234 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:54.234 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.235 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:54.235 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048851+0000", 2024-09-21T15:50:54.235 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:54.235 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.235 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.235 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:54.235 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048851+0000", 2024-09-21T15:50:54.235 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:54.235 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.235 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.235 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:54.235 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048851+0000", 2024-09-21T15:50:54.235 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:54.235 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.235 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.236 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:54.236 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048852+0000", 2024-09-21T15:50:54.236 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.077e-06 2024-09-21T15:50:54.236 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.236 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.236 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:54.236 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048855+0000", 2024-09-21T15:50:54.236 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9859999999999999e-06 2024-09-21T15:50:54.236 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.236 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.236 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:54.236 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048858+0000", 2024-09-21T15:50:54.236 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3810000000000002e-06 2024-09-21T15:50:54.236 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.236 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.237 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:54.237 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106335+0000", 2024-09-21T15:50:54.237 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057477473000000001 2024-09-21T15:50:54.237 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.237 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.237 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:54.237 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106393+0000", 2024-09-21T15:50:54.237 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8050000000000002e-05 2024-09-21T15:50:54.237 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.237 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.237 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:54.237 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106432+0000", 2024-09-21T15:50:54.237 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9274999999999999e-05 2024-09-21T15:50:54.237 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:54.237 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:54.238 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:54.238 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.238 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.238 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116326 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 536576~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:54.238 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048860+0000", 2024-09-21T15:50:54.238 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 58.154583934000001, 2024-09-21T15:50:54.238 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057774197999999999, 2024-09-21T15:50:54.238 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:54.238 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:54.238 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:54.238 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:54.238 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:54.238 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116326 2024-09-21T15:50:54.239 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.239 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:54.239 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.239 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:54.239 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048860+0000", 2024-09-21T15:50:54.239 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:54.239 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.239 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.239 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:54.239 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048860+0000", 2024-09-21T15:50:54.239 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:54.239 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.239 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.239 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:54.239 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048859+0000", 2024-09-21T15:50:54.240 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:54.240 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.240 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.240 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:54.240 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048860+0000", 2024-09-21T15:50:54.240 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.02e-06 2024-09-21T15:50:54.240 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.240 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.240 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:54.240 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048863+0000", 2024-09-21T15:50:54.240 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1039999999999998e-06 2024-09-21T15:50:54.240 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.240 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.240 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:54.240 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048867+0000", 2024-09-21T15:50:54.241 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.563e-06 2024-09-21T15:50:54.241 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.241 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.241 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:54.241 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106465+0000", 2024-09-21T15:50:54.241 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057597789000000003 2024-09-21T15:50:54.241 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.241 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.241 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:54.241 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106552+0000", 2024-09-21T15:50:54.241 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7602000000000006e-05 2024-09-21T15:50:54.241 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.241 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.241 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:54.242 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106634+0000", 2024-09-21T15:50:54.242 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1321000000000003e-05 2024-09-21T15:50:54.242 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:54.242 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:54.242 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:54.242 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.242 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.242 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116327 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 542720~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:54.242 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048870+0000", 2024-09-21T15:50:54.242 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 58.154573061999997, 2024-09-21T15:50:54.242 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057883148000000002, 2024-09-21T15:50:54.242 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:54.242 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:54.243 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:54.243 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:54.243 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:54.243 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116327 2024-09-21T15:50:54.243 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.243 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:54.243 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.243 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:54.243 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048870+0000", 2024-09-21T15:50:54.243 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:54.243 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.243 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.243 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:54.243 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048870+0000", 2024-09-21T15:50:54.243 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:54.243 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.244 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.244 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:54.244 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048870+0000", 2024-09-21T15:50:54.244 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:54.244 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.244 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.244 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:54.244 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048872+0000", 2024-09-21T15:50:54.244 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.824e-06 2024-09-21T15:50:54.244 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.244 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.244 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:54.244 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048879+0000", 2024-09-21T15:50:54.244 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6749999999999996e-06 2024-09-21T15:50:54.244 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.245 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.245 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:54.245 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048883+0000", 2024-09-21T15:50:54.245 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2200000000000003e-06 2024-09-21T15:50:54.245 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.245 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.245 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:54.245 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106663+0000", 2024-09-21T15:50:54.245 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057780049 2024-09-21T15:50:54.245 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.245 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.245 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:54.245 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106711+0000", 2024-09-21T15:50:54.245 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8433999999999997e-05 2024-09-21T15:50:54.246 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.246 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.246 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:54.246 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106754+0000", 2024-09-21T15:50:54.246 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2236e-05 2024-09-21T15:50:54.246 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:54.246 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:54.246 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:54.246 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.246 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.246 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116328 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 548864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:54.246 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048887+0000", 2024-09-21T15:50:54.246 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 58.154556542000002, 2024-09-21T15:50:54.246 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057981984, 2024-09-21T15:50:54.247 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:54.247 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:54.247 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:54.247 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:54.247 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:54.247 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116328 2024-09-21T15:50:54.247 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.247 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:54.247 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.247 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:54.247 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048887+0000", 2024-09-21T15:50:54.247 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:54.247 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.247 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.248 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:54.248 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048887+0000", 2024-09-21T15:50:54.248 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:54.248 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.248 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.248 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:54.248 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048887+0000", 2024-09-21T15:50:54.248 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:54.248 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.248 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.248 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:54.248 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048891+0000", 2024-09-21T15:50:54.248 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.843e-06 2024-09-21T15:50:54.248 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.248 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.249 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:54.249 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048897+0000", 2024-09-21T15:50:54.249 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5419999999999996e-06 2024-09-21T15:50:54.249 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.249 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.249 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:54.249 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048901+0000", 2024-09-21T15:50:54.249 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4689999999999999e-06 2024-09-21T15:50:54.249 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.249 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.249 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:54.249 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106780+0000", 2024-09-21T15:50:54.249 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057878259000000001 2024-09-21T15:50:54.249 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.249 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.250 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:54.250 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106827+0000", 2024-09-21T15:50:54.250 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7098000000000002e-05 2024-09-21T15:50:54.250 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.250 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.250 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:54.250 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106869+0000", 2024-09-21T15:50:54.250 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2129999999999998e-05 2024-09-21T15:50:54.250 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:54.250 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:54.250 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:54.250 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.250 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.250 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116329 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 555008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:54.251 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048905+0000", 2024-09-21T15:50:54.251 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 58.154538432000003, 2024-09-21T15:50:54.251 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058115554999999999, 2024-09-21T15:50:54.251 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:54.251 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:54.251 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:54.251 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:54.251 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:54.251 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116329 2024-09-21T15:50:54.251 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.251 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:54.251 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.252 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:54.252 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048905+0000", 2024-09-21T15:50:54.252 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:54.252 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.252 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.252 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:54.252 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048905+0000", 2024-09-21T15:50:54.252 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:54.252 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.252 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.252 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:54.252 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048905+0000", 2024-09-21T15:50:54.252 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:54.252 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.252 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.253 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:54.253 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048907+0000", 2024-09-21T15:50:54.253 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.086e-06 2024-09-21T15:50:54.253 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.253 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.253 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:54.253 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048912+0000", 2024-09-21T15:50:54.253 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2830000000000001e-06 2024-09-21T15:50:54.253 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.253 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.253 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:54.253 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048916+0000", 2024-09-21T15:50:54.253 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.98e-06 2024-09-21T15:50:54.253 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.253 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.254 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:54.254 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106907+0000", 2024-09-21T15:50:54.254 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057990719000000003 2024-09-21T15:50:54.254 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.254 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.254 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:54.254 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106969+0000", 2024-09-21T15:50:54.254 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1789000000000001e-05 2024-09-21T15:50:54.254 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.254 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.254 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:54.254 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107021+0000", 2024-09-21T15:50:54.254 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2030999999999997e-05 2024-09-21T15:50:54.255 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:54.255 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:54.255 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:54.255 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.255 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.255 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116330 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 561152~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:54.255 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048922+0000", 2024-09-21T15:50:54.255 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 58.154521273999997, 2024-09-21T15:50:54.255 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058242343000000002, 2024-09-21T15:50:54.255 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:54.255 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:54.255 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:54.255 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:54.255 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:54.255 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116330 2024-09-21T15:50:54.256 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.256 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:54.256 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.256 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:54.256 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048922+0000", 2024-09-21T15:50:54.256 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:54.256 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.256 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.256 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:54.256 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048922+0000", 2024-09-21T15:50:54.256 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:54.256 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.256 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.256 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:54.256 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048922+0000", 2024-09-21T15:50:54.257 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:54.257 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.257 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.257 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:54.257 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048924+0000", 2024-09-21T15:50:54.257 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1660000000000001e-06 2024-09-21T15:50:54.257 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.257 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.257 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:54.257 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048930+0000", 2024-09-21T15:50:54.257 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4529999999999999e-06 2024-09-21T15:50:54.257 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.257 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.257 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:54.257 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048934+0000", 2024-09-21T15:50:54.258 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0269999999999997e-06 2024-09-21T15:50:54.258 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.258 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.258 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:54.258 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107047+0000", 2024-09-21T15:50:54.258 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058112913000000002 2024-09-21T15:50:54.258 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.258 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.258 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:54.258 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107095+0000", 2024-09-21T15:50:54.258 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7778000000000002e-05 2024-09-21T15:50:54.258 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.258 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.258 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:54.258 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107165+0000", 2024-09-21T15:50:54.258 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9370999999999997e-05 2024-09-21T15:50:54.259 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:54.259 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:54.259 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:54.259 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.259 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.259 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116331 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 567296~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:54.259 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.049440+0000", 2024-09-21T15:50:54.259 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 58.154003179999997, 2024-09-21T15:50:54.259 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057878485, 2024-09-21T15:50:54.259 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:54.259 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:54.259 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:54.259 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:54.259 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:54.260 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116331 2024-09-21T15:50:54.260 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.260 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:54.260 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.260 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:54.260 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049440+0000", 2024-09-21T15:50:54.260 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:54.260 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.260 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.260 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:54.260 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049440+0000", 2024-09-21T15:50:54.260 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:54.260 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.260 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.261 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:54.261 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048939+0000", 2024-09-21T15:50:54.261 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9994984 2024-09-21T15:50:54.261 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.261 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.261 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:54.261 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049444+0000", 2024-09-21T15:50:54.261 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00050557200000000003 2024-09-21T15:50:54.261 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.261 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.261 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:54.261 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049460+0000", 2024-09-21T15:50:54.261 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6120000000000002e-05 2024-09-21T15:50:54.261 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.262 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.262 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:54.262 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049468+0000", 2024-09-21T15:50:54.262 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4209999999999998e-06 2024-09-21T15:50:54.262 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.262 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.262 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:54.262 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107189+0000", 2024-09-21T15:50:54.262 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057721019999999998 2024-09-21T15:50:54.262 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.262 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.262 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:54.262 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107254+0000", 2024-09-21T15:50:54.262 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5265000000000006e-05 2024-09-21T15:50:54.263 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.263 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.263 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:54.263 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107319+0000", 2024-09-21T15:50:54.263 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4784999999999997e-05 2024-09-21T15:50:54.263 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:54.263 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:54.263 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:54.263 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.263 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.263 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116332 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 573440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:54.263 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.049474+0000", 2024-09-21T15:50:54.263 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 58.153969918999998, 2024-09-21T15:50:54.263 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058013809, 2024-09-21T15:50:54.263 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:54.264 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:54.264 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:54.264 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:54.264 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:54.264 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116332 2024-09-21T15:50:54.264 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.264 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:54.264 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.264 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:54.264 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049474+0000", 2024-09-21T15:50:54.264 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:54.264 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.265 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.265 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:54.265 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049474+0000", 2024-09-21T15:50:54.265 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:54.265 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.265 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.265 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:54.265 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049473+0000", 2024-09-21T15:50:54.265 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:54.265 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.265 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.265 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:54.265 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049476+0000", 2024-09-21T15:50:54.265 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6249999999999999e-06 2024-09-21T15:50:54.265 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.266 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.266 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:54.266 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049482+0000", 2024-09-21T15:50:54.266 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0959999999999997e-06 2024-09-21T15:50:54.266 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.266 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.266 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:54.266 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049487+0000", 2024-09-21T15:50:54.266 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5949999999999998e-06 2024-09-21T15:50:54.266 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.266 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.266 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:54.266 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107344+0000", 2024-09-21T15:50:54.266 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057856594999999997 2024-09-21T15:50:54.266 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.267 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.267 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:54.267 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107422+0000", 2024-09-21T15:50:54.267 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7582000000000004e-05 2024-09-21T15:50:54.267 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.267 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.267 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:54.267 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107487+0000", 2024-09-21T15:50:54.267 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5723999999999996e-05 2024-09-21T15:50:54.267 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:54.267 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:54.267 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:54.267 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.267 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.268 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116333 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 579584~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:54.268 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.049993+0000", 2024-09-21T15:50:54.268 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 58.153450135999996, 2024-09-21T15:50:54.268 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057663537000000001, 2024-09-21T15:50:54.268 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:54.268 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:54.268 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:54.268 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:54.268 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:54.268 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116333 2024-09-21T15:50:54.268 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.268 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:54.268 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.268 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:54.268 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049993+0000", 2024-09-21T15:50:54.269 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:54.269 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.269 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.269 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:54.269 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049993+0000", 2024-09-21T15:50:54.269 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:54.269 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.269 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.269 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:54.269 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049491+0000", 2024-09-21T15:50:54.269 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9994979 2024-09-21T15:50:54.269 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.269 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.269 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:54.269 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049998+0000", 2024-09-21T15:50:54.269 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00050654200000000002 2024-09-21T15:50:54.270 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.270 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.270 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:54.270 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050010+0000", 2024-09-21T15:50:54.270 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1790999999999999e-05 2024-09-21T15:50:54.270 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.270 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.270 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:54.270 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050016+0000", 2024-09-21T15:50:54.270 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2179999999999996e-06 2024-09-21T15:50:54.270 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.270 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.270 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:54.270 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107513+0000", 2024-09-21T15:50:54.270 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057496927000000003 2024-09-21T15:50:54.271 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.271 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.271 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:54.271 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107590+0000", 2024-09-21T15:50:54.271 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7547999999999997e-05 2024-09-21T15:50:54.271 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.271 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.271 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:54.271 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107657+0000", 2024-09-21T15:50:54.271 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6630000000000004e-05 2024-09-21T15:50:54.271 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:54.271 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:54.271 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:54.271 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.272 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.272 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116334 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 585728~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:54.272 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.050091+0000", 2024-09-21T15:50:54.272 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 58.153352507999998, 2024-09-21T15:50:54.272 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057734355000000001, 2024-09-21T15:50:54.272 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:54.272 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:54.272 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:54.272 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:54.272 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:54.272 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116334 2024-09-21T15:50:54.272 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.272 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:54.272 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.272 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:54.273 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050091+0000", 2024-09-21T15:50:54.273 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:54.273 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.273 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.273 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:54.273 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050091+0000", 2024-09-21T15:50:54.273 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:54.273 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.273 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.273 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:54.273 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050021+0000", 2024-09-21T15:50:54.273 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999304 2024-09-21T15:50:54.273 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.273 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.273 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:54.273 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050095+0000", 2024-09-21T15:50:54.274 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4158000000000004e-05 2024-09-21T15:50:54.274 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.274 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.274 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:54.274 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050103+0000", 2024-09-21T15:50:54.274 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0770000000000002e-06 2024-09-21T15:50:54.274 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.274 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.274 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:54.274 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050108+0000", 2024-09-21T15:50:54.274 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6330000000000001e-06 2024-09-21T15:50:54.274 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.274 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.274 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:54.275 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107681+0000", 2024-09-21T15:50:54.275 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057572996000000001 2024-09-21T15:50:54.275 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.275 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.275 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:54.275 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107759+0000", 2024-09-21T15:50:54.275 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7368999999999997e-05 2024-09-21T15:50:54.275 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.275 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.275 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:54.275 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107825+0000", 2024-09-21T15:50:54.275 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6783000000000001e-05 2024-09-21T15:50:54.275 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:54.275 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:54.276 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:54.276 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.276 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.276 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116335 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 591872~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:54.276 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.050542+0000", 2024-09-21T15:50:54.276 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 58.152901923000002, 2024-09-21T15:50:54.276 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057466455999999999, 2024-09-21T15:50:54.276 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:54.276 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:54.276 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:54.276 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:54.276 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:54.276 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116335 2024-09-21T15:50:54.276 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.276 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:54.277 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.277 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:54.277 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050542+0000", 2024-09-21T15:50:54.277 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:54.277 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.277 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.277 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:54.277 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050542+0000", 2024-09-21T15:50:54.277 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:54.277 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.277 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.277 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:54.277 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050113+0000", 2024-09-21T15:50:54.277 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9995713 2024-09-21T15:50:54.277 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.278 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.278 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:54.278 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050546+0000", 2024-09-21T15:50:54.278 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00043328699999999997 2024-09-21T15:50:54.278 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.278 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.278 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:54.278 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050557+0000", 2024-09-21T15:50:54.278 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1011e-05 2024-09-21T15:50:54.278 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.278 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.278 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:54.278 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050564+0000", 2024-09-21T15:50:54.278 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1130000000000003e-06 2024-09-21T15:50:54.278 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.279 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.279 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:54.279 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107852+0000", 2024-09-21T15:50:54.279 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057287350000000001 2024-09-21T15:50:54.279 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.279 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.279 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:54.279 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107928+0000", 2024-09-21T15:50:54.279 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6668000000000005e-05 2024-09-21T15:50:54.279 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.279 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.279 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:54.279 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108008+0000", 2024-09-21T15:50:54.279 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9783999999999994e-05 2024-09-21T15:50:54.279 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:54.279 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:54.280 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:54.280 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.280 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.280 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116336 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 598016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:54.280 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.050834+0000", 2024-09-21T15:50:54.280 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 58.152609656999999, 2024-09-21T15:50:54.280 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057353572999999998, 2024-09-21T15:50:54.280 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:54.280 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:54.280 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:54.280 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:54.280 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:54.280 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116336 2024-09-21T15:50:54.280 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.280 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:54.280 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.281 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:54.281 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050834+0000", 2024-09-21T15:50:54.281 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:54.281 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.281 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.281 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:54.281 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050834+0000", 2024-09-21T15:50:54.281 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:54.281 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.281 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.281 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:54.281 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050571+0000", 2024-09-21T15:50:54.281 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9997373 2024-09-21T15:50:54.281 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.281 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.282 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:54.282 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050842+0000", 2024-09-21T15:50:54.282 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00027029099999999998 2024-09-21T15:50:54.282 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.282 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.282 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:54.282 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050853+0000", 2024-09-21T15:50:54.282 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1532000000000001e-05 2024-09-21T15:50:54.282 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.282 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.282 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:54.282 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050863+0000", 2024-09-21T15:50:54.282 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4239999999999999e-06 2024-09-21T15:50:54.282 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.282 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.282 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:54.283 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108045+0000", 2024-09-21T15:50:54.283 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057182415 2024-09-21T15:50:54.283 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.283 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.283 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:54.283 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108123+0000", 2024-09-21T15:50:54.283 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8275000000000005e-05 2024-09-21T15:50:54.283 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.283 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.283 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:54.283 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108187+0000", 2024-09-21T15:50:54.283 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4152000000000005e-05 2024-09-21T15:50:54.283 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:54.283 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:54.283 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:54.284 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.284 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.284 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116337 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 604160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:54.284 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.051192+0000", 2024-09-21T15:50:54.284 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 58.152251028999999, 2024-09-21T15:50:54.284 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057175981000000001, 2024-09-21T15:50:54.284 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:54.284 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:54.284 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:54.284 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:54.284 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:54.284 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116337 2024-09-21T15:50:54.284 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.284 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:54.284 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.285 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:54.285 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051192+0000", 2024-09-21T15:50:54.285 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:54.285 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.285 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.285 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:54.285 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051192+0000", 2024-09-21T15:50:54.285 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:54.285 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.285 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.285 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:54.285 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050870+0000", 2024-09-21T15:50:54.285 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9996777 2024-09-21T15:50:54.285 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.285 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.286 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:54.286 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051197+0000", 2024-09-21T15:50:54.286 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00032656299999999998 2024-09-21T15:50:54.286 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.286 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.286 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:54.286 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051209+0000", 2024-09-21T15:50:54.286 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2636000000000001e-05 2024-09-21T15:50:54.286 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.286 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.286 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:54.286 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051217+0000", 2024-09-21T15:50:54.286 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5989999999999996e-06 2024-09-21T15:50:54.287 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.287 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.287 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:54.287 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108224+0000", 2024-09-21T15:50:54.287 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057007622000000001 2024-09-21T15:50:54.287 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.287 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.287 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:54.287 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108299+0000", 2024-09-21T15:50:54.287 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5099999999999996e-05 2024-09-21T15:50:54.287 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.287 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.287 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:54.287 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108368+0000", 2024-09-21T15:50:54.287 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8918000000000007e-05 2024-09-21T15:50:54.287 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:54.288 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:54.288 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:54.288 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.288 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.288 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116338 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 610304~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:54.288 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.051583+0000", 2024-09-21T15:50:54.288 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 58.151860575999997, 2024-09-21T15:50:54.288 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056967716000000002, 2024-09-21T15:50:54.288 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:54.288 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:54.288 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:54.288 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:54.288 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:54.288 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116338 2024-09-21T15:50:54.288 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.289 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:54.289 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.289 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:54.289 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051583+0000", 2024-09-21T15:50:54.289 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:54.289 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.289 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.289 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:54.289 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051583+0000", 2024-09-21T15:50:54.289 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:54.289 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.289 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.289 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:54.289 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051223+0000", 2024-09-21T15:50:54.289 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.99964 2024-09-21T15:50:54.289 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.290 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.290 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:54.290 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051587+0000", 2024-09-21T15:50:54.290 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00036352999999999998 2024-09-21T15:50:54.290 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.290 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.290 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:54.290 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051597+0000", 2024-09-21T15:50:54.290 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.893e-06 2024-09-21T15:50:54.290 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.290 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.290 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:54.290 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051603+0000", 2024-09-21T15:50:54.290 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0519999999999998e-06 2024-09-21T15:50:54.291 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.291 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.291 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:54.291 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108394+0000", 2024-09-21T15:50:54.291 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056791531999999999 2024-09-21T15:50:54.291 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.291 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.291 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:54.291 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108480+0000", 2024-09-21T15:50:54.291 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.577e-05 2024-09-21T15:50:54.291 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:54.291 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:54.291 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:54.291 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108551+0000", 2024-09-21T15:50:54.292 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0722000000000005e-05 2024-09-21T15:50:54.292 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:54.292 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:54.292 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:54.292 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:54.292 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:54.292 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:54.293 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:50:54.293 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:54.292+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:54.394 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:54.393+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:54.404 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:54.404 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:54.404 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:50:54.404 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:54.413 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:50:54.497 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:54.496+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:54.557 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:54.557 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:54.557 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:50:54.557 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:50:54.557 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:54.566 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:50:54.607 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:54.606+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:54.688 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is set 2024-09-21T15:50:54.696 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:54.695+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:54.713 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:54.714 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:50:54.714 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:50:54.714 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:50:54.714 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.714 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155031 3.1c 3.8612741c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:54.714 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448748+0000", 2024-09-21T15:50:54.714 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.263376794999999, 2024-09-21T15:50:54.714 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064010286999999999, 2024-09-21T15:50:54.714 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:54.714 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:54.714 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:54.714 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:54.714 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:54.715 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155031 2024-09-21T15:50:54.715 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.715 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:54.715 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.715 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:54.715 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448748+0000", 2024-09-21T15:50:54.715 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:54.715 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.715 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.715 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:54.715 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448748+0000", 2024-09-21T15:50:54.715 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:54.715 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.716 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.716 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:54.716 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448747+0000", 2024-09-21T15:50:54.716 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:54.716 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.716 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.716 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:54.716 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448753+0000", 2024-09-21T15:50:54.716 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.755e-06 2024-09-21T15:50:54.716 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.716 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.716 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:54.716 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448763+0000", 2024-09-21T15:50:54.717 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7659999999999993e-06 2024-09-21T15:50:54.717 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.717 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.717 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:54.717 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448769+0000", 2024-09-21T15:50:54.717 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6060000000000002e-06 2024-09-21T15:50:54.717 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.717 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.717 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:54.717 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512750+0000", 2024-09-21T15:50:54.717 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.063981298000000006 2024-09-21T15:50:54.717 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.717 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.717 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:54.717 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512758+0000", 2024-09-21T15:50:54.717 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1669999999999999e-06 2024-09-21T15:50:54.718 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:54.718 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:54.718 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:54.718 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.718 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.718 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155091 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:54.718 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448916+0000", 2024-09-21T15:50:54.718 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.263208949999999, 2024-09-21T15:50:54.718 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064006309999999997, 2024-09-21T15:50:54.718 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:54.718 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:54.718 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:54.718 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:54.718 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:54.718 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155091 2024-09-21T15:50:54.719 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.719 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:54.719 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.719 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:54.719 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448916+0000", 2024-09-21T15:50:54.719 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:54.719 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.719 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.719 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:54.719 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448916+0000", 2024-09-21T15:50:54.719 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:54.719 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.719 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.719 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:54.719 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448915+0000", 2024-09-21T15:50:54.720 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:54.720 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.720 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.720 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:54.720 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448917+0000", 2024-09-21T15:50:54.720 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2400000000000002e-06 2024-09-21T15:50:54.720 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.720 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.720 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:54.720 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448923+0000", 2024-09-21T15:50:54.720 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0580000000000003e-06 2024-09-21T15:50:54.720 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.720 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.720 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:54.720 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448927+0000", 2024-09-21T15:50:54.721 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.031e-06 2024-09-21T15:50:54.721 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.721 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.721 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:54.721 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512916+0000", 2024-09-21T15:50:54.721 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.063988456999999999 2024-09-21T15:50:54.721 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.721 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.721 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:54.721 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512922+0000", 2024-09-21T15:50:54.721 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9009999999999999e-06 2024-09-21T15:50:54.721 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:54.721 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:54.721 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:54.721 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.722 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.722 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155092 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:54.722 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448931+0000", 2024-09-21T15:50:54.722 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.263193168000001, 2024-09-21T15:50:54.722 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064022793999999994, 2024-09-21T15:50:54.722 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:54.722 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:54.722 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:54.722 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:54.722 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:54.722 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155092 2024-09-21T15:50:54.722 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.722 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:54.722 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.722 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:54.723 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448931+0000", 2024-09-21T15:50:54.723 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:54.723 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.723 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.723 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:54.723 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448931+0000", 2024-09-21T15:50:54.723 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:54.723 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.723 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.723 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:54.723 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448931+0000", 2024-09-21T15:50:54.723 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:54.723 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.723 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.723 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:54.723 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448933+0000", 2024-09-21T15:50:54.724 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1799999999999999e-06 2024-09-21T15:50:54.724 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.724 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.724 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:54.724 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448939+0000", 2024-09-21T15:50:54.724 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.604e-06 2024-09-21T15:50:54.724 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.724 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.724 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:54.724 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448943+0000", 2024-09-21T15:50:54.724 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0740000000000003e-06 2024-09-21T15:50:54.724 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.724 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.724 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:54.724 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512946+0000", 2024-09-21T15:50:54.725 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064003612000000001 2024-09-21T15:50:54.725 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.725 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.725 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:54.725 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512954+0000", 2024-09-21T15:50:54.725 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6690000000000006e-06 2024-09-21T15:50:54.725 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:54.725 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:54.725 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:54.725 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.725 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.725 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155093 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:54.725 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448947+0000", 2024-09-21T15:50:54.725 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.263177794000001, 2024-09-21T15:50:54.726 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064020717000000005, 2024-09-21T15:50:54.726 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:54.726 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:54.726 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:54.726 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:54.726 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:54.726 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155093 2024-09-21T15:50:54.726 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.726 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:54.726 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.726 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:54.726 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448947+0000", 2024-09-21T15:50:54.726 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:54.726 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.726 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.726 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:54.727 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448947+0000", 2024-09-21T15:50:54.727 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:54.727 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.727 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.727 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:54.727 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448946+0000", 2024-09-21T15:50:54.727 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:54.727 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.727 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.727 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:54.727 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448949+0000", 2024-09-21T15:50:54.727 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3860000000000001e-06 2024-09-21T15:50:54.727 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.727 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.727 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:54.727 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448955+0000", 2024-09-21T15:50:54.728 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.823e-06 2024-09-21T15:50:54.728 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.728 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.728 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:54.728 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448959+0000", 2024-09-21T15:50:54.728 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2799999999999997e-06 2024-09-21T15:50:54.728 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.728 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.728 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:54.728 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512964+0000", 2024-09-21T15:50:54.728 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064004741000000004 2024-09-21T15:50:54.728 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.728 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.728 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:54.728 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512967+0000", 2024-09-21T15:50:54.729 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.839e-06 2024-09-21T15:50:54.729 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:54.729 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:54.729 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:54.729 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.729 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.729 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155094 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:54.729 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448962+0000", 2024-09-21T15:50:54.729 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.263162170999998, 2024-09-21T15:50:54.729 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064037378000000006, 2024-09-21T15:50:54.729 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:54.729 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:54.729 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:54.729 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:54.730 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:54.730 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155094 2024-09-21T15:50:54.730 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.730 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:54.730 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.730 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:54.730 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448962+0000", 2024-09-21T15:50:54.730 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:54.730 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.730 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.730 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:54.730 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448962+0000", 2024-09-21T15:50:54.730 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:54.730 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.731 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.731 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:54.731 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448962+0000", 2024-09-21T15:50:54.731 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:54.731 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.731 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.731 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:54.731 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448968+0000", 2024-09-21T15:50:54.731 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9710000000000001e-06 2024-09-21T15:50:54.731 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.731 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.731 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:54.731 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448974+0000", 2024-09-21T15:50:54.731 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7230000000000001e-06 2024-09-21T15:50:54.731 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.731 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.732 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:54.732 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448978+0000", 2024-09-21T15:50:54.732 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3749999999999996e-06 2024-09-21T15:50:54.732 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.732 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.732 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:54.732 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512992+0000", 2024-09-21T15:50:54.732 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064013525000000002 2024-09-21T15:50:54.732 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.732 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.732 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:54.732 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513000+0000", 2024-09-21T15:50:54.732 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0830000000000004e-06 2024-09-21T15:50:54.732 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:54.732 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:54.732 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:54.733 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.733 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.733 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155095 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:54.733 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448982+0000", 2024-09-21T15:50:54.733 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.263142889000001, 2024-09-21T15:50:54.733 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064041904999999996, 2024-09-21T15:50:54.733 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:54.733 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:54.733 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:54.733 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:54.733 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:54.733 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155095 2024-09-21T15:50:54.733 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.733 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:54.733 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.734 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:54.734 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448982+0000", 2024-09-21T15:50:54.734 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:54.734 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.734 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.734 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:54.734 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448982+0000", 2024-09-21T15:50:54.734 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:54.734 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.734 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.734 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:54.734 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448981+0000", 2024-09-21T15:50:54.734 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:54.735 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.735 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.735 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:54.735 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448983+0000", 2024-09-21T15:50:54.735 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.905e-06 2024-09-21T15:50:54.735 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.735 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.735 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:54.735 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448989+0000", 2024-09-21T15:50:54.735 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.536e-06 2024-09-21T15:50:54.735 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.735 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.735 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:54.735 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448993+0000", 2024-09-21T15:50:54.736 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2390000000000004e-06 2024-09-21T15:50:54.736 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.736 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.736 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:54.736 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513015+0000", 2024-09-21T15:50:54.736 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064022081999999994 2024-09-21T15:50:54.736 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.736 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.736 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:54.736 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513023+0000", 2024-09-21T15:50:54.736 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4500000000000004e-06 2024-09-21T15:50:54.736 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:54.736 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:54.736 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:54.736 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.736 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.736 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155096 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:54.737 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448996+0000", 2024-09-21T15:50:54.737 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.263127988000001, 2024-09-21T15:50:54.737 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064060868000000007, 2024-09-21T15:50:54.737 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:54.737 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:54.737 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:54.737 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:54.737 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:54.737 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155096 2024-09-21T15:50:54.737 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.737 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:54.737 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.737 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:54.737 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448996+0000", 2024-09-21T15:50:54.737 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:54.737 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.738 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.738 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:54.738 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448996+0000", 2024-09-21T15:50:54.738 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:54.738 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.738 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.738 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:54.738 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448996+0000", 2024-09-21T15:50:54.738 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:54.738 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.738 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.738 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:54.738 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448998+0000", 2024-09-21T15:50:54.738 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8780000000000001e-06 2024-09-21T15:50:54.739 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.739 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.739 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:54.739 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449003+0000", 2024-09-21T15:50:54.739 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4099999999999999e-06 2024-09-21T15:50:54.739 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.739 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.739 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:54.739 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449008+0000", 2024-09-21T15:50:54.739 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.335e-06 2024-09-21T15:50:54.739 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.739 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.739 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:54.739 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513049+0000", 2024-09-21T15:50:54.739 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064041059999999997 2024-09-21T15:50:54.740 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.740 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.740 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:54.740 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513057+0000", 2024-09-21T15:50:54.740 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5050000000000007e-06 2024-09-21T15:50:54.740 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:54.740 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:54.740 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:54.740 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.740 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.740 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155097 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:54.740 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449019+0000", 2024-09-21T15:50:54.740 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.263105566999997, 2024-09-21T15:50:54.740 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064052488000000005, 2024-09-21T15:50:54.740 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:54.740 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:54.741 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:54.741 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:54.741 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:54.741 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155097 2024-09-21T15:50:54.741 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.741 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:54.741 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.741 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:54.741 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449019+0000", 2024-09-21T15:50:54.741 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:54.741 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.741 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.741 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:54.741 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449019+0000", 2024-09-21T15:50:54.742 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:54.742 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.742 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.742 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:54.742 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449019+0000", 2024-09-21T15:50:54.742 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:54.742 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.742 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.742 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:54.742 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449021+0000", 2024-09-21T15:50:54.742 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.244e-06 2024-09-21T15:50:54.742 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.742 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.743 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:54.743 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449027+0000", 2024-09-21T15:50:54.743 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8619999999999999e-06 2024-09-21T15:50:54.743 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.743 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.743 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:54.743 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449031+0000", 2024-09-21T15:50:54.743 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1359999999999999e-06 2024-09-21T15:50:54.743 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.743 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.743 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:54.743 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513067+0000", 2024-09-21T15:50:54.743 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064036391999999998 2024-09-21T15:50:54.744 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.744 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.744 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:54.744 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513071+0000", 2024-09-21T15:50:54.744 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1989999999999999e-06 2024-09-21T15:50:54.744 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:54.744 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:54.744 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:54.744 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.744 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.744 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155098 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:54.744 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449035+0000", 2024-09-21T15:50:54.744 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.263089970000003, 2024-09-21T15:50:54.744 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064071761000000005, 2024-09-21T15:50:54.745 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:54.745 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:54.745 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:54.745 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:54.745 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:54.745 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155098 2024-09-21T15:50:54.745 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.745 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:54.745 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.745 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:54.745 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449035+0000", 2024-09-21T15:50:54.745 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:54.745 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.745 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.745 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:54.746 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449035+0000", 2024-09-21T15:50:54.746 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:54.746 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.746 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.746 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:54.746 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449034+0000", 2024-09-21T15:50:54.746 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:54.746 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.746 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.746 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:54.746 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449036+0000", 2024-09-21T15:50:54.746 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9989999999999998e-06 2024-09-21T15:50:54.746 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.746 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.746 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:54.747 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449042+0000", 2024-09-21T15:50:54.747 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6010000000000002e-06 2024-09-21T15:50:54.747 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.747 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.747 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:54.747 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449047+0000", 2024-09-21T15:50:54.747 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3600000000000004e-06 2024-09-21T15:50:54.747 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.747 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.747 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:54.747 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513098+0000", 2024-09-21T15:50:54.747 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064050544000000001 2024-09-21T15:50:54.747 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.747 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.747 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:54.748 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513106+0000", 2024-09-21T15:50:54.748 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5650000000000002e-06 2024-09-21T15:50:54.748 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:54.748 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:54.748 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:54.748 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.748 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.748 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155099 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:54.748 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449051+0000", 2024-09-21T15:50:54.748 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.263073429000002, 2024-09-21T15:50:54.748 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064069388000000005, 2024-09-21T15:50:54.748 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:54.748 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:54.748 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:54.749 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:54.749 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:54.749 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155099 2024-09-21T15:50:54.749 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.749 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:54.749 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.749 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:54.749 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449051+0000", 2024-09-21T15:50:54.749 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:54.749 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.749 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.749 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:54.749 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449051+0000", 2024-09-21T15:50:54.749 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:54.749 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.750 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.750 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:54.750 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449051+0000", 2024-09-21T15:50:54.750 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:54.750 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.750 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.750 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:54.750 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449053+0000", 2024-09-21T15:50:54.750 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2620000000000001e-06 2024-09-21T15:50:54.750 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.750 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.750 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:54.750 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449058+0000", 2024-09-21T15:50:54.750 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1680000000000003e-06 2024-09-21T15:50:54.751 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.751 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.751 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:54.751 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449062+0000", 2024-09-21T15:50:54.751 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9979999999999997e-06 2024-09-21T15:50:54.751 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.751 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.751 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:54.751 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513116+0000", 2024-09-21T15:50:54.751 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064053761000000001 2024-09-21T15:50:54.751 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.751 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.751 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:54.752 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513120+0000", 2024-09-21T15:50:54.752 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5399999999999997e-06 2024-09-21T15:50:54.752 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:54.752 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:54.752 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:54.752 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.752 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.752 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155100 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:54.752 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449101+0000", 2024-09-21T15:50:54.752 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.263023674000003, 2024-09-21T15:50:54.752 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064079585999999994, 2024-09-21T15:50:54.752 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:54.752 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:54.752 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:54.753 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:54.753 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:54.753 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155100 2024-09-21T15:50:54.753 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.753 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:54.753 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.753 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:54.753 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449101+0000", 2024-09-21T15:50:54.753 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:54.753 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.753 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.753 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:54.753 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449101+0000", 2024-09-21T15:50:54.753 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:54.753 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.754 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.754 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:54.754 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449100+0000", 2024-09-21T15:50:54.754 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-21T15:50:54.754 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.754 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.754 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:54.754 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449105+0000", 2024-09-21T15:50:54.754 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6700000000000002e-06 2024-09-21T15:50:54.754 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.754 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.754 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:54.754 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449117+0000", 2024-09-21T15:50:54.755 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1841000000000001e-05 2024-09-21T15:50:54.755 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.755 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.755 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:54.755 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449124+0000", 2024-09-21T15:50:54.755 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4660000000000002e-06 2024-09-21T15:50:54.755 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.755 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.755 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:54.755 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513163+0000", 2024-09-21T15:50:54.755 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064039053999999998 2024-09-21T15:50:54.755 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.755 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.755 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:54.756 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513180+0000", 2024-09-21T15:50:54.756 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7333000000000002e-05 2024-09-21T15:50:54.756 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:54.756 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:54.756 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:54.756 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.756 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.756 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155101 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:54.756 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449166+0000", 2024-09-21T15:50:54.756 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.262958969000003, 2024-09-21T15:50:54.756 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064027639999999997, 2024-09-21T15:50:54.756 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:54.756 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:54.756 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:54.757 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:54.757 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:54.757 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155101 2024-09-21T15:50:54.757 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.757 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:54.757 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.757 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:54.757 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449166+0000", 2024-09-21T15:50:54.757 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:54.757 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.757 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.757 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:54.757 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449166+0000", 2024-09-21T15:50:54.757 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:54.757 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.758 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.758 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:54.758 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449165+0000", 2024-09-21T15:50:54.758 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:54.758 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.758 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.758 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:54.758 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449168+0000", 2024-09-21T15:50:54.758 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5099999999999999e-06 2024-09-21T15:50:54.758 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.758 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.758 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:54.758 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449177+0000", 2024-09-21T15:50:54.758 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6659999999999997e-06 2024-09-21T15:50:54.759 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.759 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.759 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:54.759 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449182+0000", 2024-09-21T15:50:54.759 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1320000000000002e-06 2024-09-21T15:50:54.759 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.759 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.759 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:54.759 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513189+0000", 2024-09-21T15:50:54.759 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064007231999999997 2024-09-21T15:50:54.759 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.759 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.759 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:54.759 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513193+0000", 2024-09-21T15:50:54.759 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.692e-06 2024-09-21T15:50:54.760 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:54.760 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:54.760 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:54.760 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.760 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.760 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155134 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:54.760 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449187+0000", 2024-09-21T15:50:54.760 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.262937381, 2024-09-21T15:50:54.760 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064031331999999996, 2024-09-21T15:50:54.760 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:54.760 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:54.760 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:54.760 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:54.760 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:54.761 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155134 2024-09-21T15:50:54.761 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.761 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:54.761 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.761 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:54.761 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449187+0000", 2024-09-21T15:50:54.761 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:54.761 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.761 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.761 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:54.761 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449187+0000", 2024-09-21T15:50:54.761 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:54.761 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.761 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.761 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:54.762 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449187+0000", 2024-09-21T15:50:54.762 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:54.762 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.762 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.762 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:54.762 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449189+0000", 2024-09-21T15:50:54.762 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5469999999999999e-06 2024-09-21T15:50:54.762 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.762 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.762 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:54.762 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449196+0000", 2024-09-21T15:50:54.762 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7009999999999999e-06 2024-09-21T15:50:54.762 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.762 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.763 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:54.763 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449202+0000", 2024-09-21T15:50:54.763 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7409999999999998e-06 2024-09-21T15:50:54.763 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.763 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.763 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:54.763 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513214+0000", 2024-09-21T15:50:54.763 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064012301999999993 2024-09-21T15:50:54.763 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.763 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.763 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:54.763 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513218+0000", 2024-09-21T15:50:54.763 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5709999999999999e-06 2024-09-21T15:50:54.764 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:54.764 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:54.764 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:54.764 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.764 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.764 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155135 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:54.764 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449207+0000", 2024-09-21T15:50:54.764 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.262917870999999, 2024-09-21T15:50:54.764 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064029249999999996, 2024-09-21T15:50:54.764 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:54.764 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:54.764 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:54.764 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:54.764 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:54.764 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155135 2024-09-21T15:50:54.764 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.765 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:54.765 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.765 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:54.765 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449207+0000", 2024-09-21T15:50:54.765 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:54.765 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.765 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.765 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:54.765 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449207+0000", 2024-09-21T15:50:54.765 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:54.765 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.765 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.765 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:54.765 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449206+0000", 2024-09-21T15:50:54.765 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:54.765 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.766 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.766 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:54.766 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449209+0000", 2024-09-21T15:50:54.766 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5150000000000001e-06 2024-09-21T15:50:54.766 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.766 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.766 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:54.766 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449215+0000", 2024-09-21T15:50:54.766 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9329999999999998e-06 2024-09-21T15:50:54.766 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.766 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.766 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:54.766 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449220+0000", 2024-09-21T15:50:54.766 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0819999999999998e-06 2024-09-21T15:50:54.767 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.767 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.767 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:54.767 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513232+0000", 2024-09-21T15:50:54.767 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064012052 2024-09-21T15:50:54.767 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.767 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.767 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:54.767 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513236+0000", 2024-09-21T15:50:54.767 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2080000000000002e-06 2024-09-21T15:50:54.767 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:54.767 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:54.767 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:54.767 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.767 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.767 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155136 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:54.768 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449224+0000", 2024-09-21T15:50:54.768 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.262900307000002, 2024-09-21T15:50:54.768 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064053025999999999, 2024-09-21T15:50:54.768 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:54.768 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:54.768 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:54.768 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:54.768 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:54.768 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155136 2024-09-21T15:50:54.768 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.768 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:54.768 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.768 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:54.768 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449224+0000", 2024-09-21T15:50:54.768 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:54.769 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.769 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.769 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:54.769 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449224+0000", 2024-09-21T15:50:54.769 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:54.769 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.769 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.769 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:54.769 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449224+0000", 2024-09-21T15:50:54.769 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:54.769 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.769 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.769 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:54.769 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449226+0000", 2024-09-21T15:50:54.769 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2699999999999999e-06 2024-09-21T15:50:54.769 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.770 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.770 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:54.770 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449233+0000", 2024-09-21T15:50:54.770 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4289999999999998e-06 2024-09-21T15:50:54.770 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.770 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.770 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:54.770 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449238+0000", 2024-09-21T15:50:54.770 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0730000000000004e-06 2024-09-21T15:50:54.770 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.770 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.770 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:54.770 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513269+0000", 2024-09-21T15:50:54.770 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064030999000000005 2024-09-21T15:50:54.770 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.771 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.771 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:54.771 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513277+0000", 2024-09-21T15:50:54.771 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5520000000000005e-06 2024-09-21T15:50:54.771 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:54.771 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:54.771 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:54.771 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.771 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.771 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155137 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:54.771 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449263+0000", 2024-09-21T15:50:54.771 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.262861174999998, 2024-09-21T15:50:54.771 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.06402803, 2024-09-21T15:50:54.771 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:54.771 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:54.771 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:54.772 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:54.772 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:54.772 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155137 2024-09-21T15:50:54.772 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.772 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:54.772 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.772 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:54.772 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449263+0000", 2024-09-21T15:50:54.772 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:54.772 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.772 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.772 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:54.772 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449263+0000", 2024-09-21T15:50:54.772 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:54.772 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.772 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.773 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:54.773 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449262+0000", 2024-09-21T15:50:54.773 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-21T15:50:54.773 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.773 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.773 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:54.773 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449267+0000", 2024-09-21T15:50:54.773 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.933e-06 2024-09-21T15:50:54.773 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.773 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.773 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:54.773 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449279+0000", 2024-09-21T15:50:54.773 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.181e-05 2024-09-21T15:50:54.773 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.773 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.773 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:54.774 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449286+0000", 2024-09-21T15:50:54.774 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4749999999999998e-06 2024-09-21T15:50:54.774 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.774 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.774 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:54.774 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513287+0000", 2024-09-21T15:50:54.774 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064001369000000002 2024-09-21T15:50:54.774 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.774 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.774 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:54.774 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513291+0000", 2024-09-21T15:50:54.774 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3000000000000003e-06 2024-09-21T15:50:54.774 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:54.774 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:54.774 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:54.775 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.775 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.775 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155138 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:54.775 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449299+0000", 2024-09-21T15:50:54.775 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.262825683000003, 2024-09-21T15:50:54.775 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064028359000000007, 2024-09-21T15:50:54.775 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:54.775 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:54.775 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:54.775 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:54.775 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:54.775 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155138 2024-09-21T15:50:54.775 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.775 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:54.776 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.776 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:54.776 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449299+0000", 2024-09-21T15:50:54.776 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:54.776 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.776 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.776 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:54.776 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449299+0000", 2024-09-21T15:50:54.776 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:54.776 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.776 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.776 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:54.776 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449298+0000", 2024-09-21T15:50:54.776 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:54.776 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.776 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.777 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:54.777 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449301+0000", 2024-09-21T15:50:54.777 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0230000000000001e-06 2024-09-21T15:50:54.777 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.777 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.777 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:54.777 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449308+0000", 2024-09-21T15:50:54.777 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3119999999999999e-06 2024-09-21T15:50:54.777 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.777 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.777 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:54.777 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449313+0000", 2024-09-21T15:50:54.777 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8980000000000004e-06 2024-09-21T15:50:54.777 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.777 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.778 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:54.778 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513318+0000", 2024-09-21T15:50:54.778 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064005612000000003 2024-09-21T15:50:54.778 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.778 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.778 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:54.778 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513327+0000", 2024-09-21T15:50:54.778 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0100000000000001e-06 2024-09-21T15:50:54.778 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:54.778 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:54.778 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:54.778 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.778 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.778 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155142 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:54.778 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449392+0000", 2024-09-21T15:50:54.779 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.262732925000002, 2024-09-21T15:50:54.779 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064012464000000005, 2024-09-21T15:50:54.779 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:54.779 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:54.779 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:54.779 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:54.779 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:54.779 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155142 2024-09-21T15:50:54.779 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.779 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:54.779 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.779 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:54.779 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449392+0000", 2024-09-21T15:50:54.779 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:54.779 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.779 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.779 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:54.780 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449392+0000", 2024-09-21T15:50:54.780 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:54.780 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.780 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.780 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:54.780 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449391+0000", 2024-09-21T15:50:54.780 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:54.780 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.780 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.780 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:54.780 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449393+0000", 2024-09-21T15:50:54.780 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.26e-06 2024-09-21T15:50:54.780 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.780 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.780 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:54.780 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449399+0000", 2024-09-21T15:50:54.781 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6269999999999997e-06 2024-09-21T15:50:54.781 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.781 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.781 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:54.781 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449403+0000", 2024-09-21T15:50:54.781 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3000000000000003e-06 2024-09-21T15:50:54.781 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.781 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.781 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:54.781 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513396+0000", 2024-09-21T15:50:54.781 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.063992568999999999 2024-09-21T15:50:54.781 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.781 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.782 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:54.782 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513404+0000", 2024-09-21T15:50:54.782 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1100000000000003e-06 2024-09-21T15:50:54.782 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:54.782 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:54.782 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:54.782 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.782 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.782 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155143 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:54.782 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449407+0000", 2024-09-21T15:50:54.782 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.262717338999998, 2024-09-21T15:50:54.782 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064009916, 2024-09-21T15:50:54.782 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:54.782 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:54.782 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:54.783 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:54.783 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:54.783 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155143 2024-09-21T15:50:54.783 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.783 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:54.783 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.783 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:54.783 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449407+0000", 2024-09-21T15:50:54.783 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:54.783 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.783 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.783 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:54.783 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449407+0000", 2024-09-21T15:50:54.783 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:54.784 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.784 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.784 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:54.784 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449407+0000", 2024-09-21T15:50:54.784 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:54.784 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.784 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.784 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:54.784 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449409+0000", 2024-09-21T15:50:54.784 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1090000000000001e-06 2024-09-21T15:50:54.784 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.784 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.784 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:54.784 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449413+0000", 2024-09-21T15:50:54.784 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6650000000000001e-06 2024-09-21T15:50:54.785 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.785 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.785 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:54.785 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449415+0000", 2024-09-21T15:50:54.785 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5459999999999998e-06 2024-09-21T15:50:54.785 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.785 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.785 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:54.785 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513413+0000", 2024-09-21T15:50:54.785 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.063998037999999993 2024-09-21T15:50:54.785 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.785 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.785 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:54.785 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513417+0000", 2024-09-21T15:50:54.785 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8789999999999996e-06 2024-09-21T15:50:54.785 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:54.786 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:54.786 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:54.786 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.786 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.786 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155144 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:54.786 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449427+0000", 2024-09-21T15:50:54.786 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 32.262697330999998, 2024-09-21T15:50:54.786 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064022531999999993, 2024-09-21T15:50:54.786 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:54.786 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:54.786 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:54.786 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:54.786 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:54.787 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155144 2024-09-21T15:50:54.787 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.787 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:54.787 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.787 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:54.787 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449427+0000", 2024-09-21T15:50:54.787 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:54.787 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.787 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.787 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:54.787 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449427+0000", 2024-09-21T15:50:54.787 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:54.787 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.787 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.787 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:54.787 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449427+0000", 2024-09-21T15:50:54.788 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:54.788 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.788 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.788 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:54.788 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449430+0000", 2024-09-21T15:50:54.788 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3579999999999999e-06 2024-09-21T15:50:54.788 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.788 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.788 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:54.788 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449437+0000", 2024-09-21T15:50:54.788 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4580000000000004e-06 2024-09-21T15:50:54.788 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.788 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.788 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:54.788 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449443+0000", 2024-09-21T15:50:54.788 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5609999999999998e-06 2024-09-21T15:50:54.789 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.789 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.789 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:54.789 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513441+0000", 2024-09-21T15:50:54.789 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.063998136999999997 2024-09-21T15:50:54.789 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:54.789 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:54.789 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:54.789 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513450+0000", 2024-09-21T15:50:54.789 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4950000000000008e-06 2024-09-21T15:50:54.789 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:54.789 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:54.789 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:54.789 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:54.789 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:54.790 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:54.790 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:50:54.797 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:54.796+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:54.893 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:54.893 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:54.893 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:50:54.893 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:54.898 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:54.897+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:54.901 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:50:54.998 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:54.997+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:55.043 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:55.043 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:55.043 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:50:55.043 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:50:55.043 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:55.051 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:50:55.100 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:50:55.099+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:50:55.108 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:50:55.195 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:55.195 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:50:55.195 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:50:55.195 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:50:55.195 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.195 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120111 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 321536~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:55.195 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572216+0000", 2024-09-21T15:50:55.195 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 51.620896322999997, 2024-09-21T15:50:55.195 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054657399000000002, 2024-09-21T15:50:55.195 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:55.196 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:55.196 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:55.196 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:55.196 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:55.196 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120111 2024-09-21T15:50:55.196 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.196 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:55.196 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.196 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:55.196 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572216+0000", 2024-09-21T15:50:55.196 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:55.196 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.196 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.196 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:55.197 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572216+0000", 2024-09-21T15:50:55.197 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:55.197 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.197 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.197 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:55.197 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572216+0000", 2024-09-21T15:50:55.197 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:55.197 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.197 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.197 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:55.197 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572218+0000", 2024-09-21T15:50:55.197 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1840000000000002e-06 2024-09-21T15:50:55.197 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.197 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.197 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:55.198 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572224+0000", 2024-09-21T15:50:55.198 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7880000000000003e-06 2024-09-21T15:50:55.198 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.198 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.198 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:55.198 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572229+0000", 2024-09-21T15:50:55.198 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8239999999999999e-06 2024-09-21T15:50:55.198 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.198 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.198 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:55.198 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626680+0000", 2024-09-21T15:50:55.198 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054451431000000002 2024-09-21T15:50:55.198 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.199 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.199 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:55.199 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626755+0000", 2024-09-21T15:50:55.199 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4641000000000006e-05 2024-09-21T15:50:55.199 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.199 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.199 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:55.199 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626874+0000", 2024-09-21T15:50:55.199 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00011886 2024-09-21T15:50:55.199 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:55.199 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:55.199 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:55.199 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.199 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.199 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120112 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 342016~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:55.200 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572233+0000", 2024-09-21T15:50:55.200 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 51.620879954999999, 2024-09-21T15:50:55.200 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054823997999999999, 2024-09-21T15:50:55.200 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:55.200 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:55.200 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:55.200 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:55.200 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:55.200 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120112 2024-09-21T15:50:55.200 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.200 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:55.200 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.200 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:55.200 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572233+0000", 2024-09-21T15:50:55.200 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:55.201 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.201 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.201 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:55.201 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572233+0000", 2024-09-21T15:50:55.201 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:55.201 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.201 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.201 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:55.201 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572232+0000", 2024-09-21T15:50:55.201 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:55.201 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.201 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.201 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:55.201 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572234+0000", 2024-09-21T15:50:55.201 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0660000000000002e-06 2024-09-21T15:50:55.201 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.202 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.202 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:55.202 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572242+0000", 2024-09-21T15:50:55.202 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0430000000000002e-06 2024-09-21T15:50:55.202 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.202 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.202 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:55.202 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572246+0000", 2024-09-21T15:50:55.202 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9080000000000003e-06 2024-09-21T15:50:55.202 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.202 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.202 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:55.202 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626916+0000", 2024-09-21T15:50:55.202 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054669482999999998 2024-09-21T15:50:55.203 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.203 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.203 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:55.203 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626977+0000", 2024-09-21T15:50:55.203 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0597000000000003e-05 2024-09-21T15:50:55.203 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.203 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.203 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:55.203 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627057+0000", 2024-09-21T15:50:55.203 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0258000000000003e-05 2024-09-21T15:50:55.203 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:55.203 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:55.203 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:55.203 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.203 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.204 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120113 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 362496~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:55.204 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572253+0000", 2024-09-21T15:50:55.204 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 51.620859969999998, 2024-09-21T15:50:55.204 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.05503044, 2024-09-21T15:50:55.204 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:55.204 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:55.204 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:55.204 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:55.204 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:55.204 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120113 2024-09-21T15:50:55.204 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.204 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:55.204 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.204 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:55.204 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572253+0000", 2024-09-21T15:50:55.204 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:55.205 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.205 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.205 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:55.205 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572253+0000", 2024-09-21T15:50:55.205 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:55.205 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.205 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.205 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:55.205 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572252+0000", 2024-09-21T15:50:55.205 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:55.205 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.205 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.205 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:55.205 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572258+0000", 2024-09-21T15:50:55.206 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6169999999999998e-06 2024-09-21T15:50:55.206 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.206 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.206 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:55.206 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572266+0000", 2024-09-21T15:50:55.206 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9410000000000007e-06 2024-09-21T15:50:55.206 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.206 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.206 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:55.206 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572272+0000", 2024-09-21T15:50:55.206 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.241e-06 2024-09-21T15:50:55.206 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.206 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.206 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:55.206 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627087+0000", 2024-09-21T15:50:55.207 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054814850999999998 2024-09-21T15:50:55.207 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.207 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.207 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:55.207 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627162+0000", 2024-09-21T15:50:55.207 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5012999999999995e-05 2024-09-21T15:50:55.207 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.207 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.207 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:55.207 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627283+0000", 2024-09-21T15:50:55.207 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012113 2024-09-21T15:50:55.207 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:55.207 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:55.207 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:55.207 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.207 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.208 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120114 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 382976~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:55.208 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572277+0000", 2024-09-21T15:50:55.208 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 51.620835952, 2024-09-21T15:50:55.208 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055184395999999997, 2024-09-21T15:50:55.208 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:55.208 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:55.208 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:55.208 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:55.208 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:55.208 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120114 2024-09-21T15:50:55.208 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.208 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:55.208 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.208 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:55.208 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572277+0000", 2024-09-21T15:50:55.208 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:55.209 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.209 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.209 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:55.209 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572277+0000", 2024-09-21T15:50:55.209 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:55.209 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.209 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.209 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:55.209 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572276+0000", 2024-09-21T15:50:55.209 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:55.209 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.209 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.209 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:55.209 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572279+0000", 2024-09-21T15:50:55.209 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.373e-06 2024-09-21T15:50:55.210 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.210 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.210 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:55.210 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572285+0000", 2024-09-21T15:50:55.210 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9359999999999996e-06 2024-09-21T15:50:55.210 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.210 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.210 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:55.210 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572289+0000", 2024-09-21T15:50:55.210 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.634e-06 2024-09-21T15:50:55.210 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.210 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.210 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:55.211 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627311+0000", 2024-09-21T15:50:55.211 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055021211 2024-09-21T15:50:55.211 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.211 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.211 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:55.211 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627360+0000", 2024-09-21T15:50:55.211 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9203999999999998e-05 2024-09-21T15:50:55.211 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.211 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.211 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:55.211 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627461+0000", 2024-09-21T15:50:55.211 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000101469 2024-09-21T15:50:55.211 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:55.211 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:55.211 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:55.211 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.212 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.212 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120115 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 399360~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:55.212 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572294+0000", 2024-09-21T15:50:55.212 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 51.620818645, 2024-09-21T15:50:55.212 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055400092999999997, 2024-09-21T15:50:55.212 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:55.212 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:55.212 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:55.212 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:55.212 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:55.212 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120115 2024-09-21T15:50:55.212 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.212 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:55.212 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.212 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:55.212 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572294+0000", 2024-09-21T15:50:55.213 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:55.213 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.213 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.213 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:55.213 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572294+0000", 2024-09-21T15:50:55.213 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:55.213 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.213 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.213 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:55.213 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572294+0000", 2024-09-21T15:50:55.213 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:55.213 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.213 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.213 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:55.213 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572296+0000", 2024-09-21T15:50:55.213 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1100000000000001e-06 2024-09-21T15:50:55.214 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.214 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.214 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:55.214 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572303+0000", 2024-09-21T15:50:55.214 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8580000000000002e-06 2024-09-21T15:50:55.214 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.214 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.214 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:55.214 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572314+0000", 2024-09-21T15:50:55.214 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.101e-05 2024-09-21T15:50:55.214 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.214 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.214 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:55.214 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627496+0000", 2024-09-21T15:50:55.214 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055182522999999997 2024-09-21T15:50:55.215 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.215 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.215 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:55.215 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627568+0000", 2024-09-21T15:50:55.215 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1569999999999994e-05 2024-09-21T15:50:55.215 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.215 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.215 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:55.215 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627694+0000", 2024-09-21T15:50:55.215 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000126372 2024-09-21T15:50:55.215 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:55.215 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:55.215 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:55.215 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.215 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.215 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120116 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 419840~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:55.216 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572318+0000", 2024-09-21T15:50:55.216 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 51.620794453999999, 2024-09-21T15:50:55.216 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055539416000000001, 2024-09-21T15:50:55.216 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:55.216 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:55.216 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:55.216 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:55.216 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:55.216 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120116 2024-09-21T15:50:55.216 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.216 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:55.216 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.216 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:55.216 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572318+0000", 2024-09-21T15:50:55.216 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:55.217 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.217 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.217 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:55.217 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572318+0000", 2024-09-21T15:50:55.217 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:55.217 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.217 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.217 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:55.217 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572318+0000", 2024-09-21T15:50:55.217 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:55.217 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.217 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.217 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:55.217 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572322+0000", 2024-09-21T15:50:55.218 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4839999999999998e-06 2024-09-21T15:50:55.218 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.218 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.218 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:55.218 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572329+0000", 2024-09-21T15:50:55.218 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0959999999999997e-06 2024-09-21T15:50:55.218 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.218 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.218 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:55.218 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572333+0000", 2024-09-21T15:50:55.218 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9359999999999998e-06 2024-09-21T15:50:55.218 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.218 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.218 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:55.218 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627726+0000", 2024-09-21T15:50:55.219 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055392829999999997 2024-09-21T15:50:55.219 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.219 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.219 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:55.219 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627779+0000", 2024-09-21T15:50:55.219 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2185999999999998e-05 2024-09-21T15:50:55.219 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.219 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.219 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:55.219 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627858+0000", 2024-09-21T15:50:55.219 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9190999999999994e-05 2024-09-21T15:50:55.219 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:55.219 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:55.219 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:55.219 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.219 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.220 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120117 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 440320~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:55.220 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572338+0000", 2024-09-21T15:50:55.220 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 51.620775141000003, 2024-09-21T15:50:55.220 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055680888999999997, 2024-09-21T15:50:55.220 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:55.220 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:55.220 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:55.220 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:55.220 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:55.220 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120117 2024-09-21T15:50:55.220 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.220 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:55.220 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.220 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:55.220 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572338+0000", 2024-09-21T15:50:55.220 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:55.221 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.221 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.221 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:55.221 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572338+0000", 2024-09-21T15:50:55.221 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:55.221 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.221 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.221 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:55.221 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572337+0000", 2024-09-21T15:50:55.221 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:55.221 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.221 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.221 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:55.221 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572339+0000", 2024-09-21T15:50:55.221 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9379999999999999e-06 2024-09-21T15:50:55.221 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.222 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.222 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:55.222 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572349+0000", 2024-09-21T15:50:55.222 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7780000000000002e-06 2024-09-21T15:50:55.222 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.222 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.222 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:55.222 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572354+0000", 2024-09-21T15:50:55.222 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8740000000000003e-06 2024-09-21T15:50:55.222 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.222 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.222 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:55.222 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627885+0000", 2024-09-21T15:50:55.222 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055531448999999997 2024-09-21T15:50:55.222 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.222 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.223 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:55.223 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627935+0000", 2024-09-21T15:50:55.223 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9348999999999998e-05 2024-09-21T15:50:55.223 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.223 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.223 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:55.223 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628018+0000", 2024-09-21T15:50:55.223 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3851999999999996e-05 2024-09-21T15:50:55.223 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:55.223 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:55.223 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:55.223 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.223 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.223 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120118 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 460800~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:55.223 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572359+0000", 2024-09-21T15:50:55.224 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 51.620753899, 2024-09-21T15:50:55.224 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055951714, 2024-09-21T15:50:55.224 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:55.224 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:55.224 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:55.224 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:55.224 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:55.224 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120118 2024-09-21T15:50:55.224 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.224 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:55.224 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.224 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:55.224 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572359+0000", 2024-09-21T15:50:55.224 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:55.224 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.224 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.225 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:55.225 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572359+0000", 2024-09-21T15:50:55.225 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:55.225 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.225 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.225 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:55.225 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572358+0000", 2024-09-21T15:50:55.225 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:55.225 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.225 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.225 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:55.225 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572361+0000", 2024-09-21T15:50:55.225 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.26e-06 2024-09-21T15:50:55.225 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.225 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.225 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:55.226 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572367+0000", 2024-09-21T15:50:55.226 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3640000000000004e-06 2024-09-21T15:50:55.226 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.226 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.226 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:55.226 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572376+0000", 2024-09-21T15:50:55.226 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2310000000000002e-06 2024-09-21T15:50:55.226 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.226 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.226 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:55.226 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628096+0000", 2024-09-21T15:50:55.226 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055719369999999997 2024-09-21T15:50:55.226 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.226 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.226 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:55.227 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628178+0000", 2024-09-21T15:50:55.227 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2667e-05 2024-09-21T15:50:55.227 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.227 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.227 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:55.227 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628311+0000", 2024-09-21T15:50:55.227 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000132175 2024-09-21T15:50:55.227 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:55.227 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:55.227 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:55.227 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.227 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.227 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120119 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 481280~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:55.227 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572381+0000", 2024-09-21T15:50:55.227 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 51.620731454999998, 2024-09-21T15:50:55.227 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056088791999999998, 2024-09-21T15:50:55.228 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:55.228 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:55.228 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:55.228 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:55.228 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:55.228 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120119 2024-09-21T15:50:55.228 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.228 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:55.228 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.228 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:55.228 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572381+0000", 2024-09-21T15:50:55.228 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:55.228 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.228 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.228 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:55.228 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572381+0000", 2024-09-21T15:50:55.229 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:55.229 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.229 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.229 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:55.229 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572381+0000", 2024-09-21T15:50:55.229 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:55.229 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.229 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.229 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:55.229 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572383+0000", 2024-09-21T15:50:55.229 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0619999999999999e-06 2024-09-21T15:50:55.229 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.229 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.229 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:55.229 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572391+0000", 2024-09-21T15:50:55.230 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8769999999999993e-06 2024-09-21T15:50:55.230 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.230 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.230 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:55.230 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572399+0000", 2024-09-21T15:50:55.230 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6240000000000008e-06 2024-09-21T15:50:55.230 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.230 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.230 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:55.230 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628344+0000", 2024-09-21T15:50:55.230 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055944828000000002 2024-09-21T15:50:55.230 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.230 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.230 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:55.230 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628393+0000", 2024-09-21T15:50:55.230 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8653000000000003e-05 2024-09-21T15:50:55.231 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.231 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.231 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:55.231 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628470+0000", 2024-09-21T15:50:55.231 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7121999999999998e-05 2024-09-21T15:50:55.231 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:55.231 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:55.231 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:55.231 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.231 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.231 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120120 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 501760~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:55.231 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572418+0000", 2024-09-21T15:50:55.231 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 51.620694956999998, 2024-09-21T15:50:55.231 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056209138999999998, 2024-09-21T15:50:55.231 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:55.232 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:55.232 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:55.232 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:55.232 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:55.232 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120120 2024-09-21T15:50:55.232 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.232 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:55.232 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.232 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:55.232 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572418+0000", 2024-09-21T15:50:55.232 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:55.232 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.232 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.232 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:55.232 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572418+0000", 2024-09-21T15:50:55.232 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:55.233 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.233 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.233 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:55.233 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572417+0000", 2024-09-21T15:50:55.233 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:55.233 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.233 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.233 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:55.233 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572420+0000", 2024-09-21T15:50:55.233 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5270000000000001e-06 2024-09-21T15:50:55.233 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.233 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.233 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:55.233 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572426+0000", 2024-09-21T15:50:55.233 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3380000000000001e-06 2024-09-21T15:50:55.233 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.234 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.234 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:55.234 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572431+0000", 2024-09-21T15:50:55.234 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.865e-06 2024-09-21T15:50:55.234 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.234 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.234 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:55.234 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628500+0000", 2024-09-21T15:50:55.234 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056068917000000003 2024-09-21T15:50:55.234 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.234 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.234 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:55.234 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628542+0000", 2024-09-21T15:50:55.234 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1761999999999998e-05 2024-09-21T15:50:55.234 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.235 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.235 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:55.235 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628627+0000", 2024-09-21T15:50:55.235 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5247999999999994e-05 2024-09-21T15:50:55.235 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:55.235 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:55.235 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:55.235 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.235 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.235 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120121 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 522240~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:55.235 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572630+0000", 2024-09-21T15:50:55.235 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 51.620482570999997, 2024-09-21T15:50:55.235 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056207209000000001, 2024-09-21T15:50:55.235 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:55.235 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:55.236 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:55.236 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:55.236 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:55.236 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120121 2024-09-21T15:50:55.236 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.236 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:55.236 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.236 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:55.236 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572630+0000", 2024-09-21T15:50:55.236 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:55.236 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.236 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.236 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:55.236 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572630+0000", 2024-09-21T15:50:55.236 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:55.236 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.237 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.237 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:55.237 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572435+0000", 2024-09-21T15:50:55.237 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998045 2024-09-21T15:50:55.237 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.237 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.237 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:55.237 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572633+0000", 2024-09-21T15:50:55.237 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000198786 2024-09-21T15:50:55.237 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.237 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.237 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:55.237 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572641+0000", 2024-09-21T15:50:55.237 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9449999999999993e-06 2024-09-21T15:50:55.237 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.237 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.238 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:55.238 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572648+0000", 2024-09-21T15:50:55.238 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1929999999999998e-06 2024-09-21T15:50:55.238 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.238 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.238 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:55.238 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628669+0000", 2024-09-21T15:50:55.238 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056021690999999998 2024-09-21T15:50:55.238 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.238 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.238 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:55.238 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628751+0000", 2024-09-21T15:50:55.238 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1710000000000002e-05 2024-09-21T15:50:55.238 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.238 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.239 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:55.239 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628837+0000", 2024-09-21T15:50:55.239 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6383000000000003e-05 2024-09-21T15:50:55.239 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:55.239 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:55.239 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:55.239 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.239 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.239 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120122 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 538624~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:55.239 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.573471+0000", 2024-09-21T15:50:55.239 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 51.619641326, 2024-09-21T15:50:55.239 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055563935000000002, 2024-09-21T15:50:55.239 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:55.239 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:55.239 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:55.239 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:55.239 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:55.240 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120122 2024-09-21T15:50:55.240 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.240 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:55.240 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.240 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:55.240 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573471+0000", 2024-09-21T15:50:55.240 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:55.240 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.240 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.240 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:55.240 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573471+0000", 2024-09-21T15:50:55.240 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:55.240 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.240 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.240 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:55.240 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572653+0000", 2024-09-21T15:50:55.241 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9991813 2024-09-21T15:50:55.241 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.241 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.241 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:55.241 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573475+0000", 2024-09-21T15:50:55.241 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00082196500000000002 2024-09-21T15:50:55.241 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.241 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.241 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:55.241 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573483+0000", 2024-09-21T15:50:55.241 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5939999999999994e-06 2024-09-21T15:50:55.241 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.241 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.241 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:55.241 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573489+0000", 2024-09-21T15:50:55.241 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1689999999999997e-06 2024-09-21T15:50:55.242 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.242 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.242 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:55.242 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628882+0000", 2024-09-21T15:50:55.242 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055392230000000001 2024-09-21T15:50:55.242 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.242 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.242 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:55.242 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628934+0000", 2024-09-21T15:50:55.242 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1870000000000003e-05 2024-09-21T15:50:55.242 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.242 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.242 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:55.242 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629035+0000", 2024-09-21T15:50:55.242 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000101753 2024-09-21T15:50:55.243 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:55.243 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:55.243 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:55.243 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.243 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.243 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120123 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 559104~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:55.243 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.573495+0000", 2024-09-21T15:50:55.243 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 51.619617626999997, 2024-09-21T15:50:55.243 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055704765000000003, 2024-09-21T15:50:55.243 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:55.243 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:55.243 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:55.243 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:55.243 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:55.243 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120123 2024-09-21T15:50:55.244 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.244 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:55.244 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.244 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:55.244 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573495+0000", 2024-09-21T15:50:55.244 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:55.244 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.244 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.244 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:55.244 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573495+0000", 2024-09-21T15:50:55.244 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:55.244 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.244 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.244 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:55.244 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573495+0000", 2024-09-21T15:50:55.244 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:55.244 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.245 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.245 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:55.245 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573497+0000", 2024-09-21T15:50:55.245 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3420000000000002e-06 2024-09-21T15:50:55.245 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.245 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.245 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:55.245 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573503+0000", 2024-09-21T15:50:55.245 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7030000000000003e-06 2024-09-21T15:50:55.245 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.245 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.245 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:55.245 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573511+0000", 2024-09-21T15:50:55.245 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0790000000000001e-06 2024-09-21T15:50:55.245 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.245 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.246 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:55.246 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629067+0000", 2024-09-21T15:50:55.246 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055556573999999997 2024-09-21T15:50:55.246 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.246 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.246 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:55.246 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629118+0000", 2024-09-21T15:50:55.246 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1053000000000001e-05 2024-09-21T15:50:55.246 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.246 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.246 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:55.246 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629200+0000", 2024-09-21T15:50:55.246 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1452999999999994e-05 2024-09-21T15:50:55.246 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:55.247 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:55.247 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:55.247 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.247 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.247 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120124 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 579584~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:55.247 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.574014+0000", 2024-09-21T15:50:55.247 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 51.619098588, 2024-09-21T15:50:55.247 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055440284999999999, 2024-09-21T15:50:55.247 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:55.247 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:55.247 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:55.247 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:55.247 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:55.247 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120124 2024-09-21T15:50:55.247 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.247 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:55.248 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.248 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:55.248 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574014+0000", 2024-09-21T15:50:55.248 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:55.248 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.248 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.248 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:55.248 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574014+0000", 2024-09-21T15:50:55.248 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:55.248 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.248 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.248 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:55.248 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573515+0000", 2024-09-21T15:50:55.248 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9995008 2024-09-21T15:50:55.248 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.249 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.249 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:55.249 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574017+0000", 2024-09-21T15:50:55.249 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00050241100000000003 2024-09-21T15:50:55.249 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.249 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.249 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:55.249 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574027+0000", 2024-09-21T15:50:55.249 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4700000000000008e-06 2024-09-21T15:50:55.249 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.249 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.249 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:55.249 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574033+0000", 2024-09-21T15:50:55.249 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1489999999999999e-06 2024-09-21T15:50:55.249 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.249 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.249 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:55.250 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629238+0000", 2024-09-21T15:50:55.250 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055204880999999997 2024-09-21T15:50:55.250 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.250 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.250 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:55.250 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629317+0000", 2024-09-21T15:50:55.250 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8942000000000004e-05 2024-09-21T15:50:55.250 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.250 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.250 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:55.250 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629454+0000", 2024-09-21T15:50:55.250 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013767099999999999 2024-09-21T15:50:55.250 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:55.250 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:55.250 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:55.251 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.251 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.251 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120125 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 600064~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:55.251 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.574482+0000", 2024-09-21T15:50:55.251 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 51.618630289000002, 2024-09-21T15:50:55.251 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055243608, 2024-09-21T15:50:55.251 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:55.251 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:55.251 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:55.251 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:55.251 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:55.251 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120125 2024-09-21T15:50:55.251 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.251 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:55.251 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.251 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:55.252 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574482+0000", 2024-09-21T15:50:55.252 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:55.252 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.252 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.252 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:55.252 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574482+0000", 2024-09-21T15:50:55.252 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:55.252 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.252 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.252 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:55.252 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574038+0000", 2024-09-21T15:50:55.252 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9995556 2024-09-21T15:50:55.252 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.252 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.252 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:55.252 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574497+0000", 2024-09-21T15:50:55.253 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00045854400000000002 2024-09-21T15:50:55.253 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.253 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.253 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:55.253 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574509+0000", 2024-09-21T15:50:55.253 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2486e-05 2024-09-21T15:50:55.253 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.253 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.253 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:55.253 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574516+0000", 2024-09-21T15:50:55.253 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2269999999999996e-06 2024-09-21T15:50:55.253 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.253 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.253 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:55.253 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629497+0000", 2024-09-21T15:50:55.254 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054980785999999997 2024-09-21T15:50:55.254 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.254 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.254 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:55.254 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629580+0000", 2024-09-21T15:50:55.254 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2528000000000007e-05 2024-09-21T15:50:55.254 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.254 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.254 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:55.254 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629726+0000", 2024-09-21T15:50:55.254 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014630399999999999 2024-09-21T15:50:55.254 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:55.254 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:55.254 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:55.254 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.254 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.255 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120126 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 620544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:55.255 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.575315+0000", 2024-09-21T15:50:55.255 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 51.617797259, 2024-09-21T15:50:55.255 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054709559999999997, 2024-09-21T15:50:55.255 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:55.255 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:55.255 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:55.255 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:55.255 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:55.255 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120126 2024-09-21T15:50:55.255 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.255 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:55.255 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.255 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:55.255 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575315+0000", 2024-09-21T15:50:55.255 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:55.256 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.256 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.256 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:55.256 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575315+0000", 2024-09-21T15:50:55.256 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:55.256 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.256 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.256 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:55.256 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574522+0000", 2024-09-21T15:50:55.256 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9992065 2024-09-21T15:50:55.256 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.256 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.256 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:55.256 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575319+0000", 2024-09-21T15:50:55.256 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00079626800000000004 2024-09-21T15:50:55.256 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.257 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.257 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:55.257 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575329+0000", 2024-09-21T15:50:55.257 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.028e-05 2024-09-21T15:50:55.257 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.257 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.257 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:55.257 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575335+0000", 2024-09-21T15:50:55.257 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9290000000000003e-06 2024-09-21T15:50:55.257 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.257 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.257 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:55.257 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629806+0000", 2024-09-21T15:50:55.257 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054471558000000003 2024-09-21T15:50:55.257 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.257 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.258 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:55.258 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629885+0000", 2024-09-21T15:50:55.258 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8563e-05 2024-09-21T15:50:55.258 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.258 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.258 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:55.258 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630025+0000", 2024-09-21T15:50:55.258 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014019 2024-09-21T15:50:55.258 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:55.258 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:55.258 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:55.258 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.258 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.258 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120127 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 641024~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:55.258 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.575340+0000", 2024-09-21T15:50:55.259 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 51.617772551999998, 2024-09-21T15:50:55.259 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054895589000000002, 2024-09-21T15:50:55.259 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:55.259 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:55.259 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:55.259 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:55.259 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:55.259 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120127 2024-09-21T15:50:55.259 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.259 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:55.259 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.259 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:55.259 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575340+0000", 2024-09-21T15:50:55.259 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:55.259 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.260 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.260 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:55.260 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575340+0000", 2024-09-21T15:50:55.260 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:55.260 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.260 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.260 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:55.260 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575340+0000", 2024-09-21T15:50:55.260 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:55.260 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.260 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.260 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:55.260 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575342+0000", 2024-09-21T15:50:55.260 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5600000000000001e-06 2024-09-21T15:50:55.260 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.260 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.261 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:55.261 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575351+0000", 2024-09-21T15:50:55.261 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2349999999999999e-06 2024-09-21T15:50:55.261 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.261 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.261 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:55.261 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575356+0000", 2024-09-21T15:50:55.261 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4600000000000002e-06 2024-09-21T15:50:55.261 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.261 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.261 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:55.261 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630053+0000", 2024-09-21T15:50:55.261 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054697190999999999 2024-09-21T15:50:55.261 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.261 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.261 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:55.262 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630128+0000", 2024-09-21T15:50:55.262 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4777000000000006e-05 2024-09-21T15:50:55.262 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.262 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.262 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:55.262 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630236+0000", 2024-09-21T15:50:55.262 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000107837 2024-09-21T15:50:55.262 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:55.262 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:55.262 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:55.262 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.262 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.262 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120128 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 657408~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:55.262 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.575461+0000", 2024-09-21T15:50:55.262 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 51.617651973000001, 2024-09-21T15:50:55.263 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054945121999999999, 2024-09-21T15:50:55.263 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:55.263 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:55.263 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:55.263 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:55.263 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:55.263 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120128 2024-09-21T15:50:55.263 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.263 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:55.263 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.263 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:55.263 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575461+0000", 2024-09-21T15:50:55.263 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:55.263 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.263 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.264 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:55.264 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575461+0000", 2024-09-21T15:50:55.264 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:55.264 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.264 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.264 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:55.264 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575360+0000", 2024-09-21T15:50:55.264 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998994 2024-09-21T15:50:55.264 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.264 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.264 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:55.264 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575464+0000", 2024-09-21T15:50:55.264 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000103397 2024-09-21T15:50:55.264 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.264 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.264 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:55.265 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575472+0000", 2024-09-21T15:50:55.265 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3189999999999995e-06 2024-09-21T15:50:55.265 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.265 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.265 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:55.265 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575478+0000", 2024-09-21T15:50:55.265 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2779999999999999e-06 2024-09-21T15:50:55.265 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.265 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.265 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:55.265 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630268+0000", 2024-09-21T15:50:55.265 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054789545000000002 2024-09-21T15:50:55.265 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.265 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.265 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:55.265 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630315+0000", 2024-09-21T15:50:55.266 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7244999999999999e-05 2024-09-21T15:50:55.266 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.266 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.266 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:55.266 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630406+0000", 2024-09-21T15:50:55.266 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0811999999999997e-05 2024-09-21T15:50:55.266 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:55.266 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:55.266 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:55.266 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.266 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.266 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120129 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 677888~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:55.266 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.576016+0000", 2024-09-21T15:50:55.266 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 51.617096429, 2024-09-21T15:50:55.267 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054602823000000002, 2024-09-21T15:50:55.267 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:55.267 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:55.267 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:55.267 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:55.267 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:55.267 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120129 2024-09-21T15:50:55.267 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.267 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:55.267 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.267 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:55.267 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576016+0000", 2024-09-21T15:50:55.267 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:55.267 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.268 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.268 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:55.268 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576016+0000", 2024-09-21T15:50:55.268 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:55.268 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.268 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.268 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:55.268 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575483+0000", 2024-09-21T15:50:55.268 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9994669 2024-09-21T15:50:55.268 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.268 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.268 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:55.268 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576020+0000", 2024-09-21T15:50:55.268 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00053657900000000003 2024-09-21T15:50:55.268 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.268 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.268 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:55.269 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576030+0000", 2024-09-21T15:50:55.269 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0326000000000001e-05 2024-09-21T15:50:55.269 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.269 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.269 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:55.269 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576037+0000", 2024-09-21T15:50:55.269 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6490000000000002e-06 2024-09-21T15:50:55.269 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.269 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.269 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:55.269 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630446+0000", 2024-09-21T15:50:55.269 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054409698999999999 2024-09-21T15:50:55.269 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.269 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.269 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:55.269 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630524+0000", 2024-09-21T15:50:55.270 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7422999999999994e-05 2024-09-21T15:50:55.270 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.270 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.270 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:55.270 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630619+0000", 2024-09-21T15:50:55.270 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5332000000000001e-05 2024-09-21T15:50:55.270 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:55.270 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:55.270 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:55.270 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.270 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.270 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120130 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 698368~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:55.270 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.576043+0000", 2024-09-21T15:50:55.270 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 51.617070058000003, 2024-09-21T15:50:55.270 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054781798, 2024-09-21T15:50:55.271 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:55.271 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:55.271 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:55.271 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:55.271 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:55.271 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120130 2024-09-21T15:50:55.271 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.271 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:55.271 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.271 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:55.271 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576043+0000", 2024-09-21T15:50:55.271 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:55.271 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.271 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.271 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:55.271 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576043+0000", 2024-09-21T15:50:55.272 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:55.272 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.272 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.272 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:55.272 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576042+0000", 2024-09-21T15:50:55.272 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:55.272 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.272 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.272 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:55.272 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576044+0000", 2024-09-21T15:50:55.272 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0999999999999998e-06 2024-09-21T15:50:55.272 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.272 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.272 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:55.272 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576051+0000", 2024-09-21T15:50:55.272 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4779999999999996e-06 2024-09-21T15:50:55.273 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.273 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.273 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:55.273 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576058+0000", 2024-09-21T15:50:55.273 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2180000000000002e-06 2024-09-21T15:50:55.273 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.273 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.273 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:55.273 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630661+0000", 2024-09-21T15:50:55.273 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054603240999999997 2024-09-21T15:50:55.273 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.273 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.273 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:55.273 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630747+0000", 2024-09-21T15:50:55.273 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6046999999999998e-05 2024-09-21T15:50:55.273 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:55.274 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:55.274 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:55.274 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630824+0000", 2024-09-21T15:50:55.274 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7119999999999993e-05 2024-09-21T15:50:55.274 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:55.274 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:55.274 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:55.274 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:55.274 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:55.274 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:55.275 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-09-21T15:50:55.275 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:50:55.276 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:50:55.276 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:50:55.276 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T15:50:55.277 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:55.274+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:55.277 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:50:55.277 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:50:55.277 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:50:55.278 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:50:55.292 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:50:55.292 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:50:55.308 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:50:55.308 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:50:55.324 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:50:55.324 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:50:55.340 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:50:55.340 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:50:55.376 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:55.374+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:55.418 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:55.418 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:55.426 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T15:50:55.476 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:50:55.475+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:50:55.528 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:55.528 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:55.537 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T15:50:55.580 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:55.579+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:55.639 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:55.639 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:55.647 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T15:50:55.688 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:55.687+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:55.779 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:55.777+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:55.825 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:55.825 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:55.835 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T15:50:55.919 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:55.918+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:55.980 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:55.979+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:55.999 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:55.999 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:56.009 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T15:50:56.093 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:56.092+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:56.167 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:56.167 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:56.178 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:50:56.221 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:56.220+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:56.281 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:56.280+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:56.282 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:56.282 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:56.291 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:50:56.381 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:56.380+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:56.460 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:56.460 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:56.470 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:50:56.482 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:56.481+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:56.582 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:56.581+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:56.618 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:56.618 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:56.628 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:50:56.706 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T15:50:56.712 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:56.711+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:56.782 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:56.782 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:56.784 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:56.783+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:56.791 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:50:56.886 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:50:56.885+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:50:56.935 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:56.935 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:56.944 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:50:56.987 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:56.985+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:57.085 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:57.086 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:57.088 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:57.087+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:57.094 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:50:57.188 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:57.187+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:57.195 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:57.196 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:57.204 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:50:57.289 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:57.288+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:57.305 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:57.305 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:57.313 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:50:57.351 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:50:57.389 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:57.388+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:57.417 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:57.417 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:57.426 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:50:57.477 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-09-21T15:50:57.477 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:50:57.490 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:57.489+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:57.494 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:50:57.494 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:50:57.510 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:50:57.510 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:50:57.526 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:50:57.526 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:50:57.531 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:57.531 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:57.531 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:50:57.531 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:57.540 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:50:57.543 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:50:57.543 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:50:57.560 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:50:57.560 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:50:57.577 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:50:57.577 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:50:57.591 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:57.590+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:57.594 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:50:57.594 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:50:57.646 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:57.646 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:50:57.646 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:50:57.646 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:50:57.646 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:57.655 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:50:57.671 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is set 2024-09-21T15:50:57.731 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:57.730+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:57.770 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:50:57.770 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:50:57.770 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:50:57.770 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:50:57.770 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.770 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:14051 3.8 3:13437810:::rbd_data.1133dc5adfdf.0000000000000039:head [write 716800~796] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-21T15:50:57.770 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:50:57.770 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 100.07307297600001, 2024-09-21T15:50:57.770 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.98064002699999997, 2024-09-21T15:50:57.771 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:57.771 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:57.771 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:57.771 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:57.771 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:57.771 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 14051 2024-09-21T15:50:57.771 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.771 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:57.771 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.771 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:57.771 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:50:57.771 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:57.771 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.771 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.772 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:57.772 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:50:57.772 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:57.772 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.772 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.772 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:57.772 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:50:57.772 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:57.772 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.772 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.772 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:57.772 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695131+0000", 2024-09-21T15:50:57.772 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4680000000000003e-06 2024-09-21T15:50:57.772 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.773 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.773 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:57.773 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695139+0000", 2024-09-21T15:50:57.773 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9400000000000002e-06 2024-09-21T15:50:57.773 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.773 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.773 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:57.773 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695143+0000", 2024-09-21T15:50:57.773 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9899999999999999e-06 2024-09-21T15:50:57.773 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.773 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.773 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:57.773 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695190+0000", 2024-09-21T15:50:57.774 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6458000000000001e-05 2024-09-21T15:50:57.774 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.774 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.774 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:50:57.774 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695201+0000", 2024-09-21T15:50:57.774 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0964999999999999e-05 2024-09-21T15:50:57.774 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.774 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.774 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:57.774 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.669490+0000", 2024-09-21T15:50:57.774 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.97428980499999995 2024-09-21T15:50:57.774 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.774 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.775 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-09-21T15:50:57.775 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.670033+0000", 2024-09-21T15:50:57.775 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00054252599999999999 2024-09-21T15:50:57.775 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.775 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.775 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:57.775 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.671095+0000", 2024-09-21T15:50:57.775 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001062183 2024-09-21T15:50:57.775 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.775 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.775 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:57.775 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.671186+0000", 2024-09-21T15:50:57.775 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1191000000000002e-05 2024-09-21T15:50:57.775 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.776 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.776 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-09-21T15:50:57.776 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.671346+0000", 2024-09-21T15:50:57.776 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015980399999999999 2024-09-21T15:50:57.776 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.776 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.776 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:57.776 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.672221+0000", 2024-09-21T15:50:57.776 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00087471999999999995 2024-09-21T15:50:57.776 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.776 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.776 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:50:57.776 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.675619+0000", 2024-09-21T15:50:57.776 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00339858 2024-09-21T15:50:57.777 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.777 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.777 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:50:57.777 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.675670+0000", 2024-09-21T15:50:57.777 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0664000000000001e-05 2024-09-21T15:50:57.777 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.777 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.777 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:57.777 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.675768+0000", 2024-09-21T15:50:57.777 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7415000000000002e-05 2024-09-21T15:50:57.777 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:57.777 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:57.777 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:57.778 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.778 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.778 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116235 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 761856~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:57.778 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048806+0000", 2024-09-21T15:50:57.778 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 61.719394092000002, 2024-09-21T15:50:57.778 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056818556999999999, 2024-09-21T15:50:57.778 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:57.778 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:57.778 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:57.778 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:57.778 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:57.778 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116235 2024-09-21T15:50:57.778 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.779 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:57.779 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.779 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:57.779 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048806+0000", 2024-09-21T15:50:57.779 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:57.779 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.779 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.779 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:57.779 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048806+0000", 2024-09-21T15:50:57.779 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:57.779 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.779 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.779 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:57.780 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048806+0000", 2024-09-21T15:50:57.780 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:57.780 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.780 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.780 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:57.780 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048807+0000", 2024-09-21T15:50:57.780 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.001e-06 2024-09-21T15:50:57.780 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.780 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.780 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:57.780 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048810+0000", 2024-09-21T15:50:57.780 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.711e-06 2024-09-21T15:50:57.780 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.780 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.781 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:57.781 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048812+0000", 2024-09-21T15:50:57.781 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2290000000000002e-06 2024-09-21T15:50:57.781 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.781 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.781 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:57.781 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105495+0000", 2024-09-21T15:50:57.781 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056682679999999999 2024-09-21T15:50:57.781 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.781 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.781 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:57.781 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105565+0000", 2024-09-21T15:50:57.781 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9868000000000003e-05 2024-09-21T15:50:57.781 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.781 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.782 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:57.782 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105625+0000", 2024-09-21T15:50:57.782 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0268e-05 2024-09-21T15:50:57.782 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:57.782 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:57.782 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:57.782 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.782 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.782 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116236 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 768000~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:57.782 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048814+0000", 2024-09-21T15:50:57.782 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 61.719386428999996, 2024-09-21T15:50:57.782 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056959587999999998, 2024-09-21T15:50:57.782 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:57.782 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:57.783 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:57.783 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:57.783 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:57.783 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116236 2024-09-21T15:50:57.783 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.783 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:57.783 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.783 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:57.783 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048814+0000", 2024-09-21T15:50:57.783 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:57.783 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.783 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.783 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:57.783 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048814+0000", 2024-09-21T15:50:57.783 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:57.784 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.784 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.784 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:57.784 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048814+0000", 2024-09-21T15:50:57.784 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:57.784 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.784 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.784 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:57.784 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048815+0000", 2024-09-21T15:50:57.784 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7999999999999993e-07 2024-09-21T15:50:57.784 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.784 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.784 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:57.785 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048817+0000", 2024-09-21T15:50:57.785 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6620000000000001e-06 2024-09-21T15:50:57.785 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.785 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.785 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:57.785 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048820+0000", 2024-09-21T15:50:57.785 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2800000000000002e-06 2024-09-21T15:50:57.785 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.785 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.785 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:57.785 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105647+0000", 2024-09-21T15:50:57.785 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056827253000000001 2024-09-21T15:50:57.785 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.785 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.786 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:57.786 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105716+0000", 2024-09-21T15:50:57.786 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8761000000000001e-05 2024-09-21T15:50:57.786 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.786 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.786 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:57.786 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105774+0000", 2024-09-21T15:50:57.786 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7855000000000001e-05 2024-09-21T15:50:57.786 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:57.786 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:57.786 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:57.786 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.786 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.786 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116237 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 774144~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:57.786 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048822+0000", 2024-09-21T15:50:57.787 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 61.719378743999997, 2024-09-21T15:50:57.787 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057114028999999997, 2024-09-21T15:50:57.787 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:57.787 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:57.787 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:57.787 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:57.787 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:57.787 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116237 2024-09-21T15:50:57.787 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.787 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:57.787 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.787 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:57.787 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048822+0000", 2024-09-21T15:50:57.787 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:57.787 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.788 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.788 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:57.788 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048822+0000", 2024-09-21T15:50:57.788 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:57.788 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.788 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.788 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:57.788 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048822+0000", 2024-09-21T15:50:57.788 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:57.788 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.788 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.788 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:57.788 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048823+0000", 2024-09-21T15:50:57.788 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9699999999999994e-07 2024-09-21T15:50:57.788 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.789 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.789 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:57.789 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048825+0000", 2024-09-21T15:50:57.789 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.65e-06 2024-09-21T15:50:57.789 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.789 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.789 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:57.789 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048828+0000", 2024-09-21T15:50:57.789 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8890000000000002e-06 2024-09-21T15:50:57.789 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.789 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.789 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:57.789 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105802+0000", 2024-09-21T15:50:57.789 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056974415 2024-09-21T15:50:57.790 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.790 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.790 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:57.790 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105871+0000", 2024-09-21T15:50:57.790 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8727999999999997e-05 2024-09-21T15:50:57.790 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.790 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.790 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:57.790 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105936+0000", 2024-09-21T15:50:57.790 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4554999999999994e-05 2024-09-21T15:50:57.790 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:57.790 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:57.790 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:57.791 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.791 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.791 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116238 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 780288~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:57.791 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048830+0000", 2024-09-21T15:50:57.791 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 61.719370484999999, 2024-09-21T15:50:57.791 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057296574000000003, 2024-09-21T15:50:57.791 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:57.791 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:57.791 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:57.791 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:57.791 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:57.791 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116238 2024-09-21T15:50:57.791 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.791 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:57.791 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.792 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:57.792 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048830+0000", 2024-09-21T15:50:57.792 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:57.792 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.792 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.792 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:57.792 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048830+0000", 2024-09-21T15:50:57.792 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:57.792 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.792 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.792 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:57.792 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048830+0000", 2024-09-21T15:50:57.792 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:57.792 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.792 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.793 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:57.793 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048831+0000", 2024-09-21T15:50:57.793 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0160000000000001e-06 2024-09-21T15:50:57.793 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.793 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.793 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:57.793 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048834+0000", 2024-09-21T15:50:57.793 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.813e-06 2024-09-21T15:50:57.793 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.793 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.793 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:57.793 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048836+0000", 2024-09-21T15:50:57.793 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6790000000000001e-06 2024-09-21T15:50:57.793 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.793 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.793 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:57.794 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105961+0000", 2024-09-21T15:50:57.794 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057124705999999997 2024-09-21T15:50:57.794 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.794 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.794 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:57.794 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106042+0000", 2024-09-21T15:50:57.794 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0820000000000002e-05 2024-09-21T15:50:57.794 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.794 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.794 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:57.794 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106127+0000", 2024-09-21T15:50:57.794 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4746000000000005e-05 2024-09-21T15:50:57.794 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:57.794 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:57.795 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:57.795 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.795 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.795 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116324 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 524288~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:57.795 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048838+0000", 2024-09-21T15:50:57.795 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 61.719362175999997, 2024-09-21T15:50:57.795 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057477185, 2024-09-21T15:50:57.795 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:57.795 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:57.795 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:57.795 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:57.795 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:57.795 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116324 2024-09-21T15:50:57.795 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.795 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:57.796 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.796 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:57.796 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048838+0000", 2024-09-21T15:50:57.796 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:57.796 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.796 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.796 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:57.796 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048838+0000", 2024-09-21T15:50:57.796 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:57.796 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.796 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.796 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:57.796 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048838+0000", 2024-09-21T15:50:57.796 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:57.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.797 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.797 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:57.797 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048842+0000", 2024-09-21T15:50:57.797 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4829999999999999e-06 2024-09-21T15:50:57.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.797 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.797 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:57.797 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048846+0000", 2024-09-21T15:50:57.797 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0470000000000004e-06 2024-09-21T15:50:57.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.797 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.797 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:57.797 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048849+0000", 2024-09-21T15:50:57.797 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3689999999999998e-06 2024-09-21T15:50:57.798 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.798 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.798 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:57.798 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106157+0000", 2024-09-21T15:50:57.798 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057307942000000001 2024-09-21T15:50:57.798 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.798 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.798 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:57.798 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106237+0000", 2024-09-21T15:50:57.798 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0320000000000003e-05 2024-09-21T15:50:57.798 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.798 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.798 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:57.798 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106315+0000", 2024-09-21T15:50:57.798 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8227999999999997e-05 2024-09-21T15:50:57.799 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:57.799 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:57.799 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:57.799 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.799 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.799 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116325 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 530432~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:57.799 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048851+0000", 2024-09-21T15:50:57.799 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 61.719349004000001, 2024-09-21T15:50:57.799 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057581036000000002, 2024-09-21T15:50:57.799 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:57.799 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:57.799 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:57.799 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:57.799 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:57.799 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116325 2024-09-21T15:50:57.800 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.800 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:57.800 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.800 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:57.800 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048851+0000", 2024-09-21T15:50:57.800 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:57.800 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.800 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.800 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:57.800 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048851+0000", 2024-09-21T15:50:57.800 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:57.800 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.800 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.800 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:57.800 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048851+0000", 2024-09-21T15:50:57.800 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:57.801 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.801 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.801 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:57.801 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048852+0000", 2024-09-21T15:50:57.801 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.077e-06 2024-09-21T15:50:57.801 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.801 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.801 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:57.801 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048855+0000", 2024-09-21T15:50:57.801 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9859999999999999e-06 2024-09-21T15:50:57.801 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.801 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.801 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:57.801 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048858+0000", 2024-09-21T15:50:57.801 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3810000000000002e-06 2024-09-21T15:50:57.801 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.802 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.802 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:57.802 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106335+0000", 2024-09-21T15:50:57.802 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057477473000000001 2024-09-21T15:50:57.802 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.802 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.802 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:57.802 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106393+0000", 2024-09-21T15:50:57.802 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8050000000000002e-05 2024-09-21T15:50:57.802 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.802 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.802 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:57.802 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106432+0000", 2024-09-21T15:50:57.802 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9274999999999999e-05 2024-09-21T15:50:57.803 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:57.803 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:57.803 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:57.803 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.803 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.803 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116326 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 536576~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:57.803 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048860+0000", 2024-09-21T15:50:57.803 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 61.719340953, 2024-09-21T15:50:57.803 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057774197999999999, 2024-09-21T15:50:57.803 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:57.803 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:57.803 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:57.803 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:57.804 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:57.804 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116326 2024-09-21T15:50:57.804 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.804 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:57.804 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.804 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:57.804 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048860+0000", 2024-09-21T15:50:57.804 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:57.804 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.804 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.804 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:57.804 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048860+0000", 2024-09-21T15:50:57.804 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:57.805 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.805 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.805 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:57.805 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048859+0000", 2024-09-21T15:50:57.805 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:50:57.805 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.805 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.805 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:57.805 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048860+0000", 2024-09-21T15:50:57.805 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.02e-06 2024-09-21T15:50:57.805 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.805 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.805 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:57.805 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048863+0000", 2024-09-21T15:50:57.806 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1039999999999998e-06 2024-09-21T15:50:57.806 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.806 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.806 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:57.806 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048867+0000", 2024-09-21T15:50:57.806 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.563e-06 2024-09-21T15:50:57.806 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.806 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.806 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:57.806 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106465+0000", 2024-09-21T15:50:57.806 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057597789000000003 2024-09-21T15:50:57.806 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.806 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.806 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:57.807 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106552+0000", 2024-09-21T15:50:57.807 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7602000000000006e-05 2024-09-21T15:50:57.807 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.807 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.807 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:57.807 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106634+0000", 2024-09-21T15:50:57.807 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1321000000000003e-05 2024-09-21T15:50:57.807 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:57.807 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:57.807 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:57.807 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.807 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.808 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116327 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 542720~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:57.808 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048870+0000", 2024-09-21T15:50:57.808 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 61.719330081000003, 2024-09-21T15:50:57.808 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057883148000000002, 2024-09-21T15:50:57.808 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:57.808 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:57.808 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:57.808 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:57.808 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:57.808 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116327 2024-09-21T15:50:57.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.808 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:57.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.808 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:57.808 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048870+0000", 2024-09-21T15:50:57.808 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:57.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.809 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:57.809 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048870+0000", 2024-09-21T15:50:57.809 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:57.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.809 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:57.809 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048870+0000", 2024-09-21T15:50:57.809 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:57.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.809 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:57.809 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048872+0000", 2024-09-21T15:50:57.810 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.824e-06 2024-09-21T15:50:57.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.810 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.810 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:57.810 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048879+0000", 2024-09-21T15:50:57.810 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6749999999999996e-06 2024-09-21T15:50:57.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.810 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.810 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:57.810 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048883+0000", 2024-09-21T15:50:57.810 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2200000000000003e-06 2024-09-21T15:50:57.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.810 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.810 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:57.810 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106663+0000", 2024-09-21T15:50:57.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057780049 2024-09-21T15:50:57.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:57.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106711+0000", 2024-09-21T15:50:57.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8433999999999997e-05 2024-09-21T15:50:57.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:57.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106754+0000", 2024-09-21T15:50:57.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2236e-05 2024-09-21T15:50:57.811 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:57.811 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:57.811 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:57.812 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.812 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.812 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116328 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 548864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:57.812 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048887+0000", 2024-09-21T15:50:57.812 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 61.719313561, 2024-09-21T15:50:57.812 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057981984, 2024-09-21T15:50:57.812 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:57.812 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:57.812 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:57.812 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:57.812 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:57.812 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116328 2024-09-21T15:50:57.812 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.812 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:57.812 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.813 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:57.813 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048887+0000", 2024-09-21T15:50:57.813 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:57.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.813 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:57.813 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048887+0000", 2024-09-21T15:50:57.813 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:57.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.813 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:57.813 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048887+0000", 2024-09-21T15:50:57.813 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:57.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:57.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048891+0000", 2024-09-21T15:50:57.814 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.843e-06 2024-09-21T15:50:57.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.814 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:57.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048897+0000", 2024-09-21T15:50:57.814 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5419999999999996e-06 2024-09-21T15:50:57.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.814 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:57.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048901+0000", 2024-09-21T15:50:57.814 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4689999999999999e-06 2024-09-21T15:50:57.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.815 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.815 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:57.815 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106780+0000", 2024-09-21T15:50:57.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057878259000000001 2024-09-21T15:50:57.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.815 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.815 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:57.815 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106827+0000", 2024-09-21T15:50:57.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7098000000000002e-05 2024-09-21T15:50:57.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.815 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.815 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:57.815 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106869+0000", 2024-09-21T15:50:57.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2129999999999998e-05 2024-09-21T15:50:57.816 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:57.816 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:57.816 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:57.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.816 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.816 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116329 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 555008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:57.816 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048905+0000", 2024-09-21T15:50:57.816 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 61.719295451000001, 2024-09-21T15:50:57.816 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058115554999999999, 2024-09-21T15:50:57.816 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:57.816 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:57.816 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:57.816 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:57.816 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:57.816 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116329 2024-09-21T15:50:57.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.817 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:57.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.817 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:57.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048905+0000", 2024-09-21T15:50:57.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:57.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.817 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:57.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048905+0000", 2024-09-21T15:50:57.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:57.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.817 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:57.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048905+0000", 2024-09-21T15:50:57.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:57.818 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.818 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.818 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:57.818 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048907+0000", 2024-09-21T15:50:57.818 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.086e-06 2024-09-21T15:50:57.818 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.818 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.818 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:57.818 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048912+0000", 2024-09-21T15:50:57.818 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2830000000000001e-06 2024-09-21T15:50:57.818 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.818 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.818 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:57.819 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048916+0000", 2024-09-21T15:50:57.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.98e-06 2024-09-21T15:50:57.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.819 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:57.819 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106907+0000", 2024-09-21T15:50:57.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057990719000000003 2024-09-21T15:50:57.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.819 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:57.819 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106969+0000", 2024-09-21T15:50:57.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1789000000000001e-05 2024-09-21T15:50:57.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.820 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:57.820 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107021+0000", 2024-09-21T15:50:57.820 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2030999999999997e-05 2024-09-21T15:50:57.820 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:57.820 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:57.820 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:57.820 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.820 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.820 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116330 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 561152~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:57.820 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048922+0000", 2024-09-21T15:50:57.820 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 61.719278293000002, 2024-09-21T15:50:57.820 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058242343000000002, 2024-09-21T15:50:57.820 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:57.821 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:57.821 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:57.821 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:57.821 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:57.821 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116330 2024-09-21T15:50:57.821 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.821 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:57.821 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.821 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:57.821 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048922+0000", 2024-09-21T15:50:57.821 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:57.821 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.821 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.821 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:57.822 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048922+0000", 2024-09-21T15:50:57.822 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:57.822 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.822 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.822 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:57.822 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048922+0000", 2024-09-21T15:50:57.822 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:57.822 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.822 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.822 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:57.822 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048924+0000", 2024-09-21T15:50:57.822 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1660000000000001e-06 2024-09-21T15:50:57.822 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.823 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.823 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:57.823 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048930+0000", 2024-09-21T15:50:57.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4529999999999999e-06 2024-09-21T15:50:57.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.823 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.823 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:57.823 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048934+0000", 2024-09-21T15:50:57.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0269999999999997e-06 2024-09-21T15:50:57.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.823 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.823 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:57.823 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107047+0000", 2024-09-21T15:50:57.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058112913000000002 2024-09-21T15:50:57.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.824 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:57.824 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107095+0000", 2024-09-21T15:50:57.824 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7778000000000002e-05 2024-09-21T15:50:57.824 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.824 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:57.824 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107165+0000", 2024-09-21T15:50:57.824 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9370999999999997e-05 2024-09-21T15:50:57.824 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:57.824 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:57.824 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:57.824 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.825 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116331 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 567296~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:57.825 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.049440+0000", 2024-09-21T15:50:57.825 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 61.718760199000002, 2024-09-21T15:50:57.825 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057878485, 2024-09-21T15:50:57.825 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:57.825 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:57.825 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:57.825 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:57.825 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:57.825 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116331 2024-09-21T15:50:57.825 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.825 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:57.825 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.825 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:57.826 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049440+0000", 2024-09-21T15:50:57.826 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:57.826 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.826 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.826 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:57.826 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049440+0000", 2024-09-21T15:50:57.826 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:57.826 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.826 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.826 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:57.826 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048939+0000", 2024-09-21T15:50:57.826 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9994984 2024-09-21T15:50:57.826 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.826 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.827 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:57.827 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049444+0000", 2024-09-21T15:50:57.827 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00050557200000000003 2024-09-21T15:50:57.827 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.827 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.827 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:57.827 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049460+0000", 2024-09-21T15:50:57.827 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6120000000000002e-05 2024-09-21T15:50:57.827 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.827 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.827 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:57.827 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049468+0000", 2024-09-21T15:50:57.827 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4209999999999998e-06 2024-09-21T15:50:57.827 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.828 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.828 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:57.828 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107189+0000", 2024-09-21T15:50:57.828 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057721019999999998 2024-09-21T15:50:57.828 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.828 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.828 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:57.828 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107254+0000", 2024-09-21T15:50:57.828 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5265000000000006e-05 2024-09-21T15:50:57.828 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.828 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.828 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:57.828 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107319+0000", 2024-09-21T15:50:57.828 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4784999999999997e-05 2024-09-21T15:50:57.829 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:57.829 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:57.829 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:57.829 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.829 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.829 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116332 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 573440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:57.829 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.049474+0000", 2024-09-21T15:50:57.829 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 61.718726938000003, 2024-09-21T15:50:57.829 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058013809, 2024-09-21T15:50:57.829 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:57.829 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:57.829 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:57.829 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:57.829 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:57.830 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116332 2024-09-21T15:50:57.830 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.830 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:57.830 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.830 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:57.830 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049474+0000", 2024-09-21T15:50:57.830 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:57.830 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.830 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.830 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:57.830 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049474+0000", 2024-09-21T15:50:57.830 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:57.830 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.831 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.831 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:57.831 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049473+0000", 2024-09-21T15:50:57.831 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:57.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.831 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.831 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:57.831 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049476+0000", 2024-09-21T15:50:57.831 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6249999999999999e-06 2024-09-21T15:50:57.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.831 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.831 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:57.831 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049482+0000", 2024-09-21T15:50:57.831 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0959999999999997e-06 2024-09-21T15:50:57.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.832 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.832 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:57.832 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049487+0000", 2024-09-21T15:50:57.832 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5949999999999998e-06 2024-09-21T15:50:57.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.832 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.832 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:57.832 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107344+0000", 2024-09-21T15:50:57.832 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057856594999999997 2024-09-21T15:50:57.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.832 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.832 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:57.832 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107422+0000", 2024-09-21T15:50:57.832 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7582000000000004e-05 2024-09-21T15:50:57.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.833 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.833 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:57.833 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107487+0000", 2024-09-21T15:50:57.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5723999999999996e-05 2024-09-21T15:50:57.833 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:57.833 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:57.833 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:57.833 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.833 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.833 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116333 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 579584~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:57.833 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.049993+0000", 2024-09-21T15:50:57.833 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 61.718207155000002, 2024-09-21T15:50:57.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057663537000000001, 2024-09-21T15:50:57.833 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:57.833 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:57.833 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:57.834 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:57.834 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:57.834 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116333 2024-09-21T15:50:57.834 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.834 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:57.834 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.834 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:57.834 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049993+0000", 2024-09-21T15:50:57.834 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:57.834 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.834 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.834 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:57.834 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049993+0000", 2024-09-21T15:50:57.835 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:57.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.835 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.835 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:57.835 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049491+0000", 2024-09-21T15:50:57.835 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9994979 2024-09-21T15:50:57.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.835 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.835 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:57.835 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049998+0000", 2024-09-21T15:50:57.835 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00050654200000000002 2024-09-21T15:50:57.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.835 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.835 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:57.836 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050010+0000", 2024-09-21T15:50:57.836 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1790999999999999e-05 2024-09-21T15:50:57.836 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.836 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.836 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:57.836 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050016+0000", 2024-09-21T15:50:57.836 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2179999999999996e-06 2024-09-21T15:50:57.836 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.836 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.836 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:57.836 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107513+0000", 2024-09-21T15:50:57.836 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057496927000000003 2024-09-21T15:50:57.836 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.836 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.836 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:57.836 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107590+0000", 2024-09-21T15:50:57.837 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7547999999999997e-05 2024-09-21T15:50:57.837 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.837 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.837 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:57.837 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107657+0000", 2024-09-21T15:50:57.837 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6630000000000004e-05 2024-09-21T15:50:57.837 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:57.837 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:57.837 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:57.837 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.837 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.837 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116334 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 585728~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:57.837 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.050091+0000", 2024-09-21T15:50:57.837 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 61.718109527000003, 2024-09-21T15:50:57.837 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057734355000000001, 2024-09-21T15:50:57.838 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:57.838 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:57.838 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:57.838 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:57.838 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:57.838 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116334 2024-09-21T15:50:57.838 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.838 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:57.838 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.838 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:57.838 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050091+0000", 2024-09-21T15:50:57.838 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:57.838 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.838 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.839 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:57.839 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050091+0000", 2024-09-21T15:50:57.839 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:57.839 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.839 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.839 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:57.839 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050021+0000", 2024-09-21T15:50:57.839 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999304 2024-09-21T15:50:57.839 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.839 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.839 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:57.839 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050095+0000", 2024-09-21T15:50:57.839 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4158000000000004e-05 2024-09-21T15:50:57.839 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.840 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.840 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:57.840 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050103+0000", 2024-09-21T15:50:57.840 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0770000000000002e-06 2024-09-21T15:50:57.840 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.840 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.840 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:57.840 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050108+0000", 2024-09-21T15:50:57.840 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6330000000000001e-06 2024-09-21T15:50:57.840 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.840 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.840 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:57.840 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107681+0000", 2024-09-21T15:50:57.840 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057572996000000001 2024-09-21T15:50:57.840 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.840 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.841 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:57.841 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107759+0000", 2024-09-21T15:50:57.841 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7368999999999997e-05 2024-09-21T15:50:57.841 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.841 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.841 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:57.841 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107825+0000", 2024-09-21T15:50:57.841 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6783000000000001e-05 2024-09-21T15:50:57.841 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:57.841 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:57.841 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:57.841 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.841 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.841 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116335 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 591872~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:57.842 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.050542+0000", 2024-09-21T15:50:57.842 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 61.717658942, 2024-09-21T15:50:57.842 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057466455999999999, 2024-09-21T15:50:57.842 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:57.842 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:57.842 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:57.842 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:57.842 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:57.842 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116335 2024-09-21T15:50:57.842 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.842 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:57.842 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.842 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:57.842 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050542+0000", 2024-09-21T15:50:57.842 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:57.843 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.843 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.843 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:57.843 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050542+0000", 2024-09-21T15:50:57.843 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:57.843 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.843 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.843 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:57.843 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050113+0000", 2024-09-21T15:50:57.843 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9995713 2024-09-21T15:50:57.843 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.843 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.843 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:57.843 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050546+0000", 2024-09-21T15:50:57.843 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00043328699999999997 2024-09-21T15:50:57.844 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.844 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.844 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:57.844 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050557+0000", 2024-09-21T15:50:57.844 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1011e-05 2024-09-21T15:50:57.844 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.844 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.844 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:57.844 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050564+0000", 2024-09-21T15:50:57.844 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1130000000000003e-06 2024-09-21T15:50:57.844 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.844 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.844 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:57.844 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107852+0000", 2024-09-21T15:50:57.844 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057287350000000001 2024-09-21T15:50:57.845 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.845 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.845 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:57.845 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107928+0000", 2024-09-21T15:50:57.845 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6668000000000005e-05 2024-09-21T15:50:57.845 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.845 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.845 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:57.845 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108008+0000", 2024-09-21T15:50:57.845 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9783999999999994e-05 2024-09-21T15:50:57.845 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:57.845 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:57.845 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:57.845 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.845 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.846 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116336 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 598016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:57.846 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.050834+0000", 2024-09-21T15:50:57.846 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 61.717366675999997, 2024-09-21T15:50:57.846 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057353572999999998, 2024-09-21T15:50:57.846 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:57.846 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:57.846 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:57.846 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:57.846 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:57.846 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116336 2024-09-21T15:50:57.846 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.846 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:57.846 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.846 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:57.846 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050834+0000", 2024-09-21T15:50:57.846 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:57.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.847 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.847 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:57.847 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050834+0000", 2024-09-21T15:50:57.847 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:57.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.847 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.847 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:57.847 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050571+0000", 2024-09-21T15:50:57.847 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9997373 2024-09-21T15:50:57.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.847 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.847 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:57.847 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050842+0000", 2024-09-21T15:50:57.847 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00027029099999999998 2024-09-21T15:50:57.848 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.848 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.848 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:57.848 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050853+0000", 2024-09-21T15:50:57.848 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1532000000000001e-05 2024-09-21T15:50:57.848 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.848 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.848 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:57.848 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050863+0000", 2024-09-21T15:50:57.848 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4239999999999999e-06 2024-09-21T15:50:57.848 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.848 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.848 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:57.848 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108045+0000", 2024-09-21T15:50:57.848 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057182415 2024-09-21T15:50:57.848 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.849 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.849 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:57.849 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108123+0000", 2024-09-21T15:50:57.849 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8275000000000005e-05 2024-09-21T15:50:57.849 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.849 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.849 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:57.849 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108187+0000", 2024-09-21T15:50:57.849 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4152000000000005e-05 2024-09-21T15:50:57.849 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:57.849 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:57.849 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:57.849 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.849 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.849 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116337 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 604160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:57.849 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.051192+0000", 2024-09-21T15:50:57.850 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 61.717008047999997, 2024-09-21T15:50:57.850 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057175981000000001, 2024-09-21T15:50:57.850 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:57.850 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:57.850 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:57.850 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:57.850 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:57.850 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116337 2024-09-21T15:50:57.850 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.850 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:57.850 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.850 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:57.850 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051192+0000", 2024-09-21T15:50:57.850 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:57.850 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.851 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.851 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:57.851 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051192+0000", 2024-09-21T15:50:57.851 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:57.851 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.851 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.851 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:57.851 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050870+0000", 2024-09-21T15:50:57.851 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9996777 2024-09-21T15:50:57.851 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.851 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.851 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:57.851 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051197+0000", 2024-09-21T15:50:57.851 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00032656299999999998 2024-09-21T15:50:57.851 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.851 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.852 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:57.852 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051209+0000", 2024-09-21T15:50:57.852 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2636000000000001e-05 2024-09-21T15:50:57.852 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.852 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.852 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:57.852 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051217+0000", 2024-09-21T15:50:57.852 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5989999999999996e-06 2024-09-21T15:50:57.852 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.852 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.852 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:57.852 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108224+0000", 2024-09-21T15:50:57.852 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057007622000000001 2024-09-21T15:50:57.852 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.852 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.853 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:57.853 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108299+0000", 2024-09-21T15:50:57.853 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5099999999999996e-05 2024-09-21T15:50:57.853 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.853 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:57.853 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108368+0000", 2024-09-21T15:50:57.853 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8918000000000007e-05 2024-09-21T15:50:57.853 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:57.853 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:57.853 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:57.853 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.853 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116338 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 610304~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:50:57.853 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.051583+0000", 2024-09-21T15:50:57.853 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 61.716617595000002, 2024-09-21T15:50:57.854 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056967716000000002, 2024-09-21T15:50:57.854 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:50:57.854 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:50:57.854 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:50:57.854 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:50:57.854 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:57.854 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116338 2024-09-21T15:50:57.854 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.854 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:50:57.854 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.854 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:50:57.854 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051583+0000", 2024-09-21T15:50:57.854 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:57.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.855 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:50:57.855 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051583+0000", 2024-09-21T15:50:57.855 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:50:57.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.855 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:50:57.855 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051223+0000", 2024-09-21T15:50:57.855 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.99964 2024-09-21T15:50:57.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.855 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:50:57.855 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051587+0000", 2024-09-21T15:50:57.855 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00036352999999999998 2024-09-21T15:50:57.856 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.856 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.856 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:50:57.856 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051597+0000", 2024-09-21T15:50:57.856 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.893e-06 2024-09-21T15:50:57.856 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.856 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.856 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:50:57.856 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051603+0000", 2024-09-21T15:50:57.856 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0519999999999998e-06 2024-09-21T15:50:57.856 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.856 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.856 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:50:57.856 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108394+0000", 2024-09-21T15:50:57.856 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056791531999999999 2024-09-21T15:50:57.856 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.857 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.857 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:50:57.857 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108480+0000", 2024-09-21T15:50:57.857 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.577e-05 2024-09-21T15:50:57.857 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:50:57.857 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:50:57.857 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:50:57.857 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108551+0000", 2024-09-21T15:50:57.857 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0722000000000005e-05 2024-09-21T15:50:57.857 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:57.857 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:57.857 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:57.857 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:50:57.857 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:50:57.857 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:50:57.859 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:50:57.859 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:57.858+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:57.959 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:57.958+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:57.964 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:57.964 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:57.964 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:50:57.964 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:57.973 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:50:58.060 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:50:58.059+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:50:58.115 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:58.115 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:58.115 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:50:58.115 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:50:58.115 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:58.123 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:50:58.161 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:58.160+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:58.262 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:58.260+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:58.267 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:58.268 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:50:58.268 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:50:58.268 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:50:58.268 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.268 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155031 3.1c 3.8612741c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:58.268 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448748+0000", 2024-09-21T15:50:58.268 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 35.817201967999999, 2024-09-21T15:50:58.268 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064010286999999999, 2024-09-21T15:50:58.268 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:58.268 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:58.268 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:58.268 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:58.268 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:58.268 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155031 2024-09-21T15:50:58.269 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.269 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:58.269 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.269 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:58.269 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448748+0000", 2024-09-21T15:50:58.269 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.269 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.269 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.269 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:58.269 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448748+0000", 2024-09-21T15:50:58.269 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.269 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.269 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.269 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:58.269 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448747+0000", 2024-09-21T15:50:58.269 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:58.270 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.270 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.270 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:58.270 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448753+0000", 2024-09-21T15:50:58.270 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.755e-06 2024-09-21T15:50:58.270 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.270 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.270 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:58.270 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448763+0000", 2024-09-21T15:50:58.270 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7659999999999993e-06 2024-09-21T15:50:58.270 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.270 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.270 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:58.270 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448769+0000", 2024-09-21T15:50:58.270 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6060000000000002e-06 2024-09-21T15:50:58.271 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.271 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.271 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:58.271 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512750+0000", 2024-09-21T15:50:58.271 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.063981298000000006 2024-09-21T15:50:58.271 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.271 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.271 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:58.271 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512758+0000", 2024-09-21T15:50:58.271 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1669999999999999e-06 2024-09-21T15:50:58.271 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.271 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:58.271 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.271 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.271 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.272 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155091 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:58.272 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448916+0000", 2024-09-21T15:50:58.272 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 35.817034122999999, 2024-09-21T15:50:58.272 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064006309999999997, 2024-09-21T15:50:58.272 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:58.272 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:58.272 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:58.272 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:58.272 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:58.272 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155091 2024-09-21T15:50:58.272 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.272 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:58.272 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.272 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:58.272 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448916+0000", 2024-09-21T15:50:58.273 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.273 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.273 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.273 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:58.273 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448916+0000", 2024-09-21T15:50:58.273 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.273 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.273 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.273 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:58.273 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448915+0000", 2024-09-21T15:50:58.273 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:58.273 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.273 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.273 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:58.273 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448917+0000", 2024-09-21T15:50:58.274 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2400000000000002e-06 2024-09-21T15:50:58.274 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.274 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.274 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:58.274 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448923+0000", 2024-09-21T15:50:58.274 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0580000000000003e-06 2024-09-21T15:50:58.274 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.274 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.274 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:58.274 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448927+0000", 2024-09-21T15:50:58.274 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.031e-06 2024-09-21T15:50:58.274 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.274 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.274 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:58.274 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512916+0000", 2024-09-21T15:50:58.275 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.063988456999999999 2024-09-21T15:50:58.275 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.275 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.275 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:58.275 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512922+0000", 2024-09-21T15:50:58.275 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9009999999999999e-06 2024-09-21T15:50:58.275 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.275 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:58.275 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.275 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.275 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.275 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155092 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:58.275 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448931+0000", 2024-09-21T15:50:58.275 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 35.817018341000001, 2024-09-21T15:50:58.276 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064022793999999994, 2024-09-21T15:50:58.276 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:58.276 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:58.276 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:58.276 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:58.276 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:58.276 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155092 2024-09-21T15:50:58.276 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.276 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:58.276 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.276 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:58.276 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448931+0000", 2024-09-21T15:50:58.276 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.276 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.276 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.276 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:58.276 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448931+0000", 2024-09-21T15:50:58.277 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.277 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.277 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.277 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:58.277 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448931+0000", 2024-09-21T15:50:58.277 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:58.277 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.277 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.277 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:58.277 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448933+0000", 2024-09-21T15:50:58.277 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1799999999999999e-06 2024-09-21T15:50:58.277 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.277 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.277 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:58.277 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448939+0000", 2024-09-21T15:50:58.277 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.604e-06 2024-09-21T15:50:58.278 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.278 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.278 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:58.278 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448943+0000", 2024-09-21T15:50:58.278 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0740000000000003e-06 2024-09-21T15:50:58.278 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.278 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.278 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:58.278 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512946+0000", 2024-09-21T15:50:58.278 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064003612000000001 2024-09-21T15:50:58.278 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.278 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.278 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:58.278 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512954+0000", 2024-09-21T15:50:58.278 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6690000000000006e-06 2024-09-21T15:50:58.279 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.279 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:58.279 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.279 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.279 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.279 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155093 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:58.279 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448947+0000", 2024-09-21T15:50:58.279 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 35.817002967000001, 2024-09-21T15:50:58.279 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064020717000000005, 2024-09-21T15:50:58.279 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:58.279 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:58.279 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:58.279 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:58.279 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:58.279 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155093 2024-09-21T15:50:58.279 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.280 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:58.280 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.280 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:58.280 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448947+0000", 2024-09-21T15:50:58.280 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.280 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.280 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.280 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:58.280 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448947+0000", 2024-09-21T15:50:58.280 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.280 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.280 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.280 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:58.280 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448946+0000", 2024-09-21T15:50:58.280 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:58.280 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.281 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.281 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:58.281 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448949+0000", 2024-09-21T15:50:58.281 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3860000000000001e-06 2024-09-21T15:50:58.281 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.281 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.281 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:58.281 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448955+0000", 2024-09-21T15:50:58.281 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.823e-06 2024-09-21T15:50:58.281 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.281 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.281 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:58.281 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448959+0000", 2024-09-21T15:50:58.281 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2799999999999997e-06 2024-09-21T15:50:58.281 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.282 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.282 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:58.282 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512964+0000", 2024-09-21T15:50:58.282 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064004741000000004 2024-09-21T15:50:58.282 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.282 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.282 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:58.282 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512967+0000", 2024-09-21T15:50:58.282 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.839e-06 2024-09-21T15:50:58.282 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.282 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:58.282 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.282 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.282 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.282 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155094 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:58.283 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448962+0000", 2024-09-21T15:50:58.283 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 35.816987343999998, 2024-09-21T15:50:58.283 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064037378000000006, 2024-09-21T15:50:58.283 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:58.283 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:58.283 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:58.283 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:58.283 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:58.283 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155094 2024-09-21T15:50:58.283 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.283 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:58.283 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.283 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:58.283 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448962+0000", 2024-09-21T15:50:58.283 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.284 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.284 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.284 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:58.284 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448962+0000", 2024-09-21T15:50:58.284 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.284 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.284 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.284 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:58.284 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448962+0000", 2024-09-21T15:50:58.284 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:58.284 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.284 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.284 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:58.284 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448968+0000", 2024-09-21T15:50:58.285 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9710000000000001e-06 2024-09-21T15:50:58.285 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.285 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.285 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:58.285 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448974+0000", 2024-09-21T15:50:58.285 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7230000000000001e-06 2024-09-21T15:50:58.285 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.285 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.285 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:58.285 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448978+0000", 2024-09-21T15:50:58.285 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3749999999999996e-06 2024-09-21T15:50:58.285 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.285 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.285 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:58.286 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512992+0000", 2024-09-21T15:50:58.286 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064013525000000002 2024-09-21T15:50:58.286 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.286 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.286 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:58.286 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513000+0000", 2024-09-21T15:50:58.286 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0830000000000004e-06 2024-09-21T15:50:58.286 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.286 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:58.286 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.286 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.286 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.286 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155095 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:58.286 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448982+0000", 2024-09-21T15:50:58.287 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 35.816968062000001, 2024-09-21T15:50:58.287 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064041904999999996, 2024-09-21T15:50:58.287 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:58.287 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:58.287 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:58.287 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:58.287 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:58.287 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155095 2024-09-21T15:50:58.287 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.287 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:58.287 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.287 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:58.287 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448982+0000", 2024-09-21T15:50:58.287 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.288 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.288 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.288 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:58.288 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448982+0000", 2024-09-21T15:50:58.288 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.288 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.288 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.288 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:58.288 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448981+0000", 2024-09-21T15:50:58.288 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:58.288 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.288 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.288 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:58.288 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448983+0000", 2024-09-21T15:50:58.289 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.905e-06 2024-09-21T15:50:58.289 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.289 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.289 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:58.289 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448989+0000", 2024-09-21T15:50:58.289 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.536e-06 2024-09-21T15:50:58.289 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.289 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.289 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:58.289 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448993+0000", 2024-09-21T15:50:58.289 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2390000000000004e-06 2024-09-21T15:50:58.289 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.289 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.289 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:58.290 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513015+0000", 2024-09-21T15:50:58.290 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064022081999999994 2024-09-21T15:50:58.290 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.290 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.290 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:58.290 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513023+0000", 2024-09-21T15:50:58.290 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4500000000000004e-06 2024-09-21T15:50:58.290 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.290 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:58.290 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.290 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.290 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.290 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155096 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:58.291 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448996+0000", 2024-09-21T15:50:58.291 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 35.816953161000001, 2024-09-21T15:50:58.291 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064060868000000007, 2024-09-21T15:50:58.291 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:58.291 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:58.291 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:58.291 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:58.291 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:58.291 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155096 2024-09-21T15:50:58.291 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.291 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:58.291 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.291 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:58.291 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448996+0000", 2024-09-21T15:50:58.291 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.292 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.292 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.292 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:58.292 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448996+0000", 2024-09-21T15:50:58.292 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.292 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.292 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.292 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:58.292 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448996+0000", 2024-09-21T15:50:58.292 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:58.292 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.292 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.292 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:58.292 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448998+0000", 2024-09-21T15:50:58.293 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8780000000000001e-06 2024-09-21T15:50:58.293 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.293 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.293 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:58.293 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449003+0000", 2024-09-21T15:50:58.293 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4099999999999999e-06 2024-09-21T15:50:58.293 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.293 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.293 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:58.293 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449008+0000", 2024-09-21T15:50:58.293 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.335e-06 2024-09-21T15:50:58.293 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.293 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.293 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:58.293 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513049+0000", 2024-09-21T15:50:58.293 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064041059999999997 2024-09-21T15:50:58.294 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.294 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.294 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:58.294 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513057+0000", 2024-09-21T15:50:58.294 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5050000000000007e-06 2024-09-21T15:50:58.294 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.294 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:58.294 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.294 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.294 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.294 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155097 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:58.294 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449019+0000", 2024-09-21T15:50:58.294 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 35.816930739999997, 2024-09-21T15:50:58.294 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064052488000000005, 2024-09-21T15:50:58.295 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:58.295 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:58.295 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:58.295 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:58.295 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:58.295 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155097 2024-09-21T15:50:58.295 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.295 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:58.295 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.295 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:58.295 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449019+0000", 2024-09-21T15:50:58.295 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.295 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.295 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.295 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:58.296 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449019+0000", 2024-09-21T15:50:58.296 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.296 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.296 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.296 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:58.296 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449019+0000", 2024-09-21T15:50:58.296 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:58.296 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.296 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.296 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:58.296 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449021+0000", 2024-09-21T15:50:58.296 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.244e-06 2024-09-21T15:50:58.296 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.296 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.296 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:58.297 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449027+0000", 2024-09-21T15:50:58.297 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8619999999999999e-06 2024-09-21T15:50:58.297 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.297 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.297 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:58.297 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449031+0000", 2024-09-21T15:50:58.297 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1359999999999999e-06 2024-09-21T15:50:58.297 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.297 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.297 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:58.297 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513067+0000", 2024-09-21T15:50:58.297 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064036391999999998 2024-09-21T15:50:58.297 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.297 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.298 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:58.298 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513071+0000", 2024-09-21T15:50:58.298 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1989999999999999e-06 2024-09-21T15:50:58.298 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.298 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:58.298 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.298 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.298 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.298 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155098 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:58.298 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449035+0000", 2024-09-21T15:50:58.298 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 35.816915143000003, 2024-09-21T15:50:58.298 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064071761000000005, 2024-09-21T15:50:58.298 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:58.298 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:58.298 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:58.299 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:58.299 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:58.299 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155098 2024-09-21T15:50:58.299 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.299 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:58.299 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.299 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:58.299 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449035+0000", 2024-09-21T15:50:58.299 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.299 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.299 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.299 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:58.299 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449035+0000", 2024-09-21T15:50:58.299 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.299 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.299 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.300 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:58.300 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449034+0000", 2024-09-21T15:50:58.300 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:58.300 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.300 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.300 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:58.300 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449036+0000", 2024-09-21T15:50:58.300 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9989999999999998e-06 2024-09-21T15:50:58.300 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.300 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.300 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:58.300 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449042+0000", 2024-09-21T15:50:58.300 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6010000000000002e-06 2024-09-21T15:50:58.300 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.301 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.301 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:58.301 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449047+0000", 2024-09-21T15:50:58.301 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3600000000000004e-06 2024-09-21T15:50:58.301 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.301 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.301 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:58.301 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513098+0000", 2024-09-21T15:50:58.301 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064050544000000001 2024-09-21T15:50:58.301 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.301 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.301 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:58.301 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513106+0000", 2024-09-21T15:50:58.301 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5650000000000002e-06 2024-09-21T15:50:58.302 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.302 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:58.302 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.302 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.302 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.302 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155099 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:58.302 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449051+0000", 2024-09-21T15:50:58.302 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 35.816898602000002, 2024-09-21T15:50:58.302 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064069388000000005, 2024-09-21T15:50:58.302 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:58.302 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:58.302 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:58.302 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:58.302 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:58.302 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155099 2024-09-21T15:50:58.303 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.303 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:58.303 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.303 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:58.303 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449051+0000", 2024-09-21T15:50:58.303 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.303 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.303 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.303 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:58.303 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449051+0000", 2024-09-21T15:50:58.303 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.303 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.303 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.303 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:58.303 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449051+0000", 2024-09-21T15:50:58.304 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:58.304 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.304 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.304 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:58.304 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449053+0000", 2024-09-21T15:50:58.304 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2620000000000001e-06 2024-09-21T15:50:58.304 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.304 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.304 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:58.304 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449058+0000", 2024-09-21T15:50:58.304 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1680000000000003e-06 2024-09-21T15:50:58.304 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.304 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.304 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:58.304 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449062+0000", 2024-09-21T15:50:58.304 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9979999999999997e-06 2024-09-21T15:50:58.305 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.305 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.305 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:58.305 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513116+0000", 2024-09-21T15:50:58.305 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064053761000000001 2024-09-21T15:50:58.305 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.305 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.305 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:58.305 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513120+0000", 2024-09-21T15:50:58.305 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5399999999999997e-06 2024-09-21T15:50:58.305 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.305 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:58.305 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.305 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.305 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.305 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155100 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:58.306 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449101+0000", 2024-09-21T15:50:58.306 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 35.816848847000003, 2024-09-21T15:50:58.306 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064079585999999994, 2024-09-21T15:50:58.306 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:58.306 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:58.306 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:58.306 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:58.306 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:58.306 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155100 2024-09-21T15:50:58.306 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.306 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:58.306 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.306 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:58.306 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449101+0000", 2024-09-21T15:50:58.306 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.306 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.307 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.307 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:58.307 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449101+0000", 2024-09-21T15:50:58.307 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.307 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.307 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.307 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:58.307 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449100+0000", 2024-09-21T15:50:58.307 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-21T15:50:58.307 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.307 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.307 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:58.307 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449105+0000", 2024-09-21T15:50:58.307 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6700000000000002e-06 2024-09-21T15:50:58.307 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.307 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.308 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:58.308 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449117+0000", 2024-09-21T15:50:58.308 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1841000000000001e-05 2024-09-21T15:50:58.308 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.308 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.308 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:58.308 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449124+0000", 2024-09-21T15:50:58.308 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4660000000000002e-06 2024-09-21T15:50:58.308 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.308 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.308 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:58.308 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513163+0000", 2024-09-21T15:50:58.308 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064039053999999998 2024-09-21T15:50:58.308 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.308 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.308 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:58.309 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513180+0000", 2024-09-21T15:50:58.309 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7333000000000002e-05 2024-09-21T15:50:58.309 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.309 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:58.309 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.309 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.309 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.309 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155101 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:58.309 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449166+0000", 2024-09-21T15:50:58.309 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 35.816784142000003, 2024-09-21T15:50:58.309 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064027639999999997, 2024-09-21T15:50:58.309 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:58.309 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:58.309 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:58.309 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:58.309 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:58.310 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155101 2024-09-21T15:50:58.310 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.310 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:58.310 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.310 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:58.310 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449166+0000", 2024-09-21T15:50:58.310 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.310 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.310 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.310 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:58.310 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449166+0000", 2024-09-21T15:50:58.310 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.310 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.310 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.310 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:58.310 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449165+0000", 2024-09-21T15:50:58.311 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:58.311 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.311 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.311 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:58.311 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449168+0000", 2024-09-21T15:50:58.311 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5099999999999999e-06 2024-09-21T15:50:58.311 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.311 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.311 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:58.311 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449177+0000", 2024-09-21T15:50:58.311 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6659999999999997e-06 2024-09-21T15:50:58.311 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.311 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.311 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:58.312 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449182+0000", 2024-09-21T15:50:58.312 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1320000000000002e-06 2024-09-21T15:50:58.312 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.312 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.312 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:58.312 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513189+0000", 2024-09-21T15:50:58.312 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064007231999999997 2024-09-21T15:50:58.312 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.312 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.312 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:58.312 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513193+0000", 2024-09-21T15:50:58.312 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.692e-06 2024-09-21T15:50:58.312 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.312 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:58.313 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.313 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.313 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.313 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155134 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:58.313 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449187+0000", 2024-09-21T15:50:58.313 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 35.816762554, 2024-09-21T15:50:58.313 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064031331999999996, 2024-09-21T15:50:58.313 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:58.313 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:58.313 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:58.313 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:58.313 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:58.313 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155134 2024-09-21T15:50:58.313 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.314 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:58.314 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.314 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:58.314 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449187+0000", 2024-09-21T15:50:58.314 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.314 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.314 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.314 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:58.314 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449187+0000", 2024-09-21T15:50:58.314 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.314 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.314 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.314 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:58.315 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449187+0000", 2024-09-21T15:50:58.315 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:58.315 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.315 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.315 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:58.315 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449189+0000", 2024-09-21T15:50:58.315 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5469999999999999e-06 2024-09-21T15:50:58.315 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.315 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.315 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:58.315 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449196+0000", 2024-09-21T15:50:58.315 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7009999999999999e-06 2024-09-21T15:50:58.315 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.315 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.315 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:58.316 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449202+0000", 2024-09-21T15:50:58.316 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7409999999999998e-06 2024-09-21T15:50:58.316 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.316 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.316 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:58.316 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513214+0000", 2024-09-21T15:50:58.316 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064012301999999993 2024-09-21T15:50:58.316 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.316 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.316 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:58.316 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513218+0000", 2024-09-21T15:50:58.316 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5709999999999999e-06 2024-09-21T15:50:58.316 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.316 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:58.317 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.317 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.317 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.317 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155135 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:58.317 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449207+0000", 2024-09-21T15:50:58.317 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 35.816743043999999, 2024-09-21T15:50:58.317 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064029249999999996, 2024-09-21T15:50:58.317 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:58.317 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:58.317 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:58.317 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:58.317 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:58.317 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155135 2024-09-21T15:50:58.317 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.318 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:58.318 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.318 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:58.318 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449207+0000", 2024-09-21T15:50:58.318 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.318 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.318 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.318 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:58.318 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449207+0000", 2024-09-21T15:50:58.318 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.318 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.318 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.318 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:58.318 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449206+0000", 2024-09-21T15:50:58.319 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:58.319 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.319 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.319 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:58.319 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449209+0000", 2024-09-21T15:50:58.319 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5150000000000001e-06 2024-09-21T15:50:58.319 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.319 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.319 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:58.319 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449215+0000", 2024-09-21T15:50:58.319 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9329999999999998e-06 2024-09-21T15:50:58.319 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.319 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.320 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:58.320 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449220+0000", 2024-09-21T15:50:58.320 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0819999999999998e-06 2024-09-21T15:50:58.320 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.320 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.320 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:58.320 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513232+0000", 2024-09-21T15:50:58.320 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064012052 2024-09-21T15:50:58.320 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.320 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.320 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:58.320 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513236+0000", 2024-09-21T15:50:58.320 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2080000000000002e-06 2024-09-21T15:50:58.320 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.320 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:58.320 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.321 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.321 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.321 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155136 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:58.321 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449224+0000", 2024-09-21T15:50:58.321 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 35.816725480000002, 2024-09-21T15:50:58.321 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064053025999999999, 2024-09-21T15:50:58.321 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:58.321 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:58.321 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:58.321 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:58.321 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:58.321 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155136 2024-09-21T15:50:58.321 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.321 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:58.321 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.322 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:58.322 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449224+0000", 2024-09-21T15:50:58.322 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.322 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.322 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.322 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:58.322 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449224+0000", 2024-09-21T15:50:58.322 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.322 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.322 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.322 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:58.322 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449224+0000", 2024-09-21T15:50:58.322 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:58.322 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.322 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.323 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:58.323 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449226+0000", 2024-09-21T15:50:58.323 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2699999999999999e-06 2024-09-21T15:50:58.323 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.323 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.323 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:58.323 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449233+0000", 2024-09-21T15:50:58.323 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4289999999999998e-06 2024-09-21T15:50:58.323 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.323 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.323 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:58.323 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449238+0000", 2024-09-21T15:50:58.323 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0730000000000004e-06 2024-09-21T15:50:58.323 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.323 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.324 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:58.324 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513269+0000", 2024-09-21T15:50:58.324 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064030999000000005 2024-09-21T15:50:58.324 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.324 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.324 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:58.324 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513277+0000", 2024-09-21T15:50:58.324 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5520000000000005e-06 2024-09-21T15:50:58.324 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.324 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:58.324 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.324 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.324 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.324 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155137 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:58.324 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449263+0000", 2024-09-21T15:50:58.325 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 35.816686347999998, 2024-09-21T15:50:58.325 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.06402803, 2024-09-21T15:50:58.325 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:58.325 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:58.325 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:58.325 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:58.325 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:58.325 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155137 2024-09-21T15:50:58.325 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.325 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:58.325 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.325 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:58.325 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449263+0000", 2024-09-21T15:50:58.325 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.326 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.326 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.326 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:58.326 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449263+0000", 2024-09-21T15:50:58.326 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.326 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.326 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.326 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:58.326 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449262+0000", 2024-09-21T15:50:58.326 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-21T15:50:58.326 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.326 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.326 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:58.326 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449267+0000", 2024-09-21T15:50:58.327 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.933e-06 2024-09-21T15:50:58.327 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.327 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.327 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:58.327 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449279+0000", 2024-09-21T15:50:58.327 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.181e-05 2024-09-21T15:50:58.327 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.327 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.327 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:58.327 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449286+0000", 2024-09-21T15:50:58.327 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4749999999999998e-06 2024-09-21T15:50:58.327 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.327 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.327 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:58.327 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513287+0000", 2024-09-21T15:50:58.327 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064001369000000002 2024-09-21T15:50:58.328 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.328 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.328 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:58.328 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513291+0000", 2024-09-21T15:50:58.328 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3000000000000003e-06 2024-09-21T15:50:58.328 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.328 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:58.328 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.328 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.328 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.328 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155138 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:58.328 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449299+0000", 2024-09-21T15:50:58.328 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 35.816650856000003, 2024-09-21T15:50:58.328 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064028359000000007, 2024-09-21T15:50:58.329 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:58.329 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:58.329 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:58.329 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:58.329 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:58.329 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155138 2024-09-21T15:50:58.329 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.329 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:58.329 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.329 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:58.329 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449299+0000", 2024-09-21T15:50:58.329 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.329 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.329 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.329 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:58.329 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449299+0000", 2024-09-21T15:50:58.330 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.330 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.330 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.330 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:58.330 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449298+0000", 2024-09-21T15:50:58.330 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:58.330 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.330 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.330 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:58.330 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449301+0000", 2024-09-21T15:50:58.330 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0230000000000001e-06 2024-09-21T15:50:58.330 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.330 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.330 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:58.330 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449308+0000", 2024-09-21T15:50:58.331 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3119999999999999e-06 2024-09-21T15:50:58.331 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.331 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.331 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:58.331 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449313+0000", 2024-09-21T15:50:58.331 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8980000000000004e-06 2024-09-21T15:50:58.331 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.331 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.331 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:58.331 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513318+0000", 2024-09-21T15:50:58.331 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064005612000000003 2024-09-21T15:50:58.331 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.331 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.331 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:58.331 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513327+0000", 2024-09-21T15:50:58.332 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0100000000000001e-06 2024-09-21T15:50:58.332 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.332 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:58.332 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.332 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.332 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.332 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155142 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:58.332 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449392+0000", 2024-09-21T15:50:58.332 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 35.816558098000002, 2024-09-21T15:50:58.332 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064012464000000005, 2024-09-21T15:50:58.332 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:58.332 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:58.332 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:58.332 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:58.332 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:58.332 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155142 2024-09-21T15:50:58.333 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.333 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:58.333 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.333 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:58.333 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449392+0000", 2024-09-21T15:50:58.333 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.333 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.333 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.333 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:58.333 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449392+0000", 2024-09-21T15:50:58.333 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.333 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.333 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.333 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:58.333 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449391+0000", 2024-09-21T15:50:58.333 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:58.334 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.334 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.334 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:58.334 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449393+0000", 2024-09-21T15:50:58.334 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.26e-06 2024-09-21T15:50:58.334 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.334 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.334 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:58.334 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449399+0000", 2024-09-21T15:50:58.334 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6269999999999997e-06 2024-09-21T15:50:58.334 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.334 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.334 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:58.334 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449403+0000", 2024-09-21T15:50:58.335 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3000000000000003e-06 2024-09-21T15:50:58.335 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.335 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.335 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:58.335 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513396+0000", 2024-09-21T15:50:58.335 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.063992568999999999 2024-09-21T15:50:58.335 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.335 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.335 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:58.335 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513404+0000", 2024-09-21T15:50:58.335 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1100000000000003e-06 2024-09-21T15:50:58.335 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.335 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:58.335 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.336 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.336 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.336 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155143 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:58.336 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449407+0000", 2024-09-21T15:50:58.336 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 35.816542511999998, 2024-09-21T15:50:58.336 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064009916, 2024-09-21T15:50:58.336 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:58.336 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:58.336 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:58.336 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:58.336 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:58.336 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155143 2024-09-21T15:50:58.336 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.336 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:58.336 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.336 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:58.337 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449407+0000", 2024-09-21T15:50:58.337 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.337 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.337 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.337 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:58.337 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449407+0000", 2024-09-21T15:50:58.337 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.337 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.337 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.337 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:58.337 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449407+0000", 2024-09-21T15:50:58.337 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:58.337 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.337 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.337 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:58.337 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449409+0000", 2024-09-21T15:50:58.338 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1090000000000001e-06 2024-09-21T15:50:58.338 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.338 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.338 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:58.338 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449413+0000", 2024-09-21T15:50:58.338 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6650000000000001e-06 2024-09-21T15:50:58.338 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.338 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.338 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:58.338 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449415+0000", 2024-09-21T15:50:58.338 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5459999999999998e-06 2024-09-21T15:50:58.338 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.338 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.338 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:58.338 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513413+0000", 2024-09-21T15:50:58.339 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.063998037999999993 2024-09-21T15:50:58.339 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.339 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.339 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:58.339 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513417+0000", 2024-09-21T15:50:58.339 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8789999999999996e-06 2024-09-21T15:50:58.339 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.339 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:58.339 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.339 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.339 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.339 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155144 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:58.339 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449427+0000", 2024-09-21T15:50:58.339 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 35.816522503999998, 2024-09-21T15:50:58.339 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064022531999999993, 2024-09-21T15:50:58.340 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:58.340 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:50:58.340 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:58.340 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:58.340 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:58.340 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155144 2024-09-21T15:50:58.340 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.340 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:58.340 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.340 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:58.340 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449427+0000", 2024-09-21T15:50:58.340 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.340 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.340 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.340 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:58.341 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449427+0000", 2024-09-21T15:50:58.341 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.341 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.341 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.341 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:58.341 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449427+0000", 2024-09-21T15:50:58.341 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:58.341 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.341 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.341 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:58.341 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449430+0000", 2024-09-21T15:50:58.341 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3579999999999999e-06 2024-09-21T15:50:58.341 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.341 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.341 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:58.342 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449437+0000", 2024-09-21T15:50:58.342 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4580000000000004e-06 2024-09-21T15:50:58.342 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.342 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.342 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:58.342 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449443+0000", 2024-09-21T15:50:58.342 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5609999999999998e-06 2024-09-21T15:50:58.342 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.342 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.342 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:58.342 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513441+0000", 2024-09-21T15:50:58.342 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.063998136999999997 2024-09-21T15:50:58.342 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.342 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.343 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:58.343 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513450+0000", 2024-09-21T15:50:58.343 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4950000000000008e-06 2024-09-21T15:50:58.343 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.343 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:58.343 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.343 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.343 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:58.343 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:58.344 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:50:58.362 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:58.362+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:58.463 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:58.462+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:50:58.487 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:58.487 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:58.487 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:50:58.487 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:58.495 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:50:58.563 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:50:58.562+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:50:58.596 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:58.596 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:58.596 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:50:58.596 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:50:58.596 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:58.604 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:50:58.688 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:58.687+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:58.747 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:58.747 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:50:58.747 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:50:58.748 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:50:58.748 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.748 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120111 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 321536~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:58.748 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572216+0000", 2024-09-21T15:50:58.748 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 55.173483113000003, 2024-09-21T15:50:58.748 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054657399000000002, 2024-09-21T15:50:58.748 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:58.748 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:58.748 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:58.748 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:58.748 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:58.748 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120111 2024-09-21T15:50:58.748 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.749 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:58.749 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.749 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:58.749 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572216+0000", 2024-09-21T15:50:58.749 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.749 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.749 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.749 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:58.749 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572216+0000", 2024-09-21T15:50:58.749 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.749 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.749 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.749 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:58.749 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572216+0000", 2024-09-21T15:50:58.749 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:58.750 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.750 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.750 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:58.750 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572218+0000", 2024-09-21T15:50:58.750 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1840000000000002e-06 2024-09-21T15:50:58.750 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.750 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.750 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:58.750 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572224+0000", 2024-09-21T15:50:58.750 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7880000000000003e-06 2024-09-21T15:50:58.750 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.750 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.750 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:58.750 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572229+0000", 2024-09-21T15:50:58.750 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8239999999999999e-06 2024-09-21T15:50:58.751 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.751 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.751 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:58.751 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626680+0000", 2024-09-21T15:50:58.751 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054451431000000002 2024-09-21T15:50:58.751 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.751 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.751 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:58.751 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626755+0000", 2024-09-21T15:50:58.751 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4641000000000006e-05 2024-09-21T15:50:58.751 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.751 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.751 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:58.751 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626874+0000", 2024-09-21T15:50:58.751 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00011886 2024-09-21T15:50:58.751 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.752 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:58.752 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.752 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.752 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.752 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120112 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 342016~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:58.752 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572233+0000", 2024-09-21T15:50:58.752 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 55.173466744999999, 2024-09-21T15:50:58.752 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054823997999999999, 2024-09-21T15:50:58.752 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:58.752 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:58.752 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:58.752 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:58.752 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:58.752 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120112 2024-09-21T15:50:58.752 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.753 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:58.753 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.753 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:58.753 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572233+0000", 2024-09-21T15:50:58.753 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.753 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.753 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.753 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:58.753 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572233+0000", 2024-09-21T15:50:58.753 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.753 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.753 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.753 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:58.753 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572232+0000", 2024-09-21T15:50:58.753 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:58.754 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.754 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.754 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:58.754 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572234+0000", 2024-09-21T15:50:58.754 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0660000000000002e-06 2024-09-21T15:50:58.754 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.754 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.754 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:58.754 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572242+0000", 2024-09-21T15:50:58.754 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0430000000000002e-06 2024-09-21T15:50:58.754 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.754 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.754 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:58.754 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572246+0000", 2024-09-21T15:50:58.755 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9080000000000003e-06 2024-09-21T15:50:58.755 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.755 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.755 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:58.755 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626916+0000", 2024-09-21T15:50:58.755 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054669482999999998 2024-09-21T15:50:58.755 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.755 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.755 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:58.755 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626977+0000", 2024-09-21T15:50:58.755 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0597000000000003e-05 2024-09-21T15:50:58.755 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.755 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.755 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:58.755 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627057+0000", 2024-09-21T15:50:58.756 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0258000000000003e-05 2024-09-21T15:50:58.756 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.756 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:58.756 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.756 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.756 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.756 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120113 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 362496~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:58.756 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572253+0000", 2024-09-21T15:50:58.756 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 55.173446759999997, 2024-09-21T15:50:58.756 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.05503044, 2024-09-21T15:50:58.756 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:58.756 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:58.756 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:58.756 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:58.756 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:58.756 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120113 2024-09-21T15:50:58.757 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.757 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:58.757 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.757 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:58.757 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572253+0000", 2024-09-21T15:50:58.757 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.757 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.757 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.757 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:58.757 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572253+0000", 2024-09-21T15:50:58.757 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.757 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.757 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.757 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:58.757 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572252+0000", 2024-09-21T15:50:58.758 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:58.758 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.758 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.758 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:58.758 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572258+0000", 2024-09-21T15:50:58.758 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6169999999999998e-06 2024-09-21T15:50:58.758 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.758 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.758 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:58.758 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572266+0000", 2024-09-21T15:50:58.758 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9410000000000007e-06 2024-09-21T15:50:58.758 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.758 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.758 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:58.758 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572272+0000", 2024-09-21T15:50:58.759 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.241e-06 2024-09-21T15:50:58.759 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.759 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.759 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:58.759 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627087+0000", 2024-09-21T15:50:58.759 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054814850999999998 2024-09-21T15:50:58.759 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.759 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.759 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:58.759 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627162+0000", 2024-09-21T15:50:58.759 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5012999999999995e-05 2024-09-21T15:50:58.759 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.759 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.759 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:58.759 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627283+0000", 2024-09-21T15:50:58.760 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012113 2024-09-21T15:50:58.760 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.760 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:58.760 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.760 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.760 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.760 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120114 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 382976~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:58.760 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572277+0000", 2024-09-21T15:50:58.760 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 55.173422742, 2024-09-21T15:50:58.760 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055184395999999997, 2024-09-21T15:50:58.760 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:58.760 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:58.760 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:58.760 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:58.760 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:58.760 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120114 2024-09-21T15:50:58.761 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.761 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:58.761 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.761 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:58.761 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572277+0000", 2024-09-21T15:50:58.761 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.761 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.761 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.761 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:58.761 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572277+0000", 2024-09-21T15:50:58.761 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.761 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.761 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.761 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:58.761 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572276+0000", 2024-09-21T15:50:58.762 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:58.762 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.762 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.762 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:58.762 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572279+0000", 2024-09-21T15:50:58.762 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.373e-06 2024-09-21T15:50:58.762 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.762 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.762 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:58.762 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572285+0000", 2024-09-21T15:50:58.762 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9359999999999996e-06 2024-09-21T15:50:58.762 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.762 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.762 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:58.763 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572289+0000", 2024-09-21T15:50:58.763 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.634e-06 2024-09-21T15:50:58.763 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.763 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.763 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:58.763 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627311+0000", 2024-09-21T15:50:58.763 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055021211 2024-09-21T15:50:58.763 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.763 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.763 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:58.763 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627360+0000", 2024-09-21T15:50:58.763 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9203999999999998e-05 2024-09-21T15:50:58.763 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.763 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.764 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:58.764 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627461+0000", 2024-09-21T15:50:58.764 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000101469 2024-09-21T15:50:58.764 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.764 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:58.764 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.764 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.764 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.764 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120115 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 399360~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:58.764 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572294+0000", 2024-09-21T15:50:58.764 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 55.173405434999999, 2024-09-21T15:50:58.764 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055400092999999997, 2024-09-21T15:50:58.764 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:58.764 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:58.764 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:58.765 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:58.765 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:58.765 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120115 2024-09-21T15:50:58.765 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.765 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:58.765 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.765 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:58.765 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572294+0000", 2024-09-21T15:50:58.765 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.765 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.765 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.765 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:58.765 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572294+0000", 2024-09-21T15:50:58.765 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.765 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.765 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.765 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:58.766 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572294+0000", 2024-09-21T15:50:58.766 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:58.766 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.766 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.766 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:58.766 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572296+0000", 2024-09-21T15:50:58.766 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1100000000000001e-06 2024-09-21T15:50:58.766 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.766 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.766 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:58.766 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572303+0000", 2024-09-21T15:50:58.766 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8580000000000002e-06 2024-09-21T15:50:58.766 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.766 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.766 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:58.767 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572314+0000", 2024-09-21T15:50:58.767 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.101e-05 2024-09-21T15:50:58.767 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.767 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.767 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:58.767 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627496+0000", 2024-09-21T15:50:58.767 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055182522999999997 2024-09-21T15:50:58.767 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.767 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.767 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:58.767 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627568+0000", 2024-09-21T15:50:58.767 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1569999999999994e-05 2024-09-21T15:50:58.767 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.767 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.767 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:58.768 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627694+0000", 2024-09-21T15:50:58.768 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000126372 2024-09-21T15:50:58.768 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.768 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:58.768 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.768 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.768 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.768 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120116 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 419840~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:58.768 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572318+0000", 2024-09-21T15:50:58.768 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 55.173381243999998, 2024-09-21T15:50:58.768 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055539416000000001, 2024-09-21T15:50:58.768 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:58.768 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:58.768 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:58.768 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:58.768 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:58.769 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120116 2024-09-21T15:50:58.769 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.769 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:58.769 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.769 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:58.769 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572318+0000", 2024-09-21T15:50:58.769 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.769 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.769 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.769 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:58.769 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572318+0000", 2024-09-21T15:50:58.769 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.769 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.769 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.769 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:58.769 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572318+0000", 2024-09-21T15:50:58.769 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:58.770 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.770 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.770 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:58.770 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572322+0000", 2024-09-21T15:50:58.770 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4839999999999998e-06 2024-09-21T15:50:58.770 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.770 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.770 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:58.770 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572329+0000", 2024-09-21T15:50:58.770 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0959999999999997e-06 2024-09-21T15:50:58.770 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.770 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.770 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:58.770 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572333+0000", 2024-09-21T15:50:58.770 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9359999999999998e-06 2024-09-21T15:50:58.771 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.771 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.771 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:58.771 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627726+0000", 2024-09-21T15:50:58.771 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055392829999999997 2024-09-21T15:50:58.771 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.771 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.771 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:58.771 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627779+0000", 2024-09-21T15:50:58.771 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2185999999999998e-05 2024-09-21T15:50:58.771 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.771 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.771 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:58.771 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627858+0000", 2024-09-21T15:50:58.771 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9190999999999994e-05 2024-09-21T15:50:58.771 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.772 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:58.772 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.772 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.772 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.772 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120117 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 440320~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:58.772 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572338+0000", 2024-09-21T15:50:58.772 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 55.173361931000002, 2024-09-21T15:50:58.772 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055680888999999997, 2024-09-21T15:50:58.772 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:58.772 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:58.772 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:58.772 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:58.772 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:58.772 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120117 2024-09-21T15:50:58.772 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.772 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:58.773 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.773 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:58.773 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572338+0000", 2024-09-21T15:50:58.773 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.773 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.773 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.773 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:58.773 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572338+0000", 2024-09-21T15:50:58.773 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.773 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.773 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.773 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:58.773 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572337+0000", 2024-09-21T15:50:58.773 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:58.773 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.774 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.774 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:58.774 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572339+0000", 2024-09-21T15:50:58.774 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9379999999999999e-06 2024-09-21T15:50:58.774 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.774 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.774 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:58.774 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572349+0000", 2024-09-21T15:50:58.774 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7780000000000002e-06 2024-09-21T15:50:58.774 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.774 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.774 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:58.774 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572354+0000", 2024-09-21T15:50:58.774 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8740000000000003e-06 2024-09-21T15:50:58.774 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.775 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.775 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:58.775 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627885+0000", 2024-09-21T15:50:58.775 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055531448999999997 2024-09-21T15:50:58.775 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.775 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.775 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:58.775 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627935+0000", 2024-09-21T15:50:58.775 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9348999999999998e-05 2024-09-21T15:50:58.775 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.775 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.775 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:58.775 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628018+0000", 2024-09-21T15:50:58.775 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3851999999999996e-05 2024-09-21T15:50:58.775 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.776 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:58.776 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.776 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.776 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.776 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120118 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 460800~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:58.776 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572359+0000", 2024-09-21T15:50:58.776 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 55.173340689, 2024-09-21T15:50:58.776 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055951714, 2024-09-21T15:50:58.776 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:58.776 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:58.776 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:58.776 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:58.776 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:58.776 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120118 2024-09-21T15:50:58.776 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.776 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:58.776 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.777 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:58.777 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572359+0000", 2024-09-21T15:50:58.777 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.777 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.777 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.777 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:58.777 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572359+0000", 2024-09-21T15:50:58.777 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.777 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.777 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.777 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:58.777 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572358+0000", 2024-09-21T15:50:58.777 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:58.777 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.777 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.777 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:58.778 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572361+0000", 2024-09-21T15:50:58.778 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.26e-06 2024-09-21T15:50:58.778 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.778 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.778 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:58.778 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572367+0000", 2024-09-21T15:50:58.778 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3640000000000004e-06 2024-09-21T15:50:58.778 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.778 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.778 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:58.778 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572376+0000", 2024-09-21T15:50:58.779 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2310000000000002e-06 2024-09-21T15:50:58.779 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.779 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.779 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:58.779 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628096+0000", 2024-09-21T15:50:58.779 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055719369999999997 2024-09-21T15:50:58.779 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.779 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.779 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:58.779 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628178+0000", 2024-09-21T15:50:58.779 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2667e-05 2024-09-21T15:50:58.779 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.779 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.779 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:58.779 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628311+0000", 2024-09-21T15:50:58.779 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000132175 2024-09-21T15:50:58.780 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.780 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:58.780 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.780 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.780 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.780 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120119 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 481280~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:58.780 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572381+0000", 2024-09-21T15:50:58.780 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 55.173318244999997, 2024-09-21T15:50:58.780 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056088791999999998, 2024-09-21T15:50:58.780 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:58.780 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:58.780 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:58.780 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:58.780 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:58.780 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120119 2024-09-21T15:50:58.780 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.781 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:58.781 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.781 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:58.781 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572381+0000", 2024-09-21T15:50:58.781 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.781 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.781 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.781 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:58.781 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572381+0000", 2024-09-21T15:50:58.781 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.781 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.781 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.781 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:58.781 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572381+0000", 2024-09-21T15:50:58.781 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:58.781 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.782 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.782 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:58.782 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572383+0000", 2024-09-21T15:50:58.782 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0619999999999999e-06 2024-09-21T15:50:58.782 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.782 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.782 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:58.782 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572391+0000", 2024-09-21T15:50:58.782 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8769999999999993e-06 2024-09-21T15:50:58.782 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.782 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.782 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:58.782 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572399+0000", 2024-09-21T15:50:58.782 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6240000000000008e-06 2024-09-21T15:50:58.782 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.783 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.783 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:58.783 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628344+0000", 2024-09-21T15:50:58.783 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055944828000000002 2024-09-21T15:50:58.783 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.783 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.783 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:58.783 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628393+0000", 2024-09-21T15:50:58.783 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8653000000000003e-05 2024-09-21T15:50:58.783 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.783 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.783 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:58.783 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628470+0000", 2024-09-21T15:50:58.783 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7121999999999998e-05 2024-09-21T15:50:58.784 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.784 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:58.784 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.784 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.784 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.784 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120120 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 501760~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:58.784 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572418+0000", 2024-09-21T15:50:58.784 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 55.173281746999997, 2024-09-21T15:50:58.784 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056209138999999998, 2024-09-21T15:50:58.784 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:58.784 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:58.784 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:58.784 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:58.784 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:58.784 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120120 2024-09-21T15:50:58.784 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.785 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:58.785 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.785 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:58.785 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572418+0000", 2024-09-21T15:50:58.785 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.785 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.785 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.785 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:58.785 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572418+0000", 2024-09-21T15:50:58.785 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.785 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.785 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.785 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:58.785 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572417+0000", 2024-09-21T15:50:58.785 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:58.785 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.786 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.786 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:58.786 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572420+0000", 2024-09-21T15:50:58.786 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5270000000000001e-06 2024-09-21T15:50:58.786 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.786 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.786 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:58.786 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572426+0000", 2024-09-21T15:50:58.786 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3380000000000001e-06 2024-09-21T15:50:58.786 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.786 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.786 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:58.786 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572431+0000", 2024-09-21T15:50:58.786 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.865e-06 2024-09-21T15:50:58.786 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.787 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.787 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:58.787 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628500+0000", 2024-09-21T15:50:58.787 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056068917000000003 2024-09-21T15:50:58.787 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.787 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.787 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:58.787 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628542+0000", 2024-09-21T15:50:58.787 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1761999999999998e-05 2024-09-21T15:50:58.787 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.787 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.787 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:58.787 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628627+0000", 2024-09-21T15:50:58.787 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5247999999999994e-05 2024-09-21T15:50:58.788 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.788 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:58.788 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.788 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.788 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.788 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120121 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 522240~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:58.788 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572630+0000", 2024-09-21T15:50:58.788 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 55.173069361000003, 2024-09-21T15:50:58.788 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056207209000000001, 2024-09-21T15:50:58.788 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:58.788 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:58.788 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:58.788 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:58.788 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:58.788 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120121 2024-09-21T15:50:58.789 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.789 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:58.789 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.789 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:58.789 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572630+0000", 2024-09-21T15:50:58.789 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.789 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.789 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.789 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:58.789 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572630+0000", 2024-09-21T15:50:58.789 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.789 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.789 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.789 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:58.789 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572435+0000", 2024-09-21T15:50:58.789 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998045 2024-09-21T15:50:58.790 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.790 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.790 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:58.790 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572633+0000", 2024-09-21T15:50:58.790 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000198786 2024-09-21T15:50:58.790 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.790 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.790 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:58.790 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572641+0000", 2024-09-21T15:50:58.790 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9449999999999993e-06 2024-09-21T15:50:58.790 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.790 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.790 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:58.790 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572648+0000", 2024-09-21T15:50:58.790 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1929999999999998e-06 2024-09-21T15:50:58.791 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.791 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.791 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:58.791 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628669+0000", 2024-09-21T15:50:58.791 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056021690999999998 2024-09-21T15:50:58.791 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.791 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.791 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:58.791 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628751+0000", 2024-09-21T15:50:58.791 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1710000000000002e-05 2024-09-21T15:50:58.791 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.791 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.791 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:58.791 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628837+0000", 2024-09-21T15:50:58.791 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6383000000000003e-05 2024-09-21T15:50:58.792 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.792 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:58.792 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.792 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.792 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.792 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120122 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 538624~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:58.792 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.573471+0000", 2024-09-21T15:50:58.792 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 55.172228115999999, 2024-09-21T15:50:58.792 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055563935000000002, 2024-09-21T15:50:58.792 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:58.792 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:58.792 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:58.792 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:58.792 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:58.792 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120122 2024-09-21T15:50:58.792 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.793 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:58.793 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.793 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:58.793 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573471+0000", 2024-09-21T15:50:58.793 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.793 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.793 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.793 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:58.793 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573471+0000", 2024-09-21T15:50:58.793 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.793 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.793 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.793 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:58.793 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572653+0000", 2024-09-21T15:50:58.793 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9991813 2024-09-21T15:50:58.794 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.794 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.794 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:58.794 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573475+0000", 2024-09-21T15:50:58.794 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00082196500000000002 2024-09-21T15:50:58.794 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.794 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.794 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:58.794 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573483+0000", 2024-09-21T15:50:58.794 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5939999999999994e-06 2024-09-21T15:50:58.794 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.794 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.794 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:58.794 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573489+0000", 2024-09-21T15:50:58.795 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1689999999999997e-06 2024-09-21T15:50:58.795 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.795 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.795 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:58.795 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628882+0000", 2024-09-21T15:50:58.795 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055392230000000001 2024-09-21T15:50:58.795 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.795 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.795 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:58.795 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628934+0000", 2024-09-21T15:50:58.795 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1870000000000003e-05 2024-09-21T15:50:58.795 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.795 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.795 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:58.795 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629035+0000", 2024-09-21T15:50:58.795 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000101753 2024-09-21T15:50:58.796 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.796 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:58.796 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.796 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.796 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.796 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120123 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 559104~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:58.796 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.573495+0000", 2024-09-21T15:50:58.796 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 55.172204417000003, 2024-09-21T15:50:58.796 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055704765000000003, 2024-09-21T15:50:58.796 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:58.796 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:58.796 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:58.796 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:58.796 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:58.797 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120123 2024-09-21T15:50:58.797 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.797 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:58.797 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.797 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:58.797 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573495+0000", 2024-09-21T15:50:58.797 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.797 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.797 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.797 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:58.797 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573495+0000", 2024-09-21T15:50:58.797 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.797 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.797 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.797 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:58.797 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573495+0000", 2024-09-21T15:50:58.798 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:58.798 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.798 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.798 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:58.798 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573497+0000", 2024-09-21T15:50:58.798 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3420000000000002e-06 2024-09-21T15:50:58.798 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.798 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.798 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:58.798 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573503+0000", 2024-09-21T15:50:58.798 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7030000000000003e-06 2024-09-21T15:50:58.798 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.798 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.798 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:58.799 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573511+0000", 2024-09-21T15:50:58.799 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0790000000000001e-06 2024-09-21T15:50:58.799 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.799 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.799 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:58.799 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629067+0000", 2024-09-21T15:50:58.799 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055556573999999997 2024-09-21T15:50:58.799 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.799 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.799 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:58.799 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629118+0000", 2024-09-21T15:50:58.799 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1053000000000001e-05 2024-09-21T15:50:58.799 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.799 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.799 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:58.800 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629200+0000", 2024-09-21T15:50:58.800 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1452999999999994e-05 2024-09-21T15:50:58.800 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.800 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:58.800 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.800 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.800 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.800 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120124 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 579584~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:58.800 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.574014+0000", 2024-09-21T15:50:58.800 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 55.171685377999999, 2024-09-21T15:50:58.800 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055440284999999999, 2024-09-21T15:50:58.800 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:58.800 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:58.800 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:58.800 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:58.801 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:58.801 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120124 2024-09-21T15:50:58.801 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.801 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:58.801 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.801 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:58.801 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574014+0000", 2024-09-21T15:50:58.801 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.801 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.801 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.801 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:58.801 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574014+0000", 2024-09-21T15:50:58.801 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.801 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.801 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.802 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:58.802 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573515+0000", 2024-09-21T15:50:58.802 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9995008 2024-09-21T15:50:58.802 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.802 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.802 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:58.802 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574017+0000", 2024-09-21T15:50:58.802 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00050241100000000003 2024-09-21T15:50:58.802 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.802 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.802 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:58.802 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574027+0000", 2024-09-21T15:50:58.802 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4700000000000008e-06 2024-09-21T15:50:58.802 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.802 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.803 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:58.803 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574033+0000", 2024-09-21T15:50:58.803 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1489999999999999e-06 2024-09-21T15:50:58.803 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.803 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.803 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:58.803 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629238+0000", 2024-09-21T15:50:58.803 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055204880999999997 2024-09-21T15:50:58.803 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.803 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.803 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:58.803 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629317+0000", 2024-09-21T15:50:58.803 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8942000000000004e-05 2024-09-21T15:50:58.803 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.804 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.804 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:58.804 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629454+0000", 2024-09-21T15:50:58.804 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013767099999999999 2024-09-21T15:50:58.804 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.804 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:58.804 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.804 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.804 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.804 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120125 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 600064~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:58.804 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.574482+0000", 2024-09-21T15:50:58.804 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 55.171217079000002, 2024-09-21T15:50:58.804 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055243608, 2024-09-21T15:50:58.804 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:58.805 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:58.805 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:58.805 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:58.805 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:58.805 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120125 2024-09-21T15:50:58.805 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.805 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:58.805 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.805 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:58.805 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574482+0000", 2024-09-21T15:50:58.805 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.805 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.805 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.805 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:58.805 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574482+0000", 2024-09-21T15:50:58.806 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.806 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.806 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.806 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:58.806 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574038+0000", 2024-09-21T15:50:58.806 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9995556 2024-09-21T15:50:58.806 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.806 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.806 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:58.806 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574497+0000", 2024-09-21T15:50:58.806 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00045854400000000002 2024-09-21T15:50:58.806 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.806 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.806 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:58.806 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574509+0000", 2024-09-21T15:50:58.807 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2486e-05 2024-09-21T15:50:58.807 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.807 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.807 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:58.807 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574516+0000", 2024-09-21T15:50:58.807 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2269999999999996e-06 2024-09-21T15:50:58.807 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.807 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.807 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:58.807 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629497+0000", 2024-09-21T15:50:58.807 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054980785999999997 2024-09-21T15:50:58.807 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.807 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.807 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:58.807 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629580+0000", 2024-09-21T15:50:58.807 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2528000000000007e-05 2024-09-21T15:50:58.807 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.808 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.808 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:58.808 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629726+0000", 2024-09-21T15:50:58.808 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014630399999999999 2024-09-21T15:50:58.808 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.808 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:58.808 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.808 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.808 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.808 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120126 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 620544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:58.808 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.575315+0000", 2024-09-21T15:50:58.808 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 55.170384048999999, 2024-09-21T15:50:58.808 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054709559999999997, 2024-09-21T15:50:58.808 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:58.808 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:58.808 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:58.809 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:58.809 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:58.809 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120126 2024-09-21T15:50:58.809 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.809 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:58.809 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.809 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:58.809 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575315+0000", 2024-09-21T15:50:58.809 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.809 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.809 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.809 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:58.809 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575315+0000", 2024-09-21T15:50:58.809 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.809 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.809 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.810 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:58.810 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574522+0000", 2024-09-21T15:50:58.810 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9992065 2024-09-21T15:50:58.810 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.810 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.810 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:58.810 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575319+0000", 2024-09-21T15:50:58.810 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00079626800000000004 2024-09-21T15:50:58.810 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.810 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.810 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:58.810 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575329+0000", 2024-09-21T15:50:58.810 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.028e-05 2024-09-21T15:50:58.810 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.810 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.810 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:58.811 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575335+0000", 2024-09-21T15:50:58.811 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9290000000000003e-06 2024-09-21T15:50:58.811 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.811 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.811 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:58.811 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629806+0000", 2024-09-21T15:50:58.811 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054471558000000003 2024-09-21T15:50:58.811 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.811 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.811 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:58.811 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629885+0000", 2024-09-21T15:50:58.811 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8563e-05 2024-09-21T15:50:58.811 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.811 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.811 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:58.812 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630025+0000", 2024-09-21T15:50:58.812 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014019 2024-09-21T15:50:58.812 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.812 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:58.812 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.812 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.812 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.812 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120127 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 641024~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:58.812 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.575340+0000", 2024-09-21T15:50:58.812 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 55.170359341999998, 2024-09-21T15:50:58.812 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054895589000000002, 2024-09-21T15:50:58.812 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:58.812 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:58.812 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:58.812 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:58.812 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:58.813 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120127 2024-09-21T15:50:58.813 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.813 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:58.813 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.813 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:58.813 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575340+0000", 2024-09-21T15:50:58.813 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.813 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.813 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.813 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:58.813 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575340+0000", 2024-09-21T15:50:58.813 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.813 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.813 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.813 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:58.813 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575340+0000", 2024-09-21T15:50:58.813 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:58.814 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.814 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.814 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:58.814 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575342+0000", 2024-09-21T15:50:58.814 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5600000000000001e-06 2024-09-21T15:50:58.814 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.814 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.814 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:58.814 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575351+0000", 2024-09-21T15:50:58.814 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2349999999999999e-06 2024-09-21T15:50:58.814 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.814 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.814 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:58.814 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575356+0000", 2024-09-21T15:50:58.815 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4600000000000002e-06 2024-09-21T15:50:58.815 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.815 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.815 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:58.815 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630053+0000", 2024-09-21T15:50:58.815 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054697190999999999 2024-09-21T15:50:58.815 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.815 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.815 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:58.815 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630128+0000", 2024-09-21T15:50:58.815 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4777000000000006e-05 2024-09-21T15:50:58.815 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.816 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.816 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:58.816 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630236+0000", 2024-09-21T15:50:58.816 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000107837 2024-09-21T15:50:58.816 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.816 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:58.816 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.816 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.816 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.816 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120128 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 657408~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:58.816 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.575461+0000", 2024-09-21T15:50:58.816 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 55.170238763, 2024-09-21T15:50:58.816 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054945121999999999, 2024-09-21T15:50:58.816 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:58.816 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:58.817 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:58.817 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:58.817 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:58.817 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120128 2024-09-21T15:50:58.817 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.817 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:58.817 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.817 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:58.817 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575461+0000", 2024-09-21T15:50:58.817 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.817 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.817 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.817 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:58.817 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575461+0000", 2024-09-21T15:50:58.817 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.818 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.818 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.818 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:58.818 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575360+0000", 2024-09-21T15:50:58.818 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998994 2024-09-21T15:50:58.818 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.818 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.818 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:58.818 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575464+0000", 2024-09-21T15:50:58.818 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000103397 2024-09-21T15:50:58.818 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.818 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.818 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:58.818 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575472+0000", 2024-09-21T15:50:58.819 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3189999999999995e-06 2024-09-21T15:50:58.819 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.819 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.819 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:58.819 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575478+0000", 2024-09-21T15:50:58.819 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2779999999999999e-06 2024-09-21T15:50:58.819 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.819 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.819 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:58.819 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630268+0000", 2024-09-21T15:50:58.819 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054789545000000002 2024-09-21T15:50:58.819 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.819 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.819 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:58.819 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630315+0000", 2024-09-21T15:50:58.820 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7244999999999999e-05 2024-09-21T15:50:58.820 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.820 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.820 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:58.820 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630406+0000", 2024-09-21T15:50:58.820 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0811999999999997e-05 2024-09-21T15:50:58.820 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.820 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:58.820 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.820 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.820 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.820 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120129 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 677888~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:58.821 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.576016+0000", 2024-09-21T15:50:58.821 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 55.169683218999999, 2024-09-21T15:50:58.821 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054602823000000002, 2024-09-21T15:50:58.821 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:58.821 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:58.821 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:58.821 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:58.821 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:58.821 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120129 2024-09-21T15:50:58.821 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.821 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:58.821 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.821 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:58.821 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576016+0000", 2024-09-21T15:50:58.821 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.821 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.822 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.822 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:58.822 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576016+0000", 2024-09-21T15:50:58.822 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.822 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.822 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.822 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:58.822 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575483+0000", 2024-09-21T15:50:58.822 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9994669 2024-09-21T15:50:58.822 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.822 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.822 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:58.822 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576020+0000", 2024-09-21T15:50:58.822 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00053657900000000003 2024-09-21T15:50:58.822 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.823 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.823 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:58.823 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576030+0000", 2024-09-21T15:50:58.823 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0326000000000001e-05 2024-09-21T15:50:58.823 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.823 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.823 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:58.823 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576037+0000", 2024-09-21T15:50:58.823 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6490000000000002e-06 2024-09-21T15:50:58.823 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.823 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.823 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:58.823 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630446+0000", 2024-09-21T15:50:58.823 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054409698999999999 2024-09-21T15:50:58.824 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.824 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.824 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:58.824 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630524+0000", 2024-09-21T15:50:58.824 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7422999999999994e-05 2024-09-21T15:50:58.824 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.824 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.824 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:58.824 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630619+0000", 2024-09-21T15:50:58.824 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5332000000000001e-05 2024-09-21T15:50:58.824 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.824 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:58.824 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.824 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.824 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.825 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120130 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 698368~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:50:58.825 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.576043+0000", 2024-09-21T15:50:58.825 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 55.169656848000002, 2024-09-21T15:50:58.825 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054781798, 2024-09-21T15:50:58.825 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:58.825 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:58.825 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:58.825 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:58.825 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:58.825 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120130 2024-09-21T15:50:58.825 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.825 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:58.825 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.825 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:58.825 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576043+0000", 2024-09-21T15:50:58.826 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.826 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.826 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.826 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:58.826 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576043+0000", 2024-09-21T15:50:58.826 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:58.826 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.826 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.826 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:58.826 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576042+0000", 2024-09-21T15:50:58.826 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:58.826 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.826 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.826 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:58.826 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576044+0000", 2024-09-21T15:50:58.827 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0999999999999998e-06 2024-09-21T15:50:58.827 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.827 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.827 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:58.827 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576051+0000", 2024-09-21T15:50:58.827 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4779999999999996e-06 2024-09-21T15:50:58.827 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.827 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.827 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:58.827 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576058+0000", 2024-09-21T15:50:58.827 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2180000000000002e-06 2024-09-21T15:50:58.827 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.827 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.827 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:58.827 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630661+0000", 2024-09-21T15:50:58.828 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054603240999999997 2024-09-21T15:50:58.828 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.828 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.828 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:58.828 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630747+0000", 2024-09-21T15:50:58.828 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6046999999999998e-05 2024-09-21T15:50:58.828 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:58.828 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:58.828 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:58.828 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630824+0000", 2024-09-21T15:50:58.828 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7119999999999993e-05 2024-09-21T15:50:58.828 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.828 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:58.828 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.828 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:58.829 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:58.829 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:58.829 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-09-21T15:50:58.829 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-21T15:50:58.831 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T15:50:58.831 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:50:58.829+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:50:58.931 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:58.929+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:58.973 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:58.973 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:58.973 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:50:58.973 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:58.982 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T15:50:59.031 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:59.030+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:59.082 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:59.083 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:59.083 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:50:59.083 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:50:59.083 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:59.091 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T15:50:59.132 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:50:59.131+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:50:59.233 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:50:59.231+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:50:59.235 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:59.235 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:50:59.235 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:50:59.235 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:50:59.235 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.235 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63140 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 876544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:59.235 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.133649+0000", 2024-09-21T15:50:59.235 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 82.099459171000007, 2024-09-21T15:50:59.235 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053950201000000003, 2024-09-21T15:50:59.235 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:59.235 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:59.236 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:59.236 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:59.236 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:59.236 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63140 2024-09-21T15:50:59.236 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.236 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:59.236 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.236 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:59.236 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133649+0000", 2024-09-21T15:50:59.236 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.236 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.236 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.236 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:59.236 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133649+0000", 2024-09-21T15:50:59.236 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.237 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.237 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.237 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:59.237 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133648+0000", 2024-09-21T15:50:59.237 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:59.237 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.237 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.237 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:59.237 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133651+0000", 2024-09-21T15:50:59.237 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1549999999999999e-06 2024-09-21T15:50:59.237 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.237 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.237 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:59.238 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133660+0000", 2024-09-21T15:50:59.238 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3899999999999993e-06 2024-09-21T15:50:59.238 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.238 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.238 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:59.238 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133669+0000", 2024-09-21T15:50:59.238 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3219999999999999e-06 2024-09-21T15:50:59.238 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.238 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.238 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:59.238 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187409+0000", 2024-09-21T15:50:59.238 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053740257 2024-09-21T15:50:59.238 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.238 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.238 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:59.239 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187492+0000", 2024-09-21T15:50:59.239 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2409000000000003e-05 2024-09-21T15:50:59.239 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.239 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.239 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:59.239 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187599+0000", 2024-09-21T15:50:59.239 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000107334 2024-09-21T15:50:59.239 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.239 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:59.239 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.239 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.239 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.239 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63141 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 884736~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:59.239 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.133843+0000", 2024-09-21T15:50:59.239 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 82.099265052000007, 2024-09-21T15:50:59.239 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053954015000000001, 2024-09-21T15:50:59.240 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:59.240 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:59.240 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:59.240 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:59.240 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:59.240 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63141 2024-09-21T15:50:59.240 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.240 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:59.240 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.240 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:59.240 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133843+0000", 2024-09-21T15:50:59.240 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.240 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.240 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.240 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:59.241 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133843+0000", 2024-09-21T15:50:59.241 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.241 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.241 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.241 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:59.241 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133673+0000", 2024-09-21T15:50:59.241 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998302 2024-09-21T15:50:59.241 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.241 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.241 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:59.241 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133846+0000", 2024-09-21T15:50:59.241 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000172762 2024-09-21T15:50:59.241 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.241 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.242 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:59.242 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133855+0000", 2024-09-21T15:50:59.242 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2940000000000006e-06 2024-09-21T15:50:59.242 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.242 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.242 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:59.242 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133863+0000", 2024-09-21T15:50:59.242 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9740000000000002e-06 2024-09-21T15:50:59.242 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.242 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.242 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:59.242 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187630+0000", 2024-09-21T15:50:59.242 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053767211000000002 2024-09-21T15:50:59.242 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.243 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.243 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:59.243 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187707+0000", 2024-09-21T15:50:59.243 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7446000000000004e-05 2024-09-21T15:50:59.243 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.243 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.243 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:59.243 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187797+0000", 2024-09-21T15:50:59.243 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.9858000000000006e-05 2024-09-21T15:50:59.243 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.243 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:59.243 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.243 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.243 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.243 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63142 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 892928~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:59.243 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134046+0000", 2024-09-21T15:50:59.244 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 82.099062110000006, 2024-09-21T15:50:59.244 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054017849999999999, 2024-09-21T15:50:59.244 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:59.244 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:59.244 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:59.244 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:59.244 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:59.244 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63142 2024-09-21T15:50:59.244 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.244 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:59.244 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.244 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:59.244 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134046+0000", 2024-09-21T15:50:59.244 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.244 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.245 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.245 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:59.245 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134046+0000", 2024-09-21T15:50:59.245 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.245 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.245 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.245 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:59.245 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133867+0000", 2024-09-21T15:50:59.245 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998217 2024-09-21T15:50:59.245 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.245 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.245 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:59.245 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134049+0000", 2024-09-21T15:50:59.245 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000181695 2024-09-21T15:50:59.245 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.246 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.246 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:59.246 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134057+0000", 2024-09-21T15:50:59.246 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.303e-06 2024-09-21T15:50:59.246 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.246 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.246 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:59.246 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134063+0000", 2024-09-21T15:50:59.246 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9959999999999999e-06 2024-09-21T15:50:59.246 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.246 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.246 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:59.246 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187849+0000", 2024-09-21T15:50:59.246 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053785571999999997 2024-09-21T15:50:59.246 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.247 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.247 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:59.247 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187954+0000", 2024-09-21T15:50:59.247 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010524899999999999 2024-09-21T15:50:59.247 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.247 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.247 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:59.247 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188064+0000", 2024-09-21T15:50:59.247 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000109549 2024-09-21T15:50:59.247 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.247 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:59.247 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.247 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.247 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.247 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63143 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 901120~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:59.247 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134239+0000", 2024-09-21T15:50:59.247 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 82.098869516999997, 2024-09-21T15:50:59.248 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054026058000000002, 2024-09-21T15:50:59.248 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:59.248 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:59.248 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:59.248 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:59.248 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:59.248 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63143 2024-09-21T15:50:59.248 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.248 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:59.248 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.248 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:59.248 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134239+0000", 2024-09-21T15:50:59.248 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.248 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.248 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.248 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:59.249 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134239+0000", 2024-09-21T15:50:59.249 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.249 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.249 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.249 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:59.249 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134068+0000", 2024-09-21T15:50:59.249 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998298 2024-09-21T15:50:59.249 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.249 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.249 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:59.249 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134242+0000", 2024-09-21T15:50:59.249 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00017414199999999999 2024-09-21T15:50:59.249 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.249 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.249 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:59.250 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134264+0000", 2024-09-21T15:50:59.250 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1552e-05 2024-09-21T15:50:59.250 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.250 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.250 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:59.250 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134270+0000", 2024-09-21T15:50:59.250 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6939999999999998e-06 2024-09-21T15:50:59.250 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.250 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.250 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:59.250 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188108+0000", 2024-09-21T15:50:59.250 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.05383719 2024-09-21T15:50:59.250 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.250 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.250 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:59.250 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188186+0000", 2024-09-21T15:50:59.251 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8318000000000005e-05 2024-09-21T15:50:59.251 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.251 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.251 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:59.251 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188265+0000", 2024-09-21T15:50:59.251 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8597000000000007e-05 2024-09-21T15:50:59.251 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.251 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:59.251 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.251 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.251 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.251 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63144 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 909312~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:59.251 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134432+0000", 2024-09-21T15:50:59.251 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 82.098675960999998, 2024-09-21T15:50:59.251 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053968953, 2024-09-21T15:50:59.251 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:59.252 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:59.252 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:59.252 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:59.252 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:59.252 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63144 2024-09-21T15:50:59.252 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.252 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:59.252 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.252 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:59.252 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134432+0000", 2024-09-21T15:50:59.252 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.252 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.252 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.252 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:59.252 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134432+0000", 2024-09-21T15:50:59.252 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.252 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.253 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.253 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:59.253 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134276+0000", 2024-09-21T15:50:59.253 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998441 2024-09-21T15:50:59.253 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.253 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.253 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:59.253 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134436+0000", 2024-09-21T15:50:59.253 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000159591 2024-09-21T15:50:59.253 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.253 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.253 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:59.253 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134445+0000", 2024-09-21T15:50:59.253 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6780000000000006e-06 2024-09-21T15:50:59.253 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.254 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.254 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:59.254 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134452+0000", 2024-09-21T15:50:59.254 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6830000000000008e-06 2024-09-21T15:50:59.254 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.254 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.254 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:59.254 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188282+0000", 2024-09-21T15:50:59.254 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053829252000000001 2024-09-21T15:50:59.254 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.254 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.254 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:59.254 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188328+0000", 2024-09-21T15:50:59.254 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6734999999999999e-05 2024-09-21T15:50:59.255 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.255 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.255 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:59.255 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188401+0000", 2024-09-21T15:50:59.255 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2731000000000005e-05 2024-09-21T15:50:59.255 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.255 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:59.255 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.255 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.255 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.255 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63145 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 860160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:59.255 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134736+0000", 2024-09-21T15:50:59.255 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 82.098372175999998, 2024-09-21T15:50:59.255 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053865960999999997, 2024-09-21T15:50:59.255 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:59.255 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:59.256 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:59.256 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:59.256 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:59.256 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63145 2024-09-21T15:50:59.256 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.256 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:59.256 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.256 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:59.256 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134736+0000", 2024-09-21T15:50:59.256 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.256 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.256 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.256 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:59.256 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134736+0000", 2024-09-21T15:50:59.256 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.256 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.257 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.257 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:59.257 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134458+0000", 2024-09-21T15:50:59.257 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999722 2024-09-21T15:50:59.257 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.257 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.257 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:59.257 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134741+0000", 2024-09-21T15:50:59.257 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00028245599999999999 2024-09-21T15:50:59.257 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.257 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.257 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:59.257 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134753+0000", 2024-09-21T15:50:59.257 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2548000000000001e-05 2024-09-21T15:50:59.257 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.258 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.258 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:59.258 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134760+0000", 2024-09-21T15:50:59.258 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4189999999999996e-06 2024-09-21T15:50:59.258 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.258 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.258 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:59.258 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188426+0000", 2024-09-21T15:50:59.258 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053665894999999998 2024-09-21T15:50:59.258 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.258 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.258 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:59.258 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188502+0000", 2024-09-21T15:50:59.258 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5791000000000007e-05 2024-09-21T15:50:59.259 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.259 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.259 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:59.259 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188602+0000", 2024-09-21T15:50:59.259 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9667999999999995e-05 2024-09-21T15:50:59.259 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.259 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:59.259 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.259 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.259 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.259 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63148 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 933888~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:59.259 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.135289+0000", 2024-09-21T15:50:59.259 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 82.097819220999995, 2024-09-21T15:50:59.259 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053948488000000003, 2024-09-21T15:50:59.259 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:59.259 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:59.260 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:59.260 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:59.260 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:59.260 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63148 2024-09-21T15:50:59.260 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.260 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:59.260 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.260 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:59.260 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135289+0000", 2024-09-21T15:50:59.260 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.260 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.260 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.260 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:59.260 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135289+0000", 2024-09-21T15:50:59.260 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.260 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.261 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.261 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:59.261 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135288+0000", 2024-09-21T15:50:59.261 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:59.261 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.261 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.261 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:59.261 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135296+0000", 2024-09-21T15:50:59.261 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5399999999999998e-06 2024-09-21T15:50:59.261 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.261 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.261 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:59.261 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135303+0000", 2024-09-21T15:50:59.261 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0500000000000003e-06 2024-09-21T15:50:59.262 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.262 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.262 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:59.262 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135308+0000", 2024-09-21T15:50:59.262 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9749999999999998e-06 2024-09-21T15:50:59.262 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.262 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.262 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:59.262 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189066+0000", 2024-09-21T15:50:59.262 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053758501 2024-09-21T15:50:59.262 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.262 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.262 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:59.262 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189145+0000", 2024-09-21T15:50:59.263 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9017000000000006e-05 2024-09-21T15:50:59.263 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.263 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.263 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:59.263 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189237+0000", 2024-09-21T15:50:59.263 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1906999999999999e-05 2024-09-21T15:50:59.263 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.263 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:59.263 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.263 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.263 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.263 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63149 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 942080~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:59.263 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.135430+0000", 2024-09-21T15:50:59.263 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 82.097677891999993, 2024-09-21T15:50:59.264 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054005434999999997, 2024-09-21T15:50:59.264 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:59.264 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:59.264 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:59.264 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:59.264 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:59.264 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63149 2024-09-21T15:50:59.264 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.264 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:59.264 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.264 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:59.264 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135430+0000", 2024-09-21T15:50:59.264 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.264 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.264 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.265 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:59.265 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135430+0000", 2024-09-21T15:50:59.265 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.265 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.265 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.265 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:59.265 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135313+0000", 2024-09-21T15:50:59.265 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998827 2024-09-21T15:50:59.265 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.265 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.265 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:59.265 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135434+0000", 2024-09-21T15:50:59.265 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000120876 2024-09-21T15:50:59.265 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.265 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.265 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:59.266 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135443+0000", 2024-09-21T15:50:59.266 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3340000000000008e-06 2024-09-21T15:50:59.266 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.266 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.266 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:59.266 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135449+0000", 2024-09-21T15:50:59.266 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1580000000000002e-06 2024-09-21T15:50:59.266 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.266 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.266 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:59.266 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189267+0000", 2024-09-21T15:50:59.266 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053817775999999998 2024-09-21T15:50:59.266 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.266 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.266 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:59.266 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189336+0000", 2024-09-21T15:50:59.267 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9485000000000003e-05 2024-09-21T15:50:59.267 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.267 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.267 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:59.267 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189436+0000", 2024-09-21T15:50:59.267 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9277000000000005e-05 2024-09-21T15:50:59.267 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.267 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:59.267 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.267 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.267 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.267 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63150 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 950272~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:59.267 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.135618+0000", 2024-09-21T15:50:59.267 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 82.097490147000002, 2024-09-21T15:50:59.267 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054027393, 2024-09-21T15:50:59.267 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:59.268 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:59.268 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:59.268 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:59.268 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:59.268 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63150 2024-09-21T15:50:59.268 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.268 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:59.268 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.268 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:59.268 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135618+0000", 2024-09-21T15:50:59.268 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.268 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.268 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.268 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:59.268 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135618+0000", 2024-09-21T15:50:59.268 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.269 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.269 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.269 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:59.269 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135454+0000", 2024-09-21T15:50:59.269 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998364 2024-09-21T15:50:59.269 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.269 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.269 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:59.269 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135621+0000", 2024-09-21T15:50:59.269 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016714099999999999 2024-09-21T15:50:59.269 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.269 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.269 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:59.269 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135631+0000", 2024-09-21T15:50:59.269 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5340000000000005e-06 2024-09-21T15:50:59.269 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.269 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.270 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:59.270 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135638+0000", 2024-09-21T15:50:59.270 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9280000000000004e-06 2024-09-21T15:50:59.270 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.270 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.270 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:59.270 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189463+0000", 2024-09-21T15:50:59.270 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053825681 2024-09-21T15:50:59.270 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.270 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.270 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:59.270 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189546+0000", 2024-09-21T15:50:59.270 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2278000000000001e-05 2024-09-21T15:50:59.270 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.271 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.271 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:59.271 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189645+0000", 2024-09-21T15:50:59.271 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9543999999999994e-05 2024-09-21T15:50:59.271 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.271 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:59.271 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.271 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.271 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.271 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63152 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 966656~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:59.271 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.136029+0000", 2024-09-21T15:50:59.271 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 82.097079554999993, 2024-09-21T15:50:59.271 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054024823999999999, 2024-09-21T15:50:59.271 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:59.271 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:59.271 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:59.272 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:59.272 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:59.272 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63152 2024-09-21T15:50:59.272 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.272 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:59.272 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.272 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:59.272 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136029+0000", 2024-09-21T15:50:59.272 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.272 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.272 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.272 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:59.272 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136029+0000", 2024-09-21T15:50:59.272 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.272 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.272 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.273 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:59.273 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136028+0000", 2024-09-21T15:50:59.273 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:59.273 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.273 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.273 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:59.273 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136031+0000", 2024-09-21T15:50:59.273 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.033e-06 2024-09-21T15:50:59.273 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.273 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.273 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:59.273 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136039+0000", 2024-09-21T15:50:59.273 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7200000000000006e-06 2024-09-21T15:50:59.273 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.274 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.274 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:59.274 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136051+0000", 2024-09-21T15:50:59.274 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1987000000000001e-05 2024-09-21T15:50:59.274 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.274 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.274 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:59.274 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189876+0000", 2024-09-21T15:50:59.274 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053825264999999997 2024-09-21T15:50:59.274 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.274 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.274 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:59.274 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189955+0000", 2024-09-21T15:50:59.274 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9296999999999997e-05 2024-09-21T15:50:59.275 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.275 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.275 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:59.275 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190053+0000", 2024-09-21T15:50:59.275 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.8067999999999997e-05 2024-09-21T15:50:59.275 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.275 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:59.275 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.275 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.275 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.275 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63153 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 974848~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:59.275 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.136311+0000", 2024-09-21T15:50:59.275 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 82.096797042000006, 2024-09-21T15:50:59.275 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053933825999999997, 2024-09-21T15:50:59.275 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:59.276 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:59.276 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:59.276 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:59.276 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:59.276 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63153 2024-09-21T15:50:59.276 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.276 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:59.276 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.276 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:59.276 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136311+0000", 2024-09-21T15:50:59.276 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.276 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.276 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.276 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:59.276 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136311+0000", 2024-09-21T15:50:59.276 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.277 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.277 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.277 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:59.277 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136057+0000", 2024-09-21T15:50:59.277 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9997463 2024-09-21T15:50:59.277 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.277 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.277 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:59.277 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136316+0000", 2024-09-21T15:50:59.277 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00025872299999999998 2024-09-21T15:50:59.277 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.277 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.277 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:59.278 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136326+0000", 2024-09-21T15:50:59.278 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0552e-05 2024-09-21T15:50:59.278 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.278 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.278 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:59.278 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136334+0000", 2024-09-21T15:50:59.278 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1409999999999999e-06 2024-09-21T15:50:59.278 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.278 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.278 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:59.278 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190080+0000", 2024-09-21T15:50:59.278 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053745933000000003 2024-09-21T15:50:59.278 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.278 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.278 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:59.279 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190157+0000", 2024-09-21T15:50:59.279 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7412999999999999e-05 2024-09-21T15:50:59.279 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.279 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.279 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:59.279 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190245+0000", 2024-09-21T15:50:59.279 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7904000000000004e-05 2024-09-21T15:50:59.279 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.279 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:59.279 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.279 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.279 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.279 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63154 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 983040~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:59.279 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.136472+0000", 2024-09-21T15:50:59.279 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 82.096635691000003, 2024-09-21T15:50:59.279 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053968448000000002, 2024-09-21T15:50:59.280 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:59.280 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:59.280 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:59.280 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:59.280 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:59.280 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63154 2024-09-21T15:50:59.280 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.280 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:59.280 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.280 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:59.280 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136472+0000", 2024-09-21T15:50:59.280 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.280 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.280 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.280 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:59.280 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136472+0000", 2024-09-21T15:50:59.281 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.281 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.281 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.281 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:59.281 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136341+0000", 2024-09-21T15:50:59.281 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998684 2024-09-21T15:50:59.281 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.281 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.281 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:59.281 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136476+0000", 2024-09-21T15:50:59.281 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013528100000000001 2024-09-21T15:50:59.281 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.281 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.281 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:59.281 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136484+0000", 2024-09-21T15:50:59.282 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4760000000000001e-06 2024-09-21T15:50:59.282 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.282 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.282 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:59.282 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136490+0000", 2024-09-21T15:50:59.282 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8640000000000001e-06 2024-09-21T15:50:59.282 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.282 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.282 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:59.282 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190271+0000", 2024-09-21T15:50:59.282 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053781223000000003 2024-09-21T15:50:59.282 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.282 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.282 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:59.282 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190348+0000", 2024-09-21T15:50:59.283 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7058999999999995e-05 2024-09-21T15:50:59.283 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.283 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.283 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:59.283 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190441+0000", 2024-09-21T15:50:59.283 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2937999999999997e-05 2024-09-21T15:50:59.283 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.283 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:59.283 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.283 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.283 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.283 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63157 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1015808~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:59.283 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137112+0000", 2024-09-21T15:50:59.283 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 82.095996404000005, 2024-09-21T15:50:59.283 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053932345, 2024-09-21T15:50:59.284 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:59.284 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:59.284 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:59.284 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:59.284 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:59.284 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63157 2024-09-21T15:50:59.284 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.284 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:59.284 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.284 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:59.284 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137112+0000", 2024-09-21T15:50:59.284 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.284 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.284 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.284 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:59.285 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137112+0000", 2024-09-21T15:50:59.285 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.285 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.285 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.285 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:59.285 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137072+0000", 2024-09-21T15:50:59.285 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999604 2024-09-21T15:50:59.285 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.285 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.285 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:59.285 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137115+0000", 2024-09-21T15:50:59.285 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2379999999999997e-05 2024-09-21T15:50:59.285 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.285 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.286 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:59.286 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137121+0000", 2024-09-21T15:50:59.286 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6499999999999999e-06 2024-09-21T15:50:59.286 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.286 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.286 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:59.286 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137126+0000", 2024-09-21T15:50:59.286 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0769999999999998e-06 2024-09-21T15:50:59.286 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.286 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.286 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:59.286 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190868+0000", 2024-09-21T15:50:59.286 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053741555000000003 2024-09-21T15:50:59.287 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.287 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.287 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:59.287 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190945+0000", 2024-09-21T15:50:59.287 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7171999999999999e-05 2024-09-21T15:50:59.287 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.287 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.287 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:59.287 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191044+0000", 2024-09-21T15:50:59.287 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9031999999999996e-05 2024-09-21T15:50:59.287 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.287 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:59.287 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.287 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.288 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.288 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63158 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1007616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:59.288 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137293+0000", 2024-09-21T15:50:59.288 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 82.095815117000001, 2024-09-21T15:50:59.288 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054005642, 2024-09-21T15:50:59.288 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:59.288 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:59.288 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:59.288 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:59.288 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:59.288 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63158 2024-09-21T15:50:59.288 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.288 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:59.289 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.289 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:59.289 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137293+0000", 2024-09-21T15:50:59.289 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.289 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.289 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.289 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:59.289 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137293+0000", 2024-09-21T15:50:59.289 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.289 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.289 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.289 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:59.289 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137131+0000", 2024-09-21T15:50:59.289 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998379 2024-09-21T15:50:59.289 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.289 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.290 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:59.290 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137296+0000", 2024-09-21T15:50:59.290 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016568999999999999 2024-09-21T15:50:59.290 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.290 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.290 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:59.290 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137306+0000", 2024-09-21T15:50:59.290 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.6029999999999994e-06 2024-09-21T15:50:59.290 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.290 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.290 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:59.290 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137312+0000", 2024-09-21T15:50:59.290 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3210000000000001e-06 2024-09-21T15:50:59.290 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.291 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.291 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:59.291 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191087+0000", 2024-09-21T15:50:59.291 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053774686000000002 2024-09-21T15:50:59.291 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.291 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.291 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:59.291 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191175+0000", 2024-09-21T15:50:59.291 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8106e-05 2024-09-21T15:50:59.291 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.291 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.291 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:59.291 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191299+0000", 2024-09-21T15:50:59.291 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012343599999999999 2024-09-21T15:50:59.292 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.292 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:59.292 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.292 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.292 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.292 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63159 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1024000~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:59.292 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137568+0000", 2024-09-21T15:50:59.292 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 82.095539845000005, 2024-09-21T15:50:59.292 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053962999999999997, 2024-09-21T15:50:59.292 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:59.292 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:59.292 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:59.292 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:59.292 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:59.292 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63159 2024-09-21T15:50:59.293 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.293 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:59.293 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.293 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:59.293 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137568+0000", 2024-09-21T15:50:59.293 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.293 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.293 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.293 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:59.293 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137568+0000", 2024-09-21T15:50:59.293 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.293 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.293 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.293 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:59.293 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137317+0000", 2024-09-21T15:50:59.294 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9997492 2024-09-21T15:50:59.294 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.294 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.294 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:59.294 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137572+0000", 2024-09-21T15:50:59.294 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00025491499999999998 2024-09-21T15:50:59.294 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.294 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.294 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:59.294 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137582+0000", 2024-09-21T15:50:59.294 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4490000000000005e-06 2024-09-21T15:50:59.294 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.294 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.294 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:59.295 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137589+0000", 2024-09-21T15:50:59.295 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5580000000000003e-06 2024-09-21T15:50:59.295 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.295 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.295 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:59.295 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191344+0000", 2024-09-21T15:50:59.295 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053754272999999998 2024-09-21T15:50:59.295 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.295 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.295 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:59.295 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191429+0000", 2024-09-21T15:50:59.295 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.509e-05 2024-09-21T15:50:59.295 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.295 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.296 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:59.296 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191531+0000", 2024-09-21T15:50:59.296 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000102519 2024-09-21T15:50:59.296 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.296 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:59.296 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.296 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.296 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.296 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63160 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1032192~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:59.296 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137771+0000", 2024-09-21T15:50:59.296 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 82.095337310999994, 2024-09-21T15:50:59.296 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053957818999999997, 2024-09-21T15:50:59.296 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:59.296 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:59.296 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:59.297 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:59.297 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:59.297 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63160 2024-09-21T15:50:59.297 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.297 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:59.297 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.297 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:59.297 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137771+0000", 2024-09-21T15:50:59.297 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.297 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.297 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.297 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:59.297 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137771+0000", 2024-09-21T15:50:59.297 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.297 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.298 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.298 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:59.298 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137596+0000", 2024-09-21T15:50:59.298 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998255 2024-09-21T15:50:59.298 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.298 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.298 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:59.298 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137780+0000", 2024-09-21T15:50:59.298 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00018363900000000001 2024-09-21T15:50:59.298 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.298 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.298 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:59.298 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137791+0000", 2024-09-21T15:50:59.298 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.096e-05 2024-09-21T15:50:59.298 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.299 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.299 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:59.299 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137798+0000", 2024-09-21T15:50:59.299 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1169999999999998e-06 2024-09-21T15:50:59.299 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.299 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.299 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:59.299 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191560+0000", 2024-09-21T15:50:59.299 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053762163000000002 2024-09-21T15:50:59.299 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.299 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.299 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:59.299 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191634+0000", 2024-09-21T15:50:59.299 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4381999999999994e-05 2024-09-21T15:50:59.300 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.300 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.300 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:59.300 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191729+0000", 2024-09-21T15:50:59.300 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4098000000000006e-05 2024-09-21T15:50:59.300 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.300 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:59.300 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.300 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.300 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.300 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63161 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1040384~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:59.300 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.138018+0000", 2024-09-21T15:50:59.300 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 82.095090557000006, 2024-09-21T15:50:59.300 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053905657000000003, 2024-09-21T15:50:59.300 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:59.301 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:59.301 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:59.301 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:59.301 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:59.301 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63161 2024-09-21T15:50:59.301 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.301 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:59.301 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.301 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:59.301 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138018+0000", 2024-09-21T15:50:59.301 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.301 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.301 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.301 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:59.301 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138018+0000", 2024-09-21T15:50:59.301 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.302 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.302 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.302 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:59.302 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137804+0000", 2024-09-21T15:50:59.302 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9997859 2024-09-21T15:50:59.302 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.302 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.302 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:59.302 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138021+0000", 2024-09-21T15:50:59.302 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00021769899999999999 2024-09-21T15:50:59.302 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.302 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.302 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:59.302 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138043+0000", 2024-09-21T15:50:59.302 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1627000000000001e-05 2024-09-21T15:50:59.303 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.303 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.303 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:59.303 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138054+0000", 2024-09-21T15:50:59.303 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1321999999999999e-05 2024-09-21T15:50:59.303 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.303 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.303 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:59.303 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191755+0000", 2024-09-21T15:50:59.303 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053700354999999998 2024-09-21T15:50:59.303 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.303 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.303 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:59.304 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191830+0000", 2024-09-21T15:50:59.304 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5320000000000004e-05 2024-09-21T15:50:59.304 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.304 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.304 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:59.304 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191923+0000", 2024-09-21T15:50:59.304 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3282999999999994e-05 2024-09-21T15:50:59.304 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.304 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:59.304 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.304 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.304 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.304 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63176 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 108544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:59.304 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.140861+0000", 2024-09-21T15:50:59.304 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 82.092247237999999, 2024-09-21T15:50:59.305 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053935394999999997, 2024-09-21T15:50:59.305 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:59.305 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:59.305 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:59.305 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:59.305 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:59.305 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63176 2024-09-21T15:50:59.305 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.305 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:59.305 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.305 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:59.305 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140861+0000", 2024-09-21T15:50:59.305 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.305 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.306 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.306 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:59.306 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140861+0000", 2024-09-21T15:50:59.306 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.306 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.306 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.306 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:59.306 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140860+0000", 2024-09-21T15:50:59.306 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:59.306 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.306 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.307 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:59.307 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140863+0000", 2024-09-21T15:50:59.307 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8710000000000001e-06 2024-09-21T15:50:59.307 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.307 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.307 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:59.307 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140871+0000", 2024-09-21T15:50:59.307 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8089999999999993e-06 2024-09-21T15:50:59.307 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.307 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.307 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:59.307 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140877+0000", 2024-09-21T15:50:59.307 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5969999999999999e-06 2024-09-21T15:50:59.307 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.308 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.308 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:59.308 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.194608+0000", 2024-09-21T15:50:59.308 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053731749000000002 2024-09-21T15:50:59.308 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.308 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.308 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:59.308 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.194697+0000", 2024-09-21T15:50:59.308 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8504000000000005e-05 2024-09-21T15:50:59.308 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.308 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.308 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:59.308 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.194796+0000", 2024-09-21T15:50:59.309 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9414999999999996e-05 2024-09-21T15:50:59.309 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.309 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:59.309 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.309 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.309 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.309 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63182 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 157696~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:59.309 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:50:59.309 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 82.090974837999994, 2024-09-21T15:50:59.309 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053958825000000002, 2024-09-21T15:50:59.309 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:59.309 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:59.309 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:59.309 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:59.309 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:59.310 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63182 2024-09-21T15:50:59.310 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.310 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:59.310 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.310 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:59.310 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:50:59.310 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.310 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.310 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.310 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:59.310 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:50:59.310 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.310 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.311 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.311 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:59.311 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:50:59.311 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:59.311 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.311 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.311 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:59.311 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142136+0000", 2024-09-21T15:50:59.311 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8169999999999999e-06 2024-09-21T15:50:59.311 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.311 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.311 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:59.311 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:50:59.311 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3752e-05 2024-09-21T15:50:59.311 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.312 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.312 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:59.312 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142155+0000", 2024-09-21T15:50:59.312 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1859999999999997e-06 2024-09-21T15:50:59.312 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.312 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.312 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:59.312 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.195914+0000", 2024-09-21T15:50:59.312 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053758528999999999 2024-09-21T15:50:59.312 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.312 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.312 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:59.312 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.195995+0000", 2024-09-21T15:50:59.312 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0983999999999996e-05 2024-09-21T15:50:59.312 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.312 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.313 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:59.313 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.196092+0000", 2024-09-21T15:50:59.313 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7069000000000002e-05 2024-09-21T15:50:59.313 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.313 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:59.313 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.313 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.313 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.313 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63193 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 239616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:59.313 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:50:59.313 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 82.088887252999996, 2024-09-21T15:50:59.313 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053956577999999998, 2024-09-21T15:50:59.313 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:59.313 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:59.314 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:59.314 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:59.314 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:59.314 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63193 2024-09-21T15:50:59.314 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.314 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:59.314 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.314 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:59.314 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:50:59.314 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.314 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.314 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.314 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:59.314 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:50:59.315 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.315 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.315 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.315 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:59.315 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:50:59.315 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:59.315 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.315 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.315 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:59.315 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144223+0000", 2024-09-21T15:50:59.315 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0990000000000001e-06 2024-09-21T15:50:59.315 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.315 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.315 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:59.315 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144230+0000", 2024-09-21T15:50:59.316 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9070000000000001e-06 2024-09-21T15:50:59.316 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.316 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.316 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:59.316 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144234+0000", 2024-09-21T15:50:59.316 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9010000000000002e-06 2024-09-21T15:50:59.316 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.316 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.316 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:59.316 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.198004+0000", 2024-09-21T15:50:59.316 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053769539999999998 2024-09-21T15:50:59.316 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.316 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.317 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:59.317 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.198083+0000", 2024-09-21T15:50:59.317 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9531999999999997e-05 2024-09-21T15:50:59.317 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.317 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.317 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:59.317 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.198177+0000", 2024-09-21T15:50:59.317 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3906000000000005e-05 2024-09-21T15:50:59.317 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.317 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:59.317 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.317 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.317 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:59.318 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:59.319 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T15:50:59.333 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:50:59.332+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:50:59.421 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:59.421 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:59.421 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:50:59.421 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:59.430 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T15:50:59.473 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:59.472+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:59.532 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:59.532 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:50:59.532 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:50:59.532 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:50:59.532 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:59.534 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:50:59.534+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:50:59.540 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T15:50:59.606 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:50:59.650 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:50:59.649+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:50:59.684 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:50:59.684 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:50:59.684 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:50:59.684 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:50:59.684 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.684 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4209.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:50:59.684 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:50:59.684 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 128.49112532800001, 2024-09-21T15:50:59.684 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.19452909900000001, 2024-09-21T15:50:59.684 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:59.684 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:50:59.685 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:59.685 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4209", 2024-09-21T15:50:59.685 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/4045143050", 2024-09-21T15:50:59.685 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1 2024-09-21T15:50:59.685 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.685 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:59.685 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.685 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:59.685 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:50:59.685 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.685 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.685 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.685 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:59.685 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:50:59.685 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.685 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.686 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.686 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:59.686 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191161+0000", 2024-09-21T15:50:59.686 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-21T15:50:59.686 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.686 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.686 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:59.686 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191169+0000", 2024-09-21T15:50:59.686 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0090000000000007e-06 2024-09-21T15:50:59.686 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.686 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.686 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:59.686 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191180+0000", 2024-09-21T15:50:59.687 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0728e-05 2024-09-21T15:50:59.687 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.687 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.687 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:59.687 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191187+0000", 2024-09-21T15:50:59.687 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9009999999999994e-06 2024-09-21T15:50:59.687 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.687 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.687 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:59.687 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191238+0000", 2024-09-21T15:50:59.687 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0287000000000002e-05 2024-09-21T15:50:59.687 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.687 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.688 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-21T15:50:59.688 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191245+0000", 2024-09-21T15:50:59.688 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0709999999999997e-06 2024-09-21T15:50:59.688 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.688 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.688 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:59.688 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380309+0000", 2024-09-21T15:50:59.688 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.189064277 2024-09-21T15:50:59.688 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.688 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.688 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:59.688 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380454+0000", 2024-09-21T15:50:59.688 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014490799999999999 2024-09-21T15:50:59.688 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.688 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.688 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0", 2024-09-21T15:50:59.689 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380610+0000", 2024-09-21T15:50:59.689 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015650199999999999 2024-09-21T15:50:59.689 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.689 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.689 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:50:59.689 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.381160+0000", 2024-09-21T15:50:59.689 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00054903799999999996 2024-09-21T15:50:59.689 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.689 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.689 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:50:59.689 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385591+0000", 2024-09-21T15:50:59.689 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.004431272 2024-09-21T15:50:59.689 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.689 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.689 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:50:59.689 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385651+0000", 2024-09-21T15:50:59.690 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0455999999999999e-05 2024-09-21T15:50:59.690 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.690 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.690 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:59.690 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385691+0000", 2024-09-21T15:50:59.690 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0234999999999997e-05 2024-09-21T15:50:59.690 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.690 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:59.690 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.690 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.690 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.690 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64034 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1083392~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:59.690 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167717+0000", 2024-09-21T15:50:59.690 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 82.514570997999996, 2024-09-21T15:50:59.691 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048233316999999998, 2024-09-21T15:50:59.691 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:59.691 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:59.691 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:59.691 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:59.691 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:59.691 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64034 2024-09-21T15:50:59.691 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.691 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:59.691 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.691 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:59.691 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167717+0000", 2024-09-21T15:50:59.691 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.691 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.691 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.692 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:59.692 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167717+0000", 2024-09-21T15:50:59.692 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.692 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.692 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.692 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:59.692 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167716+0000", 2024-09-21T15:50:59.692 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:59.692 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.692 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.692 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:59.692 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167718+0000", 2024-09-21T15:50:59.692 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.165e-06 2024-09-21T15:50:59.692 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.692 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.693 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:59.693 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167725+0000", 2024-09-21T15:50:59.693 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3380000000000001e-06 2024-09-21T15:50:59.693 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.693 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.693 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:59.693 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167730+0000", 2024-09-21T15:50:59.693 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8010000000000003e-06 2024-09-21T15:50:59.693 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.693 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.693 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:59.693 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215807+0000", 2024-09-21T15:50:59.693 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048077823999999998 2024-09-21T15:50:59.693 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.694 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.694 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:59.694 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215884+0000", 2024-09-21T15:50:59.694 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6283000000000001e-05 2024-09-21T15:50:59.694 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.694 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.694 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:59.694 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215950+0000", 2024-09-21T15:50:59.694 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6290000000000004e-05 2024-09-21T15:50:59.694 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.694 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:59.694 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.694 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.694 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.695 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64035 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1091584~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:59.695 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:50:59.695 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 82.514553785000004, 2024-09-21T15:50:59.695 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048412117999999997, 2024-09-21T15:50:59.695 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:59.695 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:59.695 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:59.695 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:59.695 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:59.695 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64035 2024-09-21T15:50:59.695 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.695 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:59.695 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.695 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:59.695 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:50:59.695 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.696 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.696 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.696 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:59.696 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:50:59.696 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.696 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.696 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.696 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:59.696 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:50:59.696 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:59.696 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.696 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.696 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:59.696 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167746+0000", 2024-09-21T15:50:59.696 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2031000000000001e-05 2024-09-21T15:50:59.697 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.697 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.697 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:59.697 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167753+0000", 2024-09-21T15:50:59.697 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0620000000000003e-06 2024-09-21T15:50:59.697 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.697 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.697 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:59.697 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167759+0000", 2024-09-21T15:50:59.697 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.596e-06 2024-09-21T15:50:59.697 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.697 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.697 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:59.697 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215977+0000", 2024-09-21T15:50:59.697 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048217318000000002 2024-09-21T15:50:59.698 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.698 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.698 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:59.698 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216076+0000", 2024-09-21T15:50:59.698 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9846000000000006e-05 2024-09-21T15:50:59.698 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.698 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.698 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:59.698 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216146+0000", 2024-09-21T15:50:59.698 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9578000000000004e-05 2024-09-21T15:50:59.698 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.698 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:59.698 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.698 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.699 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.699 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64036 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1099776~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:59.699 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167853+0000", 2024-09-21T15:50:59.699 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 82.514435172000006, 2024-09-21T15:50:59.699 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048450675999999998, 2024-09-21T15:50:59.699 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:59.699 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:59.699 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:59.699 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:59.699 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:59.699 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64036 2024-09-21T15:50:59.699 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.699 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:59.699 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.699 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:59.700 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167853+0000", 2024-09-21T15:50:59.700 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.700 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.700 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.700 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:59.700 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167853+0000", 2024-09-21T15:50:59.700 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.700 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.700 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.700 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:59.700 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167852+0000", 2024-09-21T15:50:59.700 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:59.700 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.700 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.700 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:59.700 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167855+0000", 2024-09-21T15:50:59.701 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5640000000000001e-06 2024-09-21T15:50:59.701 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.701 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.701 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:59.701 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167864+0000", 2024-09-21T15:50:59.701 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8140000000000007e-06 2024-09-21T15:50:59.701 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.701 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.701 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:59.701 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167871+0000", 2024-09-21T15:50:59.701 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5729999999999996e-06 2024-09-21T15:50:59.701 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.701 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.701 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:59.701 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216174+0000", 2024-09-21T15:50:59.701 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048303437999999997 2024-09-21T15:50:59.702 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.702 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.702 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:59.702 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216250+0000", 2024-09-21T15:50:59.702 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5661999999999995e-05 2024-09-21T15:50:59.702 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.702 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.702 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:59.702 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216303+0000", 2024-09-21T15:50:59.702 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3273000000000003e-05 2024-09-21T15:50:59.702 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.702 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:59.702 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.702 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.702 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.703 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64037 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1771520~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:59.703 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:50:59.703 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 82.514411628999994, 2024-09-21T15:50:59.703 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048588771000000003, 2024-09-21T15:50:59.703 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:59.703 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:59.703 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:59.703 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:59.703 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:59.703 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64037 2024-09-21T15:50:59.703 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.703 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:59.703 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.703 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:59.703 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:50:59.704 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.704 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.704 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.704 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:59.704 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:50:59.704 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.704 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.704 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.704 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:59.704 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:50:59.704 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:59.704 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.704 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.704 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:59.704 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167878+0000", 2024-09-21T15:50:59.704 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3e-06 2024-09-21T15:50:59.705 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.705 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.705 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:59.705 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167884+0000", 2024-09-21T15:50:59.705 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3269999999999997e-06 2024-09-21T15:50:59.705 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.705 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.705 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:59.705 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167889+0000", 2024-09-21T15:50:59.705 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9269999999999996e-06 2024-09-21T15:50:59.705 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.705 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.706 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:59.706 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216331+0000", 2024-09-21T15:50:59.706 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048442102000000001 2024-09-21T15:50:59.706 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.706 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.706 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:59.706 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216393+0000", 2024-09-21T15:50:59.706 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1132999999999999e-05 2024-09-21T15:50:59.706 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.706 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.706 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:59.706 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216465+0000", 2024-09-21T15:50:59.706 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2339e-05 2024-09-21T15:50:59.706 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.707 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:59.707 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.707 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.707 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.707 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64038 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1779712~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:59.707 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:50:59.707 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 82.514393798, 2024-09-21T15:50:59.707 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048749795999999998, 2024-09-21T15:50:59.707 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:59.707 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:59.707 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:59.707 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:59.707 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:59.707 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64038 2024-09-21T15:50:59.708 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.708 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:59.708 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.708 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:59.708 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:50:59.708 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.708 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.708 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.708 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:59.708 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:50:59.708 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.708 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.708 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.708 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:59.708 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:50:59.709 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:59.709 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.709 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.709 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:59.709 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167896+0000", 2024-09-21T15:50:59.709 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2419999999999999e-06 2024-09-21T15:50:59.709 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.709 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.709 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:59.709 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167902+0000", 2024-09-21T15:50:59.709 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0859999999999998e-06 2024-09-21T15:50:59.709 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.709 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.709 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:59.709 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167907+0000", 2024-09-21T15:50:59.709 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9679999999999997e-06 2024-09-21T15:50:59.710 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.710 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.710 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:59.710 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216506+0000", 2024-09-21T15:50:59.710 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048599374000000001 2024-09-21T15:50:59.710 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.710 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.710 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:59.710 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216579+0000", 2024-09-21T15:50:59.710 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2801999999999999e-05 2024-09-21T15:50:59.710 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.710 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.710 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:59.710 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216644+0000", 2024-09-21T15:50:59.711 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4715000000000006e-05 2024-09-21T15:50:59.711 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.711 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:59.711 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.711 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.711 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.711 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64039 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1107968~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:59.711 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:50:59.711 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 82.514376443000003, 2024-09-21T15:50:59.711 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048911370000000003, 2024-09-21T15:50:59.711 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:59.711 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:59.711 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:59.711 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:59.712 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:59.712 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64039 2024-09-21T15:50:59.712 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.712 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:59.712 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.712 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:59.712 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:50:59.712 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.712 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.712 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.712 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:59.712 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:50:59.712 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.712 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.712 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.713 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:59.713 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:50:59.713 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:59.713 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.713 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.713 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:59.713 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167913+0000", 2024-09-21T15:50:59.713 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0839999999999999e-06 2024-09-21T15:50:59.713 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.713 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.713 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:59.713 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167921+0000", 2024-09-21T15:50:59.713 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8310000000000001e-06 2024-09-21T15:50:59.713 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.714 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.714 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:59.714 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167926+0000", 2024-09-21T15:50:59.714 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0270000000000003e-06 2024-09-21T15:50:59.714 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.714 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.714 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:59.714 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216680+0000", 2024-09-21T15:50:59.714 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048753890000000001 2024-09-21T15:50:59.714 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.714 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.714 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:59.714 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216757+0000", 2024-09-21T15:50:59.714 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6848999999999996e-05 2024-09-21T15:50:59.715 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.715 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.715 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:59.715 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216823+0000", 2024-09-21T15:50:59.715 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6068000000000005e-05 2024-09-21T15:50:59.715 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.715 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:59.715 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.715 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.715 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.715 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64040 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1787904~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:59.715 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167931+0000", 2024-09-21T15:50:59.715 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 82.514357211000004, 2024-09-21T15:50:59.715 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049010561000000001, 2024-09-21T15:50:59.716 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:59.716 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:59.716 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:59.716 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:59.716 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:59.716 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64040 2024-09-21T15:50:59.716 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.716 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:59.716 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.716 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:59.716 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167931+0000", 2024-09-21T15:50:59.716 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.716 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.716 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.716 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:59.716 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167931+0000", 2024-09-21T15:50:59.717 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.717 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.717 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.717 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:59.717 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167930+0000", 2024-09-21T15:50:59.717 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:59.717 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.717 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.717 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:59.717 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167932+0000", 2024-09-21T15:50:59.717 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.198e-06 2024-09-21T15:50:59.717 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.717 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.717 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:59.717 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167938+0000", 2024-09-21T15:50:59.717 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.105e-06 2024-09-21T15:50:59.718 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.718 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.718 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:59.718 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167943+0000", 2024-09-21T15:50:59.718 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9429999999999999e-06 2024-09-21T15:50:59.718 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.718 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.718 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:59.718 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216848+0000", 2024-09-21T15:50:59.718 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048905200000000003 2024-09-21T15:50:59.718 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.719 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.719 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:59.719 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216903+0000", 2024-09-21T15:50:59.719 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4793999999999997e-05 2024-09-21T15:50:59.719 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.719 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.719 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:59.719 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216941+0000", 2024-09-21T15:50:59.719 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7790999999999998e-05 2024-09-21T15:50:59.719 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.719 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:59.719 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.719 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.719 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.719 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64041 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1796096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:59.720 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:50:59.720 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 82.514340239999996, 2024-09-21T15:50:59.720 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049181457999999997, 2024-09-21T15:50:59.720 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:59.720 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:59.720 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:59.720 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:59.720 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:59.720 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64041 2024-09-21T15:50:59.720 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.720 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:59.720 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.720 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:59.720 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:50:59.720 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.720 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.721 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.721 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:59.721 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:50:59.721 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.721 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.721 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.721 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:59.721 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:50:59.721 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:59.721 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.721 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.721 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:59.721 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167949+0000", 2024-09-21T15:50:59.721 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0200000000000001e-06 2024-09-21T15:50:59.721 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.722 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.722 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:59.722 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167955+0000", 2024-09-21T15:50:59.722 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5990000000000001e-06 2024-09-21T15:50:59.722 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.722 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.722 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:59.722 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167959+0000", 2024-09-21T15:50:59.722 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5589999999999999e-06 2024-09-21T15:50:59.722 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.722 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.722 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:59.722 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216972+0000", 2024-09-21T15:50:59.722 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049012574000000003 2024-09-21T15:50:59.722 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.723 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.723 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:59.723 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217057+0000", 2024-09-21T15:50:59.723 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4820000000000004e-05 2024-09-21T15:50:59.723 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.723 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.723 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:59.723 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217129+0000", 2024-09-21T15:50:59.723 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2186000000000003e-05 2024-09-21T15:50:59.723 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.723 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:59.723 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.723 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.724 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.724 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64042 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1804288~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:59.724 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167964+0000", 2024-09-21T15:50:59.724 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 82.514323949000001, 2024-09-21T15:50:59.724 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049333372, 2024-09-21T15:50:59.724 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:59.724 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:59.724 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:59.724 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:59.724 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:59.724 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64042 2024-09-21T15:50:59.724 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.724 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:59.724 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.724 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:59.724 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167964+0000", 2024-09-21T15:50:59.725 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.725 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.725 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.725 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:59.725 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167964+0000", 2024-09-21T15:50:59.725 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.725 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.725 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.725 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:59.725 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167963+0000", 2024-09-21T15:50:59.725 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:59.725 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.725 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.725 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:59.725 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167966+0000", 2024-09-21T15:50:59.726 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1119999999999999e-06 2024-09-21T15:50:59.726 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.726 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.726 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:59.726 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167975+0000", 2024-09-21T15:50:59.726 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1130000000000007e-06 2024-09-21T15:50:59.726 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.726 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.726 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:59.726 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167980+0000", 2024-09-21T15:50:59.726 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9599999999999999e-06 2024-09-21T15:50:59.726 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.726 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.727 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:59.727 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217158+0000", 2024-09-21T15:50:59.727 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049178569999999998 2024-09-21T15:50:59.727 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.727 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.727 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:59.727 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217232+0000", 2024-09-21T15:50:59.727 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3589000000000003e-05 2024-09-21T15:50:59.727 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.727 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.727 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:59.727 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217297+0000", 2024-09-21T15:50:59.727 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5339999999999994e-05 2024-09-21T15:50:59.728 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.728 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:59.728 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.728 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.728 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.728 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64043 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1812480~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:50:59.728 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.168085+0000", 2024-09-21T15:50:59.728 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 82.514202882000006, 2024-09-21T15:50:59.728 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049365157999999999, 2024-09-21T15:50:59.728 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:59.728 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:59.728 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:59.728 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:59.729 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:59.729 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64043 2024-09-21T15:50:59.729 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.729 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:59.729 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.729 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:59.729 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168085+0000", 2024-09-21T15:50:59.729 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.729 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.729 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.729 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:59.729 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168085+0000", 2024-09-21T15:50:59.729 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.729 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.729 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.730 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:59.730 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168084+0000", 2024-09-21T15:50:59.730 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:59.730 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.730 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.730 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:59.730 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168088+0000", 2024-09-21T15:50:59.730 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5420000000000001e-06 2024-09-21T15:50:59.730 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.730 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.730 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:59.730 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168097+0000", 2024-09-21T15:50:59.730 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5410000000000006e-06 2024-09-21T15:50:59.730 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.731 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.731 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:59.731 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168104+0000", 2024-09-21T15:50:59.731 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6719999999999998e-06 2024-09-21T15:50:59.731 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.731 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.731 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:59.731 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217323+0000", 2024-09-21T15:50:59.731 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049218857999999997 2024-09-21T15:50:59.731 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.731 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.731 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:59.731 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217397+0000", 2024-09-21T15:50:59.731 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3932999999999998e-05 2024-09-21T15:50:59.732 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.732 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.732 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:59.732 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217450+0000", 2024-09-21T15:50:59.732 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3168000000000003e-05 2024-09-21T15:50:59.732 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.732 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:59.732 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.732 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.732 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.733 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155255 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 323584~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:59.733 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.440996+0000", 2024-09-21T15:50:59.733 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 37.241291355000001, 2024-09-21T15:50:59.733 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048233848000000003, 2024-09-21T15:50:59.733 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:59.733 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:59.734 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:59.734 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:59.734 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:59.734 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155255 2024-09-21T15:50:59.734 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.734 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:59.735 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.735 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:59.735 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.440996+0000", 2024-09-21T15:50:59.735 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.735 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.735 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.736 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:59.736 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.440996+0000", 2024-09-21T15:50:59.736 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.736 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.736 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.736 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:59.736 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.440996+0000", 2024-09-21T15:50:59.737 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:59.737 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.737 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.737 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:59.737 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441004+0000", 2024-09-21T15:50:59.737 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8460000000000008e-06 2024-09-21T15:50:59.738 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.738 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.738 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:59.738 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441018+0000", 2024-09-21T15:50:59.738 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.42e-05 2024-09-21T15:50:59.738 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.739 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.739 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:59.739 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441023+0000", 2024-09-21T15:50:59.739 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0019999999999997e-06 2024-09-21T15:50:59.739 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.739 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.740 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:59.740 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489060+0000", 2024-09-21T15:50:59.740 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048036660000000002 2024-09-21T15:50:59.740 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.740 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.740 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:59.740 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489107+0000", 2024-09-21T15:50:59.741 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7153000000000001e-05 2024-09-21T15:50:59.741 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.741 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.741 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:59.741 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489230+0000", 2024-09-21T15:50:59.741 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012328600000000001 2024-09-21T15:50:59.741 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.742 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:59.742 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.742 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.742 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.742 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155256 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 356352~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:59.742 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441027+0000", 2024-09-21T15:50:59.743 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 37.241260267999998, 2024-09-21T15:50:59.743 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048677606999999998, 2024-09-21T15:50:59.743 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:59.743 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:59.743 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:59.744 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:59.744 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:59.744 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155256 2024-09-21T15:50:59.744 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.744 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:59.744 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.744 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:59.745 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441027+0000", 2024-09-21T15:50:59.745 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.745 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.745 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.745 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:59.745 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441027+0000", 2024-09-21T15:50:59.745 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.746 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.746 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.746 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:59.746 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441027+0000", 2024-09-21T15:50:59.746 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:59.746 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.746 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.746 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:59.747 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441029+0000", 2024-09-21T15:50:59.747 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.39e-06 2024-09-21T15:50:59.747 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.747 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.747 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:59.747 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441036+0000", 2024-09-21T15:50:59.748 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1129999999999997e-06 2024-09-21T15:50:59.748 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.748 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.748 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:59.748 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441040+0000", 2024-09-21T15:50:59.748 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8459999999999999e-06 2024-09-21T15:50:59.748 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.748 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.748 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:59.749 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489244+0000", 2024-09-21T15:50:59.749 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048203751000000003 2024-09-21T15:50:59.749 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.749 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.749 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:59.749 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489288+0000", 2024-09-21T15:50:59.749 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4153999999999998e-05 2024-09-21T15:50:59.749 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.749 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.749 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:59.749 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489705+0000", 2024-09-21T15:50:59.749 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000416773 2024-09-21T15:50:59.749 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.749 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:59.749 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.749 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.750 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.750 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155257 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 389120~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:59.750 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441045+0000", 2024-09-21T15:50:59.750 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 37.241243177000001, 2024-09-21T15:50:59.750 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049000325999999997, 2024-09-21T15:50:59.750 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:59.750 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:59.750 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:59.750 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:59.750 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:59.750 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155257 2024-09-21T15:50:59.750 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.750 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:59.751 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.751 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:59.751 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441045+0000", 2024-09-21T15:50:59.751 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.751 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.751 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.751 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:59.751 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441045+0000", 2024-09-21T15:50:59.751 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.751 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.751 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.751 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:59.751 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441044+0000", 2024-09-21T15:50:59.751 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:59.751 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.752 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.752 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:59.752 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441046+0000", 2024-09-21T15:50:59.752 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1909999999999999e-06 2024-09-21T15:50:59.752 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.752 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.752 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:59.752 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441054+0000", 2024-09-21T15:50:59.752 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3900000000000004e-06 2024-09-21T15:50:59.752 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.752 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.752 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:59.752 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441058+0000", 2024-09-21T15:50:59.752 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4619999999999998e-06 2024-09-21T15:50:59.752 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.752 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.753 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:59.753 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489741+0000", 2024-09-21T15:50:59.753 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048682660000000003 2024-09-21T15:50:59.753 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.753 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.753 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:59.753 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489813+0000", 2024-09-21T15:50:59.753 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2051999999999994e-05 2024-09-21T15:50:59.753 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.753 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.753 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:59.753 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490045+0000", 2024-09-21T15:50:59.753 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00023191299999999999 2024-09-21T15:50:59.754 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.754 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:59.754 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.754 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.754 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.754 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155258 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 421888~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:59.754 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441062+0000", 2024-09-21T15:50:59.754 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 37.241225634999999, 2024-09-21T15:50:59.754 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049339651999999998, 2024-09-21T15:50:59.754 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:59.754 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:59.754 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:59.754 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:59.754 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:59.755 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155258 2024-09-21T15:50:59.755 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.755 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:59.755 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.755 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:59.755 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441062+0000", 2024-09-21T15:50:59.755 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.755 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.755 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.755 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:59.755 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441062+0000", 2024-09-21T15:50:59.755 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.755 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.755 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.755 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:59.755 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441062+0000", 2024-09-21T15:50:59.756 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:59.756 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.756 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.756 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:59.756 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441064+0000", 2024-09-21T15:50:59.756 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.967e-06 2024-09-21T15:50:59.756 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.756 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.756 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:59.756 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441069+0000", 2024-09-21T15:50:59.756 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1529999999999996e-06 2024-09-21T15:50:59.756 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.756 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.756 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:59.756 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441073+0000", 2024-09-21T15:50:59.756 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3850000000000004e-06 2024-09-21T15:50:59.757 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.757 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.757 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:59.757 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490082+0000", 2024-09-21T15:50:59.757 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.04900885 2024-09-21T15:50:59.757 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.757 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.757 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:59.757 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490166+0000", 2024-09-21T15:50:59.757 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3640999999999994e-05 2024-09-21T15:50:59.757 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.757 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.757 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:59.757 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490402+0000", 2024-09-21T15:50:59.758 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000235978 2024-09-21T15:50:59.758 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.758 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:59.758 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.758 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.758 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.758 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155259 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 454656~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:59.758 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441086+0000", 2024-09-21T15:50:59.758 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 37.241201830000001, 2024-09-21T15:50:59.758 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049624397000000001, 2024-09-21T15:50:59.758 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:59.758 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:59.758 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:59.759 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:59.759 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:59.759 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155259 2024-09-21T15:50:59.759 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.759 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:59.759 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.759 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:59.759 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441086+0000", 2024-09-21T15:50:59.759 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.759 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.759 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.759 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:59.759 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441086+0000", 2024-09-21T15:50:59.759 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.760 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.760 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.760 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:59.760 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441086+0000", 2024-09-21T15:50:59.760 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:59.760 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.760 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.760 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:59.760 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441088+0000", 2024-09-21T15:50:59.760 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2630000000000002e-06 2024-09-21T15:50:59.760 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.760 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.760 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:59.760 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441093+0000", 2024-09-21T15:50:59.760 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5010000000000004e-06 2024-09-21T15:50:59.761 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.761 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.761 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:59.761 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441098+0000", 2024-09-21T15:50:59.761 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.707e-06 2024-09-21T15:50:59.761 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.761 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.761 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:59.761 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490447+0000", 2024-09-21T15:50:59.761 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049349125000000001 2024-09-21T15:50:59.761 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.761 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.761 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:59.761 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490504+0000", 2024-09-21T15:50:59.762 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6521000000000003e-05 2024-09-21T15:50:59.762 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.762 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.762 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:59.762 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490710+0000", 2024-09-21T15:50:59.762 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00020663999999999999 2024-09-21T15:50:59.762 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.762 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:59.762 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.762 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.762 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.763 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155260 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 487424~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:59.763 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441160+0000", 2024-09-21T15:50:59.763 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 37.241127868, 2024-09-21T15:50:59.763 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049876810000000001, 2024-09-21T15:50:59.763 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:59.763 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:59.763 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:59.763 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:59.763 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:59.763 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155260 2024-09-21T15:50:59.763 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.764 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:59.764 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.764 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:59.764 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441160+0000", 2024-09-21T15:50:59.764 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.764 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.764 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.764 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:59.764 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441160+0000", 2024-09-21T15:50:59.764 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.764 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.764 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.765 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:59.765 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441159+0000", 2024-09-21T15:50:59.765 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:59.765 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.765 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.765 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:59.765 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441164+0000", 2024-09-21T15:50:59.765 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4059999999999998e-06 2024-09-21T15:50:59.765 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.765 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.765 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:59.765 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441175+0000", 2024-09-21T15:50:59.765 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1095e-05 2024-09-21T15:50:59.766 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.766 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.766 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:59.766 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441182+0000", 2024-09-21T15:50:59.766 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2139999999999999e-06 2024-09-21T15:50:59.766 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.766 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.766 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:59.766 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490751+0000", 2024-09-21T15:50:59.766 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049569230999999998 2024-09-21T15:50:59.766 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.766 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.766 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:59.767 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490837+0000", 2024-09-21T15:50:59.767 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6017e-05 2024-09-21T15:50:59.767 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.767 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.767 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:59.767 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491037+0000", 2024-09-21T15:50:59.767 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00019952300000000001 2024-09-21T15:50:59.767 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.767 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:59.767 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.767 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.768 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.768 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155261 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 520192~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:59.768 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441187+0000", 2024-09-21T15:50:59.768 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 37.241100684999999, 2024-09-21T15:50:59.768 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.050124931999999997, 2024-09-21T15:50:59.768 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:59.768 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:59.768 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:59.768 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:59.768 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:59.768 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155261 2024-09-21T15:50:59.768 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.768 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:59.768 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.769 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:59.769 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441187+0000", 2024-09-21T15:50:59.769 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.769 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.769 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.769 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:59.769 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441187+0000", 2024-09-21T15:50:59.769 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.769 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.769 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.769 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:59.769 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441187+0000", 2024-09-21T15:50:59.769 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:59.769 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.770 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.770 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:59.770 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441189+0000", 2024-09-21T15:50:59.770 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3649999999999998e-06 2024-09-21T15:50:59.770 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.770 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.770 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:59.770 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441196+0000", 2024-09-21T15:50:59.770 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5379999999999999e-06 2024-09-21T15:50:59.770 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.770 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.770 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:59.771 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441200+0000", 2024-09-21T15:50:59.771 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4719999999999997e-06 2024-09-21T15:50:59.771 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.771 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.771 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:59.771 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491070+0000", 2024-09-21T15:50:59.771 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049869915000000001 2024-09-21T15:50:59.771 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.771 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.771 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:59.771 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491146+0000", 2024-09-21T15:50:59.771 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6589999999999997e-05 2024-09-21T15:50:59.771 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.772 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.772 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:59.772 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491312+0000", 2024-09-21T15:50:59.772 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000165475 2024-09-21T15:50:59.772 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.772 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:59.772 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.772 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.772 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.772 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155308 3.11 3:8b5183e5:::rbd_data.11335aef9442.0000000000000000:2a [read 356352~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:59.772 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441369+0000", 2024-09-21T15:50:59.772 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 37.240918837000002, 2024-09-21T15:50:59.772 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.050090672000000003, 2024-09-21T15:50:59.772 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:59.772 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:59.772 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:59.773 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:59.773 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:59.773 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155308 2024-09-21T15:50:59.773 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.773 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:59.773 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.773 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:59.773 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441369+0000", 2024-09-21T15:50:59.773 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.773 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.773 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.773 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:59.773 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441369+0000", 2024-09-21T15:50:59.773 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.774 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.774 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.774 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:59.774 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441368+0000", 2024-09-21T15:50:59.774 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:59.774 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.774 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.774 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:59.774 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441371+0000", 2024-09-21T15:50:59.774 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1100000000000001e-06 2024-09-21T15:50:59.774 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.774 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.774 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:59.774 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441377+0000", 2024-09-21T15:50:59.775 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5649999999999998e-06 2024-09-21T15:50:59.775 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.775 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.775 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:59.775 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441382+0000", 2024-09-21T15:50:59.775 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5929999999999999e-06 2024-09-21T15:50:59.775 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.775 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.775 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:59.775 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491349+0000", 2024-09-21T15:50:59.775 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049967122000000003 2024-09-21T15:50:59.775 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.775 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.775 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:59.776 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491400+0000", 2024-09-21T15:50:59.776 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1589999999999999e-05 2024-09-21T15:50:59.776 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.776 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.776 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:59.776 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491460+0000", 2024-09-21T15:50:59.776 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9070000000000002e-05 2024-09-21T15:50:59.776 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.776 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:59.776 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.776 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.776 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.776 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155309 3.11 3:8b5183e5:::rbd_data.11335aef9442.0000000000000000:2a [read 389120~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:50:59.776 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441386+0000", 2024-09-21T15:50:59.776 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 37.240901487999999, 2024-09-21T15:50:59.777 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.050184777, 2024-09-21T15:50:59.777 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:50:59.777 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:50:59.777 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:50:59.777 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:50:59.777 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:50:59.777 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155309 2024-09-21T15:50:59.777 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.777 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:50:59.777 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.777 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:50:59.777 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441386+0000", 2024-09-21T15:50:59.777 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.777 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.777 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.778 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:50:59.778 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441386+0000", 2024-09-21T15:50:59.778 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:50:59.778 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.778 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.778 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:50:59.778 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441386+0000", 2024-09-21T15:50:59.778 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:50:59.778 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.778 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.778 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:50:59.778 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441388+0000", 2024-09-21T15:50:59.778 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4420000000000001e-06 2024-09-21T15:50:59.779 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.779 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.779 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:50:59.779 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441395+0000", 2024-09-21T15:50:59.779 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6189999999999996e-06 2024-09-21T15:50:59.779 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.779 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.779 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:50:59.779 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441400+0000", 2024-09-21T15:50:59.779 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.118e-06 2024-09-21T15:50:59.779 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.779 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.779 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:50:59.779 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491473+0000", 2024-09-21T15:50:59.779 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.050072783000000003 2024-09-21T15:50:59.780 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.780 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.780 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:50:59.780 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491518+0000", 2024-09-21T15:50:59.780 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4854000000000001e-05 2024-09-21T15:50:59.780 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:50:59.780 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:50:59.780 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:50:59.780 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491571+0000", 2024-09-21T15:50:59.780 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3358e-05 2024-09-21T15:50:59.780 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.780 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:59.780 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.780 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:50:59.781 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:50:59.781 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:50:59.781 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T15:50:59.782 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-09-21T15:50:59.782 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:50:59.783 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:50:59.783 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:50:59.784 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:50:59.784 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:59.781+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:59.784 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:50:59.784 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:50:59.785 INFO:teuthology.orchestra.run.smithi089.stderr:reweighted osd.0 to 1 (10000) 2024-09-21T15:50:59.785 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:50:59.785 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:50:59.786 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-21T15:50:59.797 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:50:59.797 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:50:59.816 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:50:59.816 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:50:59.837 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:50:59.837 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:50:59.868 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:50:59.868 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:50:59.883 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:50:59.882+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:50:59.925 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:50:59.925 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:50:59.936 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:50:59.984 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:50:59.983+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:51:00.041 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:00.041 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:00.049 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:51:00.090 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:51:00.089+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:51:00.186 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:51:00.184+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:51:00.192 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:00.192 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:00.200 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:51:00.286 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:51:00.285+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:51:00.301 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:00.301 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:00.309 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:51:00.388 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:51:00.386+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:51:00.411 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:00.411 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:00.420 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:51:00.488 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:51:00.487+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:51:00.522 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:00.522 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:00.531 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:51:00.589 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:51:00.588+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:51:00.633 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:00.699 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:00.701 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:51:00.701 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:51:00.700+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:51:00.783 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is unset 2024-09-21T15:51:00.784 INFO:teuthology.orchestra.run.smithi089.stderr:reweighted osd.1 to 1 (10000) 2024-09-21T15:51:00.799 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-21T15:51:00.843 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:51:00.841+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:51:00.857 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:00.857 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:00.867 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:51:00.902 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:51:00.901+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:51:01.004 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:51:01.003+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:51:01.012 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:01.012 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:01.021 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:51:01.104 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:51:01.103+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:51:01.124 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:01.124 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:01.133 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:51:01.217 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:51:01.216+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:51:01.276 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:01.276 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:01.284 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:51:01.306 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:51:01.304+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:51:01.406 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:51:01.405+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:51:01.426 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:01.427 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:01.435 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:51:01.507 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:51:01.506+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:51:01.549 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:01.549 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:01.559 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:51:01.608 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:51:01.607+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:51:01.702 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:01.702 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:01.709 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:51:01.708+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:51:01.710 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:51:01.780 INFO:teuthology.orchestra.run.smithi089.stderr:reweighted osd.2 to 1 (10000) 2024-09-21T15:51:01.796 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-21T15:51:01.837 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:51:01.836+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:51:01.861 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:01.862 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:01.871 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:51:01.882 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:51:01.926 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:51:01.925+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:51:01.975 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:01.975 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:01.984 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:51:02.008 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-09-21T15:51:02.008 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:51:02.025 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:51:02.025 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:51:02.027 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:51:02.026+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:51:02.041 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:51:02.041 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:51:02.057 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:51:02.057 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:51:02.086 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:02.086 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:02.095 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:51:02.136 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:51:02.135+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:51:02.214 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:51:02.213+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:51:02.237 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:02.237 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:02.246 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T15:51:02.273 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:51:02.274 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:51:02.291 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:51:02.291 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:51:02.307 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:51:02.307 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:51:02.314 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:51:02.313+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:51:02.323 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:51:02.323 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:51:02.350 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:51:02.350 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:51:02.350 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:51:02.350 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:51:02.358 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T15:51:02.415 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:51:02.414+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:51:02.460 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:51:02.460 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:51:02.460 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:51:02.460 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:51:02.460 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:51:02.469 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T15:51:02.516 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:51:02.515+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:51:02.572 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:51:02.572 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:51:02.572 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:51:02.572 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:51:02.572 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.572 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63140 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 876544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:02.572 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.133649+0000", 2024-09-21T15:51:02.572 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 85.436436499999999, 2024-09-21T15:51:02.572 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053950201000000003, 2024-09-21T15:51:02.573 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:02.573 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:02.573 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:02.573 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:02.573 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:02.573 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63140 2024-09-21T15:51:02.573 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.573 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:02.573 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.573 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:02.573 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133649+0000", 2024-09-21T15:51:02.573 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:02.573 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.573 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.574 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:02.574 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133649+0000", 2024-09-21T15:51:02.574 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:02.574 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.574 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.574 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:02.574 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133648+0000", 2024-09-21T15:51:02.574 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:02.574 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.574 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.574 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:02.574 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133651+0000", 2024-09-21T15:51:02.574 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1549999999999999e-06 2024-09-21T15:51:02.575 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.575 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.575 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:02.575 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133660+0000", 2024-09-21T15:51:02.575 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3899999999999993e-06 2024-09-21T15:51:02.575 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.575 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.575 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:02.575 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133669+0000", 2024-09-21T15:51:02.575 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3219999999999999e-06 2024-09-21T15:51:02.575 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.575 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.575 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:02.575 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187409+0000", 2024-09-21T15:51:02.575 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053740257 2024-09-21T15:51:02.576 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.576 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.576 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:02.576 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187492+0000", 2024-09-21T15:51:02.576 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2409000000000003e-05 2024-09-21T15:51:02.576 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.576 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.576 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:02.576 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187599+0000", 2024-09-21T15:51:02.576 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000107334 2024-09-21T15:51:02.576 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:02.576 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:02.576 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:02.576 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.576 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.577 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63141 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 884736~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:02.577 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.133843+0000", 2024-09-21T15:51:02.577 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 85.436242381, 2024-09-21T15:51:02.577 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053954015000000001, 2024-09-21T15:51:02.577 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:02.577 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:02.577 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:02.577 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:02.577 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:02.577 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63141 2024-09-21T15:51:02.577 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.577 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:02.577 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.577 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:02.577 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133843+0000", 2024-09-21T15:51:02.578 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:02.578 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.578 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.578 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:02.578 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133843+0000", 2024-09-21T15:51:02.578 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:02.578 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.578 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.578 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:02.578 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133673+0000", 2024-09-21T15:51:02.578 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998302 2024-09-21T15:51:02.578 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.578 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.578 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:02.578 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133846+0000", 2024-09-21T15:51:02.578 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000172762 2024-09-21T15:51:02.579 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.579 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.579 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:02.579 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133855+0000", 2024-09-21T15:51:02.579 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2940000000000006e-06 2024-09-21T15:51:02.579 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.579 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.579 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:02.579 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133863+0000", 2024-09-21T15:51:02.579 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9740000000000002e-06 2024-09-21T15:51:02.579 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.579 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.579 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:02.579 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187630+0000", 2024-09-21T15:51:02.579 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053767211000000002 2024-09-21T15:51:02.580 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.580 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.580 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:02.580 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187707+0000", 2024-09-21T15:51:02.580 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7446000000000004e-05 2024-09-21T15:51:02.580 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.580 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.580 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:02.580 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187797+0000", 2024-09-21T15:51:02.580 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.9858000000000006e-05 2024-09-21T15:51:02.580 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:02.580 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:02.580 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:02.580 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.580 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.580 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63142 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 892928~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:02.581 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134046+0000", 2024-09-21T15:51:02.581 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 85.436039438999998, 2024-09-21T15:51:02.581 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054017849999999999, 2024-09-21T15:51:02.581 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:02.581 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:02.581 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:02.581 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:02.581 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:02.581 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63142 2024-09-21T15:51:02.581 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.581 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:02.581 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.581 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:02.581 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134046+0000", 2024-09-21T15:51:02.581 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:02.582 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.582 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.582 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:02.582 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134046+0000", 2024-09-21T15:51:02.582 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:02.582 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.582 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.582 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:02.582 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133867+0000", 2024-09-21T15:51:02.582 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998217 2024-09-21T15:51:02.582 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.582 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.582 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:02.582 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134049+0000", 2024-09-21T15:51:02.582 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000181695 2024-09-21T15:51:02.583 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.583 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.583 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:02.583 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134057+0000", 2024-09-21T15:51:02.583 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.303e-06 2024-09-21T15:51:02.583 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.583 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.583 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:02.583 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134063+0000", 2024-09-21T15:51:02.583 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9959999999999999e-06 2024-09-21T15:51:02.583 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.583 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.583 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:02.583 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187849+0000", 2024-09-21T15:51:02.583 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053785571999999997 2024-09-21T15:51:02.584 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.584 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.584 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:02.584 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187954+0000", 2024-09-21T15:51:02.584 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010524899999999999 2024-09-21T15:51:02.584 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.584 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.584 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:02.584 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188064+0000", 2024-09-21T15:51:02.584 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000109549 2024-09-21T15:51:02.584 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:02.584 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:02.584 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:02.584 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.584 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.584 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63143 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 901120~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:02.585 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134239+0000", 2024-09-21T15:51:02.585 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 85.435846846000004, 2024-09-21T15:51:02.585 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054026058000000002, 2024-09-21T15:51:02.585 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:02.585 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:02.585 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:02.585 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:02.585 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:02.585 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63143 2024-09-21T15:51:02.585 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.585 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:02.585 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.585 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:02.585 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134239+0000", 2024-09-21T15:51:02.585 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:02.586 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.586 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.586 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:02.586 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134239+0000", 2024-09-21T15:51:02.586 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:02.586 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.586 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.586 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:02.586 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134068+0000", 2024-09-21T15:51:02.586 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998298 2024-09-21T15:51:02.586 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.586 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.586 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:02.586 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134242+0000", 2024-09-21T15:51:02.586 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00017414199999999999 2024-09-21T15:51:02.587 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.587 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.587 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:02.587 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134264+0000", 2024-09-21T15:51:02.587 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1552e-05 2024-09-21T15:51:02.587 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.587 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.587 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:02.587 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134270+0000", 2024-09-21T15:51:02.587 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6939999999999998e-06 2024-09-21T15:51:02.587 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.587 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.587 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:02.587 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188108+0000", 2024-09-21T15:51:02.587 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.05383719 2024-09-21T15:51:02.588 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.588 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.588 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:02.588 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188186+0000", 2024-09-21T15:51:02.588 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8318000000000005e-05 2024-09-21T15:51:02.588 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.588 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.588 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:02.588 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188265+0000", 2024-09-21T15:51:02.588 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8597000000000007e-05 2024-09-21T15:51:02.588 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:02.588 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:02.588 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:02.588 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.588 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.589 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63144 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 909312~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:02.589 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134432+0000", 2024-09-21T15:51:02.589 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 85.435653290000005, 2024-09-21T15:51:02.589 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053968953, 2024-09-21T15:51:02.589 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:02.589 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:02.589 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:02.589 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:02.589 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:02.589 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63144 2024-09-21T15:51:02.589 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.589 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:02.589 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.589 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:02.589 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134432+0000", 2024-09-21T15:51:02.589 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:02.590 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.590 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.590 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:02.590 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134432+0000", 2024-09-21T15:51:02.590 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:02.590 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.590 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.590 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:02.590 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134276+0000", 2024-09-21T15:51:02.590 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998441 2024-09-21T15:51:02.591 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.591 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.591 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:02.591 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134436+0000", 2024-09-21T15:51:02.591 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000159591 2024-09-21T15:51:02.591 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.591 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.591 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:02.591 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134445+0000", 2024-09-21T15:51:02.591 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6780000000000006e-06 2024-09-21T15:51:02.591 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.591 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.591 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:02.592 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134452+0000", 2024-09-21T15:51:02.592 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6830000000000008e-06 2024-09-21T15:51:02.592 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.592 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.592 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:02.592 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188282+0000", 2024-09-21T15:51:02.592 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053829252000000001 2024-09-21T15:51:02.592 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.592 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.592 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:02.592 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188328+0000", 2024-09-21T15:51:02.592 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6734999999999999e-05 2024-09-21T15:51:02.592 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.592 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.593 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:02.593 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188401+0000", 2024-09-21T15:51:02.593 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2731000000000005e-05 2024-09-21T15:51:02.593 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:02.593 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:02.593 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:02.593 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.593 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.593 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63145 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 860160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:02.593 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134736+0000", 2024-09-21T15:51:02.593 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 85.435349505000005, 2024-09-21T15:51:02.593 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053865960999999997, 2024-09-21T15:51:02.593 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:02.593 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:02.593 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:02.593 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:02.594 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:02.594 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63145 2024-09-21T15:51:02.594 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.594 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:02.594 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.594 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:02.594 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134736+0000", 2024-09-21T15:51:02.594 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:02.594 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.594 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.594 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:02.594 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134736+0000", 2024-09-21T15:51:02.594 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:02.594 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.594 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.595 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:02.595 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134458+0000", 2024-09-21T15:51:02.595 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999722 2024-09-21T15:51:02.595 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.595 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.595 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:02.595 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134741+0000", 2024-09-21T15:51:02.595 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00028245599999999999 2024-09-21T15:51:02.595 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.595 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.595 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:02.595 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134753+0000", 2024-09-21T15:51:02.595 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2548000000000001e-05 2024-09-21T15:51:02.595 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.595 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.596 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:02.596 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134760+0000", 2024-09-21T15:51:02.596 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4189999999999996e-06 2024-09-21T15:51:02.596 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.596 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.596 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:02.596 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188426+0000", 2024-09-21T15:51:02.596 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053665894999999998 2024-09-21T15:51:02.596 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.596 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.596 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:02.596 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188502+0000", 2024-09-21T15:51:02.596 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5791000000000007e-05 2024-09-21T15:51:02.596 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.597 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.597 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:02.597 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188602+0000", 2024-09-21T15:51:02.597 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9667999999999995e-05 2024-09-21T15:51:02.597 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:02.597 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:02.597 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:02.597 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.597 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.597 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63148 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 933888~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:02.597 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.135289+0000", 2024-09-21T15:51:02.597 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 85.434796550000002, 2024-09-21T15:51:02.597 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053948488000000003, 2024-09-21T15:51:02.597 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:02.597 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:02.597 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:02.598 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:02.598 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:02.598 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63148 2024-09-21T15:51:02.598 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.598 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:02.598 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.598 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:02.598 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135289+0000", 2024-09-21T15:51:02.598 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:02.598 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.598 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.598 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:02.598 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135289+0000", 2024-09-21T15:51:02.598 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:02.599 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.599 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.599 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:02.599 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135288+0000", 2024-09-21T15:51:02.599 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:02.599 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.599 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.599 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:02.599 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135296+0000", 2024-09-21T15:51:02.599 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5399999999999998e-06 2024-09-21T15:51:02.599 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.599 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.599 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:02.599 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135303+0000", 2024-09-21T15:51:02.600 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0500000000000003e-06 2024-09-21T15:51:02.600 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.600 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.600 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:02.600 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135308+0000", 2024-09-21T15:51:02.600 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9749999999999998e-06 2024-09-21T15:51:02.600 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.600 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.600 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:02.600 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189066+0000", 2024-09-21T15:51:02.600 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053758501 2024-09-21T15:51:02.600 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.600 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.600 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:02.600 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189145+0000", 2024-09-21T15:51:02.601 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9017000000000006e-05 2024-09-21T15:51:02.601 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.601 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.601 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:02.601 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189237+0000", 2024-09-21T15:51:02.601 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1906999999999999e-05 2024-09-21T15:51:02.601 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:02.601 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:02.601 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:02.601 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.601 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.601 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63149 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 942080~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:02.601 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.135430+0000", 2024-09-21T15:51:02.601 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 85.434655221, 2024-09-21T15:51:02.601 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054005434999999997, 2024-09-21T15:51:02.602 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:02.602 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:02.602 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:02.602 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:02.602 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:02.602 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63149 2024-09-21T15:51:02.602 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.602 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:02.602 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.602 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:02.602 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135430+0000", 2024-09-21T15:51:02.602 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:02.602 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.602 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.603 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:02.603 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135430+0000", 2024-09-21T15:51:02.603 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:02.603 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.603 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.603 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:02.603 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135313+0000", 2024-09-21T15:51:02.603 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998827 2024-09-21T15:51:02.603 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.603 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.603 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:02.603 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135434+0000", 2024-09-21T15:51:02.603 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000120876 2024-09-21T15:51:02.603 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.603 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.604 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:02.604 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135443+0000", 2024-09-21T15:51:02.604 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3340000000000008e-06 2024-09-21T15:51:02.604 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.604 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.604 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:02.604 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135449+0000", 2024-09-21T15:51:02.604 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1580000000000002e-06 2024-09-21T15:51:02.604 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.604 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.604 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:02.604 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189267+0000", 2024-09-21T15:51:02.604 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053817775999999998 2024-09-21T15:51:02.604 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.605 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.605 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:02.605 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189336+0000", 2024-09-21T15:51:02.605 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9485000000000003e-05 2024-09-21T15:51:02.605 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.605 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.605 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:02.605 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189436+0000", 2024-09-21T15:51:02.605 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9277000000000005e-05 2024-09-21T15:51:02.605 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:02.605 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:02.605 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:02.605 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.605 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.605 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63150 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 950272~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:02.606 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.135618+0000", 2024-09-21T15:51:02.606 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 85.434467475999995, 2024-09-21T15:51:02.606 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054027393, 2024-09-21T15:51:02.606 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:02.606 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:02.606 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:02.606 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:02.606 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:02.606 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63150 2024-09-21T15:51:02.606 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.606 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:02.606 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.606 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:02.606 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135618+0000", 2024-09-21T15:51:02.607 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:02.607 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.607 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.607 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:02.607 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135618+0000", 2024-09-21T15:51:02.607 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:02.607 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.607 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.607 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:02.607 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135454+0000", 2024-09-21T15:51:02.607 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998364 2024-09-21T15:51:02.607 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.607 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.607 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:02.607 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135621+0000", 2024-09-21T15:51:02.608 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016714099999999999 2024-09-21T15:51:02.608 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.608 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.608 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:02.608 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135631+0000", 2024-09-21T15:51:02.608 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5340000000000005e-06 2024-09-21T15:51:02.608 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.608 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.608 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:02.608 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135638+0000", 2024-09-21T15:51:02.608 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9280000000000004e-06 2024-09-21T15:51:02.608 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.608 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.608 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:02.609 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189463+0000", 2024-09-21T15:51:02.609 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053825681 2024-09-21T15:51:02.609 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.609 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.609 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:02.609 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189546+0000", 2024-09-21T15:51:02.609 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2278000000000001e-05 2024-09-21T15:51:02.609 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.609 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.609 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:02.609 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189645+0000", 2024-09-21T15:51:02.609 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9543999999999994e-05 2024-09-21T15:51:02.609 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:02.609 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:02.609 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:02.609 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.610 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.610 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63152 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 966656~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:02.610 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.136029+0000", 2024-09-21T15:51:02.610 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 85.434056884, 2024-09-21T15:51:02.610 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054024823999999999, 2024-09-21T15:51:02.610 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:02.610 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:02.610 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:02.610 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:02.610 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:02.610 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63152 2024-09-21T15:51:02.610 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.610 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:02.610 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.610 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:02.611 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136029+0000", 2024-09-21T15:51:02.611 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:02.611 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.611 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.611 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:02.611 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136029+0000", 2024-09-21T15:51:02.611 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:02.611 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.611 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.611 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:02.611 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136028+0000", 2024-09-21T15:51:02.611 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:02.611 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.611 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.611 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:02.612 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136031+0000", 2024-09-21T15:51:02.612 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.033e-06 2024-09-21T15:51:02.612 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.612 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.612 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:02.612 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136039+0000", 2024-09-21T15:51:02.612 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7200000000000006e-06 2024-09-21T15:51:02.612 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.612 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.612 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:02.612 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136051+0000", 2024-09-21T15:51:02.612 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1987000000000001e-05 2024-09-21T15:51:02.612 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.612 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.612 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:02.613 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189876+0000", 2024-09-21T15:51:02.613 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053825264999999997 2024-09-21T15:51:02.613 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.613 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.613 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:02.613 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189955+0000", 2024-09-21T15:51:02.613 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9296999999999997e-05 2024-09-21T15:51:02.613 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.613 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.613 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:02.613 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190053+0000", 2024-09-21T15:51:02.613 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.8067999999999997e-05 2024-09-21T15:51:02.613 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:02.613 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:02.613 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:02.614 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.614 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.614 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63153 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 974848~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:02.614 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.136311+0000", 2024-09-21T15:51:02.614 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 85.433774370999998, 2024-09-21T15:51:02.614 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053933825999999997, 2024-09-21T15:51:02.614 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:02.614 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:02.614 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:02.614 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:02.614 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:02.614 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63153 2024-09-21T15:51:02.614 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.614 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:02.615 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.615 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:02.615 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136311+0000", 2024-09-21T15:51:02.615 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:02.615 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.615 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.615 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:02.615 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136311+0000", 2024-09-21T15:51:02.615 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:02.615 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.615 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.615 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:02.615 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136057+0000", 2024-09-21T15:51:02.615 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9997463 2024-09-21T15:51:02.615 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.615 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.616 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:02.616 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136316+0000", 2024-09-21T15:51:02.616 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00025872299999999998 2024-09-21T15:51:02.616 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.616 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.616 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:02.616 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136326+0000", 2024-09-21T15:51:02.616 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0552e-05 2024-09-21T15:51:02.616 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.616 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.616 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:02.616 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136334+0000", 2024-09-21T15:51:02.616 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1409999999999999e-06 2024-09-21T15:51:02.616 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.616 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.617 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:02.617 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190080+0000", 2024-09-21T15:51:02.617 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053745933000000003 2024-09-21T15:51:02.617 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.617 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.617 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:02.617 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190157+0000", 2024-09-21T15:51:02.617 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7412999999999999e-05 2024-09-21T15:51:02.617 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.617 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.617 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:02.617 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190245+0000", 2024-09-21T15:51:02.617 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7904000000000004e-05 2024-09-21T15:51:02.617 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:02.617 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:02.618 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:02.618 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.618 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.618 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63154 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 983040~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:02.618 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.136472+0000", 2024-09-21T15:51:02.618 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 85.433613019999996, 2024-09-21T15:51:02.618 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053968448000000002, 2024-09-21T15:51:02.618 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:02.618 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:02.618 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:02.618 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:02.618 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:02.618 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63154 2024-09-21T15:51:02.618 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.618 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:02.619 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.619 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:02.619 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136472+0000", 2024-09-21T15:51:02.619 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:02.619 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.619 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.619 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:02.619 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136472+0000", 2024-09-21T15:51:02.619 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:02.619 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.619 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.619 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:02.619 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136341+0000", 2024-09-21T15:51:02.620 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998684 2024-09-21T15:51:02.620 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.620 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.620 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:02.620 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136476+0000", 2024-09-21T15:51:02.620 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013528100000000001 2024-09-21T15:51:02.620 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.620 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.620 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:02.620 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136484+0000", 2024-09-21T15:51:02.620 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4760000000000001e-06 2024-09-21T15:51:02.620 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.620 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.620 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:02.620 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136490+0000", 2024-09-21T15:51:02.620 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8640000000000001e-06 2024-09-21T15:51:02.621 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.621 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.621 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:02.621 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190271+0000", 2024-09-21T15:51:02.621 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053781223000000003 2024-09-21T15:51:02.621 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.621 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.621 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:02.621 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190348+0000", 2024-09-21T15:51:02.621 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7058999999999995e-05 2024-09-21T15:51:02.621 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.621 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.621 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:02.621 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190441+0000", 2024-09-21T15:51:02.621 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2937999999999997e-05 2024-09-21T15:51:02.622 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:02.622 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:02.622 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:02.622 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.622 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.622 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63157 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1015808~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:02.622 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137112+0000", 2024-09-21T15:51:02.622 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 85.432973732999997, 2024-09-21T15:51:02.622 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053932345, 2024-09-21T15:51:02.622 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:02.622 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:02.622 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:02.622 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:02.622 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:02.622 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63157 2024-09-21T15:51:02.623 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.623 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:02.623 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.623 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:02.623 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137112+0000", 2024-09-21T15:51:02.623 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:02.623 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.623 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.623 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:02.623 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137112+0000", 2024-09-21T15:51:02.623 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:02.623 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.623 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.623 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:02.623 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137072+0000", 2024-09-21T15:51:02.624 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999604 2024-09-21T15:51:02.624 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.624 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.624 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:02.624 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137115+0000", 2024-09-21T15:51:02.624 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2379999999999997e-05 2024-09-21T15:51:02.624 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.624 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.624 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:02.624 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137121+0000", 2024-09-21T15:51:02.624 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6499999999999999e-06 2024-09-21T15:51:02.624 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.624 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.624 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:02.624 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137126+0000", 2024-09-21T15:51:02.625 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0769999999999998e-06 2024-09-21T15:51:02.625 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.625 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.625 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:02.625 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190868+0000", 2024-09-21T15:51:02.625 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053741555000000003 2024-09-21T15:51:02.625 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.625 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.625 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:02.625 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190945+0000", 2024-09-21T15:51:02.625 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7171999999999999e-05 2024-09-21T15:51:02.625 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.625 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.625 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:02.626 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191044+0000", 2024-09-21T15:51:02.626 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9031999999999996e-05 2024-09-21T15:51:02.626 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:02.626 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:02.626 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:02.626 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.626 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.626 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63158 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1007616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:02.626 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137293+0000", 2024-09-21T15:51:02.626 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 85.432792445999993, 2024-09-21T15:51:02.626 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054005642, 2024-09-21T15:51:02.626 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:02.626 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:02.627 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:02.627 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:02.627 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:02.627 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63158 2024-09-21T15:51:02.627 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.627 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:02.627 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.627 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:02.627 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137293+0000", 2024-09-21T15:51:02.627 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:02.627 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.627 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.627 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:02.627 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137293+0000", 2024-09-21T15:51:02.628 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:02.628 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.628 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.628 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:02.628 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137131+0000", 2024-09-21T15:51:02.628 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998379 2024-09-21T15:51:02.628 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.628 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.628 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:02.628 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137296+0000", 2024-09-21T15:51:02.628 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016568999999999999 2024-09-21T15:51:02.628 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.628 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.628 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:02.629 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137306+0000", 2024-09-21T15:51:02.629 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.6029999999999994e-06 2024-09-21T15:51:02.629 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.629 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.629 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:02.629 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137312+0000", 2024-09-21T15:51:02.629 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3210000000000001e-06 2024-09-21T15:51:02.629 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.629 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.629 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:02.629 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191087+0000", 2024-09-21T15:51:02.629 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053774686000000002 2024-09-21T15:51:02.629 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.629 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.629 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:02.630 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191175+0000", 2024-09-21T15:51:02.630 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8106e-05 2024-09-21T15:51:02.630 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.630 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.630 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:02.630 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191299+0000", 2024-09-21T15:51:02.630 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012343599999999999 2024-09-21T15:51:02.630 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:02.630 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:02.630 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:02.630 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.630 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.630 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63159 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1024000~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:02.630 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137568+0000", 2024-09-21T15:51:02.630 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 85.432517173999997, 2024-09-21T15:51:02.631 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053962999999999997, 2024-09-21T15:51:02.631 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:02.631 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:02.631 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:02.631 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:02.631 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:02.631 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63159 2024-09-21T15:51:02.631 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.631 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:02.631 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.631 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:02.631 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137568+0000", 2024-09-21T15:51:02.631 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:02.631 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.632 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.632 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:02.632 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137568+0000", 2024-09-21T15:51:02.632 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:02.632 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.632 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.632 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:02.632 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137317+0000", 2024-09-21T15:51:02.632 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9997492 2024-09-21T15:51:02.632 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.632 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.632 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:02.632 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137572+0000", 2024-09-21T15:51:02.632 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00025491499999999998 2024-09-21T15:51:02.633 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.633 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.633 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:02.633 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137582+0000", 2024-09-21T15:51:02.633 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4490000000000005e-06 2024-09-21T15:51:02.633 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.633 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.633 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:02.633 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137589+0000", 2024-09-21T15:51:02.633 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5580000000000003e-06 2024-09-21T15:51:02.633 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.633 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.633 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:02.633 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191344+0000", 2024-09-21T15:51:02.634 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053754272999999998 2024-09-21T15:51:02.634 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.634 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.634 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:02.634 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191429+0000", 2024-09-21T15:51:02.634 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.509e-05 2024-09-21T15:51:02.634 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.634 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.634 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:02.634 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191531+0000", 2024-09-21T15:51:02.634 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000102519 2024-09-21T15:51:02.634 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:02.634 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:02.634 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:02.635 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.635 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.635 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63160 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1032192~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:02.635 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137771+0000", 2024-09-21T15:51:02.635 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 85.432314640000001, 2024-09-21T15:51:02.635 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053957818999999997, 2024-09-21T15:51:02.635 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:02.635 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:02.635 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:02.635 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:02.635 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:02.635 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63160 2024-09-21T15:51:02.635 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.635 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:02.635 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.635 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:02.636 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137771+0000", 2024-09-21T15:51:02.636 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:02.636 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.636 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.636 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:02.636 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137771+0000", 2024-09-21T15:51:02.636 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:02.636 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.636 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.636 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:02.636 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137596+0000", 2024-09-21T15:51:02.636 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998255 2024-09-21T15:51:02.636 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.636 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.636 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:02.636 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137780+0000", 2024-09-21T15:51:02.637 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00018363900000000001 2024-09-21T15:51:02.637 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.637 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.637 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:02.637 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137791+0000", 2024-09-21T15:51:02.637 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.096e-05 2024-09-21T15:51:02.637 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.637 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.637 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:02.637 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137798+0000", 2024-09-21T15:51:02.637 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1169999999999998e-06 2024-09-21T15:51:02.637 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.637 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.637 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:02.637 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191560+0000", 2024-09-21T15:51:02.637 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053762163000000002 2024-09-21T15:51:02.637 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.638 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.638 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:02.638 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191634+0000", 2024-09-21T15:51:02.638 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4381999999999994e-05 2024-09-21T15:51:02.638 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.638 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.638 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:02.638 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191729+0000", 2024-09-21T15:51:02.638 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4098000000000006e-05 2024-09-21T15:51:02.638 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:02.638 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:02.638 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:02.638 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.638 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.639 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63161 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1040384~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:02.639 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.138018+0000", 2024-09-21T15:51:02.639 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 85.432067885999999, 2024-09-21T15:51:02.639 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053905657000000003, 2024-09-21T15:51:02.639 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:02.639 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:02.639 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:02.639 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:02.639 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:02.639 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63161 2024-09-21T15:51:02.639 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.639 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:02.639 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.639 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:02.639 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138018+0000", 2024-09-21T15:51:02.640 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:02.640 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.640 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.640 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:02.640 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138018+0000", 2024-09-21T15:51:02.640 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:02.640 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.640 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.640 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:02.640 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137804+0000", 2024-09-21T15:51:02.640 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9997859 2024-09-21T15:51:02.640 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.640 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.640 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:02.640 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138021+0000", 2024-09-21T15:51:02.640 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00021769899999999999 2024-09-21T15:51:02.641 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.641 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.641 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:02.641 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138043+0000", 2024-09-21T15:51:02.641 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1627000000000001e-05 2024-09-21T15:51:02.641 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.641 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.641 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:02.641 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138054+0000", 2024-09-21T15:51:02.641 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1321999999999999e-05 2024-09-21T15:51:02.641 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.641 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.641 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:02.641 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191755+0000", 2024-09-21T15:51:02.641 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053700354999999998 2024-09-21T15:51:02.641 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.642 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.642 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:02.642 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191830+0000", 2024-09-21T15:51:02.642 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5320000000000004e-05 2024-09-21T15:51:02.642 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.642 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.642 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:02.642 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191923+0000", 2024-09-21T15:51:02.642 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3282999999999994e-05 2024-09-21T15:51:02.642 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:02.642 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:02.642 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:02.642 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.642 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.643 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63176 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 108544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:02.643 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.140861+0000", 2024-09-21T15:51:02.643 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 85.429224567000006, 2024-09-21T15:51:02.643 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053935394999999997, 2024-09-21T15:51:02.643 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:02.643 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:02.643 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:02.643 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:02.643 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:02.643 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63176 2024-09-21T15:51:02.643 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.643 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:02.643 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.643 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:02.643 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140861+0000", 2024-09-21T15:51:02.644 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:02.644 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.644 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.644 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:02.644 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140861+0000", 2024-09-21T15:51:02.644 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:02.644 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.644 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.644 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:02.644 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140860+0000", 2024-09-21T15:51:02.644 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:02.644 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.644 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.644 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:02.644 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140863+0000", 2024-09-21T15:51:02.644 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8710000000000001e-06 2024-09-21T15:51:02.644 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.645 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.645 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:02.645 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140871+0000", 2024-09-21T15:51:02.645 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8089999999999993e-06 2024-09-21T15:51:02.645 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.645 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.645 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:02.645 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140877+0000", 2024-09-21T15:51:02.645 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5969999999999999e-06 2024-09-21T15:51:02.645 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.645 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.645 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:02.645 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.194608+0000", 2024-09-21T15:51:02.645 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053731749000000002 2024-09-21T15:51:02.645 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.646 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.646 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:02.646 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.194697+0000", 2024-09-21T15:51:02.646 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8504000000000005e-05 2024-09-21T15:51:02.646 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.646 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.646 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:02.646 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.194796+0000", 2024-09-21T15:51:02.646 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9414999999999996e-05 2024-09-21T15:51:02.646 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:02.646 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:02.646 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:02.646 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.646 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.646 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63182 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 157696~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:02.647 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:51:02.647 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 85.427952167000001, 2024-09-21T15:51:02.647 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053958825000000002, 2024-09-21T15:51:02.647 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:02.647 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:02.647 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:02.647 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:02.647 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:02.647 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63182 2024-09-21T15:51:02.647 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.647 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:02.647 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.647 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:02.647 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:51:02.647 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:02.648 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.648 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.648 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:02.648 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:51:02.648 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:02.648 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.648 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.648 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:02.648 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:51:02.648 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:02.648 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.648 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.648 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:02.648 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142136+0000", 2024-09-21T15:51:02.648 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8169999999999999e-06 2024-09-21T15:51:02.649 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.649 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.649 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:02.649 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:51:02.649 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3752e-05 2024-09-21T15:51:02.649 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.649 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.649 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:02.649 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142155+0000", 2024-09-21T15:51:02.649 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1859999999999997e-06 2024-09-21T15:51:02.649 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.649 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.649 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:02.649 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.195914+0000", 2024-09-21T15:51:02.649 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053758528999999999 2024-09-21T15:51:02.649 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.649 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.650 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:02.650 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.195995+0000", 2024-09-21T15:51:02.650 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0983999999999996e-05 2024-09-21T15:51:02.650 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.650 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.650 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:02.650 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.196092+0000", 2024-09-21T15:51:02.650 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7069000000000002e-05 2024-09-21T15:51:02.650 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:02.650 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:02.650 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:02.650 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.650 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.650 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63193 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 239616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:02.650 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:51:02.651 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 85.425864582000003, 2024-09-21T15:51:02.651 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053956577999999998, 2024-09-21T15:51:02.651 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:02.651 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:02.651 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:02.651 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:02.651 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:02.651 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63193 2024-09-21T15:51:02.651 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.651 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:02.651 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.651 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:02.651 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:51:02.651 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:02.652 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.652 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.652 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:02.652 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:51:02.652 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:02.652 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.652 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.652 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:02.652 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:51:02.652 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:02.652 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.652 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.652 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:02.652 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144223+0000", 2024-09-21T15:51:02.652 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0990000000000001e-06 2024-09-21T15:51:02.652 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.653 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.653 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:02.653 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144230+0000", 2024-09-21T15:51:02.653 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9070000000000001e-06 2024-09-21T15:51:02.653 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.653 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.653 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:02.653 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144234+0000", 2024-09-21T15:51:02.653 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9010000000000002e-06 2024-09-21T15:51:02.653 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.653 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.653 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:02.653 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.198004+0000", 2024-09-21T15:51:02.653 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053769539999999998 2024-09-21T15:51:02.653 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.654 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.654 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:02.654 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.198083+0000", 2024-09-21T15:51:02.654 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9531999999999997e-05 2024-09-21T15:51:02.654 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:02.654 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:02.654 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:02.654 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.198177+0000", 2024-09-21T15:51:02.654 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3906000000000005e-05 2024-09-21T15:51:02.654 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:02.654 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:02.654 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:02.654 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:02.654 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:02.654 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:51:02.656 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T15:51:02.656 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:51:02.655+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:51:02.757 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:51:02.755+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:51:02.758 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:51:02.758 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:51:02.758 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:51:02.758 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:51:02.768 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T15:51:02.792 INFO:teuthology.orchestra.run.smithi089.stderr:reweighted osd.4 to 1 (10000) 2024-09-21T15:51:02.801 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T15:51:02.809 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-21T15:51:02.893 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:51:02.892+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:51:02.920 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:51:02.920 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:51:02.920 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:51:02.921 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:51:02.921 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:51:02.930 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T15:51:02.958 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:51:02.957+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:51:03.034 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:51:03.034 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:51:03.034 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:51:03.034 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:51:03.034 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.034 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4209.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:51:03.034 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:51:03.035 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 131.840838733, 2024-09-21T15:51:03.035 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.19452909900000001, 2024-09-21T15:51:03.035 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:03.035 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:03.035 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:03.035 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4209", 2024-09-21T15:51:03.035 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/4045143050", 2024-09-21T15:51:03.035 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1 2024-09-21T15:51:03.035 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.035 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:03.035 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.035 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:03.035 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:51:03.035 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:03.036 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.036 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.036 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:03.036 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:51:03.036 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:03.036 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.036 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.036 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:03.036 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191161+0000", 2024-09-21T15:51:03.036 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-21T15:51:03.036 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.036 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.036 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:03.036 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191169+0000", 2024-09-21T15:51:03.037 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0090000000000007e-06 2024-09-21T15:51:03.037 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.037 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.037 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:03.037 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191180+0000", 2024-09-21T15:51:03.037 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0728e-05 2024-09-21T15:51:03.037 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.037 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.037 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:03.037 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191187+0000", 2024-09-21T15:51:03.037 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9009999999999994e-06 2024-09-21T15:51:03.037 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.037 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.037 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:03.037 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191238+0000", 2024-09-21T15:51:03.037 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0287000000000002e-05 2024-09-21T15:51:03.038 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.038 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.038 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-21T15:51:03.038 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191245+0000", 2024-09-21T15:51:03.038 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0709999999999997e-06 2024-09-21T15:51:03.038 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.038 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.038 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:03.038 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380309+0000", 2024-09-21T15:51:03.038 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.189064277 2024-09-21T15:51:03.038 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.038 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.038 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:03.038 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380454+0000", 2024-09-21T15:51:03.039 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014490799999999999 2024-09-21T15:51:03.039 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.039 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.039 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0", 2024-09-21T15:51:03.039 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380610+0000", 2024-09-21T15:51:03.039 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015650199999999999 2024-09-21T15:51:03.039 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.039 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.039 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:51:03.039 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.381160+0000", 2024-09-21T15:51:03.039 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00054903799999999996 2024-09-21T15:51:03.039 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.039 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.039 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:03.040 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385591+0000", 2024-09-21T15:51:03.040 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.004431272 2024-09-21T15:51:03.040 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.040 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.040 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:51:03.040 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385651+0000", 2024-09-21T15:51:03.040 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0455999999999999e-05 2024-09-21T15:51:03.040 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.040 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.040 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:03.040 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385691+0000", 2024-09-21T15:51:03.040 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0234999999999997e-05 2024-09-21T15:51:03.040 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:03.040 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:03.040 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:03.041 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.041 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.041 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64034 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1083392~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:03.041 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167717+0000", 2024-09-21T15:51:03.041 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 85.864284402999999, 2024-09-21T15:51:03.041 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048233316999999998, 2024-09-21T15:51:03.041 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:03.041 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:03.041 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:03.041 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:03.041 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:03.041 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64034 2024-09-21T15:51:03.041 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.041 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:03.041 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.042 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:03.042 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167717+0000", 2024-09-21T15:51:03.042 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:03.042 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.042 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.042 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:03.042 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167717+0000", 2024-09-21T15:51:03.042 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:03.042 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.042 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.042 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:03.042 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167716+0000", 2024-09-21T15:51:03.042 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:03.042 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.042 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.043 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:03.043 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167718+0000", 2024-09-21T15:51:03.043 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.165e-06 2024-09-21T15:51:03.043 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.043 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.043 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:03.043 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167725+0000", 2024-09-21T15:51:03.043 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3380000000000001e-06 2024-09-21T15:51:03.043 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.043 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.043 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:03.043 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167730+0000", 2024-09-21T15:51:03.043 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8010000000000003e-06 2024-09-21T15:51:03.043 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.043 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.043 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:03.044 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215807+0000", 2024-09-21T15:51:03.044 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048077823999999998 2024-09-21T15:51:03.044 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.044 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.044 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:03.044 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215884+0000", 2024-09-21T15:51:03.044 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6283000000000001e-05 2024-09-21T15:51:03.044 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.044 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.044 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:03.044 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215950+0000", 2024-09-21T15:51:03.044 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6290000000000004e-05 2024-09-21T15:51:03.044 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:03.045 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:03.045 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:03.045 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.045 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.045 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64035 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1091584~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:03.045 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:51:03.045 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 85.864267190000007, 2024-09-21T15:51:03.045 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048412117999999997, 2024-09-21T15:51:03.045 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:03.045 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:03.045 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:03.045 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:03.045 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:03.045 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64035 2024-09-21T15:51:03.046 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.046 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:03.046 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.046 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:03.046 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:51:03.046 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:03.046 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.046 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.046 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:03.046 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:51:03.046 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:03.046 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.046 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.046 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:03.046 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:51:03.046 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:03.047 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.047 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.047 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:03.047 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167746+0000", 2024-09-21T15:51:03.047 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2031000000000001e-05 2024-09-21T15:51:03.047 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.047 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.047 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:03.047 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167753+0000", 2024-09-21T15:51:03.047 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0620000000000003e-06 2024-09-21T15:51:03.047 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.047 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.048 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:03.048 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167759+0000", 2024-09-21T15:51:03.048 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.596e-06 2024-09-21T15:51:03.048 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.048 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.048 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:03.048 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215977+0000", 2024-09-21T15:51:03.048 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048217318000000002 2024-09-21T15:51:03.048 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.048 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.048 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:03.048 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216076+0000", 2024-09-21T15:51:03.048 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9846000000000006e-05 2024-09-21T15:51:03.048 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.048 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.049 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:03.049 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216146+0000", 2024-09-21T15:51:03.049 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9578000000000004e-05 2024-09-21T15:51:03.049 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:03.049 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:03.049 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:03.049 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.049 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.049 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64036 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1099776~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:03.049 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167853+0000", 2024-09-21T15:51:03.049 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 85.864148576999995, 2024-09-21T15:51:03.049 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048450675999999998, 2024-09-21T15:51:03.049 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:03.049 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:03.049 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:03.049 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:03.050 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:03.050 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64036 2024-09-21T15:51:03.050 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.050 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:03.050 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.050 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:03.050 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167853+0000", 2024-09-21T15:51:03.050 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:03.050 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.050 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.050 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:03.050 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167853+0000", 2024-09-21T15:51:03.050 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:03.050 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.050 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.051 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:03.051 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167852+0000", 2024-09-21T15:51:03.051 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:03.051 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.051 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.051 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:03.051 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167855+0000", 2024-09-21T15:51:03.051 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5640000000000001e-06 2024-09-21T15:51:03.051 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.051 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.051 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:03.051 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167864+0000", 2024-09-21T15:51:03.051 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8140000000000007e-06 2024-09-21T15:51:03.051 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.051 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.051 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:03.052 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167871+0000", 2024-09-21T15:51:03.052 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5729999999999996e-06 2024-09-21T15:51:03.052 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.052 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.052 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:03.052 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216174+0000", 2024-09-21T15:51:03.052 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048303437999999997 2024-09-21T15:51:03.052 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.052 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.052 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:03.052 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216250+0000", 2024-09-21T15:51:03.052 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5661999999999995e-05 2024-09-21T15:51:03.052 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.052 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.052 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:03.053 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216303+0000", 2024-09-21T15:51:03.053 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3273000000000003e-05 2024-09-21T15:51:03.053 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:03.053 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:03.053 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:03.053 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.053 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.053 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64037 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1771520~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:03.053 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:51:03.053 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 85.864125033999997, 2024-09-21T15:51:03.053 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048588771000000003, 2024-09-21T15:51:03.053 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:03.053 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:03.053 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:03.053 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:03.053 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:03.054 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64037 2024-09-21T15:51:03.054 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.054 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:03.054 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.054 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:03.054 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:51:03.054 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:03.054 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.054 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.054 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:03.054 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:51:03.054 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:03.054 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.055 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.055 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:03.055 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:51:03.055 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:03.055 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.055 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.055 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:03.055 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167878+0000", 2024-09-21T15:51:03.055 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3e-06 2024-09-21T15:51:03.055 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.055 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.055 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:03.055 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167884+0000", 2024-09-21T15:51:03.055 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3269999999999997e-06 2024-09-21T15:51:03.055 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.055 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.056 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:03.056 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167889+0000", 2024-09-21T15:51:03.056 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9269999999999996e-06 2024-09-21T15:51:03.056 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.056 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.056 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:03.056 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216331+0000", 2024-09-21T15:51:03.056 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048442102000000001 2024-09-21T15:51:03.056 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.056 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.056 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:03.056 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216393+0000", 2024-09-21T15:51:03.056 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1132999999999999e-05 2024-09-21T15:51:03.056 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.056 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.056 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:03.057 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216465+0000", 2024-09-21T15:51:03.057 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2339e-05 2024-09-21T15:51:03.057 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:03.057 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:03.057 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:03.057 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.057 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.057 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64038 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1779712~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:03.057 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:51:03.057 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 85.864107203000003, 2024-09-21T15:51:03.057 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048749795999999998, 2024-09-21T15:51:03.057 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:03.057 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:03.057 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:03.057 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:03.057 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:03.058 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64038 2024-09-21T15:51:03.058 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.058 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:03.058 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.058 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:03.058 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:51:03.058 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:03.058 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.058 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.058 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:03.058 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:51:03.058 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:03.058 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.058 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.058 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:03.059 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:51:03.059 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:03.059 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.059 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.059 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:03.059 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167896+0000", 2024-09-21T15:51:03.059 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2419999999999999e-06 2024-09-21T15:51:03.059 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.059 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.059 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:03.059 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167902+0000", 2024-09-21T15:51:03.059 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0859999999999998e-06 2024-09-21T15:51:03.059 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.059 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.059 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:03.060 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167907+0000", 2024-09-21T15:51:03.060 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9679999999999997e-06 2024-09-21T15:51:03.060 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.060 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.060 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:03.060 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216506+0000", 2024-09-21T15:51:03.060 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048599374000000001 2024-09-21T15:51:03.060 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.060 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.060 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:03.060 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216579+0000", 2024-09-21T15:51:03.060 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2801999999999999e-05 2024-09-21T15:51:03.060 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.060 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.060 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:03.061 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216644+0000", 2024-09-21T15:51:03.061 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4715000000000006e-05 2024-09-21T15:51:03.061 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:03.061 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:03.061 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:03.061 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.061 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.061 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64039 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1107968~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:03.061 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:51:03.061 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 85.864089848000006, 2024-09-21T15:51:03.061 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048911370000000003, 2024-09-21T15:51:03.061 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:03.061 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:03.061 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:03.061 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:03.062 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:03.062 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64039 2024-09-21T15:51:03.062 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.062 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:03.062 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.062 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:03.062 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:51:03.062 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:03.062 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.062 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.062 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:03.062 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:51:03.062 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:03.062 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.063 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.063 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:03.063 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:51:03.063 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:03.063 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.063 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.063 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:03.063 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167913+0000", 2024-09-21T15:51:03.063 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0839999999999999e-06 2024-09-21T15:51:03.063 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.063 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.063 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:03.063 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167921+0000", 2024-09-21T15:51:03.063 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8310000000000001e-06 2024-09-21T15:51:03.064 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.064 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.064 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:03.064 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167926+0000", 2024-09-21T15:51:03.064 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0270000000000003e-06 2024-09-21T15:51:03.064 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.064 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.064 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:03.064 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216680+0000", 2024-09-21T15:51:03.064 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048753890000000001 2024-09-21T15:51:03.064 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.064 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.064 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:03.065 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216757+0000", 2024-09-21T15:51:03.065 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6848999999999996e-05 2024-09-21T15:51:03.065 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.065 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.065 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:03.065 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216823+0000", 2024-09-21T15:51:03.065 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6068000000000005e-05 2024-09-21T15:51:03.065 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:03.065 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:03.065 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:03.066 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.066 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.066 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64040 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1787904~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:03.066 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167931+0000", 2024-09-21T15:51:03.066 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 85.864070616000006, 2024-09-21T15:51:03.066 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049010561000000001, 2024-09-21T15:51:03.066 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:03.066 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:03.066 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:03.066 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:03.066 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:03.066 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64040 2024-09-21T15:51:03.067 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.067 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:03.067 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.067 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:03.067 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167931+0000", 2024-09-21T15:51:03.067 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:03.067 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.067 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.067 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:03.067 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167931+0000", 2024-09-21T15:51:03.067 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:03.067 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.067 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.067 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:03.068 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167930+0000", 2024-09-21T15:51:03.068 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:03.068 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.068 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.068 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:03.068 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167932+0000", 2024-09-21T15:51:03.068 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.198e-06 2024-09-21T15:51:03.068 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.068 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.068 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:03.068 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167938+0000", 2024-09-21T15:51:03.068 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.105e-06 2024-09-21T15:51:03.068 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.068 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.069 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:03.069 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167943+0000", 2024-09-21T15:51:03.069 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9429999999999999e-06 2024-09-21T15:51:03.069 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.069 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.069 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:03.070 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216848+0000", 2024-09-21T15:51:03.070 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048905200000000003 2024-09-21T15:51:03.070 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.070 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.070 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:03.070 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216903+0000", 2024-09-21T15:51:03.070 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4793999999999997e-05 2024-09-21T15:51:03.070 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.070 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.070 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:03.070 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216941+0000", 2024-09-21T15:51:03.070 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7790999999999998e-05 2024-09-21T15:51:03.070 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:03.071 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:03.071 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:03.071 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.071 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.071 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64041 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1796096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:03.071 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:51:03.071 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 85.864053644999998, 2024-09-21T15:51:03.071 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049181457999999997, 2024-09-21T15:51:03.071 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:03.071 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:03.071 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:03.071 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:03.071 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:03.071 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64041 2024-09-21T15:51:03.072 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.072 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:03.072 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.072 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:03.072 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:51:03.072 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:03.072 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.072 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.072 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:03.072 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:51:03.072 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:03.072 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.072 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.072 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:03.072 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:51:03.073 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:03.073 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.073 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.073 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:03.073 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167949+0000", 2024-09-21T15:51:03.073 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0200000000000001e-06 2024-09-21T15:51:03.073 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.073 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.073 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:03.073 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167955+0000", 2024-09-21T15:51:03.073 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5990000000000001e-06 2024-09-21T15:51:03.073 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.073 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.073 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:03.074 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167959+0000", 2024-09-21T15:51:03.074 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5589999999999999e-06 2024-09-21T15:51:03.074 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.074 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.074 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:03.074 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216972+0000", 2024-09-21T15:51:03.074 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049012574000000003 2024-09-21T15:51:03.074 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.074 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.074 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:03.074 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217057+0000", 2024-09-21T15:51:03.074 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4820000000000004e-05 2024-09-21T15:51:03.074 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.075 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.075 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:03.075 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217129+0000", 2024-09-21T15:51:03.075 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2186000000000003e-05 2024-09-21T15:51:03.075 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:03.075 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:03.075 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:03.075 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.075 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.075 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64042 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1804288~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:03.075 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167964+0000", 2024-09-21T15:51:03.075 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 85.864037354000004, 2024-09-21T15:51:03.075 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049333372, 2024-09-21T15:51:03.075 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:03.076 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:03.076 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:03.076 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:03.076 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:03.076 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64042 2024-09-21T15:51:03.076 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.076 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:03.076 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.076 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:03.076 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167964+0000", 2024-09-21T15:51:03.076 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:03.076 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.076 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.076 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:03.076 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167964+0000", 2024-09-21T15:51:03.076 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:03.076 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.077 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.077 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:03.077 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167963+0000", 2024-09-21T15:51:03.077 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:03.077 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.077 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.077 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:03.077 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167966+0000", 2024-09-21T15:51:03.077 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1119999999999999e-06 2024-09-21T15:51:03.077 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.077 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.077 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:03.077 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167975+0000", 2024-09-21T15:51:03.077 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1130000000000007e-06 2024-09-21T15:51:03.077 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.077 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.078 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:03.078 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167980+0000", 2024-09-21T15:51:03.078 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9599999999999999e-06 2024-09-21T15:51:03.078 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.078 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.078 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:03.078 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217158+0000", 2024-09-21T15:51:03.078 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049178569999999998 2024-09-21T15:51:03.078 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.078 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.078 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:03.078 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217232+0000", 2024-09-21T15:51:03.078 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3589000000000003e-05 2024-09-21T15:51:03.078 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.078 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.079 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:03.079 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217297+0000", 2024-09-21T15:51:03.079 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5339999999999994e-05 2024-09-21T15:51:03.079 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:03.079 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:03.079 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:03.079 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.079 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.079 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64043 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1812480~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:03.079 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.168085+0000", 2024-09-21T15:51:03.079 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 85.863916286999995, 2024-09-21T15:51:03.079 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049365157999999999, 2024-09-21T15:51:03.079 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:03.079 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:03.079 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:03.079 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:03.080 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:03.080 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64043 2024-09-21T15:51:03.080 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.080 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:03.080 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.080 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:03.080 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168085+0000", 2024-09-21T15:51:03.080 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:03.080 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.080 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.080 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:03.080 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168085+0000", 2024-09-21T15:51:03.080 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:03.080 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.080 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.080 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:03.081 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168084+0000", 2024-09-21T15:51:03.081 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:03.081 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.081 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.081 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:03.081 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168088+0000", 2024-09-21T15:51:03.081 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5420000000000001e-06 2024-09-21T15:51:03.081 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.081 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.081 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:03.081 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168097+0000", 2024-09-21T15:51:03.081 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5410000000000006e-06 2024-09-21T15:51:03.081 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.081 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.081 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:03.081 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168104+0000", 2024-09-21T15:51:03.081 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6719999999999998e-06 2024-09-21T15:51:03.082 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.082 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.082 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:03.082 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217323+0000", 2024-09-21T15:51:03.082 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049218857999999997 2024-09-21T15:51:03.082 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.082 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.082 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:03.082 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217397+0000", 2024-09-21T15:51:03.082 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3932999999999998e-05 2024-09-21T15:51:03.082 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.082 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.082 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:03.082 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217450+0000", 2024-09-21T15:51:03.082 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3168000000000003e-05 2024-09-21T15:51:03.083 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:03.083 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:03.083 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:03.083 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.083 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.083 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155255 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 323584~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:03.083 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.440996+0000", 2024-09-21T15:51:03.083 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 40.591004759999997, 2024-09-21T15:51:03.083 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048233848000000003, 2024-09-21T15:51:03.083 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:03.083 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:03.083 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:03.083 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:03.083 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:03.084 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155255 2024-09-21T15:51:03.084 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.084 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:03.084 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.084 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:03.084 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.440996+0000", 2024-09-21T15:51:03.084 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:03.084 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.084 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.084 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:03.084 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.440996+0000", 2024-09-21T15:51:03.084 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:03.084 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.084 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.084 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:03.085 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.440996+0000", 2024-09-21T15:51:03.085 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:03.085 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.085 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.085 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:03.085 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441004+0000", 2024-09-21T15:51:03.085 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8460000000000008e-06 2024-09-21T15:51:03.085 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.085 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.085 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:03.085 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441018+0000", 2024-09-21T15:51:03.085 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.42e-05 2024-09-21T15:51:03.085 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.085 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.085 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:03.086 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441023+0000", 2024-09-21T15:51:03.086 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0019999999999997e-06 2024-09-21T15:51:03.086 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.086 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.086 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:03.086 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489060+0000", 2024-09-21T15:51:03.086 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048036660000000002 2024-09-21T15:51:03.086 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.086 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.086 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:03.086 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489107+0000", 2024-09-21T15:51:03.086 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7153000000000001e-05 2024-09-21T15:51:03.086 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.086 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.086 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:03.087 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489230+0000", 2024-09-21T15:51:03.087 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012328600000000001 2024-09-21T15:51:03.087 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:03.087 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:03.087 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:03.087 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.087 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.087 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155256 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 356352~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:03.087 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441027+0000", 2024-09-21T15:51:03.087 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 40.590973673000001, 2024-09-21T15:51:03.087 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048677606999999998, 2024-09-21T15:51:03.087 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:03.087 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:03.088 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:03.088 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:03.088 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:03.088 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155256 2024-09-21T15:51:03.088 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.088 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:03.088 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.088 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:03.088 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441027+0000", 2024-09-21T15:51:03.088 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:03.088 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.088 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.088 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:03.088 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441027+0000", 2024-09-21T15:51:03.088 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:03.088 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.089 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.089 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:03.089 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441027+0000", 2024-09-21T15:51:03.089 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:03.089 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.089 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.089 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:03.089 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441029+0000", 2024-09-21T15:51:03.089 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.39e-06 2024-09-21T15:51:03.089 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.089 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.089 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:03.089 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441036+0000", 2024-09-21T15:51:03.089 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1129999999999997e-06 2024-09-21T15:51:03.089 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.090 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.090 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:03.090 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441040+0000", 2024-09-21T15:51:03.090 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8459999999999999e-06 2024-09-21T15:51:03.090 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.090 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.090 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:03.090 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489244+0000", 2024-09-21T15:51:03.090 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048203751000000003 2024-09-21T15:51:03.090 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.090 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.090 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:03.090 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489288+0000", 2024-09-21T15:51:03.090 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4153999999999998e-05 2024-09-21T15:51:03.091 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.091 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.091 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:03.091 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489705+0000", 2024-09-21T15:51:03.091 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000416773 2024-09-21T15:51:03.091 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:03.091 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:03.091 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:03.091 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.091 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.091 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155257 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 389120~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:03.091 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441045+0000", 2024-09-21T15:51:03.091 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 40.590956581999997, 2024-09-21T15:51:03.091 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049000325999999997, 2024-09-21T15:51:03.091 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:03.091 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:03.092 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:03.092 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:03.092 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:03.092 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155257 2024-09-21T15:51:03.092 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.092 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:03.092 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.092 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:03.092 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441045+0000", 2024-09-21T15:51:03.092 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:03.092 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.092 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.092 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:03.092 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441045+0000", 2024-09-21T15:51:03.092 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:03.093 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.093 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.093 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:03.093 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441044+0000", 2024-09-21T15:51:03.093 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:03.093 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.093 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.093 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:03.093 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441046+0000", 2024-09-21T15:51:03.093 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1909999999999999e-06 2024-09-21T15:51:03.093 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.093 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.093 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:03.093 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441054+0000", 2024-09-21T15:51:03.093 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3900000000000004e-06 2024-09-21T15:51:03.093 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.094 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.094 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:03.094 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441058+0000", 2024-09-21T15:51:03.094 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4619999999999998e-06 2024-09-21T15:51:03.094 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.094 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.094 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:03.094 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489741+0000", 2024-09-21T15:51:03.094 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048682660000000003 2024-09-21T15:51:03.094 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.094 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.094 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:03.094 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489813+0000", 2024-09-21T15:51:03.094 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2051999999999994e-05 2024-09-21T15:51:03.095 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.095 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.095 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:03.095 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490045+0000", 2024-09-21T15:51:03.095 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00023191299999999999 2024-09-21T15:51:03.095 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:03.095 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:03.095 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:03.095 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.095 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.095 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155258 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 421888~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:03.095 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441062+0000", 2024-09-21T15:51:03.095 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 40.590939040000002, 2024-09-21T15:51:03.096 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049339651999999998, 2024-09-21T15:51:03.096 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:03.096 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:03.096 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:03.096 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:03.096 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:03.096 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155258 2024-09-21T15:51:03.096 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.096 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:03.096 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.096 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:03.096 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441062+0000", 2024-09-21T15:51:03.096 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:03.096 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.096 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.097 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:03.097 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441062+0000", 2024-09-21T15:51:03.097 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:03.097 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.097 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.097 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:03.097 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441062+0000", 2024-09-21T15:51:03.097 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:03.097 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.097 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.097 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:03.097 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441064+0000", 2024-09-21T15:51:03.097 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.967e-06 2024-09-21T15:51:03.097 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.097 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.097 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:03.098 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441069+0000", 2024-09-21T15:51:03.098 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1529999999999996e-06 2024-09-21T15:51:03.098 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.098 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.098 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:03.098 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441073+0000", 2024-09-21T15:51:03.098 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3850000000000004e-06 2024-09-21T15:51:03.098 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.098 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.098 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:03.098 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490082+0000", 2024-09-21T15:51:03.098 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.04900885 2024-09-21T15:51:03.098 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.098 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.098 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:03.098 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490166+0000", 2024-09-21T15:51:03.099 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3640999999999994e-05 2024-09-21T15:51:03.099 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.099 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.099 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:03.099 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490402+0000", 2024-09-21T15:51:03.099 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000235978 2024-09-21T15:51:03.099 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:03.099 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:03.099 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:03.099 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.099 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.099 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155259 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 454656~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:03.099 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441086+0000", 2024-09-21T15:51:03.099 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 40.590915234999997, 2024-09-21T15:51:03.099 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049624397000000001, 2024-09-21T15:51:03.099 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:03.099 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:03.100 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:03.100 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:03.100 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:03.100 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155259 2024-09-21T15:51:03.100 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.100 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:03.100 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.100 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:03.100 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441086+0000", 2024-09-21T15:51:03.100 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:03.100 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.100 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.100 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:03.100 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441086+0000", 2024-09-21T15:51:03.100 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:03.100 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.101 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.101 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:03.101 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441086+0000", 2024-09-21T15:51:03.101 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:03.101 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.101 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.101 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:03.101 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441088+0000", 2024-09-21T15:51:03.101 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2630000000000002e-06 2024-09-21T15:51:03.101 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.101 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.101 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:03.101 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441093+0000", 2024-09-21T15:51:03.101 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5010000000000004e-06 2024-09-21T15:51:03.101 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.102 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.102 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:03.102 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441098+0000", 2024-09-21T15:51:03.102 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.707e-06 2024-09-21T15:51:03.102 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.102 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.102 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:03.102 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490447+0000", 2024-09-21T15:51:03.102 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049349125000000001 2024-09-21T15:51:03.102 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.102 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.102 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:03.102 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490504+0000", 2024-09-21T15:51:03.102 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6521000000000003e-05 2024-09-21T15:51:03.103 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.103 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.103 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:03.103 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490710+0000", 2024-09-21T15:51:03.103 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00020663999999999999 2024-09-21T15:51:03.103 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:03.103 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:03.103 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:03.103 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.103 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.103 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155260 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 487424~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:03.103 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441160+0000", 2024-09-21T15:51:03.103 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 40.590841273000002, 2024-09-21T15:51:03.103 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049876810000000001, 2024-09-21T15:51:03.103 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:03.104 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:03.104 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:03.104 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:03.104 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:03.104 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155260 2024-09-21T15:51:03.104 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.104 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:03.104 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.104 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:03.104 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441160+0000", 2024-09-21T15:51:03.104 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:03.104 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.104 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.104 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:03.105 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441160+0000", 2024-09-21T15:51:03.105 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:03.105 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.105 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.105 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:03.105 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441159+0000", 2024-09-21T15:51:03.105 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:03.105 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.105 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.105 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:03.105 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441164+0000", 2024-09-21T15:51:03.105 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4059999999999998e-06 2024-09-21T15:51:03.105 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.105 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.105 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:03.105 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441175+0000", 2024-09-21T15:51:03.106 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1095e-05 2024-09-21T15:51:03.106 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.106 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.106 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:03.106 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441182+0000", 2024-09-21T15:51:03.106 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2139999999999999e-06 2024-09-21T15:51:03.106 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.106 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.106 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:03.106 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490751+0000", 2024-09-21T15:51:03.106 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049569230999999998 2024-09-21T15:51:03.106 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.106 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.106 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:03.107 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490837+0000", 2024-09-21T15:51:03.107 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6017e-05 2024-09-21T15:51:03.107 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.107 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.107 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:03.107 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491037+0000", 2024-09-21T15:51:03.107 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00019952300000000001 2024-09-21T15:51:03.107 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:03.107 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:03.107 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:03.107 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.107 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.107 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155261 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 520192~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:03.107 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441187+0000", 2024-09-21T15:51:03.107 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 40.590814090000002, 2024-09-21T15:51:03.107 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.050124931999999997, 2024-09-21T15:51:03.108 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:03.108 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:03.108 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:03.108 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:03.108 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:03.108 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155261 2024-09-21T15:51:03.108 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.108 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:03.108 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.108 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:03.108 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441187+0000", 2024-09-21T15:51:03.108 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:03.108 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.108 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.108 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:03.109 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441187+0000", 2024-09-21T15:51:03.109 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:03.109 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.109 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.109 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:03.109 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441187+0000", 2024-09-21T15:51:03.109 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:03.109 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.109 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.109 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:03.109 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441189+0000", 2024-09-21T15:51:03.109 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3649999999999998e-06 2024-09-21T15:51:03.109 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.109 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.109 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:03.109 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441196+0000", 2024-09-21T15:51:03.110 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5379999999999999e-06 2024-09-21T15:51:03.110 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.110 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.110 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:03.110 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441200+0000", 2024-09-21T15:51:03.110 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4719999999999997e-06 2024-09-21T15:51:03.110 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.110 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.110 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:03.110 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491070+0000", 2024-09-21T15:51:03.110 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049869915000000001 2024-09-21T15:51:03.110 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.110 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.110 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:03.111 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491146+0000", 2024-09-21T15:51:03.111 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6589999999999997e-05 2024-09-21T15:51:03.111 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.111 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.111 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:03.111 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491312+0000", 2024-09-21T15:51:03.111 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000165475 2024-09-21T15:51:03.111 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:03.111 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:03.111 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:03.111 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.111 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.111 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155308 3.11 3:8b5183e5:::rbd_data.11335aef9442.0000000000000000:2a [read 356352~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:03.111 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441369+0000", 2024-09-21T15:51:03.111 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 40.590632241999998, 2024-09-21T15:51:03.112 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.050090672000000003, 2024-09-21T15:51:03.112 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:03.112 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:03.112 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:03.112 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:03.112 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:03.112 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155308 2024-09-21T15:51:03.112 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.112 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:03.112 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.112 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:03.112 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441369+0000", 2024-09-21T15:51:03.112 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:03.112 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.112 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.113 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:03.113 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441369+0000", 2024-09-21T15:51:03.113 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:03.113 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.113 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.113 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:03.113 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441368+0000", 2024-09-21T15:51:03.113 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:03.113 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.113 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.113 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:03.113 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441371+0000", 2024-09-21T15:51:03.113 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1100000000000001e-06 2024-09-21T15:51:03.113 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.113 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.113 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:03.114 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441377+0000", 2024-09-21T15:51:03.114 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5649999999999998e-06 2024-09-21T15:51:03.114 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.114 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.114 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:03.114 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441382+0000", 2024-09-21T15:51:03.114 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5929999999999999e-06 2024-09-21T15:51:03.114 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.114 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.114 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:03.114 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491349+0000", 2024-09-21T15:51:03.114 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049967122000000003 2024-09-21T15:51:03.114 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.114 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.115 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:03.115 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491400+0000", 2024-09-21T15:51:03.115 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1589999999999999e-05 2024-09-21T15:51:03.115 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.115 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.115 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:03.115 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491460+0000", 2024-09-21T15:51:03.115 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9070000000000002e-05 2024-09-21T15:51:03.115 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:03.115 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:03.115 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:03.115 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.115 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.115 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155309 3.11 3:8b5183e5:::rbd_data.11335aef9442.0000000000000000:2a [read 389120~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:03.115 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441386+0000", 2024-09-21T15:51:03.115 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 40.590614893000001, 2024-09-21T15:51:03.116 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.050184777, 2024-09-21T15:51:03.116 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:03.116 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:03.116 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:03.116 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:03.116 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:03.116 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155309 2024-09-21T15:51:03.116 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.116 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:03.116 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.116 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:03.116 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441386+0000", 2024-09-21T15:51:03.116 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:03.116 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.117 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.117 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:03.117 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441386+0000", 2024-09-21T15:51:03.117 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:03.117 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.117 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.117 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:03.117 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441386+0000", 2024-09-21T15:51:03.117 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:03.117 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.117 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.117 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:03.117 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441388+0000", 2024-09-21T15:51:03.117 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4420000000000001e-06 2024-09-21T15:51:03.117 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.118 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.118 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:03.118 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441395+0000", 2024-09-21T15:51:03.118 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6189999999999996e-06 2024-09-21T15:51:03.118 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.118 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.118 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:03.118 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441400+0000", 2024-09-21T15:51:03.118 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.118e-06 2024-09-21T15:51:03.118 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.118 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.118 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:03.118 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491473+0000", 2024-09-21T15:51:03.118 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.050072783000000003 2024-09-21T15:51:03.118 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.118 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.119 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:03.119 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491518+0000", 2024-09-21T15:51:03.119 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4854000000000001e-05 2024-09-21T15:51:03.119 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:03.119 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:03.119 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:03.119 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491571+0000", 2024-09-21T15:51:03.119 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3358e-05 2024-09-21T15:51:03.119 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:03.119 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:03.119 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:03.119 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:03.119 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:03.119 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:51:03.121 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:51:03.121 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:51:03.120+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:51:03.222 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:51:03.220+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:51:03.223 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:03.223 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:51:03.223 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:51:03.223 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:03.231 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:51:03.323 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:51:03.321+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:51:03.373 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:03.373 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:51:03.373 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:51:03.373 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:51:03.374 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:03.382 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:51:03.423 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:51:03.422+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:51:03.524 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:51:03.522+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:51:03.526 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:03.526 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:51:03.526 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:51:03.526 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:51:03.526 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.526 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160505 3.b 3:d297695c:::rbd_header.11338bae974a:head [watch reconnect cookie 140381478598448 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-21T15:51:03.526 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:29.138810+0000", 2024-09-21T15:51:03.526 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 34.385047513000004, 2024-09-21T15:51:03.526 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0172598319999999, 2024-09-21T15:51:03.526 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:03.526 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:03.527 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:03.527 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:03.527 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:03.527 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160505 2024-09-21T15:51:03.527 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.527 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:03.527 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.527 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:03.527 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138810+0000", 2024-09-21T15:51:03.527 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:03.527 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.527 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.527 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:03.528 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138810+0000", 2024-09-21T15:51:03.528 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:03.528 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.528 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.528 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:03.528 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138808+0000", 2024-09-21T15:51:03.528 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:51:03.528 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.528 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.528 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:03.528 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138815+0000", 2024-09-21T15:51:03.528 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6549999999999998e-06 2024-09-21T15:51:03.528 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.528 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.528 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:03.529 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138825+0000", 2024-09-21T15:51:03.529 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9790000000000005e-06 2024-09-21T15:51:03.529 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.529 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.529 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:03.529 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138834+0000", 2024-09-21T15:51:03.529 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5179999999999993e-06 2024-09-21T15:51:03.529 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.529 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.529 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:03.529 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.140973+0000", 2024-09-21T15:51:03.529 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0021392049999999999 2024-09-21T15:51:03.529 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.529 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.530 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:51:03.530 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.140981+0000", 2024-09-21T15:51:03.530 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8909999999999995e-06 2024-09-21T15:51:03.530 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.530 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.530 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:03.530 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.150457+0000", 2024-09-21T15:51:03.530 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.009475645 2024-09-21T15:51:03.530 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.530 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.530 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for missing object", 2024-09-21T15:51:03.530 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.150589+0000", 2024-09-21T15:51:03.530 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000131779 2024-09-21T15:51:03.531 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.531 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.531 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:03.531 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.151723+0000", 2024-09-21T15:51:03.531 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0011345979999999999 2024-09-21T15:51:03.531 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.531 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.531 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-09-21T15:51:03.531 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.151872+0000", 2024-09-21T15:51:03.531 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000149021 2024-09-21T15:51:03.531 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.531 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.531 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:03.531 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.152563+0000", 2024-09-21T15:51:03.532 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00069051500000000005 2024-09-21T15:51:03.532 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.532 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.532 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:03.532 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.152673+0000", 2024-09-21T15:51:03.532 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000109795 2024-09-21T15:51:03.532 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.532 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.532 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:03.532 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.156069+0000", 2024-09-21T15:51:03.532 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003396736 2024-09-21T15:51:03.532 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:03.532 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:03.532 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:03.532 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.533 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.533 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160506 3.b 3:d297695c:::rbd_header.11338bae974a:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-21T15:51:03.533 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:29.138844+0000", 2024-09-21T15:51:03.533 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 34.385012625999998, 2024-09-21T15:51:03.533 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0174592680000001, 2024-09-21T15:51:03.533 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:03.533 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:03.533 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:03.533 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:03.533 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:03.533 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160506 2024-09-21T15:51:03.533 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.533 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:03.533 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.534 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:03.534 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138844+0000", 2024-09-21T15:51:03.534 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:03.534 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.534 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.534 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:03.534 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138844+0000", 2024-09-21T15:51:03.534 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:03.534 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.534 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.534 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:03.534 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138844+0000", 2024-09-21T15:51:03.534 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:03.534 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.534 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.535 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:03.535 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138848+0000", 2024-09-21T15:51:03.535 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.76e-06 2024-09-21T15:51:03.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.535 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.535 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:03.535 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138854+0000", 2024-09-21T15:51:03.535 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3890000000000002e-06 2024-09-21T15:51:03.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.535 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.535 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:03.535 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138860+0000", 2024-09-21T15:51:03.535 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6409999999999999e-06 2024-09-21T15:51:03.535 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.535 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.536 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:03.536 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.141015+0000", 2024-09-21T15:51:03.536 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0021553779999999999 2024-09-21T15:51:03.536 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.536 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.536 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:51:03.536 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.141021+0000", 2024-09-21T15:51:03.536 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7629999999999997e-06 2024-09-21T15:51:03.536 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.536 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.536 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:03.536 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.150858+0000", 2024-09-21T15:51:03.536 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0098372200000001 2024-09-21T15:51:03.536 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.537 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for missing object", 2024-09-21T15:51:03.537 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.150899+0000", 2024-09-21T15:51:03.537 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1112000000000002e-05 2024-09-21T15:51:03.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.537 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:03.537 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.152069+0000", 2024-09-21T15:51:03.537 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001170307 2024-09-21T15:51:03.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.537 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.537 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-09-21T15:51:03.537 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.152192+0000", 2024-09-21T15:51:03.537 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012289700000000001 2024-09-21T15:51:03.537 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.538 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.538 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:03.538 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.153362+0000", 2024-09-21T15:51:03.538 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0011700549999999999 2024-09-21T15:51:03.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.538 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.538 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:03.538 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.153457+0000", 2024-09-21T15:51:03.538 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4687999999999999e-05 2024-09-21T15:51:03.538 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.538 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.538 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:51:03.538 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.153893+0000", 2024-09-21T15:51:03.538 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000436363 2024-09-21T15:51:03.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.539 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:03.539 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.156134+0000", 2024-09-21T15:51:03.539 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0022410009999999998 2024-09-21T15:51:03.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.539 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:03.539 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.156148+0000", 2024-09-21T15:51:03.539 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3574000000000001e-05 2024-09-21T15:51:03.539 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.539 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.539 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:03.539 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.156214+0000", 2024-09-21T15:51:03.539 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6438000000000003e-05 2024-09-21T15:51:03.540 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.540 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.540 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:03.540 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.156304+0000", 2024-09-21T15:51:03.540 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9278999999999997e-05 2024-09-21T15:51:03.540 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:03.540 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:03.540 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:03.540 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.540 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.540 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160512 3.b 3:d297695c:::rbd_header.11338bae974a:head [watch unwatch cookie 140381478598448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T15:51:03.540 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:30.161362+0000", 2024-09-21T15:51:03.540 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 33.362494818000002, 2024-09-21T15:51:03.540 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0074105029999999997, 2024-09-21T15:51:03.540 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:03.541 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:03.541 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:03.541 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:03.541 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:03.541 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160512 2024-09-21T15:51:03.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.541 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:03.541 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.541 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:03.541 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161362+0000", 2024-09-21T15:51:03.541 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:03.541 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.541 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.541 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:03.541 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161362+0000", 2024-09-21T15:51:03.542 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:03.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.542 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:03.542 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161362+0000", 2024-09-21T15:51:03.542 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:03.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.542 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:03.542 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161365+0000", 2024-09-21T15:51:03.542 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8669999999999998e-06 2024-09-21T15:51:03.542 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.542 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.542 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:03.542 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161371+0000", 2024-09-21T15:51:03.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8009999999999998e-06 2024-09-21T15:51:03.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.543 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.543 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:03.543 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161376+0000", 2024-09-21T15:51:03.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0340000000000004e-06 2024-09-21T15:51:03.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.543 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.543 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:03.543 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161762+0000", 2024-09-21T15:51:03.543 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038575099999999999 2024-09-21T15:51:03.543 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.543 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.543 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:03.543 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161838+0000", 2024-09-21T15:51:03.544 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5697000000000004e-05 2024-09-21T15:51:03.544 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.544 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.544 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:51:03.544 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161952+0000", 2024-09-21T15:51:03.544 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000114034 2024-09-21T15:51:03.544 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.544 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.544 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:03.544 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.163727+0000", 2024-09-21T15:51:03.544 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0017749560000000001 2024-09-21T15:51:03.544 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.544 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.544 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:03.544 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168545+0000", 2024-09-21T15:51:03.545 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0048186080000000003 2024-09-21T15:51:03.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.545 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.545 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:03.545 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168593+0000", 2024-09-21T15:51:03.545 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7549999999999997e-05 2024-09-21T15:51:03.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.545 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.545 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:03.545 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168773+0000", 2024-09-21T15:51:03.545 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00017976600000000001 2024-09-21T15:51:03.545 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:03.545 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:03.545 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:03.545 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.546 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160513 3.b 3:d297695c:::rbd_header.11338bae974a:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-09-21T15:51:03.546 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:30.161755+0000", 2024-09-21T15:51:03.546 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 33.362101887000001, 2024-09-21T15:51:03.546 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0072250300000000003, 2024-09-21T15:51:03.546 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:03.546 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:03.546 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:03.546 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:03.546 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:03.546 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160513 2024-09-21T15:51:03.546 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.546 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:03.546 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.546 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:03.547 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161755+0000", 2024-09-21T15:51:03.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:03.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.547 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:03.547 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161755+0000", 2024-09-21T15:51:03.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:03.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.547 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:03.547 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161754+0000", 2024-09-21T15:51:03.547 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:03.547 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.547 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.547 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:03.548 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161759+0000", 2024-09-21T15:51:03.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9699999999999998e-06 2024-09-21T15:51:03.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.548 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.548 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:03.548 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161768+0000", 2024-09-21T15:51:03.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9949999999999994e-06 2024-09-21T15:51:03.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.548 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.548 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:03.548 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161774+0000", 2024-09-21T15:51:03.548 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1290000000000001e-06 2024-09-21T15:51:03.548 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.548 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.548 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:03.548 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.162239+0000", 2024-09-21T15:51:03.549 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00046406600000000001 2024-09-21T15:51:03.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.549 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.549 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-09-21T15:51:03.549 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.162296+0000", 2024-09-21T15:51:03.549 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7122e-05 2024-09-21T15:51:03.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.549 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.549 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:03.549 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168802+0000", 2024-09-21T15:51:03.549 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0065061700000000004 2024-09-21T15:51:03.549 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.549 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.550 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:03.550 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168877+0000", 2024-09-21T15:51:03.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5214000000000002e-05 2024-09-21T15:51:03.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.550 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.550 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:03.550 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168980+0000", 2024-09-21T15:51:03.550 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000103182 2024-09-21T15:51:03.550 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:03.550 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:03.550 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:03.550 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.550 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.550 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160527 3.b 3:d297695c:::rbd_header.11338bae974a:head [watch watch cookie 140381476148256] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T15:51:03.550 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:30.182756+0000", 2024-09-21T15:51:03.551 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 33.341101305999999, 2024-09-21T15:51:03.551 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0054202579999999998, 2024-09-21T15:51:03.551 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:03.551 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:03.551 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:03.551 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:03.551 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:03.551 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160527 2024-09-21T15:51:03.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.551 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:03.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:03.551 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182756+0000", 2024-09-21T15:51:03.551 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:03.551 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.551 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.551 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:03.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182756+0000", 2024-09-21T15:51:03.552 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:03.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.552 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:03.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182755+0000", 2024-09-21T15:51:03.552 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:03.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.552 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:03.552 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182760+0000", 2024-09-21T15:51:03.552 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4269999999999999e-06 2024-09-21T15:51:03.552 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.552 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.552 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:03.553 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182771+0000", 2024-09-21T15:51:03.553 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0312999999999999e-05 2024-09-21T15:51:03.553 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.553 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.553 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:03.553 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182778+0000", 2024-09-21T15:51:03.553 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5700000000000004e-06 2024-09-21T15:51:03.553 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.553 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.553 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:03.553 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.184745+0000", 2024-09-21T15:51:03.553 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0019667170000000002 2024-09-21T15:51:03.553 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.553 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.553 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:03.553 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.184867+0000", 2024-09-21T15:51:03.554 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000122504 2024-09-21T15:51:03.554 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.554 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.554 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:51:03.554 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.185135+0000", 2024-09-21T15:51:03.554 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000267135 2024-09-21T15:51:03.554 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.554 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.554 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:03.554 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.187721+0000", 2024-09-21T15:51:03.554 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002586891 2024-09-21T15:51:03.554 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.554 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.554 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:03.555 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.187874+0000", 2024-09-21T15:51:03.555 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000152726 2024-09-21T15:51:03.555 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.555 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.555 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:03.555 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.187951+0000", 2024-09-21T15:51:03.555 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7132000000000006e-05 2024-09-21T15:51:03.555 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.555 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.555 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:03.555 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.188176+0000", 2024-09-21T15:51:03.555 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00022477900000000001 2024-09-21T15:51:03.555 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:03.555 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:03.555 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:03.556 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.556 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.556 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160605 3.b 3:d1863919:::rbd_data.11338bae974a.0000000000000008:head [delete] snapc 2d=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-21T15:51:03.556 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.213794+0000", 2024-09-21T15:51:03.556 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 32.310063425999999, 2024-09-21T15:51:03.556 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046152490000000001, 2024-09-21T15:51:03.556 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:03.556 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:03.556 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:03.556 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:03.556 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:03.556 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160605 2024-09-21T15:51:03.556 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.556 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:03.556 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.556 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:03.557 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.213794+0000", 2024-09-21T15:51:03.557 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:03.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.557 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.557 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:03.557 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.213794+0000", 2024-09-21T15:51:03.557 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:03.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.557 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.557 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:03.557 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.213793+0000", 2024-09-21T15:51:03.557 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:03.557 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.557 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.557 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:03.557 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.213799+0000", 2024-09-21T15:51:03.558 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9959999999999999e-06 2024-09-21T15:51:03.558 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.558 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.558 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:03.558 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.213815+0000", 2024-09-21T15:51:03.558 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5880999999999999e-05 2024-09-21T15:51:03.558 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.558 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.558 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:03.558 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.213838+0000", 2024-09-21T15:51:03.558 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3671000000000001e-05 2024-09-21T15:51:03.558 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.558 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.558 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:03.558 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.213887+0000", 2024-09-21T15:51:03.558 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8865000000000001e-05 2024-09-21T15:51:03.559 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.559 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.559 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:03.559 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.213962+0000", 2024-09-21T15:51:03.559 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4592999999999995e-05 2024-09-21T15:51:03.559 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.559 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.559 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:51:03.559 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.214088+0000", 2024-09-21T15:51:03.559 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012666199999999999 2024-09-21T15:51:03.559 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.559 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.559 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:03.559 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216917+0000", 2024-09-21T15:51:03.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0028284059999999999 2024-09-21T15:51:03.560 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.560 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:03.560 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.218319+0000", 2024-09-21T15:51:03.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001402613 2024-09-21T15:51:03.560 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.560 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:03.560 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.218372+0000", 2024-09-21T15:51:03.560 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2636000000000002e-05 2024-09-21T15:51:03.560 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.560 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.561 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:03.561 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.218409+0000", 2024-09-21T15:51:03.561 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6942e-05 2024-09-21T15:51:03.561 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:03.561 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:03.561 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:03.561 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.561 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.561 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:160602 3.12 e95/94 3:49a61a92:::rbd_data.11338bae974a.0000000000000005:head v 95'344, mlcod=0'0)", 2024-09-21T15:51:03.561 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.215034+0000", 2024-09-21T15:51:03.561 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 32.308823009000001, 2024-09-21T15:51:03.561 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045544870000000003, 2024-09-21T15:51:03.561 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:03.561 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:03.561 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:03.562 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.562 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:03.562 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215034+0000", 2024-09-21T15:51:03.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:03.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.562 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.562 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:03.562 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215034+0000", 2024-09-21T15:51:03.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:03.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.562 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.562 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:03.562 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215034+0000", 2024-09-21T15:51:03.562 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:03.562 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.562 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.563 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:03.563 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215039+0000", 2024-09-21T15:51:03.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1549999999999998e-06 2024-09-21T15:51:03.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.563 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:03.563 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215050+0000", 2024-09-21T15:51:03.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1392e-05 2024-09-21T15:51:03.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.563 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:03.563 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215057+0000", 2024-09-21T15:51:03.563 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7399999999999998e-06 2024-09-21T15:51:03.563 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.563 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:03.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215187+0000", 2024-09-21T15:51:03.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013010100000000001 2024-09-21T15:51:03.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:03.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215208+0000", 2024-09-21T15:51:03.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1013999999999999e-05 2024-09-21T15:51:03.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:03.564 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.219549+0000", 2024-09-21T15:51:03.564 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0043404990000000003 2024-09-21T15:51:03.564 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.564 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.564 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:03.565 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.219589+0000", 2024-09-21T15:51:03.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9878e-05 2024-09-21T15:51:03.565 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:03.565 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:03.565 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:03.565 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.565 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.565 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:160597 3.1 e95/94 3:84154dfc:::rbd_data.11338bae974a.0000000000000000:head v 95'451, mlcod=0'0)", 2024-09-21T15:51:03.565 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.215080+0000", 2024-09-21T15:51:03.565 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 32.308777552999999, 2024-09-21T15:51:03.565 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045596819999999998, 2024-09-21T15:51:03.565 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:03.565 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:03.565 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:03.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.566 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:03.566 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215080+0000", 2024-09-21T15:51:03.566 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:03.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.566 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:03.566 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215080+0000", 2024-09-21T15:51:03.566 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:03.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.566 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.566 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:03.566 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215079+0000", 2024-09-21T15:51:03.566 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:03.566 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.567 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.567 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:03.567 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215083+0000", 2024-09-21T15:51:03.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2570000000000001e-06 2024-09-21T15:51:03.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.567 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.567 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:03.567 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215091+0000", 2024-09-21T15:51:03.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7829999999999999e-06 2024-09-21T15:51:03.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.567 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.567 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:03.567 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215096+0000", 2024-09-21T15:51:03.567 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0370000000000002e-06 2024-09-21T15:51:03.567 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.568 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:03.568 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216055+0000", 2024-09-21T15:51:03.568 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00095892299999999998 2024-09-21T15:51:03.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.568 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:03.568 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216078+0000", 2024-09-21T15:51:03.568 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2376000000000001e-05 2024-09-21T15:51:03.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.568 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:03.568 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.219598+0000", 2024-09-21T15:51:03.568 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035209299999999998 2024-09-21T15:51:03.568 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.568 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.569 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:03.569 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.219639+0000", 2024-09-21T15:51:03.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0766000000000002e-05 2024-09-21T15:51:03.569 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:03.569 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:03.569 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:03.569 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.569 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:160608 3.12 e95/94 3:48a47e1c:::rbd_data.11338bae974a.000000000000000b:head v 95'345, mlcod=0'0)", 2024-09-21T15:51:03.569 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.216934+0000", 2024-09-21T15:51:03.569 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 32.306923167999997, 2024-09-21T15:51:03.569 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053335759999999996, 2024-09-21T15:51:03.569 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:03.569 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:03.569 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:03.569 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:03.570 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216934+0000", 2024-09-21T15:51:03.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:03.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:03.570 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216934+0000", 2024-09-21T15:51:03.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:03.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:03.570 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216933+0000", 2024-09-21T15:51:03.570 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:03.570 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.570 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.570 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:03.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216939+0000", 2024-09-21T15:51:03.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6459999999999998e-06 2024-09-21T15:51:03.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:03.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216947+0000", 2024-09-21T15:51:03.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.2479999999999996e-06 2024-09-21T15:51:03.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:03.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216951+0000", 2024-09-21T15:51:03.571 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3760000000000001e-06 2024-09-21T15:51:03.571 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.571 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.571 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:03.571 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216998+0000", 2024-09-21T15:51:03.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6244e-05 2024-09-21T15:51:03.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.572 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:03.572 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.217019+0000", 2024-09-21T15:51:03.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1191000000000001e-05 2024-09-21T15:51:03.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.572 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:03.572 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.222230+0000", 2024-09-21T15:51:03.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052114350000000004 2024-09-21T15:51:03.572 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.572 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.572 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:03.572 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.222267+0000", 2024-09-21T15:51:03.572 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7169999999999998e-05 2024-09-21T15:51:03.573 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:03.573 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:03.573 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:03.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.573 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160618 3.b 3:d297695c:::rbd_header.11338bae974a:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-21T15:51:03.573 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.233457+0000", 2024-09-21T15:51:03.573 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 32.290400331000001, 2024-09-21T15:51:03.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.005085198, 2024-09-21T15:51:03.573 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:03.573 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:03.573 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:03.573 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:03.573 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:03.573 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160618 2024-09-21T15:51:03.573 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.574 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:03.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.574 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:03.574 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233457+0000", 2024-09-21T15:51:03.574 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:03.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.574 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:03.574 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233457+0000", 2024-09-21T15:51:03.574 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:03.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.574 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:03.575 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233456+0000", 2024-09-21T15:51:03.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:03.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:03.575 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233463+0000", 2024-09-21T15:51:03.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9310000000000002e-06 2024-09-21T15:51:03.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:03.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233473+0000", 2024-09-21T15:51:03.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0519e-05 2024-09-21T15:51:03.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.576 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:03.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233481+0000", 2024-09-21T15:51:03.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4009999999999997e-06 2024-09-21T15:51:03.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.576 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:03.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233513+0000", 2024-09-21T15:51:03.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1328999999999998e-05 2024-09-21T15:51:03.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.576 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:03.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233568+0000", 2024-09-21T15:51:03.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5606999999999998e-05 2024-09-21T15:51:03.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:51:03.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233670+0000", 2024-09-21T15:51:03.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010180599999999999 2024-09-21T15:51:03.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:03.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.234761+0000", 2024-09-21T15:51:03.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010905909999999999 2024-09-21T15:51:03.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:03.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.238442+0000", 2024-09-21T15:51:03.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0036809429999999999 2024-09-21T15:51:03.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.578 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:03.578 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.238498+0000", 2024-09-21T15:51:03.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6116999999999998e-05 2024-09-21T15:51:03.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.578 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:03.578 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.238542+0000", 2024-09-21T15:51:03.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4134000000000001e-05 2024-09-21T15:51:03.578 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:03.578 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:03.578 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:03.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.579 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160662 3.b 3:d7701915:::rbd_object_map.1133fd7483d4.000000000000002c:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-21T15:51:03.579 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.279369+0000", 2024-09-21T15:51:03.579 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 32.244488400999998, 2024-09-21T15:51:03.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0056948040000000004, 2024-09-21T15:51:03.579 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:03.579 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:03.579 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:03.579 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:03.579 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:03.579 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160662 2024-09-21T15:51:03.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.579 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:03.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:03.580 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279369+0000", 2024-09-21T15:51:03.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:03.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:03.580 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279369+0000", 2024-09-21T15:51:03.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:03.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:03.580 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279368+0000", 2024-09-21T15:51:03.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:03.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:03.581 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279373+0000", 2024-09-21T15:51:03.581 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6529999999999999e-06 2024-09-21T15:51:03.581 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.581 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:03.581 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279384+0000", 2024-09-21T15:51:03.581 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0256e-05 2024-09-21T15:51:03.581 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.581 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:03.581 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279388+0000", 2024-09-21T15:51:03.581 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5630000000000002e-06 2024-09-21T15:51:03.581 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.581 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:03.581 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279424+0000", 2024-09-21T15:51:03.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5985999999999997e-05 2024-09-21T15:51:03.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.582 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.582 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:03.582 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279478+0000", 2024-09-21T15:51:03.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3944999999999999e-05 2024-09-21T15:51:03.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.582 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.582 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:51:03.582 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279575+0000", 2024-09-21T15:51:03.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6817999999999994e-05 2024-09-21T15:51:03.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.582 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.582 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:03.582 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.282391+0000", 2024-09-21T15:51:03.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0028156700000000002 2024-09-21T15:51:03.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:03.583 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.284977+0000", 2024-09-21T15:51:03.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0025859640000000001 2024-09-21T15:51:03.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:03.583 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.285021+0000", 2024-09-21T15:51:03.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4415000000000001e-05 2024-09-21T15:51:03.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:03.583 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.285063+0000", 2024-09-21T15:51:03.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2447000000000002e-05 2024-09-21T15:51:03.583 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:03.584 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:03.584 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:03.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.584 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:161122 3.1 3:82ccd16f:::rbd_header.11335aef9442:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-21T15:51:03.584 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:36.703081+0000", 2024-09-21T15:51:03.584 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 26.820776238000001, 2024-09-21T15:51:03.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0051212200000000001, 2024-09-21T15:51:03.584 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:03.584 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:03.584 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:03.584 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:03.584 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:03.585 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 161122 2024-09-21T15:51:03.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.585 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:03.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.585 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:03.585 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.703081+0000", 2024-09-21T15:51:03.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:03.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.585 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:03.585 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.703081+0000", 2024-09-21T15:51:03.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:03.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.586 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:03.586 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.703079+0000", 2024-09-21T15:51:03.586 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-09-21T15:51:03.586 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.586 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:03.586 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.703091+0000", 2024-09-21T15:51:03.586 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2584e-05 2024-09-21T15:51:03.586 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.586 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:03.586 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.703107+0000", 2024-09-21T15:51:03.586 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6313000000000001e-05 2024-09-21T15:51:03.587 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.587 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:03.587 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.703119+0000", 2024-09-21T15:51:03.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1404e-05 2024-09-21T15:51:03.587 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.587 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:03.587 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.703176+0000", 2024-09-21T15:51:03.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6740999999999998e-05 2024-09-21T15:51:03.587 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.587 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:03.587 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.703281+0000", 2024-09-21T15:51:03.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010508 2024-09-21T15:51:03.588 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.588 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-09-21T15:51:03.588 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.703483+0000", 2024-09-21T15:51:03.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00020257399999999999 2024-09-21T15:51:03.588 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.588 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:03.588 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.705657+0000", 2024-09-21T15:51:03.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0021735610000000001 2024-09-21T15:51:03.588 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.588 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:03.588 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.708144+0000", 2024-09-21T15:51:03.589 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002487122 2024-09-21T15:51:03.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.589 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.589 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:03.589 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.708177+0000", 2024-09-21T15:51:03.589 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2574999999999999e-05 2024-09-21T15:51:03.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.589 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.589 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:03.589 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.708202+0000", 2024-09-21T15:51:03.589 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5513000000000001e-05 2024-09-21T15:51:03.589 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:03.589 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:03.589 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:03.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.589 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.590 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:161219 3.17 3:e87cb47d:::rbd_data.11335aef9442.000000000000000a:head [delete] snapc 2a=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-21T15:51:03.590 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:36.771292+0000", 2024-09-21T15:51:03.590 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 26.752564588999999, 2024-09-21T15:51:03.590 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0058540529999999997, 2024-09-21T15:51:03.590 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:03.590 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:03.590 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:03.590 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:03.590 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:03.590 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 161219 2024-09-21T15:51:03.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.590 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:03.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.590 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:03.591 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771292+0000", 2024-09-21T15:51:03.591 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:03.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.591 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.591 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:03.591 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771292+0000", 2024-09-21T15:51:03.591 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:03.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.591 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.591 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:03.591 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771292+0000", 2024-09-21T15:51:03.591 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:03.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.591 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.591 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:03.592 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771298+0000", 2024-09-21T15:51:03.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9519999999999999e-06 2024-09-21T15:51:03.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.592 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:03.592 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771308+0000", 2024-09-21T15:51:03.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0601000000000001e-05 2024-09-21T15:51:03.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.592 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:03.592 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771314+0000", 2024-09-21T15:51:03.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.913e-06 2024-09-21T15:51:03.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.592 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:03.592 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771346+0000", 2024-09-21T15:51:03.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1711000000000002e-05 2024-09-21T15:51:03.593 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.593 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.593 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:03.593 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771399+0000", 2024-09-21T15:51:03.593 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2883000000000002e-05 2024-09-21T15:51:03.593 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.593 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.593 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-09-21T15:51:03.593 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771470+0000", 2024-09-21T15:51:03.593 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0934999999999998e-05 2024-09-21T15:51:03.593 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.593 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.593 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:03.593 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.772234+0000", 2024-09-21T15:51:03.593 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00076485400000000003 2024-09-21T15:51:03.593 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.594 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.594 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:03.594 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.777083+0000", 2024-09-21T15:51:03.594 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0048486329999999998 2024-09-21T15:51:03.594 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.594 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.594 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:03.594 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.777111+0000", 2024-09-21T15:51:03.594 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8399999999999999e-05 2024-09-21T15:51:03.594 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.594 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.594 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:03.594 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.777147+0000", 2024-09-21T15:51:03.594 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5049000000000003e-05 2024-09-21T15:51:03.594 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:03.595 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:03.595 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:03.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.595 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.595 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:161762 3.17 3:ef707828:::rbd_header.11338db70e6f:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T15:51:03.595 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:40.705291+0000", 2024-09-21T15:51:03.595 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 22.818566538999999, 2024-09-21T15:51:03.595 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0048878120000000001, 2024-09-21T15:51:03.595 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:03.595 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:03.595 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:03.595 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:03.595 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:03.595 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 161762 2024-09-21T15:51:03.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.596 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:03.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:03.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.705291+0000", 2024-09-21T15:51:03.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:03.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:03.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.705291+0000", 2024-09-21T15:51:03.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:03.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:03.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.705289+0000", 2024-09-21T15:51:03.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:51:03.597 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.597 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.597 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:03.597 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.705296+0000", 2024-09-21T15:51:03.597 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1999999999999997e-06 2024-09-21T15:51:03.597 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.597 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.597 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:03.597 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.705309+0000", 2024-09-21T15:51:03.597 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2723e-05 2024-09-21T15:51:03.597 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.597 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.597 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:03.597 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.705318+0000", 2024-09-21T15:51:03.597 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4890000000000001e-06 2024-09-21T15:51:03.597 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.598 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.598 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:03.598 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.705370+0000", 2024-09-21T15:51:03.598 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1493000000000003e-05 2024-09-21T15:51:03.598 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.598 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.598 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:03.598 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.705480+0000", 2024-09-21T15:51:03.598 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011059 2024-09-21T15:51:03.598 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.598 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.598 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-09-21T15:51:03.598 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.705800+0000", 2024-09-21T15:51:03.598 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00031958399999999998 2024-09-21T15:51:03.599 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.599 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.599 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:03.599 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.707140+0000", 2024-09-21T15:51:03.599 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001339954 2024-09-21T15:51:03.599 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.599 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.599 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:03.599 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.710103+0000", 2024-09-21T15:51:03.599 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029629109999999999 2024-09-21T15:51:03.599 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.599 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.599 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:03.600 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.710136+0000", 2024-09-21T15:51:03.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3300000000000003e-05 2024-09-21T15:51:03.600 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.600 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.600 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:03.600 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.710178+0000", 2024-09-21T15:51:03.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2172999999999998e-05 2024-09-21T15:51:03.600 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:03.600 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:03.600 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:03.600 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.600 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.600 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:161859 3.17 3:ee7ab022:::rbd_data.11338db70e6f.0000000000000007:head [delete] snapc 26=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T15:51:03.600 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:40.782264+0000", 2024-09-21T15:51:03.600 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 22.741592730000001, 2024-09-21T15:51:03.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052663939999999998, 2024-09-21T15:51:03.601 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:03.601 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:03.601 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:03.601 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:03.601 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:03.601 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 161859 2024-09-21T15:51:03.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.601 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:03.601 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.601 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:03.601 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782264+0000", 2024-09-21T15:51:03.601 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:03.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.601 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.601 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:03.601 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782264+0000", 2024-09-21T15:51:03.602 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:03.602 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.602 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.602 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:03.602 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782264+0000", 2024-09-21T15:51:03.602 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:03.602 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.602 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.602 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:03.602 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782266+0000", 2024-09-21T15:51:03.602 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4590000000000001e-06 2024-09-21T15:51:03.602 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.602 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.602 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:03.602 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782275+0000", 2024-09-21T15:51:03.603 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5450000000000003e-06 2024-09-21T15:51:03.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.603 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:03.603 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782281+0000", 2024-09-21T15:51:03.603 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7180000000000001e-06 2024-09-21T15:51:03.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.603 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:03.603 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782312+0000", 2024-09-21T15:51:03.603 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1059000000000002e-05 2024-09-21T15:51:03.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.603 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:03.603 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782470+0000", 2024-09-21T15:51:03.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000157818 2024-09-21T15:51:03.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.604 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-09-21T15:51:03.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782574+0000", 2024-09-21T15:51:03.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000104536 2024-09-21T15:51:03.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.604 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:03.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.783499+0000", 2024-09-21T15:51:03.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00092479599999999997 2024-09-21T15:51:03.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.604 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:03.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.787472+0000", 2024-09-21T15:51:03.605 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0039730319999999996 2024-09-21T15:51:03.605 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.605 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.605 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:03.605 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.787507+0000", 2024-09-21T15:51:03.605 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5552000000000001e-05 2024-09-21T15:51:03.605 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.605 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.605 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:03.605 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.787531+0000", 2024-09-21T15:51:03.605 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3281999999999998e-05 2024-09-21T15:51:03.605 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:03.605 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:03.605 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:03.606 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.606 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.606 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:162390 3.17 3:ea8605c7:::rbd_header.1133fc941201:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-09-21T15:51:03.606 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:44.740587+0000", 2024-09-21T15:51:03.606 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 18.783270133999999, 2024-09-21T15:51:03.606 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052218530000000003, 2024-09-21T15:51:03.606 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:03.606 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:03.606 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:03.606 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:03.606 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:03.606 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 162390 2024-09-21T15:51:03.606 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.606 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:03.606 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.606 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:03.607 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.740587+0000", 2024-09-21T15:51:03.607 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:03.607 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.607 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.607 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:03.607 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.740587+0000", 2024-09-21T15:51:03.607 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:03.607 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.607 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.607 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:03.607 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.740585+0000", 2024-09-21T15:51:03.607 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:51:03.607 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.607 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.607 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:03.607 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.740595+0000", 2024-09-21T15:51:03.608 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7839999999999998e-06 2024-09-21T15:51:03.608 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.608 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.608 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:03.608 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.740609+0000", 2024-09-21T15:51:03.608 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.384e-05 2024-09-21T15:51:03.608 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.608 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.608 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:03.608 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.740617+0000", 2024-09-21T15:51:03.608 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7560000000000005e-06 2024-09-21T15:51:03.608 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.608 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.608 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:03.608 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.740671+0000", 2024-09-21T15:51:03.608 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4073999999999998e-05 2024-09-21T15:51:03.609 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.609 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.609 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:03.609 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.740799+0000", 2024-09-21T15:51:03.609 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000128057 2024-09-21T15:51:03.609 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.609 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.609 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-09-21T15:51:03.609 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.741258+0000", 2024-09-21T15:51:03.609 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00045892900000000001 2024-09-21T15:51:03.609 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.610 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.610 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:03.610 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.742101+0000", 2024-09-21T15:51:03.610 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00084276499999999999 2024-09-21T15:51:03.610 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.610 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.610 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:03.610 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.745743+0000", 2024-09-21T15:51:03.610 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0036415280000000002 2024-09-21T15:51:03.610 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.610 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.610 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:03.610 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.745781+0000", 2024-09-21T15:51:03.611 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8398000000000001e-05 2024-09-21T15:51:03.611 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.611 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.611 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:03.611 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.745809+0000", 2024-09-21T15:51:03.611 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7691999999999998e-05 2024-09-21T15:51:03.611 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:03.611 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:03.611 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:03.611 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.611 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.611 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:163005 3.6 3:67cae2ed:::rbd_mirroring:head [call rbd.mirror_image_remove in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-21T15:51:03.611 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:49.711037+0000", 2024-09-21T15:51:03.611 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.812819709999999, 2024-09-21T15:51:03.612 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0059230919999999996, 2024-09-21T15:51:03.612 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:03.612 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:03.612 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:03.612 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:03.612 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:03.612 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 163005 2024-09-21T15:51:03.612 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.612 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:03.612 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.612 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:03.612 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:49.711037+0000", 2024-09-21T15:51:03.612 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:03.612 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.613 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.613 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:03.613 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:49.711037+0000", 2024-09-21T15:51:03.613 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:03.613 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.613 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.613 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:03.613 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:49.711037+0000", 2024-09-21T15:51:03.613 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:03.613 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.613 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.613 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:03.613 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:49.711041+0000", 2024-09-21T15:51:03.613 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8199999999999996e-06 2024-09-21T15:51:03.613 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.613 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.614 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:03.614 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:49.711052+0000", 2024-09-21T15:51:03.614 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0567999999999999e-05 2024-09-21T15:51:03.614 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.614 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.614 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:03.614 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:49.711057+0000", 2024-09-21T15:51:03.614 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7119999999999999e-06 2024-09-21T15:51:03.614 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.614 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.614 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:03.614 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:49.711096+0000", 2024-09-21T15:51:03.614 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8872999999999999e-05 2024-09-21T15:51:03.615 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.615 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.615 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:03.615 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:49.711151+0000", 2024-09-21T15:51:03.615 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5343999999999997e-05 2024-09-21T15:51:03.615 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.615 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.615 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:03.615 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:49.716960+0000", 2024-09-21T15:51:03.615 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0058095150000000003 2024-09-21T15:51:03.615 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:03.615 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:03.615 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:03.616 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.616 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.616 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:163048 3.18 3:1ebf3ed8:::rbd_header.1133545d3fa6:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-21T15:51:03.616 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:50.675796+0000", 2024-09-21T15:51:03.616 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.848061002, 2024-09-21T15:51:03.616 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.004266784, 2024-09-21T15:51:03.616 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:03.616 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:03.616 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:03.616 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:03.616 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:03.616 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 163048 2024-09-21T15:51:03.616 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.616 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:03.617 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.617 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:03.617 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:50.675796+0000", 2024-09-21T15:51:03.617 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:03.617 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.617 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.617 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:03.617 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:50.675796+0000", 2024-09-21T15:51:03.617 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:03.617 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.617 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.617 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:03.617 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:50.675794+0000", 2024-09-21T15:51:03.617 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:51:03.617 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.617 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.618 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:03.618 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:50.675804+0000", 2024-09-21T15:51:03.618 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6660000000000003e-06 2024-09-21T15:51:03.618 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.618 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.618 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:03.618 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:50.675826+0000", 2024-09-21T15:51:03.618 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2388e-05 2024-09-21T15:51:03.618 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.618 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.618 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:03.618 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:50.675837+0000", 2024-09-21T15:51:03.618 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0764999999999999e-05 2024-09-21T15:51:03.618 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.619 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.619 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:03.619 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:50.675894+0000", 2024-09-21T15:51:03.619 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6566e-05 2024-09-21T15:51:03.619 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.619 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.619 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op must wait for map", 2024-09-21T15:51:03.619 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:50.675902+0000", 2024-09-21T15:51:03.619 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9820000000000008e-06 2024-09-21T15:51:03.619 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.619 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.619 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:03.619 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:50.677909+0000", 2024-09-21T15:51:03.619 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0020078890000000001 2024-09-21T15:51:03.619 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.620 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.620 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:03.620 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:50.678019+0000", 2024-09-21T15:51:03.620 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000109645 2024-09-21T15:51:03.620 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.620 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.620 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:51:03.620 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:50.678516+0000", 2024-09-21T15:51:03.620 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00049678400000000001 2024-09-21T15:51:03.620 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.620 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.620 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:03.620 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:50.679754+0000", 2024-09-21T15:51:03.620 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001237644 2024-09-21T15:51:03.620 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.620 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.621 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:03.621 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:50.680012+0000", 2024-09-21T15:51:03.621 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00025866700000000002 2024-09-21T15:51:03.621 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.621 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.621 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:03.621 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:50.680042+0000", 2024-09-21T15:51:03.621 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9635e-05 2024-09-21T15:51:03.621 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.621 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.621 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:03.621 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:50.680063+0000", 2024-09-21T15:51:03.621 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0999999999999999e-05 2024-09-21T15:51:03.621 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:03.621 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:03.622 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:03.622 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.622 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.622 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:163051 3.6 3:64ca7355:::rbd_object_map.1133545d3fa6:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-21T15:51:03.622 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:50.680972+0000", 2024-09-21T15:51:03.622 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 12.842884836, 2024-09-21T15:51:03.622 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.004798353, 2024-09-21T15:51:03.622 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:03.622 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:03.622 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:03.622 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:03.622 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:03.622 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 163051 2024-09-21T15:51:03.622 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.622 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:03.622 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.623 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:03.623 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:50.680972+0000", 2024-09-21T15:51:03.623 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:03.623 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.623 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.623 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:03.623 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:50.680972+0000", 2024-09-21T15:51:03.623 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:03.623 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.623 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.623 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:03.623 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:50.680972+0000", 2024-09-21T15:51:03.623 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:03.623 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.623 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.623 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:03.624 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:50.680976+0000", 2024-09-21T15:51:03.624 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6430000000000003e-06 2024-09-21T15:51:03.624 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.624 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.624 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:03.624 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:50.680993+0000", 2024-09-21T15:51:03.624 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7181e-05 2024-09-21T15:51:03.624 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.624 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.624 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:03.624 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:50.680998+0000", 2024-09-21T15:51:03.624 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8389999999999997e-06 2024-09-21T15:51:03.624 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.624 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.624 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:03.625 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:50.681035+0000", 2024-09-21T15:51:03.625 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6386e-05 2024-09-21T15:51:03.625 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.625 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.625 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:03.625 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:50.681114+0000", 2024-09-21T15:51:03.625 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9453e-05 2024-09-21T15:51:03.625 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.625 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.625 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-09-21T15:51:03.625 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:50.681279+0000", 2024-09-21T15:51:03.625 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00016474600000000001 2024-09-21T15:51:03.625 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.625 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.625 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:03.626 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:50.682375+0000", 2024-09-21T15:51:03.626 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010965969999999999 2024-09-21T15:51:03.626 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.626 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.626 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:03.626 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:50.685707+0000", 2024-09-21T15:51:03.626 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033312329999999999 2024-09-21T15:51:03.626 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.626 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.626 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:03.626 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:50.685743+0000", 2024-09-21T15:51:03.626 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6474999999999999e-05 2024-09-21T15:51:03.626 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.626 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.626 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:03.627 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:50.685771+0000", 2024-09-21T15:51:03.627 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7504e-05 2024-09-21T15:51:03.627 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:03.627 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:03.627 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:03.627 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.627 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.627 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.0:3628 3.3 e127/96 3:c171ff66:::rbd_data.113336ae1075.000000000000002d:16 v 127'794, mlcod=126'789)", 2024-09-21T15:51:03.627 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:51:02.805169+0000", 2024-09-21T15:51:03.627 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.71868850299999998, 2024-09-21T15:51:03.627 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0043620869999999997, 2024-09-21T15:51:03.627 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:03.627 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:03.627 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:03.627 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.627 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:03.628 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.805169+0000", 2024-09-21T15:51:03.628 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:03.628 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.628 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.628 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:03.628 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.805169+0000", 2024-09-21T15:51:03.628 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:03.628 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.628 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.628 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:03.628 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.805168+0000", 2024-09-21T15:51:03.628 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:03.628 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.628 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.628 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:03.628 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.805173+0000", 2024-09-21T15:51:03.629 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2569999999999998e-06 2024-09-21T15:51:03.629 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.629 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.629 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:03.629 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.805190+0000", 2024-09-21T15:51:03.629 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6509999999999999e-05 2024-09-21T15:51:03.629 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.629 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.629 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:03.629 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.805196+0000", 2024-09-21T15:51:03.629 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5579999999999997e-06 2024-09-21T15:51:03.629 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.629 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.629 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:03.629 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.805669+0000", 2024-09-21T15:51:03.629 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00047213000000000002 2024-09-21T15:51:03.630 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.630 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.630 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:03.630 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.805686+0000", 2024-09-21T15:51:03.630 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6991999999999999e-05 2024-09-21T15:51:03.630 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.630 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.630 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:03.630 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.809499+0000", 2024-09-21T15:51:03.630 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0038134520000000002 2024-09-21T15:51:03.630 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:03.630 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:03.630 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:03.630 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.809531+0000", 2024-09-21T15:51:03.631 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1702999999999998e-05 2024-09-21T15:51:03.631 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:03.631 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:03.631 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:03.631 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:03.631 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:03.631 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:03.632 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:51:03.631+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:51:03.632 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:51:03.733 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:51:03.732+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:51:03.773 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:03.774 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:51:03.774 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:51:03.774 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:03.782 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:51:03.789 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is unset 2024-09-21T15:51:03.789 INFO:teuthology.orchestra.run.smithi089.stderr:reweighted osd.5 to 1 (10000) 2024-09-21T15:51:03.807 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-09-21T15:51:03.834 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:51:03.833+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:51:03.897 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:03.897 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:51:03.897 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:51:03.898 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:51:03.898 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:03.907 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:51:03.950 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:51:03.949+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:51:04.027 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:04.027 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:51:04.027 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:51:04.027 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:51:04.027 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.027 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62833 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 192512~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:04.027 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:51:04.027 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 86.883257916999995, 2024-09-21T15:51:04.027 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050387007999999997, 2024-09-21T15:51:04.028 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:04.028 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:04.028 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:04.028 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:04.028 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:04.028 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62833 2024-09-21T15:51:04.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.028 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:04.028 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.028 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:04.028 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:51:04.028 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:04.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.028 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.028 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:04.028 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:51:04.029 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:04.029 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.029 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.029 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:04.029 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:51:04.029 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:51:04.029 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.029 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.029 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:04.029 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142150+0000", 2024-09-21T15:51:04.029 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1999999999999999e-06 2024-09-21T15:51:04.029 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.029 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.029 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:04.029 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142153+0000", 2024-09-21T15:51:04.029 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7410000000000001e-06 2024-09-21T15:51:04.030 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.030 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.030 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:04.030 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142156+0000", 2024-09-21T15:51:04.030 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5739999999999998e-06 2024-09-21T15:51:04.030 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.030 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.030 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:04.030 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.192358+0000", 2024-09-21T15:51:04.030 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050202452000000002 2024-09-21T15:51:04.030 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.030 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.030 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:04.030 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.192435+0000", 2024-09-21T15:51:04.030 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6738999999999998e-05 2024-09-21T15:51:04.031 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.031 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.031 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:04.031 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.192536+0000", 2024-09-21T15:51:04.031 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101508 2024-09-21T15:51:04.031 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:04.031 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:04.031 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:04.031 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.031 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.031 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62849 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 208896~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:04.031 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.144181+0000", 2024-09-21T15:51:04.031 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 86.881225845000003, 2024-09-21T15:51:04.032 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050460390000000001, 2024-09-21T15:51:04.032 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:04.032 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:04.032 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:04.032 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:04.032 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:04.032 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62849 2024-09-21T15:51:04.032 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.032 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:04.032 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.032 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:04.032 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144181+0000", 2024-09-21T15:51:04.032 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:04.032 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.032 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.032 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:04.033 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144181+0000", 2024-09-21T15:51:04.033 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:04.033 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.033 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.033 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:04.033 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144181+0000", 2024-09-21T15:51:04.033 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:04.033 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.033 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.033 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:04.033 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144183+0000", 2024-09-21T15:51:04.033 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3860000000000001e-06 2024-09-21T15:51:04.033 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.033 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.033 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:04.033 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144189+0000", 2024-09-21T15:51:04.034 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0599999999999996e-06 2024-09-21T15:51:04.034 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.034 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:04.034 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144194+0000", 2024-09-21T15:51:04.034 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4379999999999997e-06 2024-09-21T15:51:04.034 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.034 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:04.034 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.194457+0000", 2024-09-21T15:51:04.034 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050263317000000002 2024-09-21T15:51:04.034 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.034 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:04.034 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.194548+0000", 2024-09-21T15:51:04.034 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0785000000000006e-05 2024-09-21T15:51:04.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.035 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:04.035 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.194642+0000", 2024-09-21T15:51:04.035 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3837000000000003e-05 2024-09-21T15:51:04.035 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:04.035 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:04.035 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:04.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.035 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62862 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 954368~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:04.035 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.146176+0000", 2024-09-21T15:51:04.035 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 86.879230781999993, 2024-09-21T15:51:04.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050442901999999998, 2024-09-21T15:51:04.036 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:04.036 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:04.036 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:04.036 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:04.036 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:04.036 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62862 2024-09-21T15:51:04.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.036 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:04.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.036 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:04.036 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146176+0000", 2024-09-21T15:51:04.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:04.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.037 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:04.037 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146176+0000", 2024-09-21T15:51:04.037 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:04.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.037 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:04.037 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146176+0000", 2024-09-21T15:51:04.037 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:04.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.037 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:04.037 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146178+0000", 2024-09-21T15:51:04.037 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4739999999999999e-06 2024-09-21T15:51:04.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.037 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:04.038 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146186+0000", 2024-09-21T15:51:04.038 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4989999999999997e-06 2024-09-21T15:51:04.038 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.038 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.038 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:04.038 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146191+0000", 2024-09-21T15:51:04.038 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7999999999999998e-06 2024-09-21T15:51:04.038 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.038 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.038 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:04.038 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196442+0000", 2024-09-21T15:51:04.038 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050251665000000001 2024-09-21T15:51:04.038 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.038 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.038 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:04.039 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196518+0000", 2024-09-21T15:51:04.039 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5677000000000001e-05 2024-09-21T15:51:04.039 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.039 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.039 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:04.039 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196619+0000", 2024-09-21T15:51:04.039 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101247 2024-09-21T15:51:04.039 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:04.039 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:04.039 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:04.039 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.039 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.039 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62863 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 937984~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:04.039 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.146195+0000", 2024-09-21T15:51:04.040 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 86.879212590999998, 2024-09-21T15:51:04.040 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050642305999999998, 2024-09-21T15:51:04.040 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:04.040 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:04.040 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:04.040 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:04.040 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:04.040 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62863 2024-09-21T15:51:04.040 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.040 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:04.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.040 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:04.040 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146195+0000", 2024-09-21T15:51:04.040 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:04.040 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.041 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:04.041 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146195+0000", 2024-09-21T15:51:04.041 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:04.041 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.041 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.041 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:04.041 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146194+0000", 2024-09-21T15:51:04.041 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:04.041 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.041 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.041 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:04.041 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146196+0000", 2024-09-21T15:51:04.041 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1900000000000002e-06 2024-09-21T15:51:04.041 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.042 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.042 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:04.042 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146202+0000", 2024-09-21T15:51:04.042 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.57e-06 2024-09-21T15:51:04.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.042 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.042 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:04.042 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146206+0000", 2024-09-21T15:51:04.042 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5079999999999999e-06 2024-09-21T15:51:04.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.042 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.042 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:04.042 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196655+0000", 2024-09-21T15:51:04.043 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050448973000000001 2024-09-21T15:51:04.043 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.043 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.043 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:04.043 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196730+0000", 2024-09-21T15:51:04.043 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4728999999999996e-05 2024-09-21T15:51:04.043 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.043 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.043 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:04.043 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196837+0000", 2024-09-21T15:51:04.043 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000106735 2024-09-21T15:51:04.043 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:04.043 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:04.043 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:04.043 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.043 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.044 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62872 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 290816~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:04.044 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148149+0000", 2024-09-21T15:51:04.044 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 86.877257915000001, 2024-09-21T15:51:04.044 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.05046991, 2024-09-21T15:51:04.044 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:04.044 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:04.044 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:04.044 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:04.044 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:04.044 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62872 2024-09-21T15:51:04.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.044 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:04.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.044 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:04.044 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148149+0000", 2024-09-21T15:51:04.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:04.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:04.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148149+0000", 2024-09-21T15:51:04.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:04.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:04.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148149+0000", 2024-09-21T15:51:04.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:04.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:04.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148157+0000", 2024-09-21T15:51:04.046 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4749999999999993e-06 2024-09-21T15:51:04.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.046 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:04.046 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148163+0000", 2024-09-21T15:51:04.046 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8250000000000001e-06 2024-09-21T15:51:04.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.046 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:04.046 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148168+0000", 2024-09-21T15:51:04.046 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4730000000000002e-06 2024-09-21T15:51:04.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.046 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:04.046 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198471+0000", 2024-09-21T15:51:04.047 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050302974 2024-09-21T15:51:04.047 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.047 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.047 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:04.047 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198555+0000", 2024-09-21T15:51:04.047 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4783999999999993e-05 2024-09-21T15:51:04.047 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.047 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.047 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:04.047 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198619+0000", 2024-09-21T15:51:04.047 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3751000000000007e-05 2024-09-21T15:51:04.047 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:04.047 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:04.047 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:04.048 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.048 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.048 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62873 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 315392~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:04.048 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148172+0000", 2024-09-21T15:51:04.048 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 86.877235623999994, 2024-09-21T15:51:04.048 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050707998999999997, 2024-09-21T15:51:04.048 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:04.048 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:04.048 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:04.048 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:04.048 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:04.048 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62873 2024-09-21T15:51:04.048 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.048 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:04.048 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.049 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:04.049 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148172+0000", 2024-09-21T15:51:04.049 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:04.049 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.049 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.049 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:04.049 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148172+0000", 2024-09-21T15:51:04.049 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:04.049 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.049 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.049 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:04.049 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148171+0000", 2024-09-21T15:51:04.049 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:04.049 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.049 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.050 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:04.050 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148173+0000", 2024-09-21T15:51:04.050 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2340000000000001e-06 2024-09-21T15:51:04.050 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.050 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.050 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:04.050 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148179+0000", 2024-09-21T15:51:04.050 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9340000000000003e-06 2024-09-21T15:51:04.050 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.050 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.050 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:04.050 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148186+0000", 2024-09-21T15:51:04.050 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2349999999999996e-06 2024-09-21T15:51:04.050 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.050 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.050 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:04.051 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198684+0000", 2024-09-21T15:51:04.051 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050498890999999997 2024-09-21T15:51:04.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.051 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.051 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:04.051 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198778+0000", 2024-09-21T15:51:04.051 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3134999999999995e-05 2024-09-21T15:51:04.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.051 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.051 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:04.051 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198880+0000", 2024-09-21T15:51:04.051 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101906 2024-09-21T15:51:04.051 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:04.051 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:04.051 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:04.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.052 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.052 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62874 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 323584~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:04.052 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148211+0000", 2024-09-21T15:51:04.052 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 86.877195864000001, 2024-09-21T15:51:04.052 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050887766000000001, 2024-09-21T15:51:04.052 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:04.052 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:04.052 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:04.052 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:04.052 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:04.052 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62874 2024-09-21T15:51:04.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.052 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:04.052 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.052 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:04.052 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148211+0000", 2024-09-21T15:51:04.052 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:04.053 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.053 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.053 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:04.053 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148211+0000", 2024-09-21T15:51:04.053 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:04.053 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.053 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.053 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:04.053 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148211+0000", 2024-09-21T15:51:04.053 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:04.053 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.053 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.053 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:04.053 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148213+0000", 2024-09-21T15:51:04.053 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6860000000000002e-06 2024-09-21T15:51:04.054 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.054 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.054 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:04.054 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148220+0000", 2024-09-21T15:51:04.054 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4740000000000002e-06 2024-09-21T15:51:04.054 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.054 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.054 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:04.054 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148226+0000", 2024-09-21T15:51:04.054 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0120000000000002e-06 2024-09-21T15:51:04.054 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.054 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.054 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:04.054 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198940+0000", 2024-09-21T15:51:04.055 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050714412 2024-09-21T15:51:04.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.055 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.055 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:04.055 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198995+0000", 2024-09-21T15:51:04.055 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4863e-05 2024-09-21T15:51:04.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.055 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.055 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:04.055 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.199099+0000", 2024-09-21T15:51:04.055 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010383299999999999 2024-09-21T15:51:04.055 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:04.055 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:04.055 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:04.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.056 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.056 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62877 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 995328~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:04.056 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148230+0000", 2024-09-21T15:51:04.056 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 86.877177004000004, 2024-09-21T15:51:04.056 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.051108949000000001, 2024-09-21T15:51:04.056 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:04.056 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:04.056 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:04.056 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:04.056 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:04.056 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62877 2024-09-21T15:51:04.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.056 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:04.056 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.057 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:04.057 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148230+0000", 2024-09-21T15:51:04.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:04.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.057 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:04.057 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148230+0000", 2024-09-21T15:51:04.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:04.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.057 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:04.057 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148230+0000", 2024-09-21T15:51:04.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:04.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.058 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:04.058 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148232+0000", 2024-09-21T15:51:04.058 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1220000000000002e-06 2024-09-21T15:51:04.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.058 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:04.058 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148238+0000", 2024-09-21T15:51:04.058 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.789e-06 2024-09-21T15:51:04.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.058 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:04.058 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148242+0000", 2024-09-21T15:51:04.058 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7369999999999997e-06 2024-09-21T15:51:04.059 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.059 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.059 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:04.059 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.199201+0000", 2024-09-21T15:51:04.059 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050959088 2024-09-21T15:51:04.059 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.059 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.059 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:04.059 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.199285+0000", 2024-09-21T15:51:04.059 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3147999999999997e-05 2024-09-21T15:51:04.059 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.059 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.059 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:04.059 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.199339+0000", 2024-09-21T15:51:04.060 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4484999999999998e-05 2024-09-21T15:51:04.060 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:04.060 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:04.060 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:04.060 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.060 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.060 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62888 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 331776~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:04.060 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150154+0000", 2024-09-21T15:51:04.060 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 86.875253409999999, 2024-09-21T15:51:04.060 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050483169000000001, 2024-09-21T15:51:04.060 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:04.060 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:04.060 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:04.060 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:04.060 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:04.060 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62888 2024-09-21T15:51:04.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.061 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:04.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.061 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:04.061 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150154+0000", 2024-09-21T15:51:04.061 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:04.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.061 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:04.061 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150154+0000", 2024-09-21T15:51:04.061 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:04.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.061 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:04.062 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150153+0000", 2024-09-21T15:51:04.062 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:04.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.062 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:04.062 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150155+0000", 2024-09-21T15:51:04.062 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9920000000000002e-06 2024-09-21T15:51:04.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.062 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:04.062 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150161+0000", 2024-09-21T15:51:04.062 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5439999999999998e-06 2024-09-21T15:51:04.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.063 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.063 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:04.063 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150165+0000", 2024-09-21T15:51:04.063 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3460000000000004e-06 2024-09-21T15:51:04.063 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.063 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.063 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:04.063 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200502+0000", 2024-09-21T15:51:04.063 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050337150999999997 2024-09-21T15:51:04.063 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.063 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.063 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:04.063 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200575+0000", 2024-09-21T15:51:04.063 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2250999999999997e-05 2024-09-21T15:51:04.063 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.064 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.064 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:04.064 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200637+0000", 2024-09-21T15:51:04.064 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2195999999999999e-05 2024-09-21T15:51:04.064 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:04.064 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:04.064 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:04.064 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.064 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.064 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62889 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 339968~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:04.064 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150169+0000", 2024-09-21T15:51:04.064 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 86.875237910999999, 2024-09-21T15:51:04.064 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050629678999999997, 2024-09-21T15:51:04.064 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:04.064 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:04.065 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:04.065 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:04.065 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:04.065 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62889 2024-09-21T15:51:04.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.065 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:04.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.065 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:04.065 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150169+0000", 2024-09-21T15:51:04.065 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:04.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.065 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:04.065 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150169+0000", 2024-09-21T15:51:04.065 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:04.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.066 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.066 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:04.066 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150169+0000", 2024-09-21T15:51:04.066 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:04.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.066 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.066 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:04.066 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150171+0000", 2024-09-21T15:51:04.066 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3910000000000001e-06 2024-09-21T15:51:04.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.066 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.066 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:04.067 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150177+0000", 2024-09-21T15:51:04.067 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4079999999999997e-06 2024-09-21T15:51:04.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.067 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:04.067 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150182+0000", 2024-09-21T15:51:04.067 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3890000000000004e-06 2024-09-21T15:51:04.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.067 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:04.067 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200664+0000", 2024-09-21T15:51:04.067 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050482212999999998 2024-09-21T15:51:04.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.067 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:04.068 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200738+0000", 2024-09-21T15:51:04.068 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4114000000000002e-05 2024-09-21T15:51:04.068 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.068 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.068 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:04.068 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200799+0000", 2024-09-21T15:51:04.068 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0503e-05 2024-09-21T15:51:04.068 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:04.068 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:04.068 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:04.068 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.068 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.068 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62891 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 348160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:04.068 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150186+0000", 2024-09-21T15:51:04.069 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 86.875220768999995, 2024-09-21T15:51:04.069 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050794552, 2024-09-21T15:51:04.069 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:04.069 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:04.069 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:04.069 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:04.069 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:04.069 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62891 2024-09-21T15:51:04.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.069 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:04.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.069 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:04.069 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150186+0000", 2024-09-21T15:51:04.069 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:04.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.069 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:04.070 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150186+0000", 2024-09-21T15:51:04.070 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:04.070 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.070 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:04.070 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150186+0000", 2024-09-21T15:51:04.070 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:04.070 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.070 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:04.070 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150193+0000", 2024-09-21T15:51:04.070 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6830000000000002e-06 2024-09-21T15:51:04.070 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.070 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:04.071 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150199+0000", 2024-09-21T15:51:04.071 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9859999999999999e-06 2024-09-21T15:51:04.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.071 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:04.071 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150203+0000", 2024-09-21T15:51:04.071 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5920000000000002e-06 2024-09-21T15:51:04.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.071 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:04.071 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200835+0000", 2024-09-21T15:51:04.071 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050631783 2024-09-21T15:51:04.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.072 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.072 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:04.072 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200894+0000", 2024-09-21T15:51:04.072 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8560999999999997e-05 2024-09-21T15:51:04.072 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.072 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.072 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:04.072 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200981+0000", 2024-09-21T15:51:04.072 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7305999999999994e-05 2024-09-21T15:51:04.072 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:04.072 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:04.072 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:04.072 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.072 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.072 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62892 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 356352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:04.073 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150208+0000", 2024-09-21T15:51:04.073 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 86.875199381000002, 2024-09-21T15:51:04.073 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050951769000000001, 2024-09-21T15:51:04.073 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:04.073 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:04.073 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:04.073 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:04.073 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:04.073 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62892 2024-09-21T15:51:04.073 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.073 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:04.073 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.073 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:04.073 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150208+0000", 2024-09-21T15:51:04.073 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:04.074 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.074 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.074 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:04.074 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150208+0000", 2024-09-21T15:51:04.074 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:04.074 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.074 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.074 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:04.074 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150207+0000", 2024-09-21T15:51:04.074 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:04.074 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.074 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.074 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:04.074 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150210+0000", 2024-09-21T15:51:04.074 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.238e-06 2024-09-21T15:51:04.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:04.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150215+0000", 2024-09-21T15:51:04.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6350000000000003e-06 2024-09-21T15:51:04.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:04.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150220+0000", 2024-09-21T15:51:04.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.544e-06 2024-09-21T15:51:04.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:04.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201014+0000", 2024-09-21T15:51:04.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050793932999999999 2024-09-21T15:51:04.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.076 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:04.076 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201092+0000", 2024-09-21T15:51:04.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7961999999999997e-05 2024-09-21T15:51:04.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.076 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:04.076 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201160+0000", 2024-09-21T15:51:04.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7836000000000006e-05 2024-09-21T15:51:04.076 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:04.076 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:04.076 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:04.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.077 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.077 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62894 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 364544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:04.077 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150820+0000", 2024-09-21T15:51:04.077 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 86.874587180999995, 2024-09-21T15:51:04.077 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050489196, 2024-09-21T15:51:04.077 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:04.077 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:04.077 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:04.077 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:04.077 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:04.077 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62894 2024-09-21T15:51:04.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.077 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:04.077 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.077 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:04.077 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150820+0000", 2024-09-21T15:51:04.078 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:04.078 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.078 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:04.078 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150820+0000", 2024-09-21T15:51:04.078 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:04.078 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.078 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:04.078 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150223+0000", 2024-09-21T15:51:04.078 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9994035 2024-09-21T15:51:04.078 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.078 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:04.078 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150825+0000", 2024-09-21T15:51:04.079 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00060137099999999996 2024-09-21T15:51:04.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.079 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.079 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:04.079 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150837+0000", 2024-09-21T15:51:04.079 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2503999999999999e-05 2024-09-21T15:51:04.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.079 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.079 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:04.079 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150844+0000", 2024-09-21T15:51:04.079 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6900000000000003e-06 2024-09-21T15:51:04.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.079 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.079 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:04.079 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201187+0000", 2024-09-21T15:51:04.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050342973999999999 2024-09-21T15:51:04.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.080 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:04.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201250+0000", 2024-09-21T15:51:04.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3274000000000004e-05 2024-09-21T15:51:04.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.080 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:04.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201309+0000", 2024-09-21T15:51:04.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9058000000000003e-05 2024-09-21T15:51:04.080 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:04.080 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:04.080 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:04.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.081 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62895 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 380928~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:04.081 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150851+0000", 2024-09-21T15:51:04.081 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 86.874556584999993, 2024-09-21T15:51:04.081 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050611097000000001, 2024-09-21T15:51:04.081 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:04.081 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:04.081 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:04.081 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:04.081 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:04.081 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62895 2024-09-21T15:51:04.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.081 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:04.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.081 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:04.081 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150851+0000", 2024-09-21T15:51:04.082 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:04.082 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.082 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.082 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:04.082 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150851+0000", 2024-09-21T15:51:04.082 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:04.082 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.082 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.082 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:04.082 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150850+0000", 2024-09-21T15:51:04.082 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:04.082 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.082 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.083 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:04.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150853+0000", 2024-09-21T15:51:04.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1039999999999998e-06 2024-09-21T15:51:04.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.083 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:04.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150860+0000", 2024-09-21T15:51:04.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3180000000000001e-06 2024-09-21T15:51:04.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.083 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:04.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150866+0000", 2024-09-21T15:51:04.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1599999999999997e-06 2024-09-21T15:51:04.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.084 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:04.084 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201332+0000", 2024-09-21T15:51:04.084 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050466100999999999 2024-09-21T15:51:04.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.084 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:04.084 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201392+0000", 2024-09-21T15:51:04.084 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0053999999999998e-05 2024-09-21T15:51:04.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.084 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:04.084 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201462+0000", 2024-09-21T15:51:04.084 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9861000000000001e-05 2024-09-21T15:51:04.084 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:04.085 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:04.085 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:04.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.085 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.085 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62896 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 372736~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:04.085 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150870+0000", 2024-09-21T15:51:04.085 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 86.874536805000005, 2024-09-21T15:51:04.085 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050753292999999998, 2024-09-21T15:51:04.085 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:04.085 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:04.085 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:04.085 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:04.086 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:04.086 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62896 2024-09-21T15:51:04.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.086 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:04.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.086 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:04.086 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150870+0000", 2024-09-21T15:51:04.086 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:04.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.086 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:04.086 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150870+0000", 2024-09-21T15:51:04.086 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:04.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.087 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.087 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:04.087 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150870+0000", 2024-09-21T15:51:04.087 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:04.087 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.087 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.087 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:04.087 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150872+0000", 2024-09-21T15:51:04.087 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.644e-06 2024-09-21T15:51:04.087 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.087 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.087 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:04.087 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150879+0000", 2024-09-21T15:51:04.087 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3509999999999998e-06 2024-09-21T15:51:04.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.088 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:04.088 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150884+0000", 2024-09-21T15:51:04.088 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9150000000000004e-06 2024-09-21T15:51:04.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.088 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:04.088 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201487+0000", 2024-09-21T15:51:04.088 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050603276000000003 2024-09-21T15:51:04.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.088 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:04.088 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201557+0000", 2024-09-21T15:51:04.089 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9847999999999999e-05 2024-09-21T15:51:04.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.089 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.089 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:04.089 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201624+0000", 2024-09-21T15:51:04.089 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6730999999999995e-05 2024-09-21T15:51:04.089 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:04.089 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:04.089 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:04.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.089 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.089 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62897 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 389120~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:04.089 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.151044+0000", 2024-09-21T15:51:04.089 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 86.874362765000001, 2024-09-21T15:51:04.089 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050720531999999999, 2024-09-21T15:51:04.090 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:04.090 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:04.090 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:04.090 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:04.090 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:04.090 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62897 2024-09-21T15:51:04.090 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.090 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:04.090 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.090 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:04.090 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151044+0000", 2024-09-21T15:51:04.090 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:04.090 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.090 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.091 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:04.091 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151044+0000", 2024-09-21T15:51:04.091 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:04.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.091 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:04.091 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150888+0000", 2024-09-21T15:51:04.091 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9998431 2024-09-21T15:51:04.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.091 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:04.091 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151048+0000", 2024-09-21T15:51:04.091 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00016077299999999999 2024-09-21T15:51:04.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.091 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:04.092 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151058+0000", 2024-09-21T15:51:04.092 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.465e-06 2024-09-21T15:51:04.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.092 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:04.092 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151064+0000", 2024-09-21T15:51:04.092 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3829999999999997e-06 2024-09-21T15:51:04.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.092 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:04.092 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201648+0000", 2024-09-21T15:51:04.092 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050583597000000001 2024-09-21T15:51:04.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.093 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:04.093 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201711+0000", 2024-09-21T15:51:04.093 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3616999999999998e-05 2024-09-21T15:51:04.093 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.093 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.093 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:04.093 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201765+0000", 2024-09-21T15:51:04.093 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3464000000000002e-05 2024-09-21T15:51:04.093 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:04.093 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:04.093 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:04.093 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.093 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.093 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:118351 3.4 3:25980f86:::rbd_data.1133545d3fa6.000000000000005a:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-09-21T15:51:04.093 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:58.975795+0000", 2024-09-21T15:51:04.093 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 65.049612131000004, 2024-09-21T15:51:04.094 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.99016351599999997, 2024-09-21T15:51:04.094 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:04.094 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:04.094 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:04.094 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:04.094 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:04.094 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 118351 2024-09-21T15:51:04.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.094 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:04.094 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.094 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:04.094 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975795+0000", 2024-09-21T15:51:04.094 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:04.095 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.095 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:04.095 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975795+0000", 2024-09-21T15:51:04.095 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:04.095 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.095 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:04.095 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975793+0000", 2024-09-21T15:51:04.095 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:51:04.095 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.095 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:04.095 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975800+0000", 2024-09-21T15:51:04.095 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8220000000000001e-06 2024-09-21T15:51:04.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:04.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975814+0000", 2024-09-21T15:51:04.096 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3893000000000001e-05 2024-09-21T15:51:04.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:04.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975821+0000", 2024-09-21T15:51:04.096 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1319999999999996e-06 2024-09-21T15:51:04.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:04.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975878+0000", 2024-09-21T15:51:04.096 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7238999999999999e-05 2024-09-21T15:51:04.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.097 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:51:04.097 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975889+0000", 2024-09-21T15:51:04.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0509000000000001e-05 2024-09-21T15:51:04.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.097 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:04.097 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.964573+0000", 2024-09-21T15:51:04.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.98868396300000005 2024-09-21T15:51:04.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.097 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:04.097 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.964672+0000", 2024-09-21T15:51:04.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9601999999999999e-05 2024-09-21T15:51:04.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-09-21T15:51:04.098 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.964758+0000", 2024-09-21T15:51:04.098 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5537000000000005e-05 2024-09-21T15:51:04.098 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:04.098 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965847+0000", 2024-09-21T15:51:04.098 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001089404 2024-09-21T15:51:04.098 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:04.098 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965890+0000", 2024-09-21T15:51:04.098 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2521999999999997e-05 2024-09-21T15:51:04.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.099 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:04.099 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965934+0000", 2024-09-21T15:51:04.099 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4489e-05 2024-09-21T15:51:04.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.099 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:04.099 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965959+0000", 2024-09-21T15:51:04.099 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4301999999999999e-05 2024-09-21T15:51:04.099 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:04.099 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:04.099 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:04.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.100 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:118353 3.4 3:23841e18:::rbd_data.1133545d3fa6.0000000000000061:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-09-21T15:51:04.100 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:58.975836+0000", 2024-09-21T15:51:04.100 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 65.049570631999998, 2024-09-21T15:51:04.100 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.99077746700000002, 2024-09-21T15:51:04.100 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:04.100 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:04.100 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:04.100 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:04.100 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:04.100 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 118353 2024-09-21T15:51:04.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.100 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:04.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.100 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:04.100 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975836+0000", 2024-09-21T15:51:04.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:04.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.101 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:04.101 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975836+0000", 2024-09-21T15:51:04.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:04.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.101 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:04.101 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975836+0000", 2024-09-21T15:51:04.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:04.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.101 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:04.101 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975839+0000", 2024-09-21T15:51:04.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6129999999999998e-06 2024-09-21T15:51:04.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.102 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:04.102 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975844+0000", 2024-09-21T15:51:04.102 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5130000000000004e-06 2024-09-21T15:51:04.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.102 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:04.102 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975850+0000", 2024-09-21T15:51:04.102 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3179999999999997e-06 2024-09-21T15:51:04.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.102 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:04.102 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975915+0000", 2024-09-21T15:51:04.103 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5168999999999999e-05 2024-09-21T15:51:04.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.103 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:51:04.103 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975922+0000", 2024-09-21T15:51:04.103 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2749999999999998e-06 2024-09-21T15:51:04.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.103 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:04.103 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965431+0000", 2024-09-21T15:51:04.103 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.98950864699999996 2024-09-21T15:51:04.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.104 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:04.104 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965535+0000", 2024-09-21T15:51:04.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000103949 2024-09-21T15:51:04.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.104 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-09-21T15:51:04.104 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965607+0000", 2024-09-21T15:51:04.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2478000000000006e-05 2024-09-21T15:51:04.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.104 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:04.104 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.966205+0000", 2024-09-21T15:51:04.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00059775200000000001 2024-09-21T15:51:04.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.105 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:04.105 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.966562+0000", 2024-09-21T15:51:04.105 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00035732199999999998 2024-09-21T15:51:04.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.105 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:04.105 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.966593+0000", 2024-09-21T15:51:04.105 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0642999999999998e-05 2024-09-21T15:51:04.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.105 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:04.105 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.966614+0000", 2024-09-21T15:51:04.105 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1149999999999999e-05 2024-09-21T15:51:04.105 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:04.106 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:04.106 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:04.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.106 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160535 3.6 3:67cae2ed:::rbd_mirroring:head [list-watchers] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-09-21T15:51:04.106 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:30.195408+0000", 2024-09-21T15:51:04.106 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 33.829999549999997, 2024-09-21T15:51:04.106 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.96025944900000004, 2024-09-21T15:51:04.106 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:04.106 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-09-21T15:51:04.106 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:04.106 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:04.106 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:04.106 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160535 2024-09-21T15:51:04.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.107 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:04.107 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.107 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:04.107 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195408+0000", 2024-09-21T15:51:04.107 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:04.107 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.107 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.107 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:04.107 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195408+0000", 2024-09-21T15:51:04.107 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:04.107 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.107 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.107 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:04.107 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195406+0000", 2024-09-21T15:51:04.107 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:51:04.108 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.108 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.108 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:04.108 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195414+0000", 2024-09-21T15:51:04.108 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.2870000000000004e-06 2024-09-21T15:51:04.108 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.108 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.108 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:04.108 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195432+0000", 2024-09-21T15:51:04.108 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8110000000000001e-05 2024-09-21T15:51:04.108 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.108 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.108 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:04.108 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195441+0000", 2024-09-21T15:51:04.108 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7679999999999997e-06 2024-09-21T15:51:04.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.109 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:04.109 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195505+0000", 2024-09-21T15:51:04.109 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3769000000000006e-05 2024-09-21T15:51:04.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.109 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:51:04.109 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195518+0000", 2024-09-21T15:51:04.109 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3341e-05 2024-09-21T15:51:04.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.109 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:04.109 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.155509+0000", 2024-09-21T15:51:04.109 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.959990393 2024-09-21T15:51:04.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.110 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.110 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:04.110 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.155667+0000", 2024-09-21T15:51:04.110 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000158505 2024-09-21T15:51:04.110 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:04.110 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:04.110 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:04.110 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.110 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.110 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160974 3.4 3:22dd2475:::rbd_id.image_client.0-clone39:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-21T15:51:04.110 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:34.748093+0000", 2024-09-21T15:51:04.110 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 29.277313999, 2024-09-21T15:51:04.110 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.94006199499999998, 2024-09-21T15:51:04.110 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:04.111 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:04.111 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:04.111 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:04.111 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:04.111 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160974 2024-09-21T15:51:04.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.111 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:04.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.111 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:04.111 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748093+0000", 2024-09-21T15:51:04.111 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:04.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.111 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:04.111 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748093+0000", 2024-09-21T15:51:04.111 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:04.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.112 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:04.112 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748091+0000", 2024-09-21T15:51:04.112 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:51:04.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.112 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:04.112 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748101+0000", 2024-09-21T15:51:04.112 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5950000000000005e-06 2024-09-21T15:51:04.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.112 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:04.112 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748122+0000", 2024-09-21T15:51:04.112 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1611e-05 2024-09-21T15:51:04.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.113 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:04.113 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748133+0000", 2024-09-21T15:51:04.113 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0224000000000001e-05 2024-09-21T15:51:04.113 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.113 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:04.113 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748185+0000", 2024-09-21T15:51:04.113 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2005000000000001e-05 2024-09-21T15:51:04.113 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.113 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:51:04.113 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748194+0000", 2024-09-21T15:51:04.113 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0180000000000007e-06 2024-09-21T15:51:04.113 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.114 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:04.114 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.683582+0000", 2024-09-21T15:51:04.114 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.93538800499999997 2024-09-21T15:51:04.114 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.114 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:04.114 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.683711+0000", 2024-09-21T15:51:04.114 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000129746 2024-09-21T15:51:04.114 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.114 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-09-21T15:51:04.114 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.683838+0000", 2024-09-21T15:51:04.114 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012693200000000001 2024-09-21T15:51:04.114 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:04.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.687445+0000", 2024-09-21T15:51:04.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0036071340000000001 2024-09-21T15:51:04.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:04.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.688064+0000", 2024-09-21T15:51:04.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00061857100000000005 2024-09-21T15:51:04.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:04.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.688113+0000", 2024-09-21T15:51:04.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9128000000000001e-05 2024-09-21T15:51:04.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:04.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:04.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:04.116 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.688155+0000", 2024-09-21T15:51:04.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2026e-05 2024-09-21T15:51:04.116 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:04.116 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:04.116 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:04.116 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:04.116 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:04.116 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:04.117 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:51:04.118 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:51:04.116+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:51:04.218 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:51:04.216+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:51:04.258 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:04.259 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:51:04.259 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:51:04.259 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:04.267 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:51:04.318 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:51:04.318+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:51:04.335 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:51:04.370 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:04.371 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:51:04.371 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:51:04.371 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:51:04.371 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:04.380 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:51:04.419 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:51:04.419+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:51:04.457 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-09-21T15:51:04.457 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:51:04.474 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:51:04.474 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:51:04.489 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:51:04.489 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:51:04.505 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:51:04.506 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:51:04.520 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:51:04.519+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:51:04.522 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:51:04.523 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:51:04.526 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:04.527 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:04.536 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:51:04.539 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:51:04.540 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:51:04.556 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:51:04.557 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:51:04.573 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:51:04.574 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:51:04.620 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:51:04.619+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:51:04.643 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:04.643 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:04.652 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:51:04.721 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:51:04.720+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:51:04.753 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:04.753 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:04.762 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:51:04.795 INFO:teuthology.orchestra.run.smithi089.stderr:reweighted osd.6 to 1 (10000) 2024-09-21T15:51:04.811 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-09-21T15:51:04.852 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:51:04.851+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:51:04.872 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:04.872 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:04.882 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:51:04.926 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:51:04.925+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:51:04.999 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:05.000 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:05.008 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:51:05.049 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:51:05.048+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:51:05.125 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:51:05.123+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:51:05.151 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:05.151 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:05.160 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:51:05.225 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:51:05.225+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:51:05.261 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:05.261 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:05.270 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:51:05.326 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:51:05.325+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:51:05.372 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:05.372 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:05.381 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:51:05.428 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:51:05.427+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:51:05.523 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:05.523 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:05.528 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:51:05.527+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:51:05.531 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:51:05.629 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:51:05.628+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:51:05.675 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:05.675 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:05.684 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T15:51:05.768 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:51:05.767+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:51:05.809 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T15:51:05.811 INFO:teuthology.orchestra.run.smithi089.stderr:reweighted osd.7 to 1 (10000) 2024-09-21T15:51:05.825 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-21T15:51:05.825 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T15:51:05.833 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:05.833 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:05.843 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T15:51:05.866 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:51:05.865+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:51:05.931 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:51:05.930+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:51:05.995 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:05.995 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:06.004 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T15:51:06.045 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:51:06.044+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:51:06.133 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:51:06.132+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:51:06.145 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:06.145 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:06.155 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T15:51:06.155 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:51:06.156 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-09-21T15:51:06.166 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":191,"stamp":"2024-09-21T15:51:05.911542+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":400127206,"num_objects":1596,"num_object_clones":99,"num_object_copies":3192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1596,"num_whiteouts":1116,"num_read":203013,"num_read_kb":838657,"num_write":22619,"num_write_kb":305445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2186,"num_bytes_recovered":83031061,"num_keys_recovered":281,"num_objects_omap":19,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19150,"ondisk_log_size":19150,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":475288,"kb_used_data":132976,"kb_used_omap":22,"kb_used_meta":338025,"kb_avail":660127592,"statfs":{"total":676457349120,"available":675970654208,"internally_reserved":0,"allocated":136167424,"data_stored":147769283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23526,"internal_metadata":346137626},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-289564545,"num_objects":-178,"num_object_clones":-59,"num_object_copies":-356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-178,"num_whiteouts":-43,"num_read":1178,"num_read_kb":1015,"num_write":250,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000631"},"pg_stats":[{"pgid":"3.1f","version":"124'400","reported_seq":2000,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-21T15:51:02.816879+0000","last_change":"2024-09-21T15:51:02.817045+0000","last_active":"2024-09-21T15:51:02.816879+0000","last_peered":"2024-09-21T15:51:02.816879+0000","last_clean":"2024-09-21T15:51:02.816879+0000","last_became_active":"2024-09-21T15:50:32.670964+0000","last_became_peered":"2024-09-21T15:50:32.670964+0000","last_unstale":"2024-09-21T15:51:02.816879+0000","last_undegraded":"2024-09-21T15:51:02.816879+0000","last_fullsized":"2024-09-21T15:51:02.816879+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":400,"log_dups_size":0,"ondisk_log_size":400,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:09:09.918864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010020006,"stat_sum":{"num_bytes":4571152,"num_objects":41,"num_object_clones":1,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":29,"num_read":2410,"num_read_kb":10418,"num_write":431,"num_write_kb":5956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":1974433,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"16","length":"7"}]},{"pgid":"3.1e","version":"129'550","reported_seq":6021,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-21T15:51:04.925611+0000","last_change":"2024-09-21T15:51:04.808889+0000","last_active":"2024-09-21T15:51:04.925611+0000","last_peered":"2024-09-21T15:51:04.925611+0000","last_clean":"2024-09-21T15:51:04.925611+0000","last_became_active":"2024-09-21T15:50:57.685368+0000","last_became_peered":"2024-09-21T15:50:57.685368+0000","last_unstale":"2024-09-21T15:51:04.925611+0000","last_undegraded":"2024-09-21T15:51:04.925611+0000","last_fullsized":"2024-09-21T15:51:04.925611+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":550,"log_dups_size":0,"ondisk_log_size":550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:23:42.939446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054089100000000005,"stat_sum":{"num_bytes":18810510,"num_objects":55,"num_object_clones":3,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":41,"num_read":7252,"num_read_kb":25737,"num_write":540,"num_write_kb":4350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":2730860,"num_keys_recovered":23,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"14","length":"9"}]},{"pgid":"3.1d","version":"129'596","reported_seq":9958,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-21T15:51:04.905563+0000","last_change":"2024-09-21T15:51:04.795258+0000","last_active":"2024-09-21T15:51:04.905563+0000","last_peered":"2024-09-21T15:51:04.905563+0000","last_clean":"2024-09-21T15:51:04.905563+0000","last_became_active":"2024-09-21T15:49:39.833206+0000","last_became_peered":"2024-09-21T15:49:39.833206+0000","last_unstale":"2024-09-21T15:51:04.905563+0000","last_undegraded":"2024-09-21T15:51:04.905563+0000","last_fullsized":"2024-09-21T15:51:04.905563+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":596,"log_dups_size":0,"ondisk_log_size":596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:36:14.082318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000386733,"stat_sum":{"num_bytes":32540173,"num_objects":59,"num_object_clones":6,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":38,"num_read":10487,"num_read_kb":28946,"num_write":659,"num_write_kb":8871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"14","length":"9"}]},{"pgid":"3.1c","version":"129'554","reported_seq":4313,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-21T15:51:04.870292+0000","last_change":"2024-09-21T15:51:04.795617+0000","last_active":"2024-09-21T15:51:04.870292+0000","last_peered":"2024-09-21T15:51:04.870292+0000","last_clean":"2024-09-21T15:51:04.870292+0000","last_became_active":"2024-09-21T15:49:39.833113+0000","last_became_peered":"2024-09-21T15:49:39.833113+0000","last_unstale":"2024-09-21T15:51:04.870292+0000","last_undegraded":"2024-09-21T15:51:04.870292+0000","last_fullsized":"2024-09-21T15:51:04.870292+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":554,"log_dups_size":0,"ondisk_log_size":554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:43:57.592707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064970700000000002,"stat_sum":{"num_bytes":8652416,"num_objects":42,"num_object_clones":1,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":28,"num_read":5136,"num_read_kb":17235,"num_write":600,"num_write_kb":4787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"14","length":"9"}]},{"pgid":"3.1b","version":"127'460","reported_seq":4159,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-21T15:51:02.814699+0000","last_change":"2024-09-21T15:51:02.815040+0000","last_active":"2024-09-21T15:51:02.814699+0000","last_peered":"2024-09-21T15:51:02.814699+0000","last_clean":"2024-09-21T15:51:02.814699+0000","last_became_active":"2024-09-21T15:50:32.674250+0000","last_became_peered":"2024-09-21T15:50:32.674250+0000","last_unstale":"2024-09-21T15:51:02.814699+0000","last_undegraded":"2024-09-21T15:51:02.814699+0000","last_fullsized":"2024-09-21T15:51:02.814699+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":460,"log_dups_size":0,"ondisk_log_size":460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:54:57.399818+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0082588109999999996,"stat_sum":{"num_bytes":6701956,"num_objects":57,"num_object_clones":1,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":40,"num_read":4618,"num_read_kb":29923,"num_write":458,"num_write_kb":13501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":4759000,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"16","length":"7"}]},{"pgid":"3.1a","version":"129'392","reported_seq":6587,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-21T15:51:04.897322+0000","last_change":"2024-09-21T15:51:04.795498+0000","last_active":"2024-09-21T15:51:04.897322+0000","last_peered":"2024-09-21T15:51:04.897322+0000","last_clean":"2024-09-21T15:51:04.897322+0000","last_became_active":"2024-09-21T15:49:06.693903+0000","last_became_peered":"2024-09-21T15:49:06.693903+0000","last_unstale":"2024-09-21T15:51:04.897322+0000","last_undegraded":"2024-09-21T15:51:04.897322+0000","last_fullsized":"2024-09-21T15:51:04.897322+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":392,"log_dups_size":0,"ondisk_log_size":392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:41:45.432893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076663199999999995,"stat_sum":{"num_bytes":15065327,"num_objects":49,"num_object_clones":2,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":32,"num_read":6890,"num_read_kb":31459,"num_write":391,"num_write_kb":9470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"14","length":"9"}]},{"pgid":"3.19","version":"129'442","reported_seq":8457,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-21T15:51:04.902788+0000","last_change":"2024-09-21T15:51:04.864682+0000","last_active":"2024-09-21T15:51:04.902788+0000","last_peered":"2024-09-21T15:51:04.902788+0000","last_clean":"2024-09-21T15:51:04.902788+0000","last_became_active":"2024-09-21T15:50:57.677431+0000","last_became_peered":"2024-09-21T15:50:57.677431+0000","last_unstale":"2024-09-21T15:51:04.902788+0000","last_undegraded":"2024-09-21T15:51:04.902788+0000","last_fullsized":"2024-09-21T15:51:04.902788+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":442,"log_dups_size":0,"ondisk_log_size":442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:53:19.596041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010939809999999999,"stat_sum":{"num_bytes":23099668,"num_objects":49,"num_object_clones":4,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":35,"num_read":8281,"num_read_kb":39578,"num_write":508,"num_write_kb":7408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":3488172,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"14","length":"9"}]},{"pgid":"3.18","version":"127'476","reported_seq":4093,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-21T15:51:02.812328+0000","last_change":"2024-09-21T15:51:02.812328+0000","last_active":"2024-09-21T15:51:02.812328+0000","last_peered":"2024-09-21T15:51:02.812328+0000","last_clean":"2024-09-21T15:51:02.812328+0000","last_became_active":"2024-09-21T15:50:57.675432+0000","last_became_peered":"2024-09-21T15:50:57.675432+0000","last_unstale":"2024-09-21T15:51:02.812328+0000","last_undegraded":"2024-09-21T15:51:02.812328+0000","last_fullsized":"2024-09-21T15:51:02.812328+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":476,"log_dups_size":0,"ondisk_log_size":476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:43:10.553541+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0057213999999999997,"stat_sum":{"num_bytes":6448637,"num_objects":45,"num_object_clones":2,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":28,"num_read":5684,"num_read_kb":30500,"num_write":487,"num_write_kb":11063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":10150338,"num_keys_recovered":46,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"16","length":"7"}]},{"pgid":"3.17","version":"115'996","reported_seq":9243,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-21T15:51:02.815096+0000","last_change":"2024-09-21T15:51:02.815471+0000","last_active":"2024-09-21T15:51:02.815096+0000","last_peered":"2024-09-21T15:51:02.815096+0000","last_clean":"2024-09-21T15:51:02.815096+0000","last_became_active":"2024-09-21T15:50:32.668771+0000","last_became_peered":"2024-09-21T15:50:32.668771+0000","last_unstale":"2024-09-21T15:51:02.815096+0000","last_undegraded":"2024-09-21T15:51:02.815096+0000","last_fullsized":"2024-09-21T15:51:02.815096+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":996,"log_dups_size":0,"ondisk_log_size":996,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:39:43.554150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086097599999999993,"stat_sum":{"num_bytes":6308777,"num_objects":59,"num_object_clones":5,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":40,"num_read":12662,"num_read_kb":21448,"num_write":1268,"num_write_kb":8252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":3319185,"num_keys_recovered":34,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"16","length":"7"}]},{"pgid":"3.16","version":"129'491","reported_seq":6459,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-21T15:51:04.815497+0000","last_change":"2024-09-21T15:51:04.815497+0000","last_active":"2024-09-21T15:51:04.815497+0000","last_peered":"2024-09-21T15:51:04.815497+0000","last_clean":"2024-09-21T15:51:04.815497+0000","last_became_active":"2024-09-21T15:49:06.688808+0000","last_became_peered":"2024-09-21T15:49:06.688808+0000","last_unstale":"2024-09-21T15:51:04.815497+0000","last_undegraded":"2024-09-21T15:51:04.815497+0000","last_fullsized":"2024-09-21T15:51:04.815497+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":491,"log_dups_size":0,"ondisk_log_size":491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:43:02.012254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000317251,"stat_sum":{"num_bytes":10165651,"num_objects":47,"num_object_clones":2,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":32,"num_read":7324,"num_read_kb":49627,"num_write":500,"num_write_kb":13812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"14","length":"9"}]},{"pgid":"3.15","version":"127'515","reported_seq":3963,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-21T15:51:04.894471+0000","last_change":"2024-09-21T15:51:04.894668+0000","last_active":"2024-09-21T15:51:04.894471+0000","last_peered":"2024-09-21T15:51:04.894471+0000","last_clean":"2024-09-21T15:51:04.894471+0000","last_became_active":"2024-09-21T15:50:57.680672+0000","last_became_peered":"2024-09-21T15:50:57.680672+0000","last_unstale":"2024-09-21T15:51:04.894471+0000","last_undegraded":"2024-09-21T15:51:04.894471+0000","last_fullsized":"2024-09-21T15:51:04.894471+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":515,"log_dups_size":0,"ondisk_log_size":515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:00:16.784121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030861170000000002,"stat_sum":{"num_bytes":12623976,"num_objects":44,"num_object_clones":3,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":32,"num_read":4052,"num_read_kb":27144,"num_write":628,"num_write_kb":17935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":2234893,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"14","length":"9"}]},{"pgid":"3.14","version":"129'1171","reported_seq":4230,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-21T15:51:04.902658+0000","last_change":"2024-09-21T15:51:04.795871+0000","last_active":"2024-09-21T15:51:04.902658+0000","last_peered":"2024-09-21T15:51:04.902658+0000","last_clean":"2024-09-21T15:51:04.902658+0000","last_became_active":"2024-09-21T15:49:06.688803+0000","last_became_peered":"2024-09-21T15:49:06.688803+0000","last_unstale":"2024-09-21T15:51:04.902658+0000","last_undegraded":"2024-09-21T15:51:04.902658+0000","last_fullsized":"2024-09-21T15:51:04.902658+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":1171,"log_dups_size":0,"ondisk_log_size":1171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:18:47.662631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096029700000000004,"stat_sum":{"num_bytes":5574397,"num_objects":39,"num_object_clones":2,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":27,"num_read":7697,"num_read_kb":18182,"num_write":1890,"num_write_kb":6180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"14","length":"9"}]},{"pgid":"2.7","version":"0'0","reported_seq":125,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-21T15:51:03.788040+0000","last_change":"2024-09-21T15:48:59.666431+0000","last_active":"2024-09-21T15:51:03.788040+0000","last_peered":"2024-09-21T15:51:03.788040+0000","last_clean":"2024-09-21T15:51:03.788040+0000","last_became_active":"2024-09-21T15:48:57.583665+0000","last_became_peered":"2024-09-21T15:48:57.583665+0000","last_unstale":"2024-09-21T15:51:03.788040+0000","last_undegraded":"2024-09-21T15:51:03.788040+0000","last_fullsized":"2024-09-21T15:51:03.788040+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:52:09.228964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031098399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"127'408","reported_seq":2917,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-21T15:51:02.865904+0000","last_change":"2024-09-21T15:51:02.816830+0000","last_active":"2024-09-21T15:51:02.865904+0000","last_peered":"2024-09-21T15:51:02.865904+0000","last_clean":"2024-09-21T15:51:02.865904+0000","last_became_active":"2024-09-21T15:50:32.671424+0000","last_became_peered":"2024-09-21T15:50:32.671424+0000","last_unstale":"2024-09-21T15:51:02.865904+0000","last_undegraded":"2024-09-21T15:51:02.865904+0000","last_fullsized":"2024-09-21T15:51:02.865904+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":408,"log_dups_size":0,"ondisk_log_size":408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:04:14.554732+0000","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0.010156573,"stat_sum":{"num_bytes":7990384,"num_objects":47,"num_object_clones":3,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":38,"num_read":2637,"num_read_kb":12956,"num_write":382,"num_write_kb":8392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":4494388,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"16","length":"7"}]},{"pgid":"2.6","version":"0'0","reported_seq":125,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-21T15:51:03.787702+0000","last_change":"2024-09-21T15:48:59.589790+0000","last_active":"2024-09-21T15:51:03.787702+0000","last_peered":"2024-09-21T15:51:03.787702+0000","last_clean":"2024-09-21T15:51:03.787702+0000","last_became_active":"2024-09-21T15:48:57.582121+0000","last_became_peered":"2024-09-21T15:48:57.582121+0000","last_unstale":"2024-09-21T15:51:03.787702+0000","last_undegraded":"2024-09-21T15:51:03.787702+0000","last_fullsized":"2024-09-21T15:51:03.787702+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:33:47.592624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039654700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"129'552","reported_seq":7603,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-21T15:51:04.895230+0000","last_change":"2024-09-21T15:51:04.895230+0000","last_active":"2024-09-21T15:51:04.895230+0000","last_peered":"2024-09-21T15:51:04.895230+0000","last_clean":"2024-09-21T15:51:04.895230+0000","last_became_active":"2024-09-21T15:50:57.680364+0000","last_became_peered":"2024-09-21T15:50:57.680364+0000","last_unstale":"2024-09-21T15:51:04.895230+0000","last_undegraded":"2024-09-21T15:51:04.895230+0000","last_fullsized":"2024-09-21T15:51:04.895230+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":552,"log_dups_size":0,"ondisk_log_size":552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:26:50.885554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003339421,"stat_sum":{"num_bytes":15685509,"num_objects":60,"num_object_clones":7,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":40,"num_read":8378,"num_read_kb":21194,"num_write":531,"num_write_kb":5397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":2019937,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"14","length":"9"}]},{"pgid":"2.5","version":"0'0","reported_seq":139,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-21T15:51:04.891749+0000","last_change":"2024-09-21T15:50:30.146743+0000","last_active":"2024-09-21T15:51:04.891749+0000","last_peered":"2024-09-21T15:51:04.891749+0000","last_clean":"2024-09-21T15:51:04.891749+0000","last_became_active":"2024-09-21T15:50:30.146428+0000","last_became_peered":"2024-09-21T15:50:30.146428+0000","last_unstale":"2024-09-21T15:51:04.891749+0000","last_undegraded":"2024-09-21T15:51:04.891749+0000","last_fullsized":"2024-09-21T15:51:04.891749+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:58:38.689807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036074600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"129'668","reported_seq":8597,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-21T15:51:04.864095+0000","last_change":"2024-09-21T15:51:04.864095+0000","last_active":"2024-09-21T15:51:04.864095+0000","last_peered":"2024-09-21T15:51:04.864095+0000","last_clean":"2024-09-21T15:51:04.864095+0000","last_became_active":"2024-09-21T15:50:35.682788+0000","last_became_peered":"2024-09-21T15:50:35.682788+0000","last_unstale":"2024-09-21T15:51:04.864095+0000","last_undegraded":"2024-09-21T15:51:04.864095+0000","last_fullsized":"2024-09-21T15:51:04.864095+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":668,"log_dups_size":0,"ondisk_log_size":668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:20:40.695690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055773200000000002,"stat_sum":{"num_bytes":8855371,"num_objects":38,"num_object_clones":2,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":26,"num_read":10711,"num_read_kb":27404,"num_write":842,"num_write_kb":11011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":175,"num_bytes_recovered":14134268,"num_keys_recovered":36,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"14","length":"9"}]},{"pgid":"2.4","version":"0'0","reported_seq":139,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-21T15:51:04.863562+0000","last_change":"2024-09-21T15:50:30.146654+0000","last_active":"2024-09-21T15:51:04.863562+0000","last_peered":"2024-09-21T15:51:04.863562+0000","last_clean":"2024-09-21T15:51:04.863562+0000","last_became_active":"2024-09-21T15:50:30.146273+0000","last_became_peered":"2024-09-21T15:50:30.146273+0000","last_unstale":"2024-09-21T15:51:04.863562+0000","last_undegraded":"2024-09-21T15:51:04.863562+0000","last_fullsized":"2024-09-21T15:51:04.863562+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:21:48.178519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040525700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"129'588","reported_seq":5239,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-21T15:51:04.893106+0000","last_change":"2024-09-21T15:51:04.893106+0000","last_active":"2024-09-21T15:51:04.893106+0000","last_peered":"2024-09-21T15:51:04.893106+0000","last_clean":"2024-09-21T15:51:04.893106+0000","last_became_active":"2024-09-21T15:50:57.683205+0000","last_became_peered":"2024-09-21T15:50:57.683205+0000","last_unstale":"2024-09-21T15:51:04.893106+0000","last_undegraded":"2024-09-21T15:51:04.893106+0000","last_fullsized":"2024-09-21T15:51:04.893106+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":588,"log_dups_size":0,"ondisk_log_size":588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:45:47.756413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011424860000000001,"stat_sum":{"num_bytes":10526937,"num_objects":49,"num_object_clones":1,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":35,"num_read":5720,"num_read_kb":33936,"num_write":673,"num_write_kb":16332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":6293408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"14","length":"9"}]},{"pgid":"3.3","version":"129'811","reported_seq":3842,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-21T15:51:04.923056+0000","last_change":"2024-09-21T15:51:04.795799+0000","last_active":"2024-09-21T15:51:04.923056+0000","last_peered":"2024-09-21T15:51:04.923056+0000","last_clean":"2024-09-21T15:51:04.923056+0000","last_became_active":"2024-09-21T15:50:32.677456+0000","last_became_peered":"2024-09-21T15:50:32.677456+0000","last_unstale":"2024-09-21T15:51:04.923056+0000","last_undegraded":"2024-09-21T15:51:04.923056+0000","last_fullsized":"2024-09-21T15:51:04.923056+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":811,"log_dups_size":0,"ondisk_log_size":811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:46:31.194345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080406800000000001,"stat_sum":{"num_bytes":18270079,"num_objects":60,"num_object_clones":7,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":41,"num_read":4876,"num_read_kb":19102,"num_write":858,"num_write_kb":7351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":632176,"num_keys_recovered":24,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"14","length":"9"}]},{"pgid":"2.2","version":"20'2","reported_seq":104,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-21T15:51:03.787573+0000","last_change":"2024-09-21T15:49:39.834640+0000","last_active":"2024-09-21T15:51:03.787573+0000","last_peered":"2024-09-21T15:51:03.787573+0000","last_clean":"2024-09-21T15:51:03.787573+0000","last_became_active":"2024-09-21T15:49:39.834207+0000","last_became_peered":"2024-09-21T15:49:39.834207+0000","last_unstale":"2024-09-21T15:51:03.787573+0000","last_undegraded":"2024-09-21T15:51:03.787573+0000","last_fullsized":"2024-09-21T15:51:03.787573+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:02:54.255716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"129'356","reported_seq":4589,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-21T15:51:04.864767+0000","last_change":"2024-09-21T15:51:04.864767+0000","last_active":"2024-09-21T15:51:04.864767+0000","last_peered":"2024-09-21T15:51:04.864767+0000","last_clean":"2024-09-21T15:51:04.864767+0000","last_became_active":"2024-09-21T15:50:35.682001+0000","last_became_peered":"2024-09-21T15:50:35.682001+0000","last_unstale":"2024-09-21T15:51:04.864767+0000","last_undegraded":"2024-09-21T15:51:04.864767+0000","last_fullsized":"2024-09-21T15:51:04.864767+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":356,"log_dups_size":0,"ondisk_log_size":356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:09:15.325457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001059115,"stat_sum":{"num_bytes":9325711,"num_objects":47,"num_object_clones":2,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":33,"num_read":4043,"num_read_kb":23169,"num_write":351,"num_write_kb":4139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":187,"num_bytes_recovered":4305276,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"14","length":"9"}]},{"pgid":"2.1","version":"0'0","reported_seq":50,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-21T15:51:03.787382+0000","last_change":"2024-09-21T15:50:54.694573+0000","last_active":"2024-09-21T15:51:03.787382+0000","last_peered":"2024-09-21T15:51:03.787382+0000","last_clean":"2024-09-21T15:51:03.787382+0000","last_became_active":"2024-09-21T15:50:54.694234+0000","last_became_peered":"2024-09-21T15:50:54.694234+0000","last_unstale":"2024-09-21T15:51:03.787382+0000","last_undegraded":"2024-09-21T15:51:03.787382+0000","last_fullsized":"2024-09-21T15:51:03.787382+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:56:23.067030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"127'500","reported_seq":2254,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-21T15:51:02.811670+0000","last_change":"2024-09-21T15:51:02.812254+0000","last_active":"2024-09-21T15:51:02.811670+0000","last_peered":"2024-09-21T15:51:02.811670+0000","last_clean":"2024-09-21T15:51:02.811670+0000","last_became_active":"2024-09-21T15:50:32.671413+0000","last_became_peered":"2024-09-21T15:50:32.671413+0000","last_unstale":"2024-09-21T15:51:02.811670+0000","last_undegraded":"2024-09-21T15:51:02.811670+0000","last_fullsized":"2024-09-21T15:51:02.811670+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":500,"log_dups_size":0,"ondisk_log_size":500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:04:19.509402+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0056738329999999997,"stat_sum":{"num_bytes":7850384,"num_objects":54,"num_object_clones":1,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":39,"num_read":3236,"num_read_kb":16171,"num_write":544,"num_write_kb":5965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":1696547,"num_keys_recovered":12,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"16","length":"7"}]},{"pgid":"2.0","version":"0'0","reported_seq":126,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-21T15:51:03.792496+0000","last_change":"2024-09-21T15:49:00.114730+0000","last_active":"2024-09-21T15:51:03.792496+0000","last_peered":"2024-09-21T15:51:03.792496+0000","last_clean":"2024-09-21T15:51:03.792496+0000","last_became_active":"2024-09-21T15:48:57.585525+0000","last_became_peered":"2024-09-21T15:48:57.585525+0000","last_unstale":"2024-09-21T15:51:03.792496+0000","last_undegraded":"2024-09-21T15:51:03.792496+0000","last_fullsized":"2024-09-21T15:51:03.792496+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:35:35.786526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031691100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"129'395","reported_seq":2593,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-21T15:51:04.815475+0000","last_change":"2024-09-21T15:51:04.815475+0000","last_active":"2024-09-21T15:51:04.815475+0000","last_peered":"2024-09-21T15:51:04.815475+0000","last_clean":"2024-09-21T15:51:04.815475+0000","last_became_active":"2024-09-21T15:50:57.685697+0000","last_became_peered":"2024-09-21T15:50:57.685697+0000","last_unstale":"2024-09-21T15:51:04.815475+0000","last_undegraded":"2024-09-21T15:51:04.815475+0000","last_fullsized":"2024-09-21T15:51:04.815475+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":395,"log_dups_size":0,"ondisk_log_size":395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:06:02.205965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037972,"stat_sum":{"num_bytes":11232373,"num_objects":47,"num_object_clones":3,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":32,"num_read":2138,"num_read_kb":34447,"num_write":394,"num_write_kb":12612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":1782045,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"14","length":"9"}]},{"pgid":"2.3","version":"18'1","reported_seq":97,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-21T15:51:03.787445+0000","last_change":"2024-09-21T15:49:39.834423+0000","last_active":"2024-09-21T15:51:03.787445+0000","last_peered":"2024-09-21T15:51:03.787445+0000","last_clean":"2024-09-21T15:51:03.787445+0000","last_became_active":"2024-09-21T15:49:39.833860+0000","last_became_peered":"2024-09-21T15:49:39.833860+0000","last_unstale":"2024-09-21T15:51:03.787445+0000","last_undegraded":"2024-09-21T15:51:03.787445+0000","last_fullsized":"2024-09-21T15:51:03.787445+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:11:43.630472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":287,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-21T15:51:03.792578+0000","last_change":"2024-09-21T15:50:30.147280+0000","last_active":"2024-09-21T15:51:03.792578+0000","last_peered":"2024-09-21T15:51:03.792578+0000","last_clean":"2024-09-21T15:51:03.792578+0000","last_became_active":"2024-09-21T15:50:30.146987+0000","last_became_peered":"2024-09-21T15:50:30.146987+0000","last_unstale":"2024-09-21T15:51:03.792578+0000","last_undegraded":"2024-09-21T15:51:03.792578+0000","last_fullsized":"2024-09-21T15:51:03.792578+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:50.161884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:50.161884+0000","last_clean_scrub_stamp":"2024-09-21T15:48:50.161884+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:26:32.853650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"129'505","reported_seq":7788,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-21T15:51:04.864159+0000","last_change":"2024-09-21T15:51:04.864159+0000","last_active":"2024-09-21T15:51:04.864159+0000","last_peered":"2024-09-21T15:51:04.864159+0000","last_clean":"2024-09-21T15:51:04.864159+0000","last_became_active":"2024-09-21T15:50:57.680570+0000","last_became_peered":"2024-09-21T15:50:57.680570+0000","last_unstale":"2024-09-21T15:51:04.864159+0000","last_undegraded":"2024-09-21T15:51:04.864159+0000","last_fullsized":"2024-09-21T15:51:04.864159+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":505,"log_dups_size":0,"ondisk_log_size":505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:47:54.516042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060116,"stat_sum":{"num_bytes":24028054,"num_objects":55,"num_object_clones":3,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":34,"num_read":7203,"num_read_kb":29554,"num_write":528,"num_write_kb":4789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":284,"num_bytes_recovered":4370567,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"14","length":"9"}]},{"pgid":"3.9","version":"129'358","reported_seq":3025,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-21T15:51:04.897288+0000","last_change":"2024-09-21T15:51:04.795240+0000","last_active":"2024-09-21T15:51:04.897288+0000","last_peered":"2024-09-21T15:51:04.897288+0000","last_clean":"2024-09-21T15:51:04.897288+0000","last_became_active":"2024-09-21T15:49:07.210506+0000","last_became_peered":"2024-09-21T15:49:07.210506+0000","last_unstale":"2024-09-21T15:51:04.897288+0000","last_undegraded":"2024-09-21T15:51:04.897288+0000","last_fullsized":"2024-09-21T15:51:04.897288+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":358,"log_dups_size":0,"ondisk_log_size":358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:55:18.046738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052564500000000004,"stat_sum":{"num_bytes":20219334,"num_objects":50,"num_object_clones":2,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":32,"num_read":2685,"num_read_kb":28221,"num_write":425,"num_write_kb":12085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"14","length":"9"}]},{"pgid":"3.a","version":"129'543","reported_seq":12048,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-21T15:51:04.808641+0000","last_change":"2024-09-21T15:51:04.808641+0000","last_active":"2024-09-21T15:51:04.808641+0000","last_peered":"2024-09-21T15:51:04.808641+0000","last_clean":"2024-09-21T15:51:04.808641+0000","last_became_active":"2024-09-21T15:49:06.694202+0000","last_became_peered":"2024-09-21T15:49:06.694202+0000","last_unstale":"2024-09-21T15:51:04.808641+0000","last_undegraded":"2024-09-21T15:51:04.808641+0000","last_fullsized":"2024-09-21T15:51:04.808641+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":543,"log_dups_size":0,"ondisk_log_size":543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:02:05.237705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000343273,"stat_sum":{"num_bytes":15188906,"num_objects":50,"num_object_clones":4,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":32,"num_read":12630,"num_read_kb":63299,"num_write":541,"num_write_kb":14206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"14","length":"9"}]},{"pgid":"3.b","version":"127'824","reported_seq":3477,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-21T15:51:02.817416+0000","last_change":"2024-09-21T15:51:02.817674+0000","last_active":"2024-09-21T15:51:02.817416+0000","last_peered":"2024-09-21T15:51:02.817416+0000","last_clean":"2024-09-21T15:51:02.817416+0000","last_became_active":"2024-09-21T15:50:57.677421+0000","last_became_peered":"2024-09-21T15:50:57.677421+0000","last_unstale":"2024-09-21T15:51:02.817416+0000","last_undegraded":"2024-09-21T15:51:02.817416+0000","last_fullsized":"2024-09-21T15:51:02.817416+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":824,"log_dups_size":0,"ondisk_log_size":824,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:11:10.826178+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.010713808999999999,"stat_sum":{"num_bytes":7831317,"num_objects":42,"num_object_clones":2,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":28,"num_read":5909,"num_read_kb":20142,"num_write":1219,"num_write_kb":7745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":152,"num_bytes_recovered":7108665,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"16","length":"7"}]},{"pgid":"3.c","version":"129'721","reported_seq":6637,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-21T15:51:04.918634+0000","last_change":"2024-09-21T15:51:04.809656+0000","last_active":"2024-09-21T15:51:04.918634+0000","last_peered":"2024-09-21T15:51:04.918634+0000","last_clean":"2024-09-21T15:51:04.918634+0000","last_became_active":"2024-09-21T15:50:57.685366+0000","last_became_peered":"2024-09-21T15:50:57.685366+0000","last_unstale":"2024-09-21T15:51:04.918634+0000","last_undegraded":"2024-09-21T15:51:04.918634+0000","last_fullsized":"2024-09-21T15:51:04.918634+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":721,"log_dups_size":0,"ondisk_log_size":721,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:19:24.171447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012787390000000001,"stat_sum":{"num_bytes":12003256,"num_objects":53,"num_object_clones":2,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":37,"num_read":8835,"num_read_kb":25671,"num_write":854,"num_write_kb":5005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":2985423,"num_keys_recovered":23,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"14","length":"9"}]},{"pgid":"3.d","version":"124'854","reported_seq":4035,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-21T15:51:04.894525+0000","last_change":"2024-09-21T15:51:04.894721+0000","last_active":"2024-09-21T15:51:04.894525+0000","last_peered":"2024-09-21T15:51:04.894525+0000","last_clean":"2024-09-21T15:51:04.894525+0000","last_became_active":"2024-09-21T15:49:06.689203+0000","last_became_peered":"2024-09-21T15:49:06.689203+0000","last_unstale":"2024-09-21T15:51:04.894525+0000","last_undegraded":"2024-09-21T15:51:04.894525+0000","last_fullsized":"2024-09-21T15:51:04.894525+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:22:17.578359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029359149999999999,"stat_sum":{"num_bytes":14268393,"num_objects":54,"num_object_clones":5,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":40,"num_read":6938,"num_read_kb":27800,"num_write":1121,"num_write_kb":14803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"14","length":"9"}]},{"pgid":"3.e","version":"127'522","reported_seq":7921,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-21T15:51:04.894981+0000","last_change":"2024-09-21T15:51:04.895197+0000","last_active":"2024-09-21T15:51:04.894981+0000","last_peered":"2024-09-21T15:51:04.894981+0000","last_clean":"2024-09-21T15:51:04.894981+0000","last_became_active":"2024-09-21T15:49:06.690101+0000","last_became_peered":"2024-09-21T15:49:06.690101+0000","last_unstale":"2024-09-21T15:51:04.894981+0000","last_undegraded":"2024-09-21T15:51:04.894981+0000","last_fullsized":"2024-09-21T15:51:04.894981+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":522,"log_dups_size":0,"ondisk_log_size":522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:43:30.659104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033460719999999998,"stat_sum":{"num_bytes":35076593,"num_objects":53,"num_object_clones":8,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":36,"num_read":8562,"num_read_kb":42832,"num_write":484,"num_write_kb":16267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"14","length":"9"}]},{"pgid":"3.f","version":"129'686","reported_seq":3674,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-21T15:51:04.897602+0000","last_change":"2024-09-21T15:51:04.895459+0000","last_active":"2024-09-21T15:51:04.897602+0000","last_peered":"2024-09-21T15:51:04.897602+0000","last_clean":"2024-09-21T15:51:04.897602+0000","last_became_active":"2024-09-21T15:49:06.694250+0000","last_became_peered":"2024-09-21T15:49:06.694250+0000","last_unstale":"2024-09-21T15:51:04.897602+0000","last_undegraded":"2024-09-21T15:51:04.897602+0000","last_fullsized":"2024-09-21T15:51:04.897602+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":686,"log_dups_size":0,"ondisk_log_size":686,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:21:37.777769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003517409,"stat_sum":{"num_bytes":5002603,"num_objects":54,"num_object_clones":2,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":44,"num_read":4936,"num_read_kb":15916,"num_write":881,"num_write_kb":5075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"14","length":"9"}]},{"pgid":"3.10","version":"128'441","reported_seq":2937,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-21T15:51:04.808417+0000","last_change":"2024-09-21T15:51:04.808673+0000","last_active":"2024-09-21T15:51:04.808417+0000","last_peered":"2024-09-21T15:51:04.808417+0000","last_clean":"2024-09-21T15:51:04.808417+0000","last_became_active":"2024-09-21T15:50:32.676147+0000","last_became_peered":"2024-09-21T15:50:32.676147+0000","last_unstale":"2024-09-21T15:51:04.808417+0000","last_undegraded":"2024-09-21T15:51:04.808417+0000","last_fullsized":"2024-09-21T15:51:04.808417+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":441,"log_dups_size":0,"ondisk_log_size":441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:44:18.821310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042708200000000002,"stat_sum":{"num_bytes":4606610,"num_objects":47,"num_object_clones":2,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":42,"num_read":2958,"num_read_kb":11674,"num_write":362,"num_write_kb":4465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":1191778,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"14","length":"9"}]},{"pgid":"3.11","version":"129'965","reported_seq":4030,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-21T15:51:04.895548+0000","last_change":"2024-09-21T15:51:04.894481+0000","last_active":"2024-09-21T15:51:04.895548+0000","last_peered":"2024-09-21T15:51:04.895548+0000","last_clean":"2024-09-21T15:51:04.895548+0000","last_became_active":"2024-09-21T15:49:06.688798+0000","last_became_peered":"2024-09-21T15:49:06.688798+0000","last_unstale":"2024-09-21T15:51:04.895548+0000","last_undegraded":"2024-09-21T15:51:04.895548+0000","last_fullsized":"2024-09-21T15:51:04.895548+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":965,"log_dups_size":0,"ondisk_log_size":965,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:48:33.679133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00315972,"stat_sum":{"num_bytes":9847433,"num_objects":43,"num_object_clones":3,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":32,"num_read":6298,"num_read_kb":16898,"num_write":1309,"num_write_kb":8394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"14","length":"9"}]},{"pgid":"3.12","version":"125'364","reported_seq":2873,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-21T15:51:02.806687+0000","last_change":"2024-09-21T15:51:02.807046+0000","last_active":"2024-09-21T15:51:02.806687+0000","last_peered":"2024-09-21T15:51:02.806687+0000","last_clean":"2024-09-21T15:51:02.806687+0000","last_became_active":"2024-09-21T15:50:32.671686+0000","last_became_peered":"2024-09-21T15:50:32.671686+0000","last_unstale":"2024-09-21T15:51:02.806687+0000","last_undegraded":"2024-09-21T15:51:02.806687+0000","last_fullsized":"2024-09-21T15:51:02.806687+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":364,"log_dups_size":0,"ondisk_log_size":364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:40:53.968477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000479734,"stat_sum":{"num_bytes":6332700,"num_objects":53,"num_object_clones":4,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":38,"num_read":2451,"num_read_kb":11667,"num_write":390,"num_write_kb":13839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":3359702,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"16","length":"7"}]},{"pgid":"3.13","version":"129'926","reported_seq":7272,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-21T15:51:04.894383+0000","last_change":"2024-09-21T15:51:04.894383+0000","last_active":"2024-09-21T15:51:04.894383+0000","last_peered":"2024-09-21T15:51:04.894383+0000","last_clean":"2024-09-21T15:51:04.894383+0000","last_became_active":"2024-09-21T15:49:06.690128+0000","last_became_peered":"2024-09-21T15:49:06.690128+0000","last_unstale":"2024-09-21T15:51:04.894383+0000","last_undegraded":"2024-09-21T15:51:04.894383+0000","last_fullsized":"2024-09-21T15:51:04.894383+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":926,"log_dups_size":0,"ondisk_log_size":926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:11.763959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030254129999999998,"stat_sum":{"num_bytes":4832232,"num_objects":50,"num_object_clones":4,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":35,"num_read":9260,"num_read_kb":26307,"num_write":1409,"num_write_kb":13570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"14","length":"9"}]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":399536819,"num_objects":1592,"num_object_clones":99,"num_object_copies":3184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1592,"num_whiteouts":1116,"num_read":202897,"num_read_kb":838557,"num_write":22458,"num_write_kb":303027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2186,"num_bytes_recovered":83031061,"num_keys_recovered":281,"num_objects_omap":19,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":145104896,"data_stored":172043819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5973,"internal_metadata":0},"log_size":19030,"ondisk_log_size":19030,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574878,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95744,"kb_used_data":20340,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94276096,"statfs":{"total":96636764160,"available":96538722304,"internally_reserved":0,"allocated":20828160,"data_stored":21085203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":77199819},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.372}]},{"osd":1,"last update":"Sat Sep 21 15:50:52 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.48299999999999998,"15min":0.495},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":0.73199999999999998,"5min":1.04,"15min":1.04},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.44,"5min":0.46999999999999997,"15min":0.47499999999999998},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.64500000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.438}]},{"osd":2,"last update":"Sat Sep 21 15:50:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.439,"15min":0.44400000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.53500000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.437,"5min":0.498,"15min":0.50800000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.70099999999999996,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.38100000000000001}]},{"osd":3,"last update":"Sat Sep 21 15:50:52 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.49099999999999999,"15min":0.499},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.65300000000000002,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.432,"5min":0.47099999999999997,"15min":0.47699999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68200000000000005,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.41999999999999998}]},{"osd":4,"last update":"Sat Sep 21 15:50:52 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.41699999999999998,"15min":0.42299999999999999},"min":{"1min":0.24099999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.59999999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.38,"5min":0.39000000000000001,"15min":0.39200000000000002},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.28999999999999998}]},{"osd":5,"last update":"Sat Sep 21 15:50:52 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.439,"15min":0.45000000000000001},"min":{"1min":0.25900000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.52900000000000003,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.317},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.44700000000000001,"15min":0.45000000000000001},"min":{"1min":0.29499999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.68600000000000005,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.40999999999999998}]},{"osd":6,"last update":"Sat Sep 21 15:50:52 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.49399999999999999,"15min":0.5},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.72099999999999997,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.74199999999999999,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.39300000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574881,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52720,"kb_used_data":9528,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94319120,"statfs":{"total":96636764160,"available":96582778880,"internally_reserved":0,"allocated":9756672,"data_stored":10476507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.499}]},{"osd":1,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.44800000000000001,"15min":0.45800000000000002},"min":{"1min":0.26900000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.53300000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.46800000000000003,"15min":0.47999999999999998},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.53300000000000003,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.39300000000000002}]},{"osd":2,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.47099999999999997,"15min":0.47699999999999998},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.45800000000000002,"15min":0.46200000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.60999999999999999,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.52100000000000002}]},{"osd":3,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.51400000000000001,"15min":0.52400000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.70399999999999996,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.53100000000000003,"15min":0.53800000000000003},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.73699999999999999,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.48399999999999999}]},{"osd":4,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.40600000000000003,"15min":0.41399999999999998},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.222},"max":{"1min":0.57499999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.379,"5min":0.46700000000000003,"15min":0.48199999999999998},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.76300000000000001,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.441}]},{"osd":5,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.44600000000000001,"15min":0.45600000000000002},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.57399999999999995,"5min":0.755,"15min":0.755},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.372,"5min":0.45000000000000001,"15min":0.46300000000000002},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.628,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.35999999999999999}]},{"osd":7,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.45000000000000001,"15min":0.45700000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.55400000000000005,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.497,"15min":0.51100000000000001},"min":{"1min":0.30499999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.748,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.42799999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574880,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72236,"kb_used_data":18976,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94299604,"statfs":{"total":96636764160,"available":96562794496,"internally_reserved":0,"allocated":19431424,"data_stored":22501581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.48899999999999999}]},{"osd":1,"last update":"Sat Sep 21 15:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.71699999999999997,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.5,"15min":0.51400000000000001},"min":{"1min":0.34399999999999997,"5min":0.314,"15min":0.314},"max":{"1min":0.52200000000000002,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.50800000000000001}]},{"osd":2,"last update":"Sat Sep 21 15:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.49399999999999999,"15min":0.50600000000000001},"min":{"1min":0.34200000000000003,"5min":0.311,"15min":0.311},"max":{"1min":0.56899999999999995,"5min":1.046,"15min":1.046},"last":0.434},{"interface":"front","average":{"1min":0.437,"5min":0.46800000000000003,"15min":0.47299999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.68999999999999995,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.46700000000000003}]},{"osd":3,"last update":"Sat Sep 21 15:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.47699999999999998,"15min":0.48699999999999999},"min":{"1min":0.29599999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.55200000000000005,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.53800000000000003,"15min":0.55200000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.76900000000000002,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.53100000000000003}]},{"osd":4,"last update":"Sat Sep 21 15:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.65400000000000003,"15min":0.70799999999999996},"min":{"1min":0.254,"5min":0.253,"15min":0.253},"max":{"1min":0.503,"5min":5.6399999999999997,"15min":5.6399999999999997},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.45000000000000001,"15min":0.47199999999999998},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.48899999999999999,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.45500000000000002}]},{"osd":6,"last update":"Sat Sep 21 15:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.443,"15min":0.45500000000000002},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.51700000000000002,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.48199999999999998,"15min":0.499},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.55200000000000005,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.56200000000000006}]},{"osd":7,"last update":"Sat Sep 21 15:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.47199999999999998,"15min":0.48599999999999999},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.56000000000000005,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.504,"15min":0.52100000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.61599999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574879,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":96636,"kb_used_data":33516,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94275204,"statfs":{"total":96636764160,"available":96537808896,"internally_reserved":0,"allocated":34320384,"data_stored":36968991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:49:52 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.56499999999999995}]},{"osd":1,"last update":"Sat Sep 21 15:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.433,"15min":0.41899999999999998},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":1.8839999999999999,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.33600000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.51800000000000002}]},{"osd":2,"last update":"Sat Sep 21 15:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.443,"15min":0.439},"min":{"1min":0.311,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.433,"15min":0.433},"min":{"1min":0.31,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.54000000000000004,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.432}]},{"osd":3,"last update":"Sat Sep 21 15:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.34200000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.66600000000000004,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.46800000000000003,"15min":0.46999999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.60899999999999999,"5min":0.626,"15min":0.626},"last":0.505}]},{"osd":5,"last update":"Sat Sep 21 15:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.47199999999999998,"15min":0.47599999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":1.919,"5min":2.4489999999999998,"15min":2.4489999999999998},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.42399999999999999,"15min":0.42499999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.748,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.63700000000000001}]},{"osd":6,"last update":"Sat Sep 21 15:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.41599999999999998,"15min":0.41399999999999998},"min":{"1min":0.29399999999999998,"5min":0.245,"15min":0.245},"max":{"1min":0.63800000000000001,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.55700000000000005,"15min":0.54500000000000004},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":4.1340000000000003,"5min":4.1340000000000003,"15min":4.1340000000000003},"last":0.67000000000000004}]},{"osd":7,"last update":"Sat Sep 21 15:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.45400000000000001,"15min":0.434},"min":{"1min":0.28199999999999997,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":2.7599999999999998,"5min":2.7599999999999998,"15min":2.7599999999999998},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.52400000000000002,"15min":0.52200000000000002},"min":{"1min":0.32600000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":2.016,"5min":2.016,"15min":2.016},"last":0.68899999999999995}]}]},{"osd":3,"up_from":0,"seq":55834574880,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":13,"seq":55834574880,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44584,"kb_used_data":9104,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94327256,"statfs":{"total":96636764160,"available":96591110144,"internally_reserved":0,"allocated":9322496,"data_stored":9636862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.316}]},{"osd":1,"last update":"Sat Sep 21 15:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.40300000000000002,"15min":0.41999999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.439,"5min":1.621,"15min":1.621},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.308,"5min":0.36399999999999999,"15min":0.374},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.46400000000000002,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.23799999999999999}]},{"osd":3,"last update":"Sat Sep 21 15:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.90800000000000003,"15min":0.97099999999999997},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":4.6580000000000004,"5min":11.56,"15min":11.56},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.40200000000000002,"15min":0.41299999999999998},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.46999999999999997,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.35399999999999998}]},{"osd":4,"last update":"Sat Sep 21 15:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.57999999999999996,"15min":0.60199999999999998},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.60499999999999998,"5min":1.653,"15min":1.653},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.60399999999999998,"15min":0.628},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.74199999999999999,"5min":2.4660000000000002,"15min":2.4660000000000002},"last":0.41099999999999998}]},{"osd":5,"last update":"Sat Sep 21 15:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.501,"15min":0.51100000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.61199999999999999,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.372},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.52700000000000002,"15min":0.53900000000000003},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.65400000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.38400000000000001}]},{"osd":6,"last update":"Sat Sep 21 15:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.54900000000000004,"15min":0.56299999999999994},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.67500000000000004,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.54000000000000004,"15min":0.55600000000000005},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.629,"5min":0.873,"15min":0.873},"last":0.39300000000000002}]},{"osd":7,"last update":"Sat Sep 21 15:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.54900000000000004,"15min":0.56100000000000005},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.67000000000000004,"5min":0.876,"15min":0.876},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.58199999999999996,"15min":0.59799999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.754,"5min":1.089,"15min":1.089},"last":0.42599999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574879,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54400,"kb_used_data":15076,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94317440,"statfs":{"total":96636764160,"available":96581058560,"internally_reserved":0,"allocated":15437824,"data_stored":15612328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.315,"15min":0.315},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"last":0.623}]},{"osd":2,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":0.33000000000000002,"15min":0.34000000000000002},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.43099999999999999,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.30299999999999999,"5min":0.499,"15min":0.53200000000000003},"min":{"1min":0.21299999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.52400000000000002,"5min":5.0670000000000002,"15min":5.0670000000000002},"last":0.46400000000000002}]},{"osd":3,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.91700000000000004,"15min":1.0189999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.46000000000000002,"5min":16.032,"15min":16.032},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.309,"5min":0.34999999999999998,"15min":0.35699999999999998},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":0.47899999999999998,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.86799999999999999}]},{"osd":4,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.46800000000000003,"15min":0.47399999999999998},"min":{"1min":0.25900000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.63200000000000001,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.53600000000000003,"15min":0.55100000000000005},"min":{"1min":0.311,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.56799999999999995,"5min":2.2269999999999999,"15min":2.2269999999999999},"last":0.752}]},{"osd":5,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.55300000000000005,"15min":0.57299999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.64700000000000002,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.66000000000000003,"15min":0.69599999999999995},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.66200000000000003,"5min":4.6989999999999998,"15min":4.6989999999999998},"last":0.68200000000000005}]},{"osd":6,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.53000000000000003,"15min":0.53900000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.68200000000000005,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.59999999999999998,"15min":0.62},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.64500000000000002,"5min":2.262,"15min":2.262},"last":0.69599999999999995}]},{"osd":7,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.67000000000000004,"15min":0.70399999999999996},"min":{"1min":0.33400000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.69399999999999995,"5min":4.742,"15min":4.742},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.51300000000000001,"15min":0.51800000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.70199999999999996,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.63900000000000001}]}]},{"osd":0,"up_from":93,"seq":399431958536,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58968,"kb_used_data":26436,"kb_used_omap":13,"kb_used_meta":32498,"kb_avail":94312872,"statfs":{"total":96636764160,"available":96576380928,"internally_reserved":0,"allocated":27070464,"data_stored":31487811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13968,"internal_metadata":33278320},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.20499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.307}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":19070976,"data_stored":20811093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5973,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12279808,"data_stored":13273764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7778304,"data_stored":8899678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":33562624,"data_stored":46709369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":29290496,"data_stored":32745123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":17555456,"data_stored":21441417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8208384,"data_stored":9735893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":17358848,"data_stored":18427482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T15:51:06.169 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T15:51:06.238 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:51:06.237+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:51:06.298 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:06.298 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:06.307 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T15:51:06.334 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:51:06.333+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:51:06.436 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:51:06.435+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:51:06.440 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:51:06.440 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-09-21T15:51:06.448 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:06.448 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:06.452 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":191,"stamp":"2024-09-21T15:51:05.911542+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":400127206,"num_objects":1596,"num_object_clones":99,"num_object_copies":3192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1596,"num_whiteouts":1116,"num_read":203013,"num_read_kb":838657,"num_write":22619,"num_write_kb":305445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2186,"num_bytes_recovered":83031061,"num_keys_recovered":281,"num_objects_omap":19,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19150,"ondisk_log_size":19150,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":660602880,"kb_used":475288,"kb_used_data":132976,"kb_used_omap":22,"kb_used_meta":338025,"kb_avail":660127592,"statfs":{"total":676457349120,"available":675970654208,"internally_reserved":0,"allocated":136167424,"data_stored":147769283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":23526,"internal_metadata":346137626},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-289564545,"num_objects":-178,"num_object_clones":-59,"num_object_copies":-356,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-178,"num_whiteouts":-43,"num_read":1178,"num_read_kb":1015,"num_write":250,"num_write_kb":60,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000631"},"pg_stats":[{"pgid":"3.1f","version":"124'400","reported_seq":2000,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-21T15:51:02.816879+0000","last_change":"2024-09-21T15:51:02.817045+0000","last_active":"2024-09-21T15:51:02.816879+0000","last_peered":"2024-09-21T15:51:02.816879+0000","last_clean":"2024-09-21T15:51:02.816879+0000","last_became_active":"2024-09-21T15:50:32.670964+0000","last_became_peered":"2024-09-21T15:50:32.670964+0000","last_unstale":"2024-09-21T15:51:02.816879+0000","last_undegraded":"2024-09-21T15:51:02.816879+0000","last_fullsized":"2024-09-21T15:51:02.816879+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":400,"log_dups_size":0,"ondisk_log_size":400,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:09:09.918864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.010020006,"stat_sum":{"num_bytes":4571152,"num_objects":41,"num_object_clones":1,"num_object_copies":82,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":29,"num_read":2410,"num_read_kb":10418,"num_write":431,"num_write_kb":5956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":1974433,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"16","length":"7"}]},{"pgid":"3.1e","version":"129'550","reported_seq":6021,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-21T15:51:04.925611+0000","last_change":"2024-09-21T15:51:04.808889+0000","last_active":"2024-09-21T15:51:04.925611+0000","last_peered":"2024-09-21T15:51:04.925611+0000","last_clean":"2024-09-21T15:51:04.925611+0000","last_became_active":"2024-09-21T15:50:57.685368+0000","last_became_peered":"2024-09-21T15:50:57.685368+0000","last_unstale":"2024-09-21T15:51:04.925611+0000","last_undegraded":"2024-09-21T15:51:04.925611+0000","last_fullsized":"2024-09-21T15:51:04.925611+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":550,"log_dups_size":0,"ondisk_log_size":550,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:23:42.939446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054089100000000005,"stat_sum":{"num_bytes":18810510,"num_objects":55,"num_object_clones":3,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":41,"num_read":7252,"num_read_kb":25737,"num_write":540,"num_write_kb":4350,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":125,"num_bytes_recovered":2730860,"num_keys_recovered":23,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"14","length":"9"}]},{"pgid":"3.1d","version":"129'596","reported_seq":9958,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-21T15:51:04.905563+0000","last_change":"2024-09-21T15:51:04.795258+0000","last_active":"2024-09-21T15:51:04.905563+0000","last_peered":"2024-09-21T15:51:04.905563+0000","last_clean":"2024-09-21T15:51:04.905563+0000","last_became_active":"2024-09-21T15:49:39.833206+0000","last_became_peered":"2024-09-21T15:49:39.833206+0000","last_unstale":"2024-09-21T15:51:04.905563+0000","last_undegraded":"2024-09-21T15:51:04.905563+0000","last_fullsized":"2024-09-21T15:51:04.905563+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":596,"log_dups_size":0,"ondisk_log_size":596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:36:14.082318+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000386733,"stat_sum":{"num_bytes":32540173,"num_objects":59,"num_object_clones":6,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":38,"num_read":10487,"num_read_kb":28946,"num_write":659,"num_write_kb":8871,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"14","length":"9"}]},{"pgid":"3.1c","version":"129'554","reported_seq":4313,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-21T15:51:04.870292+0000","last_change":"2024-09-21T15:51:04.795617+0000","last_active":"2024-09-21T15:51:04.870292+0000","last_peered":"2024-09-21T15:51:04.870292+0000","last_clean":"2024-09-21T15:51:04.870292+0000","last_became_active":"2024-09-21T15:49:39.833113+0000","last_became_peered":"2024-09-21T15:49:39.833113+0000","last_unstale":"2024-09-21T15:51:04.870292+0000","last_undegraded":"2024-09-21T15:51:04.870292+0000","last_fullsized":"2024-09-21T15:51:04.870292+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":554,"log_dups_size":0,"ondisk_log_size":554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:43:57.592707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00064970700000000002,"stat_sum":{"num_bytes":8652416,"num_objects":42,"num_object_clones":1,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":28,"num_read":5136,"num_read_kb":17235,"num_write":600,"num_write_kb":4787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"14","length":"9"}]},{"pgid":"3.1b","version":"127'460","reported_seq":4159,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-21T15:51:02.814699+0000","last_change":"2024-09-21T15:51:02.815040+0000","last_active":"2024-09-21T15:51:02.814699+0000","last_peered":"2024-09-21T15:51:02.814699+0000","last_clean":"2024-09-21T15:51:02.814699+0000","last_became_active":"2024-09-21T15:50:32.674250+0000","last_became_peered":"2024-09-21T15:50:32.674250+0000","last_unstale":"2024-09-21T15:51:02.814699+0000","last_undegraded":"2024-09-21T15:51:02.814699+0000","last_fullsized":"2024-09-21T15:51:02.814699+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":460,"log_dups_size":0,"ondisk_log_size":460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:54:57.399818+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0082588109999999996,"stat_sum":{"num_bytes":6701956,"num_objects":57,"num_object_clones":1,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":57,"num_whiteouts":40,"num_read":4618,"num_read_kb":29923,"num_write":458,"num_write_kb":13501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":4759000,"num_keys_recovered":12,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"16","length":"7"}]},{"pgid":"3.1a","version":"129'392","reported_seq":6587,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-21T15:51:04.897322+0000","last_change":"2024-09-21T15:51:04.795498+0000","last_active":"2024-09-21T15:51:04.897322+0000","last_peered":"2024-09-21T15:51:04.897322+0000","last_clean":"2024-09-21T15:51:04.897322+0000","last_became_active":"2024-09-21T15:49:06.693903+0000","last_became_peered":"2024-09-21T15:49:06.693903+0000","last_unstale":"2024-09-21T15:51:04.897322+0000","last_undegraded":"2024-09-21T15:51:04.897322+0000","last_fullsized":"2024-09-21T15:51:04.897322+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":392,"log_dups_size":0,"ondisk_log_size":392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:41:45.432893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00076663199999999995,"stat_sum":{"num_bytes":15065327,"num_objects":49,"num_object_clones":2,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":32,"num_read":6890,"num_read_kb":31459,"num_write":391,"num_write_kb":9470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"14","length":"9"}]},{"pgid":"3.19","version":"129'442","reported_seq":8457,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-21T15:51:04.902788+0000","last_change":"2024-09-21T15:51:04.864682+0000","last_active":"2024-09-21T15:51:04.902788+0000","last_peered":"2024-09-21T15:51:04.902788+0000","last_clean":"2024-09-21T15:51:04.902788+0000","last_became_active":"2024-09-21T15:50:57.677431+0000","last_became_peered":"2024-09-21T15:50:57.677431+0000","last_unstale":"2024-09-21T15:51:04.902788+0000","last_undegraded":"2024-09-21T15:51:04.902788+0000","last_fullsized":"2024-09-21T15:51:04.902788+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":442,"log_dups_size":0,"ondisk_log_size":442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:53:19.596041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010939809999999999,"stat_sum":{"num_bytes":23099668,"num_objects":49,"num_object_clones":4,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":35,"num_read":8281,"num_read_kb":39578,"num_write":508,"num_write_kb":7408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":118,"num_bytes_recovered":3488172,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"14","length":"9"}]},{"pgid":"3.18","version":"127'476","reported_seq":4093,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-21T15:51:02.812328+0000","last_change":"2024-09-21T15:51:02.812328+0000","last_active":"2024-09-21T15:51:02.812328+0000","last_peered":"2024-09-21T15:51:02.812328+0000","last_clean":"2024-09-21T15:51:02.812328+0000","last_became_active":"2024-09-21T15:50:57.675432+0000","last_became_peered":"2024-09-21T15:50:57.675432+0000","last_unstale":"2024-09-21T15:51:02.812328+0000","last_undegraded":"2024-09-21T15:51:02.812328+0000","last_fullsized":"2024-09-21T15:51:02.812328+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":476,"log_dups_size":0,"ondisk_log_size":476,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:43:10.553541+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0057213999999999997,"stat_sum":{"num_bytes":6448637,"num_objects":45,"num_object_clones":2,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":28,"num_read":5684,"num_read_kb":30500,"num_write":487,"num_write_kb":11063,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":161,"num_bytes_recovered":10150338,"num_keys_recovered":46,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"16","length":"7"}]},{"pgid":"3.17","version":"115'996","reported_seq":9243,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-21T15:51:02.815096+0000","last_change":"2024-09-21T15:51:02.815471+0000","last_active":"2024-09-21T15:51:02.815096+0000","last_peered":"2024-09-21T15:51:02.815096+0000","last_clean":"2024-09-21T15:51:02.815096+0000","last_became_active":"2024-09-21T15:50:32.668771+0000","last_became_peered":"2024-09-21T15:50:32.668771+0000","last_unstale":"2024-09-21T15:51:02.815096+0000","last_undegraded":"2024-09-21T15:51:02.815096+0000","last_fullsized":"2024-09-21T15:51:02.815096+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":996,"log_dups_size":0,"ondisk_log_size":996,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:39:43.554150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0086097599999999993,"stat_sum":{"num_bytes":6308777,"num_objects":59,"num_object_clones":5,"num_object_copies":118,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":59,"num_whiteouts":40,"num_read":12662,"num_read_kb":21448,"num_write":1268,"num_write_kb":8252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":3319185,"num_keys_recovered":34,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"16","length":"7"}]},{"pgid":"3.16","version":"129'491","reported_seq":6459,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-21T15:51:04.815497+0000","last_change":"2024-09-21T15:51:04.815497+0000","last_active":"2024-09-21T15:51:04.815497+0000","last_peered":"2024-09-21T15:51:04.815497+0000","last_clean":"2024-09-21T15:51:04.815497+0000","last_became_active":"2024-09-21T15:49:06.688808+0000","last_became_peered":"2024-09-21T15:49:06.688808+0000","last_unstale":"2024-09-21T15:51:04.815497+0000","last_undegraded":"2024-09-21T15:51:04.815497+0000","last_fullsized":"2024-09-21T15:51:04.815497+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":491,"log_dups_size":0,"ondisk_log_size":491,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:43:02.012254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000317251,"stat_sum":{"num_bytes":10165651,"num_objects":47,"num_object_clones":2,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":32,"num_read":7324,"num_read_kb":49627,"num_write":500,"num_write_kb":13812,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"14","length":"9"}]},{"pgid":"3.15","version":"127'515","reported_seq":3963,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-21T15:51:04.894471+0000","last_change":"2024-09-21T15:51:04.894668+0000","last_active":"2024-09-21T15:51:04.894471+0000","last_peered":"2024-09-21T15:51:04.894471+0000","last_clean":"2024-09-21T15:51:04.894471+0000","last_became_active":"2024-09-21T15:50:57.680672+0000","last_became_peered":"2024-09-21T15:50:57.680672+0000","last_unstale":"2024-09-21T15:51:04.894471+0000","last_undegraded":"2024-09-21T15:51:04.894471+0000","last_fullsized":"2024-09-21T15:51:04.894471+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":515,"log_dups_size":0,"ondisk_log_size":515,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:00:16.784121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030861170000000002,"stat_sum":{"num_bytes":12623976,"num_objects":44,"num_object_clones":3,"num_object_copies":88,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":32,"num_read":4052,"num_read_kb":27144,"num_write":628,"num_write_kb":17935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":111,"num_bytes_recovered":2234893,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"14","length":"9"}]},{"pgid":"3.14","version":"129'1171","reported_seq":4230,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-21T15:51:04.902658+0000","last_change":"2024-09-21T15:51:04.795871+0000","last_active":"2024-09-21T15:51:04.902658+0000","last_peered":"2024-09-21T15:51:04.902658+0000","last_clean":"2024-09-21T15:51:04.902658+0000","last_became_active":"2024-09-21T15:49:06.688803+0000","last_became_peered":"2024-09-21T15:49:06.688803+0000","last_unstale":"2024-09-21T15:51:04.902658+0000","last_undegraded":"2024-09-21T15:51:04.902658+0000","last_fullsized":"2024-09-21T15:51:04.902658+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":1171,"log_dups_size":0,"ondisk_log_size":1171,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:18:47.662631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00096029700000000004,"stat_sum":{"num_bytes":5574397,"num_objects":39,"num_object_clones":2,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":27,"num_read":7697,"num_read_kb":18182,"num_write":1890,"num_write_kb":6180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"14","length":"9"}]},{"pgid":"2.7","version":"0'0","reported_seq":125,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-21T15:51:03.788040+0000","last_change":"2024-09-21T15:48:59.666431+0000","last_active":"2024-09-21T15:51:03.788040+0000","last_peered":"2024-09-21T15:51:03.788040+0000","last_clean":"2024-09-21T15:51:03.788040+0000","last_became_active":"2024-09-21T15:48:57.583665+0000","last_became_peered":"2024-09-21T15:48:57.583665+0000","last_unstale":"2024-09-21T15:51:03.788040+0000","last_undegraded":"2024-09-21T15:51:03.788040+0000","last_fullsized":"2024-09-21T15:51:03.788040+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:52:09.228964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031098399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"127'408","reported_seq":2917,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-21T15:51:02.865904+0000","last_change":"2024-09-21T15:51:02.816830+0000","last_active":"2024-09-21T15:51:02.865904+0000","last_peered":"2024-09-21T15:51:02.865904+0000","last_clean":"2024-09-21T15:51:02.865904+0000","last_became_active":"2024-09-21T15:50:32.671424+0000","last_became_peered":"2024-09-21T15:50:32.671424+0000","last_unstale":"2024-09-21T15:51:02.865904+0000","last_undegraded":"2024-09-21T15:51:02.865904+0000","last_fullsized":"2024-09-21T15:51:02.865904+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":408,"log_dups_size":0,"ondisk_log_size":408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:04:14.554732+0000","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0.010156573,"stat_sum":{"num_bytes":7990384,"num_objects":47,"num_object_clones":3,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":38,"num_read":2637,"num_read_kb":12956,"num_write":382,"num_write_kb":8392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":4494388,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"16","length":"7"}]},{"pgid":"2.6","version":"0'0","reported_seq":125,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-21T15:51:03.787702+0000","last_change":"2024-09-21T15:48:59.589790+0000","last_active":"2024-09-21T15:51:03.787702+0000","last_peered":"2024-09-21T15:51:03.787702+0000","last_clean":"2024-09-21T15:51:03.787702+0000","last_became_active":"2024-09-21T15:48:57.582121+0000","last_became_peered":"2024-09-21T15:48:57.582121+0000","last_unstale":"2024-09-21T15:51:03.787702+0000","last_undegraded":"2024-09-21T15:51:03.787702+0000","last_fullsized":"2024-09-21T15:51:03.787702+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:33:47.592624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039654700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"129'552","reported_seq":7603,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-21T15:51:04.895230+0000","last_change":"2024-09-21T15:51:04.895230+0000","last_active":"2024-09-21T15:51:04.895230+0000","last_peered":"2024-09-21T15:51:04.895230+0000","last_clean":"2024-09-21T15:51:04.895230+0000","last_became_active":"2024-09-21T15:50:57.680364+0000","last_became_peered":"2024-09-21T15:50:57.680364+0000","last_unstale":"2024-09-21T15:51:04.895230+0000","last_undegraded":"2024-09-21T15:51:04.895230+0000","last_fullsized":"2024-09-21T15:51:04.895230+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":552,"log_dups_size":0,"ondisk_log_size":552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:26:50.885554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003339421,"stat_sum":{"num_bytes":15685509,"num_objects":60,"num_object_clones":7,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":40,"num_read":8378,"num_read_kb":21194,"num_write":531,"num_write_kb":5397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":2019937,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"14","length":"9"}]},{"pgid":"2.5","version":"0'0","reported_seq":139,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-21T15:51:04.891749+0000","last_change":"2024-09-21T15:50:30.146743+0000","last_active":"2024-09-21T15:51:04.891749+0000","last_peered":"2024-09-21T15:51:04.891749+0000","last_clean":"2024-09-21T15:51:04.891749+0000","last_became_active":"2024-09-21T15:50:30.146428+0000","last_became_peered":"2024-09-21T15:50:30.146428+0000","last_unstale":"2024-09-21T15:51:04.891749+0000","last_undegraded":"2024-09-21T15:51:04.891749+0000","last_fullsized":"2024-09-21T15:51:04.891749+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:58:38.689807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036074600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"129'668","reported_seq":8597,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-21T15:51:04.864095+0000","last_change":"2024-09-21T15:51:04.864095+0000","last_active":"2024-09-21T15:51:04.864095+0000","last_peered":"2024-09-21T15:51:04.864095+0000","last_clean":"2024-09-21T15:51:04.864095+0000","last_became_active":"2024-09-21T15:50:35.682788+0000","last_became_peered":"2024-09-21T15:50:35.682788+0000","last_unstale":"2024-09-21T15:51:04.864095+0000","last_undegraded":"2024-09-21T15:51:04.864095+0000","last_fullsized":"2024-09-21T15:51:04.864095+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":668,"log_dups_size":0,"ondisk_log_size":668,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:20:40.695690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00055773200000000002,"stat_sum":{"num_bytes":8855371,"num_objects":38,"num_object_clones":2,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":26,"num_read":10711,"num_read_kb":27404,"num_write":842,"num_write_kb":11011,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":175,"num_bytes_recovered":14134268,"num_keys_recovered":36,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"14","length":"9"}]},{"pgid":"2.4","version":"0'0","reported_seq":139,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-21T15:51:04.863562+0000","last_change":"2024-09-21T15:50:30.146654+0000","last_active":"2024-09-21T15:51:04.863562+0000","last_peered":"2024-09-21T15:51:04.863562+0000","last_clean":"2024-09-21T15:51:04.863562+0000","last_became_active":"2024-09-21T15:50:30.146273+0000","last_became_peered":"2024-09-21T15:50:30.146273+0000","last_unstale":"2024-09-21T15:51:04.863562+0000","last_undegraded":"2024-09-21T15:51:04.863562+0000","last_fullsized":"2024-09-21T15:51:04.863562+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:21:48.178519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040525700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"129'588","reported_seq":5239,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-21T15:51:04.893106+0000","last_change":"2024-09-21T15:51:04.893106+0000","last_active":"2024-09-21T15:51:04.893106+0000","last_peered":"2024-09-21T15:51:04.893106+0000","last_clean":"2024-09-21T15:51:04.893106+0000","last_became_active":"2024-09-21T15:50:57.683205+0000","last_became_peered":"2024-09-21T15:50:57.683205+0000","last_unstale":"2024-09-21T15:51:04.893106+0000","last_undegraded":"2024-09-21T15:51:04.893106+0000","last_fullsized":"2024-09-21T15:51:04.893106+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":588,"log_dups_size":0,"ondisk_log_size":588,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:45:47.756413+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011424860000000001,"stat_sum":{"num_bytes":10526937,"num_objects":49,"num_object_clones":1,"num_object_copies":98,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":35,"num_read":5720,"num_read_kb":33936,"num_write":673,"num_write_kb":16332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":6293408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"14","length":"9"}]},{"pgid":"3.3","version":"129'811","reported_seq":3842,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-21T15:51:04.923056+0000","last_change":"2024-09-21T15:51:04.795799+0000","last_active":"2024-09-21T15:51:04.923056+0000","last_peered":"2024-09-21T15:51:04.923056+0000","last_clean":"2024-09-21T15:51:04.923056+0000","last_became_active":"2024-09-21T15:50:32.677456+0000","last_became_peered":"2024-09-21T15:50:32.677456+0000","last_unstale":"2024-09-21T15:51:04.923056+0000","last_undegraded":"2024-09-21T15:51:04.923056+0000","last_fullsized":"2024-09-21T15:51:04.923056+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":811,"log_dups_size":0,"ondisk_log_size":811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:46:31.194345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080406800000000001,"stat_sum":{"num_bytes":18270079,"num_objects":60,"num_object_clones":7,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":41,"num_read":4876,"num_read_kb":19102,"num_write":858,"num_write_kb":7351,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":632176,"num_keys_recovered":24,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"14","length":"9"}]},{"pgid":"2.2","version":"20'2","reported_seq":104,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-21T15:51:03.787573+0000","last_change":"2024-09-21T15:49:39.834640+0000","last_active":"2024-09-21T15:51:03.787573+0000","last_peered":"2024-09-21T15:51:03.787573+0000","last_clean":"2024-09-21T15:51:03.787573+0000","last_became_active":"2024-09-21T15:49:39.834207+0000","last_became_peered":"2024-09-21T15:49:39.834207+0000","last_unstale":"2024-09-21T15:51:03.787573+0000","last_undegraded":"2024-09-21T15:51:03.787573+0000","last_fullsized":"2024-09-21T15:51:03.787573+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:02:54.255716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"129'356","reported_seq":4589,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-21T15:51:04.864767+0000","last_change":"2024-09-21T15:51:04.864767+0000","last_active":"2024-09-21T15:51:04.864767+0000","last_peered":"2024-09-21T15:51:04.864767+0000","last_clean":"2024-09-21T15:51:04.864767+0000","last_became_active":"2024-09-21T15:50:35.682001+0000","last_became_peered":"2024-09-21T15:50:35.682001+0000","last_unstale":"2024-09-21T15:51:04.864767+0000","last_undegraded":"2024-09-21T15:51:04.864767+0000","last_fullsized":"2024-09-21T15:51:04.864767+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":356,"log_dups_size":0,"ondisk_log_size":356,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:09:15.325457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001059115,"stat_sum":{"num_bytes":9325711,"num_objects":47,"num_object_clones":2,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":33,"num_read":4043,"num_read_kb":23169,"num_write":351,"num_write_kb":4139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":187,"num_bytes_recovered":4305276,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"14","length":"9"}]},{"pgid":"2.1","version":"0'0","reported_seq":50,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-21T15:51:03.787382+0000","last_change":"2024-09-21T15:50:54.694573+0000","last_active":"2024-09-21T15:51:03.787382+0000","last_peered":"2024-09-21T15:51:03.787382+0000","last_clean":"2024-09-21T15:51:03.787382+0000","last_became_active":"2024-09-21T15:50:54.694234+0000","last_became_peered":"2024-09-21T15:50:54.694234+0000","last_unstale":"2024-09-21T15:51:03.787382+0000","last_undegraded":"2024-09-21T15:51:03.787382+0000","last_fullsized":"2024-09-21T15:51:03.787382+0000","mapping_epoch":118,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":119,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:56:23.067030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"127'500","reported_seq":2254,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-21T15:51:02.811670+0000","last_change":"2024-09-21T15:51:02.812254+0000","last_active":"2024-09-21T15:51:02.811670+0000","last_peered":"2024-09-21T15:51:02.811670+0000","last_clean":"2024-09-21T15:51:02.811670+0000","last_became_active":"2024-09-21T15:50:32.671413+0000","last_became_peered":"2024-09-21T15:50:32.671413+0000","last_unstale":"2024-09-21T15:51:02.811670+0000","last_undegraded":"2024-09-21T15:51:02.811670+0000","last_fullsized":"2024-09-21T15:51:02.811670+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":500,"log_dups_size":0,"ondisk_log_size":500,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:04:19.509402+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0056738329999999997,"stat_sum":{"num_bytes":7850384,"num_objects":54,"num_object_clones":1,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":39,"num_read":3236,"num_read_kb":16171,"num_write":544,"num_write_kb":5965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":1696547,"num_keys_recovered":12,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"16","length":"7"}]},{"pgid":"2.0","version":"0'0","reported_seq":126,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-21T15:51:03.792496+0000","last_change":"2024-09-21T15:49:00.114730+0000","last_active":"2024-09-21T15:51:03.792496+0000","last_peered":"2024-09-21T15:51:03.792496+0000","last_clean":"2024-09-21T15:51:03.792496+0000","last_became_active":"2024-09-21T15:48:57.585525+0000","last_became_peered":"2024-09-21T15:48:57.585525+0000","last_unstale":"2024-09-21T15:51:03.792496+0000","last_undegraded":"2024-09-21T15:51:03.792496+0000","last_fullsized":"2024-09-21T15:51:03.792496+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:35:35.786526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031691100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"129'395","reported_seq":2593,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-21T15:51:04.815475+0000","last_change":"2024-09-21T15:51:04.815475+0000","last_active":"2024-09-21T15:51:04.815475+0000","last_peered":"2024-09-21T15:51:04.815475+0000","last_clean":"2024-09-21T15:51:04.815475+0000","last_became_active":"2024-09-21T15:50:57.685697+0000","last_became_peered":"2024-09-21T15:50:57.685697+0000","last_unstale":"2024-09-21T15:51:04.815475+0000","last_undegraded":"2024-09-21T15:51:04.815475+0000","last_fullsized":"2024-09-21T15:51:04.815475+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":395,"log_dups_size":0,"ondisk_log_size":395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:06:02.205965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037972,"stat_sum":{"num_bytes":11232373,"num_objects":47,"num_object_clones":3,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":32,"num_read":2138,"num_read_kb":34447,"num_write":394,"num_write_kb":12612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":107,"num_bytes_recovered":1782045,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"14","length":"9"}]},{"pgid":"2.3","version":"18'1","reported_seq":97,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-21T15:51:03.787445+0000","last_change":"2024-09-21T15:49:39.834423+0000","last_active":"2024-09-21T15:51:03.787445+0000","last_peered":"2024-09-21T15:51:03.787445+0000","last_clean":"2024-09-21T15:51:03.787445+0000","last_became_active":"2024-09-21T15:49:39.833860+0000","last_became_peered":"2024-09-21T15:49:39.833860+0000","last_unstale":"2024-09-21T15:51:03.787445+0000","last_undegraded":"2024-09-21T15:51:03.787445+0000","last_fullsized":"2024-09-21T15:51:03.787445+0000","mapping_epoch":48,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":49,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:11:43.630472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":287,"reported_epoch":128,"state":"active+clean","last_fresh":"2024-09-21T15:51:03.792578+0000","last_change":"2024-09-21T15:50:30.147280+0000","last_active":"2024-09-21T15:51:03.792578+0000","last_peered":"2024-09-21T15:51:03.792578+0000","last_clean":"2024-09-21T15:51:03.792578+0000","last_became_active":"2024-09-21T15:50:30.146987+0000","last_became_peered":"2024-09-21T15:50:30.146987+0000","last_unstale":"2024-09-21T15:51:03.792578+0000","last_undegraded":"2024-09-21T15:51:03.792578+0000","last_fullsized":"2024-09-21T15:51:03.792578+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:50.161884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:50.161884+0000","last_clean_scrub_stamp":"2024-09-21T15:48:50.161884+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:26:32.853650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"129'505","reported_seq":7788,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-21T15:51:04.864159+0000","last_change":"2024-09-21T15:51:04.864159+0000","last_active":"2024-09-21T15:51:04.864159+0000","last_peered":"2024-09-21T15:51:04.864159+0000","last_clean":"2024-09-21T15:51:04.864159+0000","last_became_active":"2024-09-21T15:50:57.680570+0000","last_became_peered":"2024-09-21T15:50:57.680570+0000","last_unstale":"2024-09-21T15:51:04.864159+0000","last_undegraded":"2024-09-21T15:51:04.864159+0000","last_fullsized":"2024-09-21T15:51:04.864159+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":505,"log_dups_size":0,"ondisk_log_size":505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:47:54.516042+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060116,"stat_sum":{"num_bytes":24028054,"num_objects":55,"num_object_clones":3,"num_object_copies":110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":55,"num_whiteouts":34,"num_read":7203,"num_read_kb":29554,"num_write":528,"num_write_kb":4789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":284,"num_bytes_recovered":4370567,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"14","length":"9"}]},{"pgid":"3.9","version":"129'358","reported_seq":3025,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-21T15:51:04.897288+0000","last_change":"2024-09-21T15:51:04.795240+0000","last_active":"2024-09-21T15:51:04.897288+0000","last_peered":"2024-09-21T15:51:04.897288+0000","last_clean":"2024-09-21T15:51:04.897288+0000","last_became_active":"2024-09-21T15:49:07.210506+0000","last_became_peered":"2024-09-21T15:49:07.210506+0000","last_unstale":"2024-09-21T15:51:04.897288+0000","last_undegraded":"2024-09-21T15:51:04.897288+0000","last_fullsized":"2024-09-21T15:51:04.897288+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":358,"log_dups_size":0,"ondisk_log_size":358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:55:18.046738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052564500000000004,"stat_sum":{"num_bytes":20219334,"num_objects":50,"num_object_clones":2,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":32,"num_read":2685,"num_read_kb":28221,"num_write":425,"num_write_kb":12085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"14","length":"9"}]},{"pgid":"3.a","version":"129'543","reported_seq":12048,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-21T15:51:04.808641+0000","last_change":"2024-09-21T15:51:04.808641+0000","last_active":"2024-09-21T15:51:04.808641+0000","last_peered":"2024-09-21T15:51:04.808641+0000","last_clean":"2024-09-21T15:51:04.808641+0000","last_became_active":"2024-09-21T15:49:06.694202+0000","last_became_peered":"2024-09-21T15:49:06.694202+0000","last_unstale":"2024-09-21T15:51:04.808641+0000","last_undegraded":"2024-09-21T15:51:04.808641+0000","last_fullsized":"2024-09-21T15:51:04.808641+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":543,"log_dups_size":0,"ondisk_log_size":543,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:02:05.237705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000343273,"stat_sum":{"num_bytes":15188906,"num_objects":50,"num_object_clones":4,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":32,"num_read":12630,"num_read_kb":63299,"num_write":541,"num_write_kb":14206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"14","length":"9"}]},{"pgid":"3.b","version":"127'824","reported_seq":3477,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-21T15:51:02.817416+0000","last_change":"2024-09-21T15:51:02.817674+0000","last_active":"2024-09-21T15:51:02.817416+0000","last_peered":"2024-09-21T15:51:02.817416+0000","last_clean":"2024-09-21T15:51:02.817416+0000","last_became_active":"2024-09-21T15:50:57.677421+0000","last_became_peered":"2024-09-21T15:50:57.677421+0000","last_unstale":"2024-09-21T15:51:02.817416+0000","last_undegraded":"2024-09-21T15:51:02.817416+0000","last_fullsized":"2024-09-21T15:51:02.817416+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":824,"log_dups_size":0,"ondisk_log_size":824,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:11:10.826178+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.010713808999999999,"stat_sum":{"num_bytes":7831317,"num_objects":42,"num_object_clones":2,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":28,"num_read":5909,"num_read_kb":20142,"num_write":1219,"num_write_kb":7745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":152,"num_bytes_recovered":7108665,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"16","length":"7"}]},{"pgid":"3.c","version":"129'721","reported_seq":6637,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-21T15:51:04.918634+0000","last_change":"2024-09-21T15:51:04.809656+0000","last_active":"2024-09-21T15:51:04.918634+0000","last_peered":"2024-09-21T15:51:04.918634+0000","last_clean":"2024-09-21T15:51:04.918634+0000","last_became_active":"2024-09-21T15:50:57.685366+0000","last_became_peered":"2024-09-21T15:50:57.685366+0000","last_unstale":"2024-09-21T15:51:04.918634+0000","last_undegraded":"2024-09-21T15:51:04.918634+0000","last_fullsized":"2024-09-21T15:51:04.918634+0000","mapping_epoch":121,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":122,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":721,"log_dups_size":0,"ondisk_log_size":721,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:19:24.171447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012787390000000001,"stat_sum":{"num_bytes":12003256,"num_objects":53,"num_object_clones":2,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":37,"num_read":8835,"num_read_kb":25671,"num_write":854,"num_write_kb":5005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":2985423,"num_keys_recovered":23,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"14","length":"9"}]},{"pgid":"3.d","version":"124'854","reported_seq":4035,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-21T15:51:04.894525+0000","last_change":"2024-09-21T15:51:04.894721+0000","last_active":"2024-09-21T15:51:04.894525+0000","last_peered":"2024-09-21T15:51:04.894525+0000","last_clean":"2024-09-21T15:51:04.894525+0000","last_became_active":"2024-09-21T15:49:06.689203+0000","last_became_peered":"2024-09-21T15:49:06.689203+0000","last_unstale":"2024-09-21T15:51:04.894525+0000","last_undegraded":"2024-09-21T15:51:04.894525+0000","last_fullsized":"2024-09-21T15:51:04.894525+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":854,"log_dups_size":0,"ondisk_log_size":854,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:22:17.578359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0029359149999999999,"stat_sum":{"num_bytes":14268393,"num_objects":54,"num_object_clones":5,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":40,"num_read":6938,"num_read_kb":27800,"num_write":1121,"num_write_kb":14803,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"14","length":"9"}]},{"pgid":"3.e","version":"127'522","reported_seq":7921,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-21T15:51:04.894981+0000","last_change":"2024-09-21T15:51:04.895197+0000","last_active":"2024-09-21T15:51:04.894981+0000","last_peered":"2024-09-21T15:51:04.894981+0000","last_clean":"2024-09-21T15:51:04.894981+0000","last_became_active":"2024-09-21T15:49:06.690101+0000","last_became_peered":"2024-09-21T15:49:06.690101+0000","last_unstale":"2024-09-21T15:51:04.894981+0000","last_undegraded":"2024-09-21T15:51:04.894981+0000","last_fullsized":"2024-09-21T15:51:04.894981+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":522,"log_dups_size":0,"ondisk_log_size":522,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:43:30.659104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0033460719999999998,"stat_sum":{"num_bytes":35076593,"num_objects":53,"num_object_clones":8,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":36,"num_read":8562,"num_read_kb":42832,"num_write":484,"num_write_kb":16267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"14","length":"9"}]},{"pgid":"3.f","version":"129'686","reported_seq":3674,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-21T15:51:04.897602+0000","last_change":"2024-09-21T15:51:04.895459+0000","last_active":"2024-09-21T15:51:04.897602+0000","last_peered":"2024-09-21T15:51:04.897602+0000","last_clean":"2024-09-21T15:51:04.897602+0000","last_became_active":"2024-09-21T15:49:06.694250+0000","last_became_peered":"2024-09-21T15:49:06.694250+0000","last_unstale":"2024-09-21T15:51:04.897602+0000","last_undegraded":"2024-09-21T15:51:04.897602+0000","last_fullsized":"2024-09-21T15:51:04.897602+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":686,"log_dups_size":0,"ondisk_log_size":686,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:21:37.777769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003517409,"stat_sum":{"num_bytes":5002603,"num_objects":54,"num_object_clones":2,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":44,"num_read":4936,"num_read_kb":15916,"num_write":881,"num_write_kb":5075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"14","length":"9"}]},{"pgid":"3.10","version":"128'441","reported_seq":2937,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-21T15:51:04.808417+0000","last_change":"2024-09-21T15:51:04.808673+0000","last_active":"2024-09-21T15:51:04.808417+0000","last_peered":"2024-09-21T15:51:04.808417+0000","last_clean":"2024-09-21T15:51:04.808417+0000","last_became_active":"2024-09-21T15:50:32.676147+0000","last_became_peered":"2024-09-21T15:50:32.676147+0000","last_unstale":"2024-09-21T15:51:04.808417+0000","last_undegraded":"2024-09-21T15:51:04.808417+0000","last_fullsized":"2024-09-21T15:51:04.808417+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":441,"log_dups_size":0,"ondisk_log_size":441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:44:18.821310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042708200000000002,"stat_sum":{"num_bytes":4606610,"num_objects":47,"num_object_clones":2,"num_object_copies":94,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":42,"num_read":2958,"num_read_kb":11674,"num_write":362,"num_write_kb":4465,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":1191778,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"14","length":"9"}]},{"pgid":"3.11","version":"129'965","reported_seq":4030,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-21T15:51:04.895548+0000","last_change":"2024-09-21T15:51:04.894481+0000","last_active":"2024-09-21T15:51:04.895548+0000","last_peered":"2024-09-21T15:51:04.895548+0000","last_clean":"2024-09-21T15:51:04.895548+0000","last_became_active":"2024-09-21T15:49:06.688798+0000","last_became_peered":"2024-09-21T15:49:06.688798+0000","last_unstale":"2024-09-21T15:51:04.895548+0000","last_undegraded":"2024-09-21T15:51:04.895548+0000","last_fullsized":"2024-09-21T15:51:04.895548+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":965,"log_dups_size":0,"ondisk_log_size":965,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:48:33.679133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00315972,"stat_sum":{"num_bytes":9847433,"num_objects":43,"num_object_clones":3,"num_object_copies":86,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":43,"num_whiteouts":32,"num_read":6298,"num_read_kb":16898,"num_write":1309,"num_write_kb":8394,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"14","length":"9"}]},{"pgid":"3.12","version":"125'364","reported_seq":2873,"reported_epoch":127,"state":"active+clean","last_fresh":"2024-09-21T15:51:02.806687+0000","last_change":"2024-09-21T15:51:02.807046+0000","last_active":"2024-09-21T15:51:02.806687+0000","last_peered":"2024-09-21T15:51:02.806687+0000","last_clean":"2024-09-21T15:51:02.806687+0000","last_became_active":"2024-09-21T15:50:32.671686+0000","last_became_peered":"2024-09-21T15:50:32.671686+0000","last_unstale":"2024-09-21T15:51:02.806687+0000","last_undegraded":"2024-09-21T15:51:02.806687+0000","last_fullsized":"2024-09-21T15:51:02.806687+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":364,"log_dups_size":0,"ondisk_log_size":364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:40:53.968477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000479734,"stat_sum":{"num_bytes":6332700,"num_objects":53,"num_object_clones":4,"num_object_copies":106,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53,"num_whiteouts":38,"num_read":2451,"num_read_kb":11667,"num_write":390,"num_write_kb":13839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":3359702,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"16","length":"7"}]},{"pgid":"3.13","version":"129'926","reported_seq":7272,"reported_epoch":129,"state":"active+clean","last_fresh":"2024-09-21T15:51:04.894383+0000","last_change":"2024-09-21T15:51:04.894383+0000","last_active":"2024-09-21T15:51:04.894383+0000","last_peered":"2024-09-21T15:51:04.894383+0000","last_clean":"2024-09-21T15:51:04.894383+0000","last_became_active":"2024-09-21T15:49:06.690128+0000","last_became_peered":"2024-09-21T15:49:06.690128+0000","last_unstale":"2024-09-21T15:51:04.894383+0000","last_undegraded":"2024-09-21T15:51:04.894383+0000","last_fullsized":"2024-09-21T15:51:04.894383+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":926,"log_dups_size":0,"ondisk_log_size":926,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:11.763959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030254129999999998,"stat_sum":{"num_bytes":4832232,"num_objects":50,"num_object_clones":4,"num_object_copies":100,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":35,"num_read":9260,"num_read_kb":26307,"num_write":1409,"num_write_kb":13570,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"14","length":"9"}]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":399536819,"num_objects":1592,"num_object_clones":99,"num_object_copies":3184,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1592,"num_whiteouts":1116,"num_read":202897,"num_read_kb":838557,"num_write":22458,"num_write_kb":303027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2186,"num_bytes_recovered":83031061,"num_keys_recovered":281,"num_objects_omap":19,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":145104896,"data_stored":172043819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5973,"internal_metadata":0},"log_size":19030,"ondisk_log_size":19030,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574878,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":95744,"kb_used_data":20340,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94276096,"statfs":{"total":96636764160,"available":96538722304,"internally_reserved":0,"allocated":20828160,"data_stored":21085203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":77199819},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:49:50 2024","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.46999999999999997},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.372}]},{"osd":1,"last update":"Sat Sep 21 15:50:52 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.48299999999999998,"15min":0.495},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":0.73199999999999998,"5min":1.04,"15min":1.04},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.44,"5min":0.46999999999999997,"15min":0.47499999999999998},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.64500000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.438}]},{"osd":2,"last update":"Sat Sep 21 15:50:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.439,"15min":0.44400000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.53500000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.437,"5min":0.498,"15min":0.50800000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.70099999999999996,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.38100000000000001}]},{"osd":3,"last update":"Sat Sep 21 15:50:52 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.49099999999999999,"15min":0.499},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.65300000000000002,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.432,"5min":0.47099999999999997,"15min":0.47699999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68200000000000005,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.41999999999999998}]},{"osd":4,"last update":"Sat Sep 21 15:50:52 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.41699999999999998,"15min":0.42299999999999999},"min":{"1min":0.24099999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.59999999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.38,"5min":0.39000000000000001,"15min":0.39200000000000002},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.28999999999999998}]},{"osd":5,"last update":"Sat Sep 21 15:50:52 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.439,"15min":0.45000000000000001},"min":{"1min":0.25900000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.52900000000000003,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.317},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.44700000000000001,"15min":0.45000000000000001},"min":{"1min":0.29499999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.68600000000000005,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.40999999999999998}]},{"osd":6,"last update":"Sat Sep 21 15:50:52 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.49399999999999999,"15min":0.5},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.72099999999999997,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.74199999999999999,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.39300000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574881,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":52720,"kb_used_data":9528,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94319120,"statfs":{"total":96636764160,"available":96582778880,"internally_reserved":0,"allocated":9756672,"data_stored":10476507,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.499}]},{"osd":1,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.44800000000000001,"15min":0.45800000000000002},"min":{"1min":0.26900000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.53300000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.46800000000000003,"15min":0.47999999999999998},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.53300000000000003,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.39300000000000002}]},{"osd":2,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.47099999999999997,"15min":0.47699999999999998},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.45800000000000002,"15min":0.46200000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.60999999999999999,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.52100000000000002}]},{"osd":3,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.51400000000000001,"15min":0.52400000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.70399999999999996,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.53100000000000003,"15min":0.53800000000000003},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.73699999999999999,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.48399999999999999}]},{"osd":4,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.40600000000000003,"15min":0.41399999999999998},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.222},"max":{"1min":0.57499999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.379,"5min":0.46700000000000003,"15min":0.48199999999999998},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.76300000000000001,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.441}]},{"osd":5,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.44600000000000001,"15min":0.45600000000000002},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.57399999999999995,"5min":0.755,"15min":0.755},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.372,"5min":0.45000000000000001,"15min":0.46300000000000002},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.628,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.35999999999999999}]},{"osd":7,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.45000000000000001,"15min":0.45700000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.55400000000000005,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.497,"15min":0.51100000000000001},"min":{"1min":0.30499999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.748,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.42799999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574880,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72236,"kb_used_data":18976,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94299604,"statfs":{"total":96636764160,"available":96562794496,"internally_reserved":0,"allocated":19431424,"data_stored":22501581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:49:53 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.86699999999999999,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.48899999999999999}]},{"osd":1,"last update":"Sat Sep 21 15:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.71699999999999997,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.5,"15min":0.51400000000000001},"min":{"1min":0.34399999999999997,"5min":0.314,"15min":0.314},"max":{"1min":0.52200000000000002,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.50800000000000001}]},{"osd":2,"last update":"Sat Sep 21 15:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.49399999999999999,"15min":0.50600000000000001},"min":{"1min":0.34200000000000003,"5min":0.311,"15min":0.311},"max":{"1min":0.56899999999999995,"5min":1.046,"15min":1.046},"last":0.434},{"interface":"front","average":{"1min":0.437,"5min":0.46800000000000003,"15min":0.47299999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.68999999999999995,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.46700000000000003}]},{"osd":3,"last update":"Sat Sep 21 15:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.47699999999999998,"15min":0.48699999999999999},"min":{"1min":0.29599999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.55200000000000005,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.53800000000000003,"15min":0.55200000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.76900000000000002,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.53100000000000003}]},{"osd":4,"last update":"Sat Sep 21 15:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.65400000000000003,"15min":0.70799999999999996},"min":{"1min":0.254,"5min":0.253,"15min":0.253},"max":{"1min":0.503,"5min":5.6399999999999997,"15min":5.6399999999999997},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.45000000000000001,"15min":0.47199999999999998},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.48899999999999999,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.45500000000000002}]},{"osd":6,"last update":"Sat Sep 21 15:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.443,"15min":0.45500000000000002},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.51700000000000002,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.48199999999999998,"15min":0.499},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.55200000000000005,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.56200000000000006}]},{"osd":7,"last update":"Sat Sep 21 15:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.47199999999999998,"15min":0.48599999999999999},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.56000000000000005,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.504,"15min":0.52100000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.61599999999999999}]}]},{"osd":4,"up_from":13,"seq":55834574879,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":96636,"kb_used_data":33516,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94275204,"statfs":{"total":96636764160,"available":96537808896,"internally_reserved":0,"allocated":34320384,"data_stored":36968991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:49:52 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.20399999999999999,"5min":0.20399999999999999,"15min":0.20399999999999999},"max":{"1min":0.96599999999999997,"5min":0.96599999999999997,"15min":0.96599999999999997},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.64100000000000001,"5min":0.64100000000000001,"15min":0.64100000000000001},"last":0.56499999999999995}]},{"osd":1,"last update":"Sat Sep 21 15:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.433,"15min":0.41899999999999998},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":1.8839999999999999,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.33600000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.51800000000000002}]},{"osd":2,"last update":"Sat Sep 21 15:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.443,"15min":0.439},"min":{"1min":0.311,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.433,"15min":0.433},"min":{"1min":0.31,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.54000000000000004,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.432}]},{"osd":3,"last update":"Sat Sep 21 15:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.34200000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.66600000000000004,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.46800000000000003,"15min":0.46999999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.60899999999999999,"5min":0.626,"15min":0.626},"last":0.505}]},{"osd":5,"last update":"Sat Sep 21 15:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.47199999999999998,"15min":0.47599999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":1.919,"5min":2.4489999999999998,"15min":2.4489999999999998},"last":0.64600000000000002},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.42399999999999999,"15min":0.42499999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.748,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.63700000000000001}]},{"osd":6,"last update":"Sat Sep 21 15:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.41599999999999998,"15min":0.41399999999999998},"min":{"1min":0.29399999999999998,"5min":0.245,"15min":0.245},"max":{"1min":0.63800000000000001,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.55700000000000005,"15min":0.54500000000000004},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":4.1340000000000003,"5min":4.1340000000000003,"15min":4.1340000000000003},"last":0.67000000000000004}]},{"osd":7,"last update":"Sat Sep 21 15:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.45400000000000001,"15min":0.434},"min":{"1min":0.28199999999999997,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":2.7599999999999998,"5min":2.7599999999999998,"15min":2.7599999999999998},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.52400000000000002,"15min":0.52200000000000002},"min":{"1min":0.32600000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":2.016,"5min":2.016,"15min":2.016},"last":0.68899999999999995}]}]},{"osd":3,"up_from":0,"seq":55834574880,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":13,"seq":55834574880,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":44584,"kb_used_data":9104,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94327256,"statfs":{"total":96636764160,"available":96591110144,"internally_reserved":0,"allocated":9322496,"data_stored":9636862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.77100000000000002,"5min":0.77100000000000002,"15min":0.77100000000000002},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.19600000000000001,"5min":0.19600000000000001,"15min":0.19600000000000001},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.316}]},{"osd":1,"last update":"Sat Sep 21 15:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.40300000000000002,"15min":0.41999999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.439,"5min":1.621,"15min":1.621},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.308,"5min":0.36399999999999999,"15min":0.374},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.46400000000000002,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.23799999999999999}]},{"osd":3,"last update":"Sat Sep 21 15:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.90800000000000003,"15min":0.97099999999999997},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":4.6580000000000004,"5min":11.56,"15min":11.56},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.40200000000000002,"15min":0.41299999999999998},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.46999999999999997,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.35399999999999998}]},{"osd":4,"last update":"Sat Sep 21 15:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.57999999999999996,"15min":0.60199999999999998},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.60499999999999998,"5min":1.653,"15min":1.653},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.60399999999999998,"15min":0.628},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.74199999999999999,"5min":2.4660000000000002,"15min":2.4660000000000002},"last":0.41099999999999998}]},{"osd":5,"last update":"Sat Sep 21 15:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.501,"15min":0.51100000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.61199999999999999,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.372},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.52700000000000002,"15min":0.53900000000000003},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.65400000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.38400000000000001}]},{"osd":6,"last update":"Sat Sep 21 15:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.54900000000000004,"15min":0.56299999999999994},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.67500000000000004,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.54000000000000004,"15min":0.55600000000000005},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.629,"5min":0.873,"15min":0.873},"last":0.39300000000000002}]},{"osd":7,"last update":"Sat Sep 21 15:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.54900000000000004,"15min":0.56100000000000005},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.67000000000000004,"5min":0.876,"15min":0.876},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.58199999999999996,"15min":0.59799999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.754,"5min":1.089,"15min":1.089},"last":0.42599999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574879,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":54400,"kb_used_data":15076,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94317440,"statfs":{"total":96636764160,"available":96581058560,"internally_reserved":0,"allocated":15437824,"data_stored":15612328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:49:49 2024","interfaces":[{"interface":"back","average":{"1min":0.315,"5min":0.315,"15min":0.315},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.54200000000000004,"5min":0.54200000000000004,"15min":0.54200000000000004},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.52000000000000002,"5min":0.52000000000000002,"15min":0.52000000000000002},"last":0.623}]},{"osd":2,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":0.33000000000000002,"15min":0.34000000000000002},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.43099999999999999,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.30299999999999999,"5min":0.499,"15min":0.53200000000000003},"min":{"1min":0.21299999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.52400000000000002,"5min":5.0670000000000002,"15min":5.0670000000000002},"last":0.46400000000000002}]},{"osd":3,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.91700000000000004,"15min":1.0189999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.46000000000000002,"5min":16.032,"15min":16.032},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.309,"5min":0.34999999999999998,"15min":0.35699999999999998},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":0.47899999999999998,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.86799999999999999}]},{"osd":4,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.46800000000000003,"15min":0.47399999999999998},"min":{"1min":0.25900000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.63200000000000001,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.53600000000000003,"15min":0.55100000000000005},"min":{"1min":0.311,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.56799999999999995,"5min":2.2269999999999999,"15min":2.2269999999999999},"last":0.752}]},{"osd":5,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.55300000000000005,"15min":0.57299999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.64700000000000002,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.72699999999999998},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.66000000000000003,"15min":0.69599999999999995},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.66200000000000003,"5min":4.6989999999999998,"15min":4.6989999999999998},"last":0.68200000000000005}]},{"osd":6,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.53000000000000003,"15min":0.53900000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.68200000000000005,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.59999999999999998,"15min":0.62},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.64500000000000002,"5min":2.262,"15min":2.262},"last":0.69599999999999995}]},{"osd":7,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.67000000000000004,"15min":0.70399999999999996},"min":{"1min":0.33400000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.69399999999999995,"5min":4.742,"15min":4.742},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.51300000000000001,"15min":0.51800000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.70199999999999996,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.63900000000000001}]}]},{"osd":0,"up_from":93,"seq":399431958536,"num_pgs":15,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":58968,"kb_used_data":26436,"kb_used_omap":13,"kb_used_meta":32498,"kb_avail":94312872,"statfs":{"total":96636764160,"available":96576380928,"internally_reserved":0,"allocated":27070464,"data_stored":31487811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13968,"internal_metadata":33278320},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.20499999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45000000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33800000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.307}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42399999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.442}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":19070976,"data_stored":20811093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5973,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12279808,"data_stored":13273764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":7778304,"data_stored":8899678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":33562624,"data_stored":46709369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":29290496,"data_stored":32745123,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":17555456,"data_stored":21441417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8208384,"data_stored":9735893,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":17358848,"data_stored":18427482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T15:51:06.455 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-21T15:51:06.455 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T15:51:06.455 INFO:tasks.thrashosds.thrasher:Adding osd 3 2024-09-21T15:51:06.455 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd in 3 2024-09-21T15:51:06.456 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T15:51:06.536 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:51:06.535+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:51:06.585 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:51:06.598 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:06.598 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:06.607 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:51:06.637 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:51:06.636+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:51:06.708 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-09-21T15:51:06.708 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:51:06.724 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:51:06.724 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:51:06.735 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:06.735 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:51:06.736 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:51:06.736 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:06.738 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:51:06.737+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:51:06.740 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:51:06.740 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:51:06.745 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:51:06.757 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:51:06.757 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:51:06.774 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:51:06.774 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:51:06.783 INFO:tasks.ceph.mon.a.smithi089.stderr:2024-09-21T15:51:06.782+0000 7f17447bb640 -1 mon.a@0(leader).osd e130 definitely_dead 0 2024-09-21T15:51:06.790 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:51:06.791 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:51:06.807 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:51:06.807 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:51:06.823 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is set 2024-09-21T15:51:06.825 INFO:teuthology.orchestra.run.smithi089.stderr:marked in osd.3. 2024-09-21T15:51:06.830 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:51:06.830 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:51:06.839 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:51:06.838+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:51:06.847 INFO:tasks.thrashosds.thrasher:Added osd 3 2024-09-21T15:51:06.863 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:06.864 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:51:06.864 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:51:06.864 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:51:06.864 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:06.873 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:51:06.940 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:51:06.939+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:51:07.017 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:07.018 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:51:07.018 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:51:07.018 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:51:07.018 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.018 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160505 3.b 3:d297695c:::rbd_header.11338bae974a:head [watch reconnect cookie 140381478598448 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-21T15:51:07.018 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:29.138810+0000", 2024-09-21T15:51:07.018 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 37.876675509999998, 2024-09-21T15:51:07.018 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0172598319999999, 2024-09-21T15:51:07.018 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:07.018 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:07.018 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:07.019 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:07.019 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:07.019 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160505 2024-09-21T15:51:07.019 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.019 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:07.019 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.019 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:07.019 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138810+0000", 2024-09-21T15:51:07.019 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.019 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.019 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.019 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:07.019 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138810+0000", 2024-09-21T15:51:07.020 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.020 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.020 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.020 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:07.020 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138808+0000", 2024-09-21T15:51:07.020 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:51:07.020 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.020 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.020 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:07.020 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138815+0000", 2024-09-21T15:51:07.020 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6549999999999998e-06 2024-09-21T15:51:07.020 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.020 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.021 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:07.021 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138825+0000", 2024-09-21T15:51:07.021 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9790000000000005e-06 2024-09-21T15:51:07.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.021 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.021 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:07.021 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138834+0000", 2024-09-21T15:51:07.021 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5179999999999993e-06 2024-09-21T15:51:07.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.021 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.021 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:07.021 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.140973+0000", 2024-09-21T15:51:07.021 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0021392049999999999 2024-09-21T15:51:07.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.022 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.022 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:51:07.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.140981+0000", 2024-09-21T15:51:07.022 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8909999999999995e-06 2024-09-21T15:51:07.022 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.022 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.022 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:07.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.150457+0000", 2024-09-21T15:51:07.022 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.009475645 2024-09-21T15:51:07.022 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.022 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.022 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for missing object", 2024-09-21T15:51:07.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.150589+0000", 2024-09-21T15:51:07.022 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000131779 2024-09-21T15:51:07.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.023 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.023 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:07.023 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.151723+0000", 2024-09-21T15:51:07.023 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0011345979999999999 2024-09-21T15:51:07.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.023 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.023 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-09-21T15:51:07.023 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.151872+0000", 2024-09-21T15:51:07.023 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000149021 2024-09-21T15:51:07.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.023 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.023 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:07.023 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.152563+0000", 2024-09-21T15:51:07.024 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00069051500000000005 2024-09-21T15:51:07.024 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.024 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.024 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:07.024 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.152673+0000", 2024-09-21T15:51:07.024 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000109795 2024-09-21T15:51:07.024 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.024 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.024 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:07.024 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.156069+0000", 2024-09-21T15:51:07.024 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003396736 2024-09-21T15:51:07.024 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.024 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:07.024 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.025 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.025 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.025 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160506 3.b 3:d297695c:::rbd_header.11338bae974a:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-21T15:51:07.025 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:29.138844+0000", 2024-09-21T15:51:07.025 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 37.876640623, 2024-09-21T15:51:07.025 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0174592680000001, 2024-09-21T15:51:07.025 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:07.025 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:07.025 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:07.025 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:07.025 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:07.025 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160506 2024-09-21T15:51:07.025 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.026 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:07.026 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.026 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:07.026 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138844+0000", 2024-09-21T15:51:07.026 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.026 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.026 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.026 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:07.026 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138844+0000", 2024-09-21T15:51:07.026 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.026 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.026 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.026 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:07.026 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138844+0000", 2024-09-21T15:51:07.027 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:07.027 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.027 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.027 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:07.027 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138848+0000", 2024-09-21T15:51:07.027 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.76e-06 2024-09-21T15:51:07.027 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.027 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.027 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:07.027 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138854+0000", 2024-09-21T15:51:07.027 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3890000000000002e-06 2024-09-21T15:51:07.027 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.027 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.028 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:07.028 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138860+0000", 2024-09-21T15:51:07.028 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6409999999999999e-06 2024-09-21T15:51:07.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.028 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.028 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:07.028 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.141015+0000", 2024-09-21T15:51:07.028 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0021553779999999999 2024-09-21T15:51:07.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.028 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.028 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:51:07.029 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.141021+0000", 2024-09-21T15:51:07.029 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7629999999999997e-06 2024-09-21T15:51:07.029 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.029 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.029 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:07.029 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.150858+0000", 2024-09-21T15:51:07.029 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0098372200000001 2024-09-21T15:51:07.029 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.029 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.029 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for missing object", 2024-09-21T15:51:07.029 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.150899+0000", 2024-09-21T15:51:07.029 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1112000000000002e-05 2024-09-21T15:51:07.029 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.029 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.030 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:07.030 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.152069+0000", 2024-09-21T15:51:07.030 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001170307 2024-09-21T15:51:07.030 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.030 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.030 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-09-21T15:51:07.030 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.152192+0000", 2024-09-21T15:51:07.030 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012289700000000001 2024-09-21T15:51:07.030 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.030 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.030 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:07.030 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.153362+0000", 2024-09-21T15:51:07.030 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0011700549999999999 2024-09-21T15:51:07.031 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.031 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.031 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:07.031 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.153457+0000", 2024-09-21T15:51:07.031 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4687999999999999e-05 2024-09-21T15:51:07.031 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.031 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.031 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:51:07.031 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.153893+0000", 2024-09-21T15:51:07.031 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000436363 2024-09-21T15:51:07.031 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.031 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.031 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:07.031 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.156134+0000", 2024-09-21T15:51:07.031 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0022410009999999998 2024-09-21T15:51:07.032 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.032 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.032 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:07.032 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.156148+0000", 2024-09-21T15:51:07.032 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3574000000000001e-05 2024-09-21T15:51:07.032 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.032 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.032 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:07.032 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.156214+0000", 2024-09-21T15:51:07.032 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6438000000000003e-05 2024-09-21T15:51:07.032 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.032 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.033 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:07.033 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.156304+0000", 2024-09-21T15:51:07.033 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9278999999999997e-05 2024-09-21T15:51:07.033 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.033 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:07.033 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.033 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.033 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.033 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160512 3.b 3:d297695c:::rbd_header.11338bae974a:head [watch unwatch cookie 140381478598448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T15:51:07.033 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:30.161362+0000", 2024-09-21T15:51:07.033 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 36.854122814999997, 2024-09-21T15:51:07.033 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0074105029999999997, 2024-09-21T15:51:07.033 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:07.033 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:07.034 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:07.034 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:07.034 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:07.034 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160512 2024-09-21T15:51:07.034 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.034 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:07.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.034 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:07.034 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161362+0000", 2024-09-21T15:51:07.034 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.034 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.034 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:07.034 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161362+0000", 2024-09-21T15:51:07.035 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.035 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:07.035 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161362+0000", 2024-09-21T15:51:07.035 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:07.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.035 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:07.035 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161365+0000", 2024-09-21T15:51:07.035 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8669999999999998e-06 2024-09-21T15:51:07.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.035 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:07.036 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161371+0000", 2024-09-21T15:51:07.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8009999999999998e-06 2024-09-21T15:51:07.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.036 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:07.036 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161376+0000", 2024-09-21T15:51:07.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0340000000000004e-06 2024-09-21T15:51:07.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.036 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:07.036 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161762+0000", 2024-09-21T15:51:07.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038575099999999999 2024-09-21T15:51:07.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.036 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:07.037 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161838+0000", 2024-09-21T15:51:07.037 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5697000000000004e-05 2024-09-21T15:51:07.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.037 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:51:07.037 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161952+0000", 2024-09-21T15:51:07.037 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000114034 2024-09-21T15:51:07.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.037 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:07.037 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.163727+0000", 2024-09-21T15:51:07.037 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0017749560000000001 2024-09-21T15:51:07.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.037 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:07.038 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168545+0000", 2024-09-21T15:51:07.038 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0048186080000000003 2024-09-21T15:51:07.038 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.038 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.038 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:07.038 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168593+0000", 2024-09-21T15:51:07.038 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7549999999999997e-05 2024-09-21T15:51:07.038 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.038 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.038 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:07.038 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168773+0000", 2024-09-21T15:51:07.038 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00017976600000000001 2024-09-21T15:51:07.039 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.039 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:07.039 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.039 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.039 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.039 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160513 3.b 3:d297695c:::rbd_header.11338bae974a:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-09-21T15:51:07.039 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:30.161755+0000", 2024-09-21T15:51:07.039 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 36.853729884000003, 2024-09-21T15:51:07.039 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0072250300000000003, 2024-09-21T15:51:07.039 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:07.039 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:07.039 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:07.039 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:07.039 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:07.040 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160513 2024-09-21T15:51:07.040 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.040 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:07.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.040 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:07.040 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161755+0000", 2024-09-21T15:51:07.040 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.040 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.040 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:07.040 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161755+0000", 2024-09-21T15:51:07.040 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.040 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.040 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:07.041 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161754+0000", 2024-09-21T15:51:07.041 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:07.041 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.041 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.041 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:07.041 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161759+0000", 2024-09-21T15:51:07.041 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9699999999999998e-06 2024-09-21T15:51:07.041 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.041 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.041 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:07.041 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161768+0000", 2024-09-21T15:51:07.041 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9949999999999994e-06 2024-09-21T15:51:07.041 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.041 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.042 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:07.042 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161774+0000", 2024-09-21T15:51:07.042 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1290000000000001e-06 2024-09-21T15:51:07.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.042 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.042 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:07.042 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.162239+0000", 2024-09-21T15:51:07.042 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00046406600000000001 2024-09-21T15:51:07.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.042 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.042 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-09-21T15:51:07.042 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.162296+0000", 2024-09-21T15:51:07.042 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7122e-05 2024-09-21T15:51:07.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.043 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.043 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:07.043 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168802+0000", 2024-09-21T15:51:07.043 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0065061700000000004 2024-09-21T15:51:07.043 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.043 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.043 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:07.043 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168877+0000", 2024-09-21T15:51:07.043 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5214000000000002e-05 2024-09-21T15:51:07.043 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.043 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.043 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:07.043 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168980+0000", 2024-09-21T15:51:07.043 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000103182 2024-09-21T15:51:07.043 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.044 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:07.044 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.044 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160527 3.b 3:d297695c:::rbd_header.11338bae974a:head [watch watch cookie 140381476148256] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T15:51:07.044 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:30.182756+0000", 2024-09-21T15:51:07.044 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 36.832729303000001, 2024-09-21T15:51:07.044 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0054202579999999998, 2024-09-21T15:51:07.044 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:07.044 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:07.044 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:07.044 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:07.044 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:07.044 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160527 2024-09-21T15:51:07.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.044 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:07.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:07.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182756+0000", 2024-09-21T15:51:07.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:07.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182756+0000", 2024-09-21T15:51:07.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:07.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182755+0000", 2024-09-21T15:51:07.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:07.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.046 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:07.046 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182760+0000", 2024-09-21T15:51:07.046 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4269999999999999e-06 2024-09-21T15:51:07.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.046 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:07.046 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182771+0000", 2024-09-21T15:51:07.046 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0312999999999999e-05 2024-09-21T15:51:07.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.046 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:07.046 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182778+0000", 2024-09-21T15:51:07.046 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5700000000000004e-06 2024-09-21T15:51:07.047 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.047 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.047 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:07.047 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.184745+0000", 2024-09-21T15:51:07.047 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0019667170000000002 2024-09-21T15:51:07.047 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.047 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.047 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:07.047 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.184867+0000", 2024-09-21T15:51:07.047 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000122504 2024-09-21T15:51:07.047 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.047 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.047 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:51:07.047 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.185135+0000", 2024-09-21T15:51:07.048 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000267135 2024-09-21T15:51:07.048 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.048 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.048 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:07.048 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.187721+0000", 2024-09-21T15:51:07.048 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002586891 2024-09-21T15:51:07.048 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.048 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.048 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:07.048 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.187874+0000", 2024-09-21T15:51:07.048 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000152726 2024-09-21T15:51:07.048 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.048 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.048 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:07.048 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.187951+0000", 2024-09-21T15:51:07.049 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7132000000000006e-05 2024-09-21T15:51:07.049 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.049 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.049 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:07.049 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.188176+0000", 2024-09-21T15:51:07.049 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00022477900000000001 2024-09-21T15:51:07.049 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.049 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:07.049 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.049 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.049 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.049 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160605 3.b 3:d1863919:::rbd_data.11338bae974a.0000000000000008:head [delete] snapc 2d=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-21T15:51:07.049 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.213794+0000", 2024-09-21T15:51:07.049 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 35.801691423000001, 2024-09-21T15:51:07.049 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046152490000000001, 2024-09-21T15:51:07.050 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:07.050 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:07.050 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:07.050 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:07.050 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:07.050 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160605 2024-09-21T15:51:07.050 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.050 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:07.050 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.050 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:07.050 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.213794+0000", 2024-09-21T15:51:07.050 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.050 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.050 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.051 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:07.051 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.213794+0000", 2024-09-21T15:51:07.051 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.051 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.051 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:07.051 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.213793+0000", 2024-09-21T15:51:07.051 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:07.051 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.051 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.051 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:07.051 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.213799+0000", 2024-09-21T15:51:07.051 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9959999999999999e-06 2024-09-21T15:51:07.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.052 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.052 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:07.052 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.213815+0000", 2024-09-21T15:51:07.052 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5880999999999999e-05 2024-09-21T15:51:07.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.052 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.052 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:07.052 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.213838+0000", 2024-09-21T15:51:07.052 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3671000000000001e-05 2024-09-21T15:51:07.052 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.052 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.052 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:07.052 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.213887+0000", 2024-09-21T15:51:07.053 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8865000000000001e-05 2024-09-21T15:51:07.053 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.053 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.053 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:07.053 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.213962+0000", 2024-09-21T15:51:07.053 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4592999999999995e-05 2024-09-21T15:51:07.053 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.053 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.053 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:51:07.053 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.214088+0000", 2024-09-21T15:51:07.053 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012666199999999999 2024-09-21T15:51:07.053 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.053 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.053 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:07.054 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216917+0000", 2024-09-21T15:51:07.054 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0028284059999999999 2024-09-21T15:51:07.054 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.054 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.054 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:07.054 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.218319+0000", 2024-09-21T15:51:07.054 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001402613 2024-09-21T15:51:07.054 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.054 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.054 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:07.054 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.218372+0000", 2024-09-21T15:51:07.054 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2636000000000002e-05 2024-09-21T15:51:07.054 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.054 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.055 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:07.055 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.218409+0000", 2024-09-21T15:51:07.055 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6942e-05 2024-09-21T15:51:07.055 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.055 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:07.055 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.055 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.055 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.055 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:160597 3.1 e95/94 3:84154dfc:::rbd_data.11338bae974a.0000000000000000:head v 95'451, mlcod=0'0)", 2024-09-21T15:51:07.055 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.215080+0000", 2024-09-21T15:51:07.055 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 35.800405550000001, 2024-09-21T15:51:07.055 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045596819999999998, 2024-09-21T15:51:07.055 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:07.055 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:07.056 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:07.056 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.056 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:07.056 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215080+0000", 2024-09-21T15:51:07.056 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.056 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.056 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:07.056 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215080+0000", 2024-09-21T15:51:07.056 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.056 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.056 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.056 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:07.056 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215079+0000", 2024-09-21T15:51:07.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:07.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.057 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:07.057 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215083+0000", 2024-09-21T15:51:07.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2570000000000001e-06 2024-09-21T15:51:07.057 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.057 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:07.057 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215091+0000", 2024-09-21T15:51:07.058 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7829999999999999e-06 2024-09-21T15:51:07.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.058 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:07.058 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215096+0000", 2024-09-21T15:51:07.058 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0370000000000002e-06 2024-09-21T15:51:07.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.058 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:07.058 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216055+0000", 2024-09-21T15:51:07.058 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00095892299999999998 2024-09-21T15:51:07.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.059 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:07.059 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216078+0000", 2024-09-21T15:51:07.059 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2376000000000001e-05 2024-09-21T15:51:07.059 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.059 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.059 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:07.059 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.219598+0000", 2024-09-21T15:51:07.059 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035209299999999998 2024-09-21T15:51:07.059 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.059 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.059 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:07.059 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.219639+0000", 2024-09-21T15:51:07.059 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0766000000000002e-05 2024-09-21T15:51:07.059 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.060 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:07.060 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.060 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.060 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.060 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:160608 3.12 e95/94 3:48a47e1c:::rbd_data.11338bae974a.000000000000000b:head v 95'345, mlcod=0'0)", 2024-09-21T15:51:07.060 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.216934+0000", 2024-09-21T15:51:07.060 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 35.798551164999999, 2024-09-21T15:51:07.060 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053335759999999996, 2024-09-21T15:51:07.060 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:07.060 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:07.060 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:07.060 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.060 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:07.060 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216934+0000", 2024-09-21T15:51:07.061 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.061 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:07.061 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216934+0000", 2024-09-21T15:51:07.061 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.061 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:07.061 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216933+0000", 2024-09-21T15:51:07.061 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:07.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.061 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:07.062 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216939+0000", 2024-09-21T15:51:07.062 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6459999999999998e-06 2024-09-21T15:51:07.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.062 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:07.062 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216947+0000", 2024-09-21T15:51:07.062 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.2479999999999996e-06 2024-09-21T15:51:07.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.062 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:07.062 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216951+0000", 2024-09-21T15:51:07.062 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3760000000000001e-06 2024-09-21T15:51:07.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.063 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:07.063 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216998+0000", 2024-09-21T15:51:07.063 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6244e-05 2024-09-21T15:51:07.063 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.063 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.063 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:07.063 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.217019+0000", 2024-09-21T15:51:07.063 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1191000000000001e-05 2024-09-21T15:51:07.063 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.063 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.063 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:07.063 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.222230+0000", 2024-09-21T15:51:07.063 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052114350000000004 2024-09-21T15:51:07.063 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.064 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.064 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:07.064 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.222267+0000", 2024-09-21T15:51:07.064 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7169999999999998e-05 2024-09-21T15:51:07.064 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.064 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:07.064 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.064 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.064 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.064 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160618 3.b 3:d297695c:::rbd_header.11338bae974a:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-21T15:51:07.064 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.233457+0000", 2024-09-21T15:51:07.064 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 35.782028328000003, 2024-09-21T15:51:07.064 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.005085198, 2024-09-21T15:51:07.064 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:07.065 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:07.065 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:07.065 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:07.065 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:07.065 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160618 2024-09-21T15:51:07.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.065 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:07.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.065 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:07.065 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233457+0000", 2024-09-21T15:51:07.065 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.065 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:07.066 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233457+0000", 2024-09-21T15:51:07.066 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.066 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.066 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:07.066 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233456+0000", 2024-09-21T15:51:07.066 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:07.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.066 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.066 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:07.066 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233463+0000", 2024-09-21T15:51:07.066 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9310000000000002e-06 2024-09-21T15:51:07.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.067 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:07.067 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233473+0000", 2024-09-21T15:51:07.067 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0519e-05 2024-09-21T15:51:07.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.067 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:07.067 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233481+0000", 2024-09-21T15:51:07.067 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4009999999999997e-06 2024-09-21T15:51:07.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.067 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:07.067 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233513+0000", 2024-09-21T15:51:07.068 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1328999999999998e-05 2024-09-21T15:51:07.068 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.068 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.068 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:07.068 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233568+0000", 2024-09-21T15:51:07.068 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5606999999999998e-05 2024-09-21T15:51:07.068 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.068 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.068 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:51:07.068 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233670+0000", 2024-09-21T15:51:07.068 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010180599999999999 2024-09-21T15:51:07.068 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.068 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.068 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:07.069 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.234761+0000", 2024-09-21T15:51:07.069 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010905909999999999 2024-09-21T15:51:07.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.069 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:07.069 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.238442+0000", 2024-09-21T15:51:07.069 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0036809429999999999 2024-09-21T15:51:07.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.069 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:07.069 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.238498+0000", 2024-09-21T15:51:07.069 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6116999999999998e-05 2024-09-21T15:51:07.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.069 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:07.069 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.238542+0000", 2024-09-21T15:51:07.070 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4134000000000001e-05 2024-09-21T15:51:07.070 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.070 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:07.070 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.070 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.070 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160662 3.b 3:d7701915:::rbd_object_map.1133fd7483d4.000000000000002c:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-21T15:51:07.070 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.279369+0000", 2024-09-21T15:51:07.070 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 35.736116398, 2024-09-21T15:51:07.070 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0056948040000000004, 2024-09-21T15:51:07.070 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:07.070 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:07.070 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:07.070 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:07.071 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:07.071 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160662 2024-09-21T15:51:07.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.071 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:07.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.071 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:07.071 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279369+0000", 2024-09-21T15:51:07.071 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.071 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:07.071 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279369+0000", 2024-09-21T15:51:07.071 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.072 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.072 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:07.072 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279368+0000", 2024-09-21T15:51:07.072 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:07.072 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.072 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.072 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:07.072 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279373+0000", 2024-09-21T15:51:07.072 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6529999999999999e-06 2024-09-21T15:51:07.072 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.072 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.072 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:07.072 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279384+0000", 2024-09-21T15:51:07.072 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0256e-05 2024-09-21T15:51:07.072 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.072 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.073 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:07.073 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279388+0000", 2024-09-21T15:51:07.073 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5630000000000002e-06 2024-09-21T15:51:07.073 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.073 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.073 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:07.073 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279424+0000", 2024-09-21T15:51:07.073 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5985999999999997e-05 2024-09-21T15:51:07.073 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.073 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.073 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:07.073 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279478+0000", 2024-09-21T15:51:07.073 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3944999999999999e-05 2024-09-21T15:51:07.073 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.074 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.074 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:51:07.074 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279575+0000", 2024-09-21T15:51:07.074 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6817999999999994e-05 2024-09-21T15:51:07.074 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.074 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.074 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:07.074 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.282391+0000", 2024-09-21T15:51:07.074 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0028156700000000002 2024-09-21T15:51:07.074 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.074 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.074 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:07.074 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.284977+0000", 2024-09-21T15:51:07.074 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0025859640000000001 2024-09-21T15:51:07.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:07.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.285021+0000", 2024-09-21T15:51:07.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4415000000000001e-05 2024-09-21T15:51:07.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:07.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.285063+0000", 2024-09-21T15:51:07.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2447000000000002e-05 2024-09-21T15:51:07.075 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.075 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:07.075 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.076 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:161122 3.1 3:82ccd16f:::rbd_header.11335aef9442:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-21T15:51:07.076 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:36.703081+0000", 2024-09-21T15:51:07.076 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 30.312404234999999, 2024-09-21T15:51:07.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0051212200000000001, 2024-09-21T15:51:07.076 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:07.076 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:07.076 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:07.076 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:07.076 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:07.076 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 161122 2024-09-21T15:51:07.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.076 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:07.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.077 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:07.077 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.703081+0000", 2024-09-21T15:51:07.077 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.077 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.077 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:07.077 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.703081+0000", 2024-09-21T15:51:07.077 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.077 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.077 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:07.077 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.703079+0000", 2024-09-21T15:51:07.077 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-09-21T15:51:07.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.077 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.078 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:07.078 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.703091+0000", 2024-09-21T15:51:07.078 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2584e-05 2024-09-21T15:51:07.078 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.078 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:07.078 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.703107+0000", 2024-09-21T15:51:07.078 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6313000000000001e-05 2024-09-21T15:51:07.078 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.078 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:07.078 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.703119+0000", 2024-09-21T15:51:07.078 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1404e-05 2024-09-21T15:51:07.078 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.079 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.079 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:07.079 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.703176+0000", 2024-09-21T15:51:07.079 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6740999999999998e-05 2024-09-21T15:51:07.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.079 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.079 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:07.079 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.703281+0000", 2024-09-21T15:51:07.079 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010508 2024-09-21T15:51:07.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.079 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.079 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-09-21T15:51:07.079 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.703483+0000", 2024-09-21T15:51:07.079 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00020257399999999999 2024-09-21T15:51:07.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.080 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:07.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.705657+0000", 2024-09-21T15:51:07.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0021735610000000001 2024-09-21T15:51:07.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.080 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:07.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.708144+0000", 2024-09-21T15:51:07.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002487122 2024-09-21T15:51:07.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.080 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:07.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.708177+0000", 2024-09-21T15:51:07.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2574999999999999e-05 2024-09-21T15:51:07.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.081 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:07.081 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.708202+0000", 2024-09-21T15:51:07.081 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5513000000000001e-05 2024-09-21T15:51:07.081 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.081 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:07.081 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.081 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:161219 3.17 3:e87cb47d:::rbd_data.11335aef9442.000000000000000a:head [delete] snapc 2a=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-21T15:51:07.081 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:36.771292+0000", 2024-09-21T15:51:07.081 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 30.244192586, 2024-09-21T15:51:07.081 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0058540529999999997, 2024-09-21T15:51:07.082 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:07.082 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:07.082 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:07.082 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:07.082 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:07.082 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 161219 2024-09-21T15:51:07.082 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.082 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:07.082 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.082 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:07.082 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771292+0000", 2024-09-21T15:51:07.082 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.082 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.082 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.083 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:07.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771292+0000", 2024-09-21T15:51:07.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.083 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:07.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771292+0000", 2024-09-21T15:51:07.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:07.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.083 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:07.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771298+0000", 2024-09-21T15:51:07.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9519999999999999e-06 2024-09-21T15:51:07.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.084 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:07.084 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771308+0000", 2024-09-21T15:51:07.084 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0601000000000001e-05 2024-09-21T15:51:07.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.084 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:07.084 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771314+0000", 2024-09-21T15:51:07.084 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.913e-06 2024-09-21T15:51:07.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.084 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:07.085 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771346+0000", 2024-09-21T15:51:07.085 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1711000000000002e-05 2024-09-21T15:51:07.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.085 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.085 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:07.085 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771399+0000", 2024-09-21T15:51:07.085 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2883000000000002e-05 2024-09-21T15:51:07.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.085 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.085 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-09-21T15:51:07.085 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771470+0000", 2024-09-21T15:51:07.085 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0934999999999998e-05 2024-09-21T15:51:07.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.085 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.085 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:07.086 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.772234+0000", 2024-09-21T15:51:07.086 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00076485400000000003 2024-09-21T15:51:07.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.086 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:07.086 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.777083+0000", 2024-09-21T15:51:07.086 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0048486329999999998 2024-09-21T15:51:07.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.086 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:07.086 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.777111+0000", 2024-09-21T15:51:07.086 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8399999999999999e-05 2024-09-21T15:51:07.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.087 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:07.087 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.777147+0000", 2024-09-21T15:51:07.087 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5049000000000003e-05 2024-09-21T15:51:07.087 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.087 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:07.087 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.087 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.087 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.087 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:161762 3.17 3:ef707828:::rbd_header.11338db70e6f:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T15:51:07.087 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:40.705291+0000", 2024-09-21T15:51:07.087 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 26.310194536000001, 2024-09-21T15:51:07.087 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0048878120000000001, 2024-09-21T15:51:07.087 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:07.087 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:07.088 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:07.088 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:07.088 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:07.088 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 161762 2024-09-21T15:51:07.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.088 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:07.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.088 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:07.088 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.705291+0000", 2024-09-21T15:51:07.088 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.088 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:07.088 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.705291+0000", 2024-09-21T15:51:07.088 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.089 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.089 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:07.089 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.705289+0000", 2024-09-21T15:51:07.089 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:51:07.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.089 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.089 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:07.089 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.705296+0000", 2024-09-21T15:51:07.089 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1999999999999997e-06 2024-09-21T15:51:07.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.089 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.089 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:07.089 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.705309+0000", 2024-09-21T15:51:07.089 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2723e-05 2024-09-21T15:51:07.090 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.090 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.090 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:07.090 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.705318+0000", 2024-09-21T15:51:07.090 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4890000000000001e-06 2024-09-21T15:51:07.090 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.090 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.090 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:07.090 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.705370+0000", 2024-09-21T15:51:07.090 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1493000000000003e-05 2024-09-21T15:51:07.090 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.090 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.090 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:07.090 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.705480+0000", 2024-09-21T15:51:07.091 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011059 2024-09-21T15:51:07.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.091 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-09-21T15:51:07.091 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.705800+0000", 2024-09-21T15:51:07.091 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00031958399999999998 2024-09-21T15:51:07.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.091 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:07.091 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.707140+0000", 2024-09-21T15:51:07.091 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001339954 2024-09-21T15:51:07.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.091 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:07.091 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.710103+0000", 2024-09-21T15:51:07.092 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0029629109999999999 2024-09-21T15:51:07.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.092 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:07.092 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.710136+0000", 2024-09-21T15:51:07.092 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3300000000000003e-05 2024-09-21T15:51:07.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.092 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:07.092 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.710178+0000", 2024-09-21T15:51:07.092 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2172999999999998e-05 2024-09-21T15:51:07.092 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.092 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:07.092 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.093 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.093 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:161859 3.17 3:ee7ab022:::rbd_data.11338db70e6f.0000000000000007:head [delete] snapc 26=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T15:51:07.093 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:40.782264+0000", 2024-09-21T15:51:07.093 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 26.233220726999999, 2024-09-21T15:51:07.093 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052663939999999998, 2024-09-21T15:51:07.093 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:07.093 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:07.093 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:07.093 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:07.093 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:07.093 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 161859 2024-09-21T15:51:07.093 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.093 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:07.093 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.093 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:07.094 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782264+0000", 2024-09-21T15:51:07.094 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.094 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.094 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:07.094 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782264+0000", 2024-09-21T15:51:07.094 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.094 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.094 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:07.094 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782264+0000", 2024-09-21T15:51:07.094 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:07.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.094 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.094 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:07.095 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782266+0000", 2024-09-21T15:51:07.095 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4590000000000001e-06 2024-09-21T15:51:07.095 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.095 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:07.095 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782275+0000", 2024-09-21T15:51:07.095 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5450000000000003e-06 2024-09-21T15:51:07.095 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.095 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:07.095 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782281+0000", 2024-09-21T15:51:07.095 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7180000000000001e-06 2024-09-21T15:51:07.095 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:07.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782312+0000", 2024-09-21T15:51:07.096 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1059000000000002e-05 2024-09-21T15:51:07.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:07.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782470+0000", 2024-09-21T15:51:07.096 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000157818 2024-09-21T15:51:07.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-09-21T15:51:07.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782574+0000", 2024-09-21T15:51:07.096 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000104536 2024-09-21T15:51:07.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:07.097 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.783499+0000", 2024-09-21T15:51:07.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00092479599999999997 2024-09-21T15:51:07.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.097 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:07.097 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.787472+0000", 2024-09-21T15:51:07.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0039730319999999996 2024-09-21T15:51:07.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.097 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:07.097 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.787507+0000", 2024-09-21T15:51:07.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5552000000000001e-05 2024-09-21T15:51:07.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:07.098 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.787531+0000", 2024-09-21T15:51:07.098 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3281999999999998e-05 2024-09-21T15:51:07.098 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.098 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:07.098 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.098 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.098 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:162390 3.17 3:ea8605c7:::rbd_header.1133fc941201:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-09-21T15:51:07.098 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:44.740587+0000", 2024-09-21T15:51:07.098 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 22.274898131, 2024-09-21T15:51:07.098 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052218530000000003, 2024-09-21T15:51:07.098 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:07.098 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:07.098 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:07.099 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:07.099 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:07.099 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 162390 2024-09-21T15:51:07.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.099 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:07.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.099 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:07.099 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.740587+0000", 2024-09-21T15:51:07.099 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.099 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:07.099 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.740587+0000", 2024-09-21T15:51:07.099 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.100 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:07.100 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.740585+0000", 2024-09-21T15:51:07.100 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:51:07.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.100 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:07.100 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.740595+0000", 2024-09-21T15:51:07.100 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7839999999999998e-06 2024-09-21T15:51:07.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.100 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:07.100 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.740609+0000", 2024-09-21T15:51:07.100 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.384e-05 2024-09-21T15:51:07.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.101 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:07.101 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.740617+0000", 2024-09-21T15:51:07.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7560000000000005e-06 2024-09-21T15:51:07.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.101 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:07.101 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.740671+0000", 2024-09-21T15:51:07.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4073999999999998e-05 2024-09-21T15:51:07.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.101 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:07.101 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.740799+0000", 2024-09-21T15:51:07.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000128057 2024-09-21T15:51:07.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.102 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-09-21T15:51:07.102 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.741258+0000", 2024-09-21T15:51:07.102 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00045892900000000001 2024-09-21T15:51:07.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.102 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:07.102 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.742101+0000", 2024-09-21T15:51:07.102 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00084276499999999999 2024-09-21T15:51:07.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.102 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:07.102 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.745743+0000", 2024-09-21T15:51:07.103 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0036415280000000002 2024-09-21T15:51:07.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.103 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:07.103 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.745781+0000", 2024-09-21T15:51:07.103 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8398000000000001e-05 2024-09-21T15:51:07.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.103 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:07.103 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.745809+0000", 2024-09-21T15:51:07.103 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7691999999999998e-05 2024-09-21T15:51:07.103 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.103 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:07.103 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.104 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:163005 3.6 3:67cae2ed:::rbd_mirroring:head [call rbd.mirror_image_remove in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-21T15:51:07.104 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:49.711037+0000", 2024-09-21T15:51:07.104 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 17.304447707000001, 2024-09-21T15:51:07.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0059230919999999996, 2024-09-21T15:51:07.104 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:07.104 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:07.104 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:07.104 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:07.104 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:07.104 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 163005 2024-09-21T15:51:07.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.104 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:07.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.105 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:07.105 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:49.711037+0000", 2024-09-21T15:51:07.105 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.105 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:07.105 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:49.711037+0000", 2024-09-21T15:51:07.105 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.105 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:07.105 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:49.711037+0000", 2024-09-21T15:51:07.105 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:07.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.106 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:07.106 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:49.711041+0000", 2024-09-21T15:51:07.106 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8199999999999996e-06 2024-09-21T15:51:07.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.106 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:07.106 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:49.711052+0000", 2024-09-21T15:51:07.106 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0567999999999999e-05 2024-09-21T15:51:07.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.106 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:07.106 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:49.711057+0000", 2024-09-21T15:51:07.106 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7119999999999999e-06 2024-09-21T15:51:07.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.107 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:07.107 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:49.711096+0000", 2024-09-21T15:51:07.107 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8872999999999999e-05 2024-09-21T15:51:07.107 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.107 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.107 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:07.107 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:49.711151+0000", 2024-09-21T15:51:07.107 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5343999999999997e-05 2024-09-21T15:51:07.107 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.107 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.107 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:07.107 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:49.716960+0000", 2024-09-21T15:51:07.107 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0058095150000000003 2024-09-21T15:51:07.107 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.107 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:07.108 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.108 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.108 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.108 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:163051 3.6 3:64ca7355:::rbd_object_map.1133545d3fa6:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e115)", 2024-09-21T15:51:07.108 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:50.680972+0000", 2024-09-21T15:51:07.108 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 16.334512833000002, 2024-09-21T15:51:07.108 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.004798353, 2024-09-21T15:51:07.108 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:07.108 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:07.108 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:07.108 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:07.108 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:07.108 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 163051 2024-09-21T15:51:07.108 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.109 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:07.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.109 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:07.109 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:50.680972+0000", 2024-09-21T15:51:07.109 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.109 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:07.109 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:50.680972+0000", 2024-09-21T15:51:07.109 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.109 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:07.109 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:50.680972+0000", 2024-09-21T15:51:07.109 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:07.110 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.110 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.110 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:07.110 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:50.680976+0000", 2024-09-21T15:51:07.110 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6430000000000003e-06 2024-09-21T15:51:07.110 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.110 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.110 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:07.110 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:50.680993+0000", 2024-09-21T15:51:07.110 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.7181e-05 2024-09-21T15:51:07.110 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.110 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.110 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:07.110 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:50.680998+0000", 2024-09-21T15:51:07.110 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8389999999999997e-06 2024-09-21T15:51:07.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.111 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:07.111 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:50.681035+0000", 2024-09-21T15:51:07.111 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6386e-05 2024-09-21T15:51:07.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.111 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:07.111 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:50.681114+0000", 2024-09-21T15:51:07.111 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9453e-05 2024-09-21T15:51:07.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.111 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 1", 2024-09-21T15:51:07.111 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:50.681279+0000", 2024-09-21T15:51:07.111 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00016474600000000001 2024-09-21T15:51:07.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.112 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:07.112 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:50.682375+0000", 2024-09-21T15:51:07.112 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010965969999999999 2024-09-21T15:51:07.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.112 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:07.112 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:50.685707+0000", 2024-09-21T15:51:07.112 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033312329999999999 2024-09-21T15:51:07.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.112 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:07.112 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:50.685743+0000", 2024-09-21T15:51:07.112 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.6474999999999999e-05 2024-09-21T15:51:07.113 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.113 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:07.113 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:50.685771+0000", 2024-09-21T15:51:07.113 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7504e-05 2024-09-21T15:51:07.113 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.113 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:07.113 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.113 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.113 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:165147 3.10 e131/96 3:0b23e447:::rbd_header.11339e87a32a:head v 131'448, mlcod=130'446)", 2024-09-21T15:51:07.113 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:51:06.827072+0000", 2024-09-21T15:51:07.113 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.18841292300000001, 2024-09-21T15:51:07.113 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.011683913000000001, 2024-09-21T15:51:07.114 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:07.114 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:07.114 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:07.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.114 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:07.114 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.827072+0000", 2024-09-21T15:51:07.114 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.114 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.114 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:07.114 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.827072+0000", 2024-09-21T15:51:07.114 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.114 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.114 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:07.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.827072+0000", 2024-09-21T15:51:07.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:07.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:07.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.827078+0000", 2024-09-21T15:51:07.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4840000000000001e-06 2024-09-21T15:51:07.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:07.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.827094+0000", 2024-09-21T15:51:07.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5639999999999999e-05 2024-09-21T15:51:07.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:07.116 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.827104+0000", 2024-09-21T15:51:07.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.8300000000000008e-06 2024-09-21T15:51:07.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.116 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:07.116 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.827170+0000", 2024-09-21T15:51:07.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5913000000000004e-05 2024-09-21T15:51:07.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.116 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:07.116 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.827196+0000", 2024-09-21T15:51:07.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6210000000000001e-05 2024-09-21T15:51:07.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.116 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:07.117 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.838689+0000", 2024-09-21T15:51:07.117 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.011492979 2024-09-21T15:51:07.117 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.117 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.117 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:07.117 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.838756+0000", 2024-09-21T15:51:07.117 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7155000000000003e-05 2024-09-21T15:51:07.117 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.117 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:07.117 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.117 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.117 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.117 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.0:3709 3.3 e131/96 3:c5f4482f:::rbd_data.11339e87a32a.000000000000013a:12 v 131'841, mlcod=130'838)", 2024-09-21T15:51:07.117 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:51:06.828155+0000", 2024-09-21T15:51:07.117 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.18733048099999999, 2024-09-21T15:51:07.118 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.010631115999999999, 2024-09-21T15:51:07.118 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:07.118 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:07.118 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:07.118 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.118 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:07.118 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.828155+0000", 2024-09-21T15:51:07.118 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.118 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.118 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.118 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:07.118 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.828155+0000", 2024-09-21T15:51:07.118 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.118 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.119 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.119 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:07.119 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.828154+0000", 2024-09-21T15:51:07.119 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:07.119 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.119 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.119 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:07.119 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.828162+0000", 2024-09-21T15:51:07.119 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.2269999999999993e-06 2024-09-21T15:51:07.119 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.119 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.119 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:07.119 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.828188+0000", 2024-09-21T15:51:07.119 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5626000000000002e-05 2024-09-21T15:51:07.119 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.120 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.120 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:07.120 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.828210+0000", 2024-09-21T15:51:07.120 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1792000000000001e-05 2024-09-21T15:51:07.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.120 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.120 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:07.120 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.828251+0000", 2024-09-21T15:51:07.120 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1329999999999999e-05 2024-09-21T15:51:07.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.120 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.120 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:07.120 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.828277+0000", 2024-09-21T15:51:07.120 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6236000000000001e-05 2024-09-21T15:51:07.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.121 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:07.121 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.838756+0000", 2024-09-21T15:51:07.121 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.010478919 2024-09-21T15:51:07.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.121 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:07.121 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.838786+0000", 2024-09-21T15:51:07.121 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9560000000000002e-05 2024-09-21T15:51:07.121 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.121 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:07.121 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.121 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.0:3710 3.3 e131/96 3:c7ccd1db:::rbd_data.11339e87a32a.0000000000000136:12 v 131'843, mlcod=130'838)", 2024-09-21T15:51:07.122 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:51:06.831274+0000", 2024-09-21T15:51:07.122 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.18421063100000001, 2024-09-21T15:51:07.122 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0077927079999999998, 2024-09-21T15:51:07.122 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:07.122 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:07.122 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:07.122 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.122 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:07.122 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.831274+0000", 2024-09-21T15:51:07.122 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.122 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.122 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.122 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:07.122 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.831274+0000", 2024-09-21T15:51:07.123 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.123 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.123 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:07.123 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.831274+0000", 2024-09-21T15:51:07.123 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:07.123 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.123 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:07.123 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.831286+0000", 2024-09-21T15:51:07.123 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1853e-05 2024-09-21T15:51:07.123 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.123 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:07.123 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.831296+0000", 2024-09-21T15:51:07.124 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0713999999999999e-05 2024-09-21T15:51:07.124 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.124 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.124 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:07.124 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.831305+0000", 2024-09-21T15:51:07.124 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4749999999999993e-06 2024-09-21T15:51:07.124 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.124 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.124 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:07.124 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.831364+0000", 2024-09-21T15:51:07.124 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9320999999999997e-05 2024-09-21T15:51:07.124 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.124 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.124 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:07.124 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.831386+0000", 2024-09-21T15:51:07.125 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2058000000000001e-05 2024-09-21T15:51:07.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.125 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.125 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:07.125 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.839039+0000", 2024-09-21T15:51:07.125 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0076530490000000003 2024-09-21T15:51:07.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.125 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.125 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:07.125 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.839067+0000", 2024-09-21T15:51:07.125 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8005999999999999e-05 2024-09-21T15:51:07.125 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.125 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:07.125 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.125 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.126 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:07.126 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:07.127 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:51:07.126+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:51:07.127 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:51:07.228 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:51:07.227+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:51:07.270 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:07.270 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:51:07.271 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:51:07.271 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:07.279 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:51:07.329 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:51:07.328+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:51:07.421 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:07.421 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:51:07.421 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:51:07.421 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:51:07.421 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:07.430 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:51:07.430 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:51:07.428+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:51:07.530 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:51:07.529+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:51:07.573 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:07.573 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:51:07.573 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:51:07.573 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:51:07.573 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.573 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62833 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 192512~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:07.574 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:51:07.574 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 90.428904751000005, 2024-09-21T15:51:07.574 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050387007999999997, 2024-09-21T15:51:07.574 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:07.574 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:07.574 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:07.574 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:07.574 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:07.574 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62833 2024-09-21T15:51:07.574 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.574 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:07.574 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.574 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:07.575 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:51:07.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:07.575 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:51:07.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.575 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:07.575 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:51:07.575 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:51:07.575 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.575 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.576 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:07.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142150+0000", 2024-09-21T15:51:07.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1999999999999999e-06 2024-09-21T15:51:07.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.576 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:07.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142153+0000", 2024-09-21T15:51:07.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7410000000000001e-06 2024-09-21T15:51:07.576 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.576 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:07.576 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.142156+0000", 2024-09-21T15:51:07.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5739999999999998e-06 2024-09-21T15:51:07.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:07.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.192358+0000", 2024-09-21T15:51:07.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050202452000000002 2024-09-21T15:51:07.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:07.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.192435+0000", 2024-09-21T15:51:07.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6738999999999998e-05 2024-09-21T15:51:07.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:07.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.192536+0000", 2024-09-21T15:51:07.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101508 2024-09-21T15:51:07.578 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.578 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:07.578 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.578 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62849 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 208896~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:07.578 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.144181+0000", 2024-09-21T15:51:07.578 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 90.426872678999999, 2024-09-21T15:51:07.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050460390000000001, 2024-09-21T15:51:07.578 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:07.578 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:07.579 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:07.579 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:07.579 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:07.579 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62849 2024-09-21T15:51:07.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.579 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:07.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:07.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144181+0000", 2024-09-21T15:51:07.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:07.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144181+0000", 2024-09-21T15:51:07.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:07.580 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144181+0000", 2024-09-21T15:51:07.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:07.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:07.580 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144183+0000", 2024-09-21T15:51:07.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3860000000000001e-06 2024-09-21T15:51:07.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:07.580 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144189+0000", 2024-09-21T15:51:07.581 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0599999999999996e-06 2024-09-21T15:51:07.581 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.581 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:07.581 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144194+0000", 2024-09-21T15:51:07.581 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4379999999999997e-06 2024-09-21T15:51:07.581 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.581 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:07.581 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.194457+0000", 2024-09-21T15:51:07.581 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050263317000000002 2024-09-21T15:51:07.581 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.581 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:07.581 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.194548+0000", 2024-09-21T15:51:07.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0785000000000006e-05 2024-09-21T15:51:07.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.582 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.582 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:07.582 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.194642+0000", 2024-09-21T15:51:07.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3837000000000003e-05 2024-09-21T15:51:07.582 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.582 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:07.582 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.582 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.583 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62862 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 954368~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:07.583 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.146176+0000", 2024-09-21T15:51:07.583 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 90.424877616000003, 2024-09-21T15:51:07.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050442901999999998, 2024-09-21T15:51:07.583 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:07.583 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:07.583 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:07.583 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:07.583 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:07.583 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62862 2024-09-21T15:51:07.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.584 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:07.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:07.584 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146176+0000", 2024-09-21T15:51:07.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.584 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:07.585 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146176+0000", 2024-09-21T15:51:07.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.585 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:07.585 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146176+0000", 2024-09-21T15:51:07.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:07.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.585 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:07.585 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146178+0000", 2024-09-21T15:51:07.586 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4739999999999999e-06 2024-09-21T15:51:07.586 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.586 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:07.586 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146186+0000", 2024-09-21T15:51:07.586 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4989999999999997e-06 2024-09-21T15:51:07.586 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.586 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:07.586 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146191+0000", 2024-09-21T15:51:07.586 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7999999999999998e-06 2024-09-21T15:51:07.586 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.587 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:07.587 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196442+0000", 2024-09-21T15:51:07.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050251665000000001 2024-09-21T15:51:07.587 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.587 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:07.587 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196518+0000", 2024-09-21T15:51:07.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5677000000000001e-05 2024-09-21T15:51:07.587 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.587 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:07.587 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196619+0000", 2024-09-21T15:51:07.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101247 2024-09-21T15:51:07.588 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.588 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:07.588 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.588 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.588 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62863 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 937984~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:07.588 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.146195+0000", 2024-09-21T15:51:07.588 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 90.424859424999994, 2024-09-21T15:51:07.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050642305999999998, 2024-09-21T15:51:07.589 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:07.589 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:07.589 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:07.589 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:07.589 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:07.589 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62863 2024-09-21T15:51:07.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.589 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:07.589 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.589 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:07.589 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146195+0000", 2024-09-21T15:51:07.589 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.589 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.590 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:07.590 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146195+0000", 2024-09-21T15:51:07.590 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.590 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:07.590 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146194+0000", 2024-09-21T15:51:07.590 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:07.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.590 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:07.590 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146196+0000", 2024-09-21T15:51:07.590 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1900000000000002e-06 2024-09-21T15:51:07.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.591 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:07.591 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146202+0000", 2024-09-21T15:51:07.591 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.57e-06 2024-09-21T15:51:07.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.591 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.591 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:07.591 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146206+0000", 2024-09-21T15:51:07.591 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5079999999999999e-06 2024-09-21T15:51:07.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.591 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.591 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:07.591 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196655+0000", 2024-09-21T15:51:07.591 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050448973000000001 2024-09-21T15:51:07.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.592 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:07.592 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196730+0000", 2024-09-21T15:51:07.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4728999999999996e-05 2024-09-21T15:51:07.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.592 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:07.592 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196837+0000", 2024-09-21T15:51:07.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000106735 2024-09-21T15:51:07.592 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.592 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:07.592 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.592 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62872 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 290816~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:07.593 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148149+0000", 2024-09-21T15:51:07.593 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 90.422904748999997, 2024-09-21T15:51:07.593 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.05046991, 2024-09-21T15:51:07.593 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:07.593 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:07.593 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:07.593 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:07.593 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:07.593 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62872 2024-09-21T15:51:07.593 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.593 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:07.593 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.593 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:07.593 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148149+0000", 2024-09-21T15:51:07.593 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.594 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.594 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.594 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:07.594 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148149+0000", 2024-09-21T15:51:07.594 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.594 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.594 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.594 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:07.594 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148149+0000", 2024-09-21T15:51:07.594 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:07.594 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.594 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.594 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:07.594 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148157+0000", 2024-09-21T15:51:07.594 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4749999999999993e-06 2024-09-21T15:51:07.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.595 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.595 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:07.595 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148163+0000", 2024-09-21T15:51:07.595 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8250000000000001e-06 2024-09-21T15:51:07.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.595 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.595 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:07.595 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148168+0000", 2024-09-21T15:51:07.595 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4730000000000002e-06 2024-09-21T15:51:07.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.595 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.595 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:07.595 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198471+0000", 2024-09-21T15:51:07.595 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050302974 2024-09-21T15:51:07.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:07.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198555+0000", 2024-09-21T15:51:07.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4783999999999993e-05 2024-09-21T15:51:07.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:07.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198619+0000", 2024-09-21T15:51:07.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3751000000000007e-05 2024-09-21T15:51:07.596 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.596 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:07.596 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.597 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62873 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 315392~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:07.597 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148172+0000", 2024-09-21T15:51:07.597 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 90.422882458000004, 2024-09-21T15:51:07.597 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050707998999999997, 2024-09-21T15:51:07.597 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:07.597 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:07.597 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:07.597 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:07.597 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:07.597 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62873 2024-09-21T15:51:07.597 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.597 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:07.597 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.597 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:07.597 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148172+0000", 2024-09-21T15:51:07.597 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.598 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.598 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.598 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:07.598 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148172+0000", 2024-09-21T15:51:07.598 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.598 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.598 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.598 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:07.598 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148171+0000", 2024-09-21T15:51:07.598 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:07.598 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.598 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.598 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:07.598 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148173+0000", 2024-09-21T15:51:07.599 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2340000000000001e-06 2024-09-21T15:51:07.599 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.599 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.599 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:07.599 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148179+0000", 2024-09-21T15:51:07.599 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9340000000000003e-06 2024-09-21T15:51:07.599 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.599 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.599 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:07.599 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148186+0000", 2024-09-21T15:51:07.599 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2349999999999996e-06 2024-09-21T15:51:07.599 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.599 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.599 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:07.599 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198684+0000", 2024-09-21T15:51:07.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050498890999999997 2024-09-21T15:51:07.600 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.600 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.600 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:07.600 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198778+0000", 2024-09-21T15:51:07.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3134999999999995e-05 2024-09-21T15:51:07.600 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.600 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.600 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:07.600 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198880+0000", 2024-09-21T15:51:07.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101906 2024-09-21T15:51:07.600 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.600 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:07.600 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.601 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.601 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62874 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 323584~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:07.601 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148211+0000", 2024-09-21T15:51:07.601 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 90.422842697999997, 2024-09-21T15:51:07.601 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050887766000000001, 2024-09-21T15:51:07.601 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:07.601 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:07.601 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:07.601 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:07.601 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:07.601 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62874 2024-09-21T15:51:07.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.601 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:07.601 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.602 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:07.602 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148211+0000", 2024-09-21T15:51:07.602 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.602 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.602 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.602 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:07.602 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148211+0000", 2024-09-21T15:51:07.602 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.602 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.602 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.602 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:07.602 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148211+0000", 2024-09-21T15:51:07.602 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:07.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.603 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:07.603 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148213+0000", 2024-09-21T15:51:07.603 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6860000000000002e-06 2024-09-21T15:51:07.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.603 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:07.603 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148220+0000", 2024-09-21T15:51:07.603 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4740000000000002e-06 2024-09-21T15:51:07.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.603 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:07.603 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148226+0000", 2024-09-21T15:51:07.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0120000000000002e-06 2024-09-21T15:51:07.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.604 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:07.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198940+0000", 2024-09-21T15:51:07.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050714412 2024-09-21T15:51:07.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.604 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:07.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198995+0000", 2024-09-21T15:51:07.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4863e-05 2024-09-21T15:51:07.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.604 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:07.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.199099+0000", 2024-09-21T15:51:07.605 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010383299999999999 2024-09-21T15:51:07.605 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.605 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:07.605 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.605 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.605 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.605 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62877 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 995328~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:07.605 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148230+0000", 2024-09-21T15:51:07.605 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 90.422823837999999, 2024-09-21T15:51:07.605 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.051108949000000001, 2024-09-21T15:51:07.605 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:07.605 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:07.605 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:07.605 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:07.605 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:07.606 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62877 2024-09-21T15:51:07.606 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.606 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:07.606 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.606 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:07.606 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148230+0000", 2024-09-21T15:51:07.606 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.606 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.606 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.606 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:07.606 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148230+0000", 2024-09-21T15:51:07.606 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.606 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.606 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.606 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:07.607 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148230+0000", 2024-09-21T15:51:07.607 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:07.607 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.607 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.607 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:07.607 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148232+0000", 2024-09-21T15:51:07.607 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1220000000000002e-06 2024-09-21T15:51:07.607 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.607 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.607 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:07.607 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148238+0000", 2024-09-21T15:51:07.607 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.789e-06 2024-09-21T15:51:07.607 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.607 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.607 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:07.607 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148242+0000", 2024-09-21T15:51:07.608 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7369999999999997e-06 2024-09-21T15:51:07.608 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.608 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.608 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:07.608 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.199201+0000", 2024-09-21T15:51:07.608 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050959088 2024-09-21T15:51:07.608 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.608 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.608 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:07.608 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.199285+0000", 2024-09-21T15:51:07.608 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3147999999999997e-05 2024-09-21T15:51:07.608 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.608 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.608 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:07.608 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.199339+0000", 2024-09-21T15:51:07.608 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4484999999999998e-05 2024-09-21T15:51:07.608 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.609 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:07.609 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.609 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.609 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.609 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62888 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 331776~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:07.609 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150154+0000", 2024-09-21T15:51:07.609 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 90.420900243999995, 2024-09-21T15:51:07.609 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050483169000000001, 2024-09-21T15:51:07.609 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:07.609 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:07.609 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:07.609 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:07.609 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:07.609 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62888 2024-09-21T15:51:07.609 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.609 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:07.610 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.610 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:07.610 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150154+0000", 2024-09-21T15:51:07.610 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.610 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.610 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.610 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:07.610 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150154+0000", 2024-09-21T15:51:07.610 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.610 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.610 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.610 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:07.610 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150153+0000", 2024-09-21T15:51:07.610 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:07.610 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.611 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.611 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:07.611 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150155+0000", 2024-09-21T15:51:07.611 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9920000000000002e-06 2024-09-21T15:51:07.611 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.611 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.611 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:07.611 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150161+0000", 2024-09-21T15:51:07.611 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5439999999999998e-06 2024-09-21T15:51:07.611 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.611 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.611 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:07.611 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150165+0000", 2024-09-21T15:51:07.611 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3460000000000004e-06 2024-09-21T15:51:07.611 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.612 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.612 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:07.612 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200502+0000", 2024-09-21T15:51:07.612 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050337150999999997 2024-09-21T15:51:07.612 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.612 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.612 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:07.612 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200575+0000", 2024-09-21T15:51:07.612 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2250999999999997e-05 2024-09-21T15:51:07.612 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.612 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.612 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:07.612 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200637+0000", 2024-09-21T15:51:07.612 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2195999999999999e-05 2024-09-21T15:51:07.612 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.613 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:07.613 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.613 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.613 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.613 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62889 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 339968~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:07.613 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150169+0000", 2024-09-21T15:51:07.613 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 90.420884744999995, 2024-09-21T15:51:07.613 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050629678999999997, 2024-09-21T15:51:07.613 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:07.613 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:07.613 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:07.613 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:07.613 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:07.613 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62889 2024-09-21T15:51:07.613 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.613 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:07.614 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.614 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:07.614 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150169+0000", 2024-09-21T15:51:07.614 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.614 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.614 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.614 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:07.614 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150169+0000", 2024-09-21T15:51:07.614 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.614 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.614 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.614 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:07.614 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150169+0000", 2024-09-21T15:51:07.614 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:07.614 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.615 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.615 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:07.615 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150171+0000", 2024-09-21T15:51:07.615 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3910000000000001e-06 2024-09-21T15:51:07.615 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.615 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.615 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:07.615 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150177+0000", 2024-09-21T15:51:07.615 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4079999999999997e-06 2024-09-21T15:51:07.615 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.615 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.615 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:07.615 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150182+0000", 2024-09-21T15:51:07.615 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3890000000000004e-06 2024-09-21T15:51:07.615 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.616 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.616 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:07.616 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200664+0000", 2024-09-21T15:51:07.616 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050482212999999998 2024-09-21T15:51:07.616 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.616 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.616 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:07.616 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200738+0000", 2024-09-21T15:51:07.616 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4114000000000002e-05 2024-09-21T15:51:07.616 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.616 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.616 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:07.616 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200799+0000", 2024-09-21T15:51:07.616 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0503e-05 2024-09-21T15:51:07.616 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.616 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:07.617 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.617 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.617 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.617 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62891 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 348160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:07.617 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150186+0000", 2024-09-21T15:51:07.617 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 90.420867603000005, 2024-09-21T15:51:07.617 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050794552, 2024-09-21T15:51:07.617 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:07.617 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:07.617 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:07.617 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:07.617 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:07.617 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62891 2024-09-21T15:51:07.617 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.617 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:07.618 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.618 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:07.618 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150186+0000", 2024-09-21T15:51:07.618 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.618 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.618 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.618 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:07.618 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150186+0000", 2024-09-21T15:51:07.618 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.618 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.618 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.618 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:07.618 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150186+0000", 2024-09-21T15:51:07.618 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:07.619 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.619 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.619 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:07.619 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150193+0000", 2024-09-21T15:51:07.619 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6830000000000002e-06 2024-09-21T15:51:07.619 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.619 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.619 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:07.619 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150199+0000", 2024-09-21T15:51:07.619 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9859999999999999e-06 2024-09-21T15:51:07.619 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.619 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.619 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:07.619 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150203+0000", 2024-09-21T15:51:07.619 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5920000000000002e-06 2024-09-21T15:51:07.620 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.620 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.620 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:07.620 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200835+0000", 2024-09-21T15:51:07.620 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050631783 2024-09-21T15:51:07.620 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.620 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.620 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:07.620 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200894+0000", 2024-09-21T15:51:07.620 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8560999999999997e-05 2024-09-21T15:51:07.620 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.620 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.620 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:07.620 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200981+0000", 2024-09-21T15:51:07.620 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7305999999999994e-05 2024-09-21T15:51:07.621 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.621 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:07.621 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.621 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.621 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.621 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62892 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 356352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:07.621 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150208+0000", 2024-09-21T15:51:07.621 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 90.420846214999997, 2024-09-21T15:51:07.621 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050951769000000001, 2024-09-21T15:51:07.621 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:07.621 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:07.621 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:07.621 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:07.621 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:07.621 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62892 2024-09-21T15:51:07.622 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.622 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:07.622 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.622 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:07.622 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150208+0000", 2024-09-21T15:51:07.622 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.622 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.622 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.622 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:07.622 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150208+0000", 2024-09-21T15:51:07.622 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.622 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.622 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.622 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:07.623 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150207+0000", 2024-09-21T15:51:07.623 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:07.623 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.623 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.623 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:07.623 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150210+0000", 2024-09-21T15:51:07.623 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.238e-06 2024-09-21T15:51:07.623 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.623 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.623 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:07.623 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150215+0000", 2024-09-21T15:51:07.623 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6350000000000003e-06 2024-09-21T15:51:07.623 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.623 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.624 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:07.624 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150220+0000", 2024-09-21T15:51:07.624 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.544e-06 2024-09-21T15:51:07.624 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.624 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.624 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:07.624 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201014+0000", 2024-09-21T15:51:07.624 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050793932999999999 2024-09-21T15:51:07.624 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.624 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.624 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:07.624 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201092+0000", 2024-09-21T15:51:07.624 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7961999999999997e-05 2024-09-21T15:51:07.624 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.624 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.625 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:07.625 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201160+0000", 2024-09-21T15:51:07.625 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7836000000000006e-05 2024-09-21T15:51:07.625 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.625 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:07.625 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.625 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.625 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.625 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62894 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 364544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:07.625 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150820+0000", 2024-09-21T15:51:07.625 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 90.420234015000005, 2024-09-21T15:51:07.625 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050489196, 2024-09-21T15:51:07.625 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:07.625 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:07.625 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:07.626 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:07.626 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:07.626 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62894 2024-09-21T15:51:07.626 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.626 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:07.626 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.626 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:07.626 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150820+0000", 2024-09-21T15:51:07.626 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.626 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.626 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.626 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:07.626 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150820+0000", 2024-09-21T15:51:07.627 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.627 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.627 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.627 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:07.627 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150223+0000", 2024-09-21T15:51:07.627 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9994035 2024-09-21T15:51:07.627 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.627 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.627 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:07.627 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150825+0000", 2024-09-21T15:51:07.627 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00060137099999999996 2024-09-21T15:51:07.627 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.627 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.627 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:07.627 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150837+0000", 2024-09-21T15:51:07.628 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2503999999999999e-05 2024-09-21T15:51:07.628 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.628 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.628 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:07.628 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150844+0000", 2024-09-21T15:51:07.628 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6900000000000003e-06 2024-09-21T15:51:07.628 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.628 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.628 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:07.628 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201187+0000", 2024-09-21T15:51:07.628 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050342973999999999 2024-09-21T15:51:07.628 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.628 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.628 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:07.628 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201250+0000", 2024-09-21T15:51:07.629 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3274000000000004e-05 2024-09-21T15:51:07.629 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.629 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.629 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:07.629 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201309+0000", 2024-09-21T15:51:07.629 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9058000000000003e-05 2024-09-21T15:51:07.629 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.629 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:07.629 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.629 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.629 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.629 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62895 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 380928~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:07.629 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150851+0000", 2024-09-21T15:51:07.629 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 90.420203419000003, 2024-09-21T15:51:07.629 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050611097000000001, 2024-09-21T15:51:07.629 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:07.630 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:07.630 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:07.630 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:07.630 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:07.630 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62895 2024-09-21T15:51:07.630 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.630 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:07.630 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.630 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:07.630 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150851+0000", 2024-09-21T15:51:07.630 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.630 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.630 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.630 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:07.630 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150851+0000", 2024-09-21T15:51:07.631 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.631 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.631 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.631 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:07.631 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150850+0000", 2024-09-21T15:51:07.631 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:07.631 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.631 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.631 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:07.631 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150853+0000", 2024-09-21T15:51:07.631 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1039999999999998e-06 2024-09-21T15:51:07.631 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.631 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.631 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:07.631 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150860+0000", 2024-09-21T15:51:07.631 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3180000000000001e-06 2024-09-21T15:51:07.632 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.632 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.632 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:07.632 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150866+0000", 2024-09-21T15:51:07.632 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1599999999999997e-06 2024-09-21T15:51:07.632 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.632 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.632 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:07.632 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201332+0000", 2024-09-21T15:51:07.632 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050466100999999999 2024-09-21T15:51:07.632 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.632 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.632 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:07.632 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201392+0000", 2024-09-21T15:51:07.632 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0053999999999998e-05 2024-09-21T15:51:07.633 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.633 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.633 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:07.633 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201462+0000", 2024-09-21T15:51:07.633 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9861000000000001e-05 2024-09-21T15:51:07.633 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.633 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:07.633 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.633 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.633 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.633 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62896 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 372736~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:07.633 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150870+0000", 2024-09-21T15:51:07.633 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 90.420183639000001, 2024-09-21T15:51:07.633 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050753292999999998, 2024-09-21T15:51:07.634 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:07.634 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:07.634 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:07.634 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:07.634 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:07.634 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62896 2024-09-21T15:51:07.634 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.634 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:07.634 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.634 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:07.634 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150870+0000", 2024-09-21T15:51:07.634 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.634 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.634 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.635 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:07.635 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150870+0000", 2024-09-21T15:51:07.635 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.635 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.635 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.635 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:07.635 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150870+0000", 2024-09-21T15:51:07.635 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:07.635 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.635 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.635 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:07.635 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150872+0000", 2024-09-21T15:51:07.635 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.644e-06 2024-09-21T15:51:07.636 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.636 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.636 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:07.636 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150879+0000", 2024-09-21T15:51:07.636 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3509999999999998e-06 2024-09-21T15:51:07.636 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.636 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.636 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:07.636 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150884+0000", 2024-09-21T15:51:07.636 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9150000000000004e-06 2024-09-21T15:51:07.636 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.636 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.636 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:07.636 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201487+0000", 2024-09-21T15:51:07.636 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050603276000000003 2024-09-21T15:51:07.636 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.637 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.637 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:07.637 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201557+0000", 2024-09-21T15:51:07.637 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9847999999999999e-05 2024-09-21T15:51:07.637 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.637 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.637 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:07.637 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201624+0000", 2024-09-21T15:51:07.637 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6730999999999995e-05 2024-09-21T15:51:07.637 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.637 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:07.637 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.637 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.637 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.637 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62897 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 389120~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:07.638 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.151044+0000", 2024-09-21T15:51:07.638 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 90.420009598999997, 2024-09-21T15:51:07.638 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050720531999999999, 2024-09-21T15:51:07.638 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:07.638 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:07.638 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:07.638 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:07.638 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:07.638 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62897 2024-09-21T15:51:07.638 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.638 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:07.638 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.638 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:07.638 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151044+0000", 2024-09-21T15:51:07.638 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.639 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.639 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.639 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:07.639 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151044+0000", 2024-09-21T15:51:07.639 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.639 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.639 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.639 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:07.639 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150888+0000", 2024-09-21T15:51:07.639 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9998431 2024-09-21T15:51:07.639 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.639 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.639 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:07.639 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151048+0000", 2024-09-21T15:51:07.639 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00016077299999999999 2024-09-21T15:51:07.639 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.640 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.640 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:07.640 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151058+0000", 2024-09-21T15:51:07.640 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.465e-06 2024-09-21T15:51:07.640 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.640 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.640 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:07.640 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151064+0000", 2024-09-21T15:51:07.640 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3829999999999997e-06 2024-09-21T15:51:07.640 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.640 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.640 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:07.640 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201648+0000", 2024-09-21T15:51:07.640 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050583597000000001 2024-09-21T15:51:07.640 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.640 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.640 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:07.641 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201711+0000", 2024-09-21T15:51:07.641 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3616999999999998e-05 2024-09-21T15:51:07.641 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.641 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.641 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:07.641 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201765+0000", 2024-09-21T15:51:07.641 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3464000000000002e-05 2024-09-21T15:51:07.641 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.641 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:07.641 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.641 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.641 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.641 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:118351 3.4 3:25980f86:::rbd_data.1133545d3fa6.000000000000005a:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-09-21T15:51:07.641 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:58.975795+0000", 2024-09-21T15:51:07.642 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 68.595258964999999, 2024-09-21T15:51:07.642 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.99016351599999997, 2024-09-21T15:51:07.642 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:07.642 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:07.642 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:07.642 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:07.642 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:07.642 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 118351 2024-09-21T15:51:07.642 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.642 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:07.642 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.642 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:07.642 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975795+0000", 2024-09-21T15:51:07.642 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.642 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.643 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.643 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:07.643 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975795+0000", 2024-09-21T15:51:07.643 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.643 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.643 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.643 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:07.643 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975793+0000", 2024-09-21T15:51:07.643 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:51:07.643 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.643 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.643 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:07.643 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975800+0000", 2024-09-21T15:51:07.643 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8220000000000001e-06 2024-09-21T15:51:07.643 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.644 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.644 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:07.644 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975814+0000", 2024-09-21T15:51:07.644 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3893000000000001e-05 2024-09-21T15:51:07.644 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.644 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.644 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:07.644 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975821+0000", 2024-09-21T15:51:07.644 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1319999999999996e-06 2024-09-21T15:51:07.644 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.644 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.644 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:07.644 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975878+0000", 2024-09-21T15:51:07.644 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7238999999999999e-05 2024-09-21T15:51:07.644 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.645 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.645 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:51:07.645 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975889+0000", 2024-09-21T15:51:07.645 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0509000000000001e-05 2024-09-21T15:51:07.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.645 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.645 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:07.645 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.964573+0000", 2024-09-21T15:51:07.645 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.98868396300000005 2024-09-21T15:51:07.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.645 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.645 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:07.645 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.964672+0000", 2024-09-21T15:51:07.645 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9601999999999999e-05 2024-09-21T15:51:07.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.646 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.646 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-09-21T15:51:07.646 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.964758+0000", 2024-09-21T15:51:07.646 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5537000000000005e-05 2024-09-21T15:51:07.646 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.646 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.646 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:07.646 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965847+0000", 2024-09-21T15:51:07.646 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001089404 2024-09-21T15:51:07.646 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.646 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.646 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:07.646 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965890+0000", 2024-09-21T15:51:07.646 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2521999999999997e-05 2024-09-21T15:51:07.646 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.647 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.647 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:07.647 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965934+0000", 2024-09-21T15:51:07.647 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4489e-05 2024-09-21T15:51:07.647 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.647 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.647 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:07.647 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965959+0000", 2024-09-21T15:51:07.647 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4301999999999999e-05 2024-09-21T15:51:07.647 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.647 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:07.647 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.647 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.647 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.647 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:118353 3.4 3:23841e18:::rbd_data.1133545d3fa6.0000000000000061:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-09-21T15:51:07.647 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:58.975836+0000", 2024-09-21T15:51:07.648 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 68.595217465999994, 2024-09-21T15:51:07.648 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.99077746700000002, 2024-09-21T15:51:07.648 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:07.648 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:07.648 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:07.648 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:07.648 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:07.648 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 118353 2024-09-21T15:51:07.648 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.648 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:07.648 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.648 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:07.648 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975836+0000", 2024-09-21T15:51:07.648 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.648 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.648 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.649 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:07.649 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975836+0000", 2024-09-21T15:51:07.649 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.649 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.649 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.649 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:07.649 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975836+0000", 2024-09-21T15:51:07.649 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:07.649 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.649 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.649 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:07.649 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975839+0000", 2024-09-21T15:51:07.649 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6129999999999998e-06 2024-09-21T15:51:07.649 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.650 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.650 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:07.650 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975844+0000", 2024-09-21T15:51:07.650 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5130000000000004e-06 2024-09-21T15:51:07.650 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.650 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.650 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:07.650 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975850+0000", 2024-09-21T15:51:07.650 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3179999999999997e-06 2024-09-21T15:51:07.650 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.650 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.650 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:07.650 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975915+0000", 2024-09-21T15:51:07.650 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5168999999999999e-05 2024-09-21T15:51:07.651 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.651 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.651 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:51:07.651 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975922+0000", 2024-09-21T15:51:07.651 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2749999999999998e-06 2024-09-21T15:51:07.651 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.651 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.651 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:07.651 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965431+0000", 2024-09-21T15:51:07.651 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.98950864699999996 2024-09-21T15:51:07.651 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.651 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.651 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:07.651 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965535+0000", 2024-09-21T15:51:07.651 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000103949 2024-09-21T15:51:07.651 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.652 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.652 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-09-21T15:51:07.652 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965607+0000", 2024-09-21T15:51:07.652 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2478000000000006e-05 2024-09-21T15:51:07.652 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.652 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.652 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:07.652 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.966205+0000", 2024-09-21T15:51:07.652 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00059775200000000001 2024-09-21T15:51:07.652 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.652 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.652 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:07.652 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.966562+0000", 2024-09-21T15:51:07.652 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00035732199999999998 2024-09-21T15:51:07.652 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.653 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.653 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:07.653 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.966593+0000", 2024-09-21T15:51:07.653 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0642999999999998e-05 2024-09-21T15:51:07.653 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.653 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.653 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:07.653 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.966614+0000", 2024-09-21T15:51:07.653 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1149999999999999e-05 2024-09-21T15:51:07.653 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.653 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:07.653 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.653 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.653 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.653 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160535 3.6 3:67cae2ed:::rbd_mirroring:head [list-watchers] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-09-21T15:51:07.654 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:30.195408+0000", 2024-09-21T15:51:07.654 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 37.375646384, 2024-09-21T15:51:07.654 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.96025944900000004, 2024-09-21T15:51:07.654 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:07.654 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-09-21T15:51:07.654 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:07.654 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:07.654 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:07.654 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160535 2024-09-21T15:51:07.654 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.654 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:07.654 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.654 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:07.654 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195408+0000", 2024-09-21T15:51:07.655 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.655 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.655 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.655 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:07.655 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195408+0000", 2024-09-21T15:51:07.655 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.655 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.655 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.655 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:07.655 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195406+0000", 2024-09-21T15:51:07.655 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:51:07.655 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.655 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.655 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:07.655 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195414+0000", 2024-09-21T15:51:07.656 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.2870000000000004e-06 2024-09-21T15:51:07.656 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.656 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.656 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:07.656 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195432+0000", 2024-09-21T15:51:07.656 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8110000000000001e-05 2024-09-21T15:51:07.656 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.656 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.656 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:07.656 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195441+0000", 2024-09-21T15:51:07.656 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7679999999999997e-06 2024-09-21T15:51:07.656 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.656 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.656 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:07.656 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195505+0000", 2024-09-21T15:51:07.657 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3769000000000006e-05 2024-09-21T15:51:07.657 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.657 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.657 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:51:07.657 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195518+0000", 2024-09-21T15:51:07.657 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3341e-05 2024-09-21T15:51:07.657 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.657 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.657 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:07.657 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.155509+0000", 2024-09-21T15:51:07.657 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.959990393 2024-09-21T15:51:07.657 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.657 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.657 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:07.657 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.155667+0000", 2024-09-21T15:51:07.657 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000158505 2024-09-21T15:51:07.658 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.658 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:07.658 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.658 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.658 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.658 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160974 3.4 3:22dd2475:::rbd_id.image_client.0-clone39:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-21T15:51:07.658 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:34.748093+0000", 2024-09-21T15:51:07.658 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 32.822960833000003, 2024-09-21T15:51:07.658 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.94006199499999998, 2024-09-21T15:51:07.658 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:07.658 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:07.658 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:07.658 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:07.658 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:07.659 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160974 2024-09-21T15:51:07.659 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.659 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:07.659 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.659 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:07.659 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748093+0000", 2024-09-21T15:51:07.659 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.659 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.659 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.659 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:07.659 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748093+0000", 2024-09-21T15:51:07.659 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:07.659 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.659 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.659 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:07.659 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748091+0000", 2024-09-21T15:51:07.660 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:51:07.660 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.660 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.660 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:07.660 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748101+0000", 2024-09-21T15:51:07.660 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5950000000000005e-06 2024-09-21T15:51:07.660 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.660 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.660 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:07.660 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748122+0000", 2024-09-21T15:51:07.660 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1611e-05 2024-09-21T15:51:07.660 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.660 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.660 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:07.660 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748133+0000", 2024-09-21T15:51:07.660 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0224000000000001e-05 2024-09-21T15:51:07.661 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.661 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.661 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:07.661 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748185+0000", 2024-09-21T15:51:07.661 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2005000000000001e-05 2024-09-21T15:51:07.661 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.661 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.661 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:51:07.661 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748194+0000", 2024-09-21T15:51:07.661 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0180000000000007e-06 2024-09-21T15:51:07.661 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.661 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.661 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:07.661 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.683582+0000", 2024-09-21T15:51:07.661 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.93538800499999997 2024-09-21T15:51:07.661 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.662 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.662 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:07.662 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.683711+0000", 2024-09-21T15:51:07.662 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000129746 2024-09-21T15:51:07.662 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.662 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.662 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-09-21T15:51:07.662 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.683838+0000", 2024-09-21T15:51:07.662 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012693200000000001 2024-09-21T15:51:07.662 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.662 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.662 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:07.662 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.687445+0000", 2024-09-21T15:51:07.662 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0036071340000000001 2024-09-21T15:51:07.662 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.663 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.663 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:07.663 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.688064+0000", 2024-09-21T15:51:07.663 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00061857100000000005 2024-09-21T15:51:07.663 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.663 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.663 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:07.663 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.688113+0000", 2024-09-21T15:51:07.663 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9128000000000001e-05 2024-09-21T15:51:07.663 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:07.663 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:07.663 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:07.663 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.688155+0000", 2024-09-21T15:51:07.663 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2026e-05 2024-09-21T15:51:07.663 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.664 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:07.664 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.664 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:07.664 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:07.664 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:07.665 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:51:07.666 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:51:07.664+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:51:07.766 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:51:07.765+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:51:07.807 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:07.807 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:51:07.807 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:51:07.807 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:07.815 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:51:07.867 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:51:07.865+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:51:07.919 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:07.919 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:51:07.919 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:51:07.919 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:51:07.919 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:07.928 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:51:07.969 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:51:07.968+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:51:08.068 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:51:08.067+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:51:08.071 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:08.071 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:51:08.071 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:51:08.071 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:51:08.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.071 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:4957 3.9 e29/22 3:95c5ed95:::rbd_data.11335fa2fe69.0000000000000018:head v 29'4, mlcod=26'2)", 2024-09-21T15:51:08.072 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:14.453951+0000", 2024-09-21T15:51:08.072 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 113.61568073399999, 2024-09-21T15:51:08.072 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0043804559999999996, 2024-09-21T15:51:08.072 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:08.072 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:08.072 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:08.072 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.072 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:08.072 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.453951+0000", 2024-09-21T15:51:08.072 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.072 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.072 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.072 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:08.072 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.453951+0000", 2024-09-21T15:51:08.073 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.073 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.073 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.073 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:08.073 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.453950+0000", 2024-09-21T15:51:08.073 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:08.074 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.074 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.074 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:08.074 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454495+0000", 2024-09-21T15:51:08.074 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00054437799999999998 2024-09-21T15:51:08.074 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:08.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454508+0000", 2024-09-21T15:51:08.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3475000000000001e-05 2024-09-21T15:51:08.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:08.075 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454515+0000", 2024-09-21T15:51:08.075 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8990000000000004e-06 2024-09-21T15:51:08.075 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.075 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.075 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:08.076 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454581+0000", 2024-09-21T15:51:08.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5987000000000004e-05 2024-09-21T15:51:08.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.076 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:08.076 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454609+0000", 2024-09-21T15:51:08.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7466000000000001e-05 2024-09-21T15:51:08.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.076 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.076 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:08.076 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.458303+0000", 2024-09-21T15:51:08.076 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003693979 2024-09-21T15:51:08.076 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.077 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.077 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:08.077 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.458331+0000", 2024-09-21T15:51:08.077 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8932999999999999e-05 2024-09-21T15:51:08.077 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.077 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:08.077 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.077 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.077 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.077 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13420 3.0 e31/22 3:038b8840:::rbd_data.1133dc5adfdf.0000000000000035:head v 31'49, mlcod=31'47)", 2024-09-21T15:51:08.077 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.209233+0000", 2024-09-21T15:51:08.077 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 110.860399118, 2024-09-21T15:51:08.077 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0042624960000000002, 2024-09-21T15:51:08.077 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:08.078 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:08.078 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:08.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.078 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:08.078 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209233+0000", 2024-09-21T15:51:08.078 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.078 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.078 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:08.078 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209233+0000", 2024-09-21T15:51:08.078 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.078 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.078 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.079 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:08.079 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209232+0000", 2024-09-21T15:51:08.079 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:08.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.079 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.079 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:08.079 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209251+0000", 2024-09-21T15:51:08.079 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9304e-05 2024-09-21T15:51:08.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.079 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.079 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:08.079 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209265+0000", 2024-09-21T15:51:08.079 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3532e-05 2024-09-21T15:51:08.079 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.079 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.080 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:08.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209272+0000", 2024-09-21T15:51:08.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.289e-06 2024-09-21T15:51:08.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.080 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:08.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209321+0000", 2024-09-21T15:51:08.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8773999999999998e-05 2024-09-21T15:51:08.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.080 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.080 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:08.080 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209341+0000", 2024-09-21T15:51:08.080 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0387999999999999e-05 2024-09-21T15:51:08.080 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.081 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:08.081 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.213464+0000", 2024-09-21T15:51:08.081 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0041232450000000002 2024-09-21T15:51:08.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.081 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:08.081 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.213495+0000", 2024-09-21T15:51:08.081 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0793999999999997e-05 2024-09-21T15:51:08.081 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.081 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:08.081 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.081 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.081 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.081 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13445 3.0 e31/22 3:038b8840:::rbd_data.1133dc5adfdf.0000000000000035:head v 31'50, mlcod=31'48)", 2024-09-21T15:51:08.082 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.232497+0000", 2024-09-21T15:51:08.082 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 110.83713510600001, 2024-09-21T15:51:08.082 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0040493559999999996, 2024-09-21T15:51:08.082 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:08.082 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:08.082 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:08.082 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.082 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:08.082 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232497+0000", 2024-09-21T15:51:08.082 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.083 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:08.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232497+0000", 2024-09-21T15:51:08.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.083 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.083 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:08.083 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232496+0000", 2024-09-21T15:51:08.083 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:08.083 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.084 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:08.084 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232503+0000", 2024-09-21T15:51:08.084 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7320000000000001e-06 2024-09-21T15:51:08.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.084 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:08.084 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232512+0000", 2024-09-21T15:51:08.084 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9259999999999995e-06 2024-09-21T15:51:08.084 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.084 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.084 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:08.084 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232520+0000", 2024-09-21T15:51:08.085 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3699999999999997e-06 2024-09-21T15:51:08.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.085 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.085 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:08.085 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232564+0000", 2024-09-21T15:51:08.085 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4128999999999997e-05 2024-09-21T15:51:08.085 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.085 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.086 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:08.086 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232585+0000", 2024-09-21T15:51:08.086 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0704999999999999e-05 2024-09-21T15:51:08.086 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.086 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.086 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:08.087 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.236507+0000", 2024-09-21T15:51:08.087 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0039218730000000002 2024-09-21T15:51:08.087 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.087 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.087 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:08.087 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.236546+0000", 2024-09-21T15:51:08.087 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9399e-05 2024-09-21T15:51:08.087 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.087 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:08.087 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.087 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.087 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.088 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13832 3.0 e31/22 3:0792efd2:::rbd_data.1133dc5adfdf.0000000000000090:head v 31'58, mlcod=31'56)", 2024-09-21T15:51:08.088 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.527530+0000", 2024-09-21T15:51:08.088 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 110.54210157999999, 2024-09-21T15:51:08.088 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035256530000000001, 2024-09-21T15:51:08.088 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:08.088 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:08.088 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:08.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.088 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:08.088 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527530+0000", 2024-09-21T15:51:08.088 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.088 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.088 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.089 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:08.089 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527530+0000", 2024-09-21T15:51:08.089 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.089 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.089 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:08.089 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527529+0000", 2024-09-21T15:51:08.089 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:51:08.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.089 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.089 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:08.089 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527550+0000", 2024-09-21T15:51:08.089 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1387999999999999e-05 2024-09-21T15:51:08.089 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.090 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.090 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:08.090 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527568+0000", 2024-09-21T15:51:08.090 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8068e-05 2024-09-21T15:51:08.090 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.090 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.090 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:08.090 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527577+0000", 2024-09-21T15:51:08.090 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9919999999999996e-06 2024-09-21T15:51:08.090 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.090 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.090 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:08.090 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527626+0000", 2024-09-21T15:51:08.090 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8597000000000002e-05 2024-09-21T15:51:08.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.091 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:08.091 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527652+0000", 2024-09-21T15:51:08.091 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5993e-05 2024-09-21T15:51:08.091 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.091 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.091 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:08.091 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.531024+0000", 2024-09-21T15:51:08.092 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033721910000000001 2024-09-21T15:51:08.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.092 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:08.092 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.531056+0000", 2024-09-21T15:51:08.092 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1628000000000003e-05 2024-09-21T15:51:08.092 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.092 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:08.092 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.092 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.092 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.093 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13839 3.0 e31/22 3:03c4fb32:::rbd_data.1133dc5adfdf.0000000000000099:head v 31'59, mlcod=31'56)", 2024-09-21T15:51:08.093 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.528097+0000", 2024-09-21T15:51:08.093 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 110.541534982, 2024-09-21T15:51:08.093 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0057228840000000001, 2024-09-21T15:51:08.093 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:08.093 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:08.093 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:08.093 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.093 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:08.093 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528097+0000", 2024-09-21T15:51:08.094 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.094 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.094 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:08.094 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528097+0000", 2024-09-21T15:51:08.094 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.094 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.094 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:08.094 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528096+0000", 2024-09-21T15:51:08.094 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:08.094 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.095 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:08.095 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528105+0000", 2024-09-21T15:51:08.095 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7350000000000002e-06 2024-09-21T15:51:08.095 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.095 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:08.095 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528117+0000", 2024-09-21T15:51:08.095 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1909000000000001e-05 2024-09-21T15:51:08.095 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.095 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.095 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:08.095 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528124+0000", 2024-09-21T15:51:08.095 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8419999999999999e-06 2024-09-21T15:51:08.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:08.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528168+0000", 2024-09-21T15:51:08.096 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4301999999999997e-05 2024-09-21T15:51:08.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:08.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528191+0000", 2024-09-21T15:51:08.096 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2622000000000001e-05 2024-09-21T15:51:08.096 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.096 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.096 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:08.096 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.533780+0000", 2024-09-21T15:51:08.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0055897519999999999 2024-09-21T15:51:08.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.097 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:08.097 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.533820+0000", 2024-09-21T15:51:08.097 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9366999999999998e-05 2024-09-21T15:51:08.097 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.097 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:08.097 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.097 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.097 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.097 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13884 3.9 e31/22 3:93003736:::rbd_data.1133dc5adfdf.00000000000000a1:head v 31'54, mlcod=31'52)", 2024-09-21T15:51:08.097 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.562433+0000", 2024-09-21T15:51:08.097 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 110.507198234, 2024-09-21T15:51:08.098 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.005162313, 2024-09-21T15:51:08.098 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:08.098 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:08.098 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:08.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:08.098 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562433+0000", 2024-09-21T15:51:08.098 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.098 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.098 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.098 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:08.098 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562433+0000", 2024-09-21T15:51:08.098 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.099 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:08.099 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562432+0000", 2024-09-21T15:51:08.099 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:08.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.099 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:08.099 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562467+0000", 2024-09-21T15:51:08.099 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4885999999999998e-05 2024-09-21T15:51:08.099 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.099 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.099 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:08.099 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562481+0000", 2024-09-21T15:51:08.100 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3604000000000001e-05 2024-09-21T15:51:08.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.100 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:08.100 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562489+0000", 2024-09-21T15:51:08.100 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1829999999999994e-06 2024-09-21T15:51:08.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.100 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:08.100 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562536+0000", 2024-09-21T15:51:08.100 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6830999999999999e-05 2024-09-21T15:51:08.100 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.100 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.100 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:08.100 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562559+0000", 2024-09-21T15:51:08.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2765999999999999e-05 2024-09-21T15:51:08.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.101 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:08.101 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.567566+0000", 2024-09-21T15:51:08.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0050073219999999998 2024-09-21T15:51:08.101 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.101 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.101 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:08.101 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.567596+0000", 2024-09-21T15:51:08.101 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9802999999999999e-05 2024-09-21T15:51:08.101 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.101 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:08.101 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.102 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.102 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:22340 3.9 e34/22 3:910a5abe:::rbd_data.1133896adcfd.0000000000000023:head v 34'92, mlcod=34'90)", 2024-09-21T15:51:08.102 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:21.628025+0000", 2024-09-21T15:51:08.102 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 106.441606854, 2024-09-21T15:51:08.102 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046196010000000001, 2024-09-21T15:51:08.102 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:08.102 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:08.102 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:08.102 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.102 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:08.102 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628025+0000", 2024-09-21T15:51:08.103 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.103 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:08.103 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628025+0000", 2024-09-21T15:51:08.103 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.103 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:08.103 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628024+0000", 2024-09-21T15:51:08.103 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:08.103 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.103 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.103 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:08.104 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628034+0000", 2024-09-21T15:51:08.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3470000000000005e-06 2024-09-21T15:51:08.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.104 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:08.104 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628043+0000", 2024-09-21T15:51:08.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9450000000000005e-06 2024-09-21T15:51:08.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.104 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.104 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:08.104 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628048+0000", 2024-09-21T15:51:08.104 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8080000000000004e-06 2024-09-21T15:51:08.104 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.105 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:08.105 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628081+0000", 2024-09-21T15:51:08.105 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2738999999999999e-05 2024-09-21T15:51:08.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.105 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:08.105 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628094+0000", 2024-09-21T15:51:08.105 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3273e-05 2024-09-21T15:51:08.105 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.105 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.106 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:08.106 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.632616+0000", 2024-09-21T15:51:08.106 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045217269999999997 2024-09-21T15:51:08.106 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.106 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.106 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:08.106 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.632644+0000", 2024-09-21T15:51:08.106 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8469000000000001e-05 2024-09-21T15:51:08.106 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.107 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:08.107 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.107 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.107 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.107 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:22417 3.4 e34/22 3:25310a10:::rbd_data.1133896adcfd.0000000000000075:head v 34'48, mlcod=34'46)", 2024-09-21T15:51:08.107 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:21.704297+0000", 2024-09-21T15:51:08.107 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 106.365334756, 2024-09-21T15:51:08.108 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0038101799999999998, 2024-09-21T15:51:08.108 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:08.108 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:08.108 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:08.108 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.109 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:08.109 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704297+0000", 2024-09-21T15:51:08.109 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.109 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.109 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:08.109 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704297+0000", 2024-09-21T15:51:08.109 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.109 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.110 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.110 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:08.110 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704296+0000", 2024-09-21T15:51:08.110 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:51:08.110 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.110 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.110 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:08.110 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704308+0000", 2024-09-21T15:51:08.111 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2559e-05 2024-09-21T15:51:08.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.111 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:08.111 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704337+0000", 2024-09-21T15:51:08.111 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8960000000000001e-05 2024-09-21T15:51:08.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.111 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:08.111 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704345+0000", 2024-09-21T15:51:08.111 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3059999999999998e-06 2024-09-21T15:51:08.111 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.111 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.112 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:08.112 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704408+0000", 2024-09-21T15:51:08.112 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2156000000000006e-05 2024-09-21T15:51:08.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.112 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:08.112 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704430+0000", 2024-09-21T15:51:08.112 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1931000000000001e-05 2024-09-21T15:51:08.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.112 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.112 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:08.112 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.708073+0000", 2024-09-21T15:51:08.112 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003643028 2024-09-21T15:51:08.112 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.113 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:08.113 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.708107+0000", 2024-09-21T15:51:08.113 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4613999999999998e-05 2024-09-21T15:51:08.113 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.113 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:08.113 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.113 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.113 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.113 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:25278 3.4 e38/22 3:2251cc1c:::rbd_data.1133ebd6d3ba.0000000000000024:head v 38'51, mlcod=38'49)", 2024-09-21T15:51:08.113 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.317337+0000", 2024-09-21T15:51:08.113 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 101.752294996, 2024-09-21T15:51:08.113 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053904110000000003, 2024-09-21T15:51:08.113 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:08.113 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:08.114 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:08.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.114 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:08.114 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317337+0000", 2024-09-21T15:51:08.114 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.114 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.114 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.114 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:08.114 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317337+0000", 2024-09-21T15:51:08.114 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.114 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:08.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317335+0000", 2024-09-21T15:51:08.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:51:08.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:08.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317347+0000", 2024-09-21T15:51:08.115 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.19e-05 2024-09-21T15:51:08.115 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.115 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.115 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:08.115 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317369+0000", 2024-09-21T15:51:08.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2232e-05 2024-09-21T15:51:08.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.116 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:08.116 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317381+0000", 2024-09-21T15:51:08.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2231e-05 2024-09-21T15:51:08.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.116 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:08.116 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317425+0000", 2024-09-21T15:51:08.116 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3625000000000003e-05 2024-09-21T15:51:08.116 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.116 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.117 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:08.117 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317449+0000", 2024-09-21T15:51:08.117 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4301999999999999e-05 2024-09-21T15:51:08.117 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.117 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.117 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:08.117 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.322688+0000", 2024-09-21T15:51:08.117 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.005238609 2024-09-21T15:51:08.117 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.117 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.117 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:08.117 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.322727+0000", 2024-09-21T15:51:08.117 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9201e-05 2024-09-21T15:51:08.117 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.117 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:08.118 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.118 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.118 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.118 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:25286 3.0 e38/22 3:0248fb60:::rbd_data.1133ebd6d3ba.0000000000000066:head v 38'109, mlcod=34'107)", 2024-09-21T15:51:08.118 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.317638+0000", 2024-09-21T15:51:08.118 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 101.751994117, 2024-09-21T15:51:08.118 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0054074370000000002, 2024-09-21T15:51:08.118 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:08.118 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:08.118 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:08.118 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.118 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:08.118 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317638+0000", 2024-09-21T15:51:08.119 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.119 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.119 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.119 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:08.119 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317638+0000", 2024-09-21T15:51:08.119 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.119 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.119 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.119 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:08.119 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317637+0000", 2024-09-21T15:51:08.119 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:08.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.120 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.120 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:08.120 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317643+0000", 2024-09-21T15:51:08.120 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5609999999999998e-06 2024-09-21T15:51:08.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.120 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.120 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:08.120 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317653+0000", 2024-09-21T15:51:08.120 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0705e-05 2024-09-21T15:51:08.120 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.121 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:08.121 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317660+0000", 2024-09-21T15:51:08.121 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2099999999999998e-06 2024-09-21T15:51:08.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.121 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:08.121 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317704+0000", 2024-09-21T15:51:08.121 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4659000000000001e-05 2024-09-21T15:51:08.121 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.121 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.121 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:08.122 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317725+0000", 2024-09-21T15:51:08.122 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0537e-05 2024-09-21T15:51:08.122 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.122 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.122 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:08.122 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323010+0000", 2024-09-21T15:51:08.122 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052853420000000002 2024-09-21T15:51:08.122 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.122 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.122 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:08.122 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323045+0000", 2024-09-21T15:51:08.122 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4860000000000002e-05 2024-09-21T15:51:08.122 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.123 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:08.123 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.123 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.123 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:25280 3.4 e38/22 3:27fca38c:::rbd_data.1133ebd6d3ba.0000000000000026:head v 38'52, mlcod=38'49)", 2024-09-21T15:51:08.123 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.317684+0000", 2024-09-21T15:51:08.123 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 101.75194822100001, 2024-09-21T15:51:08.123 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053570809999999996, 2024-09-21T15:51:08.123 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:08.123 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:08.123 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:08.123 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.123 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:08.123 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317684+0000", 2024-09-21T15:51:08.124 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.124 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.124 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.124 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:08.124 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317684+0000", 2024-09-21T15:51:08.124 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.124 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.124 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.124 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:08.124 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317683+0000", 2024-09-21T15:51:08.124 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:08.124 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.124 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.124 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:08.124 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317688+0000", 2024-09-21T15:51:08.125 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5009999999999998e-06 2024-09-21T15:51:08.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.125 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.125 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:08.125 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317694+0000", 2024-09-21T15:51:08.125 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3579999999999999e-06 2024-09-21T15:51:08.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.125 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.125 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:08.125 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317698+0000", 2024-09-21T15:51:08.125 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4150000000000001e-06 2024-09-21T15:51:08.125 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.125 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.125 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:08.125 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317738+0000", 2024-09-21T15:51:08.126 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9575e-05 2024-09-21T15:51:08.126 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.126 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.126 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:08.126 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317762+0000", 2024-09-21T15:51:08.126 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4173999999999999e-05 2024-09-21T15:51:08.126 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.126 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.126 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:08.126 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323005+0000", 2024-09-21T15:51:08.126 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00524333 2024-09-21T15:51:08.126 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.126 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.126 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:08.127 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323041+0000", 2024-09-21T15:51:08.127 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5139999999999999e-05 2024-09-21T15:51:08.127 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.127 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:08.127 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.127 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.127 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.127 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:34837 3.0 e43/22 3:038527c8:::rbd_header.1133fed1a840:head v 43'130, mlcod=43'128)", 2024-09-21T15:51:08.127 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:51:08.127 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 97.085743523000005, 2024-09-21T15:51:08.127 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045776519999999998, 2024-09-21T15:51:08.127 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:08.127 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:08.127 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:08.128 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.128 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:08.128 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:51:08.128 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.128 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.128 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.128 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:08.128 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:51:08.128 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.128 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.128 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.128 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:08.128 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:51:08.128 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:08.128 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.129 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.129 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:08.129 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983892+0000", 2024-09-21T15:51:08.129 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8450000000000002e-06 2024-09-21T15:51:08.129 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.129 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.129 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:08.129 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983902+0000", 2024-09-21T15:51:08.129 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0299999999999999e-06 2024-09-21T15:51:08.129 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.129 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.129 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:08.129 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983906+0000", 2024-09-21T15:51:08.129 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2300000000000002e-06 2024-09-21T15:51:08.130 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.130 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.130 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:08.130 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983940+0000", 2024-09-21T15:51:08.130 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4563000000000001e-05 2024-09-21T15:51:08.130 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.130 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.130 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:08.130 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983960+0000", 2024-09-21T15:51:08.130 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9199999999999999e-05 2024-09-21T15:51:08.130 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.130 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.131 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:08.131 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.988430+0000", 2024-09-21T15:51:08.131 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0044709149999999998 2024-09-21T15:51:08.131 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.131 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.131 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:08.131 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.988466+0000", 2024-09-21T15:51:08.131 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5420000000000003e-05 2024-09-21T15:51:08.131 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.131 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:08.131 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.131 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.131 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.132 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "pg_update_log_missing(3.0 epoch 50/22 rep_tid 498 entries 50'171 (0'0) error 3:5df4a5a0:::rbd_data.11339e87a32a.000000000000008e:head by client.4403.0:80207 0.000000 -2 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 47'167)", 2024-09-21T15:51:08.132 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:51:08.132 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 87.127870650000006, 2024-09-21T15:51:08.132 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0047253119999999997, 2024-09-21T15:51:08.132 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:08.132 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-09-21T15:51:08.132 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:08.132 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.132 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:08.132 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:51:08.132 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.132 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.132 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.132 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:08.133 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:51:08.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.133 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.133 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:08.133 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:51:08.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:08.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.133 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.133 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:08.133 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941768+0000", 2024-09-21T15:51:08.133 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7309999999999993e-06 2024-09-21T15:51:08.133 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.133 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.134 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:08.134 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941780+0000", 2024-09-21T15:51:08.134 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1399999999999999e-05 2024-09-21T15:51:08.134 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.134 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.134 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:08.134 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941789+0000", 2024-09-21T15:51:08.134 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5230000000000001e-06 2024-09-21T15:51:08.134 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.134 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.134 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:08.134 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941841+0000", 2024-09-21T15:51:08.134 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1863999999999997e-05 2024-09-21T15:51:08.134 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.135 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.135 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:08.135 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.946486+0000", 2024-09-21T15:51:08.135 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046452539999999997 2024-09-21T15:51:08.135 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.135 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:08.135 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.135 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.135 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.135 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "pg_update_log_missing(3.0 epoch 53/22 rep_tid 585 entries 53'201 (0'0) error 3:6a1d4890:::rbd_data.1133e079cae4.0000000000000004:head by client.4403.0:101089 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 53'198)", 2024-09-21T15:51:08.135 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:44.832827+0000", 2024-09-21T15:51:08.135 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 83.236804841999998, 2024-09-21T15:51:08.135 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053552929999999997, 2024-09-21T15:51:08.136 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:08.136 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-09-21T15:51:08.136 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:08.136 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.136 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:08.136 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832827+0000", 2024-09-21T15:51:08.136 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.136 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.136 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.136 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:08.136 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832827+0000", 2024-09-21T15:51:08.136 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.136 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.136 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.137 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:08.137 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832826+0000", 2024-09-21T15:51:08.137 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:08.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.137 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:08.137 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832835+0000", 2024-09-21T15:51:08.137 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4509999999999992e-06 2024-09-21T15:51:08.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.137 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:08.137 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832854+0000", 2024-09-21T15:51:08.137 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9352999999999999e-05 2024-09-21T15:51:08.137 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.137 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.138 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:08.138 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832866+0000", 2024-09-21T15:51:08.138 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1766000000000001e-05 2024-09-21T15:51:08.138 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.138 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.138 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:08.138 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.833091+0000", 2024-09-21T15:51:08.138 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00022490300000000001 2024-09-21T15:51:08.138 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.138 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.138 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:08.138 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.838182+0000", 2024-09-21T15:51:08.138 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0050915420000000001 2024-09-21T15:51:08.138 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.139 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:08.139 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.139 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.139 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:102822 3.4 e54/22 3:2729ba03:::rbd_object_map.113336ae1075:head v 54'208, mlcod=54'205)", 2024-09-21T15:51:08.139 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:46.249540+0000", 2024-09-21T15:51:08.139 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 81.820091610000006, 2024-09-21T15:51:08.139 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052654859999999998, 2024-09-21T15:51:08.139 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:08.139 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:08.139 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:08.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.139 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:08.139 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249540+0000", 2024-09-21T15:51:08.139 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.140 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:08.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249540+0000", 2024-09-21T15:51:08.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.140 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:08.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249539+0000", 2024-09-21T15:51:08.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:08.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.140 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:08.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249546+0000", 2024-09-21T15:51:08.141 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7290000000000003e-06 2024-09-21T15:51:08.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.141 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:08.141 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249558+0000", 2024-09-21T15:51:08.141 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1844e-05 2024-09-21T15:51:08.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.141 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:08.141 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249565+0000", 2024-09-21T15:51:08.141 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8210000000000004e-06 2024-09-21T15:51:08.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.142 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.142 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:08.142 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249602+0000", 2024-09-21T15:51:08.142 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7057000000000001e-05 2024-09-21T15:51:08.142 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.142 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.142 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:08.142 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249622+0000", 2024-09-21T15:51:08.142 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9678999999999999e-05 2024-09-21T15:51:08.142 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.142 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.142 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:08.143 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254783+0000", 2024-09-21T15:51:08.143 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0051611449999999998 2024-09-21T15:51:08.143 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.143 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.143 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:08.143 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254806+0000", 2024-09-21T15:51:08.143 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2853e-05 2024-09-21T15:51:08.143 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.143 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:08.143 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.144 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.144 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:102823 3.4 e54/22 3:2729ba03:::rbd_object_map.113336ae1075:head v 54'209, mlcod=54'205)", 2024-09-21T15:51:08.144 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:46.250318+0000", 2024-09-21T15:51:08.144 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 81.819314024999997, 2024-09-21T15:51:08.144 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045013539999999999, 2024-09-21T15:51:08.144 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:08.144 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:08.144 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:08.144 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.144 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:08.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250318+0000", 2024-09-21T15:51:08.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.145 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:08.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250318+0000", 2024-09-21T15:51:08.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.145 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:08.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250317+0000", 2024-09-21T15:51:08.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:08.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.146 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:08.146 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250322+0000", 2024-09-21T15:51:08.146 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.074e-06 2024-09-21T15:51:08.146 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.146 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.146 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:08.146 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250330+0000", 2024-09-21T15:51:08.146 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4730000000000003e-06 2024-09-21T15:51:08.146 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.146 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.146 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:08.146 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250335+0000", 2024-09-21T15:51:08.146 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8019999999999997e-06 2024-09-21T15:51:08.147 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.147 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.147 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:08.147 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250375+0000", 2024-09-21T15:51:08.147 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9459999999999998e-05 2024-09-21T15:51:08.147 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.147 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.147 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:08.147 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250388+0000", 2024-09-21T15:51:08.147 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3288999999999999e-05 2024-09-21T15:51:08.147 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.147 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.147 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:08.147 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254807+0000", 2024-09-21T15:51:08.148 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0044192220000000004 2024-09-21T15:51:08.148 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.148 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.148 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:08.148 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254819+0000", 2024-09-21T15:51:08.148 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1603999999999999e-05 2024-09-21T15:51:08.148 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.148 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:08.148 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.148 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.148 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.148 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:160603 3.9 e95/22 3:9171211b:::rbd_data.11338bae974a.0000000000000006:head v 95'343, mlcod=92'340)", 2024-09-21T15:51:08.148 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.215621+0000", 2024-09-21T15:51:08.149 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 36.854010993999999, 2024-09-21T15:51:08.149 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0041276250000000002, 2024-09-21T15:51:08.149 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:08.149 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:08.149 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:08.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.149 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:08.149 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215621+0000", 2024-09-21T15:51:08.149 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.149 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:08.149 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215621+0000", 2024-09-21T15:51:08.150 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.150 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.150 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.150 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:08.150 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215620+0000", 2024-09-21T15:51:08.150 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:08.150 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.150 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.150 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:08.150 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215627+0000", 2024-09-21T15:51:08.150 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8850000000000001e-06 2024-09-21T15:51:08.150 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.150 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.151 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:08.151 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215803+0000", 2024-09-21T15:51:08.151 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000176422 2024-09-21T15:51:08.151 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.151 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.151 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:08.151 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215814+0000", 2024-09-21T15:51:08.151 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0135e-05 2024-09-21T15:51:08.151 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.151 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.151 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:08.151 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215866+0000", 2024-09-21T15:51:08.151 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2327999999999998e-05 2024-09-21T15:51:08.151 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.152 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.152 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:08.152 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215887+0000", 2024-09-21T15:51:08.152 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1449000000000001e-05 2024-09-21T15:51:08.152 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.152 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.152 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:08.152 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.219719+0000", 2024-09-21T15:51:08.152 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0038312630000000001 2024-09-21T15:51:08.152 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.152 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.152 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:08.152 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.219748+0000", 2024-09-21T15:51:08.152 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9737999999999999e-05 2024-09-21T15:51:08.152 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.153 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:08.153 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.153 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.153 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.153 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:161865 3.9 e105/22 3:90e88bad:::rbd_data.11338db70e6f.0000000000000015:head v 105'347, mlcod=103'345)", 2024-09-21T15:51:08.153 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:40.786848+0000", 2024-09-21T15:51:08.153 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 27.282784092, 2024-09-21T15:51:08.153 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0056408860000000003, 2024-09-21T15:51:08.153 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:08.153 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:08.153 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:08.153 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.153 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:08.153 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786848+0000", 2024-09-21T15:51:08.154 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.154 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.154 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.154 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:08.154 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786848+0000", 2024-09-21T15:51:08.154 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.154 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.154 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.154 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:08.154 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786847+0000", 2024-09-21T15:51:08.154 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:08.154 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.154 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.155 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:08.155 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786851+0000", 2024-09-21T15:51:08.155 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7419999999999999e-06 2024-09-21T15:51:08.155 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.155 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.155 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:08.155 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786872+0000", 2024-09-21T15:51:08.155 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1186e-05 2024-09-21T15:51:08.155 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.155 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.155 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:08.155 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786878+0000", 2024-09-21T15:51:08.155 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0970000000000002e-06 2024-09-21T15:51:08.155 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.156 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.156 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:08.156 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786918+0000", 2024-09-21T15:51:08.156 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.998e-05 2024-09-21T15:51:08.156 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.156 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.156 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:08.156 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786941+0000", 2024-09-21T15:51:08.156 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.245e-05 2024-09-21T15:51:08.156 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.156 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.156 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:08.156 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.792460+0000", 2024-09-21T15:51:08.156 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0055191210000000001 2024-09-21T15:51:08.157 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.157 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.157 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:08.157 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.792489+0000", 2024-09-21T15:51:08.157 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8552e-05 2024-09-21T15:51:08.157 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.157 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:08.157 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.157 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.157 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.157 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.1.0:1503 3.4 e127/99 3:26dffaa1:::rbd_data.113336ae1075.000000000000006b:16 v 127'658, mlcod=127'653)", 2024-09-21T15:51:08.157 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:51:02.797289+0000", 2024-09-21T15:51:08.157 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.2723432130000001, 2024-09-21T15:51:08.157 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045583780000000001, 2024-09-21T15:51:08.158 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:08.158 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:08.158 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:08.158 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.158 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:08.158 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.797289+0000", 2024-09-21T15:51:08.158 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.158 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.158 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.158 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:08.158 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.797289+0000", 2024-09-21T15:51:08.158 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.158 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.158 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.159 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:08.159 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.797288+0000", 2024-09-21T15:51:08.159 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:08.159 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.159 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.159 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:08.159 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.797293+0000", 2024-09-21T15:51:08.159 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4500000000000003e-06 2024-09-21T15:51:08.159 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.159 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.159 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:08.159 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.797303+0000", 2024-09-21T15:51:08.159 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0134e-05 2024-09-21T15:51:08.159 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.160 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.160 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:08.160 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.797309+0000", 2024-09-21T15:51:08.160 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2100000000000001e-06 2024-09-21T15:51:08.160 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.160 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.160 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:08.160 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.797346+0000", 2024-09-21T15:51:08.160 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7583999999999998e-05 2024-09-21T15:51:08.160 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.160 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.160 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:08.160 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.797386+0000", 2024-09-21T15:51:08.160 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9428999999999998e-05 2024-09-21T15:51:08.161 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.161 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.161 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:08.161 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.801816+0000", 2024-09-21T15:51:08.161 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0044305730000000002 2024-09-21T15:51:08.161 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.161 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.161 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:08.161 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.801847+0000", 2024-09-21T15:51:08.161 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0639000000000003e-05 2024-09-21T15:51:08.161 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.161 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:08.161 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.161 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.161 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.162 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:164505 3.5 3:a3a629db:::rbd_data.113336ae1075.0000000000000003:head [delete] snapc 16=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-21T15:51:08.162 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:51:02.863560+0000", 2024-09-21T15:51:08.162 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 5.2060716579999999, 2024-09-21T15:51:08.162 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0040794459999999996, 2024-09-21T15:51:08.162 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:08.162 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:08.162 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:08.162 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:08.162 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:08.162 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 164505 2024-09-21T15:51:08.162 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.162 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:08.162 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.162 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:08.163 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.863560+0000", 2024-09-21T15:51:08.163 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.163 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.163 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.163 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:08.163 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.863560+0000", 2024-09-21T15:51:08.163 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.163 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.163 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.163 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:08.163 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.863558+0000", 2024-09-21T15:51:08.163 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:51:08.163 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.163 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.164 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:08.164 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.863570+0000", 2024-09-21T15:51:08.164 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1960999999999999e-05 2024-09-21T15:51:08.164 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.164 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.164 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:08.164 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.863582+0000", 2024-09-21T15:51:08.164 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1368e-05 2024-09-21T15:51:08.164 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.164 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.164 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:08.164 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.863589+0000", 2024-09-21T15:51:08.164 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1670000000000002e-06 2024-09-21T15:51:08.165 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.165 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.165 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:08.165 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.863646+0000", 2024-09-21T15:51:08.165 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7433999999999999e-05 2024-09-21T15:51:08.165 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.165 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.165 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:08.165 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.863752+0000", 2024-09-21T15:51:08.165 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105214 2024-09-21T15:51:08.165 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.165 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.165 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-09-21T15:51:08.165 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.863866+0000", 2024-09-21T15:51:08.165 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011479100000000001 2024-09-21T15:51:08.166 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.166 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.166 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:08.166 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.864919+0000", 2024-09-21T15:51:08.166 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00105302 2024-09-21T15:51:08.166 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.166 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.166 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:08.166 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.867550+0000", 2024-09-21T15:51:08.166 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0026310130000000002 2024-09-21T15:51:08.166 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.166 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.166 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:08.167 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.867600+0000", 2024-09-21T15:51:08.167 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9123999999999999e-05 2024-09-21T15:51:08.167 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.167 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.167 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:08.167 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.867640+0000", 2024-09-21T15:51:08.167 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9953000000000002e-05 2024-09-21T15:51:08.167 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.167 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:08.167 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.167 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.167 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:08.167 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:08.169 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:51:08.169 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:51:08.168+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:51:08.269 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:51:08.268+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:51:08.313 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:08.313 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:51:08.313 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:51:08.313 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:08.322 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:51:08.370 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:51:08.369+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:51:08.464 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:08.464 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:51:08.464 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:51:08.465 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:51:08.465 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:08.471 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:51:08.470+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:51:08.473 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:51:08.572 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:51:08.572+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:51:08.576 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:08.576 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:51:08.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:51:08.576 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:51:08.576 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.576 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:14051 3.8 3:13437810:::rbd_data.1133dc5adfdf.0000000000000039:head [write 716800~796] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-21T15:51:08.576 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:51:08.576 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 110.87927189299999, 2024-09-21T15:51:08.576 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.98064002699999997, 2024-09-21T15:51:08.577 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:08.577 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:08.577 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:08.577 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:08.577 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:08.577 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 14051 2024-09-21T15:51:08.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.577 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:08.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.577 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:08.577 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:51:08.577 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.577 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.577 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.578 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:08.578 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:51:08.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.578 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:08.578 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:51:08.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:08.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.578 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.578 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:08.578 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695131+0000", 2024-09-21T15:51:08.578 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4680000000000003e-06 2024-09-21T15:51:08.578 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:08.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695139+0000", 2024-09-21T15:51:08.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9400000000000002e-06 2024-09-21T15:51:08.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:08.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695143+0000", 2024-09-21T15:51:08.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9899999999999999e-06 2024-09-21T15:51:08.579 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.579 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.579 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:08.579 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695190+0000", 2024-09-21T15:51:08.579 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6458000000000001e-05 2024-09-21T15:51:08.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:51:08.580 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695201+0000", 2024-09-21T15:51:08.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0964999999999999e-05 2024-09-21T15:51:08.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:08.580 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.669490+0000", 2024-09-21T15:51:08.580 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.97428980499999995 2024-09-21T15:51:08.580 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.580 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.580 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-09-21T15:51:08.581 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.670033+0000", 2024-09-21T15:51:08.581 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00054252599999999999 2024-09-21T15:51:08.581 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.581 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:08.581 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.671095+0000", 2024-09-21T15:51:08.581 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001062183 2024-09-21T15:51:08.581 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.581 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.581 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:08.581 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.671186+0000", 2024-09-21T15:51:08.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1191000000000002e-05 2024-09-21T15:51:08.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.582 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.582 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-09-21T15:51:08.582 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.671346+0000", 2024-09-21T15:51:08.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015980399999999999 2024-09-21T15:51:08.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.582 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.582 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:08.582 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.672221+0000", 2024-09-21T15:51:08.582 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00087471999999999995 2024-09-21T15:51:08.582 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.582 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.582 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:08.582 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.675619+0000", 2024-09-21T15:51:08.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00339858 2024-09-21T15:51:08.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:08.583 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.675670+0000", 2024-09-21T15:51:08.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0664000000000001e-05 2024-09-21T15:51:08.583 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.583 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.583 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:08.583 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.675768+0000", 2024-09-21T15:51:08.583 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7415000000000002e-05 2024-09-21T15:51:08.583 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.583 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:08.583 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.584 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.584 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.584 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116235 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 761856~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:08.584 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048806+0000", 2024-09-21T15:51:08.584 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 72.525593009000005, 2024-09-21T15:51:08.584 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056818556999999999, 2024-09-21T15:51:08.584 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:08.584 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:08.584 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:08.584 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:08.584 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:08.585 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116235 2024-09-21T15:51:08.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.585 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:08.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.585 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:08.585 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048806+0000", 2024-09-21T15:51:08.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.585 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.585 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.585 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:08.585 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048806+0000", 2024-09-21T15:51:08.585 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.586 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.586 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:08.586 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048806+0000", 2024-09-21T15:51:08.586 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:51:08.586 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.586 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:08.586 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048807+0000", 2024-09-21T15:51:08.586 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.001e-06 2024-09-21T15:51:08.586 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.586 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.586 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:08.587 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048810+0000", 2024-09-21T15:51:08.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.711e-06 2024-09-21T15:51:08.587 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.587 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:08.587 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048812+0000", 2024-09-21T15:51:08.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2290000000000002e-06 2024-09-21T15:51:08.587 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.587 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:08.587 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105495+0000", 2024-09-21T15:51:08.587 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056682679999999999 2024-09-21T15:51:08.587 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.587 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.588 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:08.588 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105565+0000", 2024-09-21T15:51:08.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9868000000000003e-05 2024-09-21T15:51:08.588 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.588 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:08.588 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105625+0000", 2024-09-21T15:51:08.588 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0268e-05 2024-09-21T15:51:08.588 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.588 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:08.588 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.588 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.588 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.588 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116236 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 768000~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:08.588 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048814+0000", 2024-09-21T15:51:08.589 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 72.525585346, 2024-09-21T15:51:08.589 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056959587999999998, 2024-09-21T15:51:08.589 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:08.589 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:08.589 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:08.589 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:08.589 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:08.589 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116236 2024-09-21T15:51:08.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.589 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:08.589 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.589 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:08.589 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048814+0000", 2024-09-21T15:51:08.589 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.589 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.590 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:08.590 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048814+0000", 2024-09-21T15:51:08.590 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.590 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:08.590 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048814+0000", 2024-09-21T15:51:08.590 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:51:08.590 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.590 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.590 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:08.590 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048815+0000", 2024-09-21T15:51:08.591 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7999999999999993e-07 2024-09-21T15:51:08.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.591 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.591 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:08.591 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048817+0000", 2024-09-21T15:51:08.591 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6620000000000001e-06 2024-09-21T15:51:08.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.591 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.591 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:08.591 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048820+0000", 2024-09-21T15:51:08.591 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2800000000000002e-06 2024-09-21T15:51:08.591 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.591 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.591 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:08.591 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105647+0000", 2024-09-21T15:51:08.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056827253000000001 2024-09-21T15:51:08.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.592 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:08.592 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105716+0000", 2024-09-21T15:51:08.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8761000000000001e-05 2024-09-21T15:51:08.592 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.592 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.592 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:08.592 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105774+0000", 2024-09-21T15:51:08.592 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7855000000000001e-05 2024-09-21T15:51:08.592 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.592 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:08.592 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.593 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.593 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.593 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116237 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 774144~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:08.593 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048822+0000", 2024-09-21T15:51:08.593 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 72.525577661, 2024-09-21T15:51:08.593 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057114028999999997, 2024-09-21T15:51:08.593 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:08.593 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:08.593 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:08.593 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:08.593 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:08.593 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116237 2024-09-21T15:51:08.593 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.593 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:08.594 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.594 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:08.594 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048822+0000", 2024-09-21T15:51:08.594 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.594 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.594 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.594 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:08.594 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048822+0000", 2024-09-21T15:51:08.594 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.594 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.594 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.594 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:08.594 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048822+0000", 2024-09-21T15:51:08.594 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:51:08.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.595 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.595 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:08.595 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048823+0000", 2024-09-21T15:51:08.595 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9699999999999994e-07 2024-09-21T15:51:08.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.595 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.595 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:08.595 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048825+0000", 2024-09-21T15:51:08.595 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.65e-06 2024-09-21T15:51:08.595 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.595 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.595 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:08.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048828+0000", 2024-09-21T15:51:08.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8890000000000002e-06 2024-09-21T15:51:08.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:08.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105802+0000", 2024-09-21T15:51:08.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056974415 2024-09-21T15:51:08.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.596 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:08.596 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105871+0000", 2024-09-21T15:51:08.596 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8727999999999997e-05 2024-09-21T15:51:08.596 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.596 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.597 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:08.597 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105936+0000", 2024-09-21T15:51:08.597 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4554999999999994e-05 2024-09-21T15:51:08.597 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.597 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:08.597 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.597 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.597 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.597 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116238 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 780288~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:08.597 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048830+0000", 2024-09-21T15:51:08.597 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 72.525569402000002, 2024-09-21T15:51:08.597 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057296574000000003, 2024-09-21T15:51:08.597 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:08.597 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:08.597 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:08.598 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:08.598 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:08.598 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116238 2024-09-21T15:51:08.598 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.598 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:08.598 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.598 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:08.598 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048830+0000", 2024-09-21T15:51:08.598 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.598 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.598 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.598 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:08.598 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048830+0000", 2024-09-21T15:51:08.598 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.599 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.599 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.599 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:08.599 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048830+0000", 2024-09-21T15:51:08.599 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:51:08.599 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.599 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.599 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:08.599 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048831+0000", 2024-09-21T15:51:08.599 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0160000000000001e-06 2024-09-21T15:51:08.599 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.599 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.599 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:08.600 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048834+0000", 2024-09-21T15:51:08.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.813e-06 2024-09-21T15:51:08.600 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.600 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.600 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:08.600 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048836+0000", 2024-09-21T15:51:08.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6790000000000001e-06 2024-09-21T15:51:08.600 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.600 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.600 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:08.600 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105961+0000", 2024-09-21T15:51:08.600 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057124705999999997 2024-09-21T15:51:08.600 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.600 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.600 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:08.601 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106042+0000", 2024-09-21T15:51:08.601 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0820000000000002e-05 2024-09-21T15:51:08.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.601 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.601 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:08.601 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106127+0000", 2024-09-21T15:51:08.601 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4746000000000005e-05 2024-09-21T15:51:08.601 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.601 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:08.601 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.601 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.601 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.601 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116324 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 524288~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:08.601 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048838+0000", 2024-09-21T15:51:08.601 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 72.525561092999993, 2024-09-21T15:51:08.602 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057477185, 2024-09-21T15:51:08.602 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:08.602 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:08.602 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:08.602 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:08.602 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:08.602 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116324 2024-09-21T15:51:08.602 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.602 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:08.602 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.602 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:08.602 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048838+0000", 2024-09-21T15:51:08.602 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.602 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.602 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.603 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:08.603 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048838+0000", 2024-09-21T15:51:08.603 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.603 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:08.603 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048838+0000", 2024-09-21T15:51:08.603 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:51:08.603 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.603 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.603 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:08.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048842+0000", 2024-09-21T15:51:08.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4829999999999999e-06 2024-09-21T15:51:08.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.604 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:08.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048846+0000", 2024-09-21T15:51:08.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0470000000000004e-06 2024-09-21T15:51:08.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.604 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.604 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:08.604 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048849+0000", 2024-09-21T15:51:08.604 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3689999999999998e-06 2024-09-21T15:51:08.604 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.605 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.605 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:08.605 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106157+0000", 2024-09-21T15:51:08.605 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057307942000000001 2024-09-21T15:51:08.605 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.605 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.605 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:08.605 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106237+0000", 2024-09-21T15:51:08.605 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0320000000000003e-05 2024-09-21T15:51:08.605 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.605 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.605 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:08.605 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106315+0000", 2024-09-21T15:51:08.605 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8227999999999997e-05 2024-09-21T15:51:08.606 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.606 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:08.606 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.606 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.606 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.606 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116325 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 530432~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:08.606 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048851+0000", 2024-09-21T15:51:08.606 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 72.525547920999998, 2024-09-21T15:51:08.606 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057581036000000002, 2024-09-21T15:51:08.606 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:08.606 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:08.606 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:08.606 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:08.607 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:08.607 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116325 2024-09-21T15:51:08.607 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.607 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:08.607 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.607 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:08.607 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048851+0000", 2024-09-21T15:51:08.607 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.607 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.608 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.608 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:08.608 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048851+0000", 2024-09-21T15:51:08.608 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.608 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.608 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.608 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:08.608 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048851+0000", 2024-09-21T15:51:08.608 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:51:08.608 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.608 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.608 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:08.608 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048852+0000", 2024-09-21T15:51:08.608 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.077e-06 2024-09-21T15:51:08.609 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.609 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.609 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:08.609 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048855+0000", 2024-09-21T15:51:08.609 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9859999999999999e-06 2024-09-21T15:51:08.609 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.609 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.609 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:08.609 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048858+0000", 2024-09-21T15:51:08.609 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3810000000000002e-06 2024-09-21T15:51:08.609 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.609 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.609 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:08.609 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106335+0000", 2024-09-21T15:51:08.609 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057477473000000001 2024-09-21T15:51:08.610 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.610 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.610 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:08.610 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106393+0000", 2024-09-21T15:51:08.610 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8050000000000002e-05 2024-09-21T15:51:08.610 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.610 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.610 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:08.610 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106432+0000", 2024-09-21T15:51:08.610 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9274999999999999e-05 2024-09-21T15:51:08.610 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.610 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:08.610 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.610 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.611 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.611 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116326 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 536576~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:08.611 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048860+0000", 2024-09-21T15:51:08.611 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 72.525539870000003, 2024-09-21T15:51:08.611 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057774197999999999, 2024-09-21T15:51:08.611 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:08.611 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:08.611 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:08.611 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:08.611 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:08.611 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116326 2024-09-21T15:51:08.611 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.611 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:08.611 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.611 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:08.612 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048860+0000", 2024-09-21T15:51:08.612 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.612 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.612 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.612 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:08.612 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048860+0000", 2024-09-21T15:51:08.612 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.612 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.612 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.612 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:08.612 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048859+0000", 2024-09-21T15:51:08.612 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:51:08.612 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.612 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.612 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:08.612 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048860+0000", 2024-09-21T15:51:08.613 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.02e-06 2024-09-21T15:51:08.613 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.613 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.613 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:08.613 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048863+0000", 2024-09-21T15:51:08.613 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1039999999999998e-06 2024-09-21T15:51:08.613 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.613 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.613 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:08.613 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048867+0000", 2024-09-21T15:51:08.613 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.563e-06 2024-09-21T15:51:08.613 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.613 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.613 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:08.613 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106465+0000", 2024-09-21T15:51:08.614 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057597789000000003 2024-09-21T15:51:08.614 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.614 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.614 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:08.614 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106552+0000", 2024-09-21T15:51:08.614 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7602000000000006e-05 2024-09-21T15:51:08.614 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.614 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.614 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:08.614 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106634+0000", 2024-09-21T15:51:08.614 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1321000000000003e-05 2024-09-21T15:51:08.614 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.614 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:08.614 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.614 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.615 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.615 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116327 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 542720~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:08.615 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048870+0000", 2024-09-21T15:51:08.615 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 72.525528997999999, 2024-09-21T15:51:08.615 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057883148000000002, 2024-09-21T15:51:08.615 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:08.615 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:08.615 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:08.615 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:08.615 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:08.615 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116327 2024-09-21T15:51:08.615 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.615 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:08.615 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.616 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:08.616 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048870+0000", 2024-09-21T15:51:08.616 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.616 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.616 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.616 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:08.616 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048870+0000", 2024-09-21T15:51:08.616 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.616 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.616 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.616 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:08.616 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048870+0000", 2024-09-21T15:51:08.616 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:08.616 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.616 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.616 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:08.617 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048872+0000", 2024-09-21T15:51:08.617 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.824e-06 2024-09-21T15:51:08.617 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.617 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.617 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:08.617 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048879+0000", 2024-09-21T15:51:08.617 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6749999999999996e-06 2024-09-21T15:51:08.617 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.617 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.617 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:08.618 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048883+0000", 2024-09-21T15:51:08.618 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2200000000000003e-06 2024-09-21T15:51:08.618 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.618 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.618 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:08.618 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106663+0000", 2024-09-21T15:51:08.618 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057780049 2024-09-21T15:51:08.618 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.618 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.618 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:08.618 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106711+0000", 2024-09-21T15:51:08.618 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8433999999999997e-05 2024-09-21T15:51:08.618 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.619 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.619 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:08.619 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106754+0000", 2024-09-21T15:51:08.619 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2236e-05 2024-09-21T15:51:08.619 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.619 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:08.619 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.619 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.619 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.619 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116328 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 548864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:08.619 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048887+0000", 2024-09-21T15:51:08.619 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 72.525512477999996, 2024-09-21T15:51:08.619 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057981984, 2024-09-21T15:51:08.620 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:08.620 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:08.620 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:08.620 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:08.620 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:08.620 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116328 2024-09-21T15:51:08.620 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.620 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:08.620 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.620 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:08.620 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048887+0000", 2024-09-21T15:51:08.620 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.620 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.620 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.621 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:08.621 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048887+0000", 2024-09-21T15:51:08.621 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.621 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.621 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.621 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:08.621 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048887+0000", 2024-09-21T15:51:08.621 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:08.621 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.621 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.621 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:08.621 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048891+0000", 2024-09-21T15:51:08.621 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.843e-06 2024-09-21T15:51:08.622 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.622 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.622 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:08.622 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048897+0000", 2024-09-21T15:51:08.622 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5419999999999996e-06 2024-09-21T15:51:08.622 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.622 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.622 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:08.622 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048901+0000", 2024-09-21T15:51:08.622 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4689999999999999e-06 2024-09-21T15:51:08.622 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.622 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.622 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:08.623 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106780+0000", 2024-09-21T15:51:08.623 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057878259000000001 2024-09-21T15:51:08.623 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.623 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.623 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:08.623 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106827+0000", 2024-09-21T15:51:08.623 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7098000000000002e-05 2024-09-21T15:51:08.623 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.623 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.623 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:08.623 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106869+0000", 2024-09-21T15:51:08.623 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2129999999999998e-05 2024-09-21T15:51:08.623 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.624 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:08.624 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.624 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.624 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.624 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116329 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 555008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:08.624 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048905+0000", 2024-09-21T15:51:08.624 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 72.525494367999997, 2024-09-21T15:51:08.624 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058115554999999999, 2024-09-21T15:51:08.624 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:08.624 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:08.624 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:08.624 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:08.624 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:08.624 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116329 2024-09-21T15:51:08.625 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.625 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:08.625 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.625 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:08.625 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048905+0000", 2024-09-21T15:51:08.625 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.625 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.625 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.625 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:08.625 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048905+0000", 2024-09-21T15:51:08.625 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.625 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.625 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.625 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:08.626 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048905+0000", 2024-09-21T15:51:08.626 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:08.626 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.626 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.626 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:08.626 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048907+0000", 2024-09-21T15:51:08.626 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.086e-06 2024-09-21T15:51:08.626 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.626 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.626 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:08.626 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048912+0000", 2024-09-21T15:51:08.626 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2830000000000001e-06 2024-09-21T15:51:08.626 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.626 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.627 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:08.627 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048916+0000", 2024-09-21T15:51:08.627 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.98e-06 2024-09-21T15:51:08.627 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.627 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.627 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:08.627 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106907+0000", 2024-09-21T15:51:08.627 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057990719000000003 2024-09-21T15:51:08.627 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.627 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.627 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:08.627 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106969+0000", 2024-09-21T15:51:08.627 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1789000000000001e-05 2024-09-21T15:51:08.628 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.628 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.628 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:08.628 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107021+0000", 2024-09-21T15:51:08.628 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2030999999999997e-05 2024-09-21T15:51:08.628 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.628 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:08.628 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.628 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.628 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.628 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116330 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 561152~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:08.628 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048922+0000", 2024-09-21T15:51:08.628 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 72.525477210000005, 2024-09-21T15:51:08.628 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058242343000000002, 2024-09-21T15:51:08.628 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:08.629 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:08.629 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:08.629 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:08.629 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:08.629 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116330 2024-09-21T15:51:08.629 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.629 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:08.629 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.629 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:08.629 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048922+0000", 2024-09-21T15:51:08.629 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.629 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.629 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.629 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:08.630 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048922+0000", 2024-09-21T15:51:08.630 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.630 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.630 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.630 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:08.630 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048922+0000", 2024-09-21T15:51:08.630 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:08.630 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.630 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.630 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:08.630 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048924+0000", 2024-09-21T15:51:08.630 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1660000000000001e-06 2024-09-21T15:51:08.630 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.630 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.630 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:08.631 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048930+0000", 2024-09-21T15:51:08.631 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4529999999999999e-06 2024-09-21T15:51:08.631 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.631 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.631 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:08.631 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048934+0000", 2024-09-21T15:51:08.631 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0269999999999997e-06 2024-09-21T15:51:08.631 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.631 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.631 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:08.631 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107047+0000", 2024-09-21T15:51:08.631 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058112913000000002 2024-09-21T15:51:08.631 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.631 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.631 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:08.632 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107095+0000", 2024-09-21T15:51:08.632 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7778000000000002e-05 2024-09-21T15:51:08.632 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.632 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.632 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:08.632 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107165+0000", 2024-09-21T15:51:08.632 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9370999999999997e-05 2024-09-21T15:51:08.632 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.632 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:08.632 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.632 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.632 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.632 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116331 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 567296~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:08.632 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.049440+0000", 2024-09-21T15:51:08.633 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 72.524959116000005, 2024-09-21T15:51:08.633 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057878485, 2024-09-21T15:51:08.633 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:08.633 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:08.633 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:08.633 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:08.633 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:08.633 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116331 2024-09-21T15:51:08.633 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.633 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:08.633 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.633 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:08.633 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049440+0000", 2024-09-21T15:51:08.633 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.633 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.634 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.634 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:08.634 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049440+0000", 2024-09-21T15:51:08.634 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.634 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.634 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.634 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:08.634 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048939+0000", 2024-09-21T15:51:08.634 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9994984 2024-09-21T15:51:08.634 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.634 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.634 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:08.634 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049444+0000", 2024-09-21T15:51:08.634 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00050557200000000003 2024-09-21T15:51:08.635 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.635 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.635 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:08.635 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049460+0000", 2024-09-21T15:51:08.635 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6120000000000002e-05 2024-09-21T15:51:08.635 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.635 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.635 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:08.635 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049468+0000", 2024-09-21T15:51:08.635 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4209999999999998e-06 2024-09-21T15:51:08.635 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.635 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.635 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:08.635 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107189+0000", 2024-09-21T15:51:08.635 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057721019999999998 2024-09-21T15:51:08.636 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.636 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.636 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:08.636 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107254+0000", 2024-09-21T15:51:08.636 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5265000000000006e-05 2024-09-21T15:51:08.636 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.636 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.636 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:08.636 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107319+0000", 2024-09-21T15:51:08.636 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4784999999999997e-05 2024-09-21T15:51:08.636 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.636 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:08.636 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.636 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.636 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.637 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116332 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 573440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:08.637 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.049474+0000", 2024-09-21T15:51:08.637 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 72.524925855000006, 2024-09-21T15:51:08.637 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058013809, 2024-09-21T15:51:08.637 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:08.637 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:08.637 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:08.637 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:08.637 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:08.637 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116332 2024-09-21T15:51:08.637 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.637 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:08.637 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.637 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:08.638 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049474+0000", 2024-09-21T15:51:08.638 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.638 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.638 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.638 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:08.638 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049474+0000", 2024-09-21T15:51:08.638 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.638 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.638 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.638 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:08.638 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049473+0000", 2024-09-21T15:51:08.638 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:08.638 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.638 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.638 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:08.639 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049476+0000", 2024-09-21T15:51:08.639 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6249999999999999e-06 2024-09-21T15:51:08.639 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.639 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.639 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:08.639 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049482+0000", 2024-09-21T15:51:08.639 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0959999999999997e-06 2024-09-21T15:51:08.639 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.639 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.639 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:08.639 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049487+0000", 2024-09-21T15:51:08.639 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5949999999999998e-06 2024-09-21T15:51:08.639 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.639 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.639 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:08.640 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107344+0000", 2024-09-21T15:51:08.640 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057856594999999997 2024-09-21T15:51:08.640 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.640 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.640 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:08.640 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107422+0000", 2024-09-21T15:51:08.640 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7582000000000004e-05 2024-09-21T15:51:08.640 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.640 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.640 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:08.640 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107487+0000", 2024-09-21T15:51:08.640 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5723999999999996e-05 2024-09-21T15:51:08.640 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.640 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:08.641 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.641 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.641 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.641 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116333 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 579584~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:08.641 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.049993+0000", 2024-09-21T15:51:08.641 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 72.524406072000005, 2024-09-21T15:51:08.641 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057663537000000001, 2024-09-21T15:51:08.641 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:08.641 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:08.641 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:08.641 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:08.641 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:08.641 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116333 2024-09-21T15:51:08.641 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.641 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:08.642 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.642 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:08.642 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049993+0000", 2024-09-21T15:51:08.642 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.642 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.642 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.642 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:08.642 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049993+0000", 2024-09-21T15:51:08.642 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.642 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.642 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.642 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:08.642 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049491+0000", 2024-09-21T15:51:08.642 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9994979 2024-09-21T15:51:08.643 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.643 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.643 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:08.643 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049998+0000", 2024-09-21T15:51:08.643 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00050654200000000002 2024-09-21T15:51:08.643 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.643 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.643 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:08.643 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050010+0000", 2024-09-21T15:51:08.643 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1790999999999999e-05 2024-09-21T15:51:08.643 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.643 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.643 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:08.643 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050016+0000", 2024-09-21T15:51:08.643 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2179999999999996e-06 2024-09-21T15:51:08.644 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.644 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.644 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:08.644 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107513+0000", 2024-09-21T15:51:08.644 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057496927000000003 2024-09-21T15:51:08.644 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.644 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.644 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:08.644 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107590+0000", 2024-09-21T15:51:08.644 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7547999999999997e-05 2024-09-21T15:51:08.644 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.644 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.644 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:08.644 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107657+0000", 2024-09-21T15:51:08.644 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6630000000000004e-05 2024-09-21T15:51:08.645 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.645 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:08.645 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.645 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.645 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116334 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 585728~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:08.645 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.050091+0000", 2024-09-21T15:51:08.645 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 72.524308443999999, 2024-09-21T15:51:08.645 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057734355000000001, 2024-09-21T15:51:08.645 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:08.645 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:08.645 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:08.645 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:08.645 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:08.646 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116334 2024-09-21T15:51:08.646 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.646 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:08.646 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.646 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:08.646 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050091+0000", 2024-09-21T15:51:08.646 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.646 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.646 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.646 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:08.646 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050091+0000", 2024-09-21T15:51:08.646 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.646 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.646 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.647 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:08.647 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050021+0000", 2024-09-21T15:51:08.647 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999304 2024-09-21T15:51:08.647 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.647 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.647 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:08.647 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050095+0000", 2024-09-21T15:51:08.647 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4158000000000004e-05 2024-09-21T15:51:08.647 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.647 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.647 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:08.647 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050103+0000", 2024-09-21T15:51:08.647 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0770000000000002e-06 2024-09-21T15:51:08.647 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.647 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.648 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:08.648 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050108+0000", 2024-09-21T15:51:08.648 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6330000000000001e-06 2024-09-21T15:51:08.648 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.648 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.648 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:08.648 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107681+0000", 2024-09-21T15:51:08.648 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057572996000000001 2024-09-21T15:51:08.648 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.648 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.648 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:08.648 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107759+0000", 2024-09-21T15:51:08.648 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7368999999999997e-05 2024-09-21T15:51:08.648 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.648 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.649 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:08.649 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107825+0000", 2024-09-21T15:51:08.649 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6783000000000001e-05 2024-09-21T15:51:08.649 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.649 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:08.649 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.649 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.649 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.649 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116335 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 591872~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:08.649 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.050542+0000", 2024-09-21T15:51:08.649 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 72.523857859000003, 2024-09-21T15:51:08.649 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057466455999999999, 2024-09-21T15:51:08.649 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:08.649 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:08.649 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:08.650 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:08.650 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:08.650 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116335 2024-09-21T15:51:08.650 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.650 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:08.650 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.650 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:08.650 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050542+0000", 2024-09-21T15:51:08.650 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.650 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.650 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.650 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:08.650 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050542+0000", 2024-09-21T15:51:08.651 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.651 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.651 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.651 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:08.651 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050113+0000", 2024-09-21T15:51:08.651 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9995713 2024-09-21T15:51:08.651 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.651 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.651 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:08.651 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050546+0000", 2024-09-21T15:51:08.651 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00043328699999999997 2024-09-21T15:51:08.651 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.651 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.651 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:08.651 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050557+0000", 2024-09-21T15:51:08.652 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1011e-05 2024-09-21T15:51:08.652 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.652 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.652 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:08.652 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050564+0000", 2024-09-21T15:51:08.652 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1130000000000003e-06 2024-09-21T15:51:08.652 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.652 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.652 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:08.652 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107852+0000", 2024-09-21T15:51:08.652 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057287350000000001 2024-09-21T15:51:08.652 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.652 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.652 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:08.652 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107928+0000", 2024-09-21T15:51:08.653 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6668000000000005e-05 2024-09-21T15:51:08.653 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.653 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.653 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:08.653 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108008+0000", 2024-09-21T15:51:08.653 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9783999999999994e-05 2024-09-21T15:51:08.653 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.653 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:08.653 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.653 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.653 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.653 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116336 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 598016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:08.653 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.050834+0000", 2024-09-21T15:51:08.653 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 72.523565593000001, 2024-09-21T15:51:08.654 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057353572999999998, 2024-09-21T15:51:08.654 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:08.654 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:08.654 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:08.654 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:08.654 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:08.654 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116336 2024-09-21T15:51:08.654 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.654 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:08.654 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.654 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:08.654 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050834+0000", 2024-09-21T15:51:08.654 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.654 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.655 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.655 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:08.655 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050834+0000", 2024-09-21T15:51:08.655 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.655 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.655 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.655 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:08.655 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050571+0000", 2024-09-21T15:51:08.655 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9997373 2024-09-21T15:51:08.655 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.655 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.655 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:08.655 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050842+0000", 2024-09-21T15:51:08.655 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00027029099999999998 2024-09-21T15:51:08.656 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.656 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.656 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:08.656 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050853+0000", 2024-09-21T15:51:08.656 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1532000000000001e-05 2024-09-21T15:51:08.656 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.656 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.656 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:08.656 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050863+0000", 2024-09-21T15:51:08.656 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4239999999999999e-06 2024-09-21T15:51:08.656 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.656 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.656 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:08.656 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108045+0000", 2024-09-21T15:51:08.656 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057182415 2024-09-21T15:51:08.657 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.657 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.657 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:08.657 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108123+0000", 2024-09-21T15:51:08.657 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8275000000000005e-05 2024-09-21T15:51:08.657 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.657 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.657 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:08.657 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108187+0000", 2024-09-21T15:51:08.657 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4152000000000005e-05 2024-09-21T15:51:08.657 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.657 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:08.657 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.657 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.657 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.658 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116337 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 604160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:08.658 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.051192+0000", 2024-09-21T15:51:08.658 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 72.523206965, 2024-09-21T15:51:08.658 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057175981000000001, 2024-09-21T15:51:08.658 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:08.658 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:08.658 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:08.658 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:08.658 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:08.658 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116337 2024-09-21T15:51:08.658 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.658 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:08.658 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.658 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:08.659 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051192+0000", 2024-09-21T15:51:08.659 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.659 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.659 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.659 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:08.659 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051192+0000", 2024-09-21T15:51:08.659 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.659 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.659 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.659 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:08.659 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050870+0000", 2024-09-21T15:51:08.659 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9996777 2024-09-21T15:51:08.659 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.659 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.659 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:08.660 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051197+0000", 2024-09-21T15:51:08.660 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00032656299999999998 2024-09-21T15:51:08.660 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.660 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.660 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:08.660 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051209+0000", 2024-09-21T15:51:08.660 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2636000000000001e-05 2024-09-21T15:51:08.660 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.660 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.660 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:08.660 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051217+0000", 2024-09-21T15:51:08.660 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5989999999999996e-06 2024-09-21T15:51:08.660 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.660 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.661 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:08.661 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108224+0000", 2024-09-21T15:51:08.661 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057007622000000001 2024-09-21T15:51:08.661 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.661 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.661 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:08.661 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108299+0000", 2024-09-21T15:51:08.661 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5099999999999996e-05 2024-09-21T15:51:08.661 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.661 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.661 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:08.661 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108368+0000", 2024-09-21T15:51:08.661 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8918000000000007e-05 2024-09-21T15:51:08.661 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.661 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:08.662 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.662 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.662 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.662 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116338 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 610304~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:08.662 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.051583+0000", 2024-09-21T15:51:08.662 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 72.522816512000006, 2024-09-21T15:51:08.662 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056967716000000002, 2024-09-21T15:51:08.662 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:08.662 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:08.662 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:08.662 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:08.662 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:08.662 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116338 2024-09-21T15:51:08.662 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.663 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:08.663 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.663 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:08.663 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051583+0000", 2024-09-21T15:51:08.663 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.663 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.663 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.663 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:08.663 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051583+0000", 2024-09-21T15:51:08.663 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:08.663 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.663 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.663 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:08.663 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051223+0000", 2024-09-21T15:51:08.663 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.99964 2024-09-21T15:51:08.664 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.664 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.664 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:08.664 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051587+0000", 2024-09-21T15:51:08.664 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00036352999999999998 2024-09-21T15:51:08.664 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.664 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.664 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:08.664 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051597+0000", 2024-09-21T15:51:08.664 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.893e-06 2024-09-21T15:51:08.664 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.664 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.664 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:08.664 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051603+0000", 2024-09-21T15:51:08.665 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0519999999999998e-06 2024-09-21T15:51:08.665 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.665 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.665 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:08.665 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108394+0000", 2024-09-21T15:51:08.665 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056791531999999999 2024-09-21T15:51:08.665 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.665 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.665 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:08.665 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108480+0000", 2024-09-21T15:51:08.665 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.577e-05 2024-09-21T15:51:08.665 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:08.665 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:08.665 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:08.665 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108551+0000", 2024-09-21T15:51:08.666 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0722000000000005e-05 2024-09-21T15:51:08.666 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.666 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:08.666 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.666 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:08.666 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:08.666 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:08.667 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:51:08.711 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:51:08.710+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:51:08.774 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:51:08.773+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:51:08.782 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:51:08.783 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:51:08.783 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:51:08.783 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:51:08.791 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:51:08.845 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T15:51:08.849 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:51:08.889 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:51:08.888+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:51:08.934 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:51:08.935 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:51:08.935 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:51:08.935 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:51:08.935 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:51:08.943 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:51:08.987 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:51:08.986+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:51:09.028 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-09-21T15:51:09.028 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:51:09.049 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:51:09.050 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:51:09.055 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:51:09.055 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:51:09.055 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:51:09.055 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:51:09.055 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.055 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155031 3.1c 3.8612741c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:09.055 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448748+0000", 2024-09-21T15:51:09.055 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 46.604298565000001, 2024-09-21T15:51:09.055 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064010286999999999, 2024-09-21T15:51:09.055 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:09.056 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:09.056 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:09.056 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:09.056 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:09.056 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155031 2024-09-21T15:51:09.056 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.056 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:09.056 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.056 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:09.056 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448748+0000", 2024-09-21T15:51:09.056 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:09.056 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.056 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.056 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:09.057 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448748+0000", 2024-09-21T15:51:09.057 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:09.057 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.057 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.057 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:09.057 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448747+0000", 2024-09-21T15:51:09.057 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:09.057 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.057 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.057 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:09.057 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448753+0000", 2024-09-21T15:51:09.057 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.755e-06 2024-09-21T15:51:09.057 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.057 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.057 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:09.058 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448763+0000", 2024-09-21T15:51:09.058 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7659999999999993e-06 2024-09-21T15:51:09.058 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.058 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.058 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:09.058 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448769+0000", 2024-09-21T15:51:09.058 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6060000000000002e-06 2024-09-21T15:51:09.058 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.058 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.058 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:09.058 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512750+0000", 2024-09-21T15:51:09.058 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.063981298000000006 2024-09-21T15:51:09.058 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.058 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.059 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:09.059 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512758+0000", 2024-09-21T15:51:09.059 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1669999999999999e-06 2024-09-21T15:51:09.059 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:09.059 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:09.059 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:09.059 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.059 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.059 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155091 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:09.059 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448916+0000", 2024-09-21T15:51:09.059 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 46.604130720000001, 2024-09-21T15:51:09.059 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064006309999999997, 2024-09-21T15:51:09.060 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:09.060 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:09.060 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:09.060 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:09.060 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:09.060 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155091 2024-09-21T15:51:09.060 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.060 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:09.060 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.060 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:09.060 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448916+0000", 2024-09-21T15:51:09.060 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:09.060 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.060 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.061 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:09.061 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448916+0000", 2024-09-21T15:51:09.061 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:09.061 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.061 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.061 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:09.061 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448915+0000", 2024-09-21T15:51:09.061 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:09.061 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.061 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.061 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:09.061 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448917+0000", 2024-09-21T15:51:09.061 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2400000000000002e-06 2024-09-21T15:51:09.061 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.061 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.062 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:09.062 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448923+0000", 2024-09-21T15:51:09.062 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0580000000000003e-06 2024-09-21T15:51:09.062 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.062 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.062 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:09.062 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448927+0000", 2024-09-21T15:51:09.062 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.031e-06 2024-09-21T15:51:09.063 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.063 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.063 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:09.063 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512916+0000", 2024-09-21T15:51:09.063 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.063988456999999999 2024-09-21T15:51:09.063 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.063 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.063 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:09.063 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512922+0000", 2024-09-21T15:51:09.063 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9009999999999999e-06 2024-09-21T15:51:09.063 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:09.063 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:09.063 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:09.063 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.064 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.064 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155092 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:09.064 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448931+0000", 2024-09-21T15:51:09.064 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 46.604114938000002, 2024-09-21T15:51:09.064 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064022793999999994, 2024-09-21T15:51:09.064 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:09.064 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:09.064 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:09.064 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:09.064 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:09.064 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155092 2024-09-21T15:51:09.064 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.064 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:09.064 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.064 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:09.064 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448931+0000", 2024-09-21T15:51:09.065 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:09.065 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.065 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.065 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:09.065 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448931+0000", 2024-09-21T15:51:09.065 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:09.065 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.065 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.065 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:09.065 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448931+0000", 2024-09-21T15:51:09.065 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:09.065 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.065 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.065 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:09.065 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448933+0000", 2024-09-21T15:51:09.065 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1799999999999999e-06 2024-09-21T15:51:09.066 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.066 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.066 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:09.066 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448939+0000", 2024-09-21T15:51:09.066 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.604e-06 2024-09-21T15:51:09.066 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.066 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.066 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:09.066 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448943+0000", 2024-09-21T15:51:09.066 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0740000000000003e-06 2024-09-21T15:51:09.066 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.066 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.066 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:09.066 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512946+0000", 2024-09-21T15:51:09.066 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064003612000000001 2024-09-21T15:51:09.067 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.067 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.067 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:09.067 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512954+0000", 2024-09-21T15:51:09.067 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6690000000000006e-06 2024-09-21T15:51:09.067 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:09.067 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:09.067 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:09.067 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.067 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.067 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155093 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:09.067 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448947+0000", 2024-09-21T15:51:09.067 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 46.604099564000002, 2024-09-21T15:51:09.067 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064020717000000005, 2024-09-21T15:51:09.067 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:09.067 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:09.068 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:09.068 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:09.068 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:09.068 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155093 2024-09-21T15:51:09.068 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.068 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:09.068 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.068 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:09.068 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448947+0000", 2024-09-21T15:51:09.068 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:09.068 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.068 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.068 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:09.068 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448947+0000", 2024-09-21T15:51:09.068 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:09.068 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.069 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.069 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:09.069 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448946+0000", 2024-09-21T15:51:09.069 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:09.069 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.069 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.069 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:09.069 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448949+0000", 2024-09-21T15:51:09.069 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3860000000000001e-06 2024-09-21T15:51:09.069 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.069 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.069 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:09.069 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448955+0000", 2024-09-21T15:51:09.069 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.823e-06 2024-09-21T15:51:09.069 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.069 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.070 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:09.070 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448959+0000", 2024-09-21T15:51:09.070 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2799999999999997e-06 2024-09-21T15:51:09.070 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.070 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.070 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:09.070 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512964+0000", 2024-09-21T15:51:09.070 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064004741000000004 2024-09-21T15:51:09.070 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.070 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.070 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:09.070 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512967+0000", 2024-09-21T15:51:09.070 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.839e-06 2024-09-21T15:51:09.070 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:09.070 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:09.071 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:09.071 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.071 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.071 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155094 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:09.071 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448962+0000", 2024-09-21T15:51:09.071 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 46.604083940999999, 2024-09-21T15:51:09.071 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064037378000000006, 2024-09-21T15:51:09.071 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:09.071 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:09.071 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:09.071 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:09.071 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:09.071 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155094 2024-09-21T15:51:09.071 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.071 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:09.071 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.072 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:09.072 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448962+0000", 2024-09-21T15:51:09.072 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:09.072 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.072 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.072 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:09.072 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448962+0000", 2024-09-21T15:51:09.072 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:09.072 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.072 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.072 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:09.072 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448962+0000", 2024-09-21T15:51:09.072 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:09.072 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.072 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.072 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:09.073 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448968+0000", 2024-09-21T15:51:09.073 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9710000000000001e-06 2024-09-21T15:51:09.073 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.073 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.073 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:09.073 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448974+0000", 2024-09-21T15:51:09.073 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7230000000000001e-06 2024-09-21T15:51:09.073 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.073 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.073 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:09.073 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448978+0000", 2024-09-21T15:51:09.073 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3749999999999996e-06 2024-09-21T15:51:09.073 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.073 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.074 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:09.074 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512992+0000", 2024-09-21T15:51:09.074 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064013525000000002 2024-09-21T15:51:09.074 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.074 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.074 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:09.074 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513000+0000", 2024-09-21T15:51:09.074 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0830000000000004e-06 2024-09-21T15:51:09.074 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:09.074 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:09.074 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:09.074 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.074 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.074 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155095 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:09.074 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448982+0000", 2024-09-21T15:51:09.075 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 46.604064659000002, 2024-09-21T15:51:09.075 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064041904999999996, 2024-09-21T15:51:09.075 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:09.075 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:09.075 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:09.075 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:09.075 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:09.075 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155095 2024-09-21T15:51:09.075 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.075 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:09.075 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.075 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:09.075 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448982+0000", 2024-09-21T15:51:09.075 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:09.075 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.076 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.076 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:09.076 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448982+0000", 2024-09-21T15:51:09.076 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:09.076 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.076 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.076 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:09.076 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448981+0000", 2024-09-21T15:51:09.076 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:09.076 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.076 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.076 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:09.076 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448983+0000", 2024-09-21T15:51:09.076 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.905e-06 2024-09-21T15:51:09.076 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.076 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.077 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:09.077 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448989+0000", 2024-09-21T15:51:09.077 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.536e-06 2024-09-21T15:51:09.077 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.077 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.077 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:09.077 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448993+0000", 2024-09-21T15:51:09.077 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2390000000000004e-06 2024-09-21T15:51:09.077 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.077 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.077 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:09.077 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513015+0000", 2024-09-21T15:51:09.077 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064022081999999994 2024-09-21T15:51:09.077 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.077 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.077 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:09.078 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513023+0000", 2024-09-21T15:51:09.078 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4500000000000004e-06 2024-09-21T15:51:09.078 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:09.078 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:09.078 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:09.078 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.078 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.078 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155096 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:09.078 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448996+0000", 2024-09-21T15:51:09.078 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 46.604049758000002, 2024-09-21T15:51:09.078 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064060868000000007, 2024-09-21T15:51:09.078 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:09.078 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:09.078 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:09.078 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:09.079 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:09.079 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155096 2024-09-21T15:51:09.079 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.079 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:09.079 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.079 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:09.079 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448996+0000", 2024-09-21T15:51:09.079 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:09.079 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.079 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.079 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:09.079 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448996+0000", 2024-09-21T15:51:09.079 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:09.079 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.079 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.079 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:09.080 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448996+0000", 2024-09-21T15:51:09.080 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:09.080 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.080 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.080 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:09.080 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448998+0000", 2024-09-21T15:51:09.080 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8780000000000001e-06 2024-09-21T15:51:09.080 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.080 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.080 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:09.080 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449003+0000", 2024-09-21T15:51:09.080 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4099999999999999e-06 2024-09-21T15:51:09.080 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.080 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.080 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:09.081 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449008+0000", 2024-09-21T15:51:09.081 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.335e-06 2024-09-21T15:51:09.081 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.081 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.081 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:09.081 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513049+0000", 2024-09-21T15:51:09.081 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064041059999999997 2024-09-21T15:51:09.081 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.081 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.081 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:09.081 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513057+0000", 2024-09-21T15:51:09.081 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5050000000000007e-06 2024-09-21T15:51:09.081 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:09.081 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:09.081 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:09.082 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.082 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.082 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155097 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:09.082 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449019+0000", 2024-09-21T15:51:09.082 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 46.604027336999998, 2024-09-21T15:51:09.082 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064052488000000005, 2024-09-21T15:51:09.082 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:09.082 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:09.082 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:09.082 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:09.082 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:09.082 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155097 2024-09-21T15:51:09.082 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.082 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:09.083 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.083 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:09.083 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449019+0000", 2024-09-21T15:51:09.083 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:09.083 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.083 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.083 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:09.083 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449019+0000", 2024-09-21T15:51:09.083 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:09.083 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.083 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.083 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:09.083 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449019+0000", 2024-09-21T15:51:09.083 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:09.083 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.084 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.084 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:09.084 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449021+0000", 2024-09-21T15:51:09.084 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.244e-06 2024-09-21T15:51:09.084 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.084 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.084 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:09.084 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449027+0000", 2024-09-21T15:51:09.084 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8619999999999999e-06 2024-09-21T15:51:09.084 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.084 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.084 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:09.084 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449031+0000", 2024-09-21T15:51:09.084 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1359999999999999e-06 2024-09-21T15:51:09.084 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.085 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.085 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:09.085 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513067+0000", 2024-09-21T15:51:09.085 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064036391999999998 2024-09-21T15:51:09.085 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.085 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.085 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:09.085 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513071+0000", 2024-09-21T15:51:09.085 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1989999999999999e-06 2024-09-21T15:51:09.085 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:09.085 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:09.085 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:09.085 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.085 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.085 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155098 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:09.086 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449035+0000", 2024-09-21T15:51:09.086 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 46.604011739999997, 2024-09-21T15:51:09.086 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064071761000000005, 2024-09-21T15:51:09.086 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:09.086 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:09.086 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:09.086 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:09.086 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:09.086 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155098 2024-09-21T15:51:09.086 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.086 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:09.086 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.086 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:09.086 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449035+0000", 2024-09-21T15:51:09.087 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:09.087 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.087 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.087 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:09.087 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449035+0000", 2024-09-21T15:51:09.087 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:09.087 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.087 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.087 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:09.087 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449034+0000", 2024-09-21T15:51:09.087 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:09.087 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.087 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.087 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:09.087 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449036+0000", 2024-09-21T15:51:09.088 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9989999999999998e-06 2024-09-21T15:51:09.088 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.088 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.088 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:09.088 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449042+0000", 2024-09-21T15:51:09.088 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6010000000000002e-06 2024-09-21T15:51:09.088 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.088 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.088 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:09.088 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449047+0000", 2024-09-21T15:51:09.088 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3600000000000004e-06 2024-09-21T15:51:09.088 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.088 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.088 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:09.088 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513098+0000", 2024-09-21T15:51:09.088 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064050544000000001 2024-09-21T15:51:09.089 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.089 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.089 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:09.089 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513106+0000", 2024-09-21T15:51:09.089 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5650000000000002e-06 2024-09-21T15:51:09.089 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:09.089 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:09.089 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:09.089 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.089 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.089 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155099 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:09.089 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449051+0000", 2024-09-21T15:51:09.089 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 46.603995199000003, 2024-09-21T15:51:09.089 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064069388000000005, 2024-09-21T15:51:09.089 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:09.089 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:09.090 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:09.090 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:09.090 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:09.090 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155099 2024-09-21T15:51:09.090 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.090 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:09.090 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.090 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:09.090 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449051+0000", 2024-09-21T15:51:09.090 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:09.090 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.090 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.090 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:09.090 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449051+0000", 2024-09-21T15:51:09.090 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:09.091 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.091 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.091 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:09.091 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449051+0000", 2024-09-21T15:51:09.091 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:09.091 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.091 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.091 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:09.091 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449053+0000", 2024-09-21T15:51:09.091 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2620000000000001e-06 2024-09-21T15:51:09.091 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.091 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.091 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:09.091 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449058+0000", 2024-09-21T15:51:09.091 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1680000000000003e-06 2024-09-21T15:51:09.092 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.092 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.092 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:09.092 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449062+0000", 2024-09-21T15:51:09.092 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9979999999999997e-06 2024-09-21T15:51:09.092 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.092 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.092 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:09.092 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513116+0000", 2024-09-21T15:51:09.092 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064053761000000001 2024-09-21T15:51:09.092 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.092 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.092 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:09.092 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513120+0000", 2024-09-21T15:51:09.092 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5399999999999997e-06 2024-09-21T15:51:09.093 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:09.093 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:09.093 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:09.093 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.093 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.093 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155100 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:09.093 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449101+0000", 2024-09-21T15:51:09.093 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 46.603945443999997, 2024-09-21T15:51:09.093 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064079585999999994, 2024-09-21T15:51:09.093 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:09.093 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:09.093 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:09.093 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:09.093 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:09.093 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155100 2024-09-21T15:51:09.093 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.094 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:09.094 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.094 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:09.094 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449101+0000", 2024-09-21T15:51:09.094 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:09.094 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.094 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.094 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:09.094 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449101+0000", 2024-09-21T15:51:09.094 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:09.094 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.094 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.094 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:09.094 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449100+0000", 2024-09-21T15:51:09.095 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-21T15:51:09.095 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.095 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.095 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:09.095 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449105+0000", 2024-09-21T15:51:09.095 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6700000000000002e-06 2024-09-21T15:51:09.095 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.095 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.095 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:09.095 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449117+0000", 2024-09-21T15:51:09.095 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1841000000000001e-05 2024-09-21T15:51:09.095 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.095 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.095 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:09.096 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449124+0000", 2024-09-21T15:51:09.096 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4660000000000002e-06 2024-09-21T15:51:09.096 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.096 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.096 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:09.096 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513163+0000", 2024-09-21T15:51:09.096 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064039053999999998 2024-09-21T15:51:09.096 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.096 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.096 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:09.096 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513180+0000", 2024-09-21T15:51:09.096 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7333000000000002e-05 2024-09-21T15:51:09.096 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:09.096 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:09.096 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:09.097 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.097 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.097 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155101 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:09.097 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449166+0000", 2024-09-21T15:51:09.097 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 46.603880738999997, 2024-09-21T15:51:09.097 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064027639999999997, 2024-09-21T15:51:09.097 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:09.097 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:09.097 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:09.097 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:09.097 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:09.097 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155101 2024-09-21T15:51:09.097 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.097 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:09.098 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.098 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:09.098 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449166+0000", 2024-09-21T15:51:09.098 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:09.098 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.098 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.098 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:09.098 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449166+0000", 2024-09-21T15:51:09.098 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:09.098 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.098 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.098 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:09.098 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449165+0000", 2024-09-21T15:51:09.098 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:09.099 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.099 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.099 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:09.099 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449168+0000", 2024-09-21T15:51:09.099 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5099999999999999e-06 2024-09-21T15:51:09.099 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.099 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.099 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:09.099 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449177+0000", 2024-09-21T15:51:09.099 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6659999999999997e-06 2024-09-21T15:51:09.099 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.099 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.099 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:09.099 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449182+0000", 2024-09-21T15:51:09.100 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1320000000000002e-06 2024-09-21T15:51:09.100 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.100 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.100 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:09.100 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513189+0000", 2024-09-21T15:51:09.100 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064007231999999997 2024-09-21T15:51:09.100 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.100 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.100 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:09.100 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513193+0000", 2024-09-21T15:51:09.100 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.692e-06 2024-09-21T15:51:09.100 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:09.100 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:09.100 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:09.100 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.100 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.101 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155134 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:09.101 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449187+0000", 2024-09-21T15:51:09.101 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 46.603859151000002, 2024-09-21T15:51:09.101 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064031331999999996, 2024-09-21T15:51:09.101 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:09.101 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:09.101 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:09.101 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:09.101 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:09.101 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155134 2024-09-21T15:51:09.101 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.101 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:09.101 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.101 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:09.101 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449187+0000", 2024-09-21T15:51:09.101 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:09.102 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.102 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.102 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:09.102 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449187+0000", 2024-09-21T15:51:09.102 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:09.102 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.102 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.102 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:09.102 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449187+0000", 2024-09-21T15:51:09.102 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:09.102 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.102 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.102 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:09.102 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449189+0000", 2024-09-21T15:51:09.102 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5469999999999999e-06 2024-09-21T15:51:09.103 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.103 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.103 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:09.103 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449196+0000", 2024-09-21T15:51:09.103 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7009999999999999e-06 2024-09-21T15:51:09.103 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.103 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.103 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:09.103 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449202+0000", 2024-09-21T15:51:09.103 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7409999999999998e-06 2024-09-21T15:51:09.103 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.103 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.103 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:09.103 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513214+0000", 2024-09-21T15:51:09.103 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064012301999999993 2024-09-21T15:51:09.104 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.104 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.104 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:09.104 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513218+0000", 2024-09-21T15:51:09.104 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5709999999999999e-06 2024-09-21T15:51:09.104 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:09.104 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:09.104 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:09.104 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.104 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.104 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155135 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:09.104 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449207+0000", 2024-09-21T15:51:09.104 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 46.603839641, 2024-09-21T15:51:09.104 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064029249999999996, 2024-09-21T15:51:09.104 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:09.105 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:09.105 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:09.105 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:09.105 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:09.105 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155135 2024-09-21T15:51:09.105 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.105 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:09.105 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.105 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:09.105 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449207+0000", 2024-09-21T15:51:09.105 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:09.105 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.105 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.105 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:09.105 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449207+0000", 2024-09-21T15:51:09.105 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:09.106 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.106 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.106 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:09.106 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449206+0000", 2024-09-21T15:51:09.106 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:09.106 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.106 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.106 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:09.106 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449209+0000", 2024-09-21T15:51:09.106 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5150000000000001e-06 2024-09-21T15:51:09.106 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.106 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.106 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:09.106 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449215+0000", 2024-09-21T15:51:09.107 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9329999999999998e-06 2024-09-21T15:51:09.107 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.107 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.107 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:09.107 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449220+0000", 2024-09-21T15:51:09.107 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0819999999999998e-06 2024-09-21T15:51:09.107 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.107 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.107 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:09.107 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513232+0000", 2024-09-21T15:51:09.107 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064012052 2024-09-21T15:51:09.107 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.107 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.107 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:09.108 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513236+0000", 2024-09-21T15:51:09.108 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2080000000000002e-06 2024-09-21T15:51:09.108 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:09.108 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:09.108 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:09.108 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.108 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.108 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155136 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:09.108 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449224+0000", 2024-09-21T15:51:09.108 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 46.603822076999997, 2024-09-21T15:51:09.108 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064053025999999999, 2024-09-21T15:51:09.108 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:09.108 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:09.109 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:09.109 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:09.109 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:09.109 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155136 2024-09-21T15:51:09.109 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.109 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:09.109 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.109 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:09.109 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449224+0000", 2024-09-21T15:51:09.109 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:09.109 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.109 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.109 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:09.110 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449224+0000", 2024-09-21T15:51:09.110 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:09.110 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.110 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.110 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:09.110 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449224+0000", 2024-09-21T15:51:09.110 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:09.110 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.110 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.110 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:09.110 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449226+0000", 2024-09-21T15:51:09.110 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2699999999999999e-06 2024-09-21T15:51:09.110 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.110 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.110 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:09.111 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449233+0000", 2024-09-21T15:51:09.111 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4289999999999998e-06 2024-09-21T15:51:09.111 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.111 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.111 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:09.111 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449238+0000", 2024-09-21T15:51:09.111 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0730000000000004e-06 2024-09-21T15:51:09.111 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.111 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.111 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:09.111 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513269+0000", 2024-09-21T15:51:09.111 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064030999000000005 2024-09-21T15:51:09.111 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.111 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.112 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:09.112 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513277+0000", 2024-09-21T15:51:09.112 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5520000000000005e-06 2024-09-21T15:51:09.112 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:09.112 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:09.112 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:09.112 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.112 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.112 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155137 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:09.112 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449263+0000", 2024-09-21T15:51:09.112 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 46.603782944999999, 2024-09-21T15:51:09.112 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.06402803, 2024-09-21T15:51:09.112 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:09.112 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:09.112 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:09.112 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:09.113 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:09.113 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155137 2024-09-21T15:51:09.113 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.113 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:09.113 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.113 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:09.113 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449263+0000", 2024-09-21T15:51:09.113 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:09.113 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.113 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.113 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:09.113 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449263+0000", 2024-09-21T15:51:09.113 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:09.113 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.113 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.113 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:09.114 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449262+0000", 2024-09-21T15:51:09.114 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-21T15:51:09.114 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.114 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.114 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:09.114 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449267+0000", 2024-09-21T15:51:09.114 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.933e-06 2024-09-21T15:51:09.114 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.114 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.114 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:09.114 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449279+0000", 2024-09-21T15:51:09.114 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.181e-05 2024-09-21T15:51:09.114 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.114 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.114 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:09.115 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449286+0000", 2024-09-21T15:51:09.115 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4749999999999998e-06 2024-09-21T15:51:09.115 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.115 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.115 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:09.115 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513287+0000", 2024-09-21T15:51:09.115 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064001369000000002 2024-09-21T15:51:09.115 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.115 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.115 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:09.115 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513291+0000", 2024-09-21T15:51:09.115 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3000000000000003e-06 2024-09-21T15:51:09.115 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:09.115 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:09.115 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:09.116 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.116 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.116 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155138 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:09.116 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449299+0000", 2024-09-21T15:51:09.116 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 46.603747452999997, 2024-09-21T15:51:09.116 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064028359000000007, 2024-09-21T15:51:09.116 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:09.116 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:09.116 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:09.116 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:09.116 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:09.116 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155138 2024-09-21T15:51:09.116 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.116 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:09.116 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.116 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:09.116 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449299+0000", 2024-09-21T15:51:09.117 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:09.117 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.117 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.117 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:09.117 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449299+0000", 2024-09-21T15:51:09.117 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:09.117 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.117 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.117 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:09.117 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449298+0000", 2024-09-21T15:51:09.117 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:09.117 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.117 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.117 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:09.117 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449301+0000", 2024-09-21T15:51:09.117 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0230000000000001e-06 2024-09-21T15:51:09.118 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.118 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.118 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:09.118 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449308+0000", 2024-09-21T15:51:09.118 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3119999999999999e-06 2024-09-21T15:51:09.118 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.118 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.118 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:09.118 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449313+0000", 2024-09-21T15:51:09.118 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8980000000000004e-06 2024-09-21T15:51:09.118 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.118 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.118 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:09.118 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513318+0000", 2024-09-21T15:51:09.118 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064005612000000003 2024-09-21T15:51:09.119 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.119 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.119 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:09.119 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513327+0000", 2024-09-21T15:51:09.119 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0100000000000001e-06 2024-09-21T15:51:09.119 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:09.119 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:09.119 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:09.119 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.119 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.119 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155142 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:09.119 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449392+0000", 2024-09-21T15:51:09.119 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 46.603654695000003, 2024-09-21T15:51:09.119 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064012464000000005, 2024-09-21T15:51:09.119 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:09.120 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:09.120 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:09.120 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:09.120 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:09.120 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155142 2024-09-21T15:51:09.120 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.120 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:09.120 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.120 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:09.120 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449392+0000", 2024-09-21T15:51:09.120 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:09.120 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.120 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.120 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:09.120 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449392+0000", 2024-09-21T15:51:09.121 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:09.121 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.121 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.121 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:09.121 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449391+0000", 2024-09-21T15:51:09.121 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:09.121 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.121 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.121 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:09.121 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449393+0000", 2024-09-21T15:51:09.121 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.26e-06 2024-09-21T15:51:09.121 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.121 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.121 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:09.121 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449399+0000", 2024-09-21T15:51:09.122 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6269999999999997e-06 2024-09-21T15:51:09.122 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.122 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.122 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:09.122 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449403+0000", 2024-09-21T15:51:09.122 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3000000000000003e-06 2024-09-21T15:51:09.122 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.122 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.122 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:09.122 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513396+0000", 2024-09-21T15:51:09.122 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.063992568999999999 2024-09-21T15:51:09.122 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.122 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.122 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:09.123 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513404+0000", 2024-09-21T15:51:09.123 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1100000000000003e-06 2024-09-21T15:51:09.123 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:09.123 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:09.123 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:09.123 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.123 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.123 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155143 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:09.123 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449407+0000", 2024-09-21T15:51:09.123 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 46.603639109, 2024-09-21T15:51:09.123 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064009916, 2024-09-21T15:51:09.123 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:09.123 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:09.123 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:09.123 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:09.124 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:09.124 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155143 2024-09-21T15:51:09.124 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.124 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:09.124 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.124 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:09.124 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449407+0000", 2024-09-21T15:51:09.124 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:09.124 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.124 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.124 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:09.124 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449407+0000", 2024-09-21T15:51:09.124 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:09.124 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.125 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.125 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:09.125 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449407+0000", 2024-09-21T15:51:09.125 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:09.125 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.125 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.125 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:09.125 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449409+0000", 2024-09-21T15:51:09.125 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1090000000000001e-06 2024-09-21T15:51:09.125 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.125 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.125 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:09.125 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449413+0000", 2024-09-21T15:51:09.125 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6650000000000001e-06 2024-09-21T15:51:09.125 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.125 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.125 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:09.126 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449415+0000", 2024-09-21T15:51:09.126 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5459999999999998e-06 2024-09-21T15:51:09.126 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.126 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.126 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:09.126 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513413+0000", 2024-09-21T15:51:09.126 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.063998037999999993 2024-09-21T15:51:09.126 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.126 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.126 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:09.126 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513417+0000", 2024-09-21T15:51:09.126 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8789999999999996e-06 2024-09-21T15:51:09.126 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:09.126 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:09.127 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:09.127 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.127 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.127 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155144 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:09.127 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449427+0000", 2024-09-21T15:51:09.127 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 46.603619101, 2024-09-21T15:51:09.127 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064022531999999993, 2024-09-21T15:51:09.127 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:09.127 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:09.127 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:09.127 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:09.127 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:09.127 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155144 2024-09-21T15:51:09.127 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.128 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:09.128 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.128 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:09.128 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449427+0000", 2024-09-21T15:51:09.128 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:09.128 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.128 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.128 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:09.128 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449427+0000", 2024-09-21T15:51:09.128 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:09.128 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.128 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.128 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:09.128 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449427+0000", 2024-09-21T15:51:09.128 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:09.128 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.129 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.129 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:09.129 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449430+0000", 2024-09-21T15:51:09.129 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3579999999999999e-06 2024-09-21T15:51:09.129 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.129 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.129 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:09.129 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449437+0000", 2024-09-21T15:51:09.129 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4580000000000004e-06 2024-09-21T15:51:09.129 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.129 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.129 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:09.129 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449443+0000", 2024-09-21T15:51:09.129 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5609999999999998e-06 2024-09-21T15:51:09.129 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.129 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.130 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:09.130 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513441+0000", 2024-09-21T15:51:09.130 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.063998136999999997 2024-09-21T15:51:09.130 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:09.130 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:09.130 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:09.130 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513450+0000", 2024-09-21T15:51:09.130 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4950000000000008e-06 2024-09-21T15:51:09.130 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:09.130 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:09.130 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:09.130 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:09.130 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:09.130 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:51:09.131 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:51:09.132 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:51:09.132 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:51:09.132 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:51:09.133 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:51:09.133 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:51:09.133 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:51:09.134 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:51:09.134 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:51:09.130+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:51:09.147 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:51:09.147 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:51:09.165 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:51:09.165 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:51:09.232 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:51:09.231+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:51:09.276 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:09.277 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:09.286 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:51:09.333 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:51:09.332+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:51:09.394 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:09.395 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:09.403 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:51:09.444 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:51:09.443+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:51:09.534 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:51:09.533+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:51:09.548 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:09.548 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:09.558 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T15:51:09.642 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:51:09.641+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:51:09.702 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:09.702 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:09.711 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T15:51:09.736 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:51:09.735+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:51:09.821 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is set 2024-09-21T15:51:09.837 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:51:09.835+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:51:09.863 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:09.863 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:09.872 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T15:51:09.938 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:51:09.937+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:51:09.982 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:09.982 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:09.991 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T15:51:10.039 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:51:10.037+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:51:10.093 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:10.094 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:10.102 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T15:51:10.143 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:51:10.142+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:51:10.240 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:51:10.239+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:51:10.244 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:10.244 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:10.253 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T15:51:10.340 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:51:10.340+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:51:10.399 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:10.399 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:10.408 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:51:10.452 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:51:10.451+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:51:10.520 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:10.520 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:10.530 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:51:10.571 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:51:10.570+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:51:10.643 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:51:10.641+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:51:10.678 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:10.679 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:10.688 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:51:10.744 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:51:10.742+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:51:10.800 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:10.801 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:10.810 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:51:10.852 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:51:10.851+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:51:10.946 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:51:10.944+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:51:10.968 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:10.968 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:10.979 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:51:11.062 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:51:11.061+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:51:11.125 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:11.125 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:11.135 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:51:11.148 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:51:11.147+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:51:11.177 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:51:11.249 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:51:11.248+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:51:11.280 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:11.280 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:11.290 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:51:11.339 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-09-21T15:51:11.340 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:51:11.349 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:51:11.348+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:51:11.356 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:51:11.356 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:51:11.372 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:51:11.372 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:51:11.388 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:51:11.388 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:51:11.396 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:11.396 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:51:11.396 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:51:11.396 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:11.405 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:51:11.405 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:51:11.406 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:51:11.421 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:51:11.421 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:51:11.438 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:51:11.438 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:51:11.450 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:51:11.449+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:51:11.454 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:51:11.454 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:51:11.510 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:11.510 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:51:11.510 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:51:11.510 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:51:11.510 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:11.519 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:51:11.552 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:51:11.551+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:51:11.623 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:11.623 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:51:11.623 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:51:11.623 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:51:11.623 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.623 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:4957 3.9 e29/22 3:95c5ed95:::rbd_data.11335fa2fe69.0000000000000018:head v 29'4, mlcod=26'2)", 2024-09-21T15:51:11.623 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:14.453951+0000", 2024-09-21T15:51:11.624 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 117.16711585900001, 2024-09-21T15:51:11.624 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0043804559999999996, 2024-09-21T15:51:11.624 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:11.624 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:11.624 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:11.624 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.624 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:11.624 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.453951+0000", 2024-09-21T15:51:11.624 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:11.624 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.624 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.624 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:11.624 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.453951+0000", 2024-09-21T15:51:11.624 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:11.625 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.625 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.625 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:11.625 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.453950+0000", 2024-09-21T15:51:11.625 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:11.625 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.625 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.625 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:11.625 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454495+0000", 2024-09-21T15:51:11.625 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00054437799999999998 2024-09-21T15:51:11.625 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.625 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.625 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:11.625 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454508+0000", 2024-09-21T15:51:11.625 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3475000000000001e-05 2024-09-21T15:51:11.626 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.626 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.626 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:11.626 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454515+0000", 2024-09-21T15:51:11.626 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8990000000000004e-06 2024-09-21T15:51:11.626 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.626 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.626 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:11.626 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454581+0000", 2024-09-21T15:51:11.626 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5987000000000004e-05 2024-09-21T15:51:11.626 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.626 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.626 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:11.626 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454609+0000", 2024-09-21T15:51:11.626 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7466000000000001e-05 2024-09-21T15:51:11.627 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.627 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.627 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:11.627 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.458303+0000", 2024-09-21T15:51:11.627 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003693979 2024-09-21T15:51:11.627 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.627 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.627 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:11.627 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.458331+0000", 2024-09-21T15:51:11.627 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8932999999999999e-05 2024-09-21T15:51:11.627 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:11.627 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:11.627 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:11.627 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.628 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.628 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13420 3.0 e31/22 3:038b8840:::rbd_data.1133dc5adfdf.0000000000000035:head v 31'49, mlcod=31'47)", 2024-09-21T15:51:11.628 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.209233+0000", 2024-09-21T15:51:11.628 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 114.411834243, 2024-09-21T15:51:11.628 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0042624960000000002, 2024-09-21T15:51:11.628 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:11.628 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:11.628 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:11.628 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.628 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:11.628 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209233+0000", 2024-09-21T15:51:11.628 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:11.628 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.629 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.629 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:11.629 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209233+0000", 2024-09-21T15:51:11.629 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:11.629 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.629 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.629 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:11.629 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209232+0000", 2024-09-21T15:51:11.629 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:11.629 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.629 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.629 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:11.629 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209251+0000", 2024-09-21T15:51:11.629 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9304e-05 2024-09-21T15:51:11.629 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.629 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.630 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:11.630 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209265+0000", 2024-09-21T15:51:11.630 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3532e-05 2024-09-21T15:51:11.630 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.630 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.630 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:11.630 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209272+0000", 2024-09-21T15:51:11.630 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.289e-06 2024-09-21T15:51:11.630 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.630 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.630 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:11.630 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209321+0000", 2024-09-21T15:51:11.630 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8773999999999998e-05 2024-09-21T15:51:11.630 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.631 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.631 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:11.631 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209341+0000", 2024-09-21T15:51:11.631 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0387999999999999e-05 2024-09-21T15:51:11.631 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.631 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.631 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:11.631 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.213464+0000", 2024-09-21T15:51:11.631 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0041232450000000002 2024-09-21T15:51:11.631 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.631 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.631 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:11.631 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.213495+0000", 2024-09-21T15:51:11.631 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0793999999999997e-05 2024-09-21T15:51:11.632 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:11.632 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:11.632 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:11.632 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.632 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.632 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13445 3.0 e31/22 3:038b8840:::rbd_data.1133dc5adfdf.0000000000000035:head v 31'50, mlcod=31'48)", 2024-09-21T15:51:11.632 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.232497+0000", 2024-09-21T15:51:11.632 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 114.388570231, 2024-09-21T15:51:11.632 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0040493559999999996, 2024-09-21T15:51:11.632 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:11.632 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:11.632 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:11.632 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.632 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:11.632 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232497+0000", 2024-09-21T15:51:11.632 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:11.633 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.633 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.633 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:11.633 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232497+0000", 2024-09-21T15:51:11.633 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:11.633 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.633 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.633 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:11.633 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232496+0000", 2024-09-21T15:51:11.633 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:11.633 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.633 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.633 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:11.633 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232503+0000", 2024-09-21T15:51:11.634 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7320000000000001e-06 2024-09-21T15:51:11.634 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.634 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.634 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:11.634 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232512+0000", 2024-09-21T15:51:11.634 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9259999999999995e-06 2024-09-21T15:51:11.634 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.634 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.634 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:11.634 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232520+0000", 2024-09-21T15:51:11.634 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3699999999999997e-06 2024-09-21T15:51:11.634 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.634 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.635 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:11.635 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232564+0000", 2024-09-21T15:51:11.635 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4128999999999997e-05 2024-09-21T15:51:11.635 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.635 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.635 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:11.635 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232585+0000", 2024-09-21T15:51:11.635 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0704999999999999e-05 2024-09-21T15:51:11.635 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.635 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.635 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:11.635 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.236507+0000", 2024-09-21T15:51:11.635 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0039218730000000002 2024-09-21T15:51:11.635 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.636 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.636 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:11.636 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.236546+0000", 2024-09-21T15:51:11.636 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9399e-05 2024-09-21T15:51:11.636 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:11.636 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:11.636 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:11.636 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.636 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.636 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13832 3.0 e31/22 3:0792efd2:::rbd_data.1133dc5adfdf.0000000000000090:head v 31'58, mlcod=31'56)", 2024-09-21T15:51:11.636 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.527530+0000", 2024-09-21T15:51:11.636 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 114.09353670500001, 2024-09-21T15:51:11.636 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035256530000000001, 2024-09-21T15:51:11.636 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:11.636 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:11.637 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:11.637 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.637 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:11.637 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527530+0000", 2024-09-21T15:51:11.637 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:11.637 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.637 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.637 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:11.637 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527530+0000", 2024-09-21T15:51:11.637 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:11.637 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.637 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.637 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:11.637 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527529+0000", 2024-09-21T15:51:11.638 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:51:11.638 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.638 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.638 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:11.638 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527550+0000", 2024-09-21T15:51:11.638 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1387999999999999e-05 2024-09-21T15:51:11.638 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.638 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.638 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:11.638 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527568+0000", 2024-09-21T15:51:11.638 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8068e-05 2024-09-21T15:51:11.638 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.638 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.639 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:11.639 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527577+0000", 2024-09-21T15:51:11.639 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9919999999999996e-06 2024-09-21T15:51:11.639 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.639 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.639 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:11.639 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527626+0000", 2024-09-21T15:51:11.639 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8597000000000002e-05 2024-09-21T15:51:11.639 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.639 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.639 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:11.639 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527652+0000", 2024-09-21T15:51:11.639 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5993e-05 2024-09-21T15:51:11.639 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.639 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.640 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:11.640 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.531024+0000", 2024-09-21T15:51:11.640 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033721910000000001 2024-09-21T15:51:11.640 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.640 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.640 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:11.640 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.531056+0000", 2024-09-21T15:51:11.640 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1628000000000003e-05 2024-09-21T15:51:11.640 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:11.640 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:11.640 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:11.640 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.640 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.640 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13839 3.0 e31/22 3:03c4fb32:::rbd_data.1133dc5adfdf.0000000000000099:head v 31'59, mlcod=31'56)", 2024-09-21T15:51:11.640 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.528097+0000", 2024-09-21T15:51:11.640 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 114.092970107, 2024-09-21T15:51:11.641 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0057228840000000001, 2024-09-21T15:51:11.641 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:11.641 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:11.641 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:11.641 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.641 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:11.641 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528097+0000", 2024-09-21T15:51:11.641 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:11.641 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.641 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.641 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:11.641 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528097+0000", 2024-09-21T15:51:11.641 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:11.641 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.642 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.642 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:11.642 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528096+0000", 2024-09-21T15:51:11.642 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:11.642 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.642 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.642 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:11.642 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528105+0000", 2024-09-21T15:51:11.642 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7350000000000002e-06 2024-09-21T15:51:11.642 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.642 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.642 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:11.642 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528117+0000", 2024-09-21T15:51:11.642 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1909000000000001e-05 2024-09-21T15:51:11.643 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.643 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.643 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:11.643 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528124+0000", 2024-09-21T15:51:11.643 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8419999999999999e-06 2024-09-21T15:51:11.643 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.643 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.643 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:11.643 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528168+0000", 2024-09-21T15:51:11.643 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4301999999999997e-05 2024-09-21T15:51:11.643 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.643 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.643 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:11.644 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528191+0000", 2024-09-21T15:51:11.644 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2622000000000001e-05 2024-09-21T15:51:11.644 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.644 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.644 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:11.644 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.533780+0000", 2024-09-21T15:51:11.644 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0055897519999999999 2024-09-21T15:51:11.644 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.644 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.644 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:11.644 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.533820+0000", 2024-09-21T15:51:11.644 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9366999999999998e-05 2024-09-21T15:51:11.644 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:11.644 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:11.644 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:11.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.645 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.645 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13884 3.9 e31/22 3:93003736:::rbd_data.1133dc5adfdf.00000000000000a1:head v 31'54, mlcod=31'52)", 2024-09-21T15:51:11.645 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.562433+0000", 2024-09-21T15:51:11.645 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 114.058633359, 2024-09-21T15:51:11.645 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.005162313, 2024-09-21T15:51:11.645 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:11.645 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:11.645 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:11.645 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.645 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:11.645 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562433+0000", 2024-09-21T15:51:11.645 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:11.645 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.645 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.646 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:11.646 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562433+0000", 2024-09-21T15:51:11.646 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:11.646 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.646 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.646 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:11.646 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562432+0000", 2024-09-21T15:51:11.646 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:11.646 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.646 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.646 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:11.646 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562467+0000", 2024-09-21T15:51:11.646 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4885999999999998e-05 2024-09-21T15:51:11.646 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.646 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.647 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:11.647 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562481+0000", 2024-09-21T15:51:11.647 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3604000000000001e-05 2024-09-21T15:51:11.647 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.647 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.647 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:11.647 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562489+0000", 2024-09-21T15:51:11.647 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1829999999999994e-06 2024-09-21T15:51:11.647 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.647 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.647 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:11.647 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562536+0000", 2024-09-21T15:51:11.647 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6830999999999999e-05 2024-09-21T15:51:11.647 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.648 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.648 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:11.648 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562559+0000", 2024-09-21T15:51:11.648 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2765999999999999e-05 2024-09-21T15:51:11.648 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.648 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.648 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:11.648 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.567566+0000", 2024-09-21T15:51:11.648 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0050073219999999998 2024-09-21T15:51:11.648 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.648 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.648 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:11.648 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.567596+0000", 2024-09-21T15:51:11.648 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9802999999999999e-05 2024-09-21T15:51:11.648 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:11.649 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:11.649 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:11.649 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.649 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.649 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:22340 3.9 e34/22 3:910a5abe:::rbd_data.1133896adcfd.0000000000000023:head v 34'92, mlcod=34'90)", 2024-09-21T15:51:11.649 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:21.628025+0000", 2024-09-21T15:51:11.649 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 109.993041979, 2024-09-21T15:51:11.649 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046196010000000001, 2024-09-21T15:51:11.649 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:11.649 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:11.649 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:11.649 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.649 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:11.649 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628025+0000", 2024-09-21T15:51:11.650 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:11.650 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.650 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.650 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:11.650 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628025+0000", 2024-09-21T15:51:11.650 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:11.650 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.650 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.650 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:11.650 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628024+0000", 2024-09-21T15:51:11.650 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:11.650 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.650 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.650 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:11.651 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628034+0000", 2024-09-21T15:51:11.651 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3470000000000005e-06 2024-09-21T15:51:11.651 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.651 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.651 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:11.651 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628043+0000", 2024-09-21T15:51:11.651 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9450000000000005e-06 2024-09-21T15:51:11.651 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.651 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.651 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:11.651 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628048+0000", 2024-09-21T15:51:11.651 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8080000000000004e-06 2024-09-21T15:51:11.651 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.652 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.652 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:11.652 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628081+0000", 2024-09-21T15:51:11.652 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2738999999999999e-05 2024-09-21T15:51:11.652 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.652 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.652 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:11.652 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628094+0000", 2024-09-21T15:51:11.652 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3273e-05 2024-09-21T15:51:11.652 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.652 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.652 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:11.652 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.632616+0000", 2024-09-21T15:51:11.652 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045217269999999997 2024-09-21T15:51:11.652 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.653 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.653 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:11.653 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.632644+0000", 2024-09-21T15:51:11.653 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8469000000000001e-05 2024-09-21T15:51:11.653 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:11.653 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:11.653 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:11.653 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.653 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.653 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:22417 3.4 e34/22 3:25310a10:::rbd_data.1133896adcfd.0000000000000075:head v 34'48, mlcod=34'46)", 2024-09-21T15:51:11.653 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:21.704297+0000", 2024-09-21T15:51:11.653 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 109.91676988099999, 2024-09-21T15:51:11.653 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0038101799999999998, 2024-09-21T15:51:11.653 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:11.653 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:11.654 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:11.654 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.654 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:11.654 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704297+0000", 2024-09-21T15:51:11.654 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:11.654 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.654 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.654 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:11.654 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704297+0000", 2024-09-21T15:51:11.654 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:11.654 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.654 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.654 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:11.654 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704296+0000", 2024-09-21T15:51:11.655 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:51:11.655 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.655 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.655 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:11.655 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704308+0000", 2024-09-21T15:51:11.655 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2559e-05 2024-09-21T15:51:11.655 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.655 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.655 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:11.655 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704337+0000", 2024-09-21T15:51:11.655 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8960000000000001e-05 2024-09-21T15:51:11.655 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.655 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.655 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:11.656 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704345+0000", 2024-09-21T15:51:11.656 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3059999999999998e-06 2024-09-21T15:51:11.656 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.656 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.656 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:11.656 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704408+0000", 2024-09-21T15:51:11.656 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2156000000000006e-05 2024-09-21T15:51:11.656 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.656 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.656 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:11.656 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704430+0000", 2024-09-21T15:51:11.656 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1931000000000001e-05 2024-09-21T15:51:11.656 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.656 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.657 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:11.657 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.708073+0000", 2024-09-21T15:51:11.657 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003643028 2024-09-21T15:51:11.657 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.657 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.657 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:11.657 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.708107+0000", 2024-09-21T15:51:11.657 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4613999999999998e-05 2024-09-21T15:51:11.657 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:11.657 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:11.657 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:11.657 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.657 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.657 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:25278 3.4 e38/22 3:2251cc1c:::rbd_data.1133ebd6d3ba.0000000000000024:head v 38'51, mlcod=38'49)", 2024-09-21T15:51:11.657 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.317337+0000", 2024-09-21T15:51:11.658 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 105.303730121, 2024-09-21T15:51:11.658 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053904110000000003, 2024-09-21T15:51:11.658 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:11.658 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:11.658 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:11.658 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.658 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:11.658 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317337+0000", 2024-09-21T15:51:11.658 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:11.658 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.658 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.658 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:11.658 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317337+0000", 2024-09-21T15:51:11.658 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:11.658 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.659 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.659 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:11.659 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317335+0000", 2024-09-21T15:51:11.659 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:51:11.659 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.659 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.659 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:11.659 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317347+0000", 2024-09-21T15:51:11.659 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.19e-05 2024-09-21T15:51:11.659 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.659 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.659 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:11.659 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317369+0000", 2024-09-21T15:51:11.659 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2232e-05 2024-09-21T15:51:11.659 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.660 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.660 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:11.660 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317381+0000", 2024-09-21T15:51:11.660 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2231e-05 2024-09-21T15:51:11.660 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.660 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.660 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:11.660 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317425+0000", 2024-09-21T15:51:11.660 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3625000000000003e-05 2024-09-21T15:51:11.660 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.660 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.660 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:11.660 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317449+0000", 2024-09-21T15:51:11.660 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4301999999999999e-05 2024-09-21T15:51:11.660 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.660 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.661 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:11.661 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.322688+0000", 2024-09-21T15:51:11.661 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.005238609 2024-09-21T15:51:11.661 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.661 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.661 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:11.661 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.322727+0000", 2024-09-21T15:51:11.661 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9201e-05 2024-09-21T15:51:11.661 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:11.661 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:11.661 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:11.661 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.661 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.661 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:25286 3.0 e38/22 3:0248fb60:::rbd_data.1133ebd6d3ba.0000000000000066:head v 38'109, mlcod=34'107)", 2024-09-21T15:51:11.661 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.317638+0000", 2024-09-21T15:51:11.661 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 105.30342924199999, 2024-09-21T15:51:11.662 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0054074370000000002, 2024-09-21T15:51:11.662 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:11.662 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:11.662 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:11.662 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.662 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:11.662 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317638+0000", 2024-09-21T15:51:11.662 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:11.662 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.662 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.662 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:11.662 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317638+0000", 2024-09-21T15:51:11.662 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:11.662 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.662 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.662 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:11.663 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317637+0000", 2024-09-21T15:51:11.663 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:11.663 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.663 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.663 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:11.663 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317643+0000", 2024-09-21T15:51:11.663 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5609999999999998e-06 2024-09-21T15:51:11.663 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.663 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.663 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:11.663 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317653+0000", 2024-09-21T15:51:11.663 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0705e-05 2024-09-21T15:51:11.663 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.663 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.663 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:11.663 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317660+0000", 2024-09-21T15:51:11.664 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2099999999999998e-06 2024-09-21T15:51:11.664 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.664 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.664 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:11.664 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317704+0000", 2024-09-21T15:51:11.664 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4659000000000001e-05 2024-09-21T15:51:11.664 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.664 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.664 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:11.664 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317725+0000", 2024-09-21T15:51:11.664 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0537e-05 2024-09-21T15:51:11.664 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.664 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.665 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:11.665 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323010+0000", 2024-09-21T15:51:11.665 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052853420000000002 2024-09-21T15:51:11.665 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.665 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.665 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:11.665 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323045+0000", 2024-09-21T15:51:11.665 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4860000000000002e-05 2024-09-21T15:51:11.665 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:11.665 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:11.665 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:11.665 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.665 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.665 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:25280 3.4 e38/22 3:27fca38c:::rbd_data.1133ebd6d3ba.0000000000000026:head v 38'52, mlcod=38'49)", 2024-09-21T15:51:11.665 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.317684+0000", 2024-09-21T15:51:11.665 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 105.303383346, 2024-09-21T15:51:11.666 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053570809999999996, 2024-09-21T15:51:11.666 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:11.666 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:11.666 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:11.666 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.666 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:11.666 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317684+0000", 2024-09-21T15:51:11.666 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:11.666 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.666 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.666 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:11.666 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317684+0000", 2024-09-21T15:51:11.666 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:11.666 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.667 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.667 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:11.667 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317683+0000", 2024-09-21T15:51:11.667 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:11.667 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.667 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.667 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:11.667 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317688+0000", 2024-09-21T15:51:11.667 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5009999999999998e-06 2024-09-21T15:51:11.667 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.667 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.667 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:11.667 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317694+0000", 2024-09-21T15:51:11.667 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3579999999999999e-06 2024-09-21T15:51:11.667 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.667 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.668 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:11.668 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317698+0000", 2024-09-21T15:51:11.668 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4150000000000001e-06 2024-09-21T15:51:11.668 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.668 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.668 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:11.668 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317738+0000", 2024-09-21T15:51:11.668 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9575e-05 2024-09-21T15:51:11.668 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.668 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.668 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:11.668 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317762+0000", 2024-09-21T15:51:11.668 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4173999999999999e-05 2024-09-21T15:51:11.668 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.668 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.668 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:11.669 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323005+0000", 2024-09-21T15:51:11.669 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00524333 2024-09-21T15:51:11.669 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.669 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.669 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:11.669 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323041+0000", 2024-09-21T15:51:11.669 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5139999999999999e-05 2024-09-21T15:51:11.669 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:11.669 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:11.669 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:11.669 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.669 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.669 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:34837 3.0 e43/22 3:038527c8:::rbd_header.1133fed1a840:head v 43'130, mlcod=43'128)", 2024-09-21T15:51:11.669 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:51:11.669 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 100.637178648, 2024-09-21T15:51:11.669 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045776519999999998, 2024-09-21T15:51:11.670 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:11.670 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:11.670 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:11.670 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.670 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:11.670 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:51:11.670 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:11.670 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.670 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.670 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:11.670 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:51:11.670 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:11.670 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.670 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.670 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:11.670 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:51:11.671 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:11.671 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.671 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.671 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:11.671 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983892+0000", 2024-09-21T15:51:11.671 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8450000000000002e-06 2024-09-21T15:51:11.671 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.671 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.671 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:11.671 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983902+0000", 2024-09-21T15:51:11.671 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0299999999999999e-06 2024-09-21T15:51:11.671 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.671 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.671 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:11.671 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983906+0000", 2024-09-21T15:51:11.671 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2300000000000002e-06 2024-09-21T15:51:11.672 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.672 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.672 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:11.672 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983940+0000", 2024-09-21T15:51:11.672 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4563000000000001e-05 2024-09-21T15:51:11.672 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.672 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.672 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:11.672 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983960+0000", 2024-09-21T15:51:11.672 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9199999999999999e-05 2024-09-21T15:51:11.672 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.672 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.672 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:11.672 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.988430+0000", 2024-09-21T15:51:11.672 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0044709149999999998 2024-09-21T15:51:11.672 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.673 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.673 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:11.673 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.988466+0000", 2024-09-21T15:51:11.673 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5420000000000003e-05 2024-09-21T15:51:11.673 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:11.673 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:11.673 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:11.673 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.673 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.673 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "pg_update_log_missing(3.0 epoch 50/22 rep_tid 498 entries 50'171 (0'0) error 3:5df4a5a0:::rbd_data.11339e87a32a.000000000000008e:head by client.4403.0:80207 0.000000 -2 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 47'167)", 2024-09-21T15:51:11.673 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:51:11.673 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 90.679305775000003, 2024-09-21T15:51:11.673 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0047253119999999997, 2024-09-21T15:51:11.673 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:11.673 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-09-21T15:51:11.673 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:11.673 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.674 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:11.674 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:51:11.674 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:11.674 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.674 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.674 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:11.674 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:51:11.674 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:11.674 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.674 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.674 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:11.674 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:51:11.674 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:11.674 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.674 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.675 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:11.675 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941768+0000", 2024-09-21T15:51:11.675 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7309999999999993e-06 2024-09-21T15:51:11.675 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.675 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.675 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:11.675 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941780+0000", 2024-09-21T15:51:11.675 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1399999999999999e-05 2024-09-21T15:51:11.675 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.675 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.675 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:11.675 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941789+0000", 2024-09-21T15:51:11.675 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5230000000000001e-06 2024-09-21T15:51:11.675 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.675 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.676 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:11.676 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941841+0000", 2024-09-21T15:51:11.676 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1863999999999997e-05 2024-09-21T15:51:11.676 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.676 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.676 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:11.676 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.946486+0000", 2024-09-21T15:51:11.676 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046452539999999997 2024-09-21T15:51:11.676 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:11.676 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:11.676 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:11.676 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.676 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.676 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "pg_update_log_missing(3.0 epoch 53/22 rep_tid 585 entries 53'201 (0'0) error 3:6a1d4890:::rbd_data.1133e079cae4.0000000000000004:head by client.4403.0:101089 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 53'198)", 2024-09-21T15:51:11.677 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:44.832827+0000", 2024-09-21T15:51:11.677 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 86.788239966999996, 2024-09-21T15:51:11.677 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053552929999999997, 2024-09-21T15:51:11.677 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:11.677 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-09-21T15:51:11.677 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:11.677 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.677 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:11.677 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832827+0000", 2024-09-21T15:51:11.677 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:11.677 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.677 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.677 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:11.677 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832827+0000", 2024-09-21T15:51:11.677 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:11.678 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.678 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.678 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:11.678 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832826+0000", 2024-09-21T15:51:11.678 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:11.678 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.678 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.678 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:11.678 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832835+0000", 2024-09-21T15:51:11.678 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4509999999999992e-06 2024-09-21T15:51:11.678 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.678 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.678 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:11.678 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832854+0000", 2024-09-21T15:51:11.678 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9352999999999999e-05 2024-09-21T15:51:11.679 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.679 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.679 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:11.679 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832866+0000", 2024-09-21T15:51:11.679 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1766000000000001e-05 2024-09-21T15:51:11.679 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.679 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.679 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:11.679 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.833091+0000", 2024-09-21T15:51:11.679 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00022490300000000001 2024-09-21T15:51:11.679 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.679 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.679 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:11.679 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.838182+0000", 2024-09-21T15:51:11.680 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0050915420000000001 2024-09-21T15:51:11.680 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:11.680 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:11.680 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:11.680 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.680 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.680 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:102822 3.4 e54/22 3:2729ba03:::rbd_object_map.113336ae1075:head v 54'208, mlcod=54'205)", 2024-09-21T15:51:11.680 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:46.249540+0000", 2024-09-21T15:51:11.680 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 85.371526735000003, 2024-09-21T15:51:11.680 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052654859999999998, 2024-09-21T15:51:11.680 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:11.680 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:11.680 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:11.680 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.680 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:11.681 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249540+0000", 2024-09-21T15:51:11.681 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:11.681 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.681 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.681 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:11.681 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249540+0000", 2024-09-21T15:51:11.681 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:11.681 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.681 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.681 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:11.681 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249539+0000", 2024-09-21T15:51:11.681 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:11.681 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.681 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.681 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:11.682 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249546+0000", 2024-09-21T15:51:11.682 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7290000000000003e-06 2024-09-21T15:51:11.682 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.682 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.682 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:11.682 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249558+0000", 2024-09-21T15:51:11.682 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1844e-05 2024-09-21T15:51:11.682 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.682 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.682 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:11.682 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249565+0000", 2024-09-21T15:51:11.682 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8210000000000004e-06 2024-09-21T15:51:11.682 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.682 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.682 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:11.683 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249602+0000", 2024-09-21T15:51:11.683 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7057000000000001e-05 2024-09-21T15:51:11.683 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.683 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.683 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:11.683 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249622+0000", 2024-09-21T15:51:11.683 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9678999999999999e-05 2024-09-21T15:51:11.683 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.683 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.683 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:11.683 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254783+0000", 2024-09-21T15:51:11.683 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0051611449999999998 2024-09-21T15:51:11.683 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.683 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.683 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:11.684 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254806+0000", 2024-09-21T15:51:11.684 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2853e-05 2024-09-21T15:51:11.684 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:11.684 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:11.684 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:11.684 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.684 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.684 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:102823 3.4 e54/22 3:2729ba03:::rbd_object_map.113336ae1075:head v 54'209, mlcod=54'205)", 2024-09-21T15:51:11.684 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:46.250318+0000", 2024-09-21T15:51:11.684 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 85.370749149999995, 2024-09-21T15:51:11.684 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045013539999999999, 2024-09-21T15:51:11.684 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:11.684 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:11.684 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:11.684 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.685 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:11.685 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250318+0000", 2024-09-21T15:51:11.685 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:11.685 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.685 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.685 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:11.685 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250318+0000", 2024-09-21T15:51:11.685 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:11.685 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.685 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.685 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:11.685 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250317+0000", 2024-09-21T15:51:11.685 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:11.685 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.686 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.686 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:11.686 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250322+0000", 2024-09-21T15:51:11.686 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.074e-06 2024-09-21T15:51:11.686 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.686 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.686 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:11.686 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250330+0000", 2024-09-21T15:51:11.686 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4730000000000003e-06 2024-09-21T15:51:11.686 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.686 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.686 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:11.687 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250335+0000", 2024-09-21T15:51:11.687 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8019999999999997e-06 2024-09-21T15:51:11.687 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.687 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.687 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:11.687 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250375+0000", 2024-09-21T15:51:11.687 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9459999999999998e-05 2024-09-21T15:51:11.687 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.687 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.687 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:11.687 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250388+0000", 2024-09-21T15:51:11.687 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3288999999999999e-05 2024-09-21T15:51:11.687 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.687 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.687 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:11.687 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254807+0000", 2024-09-21T15:51:11.688 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0044192220000000004 2024-09-21T15:51:11.688 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.688 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.688 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:11.688 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254819+0000", 2024-09-21T15:51:11.688 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1603999999999999e-05 2024-09-21T15:51:11.688 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:11.688 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:11.688 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:11.688 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.688 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.688 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:160603 3.9 e95/22 3:9171211b:::rbd_data.11338bae974a.0000000000000006:head v 95'343, mlcod=92'340)", 2024-09-21T15:51:11.688 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.215621+0000", 2024-09-21T15:51:11.688 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 40.405446118999997, 2024-09-21T15:51:11.688 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0041276250000000002, 2024-09-21T15:51:11.689 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:11.689 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:11.689 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:11.689 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.689 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:11.689 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215621+0000", 2024-09-21T15:51:11.689 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:11.689 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.689 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.689 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:11.689 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215621+0000", 2024-09-21T15:51:11.689 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:11.689 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.689 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.689 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:11.690 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215620+0000", 2024-09-21T15:51:11.690 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:11.690 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.690 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.690 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:11.690 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215627+0000", 2024-09-21T15:51:11.690 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8850000000000001e-06 2024-09-21T15:51:11.690 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.690 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.690 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:11.690 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215803+0000", 2024-09-21T15:51:11.690 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000176422 2024-09-21T15:51:11.690 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.690 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.691 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:11.691 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215814+0000", 2024-09-21T15:51:11.691 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0135e-05 2024-09-21T15:51:11.691 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.691 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.691 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:11.691 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215866+0000", 2024-09-21T15:51:11.691 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2327999999999998e-05 2024-09-21T15:51:11.691 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.691 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.691 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:11.691 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215887+0000", 2024-09-21T15:51:11.691 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1449000000000001e-05 2024-09-21T15:51:11.691 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.691 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.692 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:11.692 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.219719+0000", 2024-09-21T15:51:11.692 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0038312630000000001 2024-09-21T15:51:11.692 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.692 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.692 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:11.692 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.219748+0000", 2024-09-21T15:51:11.692 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9737999999999999e-05 2024-09-21T15:51:11.692 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:11.692 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:11.692 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:11.692 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.692 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.692 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:161865 3.9 e105/22 3:90e88bad:::rbd_data.11338db70e6f.0000000000000015:head v 105'347, mlcod=103'345)", 2024-09-21T15:51:11.692 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:40.786848+0000", 2024-09-21T15:51:11.693 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 30.834219217000001, 2024-09-21T15:51:11.693 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0056408860000000003, 2024-09-21T15:51:11.693 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:11.693 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:11.693 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:11.693 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.693 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:11.693 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786848+0000", 2024-09-21T15:51:11.693 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:11.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.693 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.693 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:11.693 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786848+0000", 2024-09-21T15:51:11.693 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:11.693 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.694 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.694 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:11.694 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786847+0000", 2024-09-21T15:51:11.694 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:11.694 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.694 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.694 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:11.694 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786851+0000", 2024-09-21T15:51:11.694 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7419999999999999e-06 2024-09-21T15:51:11.694 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.694 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.694 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:11.694 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786872+0000", 2024-09-21T15:51:11.694 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1186e-05 2024-09-21T15:51:11.695 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.695 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.695 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:11.695 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786878+0000", 2024-09-21T15:51:11.695 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0970000000000002e-06 2024-09-21T15:51:11.695 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.695 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.695 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:11.695 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786918+0000", 2024-09-21T15:51:11.695 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.998e-05 2024-09-21T15:51:11.695 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.695 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.695 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:11.695 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786941+0000", 2024-09-21T15:51:11.696 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.245e-05 2024-09-21T15:51:11.696 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.696 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.696 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:11.696 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.792460+0000", 2024-09-21T15:51:11.696 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0055191210000000001 2024-09-21T15:51:11.696 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.696 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.696 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:11.696 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.792489+0000", 2024-09-21T15:51:11.696 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8552e-05 2024-09-21T15:51:11.696 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:11.696 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:11.696 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:11.696 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.697 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.697 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.1.0:1503 3.4 e127/99 3:26dffaa1:::rbd_data.113336ae1075.000000000000006b:16 v 127'658, mlcod=127'653)", 2024-09-21T15:51:11.697 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:51:02.797289+0000", 2024-09-21T15:51:11.697 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.8237783380000003, 2024-09-21T15:51:11.697 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045583780000000001, 2024-09-21T15:51:11.697 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:11.697 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:11.697 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:11.697 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.697 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:11.697 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.797289+0000", 2024-09-21T15:51:11.697 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:11.697 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.697 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.698 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:11.698 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.797289+0000", 2024-09-21T15:51:11.698 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:11.698 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.698 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.698 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:11.698 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.797288+0000", 2024-09-21T15:51:11.698 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:11.698 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.698 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.698 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:11.698 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.797293+0000", 2024-09-21T15:51:11.698 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4500000000000003e-06 2024-09-21T15:51:11.698 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.698 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.699 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:11.699 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.797303+0000", 2024-09-21T15:51:11.699 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0134e-05 2024-09-21T15:51:11.699 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.699 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.699 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:11.699 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.797309+0000", 2024-09-21T15:51:11.699 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2100000000000001e-06 2024-09-21T15:51:11.699 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.699 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.699 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:11.699 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.797346+0000", 2024-09-21T15:51:11.699 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7583999999999998e-05 2024-09-21T15:51:11.699 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.699 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.700 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:11.700 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.797386+0000", 2024-09-21T15:51:11.700 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9428999999999998e-05 2024-09-21T15:51:11.700 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.700 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.700 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:11.700 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.801816+0000", 2024-09-21T15:51:11.700 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0044305730000000002 2024-09-21T15:51:11.700 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.700 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.700 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:11.700 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.801847+0000", 2024-09-21T15:51:11.700 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0639000000000003e-05 2024-09-21T15:51:11.700 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:11.700 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:11.701 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:11.701 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.701 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.701 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:164505 3.5 3:a3a629db:::rbd_data.113336ae1075.0000000000000003:head [delete] snapc 16=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-21T15:51:11.701 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:51:02.863560+0000", 2024-09-21T15:51:11.701 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 8.7575067830000002, 2024-09-21T15:51:11.701 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0040794459999999996, 2024-09-21T15:51:11.701 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:11.701 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:11.701 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:11.701 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:11.701 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:11.701 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 164505 2024-09-21T15:51:11.701 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.701 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:11.702 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.702 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:11.702 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.863560+0000", 2024-09-21T15:51:11.702 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:11.702 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.702 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.702 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:11.702 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.863560+0000", 2024-09-21T15:51:11.702 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:11.702 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.702 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.702 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:11.702 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.863558+0000", 2024-09-21T15:51:11.702 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:51:11.703 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.703 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.703 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:11.703 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.863570+0000", 2024-09-21T15:51:11.703 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1960999999999999e-05 2024-09-21T15:51:11.703 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.703 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.703 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:11.703 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.863582+0000", 2024-09-21T15:51:11.703 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1368e-05 2024-09-21T15:51:11.703 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.703 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.703 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:11.703 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.863589+0000", 2024-09-21T15:51:11.704 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1670000000000002e-06 2024-09-21T15:51:11.704 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.704 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.704 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:11.704 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.863646+0000", 2024-09-21T15:51:11.704 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7433999999999999e-05 2024-09-21T15:51:11.704 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.704 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.704 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:11.704 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.863752+0000", 2024-09-21T15:51:11.704 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105214 2024-09-21T15:51:11.704 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.704 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.704 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-09-21T15:51:11.704 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.863866+0000", 2024-09-21T15:51:11.705 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011479100000000001 2024-09-21T15:51:11.705 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.705 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.705 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:11.705 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.864919+0000", 2024-09-21T15:51:11.705 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00105302 2024-09-21T15:51:11.705 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.705 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.705 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:11.705 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.867550+0000", 2024-09-21T15:51:11.705 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0026310130000000002 2024-09-21T15:51:11.705 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.705 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.705 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:11.705 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.867600+0000", 2024-09-21T15:51:11.706 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9123999999999999e-05 2024-09-21T15:51:11.706 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:11.706 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:11.706 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:11.706 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.867640+0000", 2024-09-21T15:51:11.706 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9953000000000002e-05 2024-09-21T15:51:11.706 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:11.706 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:11.706 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:11.706 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:11.706 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:11.706 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:11.708 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:51:11.708 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:51:11.707+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:51:11.808 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:51:11.808+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:51:11.838 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T15:51:11.848 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7, 3] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-09-21T15:51:11.848 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T15:51:11.848 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-21T15:51:11.848 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump -f json-pretty 2024-09-21T15:51:11.862 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:11.862 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:51:11.862 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:51:11.862 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:11.873 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:51:11.914 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:51:11.913+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:51:12.010 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:51:12.009+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:51:12.023 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:12.024 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:51:12.024 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:51:12.024 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:51:12.024 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:12.033 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:51:12.111 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:51:12.110+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:51:12.138 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:12.138 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:51:12.138 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:51:12.138 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:51:12.138 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.139 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:14051 3.8 3:13437810:::rbd_data.1133dc5adfdf.0000000000000039:head [write 716800~796] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-21T15:51:12.139 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:51:12.139 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 114.441249814, 2024-09-21T15:51:12.139 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.98064002699999997, 2024-09-21T15:51:12.139 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:12.139 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:12.139 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:12.139 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:12.139 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:12.139 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 14051 2024-09-21T15:51:12.139 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.139 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:12.139 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.139 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:12.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:51:12.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:12.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.140 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:12.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:51:12.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:12.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.140 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:12.140 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:51:12.140 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:12.140 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.140 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.141 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:12.141 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695131+0000", 2024-09-21T15:51:12.141 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4680000000000003e-06 2024-09-21T15:51:12.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.141 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:12.141 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695139+0000", 2024-09-21T15:51:12.141 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9400000000000002e-06 2024-09-21T15:51:12.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.141 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.141 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:12.141 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695143+0000", 2024-09-21T15:51:12.141 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9899999999999999e-06 2024-09-21T15:51:12.141 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.142 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.142 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:12.142 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695190+0000", 2024-09-21T15:51:12.142 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6458000000000001e-05 2024-09-21T15:51:12.142 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.142 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.142 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:51:12.142 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695201+0000", 2024-09-21T15:51:12.142 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0964999999999999e-05 2024-09-21T15:51:12.142 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.142 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.142 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:12.142 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.669490+0000", 2024-09-21T15:51:12.142 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.97428980499999995 2024-09-21T15:51:12.143 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.143 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.143 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-09-21T15:51:12.143 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.670033+0000", 2024-09-21T15:51:12.143 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00054252599999999999 2024-09-21T15:51:12.143 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.143 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.143 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:12.143 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.671095+0000", 2024-09-21T15:51:12.143 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001062183 2024-09-21T15:51:12.143 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.143 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.144 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:12.144 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.671186+0000", 2024-09-21T15:51:12.144 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1191000000000002e-05 2024-09-21T15:51:12.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.144 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.144 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-09-21T15:51:12.144 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.671346+0000", 2024-09-21T15:51:12.144 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015980399999999999 2024-09-21T15:51:12.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.144 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.144 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:12.144 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.672221+0000", 2024-09-21T15:51:12.144 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00087471999999999995 2024-09-21T15:51:12.144 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.145 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:12.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.675619+0000", 2024-09-21T15:51:12.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00339858 2024-09-21T15:51:12.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.145 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:12.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.675670+0000", 2024-09-21T15:51:12.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0664000000000001e-05 2024-09-21T15:51:12.145 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.145 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.145 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:12.145 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.675768+0000", 2024-09-21T15:51:12.145 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7415000000000002e-05 2024-09-21T15:51:12.145 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.145 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.146 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.146 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.146 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.146 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116235 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 761856~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:12.146 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048806+0000", 2024-09-21T15:51:12.146 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 76.087570929999998, 2024-09-21T15:51:12.146 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056818556999999999, 2024-09-21T15:51:12.146 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:12.146 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:12.146 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:12.146 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:12.146 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:12.146 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116235 2024-09-21T15:51:12.146 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.147 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:12.147 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.147 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:12.147 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048806+0000", 2024-09-21T15:51:12.147 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:12.147 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.147 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.147 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:12.147 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048806+0000", 2024-09-21T15:51:12.147 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:12.147 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.147 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.147 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:12.147 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048806+0000", 2024-09-21T15:51:12.147 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:51:12.148 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.148 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.148 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:12.148 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048807+0000", 2024-09-21T15:51:12.148 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.001e-06 2024-09-21T15:51:12.148 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.148 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.148 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:12.148 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048810+0000", 2024-09-21T15:51:12.148 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.711e-06 2024-09-21T15:51:12.148 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.148 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.148 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:12.148 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048812+0000", 2024-09-21T15:51:12.148 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2290000000000002e-06 2024-09-21T15:51:12.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.149 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:12.149 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105495+0000", 2024-09-21T15:51:12.149 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056682679999999999 2024-09-21T15:51:12.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.149 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:12.149 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105565+0000", 2024-09-21T15:51:12.149 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9868000000000003e-05 2024-09-21T15:51:12.149 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.149 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.149 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:12.150 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105625+0000", 2024-09-21T15:51:12.150 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0268e-05 2024-09-21T15:51:12.150 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.150 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.150 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.150 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.150 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.150 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116236 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 768000~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:12.150 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048814+0000", 2024-09-21T15:51:12.150 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 76.087563266999993, 2024-09-21T15:51:12.150 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056959587999999998, 2024-09-21T15:51:12.150 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:12.150 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:12.151 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:12.151 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:12.151 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:12.151 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116236 2024-09-21T15:51:12.151 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.151 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:12.151 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.151 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:12.151 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048814+0000", 2024-09-21T15:51:12.151 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:12.151 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.151 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.151 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:12.151 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048814+0000", 2024-09-21T15:51:12.152 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:12.152 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.152 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.152 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:12.152 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048814+0000", 2024-09-21T15:51:12.152 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:51:12.152 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.152 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.152 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:12.152 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048815+0000", 2024-09-21T15:51:12.152 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7999999999999993e-07 2024-09-21T15:51:12.152 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.152 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.152 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:12.153 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048817+0000", 2024-09-21T15:51:12.153 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6620000000000001e-06 2024-09-21T15:51:12.153 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.153 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.153 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:12.153 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048820+0000", 2024-09-21T15:51:12.153 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2800000000000002e-06 2024-09-21T15:51:12.153 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.153 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.153 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:12.153 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105647+0000", 2024-09-21T15:51:12.153 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056827253000000001 2024-09-21T15:51:12.153 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.153 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.153 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:12.153 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105716+0000", 2024-09-21T15:51:12.154 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8761000000000001e-05 2024-09-21T15:51:12.154 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.154 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.154 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:12.154 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105774+0000", 2024-09-21T15:51:12.154 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7855000000000001e-05 2024-09-21T15:51:12.154 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.154 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.154 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.154 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.154 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.154 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116237 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 774144~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:12.154 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048822+0000", 2024-09-21T15:51:12.154 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 76.087555581999993, 2024-09-21T15:51:12.155 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057114028999999997, 2024-09-21T15:51:12.155 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:12.155 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:12.155 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:12.155 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:12.155 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:12.155 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116237 2024-09-21T15:51:12.155 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.155 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:12.155 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.155 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:12.155 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048822+0000", 2024-09-21T15:51:12.155 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:12.156 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.156 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.156 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:12.156 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048822+0000", 2024-09-21T15:51:12.156 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:12.156 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.156 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.156 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:12.156 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048822+0000", 2024-09-21T15:51:12.156 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:51:12.156 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.156 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.156 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:12.156 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048823+0000", 2024-09-21T15:51:12.156 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9699999999999994e-07 2024-09-21T15:51:12.157 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.157 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.157 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:12.157 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048825+0000", 2024-09-21T15:51:12.157 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.65e-06 2024-09-21T15:51:12.157 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.157 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.157 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:12.157 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048828+0000", 2024-09-21T15:51:12.157 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8890000000000002e-06 2024-09-21T15:51:12.157 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.157 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.157 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:12.157 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105802+0000", 2024-09-21T15:51:12.157 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056974415 2024-09-21T15:51:12.158 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.158 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.158 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:12.158 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105871+0000", 2024-09-21T15:51:12.158 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8727999999999997e-05 2024-09-21T15:51:12.158 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.158 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.158 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:12.158 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105936+0000", 2024-09-21T15:51:12.158 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4554999999999994e-05 2024-09-21T15:51:12.158 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.158 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.158 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.158 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.159 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.159 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116238 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 780288~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:12.159 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048830+0000", 2024-09-21T15:51:12.159 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 76.087547322999995, 2024-09-21T15:51:12.159 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057296574000000003, 2024-09-21T15:51:12.159 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:12.159 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:12.159 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:12.159 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:12.159 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:12.159 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116238 2024-09-21T15:51:12.159 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.159 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:12.159 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.159 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:12.159 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048830+0000", 2024-09-21T15:51:12.160 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:12.160 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.160 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.160 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:12.160 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048830+0000", 2024-09-21T15:51:12.160 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:12.160 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.160 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.160 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:12.160 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048830+0000", 2024-09-21T15:51:12.160 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:51:12.160 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.160 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.160 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:12.160 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048831+0000", 2024-09-21T15:51:12.160 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0160000000000001e-06 2024-09-21T15:51:12.161 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.161 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.161 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:12.161 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048834+0000", 2024-09-21T15:51:12.161 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.813e-06 2024-09-21T15:51:12.161 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.161 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.161 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:12.161 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048836+0000", 2024-09-21T15:51:12.161 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6790000000000001e-06 2024-09-21T15:51:12.161 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.161 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.161 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:12.161 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105961+0000", 2024-09-21T15:51:12.161 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057124705999999997 2024-09-21T15:51:12.161 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.161 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.162 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:12.162 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106042+0000", 2024-09-21T15:51:12.162 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0820000000000002e-05 2024-09-21T15:51:12.162 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.162 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.162 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:12.162 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106127+0000", 2024-09-21T15:51:12.162 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4746000000000005e-05 2024-09-21T15:51:12.162 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.162 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.162 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.162 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.162 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.162 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116324 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 524288~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:12.163 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048838+0000", 2024-09-21T15:51:12.163 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 76.087539014000001, 2024-09-21T15:51:12.163 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057477185, 2024-09-21T15:51:12.163 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:12.163 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:12.163 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:12.163 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:12.163 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:12.163 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116324 2024-09-21T15:51:12.163 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.163 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:12.163 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.163 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:12.164 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048838+0000", 2024-09-21T15:51:12.164 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:12.164 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.164 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.164 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:12.164 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048838+0000", 2024-09-21T15:51:12.164 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:12.164 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.164 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.164 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:12.164 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048838+0000", 2024-09-21T15:51:12.164 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:51:12.164 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.164 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.164 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:12.165 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048842+0000", 2024-09-21T15:51:12.165 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4829999999999999e-06 2024-09-21T15:51:12.165 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.165 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.165 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:12.165 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048846+0000", 2024-09-21T15:51:12.165 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0470000000000004e-06 2024-09-21T15:51:12.165 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.165 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.165 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:12.165 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048849+0000", 2024-09-21T15:51:12.165 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3689999999999998e-06 2024-09-21T15:51:12.166 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.166 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.166 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:12.166 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106157+0000", 2024-09-21T15:51:12.166 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057307942000000001 2024-09-21T15:51:12.166 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.166 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.166 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:12.166 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106237+0000", 2024-09-21T15:51:12.166 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0320000000000003e-05 2024-09-21T15:51:12.166 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.166 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.166 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:12.166 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106315+0000", 2024-09-21T15:51:12.166 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8227999999999997e-05 2024-09-21T15:51:12.167 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.167 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.167 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.167 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.167 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.167 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116325 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 530432~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:12.167 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048851+0000", 2024-09-21T15:51:12.167 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 76.087525842000005, 2024-09-21T15:51:12.167 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057581036000000002, 2024-09-21T15:51:12.167 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:12.167 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:12.167 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:12.167 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:12.167 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:12.167 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116325 2024-09-21T15:51:12.168 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.168 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:12.168 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.168 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:12.168 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048851+0000", 2024-09-21T15:51:12.168 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:12.168 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.168 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.168 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:12.168 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048851+0000", 2024-09-21T15:51:12.168 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:12.168 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.168 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.168 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:12.168 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048851+0000", 2024-09-21T15:51:12.168 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:51:12.169 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.169 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.169 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:12.169 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048852+0000", 2024-09-21T15:51:12.169 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.077e-06 2024-09-21T15:51:12.169 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.169 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.169 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:12.169 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048855+0000", 2024-09-21T15:51:12.169 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9859999999999999e-06 2024-09-21T15:51:12.169 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.169 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.169 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:12.169 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048858+0000", 2024-09-21T15:51:12.169 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3810000000000002e-06 2024-09-21T15:51:12.169 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.170 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.170 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:12.170 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106335+0000", 2024-09-21T15:51:12.170 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057477473000000001 2024-09-21T15:51:12.170 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.170 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.170 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:12.170 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106393+0000", 2024-09-21T15:51:12.170 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8050000000000002e-05 2024-09-21T15:51:12.170 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.170 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.170 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:12.170 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106432+0000", 2024-09-21T15:51:12.170 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9274999999999999e-05 2024-09-21T15:51:12.170 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.170 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.171 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.171 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.171 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.171 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116326 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 536576~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:12.171 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048860+0000", 2024-09-21T15:51:12.171 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 76.087517790999996, 2024-09-21T15:51:12.171 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057774197999999999, 2024-09-21T15:51:12.171 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:12.171 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:12.171 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:12.171 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:12.171 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:12.171 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116326 2024-09-21T15:51:12.171 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.171 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:12.171 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.172 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:12.172 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048860+0000", 2024-09-21T15:51:12.172 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:12.172 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.172 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.172 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:12.172 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048860+0000", 2024-09-21T15:51:12.172 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:12.172 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.172 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.172 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:12.172 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048859+0000", 2024-09-21T15:51:12.172 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:51:12.172 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.173 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.173 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:12.173 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048860+0000", 2024-09-21T15:51:12.173 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.02e-06 2024-09-21T15:51:12.173 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.173 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.173 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:12.173 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048863+0000", 2024-09-21T15:51:12.173 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1039999999999998e-06 2024-09-21T15:51:12.173 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.173 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.173 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:12.173 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048867+0000", 2024-09-21T15:51:12.173 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.563e-06 2024-09-21T15:51:12.173 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.173 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.173 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:12.174 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106465+0000", 2024-09-21T15:51:12.174 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057597789000000003 2024-09-21T15:51:12.174 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.174 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.174 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:12.174 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106552+0000", 2024-09-21T15:51:12.174 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7602000000000006e-05 2024-09-21T15:51:12.174 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.174 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.174 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:12.174 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106634+0000", 2024-09-21T15:51:12.174 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1321000000000003e-05 2024-09-21T15:51:12.174 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.174 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.174 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.175 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.175 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.175 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116327 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 542720~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:12.175 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048870+0000", 2024-09-21T15:51:12.175 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 76.087506919000006, 2024-09-21T15:51:12.175 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057883148000000002, 2024-09-21T15:51:12.175 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:12.175 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:12.175 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:12.175 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:12.175 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:12.175 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116327 2024-09-21T15:51:12.175 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.175 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:12.176 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.176 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:12.176 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048870+0000", 2024-09-21T15:51:12.176 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:12.176 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.176 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.176 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:12.176 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048870+0000", 2024-09-21T15:51:12.176 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:12.176 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.176 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.176 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:12.176 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048870+0000", 2024-09-21T15:51:12.176 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:12.176 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.177 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.177 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:12.177 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048872+0000", 2024-09-21T15:51:12.177 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.824e-06 2024-09-21T15:51:12.177 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.177 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.177 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:12.177 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048879+0000", 2024-09-21T15:51:12.177 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6749999999999996e-06 2024-09-21T15:51:12.177 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.177 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.177 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:12.177 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048883+0000", 2024-09-21T15:51:12.177 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2200000000000003e-06 2024-09-21T15:51:12.177 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.177 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.177 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:12.178 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106663+0000", 2024-09-21T15:51:12.178 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057780049 2024-09-21T15:51:12.178 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.178 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.178 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:12.178 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106711+0000", 2024-09-21T15:51:12.178 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8433999999999997e-05 2024-09-21T15:51:12.178 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.178 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.178 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:12.178 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106754+0000", 2024-09-21T15:51:12.178 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2236e-05 2024-09-21T15:51:12.178 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.178 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.179 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.179 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.179 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.179 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116328 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 548864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:12.179 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048887+0000", 2024-09-21T15:51:12.179 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 76.087490399000004, 2024-09-21T15:51:12.179 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057981984, 2024-09-21T15:51:12.179 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:12.179 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:12.179 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:12.179 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:12.179 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:12.179 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116328 2024-09-21T15:51:12.179 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.180 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:12.180 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.180 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:12.180 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048887+0000", 2024-09-21T15:51:12.180 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:12.180 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.180 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.180 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:12.180 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048887+0000", 2024-09-21T15:51:12.180 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:12.180 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.180 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.180 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:12.180 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048887+0000", 2024-09-21T15:51:12.180 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:12.180 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.180 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.181 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:12.181 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048891+0000", 2024-09-21T15:51:12.181 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.843e-06 2024-09-21T15:51:12.181 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.181 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.181 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:12.181 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048897+0000", 2024-09-21T15:51:12.181 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5419999999999996e-06 2024-09-21T15:51:12.181 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.181 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.181 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:12.181 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048901+0000", 2024-09-21T15:51:12.181 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4689999999999999e-06 2024-09-21T15:51:12.181 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.181 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.181 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:12.182 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106780+0000", 2024-09-21T15:51:12.182 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057878259000000001 2024-09-21T15:51:12.182 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.182 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.182 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:12.182 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106827+0000", 2024-09-21T15:51:12.182 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7098000000000002e-05 2024-09-21T15:51:12.182 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.182 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.182 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:12.182 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106869+0000", 2024-09-21T15:51:12.182 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2129999999999998e-05 2024-09-21T15:51:12.182 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.182 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.182 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.183 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.183 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.183 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116329 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 555008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:12.183 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048905+0000", 2024-09-21T15:51:12.183 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 76.087472289000004, 2024-09-21T15:51:12.183 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058115554999999999, 2024-09-21T15:51:12.183 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:12.183 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:12.183 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:12.183 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:12.183 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:12.183 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116329 2024-09-21T15:51:12.183 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.183 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:12.183 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.183 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:12.184 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048905+0000", 2024-09-21T15:51:12.184 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:12.184 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.184 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.184 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:12.184 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048905+0000", 2024-09-21T15:51:12.184 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:12.184 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.184 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.184 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:12.184 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048905+0000", 2024-09-21T15:51:12.184 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:12.184 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.184 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.184 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:12.184 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048907+0000", 2024-09-21T15:51:12.185 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.086e-06 2024-09-21T15:51:12.185 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.185 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.185 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:12.185 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048912+0000", 2024-09-21T15:51:12.185 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2830000000000001e-06 2024-09-21T15:51:12.185 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.185 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.185 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:12.185 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048916+0000", 2024-09-21T15:51:12.185 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.98e-06 2024-09-21T15:51:12.185 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.185 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.185 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:12.186 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106907+0000", 2024-09-21T15:51:12.186 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057990719000000003 2024-09-21T15:51:12.186 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.186 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.186 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:12.186 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106969+0000", 2024-09-21T15:51:12.186 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1789000000000001e-05 2024-09-21T15:51:12.186 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.186 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.186 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:12.186 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107021+0000", 2024-09-21T15:51:12.186 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2030999999999997e-05 2024-09-21T15:51:12.186 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.186 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.186 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.187 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.187 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.187 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116330 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 561152~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:12.187 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048922+0000", 2024-09-21T15:51:12.187 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 76.087455130999999, 2024-09-21T15:51:12.187 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058242343000000002, 2024-09-21T15:51:12.187 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:12.187 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:12.187 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:12.187 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:12.187 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:12.187 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116330 2024-09-21T15:51:12.187 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.187 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:12.187 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.187 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:12.188 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048922+0000", 2024-09-21T15:51:12.188 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:12.188 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.188 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.188 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:12.188 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048922+0000", 2024-09-21T15:51:12.188 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:12.188 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.188 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.188 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:12.188 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048922+0000", 2024-09-21T15:51:12.188 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:12.188 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.188 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.188 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:12.188 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048924+0000", 2024-09-21T15:51:12.189 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1660000000000001e-06 2024-09-21T15:51:12.189 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.189 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.189 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:12.189 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048930+0000", 2024-09-21T15:51:12.189 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4529999999999999e-06 2024-09-21T15:51:12.189 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.189 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.189 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:12.189 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048934+0000", 2024-09-21T15:51:12.189 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0269999999999997e-06 2024-09-21T15:51:12.189 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.189 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.189 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:12.189 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107047+0000", 2024-09-21T15:51:12.190 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058112913000000002 2024-09-21T15:51:12.190 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.190 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.190 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:12.190 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107095+0000", 2024-09-21T15:51:12.190 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7778000000000002e-05 2024-09-21T15:51:12.190 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.190 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.190 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:12.190 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107165+0000", 2024-09-21T15:51:12.190 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9370999999999997e-05 2024-09-21T15:51:12.190 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.190 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.190 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.191 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.191 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.191 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116331 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 567296~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:12.191 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.049440+0000", 2024-09-21T15:51:12.191 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 76.086937036999998, 2024-09-21T15:51:12.191 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057878485, 2024-09-21T15:51:12.191 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:12.191 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:12.191 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:12.191 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:12.191 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:12.191 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116331 2024-09-21T15:51:12.191 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.191 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:12.192 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.192 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:12.192 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049440+0000", 2024-09-21T15:51:12.192 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:12.192 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.192 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.192 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:12.192 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049440+0000", 2024-09-21T15:51:12.192 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:12.192 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.192 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.192 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:12.192 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048939+0000", 2024-09-21T15:51:12.192 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9994984 2024-09-21T15:51:12.192 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.192 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.193 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:12.193 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049444+0000", 2024-09-21T15:51:12.193 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00050557200000000003 2024-09-21T15:51:12.193 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.193 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.193 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:12.193 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049460+0000", 2024-09-21T15:51:12.193 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6120000000000002e-05 2024-09-21T15:51:12.193 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.193 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.193 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:12.193 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049468+0000", 2024-09-21T15:51:12.193 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4209999999999998e-06 2024-09-21T15:51:12.193 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.193 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.193 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:12.194 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107189+0000", 2024-09-21T15:51:12.194 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057721019999999998 2024-09-21T15:51:12.194 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.194 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.194 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:12.194 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107254+0000", 2024-09-21T15:51:12.194 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5265000000000006e-05 2024-09-21T15:51:12.194 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.194 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.194 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:12.194 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107319+0000", 2024-09-21T15:51:12.194 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4784999999999997e-05 2024-09-21T15:51:12.194 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.194 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.194 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.195 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.195 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.195 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116332 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 573440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:12.195 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.049474+0000", 2024-09-21T15:51:12.195 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 76.086903776, 2024-09-21T15:51:12.195 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058013809, 2024-09-21T15:51:12.195 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:12.195 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:12.195 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:12.195 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:12.195 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:12.195 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116332 2024-09-21T15:51:12.195 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.196 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:12.196 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.196 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:12.196 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049474+0000", 2024-09-21T15:51:12.196 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:12.196 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.196 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.196 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:12.196 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049474+0000", 2024-09-21T15:51:12.196 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:12.196 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.196 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.196 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:12.196 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049473+0000", 2024-09-21T15:51:12.196 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:12.196 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.197 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.197 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:12.197 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049476+0000", 2024-09-21T15:51:12.197 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6249999999999999e-06 2024-09-21T15:51:12.197 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.197 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.197 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:12.197 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049482+0000", 2024-09-21T15:51:12.197 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0959999999999997e-06 2024-09-21T15:51:12.197 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.197 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.197 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:12.197 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049487+0000", 2024-09-21T15:51:12.197 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5949999999999998e-06 2024-09-21T15:51:12.197 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:12.198 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107344+0000", 2024-09-21T15:51:12.198 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057856594999999997 2024-09-21T15:51:12.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:12.198 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107422+0000", 2024-09-21T15:51:12.198 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7582000000000004e-05 2024-09-21T15:51:12.198 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.198 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.198 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:12.198 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107487+0000", 2024-09-21T15:51:12.198 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5723999999999996e-05 2024-09-21T15:51:12.198 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.199 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.199 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.199 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.199 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.199 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116333 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 579584~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:12.199 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.049993+0000", 2024-09-21T15:51:12.199 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 76.086383992999998, 2024-09-21T15:51:12.199 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057663537000000001, 2024-09-21T15:51:12.199 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:12.199 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:12.199 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:12.199 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:12.199 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:12.199 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116333 2024-09-21T15:51:12.200 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.200 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:12.200 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.200 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:12.200 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049993+0000", 2024-09-21T15:51:12.200 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:12.200 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.200 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.200 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:12.200 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049993+0000", 2024-09-21T15:51:12.200 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:12.200 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.200 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.201 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:12.201 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049491+0000", 2024-09-21T15:51:12.201 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9994979 2024-09-21T15:51:12.201 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.201 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.201 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:12.201 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049998+0000", 2024-09-21T15:51:12.201 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00050654200000000002 2024-09-21T15:51:12.201 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.201 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.201 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:12.201 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050010+0000", 2024-09-21T15:51:12.201 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1790999999999999e-05 2024-09-21T15:51:12.201 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.201 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.202 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:12.202 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050016+0000", 2024-09-21T15:51:12.202 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2179999999999996e-06 2024-09-21T15:51:12.202 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.202 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.202 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:12.202 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107513+0000", 2024-09-21T15:51:12.202 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057496927000000003 2024-09-21T15:51:12.202 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.202 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.202 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:12.202 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107590+0000", 2024-09-21T15:51:12.202 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7547999999999997e-05 2024-09-21T15:51:12.202 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.203 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.203 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:12.203 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107657+0000", 2024-09-21T15:51:12.203 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6630000000000004e-05 2024-09-21T15:51:12.203 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.203 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.203 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.203 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.203 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.203 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116334 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 585728~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:12.203 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.050091+0000", 2024-09-21T15:51:12.203 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 76.086286365000007, 2024-09-21T15:51:12.203 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057734355000000001, 2024-09-21T15:51:12.203 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:12.204 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:12.204 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:12.204 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:12.204 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:12.204 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116334 2024-09-21T15:51:12.204 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.204 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:12.204 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.204 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:12.204 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050091+0000", 2024-09-21T15:51:12.204 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:12.204 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.204 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.204 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:12.204 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050091+0000", 2024-09-21T15:51:12.205 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:12.205 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.205 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.205 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:12.205 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050021+0000", 2024-09-21T15:51:12.205 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999304 2024-09-21T15:51:12.205 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.205 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.205 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:12.205 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050095+0000", 2024-09-21T15:51:12.205 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4158000000000004e-05 2024-09-21T15:51:12.205 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.205 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.205 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:12.205 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050103+0000", 2024-09-21T15:51:12.205 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0770000000000002e-06 2024-09-21T15:51:12.206 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.206 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.206 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:12.206 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050108+0000", 2024-09-21T15:51:12.206 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6330000000000001e-06 2024-09-21T15:51:12.206 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.206 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.206 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:12.206 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107681+0000", 2024-09-21T15:51:12.206 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057572996000000001 2024-09-21T15:51:12.206 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.206 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.206 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:12.206 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107759+0000", 2024-09-21T15:51:12.206 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7368999999999997e-05 2024-09-21T15:51:12.207 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.207 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.207 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:12.207 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107825+0000", 2024-09-21T15:51:12.207 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6783000000000001e-05 2024-09-21T15:51:12.207 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.207 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.207 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.207 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.207 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.207 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116335 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 591872~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:12.207 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.050542+0000", 2024-09-21T15:51:12.207 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 76.085835779999996, 2024-09-21T15:51:12.207 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057466455999999999, 2024-09-21T15:51:12.207 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:12.207 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:12.208 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:12.208 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:12.208 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:12.208 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116335 2024-09-21T15:51:12.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.208 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:12.208 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.208 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:12.208 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050542+0000", 2024-09-21T15:51:12.208 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:12.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.208 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.208 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:12.208 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050542+0000", 2024-09-21T15:51:12.208 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:12.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.209 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.209 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:12.209 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050113+0000", 2024-09-21T15:51:12.209 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9995713 2024-09-21T15:51:12.209 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.209 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.209 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:12.209 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050546+0000", 2024-09-21T15:51:12.209 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00043328699999999997 2024-09-21T15:51:12.209 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.209 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.209 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:12.209 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050557+0000", 2024-09-21T15:51:12.209 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1011e-05 2024-09-21T15:51:12.209 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.209 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.209 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:12.210 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050564+0000", 2024-09-21T15:51:12.210 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1130000000000003e-06 2024-09-21T15:51:12.210 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.210 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.210 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:12.210 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107852+0000", 2024-09-21T15:51:12.210 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057287350000000001 2024-09-21T15:51:12.210 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.210 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.210 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:12.210 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107928+0000", 2024-09-21T15:51:12.210 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6668000000000005e-05 2024-09-21T15:51:12.210 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.210 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.210 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:12.211 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108008+0000", 2024-09-21T15:51:12.211 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9783999999999994e-05 2024-09-21T15:51:12.211 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.211 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.211 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.211 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.211 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.211 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116336 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 598016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:12.211 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.050834+0000", 2024-09-21T15:51:12.211 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 76.085543513999994, 2024-09-21T15:51:12.211 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057353572999999998, 2024-09-21T15:51:12.211 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:12.211 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:12.211 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:12.211 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:12.212 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:12.212 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116336 2024-09-21T15:51:12.212 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.212 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:12.212 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.212 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:12.212 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050834+0000", 2024-09-21T15:51:12.212 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:12.212 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.212 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.212 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:12.212 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050834+0000", 2024-09-21T15:51:12.212 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:12.212 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.212 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.212 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:12.213 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050571+0000", 2024-09-21T15:51:12.213 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9997373 2024-09-21T15:51:12.213 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.213 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.213 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:12.213 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050842+0000", 2024-09-21T15:51:12.213 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00027029099999999998 2024-09-21T15:51:12.213 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.213 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.213 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:12.213 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050853+0000", 2024-09-21T15:51:12.213 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1532000000000001e-05 2024-09-21T15:51:12.213 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.213 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.214 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:12.214 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050863+0000", 2024-09-21T15:51:12.214 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4239999999999999e-06 2024-09-21T15:51:12.214 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.214 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.214 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:12.214 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108045+0000", 2024-09-21T15:51:12.214 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057182415 2024-09-21T15:51:12.214 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.214 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.214 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:12.214 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108123+0000", 2024-09-21T15:51:12.214 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8275000000000005e-05 2024-09-21T15:51:12.214 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.215 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.215 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:12.215 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108187+0000", 2024-09-21T15:51:12.215 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4152000000000005e-05 2024-09-21T15:51:12.215 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.215 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.215 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.215 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.215 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.215 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116337 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 604160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:12.215 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.051192+0000", 2024-09-21T15:51:12.215 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 76.085184885999993, 2024-09-21T15:51:12.215 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057175981000000001, 2024-09-21T15:51:12.216 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:12.216 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:12.216 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:12.216 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:12.216 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:12.216 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116337 2024-09-21T15:51:12.216 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.216 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:12.216 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.216 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:12.216 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051192+0000", 2024-09-21T15:51:12.216 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:12.216 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.217 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.217 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:12.217 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051192+0000", 2024-09-21T15:51:12.217 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:12.217 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.217 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.217 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:12.217 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050870+0000", 2024-09-21T15:51:12.217 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9996777 2024-09-21T15:51:12.217 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.217 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.217 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:12.217 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051197+0000", 2024-09-21T15:51:12.217 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00032656299999999998 2024-09-21T15:51:12.218 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.218 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.218 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:12.218 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051209+0000", 2024-09-21T15:51:12.218 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2636000000000001e-05 2024-09-21T15:51:12.218 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.218 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.218 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:12.218 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051217+0000", 2024-09-21T15:51:12.218 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5989999999999996e-06 2024-09-21T15:51:12.218 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.218 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.218 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:12.219 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108224+0000", 2024-09-21T15:51:12.219 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057007622000000001 2024-09-21T15:51:12.219 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.219 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.219 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:12.219 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108299+0000", 2024-09-21T15:51:12.219 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5099999999999996e-05 2024-09-21T15:51:12.219 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.219 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.219 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:12.219 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108368+0000", 2024-09-21T15:51:12.219 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8918000000000007e-05 2024-09-21T15:51:12.219 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.219 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.219 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.219 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.220 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.220 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116338 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 610304~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:12.220 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.051583+0000", 2024-09-21T15:51:12.220 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 76.084794432999999, 2024-09-21T15:51:12.220 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056967716000000002, 2024-09-21T15:51:12.220 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:12.220 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:12.220 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:12.220 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:12.220 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:12.220 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116338 2024-09-21T15:51:12.220 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.220 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:12.220 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.221 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:12.221 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051583+0000", 2024-09-21T15:51:12.221 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:12.221 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.221 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.221 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:12.221 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051583+0000", 2024-09-21T15:51:12.221 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:12.221 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.221 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.221 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:12.221 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051223+0000", 2024-09-21T15:51:12.221 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.99964 2024-09-21T15:51:12.221 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.221 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.222 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:12.222 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051587+0000", 2024-09-21T15:51:12.222 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00036352999999999998 2024-09-21T15:51:12.222 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.222 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.222 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:12.222 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051597+0000", 2024-09-21T15:51:12.222 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.893e-06 2024-09-21T15:51:12.222 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.222 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.222 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:12.222 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051603+0000", 2024-09-21T15:51:12.222 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0519999999999998e-06 2024-09-21T15:51:12.222 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.222 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.223 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:12.223 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108394+0000", 2024-09-21T15:51:12.223 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056791531999999999 2024-09-21T15:51:12.223 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.223 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.223 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:12.223 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108480+0000", 2024-09-21T15:51:12.223 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.577e-05 2024-09-21T15:51:12.223 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.223 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.223 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:12.223 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108551+0000", 2024-09-21T15:51:12.223 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0722000000000005e-05 2024-09-21T15:51:12.223 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.224 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.224 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.224 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.224 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.224 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:12.226 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:51:12.227 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:51:12.227 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:12.227 INFO:teuthology.orchestra.run.smithi089.stdout: "epoch": 136, 2024-09-21T15:51:12.227 INFO:teuthology.orchestra.run.smithi089.stdout: "fsid": "4a9b64ea-3aaf-4a40-8ce8-02163a0f6e12", 2024-09-21T15:51:12.227 INFO:teuthology.orchestra.run.smithi089.stdout: "created": "2024-09-21T15:48:37.867501+0000", 2024-09-21T15:51:12.227 INFO:teuthology.orchestra.run.smithi089.stdout: "modified": "2024-09-21T15:51:11.823335+0000", 2024-09-21T15:51:12.227 INFO:teuthology.orchestra.run.smithi089.stdout: "last_up_change": "2024-09-21T15:50:29.130795+0000", 2024-09-21T15:51:12.227 INFO:teuthology.orchestra.run.smithi089.stdout: "last_in_change": "2024-09-21T15:51:06.806747+0000", 2024-09-21T15:51:12.227 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T15:51:12.227 INFO:teuthology.orchestra.run.smithi089.stdout: "flags_num": 5806080, 2024-09-21T15:51:12.227 INFO:teuthology.orchestra.run.smithi089.stdout: "flags_set": [ 2024-09-21T15:51:12.227 INFO:teuthology.orchestra.run.smithi089.stdout: "nodeep-scrub", 2024-09-21T15:51:12.228 INFO:teuthology.orchestra.run.smithi089.stdout: "noscrub", 2024-09-21T15:51:12.228 INFO:teuthology.orchestra.run.smithi089.stdout: "pglog_hardlimit", 2024-09-21T15:51:12.228 INFO:teuthology.orchestra.run.smithi089.stdout: "purged_snapdirs", 2024-09-21T15:51:12.228 INFO:teuthology.orchestra.run.smithi089.stdout: "recovery_deletes", 2024-09-21T15:51:12.228 INFO:teuthology.orchestra.run.smithi089.stdout: "sortbitwise" 2024-09-21T15:51:12.228 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-21T15:51:12.228 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_version": 5, 2024-09-21T15:51:12.228 INFO:teuthology.orchestra.run.smithi089.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T15:51:12.228 INFO:teuthology.orchestra.run.smithi089.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T15:51:12.228 INFO:teuthology.orchestra.run.smithi089.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T15:51:12.228 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_snapshot": "", 2024-09-21T15:51:12.228 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_max": 3, 2024-09-21T15:51:12.228 INFO:teuthology.orchestra.run.smithi089.stdout: "max_osd": 8, 2024-09-21T15:51:12.228 INFO:teuthology.orchestra.run.smithi089.stdout: "require_min_compat_client": "luminous", 2024-09-21T15:51:12.229 INFO:teuthology.orchestra.run.smithi089.stdout: "min_compat_client": "jewel", 2024-09-21T15:51:12.229 INFO:teuthology.orchestra.run.smithi089.stdout: "require_osd_release": "reef", 2024-09-21T15:51:12.229 INFO:teuthology.orchestra.run.smithi089.stdout: "allow_crimson": false, 2024-09-21T15:51:12.229 INFO:teuthology.orchestra.run.smithi089.stdout: "pools": [ 2024-09-21T15:51:12.229 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.229 INFO:teuthology.orchestra.run.smithi089.stdout: "pool": 1, 2024-09-21T15:51:12.229 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_name": ".mgr", 2024-09-21T15:51:12.229 INFO:teuthology.orchestra.run.smithi089.stdout: "create_time": "2024-09-21T15:48:49.921079+0000", 2024-09-21T15:51:12.229 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 1, 2024-09-21T15:51:12.229 INFO:teuthology.orchestra.run.smithi089.stdout: "flags_names": "hashpspool", 2024-09-21T15:51:12.229 INFO:teuthology.orchestra.run.smithi089.stdout: "type": 1, 2024-09-21T15:51:12.229 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 2, 2024-09-21T15:51:12.229 INFO:teuthology.orchestra.run.smithi089.stdout: "min_size": 1, 2024-09-21T15:51:12.229 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_rule": 0, 2024-09-21T15:51:12.229 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_count": 0, 2024-09-21T15:51:12.229 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_target": 0, 2024-09-21T15:51:12.230 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T15:51:12.230 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T15:51:12.230 INFO:teuthology.orchestra.run.smithi089.stdout: "object_hash": 2, 2024-09-21T15:51:12.230 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_autoscale_mode": "off", 2024-09-21T15:51:12.230 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num": 1, 2024-09-21T15:51:12.230 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num": 1, 2024-09-21T15:51:12.230 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num_target": 1, 2024-09-21T15:51:12.230 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_target": 1, 2024-09-21T15:51:12.230 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_pending": 1, 2024-09-21T15:51:12.230 INFO:teuthology.orchestra.run.smithi089.stdout: "last_pg_merge_meta": { 2024-09-21T15:51:12.230 INFO:teuthology.orchestra.run.smithi089.stdout: "source_pgid": "0.0", 2024-09-21T15:51:12.230 INFO:teuthology.orchestra.run.smithi089.stdout: "ready_epoch": 0, 2024-09-21T15:51:12.230 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_started": 0, 2024-09-21T15:51:12.231 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_clean": 0, 2024-09-21T15:51:12.231 INFO:teuthology.orchestra.run.smithi089.stdout: "source_version": "0'0", 2024-09-21T15:51:12.231 INFO:teuthology.orchestra.run.smithi089.stdout: "target_version": "0'0" 2024-09-21T15:51:12.231 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.231 INFO:teuthology.orchestra.run.smithi089.stdout: "last_change": "16", 2024-09-21T15:51:12.231 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend": "0", 2024-09-21T15:51:12.231 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T15:51:12.231 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T15:51:12.231 INFO:teuthology.orchestra.run.smithi089.stdout: "auid": 0, 2024-09-21T15:51:12.231 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_mode": "selfmanaged", 2024-09-21T15:51:12.231 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_seq": 0, 2024-09-21T15:51:12.231 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_epoch": 0, 2024-09-21T15:51:12.231 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_snaps": [], 2024-09-21T15:51:12.231 INFO:teuthology.orchestra.run.smithi089.stdout: "removed_snaps": "[]", 2024-09-21T15:51:12.232 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_bytes": 0, 2024-09-21T15:51:12.232 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_objects": 0, 2024-09-21T15:51:12.232 INFO:teuthology.orchestra.run.smithi089.stdout: "tiers": [], 2024-09-21T15:51:12.232 INFO:teuthology.orchestra.run.smithi089.stdout: "tier_of": -1, 2024-09-21T15:51:12.232 INFO:teuthology.orchestra.run.smithi089.stdout: "read_tier": -1, 2024-09-21T15:51:12.232 INFO:teuthology.orchestra.run.smithi089.stdout: "write_tier": -1, 2024-09-21T15:51:12.232 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_mode": "none", 2024-09-21T15:51:12.232 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_bytes": 0, 2024-09-21T15:51:12.232 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_objects": 0, 2024-09-21T15:51:12.232 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T15:51:12.232 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T15:51:12.232 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T15:51:12.232 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_flush_age": 0, 2024-09-21T15:51:12.232 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_evict_age": 0, 2024-09-21T15:51:12.232 INFO:teuthology.orchestra.run.smithi089.stdout: "erasure_code_profile": "", 2024-09-21T15:51:12.232 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_params": { 2024-09-21T15:51:12.233 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "none" 2024-09-21T15:51:12.233 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.233 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_period": 0, 2024-09-21T15:51:12.233 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_count": 0, 2024-09-21T15:51:12.233 INFO:teuthology.orchestra.run.smithi089.stdout: "use_gmt_hitset": true, 2024-09-21T15:51:12.233 INFO:teuthology.orchestra.run.smithi089.stdout: "min_read_recency_for_promote": 0, 2024-09-21T15:51:12.233 INFO:teuthology.orchestra.run.smithi089.stdout: "min_write_recency_for_promote": 0, 2024-09-21T15:51:12.233 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T15:51:12.233 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_search_last_n": 0, 2024-09-21T15:51:12.233 INFO:teuthology.orchestra.run.smithi089.stdout: "grade_table": [], 2024-09-21T15:51:12.233 INFO:teuthology.orchestra.run.smithi089.stdout: "stripe_width": 0, 2024-09-21T15:51:12.233 INFO:teuthology.orchestra.run.smithi089.stdout: "expected_num_objects": 0, 2024-09-21T15:51:12.233 INFO:teuthology.orchestra.run.smithi089.stdout: "fast_read": false, 2024-09-21T15:51:12.233 INFO:teuthology.orchestra.run.smithi089.stdout: "options": { 2024-09-21T15:51:12.233 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_max": 32, 2024-09-21T15:51:12.234 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_min": 1 2024-09-21T15:51:12.234 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.234 INFO:teuthology.orchestra.run.smithi089.stdout: "application_metadata": { 2024-09-21T15:51:12.234 INFO:teuthology.orchestra.run.smithi089.stdout: "mgr": {} 2024-09-21T15:51:12.234 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.234 INFO:teuthology.orchestra.run.smithi089.stdout: "read_balance": { 2024-09-21T15:51:12.234 INFO:teuthology.orchestra.run.smithi089.stdout: "score_acting": 8, 2024-09-21T15:51:12.234 INFO:teuthology.orchestra.run.smithi089.stdout: "score_stable": 8, 2024-09-21T15:51:12.234 INFO:teuthology.orchestra.run.smithi089.stdout: "optimal_score": 0.25, 2024-09-21T15:51:12.234 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_acting": 2, 2024-09-21T15:51:12.234 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_stable": 2, 2024-09-21T15:51:12.234 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity_weighted": 0.86229705810546875, 2024-09-21T15:51:12.234 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity": 0.86000001430511475, 2024-09-21T15:51:12.234 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity_weighted": 0.86000001430511475 2024-09-21T15:51:12.234 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.235 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.235 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.235 INFO:teuthology.orchestra.run.smithi089.stdout: "pool": 2, 2024-09-21T15:51:12.235 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_name": "rbd", 2024-09-21T15:51:12.235 INFO:teuthology.orchestra.run.smithi089.stdout: "create_time": "2024-09-21T15:48:55.812982+0000", 2024-09-21T15:51:12.235 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 8193, 2024-09-21T15:51:12.235 INFO:teuthology.orchestra.run.smithi089.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T15:51:12.235 INFO:teuthology.orchestra.run.smithi089.stdout: "type": 1, 2024-09-21T15:51:12.235 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 2, 2024-09-21T15:51:12.235 INFO:teuthology.orchestra.run.smithi089.stdout: "min_size": 1, 2024-09-21T15:51:12.235 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_rule": 0, 2024-09-21T15:51:12.235 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_count": 0, 2024-09-21T15:51:12.235 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_target": 0, 2024-09-21T15:51:12.235 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T15:51:12.235 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T15:51:12.236 INFO:teuthology.orchestra.run.smithi089.stdout: "object_hash": 2, 2024-09-21T15:51:12.236 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_autoscale_mode": "off", 2024-09-21T15:51:12.236 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num": 8, 2024-09-21T15:51:12.236 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num": 8, 2024-09-21T15:51:12.236 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num_target": 8, 2024-09-21T15:51:12.236 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_target": 8, 2024-09-21T15:51:12.236 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_pending": 8, 2024-09-21T15:51:12.236 INFO:teuthology.orchestra.run.smithi089.stdout: "last_pg_merge_meta": { 2024-09-21T15:51:12.236 INFO:teuthology.orchestra.run.smithi089.stdout: "source_pgid": "0.0", 2024-09-21T15:51:12.236 INFO:teuthology.orchestra.run.smithi089.stdout: "ready_epoch": 0, 2024-09-21T15:51:12.236 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_started": 0, 2024-09-21T15:51:12.236 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_clean": 0, 2024-09-21T15:51:12.236 INFO:teuthology.orchestra.run.smithi089.stdout: "source_version": "0'0", 2024-09-21T15:51:12.236 INFO:teuthology.orchestra.run.smithi089.stdout: "target_version": "0'0" 2024-09-21T15:51:12.236 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.237 INFO:teuthology.orchestra.run.smithi089.stdout: "last_change": "20", 2024-09-21T15:51:12.237 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend": "0", 2024-09-21T15:51:12.237 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T15:51:12.237 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T15:51:12.237 INFO:teuthology.orchestra.run.smithi089.stdout: "auid": 0, 2024-09-21T15:51:12.237 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_mode": "selfmanaged", 2024-09-21T15:51:12.237 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_seq": 3, 2024-09-21T15:51:12.237 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_epoch": 20, 2024-09-21T15:51:12.237 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_snaps": [], 2024-09-21T15:51:12.237 INFO:teuthology.orchestra.run.smithi089.stdout: "removed_snaps": "[]", 2024-09-21T15:51:12.237 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_bytes": 0, 2024-09-21T15:51:12.237 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_objects": 0, 2024-09-21T15:51:12.237 INFO:teuthology.orchestra.run.smithi089.stdout: "tiers": [], 2024-09-21T15:51:12.237 INFO:teuthology.orchestra.run.smithi089.stdout: "tier_of": -1, 2024-09-21T15:51:12.237 INFO:teuthology.orchestra.run.smithi089.stdout: "read_tier": -1, 2024-09-21T15:51:12.237 INFO:teuthology.orchestra.run.smithi089.stdout: "write_tier": -1, 2024-09-21T15:51:12.237 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_mode": "none", 2024-09-21T15:51:12.238 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_bytes": 0, 2024-09-21T15:51:12.238 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_objects": 0, 2024-09-21T15:51:12.238 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T15:51:12.238 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T15:51:12.238 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T15:51:12.238 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_flush_age": 0, 2024-09-21T15:51:12.238 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_evict_age": 0, 2024-09-21T15:51:12.238 INFO:teuthology.orchestra.run.smithi089.stdout: "erasure_code_profile": "", 2024-09-21T15:51:12.238 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_params": { 2024-09-21T15:51:12.238 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "none" 2024-09-21T15:51:12.238 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.238 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_period": 0, 2024-09-21T15:51:12.238 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_count": 0, 2024-09-21T15:51:12.238 INFO:teuthology.orchestra.run.smithi089.stdout: "use_gmt_hitset": true, 2024-09-21T15:51:12.238 INFO:teuthology.orchestra.run.smithi089.stdout: "min_read_recency_for_promote": 0, 2024-09-21T15:51:12.239 INFO:teuthology.orchestra.run.smithi089.stdout: "min_write_recency_for_promote": 0, 2024-09-21T15:51:12.239 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T15:51:12.239 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_search_last_n": 0, 2024-09-21T15:51:12.239 INFO:teuthology.orchestra.run.smithi089.stdout: "grade_table": [], 2024-09-21T15:51:12.239 INFO:teuthology.orchestra.run.smithi089.stdout: "stripe_width": 0, 2024-09-21T15:51:12.239 INFO:teuthology.orchestra.run.smithi089.stdout: "expected_num_objects": 0, 2024-09-21T15:51:12.239 INFO:teuthology.orchestra.run.smithi089.stdout: "fast_read": false, 2024-09-21T15:51:12.239 INFO:teuthology.orchestra.run.smithi089.stdout: "options": {}, 2024-09-21T15:51:12.239 INFO:teuthology.orchestra.run.smithi089.stdout: "application_metadata": { 2024-09-21T15:51:12.239 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd": {} 2024-09-21T15:51:12.239 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.239 INFO:teuthology.orchestra.run.smithi089.stdout: "read_balance": { 2024-09-21T15:51:12.239 INFO:teuthology.orchestra.run.smithi089.stdout: "score_acting": 2.9900000095367432, 2024-09-21T15:51:12.239 INFO:teuthology.orchestra.run.smithi089.stdout: "score_stable": 2.9900000095367432, 2024-09-21T15:51:12.239 INFO:teuthology.orchestra.run.smithi089.stdout: "optimal_score": 0.87999999523162842, 2024-09-21T15:51:12.239 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_acting": 2.630000114440918, 2024-09-21T15:51:12.240 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_stable": 2.630000114440918, 2024-09-21T15:51:12.240 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity_weighted": 0.86229705810546875, 2024-09-21T15:51:12.240 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity": 0.86000001430511475, 2024-09-21T15:51:12.240 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity_weighted": 0.86000001430511475 2024-09-21T15:51:12.240 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.240 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.240 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.240 INFO:teuthology.orchestra.run.smithi089.stdout: "pool": 3, 2024-09-21T15:51:12.240 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_name": "pool_client.0", 2024-09-21T15:51:12.240 INFO:teuthology.orchestra.run.smithi089.stdout: "create_time": "2024-09-21T15:49:05.523888+0000", 2024-09-21T15:51:12.240 INFO:teuthology.orchestra.run.smithi089.stdout: "flags": 8193, 2024-09-21T15:51:12.240 INFO:teuthology.orchestra.run.smithi089.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T15:51:12.240 INFO:teuthology.orchestra.run.smithi089.stdout: "type": 1, 2024-09-21T15:51:12.240 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 2, 2024-09-21T15:51:12.240 INFO:teuthology.orchestra.run.smithi089.stdout: "min_size": 1, 2024-09-21T15:51:12.241 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_rule": 0, 2024-09-21T15:51:12.241 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_count": 0, 2024-09-21T15:51:12.241 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_target": 0, 2024-09-21T15:51:12.241 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T15:51:12.241 INFO:teuthology.orchestra.run.smithi089.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T15:51:12.241 INFO:teuthology.orchestra.run.smithi089.stdout: "object_hash": 2, 2024-09-21T15:51:12.241 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_autoscale_mode": "off", 2024-09-21T15:51:12.241 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num": 32, 2024-09-21T15:51:12.241 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num": 32, 2024-09-21T15:51:12.241 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_placement_num_target": 32, 2024-09-21T15:51:12.241 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_target": 32, 2024-09-21T15:51:12.241 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_num_pending": 32, 2024-09-21T15:51:12.241 INFO:teuthology.orchestra.run.smithi089.stdout: "last_pg_merge_meta": { 2024-09-21T15:51:12.241 INFO:teuthology.orchestra.run.smithi089.stdout: "source_pgid": "0.0", 2024-09-21T15:51:12.242 INFO:teuthology.orchestra.run.smithi089.stdout: "ready_epoch": 0, 2024-09-21T15:51:12.242 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_started": 0, 2024-09-21T15:51:12.242 INFO:teuthology.orchestra.run.smithi089.stdout: "last_epoch_clean": 0, 2024-09-21T15:51:12.242 INFO:teuthology.orchestra.run.smithi089.stdout: "source_version": "0'0", 2024-09-21T15:51:12.242 INFO:teuthology.orchestra.run.smithi089.stdout: "target_version": "0'0" 2024-09-21T15:51:12.242 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.242 INFO:teuthology.orchestra.run.smithi089.stdout: "last_change": "135", 2024-09-21T15:51:12.242 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend": "0", 2024-09-21T15:51:12.242 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T15:51:12.242 INFO:teuthology.orchestra.run.smithi089.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T15:51:12.242 INFO:teuthology.orchestra.run.smithi089.stdout: "auid": 0, 2024-09-21T15:51:12.242 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_mode": "selfmanaged", 2024-09-21T15:51:12.242 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_seq": 79, 2024-09-21T15:51:12.242 INFO:teuthology.orchestra.run.smithi089.stdout: "snap_epoch": 135, 2024-09-21T15:51:12.243 INFO:teuthology.orchestra.run.smithi089.stdout: "pool_snaps": [], 2024-09-21T15:51:12.243 INFO:teuthology.orchestra.run.smithi089.stdout: "removed_snaps": "[]", 2024-09-21T15:51:12.243 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_bytes": 0, 2024-09-21T15:51:12.243 INFO:teuthology.orchestra.run.smithi089.stdout: "quota_max_objects": 0, 2024-09-21T15:51:12.243 INFO:teuthology.orchestra.run.smithi089.stdout: "tiers": [], 2024-09-21T15:51:12.243 INFO:teuthology.orchestra.run.smithi089.stdout: "tier_of": -1, 2024-09-21T15:51:12.243 INFO:teuthology.orchestra.run.smithi089.stdout: "read_tier": -1, 2024-09-21T15:51:12.243 INFO:teuthology.orchestra.run.smithi089.stdout: "write_tier": -1, 2024-09-21T15:51:12.243 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_mode": "none", 2024-09-21T15:51:12.243 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_bytes": 0, 2024-09-21T15:51:12.243 INFO:teuthology.orchestra.run.smithi089.stdout: "target_max_objects": 0, 2024-09-21T15:51:12.243 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T15:51:12.243 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T15:51:12.243 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T15:51:12.243 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_flush_age": 0, 2024-09-21T15:51:12.243 INFO:teuthology.orchestra.run.smithi089.stdout: "cache_min_evict_age": 0, 2024-09-21T15:51:12.244 INFO:teuthology.orchestra.run.smithi089.stdout: "erasure_code_profile": "", 2024-09-21T15:51:12.244 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_params": { 2024-09-21T15:51:12.244 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "none" 2024-09-21T15:51:12.244 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.244 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_period": 0, 2024-09-21T15:51:12.244 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_count": 0, 2024-09-21T15:51:12.244 INFO:teuthology.orchestra.run.smithi089.stdout: "use_gmt_hitset": true, 2024-09-21T15:51:12.244 INFO:teuthology.orchestra.run.smithi089.stdout: "min_read_recency_for_promote": 0, 2024-09-21T15:51:12.244 INFO:teuthology.orchestra.run.smithi089.stdout: "min_write_recency_for_promote": 0, 2024-09-21T15:51:12.244 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T15:51:12.244 INFO:teuthology.orchestra.run.smithi089.stdout: "hit_set_search_last_n": 0, 2024-09-21T15:51:12.244 INFO:teuthology.orchestra.run.smithi089.stdout: "grade_table": [], 2024-09-21T15:51:12.244 INFO:teuthology.orchestra.run.smithi089.stdout: "stripe_width": 0, 2024-09-21T15:51:12.245 INFO:teuthology.orchestra.run.smithi089.stdout: "expected_num_objects": 0, 2024-09-21T15:51:12.245 INFO:teuthology.orchestra.run.smithi089.stdout: "fast_read": false, 2024-09-21T15:51:12.245 INFO:teuthology.orchestra.run.smithi089.stdout: "options": {}, 2024-09-21T15:51:12.245 INFO:teuthology.orchestra.run.smithi089.stdout: "application_metadata": { 2024-09-21T15:51:12.245 INFO:teuthology.orchestra.run.smithi089.stdout: "rbd": {} 2024-09-21T15:51:12.245 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.245 INFO:teuthology.orchestra.run.smithi089.stdout: "read_balance": { 2024-09-21T15:51:12.245 INFO:teuthology.orchestra.run.smithi089.stdout: "score_acting": 2, 2024-09-21T15:51:12.245 INFO:teuthology.orchestra.run.smithi089.stdout: "score_stable": 2, 2024-09-21T15:51:12.245 INFO:teuthology.orchestra.run.smithi089.stdout: "optimal_score": 1, 2024-09-21T15:51:12.245 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_acting": 2, 2024-09-21T15:51:12.245 INFO:teuthology.orchestra.run.smithi089.stdout: "raw_score_stable": 2, 2024-09-21T15:51:12.245 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity_weighted": 0.86229705810546875, 2024-09-21T15:51:12.245 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity": 0.86000001430511475, 2024-09-21T15:51:12.245 INFO:teuthology.orchestra.run.smithi089.stdout: "average_primary_affinity_weighted": 0.86000001430511475 2024-09-21T15:51:12.245 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.246 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.246 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-21T15:51:12.246 INFO:teuthology.orchestra.run.smithi089.stdout: "osds": [ 2024-09-21T15:51:12.246 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.246 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 0, 2024-09-21T15:51:12.246 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "7b3fabfa-3fe6-47c7-9d0c-a6036f738771", 2024-09-21T15:51:12.246 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-09-21T15:51:12.246 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-09-21T15:51:12.246 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-09-21T15:51:12.246 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-09-21T15:51:12.246 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 13, 2024-09-21T15:51:12.246 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 67, 2024-09-21T15:51:12.246 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 93, 2024-09-21T15:51:12.246 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 135, 2024-09-21T15:51:12.246 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 69, 2024-09-21T15:51:12.247 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-09-21T15:51:12.247 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-09-21T15:51:12.247 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:51:12.247 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.247 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:51:12.247 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6800", 2024-09-21T15:51:12.247 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 1222314186 2024-09-21T15:51:12.247 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.247 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.247 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:51:12.247 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6801", 2024-09-21T15:51:12.247 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 1222314186 2024-09-21T15:51:12.247 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.247 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.247 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.248 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-09-21T15:51:12.248 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:51:12.248 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.248 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:51:12.248 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6803", 2024-09-21T15:51:12.248 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 1222314186 2024-09-21T15:51:12.248 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.248 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.248 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:51:12.248 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6805", 2024-09-21T15:51:12.248 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 1222314186 2024-09-21T15:51:12.248 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.248 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.248 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.248 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-09-21T15:51:12.248 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:51:12.249 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.249 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:51:12.249 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6811", 2024-09-21T15:51:12.249 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 1222314186 2024-09-21T15:51:12.249 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.249 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.249 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:51:12.249 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6813", 2024-09-21T15:51:12.249 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 1222314186 2024-09-21T15:51:12.249 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.249 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.249 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.249 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-09-21T15:51:12.249 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:51:12.249 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.250 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:51:12.250 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6807", 2024-09-21T15:51:12.250 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 1222314186 2024-09-21T15:51:12.250 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.250 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.250 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:51:12.250 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6809", 2024-09-21T15:51:12.250 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 1222314186 2024-09-21T15:51:12.250 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.250 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.250 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.250 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.89:6801/1222314186", 2024-09-21T15:51:12.250 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.89:6805/1222314186", 2024-09-21T15:51:12.250 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.89:6813/1222314186", 2024-09-21T15:51:12.250 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.89:6809/1222314186", 2024-09-21T15:51:12.251 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-09-21T15:51:12.251 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-09-21T15:51:12.251 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-09-21T15:51:12.251 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.251 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.251 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.251 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 1, 2024-09-21T15:51:12.251 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "767ad1bc-84ef-4102-83f5-aacda0917ddb", 2024-09-21T15:51:12.251 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-09-21T15:51:12.251 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-09-21T15:51:12.251 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-09-21T15:51:12.251 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-09-21T15:51:12.251 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-09-21T15:51:12.251 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-09-21T15:51:12.251 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-09-21T15:51:12.251 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 135, 2024-09-21T15:51:12.252 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-09-21T15:51:12.252 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-09-21T15:51:12.252 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-09-21T15:51:12.252 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:51:12.252 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.252 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:51:12.252 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6816", 2024-09-21T15:51:12.252 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3187095367 2024-09-21T15:51:12.252 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.252 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.252 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:51:12.252 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6817", 2024-09-21T15:51:12.252 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3187095367 2024-09-21T15:51:12.252 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.253 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.253 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.253 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-09-21T15:51:12.253 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:51:12.253 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.253 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:51:12.253 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6818", 2024-09-21T15:51:12.253 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3187095367 2024-09-21T15:51:12.253 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.253 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.253 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:51:12.253 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6819", 2024-09-21T15:51:12.253 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3187095367 2024-09-21T15:51:12.253 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.253 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.253 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.254 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-09-21T15:51:12.254 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:51:12.254 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.254 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:51:12.254 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6822", 2024-09-21T15:51:12.254 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3187095367 2024-09-21T15:51:12.254 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.254 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.254 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:51:12.254 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6824", 2024-09-21T15:51:12.254 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3187095367 2024-09-21T15:51:12.254 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.254 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.254 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.255 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-09-21T15:51:12.255 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:51:12.255 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.255 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:51:12.255 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6820", 2024-09-21T15:51:12.255 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3187095367 2024-09-21T15:51:12.255 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.255 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.255 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:51:12.255 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6821", 2024-09-21T15:51:12.255 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3187095367 2024-09-21T15:51:12.255 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.255 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.255 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.255 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.89:6817/3187095367", 2024-09-21T15:51:12.255 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.89:6819/3187095367", 2024-09-21T15:51:12.256 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.89:6824/3187095367", 2024-09-21T15:51:12.256 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.89:6821/3187095367", 2024-09-21T15:51:12.256 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-09-21T15:51:12.256 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-09-21T15:51:12.256 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-09-21T15:51:12.256 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.256 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.256 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.256 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 2, 2024-09-21T15:51:12.256 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "1dbd0d50-aae9-42a7-90a1-aa92697778ae", 2024-09-21T15:51:12.256 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-09-21T15:51:12.256 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-09-21T15:51:12.256 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-09-21T15:51:12.256 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-09-21T15:51:12.256 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-09-21T15:51:12.256 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-09-21T15:51:12.257 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-09-21T15:51:12.257 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 134, 2024-09-21T15:51:12.257 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-09-21T15:51:12.257 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-09-21T15:51:12.257 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-09-21T15:51:12.257 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:51:12.257 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.257 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:51:12.257 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6802", 2024-09-21T15:51:12.257 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 4098625737 2024-09-21T15:51:12.257 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.257 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.257 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:51:12.257 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6804", 2024-09-21T15:51:12.257 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 4098625737 2024-09-21T15:51:12.257 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.258 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.258 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.258 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-09-21T15:51:12.258 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:51:12.258 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.258 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:51:12.258 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6806", 2024-09-21T15:51:12.258 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 4098625737 2024-09-21T15:51:12.258 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.258 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.258 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:51:12.258 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6808", 2024-09-21T15:51:12.258 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 4098625737 2024-09-21T15:51:12.258 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.258 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.259 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.259 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-09-21T15:51:12.259 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:51:12.259 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.259 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:51:12.259 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6814", 2024-09-21T15:51:12.259 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 4098625737 2024-09-21T15:51:12.259 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.259 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.259 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:51:12.259 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6815", 2024-09-21T15:51:12.259 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 4098625737 2024-09-21T15:51:12.259 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.259 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.260 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.260 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-09-21T15:51:12.260 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:51:12.260 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.260 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:51:12.260 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6810", 2024-09-21T15:51:12.260 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 4098625737 2024-09-21T15:51:12.260 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.260 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.260 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:51:12.260 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6812", 2024-09-21T15:51:12.260 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 4098625737 2024-09-21T15:51:12.260 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.261 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.261 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.261 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.89:6804/4098625737", 2024-09-21T15:51:12.261 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.89:6808/4098625737", 2024-09-21T15:51:12.261 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.89:6815/4098625737", 2024-09-21T15:51:12.261 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.89:6812/4098625737", 2024-09-21T15:51:12.261 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-09-21T15:51:12.261 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-09-21T15:51:12.261 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-09-21T15:51:12.261 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.261 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.261 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.261 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 3, 2024-09-21T15:51:12.261 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "f37cd295-2bac-4df4-a99b-abedc0978dce", 2024-09-21T15:51:12.261 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-09-21T15:51:12.262 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-09-21T15:51:12.262 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-09-21T15:51:12.262 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 0.6057586669921875, 2024-09-21T15:51:12.262 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-09-21T15:51:12.262 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-09-21T15:51:12.262 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-09-21T15:51:12.262 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 135, 2024-09-21T15:51:12.262 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-09-21T15:51:12.262 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-09-21T15:51:12.262 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-09-21T15:51:12.262 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:51:12.262 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.262 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:51:12.263 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6823", 2024-09-21T15:51:12.263 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 1886828410 2024-09-21T15:51:12.263 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.263 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.263 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:51:12.263 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6825", 2024-09-21T15:51:12.263 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 1886828410 2024-09-21T15:51:12.263 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.263 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.263 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.263 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-09-21T15:51:12.263 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:51:12.263 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.263 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:51:12.263 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6826", 2024-09-21T15:51:12.263 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 1886828410 2024-09-21T15:51:12.264 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.264 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.264 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:51:12.264 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6827", 2024-09-21T15:51:12.264 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 1886828410 2024-09-21T15:51:12.264 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.264 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.264 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.264 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-09-21T15:51:12.264 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:51:12.264 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.264 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:51:12.264 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6830", 2024-09-21T15:51:12.265 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 1886828410 2024-09-21T15:51:12.265 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.265 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.265 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:51:12.265 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6831", 2024-09-21T15:51:12.265 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 1886828410 2024-09-21T15:51:12.265 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.265 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.265 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.265 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-09-21T15:51:12.265 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:51:12.265 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.265 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:51:12.265 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6828", 2024-09-21T15:51:12.266 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 1886828410 2024-09-21T15:51:12.266 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.266 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.266 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:51:12.266 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.89:6829", 2024-09-21T15:51:12.266 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 1886828410 2024-09-21T15:51:12.266 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.266 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.266 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.266 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.89:6825/1886828410", 2024-09-21T15:51:12.266 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.89:6827/1886828410", 2024-09-21T15:51:12.266 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.89:6831/1886828410", 2024-09-21T15:51:12.266 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.89:6829/1886828410", 2024-09-21T15:51:12.266 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-09-21T15:51:12.267 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-09-21T15:51:12.267 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-09-21T15:51:12.267 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.267 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.267 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.267 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 4, 2024-09-21T15:51:12.267 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "2eeb351d-d51c-4fc3-884e-debecd995aea", 2024-09-21T15:51:12.267 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-09-21T15:51:12.267 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-09-21T15:51:12.267 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-09-21T15:51:12.267 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-09-21T15:51:12.267 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-09-21T15:51:12.267 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-09-21T15:51:12.267 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-09-21T15:51:12.267 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 96, 2024-09-21T15:51:12.268 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-09-21T15:51:12.268 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-09-21T15:51:12.268 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-09-21T15:51:12.268 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:51:12.268 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.268 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:51:12.268 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6817", 2024-09-21T15:51:12.268 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 22722842 2024-09-21T15:51:12.268 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.268 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.268 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:51:12.268 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6819", 2024-09-21T15:51:12.268 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 22722842 2024-09-21T15:51:12.268 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.268 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.268 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.269 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-09-21T15:51:12.269 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:51:12.269 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.269 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:51:12.269 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6820", 2024-09-21T15:51:12.269 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 22722842 2024-09-21T15:51:12.269 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.269 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.269 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:51:12.269 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6822", 2024-09-21T15:51:12.269 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 22722842 2024-09-21T15:51:12.269 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.269 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.269 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.269 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-09-21T15:51:12.269 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:51:12.269 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.270 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:51:12.270 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6828", 2024-09-21T15:51:12.270 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 22722842 2024-09-21T15:51:12.270 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.270 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.270 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:51:12.270 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6830", 2024-09-21T15:51:12.270 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 22722842 2024-09-21T15:51:12.270 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.270 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.270 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.270 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-09-21T15:51:12.270 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:51:12.270 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.270 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:51:12.271 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6824", 2024-09-21T15:51:12.271 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 22722842 2024-09-21T15:51:12.271 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.271 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.271 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:51:12.271 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6826", 2024-09-21T15:51:12.271 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 22722842 2024-09-21T15:51:12.271 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.271 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.271 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.271 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.100:6819/22722842", 2024-09-21T15:51:12.271 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.100:6822/22722842", 2024-09-21T15:51:12.271 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.100:6830/22722842", 2024-09-21T15:51:12.271 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.100:6826/22722842", 2024-09-21T15:51:12.271 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-09-21T15:51:12.272 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-09-21T15:51:12.272 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-09-21T15:51:12.272 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.272 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.272 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.272 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 5, 2024-09-21T15:51:12.272 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "b5396267-6823-42f1-bd04-bd8f144fa76c", 2024-09-21T15:51:12.272 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-09-21T15:51:12.272 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-09-21T15:51:12.272 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-09-21T15:51:12.272 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 0.2926177978515625, 2024-09-21T15:51:12.272 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-09-21T15:51:12.272 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-09-21T15:51:12.272 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-09-21T15:51:12.272 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 134, 2024-09-21T15:51:12.272 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-09-21T15:51:12.272 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-09-21T15:51:12.273 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-09-21T15:51:12.273 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:51:12.273 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.273 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:51:12.273 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6816", 2024-09-21T15:51:12.273 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 363733545 2024-09-21T15:51:12.273 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.273 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.273 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:51:12.273 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6818", 2024-09-21T15:51:12.273 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 363733545 2024-09-21T15:51:12.273 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.273 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.273 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.273 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-09-21T15:51:12.273 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:51:12.274 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.274 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:51:12.274 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6821", 2024-09-21T15:51:12.274 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 363733545 2024-09-21T15:51:12.274 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.274 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.274 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:51:12.274 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6823", 2024-09-21T15:51:12.274 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 363733545 2024-09-21T15:51:12.274 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.274 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.274 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.274 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-09-21T15:51:12.274 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:51:12.274 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.274 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:51:12.275 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6829", 2024-09-21T15:51:12.275 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 363733545 2024-09-21T15:51:12.275 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.275 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.275 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:51:12.275 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6831", 2024-09-21T15:51:12.275 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 363733545 2024-09-21T15:51:12.275 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.275 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.275 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.275 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-09-21T15:51:12.275 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:51:12.275 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.275 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:51:12.275 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6825", 2024-09-21T15:51:12.276 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 363733545 2024-09-21T15:51:12.276 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.276 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.276 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:51:12.276 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6827", 2024-09-21T15:51:12.276 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 363733545 2024-09-21T15:51:12.276 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.276 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.276 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.276 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.100:6818/363733545", 2024-09-21T15:51:12.276 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.100:6823/363733545", 2024-09-21T15:51:12.276 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.100:6831/363733545", 2024-09-21T15:51:12.276 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.100:6827/363733545", 2024-09-21T15:51:12.276 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-09-21T15:51:12.277 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-09-21T15:51:12.277 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-09-21T15:51:12.277 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.277 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.277 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.277 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 6, 2024-09-21T15:51:12.277 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "955b67e9-73bc-42c1-88e0-3356c105e1ea", 2024-09-21T15:51:12.277 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-09-21T15:51:12.277 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-09-21T15:51:12.277 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-09-21T15:51:12.277 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-09-21T15:51:12.277 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-09-21T15:51:12.277 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-09-21T15:51:12.277 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-09-21T15:51:12.277 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 134, 2024-09-21T15:51:12.277 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-09-21T15:51:12.277 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-09-21T15:51:12.278 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-09-21T15:51:12.278 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:51:12.278 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.278 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:51:12.278 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6808", 2024-09-21T15:51:12.278 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 205694879 2024-09-21T15:51:12.278 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.278 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.278 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:51:12.278 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6809", 2024-09-21T15:51:12.278 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 205694879 2024-09-21T15:51:12.278 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.278 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.278 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.278 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-09-21T15:51:12.279 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:51:12.279 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.279 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:51:12.279 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6810", 2024-09-21T15:51:12.279 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 205694879 2024-09-21T15:51:12.279 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.279 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.279 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:51:12.279 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6811", 2024-09-21T15:51:12.279 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 205694879 2024-09-21T15:51:12.279 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.279 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.279 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.279 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-09-21T15:51:12.279 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:51:12.279 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.280 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:51:12.280 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6814", 2024-09-21T15:51:12.280 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 205694879 2024-09-21T15:51:12.280 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.280 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.280 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:51:12.280 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6815", 2024-09-21T15:51:12.280 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 205694879 2024-09-21T15:51:12.280 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.280 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.280 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.280 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-09-21T15:51:12.280 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:51:12.281 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.281 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:51:12.281 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6812", 2024-09-21T15:51:12.281 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 205694879 2024-09-21T15:51:12.281 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.281 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.281 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:51:12.281 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6813", 2024-09-21T15:51:12.281 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 205694879 2024-09-21T15:51:12.281 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.281 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.281 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.281 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.100:6809/205694879", 2024-09-21T15:51:12.281 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.100:6811/205694879", 2024-09-21T15:51:12.281 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.100:6815/205694879", 2024-09-21T15:51:12.281 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.100:6813/205694879", 2024-09-21T15:51:12.282 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-09-21T15:51:12.282 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-09-21T15:51:12.282 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-09-21T15:51:12.282 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.282 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.282 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.282 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 7, 2024-09-21T15:51:12.282 INFO:teuthology.orchestra.run.smithi089.stdout: "uuid": "414ea7f0-138d-404e-bce6-fe8dedcdc938", 2024-09-21T15:51:12.282 INFO:teuthology.orchestra.run.smithi089.stdout: "up": 1, 2024-09-21T15:51:12.282 INFO:teuthology.orchestra.run.smithi089.stdout: "in": 1, 2024-09-21T15:51:12.282 INFO:teuthology.orchestra.run.smithi089.stdout: "weight": 1, 2024-09-21T15:51:12.282 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_affinity": 1, 2024-09-21T15:51:12.282 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_begin": 0, 2024-09-21T15:51:12.282 INFO:teuthology.orchestra.run.smithi089.stdout: "last_clean_end": 0, 2024-09-21T15:51:12.282 INFO:teuthology.orchestra.run.smithi089.stdout: "up_from": 13, 2024-09-21T15:51:12.282 INFO:teuthology.orchestra.run.smithi089.stdout: "up_thru": 135, 2024-09-21T15:51:12.283 INFO:teuthology.orchestra.run.smithi089.stdout: "down_at": 0, 2024-09-21T15:51:12.283 INFO:teuthology.orchestra.run.smithi089.stdout: "lost_at": 0, 2024-09-21T15:51:12.283 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addrs": { 2024-09-21T15:51:12.283 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:51:12.283 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.283 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:51:12.283 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6800", 2024-09-21T15:51:12.283 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3120011840 2024-09-21T15:51:12.283 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.283 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.283 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:51:12.283 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6801", 2024-09-21T15:51:12.283 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3120011840 2024-09-21T15:51:12.283 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.283 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.283 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.284 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addrs": { 2024-09-21T15:51:12.284 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:51:12.284 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.284 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:51:12.284 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6802", 2024-09-21T15:51:12.284 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3120011840 2024-09-21T15:51:12.284 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.284 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.284 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:51:12.284 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6803", 2024-09-21T15:51:12.284 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3120011840 2024-09-21T15:51:12.284 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.284 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.284 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.284 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addrs": { 2024-09-21T15:51:12.284 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:51:12.285 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.285 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:51:12.285 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6806", 2024-09-21T15:51:12.285 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3120011840 2024-09-21T15:51:12.285 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.285 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.285 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:51:12.285 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6807", 2024-09-21T15:51:12.285 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3120011840 2024-09-21T15:51:12.285 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.285 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.285 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.285 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addrs": { 2024-09-21T15:51:12.285 INFO:teuthology.orchestra.run.smithi089.stdout: "addrvec": [ 2024-09-21T15:51:12.286 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.286 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v2", 2024-09-21T15:51:12.286 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6804", 2024-09-21T15:51:12.286 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3120011840 2024-09-21T15:51:12.286 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.286 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.286 INFO:teuthology.orchestra.run.smithi089.stdout: "type": "v1", 2024-09-21T15:51:12.286 INFO:teuthology.orchestra.run.smithi089.stdout: "addr": "172.21.15.100:6805", 2024-09-21T15:51:12.286 INFO:teuthology.orchestra.run.smithi089.stdout: "nonce": 3120011840 2024-09-21T15:51:12.286 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.286 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.286 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.286 INFO:teuthology.orchestra.run.smithi089.stdout: "public_addr": "172.21.15.100:6801/3120011840", 2024-09-21T15:51:12.287 INFO:teuthology.orchestra.run.smithi089.stdout: "cluster_addr": "172.21.15.100:6803/3120011840", 2024-09-21T15:51:12.287 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_back_addr": "172.21.15.100:6807/3120011840", 2024-09-21T15:51:12.287 INFO:teuthology.orchestra.run.smithi089.stdout: "heartbeat_front_addr": "172.21.15.100:6805/3120011840", 2024-09-21T15:51:12.287 INFO:teuthology.orchestra.run.smithi089.stdout: "state": [ 2024-09-21T15:51:12.287 INFO:teuthology.orchestra.run.smithi089.stdout: "exists", 2024-09-21T15:51:12.287 INFO:teuthology.orchestra.run.smithi089.stdout: "up" 2024-09-21T15:51:12.287 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.287 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.287 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-21T15:51:12.287 INFO:teuthology.orchestra.run.smithi089.stdout: "osd_xinfo": [ 2024-09-21T15:51:12.287 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.287 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 0, 2024-09-21T15:51:12.287 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-09-21T15:51:12.287 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-09-21T15:51:12.288 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-09-21T15:51:12.288 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-09-21T15:51:12.288 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-09-21T15:51:12.288 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-09-21T15:48:45.230746+0000", 2024-09-21T15:51:12.288 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 69 2024-09-21T15:51:12.288 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.288 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.288 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 1, 2024-09-21T15:51:12.288 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-09-21T15:51:12.288 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-09-21T15:51:12.288 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-09-21T15:51:12.288 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-09-21T15:51:12.288 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-09-21T15:51:12.288 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-09-21T15:48:45.303567+0000", 2024-09-21T15:51:12.289 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-09-21T15:51:12.289 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.289 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.289 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 2, 2024-09-21T15:51:12.289 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-09-21T15:51:12.289 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-09-21T15:51:12.289 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-09-21T15:51:12.289 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-09-21T15:51:12.289 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-09-21T15:51:12.289 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-09-21T15:48:45.210941+0000", 2024-09-21T15:51:12.289 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-09-21T15:51:12.289 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.289 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.289 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 3, 2024-09-21T15:51:12.290 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-09-21T15:51:12.290 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-09-21T15:51:12.290 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-09-21T15:51:12.290 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-09-21T15:51:12.290 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-09-21T15:51:12.290 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-09-21T15:48:45.218918+0000", 2024-09-21T15:51:12.290 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-09-21T15:51:12.290 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.290 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.290 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 4, 2024-09-21T15:51:12.290 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-09-21T15:51:12.290 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-09-21T15:51:12.290 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-09-21T15:51:12.290 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-09-21T15:51:12.291 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-09-21T15:51:12.291 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-09-21T15:48:45.301781+0000", 2024-09-21T15:51:12.291 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-09-21T15:51:12.291 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.291 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.291 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 5, 2024-09-21T15:51:12.291 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-09-21T15:51:12.291 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-09-21T15:51:12.291 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-09-21T15:51:12.291 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-09-21T15:51:12.291 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-09-21T15:51:12.291 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-09-21T15:48:45.248444+0000", 2024-09-21T15:51:12.291 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-09-21T15:51:12.292 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.292 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.292 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 6, 2024-09-21T15:51:12.292 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-09-21T15:51:12.292 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-09-21T15:51:12.292 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-09-21T15:51:12.292 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-09-21T15:51:12.292 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-09-21T15:51:12.292 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-09-21T15:48:45.275465+0000", 2024-09-21T15:51:12.292 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-09-21T15:51:12.292 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.292 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.292 INFO:teuthology.orchestra.run.smithi089.stdout: "osd": 7, 2024-09-21T15:51:12.293 INFO:teuthology.orchestra.run.smithi089.stdout: "down_stamp": "0.000000", 2024-09-21T15:51:12.293 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_probability": 0, 2024-09-21T15:51:12.293 INFO:teuthology.orchestra.run.smithi089.stdout: "laggy_interval": 0, 2024-09-21T15:51:12.293 INFO:teuthology.orchestra.run.smithi089.stdout: "features": 4540138322906710015, 2024-09-21T15:51:12.293 INFO:teuthology.orchestra.run.smithi089.stdout: "old_weight": 0, 2024-09-21T15:51:12.293 INFO:teuthology.orchestra.run.smithi089.stdout: "last_purged_snaps_scrub": "2024-09-21T15:48:45.166209+0000", 2024-09-21T15:51:12.293 INFO:teuthology.orchestra.run.smithi089.stdout: "dead_epoch": 0 2024-09-21T15:51:12.293 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.293 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-21T15:51:12.293 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_upmap": [], 2024-09-21T15:51:12.293 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_upmap_items": [], 2024-09-21T15:51:12.293 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_upmap_primaries": [], 2024-09-21T15:51:12.293 INFO:teuthology.orchestra.run.smithi089.stdout: "pg_temp": [], 2024-09-21T15:51:12.293 INFO:teuthology.orchestra.run.smithi089.stdout: "primary_temp": [], 2024-09-21T15:51:12.293 INFO:teuthology.orchestra.run.smithi089.stdout: "blocklist": {}, 2024-09-21T15:51:12.293 INFO:teuthology.orchestra.run.smithi089.stdout: "range_blocklist": {}, 2024-09-21T15:51:12.294 INFO:teuthology.orchestra.run.smithi089.stdout: "erasure_code_profiles": { 2024-09-21T15:51:12.294 INFO:teuthology.orchestra.run.smithi089.stdout: "default": { 2024-09-21T15:51:12.294 INFO:teuthology.orchestra.run.smithi089.stdout: "crush-failure-domain": "osd", 2024-09-21T15:51:12.294 INFO:teuthology.orchestra.run.smithi089.stdout: "k": "2", 2024-09-21T15:51:12.294 INFO:teuthology.orchestra.run.smithi089.stdout: "m": "1", 2024-09-21T15:51:12.294 INFO:teuthology.orchestra.run.smithi089.stdout: "plugin": "jerasure", 2024-09-21T15:51:12.294 INFO:teuthology.orchestra.run.smithi089.stdout: "technique": "reed_sol_van" 2024-09-21T15:51:12.294 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.294 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:12.294 INFO:teuthology.orchestra.run.smithi089.stdout: "removed_snaps_queue": [ 2024-09-21T15:51:12.294 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.294 INFO:teuthology.orchestra.run.smithi089.stdout: "pool": 3, 2024-09-21T15:51:12.294 INFO:teuthology.orchestra.run.smithi089.stdout: "snaps": [ 2024-09-21T15:51:12.294 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:12.294 INFO:teuthology.orchestra.run.smithi089.stdout: "begin": 16, 2024-09-21T15:51:12.294 INFO:teuthology.orchestra.run.smithi089.stdout: "length": 6 2024-09-21T15:51:12.295 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.295 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:12.295 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.295 INFO:teuthology.orchestra.run.smithi089.stdout: ], 2024-09-21T15:51:12.295 INFO:teuthology.orchestra.run.smithi089.stdout: "new_removed_snaps": [], 2024-09-21T15:51:12.295 INFO:teuthology.orchestra.run.smithi089.stdout: "new_purged_snaps": [], 2024-09-21T15:51:12.295 INFO:teuthology.orchestra.run.smithi089.stdout: "crush_node_flags": {}, 2024-09-21T15:51:12.295 INFO:teuthology.orchestra.run.smithi089.stdout: "device_class_flags": {}, 2024-09-21T15:51:12.295 INFO:teuthology.orchestra.run.smithi089.stdout: "stretch_mode": { 2024-09-21T15:51:12.295 INFO:teuthology.orchestra.run.smithi089.stdout: "stretch_mode_enabled": false, 2024-09-21T15:51:12.295 INFO:teuthology.orchestra.run.smithi089.stdout: "stretch_bucket_count": 0, 2024-09-21T15:51:12.295 INFO:teuthology.orchestra.run.smithi089.stdout: "degraded_stretch_mode": 0, 2024-09-21T15:51:12.295 INFO:teuthology.orchestra.run.smithi089.stdout: "recovering_stretch_mode": 0, 2024-09-21T15:51:12.295 INFO:teuthology.orchestra.run.smithi089.stdout: "stretch_mode_bucket": 0 2024-09-21T15:51:12.295 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:12.296 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:12.296 INFO:tasks.thrashosds.thrasher:j is {'epoch': 136, 'fsid': '4a9b64ea-3aaf-4a40-8ce8-02163a0f6e12', 'created': '2024-09-21T15:48:37.867501+0000', 'modified': '2024-09-21T15:51:11.823335+0000', 'last_up_change': '2024-09-21T15:50:29.130795+0000', 'last_in_change': '2024-09-21T15:51:06.806747+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 5, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 3, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'reef', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T15:48:49.921079+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '16', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8622970581054688, 'average_primary_affinity': 0.8600000143051147, 'average_primary_affinity_weighted': 0.8600000143051147}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-21T15:48:55.812982+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 3, 'snap_epoch': 20, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2.990000009536743, 'score_stable': 2.990000009536743, 'optimal_score': 0.8799999952316284, 'raw_score_acting': 2.630000114440918, 'raw_score_stable': 2.630000114440918, 'primary_affinity_weighted': 0.8622970581054688, 'average_primary_affinity': 0.8600000143051147, 'average_primary_affinity_weighted': 0.8600000143051147}}, {'pool': 3, 'pool_name': 'pool_client.0', 'create_time': '2024-09-21T15:49:05.523888+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '135', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 79, 'snap_epoch': 135, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_acting': 2, 'score_stable': 2, 'optimal_score': 1, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8622970581054688, 'average_primary_affinity': 0.8600000143051147, 'average_primary_affinity_weighted': 0.8600000143051147}}], 'osds': [{'osd': 0, 'uuid': '7b3fabfa-3fe6-47c7-9d0c-a6036f738771', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 13, 'last_clean_end': 67, 'up_from': 93, 'up_thru': 135, 'down_at': 69, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6800', 'nonce': 1222314186}, {'type': 'v1', 'addr': '172.21.15.89:6801', 'nonce': 1222314186}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6803', 'nonce': 1222314186}, {'type': 'v1', 'addr': '172.21.15.89:6805', 'nonce': 1222314186}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6811', 'nonce': 1222314186}, {'type': 'v1', 'addr': '172.21.15.89:6813', 'nonce': 1222314186}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6807', 'nonce': 1222314186}, {'type': 'v1', 'addr': '172.21.15.89:6809', 'nonce': 1222314186}]}, 'public_addr': '172.21.15.89:6801/1222314186', 'cluster_addr': '172.21.15.89:6805/1222314186', 'heartbeat_back_addr': '172.21.15.89:6813/1222314186', 'heartbeat_front_addr': '172.21.15.89:6809/1222314186', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '767ad1bc-84ef-4102-83f5-aacda0917ddb', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 135, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6816', 'nonce': 3187095367}, {'type': 'v1', 'addr': '172.21.15.89:6817', 'nonce': 3187095367}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6818', 'nonce': 3187095367}, {'type': 'v1', 'addr': '172.21.15.89:6819', 'nonce': 3187095367}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6822', 'nonce': 3187095367}, {'type': 'v1', 'addr': '172.21.15.89:6824', 'nonce': 3187095367}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6820', 'nonce': 3187095367}, {'type': 'v1', 'addr': '172.21.15.89:6821', 'nonce': 3187095367}]}, 'public_addr': '172.21.15.89:6817/3187095367', 'cluster_addr': '172.21.15.89:6819/3187095367', 'heartbeat_back_addr': '172.21.15.89:6824/3187095367', 'heartbeat_front_addr': '172.21.15.89:6821/3187095367', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '1dbd0d50-aae9-42a7-90a1-aa92697778ae', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 134, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6802', 'nonce': 4098625737}, {'type': 'v1', 'addr': '172.21.15.89:6804', 'nonce': 4098625737}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6806', 'nonce': 4098625737}, {'type': 'v1', 'addr': '172.21.15.89:6808', 'nonce': 4098625737}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6814', 'nonce': 4098625737}, {'type': 'v1', 'addr': '172.21.15.89:6815', 'nonce': 4098625737}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6810', 'nonce': 4098625737}, {'type': 'v1', 'addr': '172.21.15.89:6812', 'nonce': 4098625737}]}, 'public_addr': '172.21.15.89:6804/4098625737', 'cluster_addr': '172.21.15.89:6808/4098625737', 'heartbeat_back_addr': '172.21.15.89:6815/4098625737', 'heartbeat_front_addr': '172.21.15.89:6812/4098625737', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'f37cd295-2bac-4df4-a99b-abedc0978dce', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.6057586669921875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 135, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6823', 'nonce': 1886828410}, {'type': 'v1', 'addr': '172.21.15.89:6825', 'nonce': 1886828410}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6826', 'nonce': 1886828410}, {'type': 'v1', 'addr': '172.21.15.89:6827', 'nonce': 1886828410}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6830', 'nonce': 1886828410}, {'type': 'v1', 'addr': '172.21.15.89:6831', 'nonce': 1886828410}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.89:6828', 'nonce': 1886828410}, {'type': 'v1', 'addr': '172.21.15.89:6829', 'nonce': 1886828410}]}, 'public_addr': '172.21.15.89:6825/1886828410', 'cluster_addr': '172.21.15.89:6827/1886828410', 'heartbeat_back_addr': '172.21.15.89:6831/1886828410', 'heartbeat_front_addr': '172.21.15.89:6829/1886828410', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': '2eeb351d-d51c-4fc3-884e-debecd995aea', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 96, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6817', 'nonce': 22722842}, {'type': 'v1', 'addr': '172.21.15.100:6819', 'nonce': 22722842}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6820', 'nonce': 22722842}, {'type': 'v1', 'addr': '172.21.15.100:6822', 'nonce': 22722842}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6828', 'nonce': 22722842}, {'type': 'v1', 'addr': '172.21.15.100:6830', 'nonce': 22722842}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6824', 'nonce': 22722842}, {'type': 'v1', 'addr': '172.21.15.100:6826', 'nonce': 22722842}]}, 'public_addr': '172.21.15.100:6819/22722842', 'cluster_addr': '172.21.15.100:6822/22722842', 'heartbeat_back_addr': '172.21.15.100:6830/22722842', 'heartbeat_front_addr': '172.21.15.100:6826/22722842', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'b5396267-6823-42f1-bd04-bd8f144fa76c', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.2926177978515625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 134, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6816', 'nonce': 363733545}, {'type': 'v1', 'addr': '172.21.15.100:6818', 'nonce': 363733545}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6821', 'nonce': 363733545}, {'type': 'v1', 'addr': '172.21.15.100:6823', 'nonce': 363733545}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6829', 'nonce': 363733545}, {'type': 'v1', 'addr': '172.21.15.100:6831', 'nonce': 363733545}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6825', 'nonce': 363733545}, {'type': 'v1', 'addr': '172.21.15.100:6827', 'nonce': 363733545}]}, 'public_addr': '172.21.15.100:6818/363733545', 'cluster_addr': '172.21.15.100:6823/363733545', 'heartbeat_back_addr': '172.21.15.100:6831/363733545', 'heartbeat_front_addr': '172.21.15.100:6827/363733545', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '955b67e9-73bc-42c1-88e0-3356c105e1ea', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 134, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6808', 'nonce': 205694879}, {'type': 'v1', 'addr': '172.21.15.100:6809', 'nonce': 205694879}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6810', 'nonce': 205694879}, {'type': 'v1', 'addr': '172.21.15.100:6811', 'nonce': 205694879}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6814', 'nonce': 205694879}, {'type': 'v1', 'addr': '172.21.15.100:6815', 'nonce': 205694879}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6812', 'nonce': 205694879}, {'type': 'v1', 'addr': '172.21.15.100:6813', 'nonce': 205694879}]}, 'public_addr': '172.21.15.100:6809/205694879', 'cluster_addr': '172.21.15.100:6811/205694879', 'heartbeat_back_addr': '172.21.15.100:6815/205694879', 'heartbeat_front_addr': '172.21.15.100:6813/205694879', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': '414ea7f0-138d-404e-bce6-fe8dedcdc938', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 135, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6800', 'nonce': 3120011840}, {'type': 'v1', 'addr': '172.21.15.100:6801', 'nonce': 3120011840}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6802', 'nonce': 3120011840}, {'type': 'v1', 'addr': '172.21.15.100:6803', 'nonce': 3120011840}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6806', 'nonce': 3120011840}, {'type': 'v1', 'addr': '172.21.15.100:6807', 'nonce': 3120011840}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.100:6804', 'nonce': 3120011840}, {'type': 'v1', 'addr': '172.21.15.100:6805', 'nonce': 3120011840}]}, 'public_addr': '172.21.15.100:6801/3120011840', 'cluster_addr': '172.21.15.100:6803/3120011840', 'heartbeat_back_addr': '172.21.15.100:6807/3120011840', 'heartbeat_front_addr': '172.21.15.100:6805/3120011840', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:48:45.230746+0000', 'dead_epoch': 69}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:48:45.303567+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:48:45.210941+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:48:45.218918+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:48:45.301781+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:48:45.248444+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:48:45.275465+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138322906710015, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T15:48:45.166209+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 3, 'snaps': [{'begin': 16, 'length': 6}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-21T15:51:12.296 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2024-09-21T15:51:12.297 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:51:12.224+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:51:12.326 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:51:12.326+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:51:12.410 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:51:12.410 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:51:12.410 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:51:12.410 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:51:12.419 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:51:12.463 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:51:12.462+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:51:12.522 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:51:12.523 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:51:12.523 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:51:12.523 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:51:12.523 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:51:12.527 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:51:12.527+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:51:12.531 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:51:12.628 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:51:12.627+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:51:12.676 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is unset 2024-09-21T15:51:12.677 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:51:12.677 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:51:12.677 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:51:12.677 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:51:12.677 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.677 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155031 3.1c 3.8612741c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:12.677 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448748+0000", 2024-09-21T15:51:12.677 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 50.226568397999998, 2024-09-21T15:51:12.677 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064010286999999999, 2024-09-21T15:51:12.677 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:12.678 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:12.678 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:12.678 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:12.678 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:12.678 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155031 2024-09-21T15:51:12.678 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.678 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:12.678 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.678 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:12.678 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448748+0000", 2024-09-21T15:51:12.678 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:12.678 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.678 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.678 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:12.679 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448748+0000", 2024-09-21T15:51:12.679 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:12.679 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.679 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.679 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:12.679 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448747+0000", 2024-09-21T15:51:12.679 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:12.679 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.679 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.679 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:12.679 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448753+0000", 2024-09-21T15:51:12.679 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.755e-06 2024-09-21T15:51:12.679 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.679 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.679 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:12.680 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448763+0000", 2024-09-21T15:51:12.680 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7659999999999993e-06 2024-09-21T15:51:12.680 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.680 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.680 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:12.680 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448769+0000", 2024-09-21T15:51:12.680 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6060000000000002e-06 2024-09-21T15:51:12.680 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.680 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.680 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:12.680 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512750+0000", 2024-09-21T15:51:12.680 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.063981298000000006 2024-09-21T15:51:12.680 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.680 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.680 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:12.681 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512758+0000", 2024-09-21T15:51:12.681 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1669999999999999e-06 2024-09-21T15:51:12.681 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:12.681 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:12.681 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:12.681 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.681 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.681 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155091 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:12.681 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448916+0000", 2024-09-21T15:51:12.681 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 50.226400552999998, 2024-09-21T15:51:12.681 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064006309999999997, 2024-09-21T15:51:12.681 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:12.681 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:12.681 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:12.681 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:12.682 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:12.682 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155091 2024-09-21T15:51:12.682 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.682 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:12.682 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.682 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:12.682 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448916+0000", 2024-09-21T15:51:12.682 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:12.682 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.682 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.682 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:12.682 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448916+0000", 2024-09-21T15:51:12.682 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:12.682 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.683 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.683 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:12.683 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448915+0000", 2024-09-21T15:51:12.683 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:12.683 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.683 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.683 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:12.683 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448917+0000", 2024-09-21T15:51:12.683 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2400000000000002e-06 2024-09-21T15:51:12.683 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.683 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.683 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:12.683 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448923+0000", 2024-09-21T15:51:12.684 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0580000000000003e-06 2024-09-21T15:51:12.684 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.684 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.684 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:12.684 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448927+0000", 2024-09-21T15:51:12.684 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.031e-06 2024-09-21T15:51:12.684 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.684 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.684 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:12.684 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512916+0000", 2024-09-21T15:51:12.684 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.063988456999999999 2024-09-21T15:51:12.684 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.684 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.684 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:12.685 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512922+0000", 2024-09-21T15:51:12.685 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9009999999999999e-06 2024-09-21T15:51:12.685 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:12.685 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:12.685 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:12.685 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.685 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.685 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155092 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:12.685 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448931+0000", 2024-09-21T15:51:12.685 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 50.226384770999999, 2024-09-21T15:51:12.685 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064022793999999994, 2024-09-21T15:51:12.685 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:12.685 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:12.685 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:12.686 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:12.686 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:12.686 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155092 2024-09-21T15:51:12.686 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.686 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:12.686 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.686 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:12.686 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448931+0000", 2024-09-21T15:51:12.686 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:12.686 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.686 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.686 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:12.686 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448931+0000", 2024-09-21T15:51:12.686 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:12.687 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.687 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.687 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:12.687 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448931+0000", 2024-09-21T15:51:12.687 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:12.687 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.687 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.687 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:12.687 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448933+0000", 2024-09-21T15:51:12.687 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1799999999999999e-06 2024-09-21T15:51:12.687 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.687 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.687 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:12.687 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448939+0000", 2024-09-21T15:51:12.688 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.604e-06 2024-09-21T15:51:12.688 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.688 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.688 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:12.688 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448943+0000", 2024-09-21T15:51:12.688 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0740000000000003e-06 2024-09-21T15:51:12.688 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.688 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.688 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:12.688 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512946+0000", 2024-09-21T15:51:12.688 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064003612000000001 2024-09-21T15:51:12.688 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.688 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.688 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:12.689 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512954+0000", 2024-09-21T15:51:12.689 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6690000000000006e-06 2024-09-21T15:51:12.689 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:12.689 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:12.689 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:12.689 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.689 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.689 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155093 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:12.689 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448947+0000", 2024-09-21T15:51:12.689 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 50.226369396999999, 2024-09-21T15:51:12.689 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064020717000000005, 2024-09-21T15:51:12.689 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:12.689 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:12.689 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:12.689 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:12.690 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:12.690 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155093 2024-09-21T15:51:12.690 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.690 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:12.690 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.690 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:12.690 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448947+0000", 2024-09-21T15:51:12.690 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:12.690 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.690 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.690 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:12.690 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448947+0000", 2024-09-21T15:51:12.690 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:12.690 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.691 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.691 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:12.691 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448946+0000", 2024-09-21T15:51:12.691 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:12.691 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.691 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.691 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:12.691 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448949+0000", 2024-09-21T15:51:12.691 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3860000000000001e-06 2024-09-21T15:51:12.691 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.691 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.691 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:12.691 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448955+0000", 2024-09-21T15:51:12.692 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.823e-06 2024-09-21T15:51:12.692 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.692 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.692 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:12.692 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448959+0000", 2024-09-21T15:51:12.692 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2799999999999997e-06 2024-09-21T15:51:12.692 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.692 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.692 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:12.692 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512964+0000", 2024-09-21T15:51:12.692 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064004741000000004 2024-09-21T15:51:12.692 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.692 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.693 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:12.693 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512967+0000", 2024-09-21T15:51:12.693 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.839e-06 2024-09-21T15:51:12.693 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:12.693 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:12.693 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:12.693 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.693 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.693 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155094 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:12.693 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448962+0000", 2024-09-21T15:51:12.693 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 50.226353774000003, 2024-09-21T15:51:12.693 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064037378000000006, 2024-09-21T15:51:12.693 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:12.693 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:12.694 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:12.694 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:12.694 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:12.694 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155094 2024-09-21T15:51:12.694 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.694 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:12.694 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.694 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:12.694 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448962+0000", 2024-09-21T15:51:12.694 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:12.694 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.694 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.694 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:12.695 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448962+0000", 2024-09-21T15:51:12.695 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:12.695 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.695 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.695 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:12.695 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448962+0000", 2024-09-21T15:51:12.695 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:12.695 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.695 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.695 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:12.695 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448968+0000", 2024-09-21T15:51:12.695 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9710000000000001e-06 2024-09-21T15:51:12.695 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.695 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.695 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:12.696 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448974+0000", 2024-09-21T15:51:12.696 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7230000000000001e-06 2024-09-21T15:51:12.696 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.696 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.696 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:12.696 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448978+0000", 2024-09-21T15:51:12.696 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3749999999999996e-06 2024-09-21T15:51:12.696 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.696 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.696 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:12.696 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512992+0000", 2024-09-21T15:51:12.696 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064013525000000002 2024-09-21T15:51:12.696 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.696 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.697 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:12.697 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513000+0000", 2024-09-21T15:51:12.697 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0830000000000004e-06 2024-09-21T15:51:12.697 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:12.697 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:12.697 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:12.697 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.697 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.697 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155095 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:12.697 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448982+0000", 2024-09-21T15:51:12.697 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 50.226334491999999, 2024-09-21T15:51:12.697 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064041904999999996, 2024-09-21T15:51:12.697 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:12.697 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:12.697 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:12.698 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:12.698 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:12.698 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155095 2024-09-21T15:51:12.698 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.698 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:12.698 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.698 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:12.698 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448982+0000", 2024-09-21T15:51:12.698 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:12.698 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.698 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.698 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:12.698 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448982+0000", 2024-09-21T15:51:12.698 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:12.699 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.699 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.699 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:12.699 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448981+0000", 2024-09-21T15:51:12.699 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:12.699 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.699 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.699 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:12.699 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448983+0000", 2024-09-21T15:51:12.699 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.905e-06 2024-09-21T15:51:12.699 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.699 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.699 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:12.699 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448989+0000", 2024-09-21T15:51:12.699 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.536e-06 2024-09-21T15:51:12.700 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.700 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.700 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:12.700 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448993+0000", 2024-09-21T15:51:12.700 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2390000000000004e-06 2024-09-21T15:51:12.700 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.700 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.700 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:12.700 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513015+0000", 2024-09-21T15:51:12.700 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064022081999999994 2024-09-21T15:51:12.700 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.700 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.700 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:12.700 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513023+0000", 2024-09-21T15:51:12.700 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4500000000000004e-06 2024-09-21T15:51:12.701 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:12.701 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:12.701 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:12.701 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.701 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.701 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155096 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:12.701 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448996+0000", 2024-09-21T15:51:12.701 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 50.226319590999999, 2024-09-21T15:51:12.701 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064060868000000007, 2024-09-21T15:51:12.701 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:12.701 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:12.701 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:12.701 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:12.701 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:12.701 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155096 2024-09-21T15:51:12.702 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.702 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:12.702 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.702 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:12.702 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448996+0000", 2024-09-21T15:51:12.702 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:12.702 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.702 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.702 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:12.702 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448996+0000", 2024-09-21T15:51:12.702 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:12.702 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.702 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.702 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:12.702 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448996+0000", 2024-09-21T15:51:12.703 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:12.703 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.703 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.703 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:12.703 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448998+0000", 2024-09-21T15:51:12.703 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8780000000000001e-06 2024-09-21T15:51:12.703 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.703 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.703 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:12.703 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449003+0000", 2024-09-21T15:51:12.703 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4099999999999999e-06 2024-09-21T15:51:12.703 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.703 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.703 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:12.704 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449008+0000", 2024-09-21T15:51:12.704 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.335e-06 2024-09-21T15:51:12.704 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.704 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.704 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:12.704 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513049+0000", 2024-09-21T15:51:12.704 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064041059999999997 2024-09-21T15:51:12.704 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.704 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.704 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:12.704 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513057+0000", 2024-09-21T15:51:12.704 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5050000000000007e-06 2024-09-21T15:51:12.704 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:12.704 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:12.704 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:12.705 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.705 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.705 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155097 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:12.705 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449019+0000", 2024-09-21T15:51:12.705 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 50.226297170000002, 2024-09-21T15:51:12.705 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064052488000000005, 2024-09-21T15:51:12.705 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:12.705 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:12.705 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:12.705 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:12.705 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:12.705 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155097 2024-09-21T15:51:12.705 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.705 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:12.706 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.706 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:12.706 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449019+0000", 2024-09-21T15:51:12.706 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:12.706 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.706 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.706 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:12.706 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449019+0000", 2024-09-21T15:51:12.706 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:12.706 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.706 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.706 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:12.706 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449019+0000", 2024-09-21T15:51:12.707 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:12.707 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.707 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.707 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:12.707 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449021+0000", 2024-09-21T15:51:12.707 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.244e-06 2024-09-21T15:51:12.707 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.707 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.707 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:12.707 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449027+0000", 2024-09-21T15:51:12.707 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8619999999999999e-06 2024-09-21T15:51:12.707 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.707 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.708 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:12.708 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449031+0000", 2024-09-21T15:51:12.708 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1359999999999999e-06 2024-09-21T15:51:12.708 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.708 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.708 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:12.708 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513067+0000", 2024-09-21T15:51:12.708 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064036391999999998 2024-09-21T15:51:12.708 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.708 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.708 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:12.708 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513071+0000", 2024-09-21T15:51:12.708 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1989999999999999e-06 2024-09-21T15:51:12.708 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:12.709 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:12.709 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:12.709 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.709 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.709 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155098 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:12.709 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449035+0000", 2024-09-21T15:51:12.709 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 50.226281573000001, 2024-09-21T15:51:12.709 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064071761000000005, 2024-09-21T15:51:12.709 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:12.709 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:12.709 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:12.709 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:12.709 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:12.709 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155098 2024-09-21T15:51:12.710 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.710 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:12.710 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.710 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:12.710 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449035+0000", 2024-09-21T15:51:12.710 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:12.710 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.710 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.710 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:12.710 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449035+0000", 2024-09-21T15:51:12.710 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:12.710 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.710 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.711 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:12.711 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449034+0000", 2024-09-21T15:51:12.711 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:12.711 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.711 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.711 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:12.711 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449036+0000", 2024-09-21T15:51:12.711 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9989999999999998e-06 2024-09-21T15:51:12.711 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.711 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.711 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:12.711 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449042+0000", 2024-09-21T15:51:12.711 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6010000000000002e-06 2024-09-21T15:51:12.711 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.711 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.712 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:12.712 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449047+0000", 2024-09-21T15:51:12.712 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3600000000000004e-06 2024-09-21T15:51:12.712 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.712 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.712 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:12.712 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513098+0000", 2024-09-21T15:51:12.712 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064050544000000001 2024-09-21T15:51:12.712 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.712 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.712 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:12.712 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513106+0000", 2024-09-21T15:51:12.712 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5650000000000002e-06 2024-09-21T15:51:12.712 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:12.712 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:12.712 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:12.713 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.713 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.713 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155099 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:12.713 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449051+0000", 2024-09-21T15:51:12.713 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 50.226265032000001, 2024-09-21T15:51:12.713 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064069388000000005, 2024-09-21T15:51:12.713 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:12.713 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:12.713 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:12.713 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:12.713 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:12.713 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155099 2024-09-21T15:51:12.713 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.713 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:12.714 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.714 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:12.714 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449051+0000", 2024-09-21T15:51:12.714 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:12.714 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.714 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.714 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:12.714 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449051+0000", 2024-09-21T15:51:12.714 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:12.714 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.714 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.714 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:12.714 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449051+0000", 2024-09-21T15:51:12.714 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:12.715 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.715 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.715 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:12.715 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449053+0000", 2024-09-21T15:51:12.715 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2620000000000001e-06 2024-09-21T15:51:12.715 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.715 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.715 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:12.715 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449058+0000", 2024-09-21T15:51:12.715 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1680000000000003e-06 2024-09-21T15:51:12.715 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.715 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.715 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:12.715 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449062+0000", 2024-09-21T15:51:12.715 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9979999999999997e-06 2024-09-21T15:51:12.716 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.716 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.716 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:12.716 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513116+0000", 2024-09-21T15:51:12.716 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064053761000000001 2024-09-21T15:51:12.716 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.716 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.716 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:12.716 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513120+0000", 2024-09-21T15:51:12.716 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5399999999999997e-06 2024-09-21T15:51:12.716 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:12.716 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:12.716 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:12.716 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.716 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.717 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155100 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:12.717 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449101+0000", 2024-09-21T15:51:12.717 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 50.226215277000001, 2024-09-21T15:51:12.717 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064079585999999994, 2024-09-21T15:51:12.717 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:12.717 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:12.717 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:12.717 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:12.717 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:12.717 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155100 2024-09-21T15:51:12.717 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.717 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:12.717 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.718 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:12.718 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449101+0000", 2024-09-21T15:51:12.718 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:12.718 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.718 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.718 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:12.718 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449101+0000", 2024-09-21T15:51:12.718 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:12.718 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.718 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.718 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:12.718 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449100+0000", 2024-09-21T15:51:12.718 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-21T15:51:12.718 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.719 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.719 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:12.719 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449105+0000", 2024-09-21T15:51:12.719 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6700000000000002e-06 2024-09-21T15:51:12.719 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.719 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.719 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:12.719 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449117+0000", 2024-09-21T15:51:12.719 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1841000000000001e-05 2024-09-21T15:51:12.719 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.719 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.719 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:12.719 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449124+0000", 2024-09-21T15:51:12.720 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4660000000000002e-06 2024-09-21T15:51:12.720 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.720 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.720 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:12.720 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513163+0000", 2024-09-21T15:51:12.720 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064039053999999998 2024-09-21T15:51:12.720 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.720 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.720 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:12.720 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513180+0000", 2024-09-21T15:51:12.720 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7333000000000002e-05 2024-09-21T15:51:12.720 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:12.720 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:12.720 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:12.721 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.721 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.721 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155101 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:12.721 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449166+0000", 2024-09-21T15:51:12.721 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 50.226150572000002, 2024-09-21T15:51:12.721 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064027639999999997, 2024-09-21T15:51:12.721 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:12.721 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:12.721 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:12.721 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:12.721 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:12.721 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155101 2024-09-21T15:51:12.721 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.721 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:12.722 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.722 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:12.722 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449166+0000", 2024-09-21T15:51:12.722 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:12.722 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.722 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.722 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:12.722 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449166+0000", 2024-09-21T15:51:12.722 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:12.722 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.722 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.722 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:12.722 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449165+0000", 2024-09-21T15:51:12.723 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:12.723 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.723 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.723 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:12.723 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449168+0000", 2024-09-21T15:51:12.723 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5099999999999999e-06 2024-09-21T15:51:12.723 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.723 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.723 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:12.723 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449177+0000", 2024-09-21T15:51:12.723 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6659999999999997e-06 2024-09-21T15:51:12.723 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.723 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.723 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:12.724 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449182+0000", 2024-09-21T15:51:12.724 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1320000000000002e-06 2024-09-21T15:51:12.724 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.724 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.724 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:12.724 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513189+0000", 2024-09-21T15:51:12.724 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064007231999999997 2024-09-21T15:51:12.724 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.724 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.724 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:12.724 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513193+0000", 2024-09-21T15:51:12.724 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.692e-06 2024-09-21T15:51:12.724 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:12.724 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:12.725 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:12.725 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.725 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.725 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155134 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:12.725 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449187+0000", 2024-09-21T15:51:12.725 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 50.226128983999999, 2024-09-21T15:51:12.725 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064031331999999996, 2024-09-21T15:51:12.725 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:12.725 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:12.725 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:12.725 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:12.725 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:12.725 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155134 2024-09-21T15:51:12.725 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.726 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:12.726 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.726 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:12.726 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449187+0000", 2024-09-21T15:51:12.726 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:12.726 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.726 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.726 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:12.726 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449187+0000", 2024-09-21T15:51:12.726 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:12.726 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.726 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.726 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:12.726 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449187+0000", 2024-09-21T15:51:12.726 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:12.727 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.727 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.727 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:12.727 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449189+0000", 2024-09-21T15:51:12.727 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5469999999999999e-06 2024-09-21T15:51:12.727 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.727 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.727 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:12.727 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449196+0000", 2024-09-21T15:51:12.727 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7009999999999999e-06 2024-09-21T15:51:12.727 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.727 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.727 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:12.727 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449202+0000", 2024-09-21T15:51:12.728 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7409999999999998e-06 2024-09-21T15:51:12.728 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.728 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.728 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:12.728 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513214+0000", 2024-09-21T15:51:12.728 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064012301999999993 2024-09-21T15:51:12.728 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.728 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.728 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:12.728 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513218+0000", 2024-09-21T15:51:12.728 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5709999999999999e-06 2024-09-21T15:51:12.728 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:12.728 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:12.728 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:12.728 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.728 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.729 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155135 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:12.729 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449207+0000", 2024-09-21T15:51:12.729 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 50.226109473999998, 2024-09-21T15:51:12.729 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064029249999999996, 2024-09-21T15:51:12.729 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:12.729 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:12.729 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:12.729 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:12.729 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:12.729 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155135 2024-09-21T15:51:12.729 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.729 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:12.729 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.729 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:12.729 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449207+0000", 2024-09-21T15:51:12.730 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:12.730 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.730 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.730 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:12.730 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449207+0000", 2024-09-21T15:51:12.730 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:12.730 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.730 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.730 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:12.730 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449206+0000", 2024-09-21T15:51:12.730 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:12.730 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.730 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.731 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:12.731 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449209+0000", 2024-09-21T15:51:12.731 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5150000000000001e-06 2024-09-21T15:51:12.731 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.731 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.731 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:12.731 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449215+0000", 2024-09-21T15:51:12.731 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9329999999999998e-06 2024-09-21T15:51:12.731 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.731 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.731 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:12.731 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449220+0000", 2024-09-21T15:51:12.731 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0819999999999998e-06 2024-09-21T15:51:12.731 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.732 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.732 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:12.732 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513232+0000", 2024-09-21T15:51:12.732 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064012052 2024-09-21T15:51:12.732 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.732 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.732 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:12.732 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513236+0000", 2024-09-21T15:51:12.732 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2080000000000002e-06 2024-09-21T15:51:12.732 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:12.732 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:12.732 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:12.732 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.732 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.732 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155136 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:12.732 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449224+0000", 2024-09-21T15:51:12.733 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 50.226091910000001, 2024-09-21T15:51:12.733 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064053025999999999, 2024-09-21T15:51:12.733 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:12.733 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:12.733 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:12.733 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:12.733 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:12.733 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155136 2024-09-21T15:51:12.733 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.733 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:12.733 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.733 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:12.733 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449224+0000", 2024-09-21T15:51:12.733 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:12.733 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.734 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.734 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:12.734 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449224+0000", 2024-09-21T15:51:12.734 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:12.734 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.734 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.734 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:12.734 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449224+0000", 2024-09-21T15:51:12.734 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:12.734 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.734 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.734 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:12.734 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449226+0000", 2024-09-21T15:51:12.735 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2699999999999999e-06 2024-09-21T15:51:12.735 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.735 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.735 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:12.735 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449233+0000", 2024-09-21T15:51:12.735 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4289999999999998e-06 2024-09-21T15:51:12.735 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.735 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.735 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:12.735 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449238+0000", 2024-09-21T15:51:12.735 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0730000000000004e-06 2024-09-21T15:51:12.735 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.735 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.735 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:12.736 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513269+0000", 2024-09-21T15:51:12.736 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064030999000000005 2024-09-21T15:51:12.736 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.736 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.736 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:12.736 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513277+0000", 2024-09-21T15:51:12.736 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5520000000000005e-06 2024-09-21T15:51:12.736 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:12.736 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:12.736 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:12.736 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.736 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.736 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155137 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:12.736 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449263+0000", 2024-09-21T15:51:12.737 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 50.226052778000003, 2024-09-21T15:51:12.737 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.06402803, 2024-09-21T15:51:12.737 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:12.737 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:12.737 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:12.737 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:12.737 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:12.737 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155137 2024-09-21T15:51:12.737 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.737 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:12.737 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.737 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:12.737 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449263+0000", 2024-09-21T15:51:12.737 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:12.737 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.738 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.738 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:12.738 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449263+0000", 2024-09-21T15:51:12.738 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:12.738 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.738 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.738 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:12.738 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449262+0000", 2024-09-21T15:51:12.738 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-21T15:51:12.738 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.738 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.738 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:12.738 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449267+0000", 2024-09-21T15:51:12.739 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.933e-06 2024-09-21T15:51:12.739 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.739 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.739 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:12.739 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449279+0000", 2024-09-21T15:51:12.739 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.181e-05 2024-09-21T15:51:12.739 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.739 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.739 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:12.739 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449286+0000", 2024-09-21T15:51:12.739 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4749999999999998e-06 2024-09-21T15:51:12.739 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.739 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.739 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:12.739 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513287+0000", 2024-09-21T15:51:12.740 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064001369000000002 2024-09-21T15:51:12.740 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.740 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.740 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:12.740 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513291+0000", 2024-09-21T15:51:12.740 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3000000000000003e-06 2024-09-21T15:51:12.740 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:12.740 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:12.740 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:12.740 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.740 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.740 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155138 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:12.740 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449299+0000", 2024-09-21T15:51:12.740 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 50.226017286000001, 2024-09-21T15:51:12.741 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064028359000000007, 2024-09-21T15:51:12.741 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:12.741 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:12.741 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:12.741 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:12.741 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:12.741 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155138 2024-09-21T15:51:12.741 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.741 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:12.741 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.741 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:12.741 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449299+0000", 2024-09-21T15:51:12.741 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:12.741 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.742 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.742 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:12.742 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449299+0000", 2024-09-21T15:51:12.742 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:12.742 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.742 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.742 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:12.742 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449298+0000", 2024-09-21T15:51:12.742 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:12.742 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.742 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.742 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:12.742 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449301+0000", 2024-09-21T15:51:12.742 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0230000000000001e-06 2024-09-21T15:51:12.743 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.743 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.743 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:12.743 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449308+0000", 2024-09-21T15:51:12.743 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3119999999999999e-06 2024-09-21T15:51:12.743 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.743 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.743 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:12.743 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449313+0000", 2024-09-21T15:51:12.743 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8980000000000004e-06 2024-09-21T15:51:12.743 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.743 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.743 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:12.743 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513318+0000", 2024-09-21T15:51:12.744 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064005612000000003 2024-09-21T15:51:12.744 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.744 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.744 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:12.744 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513327+0000", 2024-09-21T15:51:12.744 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0100000000000001e-06 2024-09-21T15:51:12.744 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:12.744 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:12.744 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:12.744 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.744 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.744 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155142 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:12.744 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449392+0000", 2024-09-21T15:51:12.744 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 50.225924528, 2024-09-21T15:51:12.745 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064012464000000005, 2024-09-21T15:51:12.745 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:12.745 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:12.745 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:12.745 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:12.745 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:12.745 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155142 2024-09-21T15:51:12.745 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.745 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:12.745 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.745 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:12.745 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449392+0000", 2024-09-21T15:51:12.745 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:12.745 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.746 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.746 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:12.746 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449392+0000", 2024-09-21T15:51:12.746 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:12.746 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.746 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.746 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:12.746 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449391+0000", 2024-09-21T15:51:12.746 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:12.746 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.746 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.746 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:12.746 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449393+0000", 2024-09-21T15:51:12.746 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.26e-06 2024-09-21T15:51:12.746 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.747 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.747 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:12.747 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449399+0000", 2024-09-21T15:51:12.747 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6269999999999997e-06 2024-09-21T15:51:12.747 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.747 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.747 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:12.747 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449403+0000", 2024-09-21T15:51:12.747 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3000000000000003e-06 2024-09-21T15:51:12.747 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.747 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.747 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:12.748 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513396+0000", 2024-09-21T15:51:12.748 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.063992568999999999 2024-09-21T15:51:12.748 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.748 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.748 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:12.748 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513404+0000", 2024-09-21T15:51:12.748 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1100000000000003e-06 2024-09-21T15:51:12.748 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:12.748 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:12.748 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:12.748 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.748 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.748 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155143 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:12.748 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449407+0000", 2024-09-21T15:51:12.749 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 50.225908941999997, 2024-09-21T15:51:12.749 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064009916, 2024-09-21T15:51:12.749 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:12.749 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:12.749 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:12.749 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:12.749 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:12.749 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155143 2024-09-21T15:51:12.749 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.749 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:12.749 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.749 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:12.749 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449407+0000", 2024-09-21T15:51:12.749 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:12.749 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.749 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.750 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:12.750 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449407+0000", 2024-09-21T15:51:12.750 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:12.750 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.750 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.750 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:12.750 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449407+0000", 2024-09-21T15:51:12.750 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:12.750 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.750 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.750 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:12.750 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449409+0000", 2024-09-21T15:51:12.750 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1090000000000001e-06 2024-09-21T15:51:12.750 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.750 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.751 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:12.751 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449413+0000", 2024-09-21T15:51:12.751 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6650000000000001e-06 2024-09-21T15:51:12.751 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.751 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.751 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:12.751 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449415+0000", 2024-09-21T15:51:12.751 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5459999999999998e-06 2024-09-21T15:51:12.751 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.751 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.751 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:12.751 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513413+0000", 2024-09-21T15:51:12.751 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.063998037999999993 2024-09-21T15:51:12.751 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.752 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.752 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:12.752 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513417+0000", 2024-09-21T15:51:12.752 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8789999999999996e-06 2024-09-21T15:51:12.752 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:12.752 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:12.752 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:12.752 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.752 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.752 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155144 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:12.752 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449427+0000", 2024-09-21T15:51:12.752 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 50.225888933999997, 2024-09-21T15:51:12.752 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064022531999999993, 2024-09-21T15:51:12.752 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:12.752 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:12.752 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:12.753 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:12.753 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:12.753 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155144 2024-09-21T15:51:12.753 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.753 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:12.753 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.753 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:12.753 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449427+0000", 2024-09-21T15:51:12.753 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:12.753 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.753 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.753 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:12.753 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449427+0000", 2024-09-21T15:51:12.754 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:12.754 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.754 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.754 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:12.754 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449427+0000", 2024-09-21T15:51:12.754 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:12.754 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.754 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.754 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:12.754 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449430+0000", 2024-09-21T15:51:12.754 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3579999999999999e-06 2024-09-21T15:51:12.754 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.754 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.754 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:12.754 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449437+0000", 2024-09-21T15:51:12.755 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4580000000000004e-06 2024-09-21T15:51:12.755 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.755 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.755 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:12.755 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449443+0000", 2024-09-21T15:51:12.755 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5609999999999998e-06 2024-09-21T15:51:12.755 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.755 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.755 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:12.755 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513441+0000", 2024-09-21T15:51:12.755 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.063998136999999997 2024-09-21T15:51:12.755 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:12.755 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:12.755 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:12.756 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513450+0000", 2024-09-21T15:51:12.756 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4950000000000008e-06 2024-09-21T15:51:12.756 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:12.756 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:12.756 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:12.756 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:12.756 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:12.756 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:51:12.758 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:51:12.758 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:51:12.756+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:51:12.858 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:51:12.856+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:51:12.901 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:51:12.901 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:51:12.902 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:51:12.902 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:51:12.910 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:51:12.994 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:51:12.993+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:51:13.054 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:51:13.054 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:51:13.054 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:51:13.054 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:51:13.055 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:51:13.059 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:51:13.057+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:51:13.062 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:51:13.160 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:51:13.159+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:51:13.207 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:51:13.207 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:51:13.207 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:51:13.207 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:51:13.207 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.207 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120111 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 321536~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:13.207 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572216+0000", 2024-09-21T15:51:13.207 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 69.632921108000005, 2024-09-21T15:51:13.207 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054657399000000002, 2024-09-21T15:51:13.207 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:13.208 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:13.208 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:13.208 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:13.208 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:13.208 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120111 2024-09-21T15:51:13.208 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.208 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:13.208 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.208 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:13.208 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572216+0000", 2024-09-21T15:51:13.208 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:13.208 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.208 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.208 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:13.208 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572216+0000", 2024-09-21T15:51:13.209 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:13.209 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.209 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.209 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:13.209 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572216+0000", 2024-09-21T15:51:13.209 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:13.209 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.209 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.209 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:13.209 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572218+0000", 2024-09-21T15:51:13.209 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1840000000000002e-06 2024-09-21T15:51:13.209 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.209 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.209 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:13.210 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572224+0000", 2024-09-21T15:51:13.210 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7880000000000003e-06 2024-09-21T15:51:13.210 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.210 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.210 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:13.210 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572229+0000", 2024-09-21T15:51:13.210 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8239999999999999e-06 2024-09-21T15:51:13.210 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.210 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.210 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:13.210 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626680+0000", 2024-09-21T15:51:13.210 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054451431000000002 2024-09-21T15:51:13.210 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.210 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.210 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:13.211 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626755+0000", 2024-09-21T15:51:13.211 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4641000000000006e-05 2024-09-21T15:51:13.211 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.211 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.211 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:13.211 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626874+0000", 2024-09-21T15:51:13.211 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00011886 2024-09-21T15:51:13.211 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:13.211 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:13.211 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:13.211 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.211 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.211 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120112 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 342016~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:13.212 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572233+0000", 2024-09-21T15:51:13.212 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 69.632904740000001, 2024-09-21T15:51:13.212 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054823997999999999, 2024-09-21T15:51:13.212 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:13.212 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:13.212 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:13.212 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:13.212 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:13.212 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120112 2024-09-21T15:51:13.212 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.212 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:13.212 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.212 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:13.212 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572233+0000", 2024-09-21T15:51:13.212 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:13.213 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.213 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.213 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:13.213 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572233+0000", 2024-09-21T15:51:13.213 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:13.213 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.213 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.213 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:13.213 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572232+0000", 2024-09-21T15:51:13.213 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:13.213 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.213 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.213 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:13.213 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572234+0000", 2024-09-21T15:51:13.213 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0660000000000002e-06 2024-09-21T15:51:13.214 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.214 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.214 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:13.214 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572242+0000", 2024-09-21T15:51:13.214 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0430000000000002e-06 2024-09-21T15:51:13.214 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.214 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.214 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:13.214 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572246+0000", 2024-09-21T15:51:13.214 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9080000000000003e-06 2024-09-21T15:51:13.214 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.214 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.214 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:13.214 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626916+0000", 2024-09-21T15:51:13.215 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054669482999999998 2024-09-21T15:51:13.215 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.215 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.215 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:13.215 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626977+0000", 2024-09-21T15:51:13.215 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0597000000000003e-05 2024-09-21T15:51:13.215 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.215 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.215 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:13.215 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627057+0000", 2024-09-21T15:51:13.215 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0258000000000003e-05 2024-09-21T15:51:13.215 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:13.215 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:13.215 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:13.216 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.216 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.216 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120113 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 362496~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:13.216 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572253+0000", 2024-09-21T15:51:13.216 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 69.632884755000006, 2024-09-21T15:51:13.216 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.05503044, 2024-09-21T15:51:13.216 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:13.216 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:13.216 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:13.216 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:13.216 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:13.216 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120113 2024-09-21T15:51:13.216 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.216 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:13.216 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.216 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:13.217 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572253+0000", 2024-09-21T15:51:13.217 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:13.217 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.217 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.217 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:13.217 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572253+0000", 2024-09-21T15:51:13.217 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:13.217 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.217 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.217 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:13.217 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572252+0000", 2024-09-21T15:51:13.217 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:13.217 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.217 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.217 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:13.218 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572258+0000", 2024-09-21T15:51:13.218 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6169999999999998e-06 2024-09-21T15:51:13.218 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.218 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.218 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:13.218 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572266+0000", 2024-09-21T15:51:13.218 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9410000000000007e-06 2024-09-21T15:51:13.218 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.218 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.218 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:13.218 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572272+0000", 2024-09-21T15:51:13.218 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.241e-06 2024-09-21T15:51:13.218 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.218 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.218 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:13.218 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627087+0000", 2024-09-21T15:51:13.219 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054814850999999998 2024-09-21T15:51:13.219 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.219 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.219 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:13.219 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627162+0000", 2024-09-21T15:51:13.219 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5012999999999995e-05 2024-09-21T15:51:13.219 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.219 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.219 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:13.219 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627283+0000", 2024-09-21T15:51:13.219 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012113 2024-09-21T15:51:13.219 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:13.219 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:13.219 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:13.219 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.220 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.220 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120114 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 382976~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:13.220 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572277+0000", 2024-09-21T15:51:13.220 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 69.632860737000001, 2024-09-21T15:51:13.220 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055184395999999997, 2024-09-21T15:51:13.220 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:13.220 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:13.220 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:13.220 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:13.220 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:13.220 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120114 2024-09-21T15:51:13.220 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.220 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:13.221 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.221 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:13.221 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572277+0000", 2024-09-21T15:51:13.221 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:13.221 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.221 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.221 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:13.221 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572277+0000", 2024-09-21T15:51:13.221 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:13.221 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.221 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.221 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:13.221 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572276+0000", 2024-09-21T15:51:13.221 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:13.221 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.222 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.222 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:13.222 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572279+0000", 2024-09-21T15:51:13.222 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.373e-06 2024-09-21T15:51:13.222 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.222 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.222 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:13.222 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572285+0000", 2024-09-21T15:51:13.222 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9359999999999996e-06 2024-09-21T15:51:13.222 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.222 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.222 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:13.222 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572289+0000", 2024-09-21T15:51:13.223 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.634e-06 2024-09-21T15:51:13.223 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.223 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.223 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:13.223 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627311+0000", 2024-09-21T15:51:13.223 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055021211 2024-09-21T15:51:13.223 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.223 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.223 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:13.223 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627360+0000", 2024-09-21T15:51:13.223 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9203999999999998e-05 2024-09-21T15:51:13.223 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.223 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.223 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:13.224 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627461+0000", 2024-09-21T15:51:13.224 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000101469 2024-09-21T15:51:13.224 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:13.224 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:13.224 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:13.224 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.224 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.224 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120115 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 399360~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:13.224 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572294+0000", 2024-09-21T15:51:13.224 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 69.632843429999994, 2024-09-21T15:51:13.224 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055400092999999997, 2024-09-21T15:51:13.224 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:13.224 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:13.224 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:13.225 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:13.225 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:13.225 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120115 2024-09-21T15:51:13.225 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.225 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:13.225 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.225 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:13.225 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572294+0000", 2024-09-21T15:51:13.225 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:13.225 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.225 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.225 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:13.225 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572294+0000", 2024-09-21T15:51:13.225 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:13.225 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.226 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.226 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:13.226 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572294+0000", 2024-09-21T15:51:13.226 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:13.226 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.226 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.226 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:13.226 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572296+0000", 2024-09-21T15:51:13.226 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1100000000000001e-06 2024-09-21T15:51:13.226 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.226 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.226 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:13.226 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572303+0000", 2024-09-21T15:51:13.226 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8580000000000002e-06 2024-09-21T15:51:13.226 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.227 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.227 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:13.227 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572314+0000", 2024-09-21T15:51:13.227 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.101e-05 2024-09-21T15:51:13.227 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.227 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.227 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:13.227 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627496+0000", 2024-09-21T15:51:13.227 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055182522999999997 2024-09-21T15:51:13.227 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.227 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.227 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:13.227 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627568+0000", 2024-09-21T15:51:13.227 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1569999999999994e-05 2024-09-21T15:51:13.227 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.227 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.228 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:13.228 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627694+0000", 2024-09-21T15:51:13.228 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000126372 2024-09-21T15:51:13.228 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:13.228 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:13.228 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:13.228 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.228 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.228 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120116 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 419840~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:13.228 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572318+0000", 2024-09-21T15:51:13.228 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 69.632819239, 2024-09-21T15:51:13.228 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055539416000000001, 2024-09-21T15:51:13.228 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:13.228 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:13.228 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:13.229 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:13.229 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:13.229 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120116 2024-09-21T15:51:13.229 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.229 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:13.229 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.229 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:13.229 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572318+0000", 2024-09-21T15:51:13.229 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:13.229 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.229 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.229 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:13.229 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572318+0000", 2024-09-21T15:51:13.229 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:13.229 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.230 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.230 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:13.230 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572318+0000", 2024-09-21T15:51:13.230 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:13.230 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.230 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.230 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:13.230 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572322+0000", 2024-09-21T15:51:13.230 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4839999999999998e-06 2024-09-21T15:51:13.230 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.230 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.230 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:13.230 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572329+0000", 2024-09-21T15:51:13.230 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0959999999999997e-06 2024-09-21T15:51:13.231 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.231 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.231 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:13.231 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572333+0000", 2024-09-21T15:51:13.231 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9359999999999998e-06 2024-09-21T15:51:13.231 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.231 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.231 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:13.231 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627726+0000", 2024-09-21T15:51:13.231 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055392829999999997 2024-09-21T15:51:13.231 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.231 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.231 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:13.231 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627779+0000", 2024-09-21T15:51:13.232 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2185999999999998e-05 2024-09-21T15:51:13.232 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.232 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.232 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:13.232 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627858+0000", 2024-09-21T15:51:13.232 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9190999999999994e-05 2024-09-21T15:51:13.232 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:13.232 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:13.232 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:13.232 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.232 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.232 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120117 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 440320~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:13.232 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572338+0000", 2024-09-21T15:51:13.232 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 69.632799926000004, 2024-09-21T15:51:13.232 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055680888999999997, 2024-09-21T15:51:13.233 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:13.233 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:13.233 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:13.233 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:13.233 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:13.233 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120117 2024-09-21T15:51:13.233 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.233 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:13.233 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.233 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:13.233 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572338+0000", 2024-09-21T15:51:13.233 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:13.233 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.233 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.233 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:13.233 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572338+0000", 2024-09-21T15:51:13.234 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:13.234 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.234 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.234 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:13.234 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572337+0000", 2024-09-21T15:51:13.234 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:13.234 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.234 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.234 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:13.234 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572339+0000", 2024-09-21T15:51:13.234 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9379999999999999e-06 2024-09-21T15:51:13.234 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.234 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.235 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:13.235 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572349+0000", 2024-09-21T15:51:13.235 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7780000000000002e-06 2024-09-21T15:51:13.235 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.235 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.235 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:13.235 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572354+0000", 2024-09-21T15:51:13.235 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8740000000000003e-06 2024-09-21T15:51:13.235 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.235 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.235 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:13.235 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627885+0000", 2024-09-21T15:51:13.235 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055531448999999997 2024-09-21T15:51:13.235 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.235 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.236 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:13.236 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627935+0000", 2024-09-21T15:51:13.236 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9348999999999998e-05 2024-09-21T15:51:13.236 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.236 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.236 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:13.236 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628018+0000", 2024-09-21T15:51:13.236 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3851999999999996e-05 2024-09-21T15:51:13.236 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:13.236 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:13.236 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:13.236 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.236 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.236 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120118 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 460800~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:13.237 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572359+0000", 2024-09-21T15:51:13.237 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 69.632778684000002, 2024-09-21T15:51:13.237 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055951714, 2024-09-21T15:51:13.237 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:13.237 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:13.237 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:13.237 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:13.237 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:13.237 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120118 2024-09-21T15:51:13.237 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.237 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:13.237 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.237 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:13.237 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572359+0000", 2024-09-21T15:51:13.237 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:13.238 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.238 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.238 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:13.238 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572359+0000", 2024-09-21T15:51:13.238 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:13.238 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.238 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.238 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:13.238 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572358+0000", 2024-09-21T15:51:13.238 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:13.238 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.238 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.238 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:13.238 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572361+0000", 2024-09-21T15:51:13.238 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.26e-06 2024-09-21T15:51:13.238 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.239 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.239 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:13.239 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572367+0000", 2024-09-21T15:51:13.239 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3640000000000004e-06 2024-09-21T15:51:13.239 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.239 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.239 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:13.239 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572376+0000", 2024-09-21T15:51:13.239 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2310000000000002e-06 2024-09-21T15:51:13.239 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.239 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.239 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:13.239 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628096+0000", 2024-09-21T15:51:13.239 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055719369999999997 2024-09-21T15:51:13.239 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.240 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.240 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:13.240 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628178+0000", 2024-09-21T15:51:13.240 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2667e-05 2024-09-21T15:51:13.240 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.240 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.240 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:13.240 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628311+0000", 2024-09-21T15:51:13.240 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000132175 2024-09-21T15:51:13.240 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:13.240 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:13.240 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:13.240 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.240 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.240 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120119 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 481280~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:13.240 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572381+0000", 2024-09-21T15:51:13.241 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 69.632756240000006, 2024-09-21T15:51:13.241 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056088791999999998, 2024-09-21T15:51:13.241 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:13.241 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:13.241 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:13.241 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:13.241 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:13.241 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120119 2024-09-21T15:51:13.241 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.241 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:13.241 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.241 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:13.241 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572381+0000", 2024-09-21T15:51:13.241 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:13.241 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.242 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.242 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:13.242 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572381+0000", 2024-09-21T15:51:13.242 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:13.242 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.242 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.242 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:13.242 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572381+0000", 2024-09-21T15:51:13.242 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:13.242 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.242 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.242 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:13.242 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572383+0000", 2024-09-21T15:51:13.242 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0619999999999999e-06 2024-09-21T15:51:13.242 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.243 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.243 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:13.243 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572391+0000", 2024-09-21T15:51:13.243 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8769999999999993e-06 2024-09-21T15:51:13.243 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.243 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.243 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:13.243 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572399+0000", 2024-09-21T15:51:13.243 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6240000000000008e-06 2024-09-21T15:51:13.243 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.243 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.243 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:13.243 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628344+0000", 2024-09-21T15:51:13.243 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055944828000000002 2024-09-21T15:51:13.243 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.243 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.244 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:13.244 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628393+0000", 2024-09-21T15:51:13.244 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8653000000000003e-05 2024-09-21T15:51:13.244 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.244 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.244 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:13.244 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628470+0000", 2024-09-21T15:51:13.244 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7121999999999998e-05 2024-09-21T15:51:13.244 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:13.244 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:13.244 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:13.244 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.244 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.244 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120120 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 501760~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:13.244 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572418+0000", 2024-09-21T15:51:13.245 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 69.632719742000006, 2024-09-21T15:51:13.245 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056209138999999998, 2024-09-21T15:51:13.245 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:13.245 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:13.245 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:13.245 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:13.245 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:13.245 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120120 2024-09-21T15:51:13.245 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.245 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:13.245 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.245 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:13.245 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572418+0000", 2024-09-21T15:51:13.246 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:13.246 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.246 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.246 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:13.246 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572418+0000", 2024-09-21T15:51:13.246 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:13.246 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.246 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.246 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:13.246 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572417+0000", 2024-09-21T15:51:13.246 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:13.246 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.246 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.246 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:13.247 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572420+0000", 2024-09-21T15:51:13.247 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5270000000000001e-06 2024-09-21T15:51:13.247 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.247 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.247 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:13.247 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572426+0000", 2024-09-21T15:51:13.247 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3380000000000001e-06 2024-09-21T15:51:13.247 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.247 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.247 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:13.247 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572431+0000", 2024-09-21T15:51:13.247 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.865e-06 2024-09-21T15:51:13.247 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.247 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.247 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:13.248 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628500+0000", 2024-09-21T15:51:13.248 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056068917000000003 2024-09-21T15:51:13.248 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.248 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.248 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:13.248 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628542+0000", 2024-09-21T15:51:13.248 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1761999999999998e-05 2024-09-21T15:51:13.248 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.248 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.248 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:13.248 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628627+0000", 2024-09-21T15:51:13.248 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5247999999999994e-05 2024-09-21T15:51:13.248 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:13.248 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:13.248 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:13.248 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.249 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.249 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120121 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 522240~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:13.249 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572630+0000", 2024-09-21T15:51:13.249 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 69.632507356000005, 2024-09-21T15:51:13.249 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056207209000000001, 2024-09-21T15:51:13.249 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:13.249 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:13.249 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:13.249 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:13.249 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:13.249 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120121 2024-09-21T15:51:13.249 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.249 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:13.249 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.249 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:13.249 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572630+0000", 2024-09-21T15:51:13.250 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:13.250 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.250 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.250 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:13.250 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572630+0000", 2024-09-21T15:51:13.250 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:13.250 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.250 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.250 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:13.250 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572435+0000", 2024-09-21T15:51:13.250 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998045 2024-09-21T15:51:13.250 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.250 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.250 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:13.251 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572633+0000", 2024-09-21T15:51:13.251 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000198786 2024-09-21T15:51:13.251 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.251 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.251 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:13.251 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572641+0000", 2024-09-21T15:51:13.251 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9449999999999993e-06 2024-09-21T15:51:13.251 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.251 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.251 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:13.251 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572648+0000", 2024-09-21T15:51:13.251 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1929999999999998e-06 2024-09-21T15:51:13.251 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.251 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.251 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:13.252 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628669+0000", 2024-09-21T15:51:13.252 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056021690999999998 2024-09-21T15:51:13.252 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.252 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.252 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:13.252 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628751+0000", 2024-09-21T15:51:13.252 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1710000000000002e-05 2024-09-21T15:51:13.252 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.252 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.252 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:13.252 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628837+0000", 2024-09-21T15:51:13.252 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6383000000000003e-05 2024-09-21T15:51:13.252 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:13.252 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:13.252 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:13.253 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.253 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.253 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120122 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 538624~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:13.253 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.573471+0000", 2024-09-21T15:51:13.253 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 69.631666111000001, 2024-09-21T15:51:13.253 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055563935000000002, 2024-09-21T15:51:13.253 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:13.253 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:13.253 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:13.253 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:13.253 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:13.253 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120122 2024-09-21T15:51:13.253 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.253 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:13.254 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.254 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:13.254 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573471+0000", 2024-09-21T15:51:13.254 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:13.254 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.254 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.254 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:13.254 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573471+0000", 2024-09-21T15:51:13.254 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:13.254 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.254 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.254 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:13.254 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572653+0000", 2024-09-21T15:51:13.254 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9991813 2024-09-21T15:51:13.255 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.255 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.255 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:13.255 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573475+0000", 2024-09-21T15:51:13.255 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00082196500000000002 2024-09-21T15:51:13.255 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.255 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.255 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:13.255 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573483+0000", 2024-09-21T15:51:13.255 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5939999999999994e-06 2024-09-21T15:51:13.255 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.255 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.255 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:13.256 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573489+0000", 2024-09-21T15:51:13.256 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1689999999999997e-06 2024-09-21T15:51:13.256 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.256 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.256 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:13.256 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628882+0000", 2024-09-21T15:51:13.256 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055392230000000001 2024-09-21T15:51:13.256 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.256 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.256 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:13.256 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628934+0000", 2024-09-21T15:51:13.256 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1870000000000003e-05 2024-09-21T15:51:13.256 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.256 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.256 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:13.256 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629035+0000", 2024-09-21T15:51:13.257 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000101753 2024-09-21T15:51:13.257 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:13.257 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:13.257 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:13.257 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.257 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.257 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120123 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 559104~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:13.257 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.573495+0000", 2024-09-21T15:51:13.257 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 69.631642412000005, 2024-09-21T15:51:13.257 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055704765000000003, 2024-09-21T15:51:13.257 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:13.257 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:13.257 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:13.257 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:13.257 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:13.257 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120123 2024-09-21T15:51:13.258 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.258 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:13.258 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.258 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:13.258 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573495+0000", 2024-09-21T15:51:13.258 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:13.258 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.258 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.258 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:13.258 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573495+0000", 2024-09-21T15:51:13.258 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:13.258 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.258 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.258 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:13.258 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573495+0000", 2024-09-21T15:51:13.259 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:13.259 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.259 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.259 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:13.259 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573497+0000", 2024-09-21T15:51:13.259 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3420000000000002e-06 2024-09-21T15:51:13.259 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.259 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.259 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:13.259 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573503+0000", 2024-09-21T15:51:13.259 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7030000000000003e-06 2024-09-21T15:51:13.259 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.259 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.259 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:13.260 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573511+0000", 2024-09-21T15:51:13.260 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0790000000000001e-06 2024-09-21T15:51:13.260 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.260 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.260 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:13.260 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629067+0000", 2024-09-21T15:51:13.260 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055556573999999997 2024-09-21T15:51:13.260 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.260 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.260 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:13.260 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629118+0000", 2024-09-21T15:51:13.260 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1053000000000001e-05 2024-09-21T15:51:13.260 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.260 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.260 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:13.261 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629200+0000", 2024-09-21T15:51:13.261 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1452999999999994e-05 2024-09-21T15:51:13.261 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:13.261 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:13.261 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:13.261 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.261 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.261 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120124 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 579584~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:13.261 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.574014+0000", 2024-09-21T15:51:13.261 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 69.631123372999994, 2024-09-21T15:51:13.261 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055440284999999999, 2024-09-21T15:51:13.261 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:13.261 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:13.261 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:13.261 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:13.261 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:13.262 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120124 2024-09-21T15:51:13.262 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.262 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:13.262 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.262 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:13.262 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574014+0000", 2024-09-21T15:51:13.262 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:13.262 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.262 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.262 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:13.262 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574014+0000", 2024-09-21T15:51:13.262 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:13.262 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.262 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.262 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:13.262 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573515+0000", 2024-09-21T15:51:13.263 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9995008 2024-09-21T15:51:13.263 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.263 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.263 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:13.263 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574017+0000", 2024-09-21T15:51:13.263 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00050241100000000003 2024-09-21T15:51:13.263 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.263 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.263 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:13.263 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574027+0000", 2024-09-21T15:51:13.263 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4700000000000008e-06 2024-09-21T15:51:13.263 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.263 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.263 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:13.263 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574033+0000", 2024-09-21T15:51:13.264 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1489999999999999e-06 2024-09-21T15:51:13.264 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.264 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.264 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:13.264 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629238+0000", 2024-09-21T15:51:13.264 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055204880999999997 2024-09-21T15:51:13.264 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.264 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.264 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:13.264 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629317+0000", 2024-09-21T15:51:13.264 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8942000000000004e-05 2024-09-21T15:51:13.264 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.264 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.264 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:13.264 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629454+0000", 2024-09-21T15:51:13.265 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013767099999999999 2024-09-21T15:51:13.265 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:13.265 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:13.265 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:13.265 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.265 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.265 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120125 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 600064~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:13.265 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.574482+0000", 2024-09-21T15:51:13.265 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 69.630655074000003, 2024-09-21T15:51:13.265 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055243608, 2024-09-21T15:51:13.265 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:13.265 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:13.265 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:13.265 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:13.265 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:13.265 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120125 2024-09-21T15:51:13.266 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.266 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:13.266 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.266 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:13.266 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574482+0000", 2024-09-21T15:51:13.266 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:13.266 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.266 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.266 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:13.266 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574482+0000", 2024-09-21T15:51:13.266 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:13.266 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.266 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.266 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:13.267 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574038+0000", 2024-09-21T15:51:13.267 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9995556 2024-09-21T15:51:13.267 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.267 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.267 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:13.267 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574497+0000", 2024-09-21T15:51:13.267 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00045854400000000002 2024-09-21T15:51:13.267 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.267 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.267 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:13.267 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574509+0000", 2024-09-21T15:51:13.267 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2486e-05 2024-09-21T15:51:13.267 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.267 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.267 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:13.267 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574516+0000", 2024-09-21T15:51:13.268 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2269999999999996e-06 2024-09-21T15:51:13.268 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.268 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.268 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:13.268 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629497+0000", 2024-09-21T15:51:13.268 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054980785999999997 2024-09-21T15:51:13.268 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.268 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.268 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:13.268 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629580+0000", 2024-09-21T15:51:13.268 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2528000000000007e-05 2024-09-21T15:51:13.268 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.268 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.268 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:13.269 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629726+0000", 2024-09-21T15:51:13.269 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014630399999999999 2024-09-21T15:51:13.269 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:13.269 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:13.269 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:13.269 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.269 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.269 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120126 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 620544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:13.269 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.575315+0000", 2024-09-21T15:51:13.269 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 69.629822043999994, 2024-09-21T15:51:13.269 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054709559999999997, 2024-09-21T15:51:13.269 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:13.269 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:13.269 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:13.270 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:13.270 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:13.270 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120126 2024-09-21T15:51:13.270 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.270 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:13.270 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.270 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:13.270 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575315+0000", 2024-09-21T15:51:13.270 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:13.270 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.270 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.270 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:13.270 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575315+0000", 2024-09-21T15:51:13.270 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:13.271 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.271 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.271 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:13.271 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574522+0000", 2024-09-21T15:51:13.271 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9992065 2024-09-21T15:51:13.271 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.271 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.271 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:13.271 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575319+0000", 2024-09-21T15:51:13.271 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00079626800000000004 2024-09-21T15:51:13.271 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.271 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.271 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:13.271 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575329+0000", 2024-09-21T15:51:13.271 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.028e-05 2024-09-21T15:51:13.272 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.272 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.272 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:13.272 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575335+0000", 2024-09-21T15:51:13.272 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9290000000000003e-06 2024-09-21T15:51:13.272 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.272 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.272 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:13.272 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629806+0000", 2024-09-21T15:51:13.272 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054471558000000003 2024-09-21T15:51:13.272 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.272 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.272 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:13.272 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629885+0000", 2024-09-21T15:51:13.272 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8563e-05 2024-09-21T15:51:13.273 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.273 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.273 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:13.273 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630025+0000", 2024-09-21T15:51:13.273 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014019 2024-09-21T15:51:13.273 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:13.273 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:13.273 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:13.273 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.273 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.273 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120127 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 641024~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:13.273 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.575340+0000", 2024-09-21T15:51:13.273 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 69.629797336999999, 2024-09-21T15:51:13.273 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054895589000000002, 2024-09-21T15:51:13.273 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:13.274 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:13.274 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:13.274 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:13.274 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:13.274 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120127 2024-09-21T15:51:13.274 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.274 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:13.274 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.274 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:13.274 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575340+0000", 2024-09-21T15:51:13.274 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:13.274 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.274 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.274 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:13.275 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575340+0000", 2024-09-21T15:51:13.275 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:13.275 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.275 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.275 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:13.275 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575340+0000", 2024-09-21T15:51:13.275 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:13.275 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.275 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.275 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:13.275 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575342+0000", 2024-09-21T15:51:13.275 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5600000000000001e-06 2024-09-21T15:51:13.275 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.275 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.275 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:13.276 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575351+0000", 2024-09-21T15:51:13.276 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2349999999999999e-06 2024-09-21T15:51:13.276 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.276 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.276 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:13.276 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575356+0000", 2024-09-21T15:51:13.276 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4600000000000002e-06 2024-09-21T15:51:13.276 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.276 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.276 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:13.276 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630053+0000", 2024-09-21T15:51:13.276 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054697190999999999 2024-09-21T15:51:13.276 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.276 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.276 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:13.276 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630128+0000", 2024-09-21T15:51:13.277 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4777000000000006e-05 2024-09-21T15:51:13.277 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.277 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.277 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:13.277 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630236+0000", 2024-09-21T15:51:13.277 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000107837 2024-09-21T15:51:13.277 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:13.277 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:13.277 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:13.277 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.277 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.277 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120128 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 657408~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:13.277 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.575461+0000", 2024-09-21T15:51:13.277 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 69.629676758000002, 2024-09-21T15:51:13.277 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054945121999999999, 2024-09-21T15:51:13.278 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:13.278 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:13.278 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:13.278 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:13.278 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:13.278 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120128 2024-09-21T15:51:13.278 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.278 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:13.278 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.278 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:13.278 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575461+0000", 2024-09-21T15:51:13.278 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:13.278 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.278 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.279 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:13.279 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575461+0000", 2024-09-21T15:51:13.279 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:13.279 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.279 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.279 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:13.279 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575360+0000", 2024-09-21T15:51:13.279 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998994 2024-09-21T15:51:13.279 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.279 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.279 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:13.279 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575464+0000", 2024-09-21T15:51:13.279 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000103397 2024-09-21T15:51:13.279 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.279 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.280 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:13.280 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575472+0000", 2024-09-21T15:51:13.280 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3189999999999995e-06 2024-09-21T15:51:13.280 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.280 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.280 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:13.280 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575478+0000", 2024-09-21T15:51:13.280 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2779999999999999e-06 2024-09-21T15:51:13.280 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.280 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.280 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:13.280 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630268+0000", 2024-09-21T15:51:13.280 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054789545000000002 2024-09-21T15:51:13.280 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.281 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.281 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:13.281 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630315+0000", 2024-09-21T15:51:13.281 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7244999999999999e-05 2024-09-21T15:51:13.281 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.281 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.281 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:13.281 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630406+0000", 2024-09-21T15:51:13.281 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0811999999999997e-05 2024-09-21T15:51:13.281 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:13.281 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:13.281 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:13.281 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.281 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.282 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120130 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 698368~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:13.282 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.576043+0000", 2024-09-21T15:51:13.282 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 69.629094843000004, 2024-09-21T15:51:13.282 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054781798, 2024-09-21T15:51:13.282 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:13.282 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:13.282 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:13.282 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:13.282 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:13.282 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120130 2024-09-21T15:51:13.282 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.282 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:13.282 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.282 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:13.283 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576043+0000", 2024-09-21T15:51:13.283 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:13.283 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.283 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.283 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:13.283 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576043+0000", 2024-09-21T15:51:13.283 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:13.283 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.283 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.283 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:13.283 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576042+0000", 2024-09-21T15:51:13.283 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:13.283 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.283 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.284 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:13.284 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576044+0000", 2024-09-21T15:51:13.284 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0999999999999998e-06 2024-09-21T15:51:13.284 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.284 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.284 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:13.284 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576051+0000", 2024-09-21T15:51:13.284 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4779999999999996e-06 2024-09-21T15:51:13.284 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.284 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.284 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:13.284 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576058+0000", 2024-09-21T15:51:13.284 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2180000000000002e-06 2024-09-21T15:51:13.284 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.284 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.285 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:13.285 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630661+0000", 2024-09-21T15:51:13.285 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054603240999999997 2024-09-21T15:51:13.285 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.285 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.285 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:13.285 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630747+0000", 2024-09-21T15:51:13.285 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6046999999999998e-05 2024-09-21T15:51:13.285 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.285 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.285 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:13.285 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630824+0000", 2024-09-21T15:51:13.285 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7119999999999993e-05 2024-09-21T15:51:13.285 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:13.286 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:13.286 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:13.286 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.286 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.286 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:165208 3.2 3:40caac84:::rbd_id.image_client.0-clone14:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-21T15:51:13.286 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:51:06.879846+0000", 2024-09-21T15:51:13.286 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 6.32529158, 2024-09-21T15:51:13.286 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1380672409999999, 2024-09-21T15:51:13.286 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:13.286 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:13.286 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:13.286 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:13.287 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:13.287 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 165208 2024-09-21T15:51:13.287 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.287 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:13.287 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.287 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:13.287 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:51:06.879846+0000", 2024-09-21T15:51:13.287 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:13.287 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.287 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.287 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:13.287 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:51:06.879846+0000", 2024-09-21T15:51:13.287 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:13.287 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.287 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.288 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:13.288 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:51:06.879845+0000", 2024-09-21T15:51:13.288 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:13.288 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.288 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.288 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:13.288 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:51:06.879849+0000", 2024-09-21T15:51:13.288 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.968e-06 2024-09-21T15:51:13.288 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.288 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.288 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:13.288 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:51:06.879860+0000", 2024-09-21T15:51:13.288 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0368e-05 2024-09-21T15:51:13.288 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.288 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.289 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:13.289 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:51:06.879867+0000", 2024-09-21T15:51:13.289 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0700000000000001e-06 2024-09-21T15:51:13.289 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.289 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.289 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:13.289 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:51:06.879907+0000", 2024-09-21T15:51:13.289 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9904000000000003e-05 2024-09-21T15:51:13.289 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.289 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.289 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-21T15:51:13.289 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:51:06.879914+0000", 2024-09-21T15:51:13.289 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4279999999999999e-06 2024-09-21T15:51:13.289 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.289 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.290 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:13.290 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:51:07.686406+0000", 2024-09-21T15:51:13.290 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.80649178700000002 2024-09-21T15:51:13.290 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.290 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.290 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-21T15:51:13.290 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:51:07.686421+0000", 2024-09-21T15:51:13.290 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5339e-05 2024-09-21T15:51:13.290 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.290 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.290 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:13.290 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:51:09.013823+0000", 2024-09-21T15:51:13.290 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.327402341 2024-09-21T15:51:13.290 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.290 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.291 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:13.291 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:51:09.014007+0000", 2024-09-21T15:51:13.291 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000183952 2024-09-21T15:51:13.291 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.291 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.291 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 3", 2024-09-21T15:51:13.291 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:51:09.014167+0000", 2024-09-21T15:51:13.291 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000159543 2024-09-21T15:51:13.291 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.291 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.291 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:51:13.291 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:51:09.014932+0000", 2024-09-21T15:51:13.291 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00076526800000000005 2024-09-21T15:51:13.291 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.292 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.292 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:13.292 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:51:09.017803+0000", 2024-09-21T15:51:13.292 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0028705789999999998 2024-09-21T15:51:13.292 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.292 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.292 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:51:13.292 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:51:09.017838+0000", 2024-09-21T15:51:13.292 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4782e-05 2024-09-21T15:51:13.292 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:13.292 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:13.292 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:13.292 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:51:09.017913+0000", 2024-09-21T15:51:13.292 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5570999999999999e-05 2024-09-21T15:51:13.292 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:13.293 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:13.293 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:13.293 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:13.293 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:13.293 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:51:13.294 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T15:51:13.295 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:51:13.293+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:51:13.395 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:51:13.394+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:51:13.436 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:51:13.437 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:51:13.437 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:51:13.437 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:51:13.445 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T15:51:13.466 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:51:13.496 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:51:13.495+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:51:13.546 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:51:13.546 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:51:13.546 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:51:13.546 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:51:13.546 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:51:13.554 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T15:51:13.596 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-09-21T15:51:13.596 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:51:13.597 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:51:13.596+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:51:13.612 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:51:13.612 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:51:13.628 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:51:13.628 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:51:13.643 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:51:13.644 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:51:13.660 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:51:13.660 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:51:13.677 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:51:13.677 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:51:13.697 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:51:13.697 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:51:13.698 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:51:13.696+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:51:13.704 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:13.705 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:13.714 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T15:51:13.715 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:51:13.715 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:51:13.799 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:51:13.798+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:51:13.823 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:13.823 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:13.832 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T15:51:13.900 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:51:13.899+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:51:13.973 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:13.973 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:13.981 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T15:51:14.001 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:51:14.000+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:51:14.083 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:14.083 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:14.092 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:51:14.136 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:51:14.135+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:51:14.194 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:14.195 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:14.203 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:51:14.204 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:51:14.202+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:51:14.304 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:51:14.302+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:51:14.346 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:14.346 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:14.354 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:51:14.405 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:51:14.404+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:51:14.496 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:14.496 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:14.504 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:51:14.505 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:51:14.504+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:51:14.606 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:14.606 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:14.607 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:51:14.606+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:51:14.615 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:51:14.707 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:51:14.706+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:51:14.716 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:14.716 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:14.725 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:51:14.758 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T15:51:14.807 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:51:14.807+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:51:14.830 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:14.830 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:14.841 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:51:14.908 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:51:14.907+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:51:14.958 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:14.958 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:14.967 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:51:15.010 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:51:15.009+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:51:15.111 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:15.111 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:15.111 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:51:15.111+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:51:15.120 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:51:15.212 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:51:15.211+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:51:15.222 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:15.222 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:15.230 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:51:15.313 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:51:15.312+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:51:15.331 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:15.332 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:15.340 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:51:15.413 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:51:15.412+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:51:15.441 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:15.442 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:15.450 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:51:15.534 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:51:15.533+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:51:15.593 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:15.593 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:15.601 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:51:15.645 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:51:15.644+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:51:15.710 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:15.710 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:15.716 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:51:15.715+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:51:15.719 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:51:15.726 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:51:15.817 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:51:15.816+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:51:15.846 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-09-21T15:51:15.846 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:51:15.846 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is unset 2024-09-21T15:51:15.867 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:15.867 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:15.871 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:51:15.871 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:51:15.877 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:51:15.890 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:51:15.890 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:51:15.908 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:51:15.908 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:51:15.917 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:51:15.916+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:51:15.927 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:51:15.927 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:51:15.944 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:51:15.944 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:51:15.961 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:51:15.961 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:51:15.978 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:51:15.979 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:51:15.988 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:51:15.988 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:51:15.988 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:51:15.988 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:51:15.988 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:15.988 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155031 3.1c 3.8612741c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:15.988 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448748+0000", 2024-09-21T15:51:15.989 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 53.537783720999997, 2024-09-21T15:51:15.989 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064010286999999999, 2024-09-21T15:51:15.989 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:15.989 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:15.989 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:15.989 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:15.989 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:15.989 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155031 2024-09-21T15:51:15.989 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:15.989 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:15.989 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:15.989 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:15.989 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448748+0000", 2024-09-21T15:51:15.989 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:15.989 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:15.990 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:15.990 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:15.990 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448748+0000", 2024-09-21T15:51:15.990 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:15.990 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:15.990 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:15.990 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:15.990 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448747+0000", 2024-09-21T15:51:15.990 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:15.990 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:15.990 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:15.990 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:15.990 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448753+0000", 2024-09-21T15:51:15.990 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.755e-06 2024-09-21T15:51:15.990 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:15.991 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:15.991 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:15.991 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448763+0000", 2024-09-21T15:51:15.991 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7659999999999993e-06 2024-09-21T15:51:15.991 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:15.991 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:15.991 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:15.991 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448769+0000", 2024-09-21T15:51:15.991 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6060000000000002e-06 2024-09-21T15:51:15.991 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:15.991 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:15.991 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:15.991 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512750+0000", 2024-09-21T15:51:15.991 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.063981298000000006 2024-09-21T15:51:15.991 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:15.991 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:15.992 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:15.992 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512758+0000", 2024-09-21T15:51:15.992 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1669999999999999e-06 2024-09-21T15:51:15.992 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:15.992 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:15.992 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:15.992 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:15.992 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:15.992 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155091 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:15.992 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448916+0000", 2024-09-21T15:51:15.992 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 53.537615875999997, 2024-09-21T15:51:15.992 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064006309999999997, 2024-09-21T15:51:15.992 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:15.992 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:15.992 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:15.992 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:15.993 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:15.993 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155091 2024-09-21T15:51:15.993 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:15.993 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:15.993 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:15.993 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:15.993 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448916+0000", 2024-09-21T15:51:15.993 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:15.993 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:15.993 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:15.993 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:15.993 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448916+0000", 2024-09-21T15:51:15.993 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:15.993 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:15.993 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:15.993 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:15.994 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448915+0000", 2024-09-21T15:51:15.994 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:15.994 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:15.994 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:15.994 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:15.994 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448917+0000", 2024-09-21T15:51:15.994 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2400000000000002e-06 2024-09-21T15:51:15.994 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:15.994 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:15.994 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:15.994 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448923+0000", 2024-09-21T15:51:15.995 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0580000000000003e-06 2024-09-21T15:51:15.995 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:15.995 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:15.995 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:15.995 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448927+0000", 2024-09-21T15:51:15.995 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.031e-06 2024-09-21T15:51:15.995 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:15.995 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:15.995 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:15.995 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512916+0000", 2024-09-21T15:51:15.995 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.063988456999999999 2024-09-21T15:51:15.995 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:15.996 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:15.996 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:15.996 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512922+0000", 2024-09-21T15:51:15.996 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9009999999999999e-06 2024-09-21T15:51:15.996 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:15.996 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:15.996 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:15.996 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:15.996 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:15.996 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155092 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:15.996 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448931+0000", 2024-09-21T15:51:15.996 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 53.537600093999998, 2024-09-21T15:51:15.996 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064022793999999994, 2024-09-21T15:51:15.996 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:15.996 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:15.997 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:15.997 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:15.997 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:15.997 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155092 2024-09-21T15:51:15.997 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:15.997 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:15.997 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:15.997 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:15.997 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448931+0000", 2024-09-21T15:51:15.997 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:15.997 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:15.997 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:15.997 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:15.997 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448931+0000", 2024-09-21T15:51:15.997 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:15.998 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:15.998 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:15.998 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:15.998 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448931+0000", 2024-09-21T15:51:15.998 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:15.998 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:15.998 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:15.998 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:15.998 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448933+0000", 2024-09-21T15:51:15.998 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1799999999999999e-06 2024-09-21T15:51:15.998 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:15.998 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:15.998 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:15.998 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448939+0000", 2024-09-21T15:51:15.998 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.604e-06 2024-09-21T15:51:15.999 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:15.999 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:15.999 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:15.999 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448943+0000", 2024-09-21T15:51:15.999 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0740000000000003e-06 2024-09-21T15:51:15.999 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:15.999 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:15.999 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:15.999 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512946+0000", 2024-09-21T15:51:15.999 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064003612000000001 2024-09-21T15:51:15.999 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:15.999 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:15.999 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.000 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512954+0000", 2024-09-21T15:51:16.000 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6690000000000006e-06 2024-09-21T15:51:16.000 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.000 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.000 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.000 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.000 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.000 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155093 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:16.000 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448947+0000", 2024-09-21T15:51:16.000 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 53.537584719999998, 2024-09-21T15:51:16.000 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064020717000000005, 2024-09-21T15:51:16.000 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.000 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:16.000 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.001 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.001 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.001 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155093 2024-09-21T15:51:16.001 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.001 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.001 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.001 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.001 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448947+0000", 2024-09-21T15:51:16.001 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.001 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.001 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.001 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.001 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448947+0000", 2024-09-21T15:51:16.001 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.001 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.001 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.002 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.002 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448946+0000", 2024-09-21T15:51:16.002 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:16.002 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.002 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.002 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.002 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448949+0000", 2024-09-21T15:51:16.002 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3860000000000001e-06 2024-09-21T15:51:16.002 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.002 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.002 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.002 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448955+0000", 2024-09-21T15:51:16.002 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.823e-06 2024-09-21T15:51:16.002 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.002 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.003 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.003 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448959+0000", 2024-09-21T15:51:16.003 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2799999999999997e-06 2024-09-21T15:51:16.003 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.003 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.003 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.003 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512964+0000", 2024-09-21T15:51:16.003 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064004741000000004 2024-09-21T15:51:16.003 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.003 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.003 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.003 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512967+0000", 2024-09-21T15:51:16.003 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.839e-06 2024-09-21T15:51:16.003 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.003 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.003 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.004 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.004 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.004 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155094 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:16.004 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448962+0000", 2024-09-21T15:51:16.004 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 53.537569097000002, 2024-09-21T15:51:16.004 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064037378000000006, 2024-09-21T15:51:16.004 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.004 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:16.004 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.004 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.004 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.004 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155094 2024-09-21T15:51:16.004 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.004 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.004 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.004 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.005 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448962+0000", 2024-09-21T15:51:16.005 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.005 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.005 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.005 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.005 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448962+0000", 2024-09-21T15:51:16.005 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.005 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.005 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.005 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.005 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448962+0000", 2024-09-21T15:51:16.005 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:16.005 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.005 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.005 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.005 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448968+0000", 2024-09-21T15:51:16.006 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9710000000000001e-06 2024-09-21T15:51:16.006 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.006 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.006 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.006 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448974+0000", 2024-09-21T15:51:16.006 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7230000000000001e-06 2024-09-21T15:51:16.006 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.006 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.006 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.006 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448978+0000", 2024-09-21T15:51:16.006 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3749999999999996e-06 2024-09-21T15:51:16.006 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.006 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.006 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.006 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512992+0000", 2024-09-21T15:51:16.007 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064013525000000002 2024-09-21T15:51:16.007 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.007 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.007 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.007 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513000+0000", 2024-09-21T15:51:16.007 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0830000000000004e-06 2024-09-21T15:51:16.007 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.007 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.007 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.007 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.007 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.007 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155095 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:16.007 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448982+0000", 2024-09-21T15:51:16.007 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 53.537549814999998, 2024-09-21T15:51:16.007 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064041904999999996, 2024-09-21T15:51:16.007 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.008 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:16.008 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.008 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.008 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.008 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155095 2024-09-21T15:51:16.008 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.008 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.008 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.008 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.008 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448982+0000", 2024-09-21T15:51:16.008 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.008 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.008 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.008 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.008 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448982+0000", 2024-09-21T15:51:16.008 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.008 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.009 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.009 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.009 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448981+0000", 2024-09-21T15:51:16.009 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:16.009 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.009 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.009 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.009 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448983+0000", 2024-09-21T15:51:16.009 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.905e-06 2024-09-21T15:51:16.009 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.009 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.009 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.009 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448989+0000", 2024-09-21T15:51:16.009 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.536e-06 2024-09-21T15:51:16.009 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.009 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.010 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.010 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448993+0000", 2024-09-21T15:51:16.010 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2390000000000004e-06 2024-09-21T15:51:16.010 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.010 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.010 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.010 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513015+0000", 2024-09-21T15:51:16.010 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064022081999999994 2024-09-21T15:51:16.010 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.010 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.010 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.010 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513023+0000", 2024-09-21T15:51:16.010 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4500000000000004e-06 2024-09-21T15:51:16.010 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.010 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.011 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.011 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.011 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.011 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155096 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:16.011 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448996+0000", 2024-09-21T15:51:16.011 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 53.537534913999998, 2024-09-21T15:51:16.011 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064060868000000007, 2024-09-21T15:51:16.011 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.011 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:16.011 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.011 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.011 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.011 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155096 2024-09-21T15:51:16.011 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.011 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.011 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.012 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.012 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448996+0000", 2024-09-21T15:51:16.012 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.012 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.012 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.012 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.012 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448996+0000", 2024-09-21T15:51:16.012 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.012 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.012 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.012 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.012 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448996+0000", 2024-09-21T15:51:16.012 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:16.012 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.012 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.013 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.013 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448998+0000", 2024-09-21T15:51:16.013 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8780000000000001e-06 2024-09-21T15:51:16.013 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.013 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.013 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.013 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449003+0000", 2024-09-21T15:51:16.013 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4099999999999999e-06 2024-09-21T15:51:16.013 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.013 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.013 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.013 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449008+0000", 2024-09-21T15:51:16.013 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.335e-06 2024-09-21T15:51:16.013 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.013 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.013 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.014 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513049+0000", 2024-09-21T15:51:16.014 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064041059999999997 2024-09-21T15:51:16.014 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.014 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.014 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.014 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513057+0000", 2024-09-21T15:51:16.014 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5050000000000007e-06 2024-09-21T15:51:16.014 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.014 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.014 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.014 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.014 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.014 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155097 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:16.014 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449019+0000", 2024-09-21T15:51:16.015 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 53.537512493000001, 2024-09-21T15:51:16.015 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064052488000000005, 2024-09-21T15:51:16.015 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.015 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:16.015 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.015 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.015 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.015 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155097 2024-09-21T15:51:16.015 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.015 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.015 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.015 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.015 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449019+0000", 2024-09-21T15:51:16.015 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.015 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.015 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.016 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.016 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449019+0000", 2024-09-21T15:51:16.016 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.016 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.016 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.016 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.016 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449019+0000", 2024-09-21T15:51:16.016 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:16.016 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.016 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.016 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.016 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449021+0000", 2024-09-21T15:51:16.016 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.244e-06 2024-09-21T15:51:16.016 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.016 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.016 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.017 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449027+0000", 2024-09-21T15:51:16.017 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8619999999999999e-06 2024-09-21T15:51:16.017 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.017 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.017 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.017 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449031+0000", 2024-09-21T15:51:16.017 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1359999999999999e-06 2024-09-21T15:51:16.017 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.017 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.017 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.017 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513067+0000", 2024-09-21T15:51:16.017 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064036391999999998 2024-09-21T15:51:16.017 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.017 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.017 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.017 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513071+0000", 2024-09-21T15:51:16.018 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1989999999999999e-06 2024-09-21T15:51:16.018 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.018 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.018 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.018 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.018 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.018 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155098 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:16.018 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449035+0000", 2024-09-21T15:51:16.018 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 53.537496896, 2024-09-21T15:51:16.018 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064071761000000005, 2024-09-21T15:51:16.018 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.018 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:16.018 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.018 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.018 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.019 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155098 2024-09-21T15:51:16.019 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.019 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.019 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.019 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.019 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449035+0000", 2024-09-21T15:51:16.019 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.019 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.019 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.019 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.019 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449035+0000", 2024-09-21T15:51:16.019 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.019 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.019 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.019 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.019 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449034+0000", 2024-09-21T15:51:16.020 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:16.020 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.020 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.020 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.020 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449036+0000", 2024-09-21T15:51:16.020 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9989999999999998e-06 2024-09-21T15:51:16.020 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.020 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.020 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.020 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449042+0000", 2024-09-21T15:51:16.020 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6010000000000002e-06 2024-09-21T15:51:16.020 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.020 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.020 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.020 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449047+0000", 2024-09-21T15:51:16.020 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3600000000000004e-06 2024-09-21T15:51:16.021 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.021 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.021 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.021 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513098+0000", 2024-09-21T15:51:16.021 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064050544000000001 2024-09-21T15:51:16.021 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.021 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.021 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.021 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513106+0000", 2024-09-21T15:51:16.021 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5650000000000002e-06 2024-09-21T15:51:16.021 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.021 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.021 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.021 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.021 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.021 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155099 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:16.022 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449051+0000", 2024-09-21T15:51:16.022 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 53.537480355, 2024-09-21T15:51:16.022 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064069388000000005, 2024-09-21T15:51:16.022 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.022 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:16.022 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.022 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.022 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.022 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155099 2024-09-21T15:51:16.022 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.022 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.022 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.022 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.022 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449051+0000", 2024-09-21T15:51:16.023 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.023 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.023 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.023 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.023 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449051+0000", 2024-09-21T15:51:16.023 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.023 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.023 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.023 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.023 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449051+0000", 2024-09-21T15:51:16.023 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:16.023 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.023 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.023 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.023 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449053+0000", 2024-09-21T15:51:16.024 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2620000000000001e-06 2024-09-21T15:51:16.024 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.024 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.024 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.024 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449058+0000", 2024-09-21T15:51:16.024 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1680000000000003e-06 2024-09-21T15:51:16.024 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.024 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.024 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.024 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449062+0000", 2024-09-21T15:51:16.024 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9979999999999997e-06 2024-09-21T15:51:16.024 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.024 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.024 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.024 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513116+0000", 2024-09-21T15:51:16.025 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064053761000000001 2024-09-21T15:51:16.025 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.025 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.025 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.025 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513120+0000", 2024-09-21T15:51:16.025 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5399999999999997e-06 2024-09-21T15:51:16.025 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.025 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.025 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.025 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.025 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.025 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155100 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:16.025 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449101+0000", 2024-09-21T15:51:16.025 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 53.5374306, 2024-09-21T15:51:16.025 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064079585999999994, 2024-09-21T15:51:16.026 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.026 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:16.026 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.026 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.026 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.026 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155100 2024-09-21T15:51:16.026 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.026 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.026 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.026 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.026 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449101+0000", 2024-09-21T15:51:16.026 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.026 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.026 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.027 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.027 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449101+0000", 2024-09-21T15:51:16.027 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.027 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.027 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.027 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.027 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449100+0000", 2024-09-21T15:51:16.027 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-21T15:51:16.027 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.027 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.027 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.027 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449105+0000", 2024-09-21T15:51:16.027 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6700000000000002e-06 2024-09-21T15:51:16.028 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.028 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.028 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.028 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449117+0000", 2024-09-21T15:51:16.028 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1841000000000001e-05 2024-09-21T15:51:16.028 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.028 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.028 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.028 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449124+0000", 2024-09-21T15:51:16.028 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4660000000000002e-06 2024-09-21T15:51:16.028 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.028 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.028 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.028 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513163+0000", 2024-09-21T15:51:16.029 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064039053999999998 2024-09-21T15:51:16.029 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.029 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.029 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.029 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513180+0000", 2024-09-21T15:51:16.029 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7333000000000002e-05 2024-09-21T15:51:16.029 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.029 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.029 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.029 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.029 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.029 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155101 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:16.029 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449166+0000", 2024-09-21T15:51:16.030 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 53.537365895000001, 2024-09-21T15:51:16.030 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064027639999999997, 2024-09-21T15:51:16.030 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.030 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:16.030 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.030 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.030 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.030 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155101 2024-09-21T15:51:16.030 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.030 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.030 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.030 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.031 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449166+0000", 2024-09-21T15:51:16.031 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.031 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.031 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.031 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.031 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449166+0000", 2024-09-21T15:51:16.031 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.031 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.031 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.031 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.031 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449165+0000", 2024-09-21T15:51:16.031 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:16.031 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.031 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.032 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.032 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449168+0000", 2024-09-21T15:51:16.032 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5099999999999999e-06 2024-09-21T15:51:16.032 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.032 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.032 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.032 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449177+0000", 2024-09-21T15:51:16.032 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6659999999999997e-06 2024-09-21T15:51:16.032 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.032 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.032 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.032 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449182+0000", 2024-09-21T15:51:16.032 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1320000000000002e-06 2024-09-21T15:51:16.033 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.033 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.033 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.033 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513189+0000", 2024-09-21T15:51:16.033 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064007231999999997 2024-09-21T15:51:16.033 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.033 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.033 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.033 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513193+0000", 2024-09-21T15:51:16.033 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.692e-06 2024-09-21T15:51:16.033 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.033 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.033 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.033 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.034 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.034 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155134 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:16.034 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449187+0000", 2024-09-21T15:51:16.034 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 53.537344306999998, 2024-09-21T15:51:16.034 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064031331999999996, 2024-09-21T15:51:16.034 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.034 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:16.034 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.034 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.034 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.034 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155134 2024-09-21T15:51:16.034 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.034 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.035 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.035 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.035 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449187+0000", 2024-09-21T15:51:16.035 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.035 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.035 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.035 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.035 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449187+0000", 2024-09-21T15:51:16.035 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.035 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.035 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.035 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.035 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449187+0000", 2024-09-21T15:51:16.036 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:16.036 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.036 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.036 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.036 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449189+0000", 2024-09-21T15:51:16.036 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5469999999999999e-06 2024-09-21T15:51:16.036 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.036 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.036 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.036 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449196+0000", 2024-09-21T15:51:16.036 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7009999999999999e-06 2024-09-21T15:51:16.036 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.036 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.036 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.036 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449202+0000", 2024-09-21T15:51:16.037 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7409999999999998e-06 2024-09-21T15:51:16.037 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.037 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.037 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.037 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513214+0000", 2024-09-21T15:51:16.037 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064012301999999993 2024-09-21T15:51:16.037 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.037 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.037 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.037 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513218+0000", 2024-09-21T15:51:16.037 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5709999999999999e-06 2024-09-21T15:51:16.037 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.037 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.037 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.038 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.038 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.038 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155135 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:16.038 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449207+0000", 2024-09-21T15:51:16.038 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 53.537324796999997, 2024-09-21T15:51:16.038 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064029249999999996, 2024-09-21T15:51:16.038 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.038 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:16.038 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.038 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.038 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.038 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155135 2024-09-21T15:51:16.038 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.038 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.038 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.039 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.039 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449207+0000", 2024-09-21T15:51:16.039 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.039 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.039 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.039 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.039 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449207+0000", 2024-09-21T15:51:16.039 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.039 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.039 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.039 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.039 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449206+0000", 2024-09-21T15:51:16.039 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:16.039 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.040 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.040 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.040 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449209+0000", 2024-09-21T15:51:16.040 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5150000000000001e-06 2024-09-21T15:51:16.040 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.040 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.040 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.040 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449215+0000", 2024-09-21T15:51:16.040 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9329999999999998e-06 2024-09-21T15:51:16.040 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.040 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.040 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.040 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449220+0000", 2024-09-21T15:51:16.040 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0819999999999998e-06 2024-09-21T15:51:16.041 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.041 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.041 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.041 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513232+0000", 2024-09-21T15:51:16.041 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064012052 2024-09-21T15:51:16.041 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.041 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.041 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.041 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513236+0000", 2024-09-21T15:51:16.041 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2080000000000002e-06 2024-09-21T15:51:16.041 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.041 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.041 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.041 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.042 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.042 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155136 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:16.042 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449224+0000", 2024-09-21T15:51:16.042 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 53.537307233, 2024-09-21T15:51:16.042 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064053025999999999, 2024-09-21T15:51:16.042 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.042 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:16.042 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.042 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.042 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.042 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155136 2024-09-21T15:51:16.042 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.042 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.043 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.043 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.043 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449224+0000", 2024-09-21T15:51:16.043 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.043 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.043 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.043 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.043 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449224+0000", 2024-09-21T15:51:16.043 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.043 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.043 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.043 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.043 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449224+0000", 2024-09-21T15:51:16.043 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:16.043 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.044 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.044 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.044 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449226+0000", 2024-09-21T15:51:16.044 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2699999999999999e-06 2024-09-21T15:51:16.044 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.044 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.044 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.044 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449233+0000", 2024-09-21T15:51:16.044 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4289999999999998e-06 2024-09-21T15:51:16.044 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.044 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.044 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.044 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449238+0000", 2024-09-21T15:51:16.045 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0730000000000004e-06 2024-09-21T15:51:16.045 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.045 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.045 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.045 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513269+0000", 2024-09-21T15:51:16.045 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064030999000000005 2024-09-21T15:51:16.045 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.045 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.045 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.045 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513277+0000", 2024-09-21T15:51:16.045 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5520000000000005e-06 2024-09-21T15:51:16.045 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.045 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.045 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.046 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.046 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.046 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155137 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:16.046 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449263+0000", 2024-09-21T15:51:16.046 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 53.537268101000002, 2024-09-21T15:51:16.046 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.06402803, 2024-09-21T15:51:16.046 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.046 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:16.046 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.046 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.046 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.046 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155137 2024-09-21T15:51:16.046 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.046 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.047 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.047 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.047 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449263+0000", 2024-09-21T15:51:16.047 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.047 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.047 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.047 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.047 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449263+0000", 2024-09-21T15:51:16.047 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.047 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.047 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.047 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.047 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449262+0000", 2024-09-21T15:51:16.047 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-21T15:51:16.047 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.048 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.048 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.048 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449267+0000", 2024-09-21T15:51:16.048 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.933e-06 2024-09-21T15:51:16.048 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.048 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.048 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.048 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449279+0000", 2024-09-21T15:51:16.048 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.181e-05 2024-09-21T15:51:16.048 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.048 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.048 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.048 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449286+0000", 2024-09-21T15:51:16.048 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4749999999999998e-06 2024-09-21T15:51:16.048 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.049 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.049 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.049 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513287+0000", 2024-09-21T15:51:16.049 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064001369000000002 2024-09-21T15:51:16.049 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.049 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.049 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.049 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513291+0000", 2024-09-21T15:51:16.049 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3000000000000003e-06 2024-09-21T15:51:16.049 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.049 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.049 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.049 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.049 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.049 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155138 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:16.050 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449299+0000", 2024-09-21T15:51:16.050 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 53.537232609, 2024-09-21T15:51:16.050 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064028359000000007, 2024-09-21T15:51:16.050 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.050 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:16.050 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.050 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.050 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.050 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155138 2024-09-21T15:51:16.050 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.050 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.050 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.050 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.050 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449299+0000", 2024-09-21T15:51:16.050 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.051 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.051 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.051 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.051 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449299+0000", 2024-09-21T15:51:16.051 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.051 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.051 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.051 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.051 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449298+0000", 2024-09-21T15:51:16.051 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:16.051 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.051 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.051 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.051 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449301+0000", 2024-09-21T15:51:16.051 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0230000000000001e-06 2024-09-21T15:51:16.052 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.052 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.052 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.052 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449308+0000", 2024-09-21T15:51:16.052 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3119999999999999e-06 2024-09-21T15:51:16.052 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.052 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.052 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.052 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449313+0000", 2024-09-21T15:51:16.052 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8980000000000004e-06 2024-09-21T15:51:16.052 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.052 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.052 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.052 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513318+0000", 2024-09-21T15:51:16.052 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064005612000000003 2024-09-21T15:51:16.052 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.053 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.053 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.053 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513327+0000", 2024-09-21T15:51:16.053 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0100000000000001e-06 2024-09-21T15:51:16.053 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.053 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.053 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.053 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.053 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.053 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155142 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:16.053 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449392+0000", 2024-09-21T15:51:16.053 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 53.537139850999999, 2024-09-21T15:51:16.053 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064012464000000005, 2024-09-21T15:51:16.053 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.053 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:16.054 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.054 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.054 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.054 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155142 2024-09-21T15:51:16.054 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.054 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.054 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.054 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.054 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449392+0000", 2024-09-21T15:51:16.054 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.054 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.054 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.054 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.054 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449392+0000", 2024-09-21T15:51:16.055 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.055 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.055 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.055 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.055 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449391+0000", 2024-09-21T15:51:16.055 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:16.055 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.055 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.055 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.055 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449393+0000", 2024-09-21T15:51:16.055 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.26e-06 2024-09-21T15:51:16.055 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.055 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.055 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.056 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449399+0000", 2024-09-21T15:51:16.056 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6269999999999997e-06 2024-09-21T15:51:16.056 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.056 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.056 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.056 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449403+0000", 2024-09-21T15:51:16.056 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3000000000000003e-06 2024-09-21T15:51:16.056 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.056 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.056 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.056 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513396+0000", 2024-09-21T15:51:16.056 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.063992568999999999 2024-09-21T15:51:16.056 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.056 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.056 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.057 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513404+0000", 2024-09-21T15:51:16.057 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1100000000000003e-06 2024-09-21T15:51:16.057 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.057 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.057 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.057 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.057 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.057 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155143 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:16.057 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449407+0000", 2024-09-21T15:51:16.057 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 53.537124265000003, 2024-09-21T15:51:16.057 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064009916, 2024-09-21T15:51:16.057 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.057 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:16.057 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.057 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.057 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.058 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155143 2024-09-21T15:51:16.058 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.058 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.058 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.058 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.058 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449407+0000", 2024-09-21T15:51:16.058 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.058 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.058 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.058 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.058 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449407+0000", 2024-09-21T15:51:16.058 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.058 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.058 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.058 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.059 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449407+0000", 2024-09-21T15:51:16.059 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:16.059 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.059 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.059 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.059 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449409+0000", 2024-09-21T15:51:16.059 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1090000000000001e-06 2024-09-21T15:51:16.059 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.059 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.059 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.059 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449413+0000", 2024-09-21T15:51:16.059 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6650000000000001e-06 2024-09-21T15:51:16.059 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.059 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.059 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.059 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449415+0000", 2024-09-21T15:51:16.060 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5459999999999998e-06 2024-09-21T15:51:16.060 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.060 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.060 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.060 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513413+0000", 2024-09-21T15:51:16.060 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.063998037999999993 2024-09-21T15:51:16.060 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.060 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.060 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.060 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513417+0000", 2024-09-21T15:51:16.060 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8789999999999996e-06 2024-09-21T15:51:16.060 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.060 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.060 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.060 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.060 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.061 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155144 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:16.061 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449427+0000", 2024-09-21T15:51:16.061 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 53.537104257000003, 2024-09-21T15:51:16.061 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064022531999999993, 2024-09-21T15:51:16.061 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.061 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:16.061 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.061 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.061 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.061 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155144 2024-09-21T15:51:16.061 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.061 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.061 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.061 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.062 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449427+0000", 2024-09-21T15:51:16.062 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.062 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.062 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.062 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.062 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449427+0000", 2024-09-21T15:51:16.062 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.062 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.062 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.062 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.062 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449427+0000", 2024-09-21T15:51:16.062 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:16.062 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.062 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.063 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.063 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449430+0000", 2024-09-21T15:51:16.063 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3579999999999999e-06 2024-09-21T15:51:16.063 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.063 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.063 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.063 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449437+0000", 2024-09-21T15:51:16.063 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4580000000000004e-06 2024-09-21T15:51:16.063 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.063 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.063 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.063 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449443+0000", 2024-09-21T15:51:16.063 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5609999999999998e-06 2024-09-21T15:51:16.063 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.063 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.064 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.064 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513441+0000", 2024-09-21T15:51:16.064 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.063998136999999997 2024-09-21T15:51:16.064 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.064 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.064 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.064 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513450+0000", 2024-09-21T15:51:16.064 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4950000000000008e-06 2024-09-21T15:51:16.064 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.064 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.064 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.064 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.064 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.064 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:51:16.066 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:51:16.064+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:51:16.066 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:51:16.166 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:51:16.165+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:51:16.169 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:51:16.169 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:51:16.169 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:51:16.169 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:51:16.177 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:51:16.267 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:51:16.266+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:51:16.321 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:51:16.321 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:51:16.321 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:51:16.321 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:51:16.321 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:51:16.329 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:51:16.368 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:51:16.366+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:51:16.432 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:51:16.432 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:51:16.432 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:51:16.432 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:51:16.432 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.433 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120111 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 321536~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:16.433 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572216+0000", 2024-09-21T15:51:16.433 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 72.858240300000006, 2024-09-21T15:51:16.433 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054657399000000002, 2024-09-21T15:51:16.433 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.433 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:16.433 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.433 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.433 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.433 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120111 2024-09-21T15:51:16.433 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.433 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.433 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.433 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.434 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572216+0000", 2024-09-21T15:51:16.434 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.434 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.434 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.434 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.434 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572216+0000", 2024-09-21T15:51:16.434 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.434 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.434 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.434 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.434 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572216+0000", 2024-09-21T15:51:16.434 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:16.434 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.434 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.434 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.435 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572218+0000", 2024-09-21T15:51:16.435 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1840000000000002e-06 2024-09-21T15:51:16.435 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.435 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.435 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.435 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572224+0000", 2024-09-21T15:51:16.435 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7880000000000003e-06 2024-09-21T15:51:16.435 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.435 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.435 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.435 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572229+0000", 2024-09-21T15:51:16.435 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8239999999999999e-06 2024-09-21T15:51:16.435 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.435 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.436 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.436 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626680+0000", 2024-09-21T15:51:16.436 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054451431000000002 2024-09-21T15:51:16.436 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.436 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.436 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:16.436 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626755+0000", 2024-09-21T15:51:16.436 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4641000000000006e-05 2024-09-21T15:51:16.436 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.436 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.436 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.436 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626874+0000", 2024-09-21T15:51:16.436 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00011886 2024-09-21T15:51:16.436 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.436 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.437 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.437 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.437 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.437 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120112 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 342016~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:16.437 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572233+0000", 2024-09-21T15:51:16.437 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 72.858223932000001, 2024-09-21T15:51:16.437 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054823997999999999, 2024-09-21T15:51:16.437 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.437 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:16.437 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.437 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.437 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.437 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120112 2024-09-21T15:51:16.437 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.437 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.438 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.438 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.438 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572233+0000", 2024-09-21T15:51:16.438 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.438 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.438 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.438 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.438 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572233+0000", 2024-09-21T15:51:16.438 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.438 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.438 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.438 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.438 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572232+0000", 2024-09-21T15:51:16.438 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:16.439 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.439 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.439 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.439 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572234+0000", 2024-09-21T15:51:16.439 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0660000000000002e-06 2024-09-21T15:51:16.439 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.439 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.439 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.439 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572242+0000", 2024-09-21T15:51:16.439 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0430000000000002e-06 2024-09-21T15:51:16.439 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.439 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.439 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.439 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572246+0000", 2024-09-21T15:51:16.439 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9080000000000003e-06 2024-09-21T15:51:16.440 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.440 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.440 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.440 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626916+0000", 2024-09-21T15:51:16.440 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054669482999999998 2024-09-21T15:51:16.440 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.440 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.440 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:16.440 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626977+0000", 2024-09-21T15:51:16.440 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0597000000000003e-05 2024-09-21T15:51:16.440 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.440 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.440 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.440 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627057+0000", 2024-09-21T15:51:16.440 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0258000000000003e-05 2024-09-21T15:51:16.440 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.441 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.441 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.441 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.441 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.441 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120113 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 362496~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:16.441 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572253+0000", 2024-09-21T15:51:16.441 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 72.858203947000007, 2024-09-21T15:51:16.441 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.05503044, 2024-09-21T15:51:16.441 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.441 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:16.441 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.441 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.441 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.441 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120113 2024-09-21T15:51:16.441 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.442 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.442 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.442 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.442 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572253+0000", 2024-09-21T15:51:16.442 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.442 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.442 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.442 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.442 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572253+0000", 2024-09-21T15:51:16.442 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.442 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.442 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.442 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.442 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572252+0000", 2024-09-21T15:51:16.443 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:16.443 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.443 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.443 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.443 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572258+0000", 2024-09-21T15:51:16.443 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6169999999999998e-06 2024-09-21T15:51:16.443 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.443 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.443 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.443 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572266+0000", 2024-09-21T15:51:16.443 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9410000000000007e-06 2024-09-21T15:51:16.443 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.443 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.443 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.444 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572272+0000", 2024-09-21T15:51:16.444 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.241e-06 2024-09-21T15:51:16.444 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.444 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.444 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.444 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627087+0000", 2024-09-21T15:51:16.444 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054814850999999998 2024-09-21T15:51:16.444 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.444 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.444 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:16.444 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627162+0000", 2024-09-21T15:51:16.444 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5012999999999995e-05 2024-09-21T15:51:16.444 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.445 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.445 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.445 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627283+0000", 2024-09-21T15:51:16.445 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012113 2024-09-21T15:51:16.445 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.445 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.445 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.445 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.445 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.445 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120114 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 382976~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:16.445 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572277+0000", 2024-09-21T15:51:16.445 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 72.858179929000002, 2024-09-21T15:51:16.445 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055184395999999997, 2024-09-21T15:51:16.446 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.446 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:16.446 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.446 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.446 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.446 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120114 2024-09-21T15:51:16.446 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.446 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.446 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.446 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.446 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572277+0000", 2024-09-21T15:51:16.446 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.446 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.447 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.447 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.447 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572277+0000", 2024-09-21T15:51:16.447 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.447 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.447 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.447 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.447 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572276+0000", 2024-09-21T15:51:16.447 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:16.447 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.448 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.448 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.448 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572279+0000", 2024-09-21T15:51:16.448 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.373e-06 2024-09-21T15:51:16.448 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.448 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.448 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.448 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572285+0000", 2024-09-21T15:51:16.448 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9359999999999996e-06 2024-09-21T15:51:16.448 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.448 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.448 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.448 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572289+0000", 2024-09-21T15:51:16.448 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.634e-06 2024-09-21T15:51:16.449 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.449 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.449 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.449 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627311+0000", 2024-09-21T15:51:16.449 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055021211 2024-09-21T15:51:16.449 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.449 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.449 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:16.449 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627360+0000", 2024-09-21T15:51:16.449 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9203999999999998e-05 2024-09-21T15:51:16.449 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.450 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.450 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.450 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627461+0000", 2024-09-21T15:51:16.450 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000101469 2024-09-21T15:51:16.450 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.450 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.450 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.450 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.450 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.450 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120115 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 399360~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:16.450 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572294+0000", 2024-09-21T15:51:16.450 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 72.858162621999995, 2024-09-21T15:51:16.450 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055400092999999997, 2024-09-21T15:51:16.450 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.451 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:16.451 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.451 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.451 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.451 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120115 2024-09-21T15:51:16.451 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.451 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.451 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.451 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.451 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572294+0000", 2024-09-21T15:51:16.451 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.451 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.451 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.451 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.452 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572294+0000", 2024-09-21T15:51:16.452 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.452 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.452 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.452 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.452 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572294+0000", 2024-09-21T15:51:16.452 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:16.452 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.452 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.452 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.452 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572296+0000", 2024-09-21T15:51:16.452 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1100000000000001e-06 2024-09-21T15:51:16.452 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.452 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.453 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.453 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572303+0000", 2024-09-21T15:51:16.453 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8580000000000002e-06 2024-09-21T15:51:16.453 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.453 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.453 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.453 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572314+0000", 2024-09-21T15:51:16.453 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.101e-05 2024-09-21T15:51:16.453 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.453 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.453 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.453 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627496+0000", 2024-09-21T15:51:16.453 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055182522999999997 2024-09-21T15:51:16.453 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.453 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.454 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:16.454 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627568+0000", 2024-09-21T15:51:16.454 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1569999999999994e-05 2024-09-21T15:51:16.454 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.454 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.454 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.454 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627694+0000", 2024-09-21T15:51:16.454 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000126372 2024-09-21T15:51:16.454 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.454 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.454 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.454 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.454 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.454 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120116 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 419840~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:16.455 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572318+0000", 2024-09-21T15:51:16.455 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 72.858138431, 2024-09-21T15:51:16.455 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055539416000000001, 2024-09-21T15:51:16.455 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.455 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:16.455 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.455 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.455 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.455 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120116 2024-09-21T15:51:16.455 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.455 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.455 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.455 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.455 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572318+0000", 2024-09-21T15:51:16.455 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.455 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.456 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.456 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.456 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572318+0000", 2024-09-21T15:51:16.456 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.456 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.456 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.456 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.456 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572318+0000", 2024-09-21T15:51:16.456 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:16.456 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.456 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.456 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.456 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572322+0000", 2024-09-21T15:51:16.456 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4839999999999998e-06 2024-09-21T15:51:16.456 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.456 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.457 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.457 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572329+0000", 2024-09-21T15:51:16.457 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0959999999999997e-06 2024-09-21T15:51:16.457 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.457 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.457 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.457 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572333+0000", 2024-09-21T15:51:16.457 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9359999999999998e-06 2024-09-21T15:51:16.457 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.457 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.457 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.457 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627726+0000", 2024-09-21T15:51:16.457 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055392829999999997 2024-09-21T15:51:16.457 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.457 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.457 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:16.457 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627779+0000", 2024-09-21T15:51:16.458 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2185999999999998e-05 2024-09-21T15:51:16.458 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.458 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.458 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.458 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627858+0000", 2024-09-21T15:51:16.458 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9190999999999994e-05 2024-09-21T15:51:16.458 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.458 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.458 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.458 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.458 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.458 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120117 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 440320~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:16.458 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572338+0000", 2024-09-21T15:51:16.458 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 72.858119118000005, 2024-09-21T15:51:16.458 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055680888999999997, 2024-09-21T15:51:16.459 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.459 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:16.459 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.459 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.459 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.459 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120117 2024-09-21T15:51:16.459 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.459 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.459 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.459 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.459 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572338+0000", 2024-09-21T15:51:16.459 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.459 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.459 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.459 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.460 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572338+0000", 2024-09-21T15:51:16.460 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.460 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.460 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.460 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.460 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572337+0000", 2024-09-21T15:51:16.460 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:16.460 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.460 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.460 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.460 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572339+0000", 2024-09-21T15:51:16.460 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9379999999999999e-06 2024-09-21T15:51:16.460 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.460 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.460 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.460 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572349+0000", 2024-09-21T15:51:16.461 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7780000000000002e-06 2024-09-21T15:51:16.461 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.461 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.461 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.461 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572354+0000", 2024-09-21T15:51:16.461 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8740000000000003e-06 2024-09-21T15:51:16.461 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.461 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.461 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.461 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627885+0000", 2024-09-21T15:51:16.461 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055531448999999997 2024-09-21T15:51:16.461 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.461 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.461 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:16.461 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627935+0000", 2024-09-21T15:51:16.461 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9348999999999998e-05 2024-09-21T15:51:16.462 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.462 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.462 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.462 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628018+0000", 2024-09-21T15:51:16.462 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3851999999999996e-05 2024-09-21T15:51:16.462 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.462 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.462 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.462 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.462 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.462 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120118 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 460800~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:16.462 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572359+0000", 2024-09-21T15:51:16.462 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 72.858097876000002, 2024-09-21T15:51:16.463 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055951714, 2024-09-21T15:51:16.463 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.463 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:16.463 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.463 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.463 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.463 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120118 2024-09-21T15:51:16.463 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.463 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.463 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.463 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.463 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572359+0000", 2024-09-21T15:51:16.463 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.463 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.464 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.464 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.464 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572359+0000", 2024-09-21T15:51:16.464 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.464 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.464 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.464 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.464 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572358+0000", 2024-09-21T15:51:16.464 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:16.464 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.464 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.464 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.464 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572361+0000", 2024-09-21T15:51:16.464 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.26e-06 2024-09-21T15:51:16.465 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.465 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.465 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.465 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572367+0000", 2024-09-21T15:51:16.465 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3640000000000004e-06 2024-09-21T15:51:16.465 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.465 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.465 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.465 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572376+0000", 2024-09-21T15:51:16.465 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2310000000000002e-06 2024-09-21T15:51:16.465 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.465 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.465 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.465 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628096+0000", 2024-09-21T15:51:16.466 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055719369999999997 2024-09-21T15:51:16.466 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.466 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.466 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:16.466 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628178+0000", 2024-09-21T15:51:16.466 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2667e-05 2024-09-21T15:51:16.466 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.466 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.466 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.466 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628311+0000", 2024-09-21T15:51:16.466 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000132175 2024-09-21T15:51:16.466 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.466 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.467 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.467 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.467 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.467 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120119 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 481280~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:16.467 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572381+0000", 2024-09-21T15:51:16.467 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 72.858075432000007, 2024-09-21T15:51:16.467 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056088791999999998, 2024-09-21T15:51:16.467 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.467 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:16.467 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.467 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.467 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.467 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120119 2024-09-21T15:51:16.467 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.468 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.468 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.468 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.468 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572381+0000", 2024-09-21T15:51:16.468 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.468 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.468 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.468 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.468 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572381+0000", 2024-09-21T15:51:16.468 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.468 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.468 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.468 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.468 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572381+0000", 2024-09-21T15:51:16.468 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:16.469 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.469 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.469 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.469 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572383+0000", 2024-09-21T15:51:16.469 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0619999999999999e-06 2024-09-21T15:51:16.469 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.469 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.469 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.469 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572391+0000", 2024-09-21T15:51:16.469 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8769999999999993e-06 2024-09-21T15:51:16.469 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.469 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.469 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.469 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572399+0000", 2024-09-21T15:51:16.469 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6240000000000008e-06 2024-09-21T15:51:16.469 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.470 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.470 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.470 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628344+0000", 2024-09-21T15:51:16.470 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055944828000000002 2024-09-21T15:51:16.470 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.470 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.470 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:16.470 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628393+0000", 2024-09-21T15:51:16.470 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8653000000000003e-05 2024-09-21T15:51:16.470 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.470 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.470 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.470 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628470+0000", 2024-09-21T15:51:16.470 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7121999999999998e-05 2024-09-21T15:51:16.470 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.471 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.471 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.471 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.471 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.471 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120120 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 501760~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:16.471 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572418+0000", 2024-09-21T15:51:16.471 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 72.858038934000007, 2024-09-21T15:51:16.471 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056209138999999998, 2024-09-21T15:51:16.471 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.471 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:16.471 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.471 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.471 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.471 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120120 2024-09-21T15:51:16.471 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.471 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.472 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.472 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.472 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572418+0000", 2024-09-21T15:51:16.472 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.472 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.472 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.472 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.472 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572418+0000", 2024-09-21T15:51:16.472 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.472 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.472 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.472 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.472 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572417+0000", 2024-09-21T15:51:16.472 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:16.472 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.472 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.473 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.473 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572420+0000", 2024-09-21T15:51:16.473 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5270000000000001e-06 2024-09-21T15:51:16.473 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.473 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.473 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.473 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572426+0000", 2024-09-21T15:51:16.473 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3380000000000001e-06 2024-09-21T15:51:16.473 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.473 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.473 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.473 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572431+0000", 2024-09-21T15:51:16.473 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.865e-06 2024-09-21T15:51:16.473 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.473 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.474 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.474 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628500+0000", 2024-09-21T15:51:16.474 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056068917000000003 2024-09-21T15:51:16.474 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.474 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.474 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:16.474 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628542+0000", 2024-09-21T15:51:16.474 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1761999999999998e-05 2024-09-21T15:51:16.474 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.474 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.474 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.474 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628627+0000", 2024-09-21T15:51:16.474 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5247999999999994e-05 2024-09-21T15:51:16.474 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.474 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.475 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.475 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.475 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.475 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120121 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 522240~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:16.475 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572630+0000", 2024-09-21T15:51:16.475 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 72.857826548000006, 2024-09-21T15:51:16.475 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056207209000000001, 2024-09-21T15:51:16.475 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.475 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:16.475 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.475 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.475 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.475 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120121 2024-09-21T15:51:16.475 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.475 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.476 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.476 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.476 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572630+0000", 2024-09-21T15:51:16.476 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.476 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.476 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.476 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.476 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572630+0000", 2024-09-21T15:51:16.476 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.476 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.476 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.476 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.476 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572435+0000", 2024-09-21T15:51:16.476 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998045 2024-09-21T15:51:16.476 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.477 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.477 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.477 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572633+0000", 2024-09-21T15:51:16.477 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000198786 2024-09-21T15:51:16.477 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.477 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.477 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.477 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572641+0000", 2024-09-21T15:51:16.477 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9449999999999993e-06 2024-09-21T15:51:16.477 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.477 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.477 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.477 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572648+0000", 2024-09-21T15:51:16.477 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1929999999999998e-06 2024-09-21T15:51:16.478 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.478 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.478 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.478 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628669+0000", 2024-09-21T15:51:16.478 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056021690999999998 2024-09-21T15:51:16.478 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.478 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.478 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:16.478 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628751+0000", 2024-09-21T15:51:16.478 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1710000000000002e-05 2024-09-21T15:51:16.478 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.478 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.478 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.478 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628837+0000", 2024-09-21T15:51:16.479 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6383000000000003e-05 2024-09-21T15:51:16.479 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.479 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.479 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.479 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.479 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.479 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120122 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 538624~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:16.479 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.573471+0000", 2024-09-21T15:51:16.479 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 72.856985303000002, 2024-09-21T15:51:16.479 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055563935000000002, 2024-09-21T15:51:16.479 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.479 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:16.479 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.479 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.480 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.480 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120122 2024-09-21T15:51:16.480 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.480 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.480 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.480 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.480 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573471+0000", 2024-09-21T15:51:16.480 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.480 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.480 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.480 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.480 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573471+0000", 2024-09-21T15:51:16.480 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.480 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.481 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.481 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.481 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572653+0000", 2024-09-21T15:51:16.481 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9991813 2024-09-21T15:51:16.481 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.481 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.481 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.481 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573475+0000", 2024-09-21T15:51:16.481 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00082196500000000002 2024-09-21T15:51:16.481 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.481 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.481 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.481 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573483+0000", 2024-09-21T15:51:16.481 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5939999999999994e-06 2024-09-21T15:51:16.482 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.482 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.482 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.482 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573489+0000", 2024-09-21T15:51:16.482 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1689999999999997e-06 2024-09-21T15:51:16.482 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.482 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.482 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.482 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628882+0000", 2024-09-21T15:51:16.482 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055392230000000001 2024-09-21T15:51:16.482 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.482 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.482 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:16.482 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628934+0000", 2024-09-21T15:51:16.483 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1870000000000003e-05 2024-09-21T15:51:16.483 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.483 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.483 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.483 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629035+0000", 2024-09-21T15:51:16.483 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000101753 2024-09-21T15:51:16.483 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.483 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.483 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.483 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.483 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.483 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120123 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 559104~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:16.483 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.573495+0000", 2024-09-21T15:51:16.484 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 72.856961604000006, 2024-09-21T15:51:16.484 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055704765000000003, 2024-09-21T15:51:16.484 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.484 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:16.484 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.484 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.484 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.484 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120123 2024-09-21T15:51:16.484 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.484 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.484 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.484 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.484 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573495+0000", 2024-09-21T15:51:16.484 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.484 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.485 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.485 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.485 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573495+0000", 2024-09-21T15:51:16.485 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.485 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.485 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.485 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.485 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573495+0000", 2024-09-21T15:51:16.485 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:16.485 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.485 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.485 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.485 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573497+0000", 2024-09-21T15:51:16.485 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3420000000000002e-06 2024-09-21T15:51:16.485 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.485 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.485 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.486 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573503+0000", 2024-09-21T15:51:16.486 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7030000000000003e-06 2024-09-21T15:51:16.486 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.486 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.486 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.486 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573511+0000", 2024-09-21T15:51:16.486 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0790000000000001e-06 2024-09-21T15:51:16.486 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.486 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.486 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.486 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629067+0000", 2024-09-21T15:51:16.486 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055556573999999997 2024-09-21T15:51:16.486 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.486 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.487 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:16.487 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629118+0000", 2024-09-21T15:51:16.487 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1053000000000001e-05 2024-09-21T15:51:16.487 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.487 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.487 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.487 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629200+0000", 2024-09-21T15:51:16.487 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1452999999999994e-05 2024-09-21T15:51:16.487 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.487 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.487 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.487 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.487 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.487 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120124 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 579584~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:16.488 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.574014+0000", 2024-09-21T15:51:16.488 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 72.856442564999995, 2024-09-21T15:51:16.488 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055440284999999999, 2024-09-21T15:51:16.488 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.488 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:16.488 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.488 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.488 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.488 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120124 2024-09-21T15:51:16.488 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.488 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.488 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.488 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.488 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574014+0000", 2024-09-21T15:51:16.488 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.488 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.489 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.489 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.489 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574014+0000", 2024-09-21T15:51:16.489 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.489 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.489 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.489 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.489 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573515+0000", 2024-09-21T15:51:16.489 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9995008 2024-09-21T15:51:16.489 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.489 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.489 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.489 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574017+0000", 2024-09-21T15:51:16.489 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00050241100000000003 2024-09-21T15:51:16.489 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.489 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.490 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.490 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574027+0000", 2024-09-21T15:51:16.490 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4700000000000008e-06 2024-09-21T15:51:16.490 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.490 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.490 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.490 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574033+0000", 2024-09-21T15:51:16.490 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1489999999999999e-06 2024-09-21T15:51:16.490 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.490 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.490 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.490 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629238+0000", 2024-09-21T15:51:16.490 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055204880999999997 2024-09-21T15:51:16.490 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.491 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.491 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:16.491 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629317+0000", 2024-09-21T15:51:16.491 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8942000000000004e-05 2024-09-21T15:51:16.491 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.491 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.491 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.491 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629454+0000", 2024-09-21T15:51:16.491 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013767099999999999 2024-09-21T15:51:16.491 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.491 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.491 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.491 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.492 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.492 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120125 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 600064~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:16.492 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.574482+0000", 2024-09-21T15:51:16.492 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 72.855974266000004, 2024-09-21T15:51:16.492 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055243608, 2024-09-21T15:51:16.492 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.492 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:16.492 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.492 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.492 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.492 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120125 2024-09-21T15:51:16.492 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.492 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.492 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.492 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.492 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574482+0000", 2024-09-21T15:51:16.493 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.493 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.493 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.493 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.493 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574482+0000", 2024-09-21T15:51:16.493 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.493 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.493 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.493 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.493 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574038+0000", 2024-09-21T15:51:16.493 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9995556 2024-09-21T15:51:16.493 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.493 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.493 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.493 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574497+0000", 2024-09-21T15:51:16.493 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00045854400000000002 2024-09-21T15:51:16.493 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.494 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.494 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.494 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574509+0000", 2024-09-21T15:51:16.494 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2486e-05 2024-09-21T15:51:16.494 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.494 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.494 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.494 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574516+0000", 2024-09-21T15:51:16.494 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2269999999999996e-06 2024-09-21T15:51:16.494 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.494 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.494 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.494 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629497+0000", 2024-09-21T15:51:16.494 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054980785999999997 2024-09-21T15:51:16.494 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.495 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.495 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:16.495 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629580+0000", 2024-09-21T15:51:16.495 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2528000000000007e-05 2024-09-21T15:51:16.495 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.495 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.495 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.495 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629726+0000", 2024-09-21T15:51:16.495 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014630399999999999 2024-09-21T15:51:16.495 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.495 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.495 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.495 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.496 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.496 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120126 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 620544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:16.496 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.575315+0000", 2024-09-21T15:51:16.496 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 72.855141235999994, 2024-09-21T15:51:16.496 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054709559999999997, 2024-09-21T15:51:16.496 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.496 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:16.496 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.496 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.496 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.496 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120126 2024-09-21T15:51:16.496 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.496 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.496 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.496 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.496 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575315+0000", 2024-09-21T15:51:16.497 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.497 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.497 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.497 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.497 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575315+0000", 2024-09-21T15:51:16.497 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.497 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.497 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.497 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.497 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574522+0000", 2024-09-21T15:51:16.497 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9992065 2024-09-21T15:51:16.497 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.497 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.497 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.497 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575319+0000", 2024-09-21T15:51:16.497 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00079626800000000004 2024-09-21T15:51:16.498 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.498 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.498 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.498 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575329+0000", 2024-09-21T15:51:16.498 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.028e-05 2024-09-21T15:51:16.498 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.498 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.498 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.498 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575335+0000", 2024-09-21T15:51:16.498 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9290000000000003e-06 2024-09-21T15:51:16.498 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.498 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.498 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.498 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629806+0000", 2024-09-21T15:51:16.498 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054471558000000003 2024-09-21T15:51:16.499 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.499 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.499 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:16.499 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629885+0000", 2024-09-21T15:51:16.499 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8563e-05 2024-09-21T15:51:16.499 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.499 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.499 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.499 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630025+0000", 2024-09-21T15:51:16.499 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014019 2024-09-21T15:51:16.499 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.499 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.499 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.499 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.499 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.500 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120127 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 641024~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:16.500 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.575340+0000", 2024-09-21T15:51:16.500 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 72.855116529, 2024-09-21T15:51:16.500 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054895589000000002, 2024-09-21T15:51:16.500 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.500 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:16.500 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.500 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.500 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.500 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120127 2024-09-21T15:51:16.500 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.500 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.500 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.500 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.500 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575340+0000", 2024-09-21T15:51:16.500 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.501 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.501 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.501 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.501 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575340+0000", 2024-09-21T15:51:16.501 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.501 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.501 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.501 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.501 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575340+0000", 2024-09-21T15:51:16.501 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:16.501 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.501 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.501 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.501 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575342+0000", 2024-09-21T15:51:16.501 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5600000000000001e-06 2024-09-21T15:51:16.501 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.502 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.502 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.502 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575351+0000", 2024-09-21T15:51:16.502 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2349999999999999e-06 2024-09-21T15:51:16.502 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.502 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.502 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.502 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575356+0000", 2024-09-21T15:51:16.502 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4600000000000002e-06 2024-09-21T15:51:16.502 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.502 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.502 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.502 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630053+0000", 2024-09-21T15:51:16.502 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054697190999999999 2024-09-21T15:51:16.502 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.503 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.503 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:16.503 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630128+0000", 2024-09-21T15:51:16.503 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4777000000000006e-05 2024-09-21T15:51:16.503 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.503 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.503 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.503 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630236+0000", 2024-09-21T15:51:16.503 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000107837 2024-09-21T15:51:16.503 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.503 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.503 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.503 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.503 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.503 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120128 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 657408~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:16.504 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.575461+0000", 2024-09-21T15:51:16.504 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 72.854995950000003, 2024-09-21T15:51:16.504 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054945121999999999, 2024-09-21T15:51:16.504 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.504 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:16.504 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.504 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.504 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.504 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120128 2024-09-21T15:51:16.504 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.504 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.504 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.504 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.504 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575461+0000", 2024-09-21T15:51:16.505 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.505 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.505 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.505 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.505 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575461+0000", 2024-09-21T15:51:16.505 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.505 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.505 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.505 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.505 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575360+0000", 2024-09-21T15:51:16.505 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998994 2024-09-21T15:51:16.505 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.505 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.505 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.505 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575464+0000", 2024-09-21T15:51:16.505 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000103397 2024-09-21T15:51:16.506 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.506 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.506 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.506 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575472+0000", 2024-09-21T15:51:16.506 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3189999999999995e-06 2024-09-21T15:51:16.506 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.506 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.506 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.506 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575478+0000", 2024-09-21T15:51:16.506 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2779999999999999e-06 2024-09-21T15:51:16.506 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.506 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.506 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.506 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630268+0000", 2024-09-21T15:51:16.507 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054789545000000002 2024-09-21T15:51:16.507 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.507 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.507 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:16.507 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630315+0000", 2024-09-21T15:51:16.507 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7244999999999999e-05 2024-09-21T15:51:16.507 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.507 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.507 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.507 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630406+0000", 2024-09-21T15:51:16.507 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0811999999999997e-05 2024-09-21T15:51:16.507 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.507 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.508 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.508 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.508 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.508 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120130 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 698368~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:16.508 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.576043+0000", 2024-09-21T15:51:16.508 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 72.854414035000005, 2024-09-21T15:51:16.508 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054781798, 2024-09-21T15:51:16.508 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.508 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:16.508 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.508 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.508 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.508 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120130 2024-09-21T15:51:16.508 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.508 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.508 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.508 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.509 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576043+0000", 2024-09-21T15:51:16.509 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.509 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.509 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.509 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.509 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576043+0000", 2024-09-21T15:51:16.509 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.509 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.509 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.509 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.509 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576042+0000", 2024-09-21T15:51:16.509 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:16.509 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.509 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.509 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.510 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576044+0000", 2024-09-21T15:51:16.510 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0999999999999998e-06 2024-09-21T15:51:16.510 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.510 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.510 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.510 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576051+0000", 2024-09-21T15:51:16.510 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4779999999999996e-06 2024-09-21T15:51:16.510 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.510 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.510 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.510 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576058+0000", 2024-09-21T15:51:16.510 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2180000000000002e-06 2024-09-21T15:51:16.510 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.510 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.510 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.511 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630661+0000", 2024-09-21T15:51:16.511 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054603240999999997 2024-09-21T15:51:16.511 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.511 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.511 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:16.511 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630747+0000", 2024-09-21T15:51:16.511 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6046999999999998e-05 2024-09-21T15:51:16.511 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.511 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.511 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.511 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630824+0000", 2024-09-21T15:51:16.511 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7119999999999993e-05 2024-09-21T15:51:16.511 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.511 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.511 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.512 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.512 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.512 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:165208 3.2 3:40caac84:::rbd_id.image_client.0-clone14:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-21T15:51:16.512 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:51:06.879846+0000", 2024-09-21T15:51:16.512 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 9.5506107720000006, 2024-09-21T15:51:16.512 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1380672409999999, 2024-09-21T15:51:16.512 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.512 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:16.512 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.512 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.512 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.512 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 165208 2024-09-21T15:51:16.512 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.512 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.512 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.512 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.513 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:51:06.879846+0000", 2024-09-21T15:51:16.513 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.513 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.513 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.513 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.513 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:51:06.879846+0000", 2024-09-21T15:51:16.513 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.513 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.513 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.513 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.513 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:51:06.879845+0000", 2024-09-21T15:51:16.513 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:16.513 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.513 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.513 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.513 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:51:06.879849+0000", 2024-09-21T15:51:16.514 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.968e-06 2024-09-21T15:51:16.514 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.514 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.514 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.514 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:51:06.879860+0000", 2024-09-21T15:51:16.514 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0368e-05 2024-09-21T15:51:16.514 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.514 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.514 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.514 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:51:06.879867+0000", 2024-09-21T15:51:16.514 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0700000000000001e-06 2024-09-21T15:51:16.514 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.514 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.514 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.514 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:51:06.879907+0000", 2024-09-21T15:51:16.515 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9904000000000003e-05 2024-09-21T15:51:16.515 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.515 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.515 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-21T15:51:16.515 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:51:06.879914+0000", 2024-09-21T15:51:16.515 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4279999999999999e-06 2024-09-21T15:51:16.515 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.515 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.515 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.515 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:51:07.686406+0000", 2024-09-21T15:51:16.515 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.80649178700000002 2024-09-21T15:51:16.515 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.515 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.515 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-21T15:51:16.515 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:51:07.686421+0000", 2024-09-21T15:51:16.515 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5339e-05 2024-09-21T15:51:16.516 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.516 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.516 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.516 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:51:09.013823+0000", 2024-09-21T15:51:16.516 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.327402341 2024-09-21T15:51:16.516 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.516 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.516 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:16.516 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:51:09.014007+0000", 2024-09-21T15:51:16.516 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000183952 2024-09-21T15:51:16.516 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.516 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.516 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 3", 2024-09-21T15:51:16.516 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:51:09.014167+0000", 2024-09-21T15:51:16.516 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000159543 2024-09-21T15:51:16.516 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.517 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.517 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:51:16.517 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:51:09.014932+0000", 2024-09-21T15:51:16.517 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00076526800000000005 2024-09-21T15:51:16.517 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.517 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.517 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:16.517 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:51:09.017803+0000", 2024-09-21T15:51:16.517 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0028705789999999998 2024-09-21T15:51:16.517 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.517 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.517 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:51:16.517 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:51:09.017838+0000", 2024-09-21T15:51:16.517 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4782e-05 2024-09-21T15:51:16.517 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.517 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.518 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.518 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:51:09.017913+0000", 2024-09-21T15:51:16.518 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5570999999999999e-05 2024-09-21T15:51:16.518 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.518 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.518 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.518 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.518 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.518 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:51:16.519 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T15:51:16.520 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:51:16.518+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:51:16.620 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:51:16.619+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:51:16.622 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:51:16.622 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:51:16.622 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:51:16.622 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:51:16.629 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T15:51:16.721 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:51:16.719+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:51:16.772 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:51:16.772 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:51:16.772 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:51:16.772 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:51:16.772 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:51:16.780 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T15:51:16.821 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:51:16.820+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:51:16.922 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:51:16.921+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:51:16.933 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:51:16.933 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:51:16.933 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:51:16.933 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:51:16.933 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.933 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63140 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 876544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:16.934 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.133649+0000", 2024-09-21T15:51:16.934 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 99.797955368999993, 2024-09-21T15:51:16.934 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053950201000000003, 2024-09-21T15:51:16.934 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.934 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:16.934 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.934 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.934 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.934 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63140 2024-09-21T15:51:16.934 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.934 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.934 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.934 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.935 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133649+0000", 2024-09-21T15:51:16.935 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.935 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.935 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.935 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.935 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133649+0000", 2024-09-21T15:51:16.935 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.935 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.935 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.935 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.935 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133648+0000", 2024-09-21T15:51:16.935 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:16.935 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.935 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.935 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.935 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133651+0000", 2024-09-21T15:51:16.936 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1549999999999999e-06 2024-09-21T15:51:16.936 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.936 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.936 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.936 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133660+0000", 2024-09-21T15:51:16.936 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3899999999999993e-06 2024-09-21T15:51:16.936 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.936 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.936 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.936 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133669+0000", 2024-09-21T15:51:16.936 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3219999999999999e-06 2024-09-21T15:51:16.936 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.936 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.936 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.936 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187409+0000", 2024-09-21T15:51:16.937 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053740257 2024-09-21T15:51:16.937 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.937 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.937 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:16.937 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187492+0000", 2024-09-21T15:51:16.937 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2409000000000003e-05 2024-09-21T15:51:16.937 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.937 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.937 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.937 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187599+0000", 2024-09-21T15:51:16.937 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000107334 2024-09-21T15:51:16.937 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.937 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.937 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.937 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.938 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.938 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63141 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 884736~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:16.938 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.133843+0000", 2024-09-21T15:51:16.938 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 99.797761249999994, 2024-09-21T15:51:16.938 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053954015000000001, 2024-09-21T15:51:16.938 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.938 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:16.938 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.938 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.938 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.938 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63141 2024-09-21T15:51:16.938 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.938 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.939 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.939 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.939 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133843+0000", 2024-09-21T15:51:16.939 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.939 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.939 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.939 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.939 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133843+0000", 2024-09-21T15:51:16.939 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.939 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.939 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.939 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.939 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133673+0000", 2024-09-21T15:51:16.939 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998302 2024-09-21T15:51:16.940 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.940 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.940 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.940 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133846+0000", 2024-09-21T15:51:16.940 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000172762 2024-09-21T15:51:16.940 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.940 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.940 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.940 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133855+0000", 2024-09-21T15:51:16.940 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2940000000000006e-06 2024-09-21T15:51:16.940 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.940 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.940 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.940 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133863+0000", 2024-09-21T15:51:16.941 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9740000000000002e-06 2024-09-21T15:51:16.941 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.941 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.941 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.941 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187630+0000", 2024-09-21T15:51:16.941 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053767211000000002 2024-09-21T15:51:16.941 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.941 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.941 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:16.941 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187707+0000", 2024-09-21T15:51:16.941 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7446000000000004e-05 2024-09-21T15:51:16.941 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.941 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.942 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.942 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187797+0000", 2024-09-21T15:51:16.942 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.9858000000000006e-05 2024-09-21T15:51:16.942 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.942 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.942 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.942 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.942 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.942 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63142 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 892928~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:16.942 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134046+0000", 2024-09-21T15:51:16.942 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 99.797558308000006, 2024-09-21T15:51:16.942 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054017849999999999, 2024-09-21T15:51:16.942 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.943 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:16.943 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.943 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.943 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.943 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63142 2024-09-21T15:51:16.943 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.943 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.943 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.943 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.943 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134046+0000", 2024-09-21T15:51:16.943 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.943 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.943 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.943 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.943 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134046+0000", 2024-09-21T15:51:16.944 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.944 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.944 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.944 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.944 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133867+0000", 2024-09-21T15:51:16.944 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998217 2024-09-21T15:51:16.944 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.944 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.944 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.944 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134049+0000", 2024-09-21T15:51:16.944 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000181695 2024-09-21T15:51:16.944 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.944 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.944 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.944 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134057+0000", 2024-09-21T15:51:16.945 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.303e-06 2024-09-21T15:51:16.945 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.945 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.945 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.945 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134063+0000", 2024-09-21T15:51:16.945 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9959999999999999e-06 2024-09-21T15:51:16.945 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.945 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.945 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.945 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187849+0000", 2024-09-21T15:51:16.945 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053785571999999997 2024-09-21T15:51:16.945 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.945 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.945 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:16.945 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187954+0000", 2024-09-21T15:51:16.945 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010524899999999999 2024-09-21T15:51:16.946 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.946 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.946 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.946 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188064+0000", 2024-09-21T15:51:16.946 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000109549 2024-09-21T15:51:16.946 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.946 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.946 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.946 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.946 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.946 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63143 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 901120~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:16.946 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134239+0000", 2024-09-21T15:51:16.946 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 99.797365714999998, 2024-09-21T15:51:16.946 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054026058000000002, 2024-09-21T15:51:16.946 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.947 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:16.947 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.947 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.947 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.947 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63143 2024-09-21T15:51:16.947 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.947 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.947 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.947 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.947 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134239+0000", 2024-09-21T15:51:16.947 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.947 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.947 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.947 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.947 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134239+0000", 2024-09-21T15:51:16.948 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.948 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.948 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.948 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.948 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134068+0000", 2024-09-21T15:51:16.948 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998298 2024-09-21T15:51:16.948 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.948 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.948 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.948 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134242+0000", 2024-09-21T15:51:16.948 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00017414199999999999 2024-09-21T15:51:16.948 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.948 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.948 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.948 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134264+0000", 2024-09-21T15:51:16.948 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1552e-05 2024-09-21T15:51:16.949 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.949 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.949 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.949 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134270+0000", 2024-09-21T15:51:16.949 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6939999999999998e-06 2024-09-21T15:51:16.949 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.949 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.949 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.949 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188108+0000", 2024-09-21T15:51:16.949 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.05383719 2024-09-21T15:51:16.949 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.949 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.949 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:16.949 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188186+0000", 2024-09-21T15:51:16.949 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8318000000000005e-05 2024-09-21T15:51:16.950 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.950 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.950 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.950 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188265+0000", 2024-09-21T15:51:16.950 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8597000000000007e-05 2024-09-21T15:51:16.950 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.950 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.950 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.950 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.950 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.950 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63144 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 909312~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:16.950 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134432+0000", 2024-09-21T15:51:16.950 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 99.797172158999999, 2024-09-21T15:51:16.950 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053968953, 2024-09-21T15:51:16.951 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.951 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:16.951 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.951 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.951 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.951 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63144 2024-09-21T15:51:16.951 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.951 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.951 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.951 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.951 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134432+0000", 2024-09-21T15:51:16.951 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.951 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.951 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.951 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.951 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134432+0000", 2024-09-21T15:51:16.952 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.952 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.952 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.952 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.952 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134276+0000", 2024-09-21T15:51:16.952 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998441 2024-09-21T15:51:16.952 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.952 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.952 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.952 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134436+0000", 2024-09-21T15:51:16.952 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000159591 2024-09-21T15:51:16.952 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.952 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.952 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.952 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134445+0000", 2024-09-21T15:51:16.952 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6780000000000006e-06 2024-09-21T15:51:16.953 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.953 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.953 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.953 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134452+0000", 2024-09-21T15:51:16.953 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6830000000000008e-06 2024-09-21T15:51:16.953 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.953 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.953 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.953 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188282+0000", 2024-09-21T15:51:16.953 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053829252000000001 2024-09-21T15:51:16.953 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.953 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.953 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:16.953 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188328+0000", 2024-09-21T15:51:16.953 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6734999999999999e-05 2024-09-21T15:51:16.954 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.954 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.954 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.954 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188401+0000", 2024-09-21T15:51:16.954 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2731000000000005e-05 2024-09-21T15:51:16.954 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.954 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.954 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.954 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.954 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.954 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63145 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 860160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:16.954 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134736+0000", 2024-09-21T15:51:16.954 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 99.796868373999999, 2024-09-21T15:51:16.954 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053865960999999997, 2024-09-21T15:51:16.954 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.954 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:16.955 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.955 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.955 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.955 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63145 2024-09-21T15:51:16.955 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.955 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.955 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.955 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.955 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134736+0000", 2024-09-21T15:51:16.955 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.955 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.955 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.955 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.955 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134736+0000", 2024-09-21T15:51:16.955 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.955 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.956 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.956 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.956 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134458+0000", 2024-09-21T15:51:16.956 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999722 2024-09-21T15:51:16.956 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.956 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.956 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.956 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134741+0000", 2024-09-21T15:51:16.956 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00028245599999999999 2024-09-21T15:51:16.956 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.956 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.956 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.956 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134753+0000", 2024-09-21T15:51:16.956 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2548000000000001e-05 2024-09-21T15:51:16.956 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.956 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.957 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.957 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134760+0000", 2024-09-21T15:51:16.957 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4189999999999996e-06 2024-09-21T15:51:16.957 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.957 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.957 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.957 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188426+0000", 2024-09-21T15:51:16.957 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053665894999999998 2024-09-21T15:51:16.957 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.957 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.957 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:16.957 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188502+0000", 2024-09-21T15:51:16.957 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5791000000000007e-05 2024-09-21T15:51:16.957 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.957 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.958 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.958 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188602+0000", 2024-09-21T15:51:16.958 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9667999999999995e-05 2024-09-21T15:51:16.958 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.958 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.958 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.958 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.958 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.958 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63148 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 933888~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:16.958 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.135289+0000", 2024-09-21T15:51:16.958 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 99.796315418999995, 2024-09-21T15:51:16.958 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053948488000000003, 2024-09-21T15:51:16.958 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.958 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:16.958 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.958 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.959 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.959 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63148 2024-09-21T15:51:16.959 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.959 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.959 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.959 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.959 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135289+0000", 2024-09-21T15:51:16.959 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.959 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.959 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.959 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.959 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135289+0000", 2024-09-21T15:51:16.959 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.959 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.959 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.960 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.960 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135288+0000", 2024-09-21T15:51:16.960 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:16.960 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.960 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.960 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.960 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135296+0000", 2024-09-21T15:51:16.960 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5399999999999998e-06 2024-09-21T15:51:16.960 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.960 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.960 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.960 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135303+0000", 2024-09-21T15:51:16.960 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0500000000000003e-06 2024-09-21T15:51:16.960 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.960 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.961 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.961 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135308+0000", 2024-09-21T15:51:16.961 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9749999999999998e-06 2024-09-21T15:51:16.961 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.961 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.961 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.961 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189066+0000", 2024-09-21T15:51:16.961 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053758501 2024-09-21T15:51:16.961 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.961 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.961 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:16.961 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189145+0000", 2024-09-21T15:51:16.961 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9017000000000006e-05 2024-09-21T15:51:16.961 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.961 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.961 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.962 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189237+0000", 2024-09-21T15:51:16.962 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1906999999999999e-05 2024-09-21T15:51:16.962 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.962 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.962 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.962 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.962 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.962 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63149 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 942080~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:16.962 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.135430+0000", 2024-09-21T15:51:16.962 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 99.796174089999994, 2024-09-21T15:51:16.962 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054005434999999997, 2024-09-21T15:51:16.962 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.962 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:16.963 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.963 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.963 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.963 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63149 2024-09-21T15:51:16.963 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.963 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.963 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.963 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.963 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135430+0000", 2024-09-21T15:51:16.963 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.963 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.963 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.963 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.963 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135430+0000", 2024-09-21T15:51:16.963 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.964 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.964 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.964 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.964 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135313+0000", 2024-09-21T15:51:16.964 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998827 2024-09-21T15:51:16.964 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.964 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.964 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.964 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135434+0000", 2024-09-21T15:51:16.964 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000120876 2024-09-21T15:51:16.964 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.964 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.964 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.964 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135443+0000", 2024-09-21T15:51:16.964 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3340000000000008e-06 2024-09-21T15:51:16.964 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.965 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.965 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.965 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135449+0000", 2024-09-21T15:51:16.965 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1580000000000002e-06 2024-09-21T15:51:16.965 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.965 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.965 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.965 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189267+0000", 2024-09-21T15:51:16.965 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053817775999999998 2024-09-21T15:51:16.965 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.965 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.965 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:16.965 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189336+0000", 2024-09-21T15:51:16.965 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9485000000000003e-05 2024-09-21T15:51:16.965 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.966 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.966 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.966 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189436+0000", 2024-09-21T15:51:16.966 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9277000000000005e-05 2024-09-21T15:51:16.966 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.966 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.966 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.966 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.966 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.966 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63150 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 950272~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:16.966 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.135618+0000", 2024-09-21T15:51:16.966 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 99.795986345000003, 2024-09-21T15:51:16.966 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054027393, 2024-09-21T15:51:16.967 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.967 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:16.967 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.967 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.967 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.967 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63150 2024-09-21T15:51:16.967 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.967 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.967 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.967 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.967 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135618+0000", 2024-09-21T15:51:16.967 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.967 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.967 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.968 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.968 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135618+0000", 2024-09-21T15:51:16.968 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.968 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.968 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.968 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.968 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135454+0000", 2024-09-21T15:51:16.968 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998364 2024-09-21T15:51:16.968 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.968 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.968 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.968 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135621+0000", 2024-09-21T15:51:16.968 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016714099999999999 2024-09-21T15:51:16.968 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.968 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.968 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.969 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135631+0000", 2024-09-21T15:51:16.969 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5340000000000005e-06 2024-09-21T15:51:16.969 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.969 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.969 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.969 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135638+0000", 2024-09-21T15:51:16.969 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9280000000000004e-06 2024-09-21T15:51:16.969 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.969 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.969 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.969 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189463+0000", 2024-09-21T15:51:16.969 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053825681 2024-09-21T15:51:16.969 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.969 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.969 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:16.969 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189546+0000", 2024-09-21T15:51:16.970 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2278000000000001e-05 2024-09-21T15:51:16.970 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.970 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.970 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.970 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189645+0000", 2024-09-21T15:51:16.970 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9543999999999994e-05 2024-09-21T15:51:16.970 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.970 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.970 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.970 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.970 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.970 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63152 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 966656~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:16.970 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.136029+0000", 2024-09-21T15:51:16.970 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 99.795575752999994, 2024-09-21T15:51:16.970 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054024823999999999, 2024-09-21T15:51:16.971 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.971 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:16.971 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.971 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.971 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.971 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63152 2024-09-21T15:51:16.971 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.971 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.971 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.971 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.971 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136029+0000", 2024-09-21T15:51:16.971 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.971 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.971 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.972 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.972 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136029+0000", 2024-09-21T15:51:16.972 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.972 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.972 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.972 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.972 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136028+0000", 2024-09-21T15:51:16.972 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:16.972 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.972 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.972 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.972 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136031+0000", 2024-09-21T15:51:16.972 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.033e-06 2024-09-21T15:51:16.972 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.972 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.972 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.973 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136039+0000", 2024-09-21T15:51:16.973 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7200000000000006e-06 2024-09-21T15:51:16.973 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.973 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.973 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.973 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136051+0000", 2024-09-21T15:51:16.973 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1987000000000001e-05 2024-09-21T15:51:16.973 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.973 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.973 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.973 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189876+0000", 2024-09-21T15:51:16.973 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053825264999999997 2024-09-21T15:51:16.973 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.973 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.973 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:16.974 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189955+0000", 2024-09-21T15:51:16.974 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9296999999999997e-05 2024-09-21T15:51:16.974 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.974 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.974 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.974 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190053+0000", 2024-09-21T15:51:16.974 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.8067999999999997e-05 2024-09-21T15:51:16.974 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.974 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.974 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.975 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.975 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.975 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63153 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 974848~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:16.975 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.136311+0000", 2024-09-21T15:51:16.975 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 99.795293240000007, 2024-09-21T15:51:16.975 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053933825999999997, 2024-09-21T15:51:16.975 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.975 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:16.975 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.975 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.976 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.976 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63153 2024-09-21T15:51:16.976 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.976 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.976 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.976 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.976 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136311+0000", 2024-09-21T15:51:16.976 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.976 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.976 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.976 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.976 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136311+0000", 2024-09-21T15:51:16.976 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.976 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.977 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.977 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.977 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136057+0000", 2024-09-21T15:51:16.977 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9997463 2024-09-21T15:51:16.977 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.977 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.977 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.977 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136316+0000", 2024-09-21T15:51:16.977 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00025872299999999998 2024-09-21T15:51:16.977 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.977 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.977 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.977 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136326+0000", 2024-09-21T15:51:16.977 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0552e-05 2024-09-21T15:51:16.978 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.978 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.978 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.978 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136334+0000", 2024-09-21T15:51:16.978 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1409999999999999e-06 2024-09-21T15:51:16.978 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.978 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.978 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.978 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190080+0000", 2024-09-21T15:51:16.978 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053745933000000003 2024-09-21T15:51:16.978 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.978 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.978 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:16.978 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190157+0000", 2024-09-21T15:51:16.979 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7412999999999999e-05 2024-09-21T15:51:16.979 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.979 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.979 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.979 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190245+0000", 2024-09-21T15:51:16.979 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7904000000000004e-05 2024-09-21T15:51:16.979 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.979 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.979 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.979 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.979 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.979 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63154 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 983040~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:16.979 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.136472+0000", 2024-09-21T15:51:16.979 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 99.795131889000004, 2024-09-21T15:51:16.979 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053968448000000002, 2024-09-21T15:51:16.980 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.980 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:16.980 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.980 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.980 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.980 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63154 2024-09-21T15:51:16.980 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.980 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.980 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.980 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.980 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136472+0000", 2024-09-21T15:51:16.980 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.980 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.980 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.981 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.981 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136472+0000", 2024-09-21T15:51:16.981 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.981 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.981 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.981 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.981 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136341+0000", 2024-09-21T15:51:16.981 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998684 2024-09-21T15:51:16.981 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.981 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.981 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.981 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136476+0000", 2024-09-21T15:51:16.981 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013528100000000001 2024-09-21T15:51:16.981 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.981 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.982 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.982 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136484+0000", 2024-09-21T15:51:16.982 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4760000000000001e-06 2024-09-21T15:51:16.982 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.982 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.982 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.982 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136490+0000", 2024-09-21T15:51:16.982 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8640000000000001e-06 2024-09-21T15:51:16.982 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.982 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.982 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.982 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190271+0000", 2024-09-21T15:51:16.982 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053781223000000003 2024-09-21T15:51:16.982 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.982 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.983 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:16.983 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190348+0000", 2024-09-21T15:51:16.983 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7058999999999995e-05 2024-09-21T15:51:16.983 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.983 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.983 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.983 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190441+0000", 2024-09-21T15:51:16.983 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2937999999999997e-05 2024-09-21T15:51:16.983 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.983 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.983 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.983 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.983 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.983 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63157 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1015808~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:16.983 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137112+0000", 2024-09-21T15:51:16.983 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 99.794492602000005, 2024-09-21T15:51:16.984 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053932345, 2024-09-21T15:51:16.984 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.984 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:16.984 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.984 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.984 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.984 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63157 2024-09-21T15:51:16.984 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.984 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.984 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.984 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.984 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137112+0000", 2024-09-21T15:51:16.984 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.984 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.984 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.984 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.985 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137112+0000", 2024-09-21T15:51:16.985 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.985 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.985 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.985 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.985 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137072+0000", 2024-09-21T15:51:16.985 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999604 2024-09-21T15:51:16.985 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.985 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.985 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.985 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137115+0000", 2024-09-21T15:51:16.985 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2379999999999997e-05 2024-09-21T15:51:16.985 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.985 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.985 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.986 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137121+0000", 2024-09-21T15:51:16.986 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6499999999999999e-06 2024-09-21T15:51:16.986 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.986 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.986 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.986 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137126+0000", 2024-09-21T15:51:16.986 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0769999999999998e-06 2024-09-21T15:51:16.986 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.986 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.986 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.986 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190868+0000", 2024-09-21T15:51:16.986 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053741555000000003 2024-09-21T15:51:16.986 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.987 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.987 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:16.987 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190945+0000", 2024-09-21T15:51:16.987 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7171999999999999e-05 2024-09-21T15:51:16.987 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.987 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.987 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.987 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191044+0000", 2024-09-21T15:51:16.987 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9031999999999996e-05 2024-09-21T15:51:16.987 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.987 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.987 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.987 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.987 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.988 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63158 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1007616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:16.988 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137293+0000", 2024-09-21T15:51:16.988 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 99.794311315000002, 2024-09-21T15:51:16.988 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054005642, 2024-09-21T15:51:16.988 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.988 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:16.988 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.988 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.988 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.988 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63158 2024-09-21T15:51:16.988 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.988 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.988 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.989 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.989 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137293+0000", 2024-09-21T15:51:16.989 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.989 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.989 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.989 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.989 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137293+0000", 2024-09-21T15:51:16.989 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.989 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.989 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.989 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.989 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137131+0000", 2024-09-21T15:51:16.989 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998379 2024-09-21T15:51:16.989 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.990 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.990 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.990 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137296+0000", 2024-09-21T15:51:16.990 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016568999999999999 2024-09-21T15:51:16.990 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.990 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.990 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.990 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137306+0000", 2024-09-21T15:51:16.990 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.6029999999999994e-06 2024-09-21T15:51:16.990 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.990 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.990 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.990 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137312+0000", 2024-09-21T15:51:16.990 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3210000000000001e-06 2024-09-21T15:51:16.990 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.991 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.991 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.991 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191087+0000", 2024-09-21T15:51:16.991 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053774686000000002 2024-09-21T15:51:16.991 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.991 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.991 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:16.991 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191175+0000", 2024-09-21T15:51:16.991 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8106e-05 2024-09-21T15:51:16.991 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.991 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.991 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.991 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191299+0000", 2024-09-21T15:51:16.991 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012343599999999999 2024-09-21T15:51:16.991 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.991 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.992 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.992 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.992 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.992 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63159 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1024000~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:16.992 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137568+0000", 2024-09-21T15:51:16.992 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 99.794036043000006, 2024-09-21T15:51:16.992 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053962999999999997, 2024-09-21T15:51:16.992 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.992 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:16.992 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.992 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.992 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.992 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63159 2024-09-21T15:51:16.992 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.992 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.993 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.993 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.993 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137568+0000", 2024-09-21T15:51:16.993 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.993 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.993 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.993 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.993 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137568+0000", 2024-09-21T15:51:16.993 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.993 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.993 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.993 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.993 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137317+0000", 2024-09-21T15:51:16.993 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9997492 2024-09-21T15:51:16.993 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.994 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.994 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.994 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137572+0000", 2024-09-21T15:51:16.994 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00025491499999999998 2024-09-21T15:51:16.994 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.994 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.994 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.994 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137582+0000", 2024-09-21T15:51:16.994 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4490000000000005e-06 2024-09-21T15:51:16.994 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.994 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.994 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.994 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137589+0000", 2024-09-21T15:51:16.994 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5580000000000003e-06 2024-09-21T15:51:16.995 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.995 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.995 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.995 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191344+0000", 2024-09-21T15:51:16.995 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053754272999999998 2024-09-21T15:51:16.995 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.995 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.995 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:16.995 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191429+0000", 2024-09-21T15:51:16.995 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.509e-05 2024-09-21T15:51:16.995 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.995 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.995 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.995 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191531+0000", 2024-09-21T15:51:16.995 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000102519 2024-09-21T15:51:16.995 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.996 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:16.996 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:16.996 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.996 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.996 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63160 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1032192~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:16.996 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137771+0000", 2024-09-21T15:51:16.996 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 99.793833508999995, 2024-09-21T15:51:16.996 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053957818999999997, 2024-09-21T15:51:16.996 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:16.996 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:16.996 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:16.996 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:16.996 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:16.996 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63160 2024-09-21T15:51:16.997 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.997 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:16.997 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.997 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:16.997 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137771+0000", 2024-09-21T15:51:16.997 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.997 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.997 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.997 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:16.997 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137771+0000", 2024-09-21T15:51:16.997 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:16.997 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.997 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.997 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:16.997 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137596+0000", 2024-09-21T15:51:16.998 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998255 2024-09-21T15:51:16.998 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.998 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.998 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:16.998 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137780+0000", 2024-09-21T15:51:16.998 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00018363900000000001 2024-09-21T15:51:16.998 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.998 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.998 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:16.998 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137791+0000", 2024-09-21T15:51:16.998 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.096e-05 2024-09-21T15:51:16.998 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.998 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.998 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:16.998 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137798+0000", 2024-09-21T15:51:16.999 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1169999999999998e-06 2024-09-21T15:51:16.999 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.999 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.999 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:16.999 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191560+0000", 2024-09-21T15:51:16.999 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053762163000000002 2024-09-21T15:51:16.999 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.999 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.999 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:16.999 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191634+0000", 2024-09-21T15:51:16.999 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4381999999999994e-05 2024-09-21T15:51:16.999 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:16.999 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:16.999 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:16.999 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191729+0000", 2024-09-21T15:51:17.000 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4098000000000006e-05 2024-09-21T15:51:17.000 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:17.000 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:17.000 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:17.000 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.000 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.000 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63161 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1040384~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:17.000 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.138018+0000", 2024-09-21T15:51:17.000 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 99.793586755000007, 2024-09-21T15:51:17.000 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053905657000000003, 2024-09-21T15:51:17.000 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:17.000 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:17.000 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:17.000 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:17.001 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:17.001 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63161 2024-09-21T15:51:17.001 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.001 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:17.001 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.001 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:17.001 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138018+0000", 2024-09-21T15:51:17.001 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:17.001 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.001 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.001 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:17.001 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138018+0000", 2024-09-21T15:51:17.001 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:17.001 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.002 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.002 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:17.002 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137804+0000", 2024-09-21T15:51:17.002 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9997859 2024-09-21T15:51:17.002 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.002 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.002 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:17.002 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138021+0000", 2024-09-21T15:51:17.002 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00021769899999999999 2024-09-21T15:51:17.002 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.002 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.002 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:17.003 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138043+0000", 2024-09-21T15:51:17.003 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1627000000000001e-05 2024-09-21T15:51:17.003 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.003 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.003 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:17.003 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138054+0000", 2024-09-21T15:51:17.003 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1321999999999999e-05 2024-09-21T15:51:17.003 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.003 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.003 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:17.003 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191755+0000", 2024-09-21T15:51:17.003 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053700354999999998 2024-09-21T15:51:17.003 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.003 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.003 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:17.003 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191830+0000", 2024-09-21T15:51:17.004 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5320000000000004e-05 2024-09-21T15:51:17.004 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.004 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.004 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:17.004 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191923+0000", 2024-09-21T15:51:17.004 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3282999999999994e-05 2024-09-21T15:51:17.004 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:17.004 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:17.004 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:17.004 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.004 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.004 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63176 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 108544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:17.004 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.140861+0000", 2024-09-21T15:51:17.004 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 99.790743436, 2024-09-21T15:51:17.005 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053935394999999997, 2024-09-21T15:51:17.005 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:17.005 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:17.005 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:17.005 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:17.005 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:17.005 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63176 2024-09-21T15:51:17.005 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.005 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:17.005 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.005 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:17.005 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140861+0000", 2024-09-21T15:51:17.005 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:17.005 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.005 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.005 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:17.006 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140861+0000", 2024-09-21T15:51:17.006 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:17.006 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.006 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.006 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:17.006 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140860+0000", 2024-09-21T15:51:17.006 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:17.006 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.006 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.006 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:17.006 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140863+0000", 2024-09-21T15:51:17.006 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8710000000000001e-06 2024-09-21T15:51:17.006 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.006 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.006 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:17.007 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140871+0000", 2024-09-21T15:51:17.007 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8089999999999993e-06 2024-09-21T15:51:17.007 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.007 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.007 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:17.007 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140877+0000", 2024-09-21T15:51:17.007 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5969999999999999e-06 2024-09-21T15:51:17.007 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.007 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.007 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:17.007 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.194608+0000", 2024-09-21T15:51:17.007 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053731749000000002 2024-09-21T15:51:17.007 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.007 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.007 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:17.008 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.194697+0000", 2024-09-21T15:51:17.008 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8504000000000005e-05 2024-09-21T15:51:17.008 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.008 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.008 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:17.008 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.194796+0000", 2024-09-21T15:51:17.008 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9414999999999996e-05 2024-09-21T15:51:17.008 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:17.008 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:17.008 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:17.008 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.008 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.008 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63182 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 157696~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:17.008 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:51:17.008 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 99.789471035999995, 2024-09-21T15:51:17.008 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053958825000000002, 2024-09-21T15:51:17.009 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:17.009 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:17.009 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:17.009 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:17.009 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:17.009 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63182 2024-09-21T15:51:17.009 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.009 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:17.009 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.009 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:17.009 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:51:17.009 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:17.010 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.010 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.010 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:17.010 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:51:17.010 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:17.010 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.010 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.010 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:17.010 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:51:17.010 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:17.010 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.010 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.010 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:17.011 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142136+0000", 2024-09-21T15:51:17.011 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8169999999999999e-06 2024-09-21T15:51:17.011 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.011 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.011 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:17.011 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:51:17.011 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3752e-05 2024-09-21T15:51:17.011 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.011 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.011 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:17.011 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142155+0000", 2024-09-21T15:51:17.011 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1859999999999997e-06 2024-09-21T15:51:17.011 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.012 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.012 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:17.012 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.195914+0000", 2024-09-21T15:51:17.012 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053758528999999999 2024-09-21T15:51:17.012 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.012 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.012 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:17.012 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.195995+0000", 2024-09-21T15:51:17.012 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0983999999999996e-05 2024-09-21T15:51:17.012 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.012 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.012 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:17.012 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.196092+0000", 2024-09-21T15:51:17.012 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7069000000000002e-05 2024-09-21T15:51:17.012 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:17.013 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:17.013 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:17.013 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.013 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.013 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63193 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 239616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:17.013 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:51:17.013 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 99.787383450999997, 2024-09-21T15:51:17.013 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053956577999999998, 2024-09-21T15:51:17.013 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:17.013 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:17.013 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:17.013 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:17.013 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:17.013 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63193 2024-09-21T15:51:17.014 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.014 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:17.014 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.014 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:17.014 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:51:17.014 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:17.014 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.014 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.014 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:17.014 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:51:17.014 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:17.014 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.014 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.014 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:17.015 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:51:17.015 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:17.015 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.015 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.015 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:17.015 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144223+0000", 2024-09-21T15:51:17.015 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0990000000000001e-06 2024-09-21T15:51:17.015 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.015 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.015 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:17.015 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144230+0000", 2024-09-21T15:51:17.015 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9070000000000001e-06 2024-09-21T15:51:17.015 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.015 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.015 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:17.016 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144234+0000", 2024-09-21T15:51:17.016 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9010000000000002e-06 2024-09-21T15:51:17.016 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.016 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.016 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:17.016 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.198004+0000", 2024-09-21T15:51:17.016 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053769539999999998 2024-09-21T15:51:17.016 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.016 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.016 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:17.016 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.198083+0000", 2024-09-21T15:51:17.016 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9531999999999997e-05 2024-09-21T15:51:17.016 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.016 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.016 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:17.016 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.198177+0000", 2024-09-21T15:51:17.016 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3906000000000005e-05 2024-09-21T15:51:17.017 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:17.017 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:17.017 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:17.017 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:17.017 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:17.017 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:51:17.018 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T15:51:17.023 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:51:17.022+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:51:17.119 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:51:17.119 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:51:17.119 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:51:17.120 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:51:17.124 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:51:17.123+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:51:17.128 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T15:51:17.225 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:51:17.224+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:51:17.229 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:51:17.230 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:51:17.230 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:51:17.230 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:51:17.230 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:51:17.238 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T15:51:17.297 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7, 3] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-09-21T15:51:17.298 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T15:51:17.298 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-09-21T15:51:17.298 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T15:51:17.298 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T15:51:17.298 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok config set filestore_inject_stall 3 2024-09-21T15:51:17.342 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:51:17.341+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:51:17.381 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:51:17.381 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:51:17.381 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:51:17.381 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:51:17.381 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.382 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4209.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:51:17.382 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:51:17.382 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 146.18854890200001, 2024-09-21T15:51:17.382 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.19452909900000001, 2024-09-21T15:51:17.382 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:17.382 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:17.382 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:17.382 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4209", 2024-09-21T15:51:17.382 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/4045143050", 2024-09-21T15:51:17.382 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1 2024-09-21T15:51:17.382 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.382 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:17.382 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.382 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:17.383 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:51:17.383 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:17.383 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.383 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.383 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:17.383 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:51:17.383 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:17.383 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.383 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.383 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:17.383 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191161+0000", 2024-09-21T15:51:17.383 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-21T15:51:17.383 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.383 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.383 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:17.384 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191169+0000", 2024-09-21T15:51:17.384 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0090000000000007e-06 2024-09-21T15:51:17.384 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.384 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.384 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:17.384 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191180+0000", 2024-09-21T15:51:17.384 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0728e-05 2024-09-21T15:51:17.384 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.384 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.384 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:17.384 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191187+0000", 2024-09-21T15:51:17.384 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9009999999999994e-06 2024-09-21T15:51:17.384 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.385 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.385 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:17.385 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191238+0000", 2024-09-21T15:51:17.385 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0287000000000002e-05 2024-09-21T15:51:17.385 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.385 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.385 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-21T15:51:17.385 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191245+0000", 2024-09-21T15:51:17.385 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0709999999999997e-06 2024-09-21T15:51:17.385 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.385 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.385 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:17.385 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380309+0000", 2024-09-21T15:51:17.385 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.189064277 2024-09-21T15:51:17.386 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.386 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.386 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:17.386 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380454+0000", 2024-09-21T15:51:17.386 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014490799999999999 2024-09-21T15:51:17.386 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.386 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.386 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0", 2024-09-21T15:51:17.386 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380610+0000", 2024-09-21T15:51:17.386 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015650199999999999 2024-09-21T15:51:17.386 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.386 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.386 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:51:17.386 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.381160+0000", 2024-09-21T15:51:17.387 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00054903799999999996 2024-09-21T15:51:17.387 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.387 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.387 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:17.387 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385591+0000", 2024-09-21T15:51:17.387 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.004431272 2024-09-21T15:51:17.387 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.387 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.387 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:51:17.387 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385651+0000", 2024-09-21T15:51:17.387 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0455999999999999e-05 2024-09-21T15:51:17.387 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.387 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.387 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:17.388 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385691+0000", 2024-09-21T15:51:17.388 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0234999999999997e-05 2024-09-21T15:51:17.388 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:17.388 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:17.388 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:17.388 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.388 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.388 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64034 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1083392~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:17.388 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167717+0000", 2024-09-21T15:51:17.388 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 100.21199457199999, 2024-09-21T15:51:17.389 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048233316999999998, 2024-09-21T15:51:17.389 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:17.389 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:17.389 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:17.389 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:17.389 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:17.389 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64034 2024-09-21T15:51:17.389 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.389 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:17.389 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.389 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:17.389 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167717+0000", 2024-09-21T15:51:17.389 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:17.390 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.390 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.390 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:17.390 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167717+0000", 2024-09-21T15:51:17.390 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:17.390 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.390 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.390 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:17.391 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167716+0000", 2024-09-21T15:51:17.391 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:17.391 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.391 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.391 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:17.391 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167718+0000", 2024-09-21T15:51:17.391 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.165e-06 2024-09-21T15:51:17.391 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.391 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.391 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:17.391 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167725+0000", 2024-09-21T15:51:17.391 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3380000000000001e-06 2024-09-21T15:51:17.392 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.392 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.392 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:17.392 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167730+0000", 2024-09-21T15:51:17.392 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8010000000000003e-06 2024-09-21T15:51:17.392 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.392 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.392 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:17.392 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215807+0000", 2024-09-21T15:51:17.392 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048077823999999998 2024-09-21T15:51:17.392 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.392 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.392 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:17.392 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215884+0000", 2024-09-21T15:51:17.393 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6283000000000001e-05 2024-09-21T15:51:17.393 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.393 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.393 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:17.393 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215950+0000", 2024-09-21T15:51:17.393 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6290000000000004e-05 2024-09-21T15:51:17.393 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:17.393 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:17.393 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:17.393 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.393 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.393 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64035 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1091584~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:17.393 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:51:17.393 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 100.211977359, 2024-09-21T15:51:17.393 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048412117999999997, 2024-09-21T15:51:17.394 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:17.394 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:17.394 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:17.394 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:17.394 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:17.394 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64035 2024-09-21T15:51:17.394 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.394 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:17.394 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.394 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:17.394 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:51:17.394 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:17.394 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.395 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.395 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:17.395 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:51:17.395 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:17.395 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.395 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.395 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:17.395 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:51:17.395 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:17.395 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.395 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.395 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:17.395 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167746+0000", 2024-09-21T15:51:17.395 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2031000000000001e-05 2024-09-21T15:51:17.395 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.396 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.396 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:17.396 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167753+0000", 2024-09-21T15:51:17.396 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0620000000000003e-06 2024-09-21T15:51:17.396 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.396 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.396 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:17.396 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167759+0000", 2024-09-21T15:51:17.396 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.596e-06 2024-09-21T15:51:17.396 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.396 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.396 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:17.396 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215977+0000", 2024-09-21T15:51:17.396 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048217318000000002 2024-09-21T15:51:17.396 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.397 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.397 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:17.397 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216076+0000", 2024-09-21T15:51:17.397 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9846000000000006e-05 2024-09-21T15:51:17.397 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.397 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.397 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:17.397 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216146+0000", 2024-09-21T15:51:17.397 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9578000000000004e-05 2024-09-21T15:51:17.397 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:17.397 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:17.397 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:17.397 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.397 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.397 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64036 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1099776~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:17.398 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167853+0000", 2024-09-21T15:51:17.398 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 100.211858746, 2024-09-21T15:51:17.398 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048450675999999998, 2024-09-21T15:51:17.398 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:17.398 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:17.398 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:17.398 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:17.398 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:17.398 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64036 2024-09-21T15:51:17.398 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.398 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:17.398 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.398 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:17.398 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167853+0000", 2024-09-21T15:51:17.399 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:17.399 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.399 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.399 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:17.399 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167853+0000", 2024-09-21T15:51:17.399 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:17.399 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.399 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.399 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:17.399 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167852+0000", 2024-09-21T15:51:17.399 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:17.399 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.399 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.400 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:17.400 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167855+0000", 2024-09-21T15:51:17.400 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5640000000000001e-06 2024-09-21T15:51:17.400 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.400 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.400 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:17.400 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167864+0000", 2024-09-21T15:51:17.400 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8140000000000007e-06 2024-09-21T15:51:17.400 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.400 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.400 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:17.400 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167871+0000", 2024-09-21T15:51:17.400 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5729999999999996e-06 2024-09-21T15:51:17.400 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.400 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.401 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:17.401 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216174+0000", 2024-09-21T15:51:17.401 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048303437999999997 2024-09-21T15:51:17.401 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.401 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.401 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:17.401 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216250+0000", 2024-09-21T15:51:17.401 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5661999999999995e-05 2024-09-21T15:51:17.401 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.401 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.401 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:17.401 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216303+0000", 2024-09-21T15:51:17.401 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3273000000000003e-05 2024-09-21T15:51:17.401 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:17.401 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:17.402 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:17.402 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.402 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.402 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64037 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1771520~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:17.402 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:51:17.402 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 100.21183520300001, 2024-09-21T15:51:17.402 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048588771000000003, 2024-09-21T15:51:17.402 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:17.402 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:17.402 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:17.402 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:17.402 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:17.402 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64037 2024-09-21T15:51:17.402 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.403 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:17.403 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.403 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:17.403 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:51:17.403 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:17.403 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.403 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.403 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:17.403 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:51:17.403 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:17.403 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.403 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.403 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:17.403 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:51:17.404 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:17.404 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.404 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.404 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:17.404 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167878+0000", 2024-09-21T15:51:17.404 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3e-06 2024-09-21T15:51:17.404 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.404 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.404 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:17.404 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167884+0000", 2024-09-21T15:51:17.404 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3269999999999997e-06 2024-09-21T15:51:17.404 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.404 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.404 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:17.404 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167889+0000", 2024-09-21T15:51:17.405 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9269999999999996e-06 2024-09-21T15:51:17.405 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.405 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.405 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:17.405 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216331+0000", 2024-09-21T15:51:17.405 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048442102000000001 2024-09-21T15:51:17.405 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.405 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.405 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:17.405 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216393+0000", 2024-09-21T15:51:17.405 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1132999999999999e-05 2024-09-21T15:51:17.405 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.405 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.405 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:17.406 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216465+0000", 2024-09-21T15:51:17.406 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2339e-05 2024-09-21T15:51:17.406 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:17.406 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:17.406 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:17.406 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.406 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.406 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64038 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1779712~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:17.406 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:51:17.406 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 100.211817372, 2024-09-21T15:51:17.406 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048749795999999998, 2024-09-21T15:51:17.406 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:17.406 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:17.406 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:17.407 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:17.407 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:17.407 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64038 2024-09-21T15:51:17.407 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.407 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:17.407 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.407 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:17.407 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:51:17.407 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:17.407 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.407 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.407 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:17.407 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:51:17.408 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:17.408 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.408 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.408 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:17.408 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:51:17.408 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:17.408 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.408 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.408 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:17.408 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167896+0000", 2024-09-21T15:51:17.408 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2419999999999999e-06 2024-09-21T15:51:17.408 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.408 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.408 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:17.408 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167902+0000", 2024-09-21T15:51:17.409 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0859999999999998e-06 2024-09-21T15:51:17.409 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.409 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.409 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:17.409 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167907+0000", 2024-09-21T15:51:17.409 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9679999999999997e-06 2024-09-21T15:51:17.409 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.409 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.409 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:17.409 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216506+0000", 2024-09-21T15:51:17.409 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048599374000000001 2024-09-21T15:51:17.409 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.409 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.410 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:17.410 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216579+0000", 2024-09-21T15:51:17.410 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2801999999999999e-05 2024-09-21T15:51:17.410 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.410 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.410 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:17.410 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216644+0000", 2024-09-21T15:51:17.410 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4715000000000006e-05 2024-09-21T15:51:17.410 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:17.410 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:17.410 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:17.411 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.411 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.411 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64039 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1107968~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:17.411 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:51:17.411 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 100.211800017, 2024-09-21T15:51:17.411 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048911370000000003, 2024-09-21T15:51:17.411 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:17.411 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:17.411 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:17.411 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:17.411 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:17.411 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64039 2024-09-21T15:51:17.411 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.411 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:17.412 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.412 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:17.412 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:51:17.412 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:17.412 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.412 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.412 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:17.412 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:51:17.412 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:17.412 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.412 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.412 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:17.412 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:51:17.412 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:17.412 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.413 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.413 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:17.413 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167913+0000", 2024-09-21T15:51:17.413 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0839999999999999e-06 2024-09-21T15:51:17.413 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.413 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.413 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:17.413 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167921+0000", 2024-09-21T15:51:17.413 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8310000000000001e-06 2024-09-21T15:51:17.413 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.413 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.413 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:17.413 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167926+0000", 2024-09-21T15:51:17.413 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0270000000000003e-06 2024-09-21T15:51:17.414 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.414 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.414 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:17.414 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216680+0000", 2024-09-21T15:51:17.414 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048753890000000001 2024-09-21T15:51:17.414 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.414 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.414 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:17.414 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216757+0000", 2024-09-21T15:51:17.414 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6848999999999996e-05 2024-09-21T15:51:17.414 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.414 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.414 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:17.414 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216823+0000", 2024-09-21T15:51:17.415 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6068000000000005e-05 2024-09-21T15:51:17.415 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:17.415 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:17.415 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:17.415 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.415 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.415 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64040 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1787904~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:17.415 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167931+0000", 2024-09-21T15:51:17.415 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 100.211780785, 2024-09-21T15:51:17.415 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049010561000000001, 2024-09-21T15:51:17.416 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:17.416 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:17.416 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:17.416 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:17.416 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:17.416 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64040 2024-09-21T15:51:17.416 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.416 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:17.416 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.416 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:17.416 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167931+0000", 2024-09-21T15:51:17.416 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:17.416 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.417 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.417 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:17.417 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167931+0000", 2024-09-21T15:51:17.417 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:17.417 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.417 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.417 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:17.417 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167930+0000", 2024-09-21T15:51:17.417 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:17.417 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.417 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.417 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:17.417 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167932+0000", 2024-09-21T15:51:17.417 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.198e-06 2024-09-21T15:51:17.417 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.417 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:17.418 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167938+0000", 2024-09-21T15:51:17.418 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.105e-06 2024-09-21T15:51:17.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.418 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:17.418 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167943+0000", 2024-09-21T15:51:17.418 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9429999999999999e-06 2024-09-21T15:51:17.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.418 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.418 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:17.418 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216848+0000", 2024-09-21T15:51:17.418 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048905200000000003 2024-09-21T15:51:17.418 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.418 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.419 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:17.419 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216903+0000", 2024-09-21T15:51:17.419 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4793999999999997e-05 2024-09-21T15:51:17.419 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.419 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.419 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:17.419 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216941+0000", 2024-09-21T15:51:17.419 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7790999999999998e-05 2024-09-21T15:51:17.419 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:17.419 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:17.419 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:17.419 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.419 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.419 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64041 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1796096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:17.419 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:51:17.420 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 100.21176381399999, 2024-09-21T15:51:17.420 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049181457999999997, 2024-09-21T15:51:17.420 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:17.420 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:17.420 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:17.420 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:17.420 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:17.420 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64041 2024-09-21T15:51:17.420 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.420 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:17.420 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.420 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:17.420 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:51:17.420 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:17.420 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.420 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.421 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:17.421 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:51:17.421 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:17.421 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.421 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.421 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:17.421 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:51:17.421 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:17.421 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.421 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.421 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:17.421 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167949+0000", 2024-09-21T15:51:17.421 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0200000000000001e-06 2024-09-21T15:51:17.421 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.422 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.422 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:17.422 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167955+0000", 2024-09-21T15:51:17.422 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5990000000000001e-06 2024-09-21T15:51:17.422 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.422 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.422 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:17.422 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167959+0000", 2024-09-21T15:51:17.422 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5589999999999999e-06 2024-09-21T15:51:17.422 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.422 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.422 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:17.422 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216972+0000", 2024-09-21T15:51:17.422 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049012574000000003 2024-09-21T15:51:17.422 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.423 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.423 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:17.423 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217057+0000", 2024-09-21T15:51:17.423 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4820000000000004e-05 2024-09-21T15:51:17.423 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.423 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.423 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:17.423 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217129+0000", 2024-09-21T15:51:17.423 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2186000000000003e-05 2024-09-21T15:51:17.423 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:17.423 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:17.423 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:17.423 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.424 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.424 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64042 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1804288~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:17.424 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167964+0000", 2024-09-21T15:51:17.424 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 100.211747523, 2024-09-21T15:51:17.424 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049333372, 2024-09-21T15:51:17.424 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:17.424 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:17.424 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:17.424 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:17.424 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:17.424 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64042 2024-09-21T15:51:17.424 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.424 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:17.424 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.424 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:17.425 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167964+0000", 2024-09-21T15:51:17.425 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:17.425 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.425 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.425 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:17.425 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167964+0000", 2024-09-21T15:51:17.425 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:17.425 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.425 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.425 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:17.425 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167963+0000", 2024-09-21T15:51:17.425 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:17.425 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.425 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.426 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:17.426 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167966+0000", 2024-09-21T15:51:17.426 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1119999999999999e-06 2024-09-21T15:51:17.426 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.426 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.426 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:17.426 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167975+0000", 2024-09-21T15:51:17.426 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1130000000000007e-06 2024-09-21T15:51:17.426 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.426 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.426 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:17.426 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167980+0000", 2024-09-21T15:51:17.426 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9599999999999999e-06 2024-09-21T15:51:17.426 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.426 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.427 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:17.427 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217158+0000", 2024-09-21T15:51:17.427 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049178569999999998 2024-09-21T15:51:17.427 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.427 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.427 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:17.427 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217232+0000", 2024-09-21T15:51:17.427 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3589000000000003e-05 2024-09-21T15:51:17.427 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.427 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.427 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:17.427 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217297+0000", 2024-09-21T15:51:17.427 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5339999999999994e-05 2024-09-21T15:51:17.427 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:17.427 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:17.428 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:17.428 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.428 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.428 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64043 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1812480~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:17.428 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.168085+0000", 2024-09-21T15:51:17.428 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 100.211626456, 2024-09-21T15:51:17.428 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049365157999999999, 2024-09-21T15:51:17.428 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:17.428 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:17.428 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:17.428 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:17.428 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:17.428 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64043 2024-09-21T15:51:17.428 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.428 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:17.428 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.428 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:17.429 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168085+0000", 2024-09-21T15:51:17.429 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:17.429 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.429 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.429 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:17.429 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168085+0000", 2024-09-21T15:51:17.429 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:17.429 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.429 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.429 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:17.429 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168084+0000", 2024-09-21T15:51:17.429 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:17.429 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.429 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.429 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:17.429 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168088+0000", 2024-09-21T15:51:17.430 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5420000000000001e-06 2024-09-21T15:51:17.430 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.430 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.430 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:17.430 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168097+0000", 2024-09-21T15:51:17.430 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5410000000000006e-06 2024-09-21T15:51:17.430 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.430 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.430 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:17.430 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168104+0000", 2024-09-21T15:51:17.430 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6719999999999998e-06 2024-09-21T15:51:17.430 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.430 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.431 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:17.431 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217323+0000", 2024-09-21T15:51:17.431 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049218857999999997 2024-09-21T15:51:17.431 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.431 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.431 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:17.431 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217397+0000", 2024-09-21T15:51:17.431 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3932999999999998e-05 2024-09-21T15:51:17.431 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.431 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.431 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:17.431 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217450+0000", 2024-09-21T15:51:17.431 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3168000000000003e-05 2024-09-21T15:51:17.431 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:17.431 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:17.432 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:17.432 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.432 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.432 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155255 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 323584~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:17.432 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.440996+0000", 2024-09-21T15:51:17.432 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 54.938714929, 2024-09-21T15:51:17.432 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048233848000000003, 2024-09-21T15:51:17.432 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:17.432 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:17.432 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:17.432 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:17.432 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:17.432 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155255 2024-09-21T15:51:17.432 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.433 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:17.433 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.433 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:17.433 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.440996+0000", 2024-09-21T15:51:17.433 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:17.433 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.433 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.433 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:17.433 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.440996+0000", 2024-09-21T15:51:17.433 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:17.433 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.433 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.433 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:17.433 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.440996+0000", 2024-09-21T15:51:17.434 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:17.434 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.434 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.434 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:17.434 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441004+0000", 2024-09-21T15:51:17.434 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8460000000000008e-06 2024-09-21T15:51:17.434 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.434 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.434 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:17.434 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441018+0000", 2024-09-21T15:51:17.434 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.42e-05 2024-09-21T15:51:17.434 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.434 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.434 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:17.435 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441023+0000", 2024-09-21T15:51:17.435 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0019999999999997e-06 2024-09-21T15:51:17.435 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.435 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.435 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:17.435 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489060+0000", 2024-09-21T15:51:17.435 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048036660000000002 2024-09-21T15:51:17.435 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.435 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.435 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:17.435 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489107+0000", 2024-09-21T15:51:17.435 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7153000000000001e-05 2024-09-21T15:51:17.435 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.435 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.435 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:17.436 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489230+0000", 2024-09-21T15:51:17.436 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012328600000000001 2024-09-21T15:51:17.436 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:17.436 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:17.436 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:17.436 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.436 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.436 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155256 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 356352~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:17.436 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441027+0000", 2024-09-21T15:51:17.436 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 54.938683842000003, 2024-09-21T15:51:17.436 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048677606999999998, 2024-09-21T15:51:17.436 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:17.436 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:17.436 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:17.436 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:17.437 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:17.437 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155256 2024-09-21T15:51:17.437 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.437 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:17.437 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.437 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:17.437 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441027+0000", 2024-09-21T15:51:17.437 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:17.437 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.437 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.437 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:17.437 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441027+0000", 2024-09-21T15:51:17.437 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:17.437 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.437 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.437 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:17.438 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441027+0000", 2024-09-21T15:51:17.438 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:17.438 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.438 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.438 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:17.438 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441029+0000", 2024-09-21T15:51:17.438 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.39e-06 2024-09-21T15:51:17.438 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.438 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.438 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:17.438 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441036+0000", 2024-09-21T15:51:17.438 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1129999999999997e-06 2024-09-21T15:51:17.438 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.438 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.439 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:17.439 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441040+0000", 2024-09-21T15:51:17.439 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8459999999999999e-06 2024-09-21T15:51:17.439 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.439 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.439 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:17.439 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489244+0000", 2024-09-21T15:51:17.439 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048203751000000003 2024-09-21T15:51:17.439 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.439 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.439 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:17.439 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489288+0000", 2024-09-21T15:51:17.439 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4153999999999998e-05 2024-09-21T15:51:17.439 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.439 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.439 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:17.440 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489705+0000", 2024-09-21T15:51:17.440 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000416773 2024-09-21T15:51:17.440 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:17.440 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:17.440 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:17.440 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.440 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.440 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155257 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 389120~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:17.440 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441045+0000", 2024-09-21T15:51:17.440 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 54.938666751, 2024-09-21T15:51:17.440 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049000325999999997, 2024-09-21T15:51:17.440 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:17.440 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:17.440 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:17.440 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:17.441 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:17.441 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155257 2024-09-21T15:51:17.441 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.441 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:17.441 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.441 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:17.442 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441045+0000", 2024-09-21T15:51:17.442 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:17.442 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.442 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.442 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:17.442 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441045+0000", 2024-09-21T15:51:17.442 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:17.442 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.443 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.443 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:17.443 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441044+0000", 2024-09-21T15:51:17.443 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:17.443 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.443 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.443 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:17.443 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441046+0000", 2024-09-21T15:51:17.443 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1909999999999999e-06 2024-09-21T15:51:17.443 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.443 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.443 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:17.443 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441054+0000", 2024-09-21T15:51:17.443 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3900000000000004e-06 2024-09-21T15:51:17.444 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.444 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.444 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:17.444 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441058+0000", 2024-09-21T15:51:17.444 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4619999999999998e-06 2024-09-21T15:51:17.444 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.444 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.444 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:17.444 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489741+0000", 2024-09-21T15:51:17.444 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048682660000000003 2024-09-21T15:51:17.444 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.444 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.445 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:17.445 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489813+0000", 2024-09-21T15:51:17.445 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2051999999999994e-05 2024-09-21T15:51:17.445 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.445 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.445 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:17.445 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490045+0000", 2024-09-21T15:51:17.445 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00023191299999999999 2024-09-21T15:51:17.445 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:17.445 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:17.445 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:17.445 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.445 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.445 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155258 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 421888~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:17.446 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441062+0000", 2024-09-21T15:51:17.446 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 54.938649208999998, 2024-09-21T15:51:17.446 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049339651999999998, 2024-09-21T15:51:17.446 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:17.446 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:17.446 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:17.446 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:17.446 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:17.446 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155258 2024-09-21T15:51:17.446 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.446 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:17.446 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.446 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:17.446 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441062+0000", 2024-09-21T15:51:17.447 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:17.447 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.447 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.447 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:17.447 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441062+0000", 2024-09-21T15:51:17.447 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:17.447 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.447 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.447 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:17.447 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441062+0000", 2024-09-21T15:51:17.447 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:17.447 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.447 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.447 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:17.448 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441064+0000", 2024-09-21T15:51:17.448 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.967e-06 2024-09-21T15:51:17.448 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.448 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.448 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:17.448 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441069+0000", 2024-09-21T15:51:17.448 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1529999999999996e-06 2024-09-21T15:51:17.448 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.448 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.448 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:17.448 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441073+0000", 2024-09-21T15:51:17.448 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3850000000000004e-06 2024-09-21T15:51:17.448 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.448 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.448 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:17.448 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490082+0000", 2024-09-21T15:51:17.449 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.04900885 2024-09-21T15:51:17.449 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.449 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.449 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:17.449 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490166+0000", 2024-09-21T15:51:17.449 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3640999999999994e-05 2024-09-21T15:51:17.449 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.449 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.449 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:17.449 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490402+0000", 2024-09-21T15:51:17.449 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000235978 2024-09-21T15:51:17.449 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:17.449 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:17.449 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:17.449 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.449 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.450 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155259 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 454656~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:17.450 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441086+0000", 2024-09-21T15:51:17.450 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 54.938625404, 2024-09-21T15:51:17.450 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049624397000000001, 2024-09-21T15:51:17.450 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:17.450 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:17.450 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:17.450 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:17.450 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:17.450 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155259 2024-09-21T15:51:17.450 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.450 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:17.450 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.451 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:17.451 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441086+0000", 2024-09-21T15:51:17.451 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:17.451 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.451 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.451 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:17.451 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441086+0000", 2024-09-21T15:51:17.451 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:17.451 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.451 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.451 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:17.451 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441086+0000", 2024-09-21T15:51:17.451 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:17.451 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.452 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.452 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:17.452 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441088+0000", 2024-09-21T15:51:17.452 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2630000000000002e-06 2024-09-21T15:51:17.452 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.452 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.452 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:17.452 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441093+0000", 2024-09-21T15:51:17.452 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5010000000000004e-06 2024-09-21T15:51:17.452 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.452 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.452 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:17.452 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441098+0000", 2024-09-21T15:51:17.452 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.707e-06 2024-09-21T15:51:17.452 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.453 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.453 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:17.453 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490447+0000", 2024-09-21T15:51:17.453 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049349125000000001 2024-09-21T15:51:17.453 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.453 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.453 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:17.453 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490504+0000", 2024-09-21T15:51:17.453 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6521000000000003e-05 2024-09-21T15:51:17.453 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.453 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.453 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:17.453 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490710+0000", 2024-09-21T15:51:17.454 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00020663999999999999 2024-09-21T15:51:17.454 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:17.454 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:17.454 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:17.454 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.454 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.454 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155260 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 487424~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:17.454 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441160+0000", 2024-09-21T15:51:17.454 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 54.938551441999998, 2024-09-21T15:51:17.454 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049876810000000001, 2024-09-21T15:51:17.454 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:17.454 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:17.454 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:17.454 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:17.455 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:17.455 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155260 2024-09-21T15:51:17.455 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.455 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:17.455 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.455 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:17.455 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441160+0000", 2024-09-21T15:51:17.455 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:17.455 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.455 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.455 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:17.455 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441160+0000", 2024-09-21T15:51:17.455 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:17.455 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.455 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.456 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:17.456 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441159+0000", 2024-09-21T15:51:17.456 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:17.456 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.456 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.456 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:17.456 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441164+0000", 2024-09-21T15:51:17.456 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4059999999999998e-06 2024-09-21T15:51:17.456 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.456 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.456 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:17.456 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441175+0000", 2024-09-21T15:51:17.456 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1095e-05 2024-09-21T15:51:17.456 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.456 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.457 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:17.457 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441182+0000", 2024-09-21T15:51:17.457 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2139999999999999e-06 2024-09-21T15:51:17.457 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.457 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.457 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:17.457 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490751+0000", 2024-09-21T15:51:17.457 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049569230999999998 2024-09-21T15:51:17.457 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.457 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.457 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:17.457 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490837+0000", 2024-09-21T15:51:17.457 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6017e-05 2024-09-21T15:51:17.457 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.457 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.457 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:17.458 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491037+0000", 2024-09-21T15:51:17.458 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00019952300000000001 2024-09-21T15:51:17.458 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:17.458 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:17.458 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:17.458 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.458 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.458 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155261 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 520192~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:17.458 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441187+0000", 2024-09-21T15:51:17.458 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 54.938524258999998, 2024-09-21T15:51:17.458 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.050124931999999997, 2024-09-21T15:51:17.458 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:17.458 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:17.458 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:17.458 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:17.459 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:17.459 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155261 2024-09-21T15:51:17.459 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.459 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:17.459 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.459 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:17.459 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441187+0000", 2024-09-21T15:51:17.459 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:17.459 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.459 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.459 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:17.459 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441187+0000", 2024-09-21T15:51:17.459 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:17.459 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.459 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.460 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:17.460 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441187+0000", 2024-09-21T15:51:17.460 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:17.460 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.460 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.460 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:17.460 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441189+0000", 2024-09-21T15:51:17.460 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3649999999999998e-06 2024-09-21T15:51:17.460 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.460 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.460 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:17.460 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441196+0000", 2024-09-21T15:51:17.460 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5379999999999999e-06 2024-09-21T15:51:17.460 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.460 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.460 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:17.461 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441200+0000", 2024-09-21T15:51:17.461 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4719999999999997e-06 2024-09-21T15:51:17.461 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.461 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.461 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:17.461 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491070+0000", 2024-09-21T15:51:17.461 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049869915000000001 2024-09-21T15:51:17.461 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.461 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.461 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:17.461 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491146+0000", 2024-09-21T15:51:17.461 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6589999999999997e-05 2024-09-21T15:51:17.461 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.461 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.461 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:17.461 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491312+0000", 2024-09-21T15:51:17.462 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000165475 2024-09-21T15:51:17.462 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:17.462 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:17.462 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:17.462 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.462 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.462 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155308 3.11 3:8b5183e5:::rbd_data.11335aef9442.0000000000000000:2a [read 356352~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:17.462 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441369+0000", 2024-09-21T15:51:17.462 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 54.938342411000001, 2024-09-21T15:51:17.462 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.050090672000000003, 2024-09-21T15:51:17.462 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:17.462 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:17.462 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:17.462 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:17.462 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:17.463 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155308 2024-09-21T15:51:17.463 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.463 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:17.463 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.463 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:17.463 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441369+0000", 2024-09-21T15:51:17.463 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:17.463 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.463 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.463 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:17.463 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441369+0000", 2024-09-21T15:51:17.463 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:17.463 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.463 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.464 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:17.464 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441368+0000", 2024-09-21T15:51:17.464 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:17.464 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.464 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.464 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:17.464 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441371+0000", 2024-09-21T15:51:17.464 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1100000000000001e-06 2024-09-21T15:51:17.464 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.464 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.464 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:17.464 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441377+0000", 2024-09-21T15:51:17.464 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5649999999999998e-06 2024-09-21T15:51:17.465 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.465 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.465 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:17.465 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441382+0000", 2024-09-21T15:51:17.465 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5929999999999999e-06 2024-09-21T15:51:17.465 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.465 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.465 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:17.465 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491349+0000", 2024-09-21T15:51:17.465 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049967122000000003 2024-09-21T15:51:17.465 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.465 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.465 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:17.465 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491400+0000", 2024-09-21T15:51:17.466 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1589999999999999e-05 2024-09-21T15:51:17.466 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.466 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.466 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:17.466 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491460+0000", 2024-09-21T15:51:17.466 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9070000000000002e-05 2024-09-21T15:51:17.466 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:17.466 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:17.466 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:17.466 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.466 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.466 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155309 3.11 3:8b5183e5:::rbd_data.11335aef9442.0000000000000000:2a [read 389120~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:17.466 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441386+0000", 2024-09-21T15:51:17.467 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 54.938325061999997, 2024-09-21T15:51:17.467 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.050184777, 2024-09-21T15:51:17.467 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:17.467 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:17.467 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:17.467 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:17.467 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:17.467 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155309 2024-09-21T15:51:17.467 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.467 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:17.467 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.467 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:17.467 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441386+0000", 2024-09-21T15:51:17.467 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:17.468 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.468 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.468 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:17.468 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441386+0000", 2024-09-21T15:51:17.468 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:17.468 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.468 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.468 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:17.468 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441386+0000", 2024-09-21T15:51:17.468 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:17.468 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.468 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.469 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:17.469 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441388+0000", 2024-09-21T15:51:17.469 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4420000000000001e-06 2024-09-21T15:51:17.469 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.469 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.469 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:17.469 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441395+0000", 2024-09-21T15:51:17.469 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6189999999999996e-06 2024-09-21T15:51:17.469 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.469 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.469 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:17.469 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441400+0000", 2024-09-21T15:51:17.469 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.118e-06 2024-09-21T15:51:17.469 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.469 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.470 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:17.470 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491473+0000", 2024-09-21T15:51:17.470 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.050072783000000003 2024-09-21T15:51:17.470 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.470 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.470 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:17.470 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491518+0000", 2024-09-21T15:51:17.470 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4854000000000001e-05 2024-09-21T15:51:17.470 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:17.470 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:17.470 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:17.470 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491571+0000", 2024-09-21T15:51:17.470 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3358e-05 2024-09-21T15:51:17.470 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:17.470 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:17.471 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:17.471 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:17.471 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:17.471 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:51:17.472 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:17.472 INFO:teuthology.orchestra.run.smithi089.stdout: "success": "" 2024-09-21T15:51:17.473 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:17.473 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:51:17.473 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:51:17.471+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:51:17.572 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:51:17.571+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:51:17.615 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:17.615 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:51:17.615 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:51:17.615 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:17.624 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:51:17.673 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:51:17.673+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:51:17.774 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:51:17.772+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:51:17.779 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:17.779 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:51:17.779 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:51:17.779 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:51:17.779 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:17.789 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:51:17.865 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set noscrub 2024-09-21T15:51:17.875 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:51:17.874+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:51:17.936 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:17.936 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:51:17.936 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:51:17.936 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:51:17.936 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.936 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160505 3.b 3:d297695c:::rbd_header.11338bae974a:head [watch reconnect cookie 140381478598448 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-21T15:51:17.936 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:29.138810+0000", 2024-09-21T15:51:17.936 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 48.795156839000001, 2024-09-21T15:51:17.937 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0172598319999999, 2024-09-21T15:51:17.937 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:17.937 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:17.937 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:17.937 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:17.937 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:17.937 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160505 2024-09-21T15:51:17.937 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.937 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:17.937 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.937 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:17.937 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138810+0000", 2024-09-21T15:51:17.937 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:17.937 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.937 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.937 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:17.938 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138810+0000", 2024-09-21T15:51:17.938 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:17.938 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.938 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.938 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:17.938 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138808+0000", 2024-09-21T15:51:17.938 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:51:17.938 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.938 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.938 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:17.938 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138815+0000", 2024-09-21T15:51:17.938 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6549999999999998e-06 2024-09-21T15:51:17.938 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.938 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.938 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:17.938 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138825+0000", 2024-09-21T15:51:17.939 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9790000000000005e-06 2024-09-21T15:51:17.939 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.939 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.939 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:17.939 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138834+0000", 2024-09-21T15:51:17.939 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5179999999999993e-06 2024-09-21T15:51:17.939 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.939 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.939 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:17.939 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.140973+0000", 2024-09-21T15:51:17.939 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0021392049999999999 2024-09-21T15:51:17.939 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.939 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.939 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:51:17.940 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.140981+0000", 2024-09-21T15:51:17.940 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8909999999999995e-06 2024-09-21T15:51:17.940 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.940 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.940 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:17.940 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.150457+0000", 2024-09-21T15:51:17.940 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.009475645 2024-09-21T15:51:17.940 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.940 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.940 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for missing object", 2024-09-21T15:51:17.940 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.150589+0000", 2024-09-21T15:51:17.940 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000131779 2024-09-21T15:51:17.940 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.940 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.940 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:17.941 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.151723+0000", 2024-09-21T15:51:17.941 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0011345979999999999 2024-09-21T15:51:17.941 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.941 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.941 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-09-21T15:51:17.941 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.151872+0000", 2024-09-21T15:51:17.941 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000149021 2024-09-21T15:51:17.941 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.941 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.941 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:17.941 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.152563+0000", 2024-09-21T15:51:17.941 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00069051500000000005 2024-09-21T15:51:17.941 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.941 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.941 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:17.942 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.152673+0000", 2024-09-21T15:51:17.942 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000109795 2024-09-21T15:51:17.942 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.942 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.942 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:17.942 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.156069+0000", 2024-09-21T15:51:17.942 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003396736 2024-09-21T15:51:17.942 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:17.942 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:17.942 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:17.942 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.942 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.942 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160506 3.b 3:d297695c:::rbd_header.11338bae974a:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-21T15:51:17.942 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:29.138844+0000", 2024-09-21T15:51:17.942 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 48.795121952000002, 2024-09-21T15:51:17.943 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0174592680000001, 2024-09-21T15:51:17.943 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:17.943 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:17.943 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:17.943 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:17.943 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:17.943 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160506 2024-09-21T15:51:17.943 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.943 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:17.943 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.943 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:17.943 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138844+0000", 2024-09-21T15:51:17.943 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:17.943 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.944 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.944 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:17.944 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138844+0000", 2024-09-21T15:51:17.944 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:17.944 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.944 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.944 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:17.944 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138844+0000", 2024-09-21T15:51:17.944 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:17.944 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.944 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.944 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:17.944 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138848+0000", 2024-09-21T15:51:17.944 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.76e-06 2024-09-21T15:51:17.945 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.945 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.945 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:17.945 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138854+0000", 2024-09-21T15:51:17.945 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3890000000000002e-06 2024-09-21T15:51:17.945 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.945 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.945 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:17.945 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138860+0000", 2024-09-21T15:51:17.945 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6409999999999999e-06 2024-09-21T15:51:17.945 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.945 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.945 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:17.945 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.141015+0000", 2024-09-21T15:51:17.946 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0021553779999999999 2024-09-21T15:51:17.946 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.946 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.946 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:51:17.946 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.141021+0000", 2024-09-21T15:51:17.946 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7629999999999997e-06 2024-09-21T15:51:17.946 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.946 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.946 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:17.946 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.150858+0000", 2024-09-21T15:51:17.946 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0098372200000001 2024-09-21T15:51:17.946 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.946 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.946 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for missing object", 2024-09-21T15:51:17.947 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.150899+0000", 2024-09-21T15:51:17.947 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1112000000000002e-05 2024-09-21T15:51:17.947 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.947 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.947 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:17.947 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.152069+0000", 2024-09-21T15:51:17.947 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001170307 2024-09-21T15:51:17.947 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.947 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.947 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-09-21T15:51:17.947 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.152192+0000", 2024-09-21T15:51:17.947 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012289700000000001 2024-09-21T15:51:17.947 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.947 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.947 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:17.947 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.153362+0000", 2024-09-21T15:51:17.948 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0011700549999999999 2024-09-21T15:51:17.948 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.948 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.948 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:17.948 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.153457+0000", 2024-09-21T15:51:17.948 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4687999999999999e-05 2024-09-21T15:51:17.948 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.948 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.948 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:51:17.948 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.153893+0000", 2024-09-21T15:51:17.948 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000436363 2024-09-21T15:51:17.948 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.948 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.948 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:17.948 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.156134+0000", 2024-09-21T15:51:17.948 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0022410009999999998 2024-09-21T15:51:17.949 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.949 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.949 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:17.949 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.156148+0000", 2024-09-21T15:51:17.949 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3574000000000001e-05 2024-09-21T15:51:17.949 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.949 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.949 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:17.949 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.156214+0000", 2024-09-21T15:51:17.949 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6438000000000003e-05 2024-09-21T15:51:17.949 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.949 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.949 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:17.949 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.156304+0000", 2024-09-21T15:51:17.950 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9278999999999997e-05 2024-09-21T15:51:17.950 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:17.950 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:17.950 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:17.950 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.950 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.950 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160512 3.b 3:d297695c:::rbd_header.11338bae974a:head [watch unwatch cookie 140381478598448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T15:51:17.950 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:30.161362+0000", 2024-09-21T15:51:17.950 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 47.772604143999999, 2024-09-21T15:51:17.950 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0074105029999999997, 2024-09-21T15:51:17.950 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:17.950 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:17.950 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:17.950 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:17.950 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:17.951 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160512 2024-09-21T15:51:17.951 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.951 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:17.951 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.951 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:17.951 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161362+0000", 2024-09-21T15:51:17.951 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:17.951 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.951 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.951 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:17.951 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161362+0000", 2024-09-21T15:51:17.951 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:17.951 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.952 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.952 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:17.952 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161362+0000", 2024-09-21T15:51:17.952 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:17.952 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.952 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.952 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:17.952 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161365+0000", 2024-09-21T15:51:17.952 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8669999999999998e-06 2024-09-21T15:51:17.952 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.952 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.953 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:17.953 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161371+0000", 2024-09-21T15:51:17.953 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8009999999999998e-06 2024-09-21T15:51:17.953 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.953 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.953 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:17.953 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161376+0000", 2024-09-21T15:51:17.954 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0340000000000004e-06 2024-09-21T15:51:17.954 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.954 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.954 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:17.954 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161762+0000", 2024-09-21T15:51:17.954 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038575099999999999 2024-09-21T15:51:17.954 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.954 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.954 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:17.954 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161838+0000", 2024-09-21T15:51:17.955 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5697000000000004e-05 2024-09-21T15:51:17.955 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.955 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.955 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:51:17.955 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161952+0000", 2024-09-21T15:51:17.955 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000114034 2024-09-21T15:51:17.955 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.955 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.955 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:17.955 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.163727+0000", 2024-09-21T15:51:17.955 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0017749560000000001 2024-09-21T15:51:17.955 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.955 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.955 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:17.956 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168545+0000", 2024-09-21T15:51:17.956 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0048186080000000003 2024-09-21T15:51:17.956 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.956 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.956 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:17.956 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168593+0000", 2024-09-21T15:51:17.956 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7549999999999997e-05 2024-09-21T15:51:17.956 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.956 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.956 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:17.956 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168773+0000", 2024-09-21T15:51:17.956 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00017976600000000001 2024-09-21T15:51:17.956 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:17.957 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:17.957 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:17.957 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.957 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.957 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160513 3.b 3:d297695c:::rbd_header.11338bae974a:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-09-21T15:51:17.957 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:30.161755+0000", 2024-09-21T15:51:17.957 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 47.772211212999999, 2024-09-21T15:51:17.957 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0072250300000000003, 2024-09-21T15:51:17.957 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:17.957 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:17.957 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:17.957 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:17.957 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:17.957 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160513 2024-09-21T15:51:17.958 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.958 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:17.958 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.958 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:17.958 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161755+0000", 2024-09-21T15:51:17.958 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:17.958 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.958 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.958 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:17.958 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161755+0000", 2024-09-21T15:51:17.958 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:17.958 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.958 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.959 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:17.959 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161754+0000", 2024-09-21T15:51:17.959 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:17.959 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.959 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.959 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:17.959 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161759+0000", 2024-09-21T15:51:17.959 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9699999999999998e-06 2024-09-21T15:51:17.959 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.959 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.959 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:17.959 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161768+0000", 2024-09-21T15:51:17.959 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9949999999999994e-06 2024-09-21T15:51:17.959 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.959 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.959 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:17.960 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161774+0000", 2024-09-21T15:51:17.960 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1290000000000001e-06 2024-09-21T15:51:17.960 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.960 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.960 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:17.960 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.162239+0000", 2024-09-21T15:51:17.960 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00046406600000000001 2024-09-21T15:51:17.960 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.960 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.960 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-09-21T15:51:17.960 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.162296+0000", 2024-09-21T15:51:17.960 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7122e-05 2024-09-21T15:51:17.960 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.960 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.960 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:17.960 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168802+0000", 2024-09-21T15:51:17.960 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0065061700000000004 2024-09-21T15:51:17.961 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.961 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.961 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:17.961 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168877+0000", 2024-09-21T15:51:17.961 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5214000000000002e-05 2024-09-21T15:51:17.961 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.961 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.961 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:17.961 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168980+0000", 2024-09-21T15:51:17.961 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000103182 2024-09-21T15:51:17.961 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:17.961 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:17.961 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:17.961 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.961 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.961 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160527 3.b 3:d297695c:::rbd_header.11338bae974a:head [watch watch cookie 140381476148256] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T15:51:17.962 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:30.182756+0000", 2024-09-21T15:51:17.962 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 47.751210632000003, 2024-09-21T15:51:17.962 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0054202579999999998, 2024-09-21T15:51:17.962 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:17.962 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:17.962 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:17.962 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:17.962 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:17.962 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160527 2024-09-21T15:51:17.962 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.962 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:17.962 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.962 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:17.962 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182756+0000", 2024-09-21T15:51:17.962 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:17.962 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.963 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.963 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:17.963 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182756+0000", 2024-09-21T15:51:17.963 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:17.963 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.963 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.963 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:17.963 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182755+0000", 2024-09-21T15:51:17.963 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:17.963 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.963 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.963 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:17.963 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182760+0000", 2024-09-21T15:51:17.963 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4269999999999999e-06 2024-09-21T15:51:17.963 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.964 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.964 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:17.964 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182771+0000", 2024-09-21T15:51:17.964 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0312999999999999e-05 2024-09-21T15:51:17.964 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.964 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.964 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:17.964 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182778+0000", 2024-09-21T15:51:17.964 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5700000000000004e-06 2024-09-21T15:51:17.964 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.964 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.964 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:17.964 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.184745+0000", 2024-09-21T15:51:17.964 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0019667170000000002 2024-09-21T15:51:17.965 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.965 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.965 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:17.965 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.184867+0000", 2024-09-21T15:51:17.965 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000122504 2024-09-21T15:51:17.965 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.965 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.965 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:51:17.965 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.185135+0000", 2024-09-21T15:51:17.965 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000267135 2024-09-21T15:51:17.965 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.965 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.965 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:17.965 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.187721+0000", 2024-09-21T15:51:17.966 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002586891 2024-09-21T15:51:17.966 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.966 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.966 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:17.966 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.187874+0000", 2024-09-21T15:51:17.966 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000152726 2024-09-21T15:51:17.966 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.966 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.966 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:17.966 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.187951+0000", 2024-09-21T15:51:17.966 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7132000000000006e-05 2024-09-21T15:51:17.966 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.966 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.966 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:17.966 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.188176+0000", 2024-09-21T15:51:17.967 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00022477900000000001 2024-09-21T15:51:17.967 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:17.967 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:17.967 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:17.967 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.967 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.967 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:160608 3.12 e95/94 3:48a47e1c:::rbd_data.11338bae974a.000000000000000b:head v 95'345, mlcod=0'0)", 2024-09-21T15:51:17.967 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.216934+0000", 2024-09-21T15:51:17.967 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 46.717032494000001, 2024-09-21T15:51:17.967 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053335759999999996, 2024-09-21T15:51:17.967 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:17.967 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:17.967 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:17.967 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.967 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:17.968 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216934+0000", 2024-09-21T15:51:17.968 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:17.968 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.968 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.968 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:17.968 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216934+0000", 2024-09-21T15:51:17.968 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:17.968 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.968 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.968 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:17.968 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216933+0000", 2024-09-21T15:51:17.968 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:17.968 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.968 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.968 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:17.969 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216939+0000", 2024-09-21T15:51:17.969 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6459999999999998e-06 2024-09-21T15:51:17.969 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.969 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.969 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:17.969 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216947+0000", 2024-09-21T15:51:17.969 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.2479999999999996e-06 2024-09-21T15:51:17.969 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.969 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.969 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:17.969 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216951+0000", 2024-09-21T15:51:17.969 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3760000000000001e-06 2024-09-21T15:51:17.969 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.969 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.969 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:17.970 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216998+0000", 2024-09-21T15:51:17.970 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6244e-05 2024-09-21T15:51:17.970 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.970 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.970 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:17.970 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.217019+0000", 2024-09-21T15:51:17.970 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1191000000000001e-05 2024-09-21T15:51:17.970 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.970 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.970 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:17.970 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.222230+0000", 2024-09-21T15:51:17.970 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052114350000000004 2024-09-21T15:51:17.970 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.970 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.970 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:17.971 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.222267+0000", 2024-09-21T15:51:17.971 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7169999999999998e-05 2024-09-21T15:51:17.971 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:17.971 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:17.971 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:17.971 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.971 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.971 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160618 3.b 3:d297695c:::rbd_header.11338bae974a:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-21T15:51:17.971 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.233457+0000", 2024-09-21T15:51:17.971 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 46.700509656999998, 2024-09-21T15:51:17.971 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.005085198, 2024-09-21T15:51:17.971 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:17.971 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:17.971 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:17.972 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:17.972 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:17.972 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160618 2024-09-21T15:51:17.972 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.972 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:17.972 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.972 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:17.972 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233457+0000", 2024-09-21T15:51:17.972 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:17.972 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.972 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.972 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:17.972 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233457+0000", 2024-09-21T15:51:17.972 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:17.972 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.973 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.973 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:17.973 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233456+0000", 2024-09-21T15:51:17.973 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:17.973 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.973 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.973 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:17.973 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233463+0000", 2024-09-21T15:51:17.973 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9310000000000002e-06 2024-09-21T15:51:17.973 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.973 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.973 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:17.973 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233473+0000", 2024-09-21T15:51:17.973 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0519e-05 2024-09-21T15:51:17.973 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.974 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.974 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:17.974 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233481+0000", 2024-09-21T15:51:17.974 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4009999999999997e-06 2024-09-21T15:51:17.974 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.974 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.974 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:17.974 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233513+0000", 2024-09-21T15:51:17.974 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1328999999999998e-05 2024-09-21T15:51:17.974 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.975 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.975 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:17.975 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233568+0000", 2024-09-21T15:51:17.975 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5606999999999998e-05 2024-09-21T15:51:17.975 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.975 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.975 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:51:17.975 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.233670+0000", 2024-09-21T15:51:17.975 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010180599999999999 2024-09-21T15:51:17.975 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.975 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.975 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:17.975 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.234761+0000", 2024-09-21T15:51:17.975 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010905909999999999 2024-09-21T15:51:17.975 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.976 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.976 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:17.976 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.238442+0000", 2024-09-21T15:51:17.976 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0036809429999999999 2024-09-21T15:51:17.976 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.976 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.976 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:17.976 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.238498+0000", 2024-09-21T15:51:17.976 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6116999999999998e-05 2024-09-21T15:51:17.976 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.976 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.976 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:17.977 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.238542+0000", 2024-09-21T15:51:17.977 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4134000000000001e-05 2024-09-21T15:51:17.977 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:17.977 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:17.977 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:17.977 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.977 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.977 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160662 3.b 3:d7701915:::rbd_object_map.1133fd7483d4.000000000000002c:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-21T15:51:17.977 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.279369+0000", 2024-09-21T15:51:17.977 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 46.654597727000002, 2024-09-21T15:51:17.977 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0056948040000000004, 2024-09-21T15:51:17.977 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:17.977 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:17.977 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:17.977 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:17.978 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:17.978 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160662 2024-09-21T15:51:17.978 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.978 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:17.978 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.978 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:17.978 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279369+0000", 2024-09-21T15:51:17.978 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:17.978 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.978 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.978 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:17.978 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279369+0000", 2024-09-21T15:51:17.978 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:17.979 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.979 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.979 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:17.979 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279368+0000", 2024-09-21T15:51:17.979 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:17.979 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.979 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.980 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:17.980 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279373+0000", 2024-09-21T15:51:17.980 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6529999999999999e-06 2024-09-21T15:51:17.980 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.980 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.980 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:17.980 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279384+0000", 2024-09-21T15:51:17.980 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0256e-05 2024-09-21T15:51:17.980 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.980 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.980 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:17.980 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279388+0000", 2024-09-21T15:51:17.981 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5630000000000002e-06 2024-09-21T15:51:17.981 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.981 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.981 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:17.981 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279424+0000", 2024-09-21T15:51:17.981 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5985999999999997e-05 2024-09-21T15:51:17.981 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.981 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.981 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:17.981 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279478+0000", 2024-09-21T15:51:17.981 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3944999999999999e-05 2024-09-21T15:51:17.981 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.981 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.981 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:51:17.982 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279575+0000", 2024-09-21T15:51:17.982 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6817999999999994e-05 2024-09-21T15:51:17.982 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.982 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.982 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:17.982 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.282391+0000", 2024-09-21T15:51:17.982 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0028156700000000002 2024-09-21T15:51:17.982 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.982 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.982 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:17.982 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.284977+0000", 2024-09-21T15:51:17.982 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0025859640000000001 2024-09-21T15:51:17.982 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.983 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.983 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:17.983 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.285021+0000", 2024-09-21T15:51:17.983 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4415000000000001e-05 2024-09-21T15:51:17.983 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.983 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.983 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:17.983 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.285063+0000", 2024-09-21T15:51:17.983 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2447000000000002e-05 2024-09-21T15:51:17.983 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:17.983 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:17.983 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:17.983 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.983 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.983 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:161122 3.1 3:82ccd16f:::rbd_header.11335aef9442:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-21T15:51:17.984 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:36.703081+0000", 2024-09-21T15:51:17.984 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 41.230885563999998, 2024-09-21T15:51:17.984 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0051212200000000001, 2024-09-21T15:51:17.984 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:17.984 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:17.984 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:17.984 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:17.984 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:17.984 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 161122 2024-09-21T15:51:17.984 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.984 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:17.984 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.984 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:17.984 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.703081+0000", 2024-09-21T15:51:17.984 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:17.985 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.985 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.985 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:17.985 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.703081+0000", 2024-09-21T15:51:17.985 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:17.985 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.985 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.985 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:17.985 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.703079+0000", 2024-09-21T15:51:17.985 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-09-21T15:51:17.985 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.985 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.985 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:17.986 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.703091+0000", 2024-09-21T15:51:17.986 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2584e-05 2024-09-21T15:51:17.986 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.986 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.986 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:17.986 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.703107+0000", 2024-09-21T15:51:17.986 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6313000000000001e-05 2024-09-21T15:51:17.986 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.986 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.986 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:17.986 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.703119+0000", 2024-09-21T15:51:17.986 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1404e-05 2024-09-21T15:51:17.986 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.986 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.987 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:17.987 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.703176+0000", 2024-09-21T15:51:17.987 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6740999999999998e-05 2024-09-21T15:51:17.987 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.987 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.987 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:17.987 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.703281+0000", 2024-09-21T15:51:17.987 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010508 2024-09-21T15:51:17.987 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.987 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.987 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-09-21T15:51:17.987 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.703483+0000", 2024-09-21T15:51:17.988 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00020257399999999999 2024-09-21T15:51:17.988 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.988 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.988 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:17.988 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.705657+0000", 2024-09-21T15:51:17.988 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0021735610000000001 2024-09-21T15:51:17.988 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.988 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.988 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:17.988 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.708144+0000", 2024-09-21T15:51:17.988 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002487122 2024-09-21T15:51:17.988 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.988 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.988 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:17.988 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.708177+0000", 2024-09-21T15:51:17.989 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2574999999999999e-05 2024-09-21T15:51:17.989 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.989 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.989 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:17.989 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.708202+0000", 2024-09-21T15:51:17.989 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5513000000000001e-05 2024-09-21T15:51:17.989 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:17.989 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:17.989 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:17.989 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.989 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.989 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:161219 3.17 3:e87cb47d:::rbd_data.11335aef9442.000000000000000a:head [delete] snapc 2a=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-21T15:51:17.989 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:36.771292+0000", 2024-09-21T15:51:17.989 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 41.162673914999999, 2024-09-21T15:51:17.989 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0058540529999999997, 2024-09-21T15:51:17.990 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:17.990 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:17.990 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:17.990 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:17.990 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:17.990 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 161219 2024-09-21T15:51:17.990 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.990 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:17.990 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.990 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:17.990 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771292+0000", 2024-09-21T15:51:17.990 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:17.990 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.990 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.991 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:17.991 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771292+0000", 2024-09-21T15:51:17.991 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:17.991 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.991 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.991 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:17.991 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771292+0000", 2024-09-21T15:51:17.991 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:17.991 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.991 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.992 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:17.992 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771298+0000", 2024-09-21T15:51:17.992 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9519999999999999e-06 2024-09-21T15:51:17.992 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.992 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.992 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:17.992 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771308+0000", 2024-09-21T15:51:17.992 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0601000000000001e-05 2024-09-21T15:51:17.992 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.992 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.992 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:17.992 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771314+0000", 2024-09-21T15:51:17.992 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.913e-06 2024-09-21T15:51:17.992 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.992 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.993 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:17.993 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771346+0000", 2024-09-21T15:51:17.993 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1711000000000002e-05 2024-09-21T15:51:17.993 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.993 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.993 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:17.993 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771399+0000", 2024-09-21T15:51:17.993 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2883000000000002e-05 2024-09-21T15:51:17.993 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.993 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.993 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-09-21T15:51:17.993 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771470+0000", 2024-09-21T15:51:17.993 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0934999999999998e-05 2024-09-21T15:51:17.993 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.993 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.994 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:17.994 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.772234+0000", 2024-09-21T15:51:17.994 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00076485400000000003 2024-09-21T15:51:17.994 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.994 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.994 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:17.994 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.777083+0000", 2024-09-21T15:51:17.994 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0048486329999999998 2024-09-21T15:51:17.994 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.994 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.994 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:17.994 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.777111+0000", 2024-09-21T15:51:17.994 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8399999999999999e-05 2024-09-21T15:51:17.994 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.995 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.995 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:17.995 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.777147+0000", 2024-09-21T15:51:17.995 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5049000000000003e-05 2024-09-21T15:51:17.995 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:17.995 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:17.995 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:17.995 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.995 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.995 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:161859 3.17 3:ee7ab022:::rbd_data.11338db70e6f.0000000000000007:head [delete] snapc 26=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T15:51:17.995 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:40.782264+0000", 2024-09-21T15:51:17.995 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 37.151702055999998, 2024-09-21T15:51:17.995 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052663939999999998, 2024-09-21T15:51:17.996 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:17.996 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:17.996 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:17.996 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:17.996 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:17.996 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 161859 2024-09-21T15:51:17.996 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.996 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:17.996 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.996 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:17.996 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782264+0000", 2024-09-21T15:51:17.996 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:17.996 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.997 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.997 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:17.997 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782264+0000", 2024-09-21T15:51:17.997 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:17.997 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.997 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.997 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:17.997 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782264+0000", 2024-09-21T15:51:17.997 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:17.997 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.997 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.997 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:17.997 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782266+0000", 2024-09-21T15:51:17.997 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4590000000000001e-06 2024-09-21T15:51:17.997 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.998 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.998 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:17.998 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782275+0000", 2024-09-21T15:51:17.998 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5450000000000003e-06 2024-09-21T15:51:17.998 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.998 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.998 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:17.998 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782281+0000", 2024-09-21T15:51:17.998 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7180000000000001e-06 2024-09-21T15:51:17.998 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.998 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.998 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:17.998 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782312+0000", 2024-09-21T15:51:17.999 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1059000000000002e-05 2024-09-21T15:51:17.999 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.999 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.999 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:17.999 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782470+0000", 2024-09-21T15:51:17.999 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000157818 2024-09-21T15:51:17.999 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.999 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.999 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-09-21T15:51:17.999 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782574+0000", 2024-09-21T15:51:17.999 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000104536 2024-09-21T15:51:17.999 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:17.999 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:17.999 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:18.000 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.783499+0000", 2024-09-21T15:51:18.000 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00092479599999999997 2024-09-21T15:51:18.000 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.000 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.000 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:18.000 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.787472+0000", 2024-09-21T15:51:18.000 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0039730319999999996 2024-09-21T15:51:18.000 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.000 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.000 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:18.000 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.787507+0000", 2024-09-21T15:51:18.000 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5552000000000001e-05 2024-09-21T15:51:18.000 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.000 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.000 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:18.000 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.787531+0000", 2024-09-21T15:51:18.001 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3281999999999998e-05 2024-09-21T15:51:18.001 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:18.001 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:18.001 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:18.001 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.001 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.001 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:162390 3.17 3:ea8605c7:::rbd_header.1133fc941201:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e109)", 2024-09-21T15:51:18.001 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:44.740587+0000", 2024-09-21T15:51:18.001 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 33.193379460000003, 2024-09-21T15:51:18.001 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052218530000000003, 2024-09-21T15:51:18.001 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:18.001 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:18.001 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:18.001 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:18.001 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:18.002 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 162390 2024-09-21T15:51:18.002 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.002 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:18.002 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.002 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:18.002 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.740587+0000", 2024-09-21T15:51:18.002 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:18.002 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.002 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.002 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:18.002 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.740587+0000", 2024-09-21T15:51:18.002 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:18.002 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.002 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.003 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:18.003 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.740585+0000", 2024-09-21T15:51:18.003 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:51:18.003 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.003 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.003 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:18.003 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.740595+0000", 2024-09-21T15:51:18.003 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7839999999999998e-06 2024-09-21T15:51:18.003 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.003 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.003 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:18.003 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.740609+0000", 2024-09-21T15:51:18.003 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.384e-05 2024-09-21T15:51:18.003 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.004 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.004 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:18.004 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.740617+0000", 2024-09-21T15:51:18.004 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7560000000000005e-06 2024-09-21T15:51:18.004 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.004 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.004 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:18.004 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.740671+0000", 2024-09-21T15:51:18.004 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4073999999999998e-05 2024-09-21T15:51:18.004 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.004 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.004 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:18.004 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.740799+0000", 2024-09-21T15:51:18.004 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000128057 2024-09-21T15:51:18.004 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.005 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.005 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-09-21T15:51:18.005 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.741258+0000", 2024-09-21T15:51:18.005 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00045892900000000001 2024-09-21T15:51:18.005 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.005 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.005 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:18.005 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.742101+0000", 2024-09-21T15:51:18.005 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00084276499999999999 2024-09-21T15:51:18.005 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.005 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.005 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:18.005 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.745743+0000", 2024-09-21T15:51:18.005 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0036415280000000002 2024-09-21T15:51:18.006 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.006 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.006 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:18.006 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.745781+0000", 2024-09-21T15:51:18.006 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8398000000000001e-05 2024-09-21T15:51:18.006 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.006 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.006 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:18.006 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:44.745809+0000", 2024-09-21T15:51:18.006 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7691999999999998e-05 2024-09-21T15:51:18.006 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:18.006 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:18.006 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:18.007 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.007 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.007 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:163005 3.6 3:67cae2ed:::rbd_mirroring:head [call rbd.mirror_image_remove in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-21T15:51:18.007 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:49.711037+0000", 2024-09-21T15:51:18.007 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 28.222929036, 2024-09-21T15:51:18.007 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0059230919999999996, 2024-09-21T15:51:18.007 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:18.007 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:18.007 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:18.007 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:18.007 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:18.007 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 163005 2024-09-21T15:51:18.007 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.007 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:18.007 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.008 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:18.008 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:49.711037+0000", 2024-09-21T15:51:18.008 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:18.008 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.008 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.008 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:18.008 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:49.711037+0000", 2024-09-21T15:51:18.008 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:18.008 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.008 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.008 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:18.008 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:49.711037+0000", 2024-09-21T15:51:18.008 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:18.008 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.009 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.009 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:18.009 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:49.711041+0000", 2024-09-21T15:51:18.009 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8199999999999996e-06 2024-09-21T15:51:18.009 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.009 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.009 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:18.009 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:49.711052+0000", 2024-09-21T15:51:18.009 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0567999999999999e-05 2024-09-21T15:51:18.009 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.009 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.009 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:18.009 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:49.711057+0000", 2024-09-21T15:51:18.009 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7119999999999999e-06 2024-09-21T15:51:18.010 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.010 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.010 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:18.010 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:49.711096+0000", 2024-09-21T15:51:18.010 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8872999999999999e-05 2024-09-21T15:51:18.010 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.010 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.010 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:18.010 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:49.711151+0000", 2024-09-21T15:51:18.010 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5343999999999997e-05 2024-09-21T15:51:18.010 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.011 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.011 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:18.011 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:49.716960+0000", 2024-09-21T15:51:18.011 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0058095150000000003 2024-09-21T15:51:18.011 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:18.011 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:18.011 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:18.012 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.012 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.012 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:165147 3.10 e131/96 3:0b23e447:::rbd_header.11339e87a32a:head v 131'448, mlcod=130'446)", 2024-09-21T15:51:18.012 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:51:06.827072+0000", 2024-09-21T15:51:18.012 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 11.106894252, 2024-09-21T15:51:18.012 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.011683913000000001, 2024-09-21T15:51:18.013 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:18.013 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:18.013 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:18.013 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.013 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:18.013 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.827072+0000", 2024-09-21T15:51:18.013 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:18.013 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.014 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.014 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:18.014 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.827072+0000", 2024-09-21T15:51:18.014 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:18.014 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.014 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.014 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:18.014 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.827072+0000", 2024-09-21T15:51:18.014 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:18.014 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.015 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.015 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:18.015 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.827078+0000", 2024-09-21T15:51:18.015 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4840000000000001e-06 2024-09-21T15:51:18.015 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.015 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.015 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:18.015 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.827094+0000", 2024-09-21T15:51:18.015 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5639999999999999e-05 2024-09-21T15:51:18.015 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.015 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.015 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:18.015 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.827104+0000", 2024-09-21T15:51:18.015 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.8300000000000008e-06 2024-09-21T15:51:18.016 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.016 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.016 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:18.016 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.827170+0000", 2024-09-21T15:51:18.016 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5913000000000004e-05 2024-09-21T15:51:18.016 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.016 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.016 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:18.016 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.827196+0000", 2024-09-21T15:51:18.016 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6210000000000001e-05 2024-09-21T15:51:18.016 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.016 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.016 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:18.016 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.838689+0000", 2024-09-21T15:51:18.017 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.011492979 2024-09-21T15:51:18.017 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.017 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.017 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:18.017 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.838756+0000", 2024-09-21T15:51:18.017 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7155000000000003e-05 2024-09-21T15:51:18.017 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:18.017 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:18.017 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:18.017 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.017 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.017 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.0:3709 3.3 e131/96 3:c5f4482f:::rbd_data.11339e87a32a.000000000000013a:12 v 131'841, mlcod=130'838)", 2024-09-21T15:51:18.017 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:51:06.828155+0000", 2024-09-21T15:51:18.017 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 11.105811810000001, 2024-09-21T15:51:18.017 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.010631115999999999, 2024-09-21T15:51:18.018 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:18.018 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:18.018 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:18.018 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.018 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:18.018 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.828155+0000", 2024-09-21T15:51:18.018 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:18.018 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.018 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.018 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:18.018 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.828155+0000", 2024-09-21T15:51:18.018 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:18.018 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.018 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.019 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:18.019 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.828154+0000", 2024-09-21T15:51:18.019 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:18.019 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.019 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.019 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:18.019 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.828162+0000", 2024-09-21T15:51:18.019 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.2269999999999993e-06 2024-09-21T15:51:18.019 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.019 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.019 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:18.019 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.828188+0000", 2024-09-21T15:51:18.019 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5626000000000002e-05 2024-09-21T15:51:18.019 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.019 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.020 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:18.020 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.828210+0000", 2024-09-21T15:51:18.020 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1792000000000001e-05 2024-09-21T15:51:18.020 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.020 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.020 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:18.020 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.828251+0000", 2024-09-21T15:51:18.020 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1329999999999999e-05 2024-09-21T15:51:18.020 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.020 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.020 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:18.020 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.828277+0000", 2024-09-21T15:51:18.020 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6236000000000001e-05 2024-09-21T15:51:18.020 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.020 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.021 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:18.021 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.838756+0000", 2024-09-21T15:51:18.021 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.010478919 2024-09-21T15:51:18.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.021 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.021 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:18.021 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.838786+0000", 2024-09-21T15:51:18.021 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9560000000000002e-05 2024-09-21T15:51:18.021 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:18.021 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:18.021 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:18.021 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.021 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.021 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.0:3710 3.3 e131/96 3:c7ccd1db:::rbd_data.11339e87a32a.0000000000000136:12 v 131'843, mlcod=130'838)", 2024-09-21T15:51:18.022 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:51:06.831274+0000", 2024-09-21T15:51:18.022 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 11.10269196, 2024-09-21T15:51:18.022 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0077927079999999998, 2024-09-21T15:51:18.022 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:18.022 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:18.022 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:18.022 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.022 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:18.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.831274+0000", 2024-09-21T15:51:18.022 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:18.022 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.022 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.022 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:18.022 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.831274+0000", 2024-09-21T15:51:18.023 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:18.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.023 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.023 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:18.023 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.831274+0000", 2024-09-21T15:51:18.023 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:18.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.023 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.023 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:18.023 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.831286+0000", 2024-09-21T15:51:18.023 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1853e-05 2024-09-21T15:51:18.023 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.023 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.023 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:18.023 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.831296+0000", 2024-09-21T15:51:18.024 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0713999999999999e-05 2024-09-21T15:51:18.024 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.024 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.024 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:18.024 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.831305+0000", 2024-09-21T15:51:18.024 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4749999999999993e-06 2024-09-21T15:51:18.024 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.024 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.024 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:18.024 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.831364+0000", 2024-09-21T15:51:18.024 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9320999999999997e-05 2024-09-21T15:51:18.024 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.024 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.024 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:18.025 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.831386+0000", 2024-09-21T15:51:18.025 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2058000000000001e-05 2024-09-21T15:51:18.025 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.025 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.025 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:18.025 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.839039+0000", 2024-09-21T15:51:18.025 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0076530490000000003 2024-09-21T15:51:18.025 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.025 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.025 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:18.025 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.839067+0000", 2024-09-21T15:51:18.025 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8005999999999999e-05 2024-09-21T15:51:18.026 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:18.026 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:18.026 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:18.026 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.026 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.026 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.0:3813 3.3 e140/96 3:c72d6995:::rbd_data.1133fed1a840.0000000000000113:d v 140'872, mlcod=139'867)", 2024-09-21T15:51:18.026 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:51:15.847652+0000", 2024-09-21T15:51:18.026 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 2.0863147930000001, 2024-09-21T15:51:18.026 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0051062149999999999, 2024-09-21T15:51:18.026 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:18.026 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:18.026 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:18.026 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.026 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:18.026 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:15.847652+0000", 2024-09-21T15:51:18.027 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:18.027 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.027 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.027 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:18.027 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:15.847652+0000", 2024-09-21T15:51:18.027 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:18.027 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.027 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.027 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:18.027 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:15.847651+0000", 2024-09-21T15:51:18.027 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:18.027 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.027 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.027 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:18.027 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:15.847658+0000", 2024-09-21T15:51:18.028 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8600000000000004e-06 2024-09-21T15:51:18.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.028 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.028 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:18.028 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:15.847668+0000", 2024-09-21T15:51:18.028 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.011e-05 2024-09-21T15:51:18.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.028 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.028 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:18.028 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:15.847675+0000", 2024-09-21T15:51:18.028 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8990000000000004e-06 2024-09-21T15:51:18.028 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.028 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.028 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:18.029 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:15.847782+0000", 2024-09-21T15:51:18.029 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000107496 2024-09-21T15:51:18.029 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.029 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.029 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:18.029 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:15.847801+0000", 2024-09-21T15:51:18.029 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8320000000000001e-05 2024-09-21T15:51:18.029 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.029 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.029 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:18.029 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:15.852718+0000", 2024-09-21T15:51:18.029 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0049177600000000002 2024-09-21T15:51:18.029 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.030 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.030 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:18.030 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:15.852758+0000", 2024-09-21T15:51:18.030 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9481999999999999e-05 2024-09-21T15:51:18.030 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:18.030 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:18.030 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:18.030 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.030 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.030 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:166306 3.1 3:82f7f77c:::rbd_header.11336b2fdf4a:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-09-21T15:51:18.030 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:51:16.854113+0000", 2024-09-21T15:51:18.030 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.079853194, 2024-09-21T15:51:18.030 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0076607890000000003, 2024-09-21T15:51:18.030 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:18.031 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:18.031 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:18.031 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:18.031 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:18.031 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 166306 2024-09-21T15:51:18.031 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.031 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:18.031 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.031 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:18.031 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.854113+0000", 2024-09-21T15:51:18.031 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:18.031 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.031 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.031 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:18.032 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.854113+0000", 2024-09-21T15:51:18.032 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:18.032 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.032 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.032 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:18.032 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.854111+0000", 2024-09-21T15:51:18.032 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:51:18.032 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.032 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.032 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:18.032 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.854120+0000", 2024-09-21T15:51:18.032 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8410000000000006e-06 2024-09-21T15:51:18.032 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.032 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.033 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:18.033 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.854133+0000", 2024-09-21T15:51:18.033 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.272e-05 2024-09-21T15:51:18.033 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.033 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.033 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:18.033 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.854141+0000", 2024-09-21T15:51:18.033 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8590000000000005e-06 2024-09-21T15:51:18.033 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.033 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.033 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:18.034 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.854207+0000", 2024-09-21T15:51:18.034 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6593000000000004e-05 2024-09-21T15:51:18.034 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.034 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.034 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:18.034 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.854308+0000", 2024-09-21T15:51:18.034 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000100592 2024-09-21T15:51:18.034 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.035 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-09-21T15:51:18.035 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.854716+0000", 2024-09-21T15:51:18.035 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00040786099999999998 2024-09-21T15:51:18.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.035 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:18.035 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.858074+0000", 2024-09-21T15:51:18.035 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00335798 2024-09-21T15:51:18.035 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.035 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.035 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:18.035 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.861691+0000", 2024-09-21T15:51:18.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003617351 2024-09-21T15:51:18.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.036 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:18.036 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.861740+0000", 2024-09-21T15:51:18.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8618000000000001e-05 2024-09-21T15:51:18.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.036 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.036 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:18.036 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.861774+0000", 2024-09-21T15:51:18.036 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4313000000000002e-05 2024-09-21T15:51:18.036 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:18.036 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:18.036 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:18.036 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.037 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.0:3832 3.3 e141/96 3:c4381677:::rbd_data.11336b2fdf4a.000000000000000e:c v 141'878, mlcod=140'875)", 2024-09-21T15:51:18.037 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:51:16.856401+0000", 2024-09-21T15:51:18.037 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.077564993, 2024-09-21T15:51:18.037 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0061803919999999998, 2024-09-21T15:51:18.037 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:18.037 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:18.037 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:18.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.037 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:18.037 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.856401+0000", 2024-09-21T15:51:18.037 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:18.037 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.037 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.038 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:18.038 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.856401+0000", 2024-09-21T15:51:18.038 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:18.038 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.038 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.038 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:18.038 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.856400+0000", 2024-09-21T15:51:18.038 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:18.038 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.038 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.038 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:18.038 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.856407+0000", 2024-09-21T15:51:18.038 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9079999999999998e-06 2024-09-21T15:51:18.038 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.039 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.039 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:18.039 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.856417+0000", 2024-09-21T15:51:18.039 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5929999999999995e-06 2024-09-21T15:51:18.039 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.039 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.039 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:18.039 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.856421+0000", 2024-09-21T15:51:18.039 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6829999999999999e-06 2024-09-21T15:51:18.039 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.039 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.039 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:18.039 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.856455+0000", 2024-09-21T15:51:18.039 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3145000000000002e-05 2024-09-21T15:51:18.039 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.040 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:18.040 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.856481+0000", 2024-09-21T15:51:18.040 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5959999999999999e-05 2024-09-21T15:51:18.040 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.040 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:18.040 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.862548+0000", 2024-09-21T15:51:18.040 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0060674869999999999 2024-09-21T15:51:18.040 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.040 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.040 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:18.040 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.862582+0000", 2024-09-21T15:51:18.040 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3726000000000002e-05 2024-09-21T15:51:18.040 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:18.041 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:18.041 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:18.041 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.041 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.041 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:166310 3.1 3:82f7f77c:::rbd_header.11336b2fdf4a:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-09-21T15:51:18.041 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:51:16.867249+0000", 2024-09-21T15:51:18.041 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 1.0667171019999999, 2024-09-21T15:51:18.041 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0051734909999999997, 2024-09-21T15:51:18.041 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:18.041 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:18.041 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:18.041 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:18.041 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:18.041 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 166310 2024-09-21T15:51:18.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.042 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:18.042 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.042 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:18.042 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.867249+0000", 2024-09-21T15:51:18.042 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:18.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.042 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.042 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:18.042 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.867249+0000", 2024-09-21T15:51:18.042 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:18.042 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.042 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.043 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:18.043 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.867248+0000", 2024-09-21T15:51:18.043 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:51:18.043 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.043 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.043 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:18.043 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.867255+0000", 2024-09-21T15:51:18.043 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4490000000000002e-06 2024-09-21T15:51:18.043 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.043 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.043 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:18.043 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.867268+0000", 2024-09-21T15:51:18.043 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2456999999999999e-05 2024-09-21T15:51:18.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.044 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:18.044 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.867275+0000", 2024-09-21T15:51:18.044 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3339999999999996e-06 2024-09-21T15:51:18.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.044 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.044 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:18.044 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.867322+0000", 2024-09-21T15:51:18.044 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6937000000000001e-05 2024-09-21T15:51:18.044 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:18.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.867415+0000", 2024-09-21T15:51:18.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2712000000000003e-05 2024-09-21T15:51:18.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-09-21T15:51:18.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.867600+0000", 2024-09-21T15:51:18.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00018562200000000001 2024-09-21T15:51:18.045 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.045 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.045 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:18.045 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.868649+0000", 2024-09-21T15:51:18.045 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0010483999999999999 2024-09-21T15:51:18.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.046 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:18.046 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.872347+0000", 2024-09-21T15:51:18.046 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0036979970000000002 2024-09-21T15:51:18.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.046 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:18.046 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.872396+0000", 2024-09-21T15:51:18.046 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9094999999999997e-05 2024-09-21T15:51:18.046 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:18.046 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:18.047 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:18.047 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.872423+0000", 2024-09-21T15:51:18.047 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7038000000000001e-05 2024-09-21T15:51:18.047 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:18.047 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:18.047 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:18.047 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:18.047 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:18.047 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:18.048 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:51:18.049 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:51:18.048+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:51:18.067 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:51:18.148 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:51:18.147+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:51:18.154 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:18.155 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:51:18.155 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:51:18.155 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:18.164 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:51:18.213 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-09-21T15:51:18.213 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:51:18.230 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:51:18.230 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:51:18.246 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:51:18.246 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:51:18.249 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:51:18.249+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:51:18.262 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:51:18.262 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:51:18.278 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:51:18.278 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:51:18.295 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:51:18.295 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:51:18.308 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:18.308 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:18.311 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:51:18.311 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:51:18.317 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:51:18.328 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:51:18.328 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:51:18.350 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:51:18.350+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:51:18.430 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:18.430 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:18.438 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:51:18.451 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:51:18.450+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:51:18.540 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:18.540 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:18.549 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:51:18.552 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:51:18.551+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:51:18.651 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:18.651 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:18.652 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:51:18.651+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:51:18.662 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:51:18.678 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is set 2024-09-21T15:51:18.753 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:51:18.752+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:51:18.779 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:18.779 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:18.789 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:51:18.854 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:51:18.854+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:51:18.894 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:18.894 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:18.903 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:51:18.955 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:51:18.954+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:51:19.005 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:19.006 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:19.014 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:51:19.056 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:51:19.054+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:51:19.156 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:19.157 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:19.157 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:51:19.155+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:51:19.165 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:51:19.258 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:51:19.256+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:51:19.269 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:19.269 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:19.278 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:51:19.358 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:51:19.357+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:51:19.380 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:19.380 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:19.389 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:51:19.459 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:51:19.458+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:51:19.491 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:19.491 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:19.499 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:51:19.560 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:51:19.559+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:51:19.641 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:19.641 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:19.651 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:51:19.694 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:51:19.693+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:51:19.761 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:51:19.760+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:51:19.762 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:19.762 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:19.772 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:51:19.862 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:51:19.860+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:51:19.915 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:19.915 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:19.924 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T15:51:19.967 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:51:19.967+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:51:20.026 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:20.026 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:20.035 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T15:51:20.076 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:51:20.076+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:51:20.164 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:51:20.163+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:51:20.177 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:20.177 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:20.185 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T15:51:20.265 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:51:20.264+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:51:20.287 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:20.287 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:20.297 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T15:51:20.338 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:51:20.382 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:51:20.381+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:51:20.399 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:20.399 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:20.408 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T15:51:20.458 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-09-21T15:51:20.458 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:51:20.467 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:51:20.466+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:51:20.474 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:51:20.474 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:51:20.489 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:51:20.490 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:51:20.505 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:51:20.505 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:51:20.510 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:20.510 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:20.518 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T15:51:20.521 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:51:20.521 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:51:20.537 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:51:20.537 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:51:20.553 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:51:20.553 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:51:20.568 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:51:20.568+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:51:20.569 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:51:20.569 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:51:20.622 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:51:20.622 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:51:20.622 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:51:20.622 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:51:20.622 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.622 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4209.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:51:20.622 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:51:20.622 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 149.42895402600001, 2024-09-21T15:51:20.622 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.19452909900000001, 2024-09-21T15:51:20.622 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:20.622 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:20.622 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:20.622 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4209", 2024-09-21T15:51:20.623 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/4045143050", 2024-09-21T15:51:20.623 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1 2024-09-21T15:51:20.623 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.623 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:20.623 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.623 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:20.623 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:51:20.623 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:20.623 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.623 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.623 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:20.623 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:51:20.623 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:20.623 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.623 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.624 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:20.624 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191161+0000", 2024-09-21T15:51:20.624 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-21T15:51:20.624 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.624 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.624 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:20.624 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191169+0000", 2024-09-21T15:51:20.624 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0090000000000007e-06 2024-09-21T15:51:20.624 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.624 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.624 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:20.624 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191180+0000", 2024-09-21T15:51:20.624 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0728e-05 2024-09-21T15:51:20.624 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.624 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.625 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:20.625 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191187+0000", 2024-09-21T15:51:20.625 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9009999999999994e-06 2024-09-21T15:51:20.625 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.625 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.625 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:20.625 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191238+0000", 2024-09-21T15:51:20.625 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0287000000000002e-05 2024-09-21T15:51:20.625 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.625 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.625 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-21T15:51:20.625 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191245+0000", 2024-09-21T15:51:20.625 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0709999999999997e-06 2024-09-21T15:51:20.625 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.625 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.625 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:20.626 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380309+0000", 2024-09-21T15:51:20.626 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.189064277 2024-09-21T15:51:20.626 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.626 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.626 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:20.626 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380454+0000", 2024-09-21T15:51:20.626 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014490799999999999 2024-09-21T15:51:20.626 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.626 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.626 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0", 2024-09-21T15:51:20.626 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380610+0000", 2024-09-21T15:51:20.626 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015650199999999999 2024-09-21T15:51:20.626 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.626 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.626 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:51:20.627 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.381160+0000", 2024-09-21T15:51:20.627 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00054903799999999996 2024-09-21T15:51:20.627 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.627 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.627 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:20.627 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385591+0000", 2024-09-21T15:51:20.627 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.004431272 2024-09-21T15:51:20.627 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.627 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.627 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:51:20.627 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385651+0000", 2024-09-21T15:51:20.627 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0455999999999999e-05 2024-09-21T15:51:20.627 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.627 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.627 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:20.627 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385691+0000", 2024-09-21T15:51:20.628 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0234999999999997e-05 2024-09-21T15:51:20.628 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:20.628 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:20.628 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:20.628 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.628 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.628 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64034 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1083392~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:20.628 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167717+0000", 2024-09-21T15:51:20.628 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 103.452399696, 2024-09-21T15:51:20.628 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048233316999999998, 2024-09-21T15:51:20.628 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:20.628 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:20.628 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:20.628 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:20.629 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:20.629 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64034 2024-09-21T15:51:20.629 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.629 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:20.629 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.629 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:20.629 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167717+0000", 2024-09-21T15:51:20.629 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:20.629 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.629 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.629 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:20.629 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167717+0000", 2024-09-21T15:51:20.629 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:20.629 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.629 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.630 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:20.630 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167716+0000", 2024-09-21T15:51:20.630 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:20.630 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.630 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.630 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:20.630 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167718+0000", 2024-09-21T15:51:20.630 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.165e-06 2024-09-21T15:51:20.630 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.630 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.630 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:20.630 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167725+0000", 2024-09-21T15:51:20.630 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3380000000000001e-06 2024-09-21T15:51:20.630 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.630 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.631 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:20.631 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167730+0000", 2024-09-21T15:51:20.631 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8010000000000003e-06 2024-09-21T15:51:20.631 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.631 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.631 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:20.631 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215807+0000", 2024-09-21T15:51:20.631 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048077823999999998 2024-09-21T15:51:20.631 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.631 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.631 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:20.631 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215884+0000", 2024-09-21T15:51:20.631 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6283000000000001e-05 2024-09-21T15:51:20.631 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.631 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.631 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:20.631 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215950+0000", 2024-09-21T15:51:20.632 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6290000000000004e-05 2024-09-21T15:51:20.632 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:20.632 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:20.632 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:20.632 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.632 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.632 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64035 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1091584~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:20.632 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:51:20.632 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 103.45238248299999, 2024-09-21T15:51:20.632 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048412117999999997, 2024-09-21T15:51:20.632 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:20.632 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:20.632 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:20.632 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:20.632 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:20.633 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64035 2024-09-21T15:51:20.633 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.633 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:20.633 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.633 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:20.633 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:51:20.633 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:20.633 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.633 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.633 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:20.633 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:51:20.633 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:20.633 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.633 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.633 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:20.634 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:51:20.634 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:20.634 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.634 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.634 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:20.634 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167746+0000", 2024-09-21T15:51:20.634 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2031000000000001e-05 2024-09-21T15:51:20.634 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.634 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.634 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:20.634 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167753+0000", 2024-09-21T15:51:20.634 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0620000000000003e-06 2024-09-21T15:51:20.634 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.634 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.635 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:20.635 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167759+0000", 2024-09-21T15:51:20.635 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.596e-06 2024-09-21T15:51:20.635 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.635 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.635 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:20.635 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215977+0000", 2024-09-21T15:51:20.635 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048217318000000002 2024-09-21T15:51:20.635 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.635 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.635 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:20.635 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216076+0000", 2024-09-21T15:51:20.635 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9846000000000006e-05 2024-09-21T15:51:20.635 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.636 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.636 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:20.636 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216146+0000", 2024-09-21T15:51:20.636 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9578000000000004e-05 2024-09-21T15:51:20.636 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:20.636 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:20.636 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:20.636 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.636 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.636 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64036 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1099776~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:20.636 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167853+0000", 2024-09-21T15:51:20.636 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 103.45226387, 2024-09-21T15:51:20.636 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048450675999999998, 2024-09-21T15:51:20.636 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:20.637 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:20.637 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:20.637 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:20.637 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:20.637 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64036 2024-09-21T15:51:20.637 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.637 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:20.637 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.637 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:20.637 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167853+0000", 2024-09-21T15:51:20.637 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:20.638 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.638 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.638 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:20.638 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167853+0000", 2024-09-21T15:51:20.638 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:20.638 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.638 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.638 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:20.638 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167852+0000", 2024-09-21T15:51:20.638 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:20.638 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.638 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.638 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:20.638 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167855+0000", 2024-09-21T15:51:20.638 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5640000000000001e-06 2024-09-21T15:51:20.639 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.639 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.639 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:20.639 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167864+0000", 2024-09-21T15:51:20.639 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8140000000000007e-06 2024-09-21T15:51:20.639 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.639 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.639 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:20.639 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167871+0000", 2024-09-21T15:51:20.639 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5729999999999996e-06 2024-09-21T15:51:20.639 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.639 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.639 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:20.639 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216174+0000", 2024-09-21T15:51:20.639 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048303437999999997 2024-09-21T15:51:20.640 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.640 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.640 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:20.640 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216250+0000", 2024-09-21T15:51:20.640 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5661999999999995e-05 2024-09-21T15:51:20.640 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.640 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.640 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:20.640 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216303+0000", 2024-09-21T15:51:20.640 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3273000000000003e-05 2024-09-21T15:51:20.640 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:20.640 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:20.640 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:20.640 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.640 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.641 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64037 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1771520~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:20.641 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:51:20.641 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 103.452240327, 2024-09-21T15:51:20.641 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048588771000000003, 2024-09-21T15:51:20.641 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:20.641 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:20.641 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:20.641 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:20.641 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:20.641 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64037 2024-09-21T15:51:20.641 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.641 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:20.641 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.641 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:20.642 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:51:20.642 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:20.642 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.642 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.642 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:20.642 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:51:20.642 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:20.642 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.642 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.642 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:20.642 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:51:20.642 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:20.642 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.642 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.642 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:20.643 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167878+0000", 2024-09-21T15:51:20.643 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3e-06 2024-09-21T15:51:20.643 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.643 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.643 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:20.643 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167884+0000", 2024-09-21T15:51:20.643 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3269999999999997e-06 2024-09-21T15:51:20.643 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.643 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.643 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:20.643 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167889+0000", 2024-09-21T15:51:20.643 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9269999999999996e-06 2024-09-21T15:51:20.643 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.643 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.643 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:20.644 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216331+0000", 2024-09-21T15:51:20.644 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048442102000000001 2024-09-21T15:51:20.644 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.644 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.644 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:20.644 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216393+0000", 2024-09-21T15:51:20.644 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1132999999999999e-05 2024-09-21T15:51:20.644 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.644 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.644 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:20.644 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216465+0000", 2024-09-21T15:51:20.644 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2339e-05 2024-09-21T15:51:20.644 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:20.644 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:20.645 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:20.645 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.645 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.645 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64038 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1779712~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:20.645 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:51:20.645 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 103.452222496, 2024-09-21T15:51:20.645 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048749795999999998, 2024-09-21T15:51:20.645 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:20.645 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:20.645 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:20.645 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:20.645 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:20.645 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64038 2024-09-21T15:51:20.645 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.645 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:20.646 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.646 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:20.646 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:51:20.646 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:20.646 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.646 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.646 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:20.646 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:51:20.646 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:20.646 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.646 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.646 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:20.646 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:51:20.646 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:20.647 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.647 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.647 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:20.647 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167896+0000", 2024-09-21T15:51:20.647 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2419999999999999e-06 2024-09-21T15:51:20.647 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.647 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.647 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:20.647 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167902+0000", 2024-09-21T15:51:20.647 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0859999999999998e-06 2024-09-21T15:51:20.647 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.647 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.647 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:20.648 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167907+0000", 2024-09-21T15:51:20.648 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9679999999999997e-06 2024-09-21T15:51:20.648 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.648 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.648 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:20.648 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216506+0000", 2024-09-21T15:51:20.648 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048599374000000001 2024-09-21T15:51:20.648 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.648 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.648 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:20.648 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216579+0000", 2024-09-21T15:51:20.649 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2801999999999999e-05 2024-09-21T15:51:20.649 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.649 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.649 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:20.649 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216644+0000", 2024-09-21T15:51:20.649 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4715000000000006e-05 2024-09-21T15:51:20.649 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:20.649 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:20.649 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:20.649 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.649 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.649 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64039 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1107968~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:20.649 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:51:20.649 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 103.45220514099999, 2024-09-21T15:51:20.650 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048911370000000003, 2024-09-21T15:51:20.650 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:20.650 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:20.650 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:20.650 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:20.650 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:20.650 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64039 2024-09-21T15:51:20.650 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.650 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:20.650 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.650 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:20.650 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:51:20.650 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:20.650 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.650 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.651 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:20.651 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:51:20.651 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:20.651 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.651 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.651 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:20.651 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:51:20.651 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:20.651 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.651 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.651 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:20.651 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167913+0000", 2024-09-21T15:51:20.651 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0839999999999999e-06 2024-09-21T15:51:20.651 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.651 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.652 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:20.652 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167921+0000", 2024-09-21T15:51:20.652 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8310000000000001e-06 2024-09-21T15:51:20.652 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.652 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.652 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:20.652 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167926+0000", 2024-09-21T15:51:20.652 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0270000000000003e-06 2024-09-21T15:51:20.652 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.652 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.652 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:20.652 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216680+0000", 2024-09-21T15:51:20.652 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048753890000000001 2024-09-21T15:51:20.652 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.652 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.652 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:20.652 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216757+0000", 2024-09-21T15:51:20.653 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6848999999999996e-05 2024-09-21T15:51:20.653 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.653 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.653 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:20.653 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216823+0000", 2024-09-21T15:51:20.653 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6068000000000005e-05 2024-09-21T15:51:20.653 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:20.653 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:20.653 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:20.653 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.653 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.653 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64040 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1787904~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:20.653 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167931+0000", 2024-09-21T15:51:20.653 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 103.45218590899999, 2024-09-21T15:51:20.653 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049010561000000001, 2024-09-21T15:51:20.653 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:20.654 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:20.654 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:20.654 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:20.654 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:20.654 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64040 2024-09-21T15:51:20.654 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.654 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:20.654 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.654 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:20.654 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167931+0000", 2024-09-21T15:51:20.654 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:20.654 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.654 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.655 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:20.655 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167931+0000", 2024-09-21T15:51:20.655 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:20.655 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.655 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.655 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:20.655 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167930+0000", 2024-09-21T15:51:20.655 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:20.655 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.655 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.655 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:20.655 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167932+0000", 2024-09-21T15:51:20.655 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.198e-06 2024-09-21T15:51:20.655 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.655 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.656 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:20.656 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167938+0000", 2024-09-21T15:51:20.656 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.105e-06 2024-09-21T15:51:20.656 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.656 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.656 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:20.656 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167943+0000", 2024-09-21T15:51:20.656 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9429999999999999e-06 2024-09-21T15:51:20.656 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.656 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.656 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:20.656 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216848+0000", 2024-09-21T15:51:20.656 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048905200000000003 2024-09-21T15:51:20.656 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.656 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.657 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:20.657 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216903+0000", 2024-09-21T15:51:20.657 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4793999999999997e-05 2024-09-21T15:51:20.657 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.657 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.657 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:20.657 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216941+0000", 2024-09-21T15:51:20.657 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7790999999999998e-05 2024-09-21T15:51:20.657 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:20.657 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:20.657 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:20.657 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.657 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.657 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64041 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1796096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:20.657 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:51:20.657 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 103.452168938, 2024-09-21T15:51:20.658 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049181457999999997, 2024-09-21T15:51:20.658 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:20.658 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:20.658 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:20.658 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:20.658 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:20.658 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64041 2024-09-21T15:51:20.658 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.658 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:20.658 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.658 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:20.658 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:51:20.658 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:20.658 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.658 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.659 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:20.659 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:51:20.659 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:20.659 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.659 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.659 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:20.659 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:51:20.659 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:20.659 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.659 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.659 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:20.659 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167949+0000", 2024-09-21T15:51:20.659 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0200000000000001e-06 2024-09-21T15:51:20.659 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.660 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.660 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:20.660 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167955+0000", 2024-09-21T15:51:20.660 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5990000000000001e-06 2024-09-21T15:51:20.660 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.660 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.660 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:20.660 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167959+0000", 2024-09-21T15:51:20.660 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5589999999999999e-06 2024-09-21T15:51:20.660 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.660 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.660 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:20.660 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216972+0000", 2024-09-21T15:51:20.660 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049012574000000003 2024-09-21T15:51:20.660 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.661 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.661 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:20.661 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217057+0000", 2024-09-21T15:51:20.661 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4820000000000004e-05 2024-09-21T15:51:20.661 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.661 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.661 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:20.661 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217129+0000", 2024-09-21T15:51:20.661 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2186000000000003e-05 2024-09-21T15:51:20.661 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:20.661 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:20.661 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:20.661 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.661 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.661 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64042 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1804288~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:20.662 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167964+0000", 2024-09-21T15:51:20.662 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 103.45215264700001, 2024-09-21T15:51:20.662 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049333372, 2024-09-21T15:51:20.662 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:20.662 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:20.662 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:20.662 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:20.662 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:20.662 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64042 2024-09-21T15:51:20.662 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.662 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:20.662 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.662 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:20.662 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167964+0000", 2024-09-21T15:51:20.663 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:20.663 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.663 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.663 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:20.663 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167964+0000", 2024-09-21T15:51:20.663 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:20.663 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.663 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.663 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:20.663 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167963+0000", 2024-09-21T15:51:20.663 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:20.663 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.663 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.663 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:20.664 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167966+0000", 2024-09-21T15:51:20.664 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1119999999999999e-06 2024-09-21T15:51:20.664 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.664 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.664 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:20.664 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167975+0000", 2024-09-21T15:51:20.664 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1130000000000007e-06 2024-09-21T15:51:20.664 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.664 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.664 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:20.664 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167980+0000", 2024-09-21T15:51:20.664 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9599999999999999e-06 2024-09-21T15:51:20.664 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.664 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.664 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:20.665 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217158+0000", 2024-09-21T15:51:20.665 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049178569999999998 2024-09-21T15:51:20.665 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.665 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.665 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:20.665 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217232+0000", 2024-09-21T15:51:20.665 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3589000000000003e-05 2024-09-21T15:51:20.665 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.665 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.665 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:20.665 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217297+0000", 2024-09-21T15:51:20.665 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5339999999999994e-05 2024-09-21T15:51:20.665 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:20.666 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:20.666 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:20.666 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.666 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.666 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64043 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1812480~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:20.666 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.168085+0000", 2024-09-21T15:51:20.666 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 103.45203158, 2024-09-21T15:51:20.666 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049365157999999999, 2024-09-21T15:51:20.666 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:20.666 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:20.666 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:20.666 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:20.666 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:20.667 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64043 2024-09-21T15:51:20.667 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.667 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:20.667 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.667 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:20.667 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168085+0000", 2024-09-21T15:51:20.667 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:20.667 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.667 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.667 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:20.667 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168085+0000", 2024-09-21T15:51:20.667 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:20.667 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.668 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.668 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:20.668 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168084+0000", 2024-09-21T15:51:20.668 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:20.668 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.668 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.668 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:20.668 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168088+0000", 2024-09-21T15:51:20.668 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5420000000000001e-06 2024-09-21T15:51:20.668 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.668 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.668 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:20.668 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168097+0000", 2024-09-21T15:51:20.668 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5410000000000006e-06 2024-09-21T15:51:20.668 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.669 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.669 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:20.669 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168104+0000", 2024-09-21T15:51:20.669 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6719999999999998e-06 2024-09-21T15:51:20.669 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.669 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.669 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:20.669 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217323+0000", 2024-09-21T15:51:20.669 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049218857999999997 2024-09-21T15:51:20.669 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.669 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.669 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:20.669 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217397+0000", 2024-09-21T15:51:20.670 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3932999999999998e-05 2024-09-21T15:51:20.670 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.670 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.670 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:20.670 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217450+0000", 2024-09-21T15:51:20.670 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3168000000000003e-05 2024-09-21T15:51:20.670 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:20.670 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:20.670 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:20.670 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.670 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.670 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155255 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 323584~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:20.671 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.440996+0000", 2024-09-21T15:51:20.671 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 58.179120052999998, 2024-09-21T15:51:20.671 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048233848000000003, 2024-09-21T15:51:20.671 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:20.671 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:20.671 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:20.671 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:20.671 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:20.671 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155255 2024-09-21T15:51:20.671 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.671 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:20.671 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.671 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:20.671 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.440996+0000", 2024-09-21T15:51:20.672 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:20.672 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.672 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.672 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:20.672 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.440996+0000", 2024-09-21T15:51:20.672 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:20.672 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.672 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.672 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:20.672 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.440996+0000", 2024-09-21T15:51:20.672 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:20.672 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.672 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.672 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:20.672 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441004+0000", 2024-09-21T15:51:20.672 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8460000000000008e-06 2024-09-21T15:51:20.673 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.673 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.673 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:20.673 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441018+0000", 2024-09-21T15:51:20.673 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.42e-05 2024-09-21T15:51:20.673 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.673 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.673 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:20.673 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441023+0000", 2024-09-21T15:51:20.673 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0019999999999997e-06 2024-09-21T15:51:20.673 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.673 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.673 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:20.673 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489060+0000", 2024-09-21T15:51:20.673 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048036660000000002 2024-09-21T15:51:20.673 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.674 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.674 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:20.674 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489107+0000", 2024-09-21T15:51:20.674 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7153000000000001e-05 2024-09-21T15:51:20.674 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.674 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.674 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:20.674 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489230+0000", 2024-09-21T15:51:20.674 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012328600000000001 2024-09-21T15:51:20.674 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:20.674 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:20.674 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:20.674 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.674 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.675 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155256 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 356352~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:20.675 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441027+0000", 2024-09-21T15:51:20.675 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 58.179088966000002, 2024-09-21T15:51:20.675 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048677606999999998, 2024-09-21T15:51:20.675 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:20.675 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:20.675 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:20.675 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:20.675 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:20.675 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155256 2024-09-21T15:51:20.675 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.675 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:20.675 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.675 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:20.675 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441027+0000", 2024-09-21T15:51:20.676 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:20.676 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.676 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.676 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:20.676 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441027+0000", 2024-09-21T15:51:20.676 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:20.676 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.676 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.676 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:20.676 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441027+0000", 2024-09-21T15:51:20.676 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:20.676 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.676 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.676 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:20.676 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441029+0000", 2024-09-21T15:51:20.677 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.39e-06 2024-09-21T15:51:20.677 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.677 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.677 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:20.677 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441036+0000", 2024-09-21T15:51:20.677 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1129999999999997e-06 2024-09-21T15:51:20.677 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.677 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.677 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:20.677 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441040+0000", 2024-09-21T15:51:20.677 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8459999999999999e-06 2024-09-21T15:51:20.677 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.677 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.677 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:20.677 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489244+0000", 2024-09-21T15:51:20.677 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048203751000000003 2024-09-21T15:51:20.678 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.678 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.678 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:20.678 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489288+0000", 2024-09-21T15:51:20.678 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4153999999999998e-05 2024-09-21T15:51:20.678 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.678 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.678 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:20.678 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489705+0000", 2024-09-21T15:51:20.678 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000416773 2024-09-21T15:51:20.678 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:20.678 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:20.678 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:20.678 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.678 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.679 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155257 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 389120~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:20.679 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441045+0000", 2024-09-21T15:51:20.679 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 58.179071874999998, 2024-09-21T15:51:20.679 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049000325999999997, 2024-09-21T15:51:20.679 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:20.679 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:20.679 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:20.679 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:20.679 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:20.679 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155257 2024-09-21T15:51:20.679 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.679 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:20.679 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.679 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:20.679 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441045+0000", 2024-09-21T15:51:20.679 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:20.680 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.680 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.680 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:20.680 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441045+0000", 2024-09-21T15:51:20.680 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:20.680 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.680 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.680 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:20.680 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441044+0000", 2024-09-21T15:51:20.680 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:20.680 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.680 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.680 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:20.680 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441046+0000", 2024-09-21T15:51:20.680 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1909999999999999e-06 2024-09-21T15:51:20.680 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.681 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.681 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:20.681 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441054+0000", 2024-09-21T15:51:20.681 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3900000000000004e-06 2024-09-21T15:51:20.681 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.681 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.681 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:20.681 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441058+0000", 2024-09-21T15:51:20.681 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4619999999999998e-06 2024-09-21T15:51:20.681 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.681 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.681 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:20.681 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489741+0000", 2024-09-21T15:51:20.681 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048682660000000003 2024-09-21T15:51:20.681 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.681 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.682 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:20.682 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489813+0000", 2024-09-21T15:51:20.682 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2051999999999994e-05 2024-09-21T15:51:20.682 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.682 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.682 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:20.682 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490045+0000", 2024-09-21T15:51:20.682 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00023191299999999999 2024-09-21T15:51:20.682 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:20.682 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:20.682 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:20.682 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.682 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.682 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155258 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 421888~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:20.683 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441062+0000", 2024-09-21T15:51:20.683 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 58.179054333000003, 2024-09-21T15:51:20.683 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049339651999999998, 2024-09-21T15:51:20.683 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:20.683 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:20.683 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:20.683 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:20.683 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:20.683 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155258 2024-09-21T15:51:20.683 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.683 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:20.683 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.683 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:20.683 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441062+0000", 2024-09-21T15:51:20.683 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:20.684 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.684 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.684 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:20.684 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441062+0000", 2024-09-21T15:51:20.684 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:20.684 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.684 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.684 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:20.684 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441062+0000", 2024-09-21T15:51:20.684 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:20.684 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.684 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.684 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:20.684 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441064+0000", 2024-09-21T15:51:20.684 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.967e-06 2024-09-21T15:51:20.685 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.685 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.685 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:20.685 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441069+0000", 2024-09-21T15:51:20.685 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1529999999999996e-06 2024-09-21T15:51:20.685 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.685 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.685 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:20.685 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441073+0000", 2024-09-21T15:51:20.685 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3850000000000004e-06 2024-09-21T15:51:20.685 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.685 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.685 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:20.685 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490082+0000", 2024-09-21T15:51:20.686 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.04900885 2024-09-21T15:51:20.686 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.686 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.686 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:20.686 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490166+0000", 2024-09-21T15:51:20.686 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3640999999999994e-05 2024-09-21T15:51:20.686 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.686 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.686 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:20.686 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490402+0000", 2024-09-21T15:51:20.686 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000235978 2024-09-21T15:51:20.686 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:20.686 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:20.686 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:20.686 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.686 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.687 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155259 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 454656~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:20.687 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441086+0000", 2024-09-21T15:51:20.687 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 58.179030527999998, 2024-09-21T15:51:20.687 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049624397000000001, 2024-09-21T15:51:20.687 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:20.687 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:20.687 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:20.687 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:20.687 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:20.687 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155259 2024-09-21T15:51:20.687 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.687 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:20.687 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.687 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:20.687 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441086+0000", 2024-09-21T15:51:20.688 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:20.688 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.688 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.688 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:20.688 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441086+0000", 2024-09-21T15:51:20.688 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:20.688 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.688 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.688 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:20.688 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441086+0000", 2024-09-21T15:51:20.688 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:20.688 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.688 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.688 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:20.689 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441088+0000", 2024-09-21T15:51:20.689 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2630000000000002e-06 2024-09-21T15:51:20.689 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.689 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.689 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:20.689 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441093+0000", 2024-09-21T15:51:20.689 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5010000000000004e-06 2024-09-21T15:51:20.689 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.689 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.689 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:20.689 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441098+0000", 2024-09-21T15:51:20.689 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.707e-06 2024-09-21T15:51:20.689 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.689 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.689 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:20.689 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490447+0000", 2024-09-21T15:51:20.690 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049349125000000001 2024-09-21T15:51:20.690 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.690 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.690 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:20.690 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490504+0000", 2024-09-21T15:51:20.690 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6521000000000003e-05 2024-09-21T15:51:20.690 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.690 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.690 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:20.690 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490710+0000", 2024-09-21T15:51:20.690 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00020663999999999999 2024-09-21T15:51:20.690 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:20.690 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:20.690 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:20.691 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.691 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.691 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155260 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 487424~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:20.691 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441160+0000", 2024-09-21T15:51:20.691 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 58.178956565999997, 2024-09-21T15:51:20.691 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049876810000000001, 2024-09-21T15:51:20.691 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:20.691 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:20.691 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:20.691 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:20.691 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:20.691 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155260 2024-09-21T15:51:20.691 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.691 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:20.692 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.692 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:20.692 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441160+0000", 2024-09-21T15:51:20.692 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:20.692 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.692 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.692 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:20.692 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441160+0000", 2024-09-21T15:51:20.692 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:20.692 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.692 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.692 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:20.692 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441159+0000", 2024-09-21T15:51:20.692 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:20.692 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.693 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.693 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:20.693 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441164+0000", 2024-09-21T15:51:20.693 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4059999999999998e-06 2024-09-21T15:51:20.693 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.693 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.693 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:20.693 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441175+0000", 2024-09-21T15:51:20.693 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1095e-05 2024-09-21T15:51:20.693 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.693 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.693 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:20.693 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441182+0000", 2024-09-21T15:51:20.693 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2139999999999999e-06 2024-09-21T15:51:20.694 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.694 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.694 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:20.694 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490751+0000", 2024-09-21T15:51:20.694 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049569230999999998 2024-09-21T15:51:20.694 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.694 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.694 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:20.694 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490837+0000", 2024-09-21T15:51:20.694 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6017e-05 2024-09-21T15:51:20.694 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.694 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.695 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:20.695 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491037+0000", 2024-09-21T15:51:20.695 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00019952300000000001 2024-09-21T15:51:20.695 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:20.695 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:20.695 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:20.695 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.695 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.695 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155261 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 520192~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:20.695 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441187+0000", 2024-09-21T15:51:20.695 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 58.178929383000003, 2024-09-21T15:51:20.695 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.050124931999999997, 2024-09-21T15:51:20.695 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:20.695 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:20.695 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:20.696 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:20.696 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:20.696 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155261 2024-09-21T15:51:20.696 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.696 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:20.696 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.696 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:20.696 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441187+0000", 2024-09-21T15:51:20.696 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:20.696 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.696 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.696 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:20.696 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441187+0000", 2024-09-21T15:51:20.696 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:20.697 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.697 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.697 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:20.697 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441187+0000", 2024-09-21T15:51:20.697 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:20.697 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.697 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.697 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:20.697 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441189+0000", 2024-09-21T15:51:20.697 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3649999999999998e-06 2024-09-21T15:51:20.697 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.697 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.697 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:20.697 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441196+0000", 2024-09-21T15:51:20.698 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5379999999999999e-06 2024-09-21T15:51:20.698 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.698 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.698 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:20.698 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441200+0000", 2024-09-21T15:51:20.698 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4719999999999997e-06 2024-09-21T15:51:20.698 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.698 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.698 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:20.698 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491070+0000", 2024-09-21T15:51:20.698 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049869915000000001 2024-09-21T15:51:20.698 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.698 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.698 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:20.699 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491146+0000", 2024-09-21T15:51:20.699 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6589999999999997e-05 2024-09-21T15:51:20.699 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.699 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.699 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:20.699 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491312+0000", 2024-09-21T15:51:20.699 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000165475 2024-09-21T15:51:20.699 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:20.699 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:20.699 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:20.699 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.699 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.699 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155308 3.11 3:8b5183e5:::rbd_data.11335aef9442.0000000000000000:2a [read 356352~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:20.700 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441369+0000", 2024-09-21T15:51:20.700 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 58.178747534999999, 2024-09-21T15:51:20.700 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.050090672000000003, 2024-09-21T15:51:20.700 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:20.700 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:20.700 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:20.700 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:20.700 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:20.700 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155308 2024-09-21T15:51:20.700 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.700 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:20.700 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.700 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:20.700 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441369+0000", 2024-09-21T15:51:20.700 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:20.701 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.701 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.701 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:20.701 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441369+0000", 2024-09-21T15:51:20.701 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:20.701 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.701 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.701 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:20.701 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441368+0000", 2024-09-21T15:51:20.701 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:20.701 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.701 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.701 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:20.701 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441371+0000", 2024-09-21T15:51:20.702 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1100000000000001e-06 2024-09-21T15:51:20.702 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.702 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.702 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:20.702 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441377+0000", 2024-09-21T15:51:20.702 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5649999999999998e-06 2024-09-21T15:51:20.702 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.702 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.702 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:20.702 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441382+0000", 2024-09-21T15:51:20.702 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5929999999999999e-06 2024-09-21T15:51:20.702 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.702 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.702 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:20.702 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491349+0000", 2024-09-21T15:51:20.703 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049967122000000003 2024-09-21T15:51:20.703 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.703 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.703 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:20.703 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491400+0000", 2024-09-21T15:51:20.703 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1589999999999999e-05 2024-09-21T15:51:20.703 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.703 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.703 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:20.703 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491460+0000", 2024-09-21T15:51:20.703 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9070000000000002e-05 2024-09-21T15:51:20.703 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:20.703 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:20.703 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:20.703 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.704 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.704 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155309 3.11 3:8b5183e5:::rbd_data.11335aef9442.0000000000000000:2a [read 389120~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:20.704 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441386+0000", 2024-09-21T15:51:20.704 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 58.178730186000003, 2024-09-21T15:51:20.704 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.050184777, 2024-09-21T15:51:20.704 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:20.704 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:20.704 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:20.704 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:20.704 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:20.704 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155309 2024-09-21T15:51:20.704 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.704 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:20.704 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.705 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:20.705 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441386+0000", 2024-09-21T15:51:20.705 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:20.705 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.705 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.705 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:20.705 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441386+0000", 2024-09-21T15:51:20.705 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:20.705 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.705 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.705 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:20.705 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441386+0000", 2024-09-21T15:51:20.705 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:20.705 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.705 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.706 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:20.706 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441388+0000", 2024-09-21T15:51:20.706 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4420000000000001e-06 2024-09-21T15:51:20.706 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.706 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.706 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:20.706 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441395+0000", 2024-09-21T15:51:20.706 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6189999999999996e-06 2024-09-21T15:51:20.706 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.706 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.706 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:20.706 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441400+0000", 2024-09-21T15:51:20.706 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.118e-06 2024-09-21T15:51:20.707 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.707 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.707 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:20.707 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491473+0000", 2024-09-21T15:51:20.707 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.050072783000000003 2024-09-21T15:51:20.707 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.707 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.707 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:20.707 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491518+0000", 2024-09-21T15:51:20.707 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4854000000000001e-05 2024-09-21T15:51:20.707 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:20.707 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:20.707 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:20.707 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491571+0000", 2024-09-21T15:51:20.707 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3358e-05 2024-09-21T15:51:20.708 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:20.708 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:20.708 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:20.708 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:20.708 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:20.708 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:51:20.708 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd set nodeep-scrub 2024-09-21T15:51:20.710 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:51:20.710 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:51:20.708+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:51:20.810 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:51:20.808+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:51:20.827 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:20.827 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:51:20.827 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:51:20.827 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:20.838 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:51:20.910 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:51:20.909+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:51:20.944 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:20.944 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:51:20.944 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:51:20.944 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:51:20.944 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:20.954 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:51:21.012 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:51:21.011+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:51:21.057 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:21.057 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:51:21.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:51:21.057 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:51:21.057 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.057 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160505 3.b 3:d297695c:::rbd_header.11338bae974a:head [watch reconnect cookie 140381478598448 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-21T15:51:21.057 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:29.138810+0000", 2024-09-21T15:51:21.057 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 51.915926362, 2024-09-21T15:51:21.057 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0172598319999999, 2024-09-21T15:51:21.057 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:21.058 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:21.058 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:21.058 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:21.058 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:21.058 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160505 2024-09-21T15:51:21.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.058 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:21.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.058 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:21.058 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138810+0000", 2024-09-21T15:51:21.058 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.058 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.058 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.058 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:21.058 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138810+0000", 2024-09-21T15:51:21.059 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.059 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.059 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.059 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:21.059 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138808+0000", 2024-09-21T15:51:21.059 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:51:21.059 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.059 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.059 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:21.059 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138815+0000", 2024-09-21T15:51:21.059 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6549999999999998e-06 2024-09-21T15:51:21.059 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.059 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.059 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:21.059 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138825+0000", 2024-09-21T15:51:21.059 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9790000000000005e-06 2024-09-21T15:51:21.060 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.060 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.060 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:21.060 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138834+0000", 2024-09-21T15:51:21.060 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5179999999999993e-06 2024-09-21T15:51:21.060 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.060 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.060 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.060 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.140973+0000", 2024-09-21T15:51:21.060 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0021392049999999999 2024-09-21T15:51:21.060 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.060 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.060 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:51:21.060 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.140981+0000", 2024-09-21T15:51:21.060 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8909999999999995e-06 2024-09-21T15:51:21.060 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.061 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.061 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.150457+0000", 2024-09-21T15:51:21.061 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.009475645 2024-09-21T15:51:21.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.061 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for missing object", 2024-09-21T15:51:21.061 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.150589+0000", 2024-09-21T15:51:21.061 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000131779 2024-09-21T15:51:21.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.061 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.061 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.151723+0000", 2024-09-21T15:51:21.061 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0011345979999999999 2024-09-21T15:51:21.061 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.061 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.062 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-09-21T15:51:21.062 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.151872+0000", 2024-09-21T15:51:21.062 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000149021 2024-09-21T15:51:21.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.062 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.062 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.152563+0000", 2024-09-21T15:51:21.062 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00069051500000000005 2024-09-21T15:51:21.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.062 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:21.062 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.152673+0000", 2024-09-21T15:51:21.062 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000109795 2024-09-21T15:51:21.062 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.062 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.063 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:21.063 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.156069+0000", 2024-09-21T15:51:21.063 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003396736 2024-09-21T15:51:21.063 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.063 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:21.063 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.063 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.063 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.063 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160506 3.b 3:d297695c:::rbd_header.11338bae974a:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-21T15:51:21.063 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:29.138844+0000", 2024-09-21T15:51:21.063 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 51.915891475000002, 2024-09-21T15:51:21.063 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0174592680000001, 2024-09-21T15:51:21.063 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:21.063 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:21.063 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:21.063 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:21.064 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:21.064 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160506 2024-09-21T15:51:21.064 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.064 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:21.064 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.064 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:21.064 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138844+0000", 2024-09-21T15:51:21.064 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.064 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.064 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.064 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:21.064 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138844+0000", 2024-09-21T15:51:21.064 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.064 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.064 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.064 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:21.064 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138844+0000", 2024-09-21T15:51:21.065 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:21.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.065 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:21.065 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138848+0000", 2024-09-21T15:51:21.065 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.76e-06 2024-09-21T15:51:21.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.065 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:21.065 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138854+0000", 2024-09-21T15:51:21.065 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3890000000000002e-06 2024-09-21T15:51:21.065 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.065 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.065 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:21.065 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138860+0000", 2024-09-21T15:51:21.066 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6409999999999999e-06 2024-09-21T15:51:21.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.066 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.066 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.066 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.141015+0000", 2024-09-21T15:51:21.066 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0021553779999999999 2024-09-21T15:51:21.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.066 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.066 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:51:21.066 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.141021+0000", 2024-09-21T15:51:21.066 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7629999999999997e-06 2024-09-21T15:51:21.066 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.066 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.066 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.066 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.150858+0000", 2024-09-21T15:51:21.067 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0098372200000001 2024-09-21T15:51:21.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.067 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for missing object", 2024-09-21T15:51:21.067 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.150899+0000", 2024-09-21T15:51:21.067 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1112000000000002e-05 2024-09-21T15:51:21.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.067 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.067 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.152069+0000", 2024-09-21T15:51:21.067 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001170307 2024-09-21T15:51:21.067 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.067 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.067 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-09-21T15:51:21.067 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.152192+0000", 2024-09-21T15:51:21.068 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012289700000000001 2024-09-21T15:51:21.068 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.068 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.068 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.068 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.153362+0000", 2024-09-21T15:51:21.068 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0011700549999999999 2024-09-21T15:51:21.068 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.068 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.068 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:21.068 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.153457+0000", 2024-09-21T15:51:21.068 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4687999999999999e-05 2024-09-21T15:51:21.068 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.068 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.068 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:51:21.068 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.153893+0000", 2024-09-21T15:51:21.069 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000436363 2024-09-21T15:51:21.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.069 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:21.069 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.156134+0000", 2024-09-21T15:51:21.069 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0022410009999999998 2024-09-21T15:51:21.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.069 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:21.069 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.156148+0000", 2024-09-21T15:51:21.069 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3574000000000001e-05 2024-09-21T15:51:21.069 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.069 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.069 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:21.069 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.156214+0000", 2024-09-21T15:51:21.070 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6438000000000003e-05 2024-09-21T15:51:21.070 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.070 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:21.070 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.156304+0000", 2024-09-21T15:51:21.070 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9278999999999997e-05 2024-09-21T15:51:21.070 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.070 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:21.070 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.070 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.070 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.070 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160512 3.b 3:d297695c:::rbd_header.11338bae974a:head [watch unwatch cookie 140381478598448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T15:51:21.070 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:30.161362+0000", 2024-09-21T15:51:21.070 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 50.893373666999999, 2024-09-21T15:51:21.070 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0074105029999999997, 2024-09-21T15:51:21.071 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:21.071 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:21.071 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:21.071 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:21.071 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:21.071 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160512 2024-09-21T15:51:21.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.071 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:21.071 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.071 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:21.071 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161362+0000", 2024-09-21T15:51:21.071 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.071 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.203 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.203 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:21.203 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161362+0000", 2024-09-21T15:51:21.203 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.203 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.203 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.203 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:21.203 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161362+0000", 2024-09-21T15:51:21.204 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:21.204 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.204 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.204 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:21.204 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161365+0000", 2024-09-21T15:51:21.204 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8669999999999998e-06 2024-09-21T15:51:21.204 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.204 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.204 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:21.204 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161371+0000", 2024-09-21T15:51:21.204 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8009999999999998e-06 2024-09-21T15:51:21.204 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.204 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.205 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:21.205 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161376+0000", 2024-09-21T15:51:21.205 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0340000000000004e-06 2024-09-21T15:51:21.205 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.205 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.205 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.205 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161762+0000", 2024-09-21T15:51:21.205 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038575099999999999 2024-09-21T15:51:21.205 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.205 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.205 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:21.205 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161838+0000", 2024-09-21T15:51:21.205 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5697000000000004e-05 2024-09-21T15:51:21.205 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.206 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.206 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:51:21.206 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161952+0000", 2024-09-21T15:51:21.206 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000114034 2024-09-21T15:51:21.206 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.206 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.206 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:21.206 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.163727+0000", 2024-09-21T15:51:21.206 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0017749560000000001 2024-09-21T15:51:21.206 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.206 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.206 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:21.206 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168545+0000", 2024-09-21T15:51:21.207 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0048186080000000003 2024-09-21T15:51:21.207 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.207 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.207 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:21.207 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168593+0000", 2024-09-21T15:51:21.207 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7549999999999997e-05 2024-09-21T15:51:21.207 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.207 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.207 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:21.207 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168773+0000", 2024-09-21T15:51:21.207 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00017976600000000001 2024-09-21T15:51:21.207 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.207 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:21.208 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.208 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.208 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160513 3.b 3:d297695c:::rbd_header.11338bae974a:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-09-21T15:51:21.208 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:30.161755+0000", 2024-09-21T15:51:21.208 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 50.892980735999998, 2024-09-21T15:51:21.208 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0072250300000000003, 2024-09-21T15:51:21.208 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:21.208 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:21.208 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:21.208 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:21.208 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:21.208 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160513 2024-09-21T15:51:21.208 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.209 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:21.209 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.209 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:21.209 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161755+0000", 2024-09-21T15:51:21.209 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.209 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.209 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.209 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:21.209 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161755+0000", 2024-09-21T15:51:21.209 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.209 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.209 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.209 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:21.210 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161754+0000", 2024-09-21T15:51:21.210 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:21.210 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.210 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.210 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:21.210 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161759+0000", 2024-09-21T15:51:21.210 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9699999999999998e-06 2024-09-21T15:51:21.210 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.210 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.210 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:21.210 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161768+0000", 2024-09-21T15:51:21.210 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9949999999999994e-06 2024-09-21T15:51:21.210 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.211 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.211 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:21.211 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161774+0000", 2024-09-21T15:51:21.211 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1290000000000001e-06 2024-09-21T15:51:21.211 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.211 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.211 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.211 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.162239+0000", 2024-09-21T15:51:21.211 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00046406600000000001 2024-09-21T15:51:21.211 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.211 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.211 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-09-21T15:51:21.211 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.162296+0000", 2024-09-21T15:51:21.212 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7122e-05 2024-09-21T15:51:21.212 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.212 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.212 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.212 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168802+0000", 2024-09-21T15:51:21.212 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0065061700000000004 2024-09-21T15:51:21.212 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.212 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.212 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:21.212 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168877+0000", 2024-09-21T15:51:21.212 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5214000000000002e-05 2024-09-21T15:51:21.212 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.212 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.212 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:21.213 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168980+0000", 2024-09-21T15:51:21.213 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000103182 2024-09-21T15:51:21.213 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.213 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:21.213 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.213 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.213 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.213 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160527 3.b 3:d297695c:::rbd_header.11338bae974a:head [watch watch cookie 140381476148256] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T15:51:21.213 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:30.182756+0000", 2024-09-21T15:51:21.213 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 50.871980155000003, 2024-09-21T15:51:21.213 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0054202579999999998, 2024-09-21T15:51:21.213 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:21.213 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:21.214 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:21.214 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:21.214 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:21.214 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160527 2024-09-21T15:51:21.214 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.214 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:21.214 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.214 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:21.214 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182756+0000", 2024-09-21T15:51:21.214 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.214 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.214 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.214 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:21.214 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182756+0000", 2024-09-21T15:51:21.214 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.215 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.215 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.215 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:21.215 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182755+0000", 2024-09-21T15:51:21.215 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:21.215 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.215 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.215 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:21.215 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182760+0000", 2024-09-21T15:51:21.215 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4269999999999999e-06 2024-09-21T15:51:21.215 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.215 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.215 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:21.215 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182771+0000", 2024-09-21T15:51:21.215 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0312999999999999e-05 2024-09-21T15:51:21.215 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.216 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.216 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:21.216 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182778+0000", 2024-09-21T15:51:21.216 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5700000000000004e-06 2024-09-21T15:51:21.216 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.216 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.216 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.216 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.184745+0000", 2024-09-21T15:51:21.216 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0019667170000000002 2024-09-21T15:51:21.216 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.216 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.216 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:21.216 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.184867+0000", 2024-09-21T15:51:21.216 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000122504 2024-09-21T15:51:21.216 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.217 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.217 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:51:21.217 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.185135+0000", 2024-09-21T15:51:21.217 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000267135 2024-09-21T15:51:21.217 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.217 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.217 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:21.217 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.187721+0000", 2024-09-21T15:51:21.217 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002586891 2024-09-21T15:51:21.217 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.217 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.217 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:21.217 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.187874+0000", 2024-09-21T15:51:21.217 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000152726 2024-09-21T15:51:21.217 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.218 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.218 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:21.218 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.187951+0000", 2024-09-21T15:51:21.218 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7132000000000006e-05 2024-09-21T15:51:21.218 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.218 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.218 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:21.218 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.188176+0000", 2024-09-21T15:51:21.218 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00022477900000000001 2024-09-21T15:51:21.218 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.218 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:21.218 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.218 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.218 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.218 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:160608 3.12 e95/94 3:48a47e1c:::rbd_data.11338bae974a.000000000000000b:head v 95'345, mlcod=0'0)", 2024-09-21T15:51:21.218 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.216934+0000", 2024-09-21T15:51:21.219 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 49.837802017000001, 2024-09-21T15:51:21.219 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053335759999999996, 2024-09-21T15:51:21.219 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:21.219 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:21.219 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:21.219 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.219 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:21.219 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216934+0000", 2024-09-21T15:51:21.219 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.219 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.219 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.219 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:21.219 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216934+0000", 2024-09-21T15:51:21.219 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.219 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.220 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.220 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:21.220 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216933+0000", 2024-09-21T15:51:21.220 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:21.220 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.220 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.220 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:21.220 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216939+0000", 2024-09-21T15:51:21.220 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6459999999999998e-06 2024-09-21T15:51:21.220 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.220 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.220 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:21.220 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216947+0000", 2024-09-21T15:51:21.220 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.2479999999999996e-06 2024-09-21T15:51:21.220 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.220 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.220 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:21.221 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216951+0000", 2024-09-21T15:51:21.221 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3760000000000001e-06 2024-09-21T15:51:21.221 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.221 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.221 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.221 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216998+0000", 2024-09-21T15:51:21.221 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6244e-05 2024-09-21T15:51:21.221 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.221 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.221 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:21.221 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.217019+0000", 2024-09-21T15:51:21.221 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1191000000000001e-05 2024-09-21T15:51:21.221 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.221 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.221 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:21.221 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.222230+0000", 2024-09-21T15:51:21.222 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052114350000000004 2024-09-21T15:51:21.222 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.222 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.222 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:21.222 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.222267+0000", 2024-09-21T15:51:21.222 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7169999999999998e-05 2024-09-21T15:51:21.222 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.222 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:21.222 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.222 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.222 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.222 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160662 3.b 3:d7701915:::rbd_object_map.1133fd7483d4.000000000000002c:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-21T15:51:21.222 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.279369+0000", 2024-09-21T15:51:21.222 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 49.775367250000002, 2024-09-21T15:51:21.222 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0056948040000000004, 2024-09-21T15:51:21.223 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:21.223 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:21.223 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:21.223 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:21.223 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:21.223 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160662 2024-09-21T15:51:21.223 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.223 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:21.223 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.223 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:21.223 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279369+0000", 2024-09-21T15:51:21.223 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.223 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.223 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.223 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:21.223 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279369+0000", 2024-09-21T15:51:21.224 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.224 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.224 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.224 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:21.224 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279368+0000", 2024-09-21T15:51:21.224 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:21.224 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.224 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.224 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:21.224 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279373+0000", 2024-09-21T15:51:21.224 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6529999999999999e-06 2024-09-21T15:51:21.224 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.224 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.224 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:21.224 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279384+0000", 2024-09-21T15:51:21.224 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0256e-05 2024-09-21T15:51:21.225 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.225 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.225 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:21.225 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279388+0000", 2024-09-21T15:51:21.225 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5630000000000002e-06 2024-09-21T15:51:21.225 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.225 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.225 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.225 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279424+0000", 2024-09-21T15:51:21.225 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5985999999999997e-05 2024-09-21T15:51:21.225 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.225 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.225 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:21.225 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279478+0000", 2024-09-21T15:51:21.225 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3944999999999999e-05 2024-09-21T15:51:21.225 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.225 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.226 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:51:21.226 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279575+0000", 2024-09-21T15:51:21.226 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6817999999999994e-05 2024-09-21T15:51:21.226 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.226 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.226 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:21.226 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.282391+0000", 2024-09-21T15:51:21.226 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0028156700000000002 2024-09-21T15:51:21.226 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.226 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.226 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:21.226 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.284977+0000", 2024-09-21T15:51:21.226 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0025859640000000001 2024-09-21T15:51:21.226 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.226 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.227 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:21.227 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.285021+0000", 2024-09-21T15:51:21.227 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4415000000000001e-05 2024-09-21T15:51:21.227 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.227 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.227 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:21.227 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.285063+0000", 2024-09-21T15:51:21.227 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2447000000000002e-05 2024-09-21T15:51:21.227 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.227 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:21.227 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.227 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.227 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.227 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:161219 3.17 3:e87cb47d:::rbd_data.11335aef9442.000000000000000a:head [delete] snapc 2a=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-21T15:51:21.227 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:36.771292+0000", 2024-09-21T15:51:21.227 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 44.283443437999999, 2024-09-21T15:51:21.228 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0058540529999999997, 2024-09-21T15:51:21.228 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:21.228 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:21.228 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:21.228 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:21.228 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:21.228 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 161219 2024-09-21T15:51:21.228 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.228 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:21.228 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.228 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:21.228 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771292+0000", 2024-09-21T15:51:21.228 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.228 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.228 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.228 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:21.229 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771292+0000", 2024-09-21T15:51:21.229 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.229 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.229 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.229 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:21.229 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771292+0000", 2024-09-21T15:51:21.229 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:21.229 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.229 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.229 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:21.229 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771298+0000", 2024-09-21T15:51:21.229 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9519999999999999e-06 2024-09-21T15:51:21.229 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.229 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.229 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:21.229 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771308+0000", 2024-09-21T15:51:21.229 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0601000000000001e-05 2024-09-21T15:51:21.230 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.230 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.230 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:21.230 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771314+0000", 2024-09-21T15:51:21.230 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.913e-06 2024-09-21T15:51:21.230 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.230 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.230 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.230 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771346+0000", 2024-09-21T15:51:21.230 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1711000000000002e-05 2024-09-21T15:51:21.230 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.230 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.230 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:21.230 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771399+0000", 2024-09-21T15:51:21.230 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2883000000000002e-05 2024-09-21T15:51:21.231 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.231 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.231 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-09-21T15:51:21.231 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771470+0000", 2024-09-21T15:51:21.231 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0934999999999998e-05 2024-09-21T15:51:21.231 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.231 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.231 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:21.231 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.772234+0000", 2024-09-21T15:51:21.231 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00076485400000000003 2024-09-21T15:51:21.231 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.231 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.231 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:21.231 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.777083+0000", 2024-09-21T15:51:21.231 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0048486329999999998 2024-09-21T15:51:21.232 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.232 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.232 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:21.232 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.777111+0000", 2024-09-21T15:51:21.232 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8399999999999999e-05 2024-09-21T15:51:21.232 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.232 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.232 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:21.232 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.777147+0000", 2024-09-21T15:51:21.232 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5049000000000003e-05 2024-09-21T15:51:21.232 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.232 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:21.232 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.232 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.232 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.232 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:161859 3.17 3:ee7ab022:::rbd_data.11338db70e6f.0000000000000007:head [delete] snapc 26=[] ondisk+write+known_if_redirected+supports_pool_eio e105)", 2024-09-21T15:51:21.233 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:40.782264+0000", 2024-09-21T15:51:21.233 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 40.272471578999998, 2024-09-21T15:51:21.233 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052663939999999998, 2024-09-21T15:51:21.233 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:21.233 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:21.233 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:21.233 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:21.233 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:21.233 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 161859 2024-09-21T15:51:21.233 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.233 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:21.233 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.233 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:21.233 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782264+0000", 2024-09-21T15:51:21.233 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.234 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.234 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.234 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:21.234 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782264+0000", 2024-09-21T15:51:21.234 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.234 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.234 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.234 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:21.234 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782264+0000", 2024-09-21T15:51:21.234 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:21.234 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.234 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.234 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:21.234 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782266+0000", 2024-09-21T15:51:21.234 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4590000000000001e-06 2024-09-21T15:51:21.234 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.235 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.235 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:21.235 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782275+0000", 2024-09-21T15:51:21.235 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5450000000000003e-06 2024-09-21T15:51:21.235 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.235 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.235 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:21.235 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782281+0000", 2024-09-21T15:51:21.235 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7180000000000001e-06 2024-09-21T15:51:21.235 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.235 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.235 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.235 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782312+0000", 2024-09-21T15:51:21.235 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1059000000000002e-05 2024-09-21T15:51:21.235 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.235 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.236 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:21.236 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782470+0000", 2024-09-21T15:51:21.236 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000157818 2024-09-21T15:51:21.236 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.236 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.236 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-09-21T15:51:21.236 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.782574+0000", 2024-09-21T15:51:21.236 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000104536 2024-09-21T15:51:21.236 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.236 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.236 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:21.236 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.783499+0000", 2024-09-21T15:51:21.236 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00092479599999999997 2024-09-21T15:51:21.236 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.236 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.236 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:21.236 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.787472+0000", 2024-09-21T15:51:21.237 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0039730319999999996 2024-09-21T15:51:21.237 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.237 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.237 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:21.237 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.787507+0000", 2024-09-21T15:51:21.237 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5552000000000001e-05 2024-09-21T15:51:21.237 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.237 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.237 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:21.237 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.787531+0000", 2024-09-21T15:51:21.237 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3281999999999998e-05 2024-09-21T15:51:21.237 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.237 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:21.237 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.237 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.237 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.238 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:163005 3.6 3:67cae2ed:::rbd_mirroring:head [call rbd.mirror_image_remove in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-21T15:51:21.238 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:49.711037+0000", 2024-09-21T15:51:21.238 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 31.343698559, 2024-09-21T15:51:21.238 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0059230919999999996, 2024-09-21T15:51:21.238 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:21.238 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:21.238 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:21.238 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:21.238 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:21.238 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 163005 2024-09-21T15:51:21.238 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.238 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:21.238 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.238 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:21.238 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:49.711037+0000", 2024-09-21T15:51:21.239 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.239 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.239 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.239 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:21.239 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:49.711037+0000", 2024-09-21T15:51:21.239 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.239 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.239 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.239 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:21.239 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:49.711037+0000", 2024-09-21T15:51:21.239 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:21.239 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.239 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.239 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:21.239 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:49.711041+0000", 2024-09-21T15:51:21.239 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8199999999999996e-06 2024-09-21T15:51:21.240 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.240 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.240 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:21.240 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:49.711052+0000", 2024-09-21T15:51:21.240 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0567999999999999e-05 2024-09-21T15:51:21.240 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.240 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.240 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:21.240 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:49.711057+0000", 2024-09-21T15:51:21.240 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7119999999999999e-06 2024-09-21T15:51:21.240 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.240 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.240 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.240 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:49.711096+0000", 2024-09-21T15:51:21.240 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8872999999999999e-05 2024-09-21T15:51:21.240 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.241 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.241 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:21.241 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:49.711151+0000", 2024-09-21T15:51:21.241 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5343999999999997e-05 2024-09-21T15:51:21.241 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.241 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.241 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:21.241 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:49.716960+0000", 2024-09-21T15:51:21.241 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0058095150000000003 2024-09-21T15:51:21.241 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.241 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:21.241 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.241 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.241 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.241 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:165147 3.10 e131/96 3:0b23e447:::rbd_header.11339e87a32a:head v 131'448, mlcod=130'446)", 2024-09-21T15:51:21.241 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:51:06.827072+0000", 2024-09-21T15:51:21.242 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.227663775, 2024-09-21T15:51:21.242 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.011683913000000001, 2024-09-21T15:51:21.242 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:21.242 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:21.242 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:21.242 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.242 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:21.242 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.827072+0000", 2024-09-21T15:51:21.242 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.242 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.242 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.242 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:21.242 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.827072+0000", 2024-09-21T15:51:21.242 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.242 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.242 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.243 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:21.243 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.827072+0000", 2024-09-21T15:51:21.243 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:21.243 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.243 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.243 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:21.243 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.827078+0000", 2024-09-21T15:51:21.243 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4840000000000001e-06 2024-09-21T15:51:21.243 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.243 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.243 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:21.243 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.827094+0000", 2024-09-21T15:51:21.243 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5639999999999999e-05 2024-09-21T15:51:21.243 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.243 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.243 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:21.244 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.827104+0000", 2024-09-21T15:51:21.244 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.8300000000000008e-06 2024-09-21T15:51:21.244 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.244 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.244 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.244 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.827170+0000", 2024-09-21T15:51:21.244 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5913000000000004e-05 2024-09-21T15:51:21.244 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.244 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.244 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:21.244 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.827196+0000", 2024-09-21T15:51:21.244 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6210000000000001e-05 2024-09-21T15:51:21.244 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.244 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.244 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:21.244 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.838689+0000", 2024-09-21T15:51:21.245 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.011492979 2024-09-21T15:51:21.245 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.245 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.245 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:21.245 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.838756+0000", 2024-09-21T15:51:21.245 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7155000000000003e-05 2024-09-21T15:51:21.245 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.245 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:21.245 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.245 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.245 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.245 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.0:3709 3.3 e131/96 3:c5f4482f:::rbd_data.11339e87a32a.000000000000013a:12 v 131'841, mlcod=130'838)", 2024-09-21T15:51:21.245 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:51:06.828155+0000", 2024-09-21T15:51:21.245 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.226581333, 2024-09-21T15:51:21.245 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.010631115999999999, 2024-09-21T15:51:21.246 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:21.246 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:21.246 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:21.246 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.246 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:21.246 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.828155+0000", 2024-09-21T15:51:21.246 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.246 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.246 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.246 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:21.246 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.828155+0000", 2024-09-21T15:51:21.246 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.246 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.246 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.246 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:21.247 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.828154+0000", 2024-09-21T15:51:21.247 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:21.247 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.247 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.247 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:21.247 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.828162+0000", 2024-09-21T15:51:21.247 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.2269999999999993e-06 2024-09-21T15:51:21.247 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.247 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.247 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:21.247 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.828188+0000", 2024-09-21T15:51:21.247 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5626000000000002e-05 2024-09-21T15:51:21.247 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.247 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.247 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:21.247 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.828210+0000", 2024-09-21T15:51:21.248 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1792000000000001e-05 2024-09-21T15:51:21.248 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.248 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.248 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.248 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.828251+0000", 2024-09-21T15:51:21.248 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1329999999999999e-05 2024-09-21T15:51:21.248 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.248 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.248 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:21.248 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.828277+0000", 2024-09-21T15:51:21.248 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6236000000000001e-05 2024-09-21T15:51:21.248 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.248 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.248 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:21.248 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.838756+0000", 2024-09-21T15:51:21.248 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.010478919 2024-09-21T15:51:21.249 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.249 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.249 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:21.249 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.838786+0000", 2024-09-21T15:51:21.249 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9560000000000002e-05 2024-09-21T15:51:21.249 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.249 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:21.249 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.249 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.249 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.249 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.0:3710 3.3 e131/96 3:c7ccd1db:::rbd_data.11339e87a32a.0000000000000136:12 v 131'843, mlcod=130'838)", 2024-09-21T15:51:21.249 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:51:06.831274+0000", 2024-09-21T15:51:21.249 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 14.223461482999999, 2024-09-21T15:51:21.249 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0077927079999999998, 2024-09-21T15:51:21.249 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:21.249 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:21.250 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:21.250 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.250 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:21.250 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.831274+0000", 2024-09-21T15:51:21.250 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.250 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.250 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.250 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:21.250 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.831274+0000", 2024-09-21T15:51:21.250 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.250 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.250 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.250 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:21.250 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.831274+0000", 2024-09-21T15:51:21.250 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:21.250 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.251 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.251 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:21.251 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.831286+0000", 2024-09-21T15:51:21.251 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1853e-05 2024-09-21T15:51:21.251 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.251 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.251 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:21.251 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.831296+0000", 2024-09-21T15:51:21.251 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0713999999999999e-05 2024-09-21T15:51:21.251 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.251 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.251 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:21.251 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.831305+0000", 2024-09-21T15:51:21.251 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4749999999999993e-06 2024-09-21T15:51:21.251 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.251 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.252 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.252 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.831364+0000", 2024-09-21T15:51:21.252 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9320999999999997e-05 2024-09-21T15:51:21.252 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.252 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.252 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:21.252 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.831386+0000", 2024-09-21T15:51:21.252 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2058000000000001e-05 2024-09-21T15:51:21.252 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.252 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.252 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:21.252 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.839039+0000", 2024-09-21T15:51:21.252 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0076530490000000003 2024-09-21T15:51:21.252 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.252 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.252 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:21.253 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.839067+0000", 2024-09-21T15:51:21.253 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8005999999999999e-05 2024-09-21T15:51:21.253 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.253 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:21.253 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.253 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.253 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.253 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:166306 3.1 3:82f7f77c:::rbd_header.11336b2fdf4a:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-09-21T15:51:21.253 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:51:16.854113+0000", 2024-09-21T15:51:21.253 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.2006227169999999, 2024-09-21T15:51:21.253 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0076607890000000003, 2024-09-21T15:51:21.253 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:21.253 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:21.253 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:21.253 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:21.253 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:21.254 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 166306 2024-09-21T15:51:21.254 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.254 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:21.254 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.254 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:21.254 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.854113+0000", 2024-09-21T15:51:21.254 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.254 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.254 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.254 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:21.254 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.854113+0000", 2024-09-21T15:51:21.254 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.254 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.254 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.254 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:21.255 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.854111+0000", 2024-09-21T15:51:21.255 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:51:21.255 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.255 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.255 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:21.255 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.854120+0000", 2024-09-21T15:51:21.255 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8410000000000006e-06 2024-09-21T15:51:21.255 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.255 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.255 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:21.255 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.854133+0000", 2024-09-21T15:51:21.255 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.272e-05 2024-09-21T15:51:21.255 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.255 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.255 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:21.255 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.854141+0000", 2024-09-21T15:51:21.256 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8590000000000005e-06 2024-09-21T15:51:21.256 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.256 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.256 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.256 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.854207+0000", 2024-09-21T15:51:21.256 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6593000000000004e-05 2024-09-21T15:51:21.256 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.256 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.256 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:21.256 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.854308+0000", 2024-09-21T15:51:21.256 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000100592 2024-09-21T15:51:21.256 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.256 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.256 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-09-21T15:51:21.256 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.854716+0000", 2024-09-21T15:51:21.256 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00040786099999999998 2024-09-21T15:51:21.257 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.257 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.257 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:21.257 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.858074+0000", 2024-09-21T15:51:21.257 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00335798 2024-09-21T15:51:21.257 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.257 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.257 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:21.257 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.861691+0000", 2024-09-21T15:51:21.257 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003617351 2024-09-21T15:51:21.257 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.257 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.257 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:21.257 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.861740+0000", 2024-09-21T15:51:21.257 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8618000000000001e-05 2024-09-21T15:51:21.257 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.258 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.258 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:21.258 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.861774+0000", 2024-09-21T15:51:21.258 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4313000000000002e-05 2024-09-21T15:51:21.258 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.258 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:21.258 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.258 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.258 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.258 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.0:3832 3.3 e141/96 3:c4381677:::rbd_data.11336b2fdf4a.000000000000000e:c v 141'878, mlcod=140'875)", 2024-09-21T15:51:21.258 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:51:16.856401+0000", 2024-09-21T15:51:21.258 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 4.1983345160000001, 2024-09-21T15:51:21.258 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0061803919999999998, 2024-09-21T15:51:21.258 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:21.258 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:21.258 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:21.259 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.259 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:21.259 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.856401+0000", 2024-09-21T15:51:21.259 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.259 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.259 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.259 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:21.259 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.856401+0000", 2024-09-21T15:51:21.259 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.259 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.259 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.259 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:21.259 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.856400+0000", 2024-09-21T15:51:21.259 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:21.259 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.259 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.260 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:21.260 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.856407+0000", 2024-09-21T15:51:21.260 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9079999999999998e-06 2024-09-21T15:51:21.260 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.260 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.260 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:21.260 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.856417+0000", 2024-09-21T15:51:21.260 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5929999999999995e-06 2024-09-21T15:51:21.260 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.260 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.260 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:21.260 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.856421+0000", 2024-09-21T15:51:21.260 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6829999999999999e-06 2024-09-21T15:51:21.260 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.261 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.261 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.261 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.856455+0000", 2024-09-21T15:51:21.261 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3145000000000002e-05 2024-09-21T15:51:21.261 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.261 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.261 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:21.261 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.856481+0000", 2024-09-21T15:51:21.261 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5959999999999999e-05 2024-09-21T15:51:21.261 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.261 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.261 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:21.261 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.862548+0000", 2024-09-21T15:51:21.261 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0060674869999999999 2024-09-21T15:51:21.261 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.261 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.262 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:21.262 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.862582+0000", 2024-09-21T15:51:21.262 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3726000000000002e-05 2024-09-21T15:51:21.262 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.262 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:21.262 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.262 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.262 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.262 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:166979 3.b 3:d564a535:::rbd_object_map.1133896adcfd:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-21T15:51:21.262 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:51:20.685558+0000", 2024-09-21T15:51:21.262 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.36917811099999998, 2024-09-21T15:51:21.262 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0054097210000000001, 2024-09-21T15:51:21.262 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:21.262 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:21.262 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:21.262 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:21.263 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:21.263 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 166979 2024-09-21T15:51:21.263 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.263 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:21.263 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.263 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:21.263 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.685558+0000", 2024-09-21T15:51:21.263 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.263 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.263 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.263 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:21.263 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.685558+0000", 2024-09-21T15:51:21.263 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.263 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.263 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.263 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:21.264 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.685557+0000", 2024-09-21T15:51:21.264 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:51:21.264 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.264 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.264 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:21.264 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.685563+0000", 2024-09-21T15:51:21.264 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4699999999999999e-06 2024-09-21T15:51:21.264 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.264 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.264 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:21.264 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.685575+0000", 2024-09-21T15:51:21.264 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2113000000000001e-05 2024-09-21T15:51:21.264 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.264 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.264 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:21.265 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.685582+0000", 2024-09-21T15:51:21.265 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7830000000000001e-06 2024-09-21T15:51:21.265 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.265 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.265 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.265 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.685633+0000", 2024-09-21T15:51:21.265 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1363000000000003e-05 2024-09-21T15:51:21.265 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.265 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.265 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op must wait for map", 2024-09-21T15:51:21.265 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.685643+0000", 2024-09-21T15:51:21.265 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2310000000000002e-06 2024-09-21T15:51:21.265 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.265 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.265 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.265 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.687329+0000", 2024-09-21T15:51:21.266 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001686325 2024-09-21T15:51:21.266 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.266 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.266 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:21.266 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.687398+0000", 2024-09-21T15:51:21.266 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8666999999999998e-05 2024-09-21T15:51:21.266 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.266 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.266 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:51:21.266 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.687507+0000", 2024-09-21T15:51:21.266 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000109956 2024-09-21T15:51:21.266 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.266 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.266 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:21.266 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.688784+0000", 2024-09-21T15:51:21.267 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012768899999999999 2024-09-21T15:51:21.267 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.267 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.267 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:21.267 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.690901+0000", 2024-09-21T15:51:21.267 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0021167489999999998 2024-09-21T15:51:21.267 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.267 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.267 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:21.267 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.690943+0000", 2024-09-21T15:51:21.267 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1446999999999998e-05 2024-09-21T15:51:21.267 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.267 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.267 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:21.268 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.690968+0000", 2024-09-21T15:51:21.268 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4997000000000001e-05 2024-09-21T15:51:21.268 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.268 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:21.268 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.268 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.268 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.268 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:167071 3.1 3:851adcc3:::rbd_data.1133896adcfd.0000000000000005:head [delete] snapc 8=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-21T15:51:21.268 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:51:20.767069+0000", 2024-09-21T15:51:21.268 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.28766707699999999, 2024-09-21T15:51:21.268 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0090679330000000002, 2024-09-21T15:51:21.268 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:21.268 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:21.268 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:21.268 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:21.269 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:21.269 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 167071 2024-09-21T15:51:21.269 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.269 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:21.269 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.269 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:21.269 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.767069+0000", 2024-09-21T15:51:21.269 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.269 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.269 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.269 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:21.269 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.767069+0000", 2024-09-21T15:51:21.269 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.269 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.269 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.269 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:21.270 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.767068+0000", 2024-09-21T15:51:21.270 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:21.270 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.270 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.270 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:21.270 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.767073+0000", 2024-09-21T15:51:21.270 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.259e-06 2024-09-21T15:51:21.270 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.270 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.270 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:21.270 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.767083+0000", 2024-09-21T15:51:21.270 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0156000000000001e-05 2024-09-21T15:51:21.270 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.271 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.271 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:21.271 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.767089+0000", 2024-09-21T15:51:21.271 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.784e-06 2024-09-21T15:51:21.271 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.271 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.271 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.271 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.767152+0000", 2024-09-21T15:51:21.271 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2675000000000006e-05 2024-09-21T15:51:21.271 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.271 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.271 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:21.271 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.767326+0000", 2024-09-21T15:51:21.271 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000173962 2024-09-21T15:51:21.272 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.272 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.272 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-09-21T15:51:21.272 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.767420+0000", 2024-09-21T15:51:21.272 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4473000000000002e-05 2024-09-21T15:51:21.272 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.272 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.272 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:21.272 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.769013+0000", 2024-09-21T15:51:21.272 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001592518 2024-09-21T15:51:21.272 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.272 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.272 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:21.272 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.775716+0000", 2024-09-21T15:51:21.273 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0067037219999999996 2024-09-21T15:51:21.273 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.273 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.273 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:21.273 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.775756+0000", 2024-09-21T15:51:21.273 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9669000000000003e-05 2024-09-21T15:51:21.273 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.273 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.273 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:21.273 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.776137+0000", 2024-09-21T15:51:21.273 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038067 2024-09-21T15:51:21.273 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.273 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:21.273 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.273 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.274 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.274 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:167084 3.17 3:ebc2be9c:::rbd_data.1133896adcfd.0000000000000014:head [delete] snapc 8=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-21T15:51:21.274 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:51:20.773031+0000", 2024-09-21T15:51:21.274 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.28170514200000002, 2024-09-21T15:51:21.274 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053046810000000003, 2024-09-21T15:51:21.274 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:21.274 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:21.274 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:21.274 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:21.274 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:21.274 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 167084 2024-09-21T15:51:21.274 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.274 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:21.274 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.274 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:21.274 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.773031+0000", 2024-09-21T15:51:21.275 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.275 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.275 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.275 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:21.275 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.773031+0000", 2024-09-21T15:51:21.275 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.275 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.275 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.275 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:21.275 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.773030+0000", 2024-09-21T15:51:21.275 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:21.275 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.275 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.275 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:21.275 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.773036+0000", 2024-09-21T15:51:21.275 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7910000000000001e-06 2024-09-21T15:51:21.276 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.276 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.276 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:21.276 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.773046+0000", 2024-09-21T15:51:21.276 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.8700000000000004e-06 2024-09-21T15:51:21.276 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.276 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.276 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:21.276 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.773053+0000", 2024-09-21T15:51:21.276 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5390000000000002e-06 2024-09-21T15:51:21.276 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.276 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.276 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.276 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.773094+0000", 2024-09-21T15:51:21.276 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1307000000000003e-05 2024-09-21T15:51:21.276 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.277 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.277 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:21.277 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.773257+0000", 2024-09-21T15:51:21.277 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00016289899999999999 2024-09-21T15:51:21.277 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.277 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.277 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-09-21T15:51:21.277 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.773364+0000", 2024-09-21T15:51:21.277 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010628600000000001 2024-09-21T15:51:21.277 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.277 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.277 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:21.277 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.774168+0000", 2024-09-21T15:51:21.277 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00080437100000000001 2024-09-21T15:51:21.277 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.277 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.278 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:21.278 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.778269+0000", 2024-09-21T15:51:21.278 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0041012710000000001 2024-09-21T15:51:21.278 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.278 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.278 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:21.278 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.778300+0000", 2024-09-21T15:51:21.278 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0988999999999998e-05 2024-09-21T15:51:21.278 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.278 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.278 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:21.278 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.778335+0000", 2024-09-21T15:51:21.278 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5287000000000003e-05 2024-09-21T15:51:21.278 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.278 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:21.278 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.279 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.279 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.279 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:167092 3.1 3:87b7471f:::rbd_data.1133896adcfd.000000000000001c:head [delete] snapc 8=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-21T15:51:21.279 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:51:20.775925+0000", 2024-09-21T15:51:21.279 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.27881065500000002, 2024-09-21T15:51:21.279 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0054501690000000004, 2024-09-21T15:51:21.279 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:21.279 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:21.279 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:21.279 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:21.279 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:21.279 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 167092 2024-09-21T15:51:21.279 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.279 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:21.279 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.280 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:21.280 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.775925+0000", 2024-09-21T15:51:21.280 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.280 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.280 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.280 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:21.280 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.775925+0000", 2024-09-21T15:51:21.280 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.280 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.280 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.280 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:21.280 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.775924+0000", 2024-09-21T15:51:21.280 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:21.280 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.280 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.280 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:21.280 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.775929+0000", 2024-09-21T15:51:21.281 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1229999999999999e-06 2024-09-21T15:51:21.281 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.281 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.281 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:21.281 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.775940+0000", 2024-09-21T15:51:21.281 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0495e-05 2024-09-21T15:51:21.281 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.281 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.281 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:21.281 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.775945+0000", 2024-09-21T15:51:21.281 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7870000000000001e-06 2024-09-21T15:51:21.281 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.281 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.281 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.281 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.776184+0000", 2024-09-21T15:51:21.282 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000239196 2024-09-21T15:51:21.282 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.282 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.282 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:21.282 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.776343+0000", 2024-09-21T15:51:21.282 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000159145 2024-09-21T15:51:21.282 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.282 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.282 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-09-21T15:51:21.282 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.776450+0000", 2024-09-21T15:51:21.282 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010710899999999999 2024-09-21T15:51:21.282 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.282 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.282 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:21.282 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.777106+0000", 2024-09-21T15:51:21.283 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00065567399999999997 2024-09-21T15:51:21.283 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.283 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.283 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:21.283 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.781295+0000", 2024-09-21T15:51:21.283 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0041886120000000004 2024-09-21T15:51:21.283 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.283 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.283 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:21.283 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.781338+0000", 2024-09-21T15:51:21.283 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3906999999999998e-05 2024-09-21T15:51:21.283 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.283 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.283 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:21.283 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.781375+0000", 2024-09-21T15:51:21.283 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7030999999999998e-05 2024-09-21T15:51:21.283 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.284 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:21.284 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.284 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.284 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.284 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:167098 3.1b 3:d941edf7:::rbd_data.1133896adcfd.0000000000000022:head [delete] snapc 8=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-21T15:51:21.284 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:51:20.778455+0000", 2024-09-21T15:51:21.284 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 0.27628114399999998, 2024-09-21T15:51:21.284 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052789760000000003, 2024-09-21T15:51:21.284 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:21.284 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:21.284 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:21.284 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:21.284 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:21.284 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 167098 2024-09-21T15:51:21.284 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.284 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:21.285 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.285 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:21.285 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.778455+0000", 2024-09-21T15:51:21.285 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.285 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.285 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.285 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:21.285 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.778455+0000", 2024-09-21T15:51:21.285 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.285 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.285 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.285 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:21.285 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.778454+0000", 2024-09-21T15:51:21.285 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:21.285 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.285 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.286 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:21.286 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.778457+0000", 2024-09-21T15:51:21.286 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2579999999999998e-06 2024-09-21T15:51:21.286 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.286 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.286 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:21.286 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.778461+0000", 2024-09-21T15:51:21.286 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1609999999999996e-06 2024-09-21T15:51:21.286 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.286 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.286 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:21.286 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.778464+0000", 2024-09-21T15:51:21.286 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0690000000000002e-06 2024-09-21T15:51:21.286 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.286 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.287 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.287 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.778496+0000", 2024-09-21T15:51:21.287 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1831000000000001e-05 2024-09-21T15:51:21.287 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.287 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.287 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:21.287 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.778610+0000", 2024-09-21T15:51:21.287 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000114252 2024-09-21T15:51:21.287 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.287 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.287 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-09-21T15:51:21.287 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.778679+0000", 2024-09-21T15:51:21.287 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8717e-05 2024-09-21T15:51:21.287 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.287 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.287 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:21.288 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.779456+0000", 2024-09-21T15:51:21.288 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00077764399999999997 2024-09-21T15:51:21.288 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.288 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.288 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:21.288 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.783684+0000", 2024-09-21T15:51:21.288 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0042275519999999999 2024-09-21T15:51:21.288 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.288 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.288 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:21.288 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.783712+0000", 2024-09-21T15:51:21.288 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.792e-05 2024-09-21T15:51:21.288 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.288 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.288 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:21.288 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.783734+0000", 2024-09-21T15:51:21.289 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1942000000000001e-05 2024-09-21T15:51:21.289 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.289 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:21.289 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.289 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.289 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:21.289 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:21.290 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:51:21.289+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:51:21.290 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:51:21.391 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:51:21.390+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:51:21.432 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:21.432 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:51:21.432 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:51:21.432 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:21.441 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:51:21.491 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:51:21.490+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:51:21.542 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:21.542 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:51:21.542 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:51:21.542 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:51:21.542 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:21.550 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:51:21.592 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:51:21.590+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:51:21.683 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is set 2024-09-21T15:51:21.693 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:51:21.692+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:51:21.699 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:21.699 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:51:21.699 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:51:21.699 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:51:21.699 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.699 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62849 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 208896~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:21.699 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.144181+0000", 2024-09-21T15:51:21.699 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 104.552908519, 2024-09-21T15:51:21.699 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050460390000000001, 2024-09-21T15:51:21.699 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:21.700 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:21.700 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:21.700 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:21.700 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:21.700 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62849 2024-09-21T15:51:21.700 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.700 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:21.700 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.700 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:21.700 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144181+0000", 2024-09-21T15:51:21.700 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.700 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.700 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.700 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:21.701 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144181+0000", 2024-09-21T15:51:21.701 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.701 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.701 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.701 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:21.701 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144181+0000", 2024-09-21T15:51:21.701 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:21.701 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.701 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.701 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:21.701 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144183+0000", 2024-09-21T15:51:21.701 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3860000000000001e-06 2024-09-21T15:51:21.701 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.701 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.701 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:21.702 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144189+0000", 2024-09-21T15:51:21.702 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0599999999999996e-06 2024-09-21T15:51:21.702 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.702 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.702 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:21.702 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144194+0000", 2024-09-21T15:51:21.702 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4379999999999997e-06 2024-09-21T15:51:21.702 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.702 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.702 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.702 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.194457+0000", 2024-09-21T15:51:21.702 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050263317000000002 2024-09-21T15:51:21.702 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.702 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.703 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:21.703 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.194548+0000", 2024-09-21T15:51:21.703 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0785000000000006e-05 2024-09-21T15:51:21.703 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.703 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.703 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:21.703 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.194642+0000", 2024-09-21T15:51:21.703 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3837000000000003e-05 2024-09-21T15:51:21.703 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.703 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:21.703 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.703 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.703 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.703 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62862 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 954368~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:21.704 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.146176+0000", 2024-09-21T15:51:21.704 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 104.550913456, 2024-09-21T15:51:21.704 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050442901999999998, 2024-09-21T15:51:21.704 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:21.704 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:21.704 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:21.704 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:21.704 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:21.704 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62862 2024-09-21T15:51:21.704 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.704 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:21.704 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.704 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:21.704 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146176+0000", 2024-09-21T15:51:21.704 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.704 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.705 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.705 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:21.705 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146176+0000", 2024-09-21T15:51:21.705 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.705 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.705 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.705 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:21.705 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146176+0000", 2024-09-21T15:51:21.705 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:21.705 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.705 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.705 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:21.705 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146178+0000", 2024-09-21T15:51:21.705 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4739999999999999e-06 2024-09-21T15:51:21.705 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.705 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.706 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:21.706 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146186+0000", 2024-09-21T15:51:21.706 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4989999999999997e-06 2024-09-21T15:51:21.706 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.706 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.706 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:21.706 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146191+0000", 2024-09-21T15:51:21.706 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7999999999999998e-06 2024-09-21T15:51:21.706 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.706 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.706 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.706 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196442+0000", 2024-09-21T15:51:21.706 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050251665000000001 2024-09-21T15:51:21.706 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.707 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.707 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:21.707 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196518+0000", 2024-09-21T15:51:21.707 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5677000000000001e-05 2024-09-21T15:51:21.707 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.707 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.707 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:21.707 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196619+0000", 2024-09-21T15:51:21.707 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101247 2024-09-21T15:51:21.707 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.707 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:21.707 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.707 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.707 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.707 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62863 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 937984~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:21.708 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.146195+0000", 2024-09-21T15:51:21.708 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 104.55089526499999, 2024-09-21T15:51:21.708 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050642305999999998, 2024-09-21T15:51:21.708 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:21.708 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:21.708 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:21.708 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:21.708 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:21.708 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62863 2024-09-21T15:51:21.708 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.708 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:21.708 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.708 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:21.708 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146195+0000", 2024-09-21T15:51:21.708 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.708 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.709 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.709 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:21.709 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146195+0000", 2024-09-21T15:51:21.709 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.709 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.709 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.709 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:21.709 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146194+0000", 2024-09-21T15:51:21.709 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:21.709 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.709 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.709 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:21.709 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146196+0000", 2024-09-21T15:51:21.709 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1900000000000002e-06 2024-09-21T15:51:21.709 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.709 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.710 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:21.710 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146202+0000", 2024-09-21T15:51:21.710 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.57e-06 2024-09-21T15:51:21.710 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.710 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.710 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:21.710 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146206+0000", 2024-09-21T15:51:21.710 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5079999999999999e-06 2024-09-21T15:51:21.710 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.710 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.710 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.710 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196655+0000", 2024-09-21T15:51:21.710 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050448973000000001 2024-09-21T15:51:21.710 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.710 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.710 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:21.711 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196730+0000", 2024-09-21T15:51:21.711 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4728999999999996e-05 2024-09-21T15:51:21.711 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.711 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.711 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:21.711 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196837+0000", 2024-09-21T15:51:21.711 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000106735 2024-09-21T15:51:21.711 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.711 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:21.711 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.711 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.711 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.711 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62872 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 290816~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:21.711 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148149+0000", 2024-09-21T15:51:21.711 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 104.548940589, 2024-09-21T15:51:21.712 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.05046991, 2024-09-21T15:51:21.712 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:21.712 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:21.712 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:21.712 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:21.712 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:21.712 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62872 2024-09-21T15:51:21.712 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.712 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:21.712 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.712 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:21.712 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148149+0000", 2024-09-21T15:51:21.712 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.712 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.712 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.713 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:21.713 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148149+0000", 2024-09-21T15:51:21.713 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.713 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.713 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.713 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:21.713 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148149+0000", 2024-09-21T15:51:21.713 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:21.713 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.713 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.713 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:21.713 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148157+0000", 2024-09-21T15:51:21.713 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4749999999999993e-06 2024-09-21T15:51:21.713 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.713 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.714 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:21.714 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148163+0000", 2024-09-21T15:51:21.714 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8250000000000001e-06 2024-09-21T15:51:21.714 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.714 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.714 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:21.714 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148168+0000", 2024-09-21T15:51:21.714 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4730000000000002e-06 2024-09-21T15:51:21.714 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.714 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.714 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.714 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198471+0000", 2024-09-21T15:51:21.714 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050302974 2024-09-21T15:51:21.715 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.715 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.715 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:21.715 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198555+0000", 2024-09-21T15:51:21.715 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4783999999999993e-05 2024-09-21T15:51:21.715 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.715 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.715 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:21.715 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198619+0000", 2024-09-21T15:51:21.715 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3751000000000007e-05 2024-09-21T15:51:21.715 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.715 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:21.715 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.715 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.715 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.716 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62873 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 315392~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:21.716 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148172+0000", 2024-09-21T15:51:21.716 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 104.548918298, 2024-09-21T15:51:21.716 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050707998999999997, 2024-09-21T15:51:21.716 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:21.716 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:21.716 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:21.716 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:21.716 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:21.716 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62873 2024-09-21T15:51:21.716 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.716 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:21.716 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.716 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:21.716 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148172+0000", 2024-09-21T15:51:21.716 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.717 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.717 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.717 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:21.717 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148172+0000", 2024-09-21T15:51:21.717 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.717 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.717 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.717 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:21.717 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148171+0000", 2024-09-21T15:51:21.717 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:21.717 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.717 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.717 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:21.717 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148173+0000", 2024-09-21T15:51:21.718 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2340000000000001e-06 2024-09-21T15:51:21.718 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.718 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.718 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:21.718 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148179+0000", 2024-09-21T15:51:21.718 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9340000000000003e-06 2024-09-21T15:51:21.718 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.718 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.718 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:21.718 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148186+0000", 2024-09-21T15:51:21.718 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2349999999999996e-06 2024-09-21T15:51:21.718 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.718 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.718 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.719 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198684+0000", 2024-09-21T15:51:21.719 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050498890999999997 2024-09-21T15:51:21.719 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.719 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.719 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:21.719 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198778+0000", 2024-09-21T15:51:21.719 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3134999999999995e-05 2024-09-21T15:51:21.719 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.719 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.719 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:21.719 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198880+0000", 2024-09-21T15:51:21.719 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101906 2024-09-21T15:51:21.719 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.719 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:21.719 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.720 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.720 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.720 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62874 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 323584~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:21.720 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148211+0000", 2024-09-21T15:51:21.720 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 104.548878538, 2024-09-21T15:51:21.720 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050887766000000001, 2024-09-21T15:51:21.720 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:21.720 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:21.720 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:21.720 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:21.720 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:21.720 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62874 2024-09-21T15:51:21.720 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.720 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:21.720 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.721 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:21.721 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148211+0000", 2024-09-21T15:51:21.721 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.721 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.721 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.721 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:21.721 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148211+0000", 2024-09-21T15:51:21.721 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.721 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.721 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.721 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:21.721 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148211+0000", 2024-09-21T15:51:21.721 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:21.721 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.722 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.722 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:21.722 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148213+0000", 2024-09-21T15:51:21.722 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6860000000000002e-06 2024-09-21T15:51:21.722 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.722 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.722 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:21.722 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148220+0000", 2024-09-21T15:51:21.722 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4740000000000002e-06 2024-09-21T15:51:21.722 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.722 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.722 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:21.722 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148226+0000", 2024-09-21T15:51:21.722 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0120000000000002e-06 2024-09-21T15:51:21.723 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.723 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.723 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.723 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198940+0000", 2024-09-21T15:51:21.723 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050714412 2024-09-21T15:51:21.723 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.723 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.723 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:21.723 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198995+0000", 2024-09-21T15:51:21.723 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4863e-05 2024-09-21T15:51:21.723 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.723 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.723 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:21.723 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.199099+0000", 2024-09-21T15:51:21.724 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010383299999999999 2024-09-21T15:51:21.724 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.724 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:21.724 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.724 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.724 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.724 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62877 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 995328~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:21.724 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148230+0000", 2024-09-21T15:51:21.724 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 104.548859678, 2024-09-21T15:51:21.724 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.051108949000000001, 2024-09-21T15:51:21.724 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:21.724 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:21.724 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:21.724 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:21.724 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:21.724 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62877 2024-09-21T15:51:21.725 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.725 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:21.725 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.725 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:21.725 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148230+0000", 2024-09-21T15:51:21.725 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.725 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.725 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.725 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:21.725 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148230+0000", 2024-09-21T15:51:21.725 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.725 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.725 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.725 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:21.725 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148230+0000", 2024-09-21T15:51:21.726 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:21.726 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.726 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.726 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:21.726 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148232+0000", 2024-09-21T15:51:21.726 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1220000000000002e-06 2024-09-21T15:51:21.726 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.726 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.726 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:21.726 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148238+0000", 2024-09-21T15:51:21.726 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.789e-06 2024-09-21T15:51:21.726 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.726 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.726 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:21.726 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148242+0000", 2024-09-21T15:51:21.727 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7369999999999997e-06 2024-09-21T15:51:21.727 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.727 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.727 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.727 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.199201+0000", 2024-09-21T15:51:21.727 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050959088 2024-09-21T15:51:21.727 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.727 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.727 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:21.727 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.199285+0000", 2024-09-21T15:51:21.727 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3147999999999997e-05 2024-09-21T15:51:21.727 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.727 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.727 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:21.727 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.199339+0000", 2024-09-21T15:51:21.728 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4484999999999998e-05 2024-09-21T15:51:21.728 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.728 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:21.728 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.728 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.728 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.728 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62888 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 331776~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:21.728 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150154+0000", 2024-09-21T15:51:21.728 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 104.546936084, 2024-09-21T15:51:21.728 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050483169000000001, 2024-09-21T15:51:21.728 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:21.728 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:21.728 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:21.728 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:21.728 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:21.728 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62888 2024-09-21T15:51:21.729 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.729 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:21.729 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.729 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:21.729 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150154+0000", 2024-09-21T15:51:21.729 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.729 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.729 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.729 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:21.729 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150154+0000", 2024-09-21T15:51:21.729 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.729 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.729 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.729 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:21.729 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150153+0000", 2024-09-21T15:51:21.729 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:21.730 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.730 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.730 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:21.730 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150155+0000", 2024-09-21T15:51:21.730 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9920000000000002e-06 2024-09-21T15:51:21.730 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.730 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.730 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:21.730 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150161+0000", 2024-09-21T15:51:21.730 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5439999999999998e-06 2024-09-21T15:51:21.730 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.730 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.730 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:21.730 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150165+0000", 2024-09-21T15:51:21.730 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3460000000000004e-06 2024-09-21T15:51:21.731 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.731 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.731 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.731 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200502+0000", 2024-09-21T15:51:21.731 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050337150999999997 2024-09-21T15:51:21.731 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.731 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.731 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:21.731 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200575+0000", 2024-09-21T15:51:21.731 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2250999999999997e-05 2024-09-21T15:51:21.731 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.731 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.731 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:21.731 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200637+0000", 2024-09-21T15:51:21.731 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2195999999999999e-05 2024-09-21T15:51:21.732 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.732 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:21.732 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.732 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.732 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.732 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62889 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 339968~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:21.732 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150169+0000", 2024-09-21T15:51:21.732 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 104.546920585, 2024-09-21T15:51:21.732 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050629678999999997, 2024-09-21T15:51:21.732 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:21.732 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:21.732 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:21.732 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:21.732 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:21.732 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62889 2024-09-21T15:51:21.733 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.733 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:21.733 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.733 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:21.733 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150169+0000", 2024-09-21T15:51:21.733 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.733 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.733 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.733 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:21.733 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150169+0000", 2024-09-21T15:51:21.733 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.733 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.733 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.733 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:21.733 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150169+0000", 2024-09-21T15:51:21.734 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:21.734 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.734 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.734 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:21.734 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150171+0000", 2024-09-21T15:51:21.734 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3910000000000001e-06 2024-09-21T15:51:21.734 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.734 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.734 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:21.734 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150177+0000", 2024-09-21T15:51:21.734 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4079999999999997e-06 2024-09-21T15:51:21.734 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.734 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.734 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:21.734 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150182+0000", 2024-09-21T15:51:21.734 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3890000000000004e-06 2024-09-21T15:51:21.735 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.735 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.735 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.735 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200664+0000", 2024-09-21T15:51:21.735 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050482212999999998 2024-09-21T15:51:21.735 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.735 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.735 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:21.735 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200738+0000", 2024-09-21T15:51:21.735 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4114000000000002e-05 2024-09-21T15:51:21.735 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.735 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.735 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:21.735 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200799+0000", 2024-09-21T15:51:21.735 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0503e-05 2024-09-21T15:51:21.736 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.736 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:21.736 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.736 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.736 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.736 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62891 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 348160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:21.736 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150186+0000", 2024-09-21T15:51:21.736 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 104.54690344300001, 2024-09-21T15:51:21.736 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050794552, 2024-09-21T15:51:21.736 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:21.736 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:21.736 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:21.736 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:21.736 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:21.736 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62891 2024-09-21T15:51:21.737 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.737 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:21.737 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.737 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:21.737 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150186+0000", 2024-09-21T15:51:21.737 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.737 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.737 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.737 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:21.737 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150186+0000", 2024-09-21T15:51:21.737 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.737 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.737 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.737 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:21.738 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150186+0000", 2024-09-21T15:51:21.738 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:21.738 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.738 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.738 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:21.738 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150193+0000", 2024-09-21T15:51:21.738 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6830000000000002e-06 2024-09-21T15:51:21.738 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.738 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.738 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:21.738 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150199+0000", 2024-09-21T15:51:21.738 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9859999999999999e-06 2024-09-21T15:51:21.738 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.738 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.738 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:21.739 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150203+0000", 2024-09-21T15:51:21.739 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5920000000000002e-06 2024-09-21T15:51:21.739 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.739 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.739 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.739 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200835+0000", 2024-09-21T15:51:21.739 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050631783 2024-09-21T15:51:21.739 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.739 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.739 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:21.739 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200894+0000", 2024-09-21T15:51:21.739 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8560999999999997e-05 2024-09-21T15:51:21.739 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.739 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.739 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:21.739 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200981+0000", 2024-09-21T15:51:21.740 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7305999999999994e-05 2024-09-21T15:51:21.740 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.740 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:21.740 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.740 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.740 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.740 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62892 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 356352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:21.740 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150208+0000", 2024-09-21T15:51:21.740 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 104.546882055, 2024-09-21T15:51:21.740 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050951769000000001, 2024-09-21T15:51:21.740 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:21.740 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:21.740 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:21.740 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:21.740 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:21.741 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62892 2024-09-21T15:51:21.741 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.741 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:21.741 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.741 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:21.741 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150208+0000", 2024-09-21T15:51:21.741 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.741 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.741 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.741 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:21.741 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150208+0000", 2024-09-21T15:51:21.741 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.741 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.741 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.741 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:21.741 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150207+0000", 2024-09-21T15:51:21.741 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:21.742 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.742 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.742 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:21.742 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150210+0000", 2024-09-21T15:51:21.742 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.238e-06 2024-09-21T15:51:21.742 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.742 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.742 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:21.742 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150215+0000", 2024-09-21T15:51:21.742 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6350000000000003e-06 2024-09-21T15:51:21.742 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.742 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.742 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:21.742 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150220+0000", 2024-09-21T15:51:21.742 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.544e-06 2024-09-21T15:51:21.743 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.743 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.743 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.743 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201014+0000", 2024-09-21T15:51:21.743 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050793932999999999 2024-09-21T15:51:21.743 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.743 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.743 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:21.743 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201092+0000", 2024-09-21T15:51:21.743 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7961999999999997e-05 2024-09-21T15:51:21.743 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.743 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.743 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:21.743 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201160+0000", 2024-09-21T15:51:21.743 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7836000000000006e-05 2024-09-21T15:51:21.744 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.744 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:21.744 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.744 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.744 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.744 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62894 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 364544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:21.744 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150820+0000", 2024-09-21T15:51:21.744 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 104.54626985500001, 2024-09-21T15:51:21.744 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050489196, 2024-09-21T15:51:21.744 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:21.744 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:21.744 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:21.744 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:21.744 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:21.744 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62894 2024-09-21T15:51:21.744 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.744 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:21.745 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.745 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:21.745 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150820+0000", 2024-09-21T15:51:21.745 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.745 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.745 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.745 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:21.745 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150820+0000", 2024-09-21T15:51:21.745 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.745 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.745 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.745 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:21.745 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150223+0000", 2024-09-21T15:51:21.745 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9994035 2024-09-21T15:51:21.745 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.745 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.746 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:21.746 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150825+0000", 2024-09-21T15:51:21.746 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00060137099999999996 2024-09-21T15:51:21.746 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.746 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.746 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:21.746 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150837+0000", 2024-09-21T15:51:21.746 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2503999999999999e-05 2024-09-21T15:51:21.746 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.746 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.746 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:21.746 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150844+0000", 2024-09-21T15:51:21.746 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6900000000000003e-06 2024-09-21T15:51:21.746 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.746 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.747 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.747 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201187+0000", 2024-09-21T15:51:21.747 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050342973999999999 2024-09-21T15:51:21.747 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.747 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.747 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:21.747 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201250+0000", 2024-09-21T15:51:21.747 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3274000000000004e-05 2024-09-21T15:51:21.747 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.747 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.747 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:21.747 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201309+0000", 2024-09-21T15:51:21.747 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9058000000000003e-05 2024-09-21T15:51:21.747 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.747 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:21.748 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.748 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.748 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.748 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62895 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 380928~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:21.748 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150851+0000", 2024-09-21T15:51:21.748 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 104.546239259, 2024-09-21T15:51:21.748 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050611097000000001, 2024-09-21T15:51:21.748 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:21.748 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:21.748 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:21.748 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:21.748 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:21.748 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62895 2024-09-21T15:51:21.748 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.748 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:21.749 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.749 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:21.749 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150851+0000", 2024-09-21T15:51:21.749 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.749 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.749 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.749 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:21.749 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150851+0000", 2024-09-21T15:51:21.749 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.749 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.749 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.749 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:21.749 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150850+0000", 2024-09-21T15:51:21.749 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:21.749 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.749 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.750 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:21.750 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150853+0000", 2024-09-21T15:51:21.750 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1039999999999998e-06 2024-09-21T15:51:21.750 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.750 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.750 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:21.750 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150860+0000", 2024-09-21T15:51:21.750 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3180000000000001e-06 2024-09-21T15:51:21.750 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.750 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.750 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:21.750 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150866+0000", 2024-09-21T15:51:21.750 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1599999999999997e-06 2024-09-21T15:51:21.750 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.750 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.751 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.751 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201332+0000", 2024-09-21T15:51:21.751 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050466100999999999 2024-09-21T15:51:21.751 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.751 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.751 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:21.751 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201392+0000", 2024-09-21T15:51:21.751 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0053999999999998e-05 2024-09-21T15:51:21.751 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.751 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.751 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:21.751 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201462+0000", 2024-09-21T15:51:21.751 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9861000000000001e-05 2024-09-21T15:51:21.751 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.752 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:21.752 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.752 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.752 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.752 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62896 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 372736~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:21.752 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150870+0000", 2024-09-21T15:51:21.752 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 104.546219479, 2024-09-21T15:51:21.752 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050753292999999998, 2024-09-21T15:51:21.752 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:21.752 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:21.752 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:21.752 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:21.752 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:21.752 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62896 2024-09-21T15:51:21.752 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.753 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:21.753 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.753 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:21.753 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150870+0000", 2024-09-21T15:51:21.753 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.753 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.753 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.753 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:21.753 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150870+0000", 2024-09-21T15:51:21.753 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.753 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.753 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.753 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:21.753 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150870+0000", 2024-09-21T15:51:21.753 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:21.753 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.754 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.754 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:21.754 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150872+0000", 2024-09-21T15:51:21.754 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.644e-06 2024-09-21T15:51:21.754 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.754 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.754 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:21.754 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150879+0000", 2024-09-21T15:51:21.754 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3509999999999998e-06 2024-09-21T15:51:21.754 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.754 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.754 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:21.754 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150884+0000", 2024-09-21T15:51:21.754 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9150000000000004e-06 2024-09-21T15:51:21.755 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.755 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.755 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.755 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201487+0000", 2024-09-21T15:51:21.755 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050603276000000003 2024-09-21T15:51:21.755 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.755 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.755 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:21.755 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201557+0000", 2024-09-21T15:51:21.755 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9847999999999999e-05 2024-09-21T15:51:21.755 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.755 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.755 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:21.755 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201624+0000", 2024-09-21T15:51:21.756 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6730999999999995e-05 2024-09-21T15:51:21.756 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.756 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:21.756 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.756 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.756 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.756 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62897 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 389120~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:21.756 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.151044+0000", 2024-09-21T15:51:21.756 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 104.546045439, 2024-09-21T15:51:21.756 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050720531999999999, 2024-09-21T15:51:21.756 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:21.756 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:21.756 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:21.756 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:21.757 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:21.757 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62897 2024-09-21T15:51:21.757 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.757 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:21.757 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.757 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:21.757 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151044+0000", 2024-09-21T15:51:21.757 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.757 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.757 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.757 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:21.757 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151044+0000", 2024-09-21T15:51:21.757 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.757 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.758 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.758 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:21.758 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150888+0000", 2024-09-21T15:51:21.758 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9998431 2024-09-21T15:51:21.758 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.758 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.758 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:21.758 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151048+0000", 2024-09-21T15:51:21.758 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00016077299999999999 2024-09-21T15:51:21.758 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.758 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.758 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:21.758 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151058+0000", 2024-09-21T15:51:21.758 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.465e-06 2024-09-21T15:51:21.759 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.759 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.759 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:21.759 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151064+0000", 2024-09-21T15:51:21.759 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3829999999999997e-06 2024-09-21T15:51:21.759 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.759 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.759 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.759 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201648+0000", 2024-09-21T15:51:21.759 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050583597000000001 2024-09-21T15:51:21.759 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.759 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.759 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:21.759 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201711+0000", 2024-09-21T15:51:21.759 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3616999999999998e-05 2024-09-21T15:51:21.760 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.760 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.760 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:21.760 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201765+0000", 2024-09-21T15:51:21.760 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3464000000000002e-05 2024-09-21T15:51:21.760 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.760 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:21.760 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.760 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.760 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.760 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:118351 3.4 3:25980f86:::rbd_data.1133545d3fa6.000000000000005a:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-09-21T15:51:21.760 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:58.975795+0000", 2024-09-21T15:51:21.760 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 82.721294804999999, 2024-09-21T15:51:21.760 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.99016351599999997, 2024-09-21T15:51:21.760 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:21.761 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:21.761 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:21.761 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:21.761 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:21.761 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 118351 2024-09-21T15:51:21.761 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.761 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:21.761 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.761 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:21.761 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975795+0000", 2024-09-21T15:51:21.761 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.761 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.761 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.761 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:21.761 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975795+0000", 2024-09-21T15:51:21.762 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.762 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.762 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.762 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:21.762 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975793+0000", 2024-09-21T15:51:21.762 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:51:21.762 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.762 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.762 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:21.762 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975800+0000", 2024-09-21T15:51:21.762 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8220000000000001e-06 2024-09-21T15:51:21.762 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.762 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.763 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:21.763 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975814+0000", 2024-09-21T15:51:21.763 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3893000000000001e-05 2024-09-21T15:51:21.763 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.763 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.763 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:21.763 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975821+0000", 2024-09-21T15:51:21.763 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1319999999999996e-06 2024-09-21T15:51:21.763 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.763 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.763 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.763 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975878+0000", 2024-09-21T15:51:21.763 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7238999999999999e-05 2024-09-21T15:51:21.763 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.763 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.763 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:51:21.764 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975889+0000", 2024-09-21T15:51:21.764 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0509000000000001e-05 2024-09-21T15:51:21.764 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.764 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.764 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.764 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.964573+0000", 2024-09-21T15:51:21.764 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.98868396300000005 2024-09-21T15:51:21.764 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.764 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.764 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:21.764 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.964672+0000", 2024-09-21T15:51:21.764 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9601999999999999e-05 2024-09-21T15:51:21.764 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.764 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.764 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-09-21T15:51:21.765 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.964758+0000", 2024-09-21T15:51:21.765 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5537000000000005e-05 2024-09-21T15:51:21.765 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.765 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.765 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:21.765 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965847+0000", 2024-09-21T15:51:21.765 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001089404 2024-09-21T15:51:21.765 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.765 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.765 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:21.765 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965890+0000", 2024-09-21T15:51:21.765 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2521999999999997e-05 2024-09-21T15:51:21.765 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.765 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.765 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:21.765 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965934+0000", 2024-09-21T15:51:21.766 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4489e-05 2024-09-21T15:51:21.766 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.766 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.766 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:21.766 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965959+0000", 2024-09-21T15:51:21.766 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4301999999999999e-05 2024-09-21T15:51:21.766 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.766 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:21.766 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.766 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.766 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.766 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:118353 3.4 3:23841e18:::rbd_data.1133545d3fa6.0000000000000061:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-09-21T15:51:21.766 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:58.975836+0000", 2024-09-21T15:51:21.766 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 82.721253305999994, 2024-09-21T15:51:21.766 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.99077746700000002, 2024-09-21T15:51:21.767 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:21.767 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:21.767 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:21.767 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:21.767 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:21.767 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 118353 2024-09-21T15:51:21.767 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.767 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:21.767 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.767 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:21.767 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975836+0000", 2024-09-21T15:51:21.767 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.767 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.767 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.767 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:21.768 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975836+0000", 2024-09-21T15:51:21.768 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.768 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.768 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.768 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:21.768 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975836+0000", 2024-09-21T15:51:21.768 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:21.768 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.768 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.768 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:21.768 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975839+0000", 2024-09-21T15:51:21.768 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6129999999999998e-06 2024-09-21T15:51:21.768 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.768 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.768 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:21.769 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975844+0000", 2024-09-21T15:51:21.769 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5130000000000004e-06 2024-09-21T15:51:21.769 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.769 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.769 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:21.769 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975850+0000", 2024-09-21T15:51:21.769 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3179999999999997e-06 2024-09-21T15:51:21.769 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.769 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.769 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.769 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975915+0000", 2024-09-21T15:51:21.769 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5168999999999999e-05 2024-09-21T15:51:21.769 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.769 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.769 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:51:21.770 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975922+0000", 2024-09-21T15:51:21.770 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2749999999999998e-06 2024-09-21T15:51:21.770 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.770 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.770 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.770 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965431+0000", 2024-09-21T15:51:21.770 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.98950864699999996 2024-09-21T15:51:21.770 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.770 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.770 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:21.770 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965535+0000", 2024-09-21T15:51:21.770 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000103949 2024-09-21T15:51:21.770 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.770 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.771 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-09-21T15:51:21.771 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965607+0000", 2024-09-21T15:51:21.771 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2478000000000006e-05 2024-09-21T15:51:21.771 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.771 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.771 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:21.771 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.966205+0000", 2024-09-21T15:51:21.771 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00059775200000000001 2024-09-21T15:51:21.771 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.771 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.771 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:21.771 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.966562+0000", 2024-09-21T15:51:21.771 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00035732199999999998 2024-09-21T15:51:21.771 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.771 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.771 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:21.772 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.966593+0000", 2024-09-21T15:51:21.772 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0642999999999998e-05 2024-09-21T15:51:21.772 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.772 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.772 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:21.772 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.966614+0000", 2024-09-21T15:51:21.772 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1149999999999999e-05 2024-09-21T15:51:21.772 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.772 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:21.772 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.772 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.772 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.772 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160535 3.6 3:67cae2ed:::rbd_mirroring:head [list-watchers] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-09-21T15:51:21.772 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:30.195408+0000", 2024-09-21T15:51:21.772 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 51.501682224, 2024-09-21T15:51:21.773 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.96025944900000004, 2024-09-21T15:51:21.773 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:21.773 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-09-21T15:51:21.773 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:21.773 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:21.773 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:21.773 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160535 2024-09-21T15:51:21.773 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.773 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:21.773 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.773 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:21.773 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195408+0000", 2024-09-21T15:51:21.773 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.773 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.774 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.774 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:21.774 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195408+0000", 2024-09-21T15:51:21.774 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.774 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.774 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.774 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:21.774 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195406+0000", 2024-09-21T15:51:21.774 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:51:21.774 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.774 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.774 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:21.774 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195414+0000", 2024-09-21T15:51:21.774 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.2870000000000004e-06 2024-09-21T15:51:21.775 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.775 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.775 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:21.775 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195432+0000", 2024-09-21T15:51:21.775 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8110000000000001e-05 2024-09-21T15:51:21.775 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.775 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.775 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:21.775 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195441+0000", 2024-09-21T15:51:21.775 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7679999999999997e-06 2024-09-21T15:51:21.775 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.775 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.775 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.775 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195505+0000", 2024-09-21T15:51:21.775 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3769000000000006e-05 2024-09-21T15:51:21.775 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.776 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.776 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:51:21.776 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195518+0000", 2024-09-21T15:51:21.776 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3341e-05 2024-09-21T15:51:21.776 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.776 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.776 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.776 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.155509+0000", 2024-09-21T15:51:21.776 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.959990393 2024-09-21T15:51:21.776 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.776 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.776 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:21.776 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.155667+0000", 2024-09-21T15:51:21.776 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000158505 2024-09-21T15:51:21.776 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.777 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:21.777 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.777 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.777 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.777 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160974 3.4 3:22dd2475:::rbd_id.image_client.0-clone39:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-21T15:51:21.777 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:34.748093+0000", 2024-09-21T15:51:21.777 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 46.948996673000003, 2024-09-21T15:51:21.777 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.94006199499999998, 2024-09-21T15:51:21.777 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:21.777 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:21.777 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:21.777 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:21.777 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:21.777 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160974 2024-09-21T15:51:21.777 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.778 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:21.778 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.778 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:21.778 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748093+0000", 2024-09-21T15:51:21.778 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.778 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.778 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.778 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:21.778 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748093+0000", 2024-09-21T15:51:21.778 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.778 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.778 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.778 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:21.778 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748091+0000", 2024-09-21T15:51:21.778 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:51:21.779 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.779 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.779 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:21.779 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748101+0000", 2024-09-21T15:51:21.779 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5950000000000005e-06 2024-09-21T15:51:21.779 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.779 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.779 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:21.779 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748122+0000", 2024-09-21T15:51:21.779 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1611e-05 2024-09-21T15:51:21.779 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.779 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.779 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:21.779 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748133+0000", 2024-09-21T15:51:21.779 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0224000000000001e-05 2024-09-21T15:51:21.780 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.780 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.780 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.780 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748185+0000", 2024-09-21T15:51:21.780 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2005000000000001e-05 2024-09-21T15:51:21.780 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.780 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.780 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:51:21.780 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748194+0000", 2024-09-21T15:51:21.780 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0180000000000007e-06 2024-09-21T15:51:21.780 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.780 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.780 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.780 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.683582+0000", 2024-09-21T15:51:21.780 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.93538800499999997 2024-09-21T15:51:21.781 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.781 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.781 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:21.781 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.683711+0000", 2024-09-21T15:51:21.781 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000129746 2024-09-21T15:51:21.781 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.781 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.781 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-09-21T15:51:21.781 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.683838+0000", 2024-09-21T15:51:21.781 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012693200000000001 2024-09-21T15:51:21.781 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.781 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.781 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:21.781 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.687445+0000", 2024-09-21T15:51:21.781 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0036071340000000001 2024-09-21T15:51:21.782 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.782 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.782 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:21.782 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.688064+0000", 2024-09-21T15:51:21.782 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00061857100000000005 2024-09-21T15:51:21.782 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.782 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.782 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:21.782 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.688113+0000", 2024-09-21T15:51:21.782 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9128000000000001e-05 2024-09-21T15:51:21.782 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.782 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.782 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:21.782 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.688155+0000", 2024-09-21T15:51:21.783 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2026e-05 2024-09-21T15:51:21.783 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.783 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:21.783 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.783 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.783 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.783 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:165681 3.19 3:9c73a274:::rbd_object_map.11336829dd7d.000000000000000f:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-09-21T15:51:21.783 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:51:11.936427+0000", 2024-09-21T15:51:21.783 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 9.7606628680000007, 2024-09-21T15:51:21.783 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.75328927499999998, 2024-09-21T15:51:21.783 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:21.783 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:21.783 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:21.783 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:21.783 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:21.784 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 165681 2024-09-21T15:51:21.784 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.784 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:21.784 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.784 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:21.784 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:11.936427+0000", 2024-09-21T15:51:21.784 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.784 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.784 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.784 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:21.784 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:11.936427+0000", 2024-09-21T15:51:21.784 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:21.784 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.784 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.784 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:21.785 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:11.936425+0000", 2024-09-21T15:51:21.785 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-09-21T15:51:21.785 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.785 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.785 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:21.785 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:11.936436+0000", 2024-09-21T15:51:21.785 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1898e-05 2024-09-21T15:51:21.785 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.785 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.785 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:21.785 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:11.936456+0000", 2024-09-21T15:51:21.785 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9568999999999998e-05 2024-09-21T15:51:21.785 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.785 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.785 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:21.786 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:11.936467+0000", 2024-09-21T15:51:21.786 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.058e-05 2024-09-21T15:51:21.786 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.786 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.786 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.786 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:11.936523+0000", 2024-09-21T15:51:21.786 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5961000000000002e-05 2024-09-21T15:51:21.786 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.786 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.786 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:51:21.786 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:11.936533+0000", 2024-09-21T15:51:21.786 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0234999999999999e-05 2024-09-21T15:51:21.786 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.786 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.787 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:21.787 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:12.689430+0000", 2024-09-21T15:51:21.787 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.75289766499999999 2024-09-21T15:51:21.787 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.787 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.787 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:21.787 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:12.689563+0000", 2024-09-21T15:51:21.787 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013278199999999999 2024-09-21T15:51:21.787 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:21.787 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:21.787 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:21.787 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:12.689716+0000", 2024-09-21T15:51:21.787 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015297499999999999 2024-09-21T15:51:21.787 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.787 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:21.787 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.788 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:21.788 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:21.788 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:21.789 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:51:21.830 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:51:21.829+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:51:21.895 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:51:21.894+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:51:21.933 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:21.934 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:51:21.934 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:51:21.934 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:21.942 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:51:22.026 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:51:22.025+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:51:22.084 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:22.084 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:51:22.084 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:51:22.084 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:51:22.084 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:22.092 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:51:22.133 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:51:22.132+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:51:22.197 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:51:22.196+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:51:22.235 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:22.235 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:51:22.235 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:51:22.235 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:51:22.235 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.235 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:4957 3.9 e29/22 3:95c5ed95:::rbd_data.11335fa2fe69.0000000000000018:head v 29'4, mlcod=26'2)", 2024-09-21T15:51:22.235 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:14.453951+0000", 2024-09-21T15:51:22.235 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 127.77922012800001, 2024-09-21T15:51:22.235 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0043804559999999996, 2024-09-21T15:51:22.236 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:22.236 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:22.236 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:22.236 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.236 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:22.236 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.453951+0000", 2024-09-21T15:51:22.236 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.236 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.236 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.236 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:22.236 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.453951+0000", 2024-09-21T15:51:22.236 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.236 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.237 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.237 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:22.237 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.453950+0000", 2024-09-21T15:51:22.237 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:22.237 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.237 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.237 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:22.237 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454495+0000", 2024-09-21T15:51:22.237 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00054437799999999998 2024-09-21T15:51:22.237 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.237 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.237 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:22.237 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454508+0000", 2024-09-21T15:51:22.237 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3475000000000001e-05 2024-09-21T15:51:22.237 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.238 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.238 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:22.238 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454515+0000", 2024-09-21T15:51:22.238 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8990000000000004e-06 2024-09-21T15:51:22.238 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.238 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.238 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:22.238 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454581+0000", 2024-09-21T15:51:22.238 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5987000000000004e-05 2024-09-21T15:51:22.238 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.238 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.238 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:22.238 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454609+0000", 2024-09-21T15:51:22.239 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7466000000000001e-05 2024-09-21T15:51:22.239 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.239 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.239 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:22.239 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.458303+0000", 2024-09-21T15:51:22.239 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003693979 2024-09-21T15:51:22.239 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.239 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.239 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:22.239 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.458331+0000", 2024-09-21T15:51:22.239 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8932999999999999e-05 2024-09-21T15:51:22.239 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.239 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:22.239 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.240 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.240 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.240 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13420 3.0 e31/22 3:038b8840:::rbd_data.1133dc5adfdf.0000000000000035:head v 31'49, mlcod=31'47)", 2024-09-21T15:51:22.240 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.209233+0000", 2024-09-21T15:51:22.240 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 125.023938512, 2024-09-21T15:51:22.240 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0042624960000000002, 2024-09-21T15:51:22.240 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:22.240 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:22.240 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:22.240 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.240 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:22.240 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209233+0000", 2024-09-21T15:51:22.240 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.240 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.240 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.241 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:22.241 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209233+0000", 2024-09-21T15:51:22.241 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.241 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.241 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.241 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:22.241 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209232+0000", 2024-09-21T15:51:22.241 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:22.241 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.241 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.241 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:22.241 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209251+0000", 2024-09-21T15:51:22.241 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9304e-05 2024-09-21T15:51:22.241 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.241 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.242 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:22.242 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209265+0000", 2024-09-21T15:51:22.242 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3532e-05 2024-09-21T15:51:22.242 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.242 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.242 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:22.242 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209272+0000", 2024-09-21T15:51:22.242 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.289e-06 2024-09-21T15:51:22.242 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.242 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.242 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:22.242 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209321+0000", 2024-09-21T15:51:22.242 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8773999999999998e-05 2024-09-21T15:51:22.242 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.243 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.243 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:22.243 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209341+0000", 2024-09-21T15:51:22.243 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0387999999999999e-05 2024-09-21T15:51:22.243 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.243 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.243 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:22.243 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.213464+0000", 2024-09-21T15:51:22.243 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0041232450000000002 2024-09-21T15:51:22.243 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.243 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.243 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:22.243 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.213495+0000", 2024-09-21T15:51:22.243 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0793999999999997e-05 2024-09-21T15:51:22.243 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.244 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:22.244 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.244 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.244 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.244 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13445 3.0 e31/22 3:038b8840:::rbd_data.1133dc5adfdf.0000000000000035:head v 31'50, mlcod=31'48)", 2024-09-21T15:51:22.244 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.232497+0000", 2024-09-21T15:51:22.244 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 125.0006745, 2024-09-21T15:51:22.244 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0040493559999999996, 2024-09-21T15:51:22.244 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:22.244 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:22.244 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:22.244 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.244 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:22.244 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232497+0000", 2024-09-21T15:51:22.244 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.245 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.245 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.245 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:22.245 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232497+0000", 2024-09-21T15:51:22.245 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.245 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.245 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.245 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:22.245 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232496+0000", 2024-09-21T15:51:22.245 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:22.245 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.245 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.245 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:22.245 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232503+0000", 2024-09-21T15:51:22.245 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7320000000000001e-06 2024-09-21T15:51:22.245 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.245 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.246 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:22.246 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232512+0000", 2024-09-21T15:51:22.246 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9259999999999995e-06 2024-09-21T15:51:22.246 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.246 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.246 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:22.246 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232520+0000", 2024-09-21T15:51:22.246 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3699999999999997e-06 2024-09-21T15:51:22.246 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.246 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.246 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:22.246 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232564+0000", 2024-09-21T15:51:22.246 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4128999999999997e-05 2024-09-21T15:51:22.246 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.246 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.247 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:22.247 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232585+0000", 2024-09-21T15:51:22.247 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0704999999999999e-05 2024-09-21T15:51:22.247 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.247 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.247 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:22.247 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.236507+0000", 2024-09-21T15:51:22.247 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0039218730000000002 2024-09-21T15:51:22.247 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.247 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.247 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:22.247 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.236546+0000", 2024-09-21T15:51:22.247 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9399e-05 2024-09-21T15:51:22.247 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.247 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:22.247 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.248 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.248 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.248 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13832 3.0 e31/22 3:0792efd2:::rbd_data.1133dc5adfdf.0000000000000090:head v 31'58, mlcod=31'56)", 2024-09-21T15:51:22.248 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.527530+0000", 2024-09-21T15:51:22.248 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 124.705640974, 2024-09-21T15:51:22.248 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0035256530000000001, 2024-09-21T15:51:22.248 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:22.248 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:22.248 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:22.248 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.248 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:22.248 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527530+0000", 2024-09-21T15:51:22.248 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.248 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.249 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.249 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:22.249 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527530+0000", 2024-09-21T15:51:22.249 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.249 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.249 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.249 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:22.249 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527529+0000", 2024-09-21T15:51:22.249 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:51:22.249 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.249 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.249 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:22.249 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527550+0000", 2024-09-21T15:51:22.249 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1387999999999999e-05 2024-09-21T15:51:22.250 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.250 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.250 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:22.250 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527568+0000", 2024-09-21T15:51:22.250 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8068e-05 2024-09-21T15:51:22.250 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.250 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.250 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:22.250 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527577+0000", 2024-09-21T15:51:22.250 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9919999999999996e-06 2024-09-21T15:51:22.250 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.250 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.250 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:22.250 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527626+0000", 2024-09-21T15:51:22.250 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8597000000000002e-05 2024-09-21T15:51:22.251 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.251 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.251 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:22.251 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.527652+0000", 2024-09-21T15:51:22.251 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5993e-05 2024-09-21T15:51:22.251 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.251 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.251 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:22.251 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.531024+0000", 2024-09-21T15:51:22.251 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0033721910000000001 2024-09-21T15:51:22.251 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.251 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.251 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:22.251 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.531056+0000", 2024-09-21T15:51:22.252 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1628000000000003e-05 2024-09-21T15:51:22.252 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.252 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:22.252 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.252 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.252 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.252 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13839 3.0 e31/22 3:03c4fb32:::rbd_data.1133dc5adfdf.0000000000000099:head v 31'59, mlcod=31'56)", 2024-09-21T15:51:22.252 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.528097+0000", 2024-09-21T15:51:22.252 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 124.705074376, 2024-09-21T15:51:22.252 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0057228840000000001, 2024-09-21T15:51:22.252 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:22.252 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:22.252 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:22.252 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.252 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:22.253 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528097+0000", 2024-09-21T15:51:22.253 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.253 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.253 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.253 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:22.253 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528097+0000", 2024-09-21T15:51:22.253 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.253 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.253 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.253 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:22.253 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528096+0000", 2024-09-21T15:51:22.253 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:22.253 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.253 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.254 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:22.254 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528105+0000", 2024-09-21T15:51:22.254 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7350000000000002e-06 2024-09-21T15:51:22.254 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.254 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.254 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:22.254 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528117+0000", 2024-09-21T15:51:22.254 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1909000000000001e-05 2024-09-21T15:51:22.254 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.254 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.254 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:22.254 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528124+0000", 2024-09-21T15:51:22.254 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8419999999999999e-06 2024-09-21T15:51:22.254 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.254 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.255 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:22.255 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528168+0000", 2024-09-21T15:51:22.255 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4301999999999997e-05 2024-09-21T15:51:22.255 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.255 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.255 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:22.255 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528191+0000", 2024-09-21T15:51:22.255 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2622000000000001e-05 2024-09-21T15:51:22.255 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.255 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.255 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:22.255 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.533780+0000", 2024-09-21T15:51:22.255 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0055897519999999999 2024-09-21T15:51:22.255 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.256 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.256 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:22.256 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.533820+0000", 2024-09-21T15:51:22.256 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9366999999999998e-05 2024-09-21T15:51:22.256 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.256 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:22.256 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.256 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.256 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.256 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13884 3.9 e31/22 3:93003736:::rbd_data.1133dc5adfdf.00000000000000a1:head v 31'54, mlcod=31'52)", 2024-09-21T15:51:22.256 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.562433+0000", 2024-09-21T15:51:22.256 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 124.670737628, 2024-09-21T15:51:22.256 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.005162313, 2024-09-21T15:51:22.256 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:22.256 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:22.257 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:22.257 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.257 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:22.257 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562433+0000", 2024-09-21T15:51:22.257 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.257 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.257 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.257 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:22.257 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562433+0000", 2024-09-21T15:51:22.257 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.257 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.257 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.257 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:22.257 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562432+0000", 2024-09-21T15:51:22.257 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:22.257 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.258 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.258 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:22.258 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562467+0000", 2024-09-21T15:51:22.258 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4885999999999998e-05 2024-09-21T15:51:22.258 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.258 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.258 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:22.258 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562481+0000", 2024-09-21T15:51:22.258 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3604000000000001e-05 2024-09-21T15:51:22.258 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.258 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.258 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:22.258 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562489+0000", 2024-09-21T15:51:22.258 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1829999999999994e-06 2024-09-21T15:51:22.259 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.259 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.259 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:22.259 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562536+0000", 2024-09-21T15:51:22.259 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6830999999999999e-05 2024-09-21T15:51:22.259 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.259 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.259 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:22.259 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562559+0000", 2024-09-21T15:51:22.259 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2765999999999999e-05 2024-09-21T15:51:22.259 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.259 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.259 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:22.259 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.567566+0000", 2024-09-21T15:51:22.259 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0050073219999999998 2024-09-21T15:51:22.260 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.260 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.260 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:22.260 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.567596+0000", 2024-09-21T15:51:22.260 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9802999999999999e-05 2024-09-21T15:51:22.260 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.260 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:22.260 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.260 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.260 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.260 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:22340 3.9 e34/22 3:910a5abe:::rbd_data.1133896adcfd.0000000000000023:head v 34'92, mlcod=34'90)", 2024-09-21T15:51:22.260 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:21.628025+0000", 2024-09-21T15:51:22.260 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 120.605146248, 2024-09-21T15:51:22.260 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046196010000000001, 2024-09-21T15:51:22.260 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:22.261 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:22.261 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:22.261 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.261 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:22.261 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628025+0000", 2024-09-21T15:51:22.261 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.261 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.261 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.261 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:22.261 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628025+0000", 2024-09-21T15:51:22.261 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.261 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.261 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.261 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:22.261 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628024+0000", 2024-09-21T15:51:22.262 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:22.262 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.262 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.262 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:22.262 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628034+0000", 2024-09-21T15:51:22.262 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3470000000000005e-06 2024-09-21T15:51:22.262 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.262 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.262 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:22.262 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628043+0000", 2024-09-21T15:51:22.262 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9450000000000005e-06 2024-09-21T15:51:22.262 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.262 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.263 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:22.263 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628048+0000", 2024-09-21T15:51:22.263 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8080000000000004e-06 2024-09-21T15:51:22.263 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.263 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.263 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:22.263 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628081+0000", 2024-09-21T15:51:22.263 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2738999999999999e-05 2024-09-21T15:51:22.263 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.263 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.263 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:22.263 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628094+0000", 2024-09-21T15:51:22.263 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3273e-05 2024-09-21T15:51:22.263 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.263 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.264 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:22.264 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.632616+0000", 2024-09-21T15:51:22.264 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045217269999999997 2024-09-21T15:51:22.264 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.264 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.264 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:22.264 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.632644+0000", 2024-09-21T15:51:22.264 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8469000000000001e-05 2024-09-21T15:51:22.264 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.264 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:22.264 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.264 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.264 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.264 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:22417 3.4 e34/22 3:25310a10:::rbd_data.1133896adcfd.0000000000000075:head v 34'48, mlcod=34'46)", 2024-09-21T15:51:22.264 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:21.704297+0000", 2024-09-21T15:51:22.265 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 120.52887415000001, 2024-09-21T15:51:22.265 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0038101799999999998, 2024-09-21T15:51:22.265 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:22.265 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:22.265 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:22.265 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.265 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:22.265 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704297+0000", 2024-09-21T15:51:22.265 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.265 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.265 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.265 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:22.265 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704297+0000", 2024-09-21T15:51:22.265 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.265 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.266 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.266 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:22.266 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704296+0000", 2024-09-21T15:51:22.266 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:51:22.266 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.266 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.266 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:22.266 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704308+0000", 2024-09-21T15:51:22.266 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2559e-05 2024-09-21T15:51:22.266 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.266 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.266 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:22.266 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704337+0000", 2024-09-21T15:51:22.267 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8960000000000001e-05 2024-09-21T15:51:22.267 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.267 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.267 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:22.267 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704345+0000", 2024-09-21T15:51:22.267 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3059999999999998e-06 2024-09-21T15:51:22.267 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.267 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.267 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:22.267 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704408+0000", 2024-09-21T15:51:22.267 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2156000000000006e-05 2024-09-21T15:51:22.267 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.267 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.267 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:22.267 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704430+0000", 2024-09-21T15:51:22.268 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1931000000000001e-05 2024-09-21T15:51:22.268 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.268 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.268 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:22.268 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.708073+0000", 2024-09-21T15:51:22.268 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003643028 2024-09-21T15:51:22.268 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.268 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.268 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:22.268 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.708107+0000", 2024-09-21T15:51:22.268 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4613999999999998e-05 2024-09-21T15:51:22.268 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.268 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:22.268 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.268 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.268 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.269 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:25278 3.4 e38/22 3:2251cc1c:::rbd_data.1133ebd6d3ba.0000000000000024:head v 38'51, mlcod=38'49)", 2024-09-21T15:51:22.269 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.317337+0000", 2024-09-21T15:51:22.269 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 115.91583439, 2024-09-21T15:51:22.269 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053904110000000003, 2024-09-21T15:51:22.269 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:22.269 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:22.269 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:22.269 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.269 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:22.269 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317337+0000", 2024-09-21T15:51:22.269 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.269 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.269 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.269 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:22.270 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317337+0000", 2024-09-21T15:51:22.270 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.270 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.270 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.270 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:22.270 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317335+0000", 2024-09-21T15:51:22.270 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:51:22.270 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.270 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.270 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:22.270 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317347+0000", 2024-09-21T15:51:22.270 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.19e-05 2024-09-21T15:51:22.270 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.270 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.271 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:22.271 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317369+0000", 2024-09-21T15:51:22.271 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2232e-05 2024-09-21T15:51:22.271 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.271 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.271 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:22.271 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317381+0000", 2024-09-21T15:51:22.271 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2231e-05 2024-09-21T15:51:22.271 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.271 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.271 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:22.271 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317425+0000", 2024-09-21T15:51:22.271 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3625000000000003e-05 2024-09-21T15:51:22.271 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.271 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.271 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:22.272 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317449+0000", 2024-09-21T15:51:22.272 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4301999999999999e-05 2024-09-21T15:51:22.272 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.272 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.272 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:22.272 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.322688+0000", 2024-09-21T15:51:22.272 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.005238609 2024-09-21T15:51:22.272 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.272 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.272 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:22.272 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.322727+0000", 2024-09-21T15:51:22.272 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9201e-05 2024-09-21T15:51:22.272 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.272 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:22.272 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.273 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.273 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.273 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:25286 3.0 e38/22 3:0248fb60:::rbd_data.1133ebd6d3ba.0000000000000066:head v 38'109, mlcod=34'107)", 2024-09-21T15:51:22.273 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.317638+0000", 2024-09-21T15:51:22.273 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 115.91553351100001, 2024-09-21T15:51:22.273 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0054074370000000002, 2024-09-21T15:51:22.273 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:22.273 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:22.273 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:22.273 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.273 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:22.273 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317638+0000", 2024-09-21T15:51:22.273 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.274 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.274 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.274 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:22.274 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317638+0000", 2024-09-21T15:51:22.274 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.274 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.274 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.274 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:22.274 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317637+0000", 2024-09-21T15:51:22.274 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:22.274 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.274 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.274 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:22.274 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317643+0000", 2024-09-21T15:51:22.275 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5609999999999998e-06 2024-09-21T15:51:22.275 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.275 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.275 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:22.275 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317653+0000", 2024-09-21T15:51:22.275 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0705e-05 2024-09-21T15:51:22.275 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.275 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.275 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:22.275 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317660+0000", 2024-09-21T15:51:22.275 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2099999999999998e-06 2024-09-21T15:51:22.275 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.275 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.275 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:22.275 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317704+0000", 2024-09-21T15:51:22.275 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4659000000000001e-05 2024-09-21T15:51:22.276 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.276 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.276 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:22.276 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317725+0000", 2024-09-21T15:51:22.276 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0537e-05 2024-09-21T15:51:22.276 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.276 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.276 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:22.276 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323010+0000", 2024-09-21T15:51:22.276 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052853420000000002 2024-09-21T15:51:22.276 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.276 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.276 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:22.276 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323045+0000", 2024-09-21T15:51:22.277 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4860000000000002e-05 2024-09-21T15:51:22.277 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.277 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:22.277 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.277 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.277 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.277 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:25280 3.4 e38/22 3:27fca38c:::rbd_data.1133ebd6d3ba.0000000000000026:head v 38'52, mlcod=38'49)", 2024-09-21T15:51:22.277 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.317684+0000", 2024-09-21T15:51:22.277 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 115.915487615, 2024-09-21T15:51:22.277 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053570809999999996, 2024-09-21T15:51:22.277 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:22.277 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:22.277 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:22.277 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.278 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:22.278 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317684+0000", 2024-09-21T15:51:22.278 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.278 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.278 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.278 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:22.278 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317684+0000", 2024-09-21T15:51:22.278 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.278 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.278 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.278 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:22.278 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317683+0000", 2024-09-21T15:51:22.278 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:22.278 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.278 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.279 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:22.279 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317688+0000", 2024-09-21T15:51:22.279 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5009999999999998e-06 2024-09-21T15:51:22.279 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.279 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.279 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:22.279 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317694+0000", 2024-09-21T15:51:22.279 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3579999999999999e-06 2024-09-21T15:51:22.279 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.279 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.279 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:22.279 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317698+0000", 2024-09-21T15:51:22.279 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4150000000000001e-06 2024-09-21T15:51:22.279 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.280 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.280 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:22.280 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317738+0000", 2024-09-21T15:51:22.280 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9575e-05 2024-09-21T15:51:22.280 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.280 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.280 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:22.280 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317762+0000", 2024-09-21T15:51:22.280 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4173999999999999e-05 2024-09-21T15:51:22.280 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.280 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.280 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:22.280 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323005+0000", 2024-09-21T15:51:22.280 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00524333 2024-09-21T15:51:22.280 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.281 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.281 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:22.281 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323041+0000", 2024-09-21T15:51:22.281 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5139999999999999e-05 2024-09-21T15:51:22.281 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.281 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:22.281 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.281 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.281 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.281 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:34837 3.0 e43/22 3:038527c8:::rbd_header.1133fed1a840:head v 43'130, mlcod=43'128)", 2024-09-21T15:51:22.281 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:51:22.281 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 111.249282917, 2024-09-21T15:51:22.281 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045776519999999998, 2024-09-21T15:51:22.281 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:22.281 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:22.282 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:22.282 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.282 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:22.282 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:51:22.282 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.282 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.282 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.282 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:22.282 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:51:22.282 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.282 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.282 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.282 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:22.282 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:51:22.283 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:22.283 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.283 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.283 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:22.283 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983892+0000", 2024-09-21T15:51:22.283 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8450000000000002e-06 2024-09-21T15:51:22.283 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.283 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.283 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:22.283 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983902+0000", 2024-09-21T15:51:22.283 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0299999999999999e-06 2024-09-21T15:51:22.283 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.283 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.283 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:22.283 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983906+0000", 2024-09-21T15:51:22.283 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2300000000000002e-06 2024-09-21T15:51:22.284 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.284 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.284 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:22.284 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983940+0000", 2024-09-21T15:51:22.284 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4563000000000001e-05 2024-09-21T15:51:22.284 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.284 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.284 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:22.284 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983960+0000", 2024-09-21T15:51:22.284 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9199999999999999e-05 2024-09-21T15:51:22.284 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.284 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.284 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:22.284 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.988430+0000", 2024-09-21T15:51:22.284 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0044709149999999998 2024-09-21T15:51:22.284 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.285 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.285 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:22.285 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.988466+0000", 2024-09-21T15:51:22.285 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5420000000000003e-05 2024-09-21T15:51:22.285 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.285 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:22.285 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.285 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.285 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.285 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "pg_update_log_missing(3.0 epoch 50/22 rep_tid 498 entries 50'171 (0'0) error 3:5df4a5a0:::rbd_data.11339e87a32a.000000000000008e:head by client.4403.0:80207 0.000000 -2 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 47'167)", 2024-09-21T15:51:22.285 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:51:22.285 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 101.291410044, 2024-09-21T15:51:22.285 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0047253119999999997, 2024-09-21T15:51:22.285 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:22.285 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-09-21T15:51:22.286 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:22.286 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.286 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:22.286 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:51:22.286 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.286 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.286 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.286 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:22.286 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:51:22.286 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.286 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.286 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.286 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:22.286 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:51:22.287 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:22.287 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.287 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.287 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:22.287 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941768+0000", 2024-09-21T15:51:22.287 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7309999999999993e-06 2024-09-21T15:51:22.287 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.287 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.287 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:22.287 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941780+0000", 2024-09-21T15:51:22.287 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1399999999999999e-05 2024-09-21T15:51:22.287 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.287 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.287 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:22.287 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941789+0000", 2024-09-21T15:51:22.287 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5230000000000001e-06 2024-09-21T15:51:22.288 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.288 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.288 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:22.288 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941841+0000", 2024-09-21T15:51:22.288 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1863999999999997e-05 2024-09-21T15:51:22.288 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.288 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.288 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:22.288 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.946486+0000", 2024-09-21T15:51:22.288 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046452539999999997 2024-09-21T15:51:22.288 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.288 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:22.288 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.288 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.288 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.288 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "pg_update_log_missing(3.0 epoch 53/22 rep_tid 585 entries 53'201 (0'0) error 3:6a1d4890:::rbd_data.1133e079cae4.0000000000000004:head by client.4403.0:101089 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 53'198)", 2024-09-21T15:51:22.289 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:44.832827+0000", 2024-09-21T15:51:22.289 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 97.400344235999995, 2024-09-21T15:51:22.289 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053552929999999997, 2024-09-21T15:51:22.289 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:22.289 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-09-21T15:51:22.289 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:22.289 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.289 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:22.289 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832827+0000", 2024-09-21T15:51:22.289 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.289 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.289 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.289 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:22.289 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832827+0000", 2024-09-21T15:51:22.290 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.290 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.290 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.290 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:22.290 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832826+0000", 2024-09-21T15:51:22.290 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:22.290 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.290 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.290 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:22.290 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832835+0000", 2024-09-21T15:51:22.290 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4509999999999992e-06 2024-09-21T15:51:22.290 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.290 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.290 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:22.291 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832854+0000", 2024-09-21T15:51:22.291 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9352999999999999e-05 2024-09-21T15:51:22.291 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.291 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.291 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:22.291 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832866+0000", 2024-09-21T15:51:22.291 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1766000000000001e-05 2024-09-21T15:51:22.291 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.291 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.291 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:22.291 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.833091+0000", 2024-09-21T15:51:22.291 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00022490300000000001 2024-09-21T15:51:22.291 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.291 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.291 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:22.291 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.838182+0000", 2024-09-21T15:51:22.292 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0050915420000000001 2024-09-21T15:51:22.292 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.292 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:22.292 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.292 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.292 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.292 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:102822 3.4 e54/22 3:2729ba03:::rbd_object_map.113336ae1075:head v 54'208, mlcod=54'205)", 2024-09-21T15:51:22.292 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:46.249540+0000", 2024-09-21T15:51:22.292 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 95.983631004000003, 2024-09-21T15:51:22.292 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052654859999999998, 2024-09-21T15:51:22.292 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:22.292 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:22.292 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:22.292 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.292 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:22.292 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249540+0000", 2024-09-21T15:51:22.293 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.293 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.293 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.293 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:22.293 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249540+0000", 2024-09-21T15:51:22.293 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.293 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.293 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.293 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:22.293 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249539+0000", 2024-09-21T15:51:22.293 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:22.293 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.293 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.293 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:22.293 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249546+0000", 2024-09-21T15:51:22.294 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7290000000000003e-06 2024-09-21T15:51:22.294 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.294 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.294 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:22.294 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249558+0000", 2024-09-21T15:51:22.294 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1844e-05 2024-09-21T15:51:22.294 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.294 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.294 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:22.294 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249565+0000", 2024-09-21T15:51:22.294 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8210000000000004e-06 2024-09-21T15:51:22.294 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.294 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.294 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:22.295 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249602+0000", 2024-09-21T15:51:22.295 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7057000000000001e-05 2024-09-21T15:51:22.295 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.295 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.295 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:22.295 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249622+0000", 2024-09-21T15:51:22.295 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9678999999999999e-05 2024-09-21T15:51:22.295 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.295 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.295 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:22.295 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254783+0000", 2024-09-21T15:51:22.295 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0051611449999999998 2024-09-21T15:51:22.295 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.295 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.295 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:22.296 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254806+0000", 2024-09-21T15:51:22.296 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2853e-05 2024-09-21T15:51:22.296 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.296 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:22.296 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.296 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.296 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.296 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:102823 3.4 e54/22 3:2729ba03:::rbd_object_map.113336ae1075:head v 54'209, mlcod=54'205)", 2024-09-21T15:51:22.296 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:46.250318+0000", 2024-09-21T15:51:22.296 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 95.982853418999994, 2024-09-21T15:51:22.296 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045013539999999999, 2024-09-21T15:51:22.296 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:22.296 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:22.296 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:22.296 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.297 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:22.297 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250318+0000", 2024-09-21T15:51:22.297 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.297 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.297 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.297 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:22.297 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250318+0000", 2024-09-21T15:51:22.297 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.297 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.297 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.297 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:22.297 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250317+0000", 2024-09-21T15:51:22.297 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:22.297 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.297 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.298 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:22.298 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250322+0000", 2024-09-21T15:51:22.298 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.074e-06 2024-09-21T15:51:22.298 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.298 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.298 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:22.298 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250330+0000", 2024-09-21T15:51:22.298 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4730000000000003e-06 2024-09-21T15:51:22.298 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.298 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.298 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:22.298 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250335+0000", 2024-09-21T15:51:22.298 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8019999999999997e-06 2024-09-21T15:51:22.298 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.299 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.299 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:22.299 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250375+0000", 2024-09-21T15:51:22.299 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9459999999999998e-05 2024-09-21T15:51:22.299 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.299 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.299 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:22.299 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250388+0000", 2024-09-21T15:51:22.299 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3288999999999999e-05 2024-09-21T15:51:22.299 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.299 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.299 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:22.299 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254807+0000", 2024-09-21T15:51:22.299 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0044192220000000004 2024-09-21T15:51:22.299 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.300 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.300 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:22.300 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254819+0000", 2024-09-21T15:51:22.300 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1603999999999999e-05 2024-09-21T15:51:22.300 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.300 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:22.300 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.300 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.300 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.300 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:160603 3.9 e95/22 3:9171211b:::rbd_data.11338bae974a.0000000000000006:head v 95'343, mlcod=92'340)", 2024-09-21T15:51:22.300 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.215621+0000", 2024-09-21T15:51:22.300 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 51.017550387999997, 2024-09-21T15:51:22.300 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0041276250000000002, 2024-09-21T15:51:22.300 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:22.300 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:22.301 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:22.301 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.301 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:22.301 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215621+0000", 2024-09-21T15:51:22.301 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.301 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.301 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.301 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:22.301 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215621+0000", 2024-09-21T15:51:22.301 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.301 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.301 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.301 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:22.301 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215620+0000", 2024-09-21T15:51:22.302 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:22.302 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.302 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.302 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:22.302 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215627+0000", 2024-09-21T15:51:22.302 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8850000000000001e-06 2024-09-21T15:51:22.302 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.302 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.302 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:22.302 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215803+0000", 2024-09-21T15:51:22.302 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000176422 2024-09-21T15:51:22.302 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.302 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.303 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:22.303 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215814+0000", 2024-09-21T15:51:22.303 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0135e-05 2024-09-21T15:51:22.303 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.303 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.303 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:22.303 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215866+0000", 2024-09-21T15:51:22.303 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2327999999999998e-05 2024-09-21T15:51:22.303 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.303 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.303 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:22.303 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215887+0000", 2024-09-21T15:51:22.303 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1449000000000001e-05 2024-09-21T15:51:22.303 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.303 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.304 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:22.304 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.219719+0000", 2024-09-21T15:51:22.304 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0038312630000000001 2024-09-21T15:51:22.304 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.304 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.304 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:22.304 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.219748+0000", 2024-09-21T15:51:22.304 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9737999999999999e-05 2024-09-21T15:51:22.304 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.304 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:22.304 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.304 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.304 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.304 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:161865 3.9 e105/22 3:90e88bad:::rbd_data.11338db70e6f.0000000000000015:head v 105'347, mlcod=103'345)", 2024-09-21T15:51:22.305 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:40.786848+0000", 2024-09-21T15:51:22.305 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 41.446323485999997, 2024-09-21T15:51:22.305 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0056408860000000003, 2024-09-21T15:51:22.305 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:22.305 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:22.305 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:22.305 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.305 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:22.305 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786848+0000", 2024-09-21T15:51:22.305 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.305 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.305 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.305 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:22.305 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786848+0000", 2024-09-21T15:51:22.305 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.306 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.306 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.306 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:22.306 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786847+0000", 2024-09-21T15:51:22.306 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:22.306 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.306 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.306 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:22.306 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786851+0000", 2024-09-21T15:51:22.306 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7419999999999999e-06 2024-09-21T15:51:22.306 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.306 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.306 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:22.306 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786872+0000", 2024-09-21T15:51:22.306 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1186e-05 2024-09-21T15:51:22.307 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.307 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.307 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:22.307 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786878+0000", 2024-09-21T15:51:22.307 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0970000000000002e-06 2024-09-21T15:51:22.307 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.307 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.307 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:22.307 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786918+0000", 2024-09-21T15:51:22.307 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.998e-05 2024-09-21T15:51:22.307 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.307 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.307 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:22.307 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786941+0000", 2024-09-21T15:51:22.308 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.245e-05 2024-09-21T15:51:22.308 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.308 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.308 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:22.308 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.792460+0000", 2024-09-21T15:51:22.308 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0055191210000000001 2024-09-21T15:51:22.308 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.308 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.308 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:22.308 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.792489+0000", 2024-09-21T15:51:22.308 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8552e-05 2024-09-21T15:51:22.308 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.308 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:22.308 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.308 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.309 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.309 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.1.0:1503 3.4 e127/99 3:26dffaa1:::rbd_data.113336ae1075.000000000000006b:16 v 127'658, mlcod=127'653)", 2024-09-21T15:51:22.309 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:51:02.797289+0000", 2024-09-21T15:51:22.309 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 19.435882607, 2024-09-21T15:51:22.309 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045583780000000001, 2024-09-21T15:51:22.309 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:22.309 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:22.309 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:22.309 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.309 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:22.309 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.797289+0000", 2024-09-21T15:51:22.309 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.310 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.310 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.310 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:22.310 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.797289+0000", 2024-09-21T15:51:22.310 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.310 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.310 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.310 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:22.310 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.797288+0000", 2024-09-21T15:51:22.310 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:22.310 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.310 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.310 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:22.310 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.797293+0000", 2024-09-21T15:51:22.311 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4500000000000003e-06 2024-09-21T15:51:22.311 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.311 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.311 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:22.311 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.797303+0000", 2024-09-21T15:51:22.311 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0134e-05 2024-09-21T15:51:22.311 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.311 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.311 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:22.311 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.797309+0000", 2024-09-21T15:51:22.311 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2100000000000001e-06 2024-09-21T15:51:22.311 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.311 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.311 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:22.312 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.797346+0000", 2024-09-21T15:51:22.312 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7583999999999998e-05 2024-09-21T15:51:22.312 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.312 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.312 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:22.312 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.797386+0000", 2024-09-21T15:51:22.312 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9428999999999998e-05 2024-09-21T15:51:22.312 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.312 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.312 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:22.312 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.801816+0000", 2024-09-21T15:51:22.312 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0044305730000000002 2024-09-21T15:51:22.312 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.312 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.313 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:22.313 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.801847+0000", 2024-09-21T15:51:22.313 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0639000000000003e-05 2024-09-21T15:51:22.313 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.313 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:22.313 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.313 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.313 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.313 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:164505 3.5 3:a3a629db:::rbd_data.113336ae1075.0000000000000003:head [delete] snapc 16=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-21T15:51:22.313 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:51:02.863560+0000", 2024-09-21T15:51:22.313 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 19.369611052, 2024-09-21T15:51:22.313 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0040794459999999996, 2024-09-21T15:51:22.313 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:22.313 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:22.314 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:22.314 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:22.314 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:22.314 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 164505 2024-09-21T15:51:22.314 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.314 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:22.314 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.314 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:22.314 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.863560+0000", 2024-09-21T15:51:22.314 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.314 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.314 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.314 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:22.314 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.863560+0000", 2024-09-21T15:51:22.315 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.315 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.315 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.315 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:22.315 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.863558+0000", 2024-09-21T15:51:22.315 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:51:22.315 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.315 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.315 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:22.315 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.863570+0000", 2024-09-21T15:51:22.315 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1960999999999999e-05 2024-09-21T15:51:22.315 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.315 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.315 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:22.316 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.863582+0000", 2024-09-21T15:51:22.316 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1368e-05 2024-09-21T15:51:22.316 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.316 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.316 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:22.316 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.863589+0000", 2024-09-21T15:51:22.316 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1670000000000002e-06 2024-09-21T15:51:22.316 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.316 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.316 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:22.316 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.863646+0000", 2024-09-21T15:51:22.316 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7433999999999999e-05 2024-09-21T15:51:22.316 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.316 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.316 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:22.316 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.863752+0000", 2024-09-21T15:51:22.317 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105214 2024-09-21T15:51:22.317 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.317 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.317 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-09-21T15:51:22.317 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.863866+0000", 2024-09-21T15:51:22.317 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011479100000000001 2024-09-21T15:51:22.317 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.317 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.317 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:22.317 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.864919+0000", 2024-09-21T15:51:22.317 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00105302 2024-09-21T15:51:22.317 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.317 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.317 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:22.317 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.867550+0000", 2024-09-21T15:51:22.318 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0026310130000000002 2024-09-21T15:51:22.318 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.318 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.318 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:22.318 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.867600+0000", 2024-09-21T15:51:22.318 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9123999999999999e-05 2024-09-21T15:51:22.318 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.318 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.318 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:22.318 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.867640+0000", 2024-09-21T15:51:22.318 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9953000000000002e-05 2024-09-21T15:51:22.318 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.318 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:22.318 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.319 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.319 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:22.319 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:22.320 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:51:22.320 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:51:22.319+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:51:22.421 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:51:22.420+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:51:22.463 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:22.463 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:51:22.463 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:51:22.463 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:22.472 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:51:22.474 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7, 3] out_osds: [] dead_osds: [] live_osds: [1, 2, 3, 4, 5, 6, 7, 0] 2024-09-21T15:51:22.474 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T15:51:22.475 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-21T15:51:22.475 INFO:tasks.thrashosds.thrasher:Setting osd 6 primary_affinity to 0.444081 2024-09-21T15:51:22.475 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 0.44408110887033914 2024-09-21T15:51:22.523 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:51:22.521+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:51:22.581 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T15:51:22.618 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:22.618 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:51:22.618 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:51:22.618 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:51:22.618 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:22.623 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:51:22.622+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:51:22.631 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:51:22.723 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-09-21T15:51:22.723 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:51:22.724 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:51:22.722+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:51:22.758 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:51:22.758 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:51:22.779 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:51:22.779 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:51:22.791 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:22.791 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:51:22.791 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:51:22.791 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:51:22.791 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.791 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:14051 3.8 3:13437810:::rbd_data.1133dc5adfdf.0000000000000039:head [write 716800~796] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-21T15:51:22.792 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:51:22.792 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 125.093450041, 2024-09-21T15:51:22.792 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.98064002699999997, 2024-09-21T15:51:22.792 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:22.792 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:22.792 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:22.792 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:22.792 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:22.792 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 14051 2024-09-21T15:51:22.792 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.792 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:22.792 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.792 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:22.792 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:51:22.792 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.792 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.793 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.793 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:22.793 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:51:22.793 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.793 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.793 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.793 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:22.793 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:51:22.793 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:22.793 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.793 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.793 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:22.793 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695131+0000", 2024-09-21T15:51:22.793 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4680000000000003e-06 2024-09-21T15:51:22.793 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.793 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.793 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:22.794 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695139+0000", 2024-09-21T15:51:22.794 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9400000000000002e-06 2024-09-21T15:51:22.794 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.794 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.794 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:22.794 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695143+0000", 2024-09-21T15:51:22.794 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9899999999999999e-06 2024-09-21T15:51:22.794 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.794 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.794 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:22.794 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695190+0000", 2024-09-21T15:51:22.794 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6458000000000001e-05 2024-09-21T15:51:22.794 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.794 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.795 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:51:22.795 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695201+0000", 2024-09-21T15:51:22.795 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0964999999999999e-05 2024-09-21T15:51:22.795 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.795 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.795 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:22.795 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.669490+0000", 2024-09-21T15:51:22.795 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.97428980499999995 2024-09-21T15:51:22.795 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.795 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.795 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-09-21T15:51:22.795 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.670033+0000", 2024-09-21T15:51:22.795 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00054252599999999999 2024-09-21T15:51:22.796 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.796 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.796 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:22.796 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.671095+0000", 2024-09-21T15:51:22.796 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001062183 2024-09-21T15:51:22.796 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.796 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.796 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:22.796 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.671186+0000", 2024-09-21T15:51:22.796 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1191000000000002e-05 2024-09-21T15:51:22.796 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.796 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.796 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-09-21T15:51:22.796 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.671346+0000", 2024-09-21T15:51:22.797 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015980399999999999 2024-09-21T15:51:22.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.797 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.797 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:22.797 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.672221+0000", 2024-09-21T15:51:22.797 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00087471999999999995 2024-09-21T15:51:22.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.797 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.797 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:22.797 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.675619+0000", 2024-09-21T15:51:22.797 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00339858 2024-09-21T15:51:22.797 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.797 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.797 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:22.797 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.675670+0000", 2024-09-21T15:51:22.797 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0664000000000001e-05 2024-09-21T15:51:22.798 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.798 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.798 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:22.798 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.675768+0000", 2024-09-21T15:51:22.798 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7415000000000002e-05 2024-09-21T15:51:22.798 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.798 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:22.798 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.798 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.798 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.798 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116235 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 761856~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:22.798 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048806+0000", 2024-09-21T15:51:22.798 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 86.739771157000007, 2024-09-21T15:51:22.798 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056818556999999999, 2024-09-21T15:51:22.798 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:22.799 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:22.799 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:22.799 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:22.799 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:22.799 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116235 2024-09-21T15:51:22.799 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.799 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:22.799 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.799 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:22.799 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048806+0000", 2024-09-21T15:51:22.799 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.799 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.799 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.799 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:22.799 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048806+0000", 2024-09-21T15:51:22.799 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.800 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.800 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.800 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:22.800 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048806+0000", 2024-09-21T15:51:22.800 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:51:22.800 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.800 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.800 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:22.800 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048807+0000", 2024-09-21T15:51:22.800 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.001e-06 2024-09-21T15:51:22.800 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.800 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.800 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:22.800 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048810+0000", 2024-09-21T15:51:22.800 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.711e-06 2024-09-21T15:51:22.800 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.801 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.801 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:22.801 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048812+0000", 2024-09-21T15:51:22.801 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2290000000000002e-06 2024-09-21T15:51:22.801 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.801 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.801 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:22.801 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105495+0000", 2024-09-21T15:51:22.801 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056682679999999999 2024-09-21T15:51:22.801 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.801 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.801 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:22.801 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105565+0000", 2024-09-21T15:51:22.801 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9868000000000003e-05 2024-09-21T15:51:22.801 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.801 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.802 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:22.802 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105625+0000", 2024-09-21T15:51:22.802 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0268e-05 2024-09-21T15:51:22.802 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.802 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:22.802 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.802 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.802 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.802 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116236 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 768000~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:22.802 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048814+0000", 2024-09-21T15:51:22.802 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 86.739763494000002, 2024-09-21T15:51:22.802 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056959587999999998, 2024-09-21T15:51:22.802 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:22.802 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:22.802 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:22.803 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:22.803 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:22.803 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116236 2024-09-21T15:51:22.803 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.803 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:22.803 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.803 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:22.803 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048814+0000", 2024-09-21T15:51:22.803 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.803 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.803 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.803 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:22.803 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048814+0000", 2024-09-21T15:51:22.803 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.803 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.803 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.803 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:22.804 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048814+0000", 2024-09-21T15:51:22.804 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:51:22.804 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.804 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.804 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:22.804 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048815+0000", 2024-09-21T15:51:22.804 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7999999999999993e-07 2024-09-21T15:51:22.804 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.804 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.804 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:22.804 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048817+0000", 2024-09-21T15:51:22.804 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6620000000000001e-06 2024-09-21T15:51:22.804 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.804 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.804 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:22.805 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048820+0000", 2024-09-21T15:51:22.805 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2800000000000002e-06 2024-09-21T15:51:22.805 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.805 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.805 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:22.805 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105647+0000", 2024-09-21T15:51:22.805 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056827253000000001 2024-09-21T15:51:22.805 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.805 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.805 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:22.805 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105716+0000", 2024-09-21T15:51:22.805 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8761000000000001e-05 2024-09-21T15:51:22.805 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.805 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.805 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:22.806 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105774+0000", 2024-09-21T15:51:22.806 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7855000000000001e-05 2024-09-21T15:51:22.806 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.806 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:22.806 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.806 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.806 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.806 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116237 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 774144~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:22.806 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048822+0000", 2024-09-21T15:51:22.806 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 86.739755809000002, 2024-09-21T15:51:22.806 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057114028999999997, 2024-09-21T15:51:22.806 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:22.806 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:22.807 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:22.807 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:22.807 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:22.807 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116237 2024-09-21T15:51:22.807 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.807 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:22.807 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.807 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:22.807 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048822+0000", 2024-09-21T15:51:22.807 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.807 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.807 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.807 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:22.807 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048822+0000", 2024-09-21T15:51:22.807 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.808 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:22.808 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048822+0000", 2024-09-21T15:51:22.808 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:51:22.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.808 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:22.808 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048823+0000", 2024-09-21T15:51:22.808 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9699999999999994e-07 2024-09-21T15:51:22.808 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.808 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.808 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:22.808 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048825+0000", 2024-09-21T15:51:22.809 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.65e-06 2024-09-21T15:51:22.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.809 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:22.809 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048828+0000", 2024-09-21T15:51:22.809 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8890000000000002e-06 2024-09-21T15:51:22.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.809 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:22.809 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105802+0000", 2024-09-21T15:51:22.809 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056974415 2024-09-21T15:51:22.809 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.809 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.809 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:22.810 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105871+0000", 2024-09-21T15:51:22.810 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8727999999999997e-05 2024-09-21T15:51:22.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.810 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.810 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:22.810 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105936+0000", 2024-09-21T15:51:22.810 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4554999999999994e-05 2024-09-21T15:51:22.810 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.810 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:22.810 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.810 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.810 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.810 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116238 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 780288~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:22.810 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048830+0000", 2024-09-21T15:51:22.810 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 86.739747550000004, 2024-09-21T15:51:22.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057296574000000003, 2024-09-21T15:51:22.811 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:22.811 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:22.811 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:22.811 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:22.811 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:22.811 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116238 2024-09-21T15:51:22.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.811 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:22.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.811 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:22.811 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048830+0000", 2024-09-21T15:51:22.811 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.811 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.811 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.812 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:22.812 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048830+0000", 2024-09-21T15:51:22.812 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.812 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.812 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.812 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:22.812 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048830+0000", 2024-09-21T15:51:22.812 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:51:22.812 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.812 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.812 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:22.812 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048831+0000", 2024-09-21T15:51:22.812 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0160000000000001e-06 2024-09-21T15:51:22.812 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.813 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:22.813 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048834+0000", 2024-09-21T15:51:22.813 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.813e-06 2024-09-21T15:51:22.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.813 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:22.813 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048836+0000", 2024-09-21T15:51:22.813 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6790000000000001e-06 2024-09-21T15:51:22.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.813 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:22.813 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105961+0000", 2024-09-21T15:51:22.813 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057124705999999997 2024-09-21T15:51:22.813 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.813 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:22.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106042+0000", 2024-09-21T15:51:22.814 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0820000000000002e-05 2024-09-21T15:51:22.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.814 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.814 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:22.814 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106127+0000", 2024-09-21T15:51:22.814 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4746000000000005e-05 2024-09-21T15:51:22.814 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.814 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:22.814 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.814 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.814 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.815 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116324 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 524288~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:22.815 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048838+0000", 2024-09-21T15:51:22.815 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 86.739739240999995, 2024-09-21T15:51:22.815 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057477185, 2024-09-21T15:51:22.815 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:22.815 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:22.815 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:22.815 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:22.815 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:22.815 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116324 2024-09-21T15:51:22.815 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.815 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:22.815 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.815 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:22.816 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048838+0000", 2024-09-21T15:51:22.816 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.816 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.816 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:22.816 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048838+0000", 2024-09-21T15:51:22.816 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.816 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.816 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:22.816 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048838+0000", 2024-09-21T15:51:22.816 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:51:22.816 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.816 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.816 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:22.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048842+0000", 2024-09-21T15:51:22.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4829999999999999e-06 2024-09-21T15:51:22.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.817 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:22.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048846+0000", 2024-09-21T15:51:22.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0470000000000004e-06 2024-09-21T15:51:22.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.817 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:22.817 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048849+0000", 2024-09-21T15:51:22.817 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3689999999999998e-06 2024-09-21T15:51:22.817 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.817 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.818 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:22.818 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106157+0000", 2024-09-21T15:51:22.818 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057307942000000001 2024-09-21T15:51:22.818 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.818 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.818 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:22.818 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106237+0000", 2024-09-21T15:51:22.818 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0320000000000003e-05 2024-09-21T15:51:22.818 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.818 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.818 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:22.818 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106315+0000", 2024-09-21T15:51:22.818 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8227999999999997e-05 2024-09-21T15:51:22.818 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.818 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:22.819 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.819 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.819 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116325 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 530432~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:22.819 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048851+0000", 2024-09-21T15:51:22.819 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 86.739726069, 2024-09-21T15:51:22.819 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057581036000000002, 2024-09-21T15:51:22.819 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:22.819 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:22.819 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:22.819 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:22.819 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:22.819 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116325 2024-09-21T15:51:22.819 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.819 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:22.820 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.820 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:22.820 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048851+0000", 2024-09-21T15:51:22.820 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.820 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.820 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.820 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:22.820 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048851+0000", 2024-09-21T15:51:22.820 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.820 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.820 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.820 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:22.820 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048851+0000", 2024-09-21T15:51:22.820 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:51:22.820 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.821 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.821 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:22.821 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048852+0000", 2024-09-21T15:51:22.821 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.077e-06 2024-09-21T15:51:22.821 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.821 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.821 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:22.821 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048855+0000", 2024-09-21T15:51:22.821 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9859999999999999e-06 2024-09-21T15:51:22.821 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.821 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.821 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:22.821 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048858+0000", 2024-09-21T15:51:22.821 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3810000000000002e-06 2024-09-21T15:51:22.821 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.822 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.822 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:22.822 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106335+0000", 2024-09-21T15:51:22.822 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057477473000000001 2024-09-21T15:51:22.822 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.822 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.822 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:22.822 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106393+0000", 2024-09-21T15:51:22.822 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8050000000000002e-05 2024-09-21T15:51:22.822 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.822 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.822 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:22.822 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106432+0000", 2024-09-21T15:51:22.822 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9274999999999999e-05 2024-09-21T15:51:22.822 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.823 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:22.823 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.823 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.823 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.823 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116326 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 536576~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:22.823 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048860+0000", 2024-09-21T15:51:22.823 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 86.739718018000005, 2024-09-21T15:51:22.823 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057774197999999999, 2024-09-21T15:51:22.823 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:22.823 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:22.823 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:22.823 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:22.823 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:22.823 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116326 2024-09-21T15:51:22.824 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.824 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:22.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.824 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:22.824 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048860+0000", 2024-09-21T15:51:22.824 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.824 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.824 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:22.824 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048860+0000", 2024-09-21T15:51:22.824 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.824 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.824 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.824 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:22.825 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048859+0000", 2024-09-21T15:51:22.825 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:51:22.825 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.825 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.825 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:22.825 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048860+0000", 2024-09-21T15:51:22.825 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.02e-06 2024-09-21T15:51:22.825 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.825 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.825 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:22.825 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048863+0000", 2024-09-21T15:51:22.825 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1039999999999998e-06 2024-09-21T15:51:22.825 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.825 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.825 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:22.826 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048867+0000", 2024-09-21T15:51:22.826 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.563e-06 2024-09-21T15:51:22.826 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.826 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.826 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:22.826 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106465+0000", 2024-09-21T15:51:22.826 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057597789000000003 2024-09-21T15:51:22.826 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.826 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.826 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:22.826 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106552+0000", 2024-09-21T15:51:22.826 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7602000000000006e-05 2024-09-21T15:51:22.826 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.826 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.826 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:22.826 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106634+0000", 2024-09-21T15:51:22.827 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1321000000000003e-05 2024-09-21T15:51:22.827 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.827 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:22.827 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.827 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.827 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.827 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116327 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 542720~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:22.827 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048870+0000", 2024-09-21T15:51:22.827 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 86.739707146000001, 2024-09-21T15:51:22.827 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057883148000000002, 2024-09-21T15:51:22.827 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:22.827 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:22.827 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:22.827 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:22.827 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:22.828 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116327 2024-09-21T15:51:22.828 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.828 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:22.828 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.828 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:22.828 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048870+0000", 2024-09-21T15:51:22.828 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.828 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.828 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.828 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:22.828 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048870+0000", 2024-09-21T15:51:22.828 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.828 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.828 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.828 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:22.828 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048870+0000", 2024-09-21T15:51:22.829 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:22.829 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.829 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.829 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:22.829 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048872+0000", 2024-09-21T15:51:22.829 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.824e-06 2024-09-21T15:51:22.829 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.829 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.829 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:22.829 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048879+0000", 2024-09-21T15:51:22.829 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6749999999999996e-06 2024-09-21T15:51:22.829 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.829 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.829 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:22.829 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048883+0000", 2024-09-21T15:51:22.829 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2200000000000003e-06 2024-09-21T15:51:22.830 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.830 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.830 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:22.830 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106663+0000", 2024-09-21T15:51:22.830 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057780049 2024-09-21T15:51:22.830 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.830 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.830 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:22.830 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106711+0000", 2024-09-21T15:51:22.830 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8433999999999997e-05 2024-09-21T15:51:22.830 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.830 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.830 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:22.830 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106754+0000", 2024-09-21T15:51:22.831 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2236e-05 2024-09-21T15:51:22.831 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.831 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:22.831 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.831 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.831 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.831 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116328 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 548864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:22.831 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048887+0000", 2024-09-21T15:51:22.831 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 86.739690625999998, 2024-09-21T15:51:22.831 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057981984, 2024-09-21T15:51:22.831 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:22.831 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:22.831 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:22.831 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:22.832 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:22.832 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116328 2024-09-21T15:51:22.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.832 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:22.832 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.832 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:22.832 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048887+0000", 2024-09-21T15:51:22.832 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.832 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.832 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:22.832 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048887+0000", 2024-09-21T15:51:22.832 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.832 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.832 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.832 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:22.833 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048887+0000", 2024-09-21T15:51:22.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:22.833 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.833 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.833 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:22.833 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048891+0000", 2024-09-21T15:51:22.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.843e-06 2024-09-21T15:51:22.833 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.833 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.833 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:22.833 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048897+0000", 2024-09-21T15:51:22.833 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5419999999999996e-06 2024-09-21T15:51:22.833 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.833 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.833 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:22.834 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048901+0000", 2024-09-21T15:51:22.834 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4689999999999999e-06 2024-09-21T15:51:22.834 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.834 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.834 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:22.834 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106780+0000", 2024-09-21T15:51:22.834 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057878259000000001 2024-09-21T15:51:22.834 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.834 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.834 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:22.834 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106827+0000", 2024-09-21T15:51:22.834 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7098000000000002e-05 2024-09-21T15:51:22.834 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.834 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.835 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:22.835 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106869+0000", 2024-09-21T15:51:22.835 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2129999999999998e-05 2024-09-21T15:51:22.835 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.835 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:22.835 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.835 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.835 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.835 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116329 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 555008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:22.835 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048905+0000", 2024-09-21T15:51:22.835 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 86.739672515999999, 2024-09-21T15:51:22.835 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058115554999999999, 2024-09-21T15:51:22.835 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:22.835 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:22.835 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:22.836 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:22.836 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:22.836 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116329 2024-09-21T15:51:22.836 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.836 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:22.836 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.836 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:22.836 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048905+0000", 2024-09-21T15:51:22.836 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.836 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.836 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.836 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:22.836 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048905+0000", 2024-09-21T15:51:22.836 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.837 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.837 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.837 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:22.837 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048905+0000", 2024-09-21T15:51:22.837 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:22.837 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.837 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.837 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:22.837 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048907+0000", 2024-09-21T15:51:22.837 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.086e-06 2024-09-21T15:51:22.837 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.837 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.837 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:22.837 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048912+0000", 2024-09-21T15:51:22.837 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2830000000000001e-06 2024-09-21T15:51:22.838 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.838 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.838 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:22.838 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048916+0000", 2024-09-21T15:51:22.838 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.98e-06 2024-09-21T15:51:22.838 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.838 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.838 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:22.838 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106907+0000", 2024-09-21T15:51:22.838 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057990719000000003 2024-09-21T15:51:22.838 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.838 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.838 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:22.838 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106969+0000", 2024-09-21T15:51:22.838 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1789000000000001e-05 2024-09-21T15:51:22.838 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.839 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.839 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:22.839 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107021+0000", 2024-09-21T15:51:22.839 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2030999999999997e-05 2024-09-21T15:51:22.839 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.839 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:22.839 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.839 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.839 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.839 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116330 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 561152~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:22.839 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048922+0000", 2024-09-21T15:51:22.839 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 86.739655357999993, 2024-09-21T15:51:22.839 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058242343000000002, 2024-09-21T15:51:22.839 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:22.839 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:22.840 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:22.840 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:22.840 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:22.840 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116330 2024-09-21T15:51:22.840 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.840 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:22.840 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.840 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:22.840 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048922+0000", 2024-09-21T15:51:22.840 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.840 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.840 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.840 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:22.840 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048922+0000", 2024-09-21T15:51:22.840 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.840 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.840 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.841 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:22.841 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048922+0000", 2024-09-21T15:51:22.841 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:22.841 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.841 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.841 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:22.841 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048924+0000", 2024-09-21T15:51:22.841 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1660000000000001e-06 2024-09-21T15:51:22.841 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.841 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.841 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:22.841 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048930+0000", 2024-09-21T15:51:22.841 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4529999999999999e-06 2024-09-21T15:51:22.841 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.842 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.842 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:22.842 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048934+0000", 2024-09-21T15:51:22.842 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0269999999999997e-06 2024-09-21T15:51:22.842 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.842 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.842 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:22.842 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107047+0000", 2024-09-21T15:51:22.842 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058112913000000002 2024-09-21T15:51:22.842 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.842 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.842 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:22.842 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107095+0000", 2024-09-21T15:51:22.842 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7778000000000002e-05 2024-09-21T15:51:22.843 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.843 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.843 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:22.843 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107165+0000", 2024-09-21T15:51:22.843 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9370999999999997e-05 2024-09-21T15:51:22.843 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.843 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:22.843 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.843 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.843 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.843 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116331 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 567296~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:22.843 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.049440+0000", 2024-09-21T15:51:22.843 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 86.739137263999993, 2024-09-21T15:51:22.843 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057878485, 2024-09-21T15:51:22.844 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:22.844 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:22.844 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:22.844 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:22.844 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:22.844 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116331 2024-09-21T15:51:22.844 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.844 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:22.844 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.844 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:22.844 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049440+0000", 2024-09-21T15:51:22.844 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.844 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.844 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.844 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:22.845 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049440+0000", 2024-09-21T15:51:22.845 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.845 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.845 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.845 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:22.845 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048939+0000", 2024-09-21T15:51:22.845 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9994984 2024-09-21T15:51:22.845 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.845 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.845 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:22.845 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049444+0000", 2024-09-21T15:51:22.845 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00050557200000000003 2024-09-21T15:51:22.845 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.845 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.846 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:22.846 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049460+0000", 2024-09-21T15:51:22.846 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6120000000000002e-05 2024-09-21T15:51:22.846 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.846 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.846 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:22.846 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049468+0000", 2024-09-21T15:51:22.846 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4209999999999998e-06 2024-09-21T15:51:22.846 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.846 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.846 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:22.846 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107189+0000", 2024-09-21T15:51:22.846 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057721019999999998 2024-09-21T15:51:22.846 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.847 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.847 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:22.847 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107254+0000", 2024-09-21T15:51:22.847 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5265000000000006e-05 2024-09-21T15:51:22.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.847 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.847 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:22.847 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107319+0000", 2024-09-21T15:51:22.847 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4784999999999997e-05 2024-09-21T15:51:22.847 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.847 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:22.847 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.847 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.847 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.847 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116332 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 573440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:22.848 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.049474+0000", 2024-09-21T15:51:22.848 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 86.739104002999994, 2024-09-21T15:51:22.848 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058013809, 2024-09-21T15:51:22.848 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:22.848 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:22.848 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:22.848 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:22.848 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:22.848 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116332 2024-09-21T15:51:22.848 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.848 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:22.848 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.848 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:22.848 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049474+0000", 2024-09-21T15:51:22.848 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.848 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.849 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.849 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:22.849 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049474+0000", 2024-09-21T15:51:22.849 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.849 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.849 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.849 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:22.849 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049473+0000", 2024-09-21T15:51:22.849 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:22.849 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.849 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.849 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:22.849 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049476+0000", 2024-09-21T15:51:22.849 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6249999999999999e-06 2024-09-21T15:51:22.849 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.850 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.850 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:22.850 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049482+0000", 2024-09-21T15:51:22.850 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0959999999999997e-06 2024-09-21T15:51:22.850 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.850 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.850 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:22.850 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049487+0000", 2024-09-21T15:51:22.850 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5949999999999998e-06 2024-09-21T15:51:22.850 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.850 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.850 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:22.850 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107344+0000", 2024-09-21T15:51:22.850 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057856594999999997 2024-09-21T15:51:22.851 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.851 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.851 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:22.851 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107422+0000", 2024-09-21T15:51:22.851 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7582000000000004e-05 2024-09-21T15:51:22.851 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.851 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.851 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:22.851 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107487+0000", 2024-09-21T15:51:22.851 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5723999999999996e-05 2024-09-21T15:51:22.851 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.851 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:22.851 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.851 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.851 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.852 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116333 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 579584~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:22.852 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.049993+0000", 2024-09-21T15:51:22.852 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 86.738584220000007, 2024-09-21T15:51:22.852 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057663537000000001, 2024-09-21T15:51:22.852 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:22.852 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:22.852 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:22.852 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:22.852 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:22.852 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116333 2024-09-21T15:51:22.852 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.852 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:22.852 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.852 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:22.852 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049993+0000", 2024-09-21T15:51:22.853 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.853 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.853 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:22.853 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049993+0000", 2024-09-21T15:51:22.853 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.853 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.853 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:22.853 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049491+0000", 2024-09-21T15:51:22.853 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9994979 2024-09-21T15:51:22.853 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.853 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:22.853 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049998+0000", 2024-09-21T15:51:22.853 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00050654200000000002 2024-09-21T15:51:22.854 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.854 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.854 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:22.854 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050010+0000", 2024-09-21T15:51:22.854 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1790999999999999e-05 2024-09-21T15:51:22.854 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.854 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.854 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:22.854 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050016+0000", 2024-09-21T15:51:22.854 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2179999999999996e-06 2024-09-21T15:51:22.854 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.854 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.854 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:22.854 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107513+0000", 2024-09-21T15:51:22.855 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057496927000000003 2024-09-21T15:51:22.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.855 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:22.855 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107590+0000", 2024-09-21T15:51:22.855 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7547999999999997e-05 2024-09-21T15:51:22.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.855 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:22.855 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107657+0000", 2024-09-21T15:51:22.855 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6630000000000004e-05 2024-09-21T15:51:22.855 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.855 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:22.855 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.856 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.856 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116334 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 585728~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:22.856 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.050091+0000", 2024-09-21T15:51:22.856 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 86.738486592000001, 2024-09-21T15:51:22.856 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057734355000000001, 2024-09-21T15:51:22.856 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:22.856 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:22.856 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:22.856 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:22.856 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:22.856 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116334 2024-09-21T15:51:22.856 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.856 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:22.856 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.856 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:22.856 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050091+0000", 2024-09-21T15:51:22.857 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.857 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.857 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.857 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:22.857 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050091+0000", 2024-09-21T15:51:22.857 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.857 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.857 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.857 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:22.857 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050021+0000", 2024-09-21T15:51:22.857 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999304 2024-09-21T15:51:22.857 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.857 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.857 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:22.857 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050095+0000", 2024-09-21T15:51:22.858 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4158000000000004e-05 2024-09-21T15:51:22.858 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.858 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.858 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:22.858 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050103+0000", 2024-09-21T15:51:22.858 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0770000000000002e-06 2024-09-21T15:51:22.858 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.858 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.858 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:22.858 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050108+0000", 2024-09-21T15:51:22.858 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6330000000000001e-06 2024-09-21T15:51:22.858 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.858 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.858 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:22.859 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107681+0000", 2024-09-21T15:51:22.859 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057572996000000001 2024-09-21T15:51:22.859 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.859 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.859 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:22.859 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107759+0000", 2024-09-21T15:51:22.859 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7368999999999997e-05 2024-09-21T15:51:22.859 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.859 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.859 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:22.859 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107825+0000", 2024-09-21T15:51:22.859 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6783000000000001e-05 2024-09-21T15:51:22.859 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.859 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:22.859 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.859 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.860 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.860 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116335 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 591872~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:22.860 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.050542+0000", 2024-09-21T15:51:22.860 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 86.738036007000005, 2024-09-21T15:51:22.860 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057466455999999999, 2024-09-21T15:51:22.860 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:22.860 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:22.860 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:22.860 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:22.860 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:22.860 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116335 2024-09-21T15:51:22.860 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.860 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:22.860 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.860 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:22.861 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050542+0000", 2024-09-21T15:51:22.861 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.861 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.861 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.861 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:22.861 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050542+0000", 2024-09-21T15:51:22.861 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.861 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.861 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.861 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:22.861 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050113+0000", 2024-09-21T15:51:22.861 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9995713 2024-09-21T15:51:22.861 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.861 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.861 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:22.861 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050546+0000", 2024-09-21T15:51:22.862 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00043328699999999997 2024-09-21T15:51:22.862 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.862 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.862 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:22.862 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050557+0000", 2024-09-21T15:51:22.862 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1011e-05 2024-09-21T15:51:22.862 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.862 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.862 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:22.862 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050564+0000", 2024-09-21T15:51:22.862 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1130000000000003e-06 2024-09-21T15:51:22.862 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.862 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.862 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:22.862 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107852+0000", 2024-09-21T15:51:22.863 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057287350000000001 2024-09-21T15:51:22.863 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.863 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.863 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:22.863 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107928+0000", 2024-09-21T15:51:22.863 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6668000000000005e-05 2024-09-21T15:51:22.863 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.863 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.863 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:22.863 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108008+0000", 2024-09-21T15:51:22.863 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9783999999999994e-05 2024-09-21T15:51:22.863 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.863 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:22.863 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.864 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.864 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.864 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116336 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 598016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:22.864 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.050834+0000", 2024-09-21T15:51:22.864 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 86.737743741000003, 2024-09-21T15:51:22.864 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057353572999999998, 2024-09-21T15:51:22.864 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:22.864 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:22.864 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:22.864 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:22.864 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:22.864 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116336 2024-09-21T15:51:22.864 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.864 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:22.864 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.864 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:22.865 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050834+0000", 2024-09-21T15:51:22.865 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.865 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.865 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.865 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:22.865 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050834+0000", 2024-09-21T15:51:22.865 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.865 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.865 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.865 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:22.865 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050571+0000", 2024-09-21T15:51:22.865 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9997373 2024-09-21T15:51:22.865 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.865 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.866 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:22.866 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050842+0000", 2024-09-21T15:51:22.866 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00027029099999999998 2024-09-21T15:51:22.866 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.866 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.866 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:22.866 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050853+0000", 2024-09-21T15:51:22.866 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1532000000000001e-05 2024-09-21T15:51:22.866 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.866 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.866 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:22.866 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050863+0000", 2024-09-21T15:51:22.866 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4239999999999999e-06 2024-09-21T15:51:22.866 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.866 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.866 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:22.867 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108045+0000", 2024-09-21T15:51:22.867 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057182415 2024-09-21T15:51:22.867 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.867 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.867 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:22.867 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108123+0000", 2024-09-21T15:51:22.867 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8275000000000005e-05 2024-09-21T15:51:22.867 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.867 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.867 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:22.867 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108187+0000", 2024-09-21T15:51:22.867 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4152000000000005e-05 2024-09-21T15:51:22.867 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.867 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:22.868 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.868 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.868 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.868 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116337 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 604160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:22.868 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.051192+0000", 2024-09-21T15:51:22.868 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 86.737385113000002, 2024-09-21T15:51:22.868 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057175981000000001, 2024-09-21T15:51:22.868 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:22.868 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:22.868 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:22.868 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:22.868 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:22.868 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116337 2024-09-21T15:51:22.868 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.868 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:22.869 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.869 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:22.869 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051192+0000", 2024-09-21T15:51:22.869 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.869 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.869 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.869 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:22.869 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051192+0000", 2024-09-21T15:51:22.869 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.869 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.869 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.869 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:22.869 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050870+0000", 2024-09-21T15:51:22.869 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9996777 2024-09-21T15:51:22.869 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.869 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.870 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:22.870 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051197+0000", 2024-09-21T15:51:22.870 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00032656299999999998 2024-09-21T15:51:22.870 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.870 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.870 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:22.870 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051209+0000", 2024-09-21T15:51:22.870 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2636000000000001e-05 2024-09-21T15:51:22.870 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.870 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.870 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:22.870 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051217+0000", 2024-09-21T15:51:22.870 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5989999999999996e-06 2024-09-21T15:51:22.870 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.870 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.871 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:22.871 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108224+0000", 2024-09-21T15:51:22.871 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057007622000000001 2024-09-21T15:51:22.871 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.871 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.871 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:22.871 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108299+0000", 2024-09-21T15:51:22.871 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5099999999999996e-05 2024-09-21T15:51:22.871 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.871 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.871 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:22.871 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108368+0000", 2024-09-21T15:51:22.871 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8918000000000007e-05 2024-09-21T15:51:22.871 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.871 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:22.871 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.872 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.872 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.872 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116338 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 610304~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:22.872 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.051583+0000", 2024-09-21T15:51:22.872 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 86.736994659999993, 2024-09-21T15:51:22.872 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056967716000000002, 2024-09-21T15:51:22.872 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:22.872 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:22.872 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:22.872 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:22.872 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:22.872 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116338 2024-09-21T15:51:22.872 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.872 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:22.872 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.872 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:22.873 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051583+0000", 2024-09-21T15:51:22.873 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.873 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.873 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.873 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:22.873 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051583+0000", 2024-09-21T15:51:22.873 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:22.873 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.873 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.873 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:22.873 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051223+0000", 2024-09-21T15:51:22.873 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.99964 2024-09-21T15:51:22.873 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.873 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.873 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:22.873 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051587+0000", 2024-09-21T15:51:22.873 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00036352999999999998 2024-09-21T15:51:22.874 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.874 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.874 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:22.874 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051597+0000", 2024-09-21T15:51:22.874 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.893e-06 2024-09-21T15:51:22.874 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.874 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.874 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:22.874 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051603+0000", 2024-09-21T15:51:22.874 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0519999999999998e-06 2024-09-21T15:51:22.874 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.874 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.874 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:22.874 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108394+0000", 2024-09-21T15:51:22.874 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056791531999999999 2024-09-21T15:51:22.875 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.875 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.875 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:22.875 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108480+0000", 2024-09-21T15:51:22.875 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.577e-05 2024-09-21T15:51:22.875 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:22.875 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:22.875 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:22.875 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108551+0000", 2024-09-21T15:51:22.875 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0722000000000005e-05 2024-09-21T15:51:22.875 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.875 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:22.875 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.875 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:22.875 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:22.876 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:22.876 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:51:22.877 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:51:22.877 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:51:22.878 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:51:22.878 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:51:22.878 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:51:22.879 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:51:22.879 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:51:22.879 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:51:22.880 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:51:22.880 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:51:22.880 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:51:22.875+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:51:22.977 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:51:22.976+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:51:22.983 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:22.984 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:22.992 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:51:23.079 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:51:23.077+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:51:23.135 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:23.135 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:23.144 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:51:23.185 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:51:23.184+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:51:23.281 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:51:23.280+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:51:23.287 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:23.287 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:23.295 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:51:23.382 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:51:23.381+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:51:23.398 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:23.398 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:23.407 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:51:23.483 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:51:23.482+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:51:23.549 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:23.549 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:23.557 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:51:23.584 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:51:23.583+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:51:23.659 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:23.659 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:23.668 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T15:51:23.682 INFO:teuthology.orchestra.run.smithi089.stderr:set osd.6 primary-affinity to 0.444081 (8291032) 2024-09-21T15:51:23.698 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T15:51:23.712 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:51:23.711+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:51:23.775 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:23.775 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:23.784 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T15:51:23.785 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:51:23.784+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:51:23.886 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:51:23.885+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:51:23.926 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:23.926 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:23.935 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T15:51:24.019 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:51:24.018+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:51:24.077 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:24.078 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:24.089 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T15:51:24.089 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:51:24.087+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:51:24.190 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:51:24.189+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:51:24.231 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:24.231 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:24.240 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T15:51:24.291 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:51:24.290+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:51:24.341 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:24.341 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:24.349 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T15:51:24.392 INFO:tasks.ceph.osd.3.smithi089.stderr:2024-09-21T15:51:24.391+0000 7f6e572c4640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 3 (PID: 33332) UID: 0 2024-09-21T15:51:24.491 INFO:teuthology.orchestra.run.smithi100.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:24.491 INFO:teuthology.orchestra.run.smithi100.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:24.493 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:51:24.492+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:51:24.499 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:51:24.594 INFO:tasks.ceph.osd.5.smithi100.stderr:2024-09-21T15:51:24.592+0000 7fd096008640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 5 (PID: 33361) UID: 0 2024-09-21T15:51:24.603 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:24.603 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:24.613 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:51:24.692 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is unset 2024-09-21T15:51:24.694 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:51:24.693+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:51:24.722 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:24.722 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:24.732 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:51:24.796 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:51:24.794+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:51:24.874 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:24.874 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:24.881 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T15:51:24.883 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2024-09-21T15:51:24.924 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:51:24.923+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:51:24.996 INFO:tasks.ceph.osd.6.smithi100.stderr:2024-09-21T15:51:24.995+0000 7f11acc7e640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 6 (PID: 33366) UID: 0 2024-09-21T15:51:25.027 INFO:teuthology.orchestra.run.smithi089.stdout:ERROR: (22) Invalid argument 2024-09-21T15:51:25.027 INFO:teuthology.orchestra.run.smithi089.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T15:51:25.036 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_blocked_ops 2024-09-21T15:51:25.043 INFO:teuthology.orchestra.run.smithi089.stdout:osd.0: {} 2024-09-21T15:51:25.043 INFO:teuthology.orchestra.run.smithi089.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T15:51:25.061 INFO:teuthology.orchestra.run.smithi089.stdout:osd.1: {} 2024-09-21T15:51:25.061 INFO:teuthology.orchestra.run.smithi089.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T15:51:25.076 INFO:teuthology.orchestra.run.smithi089.stdout:osd.2: {} 2024-09-21T15:51:25.077 INFO:teuthology.orchestra.run.smithi089.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T15:51:25.092 INFO:teuthology.orchestra.run.smithi089.stdout:osd.3: {} 2024-09-21T15:51:25.093 INFO:teuthology.orchestra.run.smithi089.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T15:51:25.098 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:51:25.097+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:51:25.122 INFO:teuthology.orchestra.run.smithi089.stdout:osd.4: {} 2024-09-21T15:51:25.123 INFO:teuthology.orchestra.run.smithi089.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T15:51:25.139 INFO:teuthology.orchestra.run.smithi089.stdout:osd.5: {} 2024-09-21T15:51:25.139 INFO:teuthology.orchestra.run.smithi089.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T15:51:25.163 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:25.163 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:51:25.163 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:51:25.163 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:51:25.163 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:25.171 INFO:teuthology.orchestra.run.smithi089.stdout:osd.6: {} 2024-09-21T15:51:25.171 INFO:teuthology.orchestra.run.smithi089.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T15:51:25.174 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_historic_ops 2024-09-21T15:51:25.187 INFO:teuthology.orchestra.run.smithi089.stdout:osd.7: {} 2024-09-21T15:51:25.187 INFO:teuthology.orchestra.run.smithi089.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T15:51:25.198 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:51:25.197+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:51:25.291 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:25.291 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:51:25.291 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:51:25.291 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:51:25.291 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.291 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62849 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 208896~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:25.291 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.144181+0000", 2024-09-21T15:51:25.291 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 108.14463354599999, 2024-09-21T15:51:25.291 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050460390000000001, 2024-09-21T15:51:25.291 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:25.292 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:25.292 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:25.292 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:25.292 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:25.292 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62849 2024-09-21T15:51:25.292 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.292 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:25.292 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.292 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:25.292 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144181+0000", 2024-09-21T15:51:25.292 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.292 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.292 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.292 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:25.292 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144181+0000", 2024-09-21T15:51:25.292 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.293 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.293 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.293 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:25.293 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144181+0000", 2024-09-21T15:51:25.293 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:25.293 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.293 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.293 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:25.293 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144183+0000", 2024-09-21T15:51:25.293 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3860000000000001e-06 2024-09-21T15:51:25.293 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.293 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.293 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:25.293 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144189+0000", 2024-09-21T15:51:25.293 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0599999999999996e-06 2024-09-21T15:51:25.294 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.294 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.294 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:25.294 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.144194+0000", 2024-09-21T15:51:25.294 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4379999999999997e-06 2024-09-21T15:51:25.294 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.294 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.294 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:25.294 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.194457+0000", 2024-09-21T15:51:25.294 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050263317000000002 2024-09-21T15:51:25.294 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.294 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.294 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:25.294 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.194548+0000", 2024-09-21T15:51:25.294 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0785000000000006e-05 2024-09-21T15:51:25.295 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.295 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.295 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:25.295 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.194642+0000", 2024-09-21T15:51:25.295 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3837000000000003e-05 2024-09-21T15:51:25.295 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.295 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:25.295 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.295 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.295 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.295 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62862 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 954368~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:25.295 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.146176+0000", 2024-09-21T15:51:25.295 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 108.142638483, 2024-09-21T15:51:25.296 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050442901999999998, 2024-09-21T15:51:25.296 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:25.296 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:25.296 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:25.296 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:25.296 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:25.296 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62862 2024-09-21T15:51:25.296 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.296 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:25.296 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.296 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:25.296 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146176+0000", 2024-09-21T15:51:25.296 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.297 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.297 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.297 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:25.297 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146176+0000", 2024-09-21T15:51:25.297 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.297 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.297 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.297 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:25.297 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146176+0000", 2024-09-21T15:51:25.297 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:25.297 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.297 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.297 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:25.298 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146178+0000", 2024-09-21T15:51:25.298 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4739999999999999e-06 2024-09-21T15:51:25.298 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.298 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.298 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:25.298 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146186+0000", 2024-09-21T15:51:25.298 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4989999999999997e-06 2024-09-21T15:51:25.298 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.298 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.298 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:25.298 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146191+0000", 2024-09-21T15:51:25.298 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7999999999999998e-06 2024-09-21T15:51:25.299 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.299 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.299 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:25.299 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196442+0000", 2024-09-21T15:51:25.299 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050251665000000001 2024-09-21T15:51:25.299 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.299 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.299 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:25.299 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196518+0000", 2024-09-21T15:51:25.299 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5677000000000001e-05 2024-09-21T15:51:25.299 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.299 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.299 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:25.300 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196619+0000", 2024-09-21T15:51:25.300 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101247 2024-09-21T15:51:25.300 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.300 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:25.300 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.300 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.300 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.300 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62863 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 937984~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:25.300 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.146195+0000", 2024-09-21T15:51:25.300 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 108.142620292, 2024-09-21T15:51:25.300 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050642305999999998, 2024-09-21T15:51:25.300 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:25.300 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:25.300 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:25.301 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:25.301 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:25.301 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62863 2024-09-21T15:51:25.301 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.301 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:25.301 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.301 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:25.301 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146195+0000", 2024-09-21T15:51:25.301 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.301 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.301 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.301 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:25.301 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146195+0000", 2024-09-21T15:51:25.301 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.301 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.302 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.302 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:25.302 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146194+0000", 2024-09-21T15:51:25.302 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:25.302 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.302 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.302 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:25.302 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146196+0000", 2024-09-21T15:51:25.302 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1900000000000002e-06 2024-09-21T15:51:25.302 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.302 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.302 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:25.302 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146202+0000", 2024-09-21T15:51:25.303 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.57e-06 2024-09-21T15:51:25.303 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.303 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.303 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:25.303 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.146206+0000", 2024-09-21T15:51:25.303 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5079999999999999e-06 2024-09-21T15:51:25.303 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.303 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.303 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:25.303 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196655+0000", 2024-09-21T15:51:25.303 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050448973000000001 2024-09-21T15:51:25.303 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.303 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.303 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:25.303 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196730+0000", 2024-09-21T15:51:25.304 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4728999999999996e-05 2024-09-21T15:51:25.304 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.304 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.304 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:25.304 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.196837+0000", 2024-09-21T15:51:25.304 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000106735 2024-09-21T15:51:25.304 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.304 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:25.304 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.304 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.304 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.304 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62872 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 290816~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:25.305 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148149+0000", 2024-09-21T15:51:25.305 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 108.14066561600001, 2024-09-21T15:51:25.305 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.05046991, 2024-09-21T15:51:25.305 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:25.305 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:25.305 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:25.305 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:25.305 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:25.305 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62872 2024-09-21T15:51:25.305 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.305 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:25.305 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.305 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:25.305 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148149+0000", 2024-09-21T15:51:25.306 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.306 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.306 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.306 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:25.306 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148149+0000", 2024-09-21T15:51:25.306 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.306 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.306 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.306 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:25.306 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148149+0000", 2024-09-21T15:51:25.306 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:25.306 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.306 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.306 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:25.307 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148157+0000", 2024-09-21T15:51:25.307 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4749999999999993e-06 2024-09-21T15:51:25.307 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.307 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.307 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:25.307 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148163+0000", 2024-09-21T15:51:25.307 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8250000000000001e-06 2024-09-21T15:51:25.307 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.307 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.307 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:25.307 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148168+0000", 2024-09-21T15:51:25.307 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4730000000000002e-06 2024-09-21T15:51:25.307 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.307 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.307 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:25.308 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198471+0000", 2024-09-21T15:51:25.308 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050302974 2024-09-21T15:51:25.308 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.308 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.308 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:25.308 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198555+0000", 2024-09-21T15:51:25.308 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4783999999999993e-05 2024-09-21T15:51:25.308 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.308 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.308 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:25.308 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198619+0000", 2024-09-21T15:51:25.308 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3751000000000007e-05 2024-09-21T15:51:25.308 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.309 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:25.309 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.309 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.309 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.309 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62873 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 315392~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:25.309 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148172+0000", 2024-09-21T15:51:25.309 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 108.140643325, 2024-09-21T15:51:25.309 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050707998999999997, 2024-09-21T15:51:25.309 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:25.309 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:25.309 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:25.309 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:25.309 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:25.310 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62873 2024-09-21T15:51:25.310 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.310 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:25.310 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.310 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:25.310 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148172+0000", 2024-09-21T15:51:25.310 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.310 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.310 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.310 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:25.310 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148172+0000", 2024-09-21T15:51:25.310 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.310 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.310 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.311 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:25.311 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148171+0000", 2024-09-21T15:51:25.311 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:25.311 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.311 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.311 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:25.311 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148173+0000", 2024-09-21T15:51:25.311 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2340000000000001e-06 2024-09-21T15:51:25.311 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.311 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.311 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:25.311 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148179+0000", 2024-09-21T15:51:25.311 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9340000000000003e-06 2024-09-21T15:51:25.311 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.311 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.311 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:25.312 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148186+0000", 2024-09-21T15:51:25.312 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2349999999999996e-06 2024-09-21T15:51:25.312 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.312 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.312 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:25.312 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198684+0000", 2024-09-21T15:51:25.312 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050498890999999997 2024-09-21T15:51:25.312 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.312 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.312 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:25.312 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198778+0000", 2024-09-21T15:51:25.312 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3134999999999995e-05 2024-09-21T15:51:25.312 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.312 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.312 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:25.312 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198880+0000", 2024-09-21T15:51:25.313 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000101906 2024-09-21T15:51:25.313 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.313 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:25.313 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.313 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.313 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.313 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62874 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 323584~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:25.313 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148211+0000", 2024-09-21T15:51:25.313 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 108.14060356500001, 2024-09-21T15:51:25.313 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050887766000000001, 2024-09-21T15:51:25.313 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:25.313 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:25.313 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:25.314 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:25.314 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:25.314 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62874 2024-09-21T15:51:25.314 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.314 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:25.314 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.314 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:25.314 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148211+0000", 2024-09-21T15:51:25.314 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.314 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.314 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.314 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:25.314 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148211+0000", 2024-09-21T15:51:25.315 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.315 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.315 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.315 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:25.315 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148211+0000", 2024-09-21T15:51:25.315 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:25.315 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.315 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.315 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:25.315 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148213+0000", 2024-09-21T15:51:25.315 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6860000000000002e-06 2024-09-21T15:51:25.315 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.315 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.315 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:25.315 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148220+0000", 2024-09-21T15:51:25.316 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4740000000000002e-06 2024-09-21T15:51:25.316 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.316 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.316 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:25.316 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148226+0000", 2024-09-21T15:51:25.316 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0120000000000002e-06 2024-09-21T15:51:25.316 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.316 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.316 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:25.316 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198940+0000", 2024-09-21T15:51:25.316 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050714412 2024-09-21T15:51:25.316 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.316 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.316 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:25.317 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.198995+0000", 2024-09-21T15:51:25.317 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4863e-05 2024-09-21T15:51:25.317 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.317 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.317 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:25.317 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.199099+0000", 2024-09-21T15:51:25.317 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010383299999999999 2024-09-21T15:51:25.317 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.317 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:25.317 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.317 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.317 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.317 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62877 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 995328~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:25.317 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.148230+0000", 2024-09-21T15:51:25.317 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 108.14058470499999, 2024-09-21T15:51:25.317 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.051108949000000001, 2024-09-21T15:51:25.318 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:25.318 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:25.318 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:25.318 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:25.318 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:25.318 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62877 2024-09-21T15:51:25.318 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.318 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:25.318 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.318 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:25.318 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148230+0000", 2024-09-21T15:51:25.318 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.318 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.318 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.319 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:25.319 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148230+0000", 2024-09-21T15:51:25.319 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.319 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.319 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.319 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:25.319 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148230+0000", 2024-09-21T15:51:25.319 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:25.319 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.319 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.319 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:25.319 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148232+0000", 2024-09-21T15:51:25.319 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1220000000000002e-06 2024-09-21T15:51:25.319 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.319 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.319 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:25.320 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148238+0000", 2024-09-21T15:51:25.320 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.789e-06 2024-09-21T15:51:25.320 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.320 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.320 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:25.320 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.148242+0000", 2024-09-21T15:51:25.320 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7369999999999997e-06 2024-09-21T15:51:25.320 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.320 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.320 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:25.320 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.199201+0000", 2024-09-21T15:51:25.320 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050959088 2024-09-21T15:51:25.320 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.320 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.320 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:25.321 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.199285+0000", 2024-09-21T15:51:25.321 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3147999999999997e-05 2024-09-21T15:51:25.321 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.321 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.321 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:25.321 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.199339+0000", 2024-09-21T15:51:25.321 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4484999999999998e-05 2024-09-21T15:51:25.321 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.321 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:25.321 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.321 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.321 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.321 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62888 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 331776~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:25.322 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150154+0000", 2024-09-21T15:51:25.322 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 108.138661111, 2024-09-21T15:51:25.322 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050483169000000001, 2024-09-21T15:51:25.322 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:25.322 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:25.322 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:25.322 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:25.323 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:25.323 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62888 2024-09-21T15:51:25.323 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.323 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:25.323 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.323 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:25.323 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150154+0000", 2024-09-21T15:51:25.323 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.323 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.323 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.323 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:25.323 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150154+0000", 2024-09-21T15:51:25.324 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.324 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.324 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.324 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:25.324 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150153+0000", 2024-09-21T15:51:25.324 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:25.324 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.324 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.324 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:25.324 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150155+0000", 2024-09-21T15:51:25.324 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9920000000000002e-06 2024-09-21T15:51:25.324 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.324 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.324 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:25.325 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150161+0000", 2024-09-21T15:51:25.325 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5439999999999998e-06 2024-09-21T15:51:25.325 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.325 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.325 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:25.325 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150165+0000", 2024-09-21T15:51:25.325 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3460000000000004e-06 2024-09-21T15:51:25.325 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.325 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.325 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:25.325 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200502+0000", 2024-09-21T15:51:25.325 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050337150999999997 2024-09-21T15:51:25.325 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.326 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.326 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:25.326 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200575+0000", 2024-09-21T15:51:25.326 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2250999999999997e-05 2024-09-21T15:51:25.326 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.326 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.326 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:25.326 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200637+0000", 2024-09-21T15:51:25.326 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2195999999999999e-05 2024-09-21T15:51:25.326 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.326 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:25.326 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.326 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.326 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.326 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62889 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 339968~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:25.327 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150169+0000", 2024-09-21T15:51:25.327 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 108.138645612, 2024-09-21T15:51:25.327 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050629678999999997, 2024-09-21T15:51:25.327 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:25.327 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:25.327 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:25.327 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:25.327 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:25.327 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62889 2024-09-21T15:51:25.327 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.327 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:25.327 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.327 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:25.327 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150169+0000", 2024-09-21T15:51:25.328 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.328 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.328 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.328 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:25.328 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150169+0000", 2024-09-21T15:51:25.328 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.328 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.328 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.328 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:25.328 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150169+0000", 2024-09-21T15:51:25.328 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:25.328 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.328 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.328 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:25.328 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150171+0000", 2024-09-21T15:51:25.329 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3910000000000001e-06 2024-09-21T15:51:25.329 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.329 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.329 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:25.329 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150177+0000", 2024-09-21T15:51:25.329 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4079999999999997e-06 2024-09-21T15:51:25.329 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.329 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.329 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:25.329 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150182+0000", 2024-09-21T15:51:25.329 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3890000000000004e-06 2024-09-21T15:51:25.329 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.329 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.329 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:25.329 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200664+0000", 2024-09-21T15:51:25.330 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050482212999999998 2024-09-21T15:51:25.330 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.330 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.330 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:25.330 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200738+0000", 2024-09-21T15:51:25.330 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4114000000000002e-05 2024-09-21T15:51:25.330 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.330 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.330 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:25.330 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200799+0000", 2024-09-21T15:51:25.330 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0503e-05 2024-09-21T15:51:25.330 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.330 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:25.330 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.330 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.331 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.331 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62891 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 348160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:25.331 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150186+0000", 2024-09-21T15:51:25.331 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 108.13862847, 2024-09-21T15:51:25.331 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050794552, 2024-09-21T15:51:25.331 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:25.331 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:25.331 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:25.331 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:25.331 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:25.331 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62891 2024-09-21T15:51:25.331 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.331 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:25.331 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.331 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:25.332 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150186+0000", 2024-09-21T15:51:25.332 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.332 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.332 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.332 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:25.332 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150186+0000", 2024-09-21T15:51:25.332 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.332 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.332 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.332 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:25.332 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150186+0000", 2024-09-21T15:51:25.332 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:25.332 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.332 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.332 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:25.333 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150193+0000", 2024-09-21T15:51:25.333 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6830000000000002e-06 2024-09-21T15:51:25.333 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.333 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.333 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:25.333 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150199+0000", 2024-09-21T15:51:25.333 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9859999999999999e-06 2024-09-21T15:51:25.333 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.333 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.333 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:25.333 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150203+0000", 2024-09-21T15:51:25.333 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5920000000000002e-06 2024-09-21T15:51:25.333 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.333 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.334 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:25.334 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200835+0000", 2024-09-21T15:51:25.334 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050631783 2024-09-21T15:51:25.334 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.334 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.334 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:25.334 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200894+0000", 2024-09-21T15:51:25.334 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8560999999999997e-05 2024-09-21T15:51:25.334 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.334 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.334 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:25.334 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.200981+0000", 2024-09-21T15:51:25.334 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7305999999999994e-05 2024-09-21T15:51:25.334 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.335 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:25.335 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.335 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.335 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.335 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62892 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 356352~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:25.335 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150208+0000", 2024-09-21T15:51:25.335 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 108.13860708199999, 2024-09-21T15:51:25.335 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050951769000000001, 2024-09-21T15:51:25.335 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:25.335 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:25.335 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:25.335 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:25.335 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:25.335 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62892 2024-09-21T15:51:25.336 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.336 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:25.336 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.336 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:25.336 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150208+0000", 2024-09-21T15:51:25.336 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.336 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.336 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.336 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:25.336 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150208+0000", 2024-09-21T15:51:25.336 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.336 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.336 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.336 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:25.337 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150207+0000", 2024-09-21T15:51:25.337 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:25.337 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.337 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.337 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:25.337 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150210+0000", 2024-09-21T15:51:25.337 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.238e-06 2024-09-21T15:51:25.337 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.337 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.337 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:25.337 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150215+0000", 2024-09-21T15:51:25.337 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6350000000000003e-06 2024-09-21T15:51:25.337 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.337 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.338 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:25.338 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150220+0000", 2024-09-21T15:51:25.338 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.544e-06 2024-09-21T15:51:25.338 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.338 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.338 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:25.338 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201014+0000", 2024-09-21T15:51:25.338 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050793932999999999 2024-09-21T15:51:25.338 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.338 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.338 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:25.338 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201092+0000", 2024-09-21T15:51:25.338 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7961999999999997e-05 2024-09-21T15:51:25.338 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.338 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.339 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:25.339 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201160+0000", 2024-09-21T15:51:25.339 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7836000000000006e-05 2024-09-21T15:51:25.339 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.339 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:25.339 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.339 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.339 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.339 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62894 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 364544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:25.339 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150820+0000", 2024-09-21T15:51:25.339 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 108.137994882, 2024-09-21T15:51:25.339 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050489196, 2024-09-21T15:51:25.339 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:25.339 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:25.339 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:25.340 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:25.340 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:25.340 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62894 2024-09-21T15:51:25.340 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.340 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:25.340 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.340 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:25.340 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150820+0000", 2024-09-21T15:51:25.340 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.340 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.340 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.340 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:25.340 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150820+0000", 2024-09-21T15:51:25.340 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.340 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.340 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.341 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:25.341 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150223+0000", 2024-09-21T15:51:25.341 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9994035 2024-09-21T15:51:25.341 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.341 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.341 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:25.341 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150825+0000", 2024-09-21T15:51:25.341 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00060137099999999996 2024-09-21T15:51:25.341 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.341 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.341 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:25.341 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150837+0000", 2024-09-21T15:51:25.341 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2503999999999999e-05 2024-09-21T15:51:25.342 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.342 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.342 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:25.342 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150844+0000", 2024-09-21T15:51:25.342 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6900000000000003e-06 2024-09-21T15:51:25.342 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.342 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.342 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:25.342 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201187+0000", 2024-09-21T15:51:25.342 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050342973999999999 2024-09-21T15:51:25.342 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.342 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.342 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:25.342 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201250+0000", 2024-09-21T15:51:25.343 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3274000000000004e-05 2024-09-21T15:51:25.343 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.343 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.343 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:25.343 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201309+0000", 2024-09-21T15:51:25.343 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9058000000000003e-05 2024-09-21T15:51:25.343 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.343 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:25.343 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.343 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.343 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.343 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62895 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 380928~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:25.343 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150851+0000", 2024-09-21T15:51:25.343 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 108.137964286, 2024-09-21T15:51:25.344 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050611097000000001, 2024-09-21T15:51:25.344 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:25.344 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:25.344 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:25.344 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:25.344 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:25.344 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62895 2024-09-21T15:51:25.344 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.344 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:25.344 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.344 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:25.344 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150851+0000", 2024-09-21T15:51:25.344 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.344 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.345 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.345 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:25.345 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150851+0000", 2024-09-21T15:51:25.345 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.345 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.345 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.345 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:25.345 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150850+0000", 2024-09-21T15:51:25.345 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:25.345 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.345 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.345 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:25.345 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150853+0000", 2024-09-21T15:51:25.345 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1039999999999998e-06 2024-09-21T15:51:25.345 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.346 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.346 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:25.346 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150860+0000", 2024-09-21T15:51:25.346 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3180000000000001e-06 2024-09-21T15:51:25.346 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.346 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.346 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:25.346 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150866+0000", 2024-09-21T15:51:25.346 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1599999999999997e-06 2024-09-21T15:51:25.346 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.346 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.346 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:25.346 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201332+0000", 2024-09-21T15:51:25.346 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050466100999999999 2024-09-21T15:51:25.347 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.347 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.347 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:25.347 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201392+0000", 2024-09-21T15:51:25.347 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0053999999999998e-05 2024-09-21T15:51:25.347 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.347 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.347 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:25.347 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201462+0000", 2024-09-21T15:51:25.347 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9861000000000001e-05 2024-09-21T15:51:25.347 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.347 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:25.347 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.347 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.347 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.348 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62896 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 372736~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:25.348 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.150870+0000", 2024-09-21T15:51:25.348 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 108.137944506, 2024-09-21T15:51:25.348 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050753292999999998, 2024-09-21T15:51:25.348 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:25.348 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:25.348 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:25.348 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:25.348 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:25.348 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62896 2024-09-21T15:51:25.348 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.348 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:25.348 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.348 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:25.349 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150870+0000", 2024-09-21T15:51:25.349 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.349 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.349 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.349 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:25.349 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150870+0000", 2024-09-21T15:51:25.349 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.349 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.349 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.349 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:25.349 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150870+0000", 2024-09-21T15:51:25.349 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:25.349 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.349 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.350 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:25.350 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150872+0000", 2024-09-21T15:51:25.350 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.644e-06 2024-09-21T15:51:25.350 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.350 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.350 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:25.350 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150879+0000", 2024-09-21T15:51:25.350 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3509999999999998e-06 2024-09-21T15:51:25.350 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.350 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.350 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:25.350 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150884+0000", 2024-09-21T15:51:25.350 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9150000000000004e-06 2024-09-21T15:51:25.351 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.351 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.351 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:25.351 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201487+0000", 2024-09-21T15:51:25.351 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050603276000000003 2024-09-21T15:51:25.351 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.351 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.351 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:25.351 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201557+0000", 2024-09-21T15:51:25.351 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9847999999999999e-05 2024-09-21T15:51:25.351 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.351 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.351 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:25.351 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201624+0000", 2024-09-21T15:51:25.351 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6730999999999995e-05 2024-09-21T15:51:25.351 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.352 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:25.352 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.352 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.352 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.352 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:62897 3.4 3:20cb824d:::rbd_data.11336b2fdf4a.0000000000000003:c [read 389120~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:25.352 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:37.151044+0000", 2024-09-21T15:51:25.352 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 108.13777046600001, 2024-09-21T15:51:25.352 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050720531999999999, 2024-09-21T15:51:25.352 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:25.352 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:25.352 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:25.352 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:25.352 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:25.352 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 62897 2024-09-21T15:51:25.352 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.353 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:25.353 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.353 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:25.353 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151044+0000", 2024-09-21T15:51:25.353 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.353 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.353 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.353 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:25.353 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151044+0000", 2024-09-21T15:51:25.353 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.353 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.353 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.353 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:25.353 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.150888+0000", 2024-09-21T15:51:25.353 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9998431 2024-09-21T15:51:25.354 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.354 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.354 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:25.354 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151048+0000", 2024-09-21T15:51:25.354 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00016077299999999999 2024-09-21T15:51:25.354 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.354 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.354 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:25.354 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151058+0000", 2024-09-21T15:51:25.354 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.465e-06 2024-09-21T15:51:25.354 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.354 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.354 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:25.354 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.151064+0000", 2024-09-21T15:51:25.354 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3829999999999997e-06 2024-09-21T15:51:25.355 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.355 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.355 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:25.355 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201648+0000", 2024-09-21T15:51:25.355 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.050583597000000001 2024-09-21T15:51:25.355 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.355 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.355 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:25.355 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201711+0000", 2024-09-21T15:51:25.355 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3616999999999998e-05 2024-09-21T15:51:25.355 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.355 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.355 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:25.355 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:37.201765+0000", 2024-09-21T15:51:25.355 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3464000000000002e-05 2024-09-21T15:51:25.355 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.356 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:25.356 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.356 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.356 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.356 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:118351 3.4 3:25980f86:::rbd_data.1133545d3fa6.000000000000005a:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-09-21T15:51:25.356 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:58.975795+0000", 2024-09-21T15:51:25.356 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 86.313019831999995, 2024-09-21T15:51:25.356 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.99016351599999997, 2024-09-21T15:51:25.356 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:25.356 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:25.356 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:25.356 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:25.356 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:25.356 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 118351 2024-09-21T15:51:25.356 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.356 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:25.357 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.357 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:25.357 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975795+0000", 2024-09-21T15:51:25.357 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.357 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.357 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.357 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:25.357 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975795+0000", 2024-09-21T15:51:25.357 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.357 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.357 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.357 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:25.357 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975793+0000", 2024-09-21T15:51:25.357 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:51:25.357 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.357 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.358 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:25.358 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975800+0000", 2024-09-21T15:51:25.358 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8220000000000001e-06 2024-09-21T15:51:25.358 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.358 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.358 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:25.358 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975814+0000", 2024-09-21T15:51:25.358 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3893000000000001e-05 2024-09-21T15:51:25.358 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.358 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.358 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:25.358 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975821+0000", 2024-09-21T15:51:25.358 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1319999999999996e-06 2024-09-21T15:51:25.358 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.358 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.358 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:25.359 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975878+0000", 2024-09-21T15:51:25.359 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7238999999999999e-05 2024-09-21T15:51:25.359 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.359 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.359 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:51:25.359 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975889+0000", 2024-09-21T15:51:25.359 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0509000000000001e-05 2024-09-21T15:51:25.359 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.359 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.359 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:25.359 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.964573+0000", 2024-09-21T15:51:25.359 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.98868396300000005 2024-09-21T15:51:25.359 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.359 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.359 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:25.359 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.964672+0000", 2024-09-21T15:51:25.360 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9601999999999999e-05 2024-09-21T15:51:25.360 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.360 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.360 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-09-21T15:51:25.360 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.964758+0000", 2024-09-21T15:51:25.360 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5537000000000005e-05 2024-09-21T15:51:25.360 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.360 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.360 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:25.360 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965847+0000", 2024-09-21T15:51:25.360 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001089404 2024-09-21T15:51:25.360 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.360 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.360 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:25.360 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965890+0000", 2024-09-21T15:51:25.360 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2521999999999997e-05 2024-09-21T15:51:25.361 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.361 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.361 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:25.361 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965934+0000", 2024-09-21T15:51:25.361 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4489e-05 2024-09-21T15:51:25.361 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.361 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.361 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:25.361 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965959+0000", 2024-09-21T15:51:25.361 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4301999999999999e-05 2024-09-21T15:51:25.361 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.361 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:25.361 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.361 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.361 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.361 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:118353 3.4 3:23841e18:::rbd_data.1133545d3fa6.0000000000000061:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-09-21T15:51:25.362 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:58.975836+0000", 2024-09-21T15:51:25.362 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 86.312978333000004, 2024-09-21T15:51:25.362 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.99077746700000002, 2024-09-21T15:51:25.362 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:25.362 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:25.362 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:25.362 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:25.362 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:25.362 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 118353 2024-09-21T15:51:25.362 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.362 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:25.362 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.363 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:25.363 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975836+0000", 2024-09-21T15:51:25.363 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.363 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.363 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.363 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:25.363 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975836+0000", 2024-09-21T15:51:25.363 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.363 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.363 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.363 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:25.363 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975836+0000", 2024-09-21T15:51:25.363 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:25.363 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.363 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.364 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:25.364 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975839+0000", 2024-09-21T15:51:25.364 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6129999999999998e-06 2024-09-21T15:51:25.364 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.364 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.364 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:25.364 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975844+0000", 2024-09-21T15:51:25.364 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5130000000000004e-06 2024-09-21T15:51:25.364 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.364 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.364 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:25.364 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975850+0000", 2024-09-21T15:51:25.364 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3179999999999997e-06 2024-09-21T15:51:25.364 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.364 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.364 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:25.365 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975915+0000", 2024-09-21T15:51:25.365 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5168999999999999e-05 2024-09-21T15:51:25.365 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.365 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.365 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:51:25.365 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:58.975922+0000", 2024-09-21T15:51:25.365 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2749999999999998e-06 2024-09-21T15:51:25.365 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.365 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.365 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:25.365 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965431+0000", 2024-09-21T15:51:25.365 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.98950864699999996 2024-09-21T15:51:25.365 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.365 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.366 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:25.366 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965535+0000", 2024-09-21T15:51:25.366 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000103949 2024-09-21T15:51:25.366 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.366 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.366 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-09-21T15:51:25.366 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.965607+0000", 2024-09-21T15:51:25.366 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.2478000000000006e-05 2024-09-21T15:51:25.366 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.366 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.366 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:25.366 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.966205+0000", 2024-09-21T15:51:25.367 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00059775200000000001 2024-09-21T15:51:25.367 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.367 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.367 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:25.367 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.966562+0000", 2024-09-21T15:51:25.367 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00035732199999999998 2024-09-21T15:51:25.367 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.367 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.367 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:25.367 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.966593+0000", 2024-09-21T15:51:25.367 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0642999999999998e-05 2024-09-21T15:51:25.367 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.367 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.367 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:25.367 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:59.966614+0000", 2024-09-21T15:51:25.368 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1149999999999999e-05 2024-09-21T15:51:25.368 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.368 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:25.368 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.368 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.368 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.368 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160535 3.6 3:67cae2ed:::rbd_mirroring:head [list-watchers] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-09-21T15:51:25.368 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:30.195408+0000", 2024-09-21T15:51:25.368 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 55.093407251000002, 2024-09-21T15:51:25.368 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.96025944900000004, 2024-09-21T15:51:25.368 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:25.368 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-09-21T15:51:25.368 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:25.368 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:25.368 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:25.369 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160535 2024-09-21T15:51:25.369 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.369 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:25.369 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.369 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:25.369 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195408+0000", 2024-09-21T15:51:25.369 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.369 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.369 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.369 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:25.369 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195408+0000", 2024-09-21T15:51:25.369 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.369 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.369 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.369 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:25.370 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195406+0000", 2024-09-21T15:51:25.370 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:51:25.370 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.370 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.370 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:25.370 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195414+0000", 2024-09-21T15:51:25.370 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.2870000000000004e-06 2024-09-21T15:51:25.370 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.370 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.370 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:25.370 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195432+0000", 2024-09-21T15:51:25.370 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.8110000000000001e-05 2024-09-21T15:51:25.370 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.370 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.370 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:25.371 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195441+0000", 2024-09-21T15:51:25.371 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7679999999999997e-06 2024-09-21T15:51:25.371 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.371 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.371 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:25.371 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195505+0000", 2024-09-21T15:51:25.371 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3769000000000006e-05 2024-09-21T15:51:25.371 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.371 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.371 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:51:25.371 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.195518+0000", 2024-09-21T15:51:25.371 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3341e-05 2024-09-21T15:51:25.371 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.371 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.371 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:25.372 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.155509+0000", 2024-09-21T15:51:25.372 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.959990393 2024-09-21T15:51:25.372 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.372 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.372 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:25.372 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.155667+0000", 2024-09-21T15:51:25.372 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000158505 2024-09-21T15:51:25.372 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.372 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:25.372 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.372 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.372 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.372 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160974 3.4 3:22dd2475:::rbd_id.image_client.0-clone39:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e99)", 2024-09-21T15:51:25.372 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:34.748093+0000", 2024-09-21T15:51:25.372 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 50.540721699999999, 2024-09-21T15:51:25.373 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.94006199499999998, 2024-09-21T15:51:25.373 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:25.373 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:25.373 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:25.373 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:25.373 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:25.373 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160974 2024-09-21T15:51:25.373 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.373 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:25.373 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.373 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:25.373 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748093+0000", 2024-09-21T15:51:25.373 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.373 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.373 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.373 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:25.374 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748093+0000", 2024-09-21T15:51:25.374 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.374 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.374 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.374 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:25.374 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748091+0000", 2024-09-21T15:51:25.374 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:51:25.374 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.374 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.374 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:25.374 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748101+0000", 2024-09-21T15:51:25.374 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5950000000000005e-06 2024-09-21T15:51:25.374 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.374 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.375 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:25.375 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748122+0000", 2024-09-21T15:51:25.375 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1611e-05 2024-09-21T15:51:25.375 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.375 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.375 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:25.375 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748133+0000", 2024-09-21T15:51:25.375 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0224000000000001e-05 2024-09-21T15:51:25.375 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.375 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.375 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:25.375 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748185+0000", 2024-09-21T15:51:25.375 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2005000000000001e-05 2024-09-21T15:51:25.375 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.375 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.376 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:51:25.376 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:34.748194+0000", 2024-09-21T15:51:25.376 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0180000000000007e-06 2024-09-21T15:51:25.376 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.376 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.376 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:25.376 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.683582+0000", 2024-09-21T15:51:25.376 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.93538800499999997 2024-09-21T15:51:25.376 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.376 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.376 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:25.376 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.683711+0000", 2024-09-21T15:51:25.376 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000129746 2024-09-21T15:51:25.376 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.376 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.377 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 2", 2024-09-21T15:51:25.377 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.683838+0000", 2024-09-21T15:51:25.377 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012693200000000001 2024-09-21T15:51:25.377 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.377 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.377 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:25.377 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.687445+0000", 2024-09-21T15:51:25.377 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0036071340000000001 2024-09-21T15:51:25.377 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.377 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.377 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:25.377 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.688064+0000", 2024-09-21T15:51:25.377 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00061857100000000005 2024-09-21T15:51:25.377 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.377 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.378 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:25.378 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.688113+0000", 2024-09-21T15:51:25.378 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9128000000000001e-05 2024-09-21T15:51:25.379 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.379 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.379 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:25.379 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:35.688155+0000", 2024-09-21T15:51:25.379 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2026e-05 2024-09-21T15:51:25.379 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.379 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:25.379 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.379 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.379 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.380 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:165681 3.19 3:9c73a274:::rbd_object_map.11336829dd7d.000000000000000f:head [call] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-09-21T15:51:25.380 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:51:11.936427+0000", 2024-09-21T15:51:25.380 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 13.352387895, 2024-09-21T15:51:25.380 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.75328927499999998, 2024-09-21T15:51:25.380 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:25.380 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:25.380 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:25.380 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:25.380 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:25.380 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 165681 2024-09-21T15:51:25.381 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.381 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:25.381 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.381 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:25.381 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:11.936427+0000", 2024-09-21T15:51:25.381 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.381 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.381 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.381 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:25.381 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:11.936427+0000", 2024-09-21T15:51:25.381 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.381 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.381 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.381 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:25.382 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:11.936425+0000", 2024-09-21T15:51:25.382 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999976 2024-09-21T15:51:25.382 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.382 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.382 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:25.382 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:11.936436+0000", 2024-09-21T15:51:25.382 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1898e-05 2024-09-21T15:51:25.382 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.382 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.382 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:25.382 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:11.936456+0000", 2024-09-21T15:51:25.382 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9568999999999998e-05 2024-09-21T15:51:25.382 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.382 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.382 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:25.383 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:11.936467+0000", 2024-09-21T15:51:25.383 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.058e-05 2024-09-21T15:51:25.383 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.383 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.383 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:25.383 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:11.936523+0000", 2024-09-21T15:51:25.383 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5961000000000002e-05 2024-09-21T15:51:25.383 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.383 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.383 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:51:25.383 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:11.936533+0000", 2024-09-21T15:51:25.383 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0234999999999999e-05 2024-09-21T15:51:25.383 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.383 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.383 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:25.384 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:12.689430+0000", 2024-09-21T15:51:25.384 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.75289766499999999 2024-09-21T15:51:25.384 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.384 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.384 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:25.384 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:12.689563+0000", 2024-09-21T15:51:25.384 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00013278199999999999 2024-09-21T15:51:25.384 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.384 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.384 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:25.384 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:12.689716+0000", 2024-09-21T15:51:25.384 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015297499999999999 2024-09-21T15:51:25.384 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.385 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:25.385 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.385 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.385 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:25.385 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:25.386 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_ops_in_flight 2024-09-21T15:51:25.387 INFO:tasks.ceph.osd.0.smithi089.stderr:2024-09-21T15:51:25.385+0000 7f7444eac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 0 (PID: 51856) UID: 0 2024-09-21T15:51:25.486 INFO:tasks.ceph.osd.2.smithi089.stderr:2024-09-21T15:51:25.485+0000 7fd860d63640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 2 (PID: 33323) UID: 0 2024-09-21T15:51:25.530 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:25.530 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:51:25.530 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:51:25.530 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:25.538 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_blocked_ops 2024-09-21T15:51:25.622 INFO:tasks.ceph.osd.1.smithi089.stderr:2024-09-21T15:51:25.621+0000 7ff0d1ee7640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1 (PID: 33319) UID: 0 2024-09-21T15:51:25.680 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:25.680 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:51:25.680 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:51:25.680 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:51:25.681 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:25.689 INFO:tasks.ceph.osd.4.smithi100.stderr:2024-09-21T15:51:25.688+0000 7f2f73921640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 4 (PID: 33360) UID: 0 2024-09-21T15:51:25.691 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.2.asok dump_historic_ops 2024-09-21T15:51:25.789 INFO:tasks.ceph.osd.7.smithi100.stderr:2024-09-21T15:51:25.788+0000 7f2258cac640 -1 received signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 7 (PID: 33367) UID: 0 2024-09-21T15:51:25.814 INFO:teuthology.orchestra.run.smithi089.stdout:All operations completed A-OK! 2024-09-21T15:51:25.832 DEBUG:teuthology.parallel:result is None 2024-09-21T15:51:25.832 DEBUG:teuthology.run_tasks:Unwinding manager rbd_fsx 2024-09-21T15:51:25.842 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-09-21T15:51:25.851 INFO:tasks.thrashosds:joining thrashosds 2024-09-21T15:51:25.852 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:25.852 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:51:25.852 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:51:25.852 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:51:25.852 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.852 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:4957 3.9 e29/22 3:95c5ed95:::rbd_data.11335fa2fe69.0000000000000018:head v 29'4, mlcod=26'2)", 2024-09-21T15:51:25.852 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:14.453951+0000", 2024-09-21T15:51:25.852 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 131.394207709, 2024-09-21T15:51:25.852 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0043804559999999996, 2024-09-21T15:51:25.853 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:25.853 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:25.853 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:25.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.853 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:25.853 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.453951+0000", 2024-09-21T15:51:25.853 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.853 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.853 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:25.853 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.453951+0000", 2024-09-21T15:51:25.853 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.853 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.853 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.854 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:25.854 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.453950+0000", 2024-09-21T15:51:25.854 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:25.854 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.854 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.854 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:25.854 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454495+0000", 2024-09-21T15:51:25.854 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00054437799999999998 2024-09-21T15:51:25.854 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.854 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.854 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:25.854 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454508+0000", 2024-09-21T15:51:25.854 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3475000000000001e-05 2024-09-21T15:51:25.854 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.855 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:25.855 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454515+0000", 2024-09-21T15:51:25.855 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8990000000000004e-06 2024-09-21T15:51:25.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.855 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:25.855 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454581+0000", 2024-09-21T15:51:25.855 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5987000000000004e-05 2024-09-21T15:51:25.855 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.855 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.855 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:25.855 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.454609+0000", 2024-09-21T15:51:25.855 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.7466000000000001e-05 2024-09-21T15:51:25.856 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.856 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.856 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:25.856 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.458303+0000", 2024-09-21T15:51:25.856 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003693979 2024-09-21T15:51:25.856 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.856 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.856 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:25.856 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:14.458331+0000", 2024-09-21T15:51:25.856 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8932999999999999e-05 2024-09-21T15:51:25.856 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.856 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:25.856 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.856 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.856 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.856 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13420 3.0 e31/22 3:038b8840:::rbd_data.1133dc5adfdf.0000000000000035:head v 31'49, mlcod=31'47)", 2024-09-21T15:51:25.857 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.209233+0000", 2024-09-21T15:51:25.857 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 128.63892609300001, 2024-09-21T15:51:25.857 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0042624960000000002, 2024-09-21T15:51:25.857 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:25.857 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:25.857 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:25.857 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.857 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:25.857 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209233+0000", 2024-09-21T15:51:25.857 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.857 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.857 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.857 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:25.857 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209233+0000", 2024-09-21T15:51:25.857 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.858 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.858 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.858 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:25.858 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209232+0000", 2024-09-21T15:51:25.858 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:25.858 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.858 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.858 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:25.858 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209251+0000", 2024-09-21T15:51:25.858 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9304e-05 2024-09-21T15:51:25.858 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.858 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.858 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:25.858 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209265+0000", 2024-09-21T15:51:25.859 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3532e-05 2024-09-21T15:51:25.859 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.859 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.859 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:25.859 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209272+0000", 2024-09-21T15:51:25.859 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.289e-06 2024-09-21T15:51:25.859 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.859 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.859 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:25.859 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209321+0000", 2024-09-21T15:51:25.859 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8773999999999998e-05 2024-09-21T15:51:25.859 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.859 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.859 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:25.859 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.209341+0000", 2024-09-21T15:51:25.860 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0387999999999999e-05 2024-09-21T15:51:25.860 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.860 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.860 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:25.860 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.213464+0000", 2024-09-21T15:51:25.860 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0041232450000000002 2024-09-21T15:51:25.860 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.860 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.860 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:25.860 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.213495+0000", 2024-09-21T15:51:25.860 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0793999999999997e-05 2024-09-21T15:51:25.860 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.860 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:25.860 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.860 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.860 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.861 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13445 3.0 e31/22 3:038b8840:::rbd_data.1133dc5adfdf.0000000000000035:head v 31'50, mlcod=31'48)", 2024-09-21T15:51:25.861 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.232497+0000", 2024-09-21T15:51:25.861 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 128.61566208100001, 2024-09-21T15:51:25.861 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0040493559999999996, 2024-09-21T15:51:25.861 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:25.861 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:25.861 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:25.861 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.861 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:25.861 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232497+0000", 2024-09-21T15:51:25.861 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.861 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.861 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.861 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:25.861 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232497+0000", 2024-09-21T15:51:25.861 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.862 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.862 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.862 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:25.862 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232496+0000", 2024-09-21T15:51:25.862 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:25.862 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.862 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.862 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:25.862 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232503+0000", 2024-09-21T15:51:25.862 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7320000000000001e-06 2024-09-21T15:51:25.862 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.862 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.862 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:25.862 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232512+0000", 2024-09-21T15:51:25.862 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9259999999999995e-06 2024-09-21T15:51:25.863 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.863 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.863 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:25.863 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232520+0000", 2024-09-21T15:51:25.863 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.3699999999999997e-06 2024-09-21T15:51:25.863 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.863 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.863 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:25.863 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232564+0000", 2024-09-21T15:51:25.863 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4128999999999997e-05 2024-09-21T15:51:25.863 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.863 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.863 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:25.863 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.232585+0000", 2024-09-21T15:51:25.863 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0704999999999999e-05 2024-09-21T15:51:25.863 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.864 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.864 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:25.864 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.236507+0000", 2024-09-21T15:51:25.864 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0039218730000000002 2024-09-21T15:51:25.864 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.864 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.864 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:25.864 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.236546+0000", 2024-09-21T15:51:25.864 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9399e-05 2024-09-21T15:51:25.864 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.864 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:25.864 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.864 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.864 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.864 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13839 3.0 e31/22 3:03c4fb32:::rbd_data.1133dc5adfdf.0000000000000099:head v 31'59, mlcod=31'56)", 2024-09-21T15:51:25.864 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.528097+0000", 2024-09-21T15:51:25.864 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 128.32006195700001, 2024-09-21T15:51:25.865 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0057228840000000001, 2024-09-21T15:51:25.865 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:25.865 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:25.865 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:25.865 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.865 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:25.865 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528097+0000", 2024-09-21T15:51:25.865 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.865 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.865 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.865 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:25.865 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528097+0000", 2024-09-21T15:51:25.865 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.866 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.866 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.866 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:25.866 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528096+0000", 2024-09-21T15:51:25.866 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:25.866 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.866 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.866 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:25.866 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528105+0000", 2024-09-21T15:51:25.866 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7350000000000002e-06 2024-09-21T15:51:25.866 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.866 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.866 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:25.866 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528117+0000", 2024-09-21T15:51:25.866 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1909000000000001e-05 2024-09-21T15:51:25.867 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.867 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.867 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:25.867 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528124+0000", 2024-09-21T15:51:25.867 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8419999999999999e-06 2024-09-21T15:51:25.867 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.867 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.867 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:25.867 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528168+0000", 2024-09-21T15:51:25.867 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4301999999999997e-05 2024-09-21T15:51:25.867 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.867 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.867 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:25.867 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.528191+0000", 2024-09-21T15:51:25.867 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2622000000000001e-05 2024-09-21T15:51:25.868 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.868 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.868 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:25.868 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.533780+0000", 2024-09-21T15:51:25.868 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0055897519999999999 2024-09-21T15:51:25.868 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.868 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.868 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:25.868 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.533820+0000", 2024-09-21T15:51:25.868 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9366999999999998e-05 2024-09-21T15:51:25.868 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.868 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:25.868 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.868 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.868 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.869 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:13884 3.9 e31/22 3:93003736:::rbd_data.1133dc5adfdf.00000000000000a1:head v 31'54, mlcod=31'52)", 2024-09-21T15:51:25.869 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.562433+0000", 2024-09-21T15:51:25.869 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 128.28572520899999, 2024-09-21T15:51:25.869 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.005162313, 2024-09-21T15:51:25.869 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:25.869 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:25.869 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:25.869 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.869 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:25.869 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562433+0000", 2024-09-21T15:51:25.869 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.869 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.869 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.869 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:25.869 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562433+0000", 2024-09-21T15:51:25.869 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.869 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.870 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.870 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:25.870 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562432+0000", 2024-09-21T15:51:25.870 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:25.870 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.870 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.870 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:25.870 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562467+0000", 2024-09-21T15:51:25.870 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4885999999999998e-05 2024-09-21T15:51:25.870 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.870 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.870 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:25.870 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562481+0000", 2024-09-21T15:51:25.870 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3604000000000001e-05 2024-09-21T15:51:25.870 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.871 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.871 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:25.871 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562489+0000", 2024-09-21T15:51:25.871 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1829999999999994e-06 2024-09-21T15:51:25.871 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.871 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.871 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:25.871 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562536+0000", 2024-09-21T15:51:25.871 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6830999999999999e-05 2024-09-21T15:51:25.871 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.871 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.871 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:25.871 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.562559+0000", 2024-09-21T15:51:25.871 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2765999999999999e-05 2024-09-21T15:51:25.871 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.872 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.872 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:25.872 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.567566+0000", 2024-09-21T15:51:25.872 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0050073219999999998 2024-09-21T15:51:25.872 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.872 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.872 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:25.872 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.567596+0000", 2024-09-21T15:51:25.872 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9802999999999999e-05 2024-09-21T15:51:25.872 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.872 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:25.872 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.872 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.872 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.872 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:22340 3.9 e34/22 3:910a5abe:::rbd_data.1133896adcfd.0000000000000023:head v 34'92, mlcod=34'90)", 2024-09-21T15:51:25.873 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:21.628025+0000", 2024-09-21T15:51:25.873 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 124.22013382900001, 2024-09-21T15:51:25.873 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046196010000000001, 2024-09-21T15:51:25.873 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:25.873 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:25.873 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:25.873 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.873 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:25.873 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628025+0000", 2024-09-21T15:51:25.873 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.873 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.873 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.873 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:25.873 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628025+0000", 2024-09-21T15:51:25.873 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.873 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.874 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.874 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:25.874 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628024+0000", 2024-09-21T15:51:25.874 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:25.874 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.874 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.874 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:25.874 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628034+0000", 2024-09-21T15:51:25.874 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.3470000000000005e-06 2024-09-21T15:51:25.874 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.874 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.874 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:25.874 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628043+0000", 2024-09-21T15:51:25.874 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9450000000000005e-06 2024-09-21T15:51:25.875 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.875 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.875 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:25.875 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628048+0000", 2024-09-21T15:51:25.875 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8080000000000004e-06 2024-09-21T15:51:25.875 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.875 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.875 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:25.875 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628081+0000", 2024-09-21T15:51:25.875 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2738999999999999e-05 2024-09-21T15:51:25.875 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.875 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.875 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:25.875 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.628094+0000", 2024-09-21T15:51:25.875 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3273e-05 2024-09-21T15:51:25.876 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.876 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.876 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:25.876 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.632616+0000", 2024-09-21T15:51:25.876 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045217269999999997 2024-09-21T15:51:25.876 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.876 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.876 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:25.876 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.632644+0000", 2024-09-21T15:51:25.876 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8469000000000001e-05 2024-09-21T15:51:25.876 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.876 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:25.876 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.876 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.876 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.876 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:22417 3.4 e34/22 3:25310a10:::rbd_data.1133896adcfd.0000000000000075:head v 34'48, mlcod=34'46)", 2024-09-21T15:51:25.877 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:21.704297+0000", 2024-09-21T15:51:25.877 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 124.143861731, 2024-09-21T15:51:25.877 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0038101799999999998, 2024-09-21T15:51:25.877 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:25.877 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:25.877 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:25.877 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.877 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:25.877 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704297+0000", 2024-09-21T15:51:25.877 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.877 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.877 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.877 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:25.877 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704297+0000", 2024-09-21T15:51:25.877 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.878 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.878 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.878 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:25.878 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704296+0000", 2024-09-21T15:51:25.878 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:51:25.878 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.878 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.878 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:25.878 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704308+0000", 2024-09-21T15:51:25.878 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2559e-05 2024-09-21T15:51:25.878 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.878 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.878 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:25.878 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704337+0000", 2024-09-21T15:51:25.879 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8960000000000001e-05 2024-09-21T15:51:25.879 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.879 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.879 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:25.879 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704345+0000", 2024-09-21T15:51:25.879 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3059999999999998e-06 2024-09-21T15:51:25.879 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.879 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.879 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:25.879 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704408+0000", 2024-09-21T15:51:25.879 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2156000000000006e-05 2024-09-21T15:51:25.879 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.879 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.879 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:25.879 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.704430+0000", 2024-09-21T15:51:25.880 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1931000000000001e-05 2024-09-21T15:51:25.880 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.880 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.880 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:25.880 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.708073+0000", 2024-09-21T15:51:25.880 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003643028 2024-09-21T15:51:25.880 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.880 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.880 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:25.880 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:21.708107+0000", 2024-09-21T15:51:25.880 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4613999999999998e-05 2024-09-21T15:51:25.880 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.880 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:25.880 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.880 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.881 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.881 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:25278 3.4 e38/22 3:2251cc1c:::rbd_data.1133ebd6d3ba.0000000000000024:head v 38'51, mlcod=38'49)", 2024-09-21T15:51:25.881 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.317337+0000", 2024-09-21T15:51:25.881 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 119.53082197099999, 2024-09-21T15:51:25.881 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053904110000000003, 2024-09-21T15:51:25.881 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:25.881 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:25.881 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:25.881 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.881 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:25.881 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317337+0000", 2024-09-21T15:51:25.881 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.881 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.881 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.881 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:25.882 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317337+0000", 2024-09-21T15:51:25.882 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.882 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.882 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.882 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:25.882 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317335+0000", 2024-09-21T15:51:25.882 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:51:25.882 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.882 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.882 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:25.882 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317347+0000", 2024-09-21T15:51:25.882 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.19e-05 2024-09-21T15:51:25.882 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.882 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.883 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:25.883 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317369+0000", 2024-09-21T15:51:25.883 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2232e-05 2024-09-21T15:51:25.883 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.883 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.883 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:25.883 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317381+0000", 2024-09-21T15:51:25.883 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2231e-05 2024-09-21T15:51:25.883 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.883 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.883 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:25.883 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317425+0000", 2024-09-21T15:51:25.883 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3625000000000003e-05 2024-09-21T15:51:25.883 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.883 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.884 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:25.884 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317449+0000", 2024-09-21T15:51:25.884 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4301999999999999e-05 2024-09-21T15:51:25.884 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.884 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.884 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:25.884 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.322688+0000", 2024-09-21T15:51:25.884 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.005238609 2024-09-21T15:51:25.884 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.884 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.884 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:25.884 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.322727+0000", 2024-09-21T15:51:25.884 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9201e-05 2024-09-21T15:51:25.884 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.884 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:25.885 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.885 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.885 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.885 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:25286 3.0 e38/22 3:0248fb60:::rbd_data.1133ebd6d3ba.0000000000000066:head v 38'109, mlcod=34'107)", 2024-09-21T15:51:25.885 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.317638+0000", 2024-09-21T15:51:25.885 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 119.530521092, 2024-09-21T15:51:25.885 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0054074370000000002, 2024-09-21T15:51:25.885 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:25.885 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:25.885 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:25.885 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.885 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:25.885 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317638+0000", 2024-09-21T15:51:25.885 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.885 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.886 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.886 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:25.886 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317638+0000", 2024-09-21T15:51:25.886 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.886 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.886 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.886 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:25.886 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317637+0000", 2024-09-21T15:51:25.886 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:25.886 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.886 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.886 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:25.886 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317643+0000", 2024-09-21T15:51:25.886 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5609999999999998e-06 2024-09-21T15:51:25.887 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.887 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.887 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:25.887 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317653+0000", 2024-09-21T15:51:25.887 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0705e-05 2024-09-21T15:51:25.887 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.887 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.887 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:25.887 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317660+0000", 2024-09-21T15:51:25.887 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2099999999999998e-06 2024-09-21T15:51:25.887 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.887 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.887 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:25.887 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317704+0000", 2024-09-21T15:51:25.887 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4659000000000001e-05 2024-09-21T15:51:25.888 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.888 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.888 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:25.888 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317725+0000", 2024-09-21T15:51:25.888 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0537e-05 2024-09-21T15:51:25.888 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.888 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.888 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:25.888 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323010+0000", 2024-09-21T15:51:25.888 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052853420000000002 2024-09-21T15:51:25.888 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.888 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.888 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:25.888 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323045+0000", 2024-09-21T15:51:25.888 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4860000000000002e-05 2024-09-21T15:51:25.888 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.889 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:25.889 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.889 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.889 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.889 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:25280 3.4 e38/22 3:27fca38c:::rbd_data.1133ebd6d3ba.0000000000000026:head v 38'52, mlcod=38'49)", 2024-09-21T15:51:25.889 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:26.317684+0000", 2024-09-21T15:51:25.889 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 119.530475196, 2024-09-21T15:51:25.889 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053570809999999996, 2024-09-21T15:51:25.889 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:25.889 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:25.889 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:25.889 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.889 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:25.889 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317684+0000", 2024-09-21T15:51:25.889 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.890 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.890 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.890 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:25.890 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317684+0000", 2024-09-21T15:51:25.890 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.890 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.890 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.890 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:25.890 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317683+0000", 2024-09-21T15:51:25.890 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:25.890 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.890 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.890 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:25.890 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317688+0000", 2024-09-21T15:51:25.891 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5009999999999998e-06 2024-09-21T15:51:25.891 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.891 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.891 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:25.891 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317694+0000", 2024-09-21T15:51:25.891 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3579999999999999e-06 2024-09-21T15:51:25.891 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.891 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.891 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:25.891 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317698+0000", 2024-09-21T15:51:25.891 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4150000000000001e-06 2024-09-21T15:51:25.891 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.891 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.891 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:25.891 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317738+0000", 2024-09-21T15:51:25.892 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9575e-05 2024-09-21T15:51:25.892 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.892 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.892 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:25.892 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.317762+0000", 2024-09-21T15:51:25.892 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4173999999999999e-05 2024-09-21T15:51:25.892 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.892 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.892 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:25.892 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323005+0000", 2024-09-21T15:51:25.892 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00524333 2024-09-21T15:51:25.892 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.892 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.892 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:25.892 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:26.323041+0000", 2024-09-21T15:51:25.893 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5139999999999999e-05 2024-09-21T15:51:25.893 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.893 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:25.893 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.893 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.893 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.893 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:34837 3.0 e43/22 3:038527c8:::rbd_header.1133fed1a840:head v 43'130, mlcod=43'128)", 2024-09-21T15:51:25.893 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:51:25.893 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 114.864270498, 2024-09-21T15:51:25.893 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045776519999999998, 2024-09-21T15:51:25.893 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:25.893 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:25.893 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:25.893 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.893 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:25.894 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:51:25.894 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.894 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.894 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.894 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:25.894 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:51:25.894 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.894 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.894 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.894 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:25.894 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983888+0000", 2024-09-21T15:51:25.894 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:25.894 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.894 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.895 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:25.895 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983892+0000", 2024-09-21T15:51:25.895 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8450000000000002e-06 2024-09-21T15:51:25.895 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.895 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.895 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:25.895 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983902+0000", 2024-09-21T15:51:25.895 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.0299999999999999e-06 2024-09-21T15:51:25.895 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.895 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.895 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:25.895 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983906+0000", 2024-09-21T15:51:25.895 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2300000000000002e-06 2024-09-21T15:51:25.895 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.895 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.896 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:25.896 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983940+0000", 2024-09-21T15:51:25.896 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4563000000000001e-05 2024-09-21T15:51:25.896 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.896 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.896 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:25.896 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.983960+0000", 2024-09-21T15:51:25.896 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9199999999999999e-05 2024-09-21T15:51:25.896 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.896 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.896 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:25.896 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.988430+0000", 2024-09-21T15:51:25.896 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0044709149999999998 2024-09-21T15:51:25.896 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.896 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.897 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:25.897 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:30.988466+0000", 2024-09-21T15:51:25.897 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5420000000000003e-05 2024-09-21T15:51:25.897 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.897 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:25.897 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.897 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.897 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.897 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "pg_update_log_missing(3.0 epoch 50/22 rep_tid 498 entries 50'171 (0'0) error 3:5df4a5a0:::rbd_data.11339e87a32a.000000000000008e:head by client.4403.0:80207 0.000000 -2 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 47'167)", 2024-09-21T15:51:25.897 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:51:25.897 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 104.906397625, 2024-09-21T15:51:25.897 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0047253119999999997, 2024-09-21T15:51:25.897 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:25.897 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-09-21T15:51:25.897 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:25.898 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.898 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:25.898 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:51:25.898 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.898 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.898 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.898 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:25.898 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:51:25.898 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.898 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.898 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.898 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:25.898 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941761+0000", 2024-09-21T15:51:25.898 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:25.898 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.899 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.899 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:25.899 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941768+0000", 2024-09-21T15:51:25.899 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7309999999999993e-06 2024-09-21T15:51:25.899 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.899 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.899 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:25.899 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941780+0000", 2024-09-21T15:51:25.899 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1399999999999999e-05 2024-09-21T15:51:25.899 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.899 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.899 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:25.899 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941789+0000", 2024-09-21T15:51:25.899 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5230000000000001e-06 2024-09-21T15:51:25.899 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.900 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.900 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:25.900 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.941841+0000", 2024-09-21T15:51:25.900 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1863999999999997e-05 2024-09-21T15:51:25.900 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.900 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.900 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:25.900 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:40.946486+0000", 2024-09-21T15:51:25.900 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0046452539999999997 2024-09-21T15:51:25.900 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.900 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:25.900 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.900 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.900 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.900 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "pg_update_log_missing(3.0 epoch 53/22 rep_tid 585 entries 53'201 (0'0) error 3:6a1d4890:::rbd_data.1133e079cae4.0000000000000004:head by client.4403.0:101089 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 53'198)", 2024-09-21T15:51:25.901 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:44.832827+0000", 2024-09-21T15:51:25.901 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 101.015331817, 2024-09-21T15:51:25.901 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053552929999999997, 2024-09-21T15:51:25.901 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:25.901 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "reached pg", 2024-09-21T15:51:25.901 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:25.901 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.901 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:25.901 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832827+0000", 2024-09-21T15:51:25.901 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.901 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.901 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.901 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:25.901 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832827+0000", 2024-09-21T15:51:25.901 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.901 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.902 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.902 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:25.902 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832826+0000", 2024-09-21T15:51:25.902 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:25.902 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.902 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.902 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:25.902 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832835+0000", 2024-09-21T15:51:25.902 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4509999999999992e-06 2024-09-21T15:51:25.902 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.902 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.902 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:25.902 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832854+0000", 2024-09-21T15:51:25.902 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9352999999999999e-05 2024-09-21T15:51:25.903 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.903 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.903 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:25.903 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.832866+0000", 2024-09-21T15:51:25.903 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1766000000000001e-05 2024-09-21T15:51:25.903 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.903 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.903 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:25.903 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.833091+0000", 2024-09-21T15:51:25.903 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00022490300000000001 2024-09-21T15:51:25.903 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.903 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.903 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:25.903 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:44.838182+0000", 2024-09-21T15:51:25.903 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0050915420000000001 2024-09-21T15:51:25.904 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.904 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:25.904 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.904 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.904 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.904 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:102822 3.4 e54/22 3:2729ba03:::rbd_object_map.113336ae1075:head v 54'208, mlcod=54'205)", 2024-09-21T15:51:25.904 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:46.249540+0000", 2024-09-21T15:51:25.904 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 99.598618584999997, 2024-09-21T15:51:25.904 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052654859999999998, 2024-09-21T15:51:25.904 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:25.904 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:25.904 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:25.904 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.904 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:25.904 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249540+0000", 2024-09-21T15:51:25.905 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.905 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.905 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.905 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:25.905 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249540+0000", 2024-09-21T15:51:25.905 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.905 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.905 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.905 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:25.905 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249539+0000", 2024-09-21T15:51:25.905 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:25.905 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.905 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.905 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:25.905 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249546+0000", 2024-09-21T15:51:25.906 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7290000000000003e-06 2024-09-21T15:51:25.906 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.906 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.906 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:25.906 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249558+0000", 2024-09-21T15:51:25.906 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1844e-05 2024-09-21T15:51:25.906 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.906 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.906 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:25.906 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249565+0000", 2024-09-21T15:51:25.906 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8210000000000004e-06 2024-09-21T15:51:25.906 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.906 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.906 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:25.906 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249602+0000", 2024-09-21T15:51:25.907 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7057000000000001e-05 2024-09-21T15:51:25.907 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.907 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.907 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:25.907 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.249622+0000", 2024-09-21T15:51:25.907 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.9678999999999999e-05 2024-09-21T15:51:25.907 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.907 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.907 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:25.907 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254783+0000", 2024-09-21T15:51:25.907 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0051611449999999998 2024-09-21T15:51:25.907 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.907 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.907 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:25.907 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254806+0000", 2024-09-21T15:51:25.908 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2853e-05 2024-09-21T15:51:25.908 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.908 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:25.908 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.908 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.908 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.908 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:102823 3.4 e54/22 3:2729ba03:::rbd_object_map.113336ae1075:head v 54'209, mlcod=54'205)", 2024-09-21T15:51:25.908 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:46.250318+0000", 2024-09-21T15:51:25.908 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 99.597841000000003, 2024-09-21T15:51:25.908 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045013539999999999, 2024-09-21T15:51:25.908 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:25.908 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:25.908 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:25.908 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.908 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:25.909 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250318+0000", 2024-09-21T15:51:25.909 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.909 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.909 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.909 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:25.909 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250318+0000", 2024-09-21T15:51:25.909 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.909 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.909 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.909 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:25.909 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250317+0000", 2024-09-21T15:51:25.909 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:25.909 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.909 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.909 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:25.910 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250322+0000", 2024-09-21T15:51:25.910 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.074e-06 2024-09-21T15:51:25.910 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.910 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.910 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:25.910 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250330+0000", 2024-09-21T15:51:25.910 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4730000000000003e-06 2024-09-21T15:51:25.910 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.910 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.910 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:25.910 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250335+0000", 2024-09-21T15:51:25.910 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8019999999999997e-06 2024-09-21T15:51:25.910 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.910 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.910 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:25.911 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250375+0000", 2024-09-21T15:51:25.911 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9459999999999998e-05 2024-09-21T15:51:25.911 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.911 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.911 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:25.911 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.250388+0000", 2024-09-21T15:51:25.911 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3288999999999999e-05 2024-09-21T15:51:25.911 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.911 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.911 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:25.911 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254807+0000", 2024-09-21T15:51:25.911 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0044192220000000004 2024-09-21T15:51:25.911 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.911 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.911 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:25.912 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:46.254819+0000", 2024-09-21T15:51:25.912 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1603999999999999e-05 2024-09-21T15:51:25.912 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.912 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:25.912 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.912 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.912 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.912 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:160603 3.9 e95/22 3:9171211b:::rbd_data.11338bae974a.0000000000000006:head v 95'343, mlcod=92'340)", 2024-09-21T15:51:25.912 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.215621+0000", 2024-09-21T15:51:25.912 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 54.632537968999998, 2024-09-21T15:51:25.912 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0041276250000000002, 2024-09-21T15:51:25.912 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:25.912 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:25.912 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:25.912 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.913 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:25.913 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215621+0000", 2024-09-21T15:51:25.913 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.913 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.913 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.913 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:25.913 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215621+0000", 2024-09-21T15:51:25.913 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.913 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.913 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.913 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:25.913 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215620+0000", 2024-09-21T15:51:25.913 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:25.913 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.913 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.914 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:25.914 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215627+0000", 2024-09-21T15:51:25.914 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8850000000000001e-06 2024-09-21T15:51:25.914 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.914 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.914 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:25.914 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215803+0000", 2024-09-21T15:51:25.914 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000176422 2024-09-21T15:51:25.914 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.914 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.914 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:25.914 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215814+0000", 2024-09-21T15:51:25.914 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0135e-05 2024-09-21T15:51:25.914 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.914 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.915 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:25.915 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215866+0000", 2024-09-21T15:51:25.915 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2327999999999998e-05 2024-09-21T15:51:25.915 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.915 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.915 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:25.915 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.215887+0000", 2024-09-21T15:51:25.915 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1449000000000001e-05 2024-09-21T15:51:25.915 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.915 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.915 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:25.915 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.219719+0000", 2024-09-21T15:51:25.915 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0038312630000000001 2024-09-21T15:51:25.915 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.915 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.916 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:25.916 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.219748+0000", 2024-09-21T15:51:25.916 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9737999999999999e-05 2024-09-21T15:51:25.916 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.916 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:25.916 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.916 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.916 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.916 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:161865 3.9 e105/22 3:90e88bad:::rbd_data.11338db70e6f.0000000000000015:head v 105'347, mlcod=103'345)", 2024-09-21T15:51:25.916 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:40.786848+0000", 2024-09-21T15:51:25.916 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 45.061311066999998, 2024-09-21T15:51:25.916 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0056408860000000003, 2024-09-21T15:51:25.916 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:25.916 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:25.917 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:25.917 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.917 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:25.917 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786848+0000", 2024-09-21T15:51:25.917 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.917 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.917 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.917 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:25.917 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786848+0000", 2024-09-21T15:51:25.917 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.917 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.917 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.917 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:25.917 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786847+0000", 2024-09-21T15:51:25.917 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:25.918 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.918 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.918 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:25.918 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786851+0000", 2024-09-21T15:51:25.918 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7419999999999999e-06 2024-09-21T15:51:25.918 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.918 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.918 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:25.918 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786872+0000", 2024-09-21T15:51:25.918 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1186e-05 2024-09-21T15:51:25.918 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.918 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.918 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:25.918 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786878+0000", 2024-09-21T15:51:25.919 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0970000000000002e-06 2024-09-21T15:51:25.919 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.919 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.919 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:25.919 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786918+0000", 2024-09-21T15:51:25.919 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.998e-05 2024-09-21T15:51:25.919 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.919 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.919 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:25.919 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.786941+0000", 2024-09-21T15:51:25.919 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.245e-05 2024-09-21T15:51:25.919 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.919 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.919 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:25.919 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.792460+0000", 2024-09-21T15:51:25.920 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0055191210000000001 2024-09-21T15:51:25.920 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.920 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.920 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:25.920 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:40.792489+0000", 2024-09-21T15:51:25.920 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8552e-05 2024-09-21T15:51:25.920 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.920 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:25.920 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.920 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.920 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.920 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.1.0:1503 3.4 e127/99 3:26dffaa1:::rbd_data.113336ae1075.000000000000006b:16 v 127'658, mlcod=127'653)", 2024-09-21T15:51:25.920 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:51:02.797289+0000", 2024-09-21T15:51:25.920 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 23.050870188000001, 2024-09-21T15:51:25.920 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0045583780000000001, 2024-09-21T15:51:25.921 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:25.921 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:25.921 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:25.921 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.921 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:25.921 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.797289+0000", 2024-09-21T15:51:25.921 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.921 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.921 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.921 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:25.921 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.797289+0000", 2024-09-21T15:51:25.921 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.921 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.921 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.922 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:25.922 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.797288+0000", 2024-09-21T15:51:25.922 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:25.922 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.922 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.922 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:25.922 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.797293+0000", 2024-09-21T15:51:25.922 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4500000000000003e-06 2024-09-21T15:51:25.922 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.922 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.922 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:25.922 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.797303+0000", 2024-09-21T15:51:25.922 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0134e-05 2024-09-21T15:51:25.922 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.922 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.923 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:25.923 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.797309+0000", 2024-09-21T15:51:25.923 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2100000000000001e-06 2024-09-21T15:51:25.923 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.923 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.923 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:25.923 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.797346+0000", 2024-09-21T15:51:25.923 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7583999999999998e-05 2024-09-21T15:51:25.923 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.923 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.923 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:25.923 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.797386+0000", 2024-09-21T15:51:25.923 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9428999999999998e-05 2024-09-21T15:51:25.923 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.923 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.924 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:25.924 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.801816+0000", 2024-09-21T15:51:25.924 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0044305730000000002 2024-09-21T15:51:25.924 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.924 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.924 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:25.924 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.801847+0000", 2024-09-21T15:51:25.924 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0639000000000003e-05 2024-09-21T15:51:25.924 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.924 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:25.924 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.924 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.924 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.924 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:164505 3.5 3:a3a629db:::rbd_data.113336ae1075.0000000000000003:head [delete] snapc 16=[] ondisk+write+known_if_redirected+supports_pool_eio e127)", 2024-09-21T15:51:25.924 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:51:02.863560+0000", 2024-09-21T15:51:25.925 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 22.984598633000001, 2024-09-21T15:51:25.925 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0040794459999999996, 2024-09-21T15:51:25.925 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:25.925 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:25.925 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:25.925 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:25.925 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:25.925 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 164505 2024-09-21T15:51:25.925 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.925 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:25.925 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.925 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:25.925 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.863560+0000", 2024-09-21T15:51:25.925 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.926 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.926 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.926 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:25.926 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.863560+0000", 2024-09-21T15:51:25.926 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.926 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.926 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.926 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:25.926 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.863558+0000", 2024-09-21T15:51:25.926 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:51:25.926 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.926 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.926 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:25.926 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.863570+0000", 2024-09-21T15:51:25.927 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1960999999999999e-05 2024-09-21T15:51:25.927 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.927 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.927 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:25.927 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.863582+0000", 2024-09-21T15:51:25.927 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1368e-05 2024-09-21T15:51:25.927 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.927 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.927 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:25.927 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.863589+0000", 2024-09-21T15:51:25.927 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1670000000000002e-06 2024-09-21T15:51:25.927 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.927 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.927 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:25.928 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.863646+0000", 2024-09-21T15:51:25.928 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7433999999999999e-05 2024-09-21T15:51:25.928 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.928 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.928 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:25.928 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.863752+0000", 2024-09-21T15:51:25.928 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000105214 2024-09-21T15:51:25.928 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.928 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.928 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-09-21T15:51:25.928 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.863866+0000", 2024-09-21T15:51:25.928 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00011479100000000001 2024-09-21T15:51:25.928 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.928 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.928 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:25.929 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.864919+0000", 2024-09-21T15:51:25.929 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00105302 2024-09-21T15:51:25.929 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.929 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.929 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:25.929 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.867550+0000", 2024-09-21T15:51:25.929 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0026310130000000002 2024-09-21T15:51:25.929 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.929 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.929 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:25.929 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.867600+0000", 2024-09-21T15:51:25.929 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9123999999999999e-05 2024-09-21T15:51:25.929 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.929 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.929 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:25.929 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:02.867640+0000", 2024-09-21T15:51:25.930 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9953000000000002e-05 2024-09-21T15:51:25.930 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.930 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:25.930 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.930 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.930 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.930 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:167470 3.9 e147/22 3:96420cbd:::rbd_data.11335fa2fe69.0000000000000019:head v 147'386, mlcod=146'383)", 2024-09-21T15:51:25.930 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:51:22.764716+0000", 2024-09-21T15:51:25.930 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 3.0834429120000002, 2024-09-21T15:51:25.930 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0037969670000000001, 2024-09-21T15:51:25.930 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:25.930 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:25.930 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:25.930 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.931 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:25.931 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:22.764716+0000", 2024-09-21T15:51:25.931 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.931 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.931 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.931 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:25.931 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:22.764716+0000", 2024-09-21T15:51:25.931 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:25.931 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.931 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.931 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:25.931 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:22.764715+0000", 2024-09-21T15:51:25.931 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:25.931 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.931 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.932 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:25.932 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:22.764721+0000", 2024-09-21T15:51:25.932 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0469999999999999e-06 2024-09-21T15:51:25.932 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.932 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.932 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:25.932 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:22.764733+0000", 2024-09-21T15:51:25.932 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1719000000000001e-05 2024-09-21T15:51:25.932 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.932 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.932 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:25.932 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:22.764739+0000", 2024-09-21T15:51:25.932 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0560000000000001e-06 2024-09-21T15:51:25.932 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.932 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.933 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:25.933 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:22.764785+0000", 2024-09-21T15:51:25.933 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6029999999999998e-05 2024-09-21T15:51:25.933 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.933 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.933 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:25.933 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:22.764806+0000", 2024-09-21T15:51:25.933 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0798e-05 2024-09-21T15:51:25.933 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.933 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.933 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:25.933 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:22.768478+0000", 2024-09-21T15:51:25.933 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0036718559999999998 2024-09-21T15:51:25.933 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:25.933 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:25.934 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:25.934 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:22.768513+0000", 2024-09-21T15:51:25.934 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5108000000000003e-05 2024-09-21T15:51:25.934 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.934 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:25.934 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.934 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:25.934 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:25.934 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:25.935 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_ops_in_flight 2024-09-21T15:51:26.078 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:26.078 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:51:26.078 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:51:26.078 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:26.086 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_blocked_ops 2024-09-21T15:51:26.188 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:26.188 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:51:26.188 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:51:26.188 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:51:26.188 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:26.197 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.3.asok dump_historic_ops 2024-09-21T15:51:26.300 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:26.300 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:51:26.300 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:51:26.300 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:51:26.300 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.300 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:14051 3.8 3:13437810:::rbd_data.1133dc5adfdf.0000000000000039:head [write 716800~796] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e32)", 2024-09-21T15:51:26.300 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:51:26.300 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 128.60309785499999, 2024-09-21T15:51:26.300 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.98064002699999997, 2024-09-21T15:51:26.300 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:26.300 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:26.301 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:26.301 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:26.301 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:26.301 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 14051 2024-09-21T15:51:26.301 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.301 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:26.301 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.301 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:26.301 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:51:26.301 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:26.301 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.301 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.301 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:26.302 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:51:26.302 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:26.302 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.302 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.302 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:26.302 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695127+0000", 2024-09-21T15:51:26.302 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:26.302 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.302 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.302 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:26.302 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695131+0000", 2024-09-21T15:51:26.302 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4680000000000003e-06 2024-09-21T15:51:26.302 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.302 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.302 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:26.303 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695139+0000", 2024-09-21T15:51:26.303 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9400000000000002e-06 2024-09-21T15:51:26.303 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.303 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.303 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:26.303 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695143+0000", 2024-09-21T15:51:26.303 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9899999999999999e-06 2024-09-21T15:51:26.303 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.303 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.303 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:26.303 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695190+0000", 2024-09-21T15:51:26.303 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6458000000000001e-05 2024-09-21T15:51:26.303 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.303 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.304 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:51:26.304 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:17.695201+0000", 2024-09-21T15:51:26.304 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0964999999999999e-05 2024-09-21T15:51:26.304 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.304 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.304 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:26.304 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.669490+0000", 2024-09-21T15:51:26.304 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.97428980499999995 2024-09-21T15:51:26.304 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.304 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.304 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for degraded object", 2024-09-21T15:51:26.304 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.670033+0000", 2024-09-21T15:51:26.304 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00054252599999999999 2024-09-21T15:51:26.304 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.304 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.305 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:26.305 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.671095+0000", 2024-09-21T15:51:26.305 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001062183 2024-09-21T15:51:26.305 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.305 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.305 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:26.305 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.671186+0000", 2024-09-21T15:51:26.305 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.1191000000000002e-05 2024-09-21T15:51:26.305 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.305 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.305 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 6", 2024-09-21T15:51:26.305 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.671346+0000", 2024-09-21T15:51:26.305 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00015980399999999999 2024-09-21T15:51:26.305 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.305 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.305 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:26.306 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.672221+0000", 2024-09-21T15:51:26.306 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00087471999999999995 2024-09-21T15:51:26.306 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.306 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.306 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:26.306 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.675619+0000", 2024-09-21T15:51:26.306 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00339858 2024-09-21T15:51:26.306 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.306 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.306 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:26.306 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.675670+0000", 2024-09-21T15:51:26.306 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0664000000000001e-05 2024-09-21T15:51:26.306 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.306 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.306 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:26.306 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:18.675768+0000", 2024-09-21T15:51:26.307 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7415000000000002e-05 2024-09-21T15:51:26.307 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:26.307 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:26.307 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:26.307 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.307 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.307 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116235 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 761856~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:26.307 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048806+0000", 2024-09-21T15:51:26.307 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 90.249418970999997, 2024-09-21T15:51:26.307 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056818556999999999, 2024-09-21T15:51:26.307 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:26.307 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:26.307 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:26.307 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:26.307 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:26.308 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116235 2024-09-21T15:51:26.308 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.308 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:26.308 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.308 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:26.308 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048806+0000", 2024-09-21T15:51:26.308 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:26.308 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.308 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.308 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:26.308 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048806+0000", 2024-09-21T15:51:26.308 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:26.308 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.308 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.308 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:26.309 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048806+0000", 2024-09-21T15:51:26.309 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:51:26.309 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.309 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.309 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:26.309 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048807+0000", 2024-09-21T15:51:26.309 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.001e-06 2024-09-21T15:51:26.309 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.309 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.309 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:26.309 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048810+0000", 2024-09-21T15:51:26.309 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.711e-06 2024-09-21T15:51:26.309 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.309 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.309 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:26.309 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048812+0000", 2024-09-21T15:51:26.309 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2290000000000002e-06 2024-09-21T15:51:26.310 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.310 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.310 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:26.310 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105495+0000", 2024-09-21T15:51:26.310 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056682679999999999 2024-09-21T15:51:26.310 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.310 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.310 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:26.310 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105565+0000", 2024-09-21T15:51:26.310 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9868000000000003e-05 2024-09-21T15:51:26.310 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.310 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.310 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:26.310 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105625+0000", 2024-09-21T15:51:26.311 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0268e-05 2024-09-21T15:51:26.311 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:26.311 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:26.311 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:26.311 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.311 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.311 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116236 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 768000~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:26.311 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048814+0000", 2024-09-21T15:51:26.311 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 90.249411308000006, 2024-09-21T15:51:26.311 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056959587999999998, 2024-09-21T15:51:26.311 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:26.311 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:26.311 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:26.311 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:26.311 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:26.311 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116236 2024-09-21T15:51:26.311 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.312 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:26.312 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.312 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:26.312 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048814+0000", 2024-09-21T15:51:26.312 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:26.312 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.312 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.312 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:26.312 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048814+0000", 2024-09-21T15:51:26.312 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:26.312 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.312 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.312 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:26.312 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048814+0000", 2024-09-21T15:51:26.312 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:51:26.313 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.313 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.313 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:26.313 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048815+0000", 2024-09-21T15:51:26.313 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7999999999999993e-07 2024-09-21T15:51:26.313 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.313 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.313 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:26.313 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048817+0000", 2024-09-21T15:51:26.313 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6620000000000001e-06 2024-09-21T15:51:26.313 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.313 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.313 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:26.313 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048820+0000", 2024-09-21T15:51:26.313 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2800000000000002e-06 2024-09-21T15:51:26.314 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.314 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.314 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:26.314 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105647+0000", 2024-09-21T15:51:26.314 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056827253000000001 2024-09-21T15:51:26.314 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.314 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.314 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:26.314 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105716+0000", 2024-09-21T15:51:26.314 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8761000000000001e-05 2024-09-21T15:51:26.314 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.314 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.314 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:26.314 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105774+0000", 2024-09-21T15:51:26.314 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7855000000000001e-05 2024-09-21T15:51:26.315 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:26.315 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:26.315 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:26.315 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.315 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.315 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116237 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 774144~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:26.315 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048822+0000", 2024-09-21T15:51:26.315 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 90.249403623000006, 2024-09-21T15:51:26.315 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057114028999999997, 2024-09-21T15:51:26.315 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:26.315 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:26.315 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:26.315 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:26.315 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:26.315 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116237 2024-09-21T15:51:26.315 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.316 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:26.316 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.316 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:26.316 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048822+0000", 2024-09-21T15:51:26.316 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:26.316 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.316 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.316 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:26.316 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048822+0000", 2024-09-21T15:51:26.316 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:26.316 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.316 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.316 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:26.316 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048822+0000", 2024-09-21T15:51:26.316 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:51:26.317 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.317 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.317 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:26.317 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048823+0000", 2024-09-21T15:51:26.317 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9699999999999994e-07 2024-09-21T15:51:26.317 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.317 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.317 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:26.317 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048825+0000", 2024-09-21T15:51:26.317 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.65e-06 2024-09-21T15:51:26.317 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.318 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.318 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:26.318 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048828+0000", 2024-09-21T15:51:26.318 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8890000000000002e-06 2024-09-21T15:51:26.318 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.318 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.318 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:26.318 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105802+0000", 2024-09-21T15:51:26.318 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056974415 2024-09-21T15:51:26.318 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.318 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.318 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:26.318 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105871+0000", 2024-09-21T15:51:26.318 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8727999999999997e-05 2024-09-21T15:51:26.318 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.319 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.319 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:26.319 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105936+0000", 2024-09-21T15:51:26.319 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4554999999999994e-05 2024-09-21T15:51:26.319 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:26.319 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:26.319 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:26.319 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.319 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.319 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116238 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 780288~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:26.319 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048830+0000", 2024-09-21T15:51:26.319 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 90.249395363999994, 2024-09-21T15:51:26.319 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057296574000000003, 2024-09-21T15:51:26.319 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:26.319 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:26.320 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:26.320 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:26.320 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:26.320 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116238 2024-09-21T15:51:26.320 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.320 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:26.320 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.320 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:26.320 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048830+0000", 2024-09-21T15:51:26.320 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:26.320 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.320 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.320 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:26.320 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048830+0000", 2024-09-21T15:51:26.320 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:26.320 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.320 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.321 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:26.321 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048830+0000", 2024-09-21T15:51:26.321 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:51:26.321 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.321 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.321 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:26.321 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048831+0000", 2024-09-21T15:51:26.321 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0160000000000001e-06 2024-09-21T15:51:26.321 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.321 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.321 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:26.321 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048834+0000", 2024-09-21T15:51:26.321 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.813e-06 2024-09-21T15:51:26.321 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.321 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.322 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:26.322 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048836+0000", 2024-09-21T15:51:26.322 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6790000000000001e-06 2024-09-21T15:51:26.322 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.322 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.322 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:26.322 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.105961+0000", 2024-09-21T15:51:26.322 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057124705999999997 2024-09-21T15:51:26.322 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.322 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.322 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:26.322 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106042+0000", 2024-09-21T15:51:26.322 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0820000000000002e-05 2024-09-21T15:51:26.322 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.323 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.323 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:26.323 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106127+0000", 2024-09-21T15:51:26.323 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4746000000000005e-05 2024-09-21T15:51:26.323 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:26.323 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:26.323 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:26.323 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.323 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.323 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116324 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 524288~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:26.323 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048838+0000", 2024-09-21T15:51:26.323 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 90.249387055, 2024-09-21T15:51:26.323 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057477185, 2024-09-21T15:51:26.323 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:26.324 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:26.324 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:26.324 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:26.324 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:26.324 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116324 2024-09-21T15:51:26.324 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.324 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:26.324 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.324 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:26.324 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048838+0000", 2024-09-21T15:51:26.324 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:26.324 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.324 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.324 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:26.324 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048838+0000", 2024-09-21T15:51:26.325 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:26.325 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.325 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.325 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:26.325 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048838+0000", 2024-09-21T15:51:26.325 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:51:26.325 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.325 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.325 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:26.325 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048842+0000", 2024-09-21T15:51:26.325 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4829999999999999e-06 2024-09-21T15:51:26.325 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.325 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.325 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:26.325 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048846+0000", 2024-09-21T15:51:26.325 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0470000000000004e-06 2024-09-21T15:51:26.325 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.326 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.326 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:26.326 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048849+0000", 2024-09-21T15:51:26.326 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3689999999999998e-06 2024-09-21T15:51:26.326 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.326 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.326 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:26.326 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106157+0000", 2024-09-21T15:51:26.326 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057307942000000001 2024-09-21T15:51:26.326 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.326 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.326 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:26.326 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106237+0000", 2024-09-21T15:51:26.326 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.0320000000000003e-05 2024-09-21T15:51:26.327 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.327 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.327 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:26.327 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106315+0000", 2024-09-21T15:51:26.327 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8227999999999997e-05 2024-09-21T15:51:26.327 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:26.327 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:26.327 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:26.327 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.327 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.327 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116325 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 530432~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:26.327 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048851+0000", 2024-09-21T15:51:26.327 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 90.249373883000004, 2024-09-21T15:51:26.327 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057581036000000002, 2024-09-21T15:51:26.327 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:26.328 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:26.328 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:26.328 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:26.328 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:26.328 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116325 2024-09-21T15:51:26.328 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.328 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:26.328 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.328 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:26.328 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048851+0000", 2024-09-21T15:51:26.328 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:26.328 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.328 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.328 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:26.328 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048851+0000", 2024-09-21T15:51:26.329 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:26.329 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.329 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.329 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:26.329 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048851+0000", 2024-09-21T15:51:26.329 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:51:26.329 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.329 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.329 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:26.329 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048852+0000", 2024-09-21T15:51:26.329 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.077e-06 2024-09-21T15:51:26.329 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.329 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.329 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:26.329 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048855+0000", 2024-09-21T15:51:26.330 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9859999999999999e-06 2024-09-21T15:51:26.330 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.330 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.330 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:26.330 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048858+0000", 2024-09-21T15:51:26.330 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.3810000000000002e-06 2024-09-21T15:51:26.330 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.330 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.330 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:26.330 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106335+0000", 2024-09-21T15:51:26.330 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057477473000000001 2024-09-21T15:51:26.330 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.330 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.330 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:26.330 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106393+0000", 2024-09-21T15:51:26.331 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.8050000000000002e-05 2024-09-21T15:51:26.331 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.331 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.331 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:26.331 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106432+0000", 2024-09-21T15:51:26.331 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9274999999999999e-05 2024-09-21T15:51:26.331 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:26.331 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:26.331 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:26.331 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.331 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.331 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116326 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 536576~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:26.331 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048860+0000", 2024-09-21T15:51:26.331 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 90.249365831999995, 2024-09-21T15:51:26.332 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057774197999999999, 2024-09-21T15:51:26.332 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:26.332 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:26.332 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:26.332 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:26.332 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:26.332 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116326 2024-09-21T15:51:26.332 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.332 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:26.332 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.332 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:26.332 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048860+0000", 2024-09-21T15:51:26.332 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:26.332 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.332 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.332 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:26.333 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048860+0000", 2024-09-21T15:51:26.333 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:26.333 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.333 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.333 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:26.333 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048859+0000", 2024-09-21T15:51:26.333 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967296 2024-09-21T15:51:26.333 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.333 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.333 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:26.333 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048860+0000", 2024-09-21T15:51:26.333 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.02e-06 2024-09-21T15:51:26.333 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.333 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.334 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:26.334 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048863+0000", 2024-09-21T15:51:26.334 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1039999999999998e-06 2024-09-21T15:51:26.334 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.334 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.334 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:26.334 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048867+0000", 2024-09-21T15:51:26.334 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.563e-06 2024-09-21T15:51:26.334 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.334 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.334 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:26.334 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106465+0000", 2024-09-21T15:51:26.334 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057597789000000003 2024-09-21T15:51:26.334 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.335 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.335 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:26.335 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106552+0000", 2024-09-21T15:51:26.335 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.7602000000000006e-05 2024-09-21T15:51:26.335 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.335 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.335 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:26.335 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106634+0000", 2024-09-21T15:51:26.335 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.1321000000000003e-05 2024-09-21T15:51:26.335 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:26.335 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:26.335 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:26.335 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.335 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.335 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116327 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 542720~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:26.336 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048870+0000", 2024-09-21T15:51:26.336 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 90.249354960000005, 2024-09-21T15:51:26.336 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057883148000000002, 2024-09-21T15:51:26.336 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:26.336 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:26.336 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:26.336 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:26.336 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:26.336 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116327 2024-09-21T15:51:26.336 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.336 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:26.336 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.336 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:26.336 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048870+0000", 2024-09-21T15:51:26.336 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:26.337 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.337 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.337 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:26.337 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048870+0000", 2024-09-21T15:51:26.337 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:26.337 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.337 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.337 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:26.337 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048870+0000", 2024-09-21T15:51:26.337 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:26.337 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.337 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.337 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:26.337 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048872+0000", 2024-09-21T15:51:26.337 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.824e-06 2024-09-21T15:51:26.338 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.338 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.338 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:26.338 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048879+0000", 2024-09-21T15:51:26.338 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6749999999999996e-06 2024-09-21T15:51:26.338 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.338 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.338 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:26.338 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048883+0000", 2024-09-21T15:51:26.338 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2200000000000003e-06 2024-09-21T15:51:26.338 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.338 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.338 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:26.338 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106663+0000", 2024-09-21T15:51:26.338 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057780049 2024-09-21T15:51:26.339 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.339 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.339 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:26.339 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106711+0000", 2024-09-21T15:51:26.339 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8433999999999997e-05 2024-09-21T15:51:26.339 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.339 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.339 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:26.339 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106754+0000", 2024-09-21T15:51:26.339 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2236e-05 2024-09-21T15:51:26.339 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:26.339 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:26.340 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:26.340 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.340 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.340 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116328 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 548864~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:26.341 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048887+0000", 2024-09-21T15:51:26.341 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 90.249338440000002, 2024-09-21T15:51:26.341 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057981984, 2024-09-21T15:51:26.341 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:26.341 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:26.341 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:26.341 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:26.341 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:26.341 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116328 2024-09-21T15:51:26.341 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.341 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:26.341 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.341 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:26.341 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048887+0000", 2024-09-21T15:51:26.342 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:26.342 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.342 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.342 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:26.342 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048887+0000", 2024-09-21T15:51:26.342 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:26.342 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.342 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.342 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:26.342 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048887+0000", 2024-09-21T15:51:26.342 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:26.342 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.342 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.342 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:26.343 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048891+0000", 2024-09-21T15:51:26.343 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.843e-06 2024-09-21T15:51:26.343 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.343 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.343 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:26.343 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048897+0000", 2024-09-21T15:51:26.343 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5419999999999996e-06 2024-09-21T15:51:26.343 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.343 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.343 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:26.343 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048901+0000", 2024-09-21T15:51:26.343 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4689999999999999e-06 2024-09-21T15:51:26.343 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.343 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.343 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:26.344 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106780+0000", 2024-09-21T15:51:26.344 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057878259000000001 2024-09-21T15:51:26.344 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.344 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.344 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:26.344 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106827+0000", 2024-09-21T15:51:26.344 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7098000000000002e-05 2024-09-21T15:51:26.344 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.344 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.344 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:26.344 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106869+0000", 2024-09-21T15:51:26.344 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2129999999999998e-05 2024-09-21T15:51:26.344 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:26.344 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:26.344 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:26.345 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.345 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.345 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116329 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 555008~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:26.345 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048905+0000", 2024-09-21T15:51:26.345 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 90.249320330000003, 2024-09-21T15:51:26.345 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058115554999999999, 2024-09-21T15:51:26.345 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:26.345 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:26.345 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:26.345 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:26.345 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:26.345 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116329 2024-09-21T15:51:26.345 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.345 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:26.345 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.346 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:26.346 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048905+0000", 2024-09-21T15:51:26.346 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:26.346 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.346 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.346 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:26.346 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048905+0000", 2024-09-21T15:51:26.346 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:26.346 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.346 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.346 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:26.346 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048905+0000", 2024-09-21T15:51:26.346 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:26.346 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.346 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.347 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:26.347 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048907+0000", 2024-09-21T15:51:26.347 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.086e-06 2024-09-21T15:51:26.347 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.347 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.347 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:26.347 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048912+0000", 2024-09-21T15:51:26.347 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2830000000000001e-06 2024-09-21T15:51:26.347 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.347 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.347 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:26.347 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048916+0000", 2024-09-21T15:51:26.347 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.98e-06 2024-09-21T15:51:26.347 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.347 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.348 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:26.348 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106907+0000", 2024-09-21T15:51:26.348 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057990719000000003 2024-09-21T15:51:26.348 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.348 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.348 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:26.348 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.106969+0000", 2024-09-21T15:51:26.348 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1789000000000001e-05 2024-09-21T15:51:26.348 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.348 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.348 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:26.348 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107021+0000", 2024-09-21T15:51:26.348 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2030999999999997e-05 2024-09-21T15:51:26.348 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:26.349 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:26.349 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:26.349 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.349 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.349 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116330 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 561152~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:26.349 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.048922+0000", 2024-09-21T15:51:26.349 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 90.249303171999998, 2024-09-21T15:51:26.349 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058242343000000002, 2024-09-21T15:51:26.349 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:26.349 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:26.349 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:26.349 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:26.349 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:26.349 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116330 2024-09-21T15:51:26.350 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.350 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:26.350 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.350 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:26.350 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048922+0000", 2024-09-21T15:51:26.350 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:26.350 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.350 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.350 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:26.350 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048922+0000", 2024-09-21T15:51:26.350 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:26.350 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.350 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.350 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:26.351 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048922+0000", 2024-09-21T15:51:26.351 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:26.351 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.351 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.351 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:26.351 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048924+0000", 2024-09-21T15:51:26.351 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1660000000000001e-06 2024-09-21T15:51:26.351 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.351 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.351 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:26.351 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048930+0000", 2024-09-21T15:51:26.351 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4529999999999999e-06 2024-09-21T15:51:26.351 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.351 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.351 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:26.352 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048934+0000", 2024-09-21T15:51:26.352 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.0269999999999997e-06 2024-09-21T15:51:26.352 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.352 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.352 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:26.352 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107047+0000", 2024-09-21T15:51:26.352 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058112913000000002 2024-09-21T15:51:26.352 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.352 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.352 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:26.352 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107095+0000", 2024-09-21T15:51:26.352 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7778000000000002e-05 2024-09-21T15:51:26.352 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.352 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.352 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:26.353 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107165+0000", 2024-09-21T15:51:26.353 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9370999999999997e-05 2024-09-21T15:51:26.353 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:26.353 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:26.353 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:26.353 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.353 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.353 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116331 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 567296~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:26.353 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.049440+0000", 2024-09-21T15:51:26.353 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 90.248785077999997, 2024-09-21T15:51:26.353 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057878485, 2024-09-21T15:51:26.353 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:26.353 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:26.353 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:26.354 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:26.354 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:26.354 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116331 2024-09-21T15:51:26.354 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.354 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:26.354 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.354 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:26.354 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049440+0000", 2024-09-21T15:51:26.354 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:26.354 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.354 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.354 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:26.354 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049440+0000", 2024-09-21T15:51:26.354 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:26.355 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.355 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.355 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:26.355 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.048939+0000", 2024-09-21T15:51:26.355 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9994984 2024-09-21T15:51:26.355 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.355 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.355 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:26.355 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049444+0000", 2024-09-21T15:51:26.355 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00050557200000000003 2024-09-21T15:51:26.355 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.355 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.355 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:26.355 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049460+0000", 2024-09-21T15:51:26.355 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.6120000000000002e-05 2024-09-21T15:51:26.356 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.356 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.356 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:26.356 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049468+0000", 2024-09-21T15:51:26.356 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4209999999999998e-06 2024-09-21T15:51:26.356 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.356 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.356 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:26.356 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107189+0000", 2024-09-21T15:51:26.356 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057721019999999998 2024-09-21T15:51:26.356 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.356 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.356 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:26.356 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107254+0000", 2024-09-21T15:51:26.356 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5265000000000006e-05 2024-09-21T15:51:26.356 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.357 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.357 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:26.357 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107319+0000", 2024-09-21T15:51:26.357 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4784999999999997e-05 2024-09-21T15:51:26.357 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:26.357 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:26.357 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:26.357 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.357 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.357 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116332 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 573440~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:26.357 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.049474+0000", 2024-09-21T15:51:26.357 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 90.248751816999999, 2024-09-21T15:51:26.357 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.058013809, 2024-09-21T15:51:26.357 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:26.357 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:26.358 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:26.358 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:26.358 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:26.358 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116332 2024-09-21T15:51:26.358 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.358 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:26.358 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.358 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:26.358 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049474+0000", 2024-09-21T15:51:26.358 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:26.358 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.358 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.358 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:26.358 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049474+0000", 2024-09-21T15:51:26.359 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:26.359 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.359 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.359 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:26.359 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049473+0000", 2024-09-21T15:51:26.359 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:26.359 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.359 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.359 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:26.359 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049476+0000", 2024-09-21T15:51:26.359 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6249999999999999e-06 2024-09-21T15:51:26.359 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.359 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.360 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:26.360 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049482+0000", 2024-09-21T15:51:26.360 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0959999999999997e-06 2024-09-21T15:51:26.360 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.360 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.360 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:26.360 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049487+0000", 2024-09-21T15:51:26.360 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5949999999999998e-06 2024-09-21T15:51:26.360 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.360 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.360 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:26.360 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107344+0000", 2024-09-21T15:51:26.360 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057856594999999997 2024-09-21T15:51:26.360 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.360 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.360 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:26.361 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107422+0000", 2024-09-21T15:51:26.361 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7582000000000004e-05 2024-09-21T15:51:26.361 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.361 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.361 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:26.361 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107487+0000", 2024-09-21T15:51:26.361 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5723999999999996e-05 2024-09-21T15:51:26.361 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:26.361 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:26.361 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:26.361 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.361 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.361 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116333 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 579584~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:26.361 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.049993+0000", 2024-09-21T15:51:26.361 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 90.248232033999997, 2024-09-21T15:51:26.362 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057663537000000001, 2024-09-21T15:51:26.362 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:26.362 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:26.362 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:26.362 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:26.362 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:26.362 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116333 2024-09-21T15:51:26.362 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.362 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:26.362 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.362 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:26.362 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049993+0000", 2024-09-21T15:51:26.362 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:26.362 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.363 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.363 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:26.363 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049993+0000", 2024-09-21T15:51:26.363 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:26.363 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.363 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.363 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:26.363 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049491+0000", 2024-09-21T15:51:26.363 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9994979 2024-09-21T15:51:26.363 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.363 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.363 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:26.363 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.049998+0000", 2024-09-21T15:51:26.363 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00050654200000000002 2024-09-21T15:51:26.364 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.364 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.364 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:26.364 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050010+0000", 2024-09-21T15:51:26.364 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1790999999999999e-05 2024-09-21T15:51:26.364 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.364 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.364 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:26.364 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050016+0000", 2024-09-21T15:51:26.364 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2179999999999996e-06 2024-09-21T15:51:26.364 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.364 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.364 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:26.364 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107513+0000", 2024-09-21T15:51:26.364 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057496927000000003 2024-09-21T15:51:26.365 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.365 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.365 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:26.365 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107590+0000", 2024-09-21T15:51:26.365 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7547999999999997e-05 2024-09-21T15:51:26.365 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.365 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.365 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:26.365 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107657+0000", 2024-09-21T15:51:26.365 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6630000000000004e-05 2024-09-21T15:51:26.365 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:26.365 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:26.365 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:26.365 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.365 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.365 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116334 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 585728~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:26.366 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.050091+0000", 2024-09-21T15:51:26.366 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 90.248134406000005, 2024-09-21T15:51:26.366 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057734355000000001, 2024-09-21T15:51:26.366 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:26.366 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:26.366 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:26.366 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:26.366 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:26.366 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116334 2024-09-21T15:51:26.366 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.366 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:26.366 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.366 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:26.367 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050091+0000", 2024-09-21T15:51:26.367 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:26.367 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.367 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.367 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:26.367 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050091+0000", 2024-09-21T15:51:26.367 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:26.367 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.367 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.367 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:26.367 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050021+0000", 2024-09-21T15:51:26.367 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999304 2024-09-21T15:51:26.367 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.367 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.367 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:26.368 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050095+0000", 2024-09-21T15:51:26.368 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.4158000000000004e-05 2024-09-21T15:51:26.368 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.368 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.368 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:26.368 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050103+0000", 2024-09-21T15:51:26.368 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0770000000000002e-06 2024-09-21T15:51:26.368 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.368 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.368 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:26.368 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050108+0000", 2024-09-21T15:51:26.368 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6330000000000001e-06 2024-09-21T15:51:26.368 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.368 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.368 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:26.369 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107681+0000", 2024-09-21T15:51:26.369 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057572996000000001 2024-09-21T15:51:26.369 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.369 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.369 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:26.369 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107759+0000", 2024-09-21T15:51:26.369 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7368999999999997e-05 2024-09-21T15:51:26.369 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.369 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.369 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:26.369 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107825+0000", 2024-09-21T15:51:26.369 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6783000000000001e-05 2024-09-21T15:51:26.369 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:26.369 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:26.369 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:26.370 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.370 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.370 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116335 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 591872~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:26.370 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.050542+0000", 2024-09-21T15:51:26.370 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 90.247683820999995, 2024-09-21T15:51:26.370 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057466455999999999, 2024-09-21T15:51:26.370 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:26.370 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:26.370 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:26.370 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:26.370 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:26.370 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116335 2024-09-21T15:51:26.370 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.371 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:26.371 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.371 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:26.371 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050542+0000", 2024-09-21T15:51:26.371 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:26.371 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.371 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.371 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:26.371 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050542+0000", 2024-09-21T15:51:26.371 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:26.371 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.371 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.371 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:26.371 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050113+0000", 2024-09-21T15:51:26.372 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9995713 2024-09-21T15:51:26.372 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.372 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.372 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:26.372 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050546+0000", 2024-09-21T15:51:26.372 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00043328699999999997 2024-09-21T15:51:26.372 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.372 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.372 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:26.372 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050557+0000", 2024-09-21T15:51:26.372 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1011e-05 2024-09-21T15:51:26.372 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.372 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.372 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:26.372 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050564+0000", 2024-09-21T15:51:26.372 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.1130000000000003e-06 2024-09-21T15:51:26.373 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.373 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.373 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:26.373 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107852+0000", 2024-09-21T15:51:26.373 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057287350000000001 2024-09-21T15:51:26.373 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.373 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.373 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:26.373 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.107928+0000", 2024-09-21T15:51:26.373 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.6668000000000005e-05 2024-09-21T15:51:26.373 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.373 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.373 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:26.373 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108008+0000", 2024-09-21T15:51:26.373 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.9783999999999994e-05 2024-09-21T15:51:26.373 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:26.374 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:26.374 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:26.374 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.374 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.374 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116336 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 598016~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:26.374 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.050834+0000", 2024-09-21T15:51:26.374 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 90.247391554999993, 2024-09-21T15:51:26.374 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057353572999999998, 2024-09-21T15:51:26.374 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:26.374 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:26.374 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:26.374 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:26.374 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:26.374 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116336 2024-09-21T15:51:26.374 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.375 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:26.375 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.375 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:26.375 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050834+0000", 2024-09-21T15:51:26.375 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:26.375 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.375 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.375 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:26.375 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050834+0000", 2024-09-21T15:51:26.375 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:26.375 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.375 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.375 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:26.375 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050571+0000", 2024-09-21T15:51:26.375 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9997373 2024-09-21T15:51:26.376 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.376 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.376 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:26.376 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050842+0000", 2024-09-21T15:51:26.376 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00027029099999999998 2024-09-21T15:51:26.376 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.376 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.376 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:26.376 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050853+0000", 2024-09-21T15:51:26.376 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1532000000000001e-05 2024-09-21T15:51:26.376 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.376 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.376 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:26.376 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050863+0000", 2024-09-21T15:51:26.376 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4239999999999999e-06 2024-09-21T15:51:26.376 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.377 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.377 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:26.377 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108045+0000", 2024-09-21T15:51:26.377 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057182415 2024-09-21T15:51:26.377 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.377 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.377 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:26.377 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108123+0000", 2024-09-21T15:51:26.377 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8275000000000005e-05 2024-09-21T15:51:26.377 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.377 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.377 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:26.377 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108187+0000", 2024-09-21T15:51:26.377 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4152000000000005e-05 2024-09-21T15:51:26.377 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:26.377 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:26.378 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:26.378 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.378 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.378 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116337 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 604160~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:26.378 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.051192+0000", 2024-09-21T15:51:26.378 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 90.247032927000006, 2024-09-21T15:51:26.378 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057175981000000001, 2024-09-21T15:51:26.378 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:26.378 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:26.378 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:26.378 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:26.378 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:26.378 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116337 2024-09-21T15:51:26.378 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.378 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:26.379 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.379 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:26.379 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051192+0000", 2024-09-21T15:51:26.379 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:26.379 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.379 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.379 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:26.379 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051192+0000", 2024-09-21T15:51:26.379 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:26.379 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.379 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.379 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:26.379 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.050870+0000", 2024-09-21T15:51:26.379 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9996777 2024-09-21T15:51:26.379 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.380 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.380 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:26.380 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051197+0000", 2024-09-21T15:51:26.380 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00032656299999999998 2024-09-21T15:51:26.380 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.380 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.380 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:26.380 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051209+0000", 2024-09-21T15:51:26.380 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2636000000000001e-05 2024-09-21T15:51:26.380 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.380 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.380 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:26.380 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051217+0000", 2024-09-21T15:51:26.380 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5989999999999996e-06 2024-09-21T15:51:26.380 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.380 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.381 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:26.381 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108224+0000", 2024-09-21T15:51:26.381 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.057007622000000001 2024-09-21T15:51:26.381 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.381 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.381 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:26.381 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108299+0000", 2024-09-21T15:51:26.381 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5099999999999996e-05 2024-09-21T15:51:26.381 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.381 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.381 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:26.381 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108368+0000", 2024-09-21T15:51:26.381 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8918000000000007e-05 2024-09-21T15:51:26.381 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:26.381 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:26.382 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:26.382 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.382 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.382 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:116338 3.c 3:341c5cec:::rbd_data.11334c3240fd.0000000000000002:1c [read 610304~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e63)", 2024-09-21T15:51:26.382 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:49:56.051583+0000", 2024-09-21T15:51:26.382 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 90.246642473999998, 2024-09-21T15:51:26.382 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056967716000000002, 2024-09-21T15:51:26.382 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:26.382 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:26.382 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:26.382 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:26.382 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:26.382 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 116338 2024-09-21T15:51:26.382 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.382 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:26.383 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.383 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:26.383 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051583+0000", 2024-09-21T15:51:26.383 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:26.383 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.383 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.383 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:26.383 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051583+0000", 2024-09-21T15:51:26.383 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:26.383 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.383 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.383 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:26.383 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051223+0000", 2024-09-21T15:51:26.383 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.99964 2024-09-21T15:51:26.383 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.384 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.384 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:26.384 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051587+0000", 2024-09-21T15:51:26.384 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00036352999999999998 2024-09-21T15:51:26.384 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.384 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.384 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:26.384 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051597+0000", 2024-09-21T15:51:26.384 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.893e-06 2024-09-21T15:51:26.384 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.384 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.384 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:26.384 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.051603+0000", 2024-09-21T15:51:26.384 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0519999999999998e-06 2024-09-21T15:51:26.384 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.385 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.385 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:26.385 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108394+0000", 2024-09-21T15:51:26.385 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.056791531999999999 2024-09-21T15:51:26.385 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.385 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.385 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:26.385 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108480+0000", 2024-09-21T15:51:26.385 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.577e-05 2024-09-21T15:51:26.385 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:26.385 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:26.385 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:26.385 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:49:56.108551+0000", 2024-09-21T15:51:26.385 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0722000000000005e-05 2024-09-21T15:51:26.385 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:26.385 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:26.385 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:26.386 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:26.386 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:26.386 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:26.387 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_ops_in_flight 2024-09-21T15:51:26.497 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:51:26.497 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:51:26.497 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:51:26.497 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:51:26.506 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_blocked_ops 2024-09-21T15:51:26.608 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:51:26.608 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:51:26.609 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:51:26.609 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:51:26.609 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:51:26.617 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.4.asok dump_historic_ops 2024-09-21T15:51:26.709 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset noscrub 2024-09-21T15:51:26.721 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:51:26.721 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:51:26.721 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:51:26.721 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:51:26.721 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.722 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155031 3.1c 3.8612741c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:26.722 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448748+0000", 2024-09-21T15:51:26.722 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 64.271039998999996, 2024-09-21T15:51:26.722 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064010286999999999, 2024-09-21T15:51:26.722 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:26.722 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:26.722 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:26.722 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:26.722 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:26.722 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155031 2024-09-21T15:51:26.722 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.722 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:26.722 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.723 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:26.723 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448748+0000", 2024-09-21T15:51:26.723 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:26.723 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.723 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.723 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:26.723 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448748+0000", 2024-09-21T15:51:26.723 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:26.723 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.723 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.723 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:26.723 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448747+0000", 2024-09-21T15:51:26.723 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:26.723 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.723 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.724 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:26.724 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448753+0000", 2024-09-21T15:51:26.724 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.755e-06 2024-09-21T15:51:26.724 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.724 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.724 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:26.724 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448763+0000", 2024-09-21T15:51:26.724 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7659999999999993e-06 2024-09-21T15:51:26.724 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.724 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.724 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:26.724 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448769+0000", 2024-09-21T15:51:26.724 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6060000000000002e-06 2024-09-21T15:51:26.724 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.724 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.724 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:26.725 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512750+0000", 2024-09-21T15:51:26.725 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.063981298000000006 2024-09-21T15:51:26.725 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.725 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.725 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:26.725 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512758+0000", 2024-09-21T15:51:26.725 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1669999999999999e-06 2024-09-21T15:51:26.725 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:26.725 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:26.725 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:26.725 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.725 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.725 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155091 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:26.725 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448916+0000", 2024-09-21T15:51:26.726 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 64.270872154000003, 2024-09-21T15:51:26.726 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064006309999999997, 2024-09-21T15:51:26.726 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:26.726 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:26.726 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:26.726 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:26.726 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:26.726 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155091 2024-09-21T15:51:26.726 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.726 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:26.726 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.726 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:26.726 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448916+0000", 2024-09-21T15:51:26.726 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:26.727 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.727 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.727 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:26.727 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448916+0000", 2024-09-21T15:51:26.727 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:26.727 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.727 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.727 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:26.727 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448915+0000", 2024-09-21T15:51:26.727 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:26.727 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.727 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.727 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:26.727 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448917+0000", 2024-09-21T15:51:26.728 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2400000000000002e-06 2024-09-21T15:51:26.728 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.728 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.728 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:26.728 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448923+0000", 2024-09-21T15:51:26.728 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0580000000000003e-06 2024-09-21T15:51:26.728 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.728 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.728 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:26.728 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448927+0000", 2024-09-21T15:51:26.728 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.031e-06 2024-09-21T15:51:26.728 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.728 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.728 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:26.728 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512916+0000", 2024-09-21T15:51:26.729 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.063988456999999999 2024-09-21T15:51:26.729 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.729 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.729 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:26.729 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512922+0000", 2024-09-21T15:51:26.729 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9009999999999999e-06 2024-09-21T15:51:26.729 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:26.729 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:26.729 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:26.729 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.729 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.729 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155092 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:26.729 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448931+0000", 2024-09-21T15:51:26.729 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 64.270856371999997, 2024-09-21T15:51:26.730 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064022793999999994, 2024-09-21T15:51:26.730 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:26.730 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:26.730 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:26.730 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:26.730 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:26.730 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155092 2024-09-21T15:51:26.730 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.730 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:26.730 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.730 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:26.730 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448931+0000", 2024-09-21T15:51:26.730 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:26.730 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.730 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.731 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:26.731 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448931+0000", 2024-09-21T15:51:26.731 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:26.731 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.731 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.731 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:26.731 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448931+0000", 2024-09-21T15:51:26.731 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:26.731 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.731 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.731 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:26.731 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448933+0000", 2024-09-21T15:51:26.731 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1799999999999999e-06 2024-09-21T15:51:26.731 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.731 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.732 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:26.732 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448939+0000", 2024-09-21T15:51:26.732 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.604e-06 2024-09-21T15:51:26.732 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.732 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.732 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:26.732 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448943+0000", 2024-09-21T15:51:26.732 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0740000000000003e-06 2024-09-21T15:51:26.732 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.732 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.732 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:26.732 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512946+0000", 2024-09-21T15:51:26.732 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064003612000000001 2024-09-21T15:51:26.732 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.732 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.733 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:26.733 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512954+0000", 2024-09-21T15:51:26.733 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6690000000000006e-06 2024-09-21T15:51:26.733 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:26.733 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:26.733 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:26.733 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.733 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.733 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155093 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:26.733 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448947+0000", 2024-09-21T15:51:26.733 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 64.270840997999997, 2024-09-21T15:51:26.733 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064020717000000005, 2024-09-21T15:51:26.733 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:26.733 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:26.733 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:26.734 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:26.734 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:26.734 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155093 2024-09-21T15:51:26.734 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.734 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:26.734 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.734 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:26.734 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448947+0000", 2024-09-21T15:51:26.734 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:26.734 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.734 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.734 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:26.734 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448947+0000", 2024-09-21T15:51:26.734 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:26.735 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.735 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.735 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:26.735 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448946+0000", 2024-09-21T15:51:26.735 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:26.735 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.735 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.735 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:26.735 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448949+0000", 2024-09-21T15:51:26.735 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3860000000000001e-06 2024-09-21T15:51:26.735 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.735 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.735 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:26.735 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448955+0000", 2024-09-21T15:51:26.736 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.823e-06 2024-09-21T15:51:26.736 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.736 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.736 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:26.736 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448959+0000", 2024-09-21T15:51:26.736 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2799999999999997e-06 2024-09-21T15:51:26.736 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.736 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.736 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:26.736 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512964+0000", 2024-09-21T15:51:26.736 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064004741000000004 2024-09-21T15:51:26.736 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.736 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.737 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:26.737 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512967+0000", 2024-09-21T15:51:26.737 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.839e-06 2024-09-21T15:51:26.737 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:26.737 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:26.737 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:26.737 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.737 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.737 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155094 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:26.737 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448962+0000", 2024-09-21T15:51:26.737 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 64.270825375000001, 2024-09-21T15:51:26.737 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064037378000000006, 2024-09-21T15:51:26.737 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:26.737 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:26.738 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:26.738 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:26.738 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:26.738 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155094 2024-09-21T15:51:26.738 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.738 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:26.738 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.738 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:26.738 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448962+0000", 2024-09-21T15:51:26.738 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:26.738 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.738 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.738 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:26.738 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448962+0000", 2024-09-21T15:51:26.739 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:26.739 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.739 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.739 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:26.739 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448962+0000", 2024-09-21T15:51:26.739 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:26.739 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.739 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.739 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:26.739 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448968+0000", 2024-09-21T15:51:26.739 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9710000000000001e-06 2024-09-21T15:51:26.739 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.739 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.739 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:26.740 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448974+0000", 2024-09-21T15:51:26.740 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7230000000000001e-06 2024-09-21T15:51:26.740 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.740 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.740 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:26.740 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448978+0000", 2024-09-21T15:51:26.740 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3749999999999996e-06 2024-09-21T15:51:26.740 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.740 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.740 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:26.740 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.512992+0000", 2024-09-21T15:51:26.740 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064013525000000002 2024-09-21T15:51:26.740 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.740 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.741 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:26.741 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513000+0000", 2024-09-21T15:51:26.741 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0830000000000004e-06 2024-09-21T15:51:26.741 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:26.741 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:26.741 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:26.741 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.741 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.741 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155095 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:26.741 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448982+0000", 2024-09-21T15:51:26.741 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 64.270806093000004, 2024-09-21T15:51:26.741 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064041904999999996, 2024-09-21T15:51:26.741 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:26.741 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:26.741 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:26.742 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:26.742 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:26.742 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155095 2024-09-21T15:51:26.742 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.742 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:26.742 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.742 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:26.742 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448982+0000", 2024-09-21T15:51:26.742 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:26.742 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.742 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.742 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:26.742 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448982+0000", 2024-09-21T15:51:26.742 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:26.743 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.743 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.743 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:26.743 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448981+0000", 2024-09-21T15:51:26.743 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:26.743 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.743 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.743 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:26.743 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448983+0000", 2024-09-21T15:51:26.743 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.905e-06 2024-09-21T15:51:26.743 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.743 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.743 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:26.744 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448989+0000", 2024-09-21T15:51:26.744 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.536e-06 2024-09-21T15:51:26.744 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.744 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.744 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:26.744 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448993+0000", 2024-09-21T15:51:26.744 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2390000000000004e-06 2024-09-21T15:51:26.744 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.744 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.744 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:26.744 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513015+0000", 2024-09-21T15:51:26.744 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064022081999999994 2024-09-21T15:51:26.744 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.744 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.744 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:26.745 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513023+0000", 2024-09-21T15:51:26.745 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4500000000000004e-06 2024-09-21T15:51:26.745 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:26.745 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:26.745 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:26.745 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.745 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.745 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155096 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:26.745 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.448996+0000", 2024-09-21T15:51:26.745 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 64.270791192000004, 2024-09-21T15:51:26.745 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064060868000000007, 2024-09-21T15:51:26.745 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:26.745 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:26.745 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:26.745 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:26.746 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:26.746 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155096 2024-09-21T15:51:26.746 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.746 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:26.746 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.746 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:26.746 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448996+0000", 2024-09-21T15:51:26.746 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:26.746 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.746 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.746 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:26.746 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448996+0000", 2024-09-21T15:51:26.746 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:26.746 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.746 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.747 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:26.747 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448996+0000", 2024-09-21T15:51:26.747 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:26.747 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.747 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.747 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:26.747 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.448998+0000", 2024-09-21T15:51:26.747 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.8780000000000001e-06 2024-09-21T15:51:26.747 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.747 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.747 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:26.747 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449003+0000", 2024-09-21T15:51:26.747 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4099999999999999e-06 2024-09-21T15:51:26.747 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.747 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.748 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:26.748 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449008+0000", 2024-09-21T15:51:26.748 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.335e-06 2024-09-21T15:51:26.748 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.748 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.748 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:26.748 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513049+0000", 2024-09-21T15:51:26.748 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064041059999999997 2024-09-21T15:51:26.748 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.748 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.748 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:26.748 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513057+0000", 2024-09-21T15:51:26.748 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5050000000000007e-06 2024-09-21T15:51:26.748 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:26.748 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:26.748 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:26.749 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.749 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.749 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155097 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:26.749 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449019+0000", 2024-09-21T15:51:26.749 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 64.270768770999993, 2024-09-21T15:51:26.749 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064052488000000005, 2024-09-21T15:51:26.749 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:26.749 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:26.749 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:26.749 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:26.749 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:26.749 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155097 2024-09-21T15:51:26.749 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.749 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:26.749 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.749 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:26.750 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449019+0000", 2024-09-21T15:51:26.750 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:26.750 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.750 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.750 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:26.750 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449019+0000", 2024-09-21T15:51:26.750 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:26.750 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.750 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.750 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:26.750 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449019+0000", 2024-09-21T15:51:26.750 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:26.750 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.750 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.751 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:26.751 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449021+0000", 2024-09-21T15:51:26.751 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.244e-06 2024-09-21T15:51:26.751 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.751 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.751 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:26.751 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449027+0000", 2024-09-21T15:51:26.751 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8619999999999999e-06 2024-09-21T15:51:26.751 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.751 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.751 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:26.751 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449031+0000", 2024-09-21T15:51:26.751 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1359999999999999e-06 2024-09-21T15:51:26.751 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.752 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.752 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:26.752 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513067+0000", 2024-09-21T15:51:26.752 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064036391999999998 2024-09-21T15:51:26.752 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.752 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.752 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:26.752 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513071+0000", 2024-09-21T15:51:26.752 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1989999999999999e-06 2024-09-21T15:51:26.752 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:26.752 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:26.752 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:26.752 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.752 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.752 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155098 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:26.752 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449035+0000", 2024-09-21T15:51:26.753 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 64.270753174000006, 2024-09-21T15:51:26.753 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064071761000000005, 2024-09-21T15:51:26.753 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:26.753 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:26.753 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:26.753 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:26.753 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:26.753 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155098 2024-09-21T15:51:26.753 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.753 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:26.753 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.753 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:26.753 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449035+0000", 2024-09-21T15:51:26.753 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:26.753 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.753 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.754 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:26.754 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449035+0000", 2024-09-21T15:51:26.754 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:26.754 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.754 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.754 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:26.754 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449034+0000", 2024-09-21T15:51:26.754 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:26.754 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.754 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.754 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:26.754 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449036+0000", 2024-09-21T15:51:26.754 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9989999999999998e-06 2024-09-21T15:51:26.754 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.754 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.755 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:26.755 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449042+0000", 2024-09-21T15:51:26.755 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6010000000000002e-06 2024-09-21T15:51:26.755 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.755 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.755 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:26.755 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449047+0000", 2024-09-21T15:51:26.755 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3600000000000004e-06 2024-09-21T15:51:26.755 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.755 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.755 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:26.755 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513098+0000", 2024-09-21T15:51:26.755 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064050544000000001 2024-09-21T15:51:26.755 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.755 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.756 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:26.756 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513106+0000", 2024-09-21T15:51:26.756 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5650000000000002e-06 2024-09-21T15:51:26.756 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:26.756 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:26.756 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:26.756 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.756 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.756 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155099 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:26.756 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449051+0000", 2024-09-21T15:51:26.756 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 64.270736632999999, 2024-09-21T15:51:26.756 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064069388000000005, 2024-09-21T15:51:26.756 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:26.756 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:26.756 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:26.757 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:26.757 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:26.757 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155099 2024-09-21T15:51:26.757 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.757 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:26.757 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.757 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:26.757 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449051+0000", 2024-09-21T15:51:26.757 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:26.757 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.757 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.757 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:26.757 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449051+0000", 2024-09-21T15:51:26.757 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:26.757 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.757 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.758 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:26.758 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449051+0000", 2024-09-21T15:51:26.758 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:26.758 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.758 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.758 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:26.758 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449053+0000", 2024-09-21T15:51:26.758 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2620000000000001e-06 2024-09-21T15:51:26.758 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.758 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.758 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:26.758 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449058+0000", 2024-09-21T15:51:26.758 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1680000000000003e-06 2024-09-21T15:51:26.758 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.758 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.759 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:26.759 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449062+0000", 2024-09-21T15:51:26.759 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9979999999999997e-06 2024-09-21T15:51:26.759 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.759 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.759 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:26.759 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513116+0000", 2024-09-21T15:51:26.759 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064053761000000001 2024-09-21T15:51:26.759 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.759 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.759 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:26.759 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513120+0000", 2024-09-21T15:51:26.759 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5399999999999997e-06 2024-09-21T15:51:26.759 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:26.759 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:26.759 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:26.760 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.760 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.760 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155100 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:26.760 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449101+0000", 2024-09-21T15:51:26.760 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 64.270686878000006, 2024-09-21T15:51:26.760 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064079585999999994, 2024-09-21T15:51:26.760 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:26.760 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:26.760 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:26.760 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:26.760 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:26.760 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155100 2024-09-21T15:51:26.760 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.760 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:26.760 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.760 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:26.761 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449101+0000", 2024-09-21T15:51:26.761 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:26.761 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.761 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.761 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:26.761 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449101+0000", 2024-09-21T15:51:26.761 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:26.761 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.761 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.761 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:26.761 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449100+0000", 2024-09-21T15:51:26.761 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-21T15:51:26.761 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.761 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.761 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:26.761 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449105+0000", 2024-09-21T15:51:26.762 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6700000000000002e-06 2024-09-21T15:51:26.762 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.762 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.762 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:26.762 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449117+0000", 2024-09-21T15:51:26.762 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1841000000000001e-05 2024-09-21T15:51:26.762 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.762 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.762 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:26.762 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449124+0000", 2024-09-21T15:51:26.762 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4660000000000002e-06 2024-09-21T15:51:26.762 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.762 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.762 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:26.762 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513163+0000", 2024-09-21T15:51:26.763 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064039053999999998 2024-09-21T15:51:26.763 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.763 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.763 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:26.763 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513180+0000", 2024-09-21T15:51:26.763 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.7333000000000002e-05 2024-09-21T15:51:26.763 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:26.763 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:26.763 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:26.763 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.763 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.763 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155101 3.14 3.162cc2b4 (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:26.763 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449166+0000", 2024-09-21T15:51:26.763 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 64.270622173000007, 2024-09-21T15:51:26.763 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064027639999999997, 2024-09-21T15:51:26.764 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:26.764 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:26.764 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:26.764 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:26.764 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:26.764 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155101 2024-09-21T15:51:26.764 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.764 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:26.764 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.764 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:26.764 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449166+0000", 2024-09-21T15:51:26.764 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:26.764 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.764 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.764 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:26.764 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449166+0000", 2024-09-21T15:51:26.765 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:26.765 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.765 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.765 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:26.765 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449165+0000", 2024-09-21T15:51:26.765 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:26.765 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.765 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.765 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:26.765 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449168+0000", 2024-09-21T15:51:26.765 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5099999999999999e-06 2024-09-21T15:51:26.765 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.765 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.765 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:26.765 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449177+0000", 2024-09-21T15:51:26.766 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6659999999999997e-06 2024-09-21T15:51:26.766 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.766 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.766 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:26.766 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449182+0000", 2024-09-21T15:51:26.766 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1320000000000002e-06 2024-09-21T15:51:26.766 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.766 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.766 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:26.766 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513189+0000", 2024-09-21T15:51:26.766 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064007231999999997 2024-09-21T15:51:26.766 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.766 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.766 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:26.767 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513193+0000", 2024-09-21T15:51:26.767 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.692e-06 2024-09-21T15:51:26.767 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:26.767 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:26.767 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:26.767 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.767 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.767 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155134 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:26.767 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449187+0000", 2024-09-21T15:51:26.767 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 64.270600584999997, 2024-09-21T15:51:26.767 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064031331999999996, 2024-09-21T15:51:26.767 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:26.767 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:26.767 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:26.767 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:26.768 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:26.768 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155134 2024-09-21T15:51:26.768 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.768 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:26.768 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.768 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:26.768 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449187+0000", 2024-09-21T15:51:26.768 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:26.768 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.768 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.768 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:26.768 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449187+0000", 2024-09-21T15:51:26.768 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:26.768 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.768 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.769 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:26.769 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449187+0000", 2024-09-21T15:51:26.769 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:26.769 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.769 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.769 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:26.769 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449189+0000", 2024-09-21T15:51:26.769 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5469999999999999e-06 2024-09-21T15:51:26.769 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.769 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.769 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:26.769 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449196+0000", 2024-09-21T15:51:26.769 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.7009999999999999e-06 2024-09-21T15:51:26.769 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.769 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.769 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:26.770 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449202+0000", 2024-09-21T15:51:26.770 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7409999999999998e-06 2024-09-21T15:51:26.770 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.770 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.770 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:26.770 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513214+0000", 2024-09-21T15:51:26.770 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064012301999999993 2024-09-21T15:51:26.770 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.770 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.770 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:26.770 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513218+0000", 2024-09-21T15:51:26.770 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5709999999999999e-06 2024-09-21T15:51:26.770 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:26.770 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:26.771 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:26.771 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.771 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.771 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155135 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:26.771 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449207+0000", 2024-09-21T15:51:26.771 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 64.270581074999996, 2024-09-21T15:51:26.771 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064029249999999996, 2024-09-21T15:51:26.771 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:26.771 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:26.771 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:26.771 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:26.771 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:26.771 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155135 2024-09-21T15:51:26.771 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.772 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:26.772 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.772 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:26.772 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449207+0000", 2024-09-21T15:51:26.772 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:26.772 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.772 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.772 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:26.772 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449207+0000", 2024-09-21T15:51:26.772 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:26.772 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.772 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.772 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:26.772 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449206+0000", 2024-09-21T15:51:26.773 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:26.773 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.773 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.773 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:26.773 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449209+0000", 2024-09-21T15:51:26.773 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5150000000000001e-06 2024-09-21T15:51:26.773 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.773 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.773 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:26.773 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449215+0000", 2024-09-21T15:51:26.773 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9329999999999998e-06 2024-09-21T15:51:26.773 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.773 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.773 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:26.774 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449220+0000", 2024-09-21T15:51:26.774 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0819999999999998e-06 2024-09-21T15:51:26.774 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.774 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.774 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:26.774 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513232+0000", 2024-09-21T15:51:26.774 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064012052 2024-09-21T15:51:26.774 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.774 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.774 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:26.774 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513236+0000", 2024-09-21T15:51:26.774 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2080000000000002e-06 2024-09-21T15:51:26.774 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:26.774 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:26.775 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:26.775 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.775 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.775 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155136 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:26.775 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449224+0000", 2024-09-21T15:51:26.775 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 64.270563511000006, 2024-09-21T15:51:26.775 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064053025999999999, 2024-09-21T15:51:26.775 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:26.775 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:26.775 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:26.775 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:26.775 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:26.775 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155136 2024-09-21T15:51:26.775 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.776 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:26.776 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.776 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:26.776 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449224+0000", 2024-09-21T15:51:26.776 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:26.776 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.776 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.776 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:26.776 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449224+0000", 2024-09-21T15:51:26.776 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:26.776 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.776 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.776 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:26.776 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449224+0000", 2024-09-21T15:51:26.777 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:26.777 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.777 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.777 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:26.777 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449226+0000", 2024-09-21T15:51:26.777 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2699999999999999e-06 2024-09-21T15:51:26.777 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.777 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.777 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:26.777 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449233+0000", 2024-09-21T15:51:26.777 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4289999999999998e-06 2024-09-21T15:51:26.777 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.777 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.777 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:26.778 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449238+0000", 2024-09-21T15:51:26.778 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0730000000000004e-06 2024-09-21T15:51:26.778 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.778 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.778 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:26.778 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513269+0000", 2024-09-21T15:51:26.778 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064030999000000005 2024-09-21T15:51:26.778 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.778 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.778 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:26.778 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513277+0000", 2024-09-21T15:51:26.778 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5520000000000005e-06 2024-09-21T15:51:26.778 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:26.778 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:26.779 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:26.779 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.779 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.779 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155137 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:26.779 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449263+0000", 2024-09-21T15:51:26.779 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 64.270524378999994, 2024-09-21T15:51:26.779 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.06402803, 2024-09-21T15:51:26.779 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:26.779 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:26.779 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:26.779 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:26.779 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:26.779 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155137 2024-09-21T15:51:26.780 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.780 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:26.780 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.780 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:26.780 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449263+0000", 2024-09-21T15:51:26.780 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:26.780 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.780 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.780 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:26.780 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449263+0000", 2024-09-21T15:51:26.780 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:26.780 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.780 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.781 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:26.781 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449262+0000", 2024-09-21T15:51:26.781 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999999 2024-09-21T15:51:26.781 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.781 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.781 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:26.781 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449267+0000", 2024-09-21T15:51:26.781 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.933e-06 2024-09-21T15:51:26.781 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.781 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.781 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:26.781 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449279+0000", 2024-09-21T15:51:26.781 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.181e-05 2024-09-21T15:51:26.781 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.781 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.781 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:26.782 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449286+0000", 2024-09-21T15:51:26.782 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4749999999999998e-06 2024-09-21T15:51:26.782 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.782 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.782 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:26.782 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513287+0000", 2024-09-21T15:51:26.782 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064001369000000002 2024-09-21T15:51:26.782 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.782 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.782 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:26.782 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513291+0000", 2024-09-21T15:51:26.782 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3000000000000003e-06 2024-09-21T15:51:26.782 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:26.782 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:26.783 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:26.783 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.783 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.783 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155138 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:26.783 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449299+0000", 2024-09-21T15:51:26.783 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 64.270488886999999, 2024-09-21T15:51:26.783 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064028359000000007, 2024-09-21T15:51:26.783 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:26.783 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:26.783 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:26.783 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:26.783 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:26.783 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155138 2024-09-21T15:51:26.783 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.783 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:26.784 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.784 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:26.784 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449299+0000", 2024-09-21T15:51:26.784 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:26.784 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.784 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.784 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:26.784 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449299+0000", 2024-09-21T15:51:26.784 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:26.784 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.784 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.784 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:26.784 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449298+0000", 2024-09-21T15:51:26.784 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:26.785 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.785 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.785 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:26.785 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449301+0000", 2024-09-21T15:51:26.785 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.0230000000000001e-06 2024-09-21T15:51:26.785 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.785 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.785 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:26.785 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449308+0000", 2024-09-21T15:51:26.785 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3119999999999999e-06 2024-09-21T15:51:26.785 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.785 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.785 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:26.785 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449313+0000", 2024-09-21T15:51:26.786 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8980000000000004e-06 2024-09-21T15:51:26.786 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.786 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.786 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:26.786 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513318+0000", 2024-09-21T15:51:26.786 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064005612000000003 2024-09-21T15:51:26.786 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.786 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.786 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:26.786 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513327+0000", 2024-09-21T15:51:26.786 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0100000000000001e-06 2024-09-21T15:51:26.786 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:26.786 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:26.786 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:26.786 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.787 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.787 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155142 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:26.787 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449392+0000", 2024-09-21T15:51:26.787 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 64.270396129000005, 2024-09-21T15:51:26.787 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064012464000000005, 2024-09-21T15:51:26.787 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:26.787 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:26.787 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:26.787 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:26.787 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:26.787 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155142 2024-09-21T15:51:26.787 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.787 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:26.787 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.787 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:26.788 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449392+0000", 2024-09-21T15:51:26.788 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:26.788 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.788 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.788 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:26.788 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449392+0000", 2024-09-21T15:51:26.788 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:26.788 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.788 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.788 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:26.788 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449391+0000", 2024-09-21T15:51:26.788 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:26.788 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.788 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.788 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:26.788 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449393+0000", 2024-09-21T15:51:26.789 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.26e-06 2024-09-21T15:51:26.789 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.789 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.789 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:26.789 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449399+0000", 2024-09-21T15:51:26.789 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6269999999999997e-06 2024-09-21T15:51:26.789 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.789 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.789 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:26.789 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449403+0000", 2024-09-21T15:51:26.789 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3000000000000003e-06 2024-09-21T15:51:26.789 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.789 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.789 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:26.789 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513396+0000", 2024-09-21T15:51:26.790 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.063992568999999999 2024-09-21T15:51:26.790 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.790 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.790 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:26.790 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513404+0000", 2024-09-21T15:51:26.790 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1100000000000003e-06 2024-09-21T15:51:26.790 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:26.790 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:26.790 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:26.790 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.790 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.790 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155143 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:26.790 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449407+0000", 2024-09-21T15:51:26.790 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 64.270380543000002, 2024-09-21T15:51:26.790 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064009916, 2024-09-21T15:51:26.791 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:26.791 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:26.791 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:26.791 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:26.791 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:26.791 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155143 2024-09-21T15:51:26.791 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.791 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:26.791 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.791 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:26.791 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449407+0000", 2024-09-21T15:51:26.791 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:26.791 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.791 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.791 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:26.792 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449407+0000", 2024-09-21T15:51:26.792 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:26.792 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.792 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.792 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:26.792 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449407+0000", 2024-09-21T15:51:26.792 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:26.792 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.792 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.792 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:26.792 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449409+0000", 2024-09-21T15:51:26.792 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1090000000000001e-06 2024-09-21T15:51:26.792 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.792 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.792 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:26.792 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449413+0000", 2024-09-21T15:51:26.793 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.6650000000000001e-06 2024-09-21T15:51:26.793 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.793 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.793 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:26.793 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449415+0000", 2024-09-21T15:51:26.793 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5459999999999998e-06 2024-09-21T15:51:26.793 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.793 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.793 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:26.793 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513413+0000", 2024-09-21T15:51:26.793 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.063998037999999993 2024-09-21T15:51:26.793 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.793 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.793 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:26.793 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513417+0000", 2024-09-21T15:51:26.794 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.8789999999999996e-06 2024-09-21T15:51:26.794 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:26.794 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:26.794 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:26.794 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.794 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.794 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155144 3.1c 3.631b237c (undecoded) ondisk+retry+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:26.794 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.449427+0000", 2024-09-21T15:51:26.794 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 64.270360534999995, 2024-09-21T15:51:26.794 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.064022531999999993, 2024-09-21T15:51:26.794 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:26.794 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "reached pg", 2024-09-21T15:51:26.794 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:26.794 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:26.794 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:26.795 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155144 2024-09-21T15:51:26.795 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.795 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:26.795 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.795 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:26.795 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449427+0000", 2024-09-21T15:51:26.795 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:26.795 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.795 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.795 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:26.795 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449427+0000", 2024-09-21T15:51:26.795 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:26.795 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.795 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.795 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:26.796 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449427+0000", 2024-09-21T15:51:26.796 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:26.796 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.796 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.796 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:26.796 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449430+0000", 2024-09-21T15:51:26.796 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.3579999999999999e-06 2024-09-21T15:51:26.796 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.796 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.796 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:26.796 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449437+0000", 2024-09-21T15:51:26.796 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4580000000000004e-06 2024-09-21T15:51:26.796 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.796 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.797 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:26.797 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.449443+0000", 2024-09-21T15:51:26.797 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5609999999999998e-06 2024-09-21T15:51:26.797 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.797 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.797 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:26.797 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513441+0000", 2024-09-21T15:51:26.797 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.063998136999999997 2024-09-21T15:51:26.797 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:26.797 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:26.797 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:26.797 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.513450+0000", 2024-09-21T15:51:26.797 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4950000000000008e-06 2024-09-21T15:51:26.797 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:26.797 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:26.797 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:26.798 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:26.798 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:26.798 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:51:26.799 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight 2024-09-21T15:51:26.901 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:51:26.901 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:51:26.901 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:51:26.901 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:51:26.909 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_blocked_ops 2024-09-21T15:51:27.011 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:51:27.011 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:51:27.011 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:51:27.011 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:51:27.011 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:51:27.020 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_historic_ops 2024-09-21T15:51:27.123 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:51:27.124 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:51:27.124 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:51:27.124 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:51:27.124 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.124 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120111 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 321536~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:27.124 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572216+0000", 2024-09-21T15:51:27.124 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 83.549890809999994, 2024-09-21T15:51:27.124 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054657399000000002, 2024-09-21T15:51:27.124 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:27.124 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:27.124 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:27.124 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:27.125 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:27.125 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120111 2024-09-21T15:51:27.125 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.125 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:27.125 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.125 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:27.125 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572216+0000", 2024-09-21T15:51:27.125 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.125 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.125 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.125 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:27.125 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572216+0000", 2024-09-21T15:51:27.125 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.125 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.126 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.126 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:27.126 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572216+0000", 2024-09-21T15:51:27.126 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:27.126 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.126 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.126 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:27.126 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572218+0000", 2024-09-21T15:51:27.126 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1840000000000002e-06 2024-09-21T15:51:27.126 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.126 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.126 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:27.126 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572224+0000", 2024-09-21T15:51:27.126 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7880000000000003e-06 2024-09-21T15:51:27.126 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.127 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.127 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:27.127 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572229+0000", 2024-09-21T15:51:27.127 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8239999999999999e-06 2024-09-21T15:51:27.127 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.127 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.127 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.127 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626680+0000", 2024-09-21T15:51:27.127 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054451431000000002 2024-09-21T15:51:27.127 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.127 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.127 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:27.127 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626755+0000", 2024-09-21T15:51:27.127 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4641000000000006e-05 2024-09-21T15:51:27.127 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.128 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.128 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:27.128 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626874+0000", 2024-09-21T15:51:27.128 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00011886 2024-09-21T15:51:27.128 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.128 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.128 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.128 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.128 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.128 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120112 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 342016~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:27.128 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572233+0000", 2024-09-21T15:51:27.128 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 83.549874442000004, 2024-09-21T15:51:27.128 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054823997999999999, 2024-09-21T15:51:27.129 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:27.129 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:27.129 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:27.129 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:27.129 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:27.129 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120112 2024-09-21T15:51:27.129 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.129 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:27.129 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.129 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:27.129 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572233+0000", 2024-09-21T15:51:27.129 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.129 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.129 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.129 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:27.130 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572233+0000", 2024-09-21T15:51:27.130 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.130 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.130 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.130 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:27.130 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572232+0000", 2024-09-21T15:51:27.130 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:27.130 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.130 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.130 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:27.130 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572234+0000", 2024-09-21T15:51:27.130 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0660000000000002e-06 2024-09-21T15:51:27.130 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.130 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.130 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:27.131 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572242+0000", 2024-09-21T15:51:27.131 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0430000000000002e-06 2024-09-21T15:51:27.131 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.131 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.131 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:27.131 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572246+0000", 2024-09-21T15:51:27.131 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9080000000000003e-06 2024-09-21T15:51:27.131 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.131 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.131 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.131 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626916+0000", 2024-09-21T15:51:27.131 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054669482999999998 2024-09-21T15:51:27.131 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.131 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.131 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:27.132 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.626977+0000", 2024-09-21T15:51:27.132 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0597000000000003e-05 2024-09-21T15:51:27.132 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.132 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.132 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:27.132 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627057+0000", 2024-09-21T15:51:27.132 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0258000000000003e-05 2024-09-21T15:51:27.132 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.132 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.132 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.132 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.132 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.132 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120113 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 362496~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:27.132 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572253+0000", 2024-09-21T15:51:27.133 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 83.549854456999995, 2024-09-21T15:51:27.133 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.05503044, 2024-09-21T15:51:27.133 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:27.133 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:27.133 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:27.133 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:27.133 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:27.133 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120113 2024-09-21T15:51:27.133 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.133 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:27.133 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.133 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:27.133 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572253+0000", 2024-09-21T15:51:27.133 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.133 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.134 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.134 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:27.134 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572253+0000", 2024-09-21T15:51:27.134 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.134 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.134 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.134 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:27.134 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572252+0000", 2024-09-21T15:51:27.134 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:27.134 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.134 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.134 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:27.134 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572258+0000", 2024-09-21T15:51:27.135 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6169999999999998e-06 2024-09-21T15:51:27.135 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.135 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.135 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:27.135 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572266+0000", 2024-09-21T15:51:27.135 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9410000000000007e-06 2024-09-21T15:51:27.135 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.135 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.135 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:27.135 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572272+0000", 2024-09-21T15:51:27.135 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.241e-06 2024-09-21T15:51:27.135 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.135 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.136 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.136 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627087+0000", 2024-09-21T15:51:27.136 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054814850999999998 2024-09-21T15:51:27.136 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.136 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.136 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:27.136 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627162+0000", 2024-09-21T15:51:27.136 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5012999999999995e-05 2024-09-21T15:51:27.136 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.136 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.136 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:27.136 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627283+0000", 2024-09-21T15:51:27.136 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012113 2024-09-21T15:51:27.136 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.136 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.136 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.137 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.137 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.137 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120114 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 382976~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:27.137 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572277+0000", 2024-09-21T15:51:27.137 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 83.549830439000004, 2024-09-21T15:51:27.137 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055184395999999997, 2024-09-21T15:51:27.137 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:27.137 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:27.137 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:27.137 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:27.137 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:27.137 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120114 2024-09-21T15:51:27.137 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.137 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:27.137 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.138 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:27.138 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572277+0000", 2024-09-21T15:51:27.138 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.138 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.138 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.138 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:27.138 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572277+0000", 2024-09-21T15:51:27.138 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.138 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.138 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.138 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:27.138 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572276+0000", 2024-09-21T15:51:27.138 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:27.138 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.139 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.139 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:27.139 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572279+0000", 2024-09-21T15:51:27.139 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.373e-06 2024-09-21T15:51:27.139 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.139 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.139 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:27.139 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572285+0000", 2024-09-21T15:51:27.139 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9359999999999996e-06 2024-09-21T15:51:27.139 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.139 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.139 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:27.139 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572289+0000", 2024-09-21T15:51:27.139 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.634e-06 2024-09-21T15:51:27.140 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.140 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.140 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.140 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627311+0000", 2024-09-21T15:51:27.140 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055021211 2024-09-21T15:51:27.140 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.140 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.140 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:27.140 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627360+0000", 2024-09-21T15:51:27.140 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9203999999999998e-05 2024-09-21T15:51:27.140 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.140 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.140 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:27.140 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627461+0000", 2024-09-21T15:51:27.140 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000101469 2024-09-21T15:51:27.141 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.141 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.141 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.141 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.141 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.141 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120115 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 399360~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:27.141 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572294+0000", 2024-09-21T15:51:27.141 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 83.549813131999997, 2024-09-21T15:51:27.141 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055400092999999997, 2024-09-21T15:51:27.141 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:27.141 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:27.141 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:27.141 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:27.141 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:27.141 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120115 2024-09-21T15:51:27.141 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.142 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:27.142 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.142 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:27.142 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572294+0000", 2024-09-21T15:51:27.142 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.142 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.142 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.142 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:27.142 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572294+0000", 2024-09-21T15:51:27.142 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.142 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.142 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.142 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:27.142 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572294+0000", 2024-09-21T15:51:27.142 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:27.143 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.143 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.143 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:27.143 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572296+0000", 2024-09-21T15:51:27.143 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1100000000000001e-06 2024-09-21T15:51:27.143 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.143 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.143 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:27.143 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572303+0000", 2024-09-21T15:51:27.143 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.8580000000000002e-06 2024-09-21T15:51:27.143 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.143 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.143 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:27.143 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572314+0000", 2024-09-21T15:51:27.143 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.101e-05 2024-09-21T15:51:27.143 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.143 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.144 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.144 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627496+0000", 2024-09-21T15:51:27.144 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055182522999999997 2024-09-21T15:51:27.144 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.144 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.144 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:27.144 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627568+0000", 2024-09-21T15:51:27.144 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1569999999999994e-05 2024-09-21T15:51:27.144 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.144 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.144 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:27.144 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627694+0000", 2024-09-21T15:51:27.144 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000126372 2024-09-21T15:51:27.144 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.144 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.144 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.145 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.145 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.145 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120116 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 419840~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:27.145 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572318+0000", 2024-09-21T15:51:27.145 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 83.549788941000003, 2024-09-21T15:51:27.145 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055539416000000001, 2024-09-21T15:51:27.145 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:27.145 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:27.145 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:27.145 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:27.145 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:27.145 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120116 2024-09-21T15:51:27.145 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.145 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:27.145 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.146 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:27.146 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572318+0000", 2024-09-21T15:51:27.146 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.146 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.146 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.146 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:27.146 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572318+0000", 2024-09-21T15:51:27.146 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.146 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.146 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.146 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:27.146 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572318+0000", 2024-09-21T15:51:27.146 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:27.146 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.147 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.147 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:27.147 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572322+0000", 2024-09-21T15:51:27.147 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4839999999999998e-06 2024-09-21T15:51:27.147 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.147 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.147 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:27.147 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572329+0000", 2024-09-21T15:51:27.147 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0959999999999997e-06 2024-09-21T15:51:27.147 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.147 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.147 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:27.147 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572333+0000", 2024-09-21T15:51:27.147 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9359999999999998e-06 2024-09-21T15:51:27.147 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.147 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.148 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.148 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627726+0000", 2024-09-21T15:51:27.148 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055392829999999997 2024-09-21T15:51:27.148 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.148 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.148 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:27.148 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627779+0000", 2024-09-21T15:51:27.148 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.2185999999999998e-05 2024-09-21T15:51:27.148 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.148 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.148 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:27.148 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627858+0000", 2024-09-21T15:51:27.148 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9190999999999994e-05 2024-09-21T15:51:27.148 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.148 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.148 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.149 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.149 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.149 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120117 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 440320~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:27.149 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572338+0000", 2024-09-21T15:51:27.149 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 83.549769628000007, 2024-09-21T15:51:27.149 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055680888999999997, 2024-09-21T15:51:27.149 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:27.149 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:27.149 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:27.149 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:27.149 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:27.149 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120117 2024-09-21T15:51:27.149 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.149 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:27.149 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.150 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:27.150 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572338+0000", 2024-09-21T15:51:27.150 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.150 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.150 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.150 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:27.150 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572338+0000", 2024-09-21T15:51:27.150 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.150 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.150 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.150 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:27.150 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572337+0000", 2024-09-21T15:51:27.150 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:27.150 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.151 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.151 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:27.151 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572339+0000", 2024-09-21T15:51:27.151 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.9379999999999999e-06 2024-09-21T15:51:27.151 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.151 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.151 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:27.151 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572349+0000", 2024-09-21T15:51:27.151 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7780000000000002e-06 2024-09-21T15:51:27.151 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.151 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.151 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:27.151 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572354+0000", 2024-09-21T15:51:27.151 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8740000000000003e-06 2024-09-21T15:51:27.151 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.151 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.152 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.152 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627885+0000", 2024-09-21T15:51:27.152 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055531448999999997 2024-09-21T15:51:27.152 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.152 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.152 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:27.152 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.627935+0000", 2024-09-21T15:51:27.152 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9348999999999998e-05 2024-09-21T15:51:27.152 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.152 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.152 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:27.152 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628018+0000", 2024-09-21T15:51:27.152 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3851999999999996e-05 2024-09-21T15:51:27.152 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.152 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.152 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.153 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.153 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.153 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120118 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 460800~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:27.153 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572359+0000", 2024-09-21T15:51:27.153 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 83.549748386000005, 2024-09-21T15:51:27.153 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055951714, 2024-09-21T15:51:27.153 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:27.153 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:27.153 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:27.153 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:27.153 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:27.153 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120118 2024-09-21T15:51:27.153 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.153 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:27.153 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.153 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:27.154 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572359+0000", 2024-09-21T15:51:27.154 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.154 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.154 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.154 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:27.154 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572359+0000", 2024-09-21T15:51:27.154 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.154 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.154 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.154 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:27.154 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572358+0000", 2024-09-21T15:51:27.154 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:27.154 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.154 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.155 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:27.155 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572361+0000", 2024-09-21T15:51:27.155 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.26e-06 2024-09-21T15:51:27.155 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.155 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.155 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:27.155 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572367+0000", 2024-09-21T15:51:27.155 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3640000000000004e-06 2024-09-21T15:51:27.155 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.155 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.155 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:27.155 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572376+0000", 2024-09-21T15:51:27.155 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2310000000000002e-06 2024-09-21T15:51:27.155 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.156 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.156 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.156 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628096+0000", 2024-09-21T15:51:27.156 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055719369999999997 2024-09-21T15:51:27.156 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.156 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.156 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:27.156 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628178+0000", 2024-09-21T15:51:27.156 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2667e-05 2024-09-21T15:51:27.156 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.156 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.156 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:27.156 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628311+0000", 2024-09-21T15:51:27.156 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000132175 2024-09-21T15:51:27.156 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.157 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.157 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.157 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.157 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.157 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120119 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 481280~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:27.157 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572381+0000", 2024-09-21T15:51:27.157 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 83.549725941999995, 2024-09-21T15:51:27.157 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056088791999999998, 2024-09-21T15:51:27.157 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:27.157 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:27.157 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:27.157 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:27.157 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:27.157 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120119 2024-09-21T15:51:27.157 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.157 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:27.158 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.158 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:27.158 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572381+0000", 2024-09-21T15:51:27.158 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.158 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.158 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.158 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:27.158 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572381+0000", 2024-09-21T15:51:27.158 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.158 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.158 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.158 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:27.158 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572381+0000", 2024-09-21T15:51:27.158 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:27.158 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.159 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.159 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:27.159 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572383+0000", 2024-09-21T15:51:27.159 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0619999999999999e-06 2024-09-21T15:51:27.159 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.159 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.159 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:27.159 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572391+0000", 2024-09-21T15:51:27.159 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8769999999999993e-06 2024-09-21T15:51:27.159 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.159 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.159 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:27.159 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572399+0000", 2024-09-21T15:51:27.159 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6240000000000008e-06 2024-09-21T15:51:27.159 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.160 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.160 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.160 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628344+0000", 2024-09-21T15:51:27.160 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055944828000000002 2024-09-21T15:51:27.160 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.160 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.160 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:27.160 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628393+0000", 2024-09-21T15:51:27.160 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8653000000000003e-05 2024-09-21T15:51:27.160 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.160 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.160 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:27.160 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628470+0000", 2024-09-21T15:51:27.160 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7121999999999998e-05 2024-09-21T15:51:27.160 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.160 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.161 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.161 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.161 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.161 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120120 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 501760~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:27.161 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572418+0000", 2024-09-21T15:51:27.161 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 83.549689443999995, 2024-09-21T15:51:27.161 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056209138999999998, 2024-09-21T15:51:27.161 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:27.161 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:27.161 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:27.161 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:27.161 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:27.161 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120120 2024-09-21T15:51:27.161 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.161 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:27.161 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.162 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:27.162 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572418+0000", 2024-09-21T15:51:27.162 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.162 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.162 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.162 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:27.162 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572418+0000", 2024-09-21T15:51:27.162 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.162 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.162 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.162 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:27.162 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572417+0000", 2024-09-21T15:51:27.162 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:27.162 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.162 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.163 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:27.163 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572420+0000", 2024-09-21T15:51:27.163 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5270000000000001e-06 2024-09-21T15:51:27.163 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.163 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.163 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:27.163 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572426+0000", 2024-09-21T15:51:27.163 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3380000000000001e-06 2024-09-21T15:51:27.163 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.163 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.163 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:27.163 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572431+0000", 2024-09-21T15:51:27.163 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.865e-06 2024-09-21T15:51:27.163 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.163 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.163 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.164 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628500+0000", 2024-09-21T15:51:27.164 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056068917000000003 2024-09-21T15:51:27.164 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.164 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.164 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:27.164 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628542+0000", 2024-09-21T15:51:27.164 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.1761999999999998e-05 2024-09-21T15:51:27.164 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.164 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.164 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:27.164 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628627+0000", 2024-09-21T15:51:27.164 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5247999999999994e-05 2024-09-21T15:51:27.164 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.164 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.165 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.165 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.165 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.165 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120121 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 522240~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:27.165 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.572630+0000", 2024-09-21T15:51:27.165 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 83.549477057999994, 2024-09-21T15:51:27.165 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056207209000000001, 2024-09-21T15:51:27.165 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:27.165 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:27.165 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:27.165 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:27.165 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:27.165 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120121 2024-09-21T15:51:27.165 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.166 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:27.166 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.166 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:27.166 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572630+0000", 2024-09-21T15:51:27.166 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.166 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.166 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.166 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:27.166 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572630+0000", 2024-09-21T15:51:27.166 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.166 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.166 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.166 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:27.167 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572435+0000", 2024-09-21T15:51:27.167 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998045 2024-09-21T15:51:27.167 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.167 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.167 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:27.167 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572633+0000", 2024-09-21T15:51:27.167 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000198786 2024-09-21T15:51:27.167 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.167 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.167 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:27.167 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572641+0000", 2024-09-21T15:51:27.167 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9449999999999993e-06 2024-09-21T15:51:27.167 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.168 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.168 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:27.168 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572648+0000", 2024-09-21T15:51:27.168 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1929999999999998e-06 2024-09-21T15:51:27.168 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.168 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.168 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.168 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628669+0000", 2024-09-21T15:51:27.168 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.056021690999999998 2024-09-21T15:51:27.168 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.168 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.168 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:27.168 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628751+0000", 2024-09-21T15:51:27.168 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1710000000000002e-05 2024-09-21T15:51:27.168 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.169 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.169 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:27.169 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628837+0000", 2024-09-21T15:51:27.169 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6383000000000003e-05 2024-09-21T15:51:27.169 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.169 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.169 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.169 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.169 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.169 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120122 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 538624~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:27.169 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.573471+0000", 2024-09-21T15:51:27.169 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 83.548635813000004, 2024-09-21T15:51:27.169 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055563935000000002, 2024-09-21T15:51:27.169 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:27.170 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:27.170 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:27.170 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:27.170 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:27.170 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120122 2024-09-21T15:51:27.170 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.170 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:27.170 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.170 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:27.170 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573471+0000", 2024-09-21T15:51:27.170 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.170 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.170 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.170 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:27.171 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573471+0000", 2024-09-21T15:51:27.171 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.171 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.171 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.171 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:27.171 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.572653+0000", 2024-09-21T15:51:27.171 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9991813 2024-09-21T15:51:27.171 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.171 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.171 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:27.171 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573475+0000", 2024-09-21T15:51:27.171 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00082196500000000002 2024-09-21T15:51:27.171 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.171 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.172 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:27.172 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573483+0000", 2024-09-21T15:51:27.172 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.5939999999999994e-06 2024-09-21T15:51:27.172 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.172 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.172 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:27.172 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573489+0000", 2024-09-21T15:51:27.172 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1689999999999997e-06 2024-09-21T15:51:27.172 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.172 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.172 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.172 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628882+0000", 2024-09-21T15:51:27.172 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055392230000000001 2024-09-21T15:51:27.172 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.172 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.173 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:27.173 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.628934+0000", 2024-09-21T15:51:27.173 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1870000000000003e-05 2024-09-21T15:51:27.173 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.173 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.173 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:27.173 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629035+0000", 2024-09-21T15:51:27.173 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000101753 2024-09-21T15:51:27.173 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.173 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.173 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.173 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.173 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.173 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120123 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 559104~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:27.173 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.573495+0000", 2024-09-21T15:51:27.174 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 83.548612113999994, 2024-09-21T15:51:27.174 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055704765000000003, 2024-09-21T15:51:27.174 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:27.174 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:27.174 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:27.174 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:27.174 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:27.174 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120123 2024-09-21T15:51:27.174 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.174 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:27.174 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.174 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:27.174 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573495+0000", 2024-09-21T15:51:27.174 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.175 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.175 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.175 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:27.175 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573495+0000", 2024-09-21T15:51:27.175 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.175 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.175 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.175 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:27.175 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573495+0000", 2024-09-21T15:51:27.175 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:27.175 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.175 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.175 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:27.175 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573497+0000", 2024-09-21T15:51:27.176 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3420000000000002e-06 2024-09-21T15:51:27.176 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.176 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.176 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:27.176 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573503+0000", 2024-09-21T15:51:27.176 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.7030000000000003e-06 2024-09-21T15:51:27.176 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.176 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.176 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:27.176 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573511+0000", 2024-09-21T15:51:27.176 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0790000000000001e-06 2024-09-21T15:51:27.176 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.176 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.177 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.177 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629067+0000", 2024-09-21T15:51:27.177 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055556573999999997 2024-09-21T15:51:27.177 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.177 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.177 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:27.177 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629118+0000", 2024-09-21T15:51:27.177 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1053000000000001e-05 2024-09-21T15:51:27.177 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.177 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.177 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:27.177 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629200+0000", 2024-09-21T15:51:27.177 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.1452999999999994e-05 2024-09-21T15:51:27.177 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.178 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.178 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.178 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.178 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.178 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120124 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 579584~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:27.178 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.574014+0000", 2024-09-21T15:51:27.178 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 83.548093074999997, 2024-09-21T15:51:27.178 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055440284999999999, 2024-09-21T15:51:27.178 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:27.178 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:27.178 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:27.178 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:27.178 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:27.178 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120124 2024-09-21T15:51:27.179 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.179 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:27.179 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.179 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:27.179 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574014+0000", 2024-09-21T15:51:27.179 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.179 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.179 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.179 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:27.179 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574014+0000", 2024-09-21T15:51:27.179 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.179 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.179 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.180 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:27.180 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.573515+0000", 2024-09-21T15:51:27.180 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9995008 2024-09-21T15:51:27.180 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.180 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.180 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:27.180 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574017+0000", 2024-09-21T15:51:27.180 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00050241100000000003 2024-09-21T15:51:27.180 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.180 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.180 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:27.180 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574027+0000", 2024-09-21T15:51:27.180 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4700000000000008e-06 2024-09-21T15:51:27.180 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.180 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.180 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:27.181 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574033+0000", 2024-09-21T15:51:27.181 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1489999999999999e-06 2024-09-21T15:51:27.181 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.181 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.181 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.181 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629238+0000", 2024-09-21T15:51:27.181 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055204880999999997 2024-09-21T15:51:27.181 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.181 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.181 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:27.181 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629317+0000", 2024-09-21T15:51:27.181 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8942000000000004e-05 2024-09-21T15:51:27.181 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.181 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.181 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:27.182 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629454+0000", 2024-09-21T15:51:27.182 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013767099999999999 2024-09-21T15:51:27.182 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.182 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.182 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.182 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.182 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.182 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120125 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 600064~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:27.182 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.574482+0000", 2024-09-21T15:51:27.182 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 83.547624776000006, 2024-09-21T15:51:27.182 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.055243608, 2024-09-21T15:51:27.182 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:27.182 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:27.182 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:27.182 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:27.183 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:27.183 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120125 2024-09-21T15:51:27.183 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.183 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:27.183 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.183 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:27.183 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574482+0000", 2024-09-21T15:51:27.183 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.183 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.183 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.183 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:27.183 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574482+0000", 2024-09-21T15:51:27.183 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.183 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.184 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.184 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:27.184 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574038+0000", 2024-09-21T15:51:27.184 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9995556 2024-09-21T15:51:27.184 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.184 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.184 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:27.184 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574497+0000", 2024-09-21T15:51:27.184 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00045854400000000002 2024-09-21T15:51:27.184 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.184 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.184 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:27.184 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574509+0000", 2024-09-21T15:51:27.184 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2486e-05 2024-09-21T15:51:27.185 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.185 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.185 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:27.185 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574516+0000", 2024-09-21T15:51:27.185 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2269999999999996e-06 2024-09-21T15:51:27.185 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.185 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.185 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.185 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629497+0000", 2024-09-21T15:51:27.185 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054980785999999997 2024-09-21T15:51:27.185 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.185 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.185 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:27.185 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629580+0000", 2024-09-21T15:51:27.185 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2528000000000007e-05 2024-09-21T15:51:27.186 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.186 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.186 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:27.186 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629726+0000", 2024-09-21T15:51:27.186 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014630399999999999 2024-09-21T15:51:27.186 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.186 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.186 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.186 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.186 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.186 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120126 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 620544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:27.186 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.575315+0000", 2024-09-21T15:51:27.186 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 83.546791745999997, 2024-09-21T15:51:27.187 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054709559999999997, 2024-09-21T15:51:27.187 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:27.187 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:27.187 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:27.187 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:27.187 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:27.187 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120126 2024-09-21T15:51:27.187 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.187 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:27.187 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.187 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:27.187 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575315+0000", 2024-09-21T15:51:27.187 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.187 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.187 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.188 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:27.188 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575315+0000", 2024-09-21T15:51:27.188 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.188 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.188 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.188 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:27.188 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.574522+0000", 2024-09-21T15:51:27.188 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9992065 2024-09-21T15:51:27.188 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.188 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.188 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:27.188 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575319+0000", 2024-09-21T15:51:27.188 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00079626800000000004 2024-09-21T15:51:27.188 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.188 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.189 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:27.189 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575329+0000", 2024-09-21T15:51:27.189 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.028e-05 2024-09-21T15:51:27.189 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.189 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.189 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:27.189 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575335+0000", 2024-09-21T15:51:27.189 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9290000000000003e-06 2024-09-21T15:51:27.189 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.189 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.189 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.189 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629806+0000", 2024-09-21T15:51:27.189 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054471558000000003 2024-09-21T15:51:27.189 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.189 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.190 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:27.190 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.629885+0000", 2024-09-21T15:51:27.190 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8563e-05 2024-09-21T15:51:27.190 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.190 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.190 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:27.190 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630025+0000", 2024-09-21T15:51:27.190 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014019 2024-09-21T15:51:27.190 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.190 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.190 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.190 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.190 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.190 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120127 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 641024~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:27.191 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.575340+0000", 2024-09-21T15:51:27.191 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 83.546767039000002, 2024-09-21T15:51:27.191 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054895589000000002, 2024-09-21T15:51:27.191 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:27.191 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:27.191 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:27.191 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:27.191 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:27.191 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120127 2024-09-21T15:51:27.191 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.191 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:27.191 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.191 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:27.191 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575340+0000", 2024-09-21T15:51:27.191 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.192 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.192 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.192 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:27.192 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575340+0000", 2024-09-21T15:51:27.192 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.192 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.192 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.192 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:27.192 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575340+0000", 2024-09-21T15:51:27.192 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:27.192 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.192 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.192 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:27.192 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575342+0000", 2024-09-21T15:51:27.192 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.5600000000000001e-06 2024-09-21T15:51:27.192 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.193 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.193 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:27.193 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575351+0000", 2024-09-21T15:51:27.193 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2349999999999999e-06 2024-09-21T15:51:27.193 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.193 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.193 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:27.193 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575356+0000", 2024-09-21T15:51:27.193 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4600000000000002e-06 2024-09-21T15:51:27.193 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.193 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.193 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.193 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630053+0000", 2024-09-21T15:51:27.193 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054697190999999999 2024-09-21T15:51:27.193 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.193 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.194 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:27.194 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630128+0000", 2024-09-21T15:51:27.194 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4777000000000006e-05 2024-09-21T15:51:27.194 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.194 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.194 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:27.194 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630236+0000", 2024-09-21T15:51:27.194 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000107837 2024-09-21T15:51:27.194 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.194 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.194 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.194 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.194 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.194 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120128 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 657408~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:27.195 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.575461+0000", 2024-09-21T15:51:27.195 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 83.546646460000005, 2024-09-21T15:51:27.195 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054945121999999999, 2024-09-21T15:51:27.195 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:27.195 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:27.195 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:27.195 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:27.195 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:27.195 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120128 2024-09-21T15:51:27.195 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.195 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:27.195 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.195 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:27.195 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575461+0000", 2024-09-21T15:51:27.196 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.196 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.196 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.196 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:27.196 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575461+0000", 2024-09-21T15:51:27.196 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.196 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.196 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.196 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:27.196 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575360+0000", 2024-09-21T15:51:27.196 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998994 2024-09-21T15:51:27.196 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.196 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.196 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:27.196 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575464+0000", 2024-09-21T15:51:27.197 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000103397 2024-09-21T15:51:27.197 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.197 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.197 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:27.197 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575472+0000", 2024-09-21T15:51:27.197 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3189999999999995e-06 2024-09-21T15:51:27.197 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.197 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.197 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:27.197 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.575478+0000", 2024-09-21T15:51:27.197 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.2779999999999999e-06 2024-09-21T15:51:27.197 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.197 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.198 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.198 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630268+0000", 2024-09-21T15:51:27.198 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054789545000000002 2024-09-21T15:51:27.198 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.198 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.198 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:27.198 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630315+0000", 2024-09-21T15:51:27.198 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7244999999999999e-05 2024-09-21T15:51:27.198 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.198 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.198 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:27.198 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630406+0000", 2024-09-21T15:51:27.198 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.0811999999999997e-05 2024-09-21T15:51:27.198 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.199 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.199 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.199 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.199 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.199 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:120130 3.17 3:eacba160:::rbd_data.1133545d3fa6.0000000000000015:1e [read 698368~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T15:51:27.199 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:03.576043+0000", 2024-09-21T15:51:27.199 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 83.546064544999993, 2024-09-21T15:51:27.199 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054781798, 2024-09-21T15:51:27.199 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:27.199 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:27.199 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:27.199 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:27.199 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:27.199 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 120130 2024-09-21T15:51:27.200 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.200 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:27.200 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.200 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:27.200 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576043+0000", 2024-09-21T15:51:27.200 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.200 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.200 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.200 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:27.200 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576043+0000", 2024-09-21T15:51:27.200 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.200 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.200 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.201 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:27.201 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576042+0000", 2024-09-21T15:51:27.201 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:27.201 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.201 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.201 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:27.201 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576044+0000", 2024-09-21T15:51:27.201 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0999999999999998e-06 2024-09-21T15:51:27.201 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.201 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.201 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:27.201 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576051+0000", 2024-09-21T15:51:27.201 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4779999999999996e-06 2024-09-21T15:51:27.201 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.202 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.202 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:27.202 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.576058+0000", 2024-09-21T15:51:27.202 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2180000000000002e-06 2024-09-21T15:51:27.202 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.202 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.202 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.202 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630661+0000", 2024-09-21T15:51:27.202 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054603240999999997 2024-09-21T15:51:27.202 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.202 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.202 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:27.202 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630747+0000", 2024-09-21T15:51:27.202 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6046999999999998e-05 2024-09-21T15:51:27.203 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.203 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.203 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:27.203 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:03.630824+0000", 2024-09-21T15:51:27.203 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7119999999999993e-05 2024-09-21T15:51:27.203 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.203 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.203 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.203 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.203 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.203 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:165208 3.2 3:40caac84:::rbd_id.image_client.0-clone14:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e131)", 2024-09-21T15:51:27.203 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:51:06.879846+0000", 2024-09-21T15:51:27.203 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 20.242261282000001, 2024-09-21T15:51:27.203 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1380672409999999, 2024-09-21T15:51:27.203 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:27.204 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:27.204 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:27.204 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:27.204 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:27.204 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 165208 2024-09-21T15:51:27.204 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.204 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:27.204 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.204 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:27.204 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:51:06.879846+0000", 2024-09-21T15:51:27.204 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.204 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.204 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.204 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:27.204 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:51:06.879846+0000", 2024-09-21T15:51:27.204 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.205 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.205 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.205 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:27.205 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:51:06.879845+0000", 2024-09-21T15:51:27.205 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:27.205 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.205 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.205 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:27.205 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:51:06.879849+0000", 2024-09-21T15:51:27.205 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.968e-06 2024-09-21T15:51:27.205 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.205 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.205 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:27.205 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:51:06.879860+0000", 2024-09-21T15:51:27.205 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0368e-05 2024-09-21T15:51:27.206 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.206 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.206 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:27.206 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:51:06.879867+0000", 2024-09-21T15:51:27.206 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0700000000000001e-06 2024-09-21T15:51:27.206 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.206 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.206 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.206 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:51:06.879907+0000", 2024-09-21T15:51:27.206 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.9904000000000003e-05 2024-09-21T15:51:27.206 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.206 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.206 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-21T15:51:27.206 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:51:06.879914+0000", 2024-09-21T15:51:27.207 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4279999999999999e-06 2024-09-21T15:51:27.207 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.207 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.207 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.207 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:51:07.686406+0000", 2024-09-21T15:51:27.207 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.80649178700000002 2024-09-21T15:51:27.207 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.207 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.207 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-21T15:51:27.207 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:51:07.686421+0000", 2024-09-21T15:51:27.207 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.5339e-05 2024-09-21T15:51:27.207 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.207 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.207 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.207 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:51:09.013823+0000", 2024-09-21T15:51:27.207 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.327402341 2024-09-21T15:51:27.208 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.208 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.208 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:27.208 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:51:09.014007+0000", 2024-09-21T15:51:27.208 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000183952 2024-09-21T15:51:27.208 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.208 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.208 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 3", 2024-09-21T15:51:27.208 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:51:09.014167+0000", 2024-09-21T15:51:27.208 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000159543 2024-09-21T15:51:27.208 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.208 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.208 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:51:27.208 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:51:09.014932+0000", 2024-09-21T15:51:27.208 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00076526800000000005 2024-09-21T15:51:27.208 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.209 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.209 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:27.209 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:51:09.017803+0000", 2024-09-21T15:51:27.209 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.0028705789999999998 2024-09-21T15:51:27.209 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.209 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.209 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:51:27.209 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:51:09.017838+0000", 2024-09-21T15:51:27.209 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.4782e-05 2024-09-21T15:51:27.209 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.209 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.209 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:27.209 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:51:09.017913+0000", 2024-09-21T15:51:27.209 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5570999999999999e-05 2024-09-21T15:51:27.209 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.209 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.210 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.210 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.210 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.210 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:51:27.211 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_ops_in_flight 2024-09-21T15:51:27.313 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:51:27.313 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:51:27.313 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:51:27.314 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:51:27.322 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_blocked_ops 2024-09-21T15:51:27.423 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:51:27.423 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:51:27.423 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:51:27.423 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:51:27.423 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:51:27.431 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.6.asok dump_historic_ops 2024-09-21T15:51:27.534 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:51:27.534 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:51:27.534 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:51:27.534 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:51:27.534 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.535 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63140 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 876544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:27.535 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.133649+0000", 2024-09-21T15:51:27.535 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 110.39889453799999, 2024-09-21T15:51:27.535 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053950201000000003, 2024-09-21T15:51:27.535 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:27.535 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:27.535 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:27.535 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:27.535 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:27.535 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63140 2024-09-21T15:51:27.535 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.535 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:27.535 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.535 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:27.535 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133649+0000", 2024-09-21T15:51:27.536 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.536 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.536 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.536 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:27.536 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133649+0000", 2024-09-21T15:51:27.536 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.536 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.536 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.536 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:27.536 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133648+0000", 2024-09-21T15:51:27.536 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:27.536 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.536 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.536 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:27.536 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133651+0000", 2024-09-21T15:51:27.537 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.1549999999999999e-06 2024-09-21T15:51:27.537 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.537 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.537 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:27.537 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133660+0000", 2024-09-21T15:51:27.537 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3899999999999993e-06 2024-09-21T15:51:27.537 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.537 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.537 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:27.537 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133669+0000", 2024-09-21T15:51:27.537 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3219999999999999e-06 2024-09-21T15:51:27.537 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.537 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.537 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.537 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187409+0000", 2024-09-21T15:51:27.538 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053740257 2024-09-21T15:51:27.538 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.538 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.538 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:27.538 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187492+0000", 2024-09-21T15:51:27.538 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2409000000000003e-05 2024-09-21T15:51:27.538 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.538 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.538 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:27.538 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187599+0000", 2024-09-21T15:51:27.538 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000107334 2024-09-21T15:51:27.538 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.538 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.538 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.538 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.538 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.539 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63141 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 884736~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:27.539 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.133843+0000", 2024-09-21T15:51:27.539 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 110.39870041899999, 2024-09-21T15:51:27.539 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053954015000000001, 2024-09-21T15:51:27.539 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:27.539 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:27.539 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:27.539 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:27.539 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:27.539 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63141 2024-09-21T15:51:27.539 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.539 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:27.539 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.539 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:27.539 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133843+0000", 2024-09-21T15:51:27.540 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.540 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.540 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.540 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:27.540 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133843+0000", 2024-09-21T15:51:27.540 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.540 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.540 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.540 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:27.540 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133673+0000", 2024-09-21T15:51:27.540 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998302 2024-09-21T15:51:27.540 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.540 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.540 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:27.540 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133846+0000", 2024-09-21T15:51:27.540 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000172762 2024-09-21T15:51:27.541 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.541 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.541 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:27.541 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133855+0000", 2024-09-21T15:51:27.541 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2940000000000006e-06 2024-09-21T15:51:27.541 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.541 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.541 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:27.541 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133863+0000", 2024-09-21T15:51:27.541 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9740000000000002e-06 2024-09-21T15:51:27.541 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.541 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.541 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.541 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187630+0000", 2024-09-21T15:51:27.541 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053767211000000002 2024-09-21T15:51:27.542 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.542 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.542 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:27.542 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187707+0000", 2024-09-21T15:51:27.542 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7446000000000004e-05 2024-09-21T15:51:27.542 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.542 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.542 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:27.542 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187797+0000", 2024-09-21T15:51:27.542 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.9858000000000006e-05 2024-09-21T15:51:27.542 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.542 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.542 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.542 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.542 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.543 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63142 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 892928~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:27.543 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134046+0000", 2024-09-21T15:51:27.543 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 110.39849747700001, 2024-09-21T15:51:27.543 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054017849999999999, 2024-09-21T15:51:27.543 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:27.543 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:27.543 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:27.543 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:27.543 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:27.543 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63142 2024-09-21T15:51:27.543 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.543 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:27.543 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.543 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:27.543 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134046+0000", 2024-09-21T15:51:27.544 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.544 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.544 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.544 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:27.544 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134046+0000", 2024-09-21T15:51:27.544 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.544 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.544 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.544 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:27.544 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.133867+0000", 2024-09-21T15:51:27.544 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998217 2024-09-21T15:51:27.544 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.544 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.544 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:27.545 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134049+0000", 2024-09-21T15:51:27.545 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000181695 2024-09-21T15:51:27.545 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.545 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.545 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:27.545 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134057+0000", 2024-09-21T15:51:27.545 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.303e-06 2024-09-21T15:51:27.545 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.545 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.545 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:27.545 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134063+0000", 2024-09-21T15:51:27.545 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9959999999999999e-06 2024-09-21T15:51:27.545 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.545 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.545 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.546 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187849+0000", 2024-09-21T15:51:27.546 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053785571999999997 2024-09-21T15:51:27.546 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.546 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.546 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:27.546 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.187954+0000", 2024-09-21T15:51:27.546 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00010524899999999999 2024-09-21T15:51:27.546 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.546 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.546 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:27.546 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188064+0000", 2024-09-21T15:51:27.546 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000109549 2024-09-21T15:51:27.546 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.546 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.547 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.547 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.547 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.547 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63143 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 901120~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:27.547 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134239+0000", 2024-09-21T15:51:27.547 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 110.398304884, 2024-09-21T15:51:27.547 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054026058000000002, 2024-09-21T15:51:27.547 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:27.547 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:27.547 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:27.547 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:27.547 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:27.547 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63143 2024-09-21T15:51:27.547 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.548 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:27.548 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.548 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:27.548 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134239+0000", 2024-09-21T15:51:27.548 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.548 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.548 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.548 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:27.548 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134239+0000", 2024-09-21T15:51:27.548 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.548 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.548 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.548 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:27.548 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134068+0000", 2024-09-21T15:51:27.548 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998298 2024-09-21T15:51:27.549 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.549 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.549 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:27.549 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134242+0000", 2024-09-21T15:51:27.549 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00017414199999999999 2024-09-21T15:51:27.549 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.549 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.549 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:27.549 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134264+0000", 2024-09-21T15:51:27.549 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1552e-05 2024-09-21T15:51:27.549 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.549 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.549 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:27.549 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134270+0000", 2024-09-21T15:51:27.549 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6939999999999998e-06 2024-09-21T15:51:27.550 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.550 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.550 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.550 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188108+0000", 2024-09-21T15:51:27.550 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.05383719 2024-09-21T15:51:27.550 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.550 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.550 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:27.550 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188186+0000", 2024-09-21T15:51:27.550 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8318000000000005e-05 2024-09-21T15:51:27.550 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.550 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.550 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:27.551 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188265+0000", 2024-09-21T15:51:27.551 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8597000000000007e-05 2024-09-21T15:51:27.551 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.551 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.551 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.551 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.551 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.551 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63144 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 909312~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:27.551 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134432+0000", 2024-09-21T15:51:27.551 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 110.398111328, 2024-09-21T15:51:27.551 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053968953, 2024-09-21T15:51:27.551 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:27.551 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:27.551 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:27.552 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:27.552 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:27.552 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63144 2024-09-21T15:51:27.552 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.552 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:27.552 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.552 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:27.552 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134432+0000", 2024-09-21T15:51:27.552 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.552 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.552 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.552 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:27.552 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134432+0000", 2024-09-21T15:51:27.552 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.552 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.552 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.553 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:27.553 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134276+0000", 2024-09-21T15:51:27.553 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998441 2024-09-21T15:51:27.553 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.553 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.553 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:27.553 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134436+0000", 2024-09-21T15:51:27.553 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000159591 2024-09-21T15:51:27.553 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.553 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.553 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:27.553 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134445+0000", 2024-09-21T15:51:27.553 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6780000000000006e-06 2024-09-21T15:51:27.554 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.554 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.554 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:27.554 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134452+0000", 2024-09-21T15:51:27.554 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6830000000000008e-06 2024-09-21T15:51:27.554 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.554 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.554 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.554 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188282+0000", 2024-09-21T15:51:27.554 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053829252000000001 2024-09-21T15:51:27.554 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.554 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.554 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:27.554 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188328+0000", 2024-09-21T15:51:27.554 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.6734999999999999e-05 2024-09-21T15:51:27.555 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.555 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.555 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:27.555 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188401+0000", 2024-09-21T15:51:27.555 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2731000000000005e-05 2024-09-21T15:51:27.555 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.555 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.555 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.555 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.555 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.555 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63145 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 860160~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:27.555 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.134736+0000", 2024-09-21T15:51:27.555 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 110.397807543, 2024-09-21T15:51:27.555 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053865960999999997, 2024-09-21T15:51:27.555 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:27.556 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:27.556 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:27.556 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:27.556 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:27.556 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63145 2024-09-21T15:51:27.556 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.556 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:27.556 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.556 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:27.556 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134736+0000", 2024-09-21T15:51:27.556 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.556 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.556 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.556 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:27.556 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134736+0000", 2024-09-21T15:51:27.557 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.557 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.557 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.557 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:27.557 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134458+0000", 2024-09-21T15:51:27.557 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.999722 2024-09-21T15:51:27.557 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.557 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.557 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:27.557 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134741+0000", 2024-09-21T15:51:27.557 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00028245599999999999 2024-09-21T15:51:27.557 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.557 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.557 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:27.557 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134753+0000", 2024-09-21T15:51:27.557 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2548000000000001e-05 2024-09-21T15:51:27.558 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.558 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.558 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:27.558 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.134760+0000", 2024-09-21T15:51:27.558 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4189999999999996e-06 2024-09-21T15:51:27.558 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.558 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.558 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.558 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188426+0000", 2024-09-21T15:51:27.558 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053665894999999998 2024-09-21T15:51:27.558 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.558 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.558 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:27.558 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188502+0000", 2024-09-21T15:51:27.558 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5791000000000007e-05 2024-09-21T15:51:27.558 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.559 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.559 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:27.559 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.188602+0000", 2024-09-21T15:51:27.559 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9667999999999995e-05 2024-09-21T15:51:27.559 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.559 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.559 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.559 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.559 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.559 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63148 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 933888~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:27.559 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.135289+0000", 2024-09-21T15:51:27.559 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 110.397254588, 2024-09-21T15:51:27.559 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053948488000000003, 2024-09-21T15:51:27.559 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:27.559 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:27.559 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:27.560 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:27.560 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:27.560 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63148 2024-09-21T15:51:27.560 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.560 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:27.560 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.560 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:27.560 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135289+0000", 2024-09-21T15:51:27.560 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.560 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.560 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.560 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:27.560 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135289+0000", 2024-09-21T15:51:27.560 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.560 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.561 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.561 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:27.561 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135288+0000", 2024-09-21T15:51:27.561 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:27.561 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.561 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.561 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:27.561 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135296+0000", 2024-09-21T15:51:27.561 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5399999999999998e-06 2024-09-21T15:51:27.561 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.561 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.561 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:27.561 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135303+0000", 2024-09-21T15:51:27.561 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0500000000000003e-06 2024-09-21T15:51:27.561 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.562 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.562 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:27.562 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135308+0000", 2024-09-21T15:51:27.562 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9749999999999998e-06 2024-09-21T15:51:27.562 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.562 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.562 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.562 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189066+0000", 2024-09-21T15:51:27.562 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053758501 2024-09-21T15:51:27.562 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.562 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.562 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:27.562 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189145+0000", 2024-09-21T15:51:27.562 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9017000000000006e-05 2024-09-21T15:51:27.562 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.563 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.563 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:27.563 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189237+0000", 2024-09-21T15:51:27.563 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1906999999999999e-05 2024-09-21T15:51:27.563 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.563 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.563 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.563 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.563 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.563 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63149 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 942080~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:27.563 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.135430+0000", 2024-09-21T15:51:27.563 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 110.39711325899999, 2024-09-21T15:51:27.563 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054005434999999997, 2024-09-21T15:51:27.563 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:27.563 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:27.564 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:27.564 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:27.564 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:27.564 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63149 2024-09-21T15:51:27.564 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.564 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:27.564 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.564 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:27.564 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135430+0000", 2024-09-21T15:51:27.564 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.564 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.564 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.564 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:27.564 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135430+0000", 2024-09-21T15:51:27.564 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.565 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.565 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.565 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:27.565 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135313+0000", 2024-09-21T15:51:27.565 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998827 2024-09-21T15:51:27.565 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.565 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.565 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:27.565 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135434+0000", 2024-09-21T15:51:27.565 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000120876 2024-09-21T15:51:27.565 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.565 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.565 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:27.565 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135443+0000", 2024-09-21T15:51:27.565 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3340000000000008e-06 2024-09-21T15:51:27.565 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.566 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.566 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:27.566 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135449+0000", 2024-09-21T15:51:27.566 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1580000000000002e-06 2024-09-21T15:51:27.566 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.566 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.566 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.566 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189267+0000", 2024-09-21T15:51:27.566 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053817775999999998 2024-09-21T15:51:27.566 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.566 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.566 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:27.567 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189336+0000", 2024-09-21T15:51:27.567 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9485000000000003e-05 2024-09-21T15:51:27.567 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.567 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.567 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:27.567 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189436+0000", 2024-09-21T15:51:27.567 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9277000000000005e-05 2024-09-21T15:51:27.567 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.567 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.567 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.567 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.567 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.567 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63150 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 950272~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:27.567 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.135618+0000", 2024-09-21T15:51:27.568 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 110.396925514, 2024-09-21T15:51:27.568 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054027393, 2024-09-21T15:51:27.568 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:27.568 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:27.568 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:27.568 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:27.568 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:27.568 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63150 2024-09-21T15:51:27.568 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.568 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:27.568 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.568 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:27.568 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135618+0000", 2024-09-21T15:51:27.569 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.569 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.569 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.569 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:27.569 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135618+0000", 2024-09-21T15:51:27.569 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.569 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.569 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.569 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:27.569 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135454+0000", 2024-09-21T15:51:27.569 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998364 2024-09-21T15:51:27.569 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.569 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.569 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:27.569 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135621+0000", 2024-09-21T15:51:27.570 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016714099999999999 2024-09-21T15:51:27.570 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.570 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.570 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:27.570 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135631+0000", 2024-09-21T15:51:27.570 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5340000000000005e-06 2024-09-21T15:51:27.570 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.570 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.570 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:27.570 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.135638+0000", 2024-09-21T15:51:27.570 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9280000000000004e-06 2024-09-21T15:51:27.570 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.570 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.570 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.571 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189463+0000", 2024-09-21T15:51:27.571 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053825681 2024-09-21T15:51:27.571 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.571 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.571 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:27.571 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189546+0000", 2024-09-21T15:51:27.571 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.2278000000000001e-05 2024-09-21T15:51:27.571 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.571 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.571 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:27.571 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189645+0000", 2024-09-21T15:51:27.571 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9543999999999994e-05 2024-09-21T15:51:27.571 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.571 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.571 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.572 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.572 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.572 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63152 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 966656~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:27.572 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.136029+0000", 2024-09-21T15:51:27.572 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 110.39651492199999, 2024-09-21T15:51:27.572 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054024823999999999, 2024-09-21T15:51:27.572 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:27.572 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:27.572 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:27.572 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:27.572 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:27.572 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63152 2024-09-21T15:51:27.572 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.572 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:27.572 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.572 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:27.573 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136029+0000", 2024-09-21T15:51:27.573 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.573 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.573 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.573 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:27.573 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136029+0000", 2024-09-21T15:51:27.573 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.573 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.573 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.573 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:27.573 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136028+0000", 2024-09-21T15:51:27.573 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:27.573 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.573 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.573 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:27.574 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136031+0000", 2024-09-21T15:51:27.574 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.033e-06 2024-09-21T15:51:27.574 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.574 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.574 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:27.574 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136039+0000", 2024-09-21T15:51:27.574 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7200000000000006e-06 2024-09-21T15:51:27.574 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.574 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.574 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:27.574 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136051+0000", 2024-09-21T15:51:27.574 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1987000000000001e-05 2024-09-21T15:51:27.574 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.574 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.575 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.575 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189876+0000", 2024-09-21T15:51:27.575 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053825264999999997 2024-09-21T15:51:27.575 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.575 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.575 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:27.575 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.189955+0000", 2024-09-21T15:51:27.575 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9296999999999997e-05 2024-09-21T15:51:27.575 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.575 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.575 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:27.575 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190053+0000", 2024-09-21T15:51:27.575 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.8067999999999997e-05 2024-09-21T15:51:27.575 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.575 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.575 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.576 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.576 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.576 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63153 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 974848~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:27.576 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.136311+0000", 2024-09-21T15:51:27.576 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 110.39623240900001, 2024-09-21T15:51:27.576 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053933825999999997, 2024-09-21T15:51:27.576 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:27.576 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:27.576 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:27.576 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:27.576 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:27.576 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63153 2024-09-21T15:51:27.576 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.576 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:27.576 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.577 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:27.577 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136311+0000", 2024-09-21T15:51:27.577 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.577 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.577 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.577 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:27.577 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136311+0000", 2024-09-21T15:51:27.577 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.577 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.577 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.577 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:27.577 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136057+0000", 2024-09-21T15:51:27.577 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9997463 2024-09-21T15:51:27.577 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.577 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.578 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:27.578 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136316+0000", 2024-09-21T15:51:27.578 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00025872299999999998 2024-09-21T15:51:27.578 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.578 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.578 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:27.578 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136326+0000", 2024-09-21T15:51:27.578 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0552e-05 2024-09-21T15:51:27.578 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.578 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.578 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:27.578 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136334+0000", 2024-09-21T15:51:27.578 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1409999999999999e-06 2024-09-21T15:51:27.578 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.578 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.579 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.579 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190080+0000", 2024-09-21T15:51:27.579 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053745933000000003 2024-09-21T15:51:27.579 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.579 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.579 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:27.579 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190157+0000", 2024-09-21T15:51:27.579 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7412999999999999e-05 2024-09-21T15:51:27.579 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.579 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.579 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:27.579 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190245+0000", 2024-09-21T15:51:27.579 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.7904000000000004e-05 2024-09-21T15:51:27.579 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.579 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.579 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.580 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.580 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.580 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63154 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 983040~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:27.580 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.136472+0000", 2024-09-21T15:51:27.580 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 110.396071058, 2024-09-21T15:51:27.580 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053968448000000002, 2024-09-21T15:51:27.580 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:27.580 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:27.580 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:27.580 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:27.580 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:27.580 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63154 2024-09-21T15:51:27.580 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.580 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:27.580 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.581 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:27.581 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136472+0000", 2024-09-21T15:51:27.581 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.581 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.581 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.581 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:27.581 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136472+0000", 2024-09-21T15:51:27.581 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.581 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.581 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.581 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:27.581 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136341+0000", 2024-09-21T15:51:27.581 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998684 2024-09-21T15:51:27.581 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.581 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.581 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:27.582 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136476+0000", 2024-09-21T15:51:27.582 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00013528100000000001 2024-09-21T15:51:27.582 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.582 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.582 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:27.582 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136484+0000", 2024-09-21T15:51:27.582 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4760000000000001e-06 2024-09-21T15:51:27.582 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.582 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.582 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:27.582 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.136490+0000", 2024-09-21T15:51:27.582 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.8640000000000001e-06 2024-09-21T15:51:27.582 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.582 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.583 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.583 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190271+0000", 2024-09-21T15:51:27.583 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053781223000000003 2024-09-21T15:51:27.583 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.583 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.583 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:27.583 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190348+0000", 2024-09-21T15:51:27.583 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7058999999999995e-05 2024-09-21T15:51:27.583 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.583 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.583 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:27.583 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190441+0000", 2024-09-21T15:51:27.583 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.2937999999999997e-05 2024-09-21T15:51:27.583 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.583 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.584 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.584 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.584 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.584 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63157 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1015808~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:27.584 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137112+0000", 2024-09-21T15:51:27.584 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 110.39543177100001, 2024-09-21T15:51:27.584 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053932345, 2024-09-21T15:51:27.584 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:27.584 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:27.584 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:27.584 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:27.584 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:27.584 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63157 2024-09-21T15:51:27.584 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.584 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:27.584 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.585 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:27.585 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137112+0000", 2024-09-21T15:51:27.585 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.585 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.585 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.585 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:27.585 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137112+0000", 2024-09-21T15:51:27.585 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.585 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.585 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.585 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:27.585 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137072+0000", 2024-09-21T15:51:27.586 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999604 2024-09-21T15:51:27.586 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.586 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.586 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:27.586 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137115+0000", 2024-09-21T15:51:27.586 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.2379999999999997e-05 2024-09-21T15:51:27.586 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.586 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.586 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:27.586 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137121+0000", 2024-09-21T15:51:27.586 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6499999999999999e-06 2024-09-21T15:51:27.586 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.586 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.586 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:27.587 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137126+0000", 2024-09-21T15:51:27.587 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0769999999999998e-06 2024-09-21T15:51:27.587 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.587 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.587 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.587 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190868+0000", 2024-09-21T15:51:27.587 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053741555000000003 2024-09-21T15:51:27.587 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.587 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.587 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:27.587 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.190945+0000", 2024-09-21T15:51:27.587 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.7171999999999999e-05 2024-09-21T15:51:27.587 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.587 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.588 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:27.588 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191044+0000", 2024-09-21T15:51:27.588 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9031999999999996e-05 2024-09-21T15:51:27.588 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.588 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.588 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.588 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.588 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.588 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63158 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1007616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:27.588 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137293+0000", 2024-09-21T15:51:27.588 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 110.395250484, 2024-09-21T15:51:27.588 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.054005642, 2024-09-21T15:51:27.588 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:27.589 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:27.589 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:27.589 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:27.589 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:27.589 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63158 2024-09-21T15:51:27.589 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.589 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:27.589 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.589 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:27.589 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137293+0000", 2024-09-21T15:51:27.589 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.589 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.589 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.589 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:27.590 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137293+0000", 2024-09-21T15:51:27.590 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.590 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.590 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.590 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:27.590 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137131+0000", 2024-09-21T15:51:27.590 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998379 2024-09-21T15:51:27.590 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.590 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.590 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:27.590 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137296+0000", 2024-09-21T15:51:27.590 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00016568999999999999 2024-09-21T15:51:27.590 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.590 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.591 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:27.591 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137306+0000", 2024-09-21T15:51:27.591 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.6029999999999994e-06 2024-09-21T15:51:27.591 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.591 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.591 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:27.591 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137312+0000", 2024-09-21T15:51:27.591 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3210000000000001e-06 2024-09-21T15:51:27.591 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.591 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.591 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.591 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191087+0000", 2024-09-21T15:51:27.591 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053774686000000002 2024-09-21T15:51:27.591 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.592 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.592 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:27.592 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191175+0000", 2024-09-21T15:51:27.592 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8106e-05 2024-09-21T15:51:27.592 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.592 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.592 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:27.592 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191299+0000", 2024-09-21T15:51:27.592 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012343599999999999 2024-09-21T15:51:27.592 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.592 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.592 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.592 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.592 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.593 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63159 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1024000~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:27.593 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137568+0000", 2024-09-21T15:51:27.593 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 110.39497521200001, 2024-09-21T15:51:27.593 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053962999999999997, 2024-09-21T15:51:27.593 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:27.593 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:27.593 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:27.593 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:27.593 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:27.593 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63159 2024-09-21T15:51:27.593 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.593 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:27.593 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.593 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:27.594 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137568+0000", 2024-09-21T15:51:27.594 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.594 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.594 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.594 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:27.594 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137568+0000", 2024-09-21T15:51:27.594 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.594 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.594 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.594 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:27.594 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137317+0000", 2024-09-21T15:51:27.594 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9997492 2024-09-21T15:51:27.594 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.594 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.595 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:27.595 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137572+0000", 2024-09-21T15:51:27.595 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00025491499999999998 2024-09-21T15:51:27.595 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.595 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.595 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:27.595 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137582+0000", 2024-09-21T15:51:27.595 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4490000000000005e-06 2024-09-21T15:51:27.595 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.595 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.595 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:27.595 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137589+0000", 2024-09-21T15:51:27.595 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5580000000000003e-06 2024-09-21T15:51:27.596 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.596 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.596 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.596 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191344+0000", 2024-09-21T15:51:27.596 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053754272999999998 2024-09-21T15:51:27.596 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.596 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.596 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:27.596 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191429+0000", 2024-09-21T15:51:27.596 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.509e-05 2024-09-21T15:51:27.596 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.596 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.596 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:27.596 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191531+0000", 2024-09-21T15:51:27.597 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000102519 2024-09-21T15:51:27.597 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.597 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.597 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.597 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.597 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.597 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63160 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1032192~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:27.597 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.137771+0000", 2024-09-21T15:51:27.597 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 110.394772678, 2024-09-21T15:51:27.597 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053957818999999997, 2024-09-21T15:51:27.597 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:27.597 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:27.597 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:27.597 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:27.598 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:27.598 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63160 2024-09-21T15:51:27.598 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.598 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:27.598 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.598 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:27.598 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137771+0000", 2024-09-21T15:51:27.598 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.598 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.598 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.598 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:27.598 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137771+0000", 2024-09-21T15:51:27.598 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.598 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.599 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.599 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:27.599 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137596+0000", 2024-09-21T15:51:27.599 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9998255 2024-09-21T15:51:27.599 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.599 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.599 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:27.599 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137780+0000", 2024-09-21T15:51:27.599 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00018363900000000001 2024-09-21T15:51:27.599 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.599 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.599 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:27.599 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137791+0000", 2024-09-21T15:51:27.599 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.096e-05 2024-09-21T15:51:27.600 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.600 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.600 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:27.600 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137798+0000", 2024-09-21T15:51:27.600 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.1169999999999998e-06 2024-09-21T15:51:27.600 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.600 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.600 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.600 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191560+0000", 2024-09-21T15:51:27.600 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053762163000000002 2024-09-21T15:51:27.600 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.600 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.600 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:27.600 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191634+0000", 2024-09-21T15:51:27.601 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.4381999999999994e-05 2024-09-21T15:51:27.601 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.601 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.601 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:27.601 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191729+0000", 2024-09-21T15:51:27.601 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.4098000000000006e-05 2024-09-21T15:51:27.601 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.601 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.601 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.601 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.601 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.601 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63161 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 1040384~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:27.601 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.138018+0000", 2024-09-21T15:51:27.601 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 110.39452592400001, 2024-09-21T15:51:27.602 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053905657000000003, 2024-09-21T15:51:27.602 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:27.602 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:27.602 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:27.602 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:27.602 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:27.602 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63161 2024-09-21T15:51:27.602 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.602 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:27.602 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.602 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:27.602 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138018+0000", 2024-09-21T15:51:27.602 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.603 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.603 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.603 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:27.603 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138018+0000", 2024-09-21T15:51:27.603 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.603 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.603 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.603 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:27.603 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.137804+0000", 2024-09-21T15:51:27.603 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9997859 2024-09-21T15:51:27.603 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.603 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.603 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:27.603 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138021+0000", 2024-09-21T15:51:27.604 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00021769899999999999 2024-09-21T15:51:27.604 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.604 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.604 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:27.604 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138043+0000", 2024-09-21T15:51:27.604 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1627000000000001e-05 2024-09-21T15:51:27.604 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.604 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.604 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:27.604 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.138054+0000", 2024-09-21T15:51:27.604 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1321999999999999e-05 2024-09-21T15:51:27.604 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.604 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.604 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.605 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191755+0000", 2024-09-21T15:51:27.605 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053700354999999998 2024-09-21T15:51:27.605 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.605 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.605 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:27.605 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191830+0000", 2024-09-21T15:51:27.605 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5320000000000004e-05 2024-09-21T15:51:27.605 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.605 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.605 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:27.605 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.191923+0000", 2024-09-21T15:51:27.605 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3282999999999994e-05 2024-09-21T15:51:27.605 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.605 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.606 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.606 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.606 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.606 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63176 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 108544~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:27.606 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.140861+0000", 2024-09-21T15:51:27.606 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 110.391682605, 2024-09-21T15:51:27.606 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053935394999999997, 2024-09-21T15:51:27.606 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:27.606 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:27.606 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:27.606 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:27.606 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:27.606 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63176 2024-09-21T15:51:27.607 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.607 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:27.607 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.607 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:27.607 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140861+0000", 2024-09-21T15:51:27.607 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.607 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.607 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.607 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:27.607 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140861+0000", 2024-09-21T15:51:27.607 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.607 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.607 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.607 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:27.607 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140860+0000", 2024-09-21T15:51:27.608 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:27.608 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.608 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.608 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:27.608 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140863+0000", 2024-09-21T15:51:27.608 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8710000000000001e-06 2024-09-21T15:51:27.608 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.608 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.608 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:27.608 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140871+0000", 2024-09-21T15:51:27.608 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8089999999999993e-06 2024-09-21T15:51:27.608 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.608 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.608 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:27.608 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.140877+0000", 2024-09-21T15:51:27.609 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5969999999999999e-06 2024-09-21T15:51:27.609 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.609 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.609 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.609 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.194608+0000", 2024-09-21T15:51:27.609 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053731749000000002 2024-09-21T15:51:27.609 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.609 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.609 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:27.609 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.194697+0000", 2024-09-21T15:51:27.609 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8504000000000005e-05 2024-09-21T15:51:27.609 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.609 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.609 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:27.610 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.194796+0000", 2024-09-21T15:51:27.610 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9414999999999996e-05 2024-09-21T15:51:27.610 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.610 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.610 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.610 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.610 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.610 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63182 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 157696~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:27.610 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:51:27.610 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 110.39041020499999, 2024-09-21T15:51:27.610 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053958825000000002, 2024-09-21T15:51:27.610 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:27.610 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:27.611 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:27.611 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:27.611 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:27.611 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63182 2024-09-21T15:51:27.611 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.611 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:27.611 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.611 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:27.611 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:51:27.611 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.611 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.611 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.611 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:27.611 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:51:27.611 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.612 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.612 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.612 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:27.612 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142133+0000", 2024-09-21T15:51:27.612 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:27.612 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.612 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.612 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:27.612 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142136+0000", 2024-09-21T15:51:27.612 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.8169999999999999e-06 2024-09-21T15:51:27.612 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.612 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.612 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:27.612 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142149+0000", 2024-09-21T15:51:27.613 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.3752e-05 2024-09-21T15:51:27.613 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.613 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.613 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:27.613 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.142155+0000", 2024-09-21T15:51:27.613 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1859999999999997e-06 2024-09-21T15:51:27.613 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.613 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.613 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.613 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.195914+0000", 2024-09-21T15:51:27.613 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053758528999999999 2024-09-21T15:51:27.613 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.613 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.613 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:27.614 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.195995+0000", 2024-09-21T15:51:27.614 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0983999999999996e-05 2024-09-21T15:51:27.614 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.614 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.614 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:27.614 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.196092+0000", 2024-09-21T15:51:27.614 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.7069000000000002e-05 2024-09-21T15:51:27.614 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.614 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.614 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.614 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.614 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.614 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:63193 3.a 3:5689f281:::rbd_data.11336b2fdf4a.0000000000000005:c [read 239616~2048] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:27.615 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:51:27.615 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 110.38832262, 2024-09-21T15:51:27.615 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053956577999999998, 2024-09-21T15:51:27.615 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:27.615 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:27.615 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:27.615 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:27.615 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:27.615 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 63193 2024-09-21T15:51:27.615 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.615 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:27.615 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.615 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:27.615 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:51:27.615 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.616 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.616 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.616 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:27.616 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:51:27.616 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.616 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.616 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.616 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:27.616 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144221+0000", 2024-09-21T15:51:27.616 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:27.616 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.616 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.616 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:27.616 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144223+0000", 2024-09-21T15:51:27.617 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0990000000000001e-06 2024-09-21T15:51:27.617 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.617 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.617 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:27.617 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144230+0000", 2024-09-21T15:51:27.617 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9070000000000001e-06 2024-09-21T15:51:27.617 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.617 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.617 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:27.617 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.144234+0000", 2024-09-21T15:51:27.617 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9010000000000002e-06 2024-09-21T15:51:27.617 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.617 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.618 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.618 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.198004+0000", 2024-09-21T15:51:27.618 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.053769539999999998 2024-09-21T15:51:27.618 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.618 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.618 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:27.618 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.198083+0000", 2024-09-21T15:51:27.618 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9531999999999997e-05 2024-09-21T15:51:27.618 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.618 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.618 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:27.618 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.198177+0000", 2024-09-21T15:51:27.618 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.3906000000000005e-05 2024-09-21T15:51:27.618 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.619 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.619 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.619 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.619 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.619 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:51:27.620 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_ops_in_flight 2024-09-21T15:51:27.679 INFO:teuthology.orchestra.run.smithi089.stderr:noscrub is unset 2024-09-21T15:51:27.695 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd unset nodeep-scrub 2024-09-21T15:51:27.725 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:51:27.726 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:51:27.726 INFO:teuthology.orchestra.run.smithi100.stdout: "num_ops": 0 2024-09-21T15:51:27.726 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:51:27.734 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_blocked_ops 2024-09-21T15:51:27.838 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:51:27.839 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [], 2024-09-21T15:51:27.839 INFO:teuthology.orchestra.run.smithi100.stdout: "complaint_time": 30, 2024-09-21T15:51:27.839 INFO:teuthology.orchestra.run.smithi100.stdout: "num_blocked_ops": 0 2024-09-21T15:51:27.839 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:51:27.847 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.7.asok dump_historic_ops 2024-09-21T15:51:27.951 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-09-21T15:51:27.951 INFO:teuthology.orchestra.run.smithi100.stdout: "size": 20, 2024-09-21T15:51:27.951 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 600, 2024-09-21T15:51:27.951 INFO:teuthology.orchestra.run.smithi100.stdout: "ops": [ 2024-09-21T15:51:27.951 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.951 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4209.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e15)", 2024-09-21T15:51:27.951 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:51:27.951 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 156.758159632, 2024-09-21T15:51:27.951 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.19452909900000001, 2024-09-21T15:51:27.952 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:27.952 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:27.952 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:27.952 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4209", 2024-09-21T15:51:27.952 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/4045143050", 2024-09-21T15:51:27.952 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 1 2024-09-21T15:51:27.952 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.952 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:27.952 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.952 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:27.952 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:51:27.952 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.952 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.952 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.952 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:27.952 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191162+0000", 2024-09-21T15:51:27.953 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.953 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.953 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.953 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:27.953 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191161+0000", 2024-09-21T15:51:27.953 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999986 2024-09-21T15:51:27.953 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.953 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.953 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:27.953 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191169+0000", 2024-09-21T15:51:27.953 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.0090000000000007e-06 2024-09-21T15:51:27.953 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.953 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.953 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:27.954 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191180+0000", 2024-09-21T15:51:27.954 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.0728e-05 2024-09-21T15:51:27.954 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.954 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.954 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:27.954 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191187+0000", 2024-09-21T15:51:27.954 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.9009999999999994e-06 2024-09-21T15:51:27.954 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.954 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.954 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.954 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191238+0000", 2024-09-21T15:51:27.954 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0287000000000002e-05 2024-09-21T15:51:27.954 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.954 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.954 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for peered", 2024-09-21T15:51:27.955 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.191245+0000", 2024-09-21T15:51:27.955 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0709999999999997e-06 2024-09-21T15:51:27.955 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.955 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.955 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.955 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380309+0000", 2024-09-21T15:51:27.955 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.189064277 2024-09-21T15:51:27.955 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.955 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.955 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:27.955 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380454+0000", 2024-09-21T15:51:27.955 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00014490799999999999 2024-09-21T15:51:27.955 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.955 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.955 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "waiting for subops from 0", 2024-09-21T15:51:27.956 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.380610+0000", 2024-09-21T15:51:27.956 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00015650199999999999 2024-09-21T15:51:27.956 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.956 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.956 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "op_commit", 2024-09-21T15:51:27.956 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.381160+0000", 2024-09-21T15:51:27.956 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00054903799999999996 2024-09-21T15:51:27.956 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.956 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.956 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:27.956 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385591+0000", 2024-09-21T15:51:27.956 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.004431272 2024-09-21T15:51:27.956 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.956 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.956 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "commit_sent", 2024-09-21T15:51:27.956 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385651+0000", 2024-09-21T15:51:27.957 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0455999999999999e-05 2024-09-21T15:51:27.957 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.957 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.957 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:27.957 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:48:51.385691+0000", 2024-09-21T15:51:27.957 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.0234999999999997e-05 2024-09-21T15:51:27.957 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.957 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.957 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.957 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.957 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.957 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64034 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1083392~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:27.957 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167717+0000", 2024-09-21T15:51:27.957 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 110.781605302, 2024-09-21T15:51:27.957 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048233316999999998, 2024-09-21T15:51:27.958 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:27.958 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:27.958 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:27.958 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:27.958 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:27.958 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64034 2024-09-21T15:51:27.958 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.958 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:27.958 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.958 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:27.958 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167717+0000", 2024-09-21T15:51:27.958 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.958 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.958 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.958 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:27.959 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167717+0000", 2024-09-21T15:51:27.959 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.959 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.959 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.959 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:27.959 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167716+0000", 2024-09-21T15:51:27.959 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:27.959 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.959 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.959 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:27.959 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167718+0000", 2024-09-21T15:51:27.959 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.165e-06 2024-09-21T15:51:27.959 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.959 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.959 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:27.959 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167725+0000", 2024-09-21T15:51:27.960 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3380000000000001e-06 2024-09-21T15:51:27.960 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.960 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.960 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:27.960 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167730+0000", 2024-09-21T15:51:27.960 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8010000000000003e-06 2024-09-21T15:51:27.960 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.960 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.960 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.960 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215807+0000", 2024-09-21T15:51:27.960 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048077823999999998 2024-09-21T15:51:27.960 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.960 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.960 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:27.960 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215884+0000", 2024-09-21T15:51:27.960 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6283000000000001e-05 2024-09-21T15:51:27.961 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.961 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.961 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:27.961 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215950+0000", 2024-09-21T15:51:27.961 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6290000000000004e-05 2024-09-21T15:51:27.961 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.961 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.961 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.961 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.961 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.961 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64035 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1091584~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:27.961 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:51:27.961 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 110.781588089, 2024-09-21T15:51:27.961 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048412117999999997, 2024-09-21T15:51:27.961 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:27.961 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:27.962 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:27.962 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:27.962 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:27.962 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64035 2024-09-21T15:51:27.962 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.962 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:27.962 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.962 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:27.962 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:51:27.962 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.962 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.962 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.962 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:27.962 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:51:27.962 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.963 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.963 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.963 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:27.963 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167734+0000", 2024-09-21T15:51:27.963 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:27.963 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.963 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.963 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:27.963 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167746+0000", 2024-09-21T15:51:27.963 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.2031000000000001e-05 2024-09-21T15:51:27.963 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.963 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.963 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:27.963 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167753+0000", 2024-09-21T15:51:27.964 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.0620000000000003e-06 2024-09-21T15:51:27.964 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.964 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.964 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:27.964 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167759+0000", 2024-09-21T15:51:27.964 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.596e-06 2024-09-21T15:51:27.964 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.964 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.964 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.964 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.215977+0000", 2024-09-21T15:51:27.964 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048217318000000002 2024-09-21T15:51:27.964 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.964 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.964 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:27.964 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216076+0000", 2024-09-21T15:51:27.965 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.9846000000000006e-05 2024-09-21T15:51:27.965 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.965 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.965 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:27.965 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216146+0000", 2024-09-21T15:51:27.965 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.9578000000000004e-05 2024-09-21T15:51:27.965 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.965 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.965 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.965 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.965 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.965 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64036 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1099776~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:27.965 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167853+0000", 2024-09-21T15:51:27.965 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 110.781469476, 2024-09-21T15:51:27.965 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048450675999999998, 2024-09-21T15:51:27.965 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:27.966 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:27.966 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:27.966 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:27.966 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:27.966 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64036 2024-09-21T15:51:27.966 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.966 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:27.966 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.966 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:27.966 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167853+0000", 2024-09-21T15:51:27.966 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.966 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.966 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.966 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:27.966 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167853+0000", 2024-09-21T15:51:27.967 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.967 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.967 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.967 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:27.967 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167852+0000", 2024-09-21T15:51:27.967 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:27.967 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.967 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.967 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:27.967 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167855+0000", 2024-09-21T15:51:27.967 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5640000000000001e-06 2024-09-21T15:51:27.967 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.967 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.967 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:27.967 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167864+0000", 2024-09-21T15:51:27.968 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.8140000000000007e-06 2024-09-21T15:51:27.968 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.968 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.968 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:27.968 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167871+0000", 2024-09-21T15:51:27.968 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5729999999999996e-06 2024-09-21T15:51:27.968 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.968 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.968 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.968 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216174+0000", 2024-09-21T15:51:27.968 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048303437999999997 2024-09-21T15:51:27.968 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.968 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.968 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:27.968 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216250+0000", 2024-09-21T15:51:27.969 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.5661999999999995e-05 2024-09-21T15:51:27.969 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.969 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.969 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:27.969 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216303+0000", 2024-09-21T15:51:27.969 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3273000000000003e-05 2024-09-21T15:51:27.969 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.969 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.969 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.969 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.969 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.969 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64037 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1771520~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:27.969 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:51:27.969 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 110.781445933, 2024-09-21T15:51:27.969 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048588771000000003, 2024-09-21T15:51:27.969 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:27.970 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:27.970 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:27.970 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:27.970 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:27.970 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64037 2024-09-21T15:51:27.970 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.970 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:27.970 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.970 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:27.970 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:51:27.970 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.970 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.970 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.970 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:27.970 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:51:27.971 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.971 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.971 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.971 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:27.971 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167876+0000", 2024-09-21T15:51:27.971 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:27.971 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.971 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.971 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:27.971 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167878+0000", 2024-09-21T15:51:27.971 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3e-06 2024-09-21T15:51:27.971 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.971 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.971 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:27.972 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167884+0000", 2024-09-21T15:51:27.972 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.3269999999999997e-06 2024-09-21T15:51:27.972 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.972 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.972 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:27.972 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167889+0000", 2024-09-21T15:51:27.972 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9269999999999996e-06 2024-09-21T15:51:27.972 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.972 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.972 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.972 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216331+0000", 2024-09-21T15:51:27.972 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048442102000000001 2024-09-21T15:51:27.972 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.972 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.972 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:27.972 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216393+0000", 2024-09-21T15:51:27.973 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1132999999999999e-05 2024-09-21T15:51:27.973 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.973 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.973 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:27.973 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216465+0000", 2024-09-21T15:51:27.973 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2339e-05 2024-09-21T15:51:27.973 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.973 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.973 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.973 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.973 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.973 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64038 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1779712~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:27.973 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:51:27.973 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 110.78142810200001, 2024-09-21T15:51:27.973 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048749795999999998, 2024-09-21T15:51:27.973 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:27.974 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:27.974 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:27.974 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:27.974 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:27.974 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64038 2024-09-21T15:51:27.974 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.974 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:27.974 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.974 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:27.974 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:51:27.974 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.974 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.974 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.974 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:27.975 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:51:27.975 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.975 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.975 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.975 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:27.975 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167894+0000", 2024-09-21T15:51:27.975 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:27.975 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.975 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.975 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:27.975 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167896+0000", 2024-09-21T15:51:27.975 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2419999999999999e-06 2024-09-21T15:51:27.975 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.975 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.975 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:27.975 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167902+0000", 2024-09-21T15:51:27.976 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.0859999999999998e-06 2024-09-21T15:51:27.976 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.976 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.976 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:27.976 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167907+0000", 2024-09-21T15:51:27.976 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9679999999999997e-06 2024-09-21T15:51:27.976 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.976 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.976 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.976 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216506+0000", 2024-09-21T15:51:27.976 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048599374000000001 2024-09-21T15:51:27.976 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.976 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.976 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:27.976 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216579+0000", 2024-09-21T15:51:27.976 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2801999999999999e-05 2024-09-21T15:51:27.977 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.977 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.977 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:27.977 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216644+0000", 2024-09-21T15:51:27.977 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.4715000000000006e-05 2024-09-21T15:51:27.977 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.977 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.977 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.977 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.977 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.977 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64039 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1107968~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:27.977 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:51:27.977 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 110.781410747, 2024-09-21T15:51:27.977 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048911370000000003, 2024-09-21T15:51:27.977 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:27.978 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:27.978 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:27.978 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:27.978 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:27.978 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64039 2024-09-21T15:51:27.978 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.978 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:27.978 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.978 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:27.978 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:51:27.978 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.978 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.978 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.978 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:27.978 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:51:27.979 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.979 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.979 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.979 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:27.979 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167911+0000", 2024-09-21T15:51:27.979 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:27.979 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.979 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.979 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:27.979 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167913+0000", 2024-09-21T15:51:27.979 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0839999999999999e-06 2024-09-21T15:51:27.979 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.979 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.979 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:27.979 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167921+0000", 2024-09-21T15:51:27.980 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8310000000000001e-06 2024-09-21T15:51:27.980 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.980 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.980 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:27.980 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167926+0000", 2024-09-21T15:51:27.980 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0270000000000003e-06 2024-09-21T15:51:27.980 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.980 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.980 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.980 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216680+0000", 2024-09-21T15:51:27.980 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048753890000000001 2024-09-21T15:51:27.980 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.980 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.980 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:27.980 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216757+0000", 2024-09-21T15:51:27.980 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6848999999999996e-05 2024-09-21T15:51:27.981 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.981 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.981 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:27.981 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216823+0000", 2024-09-21T15:51:27.981 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6068000000000005e-05 2024-09-21T15:51:27.981 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.981 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.981 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.981 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.981 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.981 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64040 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1787904~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:27.981 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167931+0000", 2024-09-21T15:51:27.981 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 110.781391515, 2024-09-21T15:51:27.981 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049010561000000001, 2024-09-21T15:51:27.981 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:27.982 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:27.982 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:27.982 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:27.982 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:27.982 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64040 2024-09-21T15:51:27.982 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.982 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:27.982 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.982 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:27.982 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167931+0000", 2024-09-21T15:51:27.982 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.982 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.982 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.982 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:27.982 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167931+0000", 2024-09-21T15:51:27.983 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.983 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.983 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.983 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:27.983 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167930+0000", 2024-09-21T15:51:27.983 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:27.983 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.983 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.983 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:27.983 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167932+0000", 2024-09-21T15:51:27.983 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.198e-06 2024-09-21T15:51:27.983 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.983 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.983 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:27.984 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167938+0000", 2024-09-21T15:51:27.984 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.105e-06 2024-09-21T15:51:27.984 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.984 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.984 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:27.984 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167943+0000", 2024-09-21T15:51:27.984 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9429999999999999e-06 2024-09-21T15:51:27.984 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.984 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.984 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.984 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216848+0000", 2024-09-21T15:51:27.984 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048905200000000003 2024-09-21T15:51:27.984 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.984 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.984 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:27.984 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216903+0000", 2024-09-21T15:51:27.985 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.4793999999999997e-05 2024-09-21T15:51:27.985 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.985 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.985 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:27.985 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216941+0000", 2024-09-21T15:51:27.985 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.7790999999999998e-05 2024-09-21T15:51:27.985 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.985 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.985 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.985 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.985 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.985 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64041 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1796096~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:27.985 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:51:27.985 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 110.781374544, 2024-09-21T15:51:27.985 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049181457999999997, 2024-09-21T15:51:27.986 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:27.986 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:27.986 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:27.986 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:27.986 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:27.986 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64041 2024-09-21T15:51:27.986 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.986 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:27.986 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.986 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:27.986 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:51:27.986 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.986 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.986 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.987 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:27.987 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:51:27.987 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.987 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.987 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.987 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:27.987 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167947+0000", 2024-09-21T15:51:27.987 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:27.987 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.987 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.987 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:27.987 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167949+0000", 2024-09-21T15:51:27.987 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.0200000000000001e-06 2024-09-21T15:51:27.987 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.987 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.988 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:27.988 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167955+0000", 2024-09-21T15:51:27.988 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5990000000000001e-06 2024-09-21T15:51:27.988 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.988 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.988 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:27.988 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167959+0000", 2024-09-21T15:51:27.988 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5589999999999999e-06 2024-09-21T15:51:27.988 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.988 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.988 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.988 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.216972+0000", 2024-09-21T15:51:27.988 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049012574000000003 2024-09-21T15:51:27.988 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.988 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.989 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:27.989 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217057+0000", 2024-09-21T15:51:27.989 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.4820000000000004e-05 2024-09-21T15:51:27.989 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.989 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.989 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:27.989 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217129+0000", 2024-09-21T15:51:27.989 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2186000000000003e-05 2024-09-21T15:51:27.989 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.989 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.989 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.989 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.989 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.989 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64042 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1804288~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:27.989 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.167964+0000", 2024-09-21T15:51:27.989 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 110.78135825299999, 2024-09-21T15:51:27.990 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049333372, 2024-09-21T15:51:27.990 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:27.990 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:27.990 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:27.990 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:27.990 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:27.990 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64042 2024-09-21T15:51:27.990 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.990 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:27.990 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.990 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:27.990 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167964+0000", 2024-09-21T15:51:27.990 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.990 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.990 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.991 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:27.991 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167964+0000", 2024-09-21T15:51:27.991 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.991 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.991 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.991 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:27.991 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167963+0000", 2024-09-21T15:51:27.991 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:27.991 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.991 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.991 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:27.991 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167966+0000", 2024-09-21T15:51:27.991 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1119999999999999e-06 2024-09-21T15:51:27.991 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.991 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.991 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:27.992 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167975+0000", 2024-09-21T15:51:27.992 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.1130000000000007e-06 2024-09-21T15:51:27.992 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.992 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.992 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:27.992 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.167980+0000", 2024-09-21T15:51:27.992 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.9599999999999999e-06 2024-09-21T15:51:27.992 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.992 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.992 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.992 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217158+0000", 2024-09-21T15:51:27.992 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049178569999999998 2024-09-21T15:51:27.992 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.992 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.992 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:27.992 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217232+0000", 2024-09-21T15:51:27.993 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3589000000000003e-05 2024-09-21T15:51:27.993 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.993 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.993 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:27.993 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217297+0000", 2024-09-21T15:51:27.993 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5339999999999994e-05 2024-09-21T15:51:27.993 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.993 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.993 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.993 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.993 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.993 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:64043 3.13 3:c9666de2:::rbd_data.11336b2fdf4a.0000000000000006:c [read 1812480~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e47)", 2024-09-21T15:51:27.993 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:49:37.168085+0000", 2024-09-21T15:51:27.993 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 110.781237186, 2024-09-21T15:51:27.993 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049365157999999999, 2024-09-21T15:51:27.994 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:27.994 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:27.994 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:27.994 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:27.994 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:27.994 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 64043 2024-09-21T15:51:27.994 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.994 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:27.994 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.994 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:27.994 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168085+0000", 2024-09-21T15:51:27.994 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.994 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.994 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.994 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:27.995 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168085+0000", 2024-09-21T15:51:27.995 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.995 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.995 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.995 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:27.995 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168084+0000", 2024-09-21T15:51:27.995 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:27.995 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.995 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.995 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:27.995 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168088+0000", 2024-09-21T15:51:27.995 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 3.5420000000000001e-06 2024-09-21T15:51:27.995 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.995 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.995 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:27.996 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168097+0000", 2024-09-21T15:51:27.996 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 9.5410000000000006e-06 2024-09-21T15:51:27.996 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.996 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.996 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:27.996 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.168104+0000", 2024-09-21T15:51:27.996 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6719999999999998e-06 2024-09-21T15:51:27.996 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.996 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.996 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:27.996 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217323+0000", 2024-09-21T15:51:27.996 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049218857999999997 2024-09-21T15:51:27.996 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.996 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.997 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:27.997 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217397+0000", 2024-09-21T15:51:27.997 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3932999999999998e-05 2024-09-21T15:51:27.997 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.997 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.997 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:27.997 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:49:37.217450+0000", 2024-09-21T15:51:27.997 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3168000000000003e-05 2024-09-21T15:51:27.997 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.997 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:27.997 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:27.997 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.997 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.997 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155255 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 323584~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:27.997 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.440996+0000", 2024-09-21T15:51:27.997 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 65.508325658999993, 2024-09-21T15:51:27.998 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048233848000000003, 2024-09-21T15:51:27.998 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:27.998 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:27.998 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:27.998 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:27.998 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:27.998 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155255 2024-09-21T15:51:27.998 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.998 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:27.998 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.998 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:27.998 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.440996+0000", 2024-09-21T15:51:27.998 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.998 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.998 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.999 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:27.999 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.440996+0000", 2024-09-21T15:51:27.999 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:27.999 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.999 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.999 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:27.999 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.440996+0000", 2024-09-21T15:51:27.999 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:27.999 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.999 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:27.999 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:27.999 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441004+0000", 2024-09-21T15:51:27.999 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.8460000000000008e-06 2024-09-21T15:51:27.999 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:27.999 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.000 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:28.000 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441018+0000", 2024-09-21T15:51:28.000 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.42e-05 2024-09-21T15:51:28.000 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.000 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.000 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:28.000 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441023+0000", 2024-09-21T15:51:28.000 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.0019999999999997e-06 2024-09-21T15:51:28.000 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.000 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.000 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:28.000 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489060+0000", 2024-09-21T15:51:28.000 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048036660000000002 2024-09-21T15:51:28.000 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.000 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.000 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:28.001 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489107+0000", 2024-09-21T15:51:28.001 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.7153000000000001e-05 2024-09-21T15:51:28.001 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.001 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.001 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:28.001 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489230+0000", 2024-09-21T15:51:28.001 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00012328600000000001 2024-09-21T15:51:28.001 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:28.001 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:28.001 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:28.001 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.001 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.001 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155256 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 356352~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:28.001 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441027+0000", 2024-09-21T15:51:28.001 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 65.508294571999997, 2024-09-21T15:51:28.001 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048677606999999998, 2024-09-21T15:51:28.002 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:28.002 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:28.002 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:28.002 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:28.002 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:28.002 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155256 2024-09-21T15:51:28.002 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.002 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:28.002 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.002 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:28.002 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441027+0000", 2024-09-21T15:51:28.002 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:28.002 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.002 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.002 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:28.003 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441027+0000", 2024-09-21T15:51:28.003 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:28.003 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.003 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.003 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:28.003 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441027+0000", 2024-09-21T15:51:28.003 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:28.003 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.003 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.003 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:28.003 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441029+0000", 2024-09-21T15:51:28.003 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.39e-06 2024-09-21T15:51:28.003 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.003 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.003 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:28.003 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441036+0000", 2024-09-21T15:51:28.004 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.1129999999999997e-06 2024-09-21T15:51:28.004 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.004 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.004 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:28.004 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441040+0000", 2024-09-21T15:51:28.004 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.8459999999999999e-06 2024-09-21T15:51:28.004 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.004 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.004 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:28.004 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489244+0000", 2024-09-21T15:51:28.004 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048203751000000003 2024-09-21T15:51:28.004 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.004 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.004 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:28.004 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489288+0000", 2024-09-21T15:51:28.005 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4153999999999998e-05 2024-09-21T15:51:28.005 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.005 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.005 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:28.005 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489705+0000", 2024-09-21T15:51:28.005 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000416773 2024-09-21T15:51:28.005 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:28.005 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:28.005 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:28.005 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.005 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.005 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155257 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 389120~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:28.005 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441045+0000", 2024-09-21T15:51:28.005 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 65.508277480999993, 2024-09-21T15:51:28.006 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049000325999999997, 2024-09-21T15:51:28.006 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:28.006 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:28.006 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:28.006 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:28.006 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:28.006 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155257 2024-09-21T15:51:28.006 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.006 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:28.006 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.006 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:28.006 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441045+0000", 2024-09-21T15:51:28.006 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:28.007 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.007 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.007 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:28.007 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441045+0000", 2024-09-21T15:51:28.007 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:28.007 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.007 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.007 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:28.007 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441044+0000", 2024-09-21T15:51:28.007 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:28.007 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.007 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.007 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:28.007 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441046+0000", 2024-09-21T15:51:28.007 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1909999999999999e-06 2024-09-21T15:51:28.008 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.008 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.008 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:28.008 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441054+0000", 2024-09-21T15:51:28.008 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.3900000000000004e-06 2024-09-21T15:51:28.008 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.008 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.008 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:28.008 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441058+0000", 2024-09-21T15:51:28.008 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4619999999999998e-06 2024-09-21T15:51:28.008 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.008 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.008 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:28.008 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489741+0000", 2024-09-21T15:51:28.008 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.048682660000000003 2024-09-21T15:51:28.009 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.009 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.009 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:28.009 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.489813+0000", 2024-09-21T15:51:28.009 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2051999999999994e-05 2024-09-21T15:51:28.009 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.009 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.009 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:28.009 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490045+0000", 2024-09-21T15:51:28.009 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00023191299999999999 2024-09-21T15:51:28.009 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:28.009 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:28.009 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:28.009 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.009 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.010 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155258 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 421888~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:28.010 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441062+0000", 2024-09-21T15:51:28.010 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 65.508259938999998, 2024-09-21T15:51:28.010 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049339651999999998, 2024-09-21T15:51:28.010 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:28.010 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:28.010 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:28.010 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:28.010 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:28.010 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155258 2024-09-21T15:51:28.010 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.010 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:28.010 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.010 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:28.010 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441062+0000", 2024-09-21T15:51:28.010 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:28.011 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.011 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.011 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:28.011 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441062+0000", 2024-09-21T15:51:28.011 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:28.011 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.011 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.011 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:28.011 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441062+0000", 2024-09-21T15:51:28.011 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:28.011 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.011 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.011 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:28.011 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441064+0000", 2024-09-21T15:51:28.011 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.967e-06 2024-09-21T15:51:28.011 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.012 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.012 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:28.012 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441069+0000", 2024-09-21T15:51:28.012 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1529999999999996e-06 2024-09-21T15:51:28.012 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.012 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.012 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:28.012 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441073+0000", 2024-09-21T15:51:28.012 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.3850000000000004e-06 2024-09-21T15:51:28.012 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.012 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.012 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:28.012 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490082+0000", 2024-09-21T15:51:28.012 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.04900885 2024-09-21T15:51:28.012 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.013 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.013 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:28.013 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490166+0000", 2024-09-21T15:51:28.013 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.3640999999999994e-05 2024-09-21T15:51:28.013 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.013 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.013 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:28.013 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490402+0000", 2024-09-21T15:51:28.013 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000235978 2024-09-21T15:51:28.013 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:28.013 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:28.013 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:28.013 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.013 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.013 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155259 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 454656~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:28.014 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441086+0000", 2024-09-21T15:51:28.014 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 65.508236134000001, 2024-09-21T15:51:28.014 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049624397000000001, 2024-09-21T15:51:28.014 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:28.014 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:28.014 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:28.014 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:28.014 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:28.014 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155259 2024-09-21T15:51:28.014 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.014 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:28.014 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.014 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:28.014 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441086+0000", 2024-09-21T15:51:28.014 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:28.015 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.015 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.015 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:28.015 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441086+0000", 2024-09-21T15:51:28.015 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:28.015 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.015 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.015 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:28.015 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441086+0000", 2024-09-21T15:51:28.015 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:28.015 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.015 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.015 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:28.015 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441088+0000", 2024-09-21T15:51:28.015 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.2630000000000002e-06 2024-09-21T15:51:28.016 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.016 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.016 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:28.016 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441093+0000", 2024-09-21T15:51:28.016 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.5010000000000004e-06 2024-09-21T15:51:28.016 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.016 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.016 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:28.016 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441098+0000", 2024-09-21T15:51:28.016 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.707e-06 2024-09-21T15:51:28.016 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.016 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.016 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:28.016 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490447+0000", 2024-09-21T15:51:28.016 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049349125000000001 2024-09-21T15:51:28.017 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.017 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.017 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:28.017 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490504+0000", 2024-09-21T15:51:28.017 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.6521000000000003e-05 2024-09-21T15:51:28.017 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.017 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.017 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:28.017 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490710+0000", 2024-09-21T15:51:28.017 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00020663999999999999 2024-09-21T15:51:28.017 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:28.017 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:28.017 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:28.017 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.017 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.017 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155260 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 487424~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:28.018 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441160+0000", 2024-09-21T15:51:28.018 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 65.508162171999999, 2024-09-21T15:51:28.018 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049876810000000001, 2024-09-21T15:51:28.018 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:28.018 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:28.018 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:28.018 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:28.018 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:28.018 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155260 2024-09-21T15:51:28.018 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.018 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:28.018 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.018 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:28.018 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441160+0000", 2024-09-21T15:51:28.018 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:28.019 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.019 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.019 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:28.019 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441160+0000", 2024-09-21T15:51:28.019 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:28.019 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.019 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.019 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:28.019 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441159+0000", 2024-09-21T15:51:28.019 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:28.019 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.019 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.019 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:28.019 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441164+0000", 2024-09-21T15:51:28.020 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4059999999999998e-06 2024-09-21T15:51:28.020 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.020 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.020 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:28.020 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441175+0000", 2024-09-21T15:51:28.020 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 1.1095e-05 2024-09-21T15:51:28.020 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.020 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.020 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:28.020 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441182+0000", 2024-09-21T15:51:28.020 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.2139999999999999e-06 2024-09-21T15:51:28.020 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.020 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.020 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:28.020 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490751+0000", 2024-09-21T15:51:28.021 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049569230999999998 2024-09-21T15:51:28.021 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.021 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.021 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:28.021 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.490837+0000", 2024-09-21T15:51:28.021 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 8.6017e-05 2024-09-21T15:51:28.021 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.021 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.021 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:28.021 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491037+0000", 2024-09-21T15:51:28.021 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.00019952300000000001 2024-09-21T15:51:28.021 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:28.021 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:28.021 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:28.021 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.021 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.022 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155261 3.11 3:8a36cf12:::rbd_data.11331b06c8a5.0000000000000004:2b [read 520192~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:28.022 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441187+0000", 2024-09-21T15:51:28.022 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 65.508134988999998, 2024-09-21T15:51:28.022 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.050124931999999997, 2024-09-21T15:51:28.022 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:28.022 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:28.022 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:28.022 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:28.022 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:28.022 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155261 2024-09-21T15:51:28.022 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.022 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:28.022 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.022 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:28.022 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441187+0000", 2024-09-21T15:51:28.022 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:28.023 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.023 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.023 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:28.023 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441187+0000", 2024-09-21T15:51:28.023 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:28.023 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.023 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.023 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:28.023 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441187+0000", 2024-09-21T15:51:28.023 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:28.023 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.023 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.023 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:28.023 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441189+0000", 2024-09-21T15:51:28.023 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.3649999999999998e-06 2024-09-21T15:51:28.023 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.024 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.024 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:28.024 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441196+0000", 2024-09-21T15:51:28.024 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5379999999999999e-06 2024-09-21T15:51:28.024 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.024 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.024 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:28.024 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441200+0000", 2024-09-21T15:51:28.024 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4719999999999997e-06 2024-09-21T15:51:28.024 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.024 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.024 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:28.024 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491070+0000", 2024-09-21T15:51:28.024 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049869915000000001 2024-09-21T15:51:28.024 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.024 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.025 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:28.025 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491146+0000", 2024-09-21T15:51:28.025 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 7.6589999999999997e-05 2024-09-21T15:51:28.025 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.025 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.025 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:28.025 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491312+0000", 2024-09-21T15:51:28.025 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.000165475 2024-09-21T15:51:28.025 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:28.025 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:28.025 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:28.025 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.025 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.025 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155308 3.11 3:8b5183e5:::rbd_data.11335aef9442.0000000000000000:2a [read 356352~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:28.025 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441369+0000", 2024-09-21T15:51:28.026 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 65.507953141000002, 2024-09-21T15:51:28.026 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.050090672000000003, 2024-09-21T15:51:28.026 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:28.026 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:28.026 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:28.026 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:28.026 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:28.026 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155308 2024-09-21T15:51:28.026 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.026 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:28.026 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.026 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:28.026 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441369+0000", 2024-09-21T15:51:28.026 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:28.026 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.027 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.027 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:28.027 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441369+0000", 2024-09-21T15:51:28.027 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:28.027 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.027 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.027 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:28.027 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441368+0000", 2024-09-21T15:51:28.027 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:28.027 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.027 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.027 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:28.027 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441371+0000", 2024-09-21T15:51:28.027 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.1100000000000001e-06 2024-09-21T15:51:28.027 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.027 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.028 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:28.028 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441377+0000", 2024-09-21T15:51:28.028 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.5649999999999998e-06 2024-09-21T15:51:28.028 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.028 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.028 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:28.028 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441382+0000", 2024-09-21T15:51:28.028 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.5929999999999999e-06 2024-09-21T15:51:28.028 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.028 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.028 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:28.028 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491349+0000", 2024-09-21T15:51:28.028 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.049967122000000003 2024-09-21T15:51:28.028 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.028 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.029 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:28.029 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491400+0000", 2024-09-21T15:51:28.029 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.1589999999999999e-05 2024-09-21T15:51:28.029 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.029 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.029 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:28.029 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491460+0000", 2024-09-21T15:51:28.029 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.9070000000000002e-05 2024-09-21T15:51:28.029 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:28.029 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:28.029 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:28.029 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.029 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.029 INFO:teuthology.orchestra.run.smithi100.stdout: "description": "osd_op(client.4403.0:155309 3.11 3:8b5183e5:::rbd_data.11335aef9442.0000000000000000:2a [read 389120~4096] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e86)", 2024-09-21T15:51:28.030 INFO:teuthology.orchestra.run.smithi100.stdout: "initiated_at": "2024-09-21T15:50:22.441386+0000", 2024-09-21T15:51:28.030 INFO:teuthology.orchestra.run.smithi100.stdout: "age": 65.507935791999998, 2024-09-21T15:51:28.030 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.050184777, 2024-09-21T15:51:28.030 INFO:teuthology.orchestra.run.smithi100.stdout: "type_data": { 2024-09-21T15:51:28.030 INFO:teuthology.orchestra.run.smithi100.stdout: "flag_point": "started", 2024-09-21T15:51:28.030 INFO:teuthology.orchestra.run.smithi100.stdout: "client_info": { 2024-09-21T15:51:28.030 INFO:teuthology.orchestra.run.smithi100.stdout: "client": "client.4403", 2024-09-21T15:51:28.030 INFO:teuthology.orchestra.run.smithi100.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:28.030 INFO:teuthology.orchestra.run.smithi100.stdout: "tid": 155309 2024-09-21T15:51:28.030 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.030 INFO:teuthology.orchestra.run.smithi100.stdout: "events": [ 2024-09-21T15:51:28.030 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.030 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "initiated", 2024-09-21T15:51:28.030 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441386+0000", 2024-09-21T15:51:28.030 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:28.031 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.031 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.031 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "throttled", 2024-09-21T15:51:28.031 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441386+0000", 2024-09-21T15:51:28.031 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0 2024-09-21T15:51:28.031 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.031 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.031 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "header_read", 2024-09-21T15:51:28.031 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441386+0000", 2024-09-21T15:51:28.031 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:28.031 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.031 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.031 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "all_read", 2024-09-21T15:51:28.031 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441388+0000", 2024-09-21T15:51:28.031 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 2.4420000000000001e-06 2024-09-21T15:51:28.031 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.032 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.032 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "dispatched", 2024-09-21T15:51:28.032 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441395+0000", 2024-09-21T15:51:28.032 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 6.6189999999999996e-06 2024-09-21T15:51:28.032 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.032 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.032 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "queued_for_pg", 2024-09-21T15:51:28.032 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.441400+0000", 2024-09-21T15:51:28.032 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.118e-06 2024-09-21T15:51:28.032 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.032 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.032 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "reached_pg", 2024-09-21T15:51:28.032 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491473+0000", 2024-09-21T15:51:28.032 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 0.050072783000000003 2024-09-21T15:51:28.032 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.032 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.033 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "started", 2024-09-21T15:51:28.033 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491518+0000", 2024-09-21T15:51:28.033 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 4.4854000000000001e-05 2024-09-21T15:51:28.033 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-09-21T15:51:28.033 INFO:teuthology.orchestra.run.smithi100.stdout: { 2024-09-21T15:51:28.033 INFO:teuthology.orchestra.run.smithi100.stdout: "event": "done", 2024-09-21T15:51:28.033 INFO:teuthology.orchestra.run.smithi100.stdout: "time": "2024-09-21T15:50:22.491571+0000", 2024-09-21T15:51:28.033 INFO:teuthology.orchestra.run.smithi100.stdout: "duration": 5.3358e-05 2024-09-21T15:51:28.033 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:28.033 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:28.033 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:28.033 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-09-21T15:51:28.033 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-09-21T15:51:28.033 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-09-21T15:51:28.034 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2024-09-21T15:51:28.137 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:28.137 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:51:28.137 INFO:teuthology.orchestra.run.smithi089.stdout: "num_ops": 0 2024-09-21T15:51:28.137 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:28.146 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_blocked_ops 2024-09-21T15:51:28.246 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:28.247 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [], 2024-09-21T15:51:28.247 INFO:teuthology.orchestra.run.smithi089.stdout: "complaint_time": 30, 2024-09-21T15:51:28.247 INFO:teuthology.orchestra.run.smithi089.stdout: "num_blocked_ops": 0 2024-09-21T15:51:28.247 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:28.255 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_historic_ops 2024-09-21T15:51:28.358 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:28.358 INFO:teuthology.orchestra.run.smithi089.stdout: "size": 20, 2024-09-21T15:51:28.358 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 600, 2024-09-21T15:51:28.359 INFO:teuthology.orchestra.run.smithi089.stdout: "ops": [ 2024-09-21T15:51:28.359 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.359 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160505 3.b 3:d297695c:::rbd_header.11338bae974a:head [watch reconnect cookie 140381478598448 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-21T15:51:28.359 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:29.138810+0000", 2024-09-21T15:51:28.359 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 59.217353246000002, 2024-09-21T15:51:28.359 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0172598319999999, 2024-09-21T15:51:28.359 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:28.359 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:28.359 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:28.359 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:28.359 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:28.359 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160505 2024-09-21T15:51:28.359 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.360 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:28.360 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.360 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:28.360 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138810+0000", 2024-09-21T15:51:28.360 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:28.360 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.360 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.360 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:28.360 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138810+0000", 2024-09-21T15:51:28.360 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:28.360 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.360 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.360 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:28.360 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138808+0000", 2024-09-21T15:51:28.361 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:51:28.361 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.361 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.361 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:28.361 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138815+0000", 2024-09-21T15:51:28.361 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6549999999999998e-06 2024-09-21T15:51:28.361 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.361 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.361 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:28.361 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138825+0000", 2024-09-21T15:51:28.361 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.9790000000000005e-06 2024-09-21T15:51:28.361 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.361 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.362 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:28.362 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138834+0000", 2024-09-21T15:51:28.362 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5179999999999993e-06 2024-09-21T15:51:28.362 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.362 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.362 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:28.362 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.140973+0000", 2024-09-21T15:51:28.362 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0021392049999999999 2024-09-21T15:51:28.362 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.362 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.362 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:51:28.362 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.140981+0000", 2024-09-21T15:51:28.362 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8909999999999995e-06 2024-09-21T15:51:28.362 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.363 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.363 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:28.363 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.150457+0000", 2024-09-21T15:51:28.363 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.009475645 2024-09-21T15:51:28.363 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.363 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.363 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for missing object", 2024-09-21T15:51:28.363 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.150589+0000", 2024-09-21T15:51:28.363 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000131779 2024-09-21T15:51:28.363 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.363 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.363 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:28.363 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.151723+0000", 2024-09-21T15:51:28.363 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0011345979999999999 2024-09-21T15:51:28.364 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.364 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.364 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-09-21T15:51:28.364 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.151872+0000", 2024-09-21T15:51:28.364 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000149021 2024-09-21T15:51:28.364 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.364 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.364 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:28.364 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.152563+0000", 2024-09-21T15:51:28.364 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00069051500000000005 2024-09-21T15:51:28.364 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.364 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.364 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:28.364 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.152673+0000", 2024-09-21T15:51:28.365 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000109795 2024-09-21T15:51:28.365 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.365 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.365 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:28.365 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.156069+0000", 2024-09-21T15:51:28.365 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003396736 2024-09-21T15:51:28.365 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:28.365 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:28.365 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:28.365 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.365 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.365 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160506 3.b 3:d297695c:::rbd_header.11338bae974a:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-21T15:51:28.365 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:29.138844+0000", 2024-09-21T15:51:28.365 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 59.217318358999997, 2024-09-21T15:51:28.366 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0174592680000001, 2024-09-21T15:51:28.366 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:28.366 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:28.366 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:28.366 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:28.366 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:28.366 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160506 2024-09-21T15:51:28.366 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.366 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:28.366 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.366 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:28.366 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138844+0000", 2024-09-21T15:51:28.366 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:28.366 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.367 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.367 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:28.367 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138844+0000", 2024-09-21T15:51:28.367 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:28.367 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.367 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.367 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:28.367 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138844+0000", 2024-09-21T15:51:28.367 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:28.367 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.367 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.367 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:28.367 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138848+0000", 2024-09-21T15:51:28.367 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.76e-06 2024-09-21T15:51:28.367 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.368 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.368 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:28.368 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138854+0000", 2024-09-21T15:51:28.368 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.3890000000000002e-06 2024-09-21T15:51:28.368 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.368 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.368 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:28.368 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.138860+0000", 2024-09-21T15:51:28.368 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6409999999999999e-06 2024-09-21T15:51:28.368 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.368 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.368 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:28.368 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.141015+0000", 2024-09-21T15:51:28.368 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0021553779999999999 2024-09-21T15:51:28.368 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.369 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.369 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for peered", 2024-09-21T15:51:28.369 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:29.141021+0000", 2024-09-21T15:51:28.369 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7629999999999997e-06 2024-09-21T15:51:28.369 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.369 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.369 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:28.369 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.150858+0000", 2024-09-21T15:51:28.369 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0098372200000001 2024-09-21T15:51:28.369 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.369 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.369 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for missing object", 2024-09-21T15:51:28.369 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.150899+0000", 2024-09-21T15:51:28.369 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1112000000000002e-05 2024-09-21T15:51:28.369 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.369 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.370 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:28.370 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.152069+0000", 2024-09-21T15:51:28.370 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001170307 2024-09-21T15:51:28.370 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.370 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.370 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-09-21T15:51:28.370 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.152192+0000", 2024-09-21T15:51:28.370 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00012289700000000001 2024-09-21T15:51:28.370 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.370 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.370 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:28.370 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.153362+0000", 2024-09-21T15:51:28.370 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0011700549999999999 2024-09-21T15:51:28.370 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.370 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.370 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:28.371 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.153457+0000", 2024-09-21T15:51:28.371 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4687999999999999e-05 2024-09-21T15:51:28.371 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.371 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.371 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:51:28.371 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.153893+0000", 2024-09-21T15:51:28.371 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000436363 2024-09-21T15:51:28.371 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.371 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.371 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:28.371 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.156134+0000", 2024-09-21T15:51:28.371 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0022410009999999998 2024-09-21T15:51:28.371 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.371 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.371 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:28.372 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.156148+0000", 2024-09-21T15:51:28.372 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.3574000000000001e-05 2024-09-21T15:51:28.372 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.372 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.372 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:28.372 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.156214+0000", 2024-09-21T15:51:28.372 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6438000000000003e-05 2024-09-21T15:51:28.372 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.372 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.372 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:28.372 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.156304+0000", 2024-09-21T15:51:28.372 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9278999999999997e-05 2024-09-21T15:51:28.372 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:28.372 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:28.372 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:28.373 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.373 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.373 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160512 3.b 3:d297695c:::rbd_header.11338bae974a:head [watch unwatch cookie 140381478598448] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T15:51:28.373 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:30.161362+0000", 2024-09-21T15:51:28.373 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 58.194800551, 2024-09-21T15:51:28.373 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0074105029999999997, 2024-09-21T15:51:28.373 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:28.373 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:28.373 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:28.373 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:28.373 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:28.373 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160512 2024-09-21T15:51:28.373 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.373 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:28.374 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.374 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:28.374 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161362+0000", 2024-09-21T15:51:28.374 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:28.374 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.374 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.374 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:28.374 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161362+0000", 2024-09-21T15:51:28.374 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:28.374 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.374 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.374 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:28.374 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161362+0000", 2024-09-21T15:51:28.374 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:28.374 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.375 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.375 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:28.375 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161365+0000", 2024-09-21T15:51:28.375 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8669999999999998e-06 2024-09-21T15:51:28.375 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.375 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.375 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:28.375 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161371+0000", 2024-09-21T15:51:28.375 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8009999999999998e-06 2024-09-21T15:51:28.375 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.375 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.375 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:28.375 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161376+0000", 2024-09-21T15:51:28.375 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.0340000000000004e-06 2024-09-21T15:51:28.375 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.376 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.376 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:28.376 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161762+0000", 2024-09-21T15:51:28.376 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038575099999999999 2024-09-21T15:51:28.376 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.376 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.376 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:28.376 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161838+0000", 2024-09-21T15:51:28.376 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5697000000000004e-05 2024-09-21T15:51:28.376 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.376 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.376 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:51:28.376 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161952+0000", 2024-09-21T15:51:28.376 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000114034 2024-09-21T15:51:28.376 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.376 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.377 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:28.377 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.163727+0000", 2024-09-21T15:51:28.377 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0017749560000000001 2024-09-21T15:51:28.377 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.377 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.377 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:28.377 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168545+0000", 2024-09-21T15:51:28.377 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0048186080000000003 2024-09-21T15:51:28.377 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.377 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.377 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:28.377 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168593+0000", 2024-09-21T15:51:28.377 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7549999999999997e-05 2024-09-21T15:51:28.377 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.377 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.377 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:28.378 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168773+0000", 2024-09-21T15:51:28.378 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00017976600000000001 2024-09-21T15:51:28.378 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:28.378 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:28.378 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:28.378 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.378 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.378 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160513 3.b 3:d297695c:::rbd_header.11338bae974a:head [notify-ack] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-09-21T15:51:28.378 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:30.161755+0000", 2024-09-21T15:51:28.378 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 58.19440762, 2024-09-21T15:51:28.378 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0072250300000000003, 2024-09-21T15:51:28.378 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:28.378 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:28.378 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:28.378 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:28.379 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:28.379 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160513 2024-09-21T15:51:28.379 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.379 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:28.379 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.379 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:28.379 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161755+0000", 2024-09-21T15:51:28.379 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:28.379 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.379 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.379 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:28.379 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161755+0000", 2024-09-21T15:51:28.379 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:28.379 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.380 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.380 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:28.380 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161754+0000", 2024-09-21T15:51:28.380 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:28.380 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.380 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.380 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:28.380 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161759+0000", 2024-09-21T15:51:28.380 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.9699999999999998e-06 2024-09-21T15:51:28.380 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.380 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.380 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:28.380 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161768+0000", 2024-09-21T15:51:28.380 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.9949999999999994e-06 2024-09-21T15:51:28.380 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.380 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.381 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:28.381 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.161774+0000", 2024-09-21T15:51:28.381 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.1290000000000001e-06 2024-09-21T15:51:28.381 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.381 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.381 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:28.381 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.162239+0000", 2024-09-21T15:51:28.381 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00046406600000000001 2024-09-21T15:51:28.381 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.381 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.381 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for rw locks", 2024-09-21T15:51:28.381 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.162296+0000", 2024-09-21T15:51:28.381 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7122e-05 2024-09-21T15:51:28.381 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.381 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.381 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:28.382 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168802+0000", 2024-09-21T15:51:28.382 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0065061700000000004 2024-09-21T15:51:28.382 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.382 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.382 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:28.382 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168877+0000", 2024-09-21T15:51:28.382 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5214000000000002e-05 2024-09-21T15:51:28.382 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.382 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.382 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:28.382 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.168980+0000", 2024-09-21T15:51:28.382 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000103182 2024-09-21T15:51:28.382 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:28.382 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:28.382 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:28.383 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.383 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.383 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160527 3.b 3:d297695c:::rbd_header.11338bae974a:head [watch watch cookie 140381476148256] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T15:51:28.383 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:30.182756+0000", 2024-09-21T15:51:28.383 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 58.173407038999997, 2024-09-21T15:51:28.383 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0054202579999999998, 2024-09-21T15:51:28.383 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:28.383 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:28.383 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:28.383 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:28.383 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:28.383 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160527 2024-09-21T15:51:28.383 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.383 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:28.383 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.383 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:28.384 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182756+0000", 2024-09-21T15:51:28.384 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:28.384 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.384 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.384 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:28.384 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182756+0000", 2024-09-21T15:51:28.384 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:28.384 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.384 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.384 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:28.384 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182755+0000", 2024-09-21T15:51:28.384 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:28.384 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.384 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.385 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:28.385 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182760+0000", 2024-09-21T15:51:28.385 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.4269999999999999e-06 2024-09-21T15:51:28.385 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.385 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.385 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:28.385 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182771+0000", 2024-09-21T15:51:28.385 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0312999999999999e-05 2024-09-21T15:51:28.385 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.385 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.385 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:28.385 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.182778+0000", 2024-09-21T15:51:28.385 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5700000000000004e-06 2024-09-21T15:51:28.385 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.385 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.385 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:28.386 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.184745+0000", 2024-09-21T15:51:28.386 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0019667170000000002 2024-09-21T15:51:28.386 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.386 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.386 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:28.386 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.184867+0000", 2024-09-21T15:51:28.386 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000122504 2024-09-21T15:51:28.386 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.386 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.386 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:51:28.386 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.185135+0000", 2024-09-21T15:51:28.386 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000267135 2024-09-21T15:51:28.386 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.386 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.386 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:28.387 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.187721+0000", 2024-09-21T15:51:28.387 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.002586891 2024-09-21T15:51:28.387 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.387 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.387 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:28.387 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.187874+0000", 2024-09-21T15:51:28.387 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000152726 2024-09-21T15:51:28.387 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.387 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.387 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:28.387 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.187951+0000", 2024-09-21T15:51:28.387 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.7132000000000006e-05 2024-09-21T15:51:28.387 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.387 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.387 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:28.387 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:30.188176+0000", 2024-09-21T15:51:28.388 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00022477900000000001 2024-09-21T15:51:28.388 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:28.388 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:28.388 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:28.388 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.388 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.388 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:160608 3.12 e95/94 3:48a47e1c:::rbd_data.11338bae974a.000000000000000b:head v 95'345, mlcod=0'0)", 2024-09-21T15:51:28.388 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.216934+0000", 2024-09-21T15:51:28.388 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 57.139228901000003, 2024-09-21T15:51:28.388 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053335759999999996, 2024-09-21T15:51:28.388 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:28.388 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:28.388 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:28.388 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.388 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:28.388 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216934+0000", 2024-09-21T15:51:28.389 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:28.389 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.389 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.389 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:28.389 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216934+0000", 2024-09-21T15:51:28.389 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:28.389 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.389 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.389 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:28.389 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216933+0000", 2024-09-21T15:51:28.389 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:28.389 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.389 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.389 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:28.389 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216939+0000", 2024-09-21T15:51:28.390 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6459999999999998e-06 2024-09-21T15:51:28.390 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.390 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.390 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:28.390 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216947+0000", 2024-09-21T15:51:28.390 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.2479999999999996e-06 2024-09-21T15:51:28.390 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.390 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.390 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:28.390 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216951+0000", 2024-09-21T15:51:28.390 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3760000000000001e-06 2024-09-21T15:51:28.390 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.390 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.390 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:28.390 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.216998+0000", 2024-09-21T15:51:28.390 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6244e-05 2024-09-21T15:51:28.391 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.391 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.391 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:28.391 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.217019+0000", 2024-09-21T15:51:28.391 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1191000000000001e-05 2024-09-21T15:51:28.391 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.391 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.391 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:28.391 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.222230+0000", 2024-09-21T15:51:28.391 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0052114350000000004 2024-09-21T15:51:28.391 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.391 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.391 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:28.391 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.222267+0000", 2024-09-21T15:51:28.392 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7169999999999998e-05 2024-09-21T15:51:28.392 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:28.392 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:28.392 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:28.392 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.392 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.392 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:160662 3.b 3:d7701915:::rbd_object_map.1133fd7483d4.000000000000002c:head [delete] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-21T15:51:28.392 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:31.279369+0000", 2024-09-21T15:51:28.392 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 57.076794133999996, 2024-09-21T15:51:28.392 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0056948040000000004, 2024-09-21T15:51:28.392 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:28.392 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:28.392 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:28.392 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:28.392 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:28.393 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 160662 2024-09-21T15:51:28.393 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.393 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:28.393 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.393 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:28.393 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279369+0000", 2024-09-21T15:51:28.393 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:28.393 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.393 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.393 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:28.393 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279369+0000", 2024-09-21T15:51:28.393 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:28.393 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.393 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.393 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:28.393 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279368+0000", 2024-09-21T15:51:28.394 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:28.394 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.394 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.394 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:28.394 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279373+0000", 2024-09-21T15:51:28.394 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.6529999999999999e-06 2024-09-21T15:51:28.394 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.394 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.394 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:28.394 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279384+0000", 2024-09-21T15:51:28.394 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0256e-05 2024-09-21T15:51:28.394 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.394 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.394 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:28.394 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279388+0000", 2024-09-21T15:51:28.395 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.5630000000000002e-06 2024-09-21T15:51:28.395 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.395 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.395 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:28.395 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279424+0000", 2024-09-21T15:51:28.395 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5985999999999997e-05 2024-09-21T15:51:28.395 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.395 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.395 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:28.395 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279478+0000", 2024-09-21T15:51:28.395 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.3944999999999999e-05 2024-09-21T15:51:28.395 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.395 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.395 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:51:28.395 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.279575+0000", 2024-09-21T15:51:28.395 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.6817999999999994e-05 2024-09-21T15:51:28.396 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.396 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.396 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:28.396 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.282391+0000", 2024-09-21T15:51:28.396 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0028156700000000002 2024-09-21T15:51:28.396 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.396 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.396 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:28.396 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.284977+0000", 2024-09-21T15:51:28.396 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0025859640000000001 2024-09-21T15:51:28.396 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.396 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.396 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:28.396 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.285021+0000", 2024-09-21T15:51:28.396 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.4415000000000001e-05 2024-09-21T15:51:28.397 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.397 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.397 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:28.397 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:31.285063+0000", 2024-09-21T15:51:28.397 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2447000000000002e-05 2024-09-21T15:51:28.397 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:28.397 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:28.397 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:28.397 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.397 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.397 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:161219 3.17 3:e87cb47d:::rbd_data.11335aef9442.000000000000000a:head [delete] snapc 2a=[] ondisk+write+known_if_redirected+supports_pool_eio e101)", 2024-09-21T15:51:28.397 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:36.771292+0000", 2024-09-21T15:51:28.397 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 51.584870322, 2024-09-21T15:51:28.398 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0058540529999999997, 2024-09-21T15:51:28.398 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:28.398 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:28.398 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:28.398 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:28.398 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:28.398 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 161219 2024-09-21T15:51:28.398 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.398 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:28.398 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.398 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:28.398 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771292+0000", 2024-09-21T15:51:28.398 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:28.399 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.399 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.399 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:28.399 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771292+0000", 2024-09-21T15:51:28.399 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:28.399 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.399 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.399 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:28.399 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771292+0000", 2024-09-21T15:51:28.399 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:28.399 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.399 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.399 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:28.399 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771298+0000", 2024-09-21T15:51:28.400 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9519999999999999e-06 2024-09-21T15:51:28.400 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.400 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.400 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:28.400 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771308+0000", 2024-09-21T15:51:28.400 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0601000000000001e-05 2024-09-21T15:51:28.400 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.400 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.400 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:28.400 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771314+0000", 2024-09-21T15:51:28.400 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.913e-06 2024-09-21T15:51:28.400 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.400 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.400 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:28.401 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771346+0000", 2024-09-21T15:51:28.401 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.1711000000000002e-05 2024-09-21T15:51:28.401 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.401 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.401 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:28.401 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771399+0000", 2024-09-21T15:51:28.401 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.2883000000000002e-05 2024-09-21T15:51:28.401 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.401 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.401 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-09-21T15:51:28.401 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.771470+0000", 2024-09-21T15:51:28.401 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.0934999999999998e-05 2024-09-21T15:51:28.401 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.401 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.401 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:28.401 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.772234+0000", 2024-09-21T15:51:28.402 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00076485400000000003 2024-09-21T15:51:28.402 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.402 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.402 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:28.402 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.777083+0000", 2024-09-21T15:51:28.402 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0048486329999999998 2024-09-21T15:51:28.402 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.402 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.402 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:28.402 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.777111+0000", 2024-09-21T15:51:28.402 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8399999999999999e-05 2024-09-21T15:51:28.402 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.402 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.402 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:28.402 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:36.777147+0000", 2024-09-21T15:51:28.403 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5049000000000003e-05 2024-09-21T15:51:28.403 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:28.403 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:28.403 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:28.403 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.403 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.403 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:163005 3.6 3:67cae2ed:::rbd_mirroring:head [call rbd.mirror_image_remove in=22b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e114)", 2024-09-21T15:51:28.403 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:50:49.711037+0000", 2024-09-21T15:51:28.403 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 38.645125442999998, 2024-09-21T15:51:28.403 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0059230919999999996, 2024-09-21T15:51:28.403 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:28.403 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "started", 2024-09-21T15:51:28.403 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:28.403 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:28.403 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:28.403 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 163005 2024-09-21T15:51:28.404 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.404 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:28.404 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.404 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:28.404 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:49.711037+0000", 2024-09-21T15:51:28.404 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:28.404 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.404 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.404 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:28.404 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:49.711037+0000", 2024-09-21T15:51:28.404 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:28.404 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.404 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.404 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:28.404 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:49.711037+0000", 2024-09-21T15:51:28.405 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:28.405 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.405 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.405 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:28.405 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:49.711041+0000", 2024-09-21T15:51:28.405 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8199999999999996e-06 2024-09-21T15:51:28.405 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.405 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.405 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:28.405 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:49.711052+0000", 2024-09-21T15:51:28.405 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0567999999999999e-05 2024-09-21T15:51:28.405 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.405 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.405 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:28.405 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:49.711057+0000", 2024-09-21T15:51:28.405 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7119999999999999e-06 2024-09-21T15:51:28.406 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.406 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.406 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:28.406 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:49.711096+0000", 2024-09-21T15:51:28.406 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8872999999999999e-05 2024-09-21T15:51:28.406 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.406 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.406 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:28.406 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:49.711151+0000", 2024-09-21T15:51:28.406 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.5343999999999997e-05 2024-09-21T15:51:28.406 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.406 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.406 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:28.406 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:50:49.716960+0000", 2024-09-21T15:51:28.406 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0058095150000000003 2024-09-21T15:51:28.407 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:28.407 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:28.407 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:28.407 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.407 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.407 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(client.4403.0:165147 3.10 e131/96 3:0b23e447:::rbd_header.11339e87a32a:head v 131'448, mlcod=130'446)", 2024-09-21T15:51:28.407 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:51:06.827072+0000", 2024-09-21T15:51:28.407 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 21.529090659000001, 2024-09-21T15:51:28.407 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.011683913000000001, 2024-09-21T15:51:28.407 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:28.407 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:28.407 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:28.407 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.407 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:28.407 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.827072+0000", 2024-09-21T15:51:28.407 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:28.408 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.408 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.408 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:28.408 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.827072+0000", 2024-09-21T15:51:28.408 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:28.408 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.408 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.408 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:28.408 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.827072+0000", 2024-09-21T15:51:28.408 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:28.408 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.408 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.408 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:28.408 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.827078+0000", 2024-09-21T15:51:28.408 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4840000000000001e-06 2024-09-21T15:51:28.408 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.409 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.409 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:28.409 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.827094+0000", 2024-09-21T15:51:28.409 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.5639999999999999e-05 2024-09-21T15:51:28.409 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.409 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.409 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:28.409 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.827104+0000", 2024-09-21T15:51:28.409 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.8300000000000008e-06 2024-09-21T15:51:28.409 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.409 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.409 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:28.409 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.827170+0000", 2024-09-21T15:51:28.409 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.5913000000000004e-05 2024-09-21T15:51:28.409 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.410 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.410 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:28.410 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.827196+0000", 2024-09-21T15:51:28.410 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6210000000000001e-05 2024-09-21T15:51:28.410 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.410 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.410 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:28.410 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.838689+0000", 2024-09-21T15:51:28.410 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.011492979 2024-09-21T15:51:28.410 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.410 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.410 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:28.410 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.838756+0000", 2024-09-21T15:51:28.410 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7155000000000003e-05 2024-09-21T15:51:28.410 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:28.411 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:28.411 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:28.411 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.411 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.411 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.0:3709 3.3 e131/96 3:c5f4482f:::rbd_data.11339e87a32a.000000000000013a:12 v 131'841, mlcod=130'838)", 2024-09-21T15:51:28.411 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:51:06.828155+0000", 2024-09-21T15:51:28.411 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 21.528008217, 2024-09-21T15:51:28.411 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.010631115999999999, 2024-09-21T15:51:28.411 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:28.411 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:28.411 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:28.411 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.411 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:28.411 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.828155+0000", 2024-09-21T15:51:28.411 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:28.412 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.412 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.412 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:28.412 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.828155+0000", 2024-09-21T15:51:28.412 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:28.412 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.412 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.412 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:28.412 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.828154+0000", 2024-09-21T15:51:28.412 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:28.412 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.412 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.412 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:28.412 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.828162+0000", 2024-09-21T15:51:28.412 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.2269999999999993e-06 2024-09-21T15:51:28.413 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.413 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.413 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:28.413 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.828188+0000", 2024-09-21T15:51:28.413 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5626000000000002e-05 2024-09-21T15:51:28.413 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.413 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.413 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:28.413 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.828210+0000", 2024-09-21T15:51:28.413 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.1792000000000001e-05 2024-09-21T15:51:28.413 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.413 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.413 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:28.413 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.828251+0000", 2024-09-21T15:51:28.413 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1329999999999999e-05 2024-09-21T15:51:28.413 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.414 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.414 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:28.414 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.828277+0000", 2024-09-21T15:51:28.414 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.6236000000000001e-05 2024-09-21T15:51:28.414 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.414 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.414 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:28.414 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.838756+0000", 2024-09-21T15:51:28.414 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.010478919 2024-09-21T15:51:28.414 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.414 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.414 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:28.414 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.838786+0000", 2024-09-21T15:51:28.414 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.9560000000000002e-05 2024-09-21T15:51:28.415 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:28.415 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:28.415 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:28.415 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.415 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.415 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.0:3710 3.3 e131/96 3:c7ccd1db:::rbd_data.11339e87a32a.0000000000000136:12 v 131'843, mlcod=130'838)", 2024-09-21T15:51:28.415 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:51:06.831274+0000", 2024-09-21T15:51:28.415 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 21.524888366999999, 2024-09-21T15:51:28.415 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0077927079999999998, 2024-09-21T15:51:28.415 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:28.415 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:28.415 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:28.415 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.415 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:28.416 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.831274+0000", 2024-09-21T15:51:28.416 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:28.416 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.416 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.416 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:28.416 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.831274+0000", 2024-09-21T15:51:28.416 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:28.416 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.416 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.416 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:28.416 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.831274+0000", 2024-09-21T15:51:28.416 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:28.416 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.416 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.417 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:28.417 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.831286+0000", 2024-09-21T15:51:28.417 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.1853e-05 2024-09-21T15:51:28.417 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.417 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.417 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:28.417 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.831296+0000", 2024-09-21T15:51:28.417 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0713999999999999e-05 2024-09-21T15:51:28.417 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.417 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.417 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:28.417 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.831305+0000", 2024-09-21T15:51:28.417 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.4749999999999993e-06 2024-09-21T15:51:28.417 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.417 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.418 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:28.418 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.831364+0000", 2024-09-21T15:51:28.418 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.9320999999999997e-05 2024-09-21T15:51:28.418 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.418 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.418 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:28.418 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.831386+0000", 2024-09-21T15:51:28.418 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.2058000000000001e-05 2024-09-21T15:51:28.418 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.418 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.418 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:28.418 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.839039+0000", 2024-09-21T15:51:28.418 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0076530490000000003 2024-09-21T15:51:28.418 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.418 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.419 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:28.419 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:06.839067+0000", 2024-09-21T15:51:28.419 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.8005999999999999e-05 2024-09-21T15:51:28.419 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:28.419 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:28.419 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:28.419 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.419 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.419 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:166306 3.1 3:82f7f77c:::rbd_header.11336b2fdf4a:head [call rbd.snapshot_remove in=18b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e141)", 2024-09-21T15:51:28.419 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:51:16.854113+0000", 2024-09-21T15:51:28.419 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 11.502049601, 2024-09-21T15:51:28.419 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0076607890000000003, 2024-09-21T15:51:28.419 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:28.419 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:28.419 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:28.420 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:28.420 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:28.420 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 166306 2024-09-21T15:51:28.420 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.420 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:28.420 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.420 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:28.420 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.854113+0000", 2024-09-21T15:51:28.420 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:28.420 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.420 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.420 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:28.420 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.854113+0000", 2024-09-21T15:51:28.420 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:28.420 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.421 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.421 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:28.421 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.854111+0000", 2024-09-21T15:51:28.421 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999981 2024-09-21T15:51:28.421 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.421 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.421 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:28.421 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.854120+0000", 2024-09-21T15:51:28.421 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.8410000000000006e-06 2024-09-21T15:51:28.421 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.421 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.421 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:28.421 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.854133+0000", 2024-09-21T15:51:28.421 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.272e-05 2024-09-21T15:51:28.421 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.421 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.422 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:28.422 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.854141+0000", 2024-09-21T15:51:28.422 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.8590000000000005e-06 2024-09-21T15:51:28.422 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.422 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.422 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:28.422 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.854207+0000", 2024-09-21T15:51:28.422 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.6593000000000004e-05 2024-09-21T15:51:28.422 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.422 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.422 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:28.422 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.854308+0000", 2024-09-21T15:51:28.422 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000100592 2024-09-21T15:51:28.422 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.422 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.423 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-09-21T15:51:28.423 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.854716+0000", 2024-09-21T15:51:28.423 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00040786099999999998 2024-09-21T15:51:28.423 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.423 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.423 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:28.423 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.858074+0000", 2024-09-21T15:51:28.423 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00335798 2024-09-21T15:51:28.423 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.423 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.423 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:28.423 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.861691+0000", 2024-09-21T15:51:28.423 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.003617351 2024-09-21T15:51:28.423 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.423 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.424 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:28.424 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.861740+0000", 2024-09-21T15:51:28.424 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8618000000000001e-05 2024-09-21T15:51:28.424 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.424 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.424 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:28.424 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.861774+0000", 2024-09-21T15:51:28.424 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.4313000000000002e-05 2024-09-21T15:51:28.424 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:28.424 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:28.424 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:28.424 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.424 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.424 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_repop(osd.4.0:3832 3.3 e141/96 3:c4381677:::rbd_data.11336b2fdf4a.000000000000000e:c v 141'878, mlcod=140'875)", 2024-09-21T15:51:28.424 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:51:16.856401+0000", 2024-09-21T15:51:28.424 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 11.499761400000001, 2024-09-21T15:51:28.425 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0061803919999999998, 2024-09-21T15:51:28.425 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:28.425 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:28.425 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:28.425 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.425 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:28.425 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.856401+0000", 2024-09-21T15:51:28.425 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:28.425 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.425 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.425 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:28.425 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.856401+0000", 2024-09-21T15:51:28.425 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:28.425 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.425 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.426 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:28.426 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.856400+0000", 2024-09-21T15:51:28.426 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:28.426 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.426 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.426 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:28.426 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.856407+0000", 2024-09-21T15:51:28.426 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.9079999999999998e-06 2024-09-21T15:51:28.426 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.426 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.426 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:28.426 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.856417+0000", 2024-09-21T15:51:28.426 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.5929999999999995e-06 2024-09-21T15:51:28.426 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.426 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.427 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:28.427 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.856421+0000", 2024-09-21T15:51:28.427 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6829999999999999e-06 2024-09-21T15:51:28.427 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.427 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.427 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:28.427 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.856455+0000", 2024-09-21T15:51:28.427 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3145000000000002e-05 2024-09-21T15:51:28.427 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.427 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.427 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:28.427 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.856481+0000", 2024-09-21T15:51:28.427 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.5959999999999999e-05 2024-09-21T15:51:28.427 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.427 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.428 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:28.428 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.862548+0000", 2024-09-21T15:51:28.428 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0060674869999999999 2024-09-21T15:51:28.428 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.428 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.428 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:28.428 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:16.862582+0000", 2024-09-21T15:51:28.428 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.3726000000000002e-05 2024-09-21T15:51:28.428 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:28.428 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:28.428 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:28.428 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.428 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.428 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:166979 3.b 3:d564a535:::rbd_object_map.1133896adcfd:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-21T15:51:28.428 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:51:20.685558+0000", 2024-09-21T15:51:28.428 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 7.6706049949999997, 2024-09-21T15:51:28.429 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0054097210000000001, 2024-09-21T15:51:28.429 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:28.429 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:28.429 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:28.429 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:28.429 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:28.429 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 166979 2024-09-21T15:51:28.429 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.429 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:28.429 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.429 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:28.429 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.685558+0000", 2024-09-21T15:51:28.429 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:28.429 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.429 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.429 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:28.430 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.685558+0000", 2024-09-21T15:51:28.430 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:28.430 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.430 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.430 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:28.430 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.685557+0000", 2024-09-21T15:51:28.430 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999986 2024-09-21T15:51:28.430 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.430 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.430 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:28.430 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.685563+0000", 2024-09-21T15:51:28.430 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.4699999999999999e-06 2024-09-21T15:51:28.430 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.430 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.430 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:28.431 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.685575+0000", 2024-09-21T15:51:28.431 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.2113000000000001e-05 2024-09-21T15:51:28.431 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.431 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.431 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:28.431 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.685582+0000", 2024-09-21T15:51:28.431 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.7830000000000001e-06 2024-09-21T15:51:28.431 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.431 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.431 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:28.431 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.685633+0000", 2024-09-21T15:51:28.431 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1363000000000003e-05 2024-09-21T15:51:28.431 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.431 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.431 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op must wait for map", 2024-09-21T15:51:28.432 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.685643+0000", 2024-09-21T15:51:28.432 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.2310000000000002e-06 2024-09-21T15:51:28.432 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.432 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.432 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:28.432 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.687329+0000", 2024-09-21T15:51:28.432 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001686325 2024-09-21T15:51:28.432 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.432 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.432 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:28.432 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.687398+0000", 2024-09-21T15:51:28.432 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.8666999999999998e-05 2024-09-21T15:51:28.432 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.432 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.432 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 3", 2024-09-21T15:51:28.433 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.687507+0000", 2024-09-21T15:51:28.433 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000109956 2024-09-21T15:51:28.433 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.433 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.433 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:28.433 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.688784+0000", 2024-09-21T15:51:28.433 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0012768899999999999 2024-09-21T15:51:28.433 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.433 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.433 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:28.433 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.690901+0000", 2024-09-21T15:51:28.433 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0021167489999999998 2024-09-21T15:51:28.433 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.433 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.433 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:28.433 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.690943+0000", 2024-09-21T15:51:28.434 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1446999999999998e-05 2024-09-21T15:51:28.434 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.434 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.434 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:28.434 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.690968+0000", 2024-09-21T15:51:28.434 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.4997000000000001e-05 2024-09-21T15:51:28.434 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:28.434 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:28.434 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:28.434 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.434 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.434 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:167071 3.1 3:851adcc3:::rbd_data.1133896adcfd.0000000000000005:head [delete] snapc 8=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-21T15:51:28.434 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:51:20.767069+0000", 2024-09-21T15:51:28.434 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 7.5890939609999997, 2024-09-21T15:51:28.434 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0090679330000000002, 2024-09-21T15:51:28.435 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:28.435 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:28.435 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:28.435 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:28.435 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:28.435 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 167071 2024-09-21T15:51:28.435 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.435 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:28.435 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.435 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:28.435 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.767069+0000", 2024-09-21T15:51:28.435 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:28.435 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.435 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.435 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:28.436 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.767069+0000", 2024-09-21T15:51:28.436 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:28.436 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.436 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.436 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:28.436 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.767068+0000", 2024-09-21T15:51:28.436 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:28.436 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.436 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.436 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:28.436 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.767073+0000", 2024-09-21T15:51:28.436 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.259e-06 2024-09-21T15:51:28.436 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.436 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.436 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:28.436 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.767083+0000", 2024-09-21T15:51:28.437 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0156000000000001e-05 2024-09-21T15:51:28.437 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.437 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.437 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:28.437 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.767089+0000", 2024-09-21T15:51:28.437 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.784e-06 2024-09-21T15:51:28.437 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.437 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.437 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:28.437 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.767152+0000", 2024-09-21T15:51:28.437 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.2675000000000006e-05 2024-09-21T15:51:28.437 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.437 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.437 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:28.437 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.767326+0000", 2024-09-21T15:51:28.437 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000173962 2024-09-21T15:51:28.438 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.438 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.438 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-09-21T15:51:28.438 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.767420+0000", 2024-09-21T15:51:28.438 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.4473000000000002e-05 2024-09-21T15:51:28.438 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.438 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.438 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:28.438 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.769013+0000", 2024-09-21T15:51:28.438 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.001592518 2024-09-21T15:51:28.438 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.438 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.438 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:28.438 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.775716+0000", 2024-09-21T15:51:28.439 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0067037219999999996 2024-09-21T15:51:28.439 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.439 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.439 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:28.439 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.775756+0000", 2024-09-21T15:51:28.439 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9669000000000003e-05 2024-09-21T15:51:28.439 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.439 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.439 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:28.439 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.776137+0000", 2024-09-21T15:51:28.439 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00038067 2024-09-21T15:51:28.439 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:28.439 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:28.439 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:28.440 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.440 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.440 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:167084 3.17 3:ebc2be9c:::rbd_data.1133896adcfd.0000000000000014:head [delete] snapc 8=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-21T15:51:28.440 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:51:20.773031+0000", 2024-09-21T15:51:28.440 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 7.5831320260000004, 2024-09-21T15:51:28.440 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0053046810000000003, 2024-09-21T15:51:28.440 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:28.440 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:28.440 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:28.440 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:28.440 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:28.440 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 167084 2024-09-21T15:51:28.440 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.440 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:28.440 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.441 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:28.441 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.773031+0000", 2024-09-21T15:51:28.441 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:28.441 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.441 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.441 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:28.441 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.773031+0000", 2024-09-21T15:51:28.441 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:28.441 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.441 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.441 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:28.441 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.773030+0000", 2024-09-21T15:51:28.441 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:28.441 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.441 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.442 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:28.442 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.773036+0000", 2024-09-21T15:51:28.442 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.7910000000000001e-06 2024-09-21T15:51:28.442 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.442 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.442 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:28.442 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.773046+0000", 2024-09-21T15:51:28.442 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.8700000000000004e-06 2024-09-21T15:51:28.442 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.442 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.442 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:28.442 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.773053+0000", 2024-09-21T15:51:28.442 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 7.5390000000000002e-06 2024-09-21T15:51:28.442 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.443 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.443 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:28.443 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.773094+0000", 2024-09-21T15:51:28.443 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.1307000000000003e-05 2024-09-21T15:51:28.443 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.443 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.443 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:28.443 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.773257+0000", 2024-09-21T15:51:28.443 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00016289899999999999 2024-09-21T15:51:28.443 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.443 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.443 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 5", 2024-09-21T15:51:28.443 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.773364+0000", 2024-09-21T15:51:28.443 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010628600000000001 2024-09-21T15:51:28.443 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.443 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.444 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:28.444 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.774168+0000", 2024-09-21T15:51:28.444 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00080437100000000001 2024-09-21T15:51:28.444 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.444 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.444 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:28.444 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.778269+0000", 2024-09-21T15:51:28.444 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0041012710000000001 2024-09-21T15:51:28.444 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.444 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.444 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:28.444 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.778300+0000", 2024-09-21T15:51:28.444 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.0988999999999998e-05 2024-09-21T15:51:28.444 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.444 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.445 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:28.445 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.778335+0000", 2024-09-21T15:51:28.445 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.5287000000000003e-05 2024-09-21T15:51:28.445 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:28.445 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:28.445 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:28.445 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.445 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.445 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:167092 3.1 3:87b7471f:::rbd_data.1133896adcfd.000000000000001c:head [delete] snapc 8=[] ondisk+write+known_if_redirected+supports_pool_eio e145)", 2024-09-21T15:51:28.445 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:51:20.775925+0000", 2024-09-21T15:51:28.445 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 7.5802375389999996, 2024-09-21T15:51:28.445 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0054501690000000004, 2024-09-21T15:51:28.445 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:28.445 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:28.445 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:28.446 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:28.446 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:28.446 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 167092 2024-09-21T15:51:28.446 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.446 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:28.446 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.446 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:28.446 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.775925+0000", 2024-09-21T15:51:28.446 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:28.446 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.446 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.446 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:28.446 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.775925+0000", 2024-09-21T15:51:28.446 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:28.446 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.447 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.447 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:28.447 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.775924+0000", 2024-09-21T15:51:28.447 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:28.447 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.447 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.447 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:28.447 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.775929+0000", 2024-09-21T15:51:28.447 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1229999999999999e-06 2024-09-21T15:51:28.447 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.447 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.447 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:28.447 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.775940+0000", 2024-09-21T15:51:28.447 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 1.0495e-05 2024-09-21T15:51:28.447 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.447 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.448 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:28.448 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.775945+0000", 2024-09-21T15:51:28.448 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.7870000000000001e-06 2024-09-21T15:51:28.448 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.448 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.448 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:28.448 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.776184+0000", 2024-09-21T15:51:28.448 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000239196 2024-09-21T15:51:28.448 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.448 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.448 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:28.448 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.776343+0000", 2024-09-21T15:51:28.448 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000159145 2024-09-21T15:51:28.449 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.449 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.449 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-09-21T15:51:28.449 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.776450+0000", 2024-09-21T15:51:28.449 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00010710899999999999 2024-09-21T15:51:28.449 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.449 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.449 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:28.449 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.777106+0000", 2024-09-21T15:51:28.449 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00065567399999999997 2024-09-21T15:51:28.449 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.449 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.449 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:28.449 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.781295+0000", 2024-09-21T15:51:28.449 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0041886120000000004 2024-09-21T15:51:28.450 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.450 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.450 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:28.450 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.781338+0000", 2024-09-21T15:51:28.450 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3906999999999998e-05 2024-09-21T15:51:28.450 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.450 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.450 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:28.450 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:20.781375+0000", 2024-09-21T15:51:28.450 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7030999999999998e-05 2024-09-21T15:51:28.450 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:28.450 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:28.450 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:28.450 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.450 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.451 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:167296 3.1 3:869c7e05:::rbd_data.1133dc5adfdf.00000000000000ae:head [delete] snapc 7=[] ondisk+write+known_if_redirected+supports_pool_eio e146)", 2024-09-21T15:51:28.451 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:51:21.786220+0000", 2024-09-21T15:51:28.451 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 6.5699426489999997, 2024-09-21T15:51:28.451 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0058083359999999999, 2024-09-21T15:51:28.451 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:28.451 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:28.451 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:28.451 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:28.451 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:28.451 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 167296 2024-09-21T15:51:28.451 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.451 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:28.451 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.451 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:28.451 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:21.786220+0000", 2024-09-21T15:51:28.451 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:28.452 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.452 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.452 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:28.452 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:21.786220+0000", 2024-09-21T15:51:28.452 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:28.452 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.452 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.452 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:28.452 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:21.786219+0000", 2024-09-21T15:51:28.452 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.9999995 2024-09-21T15:51:28.452 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.452 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.452 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:28.452 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:21.786223+0000", 2024-09-21T15:51:28.452 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.9949999999999999e-06 2024-09-21T15:51:28.452 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.453 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.453 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:28.453 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:21.786232+0000", 2024-09-21T15:51:28.453 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.5350000000000004e-06 2024-09-21T15:51:28.453 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.453 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.453 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:28.453 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:21.786236+0000", 2024-09-21T15:51:28.453 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.3170000000000003e-06 2024-09-21T15:51:28.453 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.453 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.453 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:28.453 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:21.786274+0000", 2024-09-21T15:51:28.453 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.7527000000000002e-05 2024-09-21T15:51:28.453 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.454 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.454 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:28.454 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:21.786390+0000", 2024-09-21T15:51:28.454 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000116539 2024-09-21T15:51:28.454 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.454 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.454 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-09-21T15:51:28.454 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:21.786451+0000", 2024-09-21T15:51:28.454 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 6.0792999999999999e-05 2024-09-21T15:51:28.454 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.454 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.454 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:28.454 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:21.787063+0000", 2024-09-21T15:51:28.454 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00061143199999999997 2024-09-21T15:51:28.454 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.455 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.455 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:28.455 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:21.791976+0000", 2024-09-21T15:51:28.455 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.004913031 2024-09-21T15:51:28.455 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.455 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.455 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:28.455 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:21.792008+0000", 2024-09-21T15:51:28.455 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.2228999999999999e-05 2024-09-21T15:51:28.455 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.455 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.455 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:28.455 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:21.792029+0000", 2024-09-21T15:51:28.455 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 2.0628999999999998e-05 2024-09-21T15:51:28.455 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:28.456 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:28.456 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:28.456 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.456 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.456 INFO:teuthology.orchestra.run.smithi089.stdout: "description": "osd_op(client.4403.0:167730 3.1 3:878bad7e:::rbd_header.1133d8e7a407:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e150)", 2024-09-21T15:51:28.456 INFO:teuthology.orchestra.run.smithi089.stdout: "initiated_at": "2024-09-21T15:51:25.744019+0000", 2024-09-21T15:51:28.456 INFO:teuthology.orchestra.run.smithi089.stdout: "age": 2.6121433979999997, 2024-09-21T15:51:28.456 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0054516210000000002, 2024-09-21T15:51:28.456 INFO:teuthology.orchestra.run.smithi089.stdout: "type_data": { 2024-09-21T15:51:28.456 INFO:teuthology.orchestra.run.smithi089.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T15:51:28.456 INFO:teuthology.orchestra.run.smithi089.stdout: "client_info": { 2024-09-21T15:51:28.456 INFO:teuthology.orchestra.run.smithi089.stdout: "client": "client.4403", 2024-09-21T15:51:28.456 INFO:teuthology.orchestra.run.smithi089.stdout: "client_addr": "172.21.15.89:0/1194626375", 2024-09-21T15:51:28.456 INFO:teuthology.orchestra.run.smithi089.stdout: "tid": 167730 2024-09-21T15:51:28.456 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.456 INFO:teuthology.orchestra.run.smithi089.stdout: "events": [ 2024-09-21T15:51:28.457 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.457 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "initiated", 2024-09-21T15:51:28.457 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:25.744019+0000", 2024-09-21T15:51:28.457 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:28.457 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.457 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.457 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "throttled", 2024-09-21T15:51:28.457 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:25.744019+0000", 2024-09-21T15:51:28.457 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0 2024-09-21T15:51:28.457 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.457 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.457 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "header_read", 2024-09-21T15:51:28.457 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:25.744019+0000", 2024-09-21T15:51:28.457 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4294967295.999999 2024-09-21T15:51:28.457 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.457 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.458 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "all_read", 2024-09-21T15:51:28.458 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:25.744024+0000", 2024-09-21T15:51:28.458 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.8339999999999998e-06 2024-09-21T15:51:28.458 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.458 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.458 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "dispatched", 2024-09-21T15:51:28.458 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:25.744033+0000", 2024-09-21T15:51:28.458 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 9.7610000000000002e-06 2024-09-21T15:51:28.458 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.458 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.458 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "queued_for_pg", 2024-09-21T15:51:28.458 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:25.744038+0000", 2024-09-21T15:51:28.458 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.6140000000000002e-06 2024-09-21T15:51:28.458 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.458 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.459 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "reached_pg", 2024-09-21T15:51:28.459 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:25.744080+0000", 2024-09-21T15:51:28.459 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 4.2261000000000001e-05 2024-09-21T15:51:28.459 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.459 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.459 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "started", 2024-09-21T15:51:28.459 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:25.744164+0000", 2024-09-21T15:51:28.459 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 8.3509000000000003e-05 2024-09-21T15:51:28.459 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.459 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.459 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "waiting for subops from 4", 2024-09-21T15:51:28.459 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:25.744350+0000", 2024-09-21T15:51:28.459 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.000186744 2024-09-21T15:51:28.459 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.459 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.459 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "op_commit", 2024-09-21T15:51:28.460 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:25.744942+0000", 2024-09-21T15:51:28.460 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.00059126100000000004 2024-09-21T15:51:28.460 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.460 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.460 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "sub_op_commit_rec", 2024-09-21T15:51:28.460 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:25.749381+0000", 2024-09-21T15:51:28.460 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 0.0044395370000000003 2024-09-21T15:51:28.460 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.460 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.460 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "commit_sent", 2024-09-21T15:51:28.460 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:25.749433+0000", 2024-09-21T15:51:28.460 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 5.1313999999999997e-05 2024-09-21T15:51:28.460 INFO:teuthology.orchestra.run.smithi089.stdout: }, 2024-09-21T15:51:28.460 INFO:teuthology.orchestra.run.smithi089.stdout: { 2024-09-21T15:51:28.460 INFO:teuthology.orchestra.run.smithi089.stdout: "event": "done", 2024-09-21T15:51:28.460 INFO:teuthology.orchestra.run.smithi089.stdout: "time": "2024-09-21T15:51:25.749471+0000", 2024-09-21T15:51:28.461 INFO:teuthology.orchestra.run.smithi089.stdout: "duration": 3.8516000000000002e-05 2024-09-21T15:51:28.461 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:28.461 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:28.461 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:28.461 INFO:teuthology.orchestra.run.smithi089.stdout: } 2024-09-21T15:51:28.461 INFO:teuthology.orchestra.run.smithi089.stdout: ] 2024-09-21T15:51:28.461 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:28.703 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=600 2024-09-21T15:51:28.703 INFO:teuthology.orchestra.run.smithi089.stderr:nodeep-scrub is unset 2024-09-21T15:51:28.813 INFO:teuthology.orchestra.run.smithi089.stdout:mon.a: {} 2024-09-21T15:51:28.814 INFO:teuthology.orchestra.run.smithi089.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T15:51:28.824 INFO:teuthology.orchestra.run.smithi089.stdout:mon.b: {} 2024-09-21T15:51:28.824 INFO:teuthology.orchestra.run.smithi089.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T15:51:28.832 INFO:teuthology.orchestra.run.smithi089.stdout:mon.c: {} 2024-09-21T15:51:28.832 INFO:teuthology.orchestra.run.smithi089.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T15:51:28.842 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 1 1 2024-09-21T15:51:29.705 INFO:teuthology.orchestra.run.smithi089.stderr:reweighted osd.1 to 1 (10000) 2024-09-21T15:51:29.720 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 1 1 2024-09-21T15:51:30.708 INFO:teuthology.orchestra.run.smithi089.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-21T15:51:30.723 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 2 1 2024-09-21T15:51:31.715 INFO:teuthology.orchestra.run.smithi089.stderr:reweighted osd.2 to 1 (10000) 2024-09-21T15:51:31.730 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 2 1 2024-09-21T15:51:32.682 INFO:teuthology.orchestra.run.smithi089.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-21T15:51:32.698 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 3 1 2024-09-21T15:51:33.729 INFO:teuthology.orchestra.run.smithi089.stderr:reweighted osd.3 to 1 (10000) 2024-09-21T15:51:33.744 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 3 1 2024-09-21T15:51:34.736 INFO:teuthology.orchestra.run.smithi089.stderr:set osd.3 primary-affinity to 1 (8655362) 2024-09-21T15:51:34.753 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 4 1 2024-09-21T15:51:35.749 INFO:teuthology.orchestra.run.smithi089.stderr:reweighted osd.4 to 1 (10000) 2024-09-21T15:51:35.767 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 4 1 2024-09-21T15:51:36.748 INFO:teuthology.orchestra.run.smithi089.stderr:set osd.4 primary-affinity to 1 (8655362) 2024-09-21T15:51:36.764 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 5 1 2024-09-21T15:51:37.680 INFO:teuthology.orchestra.run.smithi089.stderr:reweighted osd.5 to 1 (10000) 2024-09-21T15:51:37.697 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 5 1 2024-09-21T15:51:38.761 INFO:teuthology.orchestra.run.smithi089.stderr:set osd.5 primary-affinity to 1 (8655362) 2024-09-21T15:51:38.778 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 6 1 2024-09-21T15:51:39.762 INFO:teuthology.orchestra.run.smithi089.stderr:reweighted osd.6 to 1 (10000) 2024-09-21T15:51:39.779 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 6 1 2024-09-21T15:51:40.768 INFO:teuthology.orchestra.run.smithi089.stderr:set osd.6 primary-affinity to 1 (8655362) 2024-09-21T15:51:40.784 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 7 1 2024-09-21T15:51:41.775 INFO:teuthology.orchestra.run.smithi089.stderr:reweighted osd.7 to 1 (10000) 2024-09-21T15:51:41.790 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 7 1 2024-09-21T15:51:42.782 INFO:teuthology.orchestra.run.smithi089.stderr:set osd.7 primary-affinity to 1 (8655362) 2024-09-21T15:51:42.797 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd reweight 0 1 2024-09-21T15:51:43.789 INFO:teuthology.orchestra.run.smithi089.stderr:reweighted osd.0 to 1 (10000) 2024-09-21T15:51:43.804 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd primary-affinity 0 1 2024-09-21T15:51:44.795 INFO:teuthology.orchestra.run.smithi089.stderr:set osd.0 primary-affinity to 1 (8655362) 2024-09-21T15:51:44.809 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-09-21T15:51:44.809 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-09-21T15:51:44.809 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-09-21T15:51:44.810 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-09-21T15:51:44.810 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-21T15:51:44.810 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T15:51:45.082 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:51:45.082 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":168,"fsid":"4a9b64ea-3aaf-4a40-8ce8-02163a0f6e12","created":"2024-09-21T15:48:37.867501+0000","modified":"2024-09-21T15:51:44.790288+0000","last_up_change":"2024-09-21T15:50:29.130795+0000","last_in_change":"2024-09-21T15:51:06.806747+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T15:48:49.921079+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T15:48:55.812982+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":20,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"pool_client.0","create_time":"2024-09-21T15:49:05.523888+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":91,"snap_epoch":150,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b3fabfa-3fe6-47c7-9d0c-a6036f738771","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":67,"up_from":93,"up_thru":136,"down_at":69,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1222314186},{"type":"v1","addr":"172.21.15.89:6801","nonce":1222314186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6803","nonce":1222314186},{"type":"v1","addr":"172.21.15.89:6805","nonce":1222314186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6811","nonce":1222314186},{"type":"v1","addr":"172.21.15.89:6813","nonce":1222314186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6807","nonce":1222314186},{"type":"v1","addr":"172.21.15.89:6809","nonce":1222314186}]},"public_addr":"172.21.15.89:6801/1222314186","cluster_addr":"172.21.15.89:6805/1222314186","heartbeat_back_addr":"172.21.15.89:6813/1222314186","heartbeat_front_addr":"172.21.15.89:6809/1222314186","state":["exists","up"]},{"osd":1,"uuid":"767ad1bc-84ef-4102-83f5-aacda0917ddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3187095367},{"type":"v1","addr":"172.21.15.89:6817","nonce":3187095367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":3187095367},{"type":"v1","addr":"172.21.15.89:6819","nonce":3187095367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":3187095367},{"type":"v1","addr":"172.21.15.89:6824","nonce":3187095367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":3187095367},{"type":"v1","addr":"172.21.15.89:6821","nonce":3187095367}]},"public_addr":"172.21.15.89:6817/3187095367","cluster_addr":"172.21.15.89:6819/3187095367","heartbeat_back_addr":"172.21.15.89:6824/3187095367","heartbeat_front_addr":"172.21.15.89:6821/3187095367","state":["exists","up"]},{"osd":2,"uuid":"1dbd0d50-aae9-42a7-90a1-aa92697778ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":4098625737},{"type":"v1","addr":"172.21.15.89:6804","nonce":4098625737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":4098625737},{"type":"v1","addr":"172.21.15.89:6808","nonce":4098625737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4098625737},{"type":"v1","addr":"172.21.15.89:6815","nonce":4098625737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4098625737},{"type":"v1","addr":"172.21.15.89:6812","nonce":4098625737}]},"public_addr":"172.21.15.89:6804/4098625737","cluster_addr":"172.21.15.89:6808/4098625737","heartbeat_back_addr":"172.21.15.89:6815/4098625737","heartbeat_front_addr":"172.21.15.89:6812/4098625737","state":["exists","up"]},{"osd":3,"uuid":"f37cd295-2bac-4df4-a99b-abedc0978dce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":1886828410},{"type":"v1","addr":"172.21.15.89:6825","nonce":1886828410}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1886828410},{"type":"v1","addr":"172.21.15.89:6827","nonce":1886828410}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1886828410},{"type":"v1","addr":"172.21.15.89:6831","nonce":1886828410}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1886828410},{"type":"v1","addr":"172.21.15.89:6829","nonce":1886828410}]},"public_addr":"172.21.15.89:6825/1886828410","cluster_addr":"172.21.15.89:6827/1886828410","heartbeat_back_addr":"172.21.15.89:6831/1886828410","heartbeat_front_addr":"172.21.15.89:6829/1886828410","state":["exists","up"]},{"osd":4,"uuid":"2eeb351d-d51c-4fc3-884e-debecd995aea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6817","nonce":22722842},{"type":"v1","addr":"172.21.15.100:6819","nonce":22722842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":22722842},{"type":"v1","addr":"172.21.15.100:6822","nonce":22722842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":22722842},{"type":"v1","addr":"172.21.15.100:6830","nonce":22722842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":22722842},{"type":"v1","addr":"172.21.15.100:6826","nonce":22722842}]},"public_addr":"172.21.15.100:6819/22722842","cluster_addr":"172.21.15.100:6822/22722842","heartbeat_back_addr":"172.21.15.100:6830/22722842","heartbeat_front_addr":"172.21.15.100:6826/22722842","state":["exists","up"]},{"osd":5,"uuid":"b5396267-6823-42f1-bd04-bd8f144fa76c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":363733545},{"type":"v1","addr":"172.21.15.100:6818","nonce":363733545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":363733545},{"type":"v1","addr":"172.21.15.100:6823","nonce":363733545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6829","nonce":363733545},{"type":"v1","addr":"172.21.15.100:6831","nonce":363733545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":363733545},{"type":"v1","addr":"172.21.15.100:6827","nonce":363733545}]},"public_addr":"172.21.15.100:6818/363733545","cluster_addr":"172.21.15.100:6823/363733545","heartbeat_back_addr":"172.21.15.100:6831/363733545","heartbeat_front_addr":"172.21.15.100:6827/363733545","state":["exists","up"]},{"osd":6,"uuid":"955b67e9-73bc-42c1-88e0-3356c105e1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":205694879},{"type":"v1","addr":"172.21.15.100:6809","nonce":205694879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":205694879},{"type":"v1","addr":"172.21.15.100:6811","nonce":205694879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":205694879},{"type":"v1","addr":"172.21.15.100:6815","nonce":205694879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":205694879},{"type":"v1","addr":"172.21.15.100:6813","nonce":205694879}]},"public_addr":"172.21.15.100:6809/205694879","cluster_addr":"172.21.15.100:6811/205694879","heartbeat_back_addr":"172.21.15.100:6815/205694879","heartbeat_front_addr":"172.21.15.100:6813/205694879","state":["exists","up"]},{"osd":7,"uuid":"414ea7f0-138d-404e-bce6-fe8dedcdc938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":3120011840},{"type":"v1","addr":"172.21.15.100:6801","nonce":3120011840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":3120011840},{"type":"v1","addr":"172.21.15.100:6803","nonce":3120011840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":3120011840},{"type":"v1","addr":"172.21.15.100:6807","nonce":3120011840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":3120011840},{"type":"v1","addr":"172.21.15.100:6805","nonce":3120011840}]},"public_addr":"172.21.15.100:6801/3120011840","cluster_addr":"172.21.15.100:6803/3120011840","heartbeat_back_addr":"172.21.15.100:6807/3120011840","heartbeat_front_addr":"172.21.15.100:6805/3120011840","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:48:45.230746+0000","dead_epoch":69},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:48:45.303567+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:48:45.210941+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:48:45.218918+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:48:45.301781+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:48:45.248444+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:48:45.275465+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:48:45.166209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T15:51:45.095 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-21T15:51:45.095 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T15:51:45.365 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:51:45.365 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":168,"fsid":"4a9b64ea-3aaf-4a40-8ce8-02163a0f6e12","created":"2024-09-21T15:48:37.867501+0000","modified":"2024-09-21T15:51:44.790288+0000","last_up_change":"2024-09-21T15:50:29.130795+0000","last_in_change":"2024-09-21T15:51:06.806747+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T15:48:49.921079+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T15:48:55.812982+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":20,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"pool_client.0","create_time":"2024-09-21T15:49:05.523888+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":91,"snap_epoch":150,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b3fabfa-3fe6-47c7-9d0c-a6036f738771","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":67,"up_from":93,"up_thru":136,"down_at":69,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1222314186},{"type":"v1","addr":"172.21.15.89:6801","nonce":1222314186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6803","nonce":1222314186},{"type":"v1","addr":"172.21.15.89:6805","nonce":1222314186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6811","nonce":1222314186},{"type":"v1","addr":"172.21.15.89:6813","nonce":1222314186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6807","nonce":1222314186},{"type":"v1","addr":"172.21.15.89:6809","nonce":1222314186}]},"public_addr":"172.21.15.89:6801/1222314186","cluster_addr":"172.21.15.89:6805/1222314186","heartbeat_back_addr":"172.21.15.89:6813/1222314186","heartbeat_front_addr":"172.21.15.89:6809/1222314186","state":["exists","up"]},{"osd":1,"uuid":"767ad1bc-84ef-4102-83f5-aacda0917ddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3187095367},{"type":"v1","addr":"172.21.15.89:6817","nonce":3187095367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":3187095367},{"type":"v1","addr":"172.21.15.89:6819","nonce":3187095367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":3187095367},{"type":"v1","addr":"172.21.15.89:6824","nonce":3187095367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":3187095367},{"type":"v1","addr":"172.21.15.89:6821","nonce":3187095367}]},"public_addr":"172.21.15.89:6817/3187095367","cluster_addr":"172.21.15.89:6819/3187095367","heartbeat_back_addr":"172.21.15.89:6824/3187095367","heartbeat_front_addr":"172.21.15.89:6821/3187095367","state":["exists","up"]},{"osd":2,"uuid":"1dbd0d50-aae9-42a7-90a1-aa92697778ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":4098625737},{"type":"v1","addr":"172.21.15.89:6804","nonce":4098625737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":4098625737},{"type":"v1","addr":"172.21.15.89:6808","nonce":4098625737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4098625737},{"type":"v1","addr":"172.21.15.89:6815","nonce":4098625737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4098625737},{"type":"v1","addr":"172.21.15.89:6812","nonce":4098625737}]},"public_addr":"172.21.15.89:6804/4098625737","cluster_addr":"172.21.15.89:6808/4098625737","heartbeat_back_addr":"172.21.15.89:6815/4098625737","heartbeat_front_addr":"172.21.15.89:6812/4098625737","state":["exists","up"]},{"osd":3,"uuid":"f37cd295-2bac-4df4-a99b-abedc0978dce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":1886828410},{"type":"v1","addr":"172.21.15.89:6825","nonce":1886828410}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1886828410},{"type":"v1","addr":"172.21.15.89:6827","nonce":1886828410}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1886828410},{"type":"v1","addr":"172.21.15.89:6831","nonce":1886828410}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1886828410},{"type":"v1","addr":"172.21.15.89:6829","nonce":1886828410}]},"public_addr":"172.21.15.89:6825/1886828410","cluster_addr":"172.21.15.89:6827/1886828410","heartbeat_back_addr":"172.21.15.89:6831/1886828410","heartbeat_front_addr":"172.21.15.89:6829/1886828410","state":["exists","up"]},{"osd":4,"uuid":"2eeb351d-d51c-4fc3-884e-debecd995aea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6817","nonce":22722842},{"type":"v1","addr":"172.21.15.100:6819","nonce":22722842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":22722842},{"type":"v1","addr":"172.21.15.100:6822","nonce":22722842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":22722842},{"type":"v1","addr":"172.21.15.100:6830","nonce":22722842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":22722842},{"type":"v1","addr":"172.21.15.100:6826","nonce":22722842}]},"public_addr":"172.21.15.100:6819/22722842","cluster_addr":"172.21.15.100:6822/22722842","heartbeat_back_addr":"172.21.15.100:6830/22722842","heartbeat_front_addr":"172.21.15.100:6826/22722842","state":["exists","up"]},{"osd":5,"uuid":"b5396267-6823-42f1-bd04-bd8f144fa76c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":363733545},{"type":"v1","addr":"172.21.15.100:6818","nonce":363733545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":363733545},{"type":"v1","addr":"172.21.15.100:6823","nonce":363733545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6829","nonce":363733545},{"type":"v1","addr":"172.21.15.100:6831","nonce":363733545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":363733545},{"type":"v1","addr":"172.21.15.100:6827","nonce":363733545}]},"public_addr":"172.21.15.100:6818/363733545","cluster_addr":"172.21.15.100:6823/363733545","heartbeat_back_addr":"172.21.15.100:6831/363733545","heartbeat_front_addr":"172.21.15.100:6827/363733545","state":["exists","up"]},{"osd":6,"uuid":"955b67e9-73bc-42c1-88e0-3356c105e1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":205694879},{"type":"v1","addr":"172.21.15.100:6809","nonce":205694879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":205694879},{"type":"v1","addr":"172.21.15.100:6811","nonce":205694879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":205694879},{"type":"v1","addr":"172.21.15.100:6815","nonce":205694879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":205694879},{"type":"v1","addr":"172.21.15.100:6813","nonce":205694879}]},"public_addr":"172.21.15.100:6809/205694879","cluster_addr":"172.21.15.100:6811/205694879","heartbeat_back_addr":"172.21.15.100:6815/205694879","heartbeat_front_addr":"172.21.15.100:6813/205694879","state":["exists","up"]},{"osd":7,"uuid":"414ea7f0-138d-404e-bce6-fe8dedcdc938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":3120011840},{"type":"v1","addr":"172.21.15.100:6801","nonce":3120011840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":3120011840},{"type":"v1","addr":"172.21.15.100:6803","nonce":3120011840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":3120011840},{"type":"v1","addr":"172.21.15.100:6807","nonce":3120011840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":3120011840},{"type":"v1","addr":"172.21.15.100:6805","nonce":3120011840}]},"public_addr":"172.21.15.100:6801/3120011840","cluster_addr":"172.21.15.100:6803/3120011840","heartbeat_back_addr":"172.21.15.100:6807/3120011840","heartbeat_front_addr":"172.21.15.100:6805/3120011840","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:48:45.230746+0000","dead_epoch":69},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:48:45.303567+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:48:45.210941+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:48:45.218918+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:48:45.301781+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:48:45.248444+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:48:45.275465+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:48:45.166209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T15:51:45.378 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-21T15:51:45.378 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-21T15:51:45.378 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-21T15:51:45.379 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-21T15:51:45.379 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-21T15:51:45.380 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-21T15:51:45.380 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-09-21T15:51:45.380 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-09-21T15:51:45.604 INFO:teuthology.orchestra.run.smithi089.stdout:399431958545 2024-09-21T15:51:45.605 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-21T15:51:45.615 INFO:teuthology.orchestra.run.smithi089.stdout:55834574889 2024-09-21T15:51:45.616 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-21T15:51:45.619 INFO:teuthology.orchestra.run.smithi089.stdout:55834574888 2024-09-21T15:51:45.619 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-21T15:51:45.620 INFO:teuthology.orchestra.run.smithi089.stdout:55834574888 2024-09-21T15:51:45.621 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-21T15:51:45.622 INFO:teuthology.orchestra.run.smithi089.stdout:55834574890 2024-09-21T15:51:45.623 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-21T15:51:45.623 INFO:teuthology.orchestra.run.smithi089.stdout:55834574889 2024-09-21T15:51:45.624 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-21T15:51:45.625 INFO:teuthology.orchestra.run.smithi089.stdout:55834574889 2024-09-21T15:51:45.625 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-21T15:51:45.627 INFO:teuthology.orchestra.run.smithi089.stdout:55834574887 2024-09-21T15:51:45.627 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-21T15:51:46.155 INFO:teuthology.orchestra.run.smithi089.stdout:55834574886 2024-09-21T15:51:46.167 INFO:teuthology.orchestra.run.smithi089.stdout:399431958544 2024-09-21T15:51:46.176 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574888 got 55834574886 for osd.4 2024-09-21T15:51:46.176 INFO:teuthology.orchestra.run.smithi089.stdout:55834574887 2024-09-21T15:51:46.179 INFO:teuthology.orchestra.run.smithi089.stdout:55834574886 2024-09-21T15:51:46.185 INFO:tasks.ceph.ceph_manager.ceph:need seq 399431958545 got 399431958544 for osd.0 2024-09-21T15:51:46.194 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574889 got 55834574887 for osd.2 2024-09-21T15:51:46.200 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574888 got 55834574886 for osd.1 2024-09-21T15:51:46.209 INFO:teuthology.orchestra.run.smithi089.stdout:55834574887 2024-09-21T15:51:46.215 INFO:teuthology.orchestra.run.smithi089.stdout:55834574887 2024-09-21T15:51:46.223 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574889 got 55834574887 for osd.5 2024-09-21T15:51:46.228 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574889 got 55834574887 for osd.3 2024-09-21T15:51:46.249 INFO:teuthology.orchestra.run.smithi089.stdout:55834574885 2024-09-21T15:51:46.262 INFO:teuthology.orchestra.run.smithi089.stdout:55834574888 2024-09-21T15:51:46.262 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574887 got 55834574885 for osd.7 2024-09-21T15:51:46.274 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574890 got 55834574888 for osd.6 2024-09-21T15:51:47.177 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-21T15:51:47.186 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-21T15:51:47.194 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-21T15:51:47.201 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-21T15:51:47.224 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-21T15:51:47.229 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-21T15:51:47.264 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-09-21T15:51:47.275 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-09-21T15:51:47.702 INFO:teuthology.orchestra.run.smithi089.stdout:55834574888 2024-09-21T15:51:47.719 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574888 got 55834574888 for osd.4 2024-09-21T15:51:47.720 DEBUG:teuthology.parallel:result is None 2024-09-21T15:51:47.771 INFO:teuthology.orchestra.run.smithi089.stdout:55834574889 2024-09-21T15:51:47.786 INFO:teuthology.orchestra.run.smithi089.stdout:55834574888 2024-09-21T15:51:47.789 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574889 got 55834574889 for osd.2 2024-09-21T15:51:47.790 DEBUG:teuthology.parallel:result is None 2024-09-21T15:51:47.792 INFO:teuthology.orchestra.run.smithi089.stdout:399431958545 2024-09-21T15:51:47.795 INFO:teuthology.orchestra.run.smithi089.stdout:55834574890 2024-09-21T15:51:47.801 INFO:teuthology.orchestra.run.smithi089.stdout:55834574889 2024-09-21T15:51:47.805 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574888 got 55834574888 for osd.1 2024-09-21T15:51:47.805 DEBUG:teuthology.parallel:result is None 2024-09-21T15:51:47.811 INFO:tasks.ceph.ceph_manager.ceph:need seq 399431958545 got 399431958545 for osd.0 2024-09-21T15:51:47.811 DEBUG:teuthology.parallel:result is None 2024-09-21T15:51:47.814 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574890 got 55834574890 for osd.6 2024-09-21T15:51:47.814 DEBUG:teuthology.parallel:result is None 2024-09-21T15:51:47.819 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574889 got 55834574889 for osd.5 2024-09-21T15:51:47.819 DEBUG:teuthology.parallel:result is None 2024-09-21T15:51:47.852 INFO:teuthology.orchestra.run.smithi089.stdout:55834574887 2024-09-21T15:51:47.856 INFO:teuthology.orchestra.run.smithi089.stdout:55834574889 2024-09-21T15:51:47.867 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574887 got 55834574887 for osd.7 2024-09-21T15:51:47.867 DEBUG:teuthology.parallel:result is None 2024-09-21T15:51:47.870 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574889 got 55834574889 for osd.3 2024-09-21T15:51:47.870 DEBUG:teuthology.parallel:result is None 2024-09-21T15:51:47.870 INFO:tasks.ceph.ceph_manager.ceph:waiting for recovery to complete 2024-09-21T15:51:47.870 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T15:51:48.183 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:51:48.183 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-09-21T15:51:48.194 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":250,"stamp":"2024-09-21T15:51:47.918416+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590406,"num_objects":1043,"num_object_clones":0,"num_object_copies":2086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1043,"num_whiteouts":1035,"num_read":207535,"num_read_kb":842523,"num_write":23550,"num_write_kb":305657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3397,"num_bytes_recovered":100270055,"num_keys_recovered":316,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20338,"ondisk_log_size":20338,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":416544,"kb_used_data":16932,"kb_used_omap":24,"kb_used_meta":399463,"kb_avail":754558176,"statfs":{"total":773094113280,"available":772667572224,"internally_reserved":0,"allocated":17338368,"data_stored":8940606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25118,"internal_metadata":409050594},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001005"},"pg_stats":[{"pgid":"3.1f","version":"150'413","reported_seq":2132,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.781077+0000","last_change":"2024-09-21T15:51:25.690102+0000","last_active":"2024-09-21T15:51:42.781077+0000","last_peered":"2024-09-21T15:51:42.781077+0000","last_clean":"2024-09-21T15:51:42.781077+0000","last_became_active":"2024-09-21T15:50:32.670964+0000","last_became_peered":"2024-09-21T15:50:32.670964+0000","last_unstale":"2024-09-21T15:51:42.781077+0000","last_undegraded":"2024-09-21T15:51:42.781077+0000","last_fullsized":"2024-09-21T15:51:42.781077+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":413,"log_dups_size":0,"ondisk_log_size":413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:09:09.918864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017896030000000001,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":28,"num_read":2413,"num_read_kb":10419,"num_write":442,"num_write_kb":5956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":1974433,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"149'599","reported_seq":6526,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:44.475566+0000","last_change":"2024-09-21T15:51:25.689127+0000","last_active":"2024-09-21T15:51:44.475566+0000","last_peered":"2024-09-21T15:51:44.475566+0000","last_clean":"2024-09-21T15:51:44.475566+0000","last_became_active":"2024-09-21T15:51:24.691676+0000","last_became_peered":"2024-09-21T15:51:24.691676+0000","last_unstale":"2024-09-21T15:51:44.475566+0000","last_undegraded":"2024-09-21T15:51:44.475566+0000","last_fullsized":"2024-09-21T15:51:44.475566+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":599,"log_dups_size":0,"ondisk_log_size":599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:51:29.795001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010326389999999999,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":38,"num_read":7456,"num_read_kb":25912,"num_write":580,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":250,"num_bytes_recovered":3824749,"num_keys_recovered":34,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"150'655","reported_seq":10214,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.783240+0000","last_change":"2024-09-21T15:51:39.767369+0000","last_active":"2024-09-21T15:51:42.783240+0000","last_peered":"2024-09-21T15:51:42.783240+0000","last_clean":"2024-09-21T15:51:42.783240+0000","last_became_active":"2024-09-21T15:51:39.765580+0000","last_became_peered":"2024-09-21T15:51:39.765580+0000","last_unstale":"2024-09-21T15:51:42.783240+0000","last_undegraded":"2024-09-21T15:51:42.783240+0000","last_fullsized":"2024-09-21T15:51:42.783240+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":655,"log_dups_size":0,"ondisk_log_size":655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:40:43.868959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090302100000000005,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":34,"num_read":10756,"num_read_kb":29175,"num_write":703,"num_write_kb":8882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"150'584","reported_seq":4519,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.783179+0000","last_change":"2024-09-21T15:51:39.767392+0000","last_active":"2024-09-21T15:51:42.783179+0000","last_peered":"2024-09-21T15:51:42.783179+0000","last_clean":"2024-09-21T15:51:42.783179+0000","last_became_active":"2024-09-21T15:51:39.765999+0000","last_became_peered":"2024-09-21T15:51:39.765999+0000","last_unstale":"2024-09-21T15:51:42.783179+0000","last_undegraded":"2024-09-21T15:51:42.783179+0000","last_fullsized":"2024-09-21T15:51:42.783179+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":584,"log_dups_size":0,"ondisk_log_size":584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:21:55.582116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010903670000000001,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":27,"num_read":5242,"num_read_kb":17341,"num_write":644,"num_write_kb":4787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"145'508","reported_seq":4404,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.780869+0000","last_change":"2024-09-21T15:51:25.690506+0000","last_active":"2024-09-21T15:51:42.780869+0000","last_peered":"2024-09-21T15:51:42.780869+0000","last_clean":"2024-09-21T15:51:42.780869+0000","last_became_active":"2024-09-21T15:50:32.674250+0000","last_became_peered":"2024-09-21T15:50:32.674250+0000","last_unstale":"2024-09-21T15:51:42.780869+0000","last_undegraded":"2024-09-21T15:51:42.780869+0000","last_fullsized":"2024-09-21T15:51:42.780869+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":508,"log_dups_size":0,"ondisk_log_size":508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:54:57.399818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019320559999999999,"stat_sum":{"num_bytes":0,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":39,"num_read":4815,"num_read_kb":30088,"num_write":498,"num_write_kb":13507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":4759000,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"150'431","reported_seq":6943,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581775+0000","last_change":"2024-09-21T15:51:25.689223+0000","last_active":"2024-09-21T15:51:43.581775+0000","last_peered":"2024-09-21T15:51:43.581775+0000","last_clean":"2024-09-21T15:51:43.581775+0000","last_became_active":"2024-09-21T15:49:06.693903+0000","last_became_peered":"2024-09-21T15:49:06.693903+0000","last_unstale":"2024-09-21T15:51:43.581775+0000","last_undegraded":"2024-09-21T15:51:43.581775+0000","last_fullsized":"2024-09-21T15:51:43.581775+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":431,"log_dups_size":0,"ondisk_log_size":431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:41:45.432893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047963299999999998,"stat_sum":{"num_bytes":19,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":30,"num_read":7221,"num_read_kb":31759,"num_write":421,"num_write_kb":9476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"148'460","reported_seq":8836,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.491336+0000","last_change":"2024-09-21T15:51:25.780904+0000","last_active":"2024-09-21T15:51:43.491336+0000","last_peered":"2024-09-21T15:51:43.491336+0000","last_clean":"2024-09-21T15:51:43.491336+0000","last_became_active":"2024-09-21T15:51:12.688258+0000","last_became_peered":"2024-09-21T15:51:12.688258+0000","last_unstale":"2024-09-21T15:51:43.491336+0000","last_undegraded":"2024-09-21T15:51:43.491336+0000","last_fullsized":"2024-09-21T15:51:43.491336+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":460,"log_dups_size":0,"ondisk_log_size":460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:19:38.132074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000411691,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":32,"num_read":8283,"num_read_kb":39579,"num_write":518,"num_write_kb":7408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":236,"num_bytes_recovered":6705120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"149'537","reported_seq":4716,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:44.475480+0000","last_change":"2024-09-21T15:51:35.753555+0000","last_active":"2024-09-21T15:51:44.475480+0000","last_peered":"2024-09-21T15:51:44.475480+0000","last_clean":"2024-09-21T15:51:44.475480+0000","last_became_active":"2024-09-21T15:51:35.751700+0000","last_became_peered":"2024-09-21T15:51:35.751700+0000","last_unstale":"2024-09-21T15:51:44.475480+0000","last_undegraded":"2024-09-21T15:51:44.475480+0000","last_fullsized":"2024-09-21T15:51:44.475480+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":537,"log_dups_size":0,"ondisk_log_size":537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:20:44.688916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051035439999999998,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":26,"num_read":6190,"num_read_kb":30937,"num_write":534,"num_write_kb":11075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":282,"num_bytes_recovered":11320990,"num_keys_recovered":70,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"149'1040","reported_seq":9412,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.781000+0000","last_change":"2024-09-21T15:51:25.689670+0000","last_active":"2024-09-21T15:51:42.781000+0000","last_peered":"2024-09-21T15:51:42.781000+0000","last_clean":"2024-09-21T15:51:42.781000+0000","last_became_active":"2024-09-21T15:50:32.668771+0000","last_became_peered":"2024-09-21T15:50:32.668771+0000","last_unstale":"2024-09-21T15:51:42.781000+0000","last_undegraded":"2024-09-21T15:51:42.781000+0000","last_fullsized":"2024-09-21T15:51:42.781000+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":1040,"log_dups_size":0,"ondisk_log_size":1040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:39:43.554150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001503861,"stat_sum":{"num_bytes":0,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":36,"num_read":12714,"num_read_kb":21487,"num_write":1303,"num_write_kb":8257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":3319185,"num_keys_recovered":34,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"150'540","reported_seq":6716,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.783295+0000","last_change":"2024-09-21T15:51:25.757086+0000","last_active":"2024-09-21T15:51:42.783295+0000","last_peered":"2024-09-21T15:51:42.783295+0000","last_clean":"2024-09-21T15:51:42.783295+0000","last_became_active":"2024-09-21T15:49:06.688808+0000","last_became_peered":"2024-09-21T15:49:06.688808+0000","last_unstale":"2024-09-21T15:51:42.783295+0000","last_undegraded":"2024-09-21T15:51:42.783295+0000","last_fullsized":"2024-09-21T15:51:42.783295+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":540,"log_dups_size":0,"ondisk_log_size":540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:43:02.012254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00023944,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":30,"num_read":7604,"num_read_kb":49865,"num_write":542,"num_write_kb":13823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"150'530","reported_seq":4324,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581211+0000","last_change":"2024-09-21T15:51:25.699156+0000","last_active":"2024-09-21T15:51:43.581211+0000","last_peered":"2024-09-21T15:51:43.581211+0000","last_clean":"2024-09-21T15:51:43.581211+0000","last_became_active":"2024-09-21T15:51:11.841664+0000","last_became_peered":"2024-09-21T15:51:11.841664+0000","last_unstale":"2024-09-21T15:51:43.581211+0000","last_undegraded":"2024-09-21T15:51:43.581211+0000","last_fullsized":"2024-09-21T15:51:43.581211+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:47:37.118354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001532828,"stat_sum":{"num_bytes":0,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":29,"num_read":4052,"num_read_kb":27144,"num_write":637,"num_write_kb":17935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":222,"num_bytes_recovered":4468215,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"147'1207","reported_seq":4387,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581846+0000","last_change":"2024-09-21T15:51:25.690386+0000","last_active":"2024-09-21T15:51:43.581846+0000","last_peered":"2024-09-21T15:51:43.581846+0000","last_clean":"2024-09-21T15:51:43.581846+0000","last_became_active":"2024-09-21T15:49:06.688803+0000","last_became_peered":"2024-09-21T15:49:06.688803+0000","last_unstale":"2024-09-21T15:51:43.581846+0000","last_undegraded":"2024-09-21T15:51:43.581846+0000","last_fullsized":"2024-09-21T15:51:43.581846+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":1207,"log_dups_size":0,"ondisk_log_size":1207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:18:47.662631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013586710000000001,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":26,"num_read":7764,"num_read_kb":18232,"num_write":1926,"num_write_kb":6190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":161,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581555+0000","last_change":"2024-09-21T15:48:59.666431+0000","last_active":"2024-09-21T15:51:43.581555+0000","last_peered":"2024-09-21T15:51:43.581555+0000","last_clean":"2024-09-21T15:51:43.581555+0000","last_became_active":"2024-09-21T15:48:57.583665+0000","last_became_peered":"2024-09-21T15:48:57.583665+0000","last_unstale":"2024-09-21T15:51:43.581555+0000","last_undegraded":"2024-09-21T15:51:43.581555+0000","last_fullsized":"2024-09-21T15:51:43.581555+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:52:09.228964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031098399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"150'438","reported_seq":3064,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.780970+0000","last_change":"2024-09-21T15:51:25.689587+0000","last_active":"2024-09-21T15:51:42.780970+0000","last_peered":"2024-09-21T15:51:42.780970+0000","last_clean":"2024-09-21T15:51:42.780970+0000","last_became_active":"2024-09-21T15:50:32.671424+0000","last_became_peered":"2024-09-21T15:50:32.671424+0000","last_unstale":"2024-09-21T15:51:42.780970+0000","last_undegraded":"2024-09-21T15:51:42.780970+0000","last_fullsized":"2024-09-21T15:51:42.780970+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":438,"log_dups_size":0,"ondisk_log_size":438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:04:14.554732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001507467,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":35,"num_read":2637,"num_read_kb":12956,"num_write":388,"num_write_kb":8392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":4494388,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":164,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.491395+0000","last_change":"2024-09-21T15:48:59.589790+0000","last_active":"2024-09-21T15:51:43.491395+0000","last_peered":"2024-09-21T15:51:43.491395+0000","last_clean":"2024-09-21T15:51:43.491395+0000","last_became_active":"2024-09-21T15:48:57.582121+0000","last_became_peered":"2024-09-21T15:48:57.582121+0000","last_unstale":"2024-09-21T15:51:43.491395+0000","last_undegraded":"2024-09-21T15:51:43.491395+0000","last_fullsized":"2024-09-21T15:51:43.491395+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:33:47.592624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039654700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"148'579","reported_seq":8001,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:44.475582+0000","last_change":"2024-09-21T15:51:35.754312+0000","last_active":"2024-09-21T15:51:44.475582+0000","last_peered":"2024-09-21T15:51:44.475582+0000","last_clean":"2024-09-21T15:51:44.475582+0000","last_became_active":"2024-09-21T15:51:35.752700+0000","last_became_peered":"2024-09-21T15:51:35.752700+0000","last_unstale":"2024-09-21T15:51:44.475582+0000","last_undegraded":"2024-09-21T15:51:44.475582+0000","last_fullsized":"2024-09-21T15:51:44.475582+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":579,"log_dups_size":0,"ondisk_log_size":579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:37:03.162105+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.00086244900000000005,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":35,"num_read":8380,"num_read_kb":21195,"num_write":544,"num_write_kb":5397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":256,"num_bytes_recovered":3760369,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":175,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581295+0000","last_change":"2024-09-21T15:50:30.146743+0000","last_active":"2024-09-21T15:51:43.581295+0000","last_peered":"2024-09-21T15:51:43.581295+0000","last_clean":"2024-09-21T15:51:43.581295+0000","last_became_active":"2024-09-21T15:50:30.146428+0000","last_became_peered":"2024-09-21T15:50:30.146428+0000","last_unstale":"2024-09-21T15:51:43.581295+0000","last_undegraded":"2024-09-21T15:51:43.581295+0000","last_fullsized":"2024-09-21T15:51:43.581295+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:58:38.689807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036074600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"145'703","reported_seq":8853,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.491362+0000","last_change":"2024-09-21T15:51:25.780818+0000","last_active":"2024-09-21T15:51:43.491362+0000","last_peered":"2024-09-21T15:51:43.491362+0000","last_clean":"2024-09-21T15:51:43.491362+0000","last_became_active":"2024-09-21T15:50:35.682788+0000","last_became_peered":"2024-09-21T15:50:35.682788+0000","last_unstale":"2024-09-21T15:51:43.491362+0000","last_undegraded":"2024-09-21T15:51:43.491362+0000","last_fullsized":"2024-09-21T15:51:43.491362+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":703,"log_dups_size":0,"ondisk_log_size":703,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:20:40.695690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032907100000000001,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":26,"num_read":11015,"num_read_kb":27666,"num_write":868,"num_write_kb":11017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":175,"num_bytes_recovered":14134268,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":177,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.491361+0000","last_change":"2024-09-21T15:50:30.146654+0000","last_active":"2024-09-21T15:51:43.491361+0000","last_peered":"2024-09-21T15:51:43.491361+0000","last_clean":"2024-09-21T15:51:43.491361+0000","last_became_active":"2024-09-21T15:50:30.146273+0000","last_became_peered":"2024-09-21T15:50:30.146273+0000","last_unstale":"2024-09-21T15:51:43.491361+0000","last_undegraded":"2024-09-21T15:51:43.491361+0000","last_fullsized":"2024-09-21T15:51:43.491361+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:21:48.178519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040525700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"150'603","reported_seq":5376,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.783334+0000","last_change":"2024-09-21T15:51:39.769574+0000","last_active":"2024-09-21T15:51:42.783334+0000","last_peered":"2024-09-21T15:51:42.783334+0000","last_clean":"2024-09-21T15:51:42.783334+0000","last_became_active":"2024-09-21T15:51:39.768300+0000","last_became_peered":"2024-09-21T15:51:39.768300+0000","last_unstale":"2024-09-21T15:51:42.783334+0000","last_undegraded":"2024-09-21T15:51:42.783334+0000","last_fullsized":"2024-09-21T15:51:42.783334+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":603,"log_dups_size":0,"ondisk_log_size":603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:33:47.511137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050855200000000001,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":35,"num_read":5730,"num_read_kb":33940,"num_write":686,"num_write_kb":16332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":268,"num_bytes_recovered":7823491,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"150'921","reported_seq":4195,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581881+0000","last_change":"2024-09-21T15:51:25.690079+0000","last_active":"2024-09-21T15:51:43.581881+0000","last_peered":"2024-09-21T15:51:43.581881+0000","last_clean":"2024-09-21T15:51:43.581881+0000","last_became_active":"2024-09-21T15:50:32.677456+0000","last_became_peered":"2024-09-21T15:50:32.677456+0000","last_unstale":"2024-09-21T15:51:43.581881+0000","last_undegraded":"2024-09-21T15:51:43.581881+0000","last_fullsized":"2024-09-21T15:51:43.581881+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":921,"log_dups_size":0,"ondisk_log_size":921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:46:31.194345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010737730000000001,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":34,"num_read":5136,"num_read_kb":19340,"num_write":949,"num_write_kb":7405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":632176,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"20'2","reported_seq":59,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.783126+0000","last_change":"2024-09-21T15:51:39.764743+0000","last_active":"2024-09-21T15:51:42.783126+0000","last_peered":"2024-09-21T15:51:42.783126+0000","last_clean":"2024-09-21T15:51:42.783126+0000","last_became_active":"2024-09-21T15:51:39.764409+0000","last_became_peered":"2024-09-21T15:51:39.764409+0000","last_unstale":"2024-09-21T15:51:42.783126+0000","last_undegraded":"2024-09-21T15:51:42.783126+0000","last_fullsized":"2024-09-21T15:51:42.783126+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:28:14.932986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000859595,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"148'393","reported_seq":4833,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.491393+0000","last_change":"2024-09-21T15:51:25.781082+0000","last_active":"2024-09-21T15:51:43.491393+0000","last_peered":"2024-09-21T15:51:43.491393+0000","last_clean":"2024-09-21T15:51:43.491393+0000","last_became_active":"2024-09-21T15:50:35.682001+0000","last_became_peered":"2024-09-21T15:50:35.682001+0000","last_unstale":"2024-09-21T15:51:43.491393+0000","last_undegraded":"2024-09-21T15:51:43.491393+0000","last_fullsized":"2024-09-21T15:51:43.491393+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":393,"log_dups_size":0,"ondisk_log_size":393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:09:15.325457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044281000000000002,"stat_sum":{"num_bytes":0,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":31,"num_read":4285,"num_read_kb":23375,"num_write":379,"num_write_kb":4145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":187,"num_bytes_recovered":4305276,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":97,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.788209+0000","last_change":"2024-09-21T15:51:07.683738+0000","last_active":"2024-09-21T15:51:43.788209+0000","last_peered":"2024-09-21T15:51:43.788209+0000","last_clean":"2024-09-21T15:51:43.788209+0000","last_became_active":"2024-09-21T15:51:07.683418+0000","last_became_peered":"2024-09-21T15:51:07.683418+0000","last_unstale":"2024-09-21T15:51:43.788209+0000","last_undegraded":"2024-09-21T15:51:43.788209+0000","last_fullsized":"2024-09-21T15:51:43.788209+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:25:13.468653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"150'558","reported_seq":2598,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.781077+0000","last_change":"2024-09-21T15:51:25.689707+0000","last_active":"2024-09-21T15:51:42.781077+0000","last_peered":"2024-09-21T15:51:42.781077+0000","last_clean":"2024-09-21T15:51:42.781077+0000","last_became_active":"2024-09-21T15:50:32.671413+0000","last_became_peered":"2024-09-21T15:50:32.671413+0000","last_unstale":"2024-09-21T15:51:42.781077+0000","last_undegraded":"2024-09-21T15:51:42.781077+0000","last_fullsized":"2024-09-21T15:51:42.781077+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":558,"log_dups_size":0,"ondisk_log_size":558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:04:19.509402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001292872,"stat_sum":{"num_bytes":0,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":39,"num_read":3663,"num_read_kb":16543,"num_write":590,"num_write_kb":5977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":1696547,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":162,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581078+0000","last_change":"2024-09-21T15:49:00.114730+0000","last_active":"2024-09-21T15:51:43.581078+0000","last_peered":"2024-09-21T15:51:43.581078+0000","last_clean":"2024-09-21T15:51:43.581078+0000","last_became_active":"2024-09-21T15:48:57.585525+0000","last_became_peered":"2024-09-21T15:48:57.585525+0000","last_unstale":"2024-09-21T15:51:43.581078+0000","last_undegraded":"2024-09-21T15:51:43.581078+0000","last_fullsized":"2024-09-21T15:51:43.581078+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:35:35.786526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031691100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"150'413","reported_seq":2957,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:44.475526+0000","last_change":"2024-09-21T15:51:35.754599+0000","last_active":"2024-09-21T15:51:44.475526+0000","last_peered":"2024-09-21T15:51:44.475526+0000","last_clean":"2024-09-21T15:51:44.475526+0000","last_became_active":"2024-09-21T15:51:35.752810+0000","last_became_peered":"2024-09-21T15:51:35.752810+0000","last_unstale":"2024-09-21T15:51:44.475526+0000","last_undegraded":"2024-09-21T15:51:44.475526+0000","last_fullsized":"2024-09-21T15:51:44.475526+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":413,"log_dups_size":0,"ondisk_log_size":413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:54:49.851651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033178999999999999,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":30,"num_read":2149,"num_read_kb":34452,"num_write":406,"num_write_kb":12612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":214,"num_bytes_recovered":2197196,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"18'1","reported_seq":58,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.783151+0000","last_change":"2024-09-21T15:51:39.764710+0000","last_active":"2024-09-21T15:51:42.783151+0000","last_peered":"2024-09-21T15:51:42.783151+0000","last_clean":"2024-09-21T15:51:42.783151+0000","last_became_active":"2024-09-21T15:51:39.764411+0000","last_became_peered":"2024-09-21T15:51:39.764411+0000","last_unstale":"2024-09-21T15:51:42.783151+0000","last_undegraded":"2024-09-21T15:51:42.783151+0000","last_fullsized":"2024-09-21T15:51:42.783151+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:29:50.093645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039559000000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":323,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581078+0000","last_change":"2024-09-21T15:50:30.147280+0000","last_active":"2024-09-21T15:51:43.581078+0000","last_peered":"2024-09-21T15:51:43.581078+0000","last_clean":"2024-09-21T15:51:43.581078+0000","last_became_active":"2024-09-21T15:50:30.146987+0000","last_became_peered":"2024-09-21T15:50:30.146987+0000","last_unstale":"2024-09-21T15:51:43.581078+0000","last_undegraded":"2024-09-21T15:51:43.581078+0000","last_fullsized":"2024-09-21T15:51:43.581078+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:50.161884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:50.161884+0000","last_clean_scrub_stamp":"2024-09-21T15:48:50.161884+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:26:32.853650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"150'540","reported_seq":7944,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:44.475450+0000","last_change":"2024-09-21T15:51:25.689387+0000","last_active":"2024-09-21T15:51:44.475450+0000","last_peered":"2024-09-21T15:51:44.475450+0000","last_clean":"2024-09-21T15:51:44.475450+0000","last_became_active":"2024-09-21T15:51:11.837877+0000","last_became_peered":"2024-09-21T15:51:11.837877+0000","last_unstale":"2024-09-21T15:51:44.475450+0000","last_undegraded":"2024-09-21T15:51:44.475450+0000","last_fullsized":"2024-09-21T15:51:44.475450+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":540,"log_dups_size":0,"ondisk_log_size":540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:45:58.418197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079839600000000004,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":32,"num_read":7239,"num_read_kb":29580,"num_write":558,"num_write_kb":4793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":418,"num_bytes_recovered":6262707,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"150'389","reported_seq":3170,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581718+0000","last_change":"2024-09-21T15:51:25.689626+0000","last_active":"2024-09-21T15:51:43.581718+0000","last_peered":"2024-09-21T15:51:43.581718+0000","last_clean":"2024-09-21T15:51:43.581718+0000","last_became_active":"2024-09-21T15:49:07.210506+0000","last_became_peered":"2024-09-21T15:49:07.210506+0000","last_unstale":"2024-09-21T15:51:43.581718+0000","last_undegraded":"2024-09-21T15:51:43.581718+0000","last_fullsized":"2024-09-21T15:51:43.581718+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":389,"log_dups_size":0,"ondisk_log_size":389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:55:18.046738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089193900000000001,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":30,"num_read":2716,"num_read_kb":28245,"num_write":454,"num_write_kb":12090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"149'595","reported_seq":12267,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581556+0000","last_change":"2024-09-21T15:51:41.782922+0000","last_active":"2024-09-21T15:51:43.581556+0000","last_peered":"2024-09-21T15:51:43.581556+0000","last_clean":"2024-09-21T15:51:43.581556+0000","last_became_active":"2024-09-21T15:51:41.781781+0000","last_became_peered":"2024-09-21T15:51:41.781781+0000","last_unstale":"2024-09-21T15:51:43.581556+0000","last_undegraded":"2024-09-21T15:51:43.581556+0000","last_fullsized":"2024-09-21T15:51:43.581556+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":595,"log_dups_size":0,"ondisk_log_size":595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:10:04.846054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000309432,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":29,"num_read":12828,"num_read_kb":63467,"num_write":580,"num_write_kb":14212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"147'851","reported_seq":3855,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:44.475479+0000","last_change":"2024-09-21T15:51:35.752901+0000","last_active":"2024-09-21T15:51:44.475479+0000","last_peered":"2024-09-21T15:51:44.475479+0000","last_clean":"2024-09-21T15:51:44.475479+0000","last_became_active":"2024-09-21T15:51:35.751150+0000","last_became_peered":"2024-09-21T15:51:35.751150+0000","last_unstale":"2024-09-21T15:51:44.475479+0000","last_undegraded":"2024-09-21T15:51:44.475479+0000","last_fullsized":"2024-09-21T15:51:44.475479+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":851,"log_dups_size":0,"ondisk_log_size":851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:33:02.873060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001158526,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":26,"num_read":5948,"num_read_kb":20169,"num_write":1244,"num_write_kb":7750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":262,"num_bytes_recovered":8353711,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"150'739","reported_seq":6768,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.783290+0000","last_change":"2024-09-21T15:51:39.769469+0000","last_active":"2024-09-21T15:51:42.783290+0000","last_peered":"2024-09-21T15:51:42.783290+0000","last_clean":"2024-09-21T15:51:42.783290+0000","last_became_active":"2024-09-21T15:51:39.768321+0000","last_became_peered":"2024-09-21T15:51:39.768321+0000","last_unstale":"2024-09-21T15:51:42.783290+0000","last_undegraded":"2024-09-21T15:51:42.783290+0000","last_fullsized":"2024-09-21T15:51:42.783290+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":739,"log_dups_size":0,"ondisk_log_size":739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:21:29.566035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074635999999999995,"stat_sum":{"num_bytes":0,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":36,"num_read":8849,"num_read_kb":25678,"num_write":868,"num_write_kb":5005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":246,"num_bytes_recovered":5686754,"num_keys_recovered":23,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"147'894","reported_seq":4314,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581242+0000","last_change":"2024-09-21T15:51:25.698246+0000","last_active":"2024-09-21T15:51:43.581242+0000","last_peered":"2024-09-21T15:51:43.581242+0000","last_clean":"2024-09-21T15:51:43.581242+0000","last_became_active":"2024-09-21T15:49:06.689203+0000","last_became_peered":"2024-09-21T15:49:06.689203+0000","last_unstale":"2024-09-21T15:51:43.581242+0000","last_undegraded":"2024-09-21T15:51:43.581242+0000","last_fullsized":"2024-09-21T15:51:43.581242+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":894,"log_dups_size":0,"ondisk_log_size":894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:22:17.578359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084935700000000002,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":35,"num_read":7321,"num_read_kb":28131,"num_write":1146,"num_write_kb":14809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"148'568","reported_seq":8144,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581299+0000","last_change":"2024-09-21T15:51:25.697735+0000","last_active":"2024-09-21T15:51:43.581299+0000","last_peered":"2024-09-21T15:51:43.581299+0000","last_clean":"2024-09-21T15:51:43.581299+0000","last_became_active":"2024-09-21T15:49:06.690101+0000","last_became_peered":"2024-09-21T15:49:06.690101+0000","last_unstale":"2024-09-21T15:51:43.581299+0000","last_undegraded":"2024-09-21T15:51:43.581299+0000","last_fullsized":"2024-09-21T15:51:43.581299+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":568,"log_dups_size":0,"ondisk_log_size":568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:43:30.659104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044571200000000002,"stat_sum":{"num_bytes":0,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":29,"num_read":8735,"num_read_kb":42983,"num_write":509,"num_write_kb":16273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"145'698","reported_seq":3799,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581254+0000","last_change":"2024-09-21T15:51:25.698380+0000","last_active":"2024-09-21T15:51:43.581254+0000","last_peered":"2024-09-21T15:51:43.581254+0000","last_clean":"2024-09-21T15:51:43.581254+0000","last_became_active":"2024-09-21T15:49:06.694250+0000","last_became_peered":"2024-09-21T15:49:06.694250+0000","last_unstale":"2024-09-21T15:51:43.581254+0000","last_undegraded":"2024-09-21T15:51:43.581254+0000","last_fullsized":"2024-09-21T15:51:43.581254+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":698,"log_dups_size":0,"ondisk_log_size":698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:21:37.777769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086632899999999999,"stat_sum":{"num_bytes":0,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":42,"num_read":4945,"num_read_kb":15920,"num_write":889,"num_write_kb":5075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"138'470","reported_seq":3154,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581542+0000","last_change":"2024-09-21T15:51:25.780857+0000","last_active":"2024-09-21T15:51:43.581542+0000","last_peered":"2024-09-21T15:51:43.581542+0000","last_clean":"2024-09-21T15:51:43.581542+0000","last_became_active":"2024-09-21T15:50:32.676147+0000","last_became_peered":"2024-09-21T15:50:32.676147+0000","last_unstale":"2024-09-21T15:51:43.581542+0000","last_undegraded":"2024-09-21T15:51:43.581542+0000","last_fullsized":"2024-09-21T15:51:43.581542+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":470,"log_dups_size":0,"ondisk_log_size":470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:44:18.821310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00024304599999999999,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":40,"num_read":3160,"num_read_kb":11849,"num_write":381,"num_write_kb":4471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":1191778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"148'1012","reported_seq":4197,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581091+0000","last_change":"2024-09-21T15:51:25.697763+0000","last_active":"2024-09-21T15:51:43.581091+0000","last_peered":"2024-09-21T15:51:43.581091+0000","last_clean":"2024-09-21T15:51:43.581091+0000","last_became_active":"2024-09-21T15:49:06.688798+0000","last_became_peered":"2024-09-21T15:49:06.688798+0000","last_unstale":"2024-09-21T15:51:43.581091+0000","last_undegraded":"2024-09-21T15:51:43.581091+0000","last_fullsized":"2024-09-21T15:51:43.581091+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":1012,"log_dups_size":0,"ondisk_log_size":1012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:48:33.679133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053058599999999999,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":30,"num_read":6391,"num_read_kb":16970,"num_write":1355,"num_write_kb":8408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"146'383","reported_seq":3011,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.780811+0000","last_change":"2024-09-21T15:51:25.689299+0000","last_active":"2024-09-21T15:51:42.780811+0000","last_peered":"2024-09-21T15:51:42.780811+0000","last_clean":"2024-09-21T15:51:42.780811+0000","last_became_active":"2024-09-21T15:50:32.671686+0000","last_became_peered":"2024-09-21T15:50:32.671686+0000","last_unstale":"2024-09-21T15:51:42.780811+0000","last_undegraded":"2024-09-21T15:51:42.780811+0000","last_fullsized":"2024-09-21T15:51:42.780811+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":383,"log_dups_size":0,"ondisk_log_size":383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:40:53.968477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012972649999999999,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":34,"num_read":2455,"num_read_kb":11669,"num_write":401,"num_write_kb":13839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":3359702,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"150'967","reported_seq":7431,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581127+0000","last_change":"2024-09-21T15:51:25.699098+0000","last_active":"2024-09-21T15:51:43.581127+0000","last_peered":"2024-09-21T15:51:43.581127+0000","last_clean":"2024-09-21T15:51:43.581127+0000","last_became_active":"2024-09-21T15:49:06.690128+0000","last_became_peered":"2024-09-21T15:49:06.690128+0000","last_unstale":"2024-09-21T15:51:43.581127+0000","last_undegraded":"2024-09-21T15:51:43.581127+0000","last_fullsized":"2024-09-21T15:51:43.581127+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":967,"log_dups_size":0,"ondisk_log_size":967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:11.763959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014652879999999999,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":32,"num_read":9327,"num_read_kb":26357,"num_write":1446,"num_write_kb":13580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":1039,"num_object_clones":0,"num_object_copies":2078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1039,"num_whiteouts":1035,"num_read":207419,"num_read_kb":842423,"num_write":23389,"num_write_kb":303239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3397,"num_bytes_recovered":100270055,"num_keys_recovered":316,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5973,"internal_metadata":0},"log_size":20218,"ondisk_log_size":20218,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574887,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77952,"kb_used_data":2544,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94293888,"statfs":{"total":96636764160,"available":96556941312,"internally_reserved":0,"allocated":2605056,"data_stored":1557596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":77199819},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:32 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.44},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.64100000000000001}]},{"osd":1,"last update":"Sat Sep 21 15:50:52 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.48299999999999998,"15min":0.495},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":0.73199999999999998,"5min":1.04,"15min":1.04},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.44,"5min":0.46999999999999997,"15min":0.47499999999999998},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.64500000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.58599999999999997}]},{"osd":2,"last update":"Sat Sep 21 15:50:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.439,"15min":0.44400000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.53500000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.437,"5min":0.498,"15min":0.50800000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.70099999999999996,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.55300000000000005}]},{"osd":3,"last update":"Sat Sep 21 15:50:52 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.49099999999999999,"15min":0.499},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.65300000000000002,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.432,"5min":0.47099999999999997,"15min":0.47699999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68200000000000005,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.57099999999999995}]},{"osd":4,"last update":"Sat Sep 21 15:50:52 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.41699999999999998,"15min":0.42299999999999999},"min":{"1min":0.24099999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.59999999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.38,"5min":0.39000000000000001,"15min":0.39200000000000002},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.52700000000000002}]},{"osd":5,"last update":"Sat Sep 21 15:50:52 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.439,"15min":0.45000000000000001},"min":{"1min":0.25900000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.52900000000000003,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.44700000000000001,"15min":0.45000000000000001},"min":{"1min":0.29499999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.68600000000000005,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.70199999999999996}]},{"osd":6,"last update":"Sat Sep 21 15:50:52 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.49399999999999999,"15min":0.5},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.72099999999999997,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.74199999999999999,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.77100000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574890,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41284,"kb_used_data":1976,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94330556,"statfs":{"total":96636764160,"available":96594489344,"internally_reserved":0,"allocated":2023424,"data_stored":973041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:31 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.60399999999999998}]},{"osd":1,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.44800000000000001,"15min":0.45800000000000002},"min":{"1min":0.26900000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.53300000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.46800000000000003,"15min":0.47999999999999998},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.53300000000000003,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.48899999999999999}]},{"osd":2,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.47099999999999997,"15min":0.47699999999999998},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.45800000000000002,"15min":0.46200000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.60999999999999999,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.438}]},{"osd":3,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.51400000000000001,"15min":0.52400000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.70399999999999996,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.53100000000000003,"15min":0.53800000000000003},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.73699999999999999,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.51600000000000001}]},{"osd":4,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.40600000000000003,"15min":0.41399999999999998},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.222},"max":{"1min":0.57499999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.379,"5min":0.46700000000000003,"15min":0.48199999999999998},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.76300000000000001,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.65700000000000003}]},{"osd":5,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.44600000000000001,"15min":0.45600000000000002},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.57399999999999995,"5min":0.755,"15min":0.755},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.372,"5min":0.45000000000000001,"15min":0.46300000000000002},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.628,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.626}]},{"osd":7,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.45000000000000001,"15min":0.45700000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.55400000000000005,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.497,"15min":0.51100000000000001},"min":{"1min":0.30499999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.748,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.67600000000000005}]}]},{"osd":5,"up_from":13,"seq":55834574889,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55232,"kb_used_data":1968,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94316608,"statfs":{"total":96636764160,"available":96580206592,"internally_reserved":0,"allocated":2015232,"data_stored":967247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.5},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.66400000000000003}]},{"osd":1,"last update":"Sat Sep 21 15:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.71699999999999997,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.5,"15min":0.51400000000000001},"min":{"1min":0.34399999999999997,"5min":0.314,"15min":0.314},"max":{"1min":0.52200000000000002,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.56699999999999995}]},{"osd":2,"last update":"Sat Sep 21 15:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.49399999999999999,"15min":0.50600000000000001},"min":{"1min":0.34200000000000003,"5min":0.311,"15min":0.311},"max":{"1min":0.56899999999999995,"5min":1.046,"15min":1.046},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.437,"5min":0.46800000000000003,"15min":0.47299999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.68999999999999995,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.68000000000000005}]},{"osd":3,"last update":"Sat Sep 21 15:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.47699999999999998,"15min":0.48699999999999999},"min":{"1min":0.29599999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.55200000000000005,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.53800000000000003,"15min":0.55200000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.76900000000000002,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.59099999999999997}]},{"osd":4,"last update":"Sat Sep 21 15:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.65400000000000003,"15min":0.70799999999999996},"min":{"1min":0.254,"5min":0.253,"15min":0.253},"max":{"1min":0.503,"5min":5.6399999999999997,"15min":5.6399999999999997},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.45000000000000001,"15min":0.47199999999999998},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.48899999999999999,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.69499999999999995}]},{"osd":6,"last update":"Sat Sep 21 15:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.443,"15min":0.45500000000000002},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.51700000000000002,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.48199999999999998,"15min":0.499},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.55200000000000005,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.64100000000000001}]},{"osd":7,"last update":"Sat Sep 21 15:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.47199999999999998,"15min":0.48599999999999999},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.56000000000000005,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.623},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.504,"15min":0.52100000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.439}]}]},{"osd":4,"up_from":13,"seq":55834574888,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65084,"kb_used_data":1964,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94306756,"statfs":{"total":96636764160,"available":96570118144,"internally_reserved":0,"allocated":2011136,"data_stored":965965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.56299999999999994}]},{"osd":1,"last update":"Sat Sep 21 15:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.433,"15min":0.41899999999999998},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":1.8839999999999999,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.33600000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.55000000000000004}]},{"osd":2,"last update":"Sat Sep 21 15:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.443,"15min":0.439},"min":{"1min":0.311,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.501},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.433,"15min":0.433},"min":{"1min":0.31,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.54000000000000004,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.47999999999999998}]},{"osd":3,"last update":"Sat Sep 21 15:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.34200000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.66600000000000004,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.46800000000000003,"15min":0.46999999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.60899999999999999,"5min":0.626,"15min":0.626},"last":0.52600000000000002}]},{"osd":5,"last update":"Sat Sep 21 15:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.47199999999999998,"15min":0.47599999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":1.919,"5min":2.4489999999999998,"15min":2.4489999999999998},"last":0.376},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.42399999999999999,"15min":0.42499999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.748,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.41099999999999998}]},{"osd":6,"last update":"Sat Sep 21 15:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.41599999999999998,"15min":0.41399999999999998},"min":{"1min":0.29399999999999998,"5min":0.245,"15min":0.245},"max":{"1min":0.63800000000000001,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.55700000000000005,"15min":0.54500000000000004},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":4.1340000000000003,"5min":4.1340000000000003,"15min":4.1340000000000003},"last":0.51500000000000001}]},{"osd":7,"last update":"Sat Sep 21 15:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.45400000000000001,"15min":0.434},"min":{"1min":0.28199999999999997,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":2.7599999999999998,"5min":2.7599999999999998,"15min":2.7599999999999998},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.52400000000000002,"15min":0.52200000000000002},"min":{"1min":0.32600000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":2.016,"5min":2.016,"15min":2.016},"last":0.434}]}]},{"osd":3,"up_from":13,"seq":55834574889,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59340,"kb_used_data":1976,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312500,"statfs":{"total":96636764160,"available":96576000000,"internally_reserved":0,"allocated":2023424,"data_stored":973041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":58718664},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.31,"5min":0.31,"15min":0.31},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"last":0.27800000000000002}]},{"osd":1,"last update":"Sat Sep 21 15:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.28999999999999998,"5min":0.33600000000000002,"15min":0.34399999999999997},"min":{"1min":0.219,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.42299999999999999,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.222},{"interface":"front","average":{"1min":0.30499999999999999,"5min":0.36799999999999999,"15min":0.378},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.40500000000000003,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.313}]},{"osd":2,"last update":"Sat Sep 21 15:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.38400000000000001,"15min":0.39300000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.49199999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.40999999999999998,"15min":0.42099999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.52000000000000002,"5min":0.755,"15min":0.755},"last":0.41299999999999998}]},{"osd":4,"last update":"Sat Sep 21 15:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.47899999999999998,"15min":0.49199999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.56699999999999995,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.504,"15min":0.51300000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.60399999999999998,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.51800000000000002}]},{"osd":5,"last update":"Sat Sep 21 15:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.51000000000000001,"15min":0.52500000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.68100000000000005,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.503},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.53400000000000003,"15min":0.55300000000000005},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.54100000000000004,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.48799999999999999}]},{"osd":6,"last update":"Sat Sep 21 15:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.52800000000000002,"15min":0.54200000000000004},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.622,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.53000000000000003,"15min":0.54600000000000004},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.65700000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.57499999999999996}]},{"osd":7,"last update":"Sat Sep 21 15:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.54700000000000004,"15min":0.56200000000000006},"min":{"1min":0.36299999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.60599999999999998,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.54600000000000004,"15min":0.56200000000000006},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.65000000000000002,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.42199999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574889,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37188,"kb_used_data":1976,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334652,"statfs":{"total":96636764160,"available":96598683648,"internally_reserved":0,"allocated":2023424,"data_stored":973041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:29 2024","interfaces":[{"interface":"back","average":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.439,"5min":0.439,"15min":0.439},"last":0.252},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.30099999999999999}]},{"osd":1,"last update":"Sat Sep 21 15:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.40300000000000002,"15min":0.41999999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.439,"5min":1.621,"15min":1.621},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.308,"5min":0.36399999999999999,"15min":0.374},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.46400000000000002,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.28000000000000003}]},{"osd":3,"last update":"Sat Sep 21 15:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.90800000000000003,"15min":0.97099999999999997},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":4.6580000000000004,"5min":11.56,"15min":11.56},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.40200000000000002,"15min":0.41299999999999998},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.46999999999999997,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.32500000000000001}]},{"osd":4,"last update":"Sat Sep 21 15:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.57999999999999996,"15min":0.60199999999999998},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.60499999999999998,"5min":1.653,"15min":1.653},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.60399999999999998,"15min":0.628},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.74199999999999999,"5min":2.4660000000000002,"15min":2.4660000000000002},"last":0.45800000000000002}]},{"osd":5,"last update":"Sat Sep 21 15:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.501,"15min":0.51100000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.61199999999999999,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.437},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.52700000000000002,"15min":0.53900000000000003},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.65400000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.44900000000000001}]},{"osd":6,"last update":"Sat Sep 21 15:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.54900000000000004,"15min":0.56299999999999994},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.67500000000000004,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.54000000000000004,"15min":0.55600000000000005},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.629,"5min":0.873,"15min":0.873},"last":0.41599999999999998}]},{"osd":7,"last update":"Sat Sep 21 15:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.54900000000000004,"15min":0.56100000000000005},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.67000000000000004,"5min":0.876,"15min":0.876},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.58199999999999996,"15min":0.59799999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.754,"5min":1.089,"15min":1.089},"last":0.48299999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574888,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41288,"kb_used_data":1972,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94330552,"statfs":{"total":96636764160,"available":96594485248,"internally_reserved":0,"allocated":2019328,"data_stored":967266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.312,"15min":0.312},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.40100000000000002}]},{"osd":2,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":0.33000000000000002,"15min":0.34000000000000002},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.43099999999999999,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.30299999999999999,"5min":0.499,"15min":0.53200000000000003},"min":{"1min":0.21299999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.52400000000000002,"5min":5.0670000000000002,"15min":5.0670000000000002},"last":0.34200000000000003}]},{"osd":3,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.91700000000000004,"15min":1.0189999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.46000000000000002,"5min":16.032,"15min":16.032},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.309,"5min":0.34999999999999998,"15min":0.35699999999999998},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":0.47899999999999998,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.36499999999999999}]},{"osd":4,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.46800000000000003,"15min":0.47399999999999998},"min":{"1min":0.25900000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.63200000000000001,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.53600000000000003,"15min":0.55100000000000005},"min":{"1min":0.311,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.56799999999999995,"5min":2.2269999999999999,"15min":2.2269999999999999},"last":0.53400000000000003}]},{"osd":5,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.55300000000000005,"15min":0.57299999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.64700000000000002,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.66000000000000003,"15min":0.69599999999999995},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.66200000000000003,"5min":4.6989999999999998,"15min":4.6989999999999998},"last":0.63200000000000001}]},{"osd":6,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.53000000000000003,"15min":0.53900000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.68200000000000005,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.59999999999999998,"15min":0.62},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.64500000000000002,"5min":2.262,"15min":2.262},"last":0.61399999999999999}]},{"osd":7,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.67000000000000004,"15min":0.70399999999999996},"min":{"1min":0.33400000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.69399999999999995,"5min":4.742,"15min":4.742},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.51300000000000001,"15min":0.51800000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.70199999999999996,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.52600000000000002}]}]},{"osd":0,"up_from":93,"seq":399431958545,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39176,"kb_used_data":2556,"kb_used_omap":13,"kb_used_meta":36594,"kb_avail":94332664,"statfs":{"total":96636764160,"available":96596647936,"internally_reserved":0,"allocated":2617344,"data_stored":1563409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13968,"internal_metadata":37472624},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.317,"5min":0.317,"15min":0.317},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"last":0.254}]},{"osd":2,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.313},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.28899999999999998}]},{"osd":3,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.505,"5min":0.505,"15min":0.505},"last":0.34499999999999997}]},{"osd":4,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.44500000000000001}]},{"osd":5,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.374},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.43099999999999999}]},{"osd":6,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.49099999999999999}]},{"osd":7,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.53800000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5973,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T15:51:48.197 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T15:51:48.469 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:51:48.469 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-09-21T15:51:48.480 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":250,"stamp":"2024-09-21T15:51:47.918416+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590406,"num_objects":1043,"num_object_clones":0,"num_object_copies":2086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1043,"num_whiteouts":1035,"num_read":207535,"num_read_kb":842523,"num_write":23550,"num_write_kb":305657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3397,"num_bytes_recovered":100270055,"num_keys_recovered":316,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20338,"ondisk_log_size":20338,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":416544,"kb_used_data":16932,"kb_used_omap":24,"kb_used_meta":399463,"kb_avail":754558176,"statfs":{"total":773094113280,"available":772667572224,"internally_reserved":0,"allocated":17338368,"data_stored":8940606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25118,"internal_metadata":409050594},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001005"},"pg_stats":[{"pgid":"3.1f","version":"150'413","reported_seq":2132,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.781077+0000","last_change":"2024-09-21T15:51:25.690102+0000","last_active":"2024-09-21T15:51:42.781077+0000","last_peered":"2024-09-21T15:51:42.781077+0000","last_clean":"2024-09-21T15:51:42.781077+0000","last_became_active":"2024-09-21T15:50:32.670964+0000","last_became_peered":"2024-09-21T15:50:32.670964+0000","last_unstale":"2024-09-21T15:51:42.781077+0000","last_undegraded":"2024-09-21T15:51:42.781077+0000","last_fullsized":"2024-09-21T15:51:42.781077+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":413,"log_dups_size":0,"ondisk_log_size":413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:09:09.918864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017896030000000001,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":28,"num_read":2413,"num_read_kb":10419,"num_write":442,"num_write_kb":5956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":1974433,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"149'599","reported_seq":6526,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:44.475566+0000","last_change":"2024-09-21T15:51:25.689127+0000","last_active":"2024-09-21T15:51:44.475566+0000","last_peered":"2024-09-21T15:51:44.475566+0000","last_clean":"2024-09-21T15:51:44.475566+0000","last_became_active":"2024-09-21T15:51:24.691676+0000","last_became_peered":"2024-09-21T15:51:24.691676+0000","last_unstale":"2024-09-21T15:51:44.475566+0000","last_undegraded":"2024-09-21T15:51:44.475566+0000","last_fullsized":"2024-09-21T15:51:44.475566+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":599,"log_dups_size":0,"ondisk_log_size":599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:51:29.795001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010326389999999999,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":38,"num_read":7456,"num_read_kb":25912,"num_write":580,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":250,"num_bytes_recovered":3824749,"num_keys_recovered":34,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"150'655","reported_seq":10214,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.783240+0000","last_change":"2024-09-21T15:51:39.767369+0000","last_active":"2024-09-21T15:51:42.783240+0000","last_peered":"2024-09-21T15:51:42.783240+0000","last_clean":"2024-09-21T15:51:42.783240+0000","last_became_active":"2024-09-21T15:51:39.765580+0000","last_became_peered":"2024-09-21T15:51:39.765580+0000","last_unstale":"2024-09-21T15:51:42.783240+0000","last_undegraded":"2024-09-21T15:51:42.783240+0000","last_fullsized":"2024-09-21T15:51:42.783240+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":655,"log_dups_size":0,"ondisk_log_size":655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:40:43.868959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090302100000000005,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":34,"num_read":10756,"num_read_kb":29175,"num_write":703,"num_write_kb":8882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"150'584","reported_seq":4519,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.783179+0000","last_change":"2024-09-21T15:51:39.767392+0000","last_active":"2024-09-21T15:51:42.783179+0000","last_peered":"2024-09-21T15:51:42.783179+0000","last_clean":"2024-09-21T15:51:42.783179+0000","last_became_active":"2024-09-21T15:51:39.765999+0000","last_became_peered":"2024-09-21T15:51:39.765999+0000","last_unstale":"2024-09-21T15:51:42.783179+0000","last_undegraded":"2024-09-21T15:51:42.783179+0000","last_fullsized":"2024-09-21T15:51:42.783179+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":584,"log_dups_size":0,"ondisk_log_size":584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:21:55.582116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010903670000000001,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":27,"num_read":5242,"num_read_kb":17341,"num_write":644,"num_write_kb":4787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"145'508","reported_seq":4404,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.780869+0000","last_change":"2024-09-21T15:51:25.690506+0000","last_active":"2024-09-21T15:51:42.780869+0000","last_peered":"2024-09-21T15:51:42.780869+0000","last_clean":"2024-09-21T15:51:42.780869+0000","last_became_active":"2024-09-21T15:50:32.674250+0000","last_became_peered":"2024-09-21T15:50:32.674250+0000","last_unstale":"2024-09-21T15:51:42.780869+0000","last_undegraded":"2024-09-21T15:51:42.780869+0000","last_fullsized":"2024-09-21T15:51:42.780869+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":508,"log_dups_size":0,"ondisk_log_size":508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:54:57.399818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019320559999999999,"stat_sum":{"num_bytes":0,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":39,"num_read":4815,"num_read_kb":30088,"num_write":498,"num_write_kb":13507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":4759000,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"150'431","reported_seq":6943,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581775+0000","last_change":"2024-09-21T15:51:25.689223+0000","last_active":"2024-09-21T15:51:43.581775+0000","last_peered":"2024-09-21T15:51:43.581775+0000","last_clean":"2024-09-21T15:51:43.581775+0000","last_became_active":"2024-09-21T15:49:06.693903+0000","last_became_peered":"2024-09-21T15:49:06.693903+0000","last_unstale":"2024-09-21T15:51:43.581775+0000","last_undegraded":"2024-09-21T15:51:43.581775+0000","last_fullsized":"2024-09-21T15:51:43.581775+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":431,"log_dups_size":0,"ondisk_log_size":431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:41:45.432893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047963299999999998,"stat_sum":{"num_bytes":19,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":30,"num_read":7221,"num_read_kb":31759,"num_write":421,"num_write_kb":9476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"148'460","reported_seq":8836,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.491336+0000","last_change":"2024-09-21T15:51:25.780904+0000","last_active":"2024-09-21T15:51:43.491336+0000","last_peered":"2024-09-21T15:51:43.491336+0000","last_clean":"2024-09-21T15:51:43.491336+0000","last_became_active":"2024-09-21T15:51:12.688258+0000","last_became_peered":"2024-09-21T15:51:12.688258+0000","last_unstale":"2024-09-21T15:51:43.491336+0000","last_undegraded":"2024-09-21T15:51:43.491336+0000","last_fullsized":"2024-09-21T15:51:43.491336+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":460,"log_dups_size":0,"ondisk_log_size":460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:19:38.132074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000411691,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":32,"num_read":8283,"num_read_kb":39579,"num_write":518,"num_write_kb":7408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":236,"num_bytes_recovered":6705120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"149'537","reported_seq":4716,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:44.475480+0000","last_change":"2024-09-21T15:51:35.753555+0000","last_active":"2024-09-21T15:51:44.475480+0000","last_peered":"2024-09-21T15:51:44.475480+0000","last_clean":"2024-09-21T15:51:44.475480+0000","last_became_active":"2024-09-21T15:51:35.751700+0000","last_became_peered":"2024-09-21T15:51:35.751700+0000","last_unstale":"2024-09-21T15:51:44.475480+0000","last_undegraded":"2024-09-21T15:51:44.475480+0000","last_fullsized":"2024-09-21T15:51:44.475480+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":537,"log_dups_size":0,"ondisk_log_size":537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:20:44.688916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051035439999999998,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":26,"num_read":6190,"num_read_kb":30937,"num_write":534,"num_write_kb":11075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":282,"num_bytes_recovered":11320990,"num_keys_recovered":70,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"149'1040","reported_seq":9412,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.781000+0000","last_change":"2024-09-21T15:51:25.689670+0000","last_active":"2024-09-21T15:51:42.781000+0000","last_peered":"2024-09-21T15:51:42.781000+0000","last_clean":"2024-09-21T15:51:42.781000+0000","last_became_active":"2024-09-21T15:50:32.668771+0000","last_became_peered":"2024-09-21T15:50:32.668771+0000","last_unstale":"2024-09-21T15:51:42.781000+0000","last_undegraded":"2024-09-21T15:51:42.781000+0000","last_fullsized":"2024-09-21T15:51:42.781000+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":1040,"log_dups_size":0,"ondisk_log_size":1040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:39:43.554150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001503861,"stat_sum":{"num_bytes":0,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":36,"num_read":12714,"num_read_kb":21487,"num_write":1303,"num_write_kb":8257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":3319185,"num_keys_recovered":34,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"150'540","reported_seq":6716,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.783295+0000","last_change":"2024-09-21T15:51:25.757086+0000","last_active":"2024-09-21T15:51:42.783295+0000","last_peered":"2024-09-21T15:51:42.783295+0000","last_clean":"2024-09-21T15:51:42.783295+0000","last_became_active":"2024-09-21T15:49:06.688808+0000","last_became_peered":"2024-09-21T15:49:06.688808+0000","last_unstale":"2024-09-21T15:51:42.783295+0000","last_undegraded":"2024-09-21T15:51:42.783295+0000","last_fullsized":"2024-09-21T15:51:42.783295+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":540,"log_dups_size":0,"ondisk_log_size":540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:43:02.012254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00023944,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":30,"num_read":7604,"num_read_kb":49865,"num_write":542,"num_write_kb":13823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"150'530","reported_seq":4324,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581211+0000","last_change":"2024-09-21T15:51:25.699156+0000","last_active":"2024-09-21T15:51:43.581211+0000","last_peered":"2024-09-21T15:51:43.581211+0000","last_clean":"2024-09-21T15:51:43.581211+0000","last_became_active":"2024-09-21T15:51:11.841664+0000","last_became_peered":"2024-09-21T15:51:11.841664+0000","last_unstale":"2024-09-21T15:51:43.581211+0000","last_undegraded":"2024-09-21T15:51:43.581211+0000","last_fullsized":"2024-09-21T15:51:43.581211+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:47:37.118354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001532828,"stat_sum":{"num_bytes":0,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":29,"num_read":4052,"num_read_kb":27144,"num_write":637,"num_write_kb":17935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":222,"num_bytes_recovered":4468215,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"147'1207","reported_seq":4387,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581846+0000","last_change":"2024-09-21T15:51:25.690386+0000","last_active":"2024-09-21T15:51:43.581846+0000","last_peered":"2024-09-21T15:51:43.581846+0000","last_clean":"2024-09-21T15:51:43.581846+0000","last_became_active":"2024-09-21T15:49:06.688803+0000","last_became_peered":"2024-09-21T15:49:06.688803+0000","last_unstale":"2024-09-21T15:51:43.581846+0000","last_undegraded":"2024-09-21T15:51:43.581846+0000","last_fullsized":"2024-09-21T15:51:43.581846+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":1207,"log_dups_size":0,"ondisk_log_size":1207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:18:47.662631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013586710000000001,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":26,"num_read":7764,"num_read_kb":18232,"num_write":1926,"num_write_kb":6190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":161,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581555+0000","last_change":"2024-09-21T15:48:59.666431+0000","last_active":"2024-09-21T15:51:43.581555+0000","last_peered":"2024-09-21T15:51:43.581555+0000","last_clean":"2024-09-21T15:51:43.581555+0000","last_became_active":"2024-09-21T15:48:57.583665+0000","last_became_peered":"2024-09-21T15:48:57.583665+0000","last_unstale":"2024-09-21T15:51:43.581555+0000","last_undegraded":"2024-09-21T15:51:43.581555+0000","last_fullsized":"2024-09-21T15:51:43.581555+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:52:09.228964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031098399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"150'438","reported_seq":3064,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.780970+0000","last_change":"2024-09-21T15:51:25.689587+0000","last_active":"2024-09-21T15:51:42.780970+0000","last_peered":"2024-09-21T15:51:42.780970+0000","last_clean":"2024-09-21T15:51:42.780970+0000","last_became_active":"2024-09-21T15:50:32.671424+0000","last_became_peered":"2024-09-21T15:50:32.671424+0000","last_unstale":"2024-09-21T15:51:42.780970+0000","last_undegraded":"2024-09-21T15:51:42.780970+0000","last_fullsized":"2024-09-21T15:51:42.780970+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":438,"log_dups_size":0,"ondisk_log_size":438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:04:14.554732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001507467,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":35,"num_read":2637,"num_read_kb":12956,"num_write":388,"num_write_kb":8392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":4494388,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":164,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.491395+0000","last_change":"2024-09-21T15:48:59.589790+0000","last_active":"2024-09-21T15:51:43.491395+0000","last_peered":"2024-09-21T15:51:43.491395+0000","last_clean":"2024-09-21T15:51:43.491395+0000","last_became_active":"2024-09-21T15:48:57.582121+0000","last_became_peered":"2024-09-21T15:48:57.582121+0000","last_unstale":"2024-09-21T15:51:43.491395+0000","last_undegraded":"2024-09-21T15:51:43.491395+0000","last_fullsized":"2024-09-21T15:51:43.491395+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:33:47.592624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039654700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"148'579","reported_seq":8001,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:44.475582+0000","last_change":"2024-09-21T15:51:35.754312+0000","last_active":"2024-09-21T15:51:44.475582+0000","last_peered":"2024-09-21T15:51:44.475582+0000","last_clean":"2024-09-21T15:51:44.475582+0000","last_became_active":"2024-09-21T15:51:35.752700+0000","last_became_peered":"2024-09-21T15:51:35.752700+0000","last_unstale":"2024-09-21T15:51:44.475582+0000","last_undegraded":"2024-09-21T15:51:44.475582+0000","last_fullsized":"2024-09-21T15:51:44.475582+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":579,"log_dups_size":0,"ondisk_log_size":579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:37:03.162105+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.00086244900000000005,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":35,"num_read":8380,"num_read_kb":21195,"num_write":544,"num_write_kb":5397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":256,"num_bytes_recovered":3760369,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":175,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581295+0000","last_change":"2024-09-21T15:50:30.146743+0000","last_active":"2024-09-21T15:51:43.581295+0000","last_peered":"2024-09-21T15:51:43.581295+0000","last_clean":"2024-09-21T15:51:43.581295+0000","last_became_active":"2024-09-21T15:50:30.146428+0000","last_became_peered":"2024-09-21T15:50:30.146428+0000","last_unstale":"2024-09-21T15:51:43.581295+0000","last_undegraded":"2024-09-21T15:51:43.581295+0000","last_fullsized":"2024-09-21T15:51:43.581295+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:58:38.689807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036074600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"145'703","reported_seq":8853,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.491362+0000","last_change":"2024-09-21T15:51:25.780818+0000","last_active":"2024-09-21T15:51:43.491362+0000","last_peered":"2024-09-21T15:51:43.491362+0000","last_clean":"2024-09-21T15:51:43.491362+0000","last_became_active":"2024-09-21T15:50:35.682788+0000","last_became_peered":"2024-09-21T15:50:35.682788+0000","last_unstale":"2024-09-21T15:51:43.491362+0000","last_undegraded":"2024-09-21T15:51:43.491362+0000","last_fullsized":"2024-09-21T15:51:43.491362+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":703,"log_dups_size":0,"ondisk_log_size":703,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:20:40.695690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032907100000000001,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":26,"num_read":11015,"num_read_kb":27666,"num_write":868,"num_write_kb":11017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":175,"num_bytes_recovered":14134268,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":177,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.491361+0000","last_change":"2024-09-21T15:50:30.146654+0000","last_active":"2024-09-21T15:51:43.491361+0000","last_peered":"2024-09-21T15:51:43.491361+0000","last_clean":"2024-09-21T15:51:43.491361+0000","last_became_active":"2024-09-21T15:50:30.146273+0000","last_became_peered":"2024-09-21T15:50:30.146273+0000","last_unstale":"2024-09-21T15:51:43.491361+0000","last_undegraded":"2024-09-21T15:51:43.491361+0000","last_fullsized":"2024-09-21T15:51:43.491361+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:21:48.178519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040525700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"150'603","reported_seq":5376,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.783334+0000","last_change":"2024-09-21T15:51:39.769574+0000","last_active":"2024-09-21T15:51:42.783334+0000","last_peered":"2024-09-21T15:51:42.783334+0000","last_clean":"2024-09-21T15:51:42.783334+0000","last_became_active":"2024-09-21T15:51:39.768300+0000","last_became_peered":"2024-09-21T15:51:39.768300+0000","last_unstale":"2024-09-21T15:51:42.783334+0000","last_undegraded":"2024-09-21T15:51:42.783334+0000","last_fullsized":"2024-09-21T15:51:42.783334+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":603,"log_dups_size":0,"ondisk_log_size":603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:33:47.511137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050855200000000001,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":35,"num_read":5730,"num_read_kb":33940,"num_write":686,"num_write_kb":16332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":268,"num_bytes_recovered":7823491,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"150'921","reported_seq":4195,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581881+0000","last_change":"2024-09-21T15:51:25.690079+0000","last_active":"2024-09-21T15:51:43.581881+0000","last_peered":"2024-09-21T15:51:43.581881+0000","last_clean":"2024-09-21T15:51:43.581881+0000","last_became_active":"2024-09-21T15:50:32.677456+0000","last_became_peered":"2024-09-21T15:50:32.677456+0000","last_unstale":"2024-09-21T15:51:43.581881+0000","last_undegraded":"2024-09-21T15:51:43.581881+0000","last_fullsized":"2024-09-21T15:51:43.581881+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":921,"log_dups_size":0,"ondisk_log_size":921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:46:31.194345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010737730000000001,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":34,"num_read":5136,"num_read_kb":19340,"num_write":949,"num_write_kb":7405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":632176,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"20'2","reported_seq":59,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.783126+0000","last_change":"2024-09-21T15:51:39.764743+0000","last_active":"2024-09-21T15:51:42.783126+0000","last_peered":"2024-09-21T15:51:42.783126+0000","last_clean":"2024-09-21T15:51:42.783126+0000","last_became_active":"2024-09-21T15:51:39.764409+0000","last_became_peered":"2024-09-21T15:51:39.764409+0000","last_unstale":"2024-09-21T15:51:42.783126+0000","last_undegraded":"2024-09-21T15:51:42.783126+0000","last_fullsized":"2024-09-21T15:51:42.783126+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:28:14.932986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000859595,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"148'393","reported_seq":4833,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.491393+0000","last_change":"2024-09-21T15:51:25.781082+0000","last_active":"2024-09-21T15:51:43.491393+0000","last_peered":"2024-09-21T15:51:43.491393+0000","last_clean":"2024-09-21T15:51:43.491393+0000","last_became_active":"2024-09-21T15:50:35.682001+0000","last_became_peered":"2024-09-21T15:50:35.682001+0000","last_unstale":"2024-09-21T15:51:43.491393+0000","last_undegraded":"2024-09-21T15:51:43.491393+0000","last_fullsized":"2024-09-21T15:51:43.491393+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":393,"log_dups_size":0,"ondisk_log_size":393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:09:15.325457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044281000000000002,"stat_sum":{"num_bytes":0,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":31,"num_read":4285,"num_read_kb":23375,"num_write":379,"num_write_kb":4145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":187,"num_bytes_recovered":4305276,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":97,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.788209+0000","last_change":"2024-09-21T15:51:07.683738+0000","last_active":"2024-09-21T15:51:43.788209+0000","last_peered":"2024-09-21T15:51:43.788209+0000","last_clean":"2024-09-21T15:51:43.788209+0000","last_became_active":"2024-09-21T15:51:07.683418+0000","last_became_peered":"2024-09-21T15:51:07.683418+0000","last_unstale":"2024-09-21T15:51:43.788209+0000","last_undegraded":"2024-09-21T15:51:43.788209+0000","last_fullsized":"2024-09-21T15:51:43.788209+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:25:13.468653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"150'558","reported_seq":2598,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.781077+0000","last_change":"2024-09-21T15:51:25.689707+0000","last_active":"2024-09-21T15:51:42.781077+0000","last_peered":"2024-09-21T15:51:42.781077+0000","last_clean":"2024-09-21T15:51:42.781077+0000","last_became_active":"2024-09-21T15:50:32.671413+0000","last_became_peered":"2024-09-21T15:50:32.671413+0000","last_unstale":"2024-09-21T15:51:42.781077+0000","last_undegraded":"2024-09-21T15:51:42.781077+0000","last_fullsized":"2024-09-21T15:51:42.781077+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":558,"log_dups_size":0,"ondisk_log_size":558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:04:19.509402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001292872,"stat_sum":{"num_bytes":0,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":39,"num_read":3663,"num_read_kb":16543,"num_write":590,"num_write_kb":5977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":1696547,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":162,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581078+0000","last_change":"2024-09-21T15:49:00.114730+0000","last_active":"2024-09-21T15:51:43.581078+0000","last_peered":"2024-09-21T15:51:43.581078+0000","last_clean":"2024-09-21T15:51:43.581078+0000","last_became_active":"2024-09-21T15:48:57.585525+0000","last_became_peered":"2024-09-21T15:48:57.585525+0000","last_unstale":"2024-09-21T15:51:43.581078+0000","last_undegraded":"2024-09-21T15:51:43.581078+0000","last_fullsized":"2024-09-21T15:51:43.581078+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:35:35.786526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031691100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"150'413","reported_seq":2957,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:44.475526+0000","last_change":"2024-09-21T15:51:35.754599+0000","last_active":"2024-09-21T15:51:44.475526+0000","last_peered":"2024-09-21T15:51:44.475526+0000","last_clean":"2024-09-21T15:51:44.475526+0000","last_became_active":"2024-09-21T15:51:35.752810+0000","last_became_peered":"2024-09-21T15:51:35.752810+0000","last_unstale":"2024-09-21T15:51:44.475526+0000","last_undegraded":"2024-09-21T15:51:44.475526+0000","last_fullsized":"2024-09-21T15:51:44.475526+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":413,"log_dups_size":0,"ondisk_log_size":413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:54:49.851651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033178999999999999,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":30,"num_read":2149,"num_read_kb":34452,"num_write":406,"num_write_kb":12612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":214,"num_bytes_recovered":2197196,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"18'1","reported_seq":58,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.783151+0000","last_change":"2024-09-21T15:51:39.764710+0000","last_active":"2024-09-21T15:51:42.783151+0000","last_peered":"2024-09-21T15:51:42.783151+0000","last_clean":"2024-09-21T15:51:42.783151+0000","last_became_active":"2024-09-21T15:51:39.764411+0000","last_became_peered":"2024-09-21T15:51:39.764411+0000","last_unstale":"2024-09-21T15:51:42.783151+0000","last_undegraded":"2024-09-21T15:51:42.783151+0000","last_fullsized":"2024-09-21T15:51:42.783151+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:29:50.093645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039559000000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":323,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581078+0000","last_change":"2024-09-21T15:50:30.147280+0000","last_active":"2024-09-21T15:51:43.581078+0000","last_peered":"2024-09-21T15:51:43.581078+0000","last_clean":"2024-09-21T15:51:43.581078+0000","last_became_active":"2024-09-21T15:50:30.146987+0000","last_became_peered":"2024-09-21T15:50:30.146987+0000","last_unstale":"2024-09-21T15:51:43.581078+0000","last_undegraded":"2024-09-21T15:51:43.581078+0000","last_fullsized":"2024-09-21T15:51:43.581078+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:50.161884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:50.161884+0000","last_clean_scrub_stamp":"2024-09-21T15:48:50.161884+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:26:32.853650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"150'540","reported_seq":7944,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:44.475450+0000","last_change":"2024-09-21T15:51:25.689387+0000","last_active":"2024-09-21T15:51:44.475450+0000","last_peered":"2024-09-21T15:51:44.475450+0000","last_clean":"2024-09-21T15:51:44.475450+0000","last_became_active":"2024-09-21T15:51:11.837877+0000","last_became_peered":"2024-09-21T15:51:11.837877+0000","last_unstale":"2024-09-21T15:51:44.475450+0000","last_undegraded":"2024-09-21T15:51:44.475450+0000","last_fullsized":"2024-09-21T15:51:44.475450+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":540,"log_dups_size":0,"ondisk_log_size":540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:45:58.418197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079839600000000004,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":32,"num_read":7239,"num_read_kb":29580,"num_write":558,"num_write_kb":4793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":418,"num_bytes_recovered":6262707,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"150'389","reported_seq":3170,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581718+0000","last_change":"2024-09-21T15:51:25.689626+0000","last_active":"2024-09-21T15:51:43.581718+0000","last_peered":"2024-09-21T15:51:43.581718+0000","last_clean":"2024-09-21T15:51:43.581718+0000","last_became_active":"2024-09-21T15:49:07.210506+0000","last_became_peered":"2024-09-21T15:49:07.210506+0000","last_unstale":"2024-09-21T15:51:43.581718+0000","last_undegraded":"2024-09-21T15:51:43.581718+0000","last_fullsized":"2024-09-21T15:51:43.581718+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":389,"log_dups_size":0,"ondisk_log_size":389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:55:18.046738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089193900000000001,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":30,"num_read":2716,"num_read_kb":28245,"num_write":454,"num_write_kb":12090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"149'595","reported_seq":12267,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581556+0000","last_change":"2024-09-21T15:51:41.782922+0000","last_active":"2024-09-21T15:51:43.581556+0000","last_peered":"2024-09-21T15:51:43.581556+0000","last_clean":"2024-09-21T15:51:43.581556+0000","last_became_active":"2024-09-21T15:51:41.781781+0000","last_became_peered":"2024-09-21T15:51:41.781781+0000","last_unstale":"2024-09-21T15:51:43.581556+0000","last_undegraded":"2024-09-21T15:51:43.581556+0000","last_fullsized":"2024-09-21T15:51:43.581556+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":595,"log_dups_size":0,"ondisk_log_size":595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:10:04.846054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000309432,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":29,"num_read":12828,"num_read_kb":63467,"num_write":580,"num_write_kb":14212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"147'851","reported_seq":3855,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:44.475479+0000","last_change":"2024-09-21T15:51:35.752901+0000","last_active":"2024-09-21T15:51:44.475479+0000","last_peered":"2024-09-21T15:51:44.475479+0000","last_clean":"2024-09-21T15:51:44.475479+0000","last_became_active":"2024-09-21T15:51:35.751150+0000","last_became_peered":"2024-09-21T15:51:35.751150+0000","last_unstale":"2024-09-21T15:51:44.475479+0000","last_undegraded":"2024-09-21T15:51:44.475479+0000","last_fullsized":"2024-09-21T15:51:44.475479+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":851,"log_dups_size":0,"ondisk_log_size":851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:33:02.873060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001158526,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":26,"num_read":5948,"num_read_kb":20169,"num_write":1244,"num_write_kb":7750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":262,"num_bytes_recovered":8353711,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"150'739","reported_seq":6768,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.783290+0000","last_change":"2024-09-21T15:51:39.769469+0000","last_active":"2024-09-21T15:51:42.783290+0000","last_peered":"2024-09-21T15:51:42.783290+0000","last_clean":"2024-09-21T15:51:42.783290+0000","last_became_active":"2024-09-21T15:51:39.768321+0000","last_became_peered":"2024-09-21T15:51:39.768321+0000","last_unstale":"2024-09-21T15:51:42.783290+0000","last_undegraded":"2024-09-21T15:51:42.783290+0000","last_fullsized":"2024-09-21T15:51:42.783290+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":739,"log_dups_size":0,"ondisk_log_size":739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:21:29.566035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074635999999999995,"stat_sum":{"num_bytes":0,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":36,"num_read":8849,"num_read_kb":25678,"num_write":868,"num_write_kb":5005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":246,"num_bytes_recovered":5686754,"num_keys_recovered":23,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"147'894","reported_seq":4314,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581242+0000","last_change":"2024-09-21T15:51:25.698246+0000","last_active":"2024-09-21T15:51:43.581242+0000","last_peered":"2024-09-21T15:51:43.581242+0000","last_clean":"2024-09-21T15:51:43.581242+0000","last_became_active":"2024-09-21T15:49:06.689203+0000","last_became_peered":"2024-09-21T15:49:06.689203+0000","last_unstale":"2024-09-21T15:51:43.581242+0000","last_undegraded":"2024-09-21T15:51:43.581242+0000","last_fullsized":"2024-09-21T15:51:43.581242+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":894,"log_dups_size":0,"ondisk_log_size":894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:22:17.578359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084935700000000002,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":35,"num_read":7321,"num_read_kb":28131,"num_write":1146,"num_write_kb":14809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"148'568","reported_seq":8144,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581299+0000","last_change":"2024-09-21T15:51:25.697735+0000","last_active":"2024-09-21T15:51:43.581299+0000","last_peered":"2024-09-21T15:51:43.581299+0000","last_clean":"2024-09-21T15:51:43.581299+0000","last_became_active":"2024-09-21T15:49:06.690101+0000","last_became_peered":"2024-09-21T15:49:06.690101+0000","last_unstale":"2024-09-21T15:51:43.581299+0000","last_undegraded":"2024-09-21T15:51:43.581299+0000","last_fullsized":"2024-09-21T15:51:43.581299+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":568,"log_dups_size":0,"ondisk_log_size":568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:43:30.659104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044571200000000002,"stat_sum":{"num_bytes":0,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":29,"num_read":8735,"num_read_kb":42983,"num_write":509,"num_write_kb":16273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"145'698","reported_seq":3799,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581254+0000","last_change":"2024-09-21T15:51:25.698380+0000","last_active":"2024-09-21T15:51:43.581254+0000","last_peered":"2024-09-21T15:51:43.581254+0000","last_clean":"2024-09-21T15:51:43.581254+0000","last_became_active":"2024-09-21T15:49:06.694250+0000","last_became_peered":"2024-09-21T15:49:06.694250+0000","last_unstale":"2024-09-21T15:51:43.581254+0000","last_undegraded":"2024-09-21T15:51:43.581254+0000","last_fullsized":"2024-09-21T15:51:43.581254+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":698,"log_dups_size":0,"ondisk_log_size":698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:21:37.777769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086632899999999999,"stat_sum":{"num_bytes":0,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":42,"num_read":4945,"num_read_kb":15920,"num_write":889,"num_write_kb":5075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"138'470","reported_seq":3154,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581542+0000","last_change":"2024-09-21T15:51:25.780857+0000","last_active":"2024-09-21T15:51:43.581542+0000","last_peered":"2024-09-21T15:51:43.581542+0000","last_clean":"2024-09-21T15:51:43.581542+0000","last_became_active":"2024-09-21T15:50:32.676147+0000","last_became_peered":"2024-09-21T15:50:32.676147+0000","last_unstale":"2024-09-21T15:51:43.581542+0000","last_undegraded":"2024-09-21T15:51:43.581542+0000","last_fullsized":"2024-09-21T15:51:43.581542+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":470,"log_dups_size":0,"ondisk_log_size":470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:44:18.821310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00024304599999999999,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":40,"num_read":3160,"num_read_kb":11849,"num_write":381,"num_write_kb":4471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":1191778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"148'1012","reported_seq":4197,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581091+0000","last_change":"2024-09-21T15:51:25.697763+0000","last_active":"2024-09-21T15:51:43.581091+0000","last_peered":"2024-09-21T15:51:43.581091+0000","last_clean":"2024-09-21T15:51:43.581091+0000","last_became_active":"2024-09-21T15:49:06.688798+0000","last_became_peered":"2024-09-21T15:49:06.688798+0000","last_unstale":"2024-09-21T15:51:43.581091+0000","last_undegraded":"2024-09-21T15:51:43.581091+0000","last_fullsized":"2024-09-21T15:51:43.581091+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":1012,"log_dups_size":0,"ondisk_log_size":1012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:48:33.679133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053058599999999999,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":30,"num_read":6391,"num_read_kb":16970,"num_write":1355,"num_write_kb":8408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"146'383","reported_seq":3011,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.780811+0000","last_change":"2024-09-21T15:51:25.689299+0000","last_active":"2024-09-21T15:51:42.780811+0000","last_peered":"2024-09-21T15:51:42.780811+0000","last_clean":"2024-09-21T15:51:42.780811+0000","last_became_active":"2024-09-21T15:50:32.671686+0000","last_became_peered":"2024-09-21T15:50:32.671686+0000","last_unstale":"2024-09-21T15:51:42.780811+0000","last_undegraded":"2024-09-21T15:51:42.780811+0000","last_fullsized":"2024-09-21T15:51:42.780811+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":383,"log_dups_size":0,"ondisk_log_size":383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:40:53.968477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012972649999999999,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":34,"num_read":2455,"num_read_kb":11669,"num_write":401,"num_write_kb":13839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":3359702,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"150'967","reported_seq":7431,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581127+0000","last_change":"2024-09-21T15:51:25.699098+0000","last_active":"2024-09-21T15:51:43.581127+0000","last_peered":"2024-09-21T15:51:43.581127+0000","last_clean":"2024-09-21T15:51:43.581127+0000","last_became_active":"2024-09-21T15:49:06.690128+0000","last_became_peered":"2024-09-21T15:49:06.690128+0000","last_unstale":"2024-09-21T15:51:43.581127+0000","last_undegraded":"2024-09-21T15:51:43.581127+0000","last_fullsized":"2024-09-21T15:51:43.581127+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":967,"log_dups_size":0,"ondisk_log_size":967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:11.763959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014652879999999999,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":32,"num_read":9327,"num_read_kb":26357,"num_write":1446,"num_write_kb":13580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":1039,"num_object_clones":0,"num_object_copies":2078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1039,"num_whiteouts":1035,"num_read":207419,"num_read_kb":842423,"num_write":23389,"num_write_kb":303239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3397,"num_bytes_recovered":100270055,"num_keys_recovered":316,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5973,"internal_metadata":0},"log_size":20218,"ondisk_log_size":20218,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574887,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77952,"kb_used_data":2544,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94293888,"statfs":{"total":96636764160,"available":96556941312,"internally_reserved":0,"allocated":2605056,"data_stored":1557596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":77199819},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:32 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.44},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.64100000000000001}]},{"osd":1,"last update":"Sat Sep 21 15:50:52 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.48299999999999998,"15min":0.495},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":0.73199999999999998,"5min":1.04,"15min":1.04},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.44,"5min":0.46999999999999997,"15min":0.47499999999999998},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.64500000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.58599999999999997}]},{"osd":2,"last update":"Sat Sep 21 15:50:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.439,"15min":0.44400000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.53500000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.437,"5min":0.498,"15min":0.50800000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.70099999999999996,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.55300000000000005}]},{"osd":3,"last update":"Sat Sep 21 15:50:52 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.49099999999999999,"15min":0.499},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.65300000000000002,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.432,"5min":0.47099999999999997,"15min":0.47699999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68200000000000005,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.57099999999999995}]},{"osd":4,"last update":"Sat Sep 21 15:50:52 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.41699999999999998,"15min":0.42299999999999999},"min":{"1min":0.24099999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.59999999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.38,"5min":0.39000000000000001,"15min":0.39200000000000002},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.52700000000000002}]},{"osd":5,"last update":"Sat Sep 21 15:50:52 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.439,"15min":0.45000000000000001},"min":{"1min":0.25900000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.52900000000000003,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.44700000000000001,"15min":0.45000000000000001},"min":{"1min":0.29499999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.68600000000000005,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.70199999999999996}]},{"osd":6,"last update":"Sat Sep 21 15:50:52 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.49399999999999999,"15min":0.5},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.72099999999999997,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.74199999999999999,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.77100000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574890,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41284,"kb_used_data":1976,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94330556,"statfs":{"total":96636764160,"available":96594489344,"internally_reserved":0,"allocated":2023424,"data_stored":973041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:31 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.60399999999999998}]},{"osd":1,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.44800000000000001,"15min":0.45800000000000002},"min":{"1min":0.26900000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.53300000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.46800000000000003,"15min":0.47999999999999998},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.53300000000000003,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.48899999999999999}]},{"osd":2,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.47099999999999997,"15min":0.47699999999999998},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.45800000000000002,"15min":0.46200000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.60999999999999999,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.438}]},{"osd":3,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.51400000000000001,"15min":0.52400000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.70399999999999996,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.53100000000000003,"15min":0.53800000000000003},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.73699999999999999,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.51600000000000001}]},{"osd":4,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.40600000000000003,"15min":0.41399999999999998},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.222},"max":{"1min":0.57499999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.379,"5min":0.46700000000000003,"15min":0.48199999999999998},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.76300000000000001,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.65700000000000003}]},{"osd":5,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.44600000000000001,"15min":0.45600000000000002},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.57399999999999995,"5min":0.755,"15min":0.755},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.372,"5min":0.45000000000000001,"15min":0.46300000000000002},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.628,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.626}]},{"osd":7,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.45000000000000001,"15min":0.45700000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.55400000000000005,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.497,"15min":0.51100000000000001},"min":{"1min":0.30499999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.748,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.67600000000000005}]}]},{"osd":5,"up_from":13,"seq":55834574889,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55232,"kb_used_data":1968,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94316608,"statfs":{"total":96636764160,"available":96580206592,"internally_reserved":0,"allocated":2015232,"data_stored":967247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.5},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.66400000000000003}]},{"osd":1,"last update":"Sat Sep 21 15:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.71699999999999997,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.5,"15min":0.51400000000000001},"min":{"1min":0.34399999999999997,"5min":0.314,"15min":0.314},"max":{"1min":0.52200000000000002,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.56699999999999995}]},{"osd":2,"last update":"Sat Sep 21 15:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.49399999999999999,"15min":0.50600000000000001},"min":{"1min":0.34200000000000003,"5min":0.311,"15min":0.311},"max":{"1min":0.56899999999999995,"5min":1.046,"15min":1.046},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.437,"5min":0.46800000000000003,"15min":0.47299999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.68999999999999995,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.68000000000000005}]},{"osd":3,"last update":"Sat Sep 21 15:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.47699999999999998,"15min":0.48699999999999999},"min":{"1min":0.29599999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.55200000000000005,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.53800000000000003,"15min":0.55200000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.76900000000000002,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.59099999999999997}]},{"osd":4,"last update":"Sat Sep 21 15:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.65400000000000003,"15min":0.70799999999999996},"min":{"1min":0.254,"5min":0.253,"15min":0.253},"max":{"1min":0.503,"5min":5.6399999999999997,"15min":5.6399999999999997},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.45000000000000001,"15min":0.47199999999999998},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.48899999999999999,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.69499999999999995}]},{"osd":6,"last update":"Sat Sep 21 15:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.443,"15min":0.45500000000000002},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.51700000000000002,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.48199999999999998,"15min":0.499},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.55200000000000005,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.64100000000000001}]},{"osd":7,"last update":"Sat Sep 21 15:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.47199999999999998,"15min":0.48599999999999999},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.56000000000000005,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.623},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.504,"15min":0.52100000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.439}]}]},{"osd":4,"up_from":13,"seq":55834574888,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65084,"kb_used_data":1964,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94306756,"statfs":{"total":96636764160,"available":96570118144,"internally_reserved":0,"allocated":2011136,"data_stored":965965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.56299999999999994}]},{"osd":1,"last update":"Sat Sep 21 15:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.433,"15min":0.41899999999999998},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":1.8839999999999999,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.33600000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.55000000000000004}]},{"osd":2,"last update":"Sat Sep 21 15:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.443,"15min":0.439},"min":{"1min":0.311,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.501},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.433,"15min":0.433},"min":{"1min":0.31,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.54000000000000004,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.47999999999999998}]},{"osd":3,"last update":"Sat Sep 21 15:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.34200000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.66600000000000004,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.46800000000000003,"15min":0.46999999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.60899999999999999,"5min":0.626,"15min":0.626},"last":0.52600000000000002}]},{"osd":5,"last update":"Sat Sep 21 15:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.47199999999999998,"15min":0.47599999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":1.919,"5min":2.4489999999999998,"15min":2.4489999999999998},"last":0.376},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.42399999999999999,"15min":0.42499999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.748,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.41099999999999998}]},{"osd":6,"last update":"Sat Sep 21 15:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.41599999999999998,"15min":0.41399999999999998},"min":{"1min":0.29399999999999998,"5min":0.245,"15min":0.245},"max":{"1min":0.63800000000000001,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.55700000000000005,"15min":0.54500000000000004},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":4.1340000000000003,"5min":4.1340000000000003,"15min":4.1340000000000003},"last":0.51500000000000001}]},{"osd":7,"last update":"Sat Sep 21 15:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.45400000000000001,"15min":0.434},"min":{"1min":0.28199999999999997,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":2.7599999999999998,"5min":2.7599999999999998,"15min":2.7599999999999998},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.52400000000000002,"15min":0.52200000000000002},"min":{"1min":0.32600000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":2.016,"5min":2.016,"15min":2.016},"last":0.434}]}]},{"osd":3,"up_from":13,"seq":55834574889,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59340,"kb_used_data":1976,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312500,"statfs":{"total":96636764160,"available":96576000000,"internally_reserved":0,"allocated":2023424,"data_stored":973041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":58718664},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.31,"5min":0.31,"15min":0.31},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"last":0.27800000000000002}]},{"osd":1,"last update":"Sat Sep 21 15:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.28999999999999998,"5min":0.33600000000000002,"15min":0.34399999999999997},"min":{"1min":0.219,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.42299999999999999,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.222},{"interface":"front","average":{"1min":0.30499999999999999,"5min":0.36799999999999999,"15min":0.378},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.40500000000000003,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.313}]},{"osd":2,"last update":"Sat Sep 21 15:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.38400000000000001,"15min":0.39300000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.49199999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.40999999999999998,"15min":0.42099999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.52000000000000002,"5min":0.755,"15min":0.755},"last":0.41299999999999998}]},{"osd":4,"last update":"Sat Sep 21 15:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.47899999999999998,"15min":0.49199999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.56699999999999995,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.504,"15min":0.51300000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.60399999999999998,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.51800000000000002}]},{"osd":5,"last update":"Sat Sep 21 15:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.51000000000000001,"15min":0.52500000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.68100000000000005,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.503},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.53400000000000003,"15min":0.55300000000000005},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.54100000000000004,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.48799999999999999}]},{"osd":6,"last update":"Sat Sep 21 15:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.52800000000000002,"15min":0.54200000000000004},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.622,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.53000000000000003,"15min":0.54600000000000004},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.65700000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.57499999999999996}]},{"osd":7,"last update":"Sat Sep 21 15:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.54700000000000004,"15min":0.56200000000000006},"min":{"1min":0.36299999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.60599999999999998,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.54600000000000004,"15min":0.56200000000000006},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.65000000000000002,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.42199999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574889,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37188,"kb_used_data":1976,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334652,"statfs":{"total":96636764160,"available":96598683648,"internally_reserved":0,"allocated":2023424,"data_stored":973041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:29 2024","interfaces":[{"interface":"back","average":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.439,"5min":0.439,"15min":0.439},"last":0.252},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.30099999999999999}]},{"osd":1,"last update":"Sat Sep 21 15:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.40300000000000002,"15min":0.41999999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.439,"5min":1.621,"15min":1.621},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.308,"5min":0.36399999999999999,"15min":0.374},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.46400000000000002,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.28000000000000003}]},{"osd":3,"last update":"Sat Sep 21 15:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.90800000000000003,"15min":0.97099999999999997},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":4.6580000000000004,"5min":11.56,"15min":11.56},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.40200000000000002,"15min":0.41299999999999998},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.46999999999999997,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.32500000000000001}]},{"osd":4,"last update":"Sat Sep 21 15:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.57999999999999996,"15min":0.60199999999999998},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.60499999999999998,"5min":1.653,"15min":1.653},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.60399999999999998,"15min":0.628},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.74199999999999999,"5min":2.4660000000000002,"15min":2.4660000000000002},"last":0.45800000000000002}]},{"osd":5,"last update":"Sat Sep 21 15:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.501,"15min":0.51100000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.61199999999999999,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.437},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.52700000000000002,"15min":0.53900000000000003},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.65400000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.44900000000000001}]},{"osd":6,"last update":"Sat Sep 21 15:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.54900000000000004,"15min":0.56299999999999994},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.67500000000000004,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.54000000000000004,"15min":0.55600000000000005},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.629,"5min":0.873,"15min":0.873},"last":0.41599999999999998}]},{"osd":7,"last update":"Sat Sep 21 15:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.54900000000000004,"15min":0.56100000000000005},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.67000000000000004,"5min":0.876,"15min":0.876},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.58199999999999996,"15min":0.59799999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.754,"5min":1.089,"15min":1.089},"last":0.48299999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574888,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41288,"kb_used_data":1972,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94330552,"statfs":{"total":96636764160,"available":96594485248,"internally_reserved":0,"allocated":2019328,"data_stored":967266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.312,"15min":0.312},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.40100000000000002}]},{"osd":2,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":0.33000000000000002,"15min":0.34000000000000002},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.43099999999999999,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.30299999999999999,"5min":0.499,"15min":0.53200000000000003},"min":{"1min":0.21299999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.52400000000000002,"5min":5.0670000000000002,"15min":5.0670000000000002},"last":0.34200000000000003}]},{"osd":3,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.91700000000000004,"15min":1.0189999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.46000000000000002,"5min":16.032,"15min":16.032},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.309,"5min":0.34999999999999998,"15min":0.35699999999999998},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":0.47899999999999998,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.36499999999999999}]},{"osd":4,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.46800000000000003,"15min":0.47399999999999998},"min":{"1min":0.25900000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.63200000000000001,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.53600000000000003,"15min":0.55100000000000005},"min":{"1min":0.311,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.56799999999999995,"5min":2.2269999999999999,"15min":2.2269999999999999},"last":0.53400000000000003}]},{"osd":5,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.55300000000000005,"15min":0.57299999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.64700000000000002,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.66000000000000003,"15min":0.69599999999999995},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.66200000000000003,"5min":4.6989999999999998,"15min":4.6989999999999998},"last":0.63200000000000001}]},{"osd":6,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.53000000000000003,"15min":0.53900000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.68200000000000005,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.59999999999999998,"15min":0.62},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.64500000000000002,"5min":2.262,"15min":2.262},"last":0.61399999999999999}]},{"osd":7,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.67000000000000004,"15min":0.70399999999999996},"min":{"1min":0.33400000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.69399999999999995,"5min":4.742,"15min":4.742},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.51300000000000001,"15min":0.51800000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.70199999999999996,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.52600000000000002}]}]},{"osd":0,"up_from":93,"seq":399431958545,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39176,"kb_used_data":2556,"kb_used_omap":13,"kb_used_meta":36594,"kb_avail":94332664,"statfs":{"total":96636764160,"available":96596647936,"internally_reserved":0,"allocated":2617344,"data_stored":1563409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13968,"internal_metadata":37472624},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.317,"5min":0.317,"15min":0.317},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"last":0.254}]},{"osd":2,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.313},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.28899999999999998}]},{"osd":3,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.505,"5min":0.505,"15min":0.505},"last":0.34499999999999997}]},{"osd":4,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.44500000000000001}]},{"osd":5,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.374},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.43099999999999999}]},{"osd":6,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.49099999999999999}]},{"osd":7,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.53800000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5973,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T15:51:48.483 INFO:tasks.ceph.ceph_manager.ceph:recovered! 2024-09-21T15:51:48.483 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mgr mgr_debug_aggressive_pg_num_changes 2024-09-21T15:51:48.772 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-21T15:51:48.781 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-21T15:51:48.781 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T15:51:49.055 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:51:49.056 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-09-21T15:51:49.066 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":250,"stamp":"2024-09-21T15:51:47.918416+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590406,"num_objects":1043,"num_object_clones":0,"num_object_copies":2086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1043,"num_whiteouts":1035,"num_read":207535,"num_read_kb":842523,"num_write":23550,"num_write_kb":305657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3397,"num_bytes_recovered":100270055,"num_keys_recovered":316,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20338,"ondisk_log_size":20338,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":416544,"kb_used_data":16932,"kb_used_omap":24,"kb_used_meta":399463,"kb_avail":754558176,"statfs":{"total":773094113280,"available":772667572224,"internally_reserved":0,"allocated":17338368,"data_stored":8940606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25118,"internal_metadata":409050594},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001005"},"pg_stats":[{"pgid":"3.1f","version":"150'413","reported_seq":2132,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.781077+0000","last_change":"2024-09-21T15:51:25.690102+0000","last_active":"2024-09-21T15:51:42.781077+0000","last_peered":"2024-09-21T15:51:42.781077+0000","last_clean":"2024-09-21T15:51:42.781077+0000","last_became_active":"2024-09-21T15:50:32.670964+0000","last_became_peered":"2024-09-21T15:50:32.670964+0000","last_unstale":"2024-09-21T15:51:42.781077+0000","last_undegraded":"2024-09-21T15:51:42.781077+0000","last_fullsized":"2024-09-21T15:51:42.781077+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":413,"log_dups_size":0,"ondisk_log_size":413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:09:09.918864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017896030000000001,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":28,"num_read":2413,"num_read_kb":10419,"num_write":442,"num_write_kb":5956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":1974433,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"149'599","reported_seq":6526,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:44.475566+0000","last_change":"2024-09-21T15:51:25.689127+0000","last_active":"2024-09-21T15:51:44.475566+0000","last_peered":"2024-09-21T15:51:44.475566+0000","last_clean":"2024-09-21T15:51:44.475566+0000","last_became_active":"2024-09-21T15:51:24.691676+0000","last_became_peered":"2024-09-21T15:51:24.691676+0000","last_unstale":"2024-09-21T15:51:44.475566+0000","last_undegraded":"2024-09-21T15:51:44.475566+0000","last_fullsized":"2024-09-21T15:51:44.475566+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":599,"log_dups_size":0,"ondisk_log_size":599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:51:29.795001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010326389999999999,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":38,"num_read":7456,"num_read_kb":25912,"num_write":580,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":250,"num_bytes_recovered":3824749,"num_keys_recovered":34,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"150'655","reported_seq":10214,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.783240+0000","last_change":"2024-09-21T15:51:39.767369+0000","last_active":"2024-09-21T15:51:42.783240+0000","last_peered":"2024-09-21T15:51:42.783240+0000","last_clean":"2024-09-21T15:51:42.783240+0000","last_became_active":"2024-09-21T15:51:39.765580+0000","last_became_peered":"2024-09-21T15:51:39.765580+0000","last_unstale":"2024-09-21T15:51:42.783240+0000","last_undegraded":"2024-09-21T15:51:42.783240+0000","last_fullsized":"2024-09-21T15:51:42.783240+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":655,"log_dups_size":0,"ondisk_log_size":655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:40:43.868959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090302100000000005,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":34,"num_read":10756,"num_read_kb":29175,"num_write":703,"num_write_kb":8882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"150'584","reported_seq":4519,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.783179+0000","last_change":"2024-09-21T15:51:39.767392+0000","last_active":"2024-09-21T15:51:42.783179+0000","last_peered":"2024-09-21T15:51:42.783179+0000","last_clean":"2024-09-21T15:51:42.783179+0000","last_became_active":"2024-09-21T15:51:39.765999+0000","last_became_peered":"2024-09-21T15:51:39.765999+0000","last_unstale":"2024-09-21T15:51:42.783179+0000","last_undegraded":"2024-09-21T15:51:42.783179+0000","last_fullsized":"2024-09-21T15:51:42.783179+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":584,"log_dups_size":0,"ondisk_log_size":584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:21:55.582116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010903670000000001,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":27,"num_read":5242,"num_read_kb":17341,"num_write":644,"num_write_kb":4787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"145'508","reported_seq":4404,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.780869+0000","last_change":"2024-09-21T15:51:25.690506+0000","last_active":"2024-09-21T15:51:42.780869+0000","last_peered":"2024-09-21T15:51:42.780869+0000","last_clean":"2024-09-21T15:51:42.780869+0000","last_became_active":"2024-09-21T15:50:32.674250+0000","last_became_peered":"2024-09-21T15:50:32.674250+0000","last_unstale":"2024-09-21T15:51:42.780869+0000","last_undegraded":"2024-09-21T15:51:42.780869+0000","last_fullsized":"2024-09-21T15:51:42.780869+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":508,"log_dups_size":0,"ondisk_log_size":508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:54:57.399818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019320559999999999,"stat_sum":{"num_bytes":0,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":39,"num_read":4815,"num_read_kb":30088,"num_write":498,"num_write_kb":13507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":4759000,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"150'431","reported_seq":6943,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581775+0000","last_change":"2024-09-21T15:51:25.689223+0000","last_active":"2024-09-21T15:51:43.581775+0000","last_peered":"2024-09-21T15:51:43.581775+0000","last_clean":"2024-09-21T15:51:43.581775+0000","last_became_active":"2024-09-21T15:49:06.693903+0000","last_became_peered":"2024-09-21T15:49:06.693903+0000","last_unstale":"2024-09-21T15:51:43.581775+0000","last_undegraded":"2024-09-21T15:51:43.581775+0000","last_fullsized":"2024-09-21T15:51:43.581775+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":431,"log_dups_size":0,"ondisk_log_size":431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:41:45.432893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047963299999999998,"stat_sum":{"num_bytes":19,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":30,"num_read":7221,"num_read_kb":31759,"num_write":421,"num_write_kb":9476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"148'460","reported_seq":8836,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.491336+0000","last_change":"2024-09-21T15:51:25.780904+0000","last_active":"2024-09-21T15:51:43.491336+0000","last_peered":"2024-09-21T15:51:43.491336+0000","last_clean":"2024-09-21T15:51:43.491336+0000","last_became_active":"2024-09-21T15:51:12.688258+0000","last_became_peered":"2024-09-21T15:51:12.688258+0000","last_unstale":"2024-09-21T15:51:43.491336+0000","last_undegraded":"2024-09-21T15:51:43.491336+0000","last_fullsized":"2024-09-21T15:51:43.491336+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":460,"log_dups_size":0,"ondisk_log_size":460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:19:38.132074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000411691,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":32,"num_read":8283,"num_read_kb":39579,"num_write":518,"num_write_kb":7408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":236,"num_bytes_recovered":6705120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"149'537","reported_seq":4716,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:44.475480+0000","last_change":"2024-09-21T15:51:35.753555+0000","last_active":"2024-09-21T15:51:44.475480+0000","last_peered":"2024-09-21T15:51:44.475480+0000","last_clean":"2024-09-21T15:51:44.475480+0000","last_became_active":"2024-09-21T15:51:35.751700+0000","last_became_peered":"2024-09-21T15:51:35.751700+0000","last_unstale":"2024-09-21T15:51:44.475480+0000","last_undegraded":"2024-09-21T15:51:44.475480+0000","last_fullsized":"2024-09-21T15:51:44.475480+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":537,"log_dups_size":0,"ondisk_log_size":537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:20:44.688916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051035439999999998,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":26,"num_read":6190,"num_read_kb":30937,"num_write":534,"num_write_kb":11075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":282,"num_bytes_recovered":11320990,"num_keys_recovered":70,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"149'1040","reported_seq":9412,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.781000+0000","last_change":"2024-09-21T15:51:25.689670+0000","last_active":"2024-09-21T15:51:42.781000+0000","last_peered":"2024-09-21T15:51:42.781000+0000","last_clean":"2024-09-21T15:51:42.781000+0000","last_became_active":"2024-09-21T15:50:32.668771+0000","last_became_peered":"2024-09-21T15:50:32.668771+0000","last_unstale":"2024-09-21T15:51:42.781000+0000","last_undegraded":"2024-09-21T15:51:42.781000+0000","last_fullsized":"2024-09-21T15:51:42.781000+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":1040,"log_dups_size":0,"ondisk_log_size":1040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:39:43.554150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001503861,"stat_sum":{"num_bytes":0,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":36,"num_read":12714,"num_read_kb":21487,"num_write":1303,"num_write_kb":8257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":3319185,"num_keys_recovered":34,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"150'540","reported_seq":6716,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.783295+0000","last_change":"2024-09-21T15:51:25.757086+0000","last_active":"2024-09-21T15:51:42.783295+0000","last_peered":"2024-09-21T15:51:42.783295+0000","last_clean":"2024-09-21T15:51:42.783295+0000","last_became_active":"2024-09-21T15:49:06.688808+0000","last_became_peered":"2024-09-21T15:49:06.688808+0000","last_unstale":"2024-09-21T15:51:42.783295+0000","last_undegraded":"2024-09-21T15:51:42.783295+0000","last_fullsized":"2024-09-21T15:51:42.783295+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":540,"log_dups_size":0,"ondisk_log_size":540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:43:02.012254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00023944,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":30,"num_read":7604,"num_read_kb":49865,"num_write":542,"num_write_kb":13823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"150'530","reported_seq":4324,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581211+0000","last_change":"2024-09-21T15:51:25.699156+0000","last_active":"2024-09-21T15:51:43.581211+0000","last_peered":"2024-09-21T15:51:43.581211+0000","last_clean":"2024-09-21T15:51:43.581211+0000","last_became_active":"2024-09-21T15:51:11.841664+0000","last_became_peered":"2024-09-21T15:51:11.841664+0000","last_unstale":"2024-09-21T15:51:43.581211+0000","last_undegraded":"2024-09-21T15:51:43.581211+0000","last_fullsized":"2024-09-21T15:51:43.581211+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:47:37.118354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001532828,"stat_sum":{"num_bytes":0,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":29,"num_read":4052,"num_read_kb":27144,"num_write":637,"num_write_kb":17935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":222,"num_bytes_recovered":4468215,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"147'1207","reported_seq":4387,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581846+0000","last_change":"2024-09-21T15:51:25.690386+0000","last_active":"2024-09-21T15:51:43.581846+0000","last_peered":"2024-09-21T15:51:43.581846+0000","last_clean":"2024-09-21T15:51:43.581846+0000","last_became_active":"2024-09-21T15:49:06.688803+0000","last_became_peered":"2024-09-21T15:49:06.688803+0000","last_unstale":"2024-09-21T15:51:43.581846+0000","last_undegraded":"2024-09-21T15:51:43.581846+0000","last_fullsized":"2024-09-21T15:51:43.581846+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":1207,"log_dups_size":0,"ondisk_log_size":1207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:18:47.662631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013586710000000001,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":26,"num_read":7764,"num_read_kb":18232,"num_write":1926,"num_write_kb":6190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":161,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581555+0000","last_change":"2024-09-21T15:48:59.666431+0000","last_active":"2024-09-21T15:51:43.581555+0000","last_peered":"2024-09-21T15:51:43.581555+0000","last_clean":"2024-09-21T15:51:43.581555+0000","last_became_active":"2024-09-21T15:48:57.583665+0000","last_became_peered":"2024-09-21T15:48:57.583665+0000","last_unstale":"2024-09-21T15:51:43.581555+0000","last_undegraded":"2024-09-21T15:51:43.581555+0000","last_fullsized":"2024-09-21T15:51:43.581555+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:52:09.228964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031098399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"150'438","reported_seq":3064,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.780970+0000","last_change":"2024-09-21T15:51:25.689587+0000","last_active":"2024-09-21T15:51:42.780970+0000","last_peered":"2024-09-21T15:51:42.780970+0000","last_clean":"2024-09-21T15:51:42.780970+0000","last_became_active":"2024-09-21T15:50:32.671424+0000","last_became_peered":"2024-09-21T15:50:32.671424+0000","last_unstale":"2024-09-21T15:51:42.780970+0000","last_undegraded":"2024-09-21T15:51:42.780970+0000","last_fullsized":"2024-09-21T15:51:42.780970+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":438,"log_dups_size":0,"ondisk_log_size":438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:04:14.554732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001507467,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":35,"num_read":2637,"num_read_kb":12956,"num_write":388,"num_write_kb":8392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":4494388,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":164,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.491395+0000","last_change":"2024-09-21T15:48:59.589790+0000","last_active":"2024-09-21T15:51:43.491395+0000","last_peered":"2024-09-21T15:51:43.491395+0000","last_clean":"2024-09-21T15:51:43.491395+0000","last_became_active":"2024-09-21T15:48:57.582121+0000","last_became_peered":"2024-09-21T15:48:57.582121+0000","last_unstale":"2024-09-21T15:51:43.491395+0000","last_undegraded":"2024-09-21T15:51:43.491395+0000","last_fullsized":"2024-09-21T15:51:43.491395+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:33:47.592624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039654700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"148'579","reported_seq":8001,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:44.475582+0000","last_change":"2024-09-21T15:51:35.754312+0000","last_active":"2024-09-21T15:51:44.475582+0000","last_peered":"2024-09-21T15:51:44.475582+0000","last_clean":"2024-09-21T15:51:44.475582+0000","last_became_active":"2024-09-21T15:51:35.752700+0000","last_became_peered":"2024-09-21T15:51:35.752700+0000","last_unstale":"2024-09-21T15:51:44.475582+0000","last_undegraded":"2024-09-21T15:51:44.475582+0000","last_fullsized":"2024-09-21T15:51:44.475582+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":579,"log_dups_size":0,"ondisk_log_size":579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:37:03.162105+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.00086244900000000005,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":35,"num_read":8380,"num_read_kb":21195,"num_write":544,"num_write_kb":5397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":256,"num_bytes_recovered":3760369,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":175,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581295+0000","last_change":"2024-09-21T15:50:30.146743+0000","last_active":"2024-09-21T15:51:43.581295+0000","last_peered":"2024-09-21T15:51:43.581295+0000","last_clean":"2024-09-21T15:51:43.581295+0000","last_became_active":"2024-09-21T15:50:30.146428+0000","last_became_peered":"2024-09-21T15:50:30.146428+0000","last_unstale":"2024-09-21T15:51:43.581295+0000","last_undegraded":"2024-09-21T15:51:43.581295+0000","last_fullsized":"2024-09-21T15:51:43.581295+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:58:38.689807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036074600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"145'703","reported_seq":8853,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.491362+0000","last_change":"2024-09-21T15:51:25.780818+0000","last_active":"2024-09-21T15:51:43.491362+0000","last_peered":"2024-09-21T15:51:43.491362+0000","last_clean":"2024-09-21T15:51:43.491362+0000","last_became_active":"2024-09-21T15:50:35.682788+0000","last_became_peered":"2024-09-21T15:50:35.682788+0000","last_unstale":"2024-09-21T15:51:43.491362+0000","last_undegraded":"2024-09-21T15:51:43.491362+0000","last_fullsized":"2024-09-21T15:51:43.491362+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":703,"log_dups_size":0,"ondisk_log_size":703,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:20:40.695690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032907100000000001,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":26,"num_read":11015,"num_read_kb":27666,"num_write":868,"num_write_kb":11017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":175,"num_bytes_recovered":14134268,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":177,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.491361+0000","last_change":"2024-09-21T15:50:30.146654+0000","last_active":"2024-09-21T15:51:43.491361+0000","last_peered":"2024-09-21T15:51:43.491361+0000","last_clean":"2024-09-21T15:51:43.491361+0000","last_became_active":"2024-09-21T15:50:30.146273+0000","last_became_peered":"2024-09-21T15:50:30.146273+0000","last_unstale":"2024-09-21T15:51:43.491361+0000","last_undegraded":"2024-09-21T15:51:43.491361+0000","last_fullsized":"2024-09-21T15:51:43.491361+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:21:48.178519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040525700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"150'603","reported_seq":5376,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.783334+0000","last_change":"2024-09-21T15:51:39.769574+0000","last_active":"2024-09-21T15:51:42.783334+0000","last_peered":"2024-09-21T15:51:42.783334+0000","last_clean":"2024-09-21T15:51:42.783334+0000","last_became_active":"2024-09-21T15:51:39.768300+0000","last_became_peered":"2024-09-21T15:51:39.768300+0000","last_unstale":"2024-09-21T15:51:42.783334+0000","last_undegraded":"2024-09-21T15:51:42.783334+0000","last_fullsized":"2024-09-21T15:51:42.783334+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":603,"log_dups_size":0,"ondisk_log_size":603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:33:47.511137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050855200000000001,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":35,"num_read":5730,"num_read_kb":33940,"num_write":686,"num_write_kb":16332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":268,"num_bytes_recovered":7823491,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"150'921","reported_seq":4195,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581881+0000","last_change":"2024-09-21T15:51:25.690079+0000","last_active":"2024-09-21T15:51:43.581881+0000","last_peered":"2024-09-21T15:51:43.581881+0000","last_clean":"2024-09-21T15:51:43.581881+0000","last_became_active":"2024-09-21T15:50:32.677456+0000","last_became_peered":"2024-09-21T15:50:32.677456+0000","last_unstale":"2024-09-21T15:51:43.581881+0000","last_undegraded":"2024-09-21T15:51:43.581881+0000","last_fullsized":"2024-09-21T15:51:43.581881+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":921,"log_dups_size":0,"ondisk_log_size":921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:46:31.194345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010737730000000001,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":34,"num_read":5136,"num_read_kb":19340,"num_write":949,"num_write_kb":7405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":632176,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"20'2","reported_seq":59,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.783126+0000","last_change":"2024-09-21T15:51:39.764743+0000","last_active":"2024-09-21T15:51:42.783126+0000","last_peered":"2024-09-21T15:51:42.783126+0000","last_clean":"2024-09-21T15:51:42.783126+0000","last_became_active":"2024-09-21T15:51:39.764409+0000","last_became_peered":"2024-09-21T15:51:39.764409+0000","last_unstale":"2024-09-21T15:51:42.783126+0000","last_undegraded":"2024-09-21T15:51:42.783126+0000","last_fullsized":"2024-09-21T15:51:42.783126+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:28:14.932986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000859595,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"148'393","reported_seq":4833,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.491393+0000","last_change":"2024-09-21T15:51:25.781082+0000","last_active":"2024-09-21T15:51:43.491393+0000","last_peered":"2024-09-21T15:51:43.491393+0000","last_clean":"2024-09-21T15:51:43.491393+0000","last_became_active":"2024-09-21T15:50:35.682001+0000","last_became_peered":"2024-09-21T15:50:35.682001+0000","last_unstale":"2024-09-21T15:51:43.491393+0000","last_undegraded":"2024-09-21T15:51:43.491393+0000","last_fullsized":"2024-09-21T15:51:43.491393+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":393,"log_dups_size":0,"ondisk_log_size":393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:09:15.325457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044281000000000002,"stat_sum":{"num_bytes":0,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":31,"num_read":4285,"num_read_kb":23375,"num_write":379,"num_write_kb":4145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":187,"num_bytes_recovered":4305276,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":97,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.788209+0000","last_change":"2024-09-21T15:51:07.683738+0000","last_active":"2024-09-21T15:51:43.788209+0000","last_peered":"2024-09-21T15:51:43.788209+0000","last_clean":"2024-09-21T15:51:43.788209+0000","last_became_active":"2024-09-21T15:51:07.683418+0000","last_became_peered":"2024-09-21T15:51:07.683418+0000","last_unstale":"2024-09-21T15:51:43.788209+0000","last_undegraded":"2024-09-21T15:51:43.788209+0000","last_fullsized":"2024-09-21T15:51:43.788209+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:25:13.468653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"150'558","reported_seq":2598,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.781077+0000","last_change":"2024-09-21T15:51:25.689707+0000","last_active":"2024-09-21T15:51:42.781077+0000","last_peered":"2024-09-21T15:51:42.781077+0000","last_clean":"2024-09-21T15:51:42.781077+0000","last_became_active":"2024-09-21T15:50:32.671413+0000","last_became_peered":"2024-09-21T15:50:32.671413+0000","last_unstale":"2024-09-21T15:51:42.781077+0000","last_undegraded":"2024-09-21T15:51:42.781077+0000","last_fullsized":"2024-09-21T15:51:42.781077+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":558,"log_dups_size":0,"ondisk_log_size":558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:04:19.509402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001292872,"stat_sum":{"num_bytes":0,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":39,"num_read":3663,"num_read_kb":16543,"num_write":590,"num_write_kb":5977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":1696547,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":162,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581078+0000","last_change":"2024-09-21T15:49:00.114730+0000","last_active":"2024-09-21T15:51:43.581078+0000","last_peered":"2024-09-21T15:51:43.581078+0000","last_clean":"2024-09-21T15:51:43.581078+0000","last_became_active":"2024-09-21T15:48:57.585525+0000","last_became_peered":"2024-09-21T15:48:57.585525+0000","last_unstale":"2024-09-21T15:51:43.581078+0000","last_undegraded":"2024-09-21T15:51:43.581078+0000","last_fullsized":"2024-09-21T15:51:43.581078+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:35:35.786526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031691100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"150'413","reported_seq":2957,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:44.475526+0000","last_change":"2024-09-21T15:51:35.754599+0000","last_active":"2024-09-21T15:51:44.475526+0000","last_peered":"2024-09-21T15:51:44.475526+0000","last_clean":"2024-09-21T15:51:44.475526+0000","last_became_active":"2024-09-21T15:51:35.752810+0000","last_became_peered":"2024-09-21T15:51:35.752810+0000","last_unstale":"2024-09-21T15:51:44.475526+0000","last_undegraded":"2024-09-21T15:51:44.475526+0000","last_fullsized":"2024-09-21T15:51:44.475526+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":413,"log_dups_size":0,"ondisk_log_size":413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:54:49.851651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033178999999999999,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":30,"num_read":2149,"num_read_kb":34452,"num_write":406,"num_write_kb":12612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":214,"num_bytes_recovered":2197196,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"18'1","reported_seq":58,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.783151+0000","last_change":"2024-09-21T15:51:39.764710+0000","last_active":"2024-09-21T15:51:42.783151+0000","last_peered":"2024-09-21T15:51:42.783151+0000","last_clean":"2024-09-21T15:51:42.783151+0000","last_became_active":"2024-09-21T15:51:39.764411+0000","last_became_peered":"2024-09-21T15:51:39.764411+0000","last_unstale":"2024-09-21T15:51:42.783151+0000","last_undegraded":"2024-09-21T15:51:42.783151+0000","last_fullsized":"2024-09-21T15:51:42.783151+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:29:50.093645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039559000000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":323,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581078+0000","last_change":"2024-09-21T15:50:30.147280+0000","last_active":"2024-09-21T15:51:43.581078+0000","last_peered":"2024-09-21T15:51:43.581078+0000","last_clean":"2024-09-21T15:51:43.581078+0000","last_became_active":"2024-09-21T15:50:30.146987+0000","last_became_peered":"2024-09-21T15:50:30.146987+0000","last_unstale":"2024-09-21T15:51:43.581078+0000","last_undegraded":"2024-09-21T15:51:43.581078+0000","last_fullsized":"2024-09-21T15:51:43.581078+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:50.161884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:50.161884+0000","last_clean_scrub_stamp":"2024-09-21T15:48:50.161884+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:26:32.853650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"150'540","reported_seq":7944,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:44.475450+0000","last_change":"2024-09-21T15:51:25.689387+0000","last_active":"2024-09-21T15:51:44.475450+0000","last_peered":"2024-09-21T15:51:44.475450+0000","last_clean":"2024-09-21T15:51:44.475450+0000","last_became_active":"2024-09-21T15:51:11.837877+0000","last_became_peered":"2024-09-21T15:51:11.837877+0000","last_unstale":"2024-09-21T15:51:44.475450+0000","last_undegraded":"2024-09-21T15:51:44.475450+0000","last_fullsized":"2024-09-21T15:51:44.475450+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":540,"log_dups_size":0,"ondisk_log_size":540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:45:58.418197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079839600000000004,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":32,"num_read":7239,"num_read_kb":29580,"num_write":558,"num_write_kb":4793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":418,"num_bytes_recovered":6262707,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"150'389","reported_seq":3170,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581718+0000","last_change":"2024-09-21T15:51:25.689626+0000","last_active":"2024-09-21T15:51:43.581718+0000","last_peered":"2024-09-21T15:51:43.581718+0000","last_clean":"2024-09-21T15:51:43.581718+0000","last_became_active":"2024-09-21T15:49:07.210506+0000","last_became_peered":"2024-09-21T15:49:07.210506+0000","last_unstale":"2024-09-21T15:51:43.581718+0000","last_undegraded":"2024-09-21T15:51:43.581718+0000","last_fullsized":"2024-09-21T15:51:43.581718+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":389,"log_dups_size":0,"ondisk_log_size":389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:55:18.046738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089193900000000001,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":30,"num_read":2716,"num_read_kb":28245,"num_write":454,"num_write_kb":12090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"149'595","reported_seq":12267,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581556+0000","last_change":"2024-09-21T15:51:41.782922+0000","last_active":"2024-09-21T15:51:43.581556+0000","last_peered":"2024-09-21T15:51:43.581556+0000","last_clean":"2024-09-21T15:51:43.581556+0000","last_became_active":"2024-09-21T15:51:41.781781+0000","last_became_peered":"2024-09-21T15:51:41.781781+0000","last_unstale":"2024-09-21T15:51:43.581556+0000","last_undegraded":"2024-09-21T15:51:43.581556+0000","last_fullsized":"2024-09-21T15:51:43.581556+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":595,"log_dups_size":0,"ondisk_log_size":595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:10:04.846054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000309432,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":29,"num_read":12828,"num_read_kb":63467,"num_write":580,"num_write_kb":14212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"147'851","reported_seq":3855,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:44.475479+0000","last_change":"2024-09-21T15:51:35.752901+0000","last_active":"2024-09-21T15:51:44.475479+0000","last_peered":"2024-09-21T15:51:44.475479+0000","last_clean":"2024-09-21T15:51:44.475479+0000","last_became_active":"2024-09-21T15:51:35.751150+0000","last_became_peered":"2024-09-21T15:51:35.751150+0000","last_unstale":"2024-09-21T15:51:44.475479+0000","last_undegraded":"2024-09-21T15:51:44.475479+0000","last_fullsized":"2024-09-21T15:51:44.475479+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":851,"log_dups_size":0,"ondisk_log_size":851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:33:02.873060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001158526,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":26,"num_read":5948,"num_read_kb":20169,"num_write":1244,"num_write_kb":7750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":262,"num_bytes_recovered":8353711,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"150'739","reported_seq":6768,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.783290+0000","last_change":"2024-09-21T15:51:39.769469+0000","last_active":"2024-09-21T15:51:42.783290+0000","last_peered":"2024-09-21T15:51:42.783290+0000","last_clean":"2024-09-21T15:51:42.783290+0000","last_became_active":"2024-09-21T15:51:39.768321+0000","last_became_peered":"2024-09-21T15:51:39.768321+0000","last_unstale":"2024-09-21T15:51:42.783290+0000","last_undegraded":"2024-09-21T15:51:42.783290+0000","last_fullsized":"2024-09-21T15:51:42.783290+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":739,"log_dups_size":0,"ondisk_log_size":739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:21:29.566035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074635999999999995,"stat_sum":{"num_bytes":0,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":36,"num_read":8849,"num_read_kb":25678,"num_write":868,"num_write_kb":5005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":246,"num_bytes_recovered":5686754,"num_keys_recovered":23,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"147'894","reported_seq":4314,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581242+0000","last_change":"2024-09-21T15:51:25.698246+0000","last_active":"2024-09-21T15:51:43.581242+0000","last_peered":"2024-09-21T15:51:43.581242+0000","last_clean":"2024-09-21T15:51:43.581242+0000","last_became_active":"2024-09-21T15:49:06.689203+0000","last_became_peered":"2024-09-21T15:49:06.689203+0000","last_unstale":"2024-09-21T15:51:43.581242+0000","last_undegraded":"2024-09-21T15:51:43.581242+0000","last_fullsized":"2024-09-21T15:51:43.581242+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":894,"log_dups_size":0,"ondisk_log_size":894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:22:17.578359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084935700000000002,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":35,"num_read":7321,"num_read_kb":28131,"num_write":1146,"num_write_kb":14809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"148'568","reported_seq":8144,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581299+0000","last_change":"2024-09-21T15:51:25.697735+0000","last_active":"2024-09-21T15:51:43.581299+0000","last_peered":"2024-09-21T15:51:43.581299+0000","last_clean":"2024-09-21T15:51:43.581299+0000","last_became_active":"2024-09-21T15:49:06.690101+0000","last_became_peered":"2024-09-21T15:49:06.690101+0000","last_unstale":"2024-09-21T15:51:43.581299+0000","last_undegraded":"2024-09-21T15:51:43.581299+0000","last_fullsized":"2024-09-21T15:51:43.581299+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":568,"log_dups_size":0,"ondisk_log_size":568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:43:30.659104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044571200000000002,"stat_sum":{"num_bytes":0,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":29,"num_read":8735,"num_read_kb":42983,"num_write":509,"num_write_kb":16273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"145'698","reported_seq":3799,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581254+0000","last_change":"2024-09-21T15:51:25.698380+0000","last_active":"2024-09-21T15:51:43.581254+0000","last_peered":"2024-09-21T15:51:43.581254+0000","last_clean":"2024-09-21T15:51:43.581254+0000","last_became_active":"2024-09-21T15:49:06.694250+0000","last_became_peered":"2024-09-21T15:49:06.694250+0000","last_unstale":"2024-09-21T15:51:43.581254+0000","last_undegraded":"2024-09-21T15:51:43.581254+0000","last_fullsized":"2024-09-21T15:51:43.581254+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":698,"log_dups_size":0,"ondisk_log_size":698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:21:37.777769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086632899999999999,"stat_sum":{"num_bytes":0,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":42,"num_read":4945,"num_read_kb":15920,"num_write":889,"num_write_kb":5075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"138'470","reported_seq":3154,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581542+0000","last_change":"2024-09-21T15:51:25.780857+0000","last_active":"2024-09-21T15:51:43.581542+0000","last_peered":"2024-09-21T15:51:43.581542+0000","last_clean":"2024-09-21T15:51:43.581542+0000","last_became_active":"2024-09-21T15:50:32.676147+0000","last_became_peered":"2024-09-21T15:50:32.676147+0000","last_unstale":"2024-09-21T15:51:43.581542+0000","last_undegraded":"2024-09-21T15:51:43.581542+0000","last_fullsized":"2024-09-21T15:51:43.581542+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":470,"log_dups_size":0,"ondisk_log_size":470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:44:18.821310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00024304599999999999,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":40,"num_read":3160,"num_read_kb":11849,"num_write":381,"num_write_kb":4471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":1191778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"148'1012","reported_seq":4197,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581091+0000","last_change":"2024-09-21T15:51:25.697763+0000","last_active":"2024-09-21T15:51:43.581091+0000","last_peered":"2024-09-21T15:51:43.581091+0000","last_clean":"2024-09-21T15:51:43.581091+0000","last_became_active":"2024-09-21T15:49:06.688798+0000","last_became_peered":"2024-09-21T15:49:06.688798+0000","last_unstale":"2024-09-21T15:51:43.581091+0000","last_undegraded":"2024-09-21T15:51:43.581091+0000","last_fullsized":"2024-09-21T15:51:43.581091+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":1012,"log_dups_size":0,"ondisk_log_size":1012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:48:33.679133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053058599999999999,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":30,"num_read":6391,"num_read_kb":16970,"num_write":1355,"num_write_kb":8408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"146'383","reported_seq":3011,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.780811+0000","last_change":"2024-09-21T15:51:25.689299+0000","last_active":"2024-09-21T15:51:42.780811+0000","last_peered":"2024-09-21T15:51:42.780811+0000","last_clean":"2024-09-21T15:51:42.780811+0000","last_became_active":"2024-09-21T15:50:32.671686+0000","last_became_peered":"2024-09-21T15:50:32.671686+0000","last_unstale":"2024-09-21T15:51:42.780811+0000","last_undegraded":"2024-09-21T15:51:42.780811+0000","last_fullsized":"2024-09-21T15:51:42.780811+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":383,"log_dups_size":0,"ondisk_log_size":383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:40:53.968477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012972649999999999,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":34,"num_read":2455,"num_read_kb":11669,"num_write":401,"num_write_kb":13839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":3359702,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"150'967","reported_seq":7431,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581127+0000","last_change":"2024-09-21T15:51:25.699098+0000","last_active":"2024-09-21T15:51:43.581127+0000","last_peered":"2024-09-21T15:51:43.581127+0000","last_clean":"2024-09-21T15:51:43.581127+0000","last_became_active":"2024-09-21T15:49:06.690128+0000","last_became_peered":"2024-09-21T15:49:06.690128+0000","last_unstale":"2024-09-21T15:51:43.581127+0000","last_undegraded":"2024-09-21T15:51:43.581127+0000","last_fullsized":"2024-09-21T15:51:43.581127+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":967,"log_dups_size":0,"ondisk_log_size":967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:11.763959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014652879999999999,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":32,"num_read":9327,"num_read_kb":26357,"num_write":1446,"num_write_kb":13580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":1039,"num_object_clones":0,"num_object_copies":2078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1039,"num_whiteouts":1035,"num_read":207419,"num_read_kb":842423,"num_write":23389,"num_write_kb":303239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3397,"num_bytes_recovered":100270055,"num_keys_recovered":316,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5973,"internal_metadata":0},"log_size":20218,"ondisk_log_size":20218,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574887,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77952,"kb_used_data":2544,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94293888,"statfs":{"total":96636764160,"available":96556941312,"internally_reserved":0,"allocated":2605056,"data_stored":1557596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":77199819},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:32 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.44},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.64100000000000001}]},{"osd":1,"last update":"Sat Sep 21 15:50:52 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.48299999999999998,"15min":0.495},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":0.73199999999999998,"5min":1.04,"15min":1.04},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.44,"5min":0.46999999999999997,"15min":0.47499999999999998},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.64500000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.58599999999999997}]},{"osd":2,"last update":"Sat Sep 21 15:50:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.439,"15min":0.44400000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.53500000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.437,"5min":0.498,"15min":0.50800000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.70099999999999996,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.55300000000000005}]},{"osd":3,"last update":"Sat Sep 21 15:50:52 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.49099999999999999,"15min":0.499},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.65300000000000002,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.432,"5min":0.47099999999999997,"15min":0.47699999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68200000000000005,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.57099999999999995}]},{"osd":4,"last update":"Sat Sep 21 15:50:52 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.41699999999999998,"15min":0.42299999999999999},"min":{"1min":0.24099999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.59999999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.38,"5min":0.39000000000000001,"15min":0.39200000000000002},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.52700000000000002}]},{"osd":5,"last update":"Sat Sep 21 15:50:52 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.439,"15min":0.45000000000000001},"min":{"1min":0.25900000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.52900000000000003,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.44700000000000001,"15min":0.45000000000000001},"min":{"1min":0.29499999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.68600000000000005,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.70199999999999996}]},{"osd":6,"last update":"Sat Sep 21 15:50:52 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.49399999999999999,"15min":0.5},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.72099999999999997,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.74199999999999999,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.77100000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574890,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41284,"kb_used_data":1976,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94330556,"statfs":{"total":96636764160,"available":96594489344,"internally_reserved":0,"allocated":2023424,"data_stored":973041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:31 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.60399999999999998}]},{"osd":1,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.44800000000000001,"15min":0.45800000000000002},"min":{"1min":0.26900000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.53300000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.46800000000000003,"15min":0.47999999999999998},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.53300000000000003,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.48899999999999999}]},{"osd":2,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.47099999999999997,"15min":0.47699999999999998},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.45800000000000002,"15min":0.46200000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.60999999999999999,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.438}]},{"osd":3,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.51400000000000001,"15min":0.52400000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.70399999999999996,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.53100000000000003,"15min":0.53800000000000003},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.73699999999999999,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.51600000000000001}]},{"osd":4,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.40600000000000003,"15min":0.41399999999999998},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.222},"max":{"1min":0.57499999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.379,"5min":0.46700000000000003,"15min":0.48199999999999998},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.76300000000000001,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.65700000000000003}]},{"osd":5,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.44600000000000001,"15min":0.45600000000000002},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.57399999999999995,"5min":0.755,"15min":0.755},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.372,"5min":0.45000000000000001,"15min":0.46300000000000002},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.628,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.626}]},{"osd":7,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.45000000000000001,"15min":0.45700000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.55400000000000005,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.497,"15min":0.51100000000000001},"min":{"1min":0.30499999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.748,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.67600000000000005}]}]},{"osd":5,"up_from":13,"seq":55834574889,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55232,"kb_used_data":1968,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94316608,"statfs":{"total":96636764160,"available":96580206592,"internally_reserved":0,"allocated":2015232,"data_stored":967247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.5},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.66400000000000003}]},{"osd":1,"last update":"Sat Sep 21 15:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.71699999999999997,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.5,"15min":0.51400000000000001},"min":{"1min":0.34399999999999997,"5min":0.314,"15min":0.314},"max":{"1min":0.52200000000000002,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.56699999999999995}]},{"osd":2,"last update":"Sat Sep 21 15:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.49399999999999999,"15min":0.50600000000000001},"min":{"1min":0.34200000000000003,"5min":0.311,"15min":0.311},"max":{"1min":0.56899999999999995,"5min":1.046,"15min":1.046},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.437,"5min":0.46800000000000003,"15min":0.47299999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.68999999999999995,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.68000000000000005}]},{"osd":3,"last update":"Sat Sep 21 15:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.47699999999999998,"15min":0.48699999999999999},"min":{"1min":0.29599999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.55200000000000005,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.53800000000000003,"15min":0.55200000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.76900000000000002,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.59099999999999997}]},{"osd":4,"last update":"Sat Sep 21 15:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.65400000000000003,"15min":0.70799999999999996},"min":{"1min":0.254,"5min":0.253,"15min":0.253},"max":{"1min":0.503,"5min":5.6399999999999997,"15min":5.6399999999999997},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.45000000000000001,"15min":0.47199999999999998},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.48899999999999999,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.69499999999999995}]},{"osd":6,"last update":"Sat Sep 21 15:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.443,"15min":0.45500000000000002},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.51700000000000002,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.48199999999999998,"15min":0.499},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.55200000000000005,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.64100000000000001}]},{"osd":7,"last update":"Sat Sep 21 15:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.47199999999999998,"15min":0.48599999999999999},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.56000000000000005,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.623},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.504,"15min":0.52100000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.439}]}]},{"osd":4,"up_from":13,"seq":55834574888,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65084,"kb_used_data":1964,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94306756,"statfs":{"total":96636764160,"available":96570118144,"internally_reserved":0,"allocated":2011136,"data_stored":965965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.56299999999999994}]},{"osd":1,"last update":"Sat Sep 21 15:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.433,"15min":0.41899999999999998},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":1.8839999999999999,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.33600000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.55000000000000004}]},{"osd":2,"last update":"Sat Sep 21 15:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.443,"15min":0.439},"min":{"1min":0.311,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.501},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.433,"15min":0.433},"min":{"1min":0.31,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.54000000000000004,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.47999999999999998}]},{"osd":3,"last update":"Sat Sep 21 15:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.34200000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.66600000000000004,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.46800000000000003,"15min":0.46999999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.60899999999999999,"5min":0.626,"15min":0.626},"last":0.52600000000000002}]},{"osd":5,"last update":"Sat Sep 21 15:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.47199999999999998,"15min":0.47599999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":1.919,"5min":2.4489999999999998,"15min":2.4489999999999998},"last":0.376},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.42399999999999999,"15min":0.42499999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.748,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.41099999999999998}]},{"osd":6,"last update":"Sat Sep 21 15:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.41599999999999998,"15min":0.41399999999999998},"min":{"1min":0.29399999999999998,"5min":0.245,"15min":0.245},"max":{"1min":0.63800000000000001,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.55700000000000005,"15min":0.54500000000000004},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":4.1340000000000003,"5min":4.1340000000000003,"15min":4.1340000000000003},"last":0.51500000000000001}]},{"osd":7,"last update":"Sat Sep 21 15:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.45400000000000001,"15min":0.434},"min":{"1min":0.28199999999999997,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":2.7599999999999998,"5min":2.7599999999999998,"15min":2.7599999999999998},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.52400000000000002,"15min":0.52200000000000002},"min":{"1min":0.32600000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":2.016,"5min":2.016,"15min":2.016},"last":0.434}]}]},{"osd":3,"up_from":13,"seq":55834574889,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59340,"kb_used_data":1976,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312500,"statfs":{"total":96636764160,"available":96576000000,"internally_reserved":0,"allocated":2023424,"data_stored":973041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":58718664},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.31,"5min":0.31,"15min":0.31},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"last":0.27800000000000002}]},{"osd":1,"last update":"Sat Sep 21 15:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.28999999999999998,"5min":0.33600000000000002,"15min":0.34399999999999997},"min":{"1min":0.219,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.42299999999999999,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.222},{"interface":"front","average":{"1min":0.30499999999999999,"5min":0.36799999999999999,"15min":0.378},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.40500000000000003,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.313}]},{"osd":2,"last update":"Sat Sep 21 15:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.38400000000000001,"15min":0.39300000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.49199999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.40999999999999998,"15min":0.42099999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.52000000000000002,"5min":0.755,"15min":0.755},"last":0.41299999999999998}]},{"osd":4,"last update":"Sat Sep 21 15:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.47899999999999998,"15min":0.49199999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.56699999999999995,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.504,"15min":0.51300000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.60399999999999998,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.51800000000000002}]},{"osd":5,"last update":"Sat Sep 21 15:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.51000000000000001,"15min":0.52500000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.68100000000000005,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.503},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.53400000000000003,"15min":0.55300000000000005},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.54100000000000004,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.48799999999999999}]},{"osd":6,"last update":"Sat Sep 21 15:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.52800000000000002,"15min":0.54200000000000004},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.622,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.53000000000000003,"15min":0.54600000000000004},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.65700000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.57499999999999996}]},{"osd":7,"last update":"Sat Sep 21 15:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.54700000000000004,"15min":0.56200000000000006},"min":{"1min":0.36299999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.60599999999999998,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.54600000000000004,"15min":0.56200000000000006},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.65000000000000002,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.42199999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574889,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37188,"kb_used_data":1976,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334652,"statfs":{"total":96636764160,"available":96598683648,"internally_reserved":0,"allocated":2023424,"data_stored":973041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:29 2024","interfaces":[{"interface":"back","average":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.439,"5min":0.439,"15min":0.439},"last":0.252},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.30099999999999999}]},{"osd":1,"last update":"Sat Sep 21 15:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.40300000000000002,"15min":0.41999999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.439,"5min":1.621,"15min":1.621},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.308,"5min":0.36399999999999999,"15min":0.374},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.46400000000000002,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.28000000000000003}]},{"osd":3,"last update":"Sat Sep 21 15:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.90800000000000003,"15min":0.97099999999999997},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":4.6580000000000004,"5min":11.56,"15min":11.56},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.40200000000000002,"15min":0.41299999999999998},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.46999999999999997,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.32500000000000001}]},{"osd":4,"last update":"Sat Sep 21 15:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.57999999999999996,"15min":0.60199999999999998},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.60499999999999998,"5min":1.653,"15min":1.653},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.60399999999999998,"15min":0.628},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.74199999999999999,"5min":2.4660000000000002,"15min":2.4660000000000002},"last":0.45800000000000002}]},{"osd":5,"last update":"Sat Sep 21 15:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.501,"15min":0.51100000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.61199999999999999,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.437},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.52700000000000002,"15min":0.53900000000000003},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.65400000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.44900000000000001}]},{"osd":6,"last update":"Sat Sep 21 15:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.54900000000000004,"15min":0.56299999999999994},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.67500000000000004,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.54000000000000004,"15min":0.55600000000000005},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.629,"5min":0.873,"15min":0.873},"last":0.41599999999999998}]},{"osd":7,"last update":"Sat Sep 21 15:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.54900000000000004,"15min":0.56100000000000005},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.67000000000000004,"5min":0.876,"15min":0.876},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.58199999999999996,"15min":0.59799999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.754,"5min":1.089,"15min":1.089},"last":0.48299999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574888,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41288,"kb_used_data":1972,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94330552,"statfs":{"total":96636764160,"available":96594485248,"internally_reserved":0,"allocated":2019328,"data_stored":967266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.312,"15min":0.312},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.40100000000000002}]},{"osd":2,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":0.33000000000000002,"15min":0.34000000000000002},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.43099999999999999,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.30299999999999999,"5min":0.499,"15min":0.53200000000000003},"min":{"1min":0.21299999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.52400000000000002,"5min":5.0670000000000002,"15min":5.0670000000000002},"last":0.34200000000000003}]},{"osd":3,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.91700000000000004,"15min":1.0189999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.46000000000000002,"5min":16.032,"15min":16.032},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.309,"5min":0.34999999999999998,"15min":0.35699999999999998},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":0.47899999999999998,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.36499999999999999}]},{"osd":4,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.46800000000000003,"15min":0.47399999999999998},"min":{"1min":0.25900000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.63200000000000001,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.53600000000000003,"15min":0.55100000000000005},"min":{"1min":0.311,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.56799999999999995,"5min":2.2269999999999999,"15min":2.2269999999999999},"last":0.53400000000000003}]},{"osd":5,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.55300000000000005,"15min":0.57299999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.64700000000000002,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.66000000000000003,"15min":0.69599999999999995},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.66200000000000003,"5min":4.6989999999999998,"15min":4.6989999999999998},"last":0.63200000000000001}]},{"osd":6,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.53000000000000003,"15min":0.53900000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.68200000000000005,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.59999999999999998,"15min":0.62},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.64500000000000002,"5min":2.262,"15min":2.262},"last":0.61399999999999999}]},{"osd":7,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.67000000000000004,"15min":0.70399999999999996},"min":{"1min":0.33400000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.69399999999999995,"5min":4.742,"15min":4.742},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.51300000000000001,"15min":0.51800000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.70199999999999996,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.52600000000000002}]}]},{"osd":0,"up_from":93,"seq":399431958545,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39176,"kb_used_data":2556,"kb_used_omap":13,"kb_used_meta":36594,"kb_avail":94332664,"statfs":{"total":96636764160,"available":96596647936,"internally_reserved":0,"allocated":2617344,"data_stored":1563409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13968,"internal_metadata":37472624},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.317,"5min":0.317,"15min":0.317},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"last":0.254}]},{"osd":2,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.313},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.28899999999999998}]},{"osd":3,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.505,"5min":0.505,"15min":0.505},"last":0.34499999999999997}]},{"osd":4,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.44500000000000001}]},{"osd":5,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.374},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.43099999999999999}]},{"osd":6,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.49099999999999999}]},{"osd":7,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.53800000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5973,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T15:51:49.069 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T15:51:49.341 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:51:49.341 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-09-21T15:51:49.353 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":250,"stamp":"2024-09-21T15:51:47.918416+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590406,"num_objects":1043,"num_object_clones":0,"num_object_copies":2086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1043,"num_whiteouts":1035,"num_read":207535,"num_read_kb":842523,"num_write":23550,"num_write_kb":305657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3397,"num_bytes_recovered":100270055,"num_keys_recovered":316,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20338,"ondisk_log_size":20338,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":416544,"kb_used_data":16932,"kb_used_omap":24,"kb_used_meta":399463,"kb_avail":754558176,"statfs":{"total":773094113280,"available":772667572224,"internally_reserved":0,"allocated":17338368,"data_stored":8940606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25118,"internal_metadata":409050594},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001005"},"pg_stats":[{"pgid":"3.1f","version":"150'413","reported_seq":2132,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.781077+0000","last_change":"2024-09-21T15:51:25.690102+0000","last_active":"2024-09-21T15:51:42.781077+0000","last_peered":"2024-09-21T15:51:42.781077+0000","last_clean":"2024-09-21T15:51:42.781077+0000","last_became_active":"2024-09-21T15:50:32.670964+0000","last_became_peered":"2024-09-21T15:50:32.670964+0000","last_unstale":"2024-09-21T15:51:42.781077+0000","last_undegraded":"2024-09-21T15:51:42.781077+0000","last_fullsized":"2024-09-21T15:51:42.781077+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":413,"log_dups_size":0,"ondisk_log_size":413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:09:09.918864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017896030000000001,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":28,"num_read":2413,"num_read_kb":10419,"num_write":442,"num_write_kb":5956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":1974433,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"149'599","reported_seq":6526,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:44.475566+0000","last_change":"2024-09-21T15:51:25.689127+0000","last_active":"2024-09-21T15:51:44.475566+0000","last_peered":"2024-09-21T15:51:44.475566+0000","last_clean":"2024-09-21T15:51:44.475566+0000","last_became_active":"2024-09-21T15:51:24.691676+0000","last_became_peered":"2024-09-21T15:51:24.691676+0000","last_unstale":"2024-09-21T15:51:44.475566+0000","last_undegraded":"2024-09-21T15:51:44.475566+0000","last_fullsized":"2024-09-21T15:51:44.475566+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":599,"log_dups_size":0,"ondisk_log_size":599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:51:29.795001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010326389999999999,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":38,"num_read":7456,"num_read_kb":25912,"num_write":580,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":250,"num_bytes_recovered":3824749,"num_keys_recovered":34,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"150'655","reported_seq":10214,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.783240+0000","last_change":"2024-09-21T15:51:39.767369+0000","last_active":"2024-09-21T15:51:42.783240+0000","last_peered":"2024-09-21T15:51:42.783240+0000","last_clean":"2024-09-21T15:51:42.783240+0000","last_became_active":"2024-09-21T15:51:39.765580+0000","last_became_peered":"2024-09-21T15:51:39.765580+0000","last_unstale":"2024-09-21T15:51:42.783240+0000","last_undegraded":"2024-09-21T15:51:42.783240+0000","last_fullsized":"2024-09-21T15:51:42.783240+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":655,"log_dups_size":0,"ondisk_log_size":655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:40:43.868959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090302100000000005,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":34,"num_read":10756,"num_read_kb":29175,"num_write":703,"num_write_kb":8882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"150'584","reported_seq":4519,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.783179+0000","last_change":"2024-09-21T15:51:39.767392+0000","last_active":"2024-09-21T15:51:42.783179+0000","last_peered":"2024-09-21T15:51:42.783179+0000","last_clean":"2024-09-21T15:51:42.783179+0000","last_became_active":"2024-09-21T15:51:39.765999+0000","last_became_peered":"2024-09-21T15:51:39.765999+0000","last_unstale":"2024-09-21T15:51:42.783179+0000","last_undegraded":"2024-09-21T15:51:42.783179+0000","last_fullsized":"2024-09-21T15:51:42.783179+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":584,"log_dups_size":0,"ondisk_log_size":584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:21:55.582116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010903670000000001,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":27,"num_read":5242,"num_read_kb":17341,"num_write":644,"num_write_kb":4787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"145'508","reported_seq":4404,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.780869+0000","last_change":"2024-09-21T15:51:25.690506+0000","last_active":"2024-09-21T15:51:42.780869+0000","last_peered":"2024-09-21T15:51:42.780869+0000","last_clean":"2024-09-21T15:51:42.780869+0000","last_became_active":"2024-09-21T15:50:32.674250+0000","last_became_peered":"2024-09-21T15:50:32.674250+0000","last_unstale":"2024-09-21T15:51:42.780869+0000","last_undegraded":"2024-09-21T15:51:42.780869+0000","last_fullsized":"2024-09-21T15:51:42.780869+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":508,"log_dups_size":0,"ondisk_log_size":508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:54:57.399818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019320559999999999,"stat_sum":{"num_bytes":0,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":39,"num_read":4815,"num_read_kb":30088,"num_write":498,"num_write_kb":13507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":4759000,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"150'431","reported_seq":6943,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581775+0000","last_change":"2024-09-21T15:51:25.689223+0000","last_active":"2024-09-21T15:51:43.581775+0000","last_peered":"2024-09-21T15:51:43.581775+0000","last_clean":"2024-09-21T15:51:43.581775+0000","last_became_active":"2024-09-21T15:49:06.693903+0000","last_became_peered":"2024-09-21T15:49:06.693903+0000","last_unstale":"2024-09-21T15:51:43.581775+0000","last_undegraded":"2024-09-21T15:51:43.581775+0000","last_fullsized":"2024-09-21T15:51:43.581775+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":431,"log_dups_size":0,"ondisk_log_size":431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:41:45.432893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047963299999999998,"stat_sum":{"num_bytes":19,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":30,"num_read":7221,"num_read_kb":31759,"num_write":421,"num_write_kb":9476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"148'460","reported_seq":8836,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.491336+0000","last_change":"2024-09-21T15:51:25.780904+0000","last_active":"2024-09-21T15:51:43.491336+0000","last_peered":"2024-09-21T15:51:43.491336+0000","last_clean":"2024-09-21T15:51:43.491336+0000","last_became_active":"2024-09-21T15:51:12.688258+0000","last_became_peered":"2024-09-21T15:51:12.688258+0000","last_unstale":"2024-09-21T15:51:43.491336+0000","last_undegraded":"2024-09-21T15:51:43.491336+0000","last_fullsized":"2024-09-21T15:51:43.491336+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":460,"log_dups_size":0,"ondisk_log_size":460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:19:38.132074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000411691,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":32,"num_read":8283,"num_read_kb":39579,"num_write":518,"num_write_kb":7408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":236,"num_bytes_recovered":6705120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"149'537","reported_seq":4716,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:44.475480+0000","last_change":"2024-09-21T15:51:35.753555+0000","last_active":"2024-09-21T15:51:44.475480+0000","last_peered":"2024-09-21T15:51:44.475480+0000","last_clean":"2024-09-21T15:51:44.475480+0000","last_became_active":"2024-09-21T15:51:35.751700+0000","last_became_peered":"2024-09-21T15:51:35.751700+0000","last_unstale":"2024-09-21T15:51:44.475480+0000","last_undegraded":"2024-09-21T15:51:44.475480+0000","last_fullsized":"2024-09-21T15:51:44.475480+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":537,"log_dups_size":0,"ondisk_log_size":537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:20:44.688916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051035439999999998,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":26,"num_read":6190,"num_read_kb":30937,"num_write":534,"num_write_kb":11075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":282,"num_bytes_recovered":11320990,"num_keys_recovered":70,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"149'1040","reported_seq":9412,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.781000+0000","last_change":"2024-09-21T15:51:25.689670+0000","last_active":"2024-09-21T15:51:42.781000+0000","last_peered":"2024-09-21T15:51:42.781000+0000","last_clean":"2024-09-21T15:51:42.781000+0000","last_became_active":"2024-09-21T15:50:32.668771+0000","last_became_peered":"2024-09-21T15:50:32.668771+0000","last_unstale":"2024-09-21T15:51:42.781000+0000","last_undegraded":"2024-09-21T15:51:42.781000+0000","last_fullsized":"2024-09-21T15:51:42.781000+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":1040,"log_dups_size":0,"ondisk_log_size":1040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:39:43.554150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001503861,"stat_sum":{"num_bytes":0,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":36,"num_read":12714,"num_read_kb":21487,"num_write":1303,"num_write_kb":8257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":3319185,"num_keys_recovered":34,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"150'540","reported_seq":6716,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.783295+0000","last_change":"2024-09-21T15:51:25.757086+0000","last_active":"2024-09-21T15:51:42.783295+0000","last_peered":"2024-09-21T15:51:42.783295+0000","last_clean":"2024-09-21T15:51:42.783295+0000","last_became_active":"2024-09-21T15:49:06.688808+0000","last_became_peered":"2024-09-21T15:49:06.688808+0000","last_unstale":"2024-09-21T15:51:42.783295+0000","last_undegraded":"2024-09-21T15:51:42.783295+0000","last_fullsized":"2024-09-21T15:51:42.783295+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":540,"log_dups_size":0,"ondisk_log_size":540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:43:02.012254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00023944,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":30,"num_read":7604,"num_read_kb":49865,"num_write":542,"num_write_kb":13823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"150'530","reported_seq":4324,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581211+0000","last_change":"2024-09-21T15:51:25.699156+0000","last_active":"2024-09-21T15:51:43.581211+0000","last_peered":"2024-09-21T15:51:43.581211+0000","last_clean":"2024-09-21T15:51:43.581211+0000","last_became_active":"2024-09-21T15:51:11.841664+0000","last_became_peered":"2024-09-21T15:51:11.841664+0000","last_unstale":"2024-09-21T15:51:43.581211+0000","last_undegraded":"2024-09-21T15:51:43.581211+0000","last_fullsized":"2024-09-21T15:51:43.581211+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:47:37.118354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001532828,"stat_sum":{"num_bytes":0,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":29,"num_read":4052,"num_read_kb":27144,"num_write":637,"num_write_kb":17935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":222,"num_bytes_recovered":4468215,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"147'1207","reported_seq":4387,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581846+0000","last_change":"2024-09-21T15:51:25.690386+0000","last_active":"2024-09-21T15:51:43.581846+0000","last_peered":"2024-09-21T15:51:43.581846+0000","last_clean":"2024-09-21T15:51:43.581846+0000","last_became_active":"2024-09-21T15:49:06.688803+0000","last_became_peered":"2024-09-21T15:49:06.688803+0000","last_unstale":"2024-09-21T15:51:43.581846+0000","last_undegraded":"2024-09-21T15:51:43.581846+0000","last_fullsized":"2024-09-21T15:51:43.581846+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":1207,"log_dups_size":0,"ondisk_log_size":1207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:18:47.662631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013586710000000001,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":26,"num_read":7764,"num_read_kb":18232,"num_write":1926,"num_write_kb":6190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":161,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581555+0000","last_change":"2024-09-21T15:48:59.666431+0000","last_active":"2024-09-21T15:51:43.581555+0000","last_peered":"2024-09-21T15:51:43.581555+0000","last_clean":"2024-09-21T15:51:43.581555+0000","last_became_active":"2024-09-21T15:48:57.583665+0000","last_became_peered":"2024-09-21T15:48:57.583665+0000","last_unstale":"2024-09-21T15:51:43.581555+0000","last_undegraded":"2024-09-21T15:51:43.581555+0000","last_fullsized":"2024-09-21T15:51:43.581555+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:52:09.228964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031098399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"150'438","reported_seq":3064,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.780970+0000","last_change":"2024-09-21T15:51:25.689587+0000","last_active":"2024-09-21T15:51:42.780970+0000","last_peered":"2024-09-21T15:51:42.780970+0000","last_clean":"2024-09-21T15:51:42.780970+0000","last_became_active":"2024-09-21T15:50:32.671424+0000","last_became_peered":"2024-09-21T15:50:32.671424+0000","last_unstale":"2024-09-21T15:51:42.780970+0000","last_undegraded":"2024-09-21T15:51:42.780970+0000","last_fullsized":"2024-09-21T15:51:42.780970+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":438,"log_dups_size":0,"ondisk_log_size":438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:04:14.554732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001507467,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":35,"num_read":2637,"num_read_kb":12956,"num_write":388,"num_write_kb":8392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":4494388,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":164,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.491395+0000","last_change":"2024-09-21T15:48:59.589790+0000","last_active":"2024-09-21T15:51:43.491395+0000","last_peered":"2024-09-21T15:51:43.491395+0000","last_clean":"2024-09-21T15:51:43.491395+0000","last_became_active":"2024-09-21T15:48:57.582121+0000","last_became_peered":"2024-09-21T15:48:57.582121+0000","last_unstale":"2024-09-21T15:51:43.491395+0000","last_undegraded":"2024-09-21T15:51:43.491395+0000","last_fullsized":"2024-09-21T15:51:43.491395+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:33:47.592624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039654700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"148'579","reported_seq":8001,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:44.475582+0000","last_change":"2024-09-21T15:51:35.754312+0000","last_active":"2024-09-21T15:51:44.475582+0000","last_peered":"2024-09-21T15:51:44.475582+0000","last_clean":"2024-09-21T15:51:44.475582+0000","last_became_active":"2024-09-21T15:51:35.752700+0000","last_became_peered":"2024-09-21T15:51:35.752700+0000","last_unstale":"2024-09-21T15:51:44.475582+0000","last_undegraded":"2024-09-21T15:51:44.475582+0000","last_fullsized":"2024-09-21T15:51:44.475582+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":579,"log_dups_size":0,"ondisk_log_size":579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:37:03.162105+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.00086244900000000005,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":35,"num_read":8380,"num_read_kb":21195,"num_write":544,"num_write_kb":5397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":256,"num_bytes_recovered":3760369,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":175,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581295+0000","last_change":"2024-09-21T15:50:30.146743+0000","last_active":"2024-09-21T15:51:43.581295+0000","last_peered":"2024-09-21T15:51:43.581295+0000","last_clean":"2024-09-21T15:51:43.581295+0000","last_became_active":"2024-09-21T15:50:30.146428+0000","last_became_peered":"2024-09-21T15:50:30.146428+0000","last_unstale":"2024-09-21T15:51:43.581295+0000","last_undegraded":"2024-09-21T15:51:43.581295+0000","last_fullsized":"2024-09-21T15:51:43.581295+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:58:38.689807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036074600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"145'703","reported_seq":8853,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.491362+0000","last_change":"2024-09-21T15:51:25.780818+0000","last_active":"2024-09-21T15:51:43.491362+0000","last_peered":"2024-09-21T15:51:43.491362+0000","last_clean":"2024-09-21T15:51:43.491362+0000","last_became_active":"2024-09-21T15:50:35.682788+0000","last_became_peered":"2024-09-21T15:50:35.682788+0000","last_unstale":"2024-09-21T15:51:43.491362+0000","last_undegraded":"2024-09-21T15:51:43.491362+0000","last_fullsized":"2024-09-21T15:51:43.491362+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":703,"log_dups_size":0,"ondisk_log_size":703,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:20:40.695690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032907100000000001,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":26,"num_read":11015,"num_read_kb":27666,"num_write":868,"num_write_kb":11017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":175,"num_bytes_recovered":14134268,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":177,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.491361+0000","last_change":"2024-09-21T15:50:30.146654+0000","last_active":"2024-09-21T15:51:43.491361+0000","last_peered":"2024-09-21T15:51:43.491361+0000","last_clean":"2024-09-21T15:51:43.491361+0000","last_became_active":"2024-09-21T15:50:30.146273+0000","last_became_peered":"2024-09-21T15:50:30.146273+0000","last_unstale":"2024-09-21T15:51:43.491361+0000","last_undegraded":"2024-09-21T15:51:43.491361+0000","last_fullsized":"2024-09-21T15:51:43.491361+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:21:48.178519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040525700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"150'603","reported_seq":5376,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.783334+0000","last_change":"2024-09-21T15:51:39.769574+0000","last_active":"2024-09-21T15:51:42.783334+0000","last_peered":"2024-09-21T15:51:42.783334+0000","last_clean":"2024-09-21T15:51:42.783334+0000","last_became_active":"2024-09-21T15:51:39.768300+0000","last_became_peered":"2024-09-21T15:51:39.768300+0000","last_unstale":"2024-09-21T15:51:42.783334+0000","last_undegraded":"2024-09-21T15:51:42.783334+0000","last_fullsized":"2024-09-21T15:51:42.783334+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":603,"log_dups_size":0,"ondisk_log_size":603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:33:47.511137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050855200000000001,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":35,"num_read":5730,"num_read_kb":33940,"num_write":686,"num_write_kb":16332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":268,"num_bytes_recovered":7823491,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"150'921","reported_seq":4195,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581881+0000","last_change":"2024-09-21T15:51:25.690079+0000","last_active":"2024-09-21T15:51:43.581881+0000","last_peered":"2024-09-21T15:51:43.581881+0000","last_clean":"2024-09-21T15:51:43.581881+0000","last_became_active":"2024-09-21T15:50:32.677456+0000","last_became_peered":"2024-09-21T15:50:32.677456+0000","last_unstale":"2024-09-21T15:51:43.581881+0000","last_undegraded":"2024-09-21T15:51:43.581881+0000","last_fullsized":"2024-09-21T15:51:43.581881+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":921,"log_dups_size":0,"ondisk_log_size":921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:46:31.194345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010737730000000001,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":34,"num_read":5136,"num_read_kb":19340,"num_write":949,"num_write_kb":7405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":632176,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"20'2","reported_seq":59,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.783126+0000","last_change":"2024-09-21T15:51:39.764743+0000","last_active":"2024-09-21T15:51:42.783126+0000","last_peered":"2024-09-21T15:51:42.783126+0000","last_clean":"2024-09-21T15:51:42.783126+0000","last_became_active":"2024-09-21T15:51:39.764409+0000","last_became_peered":"2024-09-21T15:51:39.764409+0000","last_unstale":"2024-09-21T15:51:42.783126+0000","last_undegraded":"2024-09-21T15:51:42.783126+0000","last_fullsized":"2024-09-21T15:51:42.783126+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:28:14.932986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000859595,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"148'393","reported_seq":4833,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.491393+0000","last_change":"2024-09-21T15:51:25.781082+0000","last_active":"2024-09-21T15:51:43.491393+0000","last_peered":"2024-09-21T15:51:43.491393+0000","last_clean":"2024-09-21T15:51:43.491393+0000","last_became_active":"2024-09-21T15:50:35.682001+0000","last_became_peered":"2024-09-21T15:50:35.682001+0000","last_unstale":"2024-09-21T15:51:43.491393+0000","last_undegraded":"2024-09-21T15:51:43.491393+0000","last_fullsized":"2024-09-21T15:51:43.491393+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":393,"log_dups_size":0,"ondisk_log_size":393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:09:15.325457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044281000000000002,"stat_sum":{"num_bytes":0,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":31,"num_read":4285,"num_read_kb":23375,"num_write":379,"num_write_kb":4145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":187,"num_bytes_recovered":4305276,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":97,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.788209+0000","last_change":"2024-09-21T15:51:07.683738+0000","last_active":"2024-09-21T15:51:43.788209+0000","last_peered":"2024-09-21T15:51:43.788209+0000","last_clean":"2024-09-21T15:51:43.788209+0000","last_became_active":"2024-09-21T15:51:07.683418+0000","last_became_peered":"2024-09-21T15:51:07.683418+0000","last_unstale":"2024-09-21T15:51:43.788209+0000","last_undegraded":"2024-09-21T15:51:43.788209+0000","last_fullsized":"2024-09-21T15:51:43.788209+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:25:13.468653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"150'558","reported_seq":2598,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.781077+0000","last_change":"2024-09-21T15:51:25.689707+0000","last_active":"2024-09-21T15:51:42.781077+0000","last_peered":"2024-09-21T15:51:42.781077+0000","last_clean":"2024-09-21T15:51:42.781077+0000","last_became_active":"2024-09-21T15:50:32.671413+0000","last_became_peered":"2024-09-21T15:50:32.671413+0000","last_unstale":"2024-09-21T15:51:42.781077+0000","last_undegraded":"2024-09-21T15:51:42.781077+0000","last_fullsized":"2024-09-21T15:51:42.781077+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":558,"log_dups_size":0,"ondisk_log_size":558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:04:19.509402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001292872,"stat_sum":{"num_bytes":0,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":39,"num_read":3663,"num_read_kb":16543,"num_write":590,"num_write_kb":5977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":1696547,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":162,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581078+0000","last_change":"2024-09-21T15:49:00.114730+0000","last_active":"2024-09-21T15:51:43.581078+0000","last_peered":"2024-09-21T15:51:43.581078+0000","last_clean":"2024-09-21T15:51:43.581078+0000","last_became_active":"2024-09-21T15:48:57.585525+0000","last_became_peered":"2024-09-21T15:48:57.585525+0000","last_unstale":"2024-09-21T15:51:43.581078+0000","last_undegraded":"2024-09-21T15:51:43.581078+0000","last_fullsized":"2024-09-21T15:51:43.581078+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:35:35.786526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031691100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"150'413","reported_seq":2957,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:44.475526+0000","last_change":"2024-09-21T15:51:35.754599+0000","last_active":"2024-09-21T15:51:44.475526+0000","last_peered":"2024-09-21T15:51:44.475526+0000","last_clean":"2024-09-21T15:51:44.475526+0000","last_became_active":"2024-09-21T15:51:35.752810+0000","last_became_peered":"2024-09-21T15:51:35.752810+0000","last_unstale":"2024-09-21T15:51:44.475526+0000","last_undegraded":"2024-09-21T15:51:44.475526+0000","last_fullsized":"2024-09-21T15:51:44.475526+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":413,"log_dups_size":0,"ondisk_log_size":413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:54:49.851651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033178999999999999,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":30,"num_read":2149,"num_read_kb":34452,"num_write":406,"num_write_kb":12612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":214,"num_bytes_recovered":2197196,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"18'1","reported_seq":58,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.783151+0000","last_change":"2024-09-21T15:51:39.764710+0000","last_active":"2024-09-21T15:51:42.783151+0000","last_peered":"2024-09-21T15:51:42.783151+0000","last_clean":"2024-09-21T15:51:42.783151+0000","last_became_active":"2024-09-21T15:51:39.764411+0000","last_became_peered":"2024-09-21T15:51:39.764411+0000","last_unstale":"2024-09-21T15:51:42.783151+0000","last_undegraded":"2024-09-21T15:51:42.783151+0000","last_fullsized":"2024-09-21T15:51:42.783151+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:29:50.093645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039559000000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":323,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581078+0000","last_change":"2024-09-21T15:50:30.147280+0000","last_active":"2024-09-21T15:51:43.581078+0000","last_peered":"2024-09-21T15:51:43.581078+0000","last_clean":"2024-09-21T15:51:43.581078+0000","last_became_active":"2024-09-21T15:50:30.146987+0000","last_became_peered":"2024-09-21T15:50:30.146987+0000","last_unstale":"2024-09-21T15:51:43.581078+0000","last_undegraded":"2024-09-21T15:51:43.581078+0000","last_fullsized":"2024-09-21T15:51:43.581078+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:50.161884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:50.161884+0000","last_clean_scrub_stamp":"2024-09-21T15:48:50.161884+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:26:32.853650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"150'540","reported_seq":7944,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:44.475450+0000","last_change":"2024-09-21T15:51:25.689387+0000","last_active":"2024-09-21T15:51:44.475450+0000","last_peered":"2024-09-21T15:51:44.475450+0000","last_clean":"2024-09-21T15:51:44.475450+0000","last_became_active":"2024-09-21T15:51:11.837877+0000","last_became_peered":"2024-09-21T15:51:11.837877+0000","last_unstale":"2024-09-21T15:51:44.475450+0000","last_undegraded":"2024-09-21T15:51:44.475450+0000","last_fullsized":"2024-09-21T15:51:44.475450+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":540,"log_dups_size":0,"ondisk_log_size":540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:45:58.418197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079839600000000004,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":32,"num_read":7239,"num_read_kb":29580,"num_write":558,"num_write_kb":4793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":418,"num_bytes_recovered":6262707,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"150'389","reported_seq":3170,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581718+0000","last_change":"2024-09-21T15:51:25.689626+0000","last_active":"2024-09-21T15:51:43.581718+0000","last_peered":"2024-09-21T15:51:43.581718+0000","last_clean":"2024-09-21T15:51:43.581718+0000","last_became_active":"2024-09-21T15:49:07.210506+0000","last_became_peered":"2024-09-21T15:49:07.210506+0000","last_unstale":"2024-09-21T15:51:43.581718+0000","last_undegraded":"2024-09-21T15:51:43.581718+0000","last_fullsized":"2024-09-21T15:51:43.581718+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":389,"log_dups_size":0,"ondisk_log_size":389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:55:18.046738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089193900000000001,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":30,"num_read":2716,"num_read_kb":28245,"num_write":454,"num_write_kb":12090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"149'595","reported_seq":12267,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581556+0000","last_change":"2024-09-21T15:51:41.782922+0000","last_active":"2024-09-21T15:51:43.581556+0000","last_peered":"2024-09-21T15:51:43.581556+0000","last_clean":"2024-09-21T15:51:43.581556+0000","last_became_active":"2024-09-21T15:51:41.781781+0000","last_became_peered":"2024-09-21T15:51:41.781781+0000","last_unstale":"2024-09-21T15:51:43.581556+0000","last_undegraded":"2024-09-21T15:51:43.581556+0000","last_fullsized":"2024-09-21T15:51:43.581556+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":595,"log_dups_size":0,"ondisk_log_size":595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:10:04.846054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000309432,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":29,"num_read":12828,"num_read_kb":63467,"num_write":580,"num_write_kb":14212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"147'851","reported_seq":3855,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:44.475479+0000","last_change":"2024-09-21T15:51:35.752901+0000","last_active":"2024-09-21T15:51:44.475479+0000","last_peered":"2024-09-21T15:51:44.475479+0000","last_clean":"2024-09-21T15:51:44.475479+0000","last_became_active":"2024-09-21T15:51:35.751150+0000","last_became_peered":"2024-09-21T15:51:35.751150+0000","last_unstale":"2024-09-21T15:51:44.475479+0000","last_undegraded":"2024-09-21T15:51:44.475479+0000","last_fullsized":"2024-09-21T15:51:44.475479+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":851,"log_dups_size":0,"ondisk_log_size":851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:33:02.873060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001158526,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":26,"num_read":5948,"num_read_kb":20169,"num_write":1244,"num_write_kb":7750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":262,"num_bytes_recovered":8353711,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"150'739","reported_seq":6768,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.783290+0000","last_change":"2024-09-21T15:51:39.769469+0000","last_active":"2024-09-21T15:51:42.783290+0000","last_peered":"2024-09-21T15:51:42.783290+0000","last_clean":"2024-09-21T15:51:42.783290+0000","last_became_active":"2024-09-21T15:51:39.768321+0000","last_became_peered":"2024-09-21T15:51:39.768321+0000","last_unstale":"2024-09-21T15:51:42.783290+0000","last_undegraded":"2024-09-21T15:51:42.783290+0000","last_fullsized":"2024-09-21T15:51:42.783290+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":739,"log_dups_size":0,"ondisk_log_size":739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:21:29.566035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074635999999999995,"stat_sum":{"num_bytes":0,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":36,"num_read":8849,"num_read_kb":25678,"num_write":868,"num_write_kb":5005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":246,"num_bytes_recovered":5686754,"num_keys_recovered":23,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"147'894","reported_seq":4314,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581242+0000","last_change":"2024-09-21T15:51:25.698246+0000","last_active":"2024-09-21T15:51:43.581242+0000","last_peered":"2024-09-21T15:51:43.581242+0000","last_clean":"2024-09-21T15:51:43.581242+0000","last_became_active":"2024-09-21T15:49:06.689203+0000","last_became_peered":"2024-09-21T15:49:06.689203+0000","last_unstale":"2024-09-21T15:51:43.581242+0000","last_undegraded":"2024-09-21T15:51:43.581242+0000","last_fullsized":"2024-09-21T15:51:43.581242+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":894,"log_dups_size":0,"ondisk_log_size":894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:22:17.578359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084935700000000002,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":35,"num_read":7321,"num_read_kb":28131,"num_write":1146,"num_write_kb":14809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"148'568","reported_seq":8144,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581299+0000","last_change":"2024-09-21T15:51:25.697735+0000","last_active":"2024-09-21T15:51:43.581299+0000","last_peered":"2024-09-21T15:51:43.581299+0000","last_clean":"2024-09-21T15:51:43.581299+0000","last_became_active":"2024-09-21T15:49:06.690101+0000","last_became_peered":"2024-09-21T15:49:06.690101+0000","last_unstale":"2024-09-21T15:51:43.581299+0000","last_undegraded":"2024-09-21T15:51:43.581299+0000","last_fullsized":"2024-09-21T15:51:43.581299+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":568,"log_dups_size":0,"ondisk_log_size":568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:43:30.659104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044571200000000002,"stat_sum":{"num_bytes":0,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":29,"num_read":8735,"num_read_kb":42983,"num_write":509,"num_write_kb":16273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"145'698","reported_seq":3799,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581254+0000","last_change":"2024-09-21T15:51:25.698380+0000","last_active":"2024-09-21T15:51:43.581254+0000","last_peered":"2024-09-21T15:51:43.581254+0000","last_clean":"2024-09-21T15:51:43.581254+0000","last_became_active":"2024-09-21T15:49:06.694250+0000","last_became_peered":"2024-09-21T15:49:06.694250+0000","last_unstale":"2024-09-21T15:51:43.581254+0000","last_undegraded":"2024-09-21T15:51:43.581254+0000","last_fullsized":"2024-09-21T15:51:43.581254+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":698,"log_dups_size":0,"ondisk_log_size":698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:21:37.777769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086632899999999999,"stat_sum":{"num_bytes":0,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":42,"num_read":4945,"num_read_kb":15920,"num_write":889,"num_write_kb":5075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"138'470","reported_seq":3154,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581542+0000","last_change":"2024-09-21T15:51:25.780857+0000","last_active":"2024-09-21T15:51:43.581542+0000","last_peered":"2024-09-21T15:51:43.581542+0000","last_clean":"2024-09-21T15:51:43.581542+0000","last_became_active":"2024-09-21T15:50:32.676147+0000","last_became_peered":"2024-09-21T15:50:32.676147+0000","last_unstale":"2024-09-21T15:51:43.581542+0000","last_undegraded":"2024-09-21T15:51:43.581542+0000","last_fullsized":"2024-09-21T15:51:43.581542+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":470,"log_dups_size":0,"ondisk_log_size":470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:44:18.821310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00024304599999999999,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":40,"num_read":3160,"num_read_kb":11849,"num_write":381,"num_write_kb":4471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":1191778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"148'1012","reported_seq":4197,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581091+0000","last_change":"2024-09-21T15:51:25.697763+0000","last_active":"2024-09-21T15:51:43.581091+0000","last_peered":"2024-09-21T15:51:43.581091+0000","last_clean":"2024-09-21T15:51:43.581091+0000","last_became_active":"2024-09-21T15:49:06.688798+0000","last_became_peered":"2024-09-21T15:49:06.688798+0000","last_unstale":"2024-09-21T15:51:43.581091+0000","last_undegraded":"2024-09-21T15:51:43.581091+0000","last_fullsized":"2024-09-21T15:51:43.581091+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":1012,"log_dups_size":0,"ondisk_log_size":1012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:48:33.679133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053058599999999999,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":30,"num_read":6391,"num_read_kb":16970,"num_write":1355,"num_write_kb":8408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"146'383","reported_seq":3011,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.780811+0000","last_change":"2024-09-21T15:51:25.689299+0000","last_active":"2024-09-21T15:51:42.780811+0000","last_peered":"2024-09-21T15:51:42.780811+0000","last_clean":"2024-09-21T15:51:42.780811+0000","last_became_active":"2024-09-21T15:50:32.671686+0000","last_became_peered":"2024-09-21T15:50:32.671686+0000","last_unstale":"2024-09-21T15:51:42.780811+0000","last_undegraded":"2024-09-21T15:51:42.780811+0000","last_fullsized":"2024-09-21T15:51:42.780811+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":383,"log_dups_size":0,"ondisk_log_size":383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:40:53.968477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012972649999999999,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":34,"num_read":2455,"num_read_kb":11669,"num_write":401,"num_write_kb":13839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":3359702,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"150'967","reported_seq":7431,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581127+0000","last_change":"2024-09-21T15:51:25.699098+0000","last_active":"2024-09-21T15:51:43.581127+0000","last_peered":"2024-09-21T15:51:43.581127+0000","last_clean":"2024-09-21T15:51:43.581127+0000","last_became_active":"2024-09-21T15:49:06.690128+0000","last_became_peered":"2024-09-21T15:49:06.690128+0000","last_unstale":"2024-09-21T15:51:43.581127+0000","last_undegraded":"2024-09-21T15:51:43.581127+0000","last_fullsized":"2024-09-21T15:51:43.581127+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":967,"log_dups_size":0,"ondisk_log_size":967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:11.763959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014652879999999999,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":32,"num_read":9327,"num_read_kb":26357,"num_write":1446,"num_write_kb":13580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":1039,"num_object_clones":0,"num_object_copies":2078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1039,"num_whiteouts":1035,"num_read":207419,"num_read_kb":842423,"num_write":23389,"num_write_kb":303239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3397,"num_bytes_recovered":100270055,"num_keys_recovered":316,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5973,"internal_metadata":0},"log_size":20218,"ondisk_log_size":20218,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574887,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77952,"kb_used_data":2544,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94293888,"statfs":{"total":96636764160,"available":96556941312,"internally_reserved":0,"allocated":2605056,"data_stored":1557596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":77199819},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:32 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.44},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.64100000000000001}]},{"osd":1,"last update":"Sat Sep 21 15:50:52 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.48299999999999998,"15min":0.495},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":0.73199999999999998,"5min":1.04,"15min":1.04},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.44,"5min":0.46999999999999997,"15min":0.47499999999999998},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.64500000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.58599999999999997}]},{"osd":2,"last update":"Sat Sep 21 15:50:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.439,"15min":0.44400000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.53500000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.437,"5min":0.498,"15min":0.50800000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.70099999999999996,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.55300000000000005}]},{"osd":3,"last update":"Sat Sep 21 15:50:52 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.49099999999999999,"15min":0.499},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.65300000000000002,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.432,"5min":0.47099999999999997,"15min":0.47699999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68200000000000005,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.57099999999999995}]},{"osd":4,"last update":"Sat Sep 21 15:50:52 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.41699999999999998,"15min":0.42299999999999999},"min":{"1min":0.24099999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.59999999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.38,"5min":0.39000000000000001,"15min":0.39200000000000002},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.52700000000000002}]},{"osd":5,"last update":"Sat Sep 21 15:50:52 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.439,"15min":0.45000000000000001},"min":{"1min":0.25900000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.52900000000000003,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.44700000000000001,"15min":0.45000000000000001},"min":{"1min":0.29499999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.68600000000000005,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.70199999999999996}]},{"osd":6,"last update":"Sat Sep 21 15:50:52 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.49399999999999999,"15min":0.5},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.72099999999999997,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.74199999999999999,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.77100000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574890,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41284,"kb_used_data":1976,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94330556,"statfs":{"total":96636764160,"available":96594489344,"internally_reserved":0,"allocated":2023424,"data_stored":973041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:31 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.60399999999999998}]},{"osd":1,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.44800000000000001,"15min":0.45800000000000002},"min":{"1min":0.26900000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.53300000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.46800000000000003,"15min":0.47999999999999998},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.53300000000000003,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.48899999999999999}]},{"osd":2,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.47099999999999997,"15min":0.47699999999999998},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.45800000000000002,"15min":0.46200000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.60999999999999999,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.438}]},{"osd":3,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.51400000000000001,"15min":0.52400000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.70399999999999996,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.53100000000000003,"15min":0.53800000000000003},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.73699999999999999,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.51600000000000001}]},{"osd":4,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.40600000000000003,"15min":0.41399999999999998},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.222},"max":{"1min":0.57499999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.379,"5min":0.46700000000000003,"15min":0.48199999999999998},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.76300000000000001,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.65700000000000003}]},{"osd":5,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.44600000000000001,"15min":0.45600000000000002},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.57399999999999995,"5min":0.755,"15min":0.755},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.372,"5min":0.45000000000000001,"15min":0.46300000000000002},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.628,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.626}]},{"osd":7,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.45000000000000001,"15min":0.45700000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.55400000000000005,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.497,"15min":0.51100000000000001},"min":{"1min":0.30499999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.748,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.67600000000000005}]}]},{"osd":5,"up_from":13,"seq":55834574889,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55232,"kb_used_data":1968,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94316608,"statfs":{"total":96636764160,"available":96580206592,"internally_reserved":0,"allocated":2015232,"data_stored":967247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.5},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.66400000000000003}]},{"osd":1,"last update":"Sat Sep 21 15:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.71699999999999997,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.5,"15min":0.51400000000000001},"min":{"1min":0.34399999999999997,"5min":0.314,"15min":0.314},"max":{"1min":0.52200000000000002,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.56699999999999995}]},{"osd":2,"last update":"Sat Sep 21 15:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.49399999999999999,"15min":0.50600000000000001},"min":{"1min":0.34200000000000003,"5min":0.311,"15min":0.311},"max":{"1min":0.56899999999999995,"5min":1.046,"15min":1.046},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.437,"5min":0.46800000000000003,"15min":0.47299999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.68999999999999995,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.68000000000000005}]},{"osd":3,"last update":"Sat Sep 21 15:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.47699999999999998,"15min":0.48699999999999999},"min":{"1min":0.29599999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.55200000000000005,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.53800000000000003,"15min":0.55200000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.76900000000000002,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.59099999999999997}]},{"osd":4,"last update":"Sat Sep 21 15:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.65400000000000003,"15min":0.70799999999999996},"min":{"1min":0.254,"5min":0.253,"15min":0.253},"max":{"1min":0.503,"5min":5.6399999999999997,"15min":5.6399999999999997},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.45000000000000001,"15min":0.47199999999999998},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.48899999999999999,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.69499999999999995}]},{"osd":6,"last update":"Sat Sep 21 15:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.443,"15min":0.45500000000000002},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.51700000000000002,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.48199999999999998,"15min":0.499},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.55200000000000005,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.64100000000000001}]},{"osd":7,"last update":"Sat Sep 21 15:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.47199999999999998,"15min":0.48599999999999999},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.56000000000000005,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.623},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.504,"15min":0.52100000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.439}]}]},{"osd":4,"up_from":13,"seq":55834574888,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65084,"kb_used_data":1964,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94306756,"statfs":{"total":96636764160,"available":96570118144,"internally_reserved":0,"allocated":2011136,"data_stored":965965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.56299999999999994}]},{"osd":1,"last update":"Sat Sep 21 15:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.433,"15min":0.41899999999999998},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":1.8839999999999999,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.33600000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.55000000000000004}]},{"osd":2,"last update":"Sat Sep 21 15:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.443,"15min":0.439},"min":{"1min":0.311,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.501},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.433,"15min":0.433},"min":{"1min":0.31,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.54000000000000004,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.47999999999999998}]},{"osd":3,"last update":"Sat Sep 21 15:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.34200000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.66600000000000004,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.46800000000000003,"15min":0.46999999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.60899999999999999,"5min":0.626,"15min":0.626},"last":0.52600000000000002}]},{"osd":5,"last update":"Sat Sep 21 15:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.47199999999999998,"15min":0.47599999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":1.919,"5min":2.4489999999999998,"15min":2.4489999999999998},"last":0.376},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.42399999999999999,"15min":0.42499999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.748,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.41099999999999998}]},{"osd":6,"last update":"Sat Sep 21 15:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.41599999999999998,"15min":0.41399999999999998},"min":{"1min":0.29399999999999998,"5min":0.245,"15min":0.245},"max":{"1min":0.63800000000000001,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.55700000000000005,"15min":0.54500000000000004},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":4.1340000000000003,"5min":4.1340000000000003,"15min":4.1340000000000003},"last":0.51500000000000001}]},{"osd":7,"last update":"Sat Sep 21 15:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.45400000000000001,"15min":0.434},"min":{"1min":0.28199999999999997,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":2.7599999999999998,"5min":2.7599999999999998,"15min":2.7599999999999998},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.52400000000000002,"15min":0.52200000000000002},"min":{"1min":0.32600000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":2.016,"5min":2.016,"15min":2.016},"last":0.434}]}]},{"osd":3,"up_from":13,"seq":55834574889,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59340,"kb_used_data":1976,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312500,"statfs":{"total":96636764160,"available":96576000000,"internally_reserved":0,"allocated":2023424,"data_stored":973041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":58718664},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.31,"5min":0.31,"15min":0.31},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"last":0.27800000000000002}]},{"osd":1,"last update":"Sat Sep 21 15:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.28999999999999998,"5min":0.33600000000000002,"15min":0.34399999999999997},"min":{"1min":0.219,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.42299999999999999,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.222},{"interface":"front","average":{"1min":0.30499999999999999,"5min":0.36799999999999999,"15min":0.378},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.40500000000000003,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.313}]},{"osd":2,"last update":"Sat Sep 21 15:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.38400000000000001,"15min":0.39300000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.49199999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.40999999999999998,"15min":0.42099999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.52000000000000002,"5min":0.755,"15min":0.755},"last":0.41299999999999998}]},{"osd":4,"last update":"Sat Sep 21 15:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.47899999999999998,"15min":0.49199999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.56699999999999995,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.504,"15min":0.51300000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.60399999999999998,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.51800000000000002}]},{"osd":5,"last update":"Sat Sep 21 15:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.51000000000000001,"15min":0.52500000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.68100000000000005,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.503},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.53400000000000003,"15min":0.55300000000000005},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.54100000000000004,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.48799999999999999}]},{"osd":6,"last update":"Sat Sep 21 15:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.52800000000000002,"15min":0.54200000000000004},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.622,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.53000000000000003,"15min":0.54600000000000004},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.65700000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.57499999999999996}]},{"osd":7,"last update":"Sat Sep 21 15:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.54700000000000004,"15min":0.56200000000000006},"min":{"1min":0.36299999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.60599999999999998,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.54600000000000004,"15min":0.56200000000000006},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.65000000000000002,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.42199999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574889,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37188,"kb_used_data":1976,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334652,"statfs":{"total":96636764160,"available":96598683648,"internally_reserved":0,"allocated":2023424,"data_stored":973041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:29 2024","interfaces":[{"interface":"back","average":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.439,"5min":0.439,"15min":0.439},"last":0.252},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.30099999999999999}]},{"osd":1,"last update":"Sat Sep 21 15:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.40300000000000002,"15min":0.41999999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.439,"5min":1.621,"15min":1.621},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.308,"5min":0.36399999999999999,"15min":0.374},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.46400000000000002,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.28000000000000003}]},{"osd":3,"last update":"Sat Sep 21 15:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.90800000000000003,"15min":0.97099999999999997},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":4.6580000000000004,"5min":11.56,"15min":11.56},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.40200000000000002,"15min":0.41299999999999998},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.46999999999999997,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.32500000000000001}]},{"osd":4,"last update":"Sat Sep 21 15:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.57999999999999996,"15min":0.60199999999999998},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.60499999999999998,"5min":1.653,"15min":1.653},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.60399999999999998,"15min":0.628},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.74199999999999999,"5min":2.4660000000000002,"15min":2.4660000000000002},"last":0.45800000000000002}]},{"osd":5,"last update":"Sat Sep 21 15:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.501,"15min":0.51100000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.61199999999999999,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.437},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.52700000000000002,"15min":0.53900000000000003},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.65400000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.44900000000000001}]},{"osd":6,"last update":"Sat Sep 21 15:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.54900000000000004,"15min":0.56299999999999994},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.67500000000000004,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.54000000000000004,"15min":0.55600000000000005},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.629,"5min":0.873,"15min":0.873},"last":0.41599999999999998}]},{"osd":7,"last update":"Sat Sep 21 15:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.54900000000000004,"15min":0.56100000000000005},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.67000000000000004,"5min":0.876,"15min":0.876},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.58199999999999996,"15min":0.59799999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.754,"5min":1.089,"15min":1.089},"last":0.48299999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574888,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41288,"kb_used_data":1972,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94330552,"statfs":{"total":96636764160,"available":96594485248,"internally_reserved":0,"allocated":2019328,"data_stored":967266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.312,"15min":0.312},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.40100000000000002}]},{"osd":2,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":0.33000000000000002,"15min":0.34000000000000002},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.43099999999999999,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.30299999999999999,"5min":0.499,"15min":0.53200000000000003},"min":{"1min":0.21299999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.52400000000000002,"5min":5.0670000000000002,"15min":5.0670000000000002},"last":0.34200000000000003}]},{"osd":3,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.91700000000000004,"15min":1.0189999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.46000000000000002,"5min":16.032,"15min":16.032},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.309,"5min":0.34999999999999998,"15min":0.35699999999999998},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":0.47899999999999998,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.36499999999999999}]},{"osd":4,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.46800000000000003,"15min":0.47399999999999998},"min":{"1min":0.25900000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.63200000000000001,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.53600000000000003,"15min":0.55100000000000005},"min":{"1min":0.311,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.56799999999999995,"5min":2.2269999999999999,"15min":2.2269999999999999},"last":0.53400000000000003}]},{"osd":5,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.55300000000000005,"15min":0.57299999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.64700000000000002,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.66000000000000003,"15min":0.69599999999999995},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.66200000000000003,"5min":4.6989999999999998,"15min":4.6989999999999998},"last":0.63200000000000001}]},{"osd":6,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.53000000000000003,"15min":0.53900000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.68200000000000005,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.59999999999999998,"15min":0.62},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.64500000000000002,"5min":2.262,"15min":2.262},"last":0.61399999999999999}]},{"osd":7,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.67000000000000004,"15min":0.70399999999999996},"min":{"1min":0.33400000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.69399999999999995,"5min":4.742,"15min":4.742},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.51300000000000001,"15min":0.51800000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.70199999999999996,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.52600000000000002}]}]},{"osd":0,"up_from":93,"seq":399431958545,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39176,"kb_used_data":2556,"kb_used_omap":13,"kb_used_meta":36594,"kb_avail":94332664,"statfs":{"total":96636764160,"available":96596647936,"internally_reserved":0,"allocated":2617344,"data_stored":1563409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13968,"internal_metadata":37472624},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.317,"5min":0.317,"15min":0.317},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"last":0.254}]},{"osd":2,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.313},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.28899999999999998}]},{"osd":3,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.505,"5min":0.505,"15min":0.505},"last":0.34499999999999997}]},{"osd":4,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.44500000000000001}]},{"osd":5,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.374},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.43099999999999999}]},{"osd":6,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.49099999999999999}]},{"osd":7,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.53800000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5973,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T15:51:49.355 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-21T15:51:49.355 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T15:51:49.628 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:51:49.628 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-09-21T15:51:49.640 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":250,"stamp":"2024-09-21T15:51:47.918416+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590406,"num_objects":1043,"num_object_clones":0,"num_object_copies":2086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1043,"num_whiteouts":1035,"num_read":207535,"num_read_kb":842523,"num_write":23550,"num_write_kb":305657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3397,"num_bytes_recovered":100270055,"num_keys_recovered":316,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20338,"ondisk_log_size":20338,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":416544,"kb_used_data":16932,"kb_used_omap":24,"kb_used_meta":399463,"kb_avail":754558176,"statfs":{"total":773094113280,"available":772667572224,"internally_reserved":0,"allocated":17338368,"data_stored":8940606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25118,"internal_metadata":409050594},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001005"},"pg_stats":[{"pgid":"3.1f","version":"150'413","reported_seq":2132,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.781077+0000","last_change":"2024-09-21T15:51:25.690102+0000","last_active":"2024-09-21T15:51:42.781077+0000","last_peered":"2024-09-21T15:51:42.781077+0000","last_clean":"2024-09-21T15:51:42.781077+0000","last_became_active":"2024-09-21T15:50:32.670964+0000","last_became_peered":"2024-09-21T15:50:32.670964+0000","last_unstale":"2024-09-21T15:51:42.781077+0000","last_undegraded":"2024-09-21T15:51:42.781077+0000","last_fullsized":"2024-09-21T15:51:42.781077+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":413,"log_dups_size":0,"ondisk_log_size":413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:09:09.918864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017896030000000001,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":28,"num_read":2413,"num_read_kb":10419,"num_write":442,"num_write_kb":5956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":1974433,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"149'599","reported_seq":6526,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:44.475566+0000","last_change":"2024-09-21T15:51:25.689127+0000","last_active":"2024-09-21T15:51:44.475566+0000","last_peered":"2024-09-21T15:51:44.475566+0000","last_clean":"2024-09-21T15:51:44.475566+0000","last_became_active":"2024-09-21T15:51:24.691676+0000","last_became_peered":"2024-09-21T15:51:24.691676+0000","last_unstale":"2024-09-21T15:51:44.475566+0000","last_undegraded":"2024-09-21T15:51:44.475566+0000","last_fullsized":"2024-09-21T15:51:44.475566+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":599,"log_dups_size":0,"ondisk_log_size":599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:51:29.795001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010326389999999999,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":38,"num_read":7456,"num_read_kb":25912,"num_write":580,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":250,"num_bytes_recovered":3824749,"num_keys_recovered":34,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"150'655","reported_seq":10214,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.783240+0000","last_change":"2024-09-21T15:51:39.767369+0000","last_active":"2024-09-21T15:51:42.783240+0000","last_peered":"2024-09-21T15:51:42.783240+0000","last_clean":"2024-09-21T15:51:42.783240+0000","last_became_active":"2024-09-21T15:51:39.765580+0000","last_became_peered":"2024-09-21T15:51:39.765580+0000","last_unstale":"2024-09-21T15:51:42.783240+0000","last_undegraded":"2024-09-21T15:51:42.783240+0000","last_fullsized":"2024-09-21T15:51:42.783240+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":655,"log_dups_size":0,"ondisk_log_size":655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:40:43.868959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090302100000000005,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":34,"num_read":10756,"num_read_kb":29175,"num_write":703,"num_write_kb":8882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"150'584","reported_seq":4519,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.783179+0000","last_change":"2024-09-21T15:51:39.767392+0000","last_active":"2024-09-21T15:51:42.783179+0000","last_peered":"2024-09-21T15:51:42.783179+0000","last_clean":"2024-09-21T15:51:42.783179+0000","last_became_active":"2024-09-21T15:51:39.765999+0000","last_became_peered":"2024-09-21T15:51:39.765999+0000","last_unstale":"2024-09-21T15:51:42.783179+0000","last_undegraded":"2024-09-21T15:51:42.783179+0000","last_fullsized":"2024-09-21T15:51:42.783179+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":584,"log_dups_size":0,"ondisk_log_size":584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:21:55.582116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010903670000000001,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":27,"num_read":5242,"num_read_kb":17341,"num_write":644,"num_write_kb":4787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"145'508","reported_seq":4404,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.780869+0000","last_change":"2024-09-21T15:51:25.690506+0000","last_active":"2024-09-21T15:51:42.780869+0000","last_peered":"2024-09-21T15:51:42.780869+0000","last_clean":"2024-09-21T15:51:42.780869+0000","last_became_active":"2024-09-21T15:50:32.674250+0000","last_became_peered":"2024-09-21T15:50:32.674250+0000","last_unstale":"2024-09-21T15:51:42.780869+0000","last_undegraded":"2024-09-21T15:51:42.780869+0000","last_fullsized":"2024-09-21T15:51:42.780869+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":508,"log_dups_size":0,"ondisk_log_size":508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:54:57.399818+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019320559999999999,"stat_sum":{"num_bytes":0,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":39,"num_read":4815,"num_read_kb":30088,"num_write":498,"num_write_kb":13507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":4759000,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"150'431","reported_seq":6943,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581775+0000","last_change":"2024-09-21T15:51:25.689223+0000","last_active":"2024-09-21T15:51:43.581775+0000","last_peered":"2024-09-21T15:51:43.581775+0000","last_clean":"2024-09-21T15:51:43.581775+0000","last_became_active":"2024-09-21T15:49:06.693903+0000","last_became_peered":"2024-09-21T15:49:06.693903+0000","last_unstale":"2024-09-21T15:51:43.581775+0000","last_undegraded":"2024-09-21T15:51:43.581775+0000","last_fullsized":"2024-09-21T15:51:43.581775+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":431,"log_dups_size":0,"ondisk_log_size":431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:41:45.432893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047963299999999998,"stat_sum":{"num_bytes":19,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":30,"num_read":7221,"num_read_kb":31759,"num_write":421,"num_write_kb":9476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"148'460","reported_seq":8836,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.491336+0000","last_change":"2024-09-21T15:51:25.780904+0000","last_active":"2024-09-21T15:51:43.491336+0000","last_peered":"2024-09-21T15:51:43.491336+0000","last_clean":"2024-09-21T15:51:43.491336+0000","last_became_active":"2024-09-21T15:51:12.688258+0000","last_became_peered":"2024-09-21T15:51:12.688258+0000","last_unstale":"2024-09-21T15:51:43.491336+0000","last_undegraded":"2024-09-21T15:51:43.491336+0000","last_fullsized":"2024-09-21T15:51:43.491336+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":460,"log_dups_size":0,"ondisk_log_size":460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:19:38.132074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000411691,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":32,"num_read":8283,"num_read_kb":39579,"num_write":518,"num_write_kb":7408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":236,"num_bytes_recovered":6705120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"149'537","reported_seq":4716,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:44.475480+0000","last_change":"2024-09-21T15:51:35.753555+0000","last_active":"2024-09-21T15:51:44.475480+0000","last_peered":"2024-09-21T15:51:44.475480+0000","last_clean":"2024-09-21T15:51:44.475480+0000","last_became_active":"2024-09-21T15:51:35.751700+0000","last_became_peered":"2024-09-21T15:51:35.751700+0000","last_unstale":"2024-09-21T15:51:44.475480+0000","last_undegraded":"2024-09-21T15:51:44.475480+0000","last_fullsized":"2024-09-21T15:51:44.475480+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":537,"log_dups_size":0,"ondisk_log_size":537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:20:44.688916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051035439999999998,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":26,"num_read":6190,"num_read_kb":30937,"num_write":534,"num_write_kb":11075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":282,"num_bytes_recovered":11320990,"num_keys_recovered":70,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"149'1040","reported_seq":9412,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.781000+0000","last_change":"2024-09-21T15:51:25.689670+0000","last_active":"2024-09-21T15:51:42.781000+0000","last_peered":"2024-09-21T15:51:42.781000+0000","last_clean":"2024-09-21T15:51:42.781000+0000","last_became_active":"2024-09-21T15:50:32.668771+0000","last_became_peered":"2024-09-21T15:50:32.668771+0000","last_unstale":"2024-09-21T15:51:42.781000+0000","last_undegraded":"2024-09-21T15:51:42.781000+0000","last_fullsized":"2024-09-21T15:51:42.781000+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":1040,"log_dups_size":0,"ondisk_log_size":1040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:39:43.554150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001503861,"stat_sum":{"num_bytes":0,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":36,"num_read":12714,"num_read_kb":21487,"num_write":1303,"num_write_kb":8257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":3319185,"num_keys_recovered":34,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"150'540","reported_seq":6716,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.783295+0000","last_change":"2024-09-21T15:51:25.757086+0000","last_active":"2024-09-21T15:51:42.783295+0000","last_peered":"2024-09-21T15:51:42.783295+0000","last_clean":"2024-09-21T15:51:42.783295+0000","last_became_active":"2024-09-21T15:49:06.688808+0000","last_became_peered":"2024-09-21T15:49:06.688808+0000","last_unstale":"2024-09-21T15:51:42.783295+0000","last_undegraded":"2024-09-21T15:51:42.783295+0000","last_fullsized":"2024-09-21T15:51:42.783295+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":540,"log_dups_size":0,"ondisk_log_size":540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:43:02.012254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00023944,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":30,"num_read":7604,"num_read_kb":49865,"num_write":542,"num_write_kb":13823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"150'530","reported_seq":4324,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581211+0000","last_change":"2024-09-21T15:51:25.699156+0000","last_active":"2024-09-21T15:51:43.581211+0000","last_peered":"2024-09-21T15:51:43.581211+0000","last_clean":"2024-09-21T15:51:43.581211+0000","last_became_active":"2024-09-21T15:51:11.841664+0000","last_became_peered":"2024-09-21T15:51:11.841664+0000","last_unstale":"2024-09-21T15:51:43.581211+0000","last_undegraded":"2024-09-21T15:51:43.581211+0000","last_fullsized":"2024-09-21T15:51:43.581211+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:47:37.118354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001532828,"stat_sum":{"num_bytes":0,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":29,"num_read":4052,"num_read_kb":27144,"num_write":637,"num_write_kb":17935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":222,"num_bytes_recovered":4468215,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"147'1207","reported_seq":4387,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581846+0000","last_change":"2024-09-21T15:51:25.690386+0000","last_active":"2024-09-21T15:51:43.581846+0000","last_peered":"2024-09-21T15:51:43.581846+0000","last_clean":"2024-09-21T15:51:43.581846+0000","last_became_active":"2024-09-21T15:49:06.688803+0000","last_became_peered":"2024-09-21T15:49:06.688803+0000","last_unstale":"2024-09-21T15:51:43.581846+0000","last_undegraded":"2024-09-21T15:51:43.581846+0000","last_fullsized":"2024-09-21T15:51:43.581846+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":1207,"log_dups_size":0,"ondisk_log_size":1207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:18:47.662631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013586710000000001,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":26,"num_read":7764,"num_read_kb":18232,"num_write":1926,"num_write_kb":6190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":161,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581555+0000","last_change":"2024-09-21T15:48:59.666431+0000","last_active":"2024-09-21T15:51:43.581555+0000","last_peered":"2024-09-21T15:51:43.581555+0000","last_clean":"2024-09-21T15:51:43.581555+0000","last_became_active":"2024-09-21T15:48:57.583665+0000","last_became_peered":"2024-09-21T15:48:57.583665+0000","last_unstale":"2024-09-21T15:51:43.581555+0000","last_undegraded":"2024-09-21T15:51:43.581555+0000","last_fullsized":"2024-09-21T15:51:43.581555+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:52:09.228964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031098399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"150'438","reported_seq":3064,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.780970+0000","last_change":"2024-09-21T15:51:25.689587+0000","last_active":"2024-09-21T15:51:42.780970+0000","last_peered":"2024-09-21T15:51:42.780970+0000","last_clean":"2024-09-21T15:51:42.780970+0000","last_became_active":"2024-09-21T15:50:32.671424+0000","last_became_peered":"2024-09-21T15:50:32.671424+0000","last_unstale":"2024-09-21T15:51:42.780970+0000","last_undegraded":"2024-09-21T15:51:42.780970+0000","last_fullsized":"2024-09-21T15:51:42.780970+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":438,"log_dups_size":0,"ondisk_log_size":438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:04:14.554732+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001507467,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":35,"num_read":2637,"num_read_kb":12956,"num_write":388,"num_write_kb":8392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":4494388,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":164,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.491395+0000","last_change":"2024-09-21T15:48:59.589790+0000","last_active":"2024-09-21T15:51:43.491395+0000","last_peered":"2024-09-21T15:51:43.491395+0000","last_clean":"2024-09-21T15:51:43.491395+0000","last_became_active":"2024-09-21T15:48:57.582121+0000","last_became_peered":"2024-09-21T15:48:57.582121+0000","last_unstale":"2024-09-21T15:51:43.491395+0000","last_undegraded":"2024-09-21T15:51:43.491395+0000","last_fullsized":"2024-09-21T15:51:43.491395+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:33:47.592624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039654700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"148'579","reported_seq":8001,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:44.475582+0000","last_change":"2024-09-21T15:51:35.754312+0000","last_active":"2024-09-21T15:51:44.475582+0000","last_peered":"2024-09-21T15:51:44.475582+0000","last_clean":"2024-09-21T15:51:44.475582+0000","last_became_active":"2024-09-21T15:51:35.752700+0000","last_became_peered":"2024-09-21T15:51:35.752700+0000","last_unstale":"2024-09-21T15:51:44.475582+0000","last_undegraded":"2024-09-21T15:51:44.475582+0000","last_fullsized":"2024-09-21T15:51:44.475582+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":579,"log_dups_size":0,"ondisk_log_size":579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:37:03.162105+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.00086244900000000005,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":35,"num_read":8380,"num_read_kb":21195,"num_write":544,"num_write_kb":5397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":256,"num_bytes_recovered":3760369,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":175,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581295+0000","last_change":"2024-09-21T15:50:30.146743+0000","last_active":"2024-09-21T15:51:43.581295+0000","last_peered":"2024-09-21T15:51:43.581295+0000","last_clean":"2024-09-21T15:51:43.581295+0000","last_became_active":"2024-09-21T15:50:30.146428+0000","last_became_peered":"2024-09-21T15:50:30.146428+0000","last_unstale":"2024-09-21T15:51:43.581295+0000","last_undegraded":"2024-09-21T15:51:43.581295+0000","last_fullsized":"2024-09-21T15:51:43.581295+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:58:38.689807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036074600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"145'703","reported_seq":8853,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.491362+0000","last_change":"2024-09-21T15:51:25.780818+0000","last_active":"2024-09-21T15:51:43.491362+0000","last_peered":"2024-09-21T15:51:43.491362+0000","last_clean":"2024-09-21T15:51:43.491362+0000","last_became_active":"2024-09-21T15:50:35.682788+0000","last_became_peered":"2024-09-21T15:50:35.682788+0000","last_unstale":"2024-09-21T15:51:43.491362+0000","last_undegraded":"2024-09-21T15:51:43.491362+0000","last_fullsized":"2024-09-21T15:51:43.491362+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":703,"log_dups_size":0,"ondisk_log_size":703,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:20:40.695690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032907100000000001,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":26,"num_read":11015,"num_read_kb":27666,"num_write":868,"num_write_kb":11017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":175,"num_bytes_recovered":14134268,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":177,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.491361+0000","last_change":"2024-09-21T15:50:30.146654+0000","last_active":"2024-09-21T15:51:43.491361+0000","last_peered":"2024-09-21T15:51:43.491361+0000","last_clean":"2024-09-21T15:51:43.491361+0000","last_became_active":"2024-09-21T15:50:30.146273+0000","last_became_peered":"2024-09-21T15:50:30.146273+0000","last_unstale":"2024-09-21T15:51:43.491361+0000","last_undegraded":"2024-09-21T15:51:43.491361+0000","last_fullsized":"2024-09-21T15:51:43.491361+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:21:48.178519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040525700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"150'603","reported_seq":5376,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.783334+0000","last_change":"2024-09-21T15:51:39.769574+0000","last_active":"2024-09-21T15:51:42.783334+0000","last_peered":"2024-09-21T15:51:42.783334+0000","last_clean":"2024-09-21T15:51:42.783334+0000","last_became_active":"2024-09-21T15:51:39.768300+0000","last_became_peered":"2024-09-21T15:51:39.768300+0000","last_unstale":"2024-09-21T15:51:42.783334+0000","last_undegraded":"2024-09-21T15:51:42.783334+0000","last_fullsized":"2024-09-21T15:51:42.783334+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":603,"log_dups_size":0,"ondisk_log_size":603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:33:47.511137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050855200000000001,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":35,"num_read":5730,"num_read_kb":33940,"num_write":686,"num_write_kb":16332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":268,"num_bytes_recovered":7823491,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"150'921","reported_seq":4195,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581881+0000","last_change":"2024-09-21T15:51:25.690079+0000","last_active":"2024-09-21T15:51:43.581881+0000","last_peered":"2024-09-21T15:51:43.581881+0000","last_clean":"2024-09-21T15:51:43.581881+0000","last_became_active":"2024-09-21T15:50:32.677456+0000","last_became_peered":"2024-09-21T15:50:32.677456+0000","last_unstale":"2024-09-21T15:51:43.581881+0000","last_undegraded":"2024-09-21T15:51:43.581881+0000","last_fullsized":"2024-09-21T15:51:43.581881+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":921,"log_dups_size":0,"ondisk_log_size":921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:46:31.194345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010737730000000001,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":34,"num_read":5136,"num_read_kb":19340,"num_write":949,"num_write_kb":7405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":632176,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"20'2","reported_seq":59,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.783126+0000","last_change":"2024-09-21T15:51:39.764743+0000","last_active":"2024-09-21T15:51:42.783126+0000","last_peered":"2024-09-21T15:51:42.783126+0000","last_clean":"2024-09-21T15:51:42.783126+0000","last_became_active":"2024-09-21T15:51:39.764409+0000","last_became_peered":"2024-09-21T15:51:39.764409+0000","last_unstale":"2024-09-21T15:51:42.783126+0000","last_undegraded":"2024-09-21T15:51:42.783126+0000","last_fullsized":"2024-09-21T15:51:42.783126+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:28:14.932986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000859595,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"148'393","reported_seq":4833,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.491393+0000","last_change":"2024-09-21T15:51:25.781082+0000","last_active":"2024-09-21T15:51:43.491393+0000","last_peered":"2024-09-21T15:51:43.491393+0000","last_clean":"2024-09-21T15:51:43.491393+0000","last_became_active":"2024-09-21T15:50:35.682001+0000","last_became_peered":"2024-09-21T15:50:35.682001+0000","last_unstale":"2024-09-21T15:51:43.491393+0000","last_undegraded":"2024-09-21T15:51:43.491393+0000","last_fullsized":"2024-09-21T15:51:43.491393+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":393,"log_dups_size":0,"ondisk_log_size":393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:09:15.325457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044281000000000002,"stat_sum":{"num_bytes":0,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":31,"num_read":4285,"num_read_kb":23375,"num_write":379,"num_write_kb":4145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":187,"num_bytes_recovered":4305276,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":97,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.788209+0000","last_change":"2024-09-21T15:51:07.683738+0000","last_active":"2024-09-21T15:51:43.788209+0000","last_peered":"2024-09-21T15:51:43.788209+0000","last_clean":"2024-09-21T15:51:43.788209+0000","last_became_active":"2024-09-21T15:51:07.683418+0000","last_became_peered":"2024-09-21T15:51:07.683418+0000","last_unstale":"2024-09-21T15:51:43.788209+0000","last_undegraded":"2024-09-21T15:51:43.788209+0000","last_fullsized":"2024-09-21T15:51:43.788209+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:25:13.468653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"150'558","reported_seq":2598,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.781077+0000","last_change":"2024-09-21T15:51:25.689707+0000","last_active":"2024-09-21T15:51:42.781077+0000","last_peered":"2024-09-21T15:51:42.781077+0000","last_clean":"2024-09-21T15:51:42.781077+0000","last_became_active":"2024-09-21T15:50:32.671413+0000","last_became_peered":"2024-09-21T15:50:32.671413+0000","last_unstale":"2024-09-21T15:51:42.781077+0000","last_undegraded":"2024-09-21T15:51:42.781077+0000","last_fullsized":"2024-09-21T15:51:42.781077+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":558,"log_dups_size":0,"ondisk_log_size":558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:04:19.509402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001292872,"stat_sum":{"num_bytes":0,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":39,"num_read":3663,"num_read_kb":16543,"num_write":590,"num_write_kb":5977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":1696547,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":162,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581078+0000","last_change":"2024-09-21T15:49:00.114730+0000","last_active":"2024-09-21T15:51:43.581078+0000","last_peered":"2024-09-21T15:51:43.581078+0000","last_clean":"2024-09-21T15:51:43.581078+0000","last_became_active":"2024-09-21T15:48:57.585525+0000","last_became_peered":"2024-09-21T15:48:57.585525+0000","last_unstale":"2024-09-21T15:51:43.581078+0000","last_undegraded":"2024-09-21T15:51:43.581078+0000","last_fullsized":"2024-09-21T15:51:43.581078+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:35:35.786526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031691100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"150'413","reported_seq":2957,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:44.475526+0000","last_change":"2024-09-21T15:51:35.754599+0000","last_active":"2024-09-21T15:51:44.475526+0000","last_peered":"2024-09-21T15:51:44.475526+0000","last_clean":"2024-09-21T15:51:44.475526+0000","last_became_active":"2024-09-21T15:51:35.752810+0000","last_became_peered":"2024-09-21T15:51:35.752810+0000","last_unstale":"2024-09-21T15:51:44.475526+0000","last_undegraded":"2024-09-21T15:51:44.475526+0000","last_fullsized":"2024-09-21T15:51:44.475526+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":413,"log_dups_size":0,"ondisk_log_size":413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:54:49.851651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033178999999999999,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":30,"num_read":2149,"num_read_kb":34452,"num_write":406,"num_write_kb":12612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":214,"num_bytes_recovered":2197196,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"18'1","reported_seq":58,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.783151+0000","last_change":"2024-09-21T15:51:39.764710+0000","last_active":"2024-09-21T15:51:42.783151+0000","last_peered":"2024-09-21T15:51:42.783151+0000","last_clean":"2024-09-21T15:51:42.783151+0000","last_became_active":"2024-09-21T15:51:39.764411+0000","last_became_peered":"2024-09-21T15:51:39.764411+0000","last_unstale":"2024-09-21T15:51:42.783151+0000","last_undegraded":"2024-09-21T15:51:42.783151+0000","last_fullsized":"2024-09-21T15:51:42.783151+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:29:50.093645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039559000000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":323,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581078+0000","last_change":"2024-09-21T15:50:30.147280+0000","last_active":"2024-09-21T15:51:43.581078+0000","last_peered":"2024-09-21T15:51:43.581078+0000","last_clean":"2024-09-21T15:51:43.581078+0000","last_became_active":"2024-09-21T15:50:30.146987+0000","last_became_peered":"2024-09-21T15:50:30.146987+0000","last_unstale":"2024-09-21T15:51:43.581078+0000","last_undegraded":"2024-09-21T15:51:43.581078+0000","last_fullsized":"2024-09-21T15:51:43.581078+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:50.161884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:50.161884+0000","last_clean_scrub_stamp":"2024-09-21T15:48:50.161884+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:26:32.853650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"150'540","reported_seq":7944,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:44.475450+0000","last_change":"2024-09-21T15:51:25.689387+0000","last_active":"2024-09-21T15:51:44.475450+0000","last_peered":"2024-09-21T15:51:44.475450+0000","last_clean":"2024-09-21T15:51:44.475450+0000","last_became_active":"2024-09-21T15:51:11.837877+0000","last_became_peered":"2024-09-21T15:51:11.837877+0000","last_unstale":"2024-09-21T15:51:44.475450+0000","last_undegraded":"2024-09-21T15:51:44.475450+0000","last_fullsized":"2024-09-21T15:51:44.475450+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":540,"log_dups_size":0,"ondisk_log_size":540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:45:58.418197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079839600000000004,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":32,"num_read":7239,"num_read_kb":29580,"num_write":558,"num_write_kb":4793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":418,"num_bytes_recovered":6262707,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"150'389","reported_seq":3170,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581718+0000","last_change":"2024-09-21T15:51:25.689626+0000","last_active":"2024-09-21T15:51:43.581718+0000","last_peered":"2024-09-21T15:51:43.581718+0000","last_clean":"2024-09-21T15:51:43.581718+0000","last_became_active":"2024-09-21T15:49:07.210506+0000","last_became_peered":"2024-09-21T15:49:07.210506+0000","last_unstale":"2024-09-21T15:51:43.581718+0000","last_undegraded":"2024-09-21T15:51:43.581718+0000","last_fullsized":"2024-09-21T15:51:43.581718+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":389,"log_dups_size":0,"ondisk_log_size":389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:55:18.046738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089193900000000001,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":30,"num_read":2716,"num_read_kb":28245,"num_write":454,"num_write_kb":12090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"149'595","reported_seq":12267,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581556+0000","last_change":"2024-09-21T15:51:41.782922+0000","last_active":"2024-09-21T15:51:43.581556+0000","last_peered":"2024-09-21T15:51:43.581556+0000","last_clean":"2024-09-21T15:51:43.581556+0000","last_became_active":"2024-09-21T15:51:41.781781+0000","last_became_peered":"2024-09-21T15:51:41.781781+0000","last_unstale":"2024-09-21T15:51:43.581556+0000","last_undegraded":"2024-09-21T15:51:43.581556+0000","last_fullsized":"2024-09-21T15:51:43.581556+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":595,"log_dups_size":0,"ondisk_log_size":595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:10:04.846054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000309432,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":29,"num_read":12828,"num_read_kb":63467,"num_write":580,"num_write_kb":14212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"147'851","reported_seq":3855,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:44.475479+0000","last_change":"2024-09-21T15:51:35.752901+0000","last_active":"2024-09-21T15:51:44.475479+0000","last_peered":"2024-09-21T15:51:44.475479+0000","last_clean":"2024-09-21T15:51:44.475479+0000","last_became_active":"2024-09-21T15:51:35.751150+0000","last_became_peered":"2024-09-21T15:51:35.751150+0000","last_unstale":"2024-09-21T15:51:44.475479+0000","last_undegraded":"2024-09-21T15:51:44.475479+0000","last_fullsized":"2024-09-21T15:51:44.475479+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":851,"log_dups_size":0,"ondisk_log_size":851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:33:02.873060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001158526,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":26,"num_read":5948,"num_read_kb":20169,"num_write":1244,"num_write_kb":7750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":262,"num_bytes_recovered":8353711,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"150'739","reported_seq":6768,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.783290+0000","last_change":"2024-09-21T15:51:39.769469+0000","last_active":"2024-09-21T15:51:42.783290+0000","last_peered":"2024-09-21T15:51:42.783290+0000","last_clean":"2024-09-21T15:51:42.783290+0000","last_became_active":"2024-09-21T15:51:39.768321+0000","last_became_peered":"2024-09-21T15:51:39.768321+0000","last_unstale":"2024-09-21T15:51:42.783290+0000","last_undegraded":"2024-09-21T15:51:42.783290+0000","last_fullsized":"2024-09-21T15:51:42.783290+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":739,"log_dups_size":0,"ondisk_log_size":739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:21:29.566035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074635999999999995,"stat_sum":{"num_bytes":0,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":36,"num_read":8849,"num_read_kb":25678,"num_write":868,"num_write_kb":5005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":246,"num_bytes_recovered":5686754,"num_keys_recovered":23,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"147'894","reported_seq":4314,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581242+0000","last_change":"2024-09-21T15:51:25.698246+0000","last_active":"2024-09-21T15:51:43.581242+0000","last_peered":"2024-09-21T15:51:43.581242+0000","last_clean":"2024-09-21T15:51:43.581242+0000","last_became_active":"2024-09-21T15:49:06.689203+0000","last_became_peered":"2024-09-21T15:49:06.689203+0000","last_unstale":"2024-09-21T15:51:43.581242+0000","last_undegraded":"2024-09-21T15:51:43.581242+0000","last_fullsized":"2024-09-21T15:51:43.581242+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":894,"log_dups_size":0,"ondisk_log_size":894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:22:17.578359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084935700000000002,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":35,"num_read":7321,"num_read_kb":28131,"num_write":1146,"num_write_kb":14809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"148'568","reported_seq":8144,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581299+0000","last_change":"2024-09-21T15:51:25.697735+0000","last_active":"2024-09-21T15:51:43.581299+0000","last_peered":"2024-09-21T15:51:43.581299+0000","last_clean":"2024-09-21T15:51:43.581299+0000","last_became_active":"2024-09-21T15:49:06.690101+0000","last_became_peered":"2024-09-21T15:49:06.690101+0000","last_unstale":"2024-09-21T15:51:43.581299+0000","last_undegraded":"2024-09-21T15:51:43.581299+0000","last_fullsized":"2024-09-21T15:51:43.581299+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":568,"log_dups_size":0,"ondisk_log_size":568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:43:30.659104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044571200000000002,"stat_sum":{"num_bytes":0,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":29,"num_read":8735,"num_read_kb":42983,"num_write":509,"num_write_kb":16273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"145'698","reported_seq":3799,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581254+0000","last_change":"2024-09-21T15:51:25.698380+0000","last_active":"2024-09-21T15:51:43.581254+0000","last_peered":"2024-09-21T15:51:43.581254+0000","last_clean":"2024-09-21T15:51:43.581254+0000","last_became_active":"2024-09-21T15:49:06.694250+0000","last_became_peered":"2024-09-21T15:49:06.694250+0000","last_unstale":"2024-09-21T15:51:43.581254+0000","last_undegraded":"2024-09-21T15:51:43.581254+0000","last_fullsized":"2024-09-21T15:51:43.581254+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":698,"log_dups_size":0,"ondisk_log_size":698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:21:37.777769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086632899999999999,"stat_sum":{"num_bytes":0,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":42,"num_read":4945,"num_read_kb":15920,"num_write":889,"num_write_kb":5075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"138'470","reported_seq":3154,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581542+0000","last_change":"2024-09-21T15:51:25.780857+0000","last_active":"2024-09-21T15:51:43.581542+0000","last_peered":"2024-09-21T15:51:43.581542+0000","last_clean":"2024-09-21T15:51:43.581542+0000","last_became_active":"2024-09-21T15:50:32.676147+0000","last_became_peered":"2024-09-21T15:50:32.676147+0000","last_unstale":"2024-09-21T15:51:43.581542+0000","last_undegraded":"2024-09-21T15:51:43.581542+0000","last_fullsized":"2024-09-21T15:51:43.581542+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":470,"log_dups_size":0,"ondisk_log_size":470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:44:18.821310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00024304599999999999,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":40,"num_read":3160,"num_read_kb":11849,"num_write":381,"num_write_kb":4471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":1191778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"148'1012","reported_seq":4197,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581091+0000","last_change":"2024-09-21T15:51:25.697763+0000","last_active":"2024-09-21T15:51:43.581091+0000","last_peered":"2024-09-21T15:51:43.581091+0000","last_clean":"2024-09-21T15:51:43.581091+0000","last_became_active":"2024-09-21T15:49:06.688798+0000","last_became_peered":"2024-09-21T15:49:06.688798+0000","last_unstale":"2024-09-21T15:51:43.581091+0000","last_undegraded":"2024-09-21T15:51:43.581091+0000","last_fullsized":"2024-09-21T15:51:43.581091+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":1012,"log_dups_size":0,"ondisk_log_size":1012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:48:33.679133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053058599999999999,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":30,"num_read":6391,"num_read_kb":16970,"num_write":1355,"num_write_kb":8408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"146'383","reported_seq":3011,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:42.780811+0000","last_change":"2024-09-21T15:51:25.689299+0000","last_active":"2024-09-21T15:51:42.780811+0000","last_peered":"2024-09-21T15:51:42.780811+0000","last_clean":"2024-09-21T15:51:42.780811+0000","last_became_active":"2024-09-21T15:50:32.671686+0000","last_became_peered":"2024-09-21T15:50:32.671686+0000","last_unstale":"2024-09-21T15:51:42.780811+0000","last_undegraded":"2024-09-21T15:51:42.780811+0000","last_fullsized":"2024-09-21T15:51:42.780811+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":383,"log_dups_size":0,"ondisk_log_size":383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:40:53.968477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012972649999999999,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":34,"num_read":2455,"num_read_kb":11669,"num_write":401,"num_write_kb":13839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":3359702,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"150'967","reported_seq":7431,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581127+0000","last_change":"2024-09-21T15:51:25.699098+0000","last_active":"2024-09-21T15:51:43.581127+0000","last_peered":"2024-09-21T15:51:43.581127+0000","last_clean":"2024-09-21T15:51:43.581127+0000","last_became_active":"2024-09-21T15:49:06.690128+0000","last_became_peered":"2024-09-21T15:49:06.690128+0000","last_unstale":"2024-09-21T15:51:43.581127+0000","last_undegraded":"2024-09-21T15:51:43.581127+0000","last_fullsized":"2024-09-21T15:51:43.581127+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":967,"log_dups_size":0,"ondisk_log_size":967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:11.763959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014652879999999999,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":32,"num_read":9327,"num_read_kb":26357,"num_write":1446,"num_write_kb":13580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":1039,"num_object_clones":0,"num_object_copies":2078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1039,"num_whiteouts":1035,"num_read":207419,"num_read_kb":842423,"num_write":23389,"num_write_kb":303239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3397,"num_bytes_recovered":100270055,"num_keys_recovered":316,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5973,"internal_metadata":0},"log_size":20218,"ondisk_log_size":20218,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574887,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77952,"kb_used_data":2544,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94293888,"statfs":{"total":96636764160,"available":96556941312,"internally_reserved":0,"allocated":2605056,"data_stored":1557596,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":77199819},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:32 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.44},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.64100000000000001}]},{"osd":1,"last update":"Sat Sep 21 15:50:52 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.48299999999999998,"15min":0.495},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":0.73199999999999998,"5min":1.04,"15min":1.04},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.44,"5min":0.46999999999999997,"15min":0.47499999999999998},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.64500000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.58599999999999997}]},{"osd":2,"last update":"Sat Sep 21 15:50:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.439,"15min":0.44400000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.53500000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.437,"5min":0.498,"15min":0.50800000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.70099999999999996,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.55300000000000005}]},{"osd":3,"last update":"Sat Sep 21 15:50:52 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.49099999999999999,"15min":0.499},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.65300000000000002,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.432,"5min":0.47099999999999997,"15min":0.47699999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68200000000000005,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.57099999999999995}]},{"osd":4,"last update":"Sat Sep 21 15:50:52 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.41699999999999998,"15min":0.42299999999999999},"min":{"1min":0.24099999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.59999999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.38,"5min":0.39000000000000001,"15min":0.39200000000000002},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.52700000000000002}]},{"osd":5,"last update":"Sat Sep 21 15:50:52 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.439,"15min":0.45000000000000001},"min":{"1min":0.25900000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.52900000000000003,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.44700000000000001,"15min":0.45000000000000001},"min":{"1min":0.29499999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.68600000000000005,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.70199999999999996}]},{"osd":6,"last update":"Sat Sep 21 15:50:52 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.49399999999999999,"15min":0.5},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.72099999999999997,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.67300000000000004},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.74199999999999999,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.77100000000000002}]}]},{"osd":6,"up_from":13,"seq":55834574890,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41284,"kb_used_data":1976,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94330556,"statfs":{"total":96636764160,"available":96594489344,"internally_reserved":0,"allocated":2023424,"data_stored":973041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:31 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.60399999999999998}]},{"osd":1,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.44800000000000001,"15min":0.45800000000000002},"min":{"1min":0.26900000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.53300000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.46800000000000003,"15min":0.47999999999999998},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.53300000000000003,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.48899999999999999}]},{"osd":2,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.47099999999999997,"15min":0.47699999999999998},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.45800000000000002,"15min":0.46200000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.60999999999999999,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.438}]},{"osd":3,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.51400000000000001,"15min":0.52400000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.70399999999999996,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.53100000000000003,"15min":0.53800000000000003},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.73699999999999999,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.51600000000000001}]},{"osd":4,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.40600000000000003,"15min":0.41399999999999998},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.222},"max":{"1min":0.57499999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.379,"5min":0.46700000000000003,"15min":0.48199999999999998},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.76300000000000001,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.65700000000000003}]},{"osd":5,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.44600000000000001,"15min":0.45600000000000002},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.57399999999999995,"5min":0.755,"15min":0.755},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.372,"5min":0.45000000000000001,"15min":0.46300000000000002},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.628,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.626}]},{"osd":7,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.45000000000000001,"15min":0.45700000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.55400000000000005,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.497,"15min":0.51100000000000001},"min":{"1min":0.30499999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.748,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.67600000000000005}]}]},{"osd":5,"up_from":13,"seq":55834574889,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55232,"kb_used_data":1968,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94316608,"statfs":{"total":96636764160,"available":96580206592,"internally_reserved":0,"allocated":2015232,"data_stored":967247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.5},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.66400000000000003}]},{"osd":1,"last update":"Sat Sep 21 15:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.71699999999999997,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.5,"15min":0.51400000000000001},"min":{"1min":0.34399999999999997,"5min":0.314,"15min":0.314},"max":{"1min":0.52200000000000002,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.56699999999999995}]},{"osd":2,"last update":"Sat Sep 21 15:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.49399999999999999,"15min":0.50600000000000001},"min":{"1min":0.34200000000000003,"5min":0.311,"15min":0.311},"max":{"1min":0.56899999999999995,"5min":1.046,"15min":1.046},"last":0.73199999999999998},{"interface":"front","average":{"1min":0.437,"5min":0.46800000000000003,"15min":0.47299999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.68999999999999995,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.68000000000000005}]},{"osd":3,"last update":"Sat Sep 21 15:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.47699999999999998,"15min":0.48699999999999999},"min":{"1min":0.29599999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.55200000000000005,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.53800000000000003,"15min":0.55200000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.76900000000000002,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.59099999999999997}]},{"osd":4,"last update":"Sat Sep 21 15:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.65400000000000003,"15min":0.70799999999999996},"min":{"1min":0.254,"5min":0.253,"15min":0.253},"max":{"1min":0.503,"5min":5.6399999999999997,"15min":5.6399999999999997},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.45000000000000001,"15min":0.47199999999999998},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.48899999999999999,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.69499999999999995}]},{"osd":6,"last update":"Sat Sep 21 15:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.443,"15min":0.45500000000000002},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.51700000000000002,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.48199999999999998,"15min":0.499},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.55200000000000005,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.64100000000000001}]},{"osd":7,"last update":"Sat Sep 21 15:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.47199999999999998,"15min":0.48599999999999999},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.56000000000000005,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.623},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.504,"15min":0.52100000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.439}]}]},{"osd":4,"up_from":13,"seq":55834574888,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65084,"kb_used_data":1964,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94306756,"statfs":{"total":96636764160,"available":96570118144,"internally_reserved":0,"allocated":2011136,"data_stored":965965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.56299999999999994}]},{"osd":1,"last update":"Sat Sep 21 15:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.433,"15min":0.41899999999999998},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":1.8839999999999999,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.33600000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.55000000000000004}]},{"osd":2,"last update":"Sat Sep 21 15:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.443,"15min":0.439},"min":{"1min":0.311,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.501},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.433,"15min":0.433},"min":{"1min":0.31,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.54000000000000004,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.47999999999999998}]},{"osd":3,"last update":"Sat Sep 21 15:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.34200000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.66600000000000004,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.46800000000000003,"15min":0.46999999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.60899999999999999,"5min":0.626,"15min":0.626},"last":0.52600000000000002}]},{"osd":5,"last update":"Sat Sep 21 15:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.47199999999999998,"15min":0.47599999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":1.919,"5min":2.4489999999999998,"15min":2.4489999999999998},"last":0.376},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.42399999999999999,"15min":0.42499999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.748,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.41099999999999998}]},{"osd":6,"last update":"Sat Sep 21 15:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.41599999999999998,"15min":0.41399999999999998},"min":{"1min":0.29399999999999998,"5min":0.245,"15min":0.245},"max":{"1min":0.63800000000000001,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.55700000000000005,"15min":0.54500000000000004},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":4.1340000000000003,"5min":4.1340000000000003,"15min":4.1340000000000003},"last":0.51500000000000001}]},{"osd":7,"last update":"Sat Sep 21 15:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.45400000000000001,"15min":0.434},"min":{"1min":0.28199999999999997,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":2.7599999999999998,"5min":2.7599999999999998,"15min":2.7599999999999998},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.52400000000000002,"15min":0.52200000000000002},"min":{"1min":0.32600000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":2.016,"5min":2.016,"15min":2.016},"last":0.434}]}]},{"osd":3,"up_from":13,"seq":55834574889,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59340,"kb_used_data":1976,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312500,"statfs":{"total":96636764160,"available":96576000000,"internally_reserved":0,"allocated":2023424,"data_stored":973041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":58718664},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.31,"5min":0.31,"15min":0.31},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"last":0.27800000000000002}]},{"osd":1,"last update":"Sat Sep 21 15:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.28999999999999998,"5min":0.33600000000000002,"15min":0.34399999999999997},"min":{"1min":0.219,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.42299999999999999,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.222},{"interface":"front","average":{"1min":0.30499999999999999,"5min":0.36799999999999999,"15min":0.378},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.40500000000000003,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.313}]},{"osd":2,"last update":"Sat Sep 21 15:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.38400000000000001,"15min":0.39300000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.49199999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.40999999999999998,"15min":0.42099999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.52000000000000002,"5min":0.755,"15min":0.755},"last":0.41299999999999998}]},{"osd":4,"last update":"Sat Sep 21 15:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.47899999999999998,"15min":0.49199999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.56699999999999995,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.504,"15min":0.51300000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.60399999999999998,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.51800000000000002}]},{"osd":5,"last update":"Sat Sep 21 15:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.51000000000000001,"15min":0.52500000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.68100000000000005,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.503},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.53400000000000003,"15min":0.55300000000000005},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.54100000000000004,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.48799999999999999}]},{"osd":6,"last update":"Sat Sep 21 15:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.52800000000000002,"15min":0.54200000000000004},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.622,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.53000000000000003,"15min":0.54600000000000004},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.65700000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.57499999999999996}]},{"osd":7,"last update":"Sat Sep 21 15:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.54700000000000004,"15min":0.56200000000000006},"min":{"1min":0.36299999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.60599999999999998,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.54600000000000004,"15min":0.56200000000000006},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.65000000000000002,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.42199999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574889,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37188,"kb_used_data":1976,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334652,"statfs":{"total":96636764160,"available":96598683648,"internally_reserved":0,"allocated":2023424,"data_stored":973041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:29 2024","interfaces":[{"interface":"back","average":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.439,"5min":0.439,"15min":0.439},"last":0.252},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.30099999999999999}]},{"osd":1,"last update":"Sat Sep 21 15:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.40300000000000002,"15min":0.41999999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.439,"5min":1.621,"15min":1.621},"last":0.22700000000000001},{"interface":"front","average":{"1min":0.308,"5min":0.36399999999999999,"15min":0.374},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.46400000000000002,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.28000000000000003}]},{"osd":3,"last update":"Sat Sep 21 15:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.90800000000000003,"15min":0.97099999999999997},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":4.6580000000000004,"5min":11.56,"15min":11.56},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.40200000000000002,"15min":0.41299999999999998},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.46999999999999997,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.32500000000000001}]},{"osd":4,"last update":"Sat Sep 21 15:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.57999999999999996,"15min":0.60199999999999998},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.60499999999999998,"5min":1.653,"15min":1.653},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.60399999999999998,"15min":0.628},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.74199999999999999,"5min":2.4660000000000002,"15min":2.4660000000000002},"last":0.45800000000000002}]},{"osd":5,"last update":"Sat Sep 21 15:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.501,"15min":0.51100000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.61199999999999999,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.437},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.52700000000000002,"15min":0.53900000000000003},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.65400000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.44900000000000001}]},{"osd":6,"last update":"Sat Sep 21 15:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.54900000000000004,"15min":0.56299999999999994},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.67500000000000004,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.54000000000000004,"15min":0.55600000000000005},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.629,"5min":0.873,"15min":0.873},"last":0.41599999999999998}]},{"osd":7,"last update":"Sat Sep 21 15:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.54900000000000004,"15min":0.56100000000000005},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.67000000000000004,"5min":0.876,"15min":0.876},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.58199999999999996,"15min":0.59799999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.754,"5min":1.089,"15min":1.089},"last":0.48299999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574888,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41288,"kb_used_data":1972,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94330552,"statfs":{"total":96636764160,"available":96594485248,"internally_reserved":0,"allocated":2019328,"data_stored":967266,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.312,"15min":0.312},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.40100000000000002}]},{"osd":2,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":0.33000000000000002,"15min":0.34000000000000002},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.43099999999999999,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.30299999999999999,"5min":0.499,"15min":0.53200000000000003},"min":{"1min":0.21299999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.52400000000000002,"5min":5.0670000000000002,"15min":5.0670000000000002},"last":0.34200000000000003}]},{"osd":3,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.91700000000000004,"15min":1.0189999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.46000000000000002,"5min":16.032,"15min":16.032},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.309,"5min":0.34999999999999998,"15min":0.35699999999999998},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":0.47899999999999998,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.36499999999999999}]},{"osd":4,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.46800000000000003,"15min":0.47399999999999998},"min":{"1min":0.25900000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.63200000000000001,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.53600000000000003,"15min":0.55100000000000005},"min":{"1min":0.311,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.56799999999999995,"5min":2.2269999999999999,"15min":2.2269999999999999},"last":0.53400000000000003}]},{"osd":5,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.55300000000000005,"15min":0.57299999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.64700000000000002,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.66000000000000003,"15min":0.69599999999999995},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.66200000000000003,"5min":4.6989999999999998,"15min":4.6989999999999998},"last":0.63200000000000001}]},{"osd":6,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.53000000000000003,"15min":0.53900000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.68200000000000005,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.59999999999999998,"15min":0.62},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.64500000000000002,"5min":2.262,"15min":2.262},"last":0.61399999999999999}]},{"osd":7,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.67000000000000004,"15min":0.70399999999999996},"min":{"1min":0.33400000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.69399999999999995,"5min":4.742,"15min":4.742},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.51300000000000001,"15min":0.51800000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.70199999999999996,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.52600000000000002}]}]},{"osd":0,"up_from":93,"seq":399431958545,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39176,"kb_used_data":2556,"kb_used_omap":13,"kb_used_meta":36594,"kb_avail":94332664,"statfs":{"total":96636764160,"available":96596647936,"internally_reserved":0,"allocated":2617344,"data_stored":1563409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13968,"internal_metadata":37472624},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"last":0.23000000000000001},{"interface":"front","average":{"1min":0.317,"5min":0.317,"15min":0.317},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"last":0.254}]},{"osd":2,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.313},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.28899999999999998}]},{"osd":3,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.505,"5min":0.505,"15min":0.505},"last":0.34499999999999997}]},{"osd":4,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.44500000000000001}]},{"osd":5,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.374},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.43099999999999999}]},{"osd":6,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.49099999999999999}]},{"osd":7,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.53800000000000003}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5973,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T15:51:49.642 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-21T15:51:49.917 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:51:49.917 INFO:teuthology.orchestra.run.smithi089.stdout:{"epoch":168,"fsid":"4a9b64ea-3aaf-4a40-8ce8-02163a0f6e12","created":"2024-09-21T15:48:37.867501+0000","modified":"2024-09-21T15:51:44.790288+0000","last_up_change":"2024-09-21T15:50:29.130795+0000","last_in_change":"2024-09-21T15:51:06.806747+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T15:48:49.921079+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-21T15:48:55.812982+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":3,"snap_epoch":20,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.9900000095367432,"score_stable":1.9900000095367432,"optimal_score":0.87999999523162842,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"pool_client.0","create_time":"2024-09-21T15:49:05.523888+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"150","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":91,"snap_epoch":150,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"7b3fabfa-3fe6-47c7-9d0c-a6036f738771","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":13,"last_clean_end":67,"up_from":93,"up_thru":136,"down_at":69,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6800","nonce":1222314186},{"type":"v1","addr":"172.21.15.89:6801","nonce":1222314186}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6803","nonce":1222314186},{"type":"v1","addr":"172.21.15.89:6805","nonce":1222314186}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6811","nonce":1222314186},{"type":"v1","addr":"172.21.15.89:6813","nonce":1222314186}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6807","nonce":1222314186},{"type":"v1","addr":"172.21.15.89:6809","nonce":1222314186}]},"public_addr":"172.21.15.89:6801/1222314186","cluster_addr":"172.21.15.89:6805/1222314186","heartbeat_back_addr":"172.21.15.89:6813/1222314186","heartbeat_front_addr":"172.21.15.89:6809/1222314186","state":["exists","up"]},{"osd":1,"uuid":"767ad1bc-84ef-4102-83f5-aacda0917ddb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":136,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6816","nonce":3187095367},{"type":"v1","addr":"172.21.15.89:6817","nonce":3187095367}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6818","nonce":3187095367},{"type":"v1","addr":"172.21.15.89:6819","nonce":3187095367}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6822","nonce":3187095367},{"type":"v1","addr":"172.21.15.89:6824","nonce":3187095367}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6820","nonce":3187095367},{"type":"v1","addr":"172.21.15.89:6821","nonce":3187095367}]},"public_addr":"172.21.15.89:6817/3187095367","cluster_addr":"172.21.15.89:6819/3187095367","heartbeat_back_addr":"172.21.15.89:6824/3187095367","heartbeat_front_addr":"172.21.15.89:6821/3187095367","state":["exists","up"]},{"osd":2,"uuid":"1dbd0d50-aae9-42a7-90a1-aa92697778ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":134,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6802","nonce":4098625737},{"type":"v1","addr":"172.21.15.89:6804","nonce":4098625737}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6806","nonce":4098625737},{"type":"v1","addr":"172.21.15.89:6808","nonce":4098625737}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6814","nonce":4098625737},{"type":"v1","addr":"172.21.15.89:6815","nonce":4098625737}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6810","nonce":4098625737},{"type":"v1","addr":"172.21.15.89:6812","nonce":4098625737}]},"public_addr":"172.21.15.89:6804/4098625737","cluster_addr":"172.21.15.89:6808/4098625737","heartbeat_back_addr":"172.21.15.89:6815/4098625737","heartbeat_front_addr":"172.21.15.89:6812/4098625737","state":["exists","up"]},{"osd":3,"uuid":"f37cd295-2bac-4df4-a99b-abedc0978dce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":158,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6823","nonce":1886828410},{"type":"v1","addr":"172.21.15.89:6825","nonce":1886828410}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6826","nonce":1886828410},{"type":"v1","addr":"172.21.15.89:6827","nonce":1886828410}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6830","nonce":1886828410},{"type":"v1","addr":"172.21.15.89:6831","nonce":1886828410}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.89:6828","nonce":1886828410},{"type":"v1","addr":"172.21.15.89:6829","nonce":1886828410}]},"public_addr":"172.21.15.89:6825/1886828410","cluster_addr":"172.21.15.89:6827/1886828410","heartbeat_back_addr":"172.21.15.89:6831/1886828410","heartbeat_front_addr":"172.21.15.89:6829/1886828410","state":["exists","up"]},{"osd":4,"uuid":"2eeb351d-d51c-4fc3-884e-debecd995aea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":148,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6817","nonce":22722842},{"type":"v1","addr":"172.21.15.100:6819","nonce":22722842}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":22722842},{"type":"v1","addr":"172.21.15.100:6822","nonce":22722842}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":22722842},{"type":"v1","addr":"172.21.15.100:6830","nonce":22722842}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":22722842},{"type":"v1","addr":"172.21.15.100:6826","nonce":22722842}]},"public_addr":"172.21.15.100:6819/22722842","cluster_addr":"172.21.15.100:6822/22722842","heartbeat_back_addr":"172.21.15.100:6830/22722842","heartbeat_front_addr":"172.21.15.100:6826/22722842","state":["exists","up"]},{"osd":5,"uuid":"b5396267-6823-42f1-bd04-bd8f144fa76c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":162,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":363733545},{"type":"v1","addr":"172.21.15.100:6818","nonce":363733545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6821","nonce":363733545},{"type":"v1","addr":"172.21.15.100:6823","nonce":363733545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6829","nonce":363733545},{"type":"v1","addr":"172.21.15.100:6831","nonce":363733545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6825","nonce":363733545},{"type":"v1","addr":"172.21.15.100:6827","nonce":363733545}]},"public_addr":"172.21.15.100:6818/363733545","cluster_addr":"172.21.15.100:6823/363733545","heartbeat_back_addr":"172.21.15.100:6831/363733545","heartbeat_front_addr":"172.21.15.100:6827/363733545","state":["exists","up"]},{"osd":6,"uuid":"955b67e9-73bc-42c1-88e0-3356c105e1ea","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":164,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":205694879},{"type":"v1","addr":"172.21.15.100:6809","nonce":205694879}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":205694879},{"type":"v1","addr":"172.21.15.100:6811","nonce":205694879}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":205694879},{"type":"v1","addr":"172.21.15.100:6815","nonce":205694879}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":205694879},{"type":"v1","addr":"172.21.15.100:6813","nonce":205694879}]},"public_addr":"172.21.15.100:6809/205694879","cluster_addr":"172.21.15.100:6811/205694879","heartbeat_back_addr":"172.21.15.100:6815/205694879","heartbeat_front_addr":"172.21.15.100:6813/205694879","state":["exists","up"]},{"osd":7,"uuid":"414ea7f0-138d-404e-bce6-fe8dedcdc938","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":135,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":3120011840},{"type":"v1","addr":"172.21.15.100:6801","nonce":3120011840}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":3120011840},{"type":"v1","addr":"172.21.15.100:6803","nonce":3120011840}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":3120011840},{"type":"v1","addr":"172.21.15.100:6807","nonce":3120011840}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":3120011840},{"type":"v1","addr":"172.21.15.100:6805","nonce":3120011840}]},"public_addr":"172.21.15.100:6801/3120011840","cluster_addr":"172.21.15.100:6803/3120011840","heartbeat_back_addr":"172.21.15.100:6807/3120011840","heartbeat_front_addr":"172.21.15.100:6805/3120011840","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:48:45.230746+0000","dead_epoch":69},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:48:45.303567+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:48:45.210941+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:48:45.218918+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:48:45.301781+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:48:45.248444+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:48:45.275465+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T15:48:45.166209+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T15:51:50.931 INFO:tasks.ceph:Scrubbing osd.0 2024-09-21T15:51:50.931 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-21T15:51:51.064 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:51.065 INFO:teuthology.orchestra.run.smithi089.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-21T15:51:51.065 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:51.076 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-21T15:51:51.347 INFO:teuthology.orchestra.run.smithi089.stderr:instructed osd(s) 0 to deep-scrub 2024-09-21T15:51:51.360 INFO:tasks.ceph:Scrubbing osd.1 2024-09-21T15:51:51.360 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-21T15:51:51.480 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:51.480 INFO:teuthology.orchestra.run.smithi089.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-21T15:51:51.480 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:51.491 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-21T15:51:51.759 INFO:teuthology.orchestra.run.smithi089.stderr:instructed osd(s) 1 to deep-scrub 2024-09-21T15:51:51.772 INFO:tasks.ceph:Scrubbing osd.2 2024-09-21T15:51:51.773 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-21T15:51:51.895 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:51.895 INFO:teuthology.orchestra.run.smithi089.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-21T15:51:51.895 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:51.906 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-21T15:51:52.174 INFO:teuthology.orchestra.run.smithi089.stderr:instructed osd(s) 2 to deep-scrub 2024-09-21T15:51:52.187 INFO:tasks.ceph:Scrubbing osd.3 2024-09-21T15:51:52.187 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-21T15:51:52.307 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:52.307 INFO:teuthology.orchestra.run.smithi089.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-21T15:51:52.307 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:52.317 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-21T15:51:52.587 INFO:teuthology.orchestra.run.smithi089.stderr:instructed osd(s) 3 to deep-scrub 2024-09-21T15:51:52.599 INFO:tasks.ceph:Scrubbing osd.4 2024-09-21T15:51:52.599 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-21T15:51:52.721 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:52.721 INFO:teuthology.orchestra.run.smithi089.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-21T15:51:52.722 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:52.732 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-21T15:51:53.005 INFO:teuthology.orchestra.run.smithi089.stderr:instructed osd(s) 4 to deep-scrub 2024-09-21T15:51:53.018 INFO:tasks.ceph:Scrubbing osd.5 2024-09-21T15:51:53.018 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-21T15:51:53.140 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:53.140 INFO:teuthology.orchestra.run.smithi089.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-21T15:51:53.140 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:53.150 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-21T15:51:53.434 INFO:teuthology.orchestra.run.smithi089.stderr:instructed osd(s) 5 to deep-scrub 2024-09-21T15:51:53.447 INFO:tasks.ceph:Scrubbing osd.6 2024-09-21T15:51:53.447 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-09-21T15:51:53.567 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:53.567 INFO:teuthology.orchestra.run.smithi089.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-21T15:51:53.567 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:53.578 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-09-21T15:51:53.849 INFO:teuthology.orchestra.run.smithi089.stderr:instructed osd(s) 6 to deep-scrub 2024-09-21T15:51:53.863 INFO:tasks.ceph:Scrubbing osd.7 2024-09-21T15:51:53.863 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-09-21T15:51:53.985 INFO:teuthology.orchestra.run.smithi089.stdout:{ 2024-09-21T15:51:53.985 INFO:teuthology.orchestra.run.smithi089.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-21T15:51:53.985 INFO:teuthology.orchestra.run.smithi089.stdout:} 2024-09-21T15:51:53.995 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-09-21T15:51:54.263 INFO:teuthology.orchestra.run.smithi089.stderr:instructed osd(s) 7 to deep-scrub 2024-09-21T15:51:54.275 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T15:51:54.550 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:51:54.551 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-09-21T15:51:54.562 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":253,"stamp":"2024-09-21T15:51:53.919381+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590406,"num_objects":1043,"num_object_clones":0,"num_object_copies":2086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1043,"num_whiteouts":1035,"num_read":207535,"num_read_kb":842523,"num_write":23550,"num_write_kb":305657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3397,"num_bytes_recovered":100270055,"num_keys_recovered":316,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20372,"ondisk_log_size":20372,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":416576,"kb_used_data":16980,"kb_used_omap":24,"kb_used_meta":399463,"kb_avail":754558144,"statfs":{"total":773094113280,"available":772667539456,"internally_reserved":0,"allocated":17387520,"data_stored":8963858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25118,"internal_metadata":409050594},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001747"},"pg_stats":[{"pgid":"3.1f","version":"150'413","reported_seq":2133,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:44.794183+0000","last_change":"2024-09-21T15:51:25.690102+0000","last_active":"2024-09-21T15:51:44.794183+0000","last_peered":"2024-09-21T15:51:44.794183+0000","last_clean":"2024-09-21T15:51:44.794183+0000","last_became_active":"2024-09-21T15:50:32.670964+0000","last_became_peered":"2024-09-21T15:50:32.670964+0000","last_unstale":"2024-09-21T15:51:44.794183+0000","last_undegraded":"2024-09-21T15:51:44.794183+0000","last_fullsized":"2024-09-21T15:51:44.794183+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":413,"log_dups_size":0,"ondisk_log_size":413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0017896030000000001,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":28,"num_read":2413,"num_read_kb":10419,"num_write":442,"num_write_kb":5956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":1974433,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"149'599","reported_seq":6526,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:44.475566+0000","last_change":"2024-09-21T15:51:25.689127+0000","last_active":"2024-09-21T15:51:44.475566+0000","last_peered":"2024-09-21T15:51:44.475566+0000","last_clean":"2024-09-21T15:51:44.475566+0000","last_became_active":"2024-09-21T15:51:24.691676+0000","last_became_peered":"2024-09-21T15:51:24.691676+0000","last_unstale":"2024-09-21T15:51:44.475566+0000","last_undegraded":"2024-09-21T15:51:44.475566+0000","last_fullsized":"2024-09-21T15:51:44.475566+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":599,"log_dups_size":0,"ondisk_log_size":599,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:51:29.795001+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010326389999999999,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":38,"num_read":7456,"num_read_kb":25912,"num_write":580,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":250,"num_bytes_recovered":3824749,"num_keys_recovered":34,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"150'655","reported_seq":10215,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.998596+0000","last_change":"2024-09-21T15:51:39.767369+0000","last_active":"2024-09-21T15:51:43.998596+0000","last_peered":"2024-09-21T15:51:43.998596+0000","last_clean":"2024-09-21T15:51:43.998596+0000","last_became_active":"2024-09-21T15:51:39.765580+0000","last_became_peered":"2024-09-21T15:51:39.765580+0000","last_unstale":"2024-09-21T15:51:43.998596+0000","last_undegraded":"2024-09-21T15:51:43.998596+0000","last_fullsized":"2024-09-21T15:51:43.998596+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":655,"log_dups_size":0,"ondisk_log_size":655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:40:43.868959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090302100000000005,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":34,"num_read":10756,"num_read_kb":29175,"num_write":703,"num_write_kb":8882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"150'584","reported_seq":4520,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.998478+0000","last_change":"2024-09-21T15:51:39.767392+0000","last_active":"2024-09-21T15:51:43.998478+0000","last_peered":"2024-09-21T15:51:43.998478+0000","last_clean":"2024-09-21T15:51:43.998478+0000","last_became_active":"2024-09-21T15:51:39.765999+0000","last_became_peered":"2024-09-21T15:51:39.765999+0000","last_unstale":"2024-09-21T15:51:43.998478+0000","last_undegraded":"2024-09-21T15:51:43.998478+0000","last_fullsized":"2024-09-21T15:51:43.998478+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":584,"log_dups_size":0,"ondisk_log_size":584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:21:55.582116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010903670000000001,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":27,"num_read":5242,"num_read_kb":17341,"num_write":644,"num_write_kb":4787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"145'508","reported_seq":4405,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:44.794045+0000","last_change":"2024-09-21T15:51:25.690506+0000","last_active":"2024-09-21T15:51:44.794045+0000","last_peered":"2024-09-21T15:51:44.794045+0000","last_clean":"2024-09-21T15:51:44.794045+0000","last_became_active":"2024-09-21T15:50:32.674250+0000","last_became_peered":"2024-09-21T15:50:32.674250+0000","last_unstale":"2024-09-21T15:51:44.794045+0000","last_undegraded":"2024-09-21T15:51:44.794045+0000","last_fullsized":"2024-09-21T15:51:44.794045+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":508,"log_dups_size":0,"ondisk_log_size":508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019320559999999999,"stat_sum":{"num_bytes":0,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":39,"num_read":4815,"num_read_kb":30088,"num_write":498,"num_write_kb":13507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":4759000,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"150'431","reported_seq":6944,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.789192+0000","last_change":"2024-09-21T15:51:25.689223+0000","last_active":"2024-09-21T15:51:43.789192+0000","last_peered":"2024-09-21T15:51:43.789192+0000","last_clean":"2024-09-21T15:51:43.789192+0000","last_became_active":"2024-09-21T15:49:06.693903+0000","last_became_peered":"2024-09-21T15:49:06.693903+0000","last_unstale":"2024-09-21T15:51:43.789192+0000","last_undegraded":"2024-09-21T15:51:43.789192+0000","last_fullsized":"2024-09-21T15:51:43.789192+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":431,"log_dups_size":0,"ondisk_log_size":431,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:41:45.432893+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00047963299999999998,"stat_sum":{"num_bytes":19,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":30,"num_read":7221,"num_read_kb":31759,"num_write":421,"num_write_kb":9476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"148'460","reported_seq":8837,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.788381+0000","last_change":"2024-09-21T15:51:25.780904+0000","last_active":"2024-09-21T15:51:43.788381+0000","last_peered":"2024-09-21T15:51:43.788381+0000","last_clean":"2024-09-21T15:51:43.788381+0000","last_became_active":"2024-09-21T15:51:12.688258+0000","last_became_peered":"2024-09-21T15:51:12.688258+0000","last_unstale":"2024-09-21T15:51:43.788381+0000","last_undegraded":"2024-09-21T15:51:43.788381+0000","last_fullsized":"2024-09-21T15:51:43.788381+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":460,"log_dups_size":0,"ondisk_log_size":460,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:19:38.132074+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000411691,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":32,"num_read":8283,"num_read_kb":39579,"num_write":518,"num_write_kb":7408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":236,"num_bytes_recovered":6705120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"149'537","reported_seq":4716,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:44.475480+0000","last_change":"2024-09-21T15:51:35.753555+0000","last_active":"2024-09-21T15:51:44.475480+0000","last_peered":"2024-09-21T15:51:44.475480+0000","last_clean":"2024-09-21T15:51:44.475480+0000","last_became_active":"2024-09-21T15:51:35.751700+0000","last_became_peered":"2024-09-21T15:51:35.751700+0000","last_unstale":"2024-09-21T15:51:44.475480+0000","last_undegraded":"2024-09-21T15:51:44.475480+0000","last_fullsized":"2024-09-21T15:51:44.475480+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":537,"log_dups_size":0,"ondisk_log_size":537,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:20:44.688916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0051035439999999998,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":26,"num_read":6190,"num_read_kb":30937,"num_write":534,"num_write_kb":11075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":282,"num_bytes_recovered":11320990,"num_keys_recovered":70,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"149'1040","reported_seq":9413,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:44.794058+0000","last_change":"2024-09-21T15:51:25.689670+0000","last_active":"2024-09-21T15:51:44.794058+0000","last_peered":"2024-09-21T15:51:44.794058+0000","last_clean":"2024-09-21T15:51:44.794058+0000","last_became_active":"2024-09-21T15:50:32.668771+0000","last_became_peered":"2024-09-21T15:50:32.668771+0000","last_unstale":"2024-09-21T15:51:44.794058+0000","last_undegraded":"2024-09-21T15:51:44.794058+0000","last_fullsized":"2024-09-21T15:51:44.794058+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":1040,"log_dups_size":0,"ondisk_log_size":1040,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001503861,"stat_sum":{"num_bytes":0,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":36,"num_read":12714,"num_read_kb":21487,"num_write":1303,"num_write_kb":8257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":3319185,"num_keys_recovered":34,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"150'540","reported_seq":6717,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.998615+0000","last_change":"2024-09-21T15:51:25.757086+0000","last_active":"2024-09-21T15:51:43.998615+0000","last_peered":"2024-09-21T15:51:43.998615+0000","last_clean":"2024-09-21T15:51:43.998615+0000","last_became_active":"2024-09-21T15:49:06.688808+0000","last_became_peered":"2024-09-21T15:49:06.688808+0000","last_unstale":"2024-09-21T15:51:43.998615+0000","last_undegraded":"2024-09-21T15:51:43.998615+0000","last_fullsized":"2024-09-21T15:51:43.998615+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":540,"log_dups_size":0,"ondisk_log_size":540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:43:02.012254+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00023944,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":30,"num_read":7604,"num_read_kb":49865,"num_write":542,"num_write_kb":13823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"150'530","reported_seq":4325,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:45.082447+0000","last_change":"2024-09-21T15:51:25.699156+0000","last_active":"2024-09-21T15:51:45.082447+0000","last_peered":"2024-09-21T15:51:45.082447+0000","last_clean":"2024-09-21T15:51:45.082447+0000","last_became_active":"2024-09-21T15:51:11.841664+0000","last_became_peered":"2024-09-21T15:51:11.841664+0000","last_unstale":"2024-09-21T15:51:45.082447+0000","last_undegraded":"2024-09-21T15:51:45.082447+0000","last_fullsized":"2024-09-21T15:51:45.082447+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:47:37.118354+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001532828,"stat_sum":{"num_bytes":0,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":29,"num_read":4052,"num_read_kb":27144,"num_write":637,"num_write_kb":17935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":222,"num_bytes_recovered":4468215,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"147'1207","reported_seq":4388,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.789362+0000","last_change":"2024-09-21T15:51:25.690386+0000","last_active":"2024-09-21T15:51:43.789362+0000","last_peered":"2024-09-21T15:51:43.789362+0000","last_clean":"2024-09-21T15:51:43.789362+0000","last_became_active":"2024-09-21T15:49:06.688803+0000","last_became_peered":"2024-09-21T15:49:06.688803+0000","last_unstale":"2024-09-21T15:51:43.789362+0000","last_undegraded":"2024-09-21T15:51:43.789362+0000","last_fullsized":"2024-09-21T15:51:43.789362+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":1207,"log_dups_size":0,"ondisk_log_size":1207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:18:47.662631+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013586710000000001,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":26,"num_read":7764,"num_read_kb":18232,"num_write":1926,"num_write_kb":6190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":161,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581555+0000","last_change":"2024-09-21T15:48:59.666431+0000","last_active":"2024-09-21T15:51:43.581555+0000","last_peered":"2024-09-21T15:51:43.581555+0000","last_clean":"2024-09-21T15:51:43.581555+0000","last_became_active":"2024-09-21T15:48:57.583665+0000","last_became_peered":"2024-09-21T15:48:57.583665+0000","last_unstale":"2024-09-21T15:51:43.581555+0000","last_undegraded":"2024-09-21T15:51:43.581555+0000","last_fullsized":"2024-09-21T15:51:43.581555+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:52:09.228964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031098399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"150'438","reported_seq":3065,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:44.793836+0000","last_change":"2024-09-21T15:51:25.689587+0000","last_active":"2024-09-21T15:51:44.793836+0000","last_peered":"2024-09-21T15:51:44.793836+0000","last_clean":"2024-09-21T15:51:44.793836+0000","last_became_active":"2024-09-21T15:50:32.671424+0000","last_became_peered":"2024-09-21T15:50:32.671424+0000","last_unstale":"2024-09-21T15:51:44.793836+0000","last_undegraded":"2024-09-21T15:51:44.793836+0000","last_fullsized":"2024-09-21T15:51:44.793836+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":438,"log_dups_size":0,"ondisk_log_size":438,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001507467,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":35,"num_read":2637,"num_read_kb":12956,"num_write":388,"num_write_kb":8392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":4494388,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":165,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.788455+0000","last_change":"2024-09-21T15:48:59.589790+0000","last_active":"2024-09-21T15:51:43.788455+0000","last_peered":"2024-09-21T15:51:43.788455+0000","last_clean":"2024-09-21T15:51:43.788455+0000","last_became_active":"2024-09-21T15:48:57.582121+0000","last_became_peered":"2024-09-21T15:48:57.582121+0000","last_unstale":"2024-09-21T15:51:43.788455+0000","last_undegraded":"2024-09-21T15:51:43.788455+0000","last_fullsized":"2024-09-21T15:51:43.788455+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:33:47.592624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039654700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"148'579","reported_seq":8001,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:44.475582+0000","last_change":"2024-09-21T15:51:35.754312+0000","last_active":"2024-09-21T15:51:44.475582+0000","last_peered":"2024-09-21T15:51:44.475582+0000","last_clean":"2024-09-21T15:51:44.475582+0000","last_became_active":"2024-09-21T15:51:35.752700+0000","last_became_peered":"2024-09-21T15:51:35.752700+0000","last_unstale":"2024-09-21T15:51:44.475582+0000","last_undegraded":"2024-09-21T15:51:44.475582+0000","last_fullsized":"2024-09-21T15:51:44.475582+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":579,"log_dups_size":0,"ondisk_log_size":579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:37:03.162105+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.00086244900000000005,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":35,"num_read":8380,"num_read_kb":21195,"num_write":544,"num_write_kb":5397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":256,"num_bytes_recovered":3760369,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":176,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:45.082527+0000","last_change":"2024-09-21T15:50:30.146743+0000","last_active":"2024-09-21T15:51:45.082527+0000","last_peered":"2024-09-21T15:51:45.082527+0000","last_clean":"2024-09-21T15:51:45.082527+0000","last_became_active":"2024-09-21T15:50:30.146428+0000","last_became_peered":"2024-09-21T15:50:30.146428+0000","last_unstale":"2024-09-21T15:51:45.082527+0000","last_undegraded":"2024-09-21T15:51:45.082527+0000","last_fullsized":"2024-09-21T15:51:45.082527+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:58:38.689807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00036074600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"145'703","reported_seq":8854,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.788507+0000","last_change":"2024-09-21T15:51:25.780818+0000","last_active":"2024-09-21T15:51:43.788507+0000","last_peered":"2024-09-21T15:51:43.788507+0000","last_clean":"2024-09-21T15:51:43.788507+0000","last_became_active":"2024-09-21T15:50:35.682788+0000","last_became_peered":"2024-09-21T15:50:35.682788+0000","last_unstale":"2024-09-21T15:51:43.788507+0000","last_undegraded":"2024-09-21T15:51:43.788507+0000","last_fullsized":"2024-09-21T15:51:43.788507+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":703,"log_dups_size":0,"ondisk_log_size":703,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:20:40.695690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00032907100000000001,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":26,"num_read":11015,"num_read_kb":27666,"num_write":868,"num_write_kb":11017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":175,"num_bytes_recovered":14134268,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":178,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.788532+0000","last_change":"2024-09-21T15:50:30.146654+0000","last_active":"2024-09-21T15:51:43.788532+0000","last_peered":"2024-09-21T15:51:43.788532+0000","last_clean":"2024-09-21T15:51:43.788532+0000","last_became_active":"2024-09-21T15:50:30.146273+0000","last_became_peered":"2024-09-21T15:50:30.146273+0000","last_unstale":"2024-09-21T15:51:43.788532+0000","last_undegraded":"2024-09-21T15:51:43.788532+0000","last_fullsized":"2024-09-21T15:51:43.788532+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:21:48.178519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00040525700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"150'603","reported_seq":5377,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.998616+0000","last_change":"2024-09-21T15:51:39.769574+0000","last_active":"2024-09-21T15:51:43.998616+0000","last_peered":"2024-09-21T15:51:43.998616+0000","last_clean":"2024-09-21T15:51:43.998616+0000","last_became_active":"2024-09-21T15:51:39.768300+0000","last_became_peered":"2024-09-21T15:51:39.768300+0000","last_unstale":"2024-09-21T15:51:43.998616+0000","last_undegraded":"2024-09-21T15:51:43.998616+0000","last_fullsized":"2024-09-21T15:51:43.998616+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":603,"log_dups_size":0,"ondisk_log_size":603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:33:47.511137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050855200000000001,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":35,"num_read":5730,"num_read_kb":33940,"num_write":686,"num_write_kb":16332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":268,"num_bytes_recovered":7823491,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"150'921","reported_seq":4196,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.789355+0000","last_change":"2024-09-21T15:51:25.690079+0000","last_active":"2024-09-21T15:51:43.789355+0000","last_peered":"2024-09-21T15:51:43.789355+0000","last_clean":"2024-09-21T15:51:43.789355+0000","last_became_active":"2024-09-21T15:50:32.677456+0000","last_became_peered":"2024-09-21T15:50:32.677456+0000","last_unstale":"2024-09-21T15:51:43.789355+0000","last_undegraded":"2024-09-21T15:51:43.789355+0000","last_fullsized":"2024-09-21T15:51:43.789355+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":921,"log_dups_size":0,"ondisk_log_size":921,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:46:31.194345+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010737730000000001,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":34,"num_read":5136,"num_read_kb":19340,"num_write":949,"num_write_kb":7405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":632176,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"20'2","reported_seq":60,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.998456+0000","last_change":"2024-09-21T15:51:39.764743+0000","last_active":"2024-09-21T15:51:43.998456+0000","last_peered":"2024-09-21T15:51:43.998456+0000","last_clean":"2024-09-21T15:51:43.998456+0000","last_became_active":"2024-09-21T15:51:39.764409+0000","last_became_peered":"2024-09-21T15:51:39.764409+0000","last_unstale":"2024-09-21T15:51:43.998456+0000","last_undegraded":"2024-09-21T15:51:43.998456+0000","last_fullsized":"2024-09-21T15:51:43.998456+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:28:14.932986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000859595,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"148'393","reported_seq":4834,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.788546+0000","last_change":"2024-09-21T15:51:25.781082+0000","last_active":"2024-09-21T15:51:43.788546+0000","last_peered":"2024-09-21T15:51:43.788546+0000","last_clean":"2024-09-21T15:51:43.788546+0000","last_became_active":"2024-09-21T15:50:35.682001+0000","last_became_peered":"2024-09-21T15:50:35.682001+0000","last_unstale":"2024-09-21T15:51:43.788546+0000","last_undegraded":"2024-09-21T15:51:43.788546+0000","last_fullsized":"2024-09-21T15:51:43.788546+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":393,"log_dups_size":0,"ondisk_log_size":393,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:09:15.325457+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044281000000000002,"stat_sum":{"num_bytes":0,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":31,"num_read":4285,"num_read_kb":23375,"num_write":379,"num_write_kb":4145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":187,"num_bytes_recovered":4305276,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":97,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.788209+0000","last_change":"2024-09-21T15:51:07.683738+0000","last_active":"2024-09-21T15:51:43.788209+0000","last_peered":"2024-09-21T15:51:43.788209+0000","last_clean":"2024-09-21T15:51:43.788209+0000","last_became_active":"2024-09-21T15:51:07.683418+0000","last_became_peered":"2024-09-21T15:51:07.683418+0000","last_unstale":"2024-09-21T15:51:43.788209+0000","last_undegraded":"2024-09-21T15:51:43.788209+0000","last_fullsized":"2024-09-21T15:51:43.788209+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:25:13.468653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"150'558","reported_seq":2599,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:44.793691+0000","last_change":"2024-09-21T15:51:25.689707+0000","last_active":"2024-09-21T15:51:44.793691+0000","last_peered":"2024-09-21T15:51:44.793691+0000","last_clean":"2024-09-21T15:51:44.793691+0000","last_became_active":"2024-09-21T15:50:32.671413+0000","last_became_peered":"2024-09-21T15:50:32.671413+0000","last_unstale":"2024-09-21T15:51:44.793691+0000","last_undegraded":"2024-09-21T15:51:44.793691+0000","last_fullsized":"2024-09-21T15:51:44.793691+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":558,"log_dups_size":0,"ondisk_log_size":558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001292872,"stat_sum":{"num_bytes":0,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":39,"num_read":3663,"num_read_kb":16543,"num_write":590,"num_write_kb":5977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":1696547,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":163,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:45.082382+0000","last_change":"2024-09-21T15:49:00.114730+0000","last_active":"2024-09-21T15:51:45.082382+0000","last_peered":"2024-09-21T15:51:45.082382+0000","last_clean":"2024-09-21T15:51:45.082382+0000","last_became_active":"2024-09-21T15:48:57.585525+0000","last_became_peered":"2024-09-21T15:48:57.585525+0000","last_unstale":"2024-09-21T15:51:45.082382+0000","last_undegraded":"2024-09-21T15:51:45.082382+0000","last_fullsized":"2024-09-21T15:51:45.082382+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:35:35.786526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00031691100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"150'413","reported_seq":2957,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:44.475526+0000","last_change":"2024-09-21T15:51:35.754599+0000","last_active":"2024-09-21T15:51:44.475526+0000","last_peered":"2024-09-21T15:51:44.475526+0000","last_clean":"2024-09-21T15:51:44.475526+0000","last_became_active":"2024-09-21T15:51:35.752810+0000","last_became_peered":"2024-09-21T15:51:35.752810+0000","last_unstale":"2024-09-21T15:51:44.475526+0000","last_undegraded":"2024-09-21T15:51:44.475526+0000","last_fullsized":"2024-09-21T15:51:44.475526+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":413,"log_dups_size":0,"ondisk_log_size":413,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:54:49.851651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00033178999999999999,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":30,"num_read":2149,"num_read_kb":34452,"num_write":406,"num_write_kb":12612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":214,"num_bytes_recovered":2197196,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"18'1","reported_seq":59,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.998478+0000","last_change":"2024-09-21T15:51:39.764710+0000","last_active":"2024-09-21T15:51:43.998478+0000","last_peered":"2024-09-21T15:51:43.998478+0000","last_clean":"2024-09-21T15:51:43.998478+0000","last_became_active":"2024-09-21T15:51:39.764411+0000","last_became_peered":"2024-09-21T15:51:39.764411+0000","last_unstale":"2024-09-21T15:51:43.998478+0000","last_undegraded":"2024-09-21T15:51:43.998478+0000","last_fullsized":"2024-09-21T15:51:43.998478+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:29:50.093645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039559000000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"16'117","reported_seq":324,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:45.082405+0000","last_change":"2024-09-21T15:50:30.147280+0000","last_active":"2024-09-21T15:51:45.082405+0000","last_peered":"2024-09-21T15:51:45.082405+0000","last_clean":"2024-09-21T15:51:45.082405+0000","last_became_active":"2024-09-21T15:50:30.146987+0000","last_became_peered":"2024-09-21T15:50:30.146987+0000","last_unstale":"2024-09-21T15:51:45.082405+0000","last_undegraded":"2024-09-21T15:51:45.082405+0000","last_fullsized":"2024-09-21T15:51:45.082405+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:50.161884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:50.161884+0000","last_clean_scrub_stamp":"2024-09-21T15:48:50.161884+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:26:32.853650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"150'540","reported_seq":7944,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:44.475450+0000","last_change":"2024-09-21T15:51:25.689387+0000","last_active":"2024-09-21T15:51:44.475450+0000","last_peered":"2024-09-21T15:51:44.475450+0000","last_clean":"2024-09-21T15:51:44.475450+0000","last_became_active":"2024-09-21T15:51:11.837877+0000","last_became_peered":"2024-09-21T15:51:11.837877+0000","last_unstale":"2024-09-21T15:51:44.475450+0000","last_undegraded":"2024-09-21T15:51:44.475450+0000","last_fullsized":"2024-09-21T15:51:44.475450+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":540,"log_dups_size":0,"ondisk_log_size":540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:45:58.418197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00079839600000000004,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":32,"num_read":7239,"num_read_kb":29580,"num_write":558,"num_write_kb":4793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":418,"num_bytes_recovered":6262707,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"150'389","reported_seq":3171,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.789196+0000","last_change":"2024-09-21T15:51:25.689626+0000","last_active":"2024-09-21T15:51:43.789196+0000","last_peered":"2024-09-21T15:51:43.789196+0000","last_clean":"2024-09-21T15:51:43.789196+0000","last_became_active":"2024-09-21T15:49:07.210506+0000","last_became_peered":"2024-09-21T15:49:07.210506+0000","last_unstale":"2024-09-21T15:51:43.789196+0000","last_undegraded":"2024-09-21T15:51:43.789196+0000","last_fullsized":"2024-09-21T15:51:43.789196+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":389,"log_dups_size":0,"ondisk_log_size":389,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:55:18.046738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00089193900000000001,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":30,"num_read":2716,"num_read_kb":28245,"num_write":454,"num_write_kb":12090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"149'595","reported_seq":12267,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581556+0000","last_change":"2024-09-21T15:51:41.782922+0000","last_active":"2024-09-21T15:51:43.581556+0000","last_peered":"2024-09-21T15:51:43.581556+0000","last_clean":"2024-09-21T15:51:43.581556+0000","last_became_active":"2024-09-21T15:51:41.781781+0000","last_became_peered":"2024-09-21T15:51:41.781781+0000","last_unstale":"2024-09-21T15:51:43.581556+0000","last_undegraded":"2024-09-21T15:51:43.581556+0000","last_fullsized":"2024-09-21T15:51:43.581556+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":595,"log_dups_size":0,"ondisk_log_size":595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:10:04.846054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000309432,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":29,"num_read":12828,"num_read_kb":63467,"num_write":580,"num_write_kb":14212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"147'851","reported_seq":3855,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:44.475479+0000","last_change":"2024-09-21T15:51:35.752901+0000","last_active":"2024-09-21T15:51:44.475479+0000","last_peered":"2024-09-21T15:51:44.475479+0000","last_clean":"2024-09-21T15:51:44.475479+0000","last_became_active":"2024-09-21T15:51:35.751150+0000","last_became_peered":"2024-09-21T15:51:35.751150+0000","last_unstale":"2024-09-21T15:51:44.475479+0000","last_undegraded":"2024-09-21T15:51:44.475479+0000","last_fullsized":"2024-09-21T15:51:44.475479+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":851,"log_dups_size":0,"ondisk_log_size":851,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:33:02.873060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001158526,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":26,"num_read":5948,"num_read_kb":20169,"num_write":1244,"num_write_kb":7750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":262,"num_bytes_recovered":8353711,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"150'739","reported_seq":6769,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.998571+0000","last_change":"2024-09-21T15:51:39.769469+0000","last_active":"2024-09-21T15:51:43.998571+0000","last_peered":"2024-09-21T15:51:43.998571+0000","last_clean":"2024-09-21T15:51:43.998571+0000","last_became_active":"2024-09-21T15:51:39.768321+0000","last_became_peered":"2024-09-21T15:51:39.768321+0000","last_unstale":"2024-09-21T15:51:43.998571+0000","last_undegraded":"2024-09-21T15:51:43.998571+0000","last_fullsized":"2024-09-21T15:51:43.998571+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":739,"log_dups_size":0,"ondisk_log_size":739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:21:29.566035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074635999999999995,"stat_sum":{"num_bytes":0,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":36,"num_read":8849,"num_read_kb":25678,"num_write":868,"num_write_kb":5005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":246,"num_bytes_recovered":5686754,"num_keys_recovered":23,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"147'894","reported_seq":4315,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:45.082526+0000","last_change":"2024-09-21T15:51:25.698246+0000","last_active":"2024-09-21T15:51:45.082526+0000","last_peered":"2024-09-21T15:51:45.082526+0000","last_clean":"2024-09-21T15:51:45.082526+0000","last_became_active":"2024-09-21T15:49:06.689203+0000","last_became_peered":"2024-09-21T15:49:06.689203+0000","last_unstale":"2024-09-21T15:51:45.082526+0000","last_undegraded":"2024-09-21T15:51:45.082526+0000","last_fullsized":"2024-09-21T15:51:45.082526+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":894,"log_dups_size":0,"ondisk_log_size":894,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:22:17.578359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084935700000000002,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":35,"num_read":7321,"num_read_kb":28131,"num_write":1146,"num_write_kb":14809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"148'568","reported_seq":8145,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:45.082509+0000","last_change":"2024-09-21T15:51:25.697735+0000","last_active":"2024-09-21T15:51:45.082509+0000","last_peered":"2024-09-21T15:51:45.082509+0000","last_clean":"2024-09-21T15:51:45.082509+0000","last_became_active":"2024-09-21T15:49:06.690101+0000","last_became_peered":"2024-09-21T15:49:06.690101+0000","last_unstale":"2024-09-21T15:51:45.082509+0000","last_undegraded":"2024-09-21T15:51:45.082509+0000","last_fullsized":"2024-09-21T15:51:45.082509+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":568,"log_dups_size":0,"ondisk_log_size":568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:43:30.659104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044571200000000002,"stat_sum":{"num_bytes":0,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":29,"num_read":8735,"num_read_kb":42983,"num_write":509,"num_write_kb":16273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"145'698","reported_seq":3800,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:45.082582+0000","last_change":"2024-09-21T15:51:25.698380+0000","last_active":"2024-09-21T15:51:45.082582+0000","last_peered":"2024-09-21T15:51:45.082582+0000","last_clean":"2024-09-21T15:51:45.082582+0000","last_became_active":"2024-09-21T15:49:06.694250+0000","last_became_peered":"2024-09-21T15:49:06.694250+0000","last_unstale":"2024-09-21T15:51:45.082582+0000","last_undegraded":"2024-09-21T15:51:45.082582+0000","last_fullsized":"2024-09-21T15:51:45.082582+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":698,"log_dups_size":0,"ondisk_log_size":698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:21:37.777769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086632899999999999,"stat_sum":{"num_bytes":0,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":42,"num_read":4945,"num_read_kb":15920,"num_write":889,"num_write_kb":5075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"138'470","reported_seq":3154,"reported_epoch":166,"state":"active+clean","last_fresh":"2024-09-21T15:51:43.581542+0000","last_change":"2024-09-21T15:51:25.780857+0000","last_active":"2024-09-21T15:51:43.581542+0000","last_peered":"2024-09-21T15:51:43.581542+0000","last_clean":"2024-09-21T15:51:43.581542+0000","last_became_active":"2024-09-21T15:50:32.676147+0000","last_became_peered":"2024-09-21T15:50:32.676147+0000","last_unstale":"2024-09-21T15:51:43.581542+0000","last_undegraded":"2024-09-21T15:51:43.581542+0000","last_fullsized":"2024-09-21T15:51:43.581542+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":470,"log_dups_size":0,"ondisk_log_size":470,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:44:18.821310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00024304599999999999,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":40,"num_read":3160,"num_read_kb":11849,"num_write":381,"num_write_kb":4471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":1191778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"148'1012","reported_seq":4198,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:45.082390+0000","last_change":"2024-09-21T15:51:25.697763+0000","last_active":"2024-09-21T15:51:45.082390+0000","last_peered":"2024-09-21T15:51:45.082390+0000","last_clean":"2024-09-21T15:51:45.082390+0000","last_became_active":"2024-09-21T15:49:06.688798+0000","last_became_peered":"2024-09-21T15:49:06.688798+0000","last_unstale":"2024-09-21T15:51:45.082390+0000","last_undegraded":"2024-09-21T15:51:45.082390+0000","last_fullsized":"2024-09-21T15:51:45.082390+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":1012,"log_dups_size":0,"ondisk_log_size":1012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:48:33.679133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053058599999999999,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":30,"num_read":6391,"num_read_kb":16970,"num_write":1355,"num_write_kb":8408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"168'417","reported_seq":3059,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:51.378061+0000","last_change":"2024-09-21T15:51:51.378167+0000","last_active":"2024-09-21T15:51:51.378061+0000","last_peered":"2024-09-21T15:51:51.378061+0000","last_clean":"2024-09-21T15:51:51.378061+0000","last_became_active":"2024-09-21T15:50:32.671686+0000","last_became_peered":"2024-09-21T15:50:32.671686+0000","last_unstale":"2024-09-21T15:51:51.378061+0000","last_undegraded":"2024-09-21T15:51:51.378061+0000","last_fullsized":"2024-09-21T15:51:51.378061+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'417","last_scrub_stamp":"2024-09-21T15:51:51.378041+0000","last_deep_scrub":"168'417","last_deep_scrub_stamp":"2024-09-21T15:51:51.378041+0000","last_clean_scrub_stamp":"2024-09-21T15:51:51.378041+0000","objects_scrubbed":34,"log_size":417,"log_dups_size":0,"ondisk_log_size":417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:40:52.256253+0000","scrub_duration":0.023972753999999999,"objects_trimmed":0,"snaptrim_duration":0.0012972649999999999,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":34,"num_read":2455,"num_read_kb":11669,"num_write":401,"num_write_kb":13839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":3359702,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"150'967","reported_seq":7432,"reported_epoch":167,"state":"active+clean","last_fresh":"2024-09-21T15:51:45.082397+0000","last_change":"2024-09-21T15:51:25.699098+0000","last_active":"2024-09-21T15:51:45.082397+0000","last_peered":"2024-09-21T15:51:45.082397+0000","last_clean":"2024-09-21T15:51:45.082397+0000","last_became_active":"2024-09-21T15:49:06.690128+0000","last_became_peered":"2024-09-21T15:49:06.690128+0000","last_unstale":"2024-09-21T15:51:45.082397+0000","last_undegraded":"2024-09-21T15:51:45.082397+0000","last_fullsized":"2024-09-21T15:51:45.082397+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":967,"log_dups_size":0,"ondisk_log_size":967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:02:11.763959+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014652879999999999,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":32,"num_read":9327,"num_read_kb":26357,"num_write":1446,"num_write_kb":13580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":1039,"num_object_clones":0,"num_object_copies":2078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1039,"num_whiteouts":1035,"num_read":207419,"num_read_kb":842423,"num_write":23389,"num_write_kb":303239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3397,"num_bytes_recovered":100270055,"num_keys_recovered":316,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5973,"internal_metadata":0},"log_size":20252,"ondisk_log_size":20252,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574888,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77960,"kb_used_data":2556,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94293880,"statfs":{"total":96636764160,"available":96556933120,"internally_reserved":0,"allocated":2617344,"data_stored":1563409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":77199819},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:32 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.53400000000000003}]},{"osd":1,"last update":"Sat Sep 21 15:50:52 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.48299999999999998,"15min":0.495},"min":{"1min":0.188,"5min":0.188,"15min":0.188},"max":{"1min":0.73199999999999998,"5min":1.04,"15min":1.04},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.44,"5min":0.46999999999999997,"15min":0.47499999999999998},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.64500000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"last":0.44600000000000001}]},{"osd":2,"last update":"Sat Sep 21 15:50:52 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.439,"15min":0.44400000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.53500000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.437,"5min":0.498,"15min":0.50800000000000001},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.70099999999999996,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.51300000000000001}]},{"osd":3,"last update":"Sat Sep 21 15:50:52 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.49099999999999999,"15min":0.499},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.65300000000000002,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.432,"5min":0.47099999999999997,"15min":0.47699999999999998},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.68200000000000005,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.47999999999999998}]},{"osd":4,"last update":"Sat Sep 21 15:50:52 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.41699999999999998,"15min":0.42299999999999999},"min":{"1min":0.24099999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.59999999999999998,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.501},{"interface":"front","average":{"1min":0.38,"5min":0.39000000000000001,"15min":0.39200000000000002},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.54600000000000004}]},{"osd":5,"last update":"Sat Sep 21 15:50:52 2024","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.439,"15min":0.45000000000000001},"min":{"1min":0.25900000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.52900000000000003,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.44700000000000001,"15min":0.45000000000000001},"min":{"1min":0.29499999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.68600000000000005,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.58299999999999996}]},{"osd":6,"last update":"Sat Sep 21 15:50:52 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.49399999999999999,"15min":0.5},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.72099999999999997,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.74199999999999999,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.57199999999999995}]}]},{"osd":6,"up_from":13,"seq":55834574891,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41284,"kb_used_data":1976,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94330556,"statfs":{"total":96636764160,"available":96594489344,"internally_reserved":0,"allocated":2023424,"data_stored":973041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:31 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.53500000000000003}]},{"osd":1,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.44800000000000001,"15min":0.45800000000000002},"min":{"1min":0.26900000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.53300000000000003,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.46800000000000003,"15min":0.47999999999999998},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.53300000000000003,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.374}]},{"osd":2,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.47099999999999997,"15min":0.47699999999999998},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.45800000000000002,"15min":0.46200000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.60999999999999999,"5min":0.66000000000000003,"15min":0.66000000000000003},"last":0.42899999999999999}]},{"osd":3,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.51400000000000001,"15min":0.52400000000000002},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.70399999999999996,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.503},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.53100000000000003,"15min":0.53800000000000003},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.73699999999999999,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.56799999999999995}]},{"osd":4,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.40600000000000003,"15min":0.41399999999999998},"min":{"1min":0.23200000000000001,"5min":0.222,"15min":0.222},"max":{"1min":0.57499999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.379,"5min":0.46700000000000003,"15min":0.48199999999999998},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.76300000000000001,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.57699999999999996}]},{"osd":5,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.44600000000000001,"15min":0.45600000000000002},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.57399999999999995,"5min":0.755,"15min":0.755},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.372,"5min":0.45000000000000001,"15min":0.46300000000000002},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.628,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.41699999999999998}]},{"osd":7,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.45000000000000001,"15min":0.45700000000000002},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.55400000000000005,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.497,"15min":0.51100000000000001},"min":{"1min":0.30499999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.748,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.45400000000000001}]}]},{"osd":5,"up_from":13,"seq":55834574890,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55240,"kb_used_data":1980,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94316600,"statfs":{"total":96636764160,"available":96580198400,"internally_reserved":0,"allocated":2027520,"data_stored":973060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.57899999999999996}]},{"osd":1,"last update":"Sat Sep 21 15:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.71699999999999997,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.5,"15min":0.51400000000000001},"min":{"1min":0.34399999999999997,"5min":0.314,"15min":0.314},"max":{"1min":0.52200000000000002,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.439}]},{"osd":2,"last update":"Sat Sep 21 15:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.49399999999999999,"15min":0.50600000000000001},"min":{"1min":0.34200000000000003,"5min":0.311,"15min":0.311},"max":{"1min":0.56899999999999995,"5min":1.046,"15min":1.046},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.437,"5min":0.46800000000000003,"15min":0.47299999999999998},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.68999999999999995,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.67900000000000005}]},{"osd":3,"last update":"Sat Sep 21 15:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.47699999999999998,"15min":0.48699999999999999},"min":{"1min":0.29599999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.55200000000000005,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.53800000000000003,"15min":0.55200000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.76900000000000002,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.56100000000000005}]},{"osd":4,"last update":"Sat Sep 21 15:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.65400000000000003,"15min":0.70799999999999996},"min":{"1min":0.254,"5min":0.253,"15min":0.253},"max":{"1min":0.503,"5min":5.6399999999999997,"15min":5.6399999999999997},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.45000000000000001,"15min":0.47199999999999998},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.48899999999999999,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.62}]},{"osd":6,"last update":"Sat Sep 21 15:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.443,"15min":0.45500000000000002},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.51700000000000002,"5min":0.58199999999999996,"15min":0.58199999999999996},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.48199999999999998,"15min":0.499},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.55200000000000005,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.54000000000000004}]},{"osd":7,"last update":"Sat Sep 21 15:50:55 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.47199999999999998,"15min":0.48599999999999999},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.56000000000000005,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.504,"15min":0.52100000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.64700000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574889,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":65100,"kb_used_data":1976,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94306740,"statfs":{"total":96636764160,"available":96570101760,"internally_reserved":0,"allocated":2023424,"data_stored":971778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":64616902},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.48599999999999999}]},{"osd":1,"last update":"Sat Sep 21 15:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.433,"15min":0.41899999999999998},"min":{"1min":0.217,"5min":0.217,"15min":0.217},"max":{"1min":1.8839999999999999,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.33600000000000002,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.52100000000000002}]},{"osd":2,"last update":"Sat Sep 21 15:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.443,"15min":0.439},"min":{"1min":0.311,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.433,"15min":0.433},"min":{"1min":0.31,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.54000000000000004,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.437}]},{"osd":3,"last update":"Sat Sep 21 15:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"min":{"1min":0.34200000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.66600000000000004,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.46800000000000003,"15min":0.46999999999999997},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.60899999999999999,"5min":0.626,"15min":0.626},"last":0.499}]},{"osd":5,"last update":"Sat Sep 21 15:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.47199999999999998,"15min":0.47599999999999998},"min":{"1min":0.20899999999999999,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":1.919,"5min":2.4489999999999998,"15min":2.4489999999999998},"last":0.373},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.42399999999999999,"15min":0.42499999999999999},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.748,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.46400000000000002}]},{"osd":6,"last update":"Sat Sep 21 15:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.41599999999999998,"15min":0.41399999999999998},"min":{"1min":0.29399999999999998,"5min":0.245,"15min":0.245},"max":{"1min":0.63800000000000001,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.63300000000000001,"5min":0.55700000000000005,"15min":0.54500000000000004},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":4.1340000000000003,"5min":4.1340000000000003,"15min":4.1340000000000003},"last":0.53300000000000003}]},{"osd":7,"last update":"Sat Sep 21 15:50:56 2024","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.45400000000000001,"15min":0.434},"min":{"1min":0.28199999999999997,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":2.7599999999999998,"5min":2.7599999999999998,"15min":2.7599999999999998},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.52400000000000002,"15min":0.52200000000000002},"min":{"1min":0.32600000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":2.016,"5min":2.016,"15min":2.016},"last":0.58499999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574890,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59340,"kb_used_data":1976,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312500,"statfs":{"total":96636764160,"available":96576000000,"internally_reserved":0,"allocated":2023424,"data_stored":973041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":58718664},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"last":0.315},{"interface":"front","average":{"1min":0.31,"5min":0.31,"15min":0.31},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"last":0.29599999999999999}]},{"osd":1,"last update":"Sat Sep 21 15:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.28999999999999998,"5min":0.33600000000000002,"15min":0.34399999999999997},"min":{"1min":0.219,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.42299999999999999,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.30499999999999999,"5min":0.36799999999999999,"15min":0.378},"min":{"1min":0.192,"5min":0.192,"15min":0.192},"max":{"1min":0.40500000000000003,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.35199999999999998}]},{"osd":2,"last update":"Sat Sep 21 15:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.38400000000000001,"15min":0.39300000000000002},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.49199999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.40999999999999998,"15min":0.42099999999999999},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.52000000000000002,"5min":0.755,"15min":0.755},"last":0.37}]},{"osd":4,"last update":"Sat Sep 21 15:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.47899999999999998,"15min":0.49199999999999999},"min":{"1min":0.23499999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.56699999999999995,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.63900000000000001},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.504,"15min":0.51300000000000001},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.60399999999999998,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.58299999999999996}]},{"osd":5,"last update":"Sat Sep 21 15:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.51000000000000001,"15min":0.52500000000000002},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.68100000000000005,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.41899999999999998,"5min":0.53400000000000003,"15min":0.55300000000000005},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.54100000000000004,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.51900000000000002}]},{"osd":6,"last update":"Sat Sep 21 15:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.52800000000000002,"15min":0.54200000000000004},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.622,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.53000000000000003,"15min":0.54600000000000004},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.65700000000000003,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.53100000000000003}]},{"osd":7,"last update":"Sat Sep 21 15:50:50 2024","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.54700000000000004,"15min":0.56200000000000006},"min":{"1min":0.36299999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.60599999999999998,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.54600000000000004,"15min":0.56200000000000006},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.65000000000000002,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.61099999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574890,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37188,"kb_used_data":1976,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334652,"statfs":{"total":96636764160,"available":96598683648,"internally_reserved":0,"allocated":2023424,"data_stored":973041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:29 2024","interfaces":[{"interface":"back","average":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.439,"5min":0.439,"15min":0.439},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.29399999999999998}]},{"osd":1,"last update":"Sat Sep 21 15:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.30299999999999999,"5min":0.40300000000000002,"15min":0.41999999999999998},"min":{"1min":0.17199999999999999,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.439,"5min":1.621,"15min":1.621},"last":0.31},{"interface":"front","average":{"1min":0.308,"5min":0.36399999999999999,"15min":0.374},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.46400000000000002,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.27200000000000002}]},{"osd":3,"last update":"Sat Sep 21 15:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.90800000000000003,"15min":0.97099999999999997},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":4.6580000000000004,"5min":11.56,"15min":11.56},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.40200000000000002,"15min":0.41299999999999998},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.46999999999999997,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.32100000000000001}]},{"osd":4,"last update":"Sat Sep 21 15:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.57999999999999996,"15min":0.60199999999999998},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.60499999999999998,"5min":1.653,"15min":1.653},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.60399999999999998,"15min":0.628},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.74199999999999999,"5min":2.4660000000000002,"15min":2.4660000000000002},"last":0.42999999999999999}]},{"osd":5,"last update":"Sat Sep 21 15:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.501,"15min":0.51100000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.61199999999999999,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.52700000000000002,"15min":0.53900000000000003},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.65400000000000003,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.36799999999999999}]},{"osd":6,"last update":"Sat Sep 21 15:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.54900000000000004,"15min":0.56299999999999994},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.67500000000000004,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.54000000000000004,"15min":0.55600000000000005},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.629,"5min":0.873,"15min":0.873},"last":0.35099999999999998}]},{"osd":7,"last update":"Sat Sep 21 15:50:54 2024","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.54900000000000004,"15min":0.56100000000000005},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.67000000000000004,"5min":0.876,"15min":0.876},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.58199999999999996,"15min":0.59799999999999998},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.754,"5min":1.089,"15min":1.089},"last":0.38200000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574889,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41292,"kb_used_data":1984,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94330548,"statfs":{"total":96636764160,"available":96594481152,"internally_reserved":0,"allocated":2031616,"data_stored":973079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.312,"15min":0.312},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"last":0.247},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.27100000000000002}]},{"osd":2,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.27100000000000002,"5min":0.33000000000000002,"15min":0.34000000000000002},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.43099999999999999,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.30299999999999999,"5min":0.499,"15min":0.53200000000000003},"min":{"1min":0.21299999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.52400000000000002,"5min":5.0670000000000002,"15min":5.0670000000000002},"last":0.32400000000000001}]},{"osd":3,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.91700000000000004,"15min":1.0189999999999999},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.46000000000000002,"5min":16.032,"15min":16.032},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.309,"5min":0.34999999999999998,"15min":0.35699999999999998},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":0.47899999999999998,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.309}]},{"osd":4,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.46800000000000003,"15min":0.47399999999999998},"min":{"1min":0.25900000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.63200000000000001,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.53600000000000003,"15min":0.55100000000000005},"min":{"1min":0.311,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.56799999999999995,"5min":2.2269999999999999,"15min":2.2269999999999999},"last":0.58599999999999997}]},{"osd":5,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.55300000000000005,"15min":0.57299999999999995},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.64700000000000002,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.66700000000000004},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.66000000000000003,"15min":0.69599999999999995},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.66200000000000003,"5min":4.6989999999999998,"15min":4.6989999999999998},"last":0.56499999999999995}]},{"osd":6,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.53000000000000003,"15min":0.53900000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.68200000000000005,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.59999999999999998,"15min":0.62},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.64500000000000002,"5min":2.262,"15min":2.262},"last":0.63900000000000001}]},{"osd":7,"last update":"Sat Sep 21 15:50:49 2024","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.67000000000000004,"15min":0.70399999999999996},"min":{"1min":0.33400000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.69399999999999995,"5min":4.742,"15min":4.742},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.51300000000000001,"15min":0.51800000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.70199999999999996,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.57199999999999995}]}]},{"osd":0,"up_from":93,"seq":399431958547,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39172,"kb_used_data":2556,"kb_used_omap":13,"kb_used_meta":36594,"kb_avail":94332668,"statfs":{"total":96636764160,"available":96596652032,"internally_reserved":0,"allocated":2617344,"data_stored":1563409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13968,"internal_metadata":37472624},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.317,"5min":0.317,"15min":0.317},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"last":0.24199999999999999}]},{"osd":2,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.309},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.29699999999999999}]},{"osd":3,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.505,"5min":0.505,"15min":0.505},"last":0.38700000000000001}]},{"osd":4,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.52000000000000002}]},{"osd":5,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.51100000000000001}]},{"osd":6,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.432},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.57399999999999995}]},{"osd":7,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.56200000000000006}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5973,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T15:51:54.567 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2024-09-21T15:49:05.663715+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=49, tm_sec=5, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=51, tm_sec=49, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:51:54.567 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2024-09-21T15:49:05.663715+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=49, tm_sec=5, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=51, tm_sec=49, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:51:54.567 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2024-09-21T15:49:05.663715+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=49, tm_sec=5, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=51, tm_sec=49, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:51:54.567 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2024-09-21T15:49:05.663715+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=49, tm_sec=5, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=51, tm_sec=49, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:51:54.567 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2024-09-21T15:49:05.663715+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=49, tm_sec=5, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=51, tm_sec=49, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:51:54.568 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2024-09-21T15:49:05.663715+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=49, tm_sec=5, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=51, tm_sec=49, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:51:54.568 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2024-09-21T15:49:05.663715+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=49, tm_sec=5, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=51, tm_sec=49, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:51:54.568 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2024-09-21T15:49:05.663715+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=49, tm_sec=5, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=51, tm_sec=49, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:51:54.568 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2024-09-21T15:49:05.663715+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=49, tm_sec=5, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=51, tm_sec=49, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:51:54.568 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2024-09-21T15:49:05.663715+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=49, tm_sec=5, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=51, tm_sec=49, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:51:54.568 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2024-09-21T15:49:05.663715+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=49, tm_sec=5, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=51, tm_sec=49, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:51:54.568 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2024-09-21T15:49:05.663715+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=49, tm_sec=5, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=51, tm_sec=49, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:51:54.568 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-21T15:48:56.568162+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=48, tm_sec=56, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=51, tm_sec=49, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:51:54.568 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2024-09-21T15:49:05.663715+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=49, tm_sec=5, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=51, tm_sec=49, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:51:54.568 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-21T15:48:56.568162+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=48, tm_sec=56, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=51, tm_sec=49, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:51:54.569 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2024-09-21T15:49:05.663715+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=49, tm_sec=5, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=51, tm_sec=49, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:51:54.569 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-21T15:48:56.568162+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=48, tm_sec=56, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=51, tm_sec=49, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:51:54.569 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-09-21T15:49:05.663715+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=49, tm_sec=5, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=51, tm_sec=49, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:51:54.569 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-21T15:48:56.568162+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=48, tm_sec=56, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=51, tm_sec=49, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:51:54.569 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2024-09-21T15:49:05.663715+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=49, tm_sec=5, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=51, tm_sec=49, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:51:54.569 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-09-21T15:49:05.663715+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=49, tm_sec=5, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=51, tm_sec=49, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:51:54.569 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-21T15:48:56.568162+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=48, tm_sec=56, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=51, tm_sec=49, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:51:54.569 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-09-21T15:49:05.663715+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=49, tm_sec=5, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=51, tm_sec=49, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:51:54.569 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-21T15:48:56.568162+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=48, tm_sec=56, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=51, tm_sec=49, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:51:54.570 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-09-21T15:49:05.663715+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=49, tm_sec=5, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=51, tm_sec=49, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:51:54.570 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-21T15:48:56.568162+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=48, tm_sec=56, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=51, tm_sec=49, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:51:54.570 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-09-21T15:49:05.663715+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=49, tm_sec=5, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=51, tm_sec=49, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:51:54.570 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-21T15:48:56.568162+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=48, tm_sec=56, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=51, tm_sec=49, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:51:54.570 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-21T15:48:50.161884+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=48, tm_sec=50, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=51, tm_sec=49, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:51:54.570 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-09-21T15:49:05.663715+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=49, tm_sec=5, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=51, tm_sec=49, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:51:54.570 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2024-09-21T15:49:05.663715+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=49, tm_sec=5, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=51, tm_sec=49, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:51:54.570 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2024-09-21T15:49:05.663715+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=49, tm_sec=5, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=51, tm_sec=49, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:51:54.570 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2024-09-21T15:49:05.663715+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=49, tm_sec=5, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=51, tm_sec=49, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:51:54.571 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2024-09-21T15:49:05.663715+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=49, tm_sec=5, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=51, tm_sec=49, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:51:54.571 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2024-09-21T15:49:05.663715+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=49, tm_sec=5, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=51, tm_sec=49, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:51:54.571 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2024-09-21T15:49:05.663715+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=49, tm_sec=5, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=51, tm_sec=49, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:51:54.571 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2024-09-21T15:49:05.663715+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=49, tm_sec=5, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=51, tm_sec=49, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:51:54.571 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2024-09-21T15:49:05.663715+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=49, tm_sec=5, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=51, tm_sec=49, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:51:54.571 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2024-09-21T15:49:05.663715+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=49, tm_sec=5, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=51, tm_sec=49, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:51:54.571 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2024-09-21T15:49:05.663715+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=49, tm_sec=5, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=51, tm_sec=49, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:51:54.571 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-21T15:52:14.572 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T15:52:14.860 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:52:14.860 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-09-21T15:52:14.871 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":263,"stamp":"2024-09-21T15:52:13.923954+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590406,"num_objects":1043,"num_object_clones":0,"num_object_copies":2086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1043,"num_whiteouts":1035,"num_read":207535,"num_read_kb":842523,"num_write":23550,"num_write_kb":305657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3397,"num_bytes_recovered":100270055,"num_keys_recovered":316,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21085,"ondisk_log_size":21085,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":420628,"kb_used_data":16980,"kb_used_omap":24,"kb_used_meta":403559,"kb_avail":754554092,"statfs":{"total":773094113280,"available":772663390208,"internally_reserved":0,"allocated":17387520,"data_stored":8963858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25118,"internal_metadata":413244898},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002759"},"pg_stats":[{"pgid":"3.1f","version":"168'441","reported_seq":2174,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:59.256550+0000","last_change":"2024-09-21T15:51:59.256650+0000","last_active":"2024-09-21T15:51:59.256550+0000","last_peered":"2024-09-21T15:51:59.256550+0000","last_clean":"2024-09-21T15:51:59.256550+0000","last_became_active":"2024-09-21T15:50:32.670964+0000","last_became_peered":"2024-09-21T15:50:32.670964+0000","last_unstale":"2024-09-21T15:51:59.256550+0000","last_undegraded":"2024-09-21T15:51:59.256550+0000","last_fullsized":"2024-09-21T15:51:59.256550+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'441","last_scrub_stamp":"2024-09-21T15:51:59.256529+0000","last_deep_scrub":"168'441","last_deep_scrub_stamp":"2024-09-21T15:51:59.256529+0000","last_clean_scrub_stamp":"2024-09-21T15:51:59.256529+0000","objects_scrubbed":28,"log_size":441,"log_dups_size":0,"ondisk_log_size":441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:45:03.562826+0000","scrub_duration":0.022020432999999999,"objects_trimmed":0,"snaptrim_duration":0.0017896030000000001,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":28,"num_read":2413,"num_read_kb":10419,"num_write":442,"num_write_kb":5956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":1974433,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"168'637","reported_seq":6578,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:54.369203+0000","last_change":"2024-09-21T15:51:54.369307+0000","last_active":"2024-09-21T15:51:54.369203+0000","last_peered":"2024-09-21T15:51:54.369203+0000","last_clean":"2024-09-21T15:51:54.369203+0000","last_became_active":"2024-09-21T15:51:24.691676+0000","last_became_peered":"2024-09-21T15:51:24.691676+0000","last_unstale":"2024-09-21T15:51:54.369203+0000","last_undegraded":"2024-09-21T15:51:54.369203+0000","last_fullsized":"2024-09-21T15:51:54.369203+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'637","last_scrub_stamp":"2024-09-21T15:51:54.369180+0000","last_deep_scrub":"168'637","last_deep_scrub_stamp":"2024-09-21T15:51:54.369180+0000","last_clean_scrub_stamp":"2024-09-21T15:51:54.369180+0000","objects_scrubbed":38,"log_size":637,"log_dups_size":0,"ondisk_log_size":637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:18:39.336637+0000","scrub_duration":0.026802408999999999,"objects_trimmed":0,"snaptrim_duration":0.0010326389999999999,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":38,"num_read":7456,"num_read_kb":25912,"num_write":580,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":250,"num_bytes_recovered":3824749,"num_keys_recovered":34,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"150'655","reported_seq":10216,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:45.784270+0000","last_change":"2024-09-21T15:51:39.767369+0000","last_active":"2024-09-21T15:51:45.784270+0000","last_peered":"2024-09-21T15:51:45.784270+0000","last_clean":"2024-09-21T15:51:45.784270+0000","last_became_active":"2024-09-21T15:51:39.765580+0000","last_became_peered":"2024-09-21T15:51:39.765580+0000","last_unstale":"2024-09-21T15:51:45.784270+0000","last_undegraded":"2024-09-21T15:51:45.784270+0000","last_fullsized":"2024-09-21T15:51:45.784270+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":655,"log_dups_size":0,"ondisk_log_size":655,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00090302100000000005,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":34,"num_read":10756,"num_read_kb":29175,"num_write":703,"num_write_kb":8882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"150'584","reported_seq":4521,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:45.784128+0000","last_change":"2024-09-21T15:51:39.767392+0000","last_active":"2024-09-21T15:51:45.784128+0000","last_peered":"2024-09-21T15:51:45.784128+0000","last_clean":"2024-09-21T15:51:45.784128+0000","last_became_active":"2024-09-21T15:51:39.765999+0000","last_became_peered":"2024-09-21T15:51:39.765999+0000","last_unstale":"2024-09-21T15:51:45.784128+0000","last_undegraded":"2024-09-21T15:51:45.784128+0000","last_fullsized":"2024-09-21T15:51:45.784128+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":584,"log_dups_size":0,"ondisk_log_size":584,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010903670000000001,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":27,"num_read":5242,"num_read_kb":17341,"num_write":644,"num_write_kb":4787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"168'547","reported_seq":4457,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:52:05.209753+0000","last_change":"2024-09-21T15:52:05.209861+0000","last_active":"2024-09-21T15:52:05.209753+0000","last_peered":"2024-09-21T15:52:05.209753+0000","last_clean":"2024-09-21T15:52:05.209753+0000","last_became_active":"2024-09-21T15:50:32.674250+0000","last_became_peered":"2024-09-21T15:50:32.674250+0000","last_unstale":"2024-09-21T15:52:05.209753+0000","last_undegraded":"2024-09-21T15:52:05.209753+0000","last_fullsized":"2024-09-21T15:52:05.209753+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'547","last_scrub_stamp":"2024-09-21T15:52:05.209731+0000","last_deep_scrub":"168'547","last_deep_scrub_stamp":"2024-09-21T15:52:05.209731+0000","last_clean_scrub_stamp":"2024-09-21T15:52:05.209731+0000","objects_scrubbed":39,"log_size":547,"log_dups_size":0,"ondisk_log_size":547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:42:51.469346+0000","scrub_duration":0.027420244,"objects_trimmed":0,"snaptrim_duration":0.0019320559999999999,"stat_sum":{"num_bytes":0,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":39,"num_read":4815,"num_read_kb":30088,"num_write":498,"num_write_kb":13507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":4759000,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"168'462","reported_seq":6989,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:53.750816+0000","last_change":"2024-09-21T15:51:53.750952+0000","last_active":"2024-09-21T15:51:53.750816+0000","last_peered":"2024-09-21T15:51:53.750816+0000","last_clean":"2024-09-21T15:51:53.750816+0000","last_became_active":"2024-09-21T15:49:06.693903+0000","last_became_peered":"2024-09-21T15:49:06.693903+0000","last_unstale":"2024-09-21T15:51:53.750816+0000","last_undegraded":"2024-09-21T15:51:53.750816+0000","last_fullsized":"2024-09-21T15:51:53.750816+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'462","last_scrub_stamp":"2024-09-21T15:51:53.750793+0000","last_deep_scrub":"168'462","last_deep_scrub_stamp":"2024-09-21T15:51:53.750793+0000","last_clean_scrub_stamp":"2024-09-21T15:51:53.750793+0000","objects_scrubbed":31,"log_size":462,"log_dups_size":0,"ondisk_log_size":462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:59:20.012704+0000","scrub_duration":0.026832650999999999,"objects_trimmed":0,"snaptrim_duration":0.00047963299999999998,"stat_sum":{"num_bytes":19,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":30,"num_read":7221,"num_read_kb":31759,"num_write":421,"num_write_kb":9476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"168'492","reported_seq":8883,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:59.302184+0000","last_change":"2024-09-21T15:51:59.302291+0000","last_active":"2024-09-21T15:51:59.302184+0000","last_peered":"2024-09-21T15:51:59.302184+0000","last_clean":"2024-09-21T15:51:59.302184+0000","last_became_active":"2024-09-21T15:51:12.688258+0000","last_became_peered":"2024-09-21T15:51:12.688258+0000","last_unstale":"2024-09-21T15:51:59.302184+0000","last_undegraded":"2024-09-21T15:51:59.302184+0000","last_fullsized":"2024-09-21T15:51:59.302184+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'492","last_scrub_stamp":"2024-09-21T15:51:59.302162+0000","last_deep_scrub":"168'492","last_deep_scrub_stamp":"2024-09-21T15:51:59.302162+0000","last_clean_scrub_stamp":"2024-09-21T15:51:59.302162+0000","objects_scrubbed":32,"log_size":492,"log_dups_size":0,"ondisk_log_size":492,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:27:39.629735+0000","scrub_duration":0.025551372999999999,"objects_trimmed":0,"snaptrim_duration":0.000411691,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":32,"num_read":8283,"num_read_kb":39579,"num_write":518,"num_write_kb":7408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":236,"num_bytes_recovered":6705120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"168'563","reported_seq":4756,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:52:00.310827+0000","last_change":"2024-09-21T15:52:00.310912+0000","last_active":"2024-09-21T15:52:00.310827+0000","last_peered":"2024-09-21T15:52:00.310827+0000","last_clean":"2024-09-21T15:52:00.310827+0000","last_became_active":"2024-09-21T15:51:35.751700+0000","last_became_peered":"2024-09-21T15:51:35.751700+0000","last_unstale":"2024-09-21T15:52:00.310827+0000","last_undegraded":"2024-09-21T15:52:00.310827+0000","last_fullsized":"2024-09-21T15:52:00.310827+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'563","last_scrub_stamp":"2024-09-21T15:52:00.310805+0000","last_deep_scrub":"168'563","last_deep_scrub_stamp":"2024-09-21T15:52:00.310805+0000","last_clean_scrub_stamp":"2024-09-21T15:52:00.310805+0000","objects_scrubbed":26,"log_size":563,"log_dups_size":0,"ondisk_log_size":563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:12:17.664579+0000","scrub_duration":0.025760488000000002,"objects_trimmed":0,"snaptrim_duration":0.0051035439999999998,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":26,"num_read":6190,"num_read_kb":30937,"num_write":534,"num_write_kb":11075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":282,"num_bytes_recovered":11320990,"num_keys_recovered":70,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"168'1076","reported_seq":9462,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:54.354291+0000","last_change":"2024-09-21T15:51:54.354423+0000","last_active":"2024-09-21T15:51:54.354291+0000","last_peered":"2024-09-21T15:51:54.354291+0000","last_clean":"2024-09-21T15:51:54.354291+0000","last_became_active":"2024-09-21T15:50:32.668771+0000","last_became_peered":"2024-09-21T15:50:32.668771+0000","last_unstale":"2024-09-21T15:51:54.354291+0000","last_undegraded":"2024-09-21T15:51:54.354291+0000","last_fullsized":"2024-09-21T15:51:54.354291+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'1076","last_scrub_stamp":"2024-09-21T15:51:54.354254+0000","last_deep_scrub":"168'1076","last_deep_scrub_stamp":"2024-09-21T15:51:54.354254+0000","last_clean_scrub_stamp":"2024-09-21T15:51:54.354254+0000","objects_scrubbed":36,"log_size":1076,"log_dups_size":0,"ondisk_log_size":1076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:31:51.364904+0000","scrub_duration":0.028413487000000001,"objects_trimmed":0,"snaptrim_duration":0.001503861,"stat_sum":{"num_bytes":0,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":36,"num_read":12714,"num_read_kb":21487,"num_write":1303,"num_write_kb":8257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":3319185,"num_keys_recovered":34,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"168'570","reported_seq":6761,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:58.112055+0000","last_change":"2024-09-21T15:51:58.112163+0000","last_active":"2024-09-21T15:51:58.112055+0000","last_peered":"2024-09-21T15:51:58.112055+0000","last_clean":"2024-09-21T15:51:58.112055+0000","last_became_active":"2024-09-21T15:49:06.688808+0000","last_became_peered":"2024-09-21T15:49:06.688808+0000","last_unstale":"2024-09-21T15:51:58.112055+0000","last_undegraded":"2024-09-21T15:51:58.112055+0000","last_fullsized":"2024-09-21T15:51:58.112055+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'570","last_scrub_stamp":"2024-09-21T15:51:58.112033+0000","last_deep_scrub":"168'570","last_deep_scrub_stamp":"2024-09-21T15:51:58.112033+0000","last_clean_scrub_stamp":"2024-09-21T15:51:58.112033+0000","objects_scrubbed":30,"log_size":570,"log_dups_size":0,"ondisk_log_size":570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:23:50.540236+0000","scrub_duration":0.021872452000000001,"objects_trimmed":0,"snaptrim_duration":0.00023944,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":30,"num_read":7604,"num_read_kb":49865,"num_write":542,"num_write_kb":13823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"150'530","reported_seq":4326,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:45.783926+0000","last_change":"2024-09-21T15:51:25.699156+0000","last_active":"2024-09-21T15:51:45.783926+0000","last_peered":"2024-09-21T15:51:45.783926+0000","last_clean":"2024-09-21T15:51:45.783926+0000","last_became_active":"2024-09-21T15:51:11.841664+0000","last_became_peered":"2024-09-21T15:51:11.841664+0000","last_unstale":"2024-09-21T15:51:45.783926+0000","last_undegraded":"2024-09-21T15:51:45.783926+0000","last_fullsized":"2024-09-21T15:51:45.783926+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":530,"log_dups_size":0,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001532828,"stat_sum":{"num_bytes":0,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":29,"num_read":4052,"num_read_kb":27144,"num_write":637,"num_write_kb":17935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":222,"num_bytes_recovered":4468215,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"168'1233","reported_seq":4428,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:57.725316+0000","last_change":"2024-09-21T15:51:57.725420+0000","last_active":"2024-09-21T15:51:57.725316+0000","last_peered":"2024-09-21T15:51:57.725316+0000","last_clean":"2024-09-21T15:51:57.725316+0000","last_became_active":"2024-09-21T15:49:06.688803+0000","last_became_peered":"2024-09-21T15:49:06.688803+0000","last_unstale":"2024-09-21T15:51:57.725316+0000","last_undegraded":"2024-09-21T15:51:57.725316+0000","last_fullsized":"2024-09-21T15:51:57.725316+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'1233","last_scrub_stamp":"2024-09-21T15:51:57.725294+0000","last_deep_scrub":"168'1233","last_deep_scrub_stamp":"2024-09-21T15:51:57.725294+0000","last_clean_scrub_stamp":"2024-09-21T15:51:57.725294+0000","objects_scrubbed":26,"log_size":1233,"log_dups_size":0,"ondisk_log_size":1233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:40:56.930628+0000","scrub_duration":0.023941679,"objects_trimmed":0,"snaptrim_duration":0.0013586710000000001,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":26,"num_read":7764,"num_read_kb":18232,"num_write":1926,"num_write_kb":6190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":173,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:52:02.022929+0000","last_change":"2024-09-21T15:52:02.023057+0000","last_active":"2024-09-21T15:52:02.022929+0000","last_peered":"2024-09-21T15:52:02.022929+0000","last_clean":"2024-09-21T15:52:02.022929+0000","last_became_active":"2024-09-21T15:48:57.583665+0000","last_became_peered":"2024-09-21T15:48:57.583665+0000","last_unstale":"2024-09-21T15:52:02.022929+0000","last_undegraded":"2024-09-21T15:52:02.022929+0000","last_fullsized":"2024-09-21T15:52:02.022929+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:52:02.022908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:52:02.022908+0000","last_clean_scrub_stamp":"2024-09-21T15:52:02.022908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:54:37.398705+0000","scrub_duration":0.0014227129999999999,"objects_trimmed":0,"snaptrim_duration":0.00031098399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"168'473","reported_seq":3113,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:53.387345+0000","last_change":"2024-09-21T15:51:53.387481+0000","last_active":"2024-09-21T15:51:53.387345+0000","last_peered":"2024-09-21T15:51:53.387345+0000","last_clean":"2024-09-21T15:51:53.387345+0000","last_became_active":"2024-09-21T15:50:32.671424+0000","last_became_peered":"2024-09-21T15:50:32.671424+0000","last_unstale":"2024-09-21T15:51:53.387345+0000","last_undegraded":"2024-09-21T15:51:53.387345+0000","last_fullsized":"2024-09-21T15:51:53.387345+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'473","last_scrub_stamp":"2024-09-21T15:51:53.387307+0000","last_deep_scrub":"168'473","last_deep_scrub_stamp":"2024-09-21T15:51:53.387307+0000","last_clean_scrub_stamp":"2024-09-21T15:51:53.387307+0000","objects_scrubbed":35,"log_size":473,"log_dups_size":0,"ondisk_log_size":473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:48:07.248726+0000","scrub_duration":0.02547369,"objects_trimmed":0,"snaptrim_duration":0.001507467,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":35,"num_read":2637,"num_read_kb":12956,"num_write":388,"num_write_kb":8392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":4494388,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":177,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:56.296589+0000","last_change":"2024-09-21T15:51:56.296699+0000","last_active":"2024-09-21T15:51:56.296589+0000","last_peered":"2024-09-21T15:51:56.296589+0000","last_clean":"2024-09-21T15:51:56.296589+0000","last_became_active":"2024-09-21T15:48:57.582121+0000","last_became_peered":"2024-09-21T15:48:57.582121+0000","last_unstale":"2024-09-21T15:51:56.296589+0000","last_undegraded":"2024-09-21T15:51:56.296589+0000","last_fullsized":"2024-09-21T15:51:56.296589+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:51:56.296569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:51:56.296569+0000","last_clean_scrub_stamp":"2024-09-21T15:51:56.296569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:31:39.212030+0000","scrub_duration":0.001405795,"objects_trimmed":0,"snaptrim_duration":0.00039654700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"168'614","reported_seq":8050,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:57.293683+0000","last_change":"2024-09-21T15:51:57.293824+0000","last_active":"2024-09-21T15:51:57.293683+0000","last_peered":"2024-09-21T15:51:57.293683+0000","last_clean":"2024-09-21T15:51:57.293683+0000","last_became_active":"2024-09-21T15:51:35.752700+0000","last_became_peered":"2024-09-21T15:51:35.752700+0000","last_unstale":"2024-09-21T15:51:57.293683+0000","last_undegraded":"2024-09-21T15:51:57.293683+0000","last_fullsized":"2024-09-21T15:51:57.293683+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'614","last_scrub_stamp":"2024-09-21T15:51:57.293645+0000","last_deep_scrub":"168'614","last_deep_scrub_stamp":"2024-09-21T15:51:57.293645+0000","last_clean_scrub_stamp":"2024-09-21T15:51:57.293645+0000","objects_scrubbed":35,"log_size":614,"log_dups_size":0,"ondisk_log_size":614,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:25:01.117384+0000","scrub_duration":0.024979000000000001,"objects_trimmed":2,"snaptrim_duration":0.00086244900000000005,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":35,"num_read":8380,"num_read_kb":21195,"num_write":544,"num_write_kb":5397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":256,"num_bytes_recovered":3760369,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":188,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:52:02.454791+0000","last_change":"2024-09-21T15:52:02.454892+0000","last_active":"2024-09-21T15:52:02.454791+0000","last_peered":"2024-09-21T15:52:02.454791+0000","last_clean":"2024-09-21T15:52:02.454791+0000","last_became_active":"2024-09-21T15:50:30.146428+0000","last_became_peered":"2024-09-21T15:50:30.146428+0000","last_unstale":"2024-09-21T15:52:02.454791+0000","last_undegraded":"2024-09-21T15:52:02.454791+0000","last_fullsized":"2024-09-21T15:52:02.454791+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:52:02.454771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:52:02.454771+0000","last_clean_scrub_stamp":"2024-09-21T15:52:02.454771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:39:26.983338+0000","scrub_duration":0.0018950830000000001,"objects_trimmed":0,"snaptrim_duration":0.00036074600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"168'729","reported_seq":8894,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:52:05.353732+0000","last_change":"2024-09-21T15:52:05.353857+0000","last_active":"2024-09-21T15:52:05.353732+0000","last_peered":"2024-09-21T15:52:05.353732+0000","last_clean":"2024-09-21T15:52:05.353732+0000","last_became_active":"2024-09-21T15:50:35.682788+0000","last_became_peered":"2024-09-21T15:50:35.682788+0000","last_unstale":"2024-09-21T15:52:05.353732+0000","last_undegraded":"2024-09-21T15:52:05.353732+0000","last_fullsized":"2024-09-21T15:52:05.353732+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'729","last_scrub_stamp":"2024-09-21T15:52:05.353709+0000","last_deep_scrub":"168'729","last_deep_scrub_stamp":"2024-09-21T15:52:05.353709+0000","last_clean_scrub_stamp":"2024-09-21T15:52:05.353709+0000","objects_scrubbed":26,"log_size":729,"log_dups_size":0,"ondisk_log_size":729,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:51:06.196186+0000","scrub_duration":0.026638608000000001,"objects_trimmed":0,"snaptrim_duration":0.00032907100000000001,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":26,"num_read":11015,"num_read_kb":27666,"num_write":868,"num_write_kb":11017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":175,"num_bytes_recovered":14134268,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":190,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:58.268172+0000","last_change":"2024-09-21T15:51:58.268273+0000","last_active":"2024-09-21T15:51:58.268172+0000","last_peered":"2024-09-21T15:51:58.268172+0000","last_clean":"2024-09-21T15:51:58.268172+0000","last_became_active":"2024-09-21T15:50:30.146273+0000","last_became_peered":"2024-09-21T15:50:30.146273+0000","last_unstale":"2024-09-21T15:51:58.268172+0000","last_undegraded":"2024-09-21T15:51:58.268172+0000","last_fullsized":"2024-09-21T15:51:58.268172+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:51:58.268153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:51:58.268153+0000","last_clean_scrub_stamp":"2024-09-21T15:51:58.268153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:38:41.425777+0000","scrub_duration":0.0014527979999999999,"objects_trimmed":0,"snaptrim_duration":0.00040525700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"150'603","reported_seq":5378,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:45.784348+0000","last_change":"2024-09-21T15:51:39.769574+0000","last_active":"2024-09-21T15:51:45.784348+0000","last_peered":"2024-09-21T15:51:45.784348+0000","last_clean":"2024-09-21T15:51:45.784348+0000","last_became_active":"2024-09-21T15:51:39.768300+0000","last_became_peered":"2024-09-21T15:51:39.768300+0000","last_unstale":"2024-09-21T15:51:45.784348+0000","last_undegraded":"2024-09-21T15:51:45.784348+0000","last_fullsized":"2024-09-21T15:51:45.784348+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":603,"log_dups_size":0,"ondisk_log_size":603,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00050855200000000001,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":35,"num_read":5730,"num_read_kb":33940,"num_write":686,"num_write_kb":16332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":268,"num_bytes_recovered":7823491,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"168'956","reported_seq":4245,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:56.688731+0000","last_change":"2024-09-21T15:51:56.688851+0000","last_active":"2024-09-21T15:51:56.688731+0000","last_peered":"2024-09-21T15:51:56.688731+0000","last_clean":"2024-09-21T15:51:56.688731+0000","last_became_active":"2024-09-21T15:50:32.677456+0000","last_became_peered":"2024-09-21T15:50:32.677456+0000","last_unstale":"2024-09-21T15:51:56.688731+0000","last_undegraded":"2024-09-21T15:51:56.688731+0000","last_fullsized":"2024-09-21T15:51:56.688731+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'956","last_scrub_stamp":"2024-09-21T15:51:56.688706+0000","last_deep_scrub":"168'956","last_deep_scrub_stamp":"2024-09-21T15:51:56.688706+0000","last_clean_scrub_stamp":"2024-09-21T15:51:56.688706+0000","objects_scrubbed":35,"log_size":956,"log_dups_size":0,"ondisk_log_size":956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:34:17.308503+0000","scrub_duration":0.033507282999999999,"objects_trimmed":0,"snaptrim_duration":0.0010737730000000001,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":34,"num_read":5136,"num_read_kb":19340,"num_write":949,"num_write_kb":7405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":632176,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"168'3","reported_seq":73,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:59.086482+0000","last_change":"2024-09-21T15:51:59.086589+0000","last_active":"2024-09-21T15:51:59.086482+0000","last_peered":"2024-09-21T15:51:59.086482+0000","last_clean":"2024-09-21T15:51:59.086482+0000","last_became_active":"2024-09-21T15:51:39.764409+0000","last_became_peered":"2024-09-21T15:51:39.764409+0000","last_unstale":"2024-09-21T15:51:59.086482+0000","last_undegraded":"2024-09-21T15:51:59.086482+0000","last_fullsized":"2024-09-21T15:51:59.086482+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'3","last_scrub_stamp":"2024-09-21T15:51:59.086459+0000","last_deep_scrub":"168'3","last_deep_scrub_stamp":"2024-09-21T15:51:59.086459+0000","last_clean_scrub_stamp":"2024-09-21T15:51:59.086459+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:04:58.492791+0000","scrub_duration":0.0036796239999999998,"objects_trimmed":0,"snaptrim_duration":0.000859595,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"168'424","reported_seq":4879,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:52:00.278311+0000","last_change":"2024-09-21T15:52:00.278425+0000","last_active":"2024-09-21T15:52:00.278311+0000","last_peered":"2024-09-21T15:52:00.278311+0000","last_clean":"2024-09-21T15:52:00.278311+0000","last_became_active":"2024-09-21T15:50:35.682001+0000","last_became_peered":"2024-09-21T15:50:35.682001+0000","last_unstale":"2024-09-21T15:52:00.278311+0000","last_undegraded":"2024-09-21T15:52:00.278311+0000","last_fullsized":"2024-09-21T15:52:00.278311+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'424","last_scrub_stamp":"2024-09-21T15:52:00.278288+0000","last_deep_scrub":"168'424","last_deep_scrub_stamp":"2024-09-21T15:52:00.278288+0000","last_clean_scrub_stamp":"2024-09-21T15:52:00.278288+0000","objects_scrubbed":31,"log_size":424,"log_dups_size":0,"ondisk_log_size":424,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:16:30.872951+0000","scrub_duration":0.023884934999999999,"objects_trimmed":0,"snaptrim_duration":0.00044281000000000002,"stat_sum":{"num_bytes":0,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":31,"num_read":4285,"num_read_kb":23375,"num_write":379,"num_write_kb":4145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":187,"num_bytes_recovered":4305276,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":109,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:52.415337+0000","last_change":"2024-09-21T15:51:52.415462+0000","last_active":"2024-09-21T15:51:52.415337+0000","last_peered":"2024-09-21T15:51:52.415337+0000","last_clean":"2024-09-21T15:51:52.415337+0000","last_became_active":"2024-09-21T15:51:07.683418+0000","last_became_peered":"2024-09-21T15:51:07.683418+0000","last_unstale":"2024-09-21T15:51:52.415337+0000","last_undegraded":"2024-09-21T15:51:52.415337+0000","last_fullsized":"2024-09-21T15:51:52.415337+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:51:52.415306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:51:52.415306+0000","last_clean_scrub_stamp":"2024-09-21T15:51:52.415306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:28:29.927181+0000","scrub_duration":0.0013079089999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"168'597","reported_seq":2651,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:56.283594+0000","last_change":"2024-09-21T15:51:56.283724+0000","last_active":"2024-09-21T15:51:56.283594+0000","last_peered":"2024-09-21T15:51:56.283594+0000","last_clean":"2024-09-21T15:51:56.283594+0000","last_became_active":"2024-09-21T15:50:32.671413+0000","last_became_peered":"2024-09-21T15:50:32.671413+0000","last_unstale":"2024-09-21T15:51:56.283594+0000","last_undegraded":"2024-09-21T15:51:56.283594+0000","last_fullsized":"2024-09-21T15:51:56.283594+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'597","last_scrub_stamp":"2024-09-21T15:51:56.283573+0000","last_deep_scrub":"168'597","last_deep_scrub_stamp":"2024-09-21T15:51:56.283573+0000","last_clean_scrub_stamp":"2024-09-21T15:51:56.283573+0000","objects_scrubbed":39,"log_size":597,"log_dups_size":0,"ondisk_log_size":597,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:35:45.562010+0000","scrub_duration":0.027644431000000001,"objects_trimmed":0,"snaptrim_duration":0.001292872,"stat_sum":{"num_bytes":0,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":39,"num_read":3663,"num_read_kb":16543,"num_write":590,"num_write_kb":5977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":1696547,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":175,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:58.512849+0000","last_change":"2024-09-21T15:51:58.512993+0000","last_active":"2024-09-21T15:51:58.512849+0000","last_peered":"2024-09-21T15:51:58.512849+0000","last_clean":"2024-09-21T15:51:58.512849+0000","last_became_active":"2024-09-21T15:48:57.585525+0000","last_became_peered":"2024-09-21T15:48:57.585525+0000","last_unstale":"2024-09-21T15:51:58.512849+0000","last_undegraded":"2024-09-21T15:51:58.512849+0000","last_fullsized":"2024-09-21T15:51:58.512849+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:51:58.512800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:51:58.512800+0000","last_clean_scrub_stamp":"2024-09-21T15:51:58.512800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:59:12.580616+0000","scrub_duration":0.001937846,"objects_trimmed":0,"snaptrim_duration":0.00031691100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"168'443","reported_seq":3001,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:55.355408+0000","last_change":"2024-09-21T15:51:55.355505+0000","last_active":"2024-09-21T15:51:55.355408+0000","last_peered":"2024-09-21T15:51:55.355408+0000","last_clean":"2024-09-21T15:51:55.355408+0000","last_became_active":"2024-09-21T15:51:35.752810+0000","last_became_peered":"2024-09-21T15:51:35.752810+0000","last_unstale":"2024-09-21T15:51:55.355408+0000","last_undegraded":"2024-09-21T15:51:55.355408+0000","last_fullsized":"2024-09-21T15:51:55.355408+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'443","last_scrub_stamp":"2024-09-21T15:51:55.355386+0000","last_deep_scrub":"168'443","last_deep_scrub_stamp":"2024-09-21T15:51:55.355386+0000","last_clean_scrub_stamp":"2024-09-21T15:51:55.355386+0000","objects_scrubbed":30,"log_size":443,"log_dups_size":0,"ondisk_log_size":443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:00:45.418578+0000","scrub_duration":0.022450193,"objects_trimmed":0,"snaptrim_duration":0.00033178999999999999,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":30,"num_read":2149,"num_read_kb":34452,"num_write":406,"num_write_kb":12612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":214,"num_bytes_recovered":2197196,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"18'1","reported_seq":60,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:45.783882+0000","last_change":"2024-09-21T15:51:39.764710+0000","last_active":"2024-09-21T15:51:45.783882+0000","last_peered":"2024-09-21T15:51:45.783882+0000","last_clean":"2024-09-21T15:51:45.783882+0000","last_became_active":"2024-09-21T15:51:39.764411+0000","last_became_peered":"2024-09-21T15:51:39.764411+0000","last_unstale":"2024-09-21T15:51:45.783882+0000","last_undegraded":"2024-09-21T15:51:45.783882+0000","last_fullsized":"2024-09-21T15:51:45.783882+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:48:56.568162+0000","last_clean_scrub_stamp":"2024-09-21T15:48:56.568162+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00039559000000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"168'119","reported_seq":338,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:57.510040+0000","last_change":"2024-09-21T15:51:57.510169+0000","last_active":"2024-09-21T15:51:57.510040+0000","last_peered":"2024-09-21T15:51:57.510040+0000","last_clean":"2024-09-21T15:51:57.510040+0000","last_became_active":"2024-09-21T15:50:30.146987+0000","last_became_peered":"2024-09-21T15:50:30.146987+0000","last_unstale":"2024-09-21T15:51:57.510040+0000","last_undegraded":"2024-09-21T15:51:57.510040+0000","last_fullsized":"2024-09-21T15:51:57.510040+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'119","last_scrub_stamp":"2024-09-21T15:51:57.510007+0000","last_deep_scrub":"168'119","last_deep_scrub_stamp":"2024-09-21T15:51:57.510007+0000","last_clean_scrub_stamp":"2024-09-21T15:51:57.510007+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:47:31.553607+0000","scrub_duration":0.0055273869999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"168'572","reported_seq":7990,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:53.395767+0000","last_change":"2024-09-21T15:51:53.395872+0000","last_active":"2024-09-21T15:51:53.395767+0000","last_peered":"2024-09-21T15:51:53.395767+0000","last_clean":"2024-09-21T15:51:53.395767+0000","last_became_active":"2024-09-21T15:51:11.837877+0000","last_became_peered":"2024-09-21T15:51:11.837877+0000","last_unstale":"2024-09-21T15:51:53.395767+0000","last_undegraded":"2024-09-21T15:51:53.395767+0000","last_fullsized":"2024-09-21T15:51:53.395767+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'572","last_scrub_stamp":"2024-09-21T15:51:53.395748+0000","last_deep_scrub":"168'572","last_deep_scrub_stamp":"2024-09-21T15:51:53.395748+0000","last_clean_scrub_stamp":"2024-09-21T15:51:53.395748+0000","objects_scrubbed":32,"log_size":572,"log_dups_size":0,"ondisk_log_size":572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:05:31.768212+0000","scrub_duration":0.029579207999999999,"objects_trimmed":0,"snaptrim_duration":0.00079839600000000004,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":32,"num_read":7239,"num_read_kb":29580,"num_write":558,"num_write_kb":4793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":418,"num_bytes_recovered":6262707,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"168'419","reported_seq":3215,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:55.697626+0000","last_change":"2024-09-21T15:51:55.697770+0000","last_active":"2024-09-21T15:51:55.697626+0000","last_peered":"2024-09-21T15:51:55.697626+0000","last_clean":"2024-09-21T15:51:55.697626+0000","last_became_active":"2024-09-21T15:49:07.210506+0000","last_became_peered":"2024-09-21T15:49:07.210506+0000","last_unstale":"2024-09-21T15:51:55.697626+0000","last_undegraded":"2024-09-21T15:51:55.697626+0000","last_fullsized":"2024-09-21T15:51:55.697626+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'419","last_scrub_stamp":"2024-09-21T15:51:55.697604+0000","last_deep_scrub":"168'419","last_deep_scrub_stamp":"2024-09-21T15:51:55.697604+0000","last_clean_scrub_stamp":"2024-09-21T15:51:55.697604+0000","objects_scrubbed":30,"log_size":419,"log_dups_size":0,"ondisk_log_size":419,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:57:05.404395+0000","scrub_duration":0.022683354999999999,"objects_trimmed":0,"snaptrim_duration":0.00089193900000000001,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":30,"num_read":2716,"num_read_kb":28245,"num_write":454,"num_write_kb":12090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"168'625","reported_seq":12311,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:52:09.055828+0000","last_change":"2024-09-21T15:52:09.055935+0000","last_active":"2024-09-21T15:52:09.055828+0000","last_peered":"2024-09-21T15:52:09.055828+0000","last_clean":"2024-09-21T15:52:09.055828+0000","last_became_active":"2024-09-21T15:51:41.781781+0000","last_became_peered":"2024-09-21T15:51:41.781781+0000","last_unstale":"2024-09-21T15:52:09.055828+0000","last_undegraded":"2024-09-21T15:52:09.055828+0000","last_fullsized":"2024-09-21T15:52:09.055828+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'625","last_scrub_stamp":"2024-09-21T15:52:09.055804+0000","last_deep_scrub":"168'625","last_deep_scrub_stamp":"2024-09-21T15:52:09.055804+0000","last_clean_scrub_stamp":"2024-09-21T15:52:09.055804+0000","objects_scrubbed":30,"log_size":625,"log_dups_size":0,"ondisk_log_size":625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:23:27.790891+0000","scrub_duration":0.030555435999999998,"objects_trimmed":0,"snaptrim_duration":0.000309432,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":29,"num_read":12828,"num_read_kb":63467,"num_write":580,"num_write_kb":14212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"168'877","reported_seq":3895,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:58.296997+0000","last_change":"2024-09-21T15:51:58.297106+0000","last_active":"2024-09-21T15:51:58.296997+0000","last_peered":"2024-09-21T15:51:58.296997+0000","last_clean":"2024-09-21T15:51:58.296997+0000","last_became_active":"2024-09-21T15:51:35.751150+0000","last_became_peered":"2024-09-21T15:51:35.751150+0000","last_unstale":"2024-09-21T15:51:58.296997+0000","last_undegraded":"2024-09-21T15:51:58.296997+0000","last_fullsized":"2024-09-21T15:51:58.296997+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'877","last_scrub_stamp":"2024-09-21T15:51:58.296964+0000","last_deep_scrub":"168'877","last_deep_scrub_stamp":"2024-09-21T15:51:58.296964+0000","last_clean_scrub_stamp":"2024-09-21T15:51:58.296964+0000","objects_scrubbed":26,"log_size":877,"log_dups_size":0,"ondisk_log_size":877,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:39:41.972358+0000","scrub_duration":0.022677787000000001,"objects_trimmed":0,"snaptrim_duration":0.001158526,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":26,"num_read":5948,"num_read_kb":20169,"num_write":1244,"num_write_kb":7750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":262,"num_bytes_recovered":8353711,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"150'739","reported_seq":6770,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:45.784243+0000","last_change":"2024-09-21T15:51:39.769469+0000","last_active":"2024-09-21T15:51:45.784243+0000","last_peered":"2024-09-21T15:51:45.784243+0000","last_clean":"2024-09-21T15:51:45.784243+0000","last_became_active":"2024-09-21T15:51:39.768321+0000","last_became_peered":"2024-09-21T15:51:39.768321+0000","last_unstale":"2024-09-21T15:51:45.784243+0000","last_undegraded":"2024-09-21T15:51:45.784243+0000","last_fullsized":"2024-09-21T15:51:45.784243+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":739,"log_dups_size":0,"ondisk_log_size":739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00074635999999999995,"stat_sum":{"num_bytes":0,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":36,"num_read":8849,"num_read_kb":25678,"num_write":868,"num_write_kb":5005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":246,"num_bytes_recovered":5686754,"num_keys_recovered":23,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"168'929","reported_seq":4364,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:52:06.450770+0000","last_change":"2024-09-21T15:52:06.450878+0000","last_active":"2024-09-21T15:52:06.450770+0000","last_peered":"2024-09-21T15:52:06.450770+0000","last_clean":"2024-09-21T15:52:06.450770+0000","last_became_active":"2024-09-21T15:49:06.689203+0000","last_became_peered":"2024-09-21T15:49:06.689203+0000","last_unstale":"2024-09-21T15:52:06.450770+0000","last_undegraded":"2024-09-21T15:52:06.450770+0000","last_fullsized":"2024-09-21T15:52:06.450770+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'929","last_scrub_stamp":"2024-09-21T15:52:06.450748+0000","last_deep_scrub":"168'929","last_deep_scrub_stamp":"2024-09-21T15:52:06.450748+0000","last_clean_scrub_stamp":"2024-09-21T15:52:06.450748+0000","objects_scrubbed":35,"log_size":929,"log_dups_size":0,"ondisk_log_size":929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:47:39.763225+0000","scrub_duration":0.029372476000000002,"objects_trimmed":0,"snaptrim_duration":0.00084935700000000002,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":35,"num_read":7321,"num_read_kb":28131,"num_write":1146,"num_write_kb":14809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"148'568","reported_seq":8146,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:45.784124+0000","last_change":"2024-09-21T15:51:25.697735+0000","last_active":"2024-09-21T15:51:45.784124+0000","last_peered":"2024-09-21T15:51:45.784124+0000","last_clean":"2024-09-21T15:51:45.784124+0000","last_became_active":"2024-09-21T15:49:06.690101+0000","last_became_peered":"2024-09-21T15:49:06.690101+0000","last_unstale":"2024-09-21T15:51:45.784124+0000","last_undegraded":"2024-09-21T15:51:45.784124+0000","last_fullsized":"2024-09-21T15:51:45.784124+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":568,"log_dups_size":0,"ondisk_log_size":568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044571200000000002,"stat_sum":{"num_bytes":0,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":29,"num_read":8735,"num_read_kb":42983,"num_write":509,"num_write_kb":16273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"145'698","reported_seq":3801,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:45.783936+0000","last_change":"2024-09-21T15:51:25.698380+0000","last_active":"2024-09-21T15:51:45.783936+0000","last_peered":"2024-09-21T15:51:45.783936+0000","last_clean":"2024-09-21T15:51:45.783936+0000","last_became_active":"2024-09-21T15:49:06.694250+0000","last_became_peered":"2024-09-21T15:49:06.694250+0000","last_unstale":"2024-09-21T15:51:45.783936+0000","last_undegraded":"2024-09-21T15:51:45.783936+0000","last_fullsized":"2024-09-21T15:51:45.783936+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":698,"log_dups_size":0,"ondisk_log_size":698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00086632899999999999,"stat_sum":{"num_bytes":0,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":42,"num_read":4945,"num_read_kb":15920,"num_write":889,"num_write_kb":5075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"168'510","reported_seq":3208,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:52:05.043840+0000","last_change":"2024-09-21T15:52:05.043971+0000","last_active":"2024-09-21T15:52:05.043840+0000","last_peered":"2024-09-21T15:52:05.043840+0000","last_clean":"2024-09-21T15:52:05.043840+0000","last_became_active":"2024-09-21T15:50:32.676147+0000","last_became_peered":"2024-09-21T15:50:32.676147+0000","last_unstale":"2024-09-21T15:52:05.043840+0000","last_undegraded":"2024-09-21T15:52:05.043840+0000","last_fullsized":"2024-09-21T15:52:05.043840+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'510","last_scrub_stamp":"2024-09-21T15:52:05.043817+0000","last_deep_scrub":"168'510","last_deep_scrub_stamp":"2024-09-21T15:52:05.043817+0000","last_clean_scrub_stamp":"2024-09-21T15:52:05.043817+0000","objects_scrubbed":40,"log_size":510,"log_dups_size":0,"ondisk_log_size":510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:29:02.342720+0000","scrub_duration":0.027963423000000001,"objects_trimmed":0,"snaptrim_duration":0.00024304599999999999,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":40,"num_read":3160,"num_read_kb":11849,"num_write":381,"num_write_kb":4471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":1191778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"148'1012","reported_seq":4199,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:45.784171+0000","last_change":"2024-09-21T15:51:25.697763+0000","last_active":"2024-09-21T15:51:45.784171+0000","last_peered":"2024-09-21T15:51:45.784171+0000","last_clean":"2024-09-21T15:51:45.784171+0000","last_became_active":"2024-09-21T15:49:06.688798+0000","last_became_peered":"2024-09-21T15:49:06.688798+0000","last_unstale":"2024-09-21T15:51:45.784171+0000","last_undegraded":"2024-09-21T15:51:45.784171+0000","last_fullsized":"2024-09-21T15:51:45.784171+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":1012,"log_dups_size":0,"ondisk_log_size":1012,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00053058599999999999,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":30,"num_read":6391,"num_read_kb":16970,"num_write":1355,"num_write_kb":8408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"168'417","reported_seq":3059,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:51.378061+0000","last_change":"2024-09-21T15:51:51.378167+0000","last_active":"2024-09-21T15:51:51.378061+0000","last_peered":"2024-09-21T15:51:51.378061+0000","last_clean":"2024-09-21T15:51:51.378061+0000","last_became_active":"2024-09-21T15:50:32.671686+0000","last_became_peered":"2024-09-21T15:50:32.671686+0000","last_unstale":"2024-09-21T15:51:51.378061+0000","last_undegraded":"2024-09-21T15:51:51.378061+0000","last_fullsized":"2024-09-21T15:51:51.378061+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'417","last_scrub_stamp":"2024-09-21T15:51:51.378041+0000","last_deep_scrub":"168'417","last_deep_scrub_stamp":"2024-09-21T15:51:51.378041+0000","last_clean_scrub_stamp":"2024-09-21T15:51:51.378041+0000","objects_scrubbed":34,"log_size":417,"log_dups_size":0,"ondisk_log_size":417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:40:52.256253+0000","scrub_duration":0.023972753999999999,"objects_trimmed":0,"snaptrim_duration":0.0012972649999999999,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":34,"num_read":2455,"num_read_kb":11669,"num_write":401,"num_write_kb":13839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":3359702,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"150'967","reported_seq":7433,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:45.783876+0000","last_change":"2024-09-21T15:51:25.699098+0000","last_active":"2024-09-21T15:51:45.783876+0000","last_peered":"2024-09-21T15:51:45.783876+0000","last_clean":"2024-09-21T15:51:45.783876+0000","last_became_active":"2024-09-21T15:49:06.690128+0000","last_became_peered":"2024-09-21T15:49:06.690128+0000","last_unstale":"2024-09-21T15:51:45.783876+0000","last_undegraded":"2024-09-21T15:51:45.783876+0000","last_fullsized":"2024-09-21T15:51:45.783876+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:49:05.663715+0000","last_clean_scrub_stamp":"2024-09-21T15:49:05.663715+0000","objects_scrubbed":0,"log_size":967,"log_dups_size":0,"ondisk_log_size":967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014652879999999999,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":32,"num_read":9327,"num_read_kb":26357,"num_write":1446,"num_write_kb":13580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":1039,"num_object_clones":0,"num_object_copies":2078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1039,"num_whiteouts":1035,"num_read":207419,"num_read_kb":842423,"num_write":23389,"num_write_kb":303239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3397,"num_bytes_recovered":100270055,"num_keys_recovered":316,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5973,"internal_metadata":0},"log_size":20962,"ondisk_log_size":20962,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574892,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77956,"kb_used_data":2556,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94293884,"statfs":{"total":96636764160,"available":96556937216,"internally_reserved":0,"allocated":2617344,"data_stored":1563409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":77199819},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:32 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.75},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.65000000000000002}]},{"osd":1,"last update":"Sat Sep 21 15:51:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.48499999999999999,"15min":0.495},"min":{"1min":0.39600000000000002,"5min":0.188,"15min":0.188},"max":{"1min":0.64000000000000001,"5min":1.04,"15min":1.04},"last":0.71899999999999997},{"interface":"front","average":{"1min":0.504,"5min":0.47499999999999998,"15min":0.47699999999999998},"min":{"1min":0.32300000000000001,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.69799999999999995}]},{"osd":2,"last update":"Sat Sep 21 15:51:56 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.34100000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.498,"15min":0.50800000000000001},"min":{"1min":0.311,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.83199999999999996,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.68400000000000005}]},{"osd":3,"last update":"Sat Sep 21 15:51:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.496,"15min":0.5},"min":{"1min":0.32900000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.86199999999999999,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.47299999999999998,"15min":0.47799999999999998},"min":{"1min":0.36399999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.74399999999999999,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.66000000000000003}]},{"osd":4,"last update":"Sat Sep 21 15:51:56 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.41599999999999998,"15min":0.42299999999999999},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.59499999999999997,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.44,"5min":0.39900000000000002,"15min":0.39500000000000002},"min":{"1min":0.28499999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.48099999999999998}]},{"osd":5,"last update":"Sat Sep 21 15:51:56 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.44900000000000001},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.75700000000000001,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.45800000000000002,"15min":0.45400000000000001},"min":{"1min":0.30099999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.73199999999999998}]},{"osd":6,"last update":"Sat Sep 21 15:51:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.495,"15min":0.501},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.5},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.46700000000000003,"15min":0.45900000000000002},"min":{"1min":0.34399999999999997,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.60999999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574895,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41280,"kb_used_data":1976,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94330560,"statfs":{"total":96636764160,"available":96594493440,"internally_reserved":0,"allocated":2023424,"data_stored":973041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:31 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.26900000000000002},{"interface":"front","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.39200000000000002}]},{"osd":1,"last update":"Sat Sep 21 15:51:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.28199999999999997,"5min":0.255,"15min":0.255},"max":{"1min":1.3120000000000001,"5min":1.3120000000000001,"15min":1.3120000000000001},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.47199999999999998,"15min":0.48099999999999998},"min":{"1min":0.34000000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":1.5229999999999999,"5min":1.5229999999999999,"15min":1.5229999999999999},"last":0.313}]},{"osd":2,"last update":"Sat Sep 21 15:51:53 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.46600000000000003,"15min":0.47499999999999998},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.64900000000000002,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.46999999999999997,"15min":0.46600000000000003},"min":{"1min":0.29799999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.411,"5min":1.411,"15min":1.411},"last":0.35299999999999998}]},{"osd":3,"last update":"Sat Sep 21 15:51:53 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.50700000000000001,"15min":0.52100000000000002},"min":{"1min":0.31,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.67700000000000005,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.52900000000000003,"15min":0.53700000000000003},"min":{"1min":0.33800000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.68100000000000005,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.47399999999999998}]},{"osd":4,"last update":"Sat Sep 21 15:51:53 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.40799999999999997,"15min":0.41399999999999998},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.67200000000000004,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.46100000000000002,"15min":0.47999999999999998},"min":{"1min":0.24199999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.68899999999999995,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.437}]},{"osd":5,"last update":"Sat Sep 21 15:51:53 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.44500000000000001,"15min":0.45500000000000002},"min":{"1min":0.34300000000000003,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.64200000000000002,"5min":0.755,"15min":0.755},"last":0.379},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.44600000000000001,"15min":0.46200000000000002},"min":{"1min":0.28100000000000003,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.626,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.36899999999999999}]},{"osd":7,"last update":"Sat Sep 21 15:51:53 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.45300000000000001,"15min":0.45800000000000002},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.48799999999999999,"15min":0.50800000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.67600000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.41199999999999998}]}]},{"osd":5,"up_from":13,"seq":55834574894,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55236,"kb_used_data":1980,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94316604,"statfs":{"total":96636764160,"available":96580202496,"internally_reserved":0,"allocated":2027520,"data_stored":973060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.502},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.441}]},{"osd":1,"last update":"Sat Sep 21 15:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.44600000000000001,"15min":0.44500000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.71599999999999997,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.48799999999999999,"15min":0.51000000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.60899999999999999,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.54200000000000004}]},{"osd":2,"last update":"Sat Sep 21 15:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.495,"15min":0.50700000000000001},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73199999999999998,"5min":1.046,"15min":1.046},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.47399999999999998,"15min":0.47499999999999998},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.72099999999999997,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.622}]},{"osd":3,"last update":"Sat Sep 21 15:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.47799999999999998,"15min":0.48799999999999999},"min":{"1min":0.29999999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.70299999999999996,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.52200000000000002,"15min":0.54700000000000004},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.62,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.48299999999999998}]},{"osd":4,"last update":"Sat Sep 21 15:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.60299999999999998,"15min":0.69099999999999995},"min":{"1min":0.28599999999999998,"5min":0.253,"15min":0.253},"max":{"1min":0.77700000000000002,"5min":5.6399999999999997,"15min":5.6399999999999997},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.44900000000000001,"15min":0.47099999999999997},"min":{"1min":0.26800000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.46300000000000002}]},{"osd":6,"last update":"Sat Sep 21 15:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.44600000000000001,"15min":0.45600000000000002},"min":{"1min":0.29099999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.47999999999999998,"15min":0.498},"min":{"1min":0.33500000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.74399999999999999,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.371}]},{"osd":7,"last update":"Sat Sep 21 15:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.47499999999999998,"15min":0.48699999999999999},"min":{"1min":0.35199999999999998,"5min":0.312,"15min":0.312},"max":{"1min":0.70599999999999996,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.503,"5min":0.499,"15min":0.51900000000000002},"min":{"1min":0.28999999999999998,"5min":0.248,"15min":0.248},"max":{"1min":0.66400000000000003,"5min":0.747,"15min":0.747},"last":0.52400000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574893,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69184,"kb_used_data":1976,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94302656,"statfs":{"total":96636764160,"available":96565919744,"internally_reserved":0,"allocated":2023424,"data_stored":971778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68811206},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.69699999999999995}]},{"osd":1,"last update":"Sat Sep 21 15:52:01 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.45000000000000001,"15min":0.42399999999999999},"min":{"1min":0.28999999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.65000000000000002,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.46999999999999997,"15min":0.46400000000000002},"min":{"1min":0.34999999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.67900000000000005,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.54300000000000004}]},{"osd":2,"last update":"Sat Sep 21 15:52:01 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.45900000000000002,"15min":0.44400000000000001},"min":{"1min":0.31,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.443,"15min":0.437},"min":{"1min":0.30499999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.58599999999999997,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.58999999999999997}]},{"osd":3,"last update":"Sat Sep 21 15:52:01 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.46500000000000002,"15min":0.45500000000000002},"min":{"1min":0.36299999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.46999999999999997,"15min":0.47099999999999997},"min":{"1min":0.33300000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.65600000000000003}]},{"osd":5,"last update":"Sat Sep 21 15:52:01 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.45900000000000002,"15min":0.47099999999999997},"min":{"1min":0.247,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.67200000000000004,"5min":2.4489999999999998,"15min":2.4489999999999998},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.434,"15min":0.42799999999999999},"min":{"1min":0.307,"5min":0.222,"15min":0.222},"max":{"1min":0.67200000000000004,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.47799999999999998}]},{"osd":6,"last update":"Sat Sep 21 15:52:01 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.42899999999999999,"15min":0.41899999999999998},"min":{"1min":0.27900000000000003,"5min":0.245,"15min":0.245},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.496},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.54800000000000004,"15min":0.54200000000000004},"min":{"1min":0.31900000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.69799999999999995,"5min":4.1340000000000003,"15min":4.1340000000000003},"last":0.57899999999999996}]},{"osd":7,"last update":"Sat Sep 21 15:52:01 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.47199999999999998,"15min":0.44},"min":{"1min":0.313,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.70899999999999996,"5min":2.7599999999999998,"15min":2.7599999999999998},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52900000000000003,"15min":0.52300000000000002},"min":{"1min":0.29399999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.77000000000000002,"5min":2.016,"15min":2.016},"last":0.67400000000000004}]}]},{"osd":3,"up_from":13,"seq":55834574894,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59328,"kb_used_data":1976,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312512,"statfs":{"total":96636764160,"available":96576012288,"internally_reserved":0,"allocated":2023424,"data_stored":973041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":58718664},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.31,"5min":0.31,"15min":0.31},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"last":0.505}]},{"osd":1,"last update":"Sat Sep 21 15:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.29699999999999999,"5min":0.32600000000000001,"15min":0.34000000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.38900000000000001,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.35899999999999999,"15min":0.375},"min":{"1min":0.26600000000000001,"5min":0.192,"15min":0.192},"max":{"1min":0.42599999999999999,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.59099999999999997}]},{"osd":2,"last update":"Sat Sep 21 15:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.371,"15min":0.38900000000000001},"min":{"1min":0.253,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.40600000000000003,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.39700000000000002,"15min":0.41699999999999998},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.57999999999999996,"5min":0.755,"15min":0.755},"last":0.627}]},{"osd":4,"last update":"Sat Sep 21 15:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.48199999999999998,"15min":0.49299999999999999},"min":{"1min":0.29699999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.505,"15min":0.51300000000000001},"min":{"1min":0.379,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.64500000000000002}]},{"osd":5,"last update":"Sat Sep 21 15:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.501,"15min":0.52200000000000002},"min":{"1min":0.28399999999999997,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.70799999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.55100000000000005},"min":{"1min":0.36599999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.73099999999999998,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.80400000000000005}]},{"osd":6,"last update":"Sat Sep 21 15:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.51200000000000001,"15min":0.53700000000000003},"min":{"1min":0.33400000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.623,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.52100000000000002,"15min":0.54300000000000004},"min":{"1min":0.36199999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.75700000000000001,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.71899999999999997}]},{"osd":7,"last update":"Sat Sep 21 15:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.53700000000000003,"15min":0.55900000000000005},"min":{"1min":0.39100000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.69999999999999996,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.53400000000000003,"15min":0.55800000000000005},"min":{"1min":0.36499999999999999,"5min":0.314,"15min":0.314},"max":{"1min":0.77600000000000002,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.61499999999999999}]}]},{"osd":2,"up_from":13,"seq":55834574894,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37184,"kb_used_data":1976,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334656,"statfs":{"total":96636764160,"available":96598687744,"internally_reserved":0,"allocated":2023424,"data_stored":973041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:29 2024","interfaces":[{"interface":"back","average":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.439,"5min":0.439,"15min":0.439},"last":0.223},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.26800000000000002}]},{"osd":1,"last update":"Sat Sep 21 15:51:57 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.377,"15min":0.41099999999999998},"min":{"1min":0.218,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.49199999999999999,"5min":1.621,"15min":1.621},"last":0.247},{"interface":"front","average":{"1min":0.313,"5min":0.35099999999999998,"15min":0.36899999999999999},"min":{"1min":0.23799999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.48299999999999998,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.316}]},{"osd":3,"last update":"Sat Sep 21 15:51:57 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.77800000000000002,"15min":0.92800000000000005},"min":{"1min":0.25700000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.54900000000000004,"5min":11.56,"15min":11.56},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.371,"5min":0.39200000000000002,"15min":0.40999999999999998},"min":{"1min":0.30399999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.51900000000000002,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.35999999999999999}]},{"osd":4,"last update":"Sat Sep 21 15:51:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.56699999999999995,"15min":0.59799999999999998},"min":{"1min":0.39800000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.76100000000000001,"5min":1.653,"15min":1.653},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.58399999999999996,"15min":0.622},"min":{"1min":0.39900000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.749,"5min":2.4660000000000002,"15min":2.4660000000000002},"last":0.71199999999999997}]},{"osd":5,"last update":"Sat Sep 21 15:51:57 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.496,"15min":0.50900000000000001},"min":{"1min":0.372,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.72999999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.496,"5min":0.51700000000000002,"15min":0.53600000000000003},"min":{"1min":0.36499999999999999,"5min":0.309,"15min":0.309},"max":{"1min":0.70599999999999996,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.69399999999999995}]},{"osd":6,"last update":"Sat Sep 21 15:51:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.53800000000000003,"15min":0.55900000000000005},"min":{"1min":0.34899999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.79700000000000004,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53400000000000003,"15min":0.55400000000000005},"min":{"1min":0.35099999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.76700000000000002,"5min":0.873,"15min":0.873},"last":0.63600000000000001}]},{"osd":7,"last update":"Sat Sep 21 15:51:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.53700000000000003,"15min":0.55700000000000005},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.76900000000000002,"5min":0.876,"15min":0.876},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57599999999999996,"15min":0.59599999999999997},"min":{"1min":0.38200000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.79600000000000004,"5min":1.089,"15min":1.089},"last":0.76100000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574893,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41288,"kb_used_data":1984,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94330552,"statfs":{"total":96636764160,"available":96594485248,"internally_reserved":0,"allocated":2031616,"data_stored":973079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.312,"15min":0.312},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"last":0.315},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.42799999999999999}]},{"osd":2,"last update":"Sat Sep 21 15:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.33100000000000002,"15min":0.34000000000000002},"min":{"1min":0.218,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.66500000000000004,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.496},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.46100000000000002,"15min":0.51900000000000002},"min":{"1min":0.24399999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.49399999999999999,"5min":5.0670000000000002,"15min":5.0670000000000002},"last":0.39800000000000002}]},{"osd":3,"last update":"Sat Sep 21 15:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.78000000000000003,"15min":0.97299999999999998},"min":{"1min":0.27200000000000002,"5min":0.218,"15min":0.218},"max":{"1min":0.625,"5min":16.032,"15min":16.032},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.35599999999999998,"15min":0.35899999999999999},"min":{"1min":0.252,"5min":0.186,"15min":0.186},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.45800000000000002}]},{"osd":4,"last update":"Sat Sep 21 15:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.46999999999999997,"15min":0.47399999999999998},"min":{"1min":0.36299999999999999,"5min":0.251,"15min":0.251},"max":{"1min":0.61799999999999999,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53100000000000003,"15min":0.54900000000000004},"min":{"1min":0.375,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.752,"5min":2.2269999999999999,"15min":2.2269999999999999},"last":0.71499999999999997}]},{"osd":5,"last update":"Sat Sep 21 15:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.53800000000000003,"15min":0.56799999999999995},"min":{"1min":0.34799999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.72699999999999998,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.623,"15min":0.68300000000000005},"min":{"1min":0.35299999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68200000000000005,"5min":4.6989999999999998,"15min":4.6989999999999998},"last":0.65900000000000003}]},{"osd":6,"last update":"Sat Sep 21 15:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.52800000000000002,"15min":0.53800000000000003},"min":{"1min":0.41199999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.68999999999999995,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.58299999999999996,"15min":0.61499999999999999},"min":{"1min":0.378,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.70099999999999996,"5min":2.262,"15min":2.262},"last":0.68500000000000005}]},{"osd":7,"last update":"Sat Sep 21 15:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.63200000000000001,"15min":0.69199999999999995},"min":{"1min":0.36499999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.71099999999999997,"5min":4.742,"15min":4.742},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.51500000000000001,"15min":0.51900000000000002},"min":{"1min":0.38,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.65500000000000003,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.64700000000000002}]}]},{"osd":0,"up_from":93,"seq":399431958551,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39172,"kb_used_data":2556,"kb_used_omap":13,"kb_used_meta":36594,"kb_avail":94332668,"statfs":{"total":96636764160,"available":96596652032,"internally_reserved":0,"allocated":2617344,"data_stored":1563409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13968,"internal_metadata":37472624},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.317,"5min":0.317,"15min":0.317},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"last":0.42199999999999999}]},{"osd":2,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.30199999999999999}]},{"osd":3,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.505,"5min":0.505,"15min":0.505},"last":0.33700000000000002}]},{"osd":4,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.60899999999999999}]},{"osd":5,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.66500000000000004}]},{"osd":6,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.747},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.63500000000000001}]},{"osd":7,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.71599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5973,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T15:52:14.875 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2024-09-21T15:49:05.663715+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=49, tm_sec=5, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=51, tm_sec=49, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:52:14.875 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2024-09-21T15:49:05.663715+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=49, tm_sec=5, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=51, tm_sec=49, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:52:14.875 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2024-09-21T15:49:05.663715+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=49, tm_sec=5, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=51, tm_sec=49, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:52:14.875 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2024-09-21T15:49:05.663715+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=49, tm_sec=5, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=51, tm_sec=49, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:52:14.875 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-21T15:48:56.568162+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=48, tm_sec=56, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=51, tm_sec=49, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:52:14.875 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2024-09-21T15:49:05.663715+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=49, tm_sec=5, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=51, tm_sec=49, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:52:14.876 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2024-09-21T15:49:05.663715+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=49, tm_sec=5, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=51, tm_sec=49, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:52:14.876 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2024-09-21T15:49:05.663715+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=49, tm_sec=5, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=51, tm_sec=49, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:52:14.876 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2024-09-21T15:49:05.663715+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=49, tm_sec=5, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=51, tm_sec=49, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:52:14.876 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2024-09-21T15:49:05.663715+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=49, tm_sec=5, tm_wday=5, tm_yday=265, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=21, tm_hour=15, tm_min=51, tm_sec=49, tm_wday=5, tm_yday=265, tm_isdst=0) 2024-09-21T15:52:14.876 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-21T15:52:34.877 DEBUG:teuthology.orchestra.run.smithi089:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-21T15:52:35.167 INFO:teuthology.orchestra.run.smithi089.stdout: 2024-09-21T15:52:35.167 INFO:teuthology.orchestra.run.smithi089.stderr:dumped all 2024-09-21T15:52:35.179 INFO:teuthology.orchestra.run.smithi089.stdout:{"pg_ready":true,"pg_map":{"version":273,"stamp":"2024-09-21T15:52:33.929017+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590406,"num_objects":1043,"num_object_clones":0,"num_object_copies":2086,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1043,"num_whiteouts":1035,"num_read":207535,"num_read_kb":842523,"num_write":23550,"num_write_kb":305657,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3397,"num_bytes_recovered":100270055,"num_keys_recovered":316,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21381,"ondisk_log_size":21381,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":420628,"kb_used_data":16980,"kb_used_omap":24,"kb_used_meta":403559,"kb_avail":754554092,"statfs":{"total":773094113280,"available":772663390208,"internally_reserved":0,"allocated":17387520,"data_stored":8963858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":25118,"internal_metadata":413244898},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003148"},"pg_stats":[{"pgid":"3.1f","version":"168'441","reported_seq":2175,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:59.256650+0000","last_change":"2024-09-21T15:51:59.256650+0000","last_active":"2024-09-21T15:51:59.256650+0000","last_peered":"2024-09-21T15:51:59.256650+0000","last_clean":"2024-09-21T15:51:59.256650+0000","last_became_active":"2024-09-21T15:50:32.670964+0000","last_became_peered":"2024-09-21T15:50:32.670964+0000","last_unstale":"2024-09-21T15:51:59.256650+0000","last_undegraded":"2024-09-21T15:51:59.256650+0000","last_fullsized":"2024-09-21T15:51:59.256650+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'441","last_scrub_stamp":"2024-09-21T15:51:59.256529+0000","last_deep_scrub":"168'441","last_deep_scrub_stamp":"2024-09-21T15:51:59.256529+0000","last_clean_scrub_stamp":"2024-09-21T15:51:59.256529+0000","objects_scrubbed":28,"log_size":441,"log_dups_size":0,"ondisk_log_size":441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:45:03.562826+0000","scrub_duration":0.022020432999999999,"objects_trimmed":0,"snaptrim_duration":0.0017896030000000001,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":28,"num_read":2413,"num_read_kb":10419,"num_write":442,"num_write_kb":5956,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":38,"num_bytes_recovered":1974433,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1e","version":"168'637","reported_seq":6578,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:54.369203+0000","last_change":"2024-09-21T15:51:54.369307+0000","last_active":"2024-09-21T15:51:54.369203+0000","last_peered":"2024-09-21T15:51:54.369203+0000","last_clean":"2024-09-21T15:51:54.369203+0000","last_became_active":"2024-09-21T15:51:24.691676+0000","last_became_peered":"2024-09-21T15:51:24.691676+0000","last_unstale":"2024-09-21T15:51:54.369203+0000","last_undegraded":"2024-09-21T15:51:54.369203+0000","last_fullsized":"2024-09-21T15:51:54.369203+0000","mapping_epoch":148,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":149,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'637","last_scrub_stamp":"2024-09-21T15:51:54.369180+0000","last_deep_scrub":"168'637","last_deep_scrub_stamp":"2024-09-21T15:51:54.369180+0000","last_clean_scrub_stamp":"2024-09-21T15:51:54.369180+0000","objects_scrubbed":38,"log_size":637,"log_dups_size":0,"ondisk_log_size":637,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:18:39.336637+0000","scrub_duration":0.026802408999999999,"objects_trimmed":0,"snaptrim_duration":0.0010326389999999999,"stat_sum":{"num_bytes":0,"num_objects":38,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":38,"num_read":7456,"num_read_kb":25912,"num_write":580,"num_write_kb":4361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":250,"num_bytes_recovered":3824749,"num_keys_recovered":34,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1d","version":"168'689","reported_seq":10263,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:52:26.423478+0000","last_change":"2024-09-21T15:52:26.423608+0000","last_active":"2024-09-21T15:52:26.423478+0000","last_peered":"2024-09-21T15:52:26.423478+0000","last_clean":"2024-09-21T15:52:26.423478+0000","last_became_active":"2024-09-21T15:51:39.765580+0000","last_became_peered":"2024-09-21T15:51:39.765580+0000","last_unstale":"2024-09-21T15:52:26.423478+0000","last_undegraded":"2024-09-21T15:52:26.423478+0000","last_fullsized":"2024-09-21T15:52:26.423478+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'689","last_scrub_stamp":"2024-09-21T15:52:26.423457+0000","last_deep_scrub":"168'689","last_deep_scrub_stamp":"2024-09-21T15:52:26.423457+0000","last_clean_scrub_stamp":"2024-09-21T15:52:26.423457+0000","objects_scrubbed":34,"log_size":689,"log_dups_size":0,"ondisk_log_size":689,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:21:33.994080+0000","scrub_duration":0.032385772,"objects_trimmed":0,"snaptrim_duration":0.00090302100000000005,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":34,"num_read":10756,"num_read_kb":29175,"num_write":703,"num_write_kb":8882,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"168'612","reported_seq":4562,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:52:19.422202+0000","last_change":"2024-09-21T15:52:19.422357+0000","last_active":"2024-09-21T15:52:19.422202+0000","last_peered":"2024-09-21T15:52:19.422202+0000","last_clean":"2024-09-21T15:52:19.422202+0000","last_became_active":"2024-09-21T15:51:39.765999+0000","last_became_peered":"2024-09-21T15:51:39.765999+0000","last_unstale":"2024-09-21T15:52:19.422202+0000","last_undegraded":"2024-09-21T15:52:19.422202+0000","last_fullsized":"2024-09-21T15:52:19.422202+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'612","last_scrub_stamp":"2024-09-21T15:52:19.422164+0000","last_deep_scrub":"168'612","last_deep_scrub_stamp":"2024-09-21T15:52:19.422164+0000","last_clean_scrub_stamp":"2024-09-21T15:52:19.422164+0000","objects_scrubbed":28,"log_size":612,"log_dups_size":0,"ondisk_log_size":612,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:04:23.753205+0000","scrub_duration":0.022674954000000001,"objects_trimmed":0,"snaptrim_duration":0.0010903670000000001,"stat_sum":{"num_bytes":0,"num_objects":28,"num_object_clones":0,"num_object_copies":56,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":27,"num_read":5242,"num_read_kb":17341,"num_write":644,"num_write_kb":4787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"168'547","reported_seq":4457,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:52:05.209753+0000","last_change":"2024-09-21T15:52:05.209861+0000","last_active":"2024-09-21T15:52:05.209753+0000","last_peered":"2024-09-21T15:52:05.209753+0000","last_clean":"2024-09-21T15:52:05.209753+0000","last_became_active":"2024-09-21T15:50:32.674250+0000","last_became_peered":"2024-09-21T15:50:32.674250+0000","last_unstale":"2024-09-21T15:52:05.209753+0000","last_undegraded":"2024-09-21T15:52:05.209753+0000","last_fullsized":"2024-09-21T15:52:05.209753+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'547","last_scrub_stamp":"2024-09-21T15:52:05.209731+0000","last_deep_scrub":"168'547","last_deep_scrub_stamp":"2024-09-21T15:52:05.209731+0000","last_clean_scrub_stamp":"2024-09-21T15:52:05.209731+0000","objects_scrubbed":39,"log_size":547,"log_dups_size":0,"ondisk_log_size":547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:42:51.469346+0000","scrub_duration":0.027420244,"objects_trimmed":0,"snaptrim_duration":0.0019320559999999999,"stat_sum":{"num_bytes":0,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":39,"num_read":4815,"num_read_kb":30088,"num_write":498,"num_write_kb":13507,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":4759000,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1a","version":"168'462","reported_seq":6989,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:53.750816+0000","last_change":"2024-09-21T15:51:53.750952+0000","last_active":"2024-09-21T15:51:53.750816+0000","last_peered":"2024-09-21T15:51:53.750816+0000","last_clean":"2024-09-21T15:51:53.750816+0000","last_became_active":"2024-09-21T15:49:06.693903+0000","last_became_peered":"2024-09-21T15:49:06.693903+0000","last_unstale":"2024-09-21T15:51:53.750816+0000","last_undegraded":"2024-09-21T15:51:53.750816+0000","last_fullsized":"2024-09-21T15:51:53.750816+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'462","last_scrub_stamp":"2024-09-21T15:51:53.750793+0000","last_deep_scrub":"168'462","last_deep_scrub_stamp":"2024-09-21T15:51:53.750793+0000","last_clean_scrub_stamp":"2024-09-21T15:51:53.750793+0000","objects_scrubbed":31,"log_size":462,"log_dups_size":0,"ondisk_log_size":462,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T22:59:20.012704+0000","scrub_duration":0.026832650999999999,"objects_trimmed":0,"snaptrim_duration":0.00047963299999999998,"stat_sum":{"num_bytes":19,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":30,"num_read":7221,"num_read_kb":31759,"num_write":421,"num_write_kb":9476,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.19","version":"168'492","reported_seq":8883,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:59.302184+0000","last_change":"2024-09-21T15:51:59.302291+0000","last_active":"2024-09-21T15:51:59.302184+0000","last_peered":"2024-09-21T15:51:59.302184+0000","last_clean":"2024-09-21T15:51:59.302184+0000","last_became_active":"2024-09-21T15:51:12.688258+0000","last_became_peered":"2024-09-21T15:51:12.688258+0000","last_unstale":"2024-09-21T15:51:59.302184+0000","last_undegraded":"2024-09-21T15:51:59.302184+0000","last_fullsized":"2024-09-21T15:51:59.302184+0000","mapping_epoch":136,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":137,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'492","last_scrub_stamp":"2024-09-21T15:51:59.302162+0000","last_deep_scrub":"168'492","last_deep_scrub_stamp":"2024-09-21T15:51:59.302162+0000","last_clean_scrub_stamp":"2024-09-21T15:51:59.302162+0000","objects_scrubbed":32,"log_size":492,"log_dups_size":0,"ondisk_log_size":492,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:27:39.629735+0000","scrub_duration":0.025551372999999999,"objects_trimmed":0,"snaptrim_duration":0.000411691,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":32,"num_read":8283,"num_read_kb":39579,"num_write":518,"num_write_kb":7408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":236,"num_bytes_recovered":6705120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"168'563","reported_seq":4756,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:52:00.310827+0000","last_change":"2024-09-21T15:52:00.310912+0000","last_active":"2024-09-21T15:52:00.310827+0000","last_peered":"2024-09-21T15:52:00.310827+0000","last_clean":"2024-09-21T15:52:00.310827+0000","last_became_active":"2024-09-21T15:51:35.751700+0000","last_became_peered":"2024-09-21T15:51:35.751700+0000","last_unstale":"2024-09-21T15:52:00.310827+0000","last_undegraded":"2024-09-21T15:52:00.310827+0000","last_fullsized":"2024-09-21T15:52:00.310827+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'563","last_scrub_stamp":"2024-09-21T15:52:00.310805+0000","last_deep_scrub":"168'563","last_deep_scrub_stamp":"2024-09-21T15:52:00.310805+0000","last_clean_scrub_stamp":"2024-09-21T15:52:00.310805+0000","objects_scrubbed":26,"log_size":563,"log_dups_size":0,"ondisk_log_size":563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:12:17.664579+0000","scrub_duration":0.025760488000000002,"objects_trimmed":0,"snaptrim_duration":0.0051035439999999998,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":26,"num_read":6190,"num_read_kb":30937,"num_write":534,"num_write_kb":11075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":282,"num_bytes_recovered":11320990,"num_keys_recovered":70,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"168'1076","reported_seq":9462,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:54.354291+0000","last_change":"2024-09-21T15:51:54.354423+0000","last_active":"2024-09-21T15:51:54.354291+0000","last_peered":"2024-09-21T15:51:54.354291+0000","last_clean":"2024-09-21T15:51:54.354291+0000","last_became_active":"2024-09-21T15:50:32.668771+0000","last_became_peered":"2024-09-21T15:50:32.668771+0000","last_unstale":"2024-09-21T15:51:54.354291+0000","last_undegraded":"2024-09-21T15:51:54.354291+0000","last_fullsized":"2024-09-21T15:51:54.354291+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'1076","last_scrub_stamp":"2024-09-21T15:51:54.354254+0000","last_deep_scrub":"168'1076","last_deep_scrub_stamp":"2024-09-21T15:51:54.354254+0000","last_clean_scrub_stamp":"2024-09-21T15:51:54.354254+0000","objects_scrubbed":36,"log_size":1076,"log_dups_size":0,"ondisk_log_size":1076,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:31:51.364904+0000","scrub_duration":0.028413487000000001,"objects_trimmed":0,"snaptrim_duration":0.001503861,"stat_sum":{"num_bytes":0,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":36,"num_read":12714,"num_read_kb":21487,"num_write":1303,"num_write_kb":8257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":3319185,"num_keys_recovered":34,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.16","version":"168'570","reported_seq":6761,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:58.112055+0000","last_change":"2024-09-21T15:51:58.112163+0000","last_active":"2024-09-21T15:51:58.112055+0000","last_peered":"2024-09-21T15:51:58.112055+0000","last_clean":"2024-09-21T15:51:58.112055+0000","last_became_active":"2024-09-21T15:49:06.688808+0000","last_became_peered":"2024-09-21T15:49:06.688808+0000","last_unstale":"2024-09-21T15:51:58.112055+0000","last_undegraded":"2024-09-21T15:51:58.112055+0000","last_fullsized":"2024-09-21T15:51:58.112055+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'570","last_scrub_stamp":"2024-09-21T15:51:58.112033+0000","last_deep_scrub":"168'570","last_deep_scrub_stamp":"2024-09-21T15:51:58.112033+0000","last_clean_scrub_stamp":"2024-09-21T15:51:58.112033+0000","objects_scrubbed":30,"log_size":570,"log_dups_size":0,"ondisk_log_size":570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:23:50.540236+0000","scrub_duration":0.021872452000000001,"objects_trimmed":0,"snaptrim_duration":0.00023944,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":30,"num_read":7604,"num_read_kb":49865,"num_write":542,"num_write_kb":13823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.15","version":"168'559","reported_seq":4368,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:52:28.378699+0000","last_change":"2024-09-21T15:52:28.378808+0000","last_active":"2024-09-21T15:52:28.378699+0000","last_peered":"2024-09-21T15:52:28.378699+0000","last_clean":"2024-09-21T15:52:28.378699+0000","last_became_active":"2024-09-21T15:51:11.841664+0000","last_became_peered":"2024-09-21T15:51:11.841664+0000","last_unstale":"2024-09-21T15:52:28.378699+0000","last_undegraded":"2024-09-21T15:52:28.378699+0000","last_fullsized":"2024-09-21T15:52:28.378699+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'559","last_scrub_stamp":"2024-09-21T15:52:28.378676+0000","last_deep_scrub":"168'559","last_deep_scrub_stamp":"2024-09-21T15:52:28.378676+0000","last_clean_scrub_stamp":"2024-09-21T15:52:28.378676+0000","objects_scrubbed":29,"log_size":559,"log_dups_size":0,"ondisk_log_size":559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:45:21.364974+0000","scrub_duration":0.020193187000000001,"objects_trimmed":0,"snaptrim_duration":0.001532828,"stat_sum":{"num_bytes":0,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":29,"num_read":4052,"num_read_kb":27144,"num_write":637,"num_write_kb":17935,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":222,"num_bytes_recovered":4468215,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.14","version":"168'1233","reported_seq":4428,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:57.725316+0000","last_change":"2024-09-21T15:51:57.725420+0000","last_active":"2024-09-21T15:51:57.725316+0000","last_peered":"2024-09-21T15:51:57.725316+0000","last_clean":"2024-09-21T15:51:57.725316+0000","last_became_active":"2024-09-21T15:49:06.688803+0000","last_became_peered":"2024-09-21T15:49:06.688803+0000","last_unstale":"2024-09-21T15:51:57.725316+0000","last_undegraded":"2024-09-21T15:51:57.725316+0000","last_fullsized":"2024-09-21T15:51:57.725316+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'1233","last_scrub_stamp":"2024-09-21T15:51:57.725294+0000","last_deep_scrub":"168'1233","last_deep_scrub_stamp":"2024-09-21T15:51:57.725294+0000","last_clean_scrub_stamp":"2024-09-21T15:51:57.725294+0000","objects_scrubbed":26,"log_size":1233,"log_dups_size":0,"ondisk_log_size":1233,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:40:56.930628+0000","scrub_duration":0.023941679,"objects_trimmed":0,"snaptrim_duration":0.0013586710000000001,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":26,"num_read":7764,"num_read_kb":18232,"num_write":1926,"num_write_kb":6190,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":173,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:52:02.022929+0000","last_change":"2024-09-21T15:52:02.023057+0000","last_active":"2024-09-21T15:52:02.022929+0000","last_peered":"2024-09-21T15:52:02.022929+0000","last_clean":"2024-09-21T15:52:02.022929+0000","last_became_active":"2024-09-21T15:48:57.583665+0000","last_became_peered":"2024-09-21T15:48:57.583665+0000","last_unstale":"2024-09-21T15:52:02.022929+0000","last_undegraded":"2024-09-21T15:52:02.022929+0000","last_fullsized":"2024-09-21T15:52:02.022929+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:52:02.022908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:52:02.022908+0000","last_clean_scrub_stamp":"2024-09-21T15:52:02.022908+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:54:37.398705+0000","scrub_duration":0.0014227129999999999,"objects_trimmed":0,"snaptrim_duration":0.00031098399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"168'473","reported_seq":3113,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:53.387345+0000","last_change":"2024-09-21T15:51:53.387481+0000","last_active":"2024-09-21T15:51:53.387345+0000","last_peered":"2024-09-21T15:51:53.387345+0000","last_clean":"2024-09-21T15:51:53.387345+0000","last_became_active":"2024-09-21T15:50:32.671424+0000","last_became_peered":"2024-09-21T15:50:32.671424+0000","last_unstale":"2024-09-21T15:51:53.387345+0000","last_undegraded":"2024-09-21T15:51:53.387345+0000","last_fullsized":"2024-09-21T15:51:53.387345+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'473","last_scrub_stamp":"2024-09-21T15:51:53.387307+0000","last_deep_scrub":"168'473","last_deep_scrub_stamp":"2024-09-21T15:51:53.387307+0000","last_clean_scrub_stamp":"2024-09-21T15:51:53.387307+0000","objects_scrubbed":35,"log_size":473,"log_dups_size":0,"ondisk_log_size":473,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:48:07.248726+0000","scrub_duration":0.02547369,"objects_trimmed":0,"snaptrim_duration":0.001507467,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":35,"num_read":2637,"num_read_kb":12956,"num_write":388,"num_write_kb":8392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":49,"num_bytes_recovered":4494388,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":177,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:56.296589+0000","last_change":"2024-09-21T15:51:56.296699+0000","last_active":"2024-09-21T15:51:56.296589+0000","last_peered":"2024-09-21T15:51:56.296589+0000","last_clean":"2024-09-21T15:51:56.296589+0000","last_became_active":"2024-09-21T15:48:57.582121+0000","last_became_peered":"2024-09-21T15:48:57.582121+0000","last_unstale":"2024-09-21T15:51:56.296589+0000","last_undegraded":"2024-09-21T15:51:56.296589+0000","last_fullsized":"2024-09-21T15:51:56.296589+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:51:56.296569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:51:56.296569+0000","last_clean_scrub_stamp":"2024-09-21T15:51:56.296569+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:31:39.212030+0000","scrub_duration":0.001405795,"objects_trimmed":0,"snaptrim_duration":0.00039654700000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"168'614","reported_seq":8050,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:57.293683+0000","last_change":"2024-09-21T15:51:57.293824+0000","last_active":"2024-09-21T15:51:57.293683+0000","last_peered":"2024-09-21T15:51:57.293683+0000","last_clean":"2024-09-21T15:51:57.293683+0000","last_became_active":"2024-09-21T15:51:35.752700+0000","last_became_peered":"2024-09-21T15:51:35.752700+0000","last_unstale":"2024-09-21T15:51:57.293683+0000","last_undegraded":"2024-09-21T15:51:57.293683+0000","last_fullsized":"2024-09-21T15:51:57.293683+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'614","last_scrub_stamp":"2024-09-21T15:51:57.293645+0000","last_deep_scrub":"168'614","last_deep_scrub_stamp":"2024-09-21T15:51:57.293645+0000","last_clean_scrub_stamp":"2024-09-21T15:51:57.293645+0000","objects_scrubbed":35,"log_size":614,"log_dups_size":0,"ondisk_log_size":614,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:25:01.117384+0000","scrub_duration":0.024979000000000001,"objects_trimmed":2,"snaptrim_duration":0.00086244900000000005,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":35,"num_read":8380,"num_read_kb":21195,"num_write":544,"num_write_kb":5397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":256,"num_bytes_recovered":3760369,"num_keys_recovered":11,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":188,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:52:02.454791+0000","last_change":"2024-09-21T15:52:02.454892+0000","last_active":"2024-09-21T15:52:02.454791+0000","last_peered":"2024-09-21T15:52:02.454791+0000","last_clean":"2024-09-21T15:52:02.454791+0000","last_became_active":"2024-09-21T15:50:30.146428+0000","last_became_peered":"2024-09-21T15:50:30.146428+0000","last_unstale":"2024-09-21T15:52:02.454791+0000","last_undegraded":"2024-09-21T15:52:02.454791+0000","last_fullsized":"2024-09-21T15:52:02.454791+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:52:02.454771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:52:02.454771+0000","last_clean_scrub_stamp":"2024-09-21T15:52:02.454771+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:39:26.983338+0000","scrub_duration":0.0018950830000000001,"objects_trimmed":0,"snaptrim_duration":0.00036074600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"168'729","reported_seq":8894,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:52:05.353732+0000","last_change":"2024-09-21T15:52:05.353857+0000","last_active":"2024-09-21T15:52:05.353732+0000","last_peered":"2024-09-21T15:52:05.353732+0000","last_clean":"2024-09-21T15:52:05.353732+0000","last_became_active":"2024-09-21T15:50:35.682788+0000","last_became_peered":"2024-09-21T15:50:35.682788+0000","last_unstale":"2024-09-21T15:52:05.353732+0000","last_undegraded":"2024-09-21T15:52:05.353732+0000","last_fullsized":"2024-09-21T15:52:05.353732+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'729","last_scrub_stamp":"2024-09-21T15:52:05.353709+0000","last_deep_scrub":"168'729","last_deep_scrub_stamp":"2024-09-21T15:52:05.353709+0000","last_clean_scrub_stamp":"2024-09-21T15:52:05.353709+0000","objects_scrubbed":26,"log_size":729,"log_dups_size":0,"ondisk_log_size":729,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:51:06.196186+0000","scrub_duration":0.026638608000000001,"objects_trimmed":0,"snaptrim_duration":0.00032907100000000001,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":26,"num_read":11015,"num_read_kb":27666,"num_write":868,"num_write_kb":11017,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":175,"num_bytes_recovered":14134268,"num_keys_recovered":36,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":190,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:58.268172+0000","last_change":"2024-09-21T15:51:58.268273+0000","last_active":"2024-09-21T15:51:58.268172+0000","last_peered":"2024-09-21T15:51:58.268172+0000","last_clean":"2024-09-21T15:51:58.268172+0000","last_became_active":"2024-09-21T15:50:30.146273+0000","last_became_peered":"2024-09-21T15:50:30.146273+0000","last_unstale":"2024-09-21T15:51:58.268172+0000","last_undegraded":"2024-09-21T15:51:58.268172+0000","last_fullsized":"2024-09-21T15:51:58.268172+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:51:58.268153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:51:58.268153+0000","last_clean_scrub_stamp":"2024-09-21T15:51:58.268153+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:38:41.425777+0000","scrub_duration":0.0014527979999999999,"objects_trimmed":0,"snaptrim_duration":0.00040525700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"168'638","reported_seq":5426,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:52:17.371189+0000","last_change":"2024-09-21T15:52:17.371291+0000","last_active":"2024-09-21T15:52:17.371189+0000","last_peered":"2024-09-21T15:52:17.371189+0000","last_clean":"2024-09-21T15:52:17.371189+0000","last_became_active":"2024-09-21T15:51:39.768300+0000","last_became_peered":"2024-09-21T15:51:39.768300+0000","last_unstale":"2024-09-21T15:52:17.371189+0000","last_undegraded":"2024-09-21T15:52:17.371189+0000","last_fullsized":"2024-09-21T15:52:17.371189+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'638","last_scrub_stamp":"2024-09-21T15:52:17.371169+0000","last_deep_scrub":"168'638","last_deep_scrub_stamp":"2024-09-21T15:52:17.371169+0000","last_clean_scrub_stamp":"2024-09-21T15:52:17.371169+0000","objects_scrubbed":35,"log_size":638,"log_dups_size":0,"ondisk_log_size":638,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:17:57.567104+0000","scrub_duration":0.028450821000000001,"objects_trimmed":0,"snaptrim_duration":0.00050855200000000001,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":35,"num_read":5730,"num_read_kb":33940,"num_write":686,"num_write_kb":16332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":268,"num_bytes_recovered":7823491,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"168'956","reported_seq":4245,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:56.688731+0000","last_change":"2024-09-21T15:51:56.688851+0000","last_active":"2024-09-21T15:51:56.688731+0000","last_peered":"2024-09-21T15:51:56.688731+0000","last_clean":"2024-09-21T15:51:56.688731+0000","last_became_active":"2024-09-21T15:50:32.677456+0000","last_became_peered":"2024-09-21T15:50:32.677456+0000","last_unstale":"2024-09-21T15:51:56.688731+0000","last_undegraded":"2024-09-21T15:51:56.688731+0000","last_fullsized":"2024-09-21T15:51:56.688731+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'956","last_scrub_stamp":"2024-09-21T15:51:56.688706+0000","last_deep_scrub":"168'956","last_deep_scrub_stamp":"2024-09-21T15:51:56.688706+0000","last_clean_scrub_stamp":"2024-09-21T15:51:56.688706+0000","objects_scrubbed":35,"log_size":956,"log_dups_size":0,"ondisk_log_size":956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T21:34:17.308503+0000","scrub_duration":0.033507282999999999,"objects_trimmed":0,"snaptrim_duration":0.0010737730000000001,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":34,"num_read":5136,"num_read_kb":19340,"num_write":949,"num_write_kb":7405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":632176,"num_keys_recovered":24,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"168'3","reported_seq":73,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:59.086482+0000","last_change":"2024-09-21T15:51:59.086589+0000","last_active":"2024-09-21T15:51:59.086482+0000","last_peered":"2024-09-21T15:51:59.086482+0000","last_clean":"2024-09-21T15:51:59.086482+0000","last_became_active":"2024-09-21T15:51:39.764409+0000","last_became_peered":"2024-09-21T15:51:39.764409+0000","last_unstale":"2024-09-21T15:51:59.086482+0000","last_undegraded":"2024-09-21T15:51:59.086482+0000","last_fullsized":"2024-09-21T15:51:59.086482+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'3","last_scrub_stamp":"2024-09-21T15:51:59.086459+0000","last_deep_scrub":"168'3","last_deep_scrub_stamp":"2024-09-21T15:51:59.086459+0000","last_clean_scrub_stamp":"2024-09-21T15:51:59.086459+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:04:58.492791+0000","scrub_duration":0.0036796239999999998,"objects_trimmed":0,"snaptrim_duration":0.000859595,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"168'424","reported_seq":4879,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:52:00.278311+0000","last_change":"2024-09-21T15:52:00.278425+0000","last_active":"2024-09-21T15:52:00.278311+0000","last_peered":"2024-09-21T15:52:00.278311+0000","last_clean":"2024-09-21T15:52:00.278311+0000","last_became_active":"2024-09-21T15:50:35.682001+0000","last_became_peered":"2024-09-21T15:50:35.682001+0000","last_unstale":"2024-09-21T15:52:00.278311+0000","last_undegraded":"2024-09-21T15:52:00.278311+0000","last_fullsized":"2024-09-21T15:52:00.278311+0000","mapping_epoch":99,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":100,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'424","last_scrub_stamp":"2024-09-21T15:52:00.278288+0000","last_deep_scrub":"168'424","last_deep_scrub_stamp":"2024-09-21T15:52:00.278288+0000","last_clean_scrub_stamp":"2024-09-21T15:52:00.278288+0000","objects_scrubbed":31,"log_size":424,"log_dups_size":0,"ondisk_log_size":424,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:16:30.872951+0000","scrub_duration":0.023884934999999999,"objects_trimmed":0,"snaptrim_duration":0.00044281000000000002,"stat_sum":{"num_bytes":0,"num_objects":31,"num_object_clones":0,"num_object_copies":62,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":31,"num_whiteouts":31,"num_read":4285,"num_read_kb":23375,"num_write":379,"num_write_kb":4145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":187,"num_bytes_recovered":4305276,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":109,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:52.415337+0000","last_change":"2024-09-21T15:51:52.415462+0000","last_active":"2024-09-21T15:51:52.415337+0000","last_peered":"2024-09-21T15:51:52.415337+0000","last_clean":"2024-09-21T15:51:52.415337+0000","last_became_active":"2024-09-21T15:51:07.683418+0000","last_became_peered":"2024-09-21T15:51:07.683418+0000","last_unstale":"2024-09-21T15:51:52.415337+0000","last_undegraded":"2024-09-21T15:51:52.415337+0000","last_fullsized":"2024-09-21T15:51:52.415337+0000","mapping_epoch":131,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":132,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:51:52.415306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:51:52.415306+0000","last_clean_scrub_stamp":"2024-09-21T15:51:52.415306+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:28:29.927181+0000","scrub_duration":0.0013079089999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1","version":"168'597","reported_seq":2652,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:56.283724+0000","last_change":"2024-09-21T15:51:56.283724+0000","last_active":"2024-09-21T15:51:56.283724+0000","last_peered":"2024-09-21T15:51:56.283724+0000","last_clean":"2024-09-21T15:51:56.283724+0000","last_became_active":"2024-09-21T15:50:32.671413+0000","last_became_peered":"2024-09-21T15:50:32.671413+0000","last_unstale":"2024-09-21T15:51:56.283724+0000","last_undegraded":"2024-09-21T15:51:56.283724+0000","last_fullsized":"2024-09-21T15:51:56.283724+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'597","last_scrub_stamp":"2024-09-21T15:51:56.283573+0000","last_deep_scrub":"168'597","last_deep_scrub_stamp":"2024-09-21T15:51:56.283573+0000","last_clean_scrub_stamp":"2024-09-21T15:51:56.283573+0000","objects_scrubbed":39,"log_size":597,"log_dups_size":0,"ondisk_log_size":597,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:35:45.562010+0000","scrub_duration":0.027644431000000001,"objects_trimmed":0,"snaptrim_duration":0.001292872,"stat_sum":{"num_bytes":0,"num_objects":39,"num_object_clones":0,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":39,"num_read":3663,"num_read_kb":16543,"num_write":590,"num_write_kb":5977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":1696547,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":175,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:58.512849+0000","last_change":"2024-09-21T15:51:58.512993+0000","last_active":"2024-09-21T15:51:58.512849+0000","last_peered":"2024-09-21T15:51:58.512849+0000","last_clean":"2024-09-21T15:51:58.512849+0000","last_became_active":"2024-09-21T15:48:57.585525+0000","last_became_peered":"2024-09-21T15:48:57.585525+0000","last_unstale":"2024-09-21T15:51:58.512849+0000","last_undegraded":"2024-09-21T15:51:58.512849+0000","last_fullsized":"2024-09-21T15:51:58.512849+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T15:51:58.512800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T15:51:58.512800+0000","last_clean_scrub_stamp":"2024-09-21T15:51:58.512800+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T01:59:12.580616+0000","scrub_duration":0.001937846,"objects_trimmed":0,"snaptrim_duration":0.00031691100000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"168'443","reported_seq":3001,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:55.355408+0000","last_change":"2024-09-21T15:51:55.355505+0000","last_active":"2024-09-21T15:51:55.355408+0000","last_peered":"2024-09-21T15:51:55.355408+0000","last_clean":"2024-09-21T15:51:55.355408+0000","last_became_active":"2024-09-21T15:51:35.752810+0000","last_became_peered":"2024-09-21T15:51:35.752810+0000","last_unstale":"2024-09-21T15:51:55.355408+0000","last_undegraded":"2024-09-21T15:51:55.355408+0000","last_fullsized":"2024-09-21T15:51:55.355408+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'443","last_scrub_stamp":"2024-09-21T15:51:55.355386+0000","last_deep_scrub":"168'443","last_deep_scrub_stamp":"2024-09-21T15:51:55.355386+0000","last_clean_scrub_stamp":"2024-09-21T15:51:55.355386+0000","objects_scrubbed":30,"log_size":443,"log_dups_size":0,"ondisk_log_size":443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T02:00:45.418578+0000","scrub_duration":0.022450193,"objects_trimmed":0,"snaptrim_duration":0.00033178999999999999,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":30,"num_read":2149,"num_read_kb":34452,"num_write":406,"num_write_kb":12612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":214,"num_bytes_recovered":2197196,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"168'2","reported_seq":72,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:52:13.302004+0000","last_change":"2024-09-21T15:52:13.302109+0000","last_active":"2024-09-21T15:52:13.302004+0000","last_peered":"2024-09-21T15:52:13.302004+0000","last_clean":"2024-09-21T15:52:13.302004+0000","last_became_active":"2024-09-21T15:51:39.764411+0000","last_became_peered":"2024-09-21T15:51:39.764411+0000","last_unstale":"2024-09-21T15:52:13.302004+0000","last_undegraded":"2024-09-21T15:52:13.302004+0000","last_fullsized":"2024-09-21T15:52:13.302004+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":17,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'2","last_scrub_stamp":"2024-09-21T15:52:13.301981+0000","last_deep_scrub":"168'2","last_deep_scrub_stamp":"2024-09-21T15:52:13.301981+0000","last_clean_scrub_stamp":"2024-09-21T15:52:13.301981+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T00:16:44.404892+0000","scrub_duration":0.0034943449999999998,"objects_trimmed":0,"snaptrim_duration":0.00039559000000000003,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"168'119","reported_seq":338,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:57.510040+0000","last_change":"2024-09-21T15:51:57.510169+0000","last_active":"2024-09-21T15:51:57.510040+0000","last_peered":"2024-09-21T15:51:57.510040+0000","last_clean":"2024-09-21T15:51:57.510040+0000","last_became_active":"2024-09-21T15:50:30.146987+0000","last_became_peered":"2024-09-21T15:50:30.146987+0000","last_unstale":"2024-09-21T15:51:57.510040+0000","last_undegraded":"2024-09-21T15:51:57.510040+0000","last_fullsized":"2024-09-21T15:51:57.510040+0000","mapping_epoch":93,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":94,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'119","last_scrub_stamp":"2024-09-21T15:51:57.510007+0000","last_deep_scrub":"168'119","last_deep_scrub_stamp":"2024-09-21T15:51:57.510007+0000","last_clean_scrub_stamp":"2024-09-21T15:51:57.510007+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:47:31.553607+0000","scrub_duration":0.0055273869999999999,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"168'572","reported_seq":7990,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:53.395767+0000","last_change":"2024-09-21T15:51:53.395872+0000","last_active":"2024-09-21T15:51:53.395767+0000","last_peered":"2024-09-21T15:51:53.395767+0000","last_clean":"2024-09-21T15:51:53.395767+0000","last_became_active":"2024-09-21T15:51:11.837877+0000","last_became_peered":"2024-09-21T15:51:11.837877+0000","last_unstale":"2024-09-21T15:51:53.395767+0000","last_undegraded":"2024-09-21T15:51:53.395767+0000","last_fullsized":"2024-09-21T15:51:53.395767+0000","mapping_epoch":135,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":136,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'572","last_scrub_stamp":"2024-09-21T15:51:53.395748+0000","last_deep_scrub":"168'572","last_deep_scrub_stamp":"2024-09-21T15:51:53.395748+0000","last_clean_scrub_stamp":"2024-09-21T15:51:53.395748+0000","objects_scrubbed":32,"log_size":572,"log_dups_size":0,"ondisk_log_size":572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:05:31.768212+0000","scrub_duration":0.029579207999999999,"objects_trimmed":0,"snaptrim_duration":0.00079839600000000004,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":32,"num_read":7239,"num_read_kb":29580,"num_write":558,"num_write_kb":4793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":418,"num_bytes_recovered":6262707,"num_keys_recovered":24,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"168'419","reported_seq":3215,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:55.697626+0000","last_change":"2024-09-21T15:51:55.697770+0000","last_active":"2024-09-21T15:51:55.697626+0000","last_peered":"2024-09-21T15:51:55.697626+0000","last_clean":"2024-09-21T15:51:55.697626+0000","last_became_active":"2024-09-21T15:49:07.210506+0000","last_became_peered":"2024-09-21T15:49:07.210506+0000","last_unstale":"2024-09-21T15:51:55.697626+0000","last_undegraded":"2024-09-21T15:51:55.697626+0000","last_fullsized":"2024-09-21T15:51:55.697626+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'419","last_scrub_stamp":"2024-09-21T15:51:55.697604+0000","last_deep_scrub":"168'419","last_deep_scrub_stamp":"2024-09-21T15:51:55.697604+0000","last_clean_scrub_stamp":"2024-09-21T15:51:55.697604+0000","objects_scrubbed":30,"log_size":419,"log_dups_size":0,"ondisk_log_size":419,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:57:05.404395+0000","scrub_duration":0.022683354999999999,"objects_trimmed":0,"snaptrim_duration":0.00089193900000000001,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":30,"num_read":2716,"num_read_kb":28245,"num_write":454,"num_write_kb":12090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"168'625","reported_seq":12311,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:52:09.055828+0000","last_change":"2024-09-21T15:52:09.055935+0000","last_active":"2024-09-21T15:52:09.055828+0000","last_peered":"2024-09-21T15:52:09.055828+0000","last_clean":"2024-09-21T15:52:09.055828+0000","last_became_active":"2024-09-21T15:51:41.781781+0000","last_became_peered":"2024-09-21T15:51:41.781781+0000","last_unstale":"2024-09-21T15:52:09.055828+0000","last_undegraded":"2024-09-21T15:52:09.055828+0000","last_fullsized":"2024-09-21T15:52:09.055828+0000","mapping_epoch":164,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":165,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'625","last_scrub_stamp":"2024-09-21T15:52:09.055804+0000","last_deep_scrub":"168'625","last_deep_scrub_stamp":"2024-09-21T15:52:09.055804+0000","last_clean_scrub_stamp":"2024-09-21T15:52:09.055804+0000","objects_scrubbed":30,"log_size":625,"log_dups_size":0,"ondisk_log_size":625,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T20:23:27.790891+0000","scrub_duration":0.030555435999999998,"objects_trimmed":0,"snaptrim_duration":0.000309432,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":29,"num_read":12828,"num_read_kb":63467,"num_write":580,"num_write_kb":14212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.b","version":"168'877","reported_seq":3895,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:58.296997+0000","last_change":"2024-09-21T15:51:58.297106+0000","last_active":"2024-09-21T15:51:58.296997+0000","last_peered":"2024-09-21T15:51:58.296997+0000","last_clean":"2024-09-21T15:51:58.296997+0000","last_became_active":"2024-09-21T15:51:35.751150+0000","last_became_peered":"2024-09-21T15:51:35.751150+0000","last_unstale":"2024-09-21T15:51:58.296997+0000","last_undegraded":"2024-09-21T15:51:58.296997+0000","last_fullsized":"2024-09-21T15:51:58.296997+0000","mapping_epoch":158,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'877","last_scrub_stamp":"2024-09-21T15:51:58.296964+0000","last_deep_scrub":"168'877","last_deep_scrub_stamp":"2024-09-21T15:51:58.296964+0000","last_clean_scrub_stamp":"2024-09-21T15:51:58.296964+0000","objects_scrubbed":26,"log_size":877,"log_dups_size":0,"ondisk_log_size":877,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:39:41.972358+0000","scrub_duration":0.022677787000000001,"objects_trimmed":0,"snaptrim_duration":0.001158526,"stat_sum":{"num_bytes":0,"num_objects":26,"num_object_clones":0,"num_object_copies":52,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":26,"num_read":5948,"num_read_kb":20169,"num_write":1244,"num_write_kb":7750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":262,"num_bytes_recovered":8353711,"num_keys_recovered":12,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.c","version":"168'775","reported_seq":6819,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:52:24.398085+0000","last_change":"2024-09-21T15:52:24.398194+0000","last_active":"2024-09-21T15:52:24.398085+0000","last_peered":"2024-09-21T15:52:24.398085+0000","last_clean":"2024-09-21T15:52:24.398085+0000","last_became_active":"2024-09-21T15:51:39.768321+0000","last_became_peered":"2024-09-21T15:51:39.768321+0000","last_unstale":"2024-09-21T15:52:24.398085+0000","last_undegraded":"2024-09-21T15:52:24.398085+0000","last_fullsized":"2024-09-21T15:52:24.398085+0000","mapping_epoch":162,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'775","last_scrub_stamp":"2024-09-21T15:52:24.398064+0000","last_deep_scrub":"168'775","last_deep_scrub_stamp":"2024-09-21T15:52:24.398064+0000","last_clean_scrub_stamp":"2024-09-21T15:52:24.398064+0000","objects_scrubbed":36,"log_size":775,"log_dups_size":0,"ondisk_log_size":775,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T17:09:22.182836+0000","scrub_duration":0.026829661000000001,"objects_trimmed":0,"snaptrim_duration":0.00074635999999999995,"stat_sum":{"num_bytes":0,"num_objects":36,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":36,"num_read":8849,"num_read_kb":25678,"num_write":868,"num_write_kb":5005,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":246,"num_bytes_recovered":5686754,"num_keys_recovered":23,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"168'929","reported_seq":4364,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:52:06.450770+0000","last_change":"2024-09-21T15:52:06.450878+0000","last_active":"2024-09-21T15:52:06.450770+0000","last_peered":"2024-09-21T15:52:06.450770+0000","last_clean":"2024-09-21T15:52:06.450770+0000","last_became_active":"2024-09-21T15:49:06.689203+0000","last_became_peered":"2024-09-21T15:49:06.689203+0000","last_unstale":"2024-09-21T15:52:06.450770+0000","last_undegraded":"2024-09-21T15:52:06.450770+0000","last_fullsized":"2024-09-21T15:52:06.450770+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'929","last_scrub_stamp":"2024-09-21T15:52:06.450748+0000","last_deep_scrub":"168'929","last_deep_scrub_stamp":"2024-09-21T15:52:06.450748+0000","last_clean_scrub_stamp":"2024-09-21T15:52:06.450748+0000","objects_scrubbed":35,"log_size":929,"log_dups_size":0,"ondisk_log_size":929,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:47:39.763225+0000","scrub_duration":0.029372476000000002,"objects_trimmed":0,"snaptrim_duration":0.00084935700000000002,"stat_sum":{"num_bytes":0,"num_objects":35,"num_object_clones":0,"num_object_copies":70,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":35,"num_read":7321,"num_read_kb":28131,"num_write":1146,"num_write_kb":14809,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"168'597","reported_seq":8188,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:52:15.393182+0000","last_change":"2024-09-21T15:52:15.393289+0000","last_active":"2024-09-21T15:52:15.393182+0000","last_peered":"2024-09-21T15:52:15.393182+0000","last_clean":"2024-09-21T15:52:15.393182+0000","last_became_active":"2024-09-21T15:49:06.690101+0000","last_became_peered":"2024-09-21T15:49:06.690101+0000","last_unstale":"2024-09-21T15:52:15.393182+0000","last_undegraded":"2024-09-21T15:52:15.393182+0000","last_fullsized":"2024-09-21T15:52:15.393182+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'597","last_scrub_stamp":"2024-09-21T15:52:15.393160+0000","last_deep_scrub":"168'597","last_deep_scrub_stamp":"2024-09-21T15:52:15.393160+0000","last_clean_scrub_stamp":"2024-09-21T15:52:15.393160+0000","objects_scrubbed":29,"log_size":597,"log_dups_size":0,"ondisk_log_size":597,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-23T03:21:02.373043+0000","scrub_duration":0.029485266999999999,"objects_trimmed":0,"snaptrim_duration":0.00044571200000000002,"stat_sum":{"num_bytes":0,"num_objects":29,"num_object_clones":0,"num_object_copies":58,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":29,"num_read":8735,"num_read_kb":42983,"num_write":509,"num_write_kb":16273,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.f","version":"168'740","reported_seq":3856,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:52:10.433020+0000","last_change":"2024-09-21T15:52:10.433137+0000","last_active":"2024-09-21T15:52:10.433020+0000","last_peered":"2024-09-21T15:52:10.433020+0000","last_clean":"2024-09-21T15:52:10.433020+0000","last_became_active":"2024-09-21T15:49:06.694250+0000","last_became_peered":"2024-09-21T15:49:06.694250+0000","last_unstale":"2024-09-21T15:52:10.433020+0000","last_undegraded":"2024-09-21T15:52:10.433020+0000","last_fullsized":"2024-09-21T15:52:10.433020+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'740","last_scrub_stamp":"2024-09-21T15:52:10.432994+0000","last_deep_scrub":"168'740","last_deep_scrub_stamp":"2024-09-21T15:52:10.432994+0000","last_clean_scrub_stamp":"2024-09-21T15:52:10.432994+0000","objects_scrubbed":42,"log_size":740,"log_dups_size":0,"ondisk_log_size":740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:03:48.131448+0000","scrub_duration":0.032396746999999997,"objects_trimmed":0,"snaptrim_duration":0.00086632899999999999,"stat_sum":{"num_bytes":0,"num_objects":42,"num_object_clones":0,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":42,"num_read":4945,"num_read_kb":15920,"num_write":889,"num_write_kb":5075,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"168'510","reported_seq":3208,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:52:05.043840+0000","last_change":"2024-09-21T15:52:05.043971+0000","last_active":"2024-09-21T15:52:05.043840+0000","last_peered":"2024-09-21T15:52:05.043840+0000","last_clean":"2024-09-21T15:52:05.043840+0000","last_became_active":"2024-09-21T15:50:32.676147+0000","last_became_peered":"2024-09-21T15:50:32.676147+0000","last_unstale":"2024-09-21T15:52:05.043840+0000","last_undegraded":"2024-09-21T15:52:05.043840+0000","last_fullsized":"2024-09-21T15:52:05.043840+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'510","last_scrub_stamp":"2024-09-21T15:52:05.043817+0000","last_deep_scrub":"168'510","last_deep_scrub_stamp":"2024-09-21T15:52:05.043817+0000","last_clean_scrub_stamp":"2024-09-21T15:52:05.043817+0000","objects_scrubbed":40,"log_size":510,"log_dups_size":0,"ondisk_log_size":510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T19:29:02.342720+0000","scrub_duration":0.027963423000000001,"objects_trimmed":0,"snaptrim_duration":0.00024304599999999999,"stat_sum":{"num_bytes":0,"num_objects":40,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":40,"num_read":3160,"num_read_kb":11849,"num_write":381,"num_write_kb":4471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":1191778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.11","version":"168'1042","reported_seq":4242,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:52:26.415600+0000","last_change":"2024-09-21T15:52:26.415743+0000","last_active":"2024-09-21T15:52:26.415600+0000","last_peered":"2024-09-21T15:52:26.415600+0000","last_clean":"2024-09-21T15:52:26.415600+0000","last_became_active":"2024-09-21T15:49:06.688798+0000","last_became_peered":"2024-09-21T15:49:06.688798+0000","last_unstale":"2024-09-21T15:52:26.415600+0000","last_undegraded":"2024-09-21T15:52:26.415600+0000","last_fullsized":"2024-09-21T15:52:26.415600+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'1042","last_scrub_stamp":"2024-09-21T15:52:26.415575+0000","last_deep_scrub":"168'1042","last_deep_scrub_stamp":"2024-09-21T15:52:26.415575+0000","last_clean_scrub_stamp":"2024-09-21T15:52:26.415575+0000","objects_scrubbed":30,"log_size":1042,"log_dups_size":0,"ondisk_log_size":1042,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T16:03:57.362119+0000","scrub_duration":0.031893248999999999,"objects_trimmed":0,"snaptrim_duration":0.00053058599999999999,"stat_sum":{"num_bytes":0,"num_objects":30,"num_object_clones":0,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":30,"num_read":6391,"num_read_kb":16970,"num_write":1355,"num_write_kb":8408,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"168'417","reported_seq":3059,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:51:51.378061+0000","last_change":"2024-09-21T15:51:51.378167+0000","last_active":"2024-09-21T15:51:51.378061+0000","last_peered":"2024-09-21T15:51:51.378061+0000","last_clean":"2024-09-21T15:51:51.378061+0000","last_became_active":"2024-09-21T15:50:32.671686+0000","last_became_peered":"2024-09-21T15:50:32.671686+0000","last_unstale":"2024-09-21T15:51:51.378061+0000","last_undegraded":"2024-09-21T15:51:51.378061+0000","last_fullsized":"2024-09-21T15:51:51.378061+0000","mapping_epoch":96,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":97,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'417","last_scrub_stamp":"2024-09-21T15:51:51.378041+0000","last_deep_scrub":"168'417","last_deep_scrub_stamp":"2024-09-21T15:51:51.378041+0000","last_clean_scrub_stamp":"2024-09-21T15:51:51.378041+0000","objects_scrubbed":34,"log_size":417,"log_dups_size":0,"ondisk_log_size":417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T18:40:52.256253+0000","scrub_duration":0.023972753999999999,"objects_trimmed":0,"snaptrim_duration":0.0012972649999999999,"stat_sum":{"num_bytes":0,"num_objects":34,"num_object_clones":0,"num_object_copies":68,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":34,"num_read":2455,"num_read_kb":11669,"num_write":401,"num_write_kb":13839,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":3359702,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.13","version":"168'999","reported_seq":7478,"reported_epoch":168,"state":"active+clean","last_fresh":"2024-09-21T15:52:17.364452+0000","last_change":"2024-09-21T15:52:17.364608+0000","last_active":"2024-09-21T15:52:17.364452+0000","last_peered":"2024-09-21T15:52:17.364452+0000","last_clean":"2024-09-21T15:52:17.364452+0000","last_became_active":"2024-09-21T15:49:06.690128+0000","last_became_peered":"2024-09-21T15:49:06.690128+0000","last_unstale":"2024-09-21T15:52:17.364452+0000","last_undegraded":"2024-09-21T15:52:17.364452+0000","last_fullsized":"2024-09-21T15:52:17.364452+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":22,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'999","last_scrub_stamp":"2024-09-21T15:52:17.364410+0000","last_deep_scrub":"168'999","last_deep_scrub_stamp":"2024-09-21T15:52:17.364410+0000","last_clean_scrub_stamp":"2024-09-21T15:52:17.364410+0000","objects_scrubbed":32,"log_size":999,"log_dups_size":0,"ondisk_log_size":999,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T23:01:05.728290+0000","scrub_duration":0.030681287000000002,"objects_trimmed":0,"snaptrim_duration":0.0014652879999999999,"stat_sum":{"num_bytes":0,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":32,"num_read":9327,"num_read_kb":26357,"num_write":1446,"num_write_kb":13580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":19,"num_objects":1039,"num_object_clones":0,"num_object_copies":2078,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1039,"num_whiteouts":1035,"num_read":207419,"num_read_kb":842423,"num_write":23389,"num_write_kb":303239,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3397,"num_bytes_recovered":100270055,"num_keys_recovered":316,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5973,"internal_metadata":0},"log_size":21257,"ondisk_log_size":21257,"up":64,"acting":64,"num_store_stats":8},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":119,"ondisk_log_size":119,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574896,"num_pgs":14,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":77956,"kb_used_data":2556,"kb_used_omap":1,"kb_used_meta":75390,"kb_avail":94293884,"statfs":{"total":96636764160,"available":96556937216,"internally_reserved":0,"allocated":2617344,"data_stored":1563409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1589,"internal_metadata":77199819},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:32 2024","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.48499999999999999}]},{"osd":1,"last update":"Sat Sep 21 15:51:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.48499999999999999,"15min":0.495},"min":{"1min":0.39600000000000002,"5min":0.188,"15min":0.188},"max":{"1min":0.64000000000000001,"5min":1.04,"15min":1.04},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.504,"5min":0.47499999999999998,"15min":0.47699999999999998},"min":{"1min":0.32300000000000001,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.68300000000000005,"5min":0.68300000000000005,"15min":0.68300000000000005},"last":0.58099999999999996}]},{"osd":2,"last update":"Sat Sep 21 15:51:56 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.44700000000000001,"15min":0.44700000000000001},"min":{"1min":0.34100000000000003,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.498,"15min":0.50800000000000001},"min":{"1min":0.311,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.83199999999999996,"5min":0.96499999999999997,"15min":0.96499999999999997},"last":0.65100000000000002}]},{"osd":3,"last update":"Sat Sep 21 15:51:56 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.496,"15min":0.5},"min":{"1min":0.32900000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.86199999999999999,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.626},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.47299999999999998,"15min":0.47799999999999998},"min":{"1min":0.36399999999999999,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.74399999999999999,"5min":0.78600000000000003,"15min":0.78600000000000003},"last":0.60599999999999998}]},{"osd":4,"last update":"Sat Sep 21 15:51:56 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.41599999999999998,"15min":0.42299999999999999},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.59499999999999997,"5min":0.97599999999999998,"15min":0.97599999999999998},"last":0.435},{"interface":"front","average":{"1min":0.44,"5min":0.39900000000000002,"15min":0.39500000000000002},"min":{"1min":0.28499999999999998,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.53400000000000003}]},{"osd":5,"last update":"Sat Sep 21 15:51:56 2024","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.435,"15min":0.44900000000000001},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.75700000000000001,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.69099999999999995},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.45800000000000002,"15min":0.45400000000000001},"min":{"1min":0.30099999999999999,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.52400000000000002}]},{"osd":6,"last update":"Sat Sep 21 15:51:56 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.495,"15min":0.501},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.46700000000000003,"15min":0.45900000000000002},"min":{"1min":0.34399999999999997,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.505}]}]},{"osd":6,"up_from":13,"seq":55834574899,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41280,"kb_used_data":1976,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94330560,"statfs":{"total":96636764160,"available":96594493440,"internally_reserved":0,"allocated":2023424,"data_stored":973041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":40237511},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:31 2024","interfaces":[{"interface":"back","average":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.19900000000000001,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.442,"5min":0.442,"15min":0.442},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.44800000000000001}]},{"osd":1,"last update":"Sat Sep 21 15:51:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.46400000000000002,"15min":0.46400000000000002},"min":{"1min":0.28199999999999997,"5min":0.255,"15min":0.255},"max":{"1min":1.3120000000000001,"5min":1.3120000000000001,"15min":1.3120000000000001},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.47199999999999998,"15min":0.48099999999999998},"min":{"1min":0.34000000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":1.5229999999999999,"5min":1.5229999999999999,"15min":1.5229999999999999},"last":0.5}]},{"osd":2,"last update":"Sat Sep 21 15:51:53 2024","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.46600000000000003,"15min":0.47499999999999998},"min":{"1min":0.25900000000000001,"5min":0.25900000000000001,"15min":0.25900000000000001},"max":{"1min":0.64900000000000002,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.46999999999999997,"15min":0.46600000000000003},"min":{"1min":0.29799999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.411,"5min":1.411,"15min":1.411},"last":0.621}]},{"osd":3,"last update":"Sat Sep 21 15:51:53 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.50700000000000001,"15min":0.52100000000000002},"min":{"1min":0.31,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.67700000000000005,"5min":0.84399999999999997,"15min":0.84399999999999997},"last":1.0469999999999999},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.52900000000000003,"15min":0.53700000000000003},"min":{"1min":0.33800000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.68100000000000005,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":1.0940000000000001}]},{"osd":4,"last update":"Sat Sep 21 15:51:53 2024","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.40799999999999997,"15min":0.41399999999999998},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.67200000000000004,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.46100000000000002,"15min":0.47999999999999998},"min":{"1min":0.24199999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.68899999999999995,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.39300000000000002}]},{"osd":5,"last update":"Sat Sep 21 15:51:53 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.44500000000000001,"15min":0.45500000000000002},"min":{"1min":0.34300000000000003,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.64200000000000002,"5min":0.755,"15min":0.755},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.44600000000000001,"15min":0.46200000000000002},"min":{"1min":0.28100000000000003,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.626,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.52000000000000002}]},{"osd":7,"last update":"Sat Sep 21 15:51:53 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.45300000000000001,"15min":0.45800000000000002},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.48799999999999999,"15min":0.50800000000000001},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.67600000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.46200000000000002}]}]},{"osd":5,"up_from":13,"seq":55834574898,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":55236,"kb_used_data":1980,"kb_used_omap":1,"kb_used_meta":53246,"kb_avail":94316604,"statfs":{"total":96636764160,"available":96580202496,"internally_reserved":0,"allocated":2027520,"data_stored":973060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":54524358},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.55600000000000005,"5min":0.55600000000000005,"15min":0.55600000000000005},"last":0.505}]},{"osd":1,"last update":"Sat Sep 21 15:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.44600000000000001,"15min":0.44500000000000001},"min":{"1min":0.218,"5min":0.218,"15min":0.218},"max":{"1min":0.71599999999999997,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.48799999999999999,"15min":0.51000000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.60899999999999999,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.55000000000000004}]},{"osd":2,"last update":"Sat Sep 21 15:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.495,"15min":0.50700000000000001},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73199999999999998,"5min":1.046,"15min":1.046},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.47399999999999998,"15min":0.47499999999999998},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.72099999999999997,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.67200000000000004}]},{"osd":3,"last update":"Sat Sep 21 15:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.47799999999999998,"15min":0.48799999999999999},"min":{"1min":0.29999999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.70299999999999996,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.52200000000000002,"15min":0.54700000000000004},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.62,"5min":1.6140000000000001,"15min":1.6140000000000001},"last":0.58599999999999997}]},{"osd":4,"last update":"Sat Sep 21 15:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.60299999999999998,"15min":0.69099999999999995},"min":{"1min":0.28599999999999998,"5min":0.253,"15min":0.253},"max":{"1min":0.77700000000000002,"5min":5.6399999999999997,"15min":5.6399999999999997},"last":0.625},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.44900000000000001,"15min":0.47099999999999997},"min":{"1min":0.26800000000000002,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.65700000000000003}]},{"osd":6,"last update":"Sat Sep 21 15:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.44600000000000001,"15min":0.45600000000000002},"min":{"1min":0.29099999999999998,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.47999999999999998,"15min":0.498},"min":{"1min":0.33500000000000002,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.74399999999999999,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.60599999999999998}]},{"osd":7,"last update":"Sat Sep 21 15:51:59 2024","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.47499999999999998,"15min":0.48699999999999999},"min":{"1min":0.35199999999999998,"5min":0.312,"15min":0.312},"max":{"1min":0.70599999999999996,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.503,"5min":0.499,"15min":0.51900000000000002},"min":{"1min":0.28999999999999998,"5min":0.248,"15min":0.248},"max":{"1min":0.66400000000000003,"5min":0.747,"15min":0.747},"last":0.69699999999999995}]}]},{"osd":4,"up_from":13,"seq":55834574897,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":69184,"kb_used_data":1976,"kb_used_omap":1,"kb_used_meta":67198,"kb_avail":94302656,"statfs":{"total":96636764160,"available":96565919744,"internally_reserved":0,"allocated":2023424,"data_stored":971778,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":68811206},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.76800000000000002},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.61199999999999999}]},{"osd":1,"last update":"Sat Sep 21 15:52:01 2024","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.45000000000000001,"15min":0.42399999999999999},"min":{"1min":0.28999999999999998,"5min":0.217,"15min":0.217},"max":{"1min":0.65000000000000002,"5min":1.8839999999999999,"15min":1.8839999999999999},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.46999999999999997,"15min":0.46400000000000002},"min":{"1min":0.34999999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.67900000000000005,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.47599999999999998}]},{"osd":2,"last update":"Sat Sep 21 15:52:01 2024","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.45900000000000002,"15min":0.44400000000000001},"min":{"1min":0.31,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.69899999999999995,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.443,"15min":0.437},"min":{"1min":0.30499999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.58599999999999997,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.58799999999999997}]},{"osd":3,"last update":"Sat Sep 21 15:52:01 2024","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.46500000000000002,"15min":0.45500000000000002},"min":{"1min":0.36299999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.46999999999999997,"15min":0.47099999999999997},"min":{"1min":0.33300000000000002,"5min":0.32000000000000001,"15min":0.32000000000000001},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.68000000000000005}]},{"osd":5,"last update":"Sat Sep 21 15:52:01 2024","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.45900000000000002,"15min":0.47099999999999997},"min":{"1min":0.247,"5min":0.20899999999999999,"15min":0.20899999999999999},"max":{"1min":0.67200000000000004,"5min":2.4489999999999998,"15min":2.4489999999999998},"last":0.79000000000000004},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.434,"15min":0.42799999999999999},"min":{"1min":0.307,"5min":0.222,"15min":0.222},"max":{"1min":0.67200000000000004,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.63800000000000001}]},{"osd":6,"last update":"Sat Sep 21 15:52:01 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.42899999999999999,"15min":0.41899999999999998},"min":{"1min":0.27900000000000003,"5min":0.245,"15min":0.245},"max":{"1min":0.752,"5min":0.752,"15min":0.752},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.54800000000000004,"15min":0.54200000000000004},"min":{"1min":0.31900000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.69799999999999995,"5min":4.1340000000000003,"15min":4.1340000000000003},"last":0.81299999999999994}]},{"osd":7,"last update":"Sat Sep 21 15:52:01 2024","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.47199999999999998,"15min":0.44},"min":{"1min":0.313,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.70899999999999996,"5min":2.7599999999999998,"15min":2.7599999999999998},"last":0.71699999999999997},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.52900000000000003,"15min":0.52300000000000002},"min":{"1min":0.29399999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.77000000000000002,"5min":2.016,"15min":2.016},"last":0.83199999999999996}]}]},{"osd":3,"up_from":13,"seq":55834574898,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":59328,"kb_used_data":1976,"kb_used_omap":1,"kb_used_meta":57342,"kb_avail":94312512,"statfs":{"total":96636764160,"available":96576012288,"internally_reserved":0,"allocated":2023424,"data_stored":973041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":58718664},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.31,"5min":0.31,"15min":0.31},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"last":0.58099999999999996}]},{"osd":1,"last update":"Sat Sep 21 15:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.29699999999999999,"5min":0.32600000000000001,"15min":0.34000000000000002},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.38900000000000001,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.34000000000000002,"5min":0.35899999999999999,"15min":0.375},"min":{"1min":0.26600000000000001,"5min":0.192,"15min":0.192},"max":{"1min":0.42599999999999999,"5min":0.69899999999999995,"15min":0.69899999999999995},"last":0.495}]},{"osd":2,"last update":"Sat Sep 21 15:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.371,"15min":0.38900000000000001},"min":{"1min":0.253,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.40600000000000003,"5min":0.86599999999999999,"15min":0.86599999999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.39700000000000002,"15min":0.41699999999999998},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.57999999999999996,"5min":0.755,"15min":0.755},"last":0.55100000000000005}]},{"osd":4,"last update":"Sat Sep 21 15:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.48199999999999998,"15min":0.49299999999999999},"min":{"1min":0.29699999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.505,"15min":0.51300000000000001},"min":{"1min":0.379,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.74099999999999999,"5min":0.74099999999999999,"15min":0.74099999999999999},"last":0.69499999999999995}]},{"osd":5,"last update":"Sat Sep 21 15:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.501,"15min":0.52200000000000002},"min":{"1min":0.28399999999999997,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.70799999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.55100000000000005},"min":{"1min":0.36599999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.73099999999999998,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.63900000000000001}]},{"osd":6,"last update":"Sat Sep 21 15:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.51200000000000001,"15min":0.53700000000000003},"min":{"1min":0.33400000000000002,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.623,"5min":0.72599999999999998,"15min":0.72599999999999998},"last":0.626},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.52100000000000002,"15min":0.54300000000000004},"min":{"1min":0.36199999999999999,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.75700000000000001,"5min":0.78300000000000003,"15min":0.78300000000000003},"last":0.52700000000000002}]},{"osd":7,"last update":"Sat Sep 21 15:51:52 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.53700000000000003,"15min":0.55900000000000005},"min":{"1min":0.39100000000000001,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.69999999999999996,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.53400000000000003,"15min":0.55800000000000005},"min":{"1min":0.36499999999999999,"5min":0.314,"15min":0.314},"max":{"1min":0.77600000000000002,"5min":0.86699999999999999,"15min":0.86699999999999999},"last":0.65300000000000002}]}]},{"osd":2,"up_from":13,"seq":55834574898,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":37184,"kb_used_data":1976,"kb_used_omap":1,"kb_used_meta":35198,"kb_avail":94334656,"statfs":{"total":96636764160,"available":96598687744,"internally_reserved":0,"allocated":2023424,"data_stored":973041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":36043206},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:52:29 2024","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.30199999999999999,"15min":0.29799999999999999},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.33700000000000002,"5min":0.35299999999999998,"15min":0.35499999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.51100000000000001,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.45500000000000002}]},{"osd":1,"last update":"Sat Sep 21 15:51:57 2024","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.377,"15min":0.41099999999999998},"min":{"1min":0.218,"5min":0.17199999999999999,"15min":0.17199999999999999},"max":{"1min":0.49199999999999999,"5min":1.621,"15min":1.621},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.313,"5min":0.35099999999999998,"15min":0.36899999999999999},"min":{"1min":0.23799999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.48299999999999998,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.34100000000000003}]},{"osd":3,"last update":"Sat Sep 21 15:51:57 2024","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.77800000000000002,"15min":0.92800000000000005},"min":{"1min":0.25700000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.54900000000000004,"5min":11.56,"15min":11.56},"last":0.441},{"interface":"front","average":{"1min":0.371,"5min":0.39200000000000002,"15min":0.40999999999999998},"min":{"1min":0.30399999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.51900000000000002,"5min":0.80900000000000005,"15min":0.80900000000000005},"last":0.39300000000000002}]},{"osd":4,"last update":"Sat Sep 21 15:51:57 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.56699999999999995,"15min":0.59799999999999998},"min":{"1min":0.39800000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.76100000000000001,"5min":1.653,"15min":1.653},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.58399999999999996,"15min":0.622},"min":{"1min":0.39900000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.749,"5min":2.4660000000000002,"15min":2.4660000000000002},"last":0.67800000000000005}]},{"osd":5,"last update":"Sat Sep 21 15:51:57 2024","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.496,"15min":0.50900000000000001},"min":{"1min":0.372,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.72999999999999998,"5min":0.84699999999999998,"15min":0.84699999999999998},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.51700000000000002,"15min":0.53600000000000003},"min":{"1min":0.36499999999999999,"5min":0.309,"15min":0.309},"max":{"1min":0.70599999999999996,"5min":0.92000000000000004,"15min":0.92000000000000004},"last":0.621}]},{"osd":6,"last update":"Sat Sep 21 15:51:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.53800000000000003,"15min":0.55900000000000005},"min":{"1min":0.34899999999999998,"5min":0.316,"15min":0.316},"max":{"1min":0.79700000000000004,"5min":1.0109999999999999,"15min":1.0109999999999999},"last":0.747},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53400000000000003,"15min":0.55400000000000005},"min":{"1min":0.35099999999999998,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.76700000000000002,"5min":0.873,"15min":0.873},"last":0.65200000000000002}]},{"osd":7,"last update":"Sat Sep 21 15:51:57 2024","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.53700000000000003,"15min":0.55700000000000005},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.76900000000000002,"5min":0.876,"15min":0.876},"last":0.70599999999999996},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57599999999999996,"15min":0.59599999999999997},"min":{"1min":0.38200000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.79600000000000004,"5min":1.089,"15min":1.089},"last":0.69099999999999995}]}]},{"osd":1,"up_from":13,"seq":55834574897,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":41288,"kb_used_data":1984,"kb_used_omap":1,"kb_used_meta":39294,"kb_avail":94330552,"statfs":{"total":96636764160,"available":96594485248,"internally_reserved":0,"allocated":2031616,"data_stored":973079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":40237510},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Sep 21 15:51:30 2024","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.312,"15min":0.312},"min":{"1min":0.20200000000000001,"5min":0.20200000000000001,"15min":0.20200000000000001},"max":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"min":{"1min":0.20599999999999999,"5min":0.20599999999999999,"15min":0.20599999999999999},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.34300000000000003}]},{"osd":2,"last update":"Sat Sep 21 15:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.33100000000000002,"15min":0.34000000000000002},"min":{"1min":0.218,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.66500000000000004,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.46100000000000002,"15min":0.51900000000000002},"min":{"1min":0.24399999999999999,"5min":0.17799999999999999,"15min":0.17799999999999999},"max":{"1min":0.49399999999999999,"5min":5.0670000000000002,"15min":5.0670000000000002},"last":0.32000000000000001}]},{"osd":3,"last update":"Sat Sep 21 15:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.78000000000000003,"15min":0.97299999999999998},"min":{"1min":0.27200000000000002,"5min":0.218,"15min":0.218},"max":{"1min":0.625,"5min":16.032,"15min":16.032},"last":0.375},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.35599999999999998,"15min":0.35899999999999999},"min":{"1min":0.252,"5min":0.186,"15min":0.186},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.39000000000000001}]},{"osd":4,"last update":"Sat Sep 21 15:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.46999999999999997,"15min":0.47399999999999998},"min":{"1min":0.36299999999999999,"5min":0.251,"15min":0.251},"max":{"1min":0.61799999999999999,"5min":0.76900000000000002,"15min":0.76900000000000002},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53100000000000003,"15min":0.54900000000000004},"min":{"1min":0.375,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.752,"5min":2.2269999999999999,"15min":2.2269999999999999},"last":0.72399999999999998}]},{"osd":5,"last update":"Sat Sep 21 15:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.53800000000000003,"15min":0.56799999999999995},"min":{"1min":0.34799999999999998,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.72699999999999998,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.503},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.623,"15min":0.68300000000000005},"min":{"1min":0.35299999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.68200000000000005,"5min":4.6989999999999998,"15min":4.6989999999999998},"last":0.68100000000000005}]},{"osd":6,"last update":"Sat Sep 21 15:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.52800000000000002,"15min":0.53800000000000003},"min":{"1min":0.41199999999999998,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.68999999999999995,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.58299999999999996,"15min":0.61499999999999999},"min":{"1min":0.378,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.70099999999999996,"5min":2.262,"15min":2.262},"last":0.69699999999999995}]},{"osd":7,"last update":"Sat Sep 21 15:51:50 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.63200000000000001,"15min":0.69199999999999995},"min":{"1min":0.36499999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.71099999999999997,"5min":4.742,"15min":4.742},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.51500000000000001,"15min":0.51900000000000002},"min":{"1min":0.38,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.65500000000000003,"5min":0.89500000000000002,"15min":0.89500000000000002},"last":0.65900000000000003}]}]},{"osd":0,"up_from":93,"seq":399431958555,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":39172,"kb_used_data":2556,"kb_used_omap":13,"kb_used_meta":36594,"kb_avail":94332668,"statfs":{"total":96636764160,"available":96596652032,"internally_reserved":0,"allocated":2617344,"data_stored":1563409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":13968,"internal_metadata":37472624},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Sep 21 15:52:31 2024","interfaces":[{"interface":"back","average":{"1min":0.41599999999999998,"5min":0.32700000000000001,"15min":0.312},"min":{"1min":0.23000000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.4370000000000001,"5min":1.4370000000000001,"15min":1.4370000000000001},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.44,"5min":0.34100000000000003,"15min":0.32500000000000001},"min":{"1min":0.24199999999999999,"5min":0.214,"15min":0.214},"max":{"1min":1.351,"5min":1.351,"15min":1.351},"last":0.45200000000000001}]},{"osd":2,"last update":"Sat Sep 21 15:52:31 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.36099999999999999,"15min":0.34499999999999997},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.532,"5min":1.532,"15min":1.532},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.34200000000000003,"15min":0.32800000000000001},"min":{"1min":0.252,"5min":0.249,"15min":0.249},"max":{"1min":1.3819999999999999,"5min":1.3819999999999999,"15min":1.3819999999999999},"last":0.38300000000000001}]},{"osd":3,"last update":"Sat Sep 21 15:52:31 2024","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.35199999999999998,"15min":0.33600000000000002},"min":{"1min":0.249,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":1.4059999999999999,"5min":1.4059999999999999,"15min":1.4059999999999999},"last":0.437},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.378,"15min":0.36099999999999999},"min":{"1min":0.33600000000000002,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":1.5569999999999999,"5min":1.5569999999999999,"15min":1.5569999999999999},"last":0.42199999999999999}]},{"osd":4,"last update":"Sat Sep 21 15:52:31 2024","interfaces":[{"interface":"back","average":{"1min":0.63,"5min":0.51900000000000002,"15min":0.501},"min":{"1min":0.44800000000000001,"5min":0.307,"15min":0.307},"max":{"1min":1.51,"5min":1.51,"15min":1.51},"last":0.55200000000000005},{"interface":"front","average":{"1min":0.64200000000000002,"5min":0.52600000000000002,"15min":0.50700000000000001},"min":{"1min":0.44500000000000001,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":1.3240000000000001,"5min":1.3240000000000001,"15min":1.3240000000000001},"last":0.61399999999999999}]},{"osd":5,"last update":"Sat Sep 21 15:52:31 2024","interfaces":[{"interface":"back","average":{"1min":0.61399999999999999,"5min":0.52200000000000002,"15min":0.50700000000000001},"min":{"1min":0.374,"5min":0.373,"15min":0.373},"max":{"1min":1.2929999999999999,"5min":1.2929999999999999,"15min":1.2929999999999999},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.53500000000000003,"15min":0.52100000000000002},"min":{"1min":0.43099999999999999,"5min":0.379,"15min":0.379},"max":{"1min":0.93899999999999995,"5min":0.93899999999999995,"15min":0.93899999999999995},"last":0.65800000000000003}]},{"osd":6,"last update":"Sat Sep 21 15:52:31 2024","interfaces":[{"interface":"back","average":{"1min":0.67200000000000004,"5min":0.53300000000000003,"15min":0.51000000000000001},"min":{"1min":0.432,"5min":0.34200000000000003,"15min":0.34200000000000003},"max":{"1min":1.488,"5min":1.488,"15min":1.488},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.52800000000000002,"15min":0.51000000000000001},"min":{"1min":0.437,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":1.262,"5min":1.262,"15min":1.262},"last":0.53800000000000003}]},{"osd":7,"last update":"Sat Sep 21 15:52:31 2024","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.53600000000000003,"15min":0.52600000000000002},"min":{"1min":0.38700000000000001,"5min":0.374,"15min":0.374},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":0.82799999999999996},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.55600000000000005,"15min":0.53500000000000003},"min":{"1min":0.42399999999999999,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.464,"5min":1.464,"15min":1.464},"last":0.67800000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":5973,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T15:52:35.182 DEBUG:teuthology.orchestra.run.smithi089:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-21T15:52:35.466 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-21T15:52:35.466 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-21T15:52:35.467 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-21T15:52:35.467 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T15:52:35.500 INFO:tasks.ceph.osd.0:Stopped 2024-09-21T15:52:35.500 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-21T15:52:35.500 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T15:52:35.568 INFO:tasks.ceph.osd.1:Stopped 2024-09-21T15:52:35.568 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-21T15:52:35.568 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T15:52:35.635 INFO:tasks.ceph.osd.2:Stopped 2024-09-21T15:52:35.635 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-21T15:52:35.635 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T15:52:35.703 INFO:tasks.ceph.osd.3:Stopped 2024-09-21T15:52:35.703 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-21T15:52:35.703 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T15:52:35.745 INFO:tasks.ceph.osd.4:Stopped 2024-09-21T15:52:35.745 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-21T15:52:35.745 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T15:52:35.789 INFO:tasks.ceph.osd.5:Stopped 2024-09-21T15:52:35.790 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-09-21T15:52:35.790 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T15:52:35.837 INFO:tasks.ceph.osd.6:Stopped 2024-09-21T15:52:35.837 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-09-21T15:52:35.837 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T15:52:35.886 INFO:tasks.ceph.osd.7:Stopped 2024-09-21T15:52:35.886 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-21T15:52:35.886 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-09-21T15:52:35.886 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T15:52:35.912 INFO:tasks.ceph.mgr.y:Stopped 2024-09-21T15:52:35.912 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-21T15:52:35.913 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T15:52:35.939 INFO:tasks.ceph.mgr.x:Stopped 2024-09-21T15:52:35.939 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-21T15:52:35.939 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-21T15:52:35.939 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T15:52:35.963 INFO:tasks.ceph.mon.a:Stopped 2024-09-21T15:52:35.963 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-21T15:52:35.963 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T15:52:36.014 INFO:tasks.ceph.mon.c:Stopped 2024-09-21T15:52:36.014 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-21T15:52:36.015 INFO:teuthology.orchestra.run:waiting for 300 2024-09-21T15:52:36.027 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2024-09-21T15:52:36.027 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a is failed for ~0s 2024-09-21T15:52:36.027 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2024-09-21T15:52:36.028 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2024-09-21T15:52:36.028 INFO:tasks.ceph.mon.b:Stopped 2024-09-21T15:52:36.028 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-21T15:52:36.028 DEBUG:teuthology.orchestra.run.smithi089:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | head -n 1 2024-09-21T15:52:36.083 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi089.front.sepia.ceph.com 2024-09-21T15:52:36.083 DEBUG:teuthology.orchestra.run.smithi089:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-21T15:52:36.281 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi089.front.sepia.ceph.com 2024-09-21T15:52:36.282 DEBUG:teuthology.orchestra.run.smithi089:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-21T15:52:36.345 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi089.front.sepia.ceph.com 2024-09-21T15:52:36.346 DEBUG:teuthology.orchestra.run.smithi089:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-21T15:52:36.411 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi089.front.sepia.ceph.com 2024-09-21T15:52:36.411 DEBUG:teuthology.orchestra.run.smithi089:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-21T15:52:36.479 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi100.front.sepia.ceph.com 2024-09-21T15:52:36.479 DEBUG:teuthology.orchestra.run.smithi100:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-21T15:52:36.605 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi100.front.sepia.ceph.com 2024-09-21T15:52:36.606 DEBUG:teuthology.orchestra.run.smithi100:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-21T15:52:36.710 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi100.front.sepia.ceph.com 2024-09-21T15:52:36.710 DEBUG:teuthology.orchestra.run.smithi100:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-09-21T15:52:36.765 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi100.front.sepia.ceph.com 2024-09-21T15:52:36.765 DEBUG:teuthology.orchestra.run.smithi100:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-09-21T15:52:36.826 INFO:tasks.ceph:Archiving mon data... 2024-09-21T15:52:36.827 DEBUG:teuthology.misc:Transferring archived files from smithi089:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi/7909998/data/mon.a.tgz 2024-09-21T15:52:36.827 DEBUG:teuthology.orchestra.run.smithi089:> mktemp 2024-09-21T15:52:36.843 INFO:teuthology.orchestra.run.smithi089.stdout:/tmp/tmp.xP3rLW4vE4 2024-09-21T15:52:36.843 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.xP3rLW4vE4 2024-09-21T15:52:36.996 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 0666 /tmp/tmp.xP3rLW4vE4 2024-09-21T15:52:37.078 DEBUG:teuthology.orchestra.remote:smithi089:/tmp/tmp.xP3rLW4vE4 is 391KB 2024-09-21T15:52:37.174 DEBUG:teuthology.orchestra.run.smithi089:> rm -fr /tmp/tmp.xP3rLW4vE4 2024-09-21T15:52:37.189 DEBUG:teuthology.misc:Transferring archived files from smithi089:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi/7909998/data/mon.c.tgz 2024-09-21T15:52:37.189 DEBUG:teuthology.orchestra.run.smithi089:> mktemp 2024-09-21T15:52:37.242 INFO:teuthology.orchestra.run.smithi089.stdout:/tmp/tmp.2JVWpjWQH5 2024-09-21T15:52:37.243 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.2JVWpjWQH5 2024-09-21T15:52:37.413 DEBUG:teuthology.orchestra.run.smithi089:> sudo chmod 0666 /tmp/tmp.2JVWpjWQH5 2024-09-21T15:52:37.496 DEBUG:teuthology.orchestra.remote:smithi089:/tmp/tmp.2JVWpjWQH5 is 524KB 2024-09-21T15:52:37.592 DEBUG:teuthology.orchestra.run.smithi089:> rm -fr /tmp/tmp.2JVWpjWQH5 2024-09-21T15:52:37.607 DEBUG:teuthology.misc:Transferring archived files from smithi100:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi/7909998/data/mon.b.tgz 2024-09-21T15:52:37.607 DEBUG:teuthology.orchestra.run.smithi100:> mktemp 2024-09-21T15:52:37.626 INFO:teuthology.orchestra.run.smithi100.stdout:/tmp/tmp.6fGvzGjSRh 2024-09-21T15:52:37.626 DEBUG:teuthology.orchestra.run.smithi100:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.6fGvzGjSRh 2024-09-21T15:52:37.816 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 0666 /tmp/tmp.6fGvzGjSRh 2024-09-21T15:52:37.899 DEBUG:teuthology.orchestra.remote:smithi100:/tmp/tmp.6fGvzGjSRh is 524KB 2024-09-21T15:52:37.996 DEBUG:teuthology.orchestra.run.smithi100:> rm -fr /tmp/tmp.6fGvzGjSRh 2024-09-21T15:52:38.011 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-21T15:52:38.011 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-21T15:52:38.013 DEBUG:teuthology.orchestra.run.smithi100:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-21T15:52:38.076 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2024-09-21T15:52:38.076 DEBUG:teuthology.orchestra.run.smithi089:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-21T15:52:38.079 DEBUG:teuthology.orchestra.run.smithi100:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2024-09-21T15:52:38.104 INFO:teuthology.orchestra.run.smithi089.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-21T15:52:38.143 INFO:teuthology.orchestra.run.smithi100.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2024-09-21T15:52:38.147 INFO:tasks.ceph:Archiving crash dumps... 2024-09-21T15:52:38.148 DEBUG:teuthology.misc:Transferring archived files from smithi089:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi/7909998/remote/smithi089/crash 2024-09-21T15:52:38.149 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-21T15:52:38.174 DEBUG:teuthology.misc:Transferring archived files from smithi100:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi/7909998/remote/smithi100/crash 2024-09-21T15:52:38.175 DEBUG:teuthology.orchestra.run.smithi100:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-21T15:52:38.211 INFO:tasks.ceph:Compressing logs... 2024-09-21T15:52:38.211 DEBUG:teuthology.orchestra.run.smithi089:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T15:52:38.218 DEBUG:teuthology.orchestra.run.smithi100:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T15:52:41.028 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2024-09-21T15:52:41.029 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.a has been restored 2024-09-21T15:52:41.029 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2024-09-21T15:52:41.029 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2024-09-21T15:52:50.998 INFO:tasks.ceph:Archiving logs... 2024-09-21T15:52:50.999 DEBUG:teuthology.misc:Transferring archived files from smithi089:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi/7909998/remote/smithi089/log 2024-09-21T15:52:51.000 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-21T15:52:52.157 DEBUG:teuthology.misc:Transferring archived files from smithi100:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi/7909998/remote/smithi100/log 2024-09-21T15:52:52.158 DEBUG:teuthology.orchestra.run.smithi100:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-21T15:52:53.333 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-21T15:52:53.342 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-21T15:52:53.342 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-21T15:52:53.345 DEBUG:teuthology.orchestra.run.smithi100:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-21T15:52:53.427 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-21T15:52:53.435 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-21T15:52:53.435 DEBUG:teuthology.orchestra.run.smithi089:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T15:52:53.438 DEBUG:teuthology.orchestra.run.smithi100:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T15:52:53.452 INFO:teuthology.orchestra.run.smithi089.stderr:bash: line 1: ntpq: command not found 2024-09-21T15:52:53.453 INFO:teuthology.orchestra.run.smithi100.stderr:bash: line 1: ntpq: command not found 2024-09-21T15:52:53.456 INFO:teuthology.orchestra.run.smithi089.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T15:52:53.456 INFO:teuthology.orchestra.run.smithi089.stdout:=============================================================================== 2024-09-21T15:52:53.456 INFO:teuthology.orchestra.run.smithi089.stdout:^* hv01.front.sepia.ceph.com 3 6 377 23 -106us[ -139us] +/- 48ms 2024-09-21T15:52:53.456 INFO:teuthology.orchestra.run.smithi089.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 24 -804us[ -836us] +/- 89ms 2024-09-21T15:52:53.456 INFO:teuthology.orchestra.run.smithi089.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 21 +704us[ +704us] +/- 62ms 2024-09-21T15:52:53.456 INFO:teuthology.orchestra.run.smithi089.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T15:52:53.457 INFO:teuthology.orchestra.run.smithi100.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T15:52:53.457 INFO:teuthology.orchestra.run.smithi100.stdout:=============================================================================== 2024-09-21T15:52:53.458 INFO:teuthology.orchestra.run.smithi100.stdout:^* hv01.front.sepia.ceph.com 3 6 377 22 -93us[ -94us] +/- 48ms 2024-09-21T15:52:53.458 INFO:teuthology.orchestra.run.smithi100.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 23 -805us[ -805us] +/- 89ms 2024-09-21T15:52:53.458 INFO:teuthology.orchestra.run.smithi100.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 23 +699us[ +698us] +/- 62ms 2024-09-21T15:52:53.458 INFO:teuthology.orchestra.run.smithi100.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T15:52:53.459 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-21T15:52:53.467 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-21T15:52:53.467 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-21T15:52:53.478 DEBUG:teuthology.orchestra.run.smithi089:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-21T15:52:53.579 DEBUG:teuthology.orchestra.run.smithi100:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-21T15:52:53.654 DEBUG:teuthology.orchestra.run.smithi089:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-21T15:52:53.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T15:52:53.684 DEBUG:teuthology.orchestra.run.smithi100:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-21T15:52:53.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T15:52:53.713 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-21T15:52:53.734 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-21T15:52:53.743 INFO:teuthology.task.internal:Duration was 645.970340 seconds 2024-09-21T15:52:53.743 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-21T15:52:53.752 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-21T15:52:53.753 DEBUG:teuthology.orchestra.run.smithi089:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-21T15:52:53.755 DEBUG:teuthology.orchestra.run.smithi100:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-21T15:52:53.791 INFO:teuthology.orchestra.run.smithi089.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T15:52:53.794 INFO:teuthology.orchestra.run.smithi100.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T15:52:54.073 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-21T15:52:54.073 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi089.front.sepia.ceph.com 2024-09-21T15:52:54.073 DEBUG:teuthology.orchestra.run.smithi089:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-21T15:52:54.093 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi100.front.sepia.ceph.com 2024-09-21T15:52:54.093 DEBUG:teuthology.orchestra.run.smithi100:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-21T15:52:54.135 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-21T15:52:54.136 DEBUG:teuthology.orchestra.run.smithi089:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T15:52:54.138 DEBUG:teuthology.orchestra.run.smithi100:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T15:52:54.201 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-21T15:52:54.201 DEBUG:teuthology.orchestra.run.smithi089:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-21T15:52:54.204 DEBUG:teuthology.orchestra.run.smithi100:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-21T15:52:54.358 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-21T15:52:54.418 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-21T15:52:54.418 DEBUG:teuthology.orchestra.run.smithi089:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-21T15:52:54.443 DEBUG:teuthology.orchestra.run.smithi100:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-21T15:52:54.468 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-21T15:52:54.479 DEBUG:teuthology.orchestra.run.smithi089:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-21T15:52:54.485 DEBUG:teuthology.orchestra.run.smithi100:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-21T15:52:54.506 INFO:teuthology.orchestra.run.smithi089.stdout:kernel.core_pattern = core 2024-09-21T15:52:54.531 INFO:teuthology.orchestra.run.smithi100.stdout:kernel.core_pattern = core 2024-09-21T15:52:54.559 DEBUG:teuthology.orchestra.run.smithi089:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-21T15:52:54.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T15:52:54.581 DEBUG:teuthology.orchestra.run.smithi100:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-21T15:52:54.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T15:52:54.613 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-21T15:52:54.623 INFO:teuthology.task.internal:Transferring archived files... 2024-09-21T15:52:54.624 DEBUG:teuthology.misc:Transferring archived files from smithi089:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi/7909998/remote/smithi089 2024-09-21T15:52:54.624 DEBUG:teuthology.orchestra.run.smithi089:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-21T15:52:54.664 DEBUG:teuthology.misc:Transferring archived files from smithi100:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-17_22:16:03-rbd-reef-distro-default-smithi/7909998/remote/smithi100 2024-09-21T15:52:54.664 DEBUG:teuthology.orchestra.run.smithi100:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-21T15:52:54.696 INFO:teuthology.task.internal:Removing archive directory... 2024-09-21T15:52:54.697 DEBUG:teuthology.orchestra.run.smithi089:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-21T15:52:54.699 DEBUG:teuthology.orchestra.run.smithi100:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-21T15:52:54.746 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-21T15:52:54.757 INFO:teuthology.task.internal:Not uploading archives. 2024-09-21T15:52:54.757 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-21T15:52:54.812 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-21T15:52:54.812 DEBUG:teuthology.orchestra.run.smithi089:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-21T15:52:54.814 DEBUG:teuthology.orchestra.run.smithi100:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-21T15:52:54.827 INFO:teuthology.orchestra.run.smithi089.stdout: 265613 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 21 15:52 /home/ubuntu/cephtest 2024-09-21T15:52:54.829 INFO:teuthology.orchestra.run.smithi100.stdout: 266037 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 21 15:52 /home/ubuntu/cephtest 2024-09-21T15:52:54.830 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-21T15:52:54.841 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-21T15:52:54.863 INFO:teuthology.run:Summary data: description: rbd/thrash/{base/install clusters/{fixed-2 openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-stupid supported-random-distro$/{centos_latest} thrashers/default thrashosds-health workloads/rbd_fsx_rate_limit} duration: 645.9703404903412 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-21T15:52:54.863 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-21T15:52:54.937 INFO:teuthology.run:pass